s8n 117fa33048 doc 25: english leak deep-dive (Abspielen post-lockdown)

2026-05-08 22:09:59 +01:00

22 KiB

Raw Blame History

25 - English Leak Deep-Dive (Post-Lockdown "Abspielen" Persistence)

Investigation triggered after the 2026-05-08 multi-agent English-only lockdown sweep landed (server-wide UICulture, per-user UICulture for 9/9, DisplayPreferences CustomPrefs.language for 32 entries, web shim with navigator.language + localStorage + Accept-Language strip + CSS hide of language switchers). Operator hard-killed Trivalent (cache + LS + SW wiped) and restarted, yet the Play button STILL renders "Abspielen". Audio + subtitle preferences correctly render English (proof the per-user preference layer IS landing for non-UI surfaces).

Date: 2026-05-08 Investigator: deep-dive sibling agent Mode: read-only on Jellyfin live state, read-only on container, no restarts, no shim modifications. Headless-Chromium reproductions used to prove behaviour rather than theorise.

Prior reading (do not repeat findings from): docs/15-force-english.md, docs/19-english-only-audit.md, docs/20-english-only-lockdown.md, docs/22-jellyfin-runtime-perf-audit.md.

1. Executive Summary — actual root cause, with proof

The multi-layer lockdown's per-user Configuration.UICulture pin is inert with respect to the web SPA's UI-string locale. The web SPA's jellyfin-web bundle does not read Configuration.UICulture from the authenticated user object at all — that field is referenced in exactly two chunks (wizard-start.<hash>.chunk.js and 25583.<hash>.chunk.js), both of which are admin dashboard forms for the SERVER-WIDE UICulture (the "Display Language" admin setting), and neither is loaded on a normal user session. Verified live:

$ docker exec jellyfin grep -lE "UICulture" /jellyfin/jellyfin-web/*.js
  /jellyfin/jellyfin-web/index.html               # ARRFLIX shim text only
$ docker exec jellyfin grep -lE "UICulture" /jellyfin/jellyfin-web/*.chunk.js
  /jellyfin/jellyfin-web/25583.95a80bf8834e61a9a8e4.chunk.js
  /jellyfin/jellyfin-web/wizard-start.a4dfcf169516d40c4e52.chunk.js
$ docker exec jellyfin grep -oE ".{40}UICulture.{60}" \
    /jellyfin/jellyfin-web/wizard-start.a4dfcf169516d40c4e52.chunk.js
  up/Configuration")).then((function(n){n.UICulture=$("#selectLocalizationLanguage",t).val(),
  e.ajax({type:"POST...

Both occurrences are POSTs to /System/Configuration (the server-wide dashboard form), not reads from /Users/{id}.Configuration.

The SPA's actual locale resolver (decompiled from main.jellyfin.bundle.js) is:

function g(){
  return document.documentElement.getAttribute("data-culture")
      || (navigator.language     ? navigator.language
       : navigator.userLanguage  ? navigator.userLanguage
       : (navigator.languages?.length) ? navigator.languages[0]
       : "en-us");
}
function w(){
  var e;
  try { e = i.currentSettings.language() }   // localStorage.getItem("language")
  catch(e){ }
  b(e = e || g());
  l = S(e);                                  // S = lowercase + replace _ with -
  document.documentElement.setAttribute("lang", l);
  ...
}

i.currentSettings.language() reads localStorage.getItem("language") (no user prefix — verified via key:"language" lookup with t=false prefix flag in the Settings.get implementation). Per-user Configuration.UICulture is never copied into this localStorage key by any code path in the bundle.

The ARRFLIX shim is the ONLY layer that actually pins the SPA UI language, by overriding Navigator.prototype.language, Navigator.prototype.languages, and pre-seeding localStorage.language to en-US. Headless Trivalent reproductions with explicit --lang=de-DE --accept-lang=de-DE,de,en confirm the shim works correctly:

$ trivalent --headless=new --lang=de-DE --accept-lang=de-DE,de,en \
    --user-data-dir=/tmp/clean-profile --enable-logging=stderr --v=1 \
    https://arrflix.s8n.ru/web/index.html
$ grep -E "json.*chunk.js" /tmp/headless.log
  ...NotifyBeforeURLRequest: https://arrflix.s8n.ru/web/en-us-json.667484b4a441712c7e05.chunk.js
$ grep "<html" /tmp/headless.dump
  <html class="preload layout-desktop" dir="ltr" lang="en-us">

So the shim produces the correct chunk request and the correct <html lang> attribute when running against a freshly-isolated browser profile that never hit arrflix.s8n.ru before. The de-json chunk is never fetched in this scenario.

Therefore the operator's persistent "Abspielen" is not a leak in any server-side or shim-side layer — it is stale browser-side state that predates the shim deploy (web-overrides/index.html mtime 2026-05-08 17:22:00) and survived the operator's wipe. The candidate stale-state vectors, in order of likelihood:

Stale index.html in HTTP disk cache. The server emits no Cache-Control header on /web/index.html; last-modified is Fri, 08 May 2026 16:22:00 GMT. Per RFC 7234 §4.2.2, Chromium heuristically caches at 0.1 * (now − Last-Modified) ≈ 24 minutes when an asset has no Cache-Control. If the operator hit /web/ between the shim deploy at 17:22 and the wipe attempt, then for the ~24-minute heuristic window the OLD pre-shim index.html could reload from disk on subsequent visits without a network round-trip. Mullvad-style "clear site data" wipes (cookies, LS, IndexedDB, SW) do NOT always include the HTTP cache for the eTLD+1 — Chromium's chrome://settings/clearBrowserData exposes "Cached images and files" as a separate checkbox from "Cookies and other site data", and a partial-wipe would leave a stale shim-less index.html in place. The operator's reported wipe path matters here — if it was "Clear site data" via DevTools (LS/SW only) or a Mullvad-style per-origin wipe scoped to storage but not cache, the index.html survives.
A second browser profile / second browser the operator forgot. The operator has multiple Chromium-family browsers installed (~/.var/app/{com.google.Chrome, com.google.ChromeDev, org.chromium.Chromium, io.github.ungoogled_software.ungoogled_chromium, net.mullvad.MullvadBrowser}) plus Trivalent at ~/.config/trivalent. Trivalent's pref selected_languages is en-GB,en-US,en (not German), so the German rendering must come from a browser the operator hasn't wiped. A profile that hit arrflix.s8n.ru weeks ago, when no shim was in place, with a de-* Accept-Language at the time, would have its in-place localStorage language=de (set by the SPA's settings persistence on first load) AND its on-disk index.html cache predating the shim.
The operator's screenshot was captured BEFORE the wipe. "I wiped and still see Abspielen" can be the operator restating an older screenshot rather than reproducing a fresh one. Verifiable only by asking the operator to take a new screenshot post-wipe.

Severity: LOW. The shim is functioning correctly; this is a deploy-day-only stale-cache window. The clean fix is two lines of docker-compose / Traefik headers config to set proper Cache-Control on /web/index.html so future shim deploys propagate without manual operator wiping.

2. Per-Hypothesis Verdicts

#	Hypothesis	Verdict	Probe + evidence
1	`<link rel="prefetch">` for the de chunk fires before the inline shim runs	Ruled out	`grep -ciE 'rel="?(prefetch\|preload\|modulepreload)' index_de.html` → `0`. The bundle uses `<script defer>`, which executes after parsing, AFTER the inline shim has already run. The shim is the first executable JS in the document.
2	Service Worker pre-cached the de chunk	Ruled out	`serviceworker.js` is 768 bytes and contains only a `notificationclick` handler + `clients.claim()`. No `fetch` event listener, no precache, no cache.put. Cannot intercept locale chunk loads. Source: `curl https://arrflix.s8n.ru/web/serviceworker.js`.
3	Bundle reads `<html lang>` attr on first paint	Ruled out	The bundle's locale resolver `g()` reads `document.documentElement.getAttribute("data-culture")` — NOT `lang`. The `lang` attribute is set by the bundle (via `document.documentElement.setAttribute("lang", l)` in `w()`), it is not read. The served HTML opens with `<html class="preload" dir="ltr">` — no `data-culture`, no `lang`.
4	Bundle reads `document.cookie` for locale	Ruled out	`grep -ciE 'document.cookie.{0,40}lang\|locale\|culture' main.bundle.js` → `0`. No cookie-based locale path in any bundle.
5	Hard-coded `de-DE` fallback in the bundle	Ruled out	The hard-coded fallback is `var f="en-us"` (decompiled from `main.bundle.js`) — used when `navigator.language`, `navigator.languages`, `navigator.userLanguage`, and `data-culture` are all absent. Falls back to English, not German.
6	Server sends `Content-Language: de`	Ruled out	`curl -sI https://arrflix.s8n.ru/web/index.html` returns no `Content-Language` header.
7	Traefik/upstream content-negotiates locale (`Vary: Accept-Language`)	Ruled out	`curl -sI` returns no `Vary` header. Both `Accept-Language: de-DE,de;q=0.9,en;q=0.5` and `Accept-Language: en-US` return byte-identical 65485-byte HTML (same etag `1dcdf06cc053bcd`). Confirmed via `diff -q` of two captures.
8	Per-user `DisplayPreferences.CustomPrefs.language` writes the wrong key	Inconclusive (irrelevant)	DisplayPreferences is read by the bundle for `chromecastVersion`, `dashboardTheme`, home-section ordering, etc. — not for UI locale. The locale-related code paths (`g()`, `w()`, `i.currentSettings.language()`) read from `Navigator.prototype.language`, `data-culture`, and `localStorage.getItem("language")` only. Per-user DisplayPreferences.CustomPrefs.language could be set to anything and the SPA UI would not change. The 32 entries written by sibling A2 are a no-op for the Abspielen bug.
9	Cineplex theme injects German strings via CSS `content:`	Ruled out	`grep -ciE 'content:.{0,80}(Abspielen\|Fortsetzen\|Anzeigen)' /opt/jellyfin/config/branding/*.css 2>&1` returns 0. Themes are CSS-only and Jellyfin's branding `CustomCss` is plain CSS, not capable of localising button labels.
10	Plugin contributes the Play string	Ruled out	`GET /Plugins` lists 6 plugins (AudioDB, MusicBrainz, OMDb, Open Subtitles, Studio Images, TMDb). All are metadata-source plugins with server-side string surfaces only; none ship web-bundle UI strings. Verified by inspecting each plugin's `Plugin.{xml,json}` for `web/` or `client/` resources — none.
11	Pre-auth chunk request races the shim	Ruled out by reproduction	Headless Trivalent run with `--lang=de-DE --accept-lang=de-DE,de,en` against a freshly-created `--user-data-dir`, capturing the full network log: the ONLY locale chunk requested is `en-us-json.667484b4a441712c7e05.chunk.js`. The de chunk URL is never touched. The shim's `Object.defineProperty(Navigator.prototype, 'language', …)` runs synchronously during HTML parsing (inline non-defer script), before any deferred bundle script executes (HTML5 spec §4.12.1 — defer scripts execute after parsing in document order; inline scripts execute when the parser reaches them). The locale resolver `g()` runs inside the deferred bundle, so the override is in effect by the time `g()` is called.
12	Browser sends `Accept-Language: de-DE` and the SPA reads it via fetch echo	Ruled out	The SPA's locale resolver does NOT make any pre-bundle network request to read its own Accept-Language. The resolver is purely synchronous and only reads `navigator.language`, `navigator.userLanguage`, `navigator.languages[0]`, plus the `data-culture` DOM attr and `localStorage.language`. Confirmed by full-text `grep -ciE 'fetch.{0,200}accept.language\|XMLHttpRequest.{0,200}accept.language' main.bundle.js` → matches only the ARRFLIX shim's STRIP code, no read.

3. Concrete remediation, ranked by blast radius

R1 — Add `Cache-Control: no-cache` on `/web/index.html` (Traefik header) — RECOMMENDED FIRST

Smallest blast radius. Forces every browser to revalidate the index.html on every visit, so future shim updates propagate within one tab refresh instead of a 10–55 minute heuristic-cache window.

# In /opt/docker/jellyfin/docker-compose.yml under the jellyfin service labels:
- "traefik.http.routers.jellyfin.middlewares=jellyfin-nocache-html@docker"
- "traefik.http.middlewares.jellyfin-nocache-html.headers.customresponseheaders.Cache-Control=no-cache, must-revalidate"

Caveat: this header would be applied to ALL responses on the jellyfin router, including the immutable hashed chunk files. Chunks SHOULD remain cacheable forever (they're hash-fingerprinted). Therefore either:

Path A (simpler): apply no-cache only to /web/index.html via a path-scoped middleware, leaving everything else alone:

- "traefik.http.middlewares.jellyfin-nocache-html.headers.customresponseheaders.Cache-Control=no-cache, must-revalidate"
- "traefik.http.routers.jellyfin-html.rule=Host(`arrflix.s8n.ru`) && Path(`/web/index.html`)"
- "traefik.http.routers.jellyfin-html.middlewares=jellyfin-nocache-html@docker"
- "traefik.http.routers.jellyfin-html.priority=100"   # higher than the catch-all
- "traefik.http.routers.jellyfin-html.service=jellyfin@docker"

Path B (cleaner, better long-term): apply Cache-Control: public, max-age=31536000, immutable to all /web/*.{js,css,chunk.js} (which Jellyfin upstream already fingerprints) AND Cache-Control: no-cache, must-revalidate to /web/index.html and /web/manifest.json. This is the conventional SPA cache strategy; we get the best of both worlds (instant chunk load + always-fresh shim).

Do NOT install yet — operator decision required on Path A vs B.

R2 — Operator-side: document the precise wipe procedure for shim updates

Add to docs/20-english-only-lockdown.md "Re-apply procedure" section:

When updating the web shim (web-overrides/index.html or any file bind-mounted into /jellyfin/jellyfin-web/), every active browser session must be wiped with "Clear browsing data" → tick BOTH "Cookies and other site data" AND "Cached images and files" for the arrflix.s8n.ru origin. DevTools "Storage → Clear site data" alone does NOT clear HTTP disk cache in all Chromium variants; the all-time wipe via chrome://settings/clearBrowserData is required.

This closes the operator-process gap that left a stale index.html in the operator's browser.

R3 — Stop investing in per-user `Configuration.UICulture` POSTs

Per the proof in §1, this field has no effect on the web SPA's UI language. It controls only:

The user object the API returns (so the dashboard form for "edit user" displays the correct value if anyone ever opens it).
Server-side string surfaces that DO honour per-user culture (Live TV EPG metadata for the API caller, some plugin responses), but NOT the web client UI strings.

Keep bin/force-english-all-users.sh and the bin/add-jellyfin-user.sh UICulture line for cosmetic consistency and future-proofing (Jellyfin upstream might wire it up someday), but stop expecting it to fix UI-string leaks. The shim is the only thing pinning the UI.

R4 — Defense-in-depth: bind-mount empty `de.json` chunk stubs

Doc 19 §"Files to Delete" (Path B) proposed this. Still valid as a belt for Path R1, but high-maintenance (chunk hashes rotate on every Jellyfin upgrade — currently de-json.1afccc006ab8bb6c5953.chunk.js but a jellyfin/jellyfin image bump could change it). Defer indefinitely unless the operator wants the German strings physically unreachable for paranoia.

R5 — `Accept-Language` rewrite at Traefik (doc 19 §"Path A — Traefik middleware")

- "traefik.http.middlewares.arrflix-lang.headers.customrequestheaders.Accept-Language=en-US,en;q=0.9"
- "traefik.http.routers.jellyfin.middlewares=arrflix-lang"

Useful but redundant with the existing shim. The shim already strips Accept-Language on outbound fetch/XHR (verified live in shim source). The browser-issued INITIAL request to /web/index.html is the only one that would ever carry Accept-Language, and the index.html is byte-identical regardless of header (proven in hypothesis 7). So this rewrite would prevent future Jellyfin upstream behaviour changes that start using Accept-Language on the index.html response, but doesn't fix anything currently broken.

4. Why prior audits (15, 19, 20) missed this

Doc 15 correctly diagnosed that the SPA "falls back to Accept-Language when UICulture is unset" — a CONJECTURE based on observing that German appeared and that Configuration.UICulture was absent on every user. The conjecture was never tested by GREPPING THE WEB BUNDLE for UICulture, which would have shown immediately that the SPA never reads it. Doc 19 inherited the conjecture verbatim. Doc 20 codified it into the lockdown procedure. Three audits in a row, all assuming a causal link that doesn't exist.

The actual causal layer (Navigator.prototype.language → <lang>-json chunk selection) was correctly identified in doc 19's §"Layer 18" remediation suggestion — which is what shipped as the shim. The shim is the fix; the per-user UICulture pin is theatre. After this deep dive, doc 20's "Layer 2" (per-user) and "Layer 1" (server-wide) sections should be re-labelled as "metadata-affecting" rather than "UI-affecting", and doc 19's primary-fix table-row should be flipped from "Layer 5 (per-user UICulture) is the biggest impact" to "Layer 18 (navigator.language shim) is the only impact on UI strings".

A subtler lesson: when doc 19 said "all 8 users have UICulture absent → that's why German leaks", the audit also noted (table row 16) that 92 non-English locale chunks are reachable and contain "Play":"Abspielen" etc. That observation alone, combined with the chunk-loading code, would have shown the actual mechanism. The fix was correctly proposed (shim with navigator.language override), but the diagnosis text emphasised the wrong layer.

5. New hypotheses uncovered during probe

H13 — `index.html` is served with NO `Cache-Control` header

Already covered above (R1). Not a leak per se but the mechanism by which ANY future shim deploy can fail to propagate without operator wiping. Critical to fix before the next shim iteration to avoid this whole "why is it still German" dance recurring.

H14 — Operator may have multiple browser profiles/binaries with stale state

Operator has Trivalent (~/.config/trivalent), Chromium (~/.config/chromium with profile Default and Profile 1), ~/.config/google-chrome-for-testing, plus Flatpak: Chrome, ChromeDev, Chromium, ungoogled-Chromium, MullvadBrowser. Eight Chromium-family installs. A wipe of "the browser" plausibly missed at least one. Probe to ask operator: which exact browser binary + which exact profile produced the screenshot? "Trivalent default profile with all storage wiped including HTTP cache" yields a different conclusion from "Mullvad ad-hoc surgical wipe targeting storage only".

H15 — Per-user `Configuration.UICulture` lockdown layer is doing literal nothing

Documented in R3. Worth flagging because the op cost (running bin/english-lockdown-runner.sh weekly via systemd timer) is nonzero and we now know the only layer that matters is the shim, which doesn't need re-running because it's a static bind-mount.

H16 — Chunk URLs in the require.context are immutable per Jellyfin upgrade

Verified: n(73125) maps ./en-us.json → [20233, 79754] and ./de.json → [99810, 89409]. These ID pairs are baked into the runtime.bundle.js at Jellyfin build time. So a Jellyfin image upgrade WILL change the chunk hashes (filename, e.g. de-json.<hash>.chunk.js) but NOT the chunk-id mapping in runtime.bundle.js. Any defense-in-depth via 1-byte stub bind-mounts (R4) MUST be regenerated after every image bump — not just the filename, but if the chunk-id stub-content (self.webpackChunk = …) .push([[CHUNK_ID], {}]) also depends on the chunk-id, then those lines need re-emission too. Treat as part of the upgrade runbook, not a one-shot install.

H17 — `localStorage.language` is shared across all Jellyfin users on the same browser

The settings store reads localStorage.getItem("language") with NO user-prefix when the prefix flag is false (verified in key: "language" getter signature with t = false). All other preferences are stored as <userId>-<key>, but language is specifically global. So if user A on a browser with German pref loads the SPA pre-shim and the SPA writes localStorage.language = "de" into the user-settings store, then user B on the same browser inherits the German preference until either the shim runs (which overwrites it on every load) or the storage is wiped. The shim's pinLocale() belt re-pins on every visibility change, so this isn't exploitable, but it IS the ONE persistence mechanism that survives both server-side UICulture pinning and per-user-DisplayPreferences writes.

Sign-off

Mode: read-only on Jellyfin live state (no POST/PATCH/PUT). Read-only on container (zero docker exec writes). Shim file unchanged. Headless Trivalent test runs used /tmp/eng-deep-dive/cprofile{,2} isolated profiles, no production browser state touched.
Live evidence captures:
- /tmp/eng-deep-dive/index_de.html (65485 bytes, has shim block)
- /tmp/eng-deep-dive/runtime.bundle.js + main.bundle.js + 37869.bundle.js (decompiled to confirm locale-resolver code path)
- /tmp/eng-deep-dive/headless2.log (only en-us-json chunk requested under explicit German Accept-Language)
Recommendation order: R1 (Cache-Control no-cache on index.html) → R2 (document wipe procedure) → R3 (stop investing in per-user UICulture for UI). R4 and R5 are optional defense-in-depth, not required to fix the screenshot.
Next-action owner: operator decides Path A vs B for R1; web agent then applies the chosen Traefik label diff in a single commit.
Severity: LOW — shim is functioning, this is a stale-cache process gap, not a continuous leak.

22 KiB Raw Blame History Unescape Escape