doc 25: english leak deep-dive (Abspielen post-lockdown)

docs/25-english-leak-deep-dive-2026-05-08.md

@@ -0,0 +1,373 @@
+# 25 - English Leak Deep-Dive (Post-Lockdown "Abspielen" Persistence)
+
+> Investigation triggered after the 2026-05-08 multi-agent English-only
+> lockdown sweep landed (server-wide UICulture, per-user UICulture for 9/9,
+> DisplayPreferences CustomPrefs.language for 32 entries, web shim with
+> `navigator.language` + localStorage + Accept-Language strip + CSS hide of
+> language switchers). Operator hard-killed Trivalent (cache + LS + SW
+> wiped) and restarted, yet the Play button STILL renders **"Abspielen"**.
+> Audio + subtitle preferences correctly render English (proof the per-user
+> preference layer IS landing for non-UI surfaces).
+
+Date: 2026-05-08
+Investigator: deep-dive sibling agent
+Mode: read-only on Jellyfin live state, read-only on container, no
+restarts, no shim modifications. Headless-Chromium reproductions used to
+prove behaviour rather than theorise.
+
+Prior reading (do not repeat findings from):
+`docs/15-force-english.md`, `docs/19-english-only-audit.md`,
+`docs/20-english-only-lockdown.md`, `docs/22-jellyfin-runtime-perf-audit.md`.
+
+---
+
+## 1. Executive Summary — actual root cause, with proof
+
+The multi-layer lockdown's **per-user `Configuration.UICulture` pin is
+inert with respect to the web SPA's UI-string locale**. The web SPA's
+`jellyfin-web` bundle does not read `Configuration.UICulture` from the
+authenticated user object at all — that field is referenced in exactly two
+chunks (`wizard-start.<hash>.chunk.js` and `25583.<hash>.chunk.js`), both
+of which are admin **dashboard** forms for the SERVER-WIDE UICulture (the
+"Display Language" admin setting), and neither is loaded on a normal user
+session. Verified live:
+
+```
+$ docker exec jellyfin grep -lE "UICulture" /jellyfin/jellyfin-web/*.js
+  /jellyfin/jellyfin-web/index.html               # ARRFLIX shim text only
+$ docker exec jellyfin grep -lE "UICulture" /jellyfin/jellyfin-web/*.chunk.js
+  /jellyfin/jellyfin-web/25583.95a80bf8834e61a9a8e4.chunk.js
+  /jellyfin/jellyfin-web/wizard-start.a4dfcf169516d40c4e52.chunk.js
+$ docker exec jellyfin grep -oE ".{40}UICulture.{60}" \
+    /jellyfin/jellyfin-web/wizard-start.a4dfcf169516d40c4e52.chunk.js
+  up/Configuration")).then((function(n){n.UICulture=$("#selectLocalizationLanguage",t).val(),
+  e.ajax({type:"POST...
+```
+
+Both occurrences are POSTs to `/System/Configuration` (the server-wide
+dashboard form), not reads from `/Users/{id}.Configuration`.
+
+**The SPA's actual locale resolver** (decompiled from
+`main.jellyfin.bundle.js`) is:
+
+```js
+function g(){
+  return document.documentElement.getAttribute("data-culture")
+      || (navigator.language     ? navigator.language
+       : navigator.userLanguage  ? navigator.userLanguage
+       : (navigator.languages?.length) ? navigator.languages[0]
+       : "en-us");
+}
+function w(){
+  var e;
+  try { e = i.currentSettings.language() }   // localStorage.getItem("language")
+  catch(e){ }
+  b(e = e || g());
+  l = S(e);                                  // S = lowercase + replace _ with -
+  document.documentElement.setAttribute("lang", l);
+  ...
+}
+```
+
+`i.currentSettings.language()` reads `localStorage.getItem("language")`
+(no user prefix — verified via `key:"language"` lookup with `t=false`
+prefix flag in the Settings.get implementation). Per-user
+`Configuration.UICulture` is never copied into this localStorage key by
+any code path in the bundle.
+
+**The ARRFLIX shim is the ONLY layer that actually pins the SPA UI
+language**, by overriding `Navigator.prototype.language`,
+`Navigator.prototype.languages`, and pre-seeding `localStorage.language`
+to `en-US`. Headless Trivalent reproductions with explicit
+`--lang=de-DE --accept-lang=de-DE,de,en` confirm the shim works correctly:
+
+```
+$ trivalent --headless=new --lang=de-DE --accept-lang=de-DE,de,en \
+    --user-data-dir=/tmp/clean-profile --enable-logging=stderr --v=1 \
+    https://arrflix.s8n.ru/web/index.html
+$ grep -E "json.*chunk.js" /tmp/headless.log
+  ...NotifyBeforeURLRequest: https://arrflix.s8n.ru/web/en-us-json.667484b4a441712c7e05.chunk.js
+$ grep "<html" /tmp/headless.dump
+  <html class="preload layout-desktop" dir="ltr" lang="en-us">
+```
+
+So the shim produces the correct chunk request and the correct `<html
+lang>` attribute when running against a freshly-isolated browser profile
+that never hit arrflix.s8n.ru before. The de-json chunk is **never**
+fetched in this scenario.
+
+**Therefore the operator's persistent "Abspielen" is not a leak in any
+server-side or shim-side layer — it is stale browser-side state that
+predates the shim deploy (web-overrides/index.html mtime
+2026-05-08 17:22:00) and survived the operator's wipe.** The candidate
+stale-state vectors, in order of likelihood:
+
+1. **Stale `index.html` in HTTP disk cache.** The server emits **no
+   `Cache-Control` header** on `/web/index.html`; `last-modified` is
+   `Fri, 08 May 2026 16:22:00 GMT`. Per RFC 7234 §4.2.2, Chromium
+   heuristically caches at `0.1 * (now − Last-Modified)` ≈ 24 minutes
+   when an asset has no Cache-Control. If the operator hit `/web/`
+   between the shim deploy at 17:22 and the wipe attempt, then for the
+   ~24-minute heuristic window the OLD pre-shim index.html could
+   reload from disk on subsequent visits without a network round-trip.
+   Mullvad-style "clear site data" wipes (cookies, LS, IndexedDB, SW)
+   do NOT always include the HTTP cache for the eTLD+1 — Chromium's
+   `chrome://settings/clearBrowserData` exposes "Cached images and
+   files" as a separate checkbox from "Cookies and other site data",
+   and a partial-wipe would leave a stale shim-less index.html in
+   place. The operator's reported wipe path matters here — if it was
+   "Clear site data" via DevTools (LS/SW only) or a Mullvad-style
+   per-origin wipe scoped to storage but not cache, the index.html
+   survives.
+
+2. **A second browser profile / second browser the operator forgot.**
+   The operator has multiple Chromium-family browsers installed
+   (`~/.var/app/{com.google.Chrome, com.google.ChromeDev,
+   org.chromium.Chromium, io.github.ungoogled_software.ungoogled_chromium,
+   net.mullvad.MullvadBrowser}`) plus Trivalent at
+   `~/.config/trivalent`. Trivalent's pref `selected_languages` is
+   `en-GB,en-US,en` (not German), so the German rendering must come
+   from a browser the operator hasn't wiped. A profile that hit
+   arrflix.s8n.ru weeks ago, when no shim was in place, with a
+   `de-*` Accept-Language at the time, would have its in-place
+   localStorage `language=de` (set by the SPA's settings persistence
+   on first load) AND its on-disk index.html cache predating the shim.
+
+3. **The operator's screenshot was captured BEFORE the wipe.** "I
+   wiped and still see Abspielen" can be the operator restating an
+   older screenshot rather than reproducing a fresh one. Verifiable
+   only by asking the operator to take a new screenshot post-wipe.
+
+**Severity: LOW.** The shim is functioning correctly; this is a
+deploy-day-only stale-cache window. The clean fix is two lines of
+docker-compose / Traefik headers config to set proper `Cache-Control`
+on `/web/index.html` so future shim deploys propagate without manual
+operator wiping.
+
+---
+
+## 2. Per-Hypothesis Verdicts
+
+| # | Hypothesis | Verdict | Probe + evidence |
+|---|---|---|---|
+| 1 | `<link rel="prefetch">` for the de chunk fires before the inline shim runs | **Ruled out** | `grep -ciE 'rel="?(prefetch\|preload\|modulepreload)' index_de.html` → `0`. The bundle uses `<script defer>`, which executes after parsing, AFTER the inline shim has already run. The shim is the first executable JS in the document. |
+| 2 | Service Worker pre-cached the de chunk | **Ruled out** | `serviceworker.js` is 768 bytes and contains only a `notificationclick` handler + `clients.claim()`. No `fetch` event listener, no precache, no cache.put. Cannot intercept locale chunk loads. Source: `curl https://arrflix.s8n.ru/web/serviceworker.js`. |
+| 3 | Bundle reads `<html lang>` attr on first paint | **Ruled out** | The bundle's locale resolver `g()` reads `document.documentElement.getAttribute("data-culture")` — NOT `lang`. The `lang` attribute is *set* by the bundle (via `document.documentElement.setAttribute("lang", l)` in `w()`), it is not read. The served HTML opens with `<html class="preload" dir="ltr">` — no `data-culture`, no `lang`. |
+| 4 | Bundle reads `document.cookie` for locale | **Ruled out** | `grep -ciE 'document.cookie.{0,40}lang\|locale\|culture' main.bundle.js` → `0`. No cookie-based locale path in any bundle. |
+| 5 | Hard-coded `de-DE` fallback in the bundle | **Ruled out** | The hard-coded fallback is `var f="en-us"` (decompiled from `main.bundle.js`) — used when `navigator.language`, `navigator.languages`, `navigator.userLanguage`, and `data-culture` are all absent. Falls back to English, not German. |
+| 6 | Server sends `Content-Language: de` | **Ruled out** | `curl -sI https://arrflix.s8n.ru/web/index.html` returns no `Content-Language` header. |
+| 7 | Traefik/upstream content-negotiates locale (`Vary: Accept-Language`) | **Ruled out** | `curl -sI` returns no `Vary` header. Both `Accept-Language: de-DE,de;q=0.9,en;q=0.5` and `Accept-Language: en-US` return byte-identical 65485-byte HTML (same etag `1dcdf06cc053bcd`). Confirmed via `diff -q` of two captures. |
+| 8 | Per-user `DisplayPreferences.CustomPrefs.language` writes the wrong key | **Inconclusive (irrelevant)** | DisplayPreferences is read by the bundle for `chromecastVersion`, `dashboardTheme`, home-section ordering, etc. — not for UI locale. The locale-related code paths (`g()`, `w()`, `i.currentSettings.language()`) read from `Navigator.prototype.language`, `data-culture`, and `localStorage.getItem("language")` only. Per-user DisplayPreferences.CustomPrefs.language could be set to anything and the SPA UI would not change. The 32 entries written by sibling A2 are a no-op for the Abspielen bug. |
+| 9 | Cineplex theme injects German strings via CSS `content:` | **Ruled out** | `grep -ciE 'content:.{0,80}(Abspielen\|Fortsetzen\|Anzeigen)' /opt/jellyfin/config/branding/*.css 2>&1` returns 0. Themes are CSS-only and Jellyfin's branding `CustomCss` is plain CSS, not capable of localising button labels. |
+| 10 | Plugin contributes the Play string | **Ruled out** | `GET /Plugins` lists 6 plugins (AudioDB, MusicBrainz, OMDb, Open Subtitles, Studio Images, TMDb). All are metadata-source plugins with server-side string surfaces only; none ship web-bundle UI strings. Verified by inspecting each plugin's `Plugin.{xml,json}` for `web/` or `client/` resources — none. |
+| 11 | Pre-auth chunk request races the shim | **Ruled out by reproduction** | Headless Trivalent run with `--lang=de-DE --accept-lang=de-DE,de,en` against a freshly-created `--user-data-dir`, capturing the full network log: the ONLY locale chunk requested is `en-us-json.667484b4a441712c7e05.chunk.js`. The de chunk URL is never touched. The shim's `Object.defineProperty(Navigator.prototype, 'language', …)` runs synchronously during HTML parsing (inline non-defer script), before any deferred bundle script executes (HTML5 spec §4.12.1 — defer scripts execute after parsing in document order; inline scripts execute when the parser reaches them). The locale resolver `g()` runs inside the deferred bundle, so the override is in effect by the time `g()` is called. |
+| 12 | Browser sends `Accept-Language: de-DE` and the SPA reads it via fetch echo | **Ruled out** | The SPA's locale resolver does NOT make any pre-bundle network request to read its own Accept-Language. The resolver is purely synchronous and only reads `navigator.language`, `navigator.userLanguage`, `navigator.languages[0]`, plus the `data-culture` DOM attr and `localStorage.language`. Confirmed by full-text `grep -ciE 'fetch.{0,200}accept.language\|XMLHttpRequest.{0,200}accept.language' main.bundle.js` → matches only the ARRFLIX shim's STRIP code, no read. |
+
+---
+
+## 3. Concrete remediation, ranked by blast radius
+
+### R1 — Add `Cache-Control: no-cache` on `/web/index.html` (Traefik header) — RECOMMENDED FIRST
+
+Smallest blast radius. Forces every browser to revalidate the index.html
+on every visit, so future shim updates propagate within one tab refresh
+instead of a 10–55 minute heuristic-cache window.
+
+```yaml
+# In /opt/docker/jellyfin/docker-compose.yml under the jellyfin service labels:
+- "traefik.http.routers.jellyfin.middlewares=jellyfin-nocache-html@docker"
+- "traefik.http.middlewares.jellyfin-nocache-html.headers.customresponseheaders.Cache-Control=no-cache, must-revalidate"
+```
+
+**Caveat:** this header would be applied to ALL responses on the
+`jellyfin` router, including the immutable hashed chunk files. Chunks
+SHOULD remain cacheable forever (they're hash-fingerprinted). Therefore
+either:
+
+- **Path A (simpler):** apply `no-cache` only to `/web/index.html` via
+  a path-scoped middleware, leaving everything else alone:
+
+  ```yaml
+  - "traefik.http.middlewares.jellyfin-nocache-html.headers.customresponseheaders.Cache-Control=no-cache, must-revalidate"
+  - "traefik.http.routers.jellyfin-html.rule=Host(`arrflix.s8n.ru`) && Path(`/web/index.html`)"
+  - "traefik.http.routers.jellyfin-html.middlewares=jellyfin-nocache-html@docker"
+  - "traefik.http.routers.jellyfin-html.priority=100"   # higher than the catch-all
+  - "traefik.http.routers.jellyfin-html.service=jellyfin@docker"
+  ```
+
+- **Path B (cleaner, better long-term):** apply `Cache-Control:
+  public, max-age=31536000, immutable` to all `/web/*.{js,css,chunk.js}`
+  (which Jellyfin upstream already fingerprints) AND `Cache-Control:
+  no-cache, must-revalidate` to `/web/index.html` and `/web/manifest.json`.
+  This is the conventional SPA cache strategy; we get the best of both
+  worlds (instant chunk load + always-fresh shim).
+
+**Do NOT install yet** — operator decision required on Path A vs B.
+
+### R2 — Operator-side: document the precise wipe procedure for shim updates
+
+Add to `docs/20-english-only-lockdown.md` "Re-apply procedure" section:
+
+> When updating the web shim (`web-overrides/index.html` or any file
+> bind-mounted into `/jellyfin/jellyfin-web/`), every active browser
+> session must be wiped with **"Clear browsing data" → tick BOTH
+> "Cookies and other site data" AND "Cached images and files"** for
+> the `arrflix.s8n.ru` origin. DevTools "Storage → Clear site data"
+> alone does NOT clear HTTP disk cache in all Chromium variants; the
+> all-time wipe via `chrome://settings/clearBrowserData` is required.
+
+This closes the operator-process gap that left a stale index.html in
+the operator's browser.
+
+### R3 — Stop investing in per-user `Configuration.UICulture` POSTs
+
+Per the proof in §1, this field has no effect on the web SPA's UI
+language. It controls only:
+
+- The user object the API returns (so the dashboard form for "edit
+  user" displays the correct value if anyone ever opens it).
+- Server-side string surfaces that DO honour per-user culture
+  (Live TV EPG metadata for the API caller, some plugin responses),
+  but NOT the web client UI strings.
+
+Keep `bin/force-english-all-users.sh` and the
+`bin/add-jellyfin-user.sh` UICulture line for cosmetic consistency
+and future-proofing (Jellyfin upstream might wire it up someday), but
+**stop expecting it to fix UI-string leaks**. The shim is the only
+thing pinning the UI.
+
+### R4 — Defense-in-depth: bind-mount empty `de.json` chunk stubs
+
+Doc 19 §"Files to Delete" (Path B) proposed this. Still valid as a
+belt for Path R1, but high-maintenance (chunk hashes rotate on every
+Jellyfin upgrade — currently `de-json.1afccc006ab8bb6c5953.chunk.js`
+but a `jellyfin/jellyfin` image bump could change it). **Defer
+indefinitely** unless the operator wants the German strings physically
+unreachable for paranoia.
+
+### R5 — `Accept-Language` rewrite at Traefik (doc 19 §"Path A — Traefik middleware")
+
+```yaml
+- "traefik.http.middlewares.arrflix-lang.headers.customrequestheaders.Accept-Language=en-US,en;q=0.9"
+- "traefik.http.routers.jellyfin.middlewares=arrflix-lang"
+```
+
+**Useful but redundant** with the existing shim. The shim already
+strips Accept-Language on outbound fetch/XHR (verified live in shim
+source). The browser-issued INITIAL request to `/web/index.html` is
+the only one that would ever carry Accept-Language, and the index.html
+is byte-identical regardless of header (proven in hypothesis 7). So
+this rewrite would prevent **future** Jellyfin upstream behaviour
+changes that start using Accept-Language on the index.html response,
+but doesn't fix anything currently broken.
+
+---
+
+## 4. Why prior audits (15, 19, 20) missed this
+
+Doc 15 correctly diagnosed that the SPA "falls back to
+`Accept-Language` when `UICulture` is unset" — a CONJECTURE based on
+observing that German appeared and that `Configuration.UICulture` was
+absent on every user. The conjecture was never tested by GREPPING THE
+WEB BUNDLE for `UICulture`, which would have shown immediately that
+the SPA never reads it. Doc 19 inherited the conjecture verbatim. Doc
+20 codified it into the lockdown procedure. Three audits in a row,
+all assuming a causal link that doesn't exist.
+
+The actual causal layer (`Navigator.prototype.language` →
+`<lang>-json` chunk selection) was correctly identified in doc 19's
+§"Layer 18" remediation suggestion — which is what shipped as the
+shim. The shim is the fix; the per-user UICulture pin is theatre.
+After this deep dive, doc 20's "Layer 2" (per-user) and "Layer 1"
+(server-wide) sections should be re-labelled as "metadata-affecting"
+rather than "UI-affecting", and doc 19's primary-fix table-row should
+be flipped from "Layer 5 (per-user UICulture) is the biggest impact"
+to "Layer 18 (navigator.language shim) is the only impact on UI
+strings".
+
+A subtler lesson: when doc 19 said "all 8 users have `UICulture` absent
+→ that's why German leaks", the audit *also* noted (table row 16)
+that 92 non-English locale chunks are reachable and contain
+`"Play":"Abspielen"` etc. That observation alone, combined with the
+chunk-loading code, would have shown the actual mechanism. The fix
+was correctly proposed (shim with `navigator.language` override),
+but the diagnosis text emphasised the wrong layer.
+
+---
+
+## 5. New hypotheses uncovered during probe
+
+### H13 — `index.html` is served with NO `Cache-Control` header
+Already covered above (R1). Not a leak per se but the mechanism by which
+ANY future shim deploy can fail to propagate without operator wiping.
+Critical to fix before the next shim iteration to avoid this whole
+"why is it still German" dance recurring.
+
+### H14 — Operator may have multiple browser profiles/binaries with stale state
+Operator has Trivalent (`~/.config/trivalent`), Chromium
+(`~/.config/chromium` with profile `Default` and `Profile 1`),
+`~/.config/google-chrome-for-testing`, plus Flatpak: Chrome,
+ChromeDev, Chromium, ungoogled-Chromium, MullvadBrowser. Eight
+Chromium-family installs. A wipe of "the browser" plausibly missed at
+least one. **Probe to ask operator:** which exact browser binary +
+which exact profile produced the screenshot? "Trivalent default
+profile with all storage wiped including HTTP cache" yields a
+different conclusion from "Mullvad ad-hoc surgical wipe targeting
+storage only".
+
+### H15 — Per-user `Configuration.UICulture` lockdown layer is doing literal nothing
+Documented in R3. Worth flagging because the op cost (running
+`bin/english-lockdown-runner.sh` weekly via systemd timer) is nonzero
+and we now know the only layer that matters is the shim, which
+doesn't need re-running because it's a static bind-mount.
+
+### H16 — Chunk URLs in the require.context are immutable per Jellyfin upgrade
+Verified: `n(73125)` maps `./en-us.json` → `[20233, 79754]` and
+`./de.json` → `[99810, 89409]`. These ID pairs are baked into the
+runtime.bundle.js at Jellyfin build time. So a Jellyfin image upgrade
+WILL change the chunk hashes (filename, e.g.
+`de-json.<hash>.chunk.js`) but NOT the chunk-id mapping in
+runtime.bundle.js. Any defense-in-depth via 1-byte stub bind-mounts
+(R4) MUST be regenerated after every image bump — not just the
+filename, but if the chunk-id stub-content `(self.webpackChunk = …)
+.push([[CHUNK_ID], {}])` also depends on the chunk-id, then those
+lines need re-emission too. Treat as part of the upgrade runbook,
+not a one-shot install.
+
+### H17 — `localStorage.language` is shared across all Jellyfin users on the same browser
+The settings store reads `localStorage.getItem("language")` with NO
+user-prefix when the prefix flag is `false` (verified in `key:
+"language"` getter signature with `t = false`). All other
+preferences are stored as `<userId>-<key>`, but `language` is
+specifically global. So if user A on a browser with German pref
+loads the SPA pre-shim and the SPA writes `localStorage.language =
+"de"` into the user-settings store, then user B on the same browser
+inherits the German preference until either the shim runs (which
+overwrites it on every load) or the storage is wiped. The shim's
+`pinLocale()` belt re-pins on every visibility change, so this isn't
+exploitable, but it IS the ONE persistence mechanism that survives
+both server-side UICulture pinning and per-user-DisplayPreferences
+writes.
+
+---
+
+## Sign-off
+
+- **Mode:** read-only on Jellyfin live state (no POST/PATCH/PUT).
+  Read-only on container (zero `docker exec` writes). Shim file
+  unchanged. Headless Trivalent test runs used `/tmp/eng-deep-dive/cprofile{,2}`
+  isolated profiles, no production browser state touched.
+- **Live evidence captures:**
+  - `/tmp/eng-deep-dive/index_de.html` (65485 bytes, has shim block)
+  - `/tmp/eng-deep-dive/runtime.bundle.js` + `main.bundle.js` +
+    `37869.bundle.js` (decompiled to confirm locale-resolver code path)
+  - `/tmp/eng-deep-dive/headless2.log` (only en-us-json chunk
+    requested under explicit German Accept-Language)
+- **Recommendation order:** R1 (Cache-Control no-cache on index.html)
+  → R2 (document wipe procedure) → R3 (stop investing in per-user
+  UICulture for UI). R4 and R5 are optional defense-in-depth, not
+  required to fix the screenshot.
+- **Next-action owner:** operator decides Path A vs B for R1; web
+  agent then applies the chosen Traefik label diff in a single commit.
+- **Severity:** LOW — shim is functioning, this is a stale-cache
+  process gap, not a continuous leak.