Player YOU500 lost full inventory to AuthLimbo void-death at 17:13:39. Investigation revealed deployed /opt/docker/backup.sh is an 88-line stub missing the Minecraft block; last successful world backup 2026-05-02 (already pruned). No recoverable .dat exists. Files: - AUDIT-2026-05-07.md — server-side findings F-01..F-06 (P0 backups, no-keepInventory, AuthLimbo silent failure, chunk preload race, Xmx > container headroom, container hardening gaps) - BACKUP-HUNT-2026-05-07.md — exhaustive backup scan; only 6-week-old archive at _archive/minecraft-old-2026-04-27.tar.gz - BACKUP-STRATEGY.md — restic-based plan; 5min/hourly/daily classes, off-host to onyx via Tailscale, monthly drill - CROSS-REFERENCE-2026-05-07.md — repo+doc landing map; flags pre-existing infra/STATE.md backup-broken note + HA-CLUSTER restic draft to extend rather than duplicate - docs/RUNBOOK-BACKUP-RESTORE.md — operator runbook for .dat restore, full-world restore, host-loss restore, drill log
22 KiB
Cross-Reference Survey — 2026-05-07
Trigger: racked.ru player YOU500 void-died via AuthLimbo
teleportAsync failure, lost full inventory, no backups exist.
Four parallel agents are writing audit + plan docs. This doc maps
them onto existing infra so nothing collides or gets orphaned.
1. Per-repo state snapshot
auth-limbo (Paper plugin source)
| Field | Value |
|---|---|
| Origin | ssh://git@192.168.0.100:222/s8n-ru/auth-limbo.git ⚠️ stale (s8n-ru rename) |
| Latest tag in CHANGELOG | 1.0.0 (2026-04-30) — single release |
| Last commit | b686380 readme: restyle to match minecraft-launcher format |
| Recent commits | README rewrites, AGPL switch, rename chain RackedLimbo → LoginLimbo → AuthLimbo |
| CI | .github/workflows/build.yml + release.yml (GitHub Actions, not .forgejo/) |
| Tests | None. src/test/ does not exist. |
| Source | 5 Java files: AuthLimbo, AuthMeDatabase, LimboWorldManager, LoginListener, VoidGenerator |
| Docs | docs/{compatibility,configuration,how-it-works,installation}.md |
| CHANGELOG style | Keep a Changelog + SemVer, date-suffixed ## [1.0.0] - 2026-04-30 |
| License | AGPL-3.0-or-later, SPDX header in every Java file |
Key existing detail relevant to the bug — LoginListener.java
already implements the documented Paper #4085 fix (chunk-ticket pin
in AuthMeAsyncPreLoginEvent + getChunkAtAsyncUrgently chained
with teleportAsync at MONITOR priority on LoginEvent, with
configurable authme.teleport-delay-ticks). If YOU500 still
void-died, the bug is in how that chain handled a return-value
of false / a thrown exception — the current code only logs a
warning and lets the player stay wherever they were (which on
login is the limbo void). See LoginListener.java:166-191.
The AuthLimbo audit agent's findings should land as:
docs/INCIDENT-2026-05-07-you500.md(new) — forensic root-cause doc, followdocs/REBRAND_2026-04-30.mdstyle (date-prefixed, scope/apply/result/rollback sections — convention shown below).CHANGELOG.md— bump to## [1.0.1] - 2026-05-07with### Fixedblock, follow Keep-a-Changelog format.src/main/java/ru/authlimbo/LoginListener.java— code patch. Likely changes: handlesuccess == falseandexceptionallywith a kick or retry rather than silent log; consider raising defaultteleport-delay-ticksfrom 10 → 20.src/test/(new directory) — unit tests for the listener. No precedent here, but pom.xml needs JUnit added.
minecraft-server (server repo — this repo)
| Field | Value |
|---|---|
| Origin | ssh://git@192.168.0.100:222/s8n-ru/minecraft-server.git ⚠️ stale |
| Last commit | ede6029 proantitab: allow lp/luckperms in global; deny essentials.motd default |
| Top-level docs | MISSION.md, README.md, RULES.md, THANKS.md, VIBE.md, TELEMETRY_AUDIT.md |
docs/ |
BACKUP.md, DEPLOY.md, PERMISSIONS.md, PLUGINS.md, PLUGIN_ALTERNATIVES.md, RACKED_BRAND.md, REBRAND_2026-04-30.md, ROADMAP.md, migrations/lands-to-landclaim.md, plugins/<name>.md (20 files) |
| Existing TODO | The README "Roadmap / TODO" section (lines 91-180) is the canonical living checklist. Tagged [P0] blocker / [P1] vision / [P2] improvement / [P3] nice-to-have. docs/ROADMAP.md is scoped narrowly to plugin-acquisition overhaul (Phases 1-3). |
live-server/ |
live config snapshot (purpur.yml, server.properties, ops.json, plugins/) — mirrors prod state, not a build input. |
| Backup script | scripts/backup.sh — note bug at line 119 (orphaned "${BACKUP_PATH}/synapse-signing-key-${TIMESTAMP}.key" block sits outside any if, will fail at runtime if signing-key path absent) |
| CI | .github/workflows/ is empty. .github/ISSUE_TEMPLATE/ empty. No .forgejo/. |
No existing files named AUDIT*, INCIDENT*, RUNBOOK*,
TODO*, CHANGELOG* at root or in docs/. The closest precedents:
docs/REBRAND_2026-04-30.md— date-prefixed event log w/ Apply/Side incident/Rollback sections. Use this as the format template for any new INCIDENT- doc.*docs/migrations/lands-to-landclaim.md— multi-section migration plan (Current State / Target / Plan / Rollback). Format template for future strategy docs.MISSION.md/VIBE.md/RULES.md— top-level "values" docs. Don't add new top-level capitalised md files unless the doc is similarly load-bearing for the project's identity. Detail goes indocs/.
infra (nullstone+cobblestone runbooks)
| Field | Value |
|---|---|
| Origin | ssh://git@192.168.0.100:222/veilor-org/infra.git ✅ org-scoped, no rename impact |
| Last commit | 381f923 runbook: distribute load + sync data (operator's HA vision) |
| Layout | forgejo/, runbooks/, repos/, root STATE.md + AUDIT-2026-05-05.md |
| Runbooks | COBBLESTONE-INTAKE.md, DE-DECISION-cobblestone.md, HA-CLUSTER-distribute-and-sync.md (already covers MC backup placement!), MIGRATION-nullstone-to-cobblestone.md |
Critical pre-existing context:
STATE.mdalready lists "/opt/docker/backup.shfixes — matrix-postgres + rocketchat-mongodb + literal CHANGE_ME pw" as open issue (line 97), AND lists Restic+autorestic as the #1 recommended addition (lines 113, 283-285 ofAUDIT-2026-05-05.md).runbooks/HA-CLUSTER-distribute-and-sync.mdline 51 already plans "Backups (offsite) — Restic to B2/Wasabi nightly" and line 72 pins MC to nullstone with "World data ZFS-replicated for DR only". The backup-strategy agent's plan must reconcile with this — don't propose a parallel scheme; either extend the HA runbook or cross-link it as the parent design.AUDIT-2026-05-05.mdlines 200-203 already flag the backup script as silently broken (RC + ex-Matrix not dumping). Confirms the symptom that caused YOU500's loss.
Format conventions in infra/:
- Audit reports:
# 5-Agent Audit Report — YYYY-MM-DDheader, TL;DR section, severity-ordered Action items section, file index. - Runbooks:
# Runbook — <topic>header, Goal blockquote, North-star diagram if applicable, phase plan, failure scenarios + RTO table, open decisions, related links. - Dating: filenames always
<TYPE>-YYYY-MM-DD.md.
minecraft-launcher
| Field | Value |
|---|---|
| Origin | ssh://git@192.168.0.100:222/s8n-ru/minecraft-launcher.git ⚠️ stale |
| Last commit | 31d25f8 readme: shrink license section to single sub line |
| Relevance to incident | None direct. Would only matter if the incident agent recommends a launcher-side patch (e.g. forced relog on void death detection) — unlikely. |
minecraft-client
Not a git repo (fatal: not a git repository). No remote to
worry about. Excluded from any rewrite list.
veilor-os
| Field | Value |
|---|---|
| Origin | ssh://git@192.168.0.100:222/veilor-org/veilor-os.git ✅ no rename impact |
| Relevance | None — separate brand (security distro), not Minecraft. Skipped per instructions. |
2. Stale s8n-ru origin URLs (per 2026-05-07 rename)
Per workspace memory user_git_identity.md the Forgejo user s8n-ru
was renamed to s8n on 2026-05-07. Forgejo serves a 307 redirect for
now but the canonical path is s8n/<repo>. The following local
clones still have the old origin:
| Repo (local clone) | Current origin | Should become |
|---|---|---|
_github/auth-limbo |
ssh://git@192.168.0.100:222/s8n-ru/auth-limbo.git |
ssh://git@192.168.0.100:222/s8n/auth-limbo.git |
_github/minecraft-server |
ssh://git@192.168.0.100:222/s8n-ru/minecraft-server.git |
ssh://git@192.168.0.100:222/s8n/minecraft-server.git |
_github/minecraft-launcher |
ssh://git@192.168.0.100:222/s8n-ru/minecraft-launcher.git |
ssh://git@192.168.0.100:222/s8n/minecraft-launcher.git |
No rename required for: _github/infra (veilor-org/),
_github/veilor-os (veilor-org/), _github/minecraft-client (not
a repo).
Recommended one-shot fix (deferred — not part of these four agents):
for r in auth-limbo minecraft-server minecraft-launcher; do
cd /home/admin/ai-lab/_github/$r
git remote set-url origin ssh://git@192.168.0.100:222/s8n/$r.git
done
Also update the in-doc URL references:
auth-limbo/src/main/resources/plugin.ymlline 7:website: https://github.com/s8n-ru/auth-limboauth-limbo/src/main/java/ru/authlimbo/*.javaSPDX header:Copyright (C) 2026 s8n-ruminecraft-server/VIBE.mdline 38:github.com/s8n-ru/auth-limbo
3. Overlap with session-noted TODO items
The session noted these TODOs that the four agents may want to fold into recommendations. State as of HEAD:
| Item | Existing mention? | Where | Status |
|---|---|---|---|
| SHA256 → BCRYPT (AuthMe hashing) | ✅ flagged 2026-05-02 | security/nullstone-server/2026-05-02-mc-audit.md summary: "AuthMe also uses unsalted SHA-256, no tempban, no captcha, and 5-char minimum passwords" |
Not yet addressed in repo. No TODO entry in README. New. |
| EZShop drop | ⚠️ Plugin loaded via PLUGINS: in docker-compose.yml:51 |
docker-compose.yml | No TODO entry yet. New. |
| CapDrop (Linux capabilities) | ❌ No mention | — | Net-new infra-side item (deploy.security level). Belongs in server-audit agent's report. |
| tracking-range | ❌ No mention | — | Net-new (purpur.yml tuning). New. |
| CO DB → MySQL (CoreProtect) | ❌ No mention | — | Net-new. Touches plugin policy (CoreProtect-CE is the one acknowledged license exception per MISSION.md — CO config change OK, plugin swap not). |
| TPS webhook | ⚠️ "Prometheus exporter + Grafana" entry exists in README:105 (P2). Webhook would be lighter-weight alternative. | README.md:105 | Adjacent to existing TODO; consider replacing or augmenting it. |
| spark baseline | ✅ spark already loaded in PLUGINS: (compose:54) and listed in VIBE.md:78 |
docker-compose.yml, VIBE.md | "Baseline" = capture a profiling run for ref. Net-new. |
| plugin folder cleanup | ⚠️ live-server/plugins/ is checked-in live config snapshot. Past cleanup happened in REBRAND_2026-04-30 (Side incident — disk full). |
docs/REBRAND_2026-04-30.md:65-74 | Operational, not docs. Net-new. |
None of the eight overlap with the existing docs/ROADMAP.md
(which is scoped narrowly to plugin-acquisition — manifest +
lockfile + CI). They all belong in the README.md "Roadmap / TODO"
checklist by current convention. The server-audit agent should
append them there, not create a new ROADMAP-* doc.
4. Existing backup-related mentions
| File | Line | Content |
|---|---|---|
docs/BACKUP.md |
all | Documents the daily 02:00 cron + retention. Critical drift: describes worlds being backed up but VIBE.md:54-58 says "no world backups". Direct contradiction. |
scripts/backup.sh |
80-117 | Minecraft block: docker-exec tar of world/world_nether/world_the_end + configs. Real, working code. |
scripts/backup.sh |
119-122 | Orphaned dead-code block outside any if (dangling from synapse-signing-key). Will trigger script failure if signing-key path missing. |
README.md |
23, 45, 164, 179 | Mentions backup feature. README:179 records "freed 11G+ (old backups, ...)". |
VIBE.md |
54-58 | "Daily configs, no world backups (it'd eat too much disk). If you lose a base to grief, that's the game." — conflicts with reality. |
docs/REBRAND_2026-04-30.md |
53, 65-74 | Records 2026-04-30 backup tarball and 2026-05-01 disk-full incident from accumulated backups. Confirms backups were running. |
SYSTEM.md |
737-749 | Workspace-level system reference says backups run daily, ~5-7GB compressed. Out-of-date plugin counts (says 25, actual ~16) and Purpur version (says 1.21.10, actual 1.21.11). |
Major contradiction the backup-strategy agent must resolve: either VIBE.md must drop the "no world backups" line (recommended — reality is that worlds are being backed up), or the operator must accept that the YOU500 loss happened because the worlds were logically excluded from the policy even though they were mechanically being archived. The latter is unlikely — daily 02:00 tarball would have caught a 2026-05-07 daytime void death.
Backup-hunt agent finding to verify: does /opt/backups/ on
nullstone actually contain any usable mc-world-backup-*.tar.gz
files? STATE.md line 97 + AUDIT-2026-05-05.md lines 200-203
suggest the script runs but its other arms are failing silently;
the MC arm at lines 80-117 of backup.sh has no obvious bug, so
backups should exist. If they don't, that's the deepest finding.
5. Forgejo runner / CI integration
Per memory project_forgejo_nullstone.md and STATE.md line 26-27,
nullstone runs a Forgejo runner with labels
ubuntu-24.04 + nullstone. No repo currently has a .forgejo/
directory — neither auth-limbo nor minecraft-server nor infra. CI
in auth-limbo is GitHub Actions (.github/workflows/).
STATE.md line 121-129 notes the v0.5.32 veilor-os ship is pending
on flipping runs-on: to nullstone to use the Forgejo runner.
Implication for the audit agents: if the AuthLimbo agent wants the fix to land via CI, two options:
- Keep
.github/workflows/build.yml, since GH-mirror is now manual-only post-2026-05-06 (STATE.md:14-18) — workflow won't trigger automatically anymore, would need manual mirror push. - Migrate to
.forgejo/workflows/build.ymlwithruns-on: ubuntu-24.04(compatible with the runner). Cleaner, matches new direction. Recommended.
Either path: pre-existing dependency on AUTHME_JAR_URL repo secret
(see .github/workflows/build.yml:21-26) needs to be re-added on
Forgejo if path 2 is taken.
6. Workspace-level SYSTEM.md updates needed after backup-strategy lands
/home/admin/ai-lab/SYSTEM.md lines 665-779 has the canonical
workspace-level Minecraft section. After the backup-strategy doc
lands, the following blocks need editing (one PR, one paragraph
each):
| SYSTEM.md location | Existing content | Drift |
|---|---|---|
| Line 677 | "Minecraft Version: 1.21.10 (Purpur build 2532)" | Actual: 1.21.11 (compose line 10) |
| Line 686-690 | "25 plugins loaded ... bulk-updated 2026-04-17" | Plugin set has shifted heavily since (LandClaimPlugin → Homestead, WorldEdit → FAWE, Vault → VaultUnlocked, LoginSecurity → AuthMe, AuthLimbo added, EZShop+AuctionHouse added). Real count ≈ 16. |
| Line 692-706 | RAM 7GB idle, Purpur 1.21.10-2535, startup 47s | Out of date; would-be benefit re-measure as part of "spark baseline" TODO. |
| Line 765-771 | "Known Issues" block | Add YOU500 incident closure note (post-fix), F10 RCON wildcard already promised in Wave 2. |
| Line 776 | "Backup frequency: Add 6-hourly world snapshots for active play sessions" | This is the existing wishlist item the backup-strategy agent will likely satisfy. Strike or replace with "Done — see infra/runbooks/MC-BACKUP-2026-05-07.md" (or wherever the strategy lands). |
Per CLAUDE.md workspace rules, technical detail belongs in
SYSTEM.md, not README.md. The README device-table line for
nullstone won't change.
7. Integration recommendations — where each parallel agent's doc lands
| Agent | Output should land at | Rationale |
|---|---|---|
| Backup hunt (find existing backups) | _github/minecraft-server/docs/INCIDENT-2026-05-07-you500-backup-hunt.md |
Date-prefixed, follows REBRAND_2026-04-30.md format. Forensic in nature → minecraft-server docs/. |
| AuthLimbo audit (root-cause + code patch) | (1) _github/auth-limbo/docs/INCIDENT-2026-05-07-teleportasync-failure.md for forensic write-up; (2) source patch + CHANGELOG.md bump in same repo; (3) optional cross-link from minecraft-server/docs/INCIDENT-2026-05-07-you500-backup-hunt.md |
Plugin source repo owns plugin bugs. INCIDENT- naming convention matches REBRAND_*.md. |
| Backup strategy (forward-looking design) | _github/infra/runbooks/MC-BACKUP-strategy-2026-05-07.md (or extend HA-CLUSTER-distribute-and-sync.md with a Phase 1.5 sub-section) |
infra owns nullstone-side cron + restic. Cross-link from minecraft-server/docs/BACKUP.md (replace its current contents with a thin pointer). |
| Server audit (broader hardening — CapDrop, plugin folder, MySQL, etc) | _github/minecraft-server/docs/AUDIT-2026-05-07.md (synthesis), then append individual TODOs to README.md "Roadmap / TODO" |
Matches infra/AUDIT-2026-05-05.md precedent. README is the canonical TODO surface for this repo per existing convention. |
Files needing edits AFTER all four agents finish:
| File | Change |
|---|---|
_github/minecraft-server/README.md |
Append new TODO entries from server-audit agent: SHA256→BCRYPT, EZShop drop, CapDrop, tracking-range, CO MySQL, TPS webhook, spark baseline, plugin folder cleanup. Add [x] for the YOU500 incident under "Done" once fix shipped. |
_github/minecraft-server/docs/BACKUP.md |
Rewrite to point to infra runbook; current Schedule/Strategy/Manual sections move to infra. Or replace contents with thin "see infra/runbooks/MC-BACKUP-strategy-2026-05-07.md". |
_github/minecraft-server/VIBE.md |
Drop or revise lines 54-58 — "no world backups" contradicts reality and is the philosophical claim that may have justified treating backups as low-priority. Important narrative fix. |
_github/minecraft-server/scripts/backup.sh |
Fix orphaned line 119-122 dead-code block. Independent of strategy agent's output. |
_github/minecraft-server/docker-compose.yml |
If EZShop drop accepted: remove line 51. (Server-audit agent decision.) |
_github/auth-limbo/CHANGELOG.md |
New ## [1.0.1] - 2026-05-07 entry. |
_github/auth-limbo/pom.xml |
Version bump 1.0.0 → 1.0.1 if patch shipped. |
_github/auth-limbo/src/main/java/ru/authlimbo/LoginListener.java |
Code fix per AuthLimbo agent. |
_github/infra/STATE.md |
Add 2026-05-07 changelog entry referencing the incident; check off "/opt/docker/backup.sh fixes" pending decision (line 97) when backup script repaired. |
_github/infra/AUDIT-2026-05-05.md |
Append addendum or leave dated; the new audit replaces/augments the F-numbered findings related to MC backups. |
/home/admin/ai-lab/SYSTEM.md |
Update Minecraft section per §6 above. Add note in Known Issues (line 765). Update Last Updated. |
/home/admin/ai-lab/README.md |
"Last Updated" stamp; one-line status mention if user wants it surfaced at workspace level. |
8. Open conflicts and duplications
-
VIBE.md vs reality (most important narrative conflict). VIBE says no world backups; backup.sh + BACKUP.md + REBRAND_2026-04-30 prove worlds are archived nightly. The YOU500 inventory loss means either (a) backups didn't run that day, (b) backup ran but the rollback isn't operationally feasible (would lose other players' progress between 02:00 and the death), or (c) operator chose not to rollback. The backup-strategy agent must address this explicitly rather than just propose a new scheme.
-
docs/ROADMAP.mdscope vs README "Roadmap / TODO" — the docs file is narrowly about plugin-acquisition Phases 1-3, while the README has the all-up living checklist. Future agents should not put generic TODO items intodocs/ROADMAP.md. Keep its scope tight or rename itdocs/PLUGIN-ACQUISITION-ROADMAP.md. -
infra
HA-CLUSTER-distribute-and-sync.mdvs new MC-backup strategy — there's a real risk the backup-strategy agent designs Restic-to-B2 in isolation while HA-CLUSTER already plans that exact service for both nullstone+cobblestone. Strategy doc must reference and extend the HA-CLUSTER plan (specifically the "Backups (offsite)" row in its layer table, line 51). -
CoreProtect MySQL migration — proposed in session TODOs.
MISSION.md:24codifies CoreProtect-CE as "the one acknowledged license exception". Switching its DB backend to MySQL is fine under that policy (config, not plugin swap), but the server-audit agent should explicitly note "this is a config change, not a plugin swap, so MISSION.md:24 still holds" so the policy isn't accidentally diluted. -
AuthLimbo CI host —
.github/workflows/lives in repo but GH push-mirror is off as of 2026-05-06. Builds will only run if someone manually pushes to GH. Worth flagging to the AuthLimbo agent that any CI step they propose may need a.forgejo/variant, otherwise the patched 1.0.1 release won't auto-build. -
_github/minecraft-clientis not a git repo — nothing to worry about for this incident, but anyone iterating on the incident later may try to commit something there expecting it to work. Worth recording.
9. Summary table — convention by repo
| Repo | Audit doc convention | Incident doc convention | TODO surface | CHANGELOG style |
|---|---|---|---|---|
auth-limbo |
(none yet) | (none yet — recommend docs/INCIDENT-YYYY-MM-DD-<slug>.md) |
(none — small repo) | Keep a Changelog + SemVer, ## [X.Y.Z] - YYYY-MM-DD |
minecraft-server |
(none yet — recommend docs/AUDIT-YYYY-MM-DD.md matching infra style) |
follow docs/REBRAND_2026-04-30.md template |
README "Roadmap / TODO" with [P0..P3] tags |
(none — uses git log) |
infra |
AUDIT-YYYY-MM-DD.md at root |
(use runbooks for forward-looking; no incident files yet) | STATE.md "Pending decisions" table |
(none — uses git log + STATE.md) |
minecraft-launcher |
n/a | n/a | (none) | (none) |
veilor-os |
(separate brand — out of scope) | — | — | — |
End of survey. Read-only. No files modified. No commits pushed.