infra/AUDIT-2026-05-05.md

# 5-Agent Audit Report — 2026-05-05

Synthesis of 5 parallel agents covering: GitHub→Forgejo migration,
ai-lab structure, nullstone services, stack rating, recommended
additions.

Source agent outputs:
1. Migration agent → `nullstone-server/forgejo/migration-report-2026-05-05.md`
2. ai-lab structural audit
3. nullstone services + deployment audit
4. Stack rating (10 axes)
5. Recommended service additions

---

## TL;DR

- **GH → Forgejo migration: complete.** 6/6 repos mirrored
  (5× s8n-ru/* + veilor-org/veilor-os). All HEADs match, branches
  match, tags match, push-mirrors back to GH all green. Repaired one
  default-branch metadata drift on `s8n-ru/x`. Zero failures.
- **Stack rating: 7/10.** Above-average self-hosted setup. Audit
  discipline + identity/CA story unusually strong. Fragile on
  monitoring + offsite backup + single-host.
- **Top 5 weaknesses (severity-ordered):** F4 no LUKS on nullstone
  (regression), no monitoring/alerting, backups local-only with
  silently broken script, `:latest` floats on most stacks, single
  point of failure (nullstone + home WAN).
- **Top 5 services to add (priority):** Restic+autorestic, Vaultwarden,
  Gatus, CrowdSec, Beszel.
- **Top 4 anti-recommendations:** Nextcloud, full LGTM stack, Mastodon,
  HashiCorp Vault.

---

## 1 — GitHub repo migration

**Status: complete.** Per migration agent's report.

- 6 repos enumerated under `s8n-ru` user + admin'd orgs.
- 6 mirrored to `git.s8n.ru` (Forgejo); 5 fresh, 1 already pre-migrated
  (`veilor-org/veilor-os`).
- HEADs / branches / tags match GH for all 6.
- Push-mirrors Forgejo → GH configured (8h interval + sync-on-commit),
  all green.
- One repair: `s8n-ru/x` default branch was stuck on
  `KisaragiEffective-patch-1` from Misskey upstream; PATCHed to
  `master`.

Detail: `nullstone-server/forgejo/migration-report-2026-05-05.md`.

---

## 2 — ai-lab structural audit

### Devices

| codename | type | OS | role |
|---|---|---|---|
| onyx | laptop | Fedora 43 KDE | Dev workstation (DHCP `.28`, registry says `.6` — drift) |
| nullstone | server | Debian 13 | Infra host — Docker stack, mesh, Matrix/Misskey/RC |
| office | workstation | Fedora 43 KDE (pending install since 2026-04-19) | Office/sales (.5) |

External: friend PC `100.64.0.3` (RTX 4080, vLLM in WSL2).

### Active projects (`_github/`)

| repo | purpose | status |
|---|---|---|
| veilor-os | Hardened Fedora 43 KDE remix | actively iterating, BlueBuild + kickstart |
| auth-limbo | Paper plugin (racked.ru AuthMe fix) | active, released jars |
| minecraft-launcher | Custom MC launcher (PrismLauncher fork) | active, v1 build script |
| minecraft-server | Purpur MC at `mc.racked.ru:25565` | live in prod |
| minecraft-client | racked.ru MC client (FO 11.3.2 fork) | active |

### Per-device security audit cadence

| device | last audit | folder |
|---|---|---|
| nullstone | 2026-05-05 (ACL hardening); full 2026-05-02 | `security/nullstone-server/` (9 reports) |
| onyx | 2026-04-15 | `security/onyx-laptop/` (2 reports) |
| office | never | `security/office-workstation/` (empty) |

### Memory record (31 files, 1 index)

- 2 user, 7 feedback, 1 reference, 21 project memos.
- Top-active: matrix_veilor, txt_cinny, x_misskey_fork, tailscale_mesh,
  friend_gpu, org_charter, brand_separation, simplex_org_chat.

### What this lab is

The operator runs a small home-lab/3-member CTO-style org
(`P M=CTO, nullstone=Runtime Owner, onyx-ai=Research/Review`) split
cleanly across **two brands** (per `project_brand_separation.md`):

1. **racked.ru** — privacy-first Minecraft platform (MC server +
   client + custom launcher + AuthLimbo plugin)
2. **veilor** — security company stack (veilor-os hardened Fedora
   ISO, veilor-server-bootstrap Debian preseed, Matrix at veilor.uk,
   Misskey-fork at x.veilor)

All self-hosted on nullstone behind Traefik+Headscale+Pi-hole. Mesh
includes friend's RTX 4080 for remote LLM inference via Tailscale.

### Drift / gaps

- `office-workstation/` registered in CLAUDE.md but install pending
  since 2026-04-19; no audit folder populated.
- README onyx IP `.6` vs actual DHCP `.28`.
- README folder tree doesn't match real repo (lists `_project_code/`
  + `scripts/`; reality has `_github/`, `_projects/`, `_archive/`,
  `archive/`, `github/`, several `.sync-conflict-*` files, 30 MB
  binary `re` at root).
- Two parallel `nullstone-server/` and `server/` device folders —
  drift candidate.
- `MEMORY.md` index missing entry for `project_forgejo_nullstone.md`
  (file present, index not updated).
- Sync-conflict files for CLAUDE.md / README.md / SYSTEM.md from
  Syncthing merge never resolved.
- SYSTEM.md still mentions Jitsi/coturn / MAS Element X test
  retired per project_matrix_veilor.md — TODO list not pruned.

---

## 3 — nullstone services + deployment audit

### Hardware

- **CPU:** AMD Ryzen 5 2600X (6c/12t)
- **RAM:** 32 GiB (15 used, 15 free, 24 GiB swap, 256 KiB used)
- **GPU:** GTX 1660 Ti 6 GB (Ollama)
- **Disk:** 477 GiB NVMe, LVM (`keystone-vg`):
  - root 30 G (35% used)
  - var 12 G (15%)
  - **home 399 G (60%, 227 G used / 153 G free)** — watch growth
  - tmp 2.7 G, swap 24 G
- **OS:** Debian 13, kernel 6.12.85+deb13
- **Docker:** v29.4.2, overlay2, **userns-remap=default**,
  live-restore=true, icc=false, no-new-privileges=true. Data root
  symlinked `/var/lib/docker → /home/user/docker-data`.

### Active services (28 containers)

Including: traefik, socket-proxy, authentik (server+worker+pg+redis),
forgejo + forgejo-runner, misskey + db + redis, x-source nginx,
rocketchat + mongodb, tuwunel + tuwunel-txt, cinny-txt, commet-web,
signup-page + signup-txt, livekit + lk-jwt-service, dl-veilor, pihole,
headscale, n8n + postgres, step-ca, filebrowser-mc, minecraft-mc,
anythingllm, plus 2 stale `alpine:3` shells from userns-host bypass.

### Domain → service map (all on `*.s8n.ru` or `*.veilor[.uk]`)

`sys.s8n.ru` (traefik dash), `git.s8n.ru` (forgejo, NEW), `auth.s8n.ru`
(authentik), `pihole.s8n.ru`, `signup.txt.s8n.ru`, `hs.s8n.ru`
(headscale), `rc.s8n.ru` (rocketchat), `n8n.s8n.ru`, `txt.s8n.ru`
(cinny), `mx.s8n.ru` (tuwunel-txt), `x.veilor` (misskey),
`matrix.veilor.uk`, `chat.veilor.uk` (commet), `livekit.veilor.uk`,
`signup.veilor.uk`, `dl.veilor.org`.

### Deployment patterns

- Compose: `/opt/docker/<svc>/docker-compose.yml`
- Data: named docker volumes under
  `/home/user/docker-data/100000.100000/volumes/` + per-service
  bind mounts. Newer services (forgejo, forgejo-runner, minecraft)
  on `/home/docker/<svc>/` to dodge 30 G root.
- userns-remap quirk: container UIDs shifted +100000.
  Workaround: alpine root container or chown to 101000.
- Docker socket exposure: traefik does NOT mount docker.sock; goes
  via tecnativa/docker-socket-proxy on socket-proxy-net.
- Networks: `proxy` + `socket-proxy-net` + `misskey-frontend` +
  per-stack internals (authentik-internal, misskey-internal, etc.).
- Middleware chain: `trusted-only@file → security-headers@file
  → rate-limit@file → <service-specific>` with `no-guest@file`
  for routers needing tailnet+LAN but blocking public.

### Auth patterns

- **Authentik (auth.s8n.ru)** — central OIDC, all 4 components healthy.
  **Currently mostly unwired.** Forgejo runs native auth (no OAUTH
  section in app.ini). RC, n8n, anythingllm, filebrowser likely
  local-auth too. Authentik present but underused.
- **Forgejo** — local users + PAT, admin `s8n-ru`, SSH 222.
- **Headscale** — preauthkey enrollment + `headscale-deny-leaks@file`.
- **Traefik dashboard** — basicauth + trusted-only@file.

### Backup state

- `/etc/cron.d/docker-backup` runs `/opt/docker/backup.sh` at 02:00
  daily, 7-day rotation to `/opt/backups/`.
- **Script silently broken (HIGH):** matrix-postgres container is
  gone (Synapse retired); rocketchat-mongodb name mismatch (script
  expects `mongodb`); Mongo password reads literal
  `CHANGE_ME_MONGO_ADMIN_PASSWORD`. So Rocket.Chat + (former) Matrix
  dumps **not happening**. Misskey side-script works.
- **No off-host replication.** Single NVMe = total loss on disk
  failure.

### Drift / risk register

- 🔴 Backup script broken (RC + ex-Matrix not dumping)
- 🔴 `anythingllm` listens 0.0.0.0:3001 with no traefik label,
  bypasses entire L7 trust model. Either bind LAN-only or front via
  traefik.
- 🟠 Resource limits: only minecraft-mc has memory/CPU limits.
  30 other containers unbounded — runaway can OOM-kill neighbours.
- 🟠 No service-level health checks on ~half the containers.
- 🔴 `no-guest@file` IPAllowList stub: declares only
  `sourceRange: ["127.0.0.0/8"]`. Routers chained with `no-guest`
  reject everything except loopback unless XFF restores client IP.
  **Verify** entryPoint forwardedHeaders.trustedIPs + middleware
  ipStrategy.depth — misconfig either 403s real users or accepts
  spoofed XFF.
- 🟡 office (100.64.0.4) not in `trusted-only@file` despite
  `tag:infra` per SYSTEM.md.
- 🟠 RocketChat: first-admin setup still pending — wizard endpoint
  takeover risk until claimed.
- 🟡 Stale `alpine:3` shell containers (userns-host bypass leftovers).
  `docker rm -f` after each one-shot.
- 🟡 Archived compose dirs (`pocket-id.archived-*`, `matrix-old`)
  contain secrets — move off docker tree.
- 🟡 `/home` 60% with growing volumes (Ollama, mongo, postgres ×3).
  No quotas.

### Mem pressure: none right now

Top consumer minecraft 9.35 / 18 GiB cap (52% of cap, ~30% host).
All others < 2.2%. Headroom good.

---

## 4 — Stack rating (10 axes)

| Axis | Score | Top weakness |
|---|---|---|
| Architectural coherence | 8 | Drift artifacts (sync-conflict files, parallel `_archive`/`archive`) |
| Security posture | 7 | F4 no LUKS on server (regression); F30 ip_forward=1; F12 partial revert |
| Reproducibility | 6 | Most stacks on `:latest`; no IaC; admin bootstrap uncoded |
| Operational maturity | **4** | **No metrics/alerts; backups untested; on-call="user reads logs"** |
| Cost discipline | 9 | Single residential ISP + single home server is "cheap because fragile" |
| Threat model clarity | 6 | No written THREAT_MODEL.md; AGPL §13 source endpoint deferred |
| Update hygiene | 5 | `:latest` floats; no staged rollout; recovery = "edit compose, restart" |
| Documentation quality | 8 | SYSTEM.md is 979 lines; CV + team-msg.txt + sync-conflicts in repo root |
| Network resilience | 5 | Single residential WAN; control + data plane same box; no Tor/SOCKS fallback |
| Branding/product discipline | 9 | "X" rebrand close to veilor — easy to confuse in logs/docs |

### Overall: **7/10**

Above-average self-hosted stack. Better-documented than 90% of
homelabs, with audit discipline most small SaaS shops don't reach,
and a coherent identity/CA story (own root CA via step-ca, own VPN
control plane via Headscale, own Matrix homeserver). Loses points on
operational maturity (no monitoring, no offsite/tested backups, no
rollback), one critical regression (no LUKS on nullstone), and
inherent fragility from single-host single-ISP design.

The gap between **known weaknesses** and **fixed weaknesses** is the
limiting factor — operator clearly *can* fix these (audit closes 27/35
findings in 3 days), they just haven't yet.

### Comparison

- vs **Stock Fedora desktop + GitHub:** wins decisively (8 vs 3) on
  network/identity/AGPL discipline.
- vs **secureblue + GH Actions:** stronger on server-side sovereignty;
  weaker on client posture and CI. Roughly tied overall, different axes.
- vs **Hetzner-VPS hobbyist stack:** loses on resilience + update
  hygiene, wins on cost + GPU inference + identity depth. This stack
  more ambitious; Hetzner more boring-and-reliable.
- vs **Cloudflare/Workers managed:** wins on sovereignty + GPU + Matrix
  ability. Loses on uptime + DDoS + zero-patching. This stack's whole
  reason to exist is the inverse tradeoff — and it makes that tradeoff
  coherently.

---

## 5 — Recommended service additions

### Top 5 priority (deploy in this order)

| # | Service | Why now | Effort | Maintenance |
|---|---|---|---|---|
| 1 | **Restic + autorestic** | Single biggest gap. nullstone NVMe failure = total loss right now. Encrypted incremental to B2/Wasabi or to onyx. | M | S |
| 2 | **Vaultwarden** | N services with N storage methods for secrets. Centralize before count grows. | S | S |
| 3 | **Gatus** | Otherwise you find out about a downed service from a friend on Matrix. Cert-expiry alone catches the silent killer. Alerts via Tuwunel webhook or ntfy. | S | S |
| 4 | **CrowdSec** | Pi-hole only sees DNS layer. Public Matrix fed candidates + RC + Misskey + signup pages = HTTP attack surface. Bouncer plugin blocks at Traefik. | M | S |
| 5 | **Beszel** | Once Restic is filling disk + CrowdSec flagging IPs, you want one dashboard. | S | S |

### Anti-recommendations

| Service | Why NOT |
|---|---|
| **Nextcloud** | Heavy (1.5 GB+ RAM idle), notorious upgrade pain. Use Seafile if you need files. |
| **Full LGTM stack** (Grafana+Prom+Loki+Alertmanager) | Five services to do what Beszel+Gatus do for solo-op. |
| **Mastodon** | You already run Misskey-fork. Federating two ActivityPub silos doubles moderation. |
| **HashiCorp Vault** | Complexity-to-benefit ratio terrible for one operator. Infisical or pass-with-git enough. |
| **Authelia** | Duplicates Authentik. Pick one. |

### Consolidation suggestions

- **Cinny + various Element/Commet forks:** pick **one** web client
  per Matrix instance. Each fork = separate audit + CSP + branding burden.
- **n8n:** if only used for 2-3 simple flows, replace with shell
  scripts in Forgejo Actions cron. n8n's value is the GUI for
  non-coders; you're a coder.
- **Step-CA + Let's Encrypt:** confirm zero overlap. If step-ca only
  issues one cert, kill it.
- **dl-veilor + signup pages:** if static, fold into single Caddy
  container behind Traefik. Two containers for static HTML is two
  too many.

### Other notable picks (lower priority)

- **Seafile CE** — file sync (much lighter than Nextcloud)
- **Karakeep** (formerly Hoarder) — bookmarks/RSS/read-later, AI tags
  via your local Ollama / friend RTX 4080
- **ntfy** — formalize the push-notification target you're already
  using ad-hoc
- **Forgejo Packages** — already implicit, just enable for container
  registry + npm/cargo/maven/generic

---

## 6 — Action items (severity-ordered)

### Ship-blocking (do this week)

1. **Fix `/opt/docker/backup.sh`** — remove dead matrix-postgres,
   correct rocketchat-mongodb container name, replace literal
   `CHANGE_ME_MONGO_ADMIN_PASSWORD`. Verify next 02:00 run produces
   non-zero RC + Mongo dumps.
2. **Bind anythingllm to LAN-only** OR add traefik front with
   `no-guest@file`. Currently public on :3001.
3. **Verify `no-guest@file` ACL** — confirm `sourceRange` covers
   LAN + tailnet, not just loopback. Verify XFF chain restores
   real client IP.
4. **Claim RocketChat first-admin** — takeover risk until then.
5. **Enable LUKS2 on nullstone** (F4 regression) — schedule reinstall
   window with TPM2 unlock; or until then, LUKS-on-file loopback
   for step-ca root key + acme.json + Mongo keyfile.

### High-value next (do this month)

6. Deploy **Restic + autorestic** with B2/Wasabi target + restore drill.
7. Deploy **Vaultwarden** + migrate secrets out of `.env` files.
8. Deploy **Gatus** with cert-expiry checks + Matrix/ntfy alerts.
9. Resolve **sync-conflict files** at ai-lab repo root.
10. **Pin docker images by digest** for critical stacks (already done
    for Misskey; do tuwunel/livekit/cinny/pihole/RC/Traefik next).

### Defer / planned

- Office workstation install + first audit
- Fold dl-veilor + signup pages into single Caddy
- Replace n8n with Forgejo Actions cron (if usage <5 flows)
- Move Headscale + step-ca to $4/mo VPS for SPOF mitigation

---

## 7 — File index

| Output | Path |
|---|---|
| This synthesis | `~/ai-lab/nullstone-server/audit-report-2026-05-05.md` |
| Migration detail | `~/ai-lab/nullstone-server/forgejo/migration-report-2026-05-05.md` |
| Forgejo runbook | `~/ai-lab/nullstone-server/forgejo/deploy-runbook.md` |
| Forgejo memory | `~/.claude/projects/-home-admin-ai-lab/memory/project_forgejo_nullstone.md` |
| veilor-os strategy | `~/ai-lab/_github/veilor-os/docs/STRATEGY.md` |
| veilor-os roadmap | `~/ai-lab/_github/veilor-os/docs/ROADMAP.md` |
| veilor-os threat model | `~/ai-lab/_github/veilor-os/docs/THREAT-MODEL.md` |