Service Audit

Service Audit & Log Scanning

Routine audit procedures for the homelab fleet. Run after each deploy and weekly.

Quick Audit (Post-Deploy)

After hey sync or reboot:

# 1. Failed units
ssh matrix_root "systemctl --failed --no-pager"

# 2. WARN+ERROR since boot, exclude known noise
ssh matrix_root 'journalctl --since "$(uptime -s)" --no-pager -p 4..3 | grep -vE "checkpoint|DNS.error|Failed.to.download|No.TLS.certificates|Blocked.IP|Configuration.build|TLS.handshake|ARC|DKIM|MTA.STS|greeting-failed|spam|matplotlib|notifyLicense|CoAuthoring|StreamingHttpResponse|url.parse|shutting.down|SIGKILL|listening|startup|shutdown" | tail -50'

# 3. High-frequency log patterns (top talkers)
ssh matrix_root 'journalctl --since "$(uptime -s)" --no-pager -p 4..3 | sed "s/^.*>://" | sort | uniq -c | sort -rn | head -15'

Weekly Deep Audit

Run every Sunday or after major changes.

1. Failed Units

ssh matrix_root "systemctl --failed --no-pager"

2. Service Restart Count

# Services that restart-loop (high count = crash-loop)
ssh matrix_root "systemctl list-units --type=service --state=running | awk '{print \$1}' | tail -n +2 | head -n -5 | xargs -I{} sh -c 'echo -n \"{}: \"; journalctl -u {} --since \"7 days ago\" | grep -c \"Started\|Starting\"' 2>/dev/null | sort -t: -k2 -rn | head -10"

3. Error/Warning Frequency Ranking

ssh matrix_root 'journalctl --since "7 days ago" --no-pager -p 4..3 | sed "s/^.*>://" | sort | uniq -c | sort -rn | head -20'

4. OAuth2-Proxy Session Health

# Failed token refreshes — spike indicates config/secrets issue
ssh matrix_root "journalctl -u oauth2-proxy --since '7 days ago' | grep -c 'Unable to refresh session'"

# Successful authentications — baseline for comparison
ssh matrix_root "journalctl -u oauth2-proxy --since '7 days ago' | grep -c 'Successfully refreshed session'"

5. PostgreSQL Health

# Connection errors — spike indicates app misconfiguration
ssh matrix_root "journalctl -u postgresql --since '7 days ago' | grep -ciE 'error|fatal'"

# Unexpected restarts — should be 0 outside planned maintenance
ssh matrix_root "journalctl -u postgresql --since '7 days ago' | grep -c 'shutdown'"

6. Nginx Rate Limiting

# Rate-limited requests — >100/week means limits too aggressive
ssh matrix_root "journalctl -u nginx --since '7 days ago' | grep -c 'limiting requests'"

7. Agenix Secret Health

# Check all secrets decrypt correctly
ssh matrix_root "ls /run/agenix/ | wc -l && ls -la /run/agenix/ | grep -E '^-' | wc -l"

# Empty secrets (0-byte files = decryption failure)
ssh matrix_root "find /run/agenix/ -type f -size 0 2>/dev/null"

8. DNS Resolution

# DNS failures during boot — persistent failures indicate network issue
ssh matrix_root "journalctl --since '7 days ago' | grep -c 'Temporary failure in name resolution'"

Audit Findings Log

2026-05-28 — Post-reboot Deep Scan

Context: Lab-matrix rebooted for litellm/ollama module updates. Full log scan across all 20+ services.

Scan commands used:

# Per-service status + tail logs
ssh matrix_root "systemctl status <unit> --no-pager"
ssh matrix_root "journalctl -u <unit> --no-pager -n 20"

# Global WARN+ERROR scan
ssh matrix_root 'journalctl --since "2026-05-28 12:00" --no-pager -p 4..3'

# Frequency ranking (deduplicated)
journalctl ... | sed 's/^.*>://' | sort | uniq -c | sort -rn

Findings and resolutions:

#	Severity	Service	Symptom	Root Cause	Fix
1	CRITICAL	Forgejo	`ensures` script: `Permission denied` reading DB password; password cleared	`forgejo-database-secret` had `owner = "git"`, but PostgreSQL post-start runs as `postgres`	Changed `owner = "postgres"` in `modules/services/git/forgejo.nix`
2	WARNING	Nginx	105+ `limiting requests, excess: 20.xxx by zone "oauth2_auth"` — Immich thumbnails throttled	`oauth2_auth` zone rate was `10r/s`; Immich service-worker prefetches many thumbnails simultaneously	Raised to `50r/s` in `modules/services/web/nginx.nix`
3	WARNING	Calibre	84 `Failed to send all data: Connection reset by peer` every 5 min	Gatus health check uses `GET` on `http://127.0.0.1:8282`, downloads 3.4MB response, then disconnects	Changed to `method = "HEAD"` in `modules/services/monitoring/gatus.nix`
4	WARNING	Litellm	Crash-loop: `Unable to find Prisma binaries` on startup	Prisma 7.x rejects bundled schema format; no compatible prisma 5.x in nixpkgs. preStart ran in read-only Nix store.	Removed PostgreSQL dependency, DATABASE_URL, and preStart. Litellm runs as pure proxy. Documented in module.
5	WARNING	Ollama/Open-WebUI	Admin password hardcoded in Nix store	`WEBUI_ADMIN_PASSWORD` was inline in environment	Moved to agenix `environmentFile`; added `ENABLE_SIGNUP=false`
7	WARNING	Open-WebUI	Admin login deadlock: `WEBUI_ADMIN_*` env vars ignored after DB exists	Open-WebUI only creates admin on first startup (empty DB). If the DB was initialized without env vars, subsequent restarts won’t create the admin, and `ENABLE_SIGNUP=false` blocks web registration	Delete `/var/lib/open-webui/data/webui.db` and restart. Documented recovery in `docs/ai-ml.md`. Module split into `modules/services/ai/open-webui.nix`.
6	WARNING	Stalwart	Dashboard shows “TOTAL USERS 0”, “TOTAL DOMAINS 0”, “Not found”	webadmin SPA (0.1.37) API paths incompatible with Stalwart 0.15.5; webadmin repo archived, replaced by webui in v0.16	Documented in web-services.md and homelab-bootstrap.md; REST API workaround; upgrade to v0.16 when available in nixpkgs

Known noise (no fix needed):

Pattern	Source	Why
Kanidm `BadCertificate` every 5 min	Gatus health check with `insecure: true`	Expected — self-signed cert on localhost
Stalwart `No TLS certificates`	lab-matrix doesn’t run ACME	Only vps-pacman issues certs; Tailscale WireGuard encrypts L1→L2
OAuth2-proxy `invalid_grant` spikes	Session expired after reboot	Normal — re-login clears it
Tailscale DNS `no fallback candidates`	Boot before DNS ready	Transient, <30s
Mealie `Secrets directory '/run/secrets'`	Docker convention	Harmless, no impact
DMS/EasyEffects Qt warnings	Desktop shell	Not web services

2026-05-30 — Port Conflict Cascade (Jitsi Meet)

Context: Adding Jitsi Meet to lab-matrix introduced 4 port conflicts that silently broke sing-box, jicofo, and stalwart.