Homelab Bootstrap

Homelab Bootstrap Guide

Step-by-step operational guide to bring lab-matrix (L2) and vps-pacman (L1) from zero to serving web traffic. Assumes the Nix code is already correct — this covers the manual steps Nix cannot automate. Last verified: Kanidm 1.10.x, oauth2-proxy 7.x

Architecture Recap

Two independent access paths — public internet and direct Tailscale:

Path A: Public Internet                    Path B: Tailscale Direct
───────────────                            ───────────────
Browser (anywhere)                         Laptop/Phone/PC (Tailscale on)
  │                                            │
  ▼                                            │
Cloudflare DNS (alienzj.org)                   │
  │                                            │
  ▼                                            ▼
vps-pacman (L1, public IP)                lab-matrix (L2, Tailscale IP)
  ├── Nginx (SSL wildcard *.alienzj.org)    ├── Nginx (internal)
  ├── ACME (DNS-01)                         ├── Kanidm (id.alienzj.org)
  ├── OAuth2-proxy (auth.alienzj.org)       ├── OAuth2-proxy (local)
  └── Sing-box (egress)                     ├── PostgreSQL 18
  │                                         └── Services: vaultwarden,
  │ Tailscale mesh (100.64.0.0/10)               forgejo, immich, freshrss,
  │                                               atuin, calibre, affine,
  ▼                                               stalwart, linkwarden...
lab-matrix (L2, no public IP)
  └── Nginx (kanidmAuth checks SSO locally)  id3-eniac (L2, GPU node)
        └── Backend services                   └── Nginx (internal)
                                                    └── Services: ollama, jupyter

Key point: Both paths work identically. The same SSO cookie (.alienzj.org) is valid regardless of which path you use. Services on lab-matrix have their own kanidmAuth via the L2 OAuth2-proxy, so direct Tailscale access is still SSO-protected — no dependency on vps-pacman being reachable.

L1→L2 DNS Resolution

vps-pacman’s nginx resolves backend hostnames via Tailscale’s internal DNS resolver (100.100.100.100). By default (server profile, --accept-dns disabled), the resolver only answers MagicDNS FQDN. The modules.services.web.nginx.homelabProxy.tailnetSuffix option in hosts/vps-pacman/modules/modules.nix must be set to the tailnet’s MagicDNS suffix. The alternative is to enable tailscale set --accept-dns=true on vps-pacman and leave tailnetSuffix = null.

Find your suffix: tailscale dns status → MagicDNS: enabled tailnet-wide (suffix = <value>).

ACME Certificate Architecture

Only vps-pacman (L1) runs ACME for public domains, using DNS-01 challenge with the Cloudflare API token. DNS-01 proves domain ownership via DNS TXT records, not HTTP reachability — so it works from any host with outbound API access, regardless of firewall state.

lab-matrix (L2) must NOT run ACME, even if you provision a Cloudflare API token for it. Reasons:

1. Wildcard conflict: vps-pacman already manages the *.alienzj.org wildcard cert. Having two ACME clients race to renew the same certificate triggers Let’s Encrypt rate limits.
2. No consumers: All services on lab-matrix use nginx.ssl = false — they don’t reference the wildcard cert. A cert issued on lab-matrix would be unused.
3. No need: L1→L2 traffic runs inside Tailscale WireGuard (already encrypted). Direct Tailscale access doesn’t need public TLS.

ACM enablement is a host-level decision: modules.services.web.acme.enable = true is set explicitly in hosts/vps-pacman/modules/modules.nix. No service module force-enables ACME — the previous side-effect in kanidm.nix (modules.services.web.acme.enable = mkIf cfg.nginx.ssl true) has been removed.

On lab-matrix, Kanidm uses a self-signed cert internally (localhost). All lab-matrix services are configured with nginx.ssl = false in the host config. SSL is terminated at vps-pacman for Path A (public), and the WireGuard transport layer provides encryption for Path B (Tailscale direct). See SSO & Identity Management for details.

Service Domains

Multi-Host Homelab & SSO Architecture

The homelab spans multiple L2 hosts. lab-matrix runs Kanidm (the single identity provider) and most services; id3-eniac runs GPU-accelerated services (Ollama, JupyterHub). The L1 gateway (vps-pacman) proxies to all L2 hosts over Tailscale — the homelabServices map in hosts/vps-pacman/modules/nginx.nix declares which service routes to which host.

Kanidm and OAuth2-Proxy are singletons per tailnet, not per host. Kanidm runs on lab-matrix only. OAuth2-Proxy runs on lab-matrix (for L2-local SSO on direct Tailscale access) and on vps-pacman (for the public auth.alienzj.org endpoint). Services on other L2 hosts (id3-eniac) do not need their own Kanidm or OAuth2-Proxy instances.

For services on lab-matrix: The L2 nginx applies auth_request /oauth2/auth via the local OAuth2-Proxy Unix socket. This works identically for Path A (traffic arrives from vps-pacman proxy) and Path B (direct Tailscale access).

For services on other L2 hosts (id3-eniac): Two approaches for SSO protection:

Currently, ollama and jupyter on id3-eniac are served without SSO — the vps-pacman proxyTo function does SSL termination + proxy only, and id3-eniac has no OAuth2-Proxy. To add SSO, choose the approach above.

The identity provider never moves. Regardless of which approach you pick, Kanidm stays on lab-matrix. The OAuth2-Proxy on any host talks to https://id.alienzj.org/oauth2/openid/oauth2_proxy — which resolves to vps-pacman → lab-matrix’s Kanidm over Tailscale. No host other than lab-matrix needs kanidm.enable = true.

Service Access Reference

There are three ways to reach homelab services, depending on where you are and what you’re debugging.

Path A: Public Domain (any browser, anywhere)

All services are reachable at https://<subdomain>.alienzj.org. Traffic flows through Cloudflare → vps-pacman (L1, SSL termination) → Tailscale WireGuard → lab-matrix (L2, backend).

# Test from any machine with internet access
curl -sI https://git.alienzj.org/
curl -sI https://bw.alienzj.org/

This is the production path. The *.alienzj.org wildcard Let’s Encrypt certificate is served by vps-pacman. SSO (Kanidm + OAuth2-Proxy) protects services that have oauth2.enable = true.

Path B: Tailscale Direct (any Tailscale-connected device)

From any device on the Tailscale mesh, you can bypass the public gateway entirely. Point your browser or curl at the domain — if your Tailscale DNS is configured to resolve your tailnet’s domains, it reaches lab-matrix directly over WireGuard.

# From a Tailscale-connected device
curl -sI "http://lab-matrix" -H "Host: git.alienzj.org"

SSO still works via lab-matrix’s own OAuth2-Proxy instance. No dependency on vps-pacman.

Note: OAuth2-Proxy sets cookie.secure = true, so browsers only send the SSO cookie over HTTPS. Direct HTTP access via Path B requires re-authentication. HTTPS on the Tailscale path requires a separate internal certificate or custom DNS setup.

Path C: Localhost IP+Port (on lab-matrix itself, for debugging)

Each backend service listens on a localhost port. You can hit them directly — bypassing nginx, SSL, and SSO entirely. This is the fastest path for debugging startup issues, checking health endpoints, and verifying the backend is alive independent of the proxy chain.

# On lab-matrix, test any backend directly:
curl -s http://127.0.0.1:3000/          # Forgejo
curl -sk https://127.0.0.1:8443/status  # Kanidm health check

Warning: Localhost access bypasses SSO entirely. Use only for debugging, never expose these ports publicly.

Quick Health Check Loop

# On lab-matrix: test all backends in one shot
for svc in \
  3000:Forgejo 8222:Vaultwarden 2283:Immich 8887:Atuin 8282:Calibre \
  8083:Stirling-PDF 8081:Glance 3010:Affine 3001:Linkwarden \
  28981:Paperless 8000:OnlyOffice 7002:HedgeDoc 4000:LiteLLM 9000:Mealie \
  9071:Gotify 8085:Stalwart-webadmin; do
  port="${svc%%:*}"
  name="${svc##*:}"
  code=$(curl -sI -o /dev/null -w '%{http_code}' "http://127.0.0.1:$port/" 2>/dev/null || echo "FAIL")
  echo "$name (:$port) → $code"
done
# Kanidm separately (uses HTTPS)
curl -sk https://127.0.0.1:8443/status && echo "Kanidm: OK"

0.1 SSH Key Setup for Agenix

Agenix decrypts secrets using the host’s SSH ed25519 key + a global key. Before first deploy, keys must exist on the target.

# Check what keys you have locally for the host
ls ~/.ssh/keys.homelab/server/ssh_host_ed25519_key_lab-matrix
ls ~/.ssh/keys.homelab/vps/ssh_host_ed25519_key_vps-pacman

# Generate if missing
ssh-keygen -t ed25519 -f ~/.ssh/keys.homelab/server/ssh_host_ed25519_key_lab-matrix -N '' -C 'lab-matrix.local'
ssh-keygen -t ed25519 -f ~/.ssh/keys.homelab/vps/ssh_host_ed25519_key_vps-pacman -N '' -C 'vps-pacman.local'

0.2 Push Keys to Targets

# For lab-matrix (local or reachable host)
hey ops push-keys lab-matrix lab_matrix_root --generate

# For vps-pacman (VPS)
hey ops push-keys vps-pacman vps_pacman_root --generate

This copies ssh_host_ed25519_key and global_ed25519 to /etc/ssh/ (or /persist/etc/ssh/ if persist enabled). Agenix uses these to decrypt secrets at boot.

0.3 Verify nix-secrets Repo Access

The nix-secrets flake input (in flake.nix) must be accessible:

# Check the remote URL
git -C $(nix eval --raw .#inputs.nix-secrets.outPath 2>/dev/null || echo /dev/null) remote -v 2>/dev/null

# Or check flake.lock for the input
grep -A5 '"nix-secrets"' flake.lock

If it’s a private repo, ensure your SSH agent has the right key loaded.

Phase 1: Secrets Preparation (dev machine)

Each encrypted .age file must be created with agenix -e and placed in the nix-secrets repo. The files are encrypted to the host’s SSH public key + global key.

1.1 Required Secrets — lab-matrix

1.2 Required Secrets — vps-pacman

1.3 Creating the oauth2-proxy-env.age

This is a multi-line environment file. Create it after Kanidm is running and the OIDC client exists (Phase 3), then return here.

Generate a cookie secret (32 bytes for AES-256):

python3 -c 'import os,base64; print(base64.urlsafe_b64encode(os.urandom(32)).decode())'

This MUST produce a base64-encoded key of exactly 16, 24, or 32 bytes when decoded. The example above generates 32 bytes (AES-256). The encoded string will be ~43 characters.

Common pitfall: Placeholder strings like <your-generated-cookie-secret> are 30 bytes — not a valid AES key length. This causes oauth2-proxy to crash-loop with: cookie_secret must be 16, 24, or 32 bytes to create an AES cipher, but is 30 bytes.

Create the encrypted file:

agenix -e secrets/oauth2-proxy-env.age

Paste:

OAUTH2_PROXY_CLIENT_ID=oauth2_proxy
OAUTH2_PROXY_CLIENT_SECRET=<output-from-kanidm-system-oauth2-show-basic-secret>
OAUTH2_PROXY_COOKIE_SECRET=<generated-32-byte-base64-cookie-secret>

Note: The OAUTH2_PROXY_CLIENT_ID is always oauth2_proxy — this is the client name used when creating the OIDC client in Kanidm. Do NOT set it to <your-kanidm-client-id>.

1.4 Tailscale Auth Key

Generate one reusable auth key for all NixOS hosts:

1. Go to tailscale.com/admin/settings/keys
2. Create auth key: Reusable = on, Ephemeral = off, Tags = tag:nixos
3. Encrypt the key for each host:

echo -n 'tskey-auth-k...' | agenix -e secrets/lab-matrix.tailscaleAuthkeyFile.age
echo -n 'tskey-auth-k...' | agenix -e secrets/vps-pacman.tailscaleAuthkeyFile.age

1.5 Cloudflare API Token for ACME

1. Cloudflare Dashboard → Profile → API Tokens → Create Token
2. Use template “Edit zone DNS” or create custom with:
   • Permissions: Zone → DNS → Edit
   • Zone Resources: Include → Specific zone → alienzj.org
3. Encrypt:

echo -n 'cf-token...' | agenix -e secrets/cloudflare-api-token.age

1.6 Secrets Declaration

Ensure the nix-secrets repo has secrets/secrets.nix entries for each host. Example:

# In nix-secrets/secrets/secrets.nix
{
  # lab-matrix secrets
  "lab-matrix.tailscaleAuthkeyFile.age" = { nodes = ["lab-matrix"]; };
  "stalwart-postgresql-password.age" = { nodes = ["lab-matrix"]; };
  "freshrss-postgresql-password.age" = { nodes = ["lab-matrix"]; };
  "vaultwarden-postgresql-password.age" = { nodes = ["lab-matrix"]; };
  "vaultwarden-smtp-pass.age" = { nodes = ["lab-matrix"]; };
  "immich-postgresql-password.age" = { nodes = ["lab-matrix"]; };
  "immich-smtp-pass.age" = { nodes = ["lab-matrix"]; };

  # vps-pacman secrets
  "vps-pacman.tailscaleAuthkeyFile.age" = { nodes = ["vps-pacman"]; };
  "cloudflare-api-token.age" = { nodes = ["vps-pacman"]; };

  # Both hosts share the same oauth2-proxy-env.age (encrypted for both host keys).
  # They MUST use the same cookie secret for cross-host SSO to work.
  "oauth2-proxy-env.age" = { nodes = ["lab-matrix" "vps-pacman"]; };
}

Phase 2: First Deployment

2.1 Deploy Both Hosts

# Deploy lab-matrix (local or reachable host)
hey ops deploy lab-matrix lab_matrix_root --boot

# Deploy vps-pacman (remote VPS)
hey ops deploy vps-pacman vps_pacman_root --boot

Use --boot for safety — the new system becomes the default bootloader entry but doesn’t activate yet. Reboot to activate.

2.2 After Reboot — Verify Agenix

# On lab-matrix (via SSH)
ssh lab_matrix_root "ls /run/agenix/"

# On vps-pacman
ssh vps_pacman_root "ls /run/agenix/"

If /run/agenix/ is empty or missing:

• Check that ssh_host_ed25519_key exists at /persist/etc/ssh/ or /etc/ssh/
• Verify the key matches what agenix encrypted against: ssh-keygen -lf /persist/etc/ssh/ssh_host_ed25519_key
• Re-encrypt secrets with the correct key if needed
• Check journalctl -u agenix-age-* for errors

2.3 Verify Tailscale

ssh lab_matrix_root "tailscale status"
ssh vps_pacman_root "tailscale status"

Both should show as connected with tag:nixos. Note lab-matrix’s Tailscale IP (e.g., 100.64.x.x) — you’ll need it for vps-pacman’s nginx proxy.

Phase 3: Database & Identity Provider (lab-matrix)

Start services in this exact order — dependencies flow downward.

3.1 Start PostgreSQL

ssh lab_matrix_root "systemctl start postgresql"
ssh lab_matrix_root "journalctl -u postgresql --no-pager -n 20"

Verify roles were created:

ssh lab_matrix_root "sudo -u postgres psql -c '\du'"

Expected roles: stalwart, freshrss, vaultwarden, atuin, immich, plus the postgres superuser. If any are missing, the self-healing postStart script will create them on next restart.

3.2 Start Kanidm

ssh lab_matrix_root "systemctl start kanidm"
ssh lab_matrix_root "journalctl -u kanidm --no-pager -n 30"

Look for: started successfully or listening on 127.0.0.1:8443. If you see config parse errors, the module fixes should have addressed them — rebuild if needed.

Verify Kanidm responds:

ssh lab_matrix_root "curl -k https://127.0.0.1:8443/status"
# Should output: true

3.3 Configure OIDC Client, Groups, and User (Declarative or Manual)

There are two options for setting up the OIDC client (oauth2_proxy), groups (access_*), and person account shell (alienzj). Option A is fully declarative via Nix (recommended), while Option B utilizes manual CLI commands (fallback).

Option A: Declarative Provisioning (Recommended)

1. Create/Configure Secrets in nix-secrets: Before deploying, create two secret files in your nix-secrets repository:

   • secrets/oauth2-proxy-env.age: Environment variables for oauth2-proxy.

     OAUTH2_PROXY_CLIENT_ID=oauth2_proxy
     OAUTH2_PROXY_CLIENT_SECRET=<a-strong-random-secret>
     OAUTH2_PROXY_COOKIE_SECRET=<generated-32-byte-base64>

   • secrets/kanidm-oauth2-proxy-secret.age: The basic secret used by the Kanidm provisioner on lab-matrix. Must contain the exact same <a-strong-random-secret> as defined above.

   Ensure both files are defined in nix-secrets/secrets/secrets.nix:

   "kanidm-oauth2-proxy-secret.age" = mkHostSecret ["lab-matrix"] (users ++ systems);
   "oauth2-proxy-env.age" = mkHostSecret ["id3-eniac" "lab-matrix" "vps-pacman"] (users ++ systems);

2. Deploy to lab-matrix and vps-pacman: Deploy the configuration. The declarative provisioner in kanidm.service will automatically:

   • Wait for Kanidm to come online.
   • Declaratively create the oauth2_proxy OIDC client and map scopes (openid email profile groups).
   • Declaratively create all the access_* groups (access_vaultwarden, access_immich, access_forgejo, access_paperless, access_stalwart).
   • Declaratively create the person account shell alienzj and add it to the correct groups.

3. Initialize User Credentials: Since passwords/credentials cannot be declared in Nix, you must manually trigger a password reset for the provisioned regular account:

   ssh lab_matrix_root "kanidm person credential create-reset-token alienzj --url https://127.0.0.1:8443 --accept-invalid-certs"

   Open the generated link, set your password, and proceed to Step 3.4 (Deploy and Verify).

Option B: Manual CLI Setup (Fallback)

3.3.1 Recover the idm_admin Account

Kanidm creates a built-in admin account called idm_admin (not admin). This is the only account with idm_admins group membership and thus the only one that can create OIDC clients. You MUST recover this specific account:

ssh lab_matrix_root "sudo kanidmd scripting recover-account -c /etc/kanidm/server.toml idm_admin"

This outputs a temporary password. Save it.

Why not admin? admin is a regular person account that doesn’t exist yet. Recovering “admin” either fails or creates an unprivileged account without idm_admins membership. Only idm_admin has the permissions to create OAuth2/OIDC clients.

3.3.2 Login and Create the OIDC Client

Login directly to Kanidm on localhost (bypass the public proxy chain — oauth2-proxy is broken until we configure it):

# Login as idm_admin on localhost
kanidm login --name idm_admin --url https://127.0.0.1:8443 --accept-invalid-certs
# Enter the temporary password from recover-account

Why --url https://127.0.0.1:8443 --accept-invalid-certs?

• Going through id.alienzj.org routes through vps-pacman → oauth2-proxy, which is currently crash-looping (502)
• Kanidm uses a self-signed cert for localhost — --accept-invalid-certs is needed until ACME certs are deployed
• These flags only affect WHERE the CLI tool connects; they have zero impact on Kanidm’s public configuration

Create the OAuth2 client (Kanidm 1.9+ uses kanidm system oauth2, not kanidm oidc):

# Create the client
kanidm system oauth2 create oauth2_proxy 'OAuth2 Proxy' \
  https://auth.alienzj.org/ \
  --url https://127.0.0.1:8443 --accept-invalid-certs

# Add the OAuth2 callback redirect URL
kanidm system oauth2 add-redirect-url oauth2_proxy \
  https://auth.alienzj.org/oauth2/callback \
  --url https://127.0.0.1:8443 --accept-invalid-certs

# Map scopes: openid is REQUIRED for OIDC to work
kanidm system oauth2 update-scope-map oauth2_proxy \
  idm_all_persons openid email profile groups \
  --url https://127.0.0.1:8443 --accept-invalid-certs

# Get the client secret
kanidm system oauth2 show-basic-secret oauth2_proxy \
  --url https://127.0.0.1:8443 --accept-invalid-certs

# Prefer short username format
kanidm system oauth2 prefer-short-username oauth2_proxy \
  --url https://127.0.0.1:8443 --accept-invalid-certs

Save the output of show-basic-secret — this is OAUTH2_PROXY_CLIENT_SECRET.

3.3.3 Create a Regular Person Account

Critical: idm_admin is a system administration account — it can manage OAuth2 clients and groups via CLI, but it is NOT a “person” account. It has no membership in idm_all_persons, so the OAuth2 scope map (idm_all_persons → openid email profile groups) grants it zero scopes. Attempting to log into services (immich, forgejo, etc.) as idm_admin will always show “Access Denied” with Kanidm logging:

WARN: Identity does not have access to the requested scopes
  ident: User( [email protected] )
  available_scopes: {}

You MUST create at least one regular person account for day-to-day SSO login. idm_admin is only for CLI administration (creating OAuth2 clients, managing groups, recovering accounts).

# Create a regular person account
kanidm person create <username> '<Display Name>' \
  --url https://127.0.0.1:8443 --accept-invalid-certs

# Set an email address (MANDATORY for OAuth2-Proxy)
kanidm person update <username> --mail '<email>' \
  --url https://127.0.0.1:8443 --accept-invalid-certs

# Generate a password reset token for the new account
kanidm person credential create-reset-token <username> \
  --url https://127.0.0.1:8443 --accept-invalid-certs

The email is mandatory: OAuth2-Proxy requests the email scope and requires an email claim in the ID token to create a session. Without it, the OAuth2 callback fails with Error creating session during OAuth2 callback: neither the id_token nor the profileURL set an email (HTTP 500).

The create-reset-token command outputs a reset link. Open it in a browser (replace 127.0.0.1:8443 with id.alienzj.org if accessing through the public domain) and set a password. After setting the password:

1. Log out of idm_admin from the Kanidm web UI
2. Visit any service (e.g. https://immich.alienzj.org)
3. When redirected to Kanidm, log in as your regular user (not idm_admin)
4. Approve the OAuth2 consent screen (“OAuth2 Proxy wants to access…”)
5. You’ll be redirected back to the service, now authenticated

The SSO cookie (.alienzj.org) is shared across all subdomains — subsequent services will load without re-login.

Optional: Enroll a YubiKey or TOTP via Kanidm’s web UI (Profile → security key) for MFA on the regular account.

3.3.4 Create oauth2-proxy-env.age (return to Phase 1.3)

Now that you have the client credentials, go back to Phase 1.3 and create the encrypted environment file with real values:

• OAUTH2_PROXY_CLIENT_ID = oauth2_proxy
• OAUTH2_PROXY_CLIENT_SECRET = output from show-basic-secret
• OAUTH2_PROXY_COOKIE_SECRET = generated 32-byte base64 string

Then rebuild and deploy both hosts:

hey ops deploy lab-matrix lab_matrix_root --boot
hey ops deploy vps-pacman vps_pacman_root --boot
ssh lab_matrix_root "reboot"
ssh vps_pacman_root "reboot"

After reboot, verify oauth2-proxy starts:

ssh lab_matrix_root "systemctl status oauth2-proxy --no-pager"
ssh vps_pacman_root "systemctl status oauth2-proxy --no-pager"

Both should show active (running), NOT activating auto-restart.

Phase 4: Auth Chain (both hosts)

4.1 Verify OAuth2-Proxy

# On vps-pacman (L1)
ssh vps_pacman_root "systemctl status oauth2-proxy --no-pager"

# On lab-matrix (L2)
ssh lab_matrix_root "systemctl status oauth2-proxy --no-pager"

Verify the Unix socket exists:

ssh vps_pacman_root "ls -la /run/oauth2-proxy/oauth2-proxy.sock"
ssh lab_matrix_root "ls -la /run/oauth2-proxy/oauth2-proxy.sock"

4.2 Start Nginx

# On vps-pacman (L1)
ssh vps_pacman_root "systemctl restart nginx"
ssh vps_pacman_root "journalctl -u nginx --no-pager -n 20"

# On lab-matrix (L2)
ssh lab_matrix_root "systemctl restart nginx"
ssh lab_matrix_root "journalctl -u nginx --no-pager -n 20"

Nginx on vps-pacman needs to resolve lab-matrix’s Tailscale hostname at request time (uses resolver 100.100.100.100 — Tailscale MagicDNS). Verify DNS works:

ssh vps_pacman_root "curl -sI http://127.0.0.1/ -H 'Host: id.alienzj.org'"

Phase 5: ACME Certificate (vps-pacman)

5.1 Trigger Certificate Issuance

ssh vps_pacman_root "systemctl start acme-${baseDomain}.service"
# Or if the timer handles it:
ssh vps_pacman_root "systemctl start acme-renew-${baseDomain}.timer"

Check progress:

ssh vps_pacman_root "journalctl -u acme-${baseDomain}.service --no-pager -f"

The wildcard cert covers *.alienzj.org + alienzj.org. All service vhosts on vps-pacman use useACMEHost = "alienzj.org" (pointing to this wildcard cert).

5.2 Verify SSL

curl -sI https://id.alienzj.org/ 2>&1 | head -10
curl -sI https://git.alienzj.org/ 2>&1 | head -10

Both should show HTTP/2 200 or HTTP/3 200 with a valid Let’s Encrypt certificate.

Phase 6: Start All Services (lab-matrix)

6.1 Start in Dependency Order

# These depend on PostgreSQL
ssh lab_matrix_root "systemctl start atuin"
ssh lab_matrix_root "systemctl start vaultwarden"
ssh lab_matrix_root "systemctl start freshrss-config"

# These depend on their own backends + nginx
ssh lab_matrix_root "systemctl start forgejo"
ssh lab_matrix_root "systemctl start immich"
ssh lab_matrix_root "systemctl start calibre-server"
ssh lab_matrix_root "systemctl start stalwart"

# Others
ssh lab_matrix_root "systemctl start linkwarden"
ssh lab_matrix_root "systemctl start stirling-pdf"
ssh lab_matrix_root "systemctl start affine"

6.2 Initialize Calibre Library (first time only)

If the postStart script didn’t auto-initialize (or an older build used the since-removed calibredb init):

ssh lab_matrix_root "sudo -u calibre-server calibre-debug -c \"from calibre.library.database2 import LibraryDatabase2; LibraryDatabase2('/var/lib/calibre-library')\""

calibredb init was removed in Calibre 7+. The postStart script in modules/services/docs/calibre.nix now uses calibre’s Python API via calibre-debug. If the directory is empty on startup, it auto-initializes. The command above is the manual fallback.

6.3 Verify All Services

ssh lab_matrix_root "systemctl --failed --no-pager"
# Expected: 0 loaded units listed

Spot-check service endpoints:

# Kanidm
curl -k https://127.0.0.1:8443/status

# Atuin (direct to backend)
curl -sI http://127.0.0.1:8887/

# Vaultwarden
curl -sI http://127.0.0.1:8222/

# Calibre
curl -sI http://127.0.0.1:8282/

# Forgejo
curl -sI http://127.0.0.1:3000/

Phase 7: End-to-End Verification

7.1 Public Access (through L1 → L2)

# Test each public service through the L1 gateway
for sub in id auth git bw rss books mail; do
    echo -n "$sub: "
    curl -sI "https://${sub}.alienzj.org/" 2>&1 | head -1
done

7.2 SSO Flow Test

1. Open a browser (not logged into anything)
2. Visit https://git.alienzj.org/
3. Should redirect to https://id.alienzj.org/ (Kanidm login)
4. Log in with admin credentials + YubiKey
5. Should redirect back to https://auth.alienzj.org/oauth2/callback
6. Then back to https://git.alienzj.org/ — Forgejo should load
7. Visit https://bw.alienzj.org/ — should load without re-login (SSO cookie)

7.3 Direct Tailscale Access (bypass L1)

From any Tailscale-connected device:

# Get lab-matrix's Tailscale hostname
tailscale status | grep lab-matrix

# Access services directly (SSO still works via L2's oauth2-proxy)
curl -sI "http://lab-matrix" -H "Host: git.alienzj.org"

Troubleshooting

/run/agenix/ is empty

1. Check host key exists: ls -la /persist/etc/ssh/ssh_host_ed25519_key
2. Check key fingerprint: ssh-keygen -lf /persist/etc/ssh/ssh_host_ed25519_key
3. Re-encrypt secrets against that key: agenix -e -i /path/to/host_key.pub secrets/foo.age
4. Verify agenix-age-* service: systemctl status 'agenix-age-*'

oauth2-proxy crash-loops: “cookie_secret must be 16, 24, or 32 bytes”

The oauth2-proxy-env.age contains invalid values. Check what’s decrypted:

ssh target "cat /run/agenix/oauth2-proxy-env"

If you see <your-generated-cookie-secret> or any placeholder — you haven’t created the real secret yet. Go to Phase 1.3.

If the value looks like a real base64 string but still fails, check its decoded length:

echo -n '<value>' | base64 -d | wc -c
# Must be 16, 24, or 32

oauth2-proxy crash-loops: “missing_kopid” or “502”

This happens when oauth2-proxy tries to reach Kanidm but fails. Check:

1. Kanidm is running: systemctl status kanidm
2. Kanidm is healthy: curl -k https://127.0.0.1:8443/status
3. The OIDC client exists: kanidm system oauth2 get oauth2_proxy --url https://127.0.0.1:8443 --accept-invalid-certs
4. oauth2-proxy can resolve the OIDC issuer: curl https://id.alienzj.org/oauth2/openid/oauth2_proxy/.well-known/openid-configuration

Kanidm shows “Access Denied” on OAuth2 consent screen (available_scopes: {})

This means you’re logging in as idm_admin — a system admin account that is NOT a “person” and has no membership in idm_all_persons. The OAuth2 scope map requires idm_all_persons membership, so idm_admin gets zero scopes.

Fix: Create a regular person account and log in as that user instead. See Phase 3.5.

To confirm this is the issue, check the Kanidm logs:

ssh matrix_root "journalctl -u kanidm --since '10min ago' --no-pager | grep -E 'available_scopes|NotAuthenticated'"

Expected for person accounts: available_scopes: {"email", "groups", "openid", "profile"} Symptom of this bug: available_scopes: {} with ident: User( [email protected] )

OAuth2 callback fails with “neither the id_token nor the profileURL set an email” (500)

OAuth2-Proxy requests the email scope but the person account has no email address set in Kanidm. The ID token contains no email claim, and session creation fails.

Fix: Set an email on the account:

kanidm person update <username> --mail '<email>' \
  --url https://127.0.0.1:8443 --accept-invalid-certs

Verify with kanidm person get <username> — look for mail: <email> in the output.

”NotAuthenticated” errors before the Access Denied

If the Kanidm logs show Invalid identity: NotAuthenticated before the available_scopes: {} warning, the browser visited the OAuth2 consent URL before logging into Kanidm. This is normal — log into Kanidm first (visit id.alienzj.org directly), then retry the service URL. Once logged in, the SSO cookie persists across subdomains.

Nginx fails with “host not found in upstream”

This should no longer happen (we use Unix socket upstreams). But if it does:

ssh target "nginx -t"
ssh target "journalctl -u nginx --no-pager -n 50"

Kanidm “unknown field” errors

Our module is updated for Kanidm 1.10.x. If you upgrade Kanidm, re-check the server config schema for new or removed fields.

ACME certificate not issuing

1. Verify Cloudflare API token: curl -H "Authorization: Bearer $(cat /run/agenix/cloudflare-api-token)" https://api.cloudflare.com/client/v4/user/tokens/verify
2. Check DNS propagation: dig TXT _acme-challenge.alienzj.org
3. Check ACME service: journalctl -u acme-alienzj.org.service --no-pager -n 50

ACME fails with HTTP-01 404 error on lab-matrix

If lab-matrix shows ACME failures for auth.alienzj.org or id.alienzj.org:

• These domains resolve to vps-pacman’s public IP (Cloudflare DNS), NOT lab-matrix
• HTTP-01 challenge can’t work because Let’s Encrypt hits vps-pacman
• Only vps-pacman should run ACME for public domains (using DNS-01 with Cloudflare API token)
• lab-matrix should NOT have ACME enabled for these domains — check the Nix config

Kanidm CLI: “missing_kopid” when not using —url

The kanidm CLI defaults to id.alienzj.org (from ~/.config/kanidm/config). This routes through:

CLI → DNS → vps-pacman → nginx → oauth2-proxy (broken) → 502

During bootstrap when oauth2-proxy is broken, ALWAYS use --url https://127.0.0.1:8443 --accept-invalid-certs to bypass the public proxy chain.

Kanidm CLI: “403 AccessDenied”

This means you’re logged in as a non-admin account. Only members of idm_admins or idm_oauth2_admins can manage OAuth2 clients. Verify:

kanidm group list-members idm_admins --url https://127.0.0.1:8443 --accept-invalid-certs

Expected: [email protected] (not admin).

PostgreSQL “role does not exist”

The self-healing postStart script creates roles before setting passwords. If it still fails:

ssh lab_matrix_root "sudo -u postgres psql -c \"CREATE USER stalwart\""
ssh lab_matrix_root "sudo -u postgres psql -c \"CREATE USER vaultwarden\""
# ... etc
ssh lab_matrix_root "systemctl restart postgresql"

Stalwart Post-Deploy: Domain, Account & Encryption-at-Rest

After first deploy, log into https://mail.alienzj.org as admin, then:

1. Add domain: Management → Directory → Domains → Add alienzj.org
2. Create user account: Management → Directory → Accounts → create account (e.g. [email protected])
3. (Optional) Enable encryption-at-rest: For each user account, generate a PGP keypair and upload the public key (see web-services.md for details)

Known dashboard bug: The webadmin dashboard may show “TOTAL USERS 0”, “TOTAL DOMAINS 0”, and “Not found” even after adding domains and accounts. This is a known issue with webadmin v0.1.37 — the backend data is correct, only the dashboard display is affected. Use the REST API directly to verify (curl -u admin:pass http://127.0.0.1:8085/api/principal). See web-services.md for details and the v0.16 upgrade path.

# Generate ECC Ed25519 keypair (NOT Ed448 — Stalwart's Sequoia PGP
# library rejects v5 LibrePGP packets)
gpg --full-generate-key
# Choose: ECC (sign and encrypt) → Curve 25519 → no expiration
# Name: alienzj  Email: [email protected]  Passphrase: <your choice>

# Export public key for Stalwart
gpg --export --armor [email protected]

Upload the public key in webadmin: Account → Public Keys → paste. Then enable: Account → Settings → Encryption at rest → OpenPGP → select key → AES-256.

The private key never leaves your machine. Thunderbird auto-discovers it from your GPG keyring for decryption. Only new incoming emails are encrypted — existing messages stay as-is.

Stalwart fallback admin “Incorrect username or password”

The authentication.fallback-admin config must include both username and secret:

[authentication.fallback-admin]
user = "admin"
secret = "$2b$12$..."  # bcrypt hash

If only secret is set without username, Stalwart v0.15 silently rejects the login. Generate a new bcrypt hash with:

mkpasswd -m bcrypt -R 12   # choose a strong password, then encrypt hash via agenix

Then update modules/services/net/stalwart.nix and redeploy.

Tailscale auth key expired

Generate a new key at tailscale.com/admin/settings/keys, re-encrypt, rebuild. Tagged keys (tag:nixos) have key expiry disabled by default — once connected, they stay connected permanently.

Kanidm TLS: CaUsedAsEndEntity error

Kanidm on lab-matrix uses a self-signed cert generated with CA flags, which some TLS clients reject. This is normal for the self-signed localhost cert. For real clients, use the public path through vps-pacman (valid Let’s Encrypt certs) or the Tailscale path (WireGuard encryption, no TLS needed).

Service crash-loop causing high CPU temperature and fan noise

A service that fails at startup and is configured with Restart=always (or a restart policy) will enter a crash-loop — repeatedly starting, failing, and restarting. Each restart cycle consumes CPU briefly, but at high frequency (every 1-2 seconds) the cumulative effect keeps the CPU hot (75-80°C) even though top shows low overall load.

Common causes:

• Invalid database URL (Mealie: POSTGRES_SERVER=/run/postgresql builds empty host)
• Missing environment variables (Affine: DATABASE_URL not set, Prisma uses env("DATABASE_URL") directly)
• Missing state directories (Affine: /var/lib/affine/storage not created on impermanent root)
• Port already in use (silent bind failure)

Diagnosis:

# Find crash-looping services
systemctl --failed --no-pager

# Count restart frequency
journalctl -u <service> --since '2min ago' --no-pager | grep -c 'Started\|Starting'

# Identify CPU-spiking short-lived processes
ps -eo user,pid,%cpu,args | grep -E 'node|python' | grep root

Immediate mitigation: systemctl stop <service> to stop the loop. The CPU should cool 30-40°C within 30 seconds.

Service binds IPv6-only, unreachable via nginx IPv4 proxy

Node.js services started with -H localhost may bind to [::1] (IPv6 loopback) only, while the nginx proxy_pass uses http://127.0.0.1:<port> (IPv4). With net.ipv6.bindv6only=1, IPv6 sockets don’t accept IPv4-mapped connections, making the backend unreachable.

Fix (preferred): Override the upstream module’s host option to "127.0.0.1" (explicit IPv4):

services.linkwarden = {
  host = "127.0.0.1";  # force IPv4, override nixpkgs default "localhost"
};

Then update the nginx proxyPass to http://127.0.0.1:<port>. This avoids the Node.js hostname resolution ambiguity entirely. If the upstream module doesn’t expose a host option, set the LINKWARDEN_HOST (or equivalent) via environment.

Alternative: Change the nginx proxyPass to use IPv6 loopback (http://[::1]:<port>) instead of IPv4. Less clean — keeps the ambiguity and requires [::1] bracket syntax in nginx.

Prisma ORM validation errors (Affine, other Prisma-based services)

Newer Prisma versions (env("DATABASE_URL")) read from DATABASE_URL directly, not from app-specific variants like AFFINE_DATABASE_URL. If the container image was upgraded, the env var name may have changed.

Also: Prisma’s URL parser is stricter than SQLAlchemy/other ORMs. Use postgresql:// scheme (not postgres://) and ensure the URL has a valid host component. Unix socket paths (/run/postgresql) don’t work in Prisma URLs — use localhost with trust auth instead.

Determining correct systemd unit names

NixOS generates systemd unit names that may differ from the service config key:

Find the actual unit name: systemctl list-units --type=service | grep -i <keyword>

OCI container crash-loop: Prisma P2021 “Table does not exist” (Affine)

Affine uses Prisma ORM with PostgreSQL. The ensures pattern creates the database and user, but the schema migrations (104 Prisma migrations) are NOT run automatically. The container entrypoint starts the NestJS server directly — if the database is empty (no tables), Prisma validation fails with:

ERROR  code: 'P2021', meta: { modelName: 'AppConfig', table: 'public.app_configs' }

Diagnosis:

# Check if tables exist
ssh matrix_root "sudo -u postgres psql -d affine -c '\dt'"

# Check for Prisma errors in container logs
ssh matrix_root "podman logs affine 2>&1 | tail -20"

Fix — run migrations manually inside the container:

ssh matrix_root "systemctl stop podman-affine"
ssh matrix_root "podman run --rm --network=host \
  -e DATABASE_URL='postgresql://[email protected]:5432/affine' \
  ghcr.io/toeverything/affine:stable \
  npx prisma migrate deploy"
ssh matrix_root "systemctl start podman-affine"

This applies all pending migrations to the existing database. The container will start successfully after the schema exists. This is a one-time manual step — subsequent rebuilds won’t wipe the database (it’s persisted under /var/lib).

To automate this in the future, add a preStart script to the podman service that runs migrations before starting the container, or set an environment variable if the image supports auto-migration on startup.

auth.alienzj.org returns 403 — this is normal

auth.alienzj.org is an OAuth2-Proxy endpoint, not a user-facing page. It only serves /oauth2/start (initiate login) and /oauth2/callback (receive OIDC code). Accessing the root / returns 403 Forbidden. The login screen you see during SSO flow is served by Kanidm at id.alienzj.org, not by auth.alienzj.org.

LiteLLM admin UI: use master_key URL to skip login loop

LiteLLM’s admin UI redirects /ui/ → /ui/login/ for its own authentication. With ui_master_key configured, access the dashboard directly via:

https://litellm.alienzj.org/ui/?master_key=litellm-admin-key

The master key auto-authenticates the SPA and skips the login page. Without it (or without the key parameter), the SPA loops between /ui/ and /ui/login/.