Web Services

Web Services & Databases

Modern, high-performance web architecture with automated certificate management and robust database backends.

Nginx Reverse Proxy

Our web architecture uses Nginx (specifically the nginxQuic fork) as a centralized reverse proxy and SSL terminator.

Core Features

• Protocols: Native support for HTTP/3 (QUIC) and kTLS (Kernel TLS acceleration) for maximum performance.
• SSL/TLS: Automated certificate management via ACME (Let’s Encrypt) using DNS-01 challenges (Cloudflare).
  • Wildcard Cert: A single *.alienzj.org wildcard certificate covers all subdomains. Defined in security.acme.certs with dnsProvider = "cloudflare" and webroot = null (DNS-01 and HTTP-01 are mutually exclusive; explicit null prevents conflict on nixpkgs 25.11+ where webroot defaults to a non-null value).
  • useACMEWildcardHost: All service vhosts set useACMEWildcardHost = true (not enableACME = true) to share the single wildcard cert. enableACME would trigger per-vhost HTTP-01 challenges, which fail on lab-matrix because its domains resolve to vps-pacman’s public IP. The useACMEWildcardHost option is added to all vhosts via modules/services/web/acme.nix using the NixOS submodule extension pattern (mkForce useACMEHost). See nix-expressions.md for the full pattern.
• Per-vhost OAuth2: modules/services/auth/oauth2-proxy.nix independently extends the same vhosts with oauth2.enable + oauth2.allowedGroups — NixOS merges submodule declarations from both modules transparently.
• Security:
  • Sinkhole: A default “deny-by-default” virtual host drops all requests using unknown hostnames or raw IPs (HTTP 444).
  • Tailscale Real-IP: Automatically restores original client IPs from Tailscale proxy headers.
• 0-RTT: Supports TLS 1.3 Early Data to reduce connection latency.
• Upstream Connection Pooling: Upstream blocks use zone <name> 64k and keepalive 2 for shared memory connection pooling (oddlama pattern).
• Security Headers: All vhosts include X-Frame-Options, X-Content-Type-Options, Referrer-Policy, and X-Permitted-Cross-Domain-Policies via genNginxVhostBase in lib/nginx.nix.
• AI Scraper Blocking: 30+ known AI scraper User-Agent patterns are blocked (return 444) at the server level.
• Rate Limiting: 50r/s + burst 50 per IP (general), 50r/s + burst 20 (OAuth2 auth), 50 concurrent connections per IP.
• Proxy Timeouts: proxyTimeout = 300s (5 min) globally. Streaming/WebSocket services (Open-WebUI, Immich) should override via locationExtraConfig for longer completions (proxy_read_timeout 1800s).

OAuth2-Proxy Upstream (Unix Socket)

The oauth2-proxy upstream uses a Unix domain socket instead of TCP:

upstreams.oauth2-proxy = {
    servers."unix:/run/oauth2-proxy/oauth2-proxy.sock" = {};
    extraConfig = ''
        zone oauth2-proxy 64k;
        keepalive 2;
    '';
};

Unix sockets are more secure than TCP localhost (no network stack exposure) and avoid port conflicts. The zone directive enables shared memory connection pooling across nginx worker processes. Unix socket upstreams do NOT fail nginx startup — the socket file is only checked at request time, not config load time.

Deployment Architecture

We employ a “Two-Layer Gateway” architecture:

1. L1 — Edge VPS (vps-pacman): Public SSL termination, ACME wildcard certs, OAuth2-proxy SSO, and reverse proxy to L2 over Tailscale. Uses runtime DNS resolution (resolver 100.100.100.100) to resolve homelab hostnames via Tailscale MagicDNS at request time.
2. L2 — Homelab Nodes (lab-matrix): Run actual services. Nginx receives traffic from L1 over Tailscale tunnels and routes to local backends. Also runs its own OAuth2-proxy for SSO on locally-served services.
3. Internal SSL: Optional “Double TLS” can be enabled for strict environments, though Tailscale’s native WireGuard encryption is the default transport.

SSL and ACME: Host-Level Constraint

nginx.ssl must match the host’s ACME status:

The chain that breaks if ssl = true on a non-ACME host:

nginx.ssl = true
  → genNginxVhostBase sets useACMEWildcardHost = true
  → acme.nix submodule sets useACMEHost = mkForce baseDomain
  → nixpkgs references security.acme.certs.<baseDomain>
  → Failed assertions: acceptTerms not set, no dnsProvider, cert unreadable

Rule: Every service on lab-matrix MUST explicitly set nginx.ssl = false. Even new services added via enable = true need this — module defaults are ssl = true (safe for L1, wrong for L2). If a deploy fails with ACME assertion errors, a service is missing nginx.ssl = false.

Why ACME only on vps-pacman: See sso-identity.md for the full rationale — wildcard cert conflict, no consumers on L2, Tailscale already encrypts.

L1 → L2 Proxy Pattern (vps-pacman)

The L1 gateway uses a declarative service map to auto-generate all proxy vhosts:

homelabServices = {
    id = "lab-matrix";     # Kanidm identity provider
    git = "lab-matrix";    # Forgejo
    bw = "lab-matrix";     # Vaultwarden
    rss = "lab-matrix";    # FreshRSS
    # ... all public services mapped here
};

Each entry generates a vhost at <name>.<baseDomain> that proxies to the named host over Tailscale. Uses variable-based proxy_pass to defer DNS resolution to request time, preventing nginx startup failures if Tailscale isn’t ready yet.

Vhost Boilerplate Reduction: genNginxVhost and genNginxVhostBase

lib/nginx.nix provides two-tier helpers:

genNginxVhost — for standard reverse-proxy services. Generates SSL, real_ip, QUIC/kTLS, oauth2, and a default "/" location with proxyPass + Alt-Svc header:

services.nginx.virtualHosts = mkIf cfg.nginx.enable {
  "${cfg.domain}" = genNginxVhost {
    domain = cfg.domain;
    ssl = cfg.nginx.ssl;
    proxyPass = "http://127.0.0.1:${toString cfg.port}";
    proxyWebsockets = true;
    quic = config.modules.services.web.nginx.quic;
    kTLS = config.modules.services.web.nginx.kTLS;
    extraConfig = "access_log /var/log/nginx/${cfg.domain}.access.log;";
    locationExtraConfig = "client_max_body_size 128M;";
    oauth2 = cfg.oauth2.enable;
    locations = { "/extra" = { proxyPass = "..."; }; };
  };
};

genNginxVhostBase — for services that don’t use proxyPass (PHP-FPM, fcgiwrap) or need custom location logic (Kanidm’s proxy_ssl_verify off, OAuth2-Proxy’s upstream alias). Provides the same boilerplate (SSL, real_ip, QUIC/kTLS, oauth2) but leaves locations entirely to the caller:

services.nginx.virtualHosts = mkIf cfg.nginx.enable {
  "${cfg.domain}" = genNginxVhostBase {
    ssl = cfg.nginx.ssl;
    quic = config.modules.services.web.nginx.quic;
    kTLS = config.modules.services.web.nginx.kTLS;
    oauth2 = cfg.oauth2.enable;
    locations."/" = {
      proxyPass = "https://127.0.0.1:8443";
      extraConfig = "proxy_ssl_verify off;";
    };
  };
};

This replaces ~15 lines of boilerplate (forceSSL, useACMEWildcardHost, quic, http3, kTLS, real_ip extraConfig, Alt-Svc header) per module.

Nixpkgs Nginx Module Internals — What AI Agents Must Know

When setting up a web service behind nginx, you must understand how nixpkgs’ module generates config — otherwise you’ll produce duplicate directives that break nginx -t at startup.

The Three Config Layers

services.nginx.recommendedProxySettings = true (set globally in modules/services/web/nginx.nix) emits these directives at the HTTP block level:

http {
    proxy_redirect          off;
    proxy_connect_timeout   300s;
    proxy_send_timeout      300s;
    proxy_read_timeout      300s;
    proxy_http_version      1.1;          # ← HTTP-level DEFAULT
    proxy_set_header        "Connection" "";
    include recommendedProxyConfig;       # Host, X-Real-IP, X-Forwarded-For/Proto/Host/Server
}

proxyWebsockets = true emits at the location block level:

location / {
    proxy_http_version 1.1;              # overrides HTTP-level default
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
}

When a location has proxyPass != null AND recommendedProxySettings = true, an additional include recommendedProxyConfig; is appended at the location level (only proxy_set_header lines, no proxy_http_version).

Location Block Emission Order

Generated by nixpkgs at nixos/modules/services/web-servers/nginx/default.nix lines 510-548, in this exact order:

1. proxy_pass <target>;
2. proxyWebsockets → proxy_http_version, Upgrade, Connection
3. extraConfig                     ← YOUR CONTENT HERE
4. include recommendedProxyConfig  ← proxy_set_header Host, X-Forwarded-*, etc.

The Critical Rule: Never Duplicate in extraConfig

proxyWebsockets = true already provides all three WebSocket headers. Do NOT add these to extraConfig:

# ❌ WRONG — duplicates proxyWebsockets output → nginx -t fails
extraConfig = ''
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
'';

# ✅ CORRECT — proxyWebsockets handles the WebSocket headers
proxyWebsockets = true;
extraConfig = ''
  auth_request off;
  proxy_set_header Host $host;
'';

The same rule applies to recommendedProxySettings — it already sets proxy_http_version 1.1 at the HTTP level (a default). Adding it again in a location’s extraConfig duplicates the location-level emission from proxyWebsockets, not the HTTP-level default.

Debugging Duplicate Directive Errors

When nginx -t fails with "<directive>" directive is duplicate, check:

1. Does the location have proxyWebsockets = true AND explicit proxy_http_version/Upgrade/Connection in extraConfig?
2. Does extraConfig repeat any of these four directives: proxy_http_version, proxy_set_header Upgrade, proxy_set_header Connection, or any line from recommendedProxyConfig (Host, X-Real-IP, X-Forwarded-*)?
3. Read the generated nginx config to confirm:

   nginx -T 2>/dev/null | grep -n 'proxy_http_version\|proxy_set_header Upgrade\|proxy_set_header Connection'

Troubleshooting ACME Permissions

If you encounter permission errors where Nginx cannot read certificates:

• The nginx user is automatically added to the acme group in our modules.
• Ensure the certificate exists in /var/lib/acme/<domain>/.
• acmeRoot = null is set on all vhosts via useACMEWildcardHost to prevent HTTP-01/DNS-01 method conflicts.

DNS & Ad Blocking

Three layers of DNS handle name resolution and content filtering across the homelab. Only one ad blocker per host — they all compete for port 53.

Layer 1: Upstream (systemd-resolved or static)

Configured in modules/security.nix via modules.security.resolved. Default is static nameservers (9.9.9.9, 1.1.1.1, 8.8.8.8). When resolved = true, systemd-resolved provides per-interface DNS (needed for Tailscale MagicDNS split-DNS) with caching and DNSSEC validation.

Layer 2: Tailscale MagicDNS (Headscale)

The self-hosted Headscale server (modules/services/net/headscale.nix) has magic_dns = true. Every Tailscale node gets a <host>.tailf0ad7b.ts.net name resolved through the Tailscale coordination server. The L1 gateway (vps-pacman) uses resolver 100.100.100.100 to resolve L2 backend hostnames at request time.

Layer 3: AdGuard Home (ad blocking)

Configured in modules/services/net/adguardhome.nix. Runs on select hosts as a network-wide ad and tracker blocking DNS server. Web admin UI behind nginx with OAuth2 SSO at adguard.<baseDomain>.

Why AdGuard Home (and not the alternatives)

Nixpkgs 26.05 ships at least four DNS/ad-blocker modules. Only one is needed per deployment — a second one just creates a port-53 conflict.

PostgreSQL Database

PostgreSQL is the primary database backend for most web services in this repository.

Module interface (modules/services/database/postgresql)

The project wraps nixpkgs’ services.postgresql with a lighter declarative interface. Service modules (vaultwarden, immich, stalwart, etc.) consume this wrapper — they never touch services.postgresql directly.

Declarative lifecycle (ensures)

No manual CREATE DATABASE / CREATE USER needed. Each service module declares its requirements:

modules.services.database.postgresql = {
  enable = true;
  ensures = [{
    username = "myapp";
    database = "myapp";
    passwordFile = config.age.secrets.myapp-postgresql-password.path;
  }];
};

This populates services.postgresql.ensureDatabases and ensureUsers. At boot, the post-start script creates missing users/databases and sets passwords from agenix secrets (if passwordFile is provided).

Extensions

Extensions are loaded per-database-role via extraExtensions. Service modules that need specific extensions declare them:

# immich needs pgvector + vectorchord
extraExtensions = ps: [ps.pgvector ps.vectorchord];
settings = { shared_preload_libraries = ["vchord.so"]; };

# minimal: just pgvector for embeddings
extraExtensions = ps: [ps.pgvector];

The package option uses withPackages to make extensions available to the PostgreSQL server.

Version upgrades

PostgreSQL data lives at /var/lib/postgresql/<version>. NixOS never auto-migrates between major versions — you must run the upgrade scripts. The module builds helpers from the pinned package list:

sudo upgrade-pg-cluster-17-18   # 17 → 18
sudo upgrade-pg-cluster-16-17   # 16 → 17
sudo upgrade-pg-cluster-15-16   # 15 → 16

Always stop dependent services and back up /var/lib/postgresql/ first.

Authentication

Generated pg_hba.conf follows a defense-in-depth model:

1. Unix socket — trust for all local users (socket permissions are the access control).
2. Loopback TCP — trust for ensured users (service-to-service on localhost), scram-sha-256 for everyone else.
3. Tailscale — trust for 100.64.0.0/10 and fd7a:115c:a1e0::/48, allowing cross-host database access over the VPN mesh.
4. All other — rejected (no matching rule = implicit deny).

password_encryption = "scram-sha-256" is the default for new passwords.

Service Inventory

All services are proxied through vps-pacman (L1) → lab-matrix (L2) over Tailscale, except Ollama and JupyterHub which run on id3-eniac.

Gatus Health Dashboard

status.alienzj.org runs Gatus — an automated health monitoring dashboard. It probes all web services every 5 minutes and reports their status. Alerts are sent via Gotify (push.alienzj.org) when any service goes down or recovers.

Configuration: modules/services/monitoring/gatus.nix. The service list, conditions, and alerting rules are declared there. The dashboard is NOT behind OAuth2 SSO (standalone web UI at 127.0.0.1:9091) so it remains accessible even when Kanidm or oauth2-proxy are down.

• No login required: Gatus has no user management or authentication. Anyone who can reach it sees the same dashboard. Protect it at the nginx level via oauth2.enable if desired.
• No per-user state: Single dashboard for all viewers. Group-based access control is meaningless here — either you can see it or you can’t.

Glance Dashboard

glance.alienzj.org runs Glance — a self-hosted dashboard for feeds, bookmarks, weather, and widgets. It serves as the homelab landing page.

Configuration: modules/services/web/glance.nix. Widgets are declared in services.web.glance.settings on the host config. Behind OAuth2 SSO by default — only members of the configured Kanidm group see the dashboard. No per-user state; Glance has no user management of its own.

• Bind address: 127.0.0.1:8081 — loopback only, proxied through nginx.
• Firewall: No ports opened — nginx handles external access.

Jitsi Meet (Currently Disabled)

Status: Disabled on lab-matrix due to Cloudflare proxy incompatibility. Module: modules/services/media/jitsi-meet.nix

Jitsi Meet requires UDP connectivity for real-time media:

Why it doesn’t work behind Cloudflare: Cloudflare’s DNS proxy (orange cloud) only proxies HTTP/HTTPS on ports 80/443. UDP ports for TURN and media relay are NOT proxied, so:

1. Client resolves meet.alienzj.org → Cloudflare IPs (not the real server)
2. Client tries TURN on Cloudflare IP:3478 → connection refused (no TURN there)
3. ICE negotiation fails → “You have been disconnected” after joining any room

Possible solutions (none adopted):

Secure domain (optional prosody auth): When modules.services.media.jitsi-meet.secureDomain.enable = true, only prosody-registered users can create rooms. When false (default), anyone who passes SSO can create rooms. To register a user:

sudo prosodyctl register <username> meet.alienzj.org <password>

Joining existing rooms never requires a prosody account — only the meeting link.

Nix Binary Cache (niks3 + R2) — DISABLED

Status: Disabled (2026-06-14). The nix-cache service has been disabled on all hosts. Module code remains for potential future re-enablement with Minio backends or after niks3 upload filtering is resolved upstream.

cache.alienzj.org ran niks3 — a self-hosted Nix binary cache backed by Cloudflare R2 (S3-compatible object storage). It was intended to replace upstream NixOS cache for homelab builds.

Configuration: modules/services/system/nix-cache.nix.

Architecture: niks3 runs as an HTTP cache proxy on lab-matrix (0.0.0.0:5751). It stores nar/narinfo files in a private R2 bucket and serves them to Nix clients via its built-in read proxy (services.niks3.readProxy.enable = true). The signing private key and R2 credentials never leave lab-matrix — clients only need the public key.

Why It Was Disabled

Despite multiple layers of filtering, niks3 still uploads packages that already exist in upstream caches (cache.nixos.org, nix-community, etc.). The attempted mitigations were:

1. Signature filtering in build script (nix path-info --json): The build script queries package signatures and filters out paths signed by known upstream caches. This works at the path-info level but only covers the derivation’s own metadata — recursive dependencies are discovered later by niks3 internals.
2. Client recursion patch (--recursive flag removed from client/nixstore.go): Patches out nix path-info --recursive in the niks3 Go source to prevent the CLI from pulling the full dependency tree. However, the niks3 server-side deduplication logic still resolves and uploads referenced paths found in narinfo References: lines.

The net effect: 9.2 GiB accumulated in R2 over a few build cycles, including many packages that are identical byte-for-byte to what’s on cache.nixos.org. This exceeds R2 free tier (10 GB) and offers no cache-hit benefit — clients already fetch these from upstream caches at wire speed.

Future direction: A self-hosted Minio S3-compatible server on lab-matrix’s local NVMe storage would remove the 10 GB ceiling entirely. With no per-GB cost, filtering becomes unnecessary — all build outputs can be cached. Minio can be set up as services.minio on NixOS, with niks3 pointing its S3 backend at http://127.0.0.1:9000. The trade-off is local disk usage instead of cloud storage, but lab-matrix has abundant NVMe capacity.

Cleanup Procedures

When decommissioning the nix-cache, three layers need cleanup: systemd services, the R2 bucket, and the PostgreSQL database.

1. Stop & Disable Services

On the cache server (lab-matrix):

# Stop the running services
ssh matrix_root 'systemctl stop niks3.service'
ssh matrix_root 'systemctl --user stop nix-cache-build.timer'

# Disable at boot — on NixOS impermanent root, /etc is read-only from the
# Nix store, so systemctl disable fails. Instead, set enable = false in
# the host config and rebuild:
#   hosts/lab-matrix/modules/modules.nix:
#     nix-cache.enable = false;
# Then: hey ops deploy lab-matrix lab_matrix_root --boot

After a rebuild with enable = false, the services are removed from the system closure entirely.

2. Empty the Cloudflare R2 Bucket

Use the AWS CLI (v2) with the R2 S3-compatible endpoint. Credentials are stored in agenix secrets on the server:

# On lab-matrix — extract credentials from agenix runtime files
ACCESS_KEY=$(ssh matrix_root 'cat /run/agenix/nix-cache-s3-access-key')
SECRET_KEY=$(ssh matrix_root 'cat /run/agenix/nix-cache-s3-secret-key')
ENDPOINT="09f1ebaf1e24350b0e521b8387168cfd.r2.cloudflarestorage.com"
BUCKET="nix-cache"

# Step 1: Check bucket size (before cleanup)
ssh matrix_root "AWS_ACCESS_KEY_ID=$ACCESS_KEY AWS_SECRET_ACCESS_KEY=$SECRET_KEY \
  nix-shell -p awscli2 --run 'aws s3 ls s3://$BUCKET \
  --endpoint-url https://$ENDPOINT --region auto --summarize --human-readable --recursive'"

# Step 2: Delete all objects recursively
ssh matrix_root "AWS_ACCESS_KEY_ID=$ACCESS_KEY AWS_SECRET_ACCESS_KEY=$SECRET_KEY \
  nix-shell -p awscli2 --run 'aws s3 rm s3://$BUCKET --recursive \
  --endpoint-url https://$ENDPOINT --region auto'"

# Step 3: Verify the bucket is empty
ssh matrix_root "AWS_ACCESS_KEY_ID=$ACCESS_KEY AWS_SECRET_ACCESS_KEY=$SECRET_KEY \
  nix-shell -p awscli2 --run 'aws s3 ls s3://$BUCKET \
  --endpoint-url https://$ENDPOINT --region auto --summarize --human-readable --recursive'"
# Expected: Total Objects: 0, Total Size: 0 Bytes

# Step 4: Abort all incomplete multipart uploads
# Even after deleting all objects, niks3 may have left orphaned multipart uploads
# (uploads that were initiated but never completed/aborted). These consume
# storage but are invisible to `aws s3 ls`. List and abort them:
ssh matrix_root "AWS_ACCESS_KEY_ID=$ACCESS_KEY AWS_SECRET_ACCESS_KEY=$SECRET_KEY \
  nix-shell -p awscli2 --run 'aws s3api list-multipart-uploads --bucket $BUCKET \
  --endpoint-url https://$ENDPOINT --region auto'"

# Abort all multipart uploads (handles pagination):
ssh matrix_root "AWS_ACCESS_KEY_ID=$ACCESS_KEY AWS_SECRET_ACCESS_KEY=$SECRET_KEY \
  nix-shell -p awscli2 --run '
    COUNT=0
    NEXT_TOKEN=\"\"
    while true; do
      if [ -z \"\$NEXT_TOKEN\" ]; then
        RESULT=\$(aws s3api list-multipart-uploads --bucket nix-cache \
          --endpoint-url https://09f1ebaf1e24350b0e521b8387168cfd.r2.cloudflarestorage.com \
          --region auto --output json)
      else
        RESULT=\$(aws s3api list-multipart-uploads --bucket nix-cache \
          --endpoint-url https://09f1ebaf1e24350b0e521b8387168cfd.r2.cloudflarestorage.com \
          --region auto --output json --key-marker \"\$NEXT_TOKEN\")
      fi
      UPLOADS=\$(echo \"\$RESULT\" | jq -r \".Uploads // [] | .[] | .Key + \\\"|\\\" + .UploadId\")
      if [ -z \"\$UPLOADS\" ]; then break; fi
      while IFS= read -r line; do
        KEY=\$(echo \"\$line\" | cut -d\"|\" -f1)
        UID=\$(echo \"\$line\" | cut -d\"|\" -f2)
        echo \"Aborting: \$KEY\"
        aws s3api abort-multipart-upload --bucket nix-cache --key \"\$KEY\" --upload-id \"\$UID\" \
          --endpoint-url https://09f1ebaf1e24350b0e521b8387168cfd.r2.cloudflarestorage.com \
          --region auto
        COUNT=\$((COUNT + 1))
      done <<< \"\$UPLOADS\"
      IS_TRUNCATED=\$(echo \"\$RESULT\" | jq -r \".IsTruncated // false\")
      if [ \"\$IS_TRUNCATED\" != \"true\" ]; then break; fi
      NEXT_TOKEN=\$(echo \"\$RESULT\" | jq -r \".NextKeyMarker // empty\")
    done
    echo \"Aborted \$COUNT multipart uploads\"
  '"

# Verify no uploads remain:
ssh matrix_root "AWS_ACCESS_KEY_ID=$ACCESS_KEY AWS_SECRET_ACCESS_KEY=$SECRET_KEY \
  nix-shell -p awscli2 --run 'aws s3api list-multipart-uploads --bucket $BUCKET \
  --endpoint-url https://$ENDPOINT --region auto --output json | jq \".Uploads // [] | length\"'"
# Expected: 0

Why multipart uploads linger: The niks3 client uses S3 multipart uploads for large NAR files (>10 MB). If the upload is interrupted (rate limiting, timeout, crash), the initiated upload stays open on R2. The PostgreSQL multipart_uploads table may have been cleaned by GC, but the S3-side state persists. These orphaned parts consume storage and count toward the 10 GB free tier limit. The R2 lifecycle rule (abort after 1 day) handles this in normal operation, but during decommission you should abort them immediately.

R2 credentials reference:

Note: R2 buckets cannot be deleted via S3 API — only emptied. To delete the bucket itself, use the Cloudflare Dashboard → R2 → nix-cache → Settings → Delete Bucket.

Alternative: rclone

If rclone is preferred over awscli2:

# On lab-matrix
rclone config create r2-nix-cache s3 \
  provider Cloudflare \
  endpoint https://09f1ebaf1e24350b0e521b8387168cfd.r2.cloudflarestorage.com \
  access_key_id <access-key> \
  secret_access_key <secret-key> \
  region auto

# Empty the bucket
rclone delete r2-nix-cache:nix-cache --fast-list
rclone rmdirs r2-nix-cache:nix-cache --leave-root

3. Clean the PostgreSQL Database

The niks3 database has 7 tables. Only 2 contain real data (objects, closures); the rest are empty in normal operation. Clean them all:

-- Read-only check first (audit what will be deleted)
SELECT 'objects' AS tbl, count(*) FROM objects
UNION ALL SELECT 'closures', count(*) FROM closures
UNION ALL SELECT 'pending_closures', count(*) FROM pending_closures
UNION ALL SELECT 'pending_objects', count(*) FROM pending_objects
UNION ALL SELECT 'multipart_uploads', count(*) FROM multipart_uploads
UNION ALL SELECT 'pins', count(*) FROM pins;

-- Destructive cleanup (requires approval)
BEGIN;
TRUNCATE TABLE objects, closures, pending_closures, pending_objects, multipart_uploads, pins CASCADE;
COMMIT;

-- Verify all tables are empty
SELECT
  (SELECT count(*) FROM objects) AS objects,
  (SELECT count(*) FROM closures) AS closures,
  (SELECT count(*) FROM pending_closures) AS pending_closures,
  (SELECT count(*) FROM pending_objects) AS pending_objects,
  (SELECT count(*) FROM multipart_uploads) AS multipart_uploads,
  (SELECT count(*) FROM pins) AS pins;

Run via SSH:

ssh matrix_root "sudo -u postgres psql -d niks3 -c \"
BEGIN;
TRUNCATE TABLE objects, closures, pending_closures, pending_objects, multipart_uploads, pins CASCADE;
COMMIT;
\""

Note: TRUNCATE is preferred over DROP TABLE — it preserves the schema and permissions. If niks3 is restarted later with a fresh bucket, it will repopulate tables from scratch.

4. Remove R2 Lifecycle Rules

If you no longer need the bucket, remove any Object Lifecycle Rules created for nix-cache from the Cloudflare Dashboard (R2 → nix-cache → Settings → Object Lifecycle Rules):

• Abort incomplete multipart uploads after 1 day — can be removed since no new multipart uploads will occur.

Complete One-Shot Cleanup Script

#!/usr/bin/env bash
# Run on lab-matrix to fully decommission nix-cache
set -euo pipefail

echo "=== Stopping services ==="
systemctl stop niks3.service
systemctl --user stop nix-cache-build.timer

echo "=== Emptying R2 bucket ==="
ACCESS_KEY=$(cat /run/agenix/nix-cache-s3-access-key)
SECRET_KEY=$(cat /run/agenix/nix-cache-s3-secret-key)
export AWS_ACCESS_KEY_ID=$ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=$SECRET_KEY
ENDPOINT="https://09f1ebaf1e24350b0e521b8387168cfd.r2.cloudflarestorage.com"
nix-shell -p awscli2 --run "
  aws s3 rm s3://nix-cache --recursive --endpoint-url $ENDPOINT --region auto
  echo '=== Aborting multipart uploads ==='
  aws s3api list-multipart-uploads --bucket nix-cache --endpoint-url $ENDPOINT --region auto \
    | jq -r '.Uploads[]? | .Key + \"|\" + .UploadId' \
    | while IFS='|' read -r key uid; do
        echo \"Aborting: \$key\"
        aws s3api abort-multipart-upload --bucket nix-cache --key \"\$key\" --upload-id \"\$uid\" \
          --endpoint-url $ENDPOINT --region auto
      done
"

echo "=== Cleaning PostgreSQL ==="
sudo -u postgres psql -d niks3 <<'SQL'
BEGIN;
TRUNCATE TABLE objects, closures, pending_closures, pending_objects, multipart_uploads, pins CASCADE;
COMMIT;
SQL

echo "=== Cleanup complete ==="
echo "Next: set nix-cache.enable = false in host configs and rebuild"

Historical: How It Worked (For Future Reference)

The module provided these features when active. Documented here for reference if the service is re-enabled with Minio or after niks3 fixes its upload filtering:

• Two access paths: Tailscale direct (http://100.88.42.65:5751) and public (https://cache.alienzj.org via nginx/OAuth2).
• OAuth2: Landing page SSO-protected; binary cache paths (/nix-cache-info, /nar/, *.narinfo) exempted.
• Bind address: 0.0.0.0:5751 — LAN + Tailscale reachable.
• Firewall: TCP 5751 restricted to LAN + Tailscale ranges.
• Storage & R2 Optimization: Cloudflare R2 (nix-cache bucket). PostgreSQL on lab-matrix stores only niks3 metadata (reference counts for GC). Build script filters out paths signed by upstream caches via nix path-info --json signature inspection.
• Client Recursion Patch: niks3 CLI patched to remove --recursive from client/nixstore.go to prevent the client from resolving and uploading full dependency trees.
• Multipart Upload Pruning: R2 bucket configured with Object Lifecycle Rule to abort incomplete multipart uploads after 1 day.
• Concurrency & Resource Control: niks3 push --max-concurrent-uploads 4 (configurable via maxConcurrentUploads).
• GC: Time-based, older than 360h, weekly on Sundays.
• Builds: Weekly timer (Sat *-*-* 03:00:00) builds configured hosts sequentially with --max-jobs 1 --cores 8 and 10-minute inter-host cooldown. Gotify notifications on completion/failure.

VictoriaMetrics

metrics.alienzj.org runs VictoriaMetrics — a Prometheus-compatible time-series database and monitoring backend. It scrapes metrics from all hosts and stores them for querying.

• Authentication: HTTP Basic Auth (not SSO). Credentials are set via basicAuthUsername/basicAuthPasswordFile in modules/services/monitoring/victoriametrics.nix. The browser shows a native username/password popup — this is normal for monitoring tools.
• vmui is a query tool, not a dashboard: The built-in UI at /vmui/ is for writing PromQL queries and exploring raw metrics. It is not a replacement for Grafana. For polished dashboards, deploy Grafana and point it at VictoriaMetrics as a Prometheus datasource.
• No additional user creation needed: The single basic-auth credential covers all access. VictoriaMetrics does not support multi-user or SSO.
• Metric sources: Node Exporter (system metrics), service endpoints (via Gatus probes), and application-level scraping targets configured in the module.

Atuin Shell History Sync

atuin.alienzj.org runs the self-hosted Atuin sync server. Clients push encrypted shell history to the server and pull it to other machines.

Module: modules/services/system/atuin.nix. The HTTP endpoint serves a sync API — there is no web UI. Browsing the URL returns JSON.

First-time user registration

Atuin gates registration behind the openRegistration toggle. It is false by default (locked down). To create your first user, use a two-deploy cycle:

# 1. In hosts/lab-matrix/modules/modules.nix, set:
#    modules.services.system.atuin.openRegistration = true;
hey ops deploy lab-matrix lab_matrix_root --boot

# 2. On your workstation:
atuin register -u alienzj -p '<password>' -e [email protected]
atuin login -u alienzj -p '<password>'
atuin sync

# 3. Set openRegistration = false and redeploy to lock registration back down:
hey ops deploy lab-matrix lab_matrix_root --boot

After registration, atuin sync pushes local history to the server and pulls history from other machines. The auto_sync = true setting in ~/.config/atuin/config.toml handles this automatically.

Usage

Atuin replaces Ctrl-R with a full-screen fuzzy search TUI. It works like this:

# Ctrl-R — opens the Atuin TUI
# Start typing any fragment — matches across all commands, all machines
# ↑/↓ to navigate, Enter to execute, Tab to select multiple

atuin search git        # search history for 'git'
atuin history list      # show recent commands
atuin stats             # show command stats (top commands, daily counts)

The architecture:

Machine 1 ── auto_sync (push) ──► atuin.alienzj.org ◄── auto_sync (pull) ── Machine 2
    │                                    │
    └── ~/.local/share/atuin/*.db        └── PostgreSQL (users only)
         (all command data                       encrypted-at-rest records
          stored locally)

Ctrl-R searches the local database — it’s instant and works offline. The sync server is purely a relay: it stores encrypted records, never sees plaintext. History is encrypted on the client with your local key (~/.local/share/atuin/key) before it leaves the machine.

If Ctrl-R shows the default shell history (single line) instead of the Atuin TUI, run eval "$(atuin init zsh)" to activate it in the current shell. New shells pick it up automatically if Atuin is configured in the zsh module.

Troubleshooting

• “username already in use” with empty database: The local ~/.local/share/atuin/key file may contain a stale identity. Delete it and re-register.
• Server has no users despite registration: Verify openRegistration = true was actually deployed. Check ssh matrix_root "sudo -u postgres psql -d atuin -c 'SELECT id, username FROM users;'".
• “Server not reporting its version”: OAuth2 is blocking the root path. nginx redirects the CLI’s GET / version check to Kanidm. Add an "= /" exact-match location with auth_request off (done in the module).

Anki Sync Server

anki.alienzj.org runs the self-hosted Anki Sync Server. It acts as a private, high-performance replacement for AnkiWeb, handling flashcard synchronizations and media uploads directly to your homelab.

Module: modules/services/home/anki-sync.nix. Backend Server Port: 27701 (managed under services.anki-sync-server on lab-matrix). Bind Address: Configurable via listenAddress option. Default 127.0.0.1 (loopback, for use behind nginx). Set to 0.0.0.0 on lab-matrix for direct Tailscale/LAN access. Firewall rules restrict port 27701 to RFC1918 + Tailscale CGNAT ranges.

Bypass of OAuth2 SSO

While the root path (/) is protected by OAuth2-Proxy SSO, the API paths /sync/ and /msync/ are configured with auth_request off; in Nginx. This bypass is critical because native Anki desktop and mobile clients cannot complete standard web-based single-sign-on (SSO) login redirects. They rely exclusively on Anki’s native username/password authentication mechanism instead.

Setup & Credentials

• Username: Defaults to your primary user (alienzj).
• Password: Managed using agenix and saved as anki-sync-password.age in your nix-secrets repository.
• Sync URL (nginx): https://anki.alienzj.org/
• Sync URL (direct, recommended): http://100.88.42.65:27701/ — connects directly over Tailscale, bypassing nginx and the vps-pacman round-trip. Requires listenAddress = "0.0.0.0" on the server host (enabled on lab-matrix).

Client Configuration

To point your Anki Desktop client to the self-hosted instance:

1. Open Anki Desktop and go to Settings/Preferences → Syncing.
2. Under the Custom sync server section, enter your preferred Sync URL:
   • Tailscale direct (fastest): http://100.88.42.65:27701/
   • Nginx (public, slower): https://anki.alienzj.org/
3. Save, close Preferences, and click Sync (or press Y).
4. Log in using your username (alienzj) and the decrypted agenix password.

Handling Sync Conflicts

When syncing a new client or making out-of-sync edits, Anki may display a conflict dialog:

“There is a conflict between decks on this device and AnkiWeb. You must choose which version to keep…”

Note that “AnkiWeb” in Anki’s UI now refers to your custom sync server.

• Upload to AnkiWeb: Select this if you are performing a first-time sync or have local changes on this device that you want to push to the server (overwriting the server’s cards).
• Download from AnkiWeb: Select this if the server contains the most up-to-date deck versions (e.g., synced from another device like AnkiDroid) and you want to pull those down, replacing your current local deck.

Downloading & Syncing Public Decks

You do not need to log in to AnkiWeb to use shared decks.

1. Browse and locate any public deck in the AnkiWeb Shared Decks directory in your web browser.
2. Click the Download button to download the deck’s .apkg file directly.
3. Open Anki Desktop (which is pointed to your self-hosted server) and click File → Import (or click Import File at the bottom).
4. Select the .apkg file and import it.
5. Click Sync (or press Y) to automatically upload the new deck to your self-hosted server.

Troubleshooting & Sync Performance

When importing and syncing massive public decks (e.g., 4000 Essential English Words with 400MiB+ of audio/image media files), you may notice that the sync is extremely slow despite your homelab server (lab-matrix) having substantial resources.

• The Cause: Anki syncs media files individually via separate HTTP POST requests. If your client points to https://anki.alienzj.org/, every request travels out through Cloudflare and vps-pacman (L1 VPS) before returning over Tailscale to lab-matrix. The round-trip latency (50ms–100ms per file) for 10,000+ files creates a massive bottleneck.
• The Solution (Tailscale Direct): Use the direct Tailscale sync URL http://100.88.42.65:27701/. Traffic flows directly over the Tailscale wireguard mesh with LAN-level latency, completely bypassing nginx, vps-pacman, and Cloudflare. This is the recommended permanent sync URL for any host on the Tailscale network.
• Alternative (LAN Bypass): If Tailscale is unavailable, you can temporarily point to the server’s local LAN IP:
  1. In Anki Desktop sync preferences, temporarily change the Sync URL to http://192.168.31.10/ (going through nginx on port 80).
  2. Perform the initial sync at local Gigabit speeds.
  3. Switch back to the Tailscale direct URL afterward for ongoing use.

WebDAV Server (Dufs)

dav.alienzj.org runs an ultra-lightweight, high-performance WebDAV server powered by dufs (written in Rust). It serves as your private cloud attachment store (e.g. for Zotero PDF sync), eliminating Nextcloud’s resource overhead.

Module: modules/services/sharing/webdav.nix. Backend Server Port: 27702 (managed via a sandboxed custom systemd service on lab-matrix).

Ultra-Lightweight Footprint

Unlike Nextcloud (which requires PHP-FPM, a heavy database, Redis, and periodic indexers, consuming 500MB to 1.5GB of RAM), dufs runs as a single static Rust binary:

• RAM Footprint: ~10MB to 15MB under active loads.
• CPU Footprint: 0% when idle.
• Storage: Stored as plain files in /var/lib/webdav for simple back-ups, without database lockouts or virtual filesystem overhead.

Selective OAuth2 SSO Protection

We employ a hybrid security model for this virtual host:

• Browser Access (/): Fully protected by OAuth2 SSO integrated with your Kanidm identity provider. When you visit https://dav.alienzj.org in a browser, it requires you to log in via Kanidm SSO first (and subsequently prompts for dufs’s Basic Auth, providing multi-layered protection).
• Zotero Sync API (/documents/zotero/): Explicitly exempted from SSO (auth_request off;). This bypass is critical because native Zotero clients (desktop and mobile apps) do not support browser-based OAuth2 login redirects. They rely exclusively on standard HTTP Basic Authentication, which Nginx passes cleanly to the backend dufs daemon.

Setup & Credentials

• Username: Defaults to your primary user (alienzj).
• Password: Managed securely using agenix and saved as webdav-password.age in your nix-secrets repository. The custom systemd service reads the decrypted secret securely at startup, keeping the plaintext password out of the world-readable Nix store.

Hybrid Zotero Sync Architecture

For details on the client-side configuration, hybrid Syncthing + WebDAV syncing, and the collection symlink mirror setup, see the Noter Documentation.

Jitsi Meet

meet.alienzj.org runs the self-hosted Jitsi Meet video conferencing platform. It provides secure, high-performance web-based video rooms natively integrated with your homelab infrastructure.

Module: modules/services/media/jitsi-meet.nix. Frontend: Served directly by Nginx (static root at ${pkgs.jitsi-meet}/share/jitsi-meet with dynamic location overrides).

Multi-Component Architecture

Jitsi Meet is a complex orchestration of four co-located backend services:

1. Prosody (XMPP chat server): Manages signaling, room states, and user sessions.
2. Jicofo (Conference focus agent): Manages video bridge allocations and coordinates participant entry/exit.
3. Jitsi Videobridge 2 (SFU media router): Routes high-bandwidth audio/video streams between participants.
4. Jitsi Excalidraw (Collaborative whiteboard): Provides shared sketching features within conference rooms.

Port Mapping & Conflict Safeguards

To avoid port exhaustion and conflicts on the lab-matrix server, the following ports are custom-mapped:

• Jicofo REST API: Binds to 8858 (exempted from 8888 to prevent conflict with Atuin).
• Jitsi Videobridge (Colibri REST): Binds to 8080 (requires Stalwart webPort override to 8085 to avoid conflict).
• Jitsi Excalidraw Whiteboard: Binds to 3002 (with Prometheus metrics custom-mapped to 9097 to prevent conflict with sing-box Clash API on 9090).
• Media Streams (UDP): Binds to port range 10000:20000 (dual-stack LAN restricted firewall rules).

Selective OAuth2 SSO Protection

• Browser Access (/): Fully protected by OAuth2 SSO at the Nginx gateway, ensuring only authenticated users can access the interface.
• API & WebSocket Paths: Exempted from SSO (auth_request off;) for /colibri/ and Prosody’s WebSocket upgrade paths. This exemption is critical because native client integrations and active video streams cannot follow HTML redirects.

Kanidm Group Model

Five services enforce per-service access control via Kanidm groups:

All groups use full SPN format ([email protected]) because Kanidm returns group claims in the OIDC token with the domain suffix. The allowedGroups option in oauth2-proxy vhost configs must match the full SPN, not the bare group name.

To add a user to a group:

kanidm group add-members access_forgejo <username>

See sso-identity.md for the complete Kanidm administration guide.

Service Operational Notes

Patterns and lessons from testing all 20 homelab services on lab-matrix. Each service has its own first-time setup ritual, auth model, and quirks.

Systemd → Backend → Browser: Three-Layer Verification

When checking whether a service works, test in this order:

# Layer 1: Systemd
ssh matrix_root "systemctl is-active <service>"

# Layer 2: Backend on localhost (confirms the daemon responds, bypasses nginx/SSO)
curl -sI http://127.0.0.1:<port>/ --max-time 3

# Layer 3: Browser through nginx → SSO → app
# Visit https://<domain>.alienzj.org, log in via Kanidm if prompted

PHP-FPM services (FreshRSS, Grocy) skip layer 2 — no standalone port. Verify via systemctl is-active phpfpm-<name> then browser test.

Auth Models

Services fall into three patterns:

SSO-only (12 services) — nginx OAuth2 is the only gate. Once you authenticate with Kanidm, the app loads without a second login:

Linkwarden, Paperless, HedgeDoc, LiteLLM, Mealie, Calibre, FreshRSS, Grocy, OnlyOffice, Gatus, Jitsi Meet

Dual-auth (4 services) — SSO gates HTTP access, then the app has its own internal login:

Affine: nginx OAuth2 controls site access. Affine has its own user system — signup is always enabled (no env var to disable it). First user signs up via the web UI and gets admin. Admin panel at /admin/setup. The nginx gate ensures only Kanidm users reach the signup page.

Forgejo: nginx OAuth2 controls site access (Kanidm group access_forgejo). Internal login uses Forgejo’s own user system. Accounts are created via CLI (forgejo admin user create), not web signup. DISABLE_REGISTRATION = true blocks web registration but CLI always works. First user created gets admin.

Stirling-PDF: Default admin/password shown on login page. Change on first login. Defense-in-depth — SSO proves you’re a valid user, app auth controls what you can do inside.

Vaultwarden: /admin page (token-gated) creates user accounts. Web vault login uses master password (not SSO). API paths (/identity/, /api/) bypass nginx OAuth2 so mobile/desktop apps connect directly.

VictoriaMetrics: HTTP Basic Auth (browser popup), not SSO. Single credential via basicAuthUsername. vmui is a PromQL query tool, not a Grafana replacement.

API-only (1 service) — no web UI, HTTP endpoint serves a protocol:

Atuin: Sync API for shell history. No browser UI — use atuin register/atuin login/atuin sync CLI. Registration gated behind openRegistration toggle (two-deploy cycle: enable → register → disable).

SSO + internal auth (1 service):

Gotify: Web UI for managing notification clients and viewing messages. First user created on initial access; signup disabled by default — admin creates additional users via the web UI.

First-Time Setup Patterns

OAuth2 Path Exemptions

The default is every path requires a valid SSO cookie — no auth_request off. Exemptions are made only when a non-browser client (mobile app, CLI, RSS reader) accesses API paths that cannot follow HTML redirects. The nginx auth_request redirect (307) works for browsers but breaks native apps.

Not exempted (works with OAuth2):

• Paperless: WebSocket /ws/status — browsers send cookies with WebSocket upgrades
• HedgeDoc: /register, /login — POST form submissions include the SSO cookie
• Forgejo, Affine, Linkwarden, Calibre, etc.: All browser-based, cookies work everywhere

The auth_request off + proxy_pass pattern in the module:

locations."/api/" = {
  proxyPass = "http://127.0.0.1:${toString cfg.port}";
  proxyWebsockets = true;
  extraConfig = "auth_request off;";
};

SMTP for Co-Located Services

Services on lab-matrix send email through Stalwart on localhost. Two patterns are used, depending on whether the service supports TLS certificate verification skip:

Pattern A — Port 587 + STARTTLS + skip cert verify (vaultwarden pattern): Services that can disable TLS certificate verification use the standard submission port with STARTTLS. The cert is for mail.alienzj.org, not 127.0.0.1, so verification must be skipped:

Pattern B — Port 10025 + plain SMTP (no TLS at all): Services that cannot disable TLS verification use Stalwart’s submission-local listener on port 10025. This is a plain-text SMTP listener with TLS disabled entirely — no cert mismatch to worry about. Stalwart’s saslMechanisms rule allows PLAIN/LOGIN auth on this port (normally only available on TLS connections).

The Stalwart saslMechanisms rule in modules/services/net/stalwart.nix:

session.auth.saslMechanisms = [
  {
    "if" = "(local_port != 25 && is_tls) || local_port == 10025";
    "then" = "[plain, login, oauthbearer, xoauth2]";
  }
  {"else" = "false";}
];

Services without an SMTP option at all: stirling-pdf, onlyoffice, calibre, freshrss, grocy.

SMTP Service Accounts

Each service authenticates with its own Stalwart account (e.g., [email protected], [email protected]). These are created in Stalwart webadmin → Directory → Accounts. The SMTP password is stored in an agenix secret per service, formatted as the env var the service expects (e.g., SMTP_PASSWORD=xxx for vaultwarden, EMAIL_SERVER_PASSWORD=xxx for linkwarden).

Common Debugging Scenarios

“username already in use” with empty database (Atuin): Local ~/.local/share/atuin/key file contains a stale identity. Delete it and re-register.

“SMTP timeout” on localhost (Vaultwarden, Forgejo): Service uses mail.alienzj.org (public VPS IP) instead of 127.0.0.1. Override mailer.host in the host config.

NextAuth NO_SECRET (Linkwarden, Affine): NextAuth.js requires a production secret. Add secretFiles.NEXTAUTH_SECRET pointing to an agenix-managed file containing openssl rand -base64 32.

Admin token plaintext warning (Vaultwarden): Hash the token with vaultwarden hash (interactive), store the $argon2id$... output in the age file.

Backend returns 404 on root (OnlyOffice): Normal — OnlyOffice serves API endpoints at specific paths, not HTML at /. Try /healthcheck instead.

Backend returns 401 on localhost (Stirling-PDF, VictoriaMetrics): Internal auth protecting the backend. Normal — test through browser with SSO.

HTTP basic auth popup instead of login page (VictoriaMetrics): Expected — VM uses HTTP Basic Auth, not a web form. The browser shows a native dialog for credentials.

Common SMTP Errors

invalid peer certificate: UnknownIssuer — cert hostname mismatch on localhost.

Fix: SMTP_ACCEPT_INVALID_CERTS = true + SMTP_ACCEPT_INVALID_HOSTNAMES = true. Already set in the vaultwarden module.

No compatible authentication mechanism — connecting to a plain SMTP port without TLS. Stalwart withholds PLAIN/LOGIN auth on non-TLS connections, only offering XOAUTH2 OAUTHBEARER which app SMTP clients don’t support.

Fix: use port 587 with STARTTLS. The TLS layer unlocks PLAIN/LOGIN auth.

535 Authentication credentials invalid — the SMTP login name doesn’t match any Stalwart directory account.

Fix: in Stalwart webadmin (Directory → Accounts), set the account’s email to the full @alienzj.org address. Stalwart resolves SMTP login names via lookup.default — the login name must match the account’s email.

Diagnostic:

SMTP_PASS="<password>"
USER="<user>@alienzj.org"
printf "EHLO test\r\nAUTH LOGIN\r\n%s\r\n%s\r\n" \
  "$(echo -n "$USER" | base64)" \
  "$(echo -n "$SMTP_PASS" | base64)" \
  | openssl s_client -connect 127.0.0.1:587 -starttls smtp -quiet 2>/dev/null

Service Specifics

Affine Setup

AFFiNE runs as an OCI container with its own user system behind nginx OAuth2. Signup is always enabled (no env var to disable it — auth.allowSignup has no env mapping). The nginx gate ensures only Kanidm-authenticated users reach it.

First-time setup:

1. Visit https://affine.alienzj.org/admin/setup → create the admin account
2. Visit https://affine.alienzj.org → log in as admin or normal user
3. Admin manages users, workspaces, and server settings from /admin

The Prisma migrations run automatically via systemd.services.podman-affine.preStart before the container starts. Database is provisioned declaratively via ensures.

Mailer uses [email protected] → Stalwart on localhost. SMTP password via affine-smtp-pass.age.

Forgejo Account Setup

Forgejo uses two-layer auth: nginx OAuth2 (Kanidm group access_forgejo) gates site access, then Forgejo’s own login handles user identity. Web registration is disabled (DISABLE_REGISTRATION = true) — create the first account via CLI:

# On lab-matrix, using the environment from the forgejo systemd service
ssh matrix_root "sudo -u git env \
  HOME=/var/lib/forgejo \
  FORGEJO_WORK_DIR=/var/lib/forgejo \
  FORGEJO_CUSTOM=/var/lib/forgejo/custom \
  /nix/store/<hash>-forgejo-<version>/bin/forgejo \
  admin user create \
  --username alienzj \
  --password '<password>' \
  --email [email protected] \
  --admin \
  --config /var/lib/forgejo/custom/conf/app.ini \
  --work-path /var/lib/forgejo"

The binary path changes with each NixOS rebuild. Find the current one with: systemctl show forgejo -p ExecStart | grep -oP '/nix/store/[^/]+-forgejo[^/]+'.

The first user created gets admin. Subsequent accounts can be added the same way. DISABLE_REGISTRATION = true prevents web signup but CLI admin user create bypasses all web-layer config and writes directly to the database.

ENABLE_BASIC_AUTHENTICATION controls HTTP Basic auth (the Authorization: Basic header) — used by API tokens and git-over-HTTP with password. It does NOT control the web login form. The web form is always shown as long as Forgejo has no OAuth2/OIDC providers configured; it’s controlled by ENABLE_INTERNAL_SIGNIN (default true).

Forgejo Mailer

Forgejo is configured to use a professional SMTP setup. By default, it uses Mailjet but can be easily pointed to your self-hosted Stalwart server.

• Sender Address: Defaults to [email protected].
• Customizing SMTP:

  modules.services.git.forgejo.mailer = {
    from = "[email protected]";
    host = "mail.alienzj.org"; # Point to your Stalwart server
    port = 587;
  };

• Secrets: SMTP credential is managed via age.secrets.forgejo-smtp-pass.

Stalwart Mail Server

For public email configuration (MX, DKIM, DMARC, outbound relay, client setup), see email.md.

The NixOS configuration is in modules/services/net/stalwart.nix. Key architectural decisions:

• Directory: Internal directory stored in PostgreSQL (directory.internal.store = "postgres"), not RocksDB. RocksDB at /var/lib/stalwart-mail/db/ gets corrupted by root-owned ExecStartPre files on restart, losing all accounts.
• Fallback admin: authentication.fallback-admin with user = "admin" and bcrypt hash from agenix secret. This bypasses the directory entirely — always works even if PostgreSQL or the directory store is broken.
• Lookup: lookup.default uses type = "directory" with directory = "internal" to resolve recipient addresses via the internal directory during SMTP recipient verification.
• IPv6-only binds: Linux forwards IPv4 to IPv6 sockets automatically, avoiding “Address already in use” dual-stack errors.
• Config isolation: config.local-keys includes directory.* and storage.* to prevent database-stored settings from overriding local Nix-generated config. lookup.* is deliberately excluded so it CAN be persisted.

Encryption-at-Rest (OpenPGP)

Stalwart supports per-account encryption-at-rest where incoming emails are encrypted with the user’s PGP public key before being written to disk. The architecture:

INCOMING: Sender → SMTP → Stalwart (plaintext)
                        → Encrypts with user's PUBLIC key
                        → Writes encrypted blob to disk

READING:  User → Thunderbird fetches encrypted blob
              → Client decrypts with PRIVATE key
              → Plaintext displayed

The private key never touches the server. Only the public key is uploaded to Stalwart. Decryption happens entirely in the mail client (Thunderbird). Even if an attacker steals the physical disk or gains root on the server, stored emails remain unreadable without the private key.

To enable for an account:

# 1. Generate an ECC Ed25519 keypair (NOT Ed448 — Stalwart's Sequoia PGP
#    library rejects v5 LibrePGP packets)
gpg --full-generate-key
# Choose: ECC (sign and encrypt) → Curve 25519 → no expiration
# Name: alienzj  Email: [email protected]  Passphrase: <your choice>

# 2. Export the PUBLIC key
gpg --export --armor [email protected] > /tmp/stalwart-pub.asc

# 3. Upload to Stalwart webadmin: Account → Public Keys → paste pub key

# 4. Enable encryption: Account → Settings → Encryption at rest
#    → OpenPGP → select uploaded key → AES-256

# 5. Export the PRIVATE key and import into Thunderbird.
#    Thunderbird uses its own RNP library, not the system GPG keyring —
#    even though the key exists in ~/.config/gnupg/, Thunderbird won't
#    find it automatically. You must import it manually.
gpg --export-secret-keys --armor [email protected] > /tmp/alienzj-pgp-priv.asc
#    Then in Thunderbird: Account Settings → End-to-End Encryption →
#    Add Key → Import from file → /tmp/alienzj-pgp-priv.asc

# 6. Securely delete exported key files
shred -u /tmp/stalwart-pub.asc /tmp/alienzj-pgp-priv.asc

Debugging: If Thunderbird shows “OpenPGP blocked”, verify the key was imported correctly:

# Check the encryption subkey ID your GPG keyring has:
gpg --list-keys --with-colons [email protected] | grep '^sub' | cut -d: -f5

# In Thunderbird, OpenPGP Key Manager → View → Show Debug Log.
# Look for "key id: 0x..." — the hex digits must match your subkey ID.
# Case doesn't matter. A mismatch means the wrong key was uploaded to Stalwart.

The --export-secret-keys command prompts for a passphrase (the one set during gpg --full-generate-key). This is expected — the private key material is passphrase-encrypted on disk. --list-keys uses only the public keyring and needs no passphrase.

Only NEW incoming emails are encrypted after enabling. Existing emails remain in plaintext (non-retroactive). Disabling encryption later also does NOT decrypt already-encrypted messages.

Known Issue: Webadmin Dashboard Shows Zero Counts

The webadmin (v0.1.37) bundled with Stalwart v0.15.x has a known bug: the dashboard shows “TOTAL USERS 0”, “TOTAL DOMAINS 0”, and “Not found” even when accounts and domains exist in the database. The backend REST API at /api/principal returns correct data; the webadmin SPA calls sub-paths like /api/principal/list and /api/domain/list which Stalwart v0.15.5 does not implement, returning {"error":"notFound"} that the dashboard interprets as empty results.

All backend functionality is unaffected — email delivery, SMTP auth, IMAP access, and the management REST API all work correctly.

Root cause: The stalwartlabs/webadmin repo has been archived. Starting with Stalwart v0.16, the management UI has been rewritten from scratch as webui (stalwartlabs/webui), and the legacy REST API (/api/...) is replaced by a JMAP Management API at /jmap. The upgrade requires a migration script (migrate_v016.py) to dump settings/principals from v0.15 and convert them to v0.16 format.

Mitigation: Use stalwart-cli or the REST API (curl -u admin:pass http://127.0.0.1:8085/api/principal) to manage accounts until v0.16 lands in nixpkgs. Then upgrade and migrate.

See Discussion #1477 for the matching upstream report.