Security Hardening

Security & Privacy Hardening

Defense-in-depth architecture: hardware token, secure boot, disk encryption, kernel hardening, application sandboxing, network isolation.

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│  Layer 7: Application Sandbox (nixpak + bubblewrap)         │
│  Discord, Telegram, Zoom, WeMeet                             │
├─────────────────────────────────────────────────────────────┤
│  Layer 6: MAC (AppArmor)                                    │
│  killUnconfinedConfinables = true                           │
├─────────────────────────────────────────────────────────────┤
│  Layer 5: Network (Firewall + Fail2Ban + Tailscale)         │
│  LAN-scoped rules, exponential bans, WireGuard mesh         │
├─────────────────────────────────────────────────────────────┤
│  Layer 4: Secrets (Agenix)                                  │
│  Per-host scoping, runtime decryption, SSH key identity     │
├─────────────────────────────────────────────────────────────┤
│  Layer 3: User Auth (YubiKey PAM U2F + SSH ed25519-sk)      │
│  systemd-lock-handler, loginctl unified lock dispatch        │
├─────────────────────────────────────────────────────────────┤
│  Layer 2: Boot (Lanzaboote Secure Boot + LUKS2 + FIDO2)     │
│  Signed kernel, encrypted root, hardware-unlocked LUKS      │
├─────────────────────────────────────────────────────────────┤
│  Layer 1: Kernel (sysctl hardening)                         │
│  kptr_restrict, dmesg_restrict, BPF hardening, ASLR         │
├─────────────────────────────────────────────────────────────┤
│  Layer 0: Hardware (YubiKey, USBGuard, TPM2)                │
│  Physical token, USB policy, measured boot                  │
└─────────────────────────────────────────────────────────────┘

Layer 0: Hardware Security

YubiKey (modules/profiles/hardware/yubikey.nix)

Your YubiKey serves multiple roles in the security stack:

Touch requirement: sudo operations require physical touch on the YubiKey, preventing remote attackers from escalating privileges even with SSH access.

Screen Lock Architecture

The screen lock uses a unified loginctl lock-session entry point backed by systemd-lock-handler. Every lock trigger converges on lock.target — the locker implementation is a systemd service detail, not something each call site decides.

any trigger  ──→  loginctl lock-session
                   → logind signal
                     → systemd-lock-handler starts lock.target
                       ├─ (DMS handles locks natively — see below)
                       ├─ diy-lock.service  (DIY)     hyprlock
                       └─ x-lock.service    (X11)     i3lock

DMS does not bind a service to lock.target. Instead, DMS listens for logind session events natively via its SessionService (onSessionLocked / onSessionUnlocked in Lock.qml). When loginctl lock-session fires, DMS detects the signal and raises the Wayland session lock directly — no systemd service needed. Hotkey binds call dms ipc call lock lock as a fast path.

Only one service is wantedBy lock.target per host — the desktop mode modules gate them on their enable flags. On DMS hosts, no service binds to lock.target at all.

The idle daemons (swayidle, hypridle) bind to graphical-session.target generically — no compositor-specific coupling. They detect inactivity and invoke the lock hook; the hook calls loginctl lock-session, which dispatches to the appropriate locker via systemd.

On unlock, the flow depends on the locker type:

• DMS: DMS listens for logind unlock signals natively via onSessionUnlocked() (Lock.qml line 115). The lock screen dismisses itself — no systemd service involvement. External unlocks (YubiKey, loginctl unlock-session) work because onSessionLocked() sets lockInitiatedLocally = false, so onSessionUnlocked() allows the dismiss.
• DIY / X11 (diy-lock.service / x-lock.service): The locker process blocks until PAM authentication succeeds, then exits with 0. onSuccess=unlock.target triggers the unlock.

A wtype key event resets the idle timer (PAM/FIDO2 auth bypasses compositor input).

USBGuard (modules/profiles/hardware/usbguard.nix)

USBGuard blocks unauthorized USB devices by default:

implicitPolicyTarget = "block";  # Deny unknown devices

Status: Rules are placeholder stubs. To populate with real device IDs:

# After plugging in all your trusted devices:
usbguard generate-policy > /etc/usbguard/rules.conf
# Or use the NixOS option:
modules.hardware.usbguard.rules = [
  "allow id 046d:c539 # Logitech receiver"
  "allow id 1050:0407 # YubiKey"
];

TPM2 (Trusted Platform Module)

TPM2 provides unattended LUKS unlock — the TPM releases the disk encryption key only if the boot chain (firmware → shim → kernel) hasn’t been tampered with. Unlike FIDO2/YubiKey, no physical interaction is needed at boot. The tradeoff: a stolen-but-powered-on machine will auto-unlock.

Hardware requirements: a physical TPM 2.0 chip. Most x86 machines sold after ~2016 have one. Check with: ls /dev/tpm*. VM TPMs (swtpm) work for testing but provide no real security — the TPM state is just a file on the host.

Step 1: Enable the TPM2 profile

Add "tpm2" to the host’s hardware profiles:

modules.profiles.hardware = [
  "cpu/amd"
  "tpm2"  # ← add this
];

This enables security.tpm2 (abrmd, PKCS11, TCTI environment), installs tpm2-tools, and ensures the TPM2 resource manager runs at boot. Module: modules/profiles/hardware/tpm2.nix.

Step 2: Verify the TPM is visible

sudo systemctl status tpm2-abrmd   # resource manager running
tpm2_getrandom 8                   # TPM generates random bytes
ls /dev/tpm0 /dev/tpmrm0           # raw device + kernel RM

Step 3: Choose PCR banks

The TPM measures boot components into Platform Configuration Registers (PCRs). You pick which PCRs to bind the LUKS key to:

Recommended for most deployments: 0+2+7 — covers firmware, boot chain, and Secure Boot without locking on every kernel update.

Strict (tamper-evident): 0+1+2+3+4+5+7+11 — locks on almost any change, including kernel updates and boot order changes. Requires re-enrollment after every nixos-rebuild that touches the kernel. Only useful for high-security static systems.

Step 4: Enroll the TPM (one-time manual step)

# Check your current LUKS device layout
sudo cryptsetup luksDump /dev/<disk-partition> | head -20

# Enroll TPM2 alongside existing passphrase/key (does NOT remove old slots)
sudo systemd-cryptenroll \
  --tpm2-device=auto \
  --tpm2-pcrs="0+2+7" \
  /dev/<disk-partition>

Example for a typical NVMe setup:

sudo systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs="0+2+7" /dev/nvme0n1p2

To require a PIN (TPM + user PIN = defense against physical theft):

sudo systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs="0+2+7" \
  --tpm2-with-pin=yes /dev/nvme0n1p2

Step 5: Add crypttab options (NixOS config)

In the host’s storage.nix, add to the existing LUKS device:

boot.initrd.luks.devices."root".crypttabExtraOpts = [
  "tpm2-device=auto"
  "tpm2-measure-pcr=yes"
];

The tpm2-measure-pcr=yes option ensures the initrd measurements extend PCR 11, which you can use if binding to PCR 11 (kernel cmdline).

Step 6: Rebuild and test

hey sync --host <host>
# On next boot, the disk should auto-unlock without a password.
# If it fails, the boot will fall back to passphrase prompt.

Recovery after firmware update

BIOS updates change PCR 0 — the TPM-unsealed key becomes invalid and boot falls back to passphrase. After booting with the passphrase, re-enroll:

# Remove the old TPM slot and re-enroll
sudo systemd-cryptenroll /dev/<disk-partition> --wipe-slot=tpm2
sudo systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs="0+2+7" /dev/<disk-partition>

Verifying the TPM protects the key

# List enrolled slots; look for the "tpm2" slot
sudo systemd-cryptenroll /dev/<disk-partition>

# Test: clear the TPM and verify the slot can't unlock
sudo tpm2_clear -c p    # WARNING: wipes ALL TPM data
# After this, only passphrase/FIDO2 slots will work.

Layer 1: Kernel Hardening (modules/security.nix)

The kernel is hardened via sysctl parameters:

Memory & Address Space

Network Hardening

Blacklisted Kernel Modules

Dangerous/uncommon protocols and known-vulnerable modules are blacklisted:

boot.blacklistedKernelModules = [
  "dccp" "sctp" "rds" "rds_tcp" "rds_rdma" "tipc"   # rare protocols
  "esp4" "esp6" "rxrpc"                              # Dirty Frag LPE mitigation
];

Vulnerability-driven blacklists are managed separately from static hardening — see docs/vulnerability-response.md for the active list, how to add new entries, and how to verify mitigations on a running host.

Coredumps

Disabled system-wide to prevent information leakage:

systemd.coredump.enable = false;

Performance (not hardening, but co-located)

net.core.default_qdisc = "cake";      # Buffer bloat protection
net.ipv4.tcp_congestion_control = "bbr"; # Google's congestion control

Layer 2: Secure Boot & Disk Encryption

Lanzaboote Secure Boot (modules/profiles/role/workstation.nix)

Lanzaboote signs the kernel and initrd with your own keys, preventing bootkit attacks:

boot.lanzaboote = {
  enable = true;
  pkiBundle = "/etc/secureboot";  # Keys persisted via impermanence
};

Key management: sbctl is installed for key creation and enrollment:

sbctl create-keys
sbctl enroll-keys --microsoft  # Include Microsoft keys for dual-boot
sbctl verify                   # Check signing status

Active on: bio-smart host (boot = "lanzaboote")

LUKS2 Disk Encryption

Full-disk encryption with multiple unlock methods:

FIDO2 enrollment (one-time setup):

# Add FIDO2 as a LUKS unlock method:
sudo systemd-cryptenroll --fido2-device=auto /dev/nvme0n1p2

# Verify enrolled methods:
sudo systemd-cryptenroll /dev/nvme0n1p2

Agenix-managed keyfiles: LUKS keyfiles are stored as agenix secrets (*.luks-keyfile), decrypted at boot to /run/agenix/.

Layer 3: User Authentication

PAM U2F (modules/profiles/hardware/yubikey.nix)

All interactive authentication requires YubiKey touch:

security.pam.u2f = {
  enable = true;
  cue = true;           # "Please touch the device"
  authFile = config.age.secrets.yubikey-auth.path;
};

Protected services: sudo, login, greetd, hyprlock

SSH Authentication (modules/services/net/ssh.nix)

SSH keys: ed25519-sk (YubiKey-backed, non-exportable) for signing and authentication.

Fail2Ban (modules/services/monitoring/fail2ban.nix)

Exponential ban times with a 7-day cap:

Tailscale IP ranges are whitelisted to prevent self-lockout.

Vaultwarden has dedicated jails for login and admin endpoints.

Layer 4: Secrets Management (Agenix)

Architecture (modules/agenix.nix)

Agenix decrypts .age secrets at build time using two SSH identity keys:

nix-secrets/secrets/secrets.nix   ← Secret definitions (encrypted)
         │
         ▼
agenix decrypts using:
  - Host key:   /persist/etc/ssh/ssh_host_ed25519_key  (per-host, Nix-generated)
  - Global key: /persist/etc/ssh/global_ed25519         (optional, shared across hosts)
         │
         ▼
/run/agenix/                      ← Runtime (tmpfs, not persisted)

The host key is generated by NixOS and persisted via environment.persistence bind-mount in ssh.nix. The global key is placed directly on the /persist btrfs subvolume (no Nix config needed — files on /persist survive reboots by nature of being on a persistent filesystem). Agenix tries both keys; any missing key is silently skipped.

For the full bootstrap workflow (how keys get onto the machine in the first place), see docs/security.md.

Per-Host Scoping

Secrets can declare which hosts should receive them:

# In nix-secrets/secrets/secrets.nix:
"smtp-password".publicKeys = [id3-eniac bio-smart];
"wireguard-key".publicKeys = [vps-pacman];
# Secrets without 'nodes' are shared across all hosts

The filterByNodes logic in modules/agenix.nix filters out secrets not belonging to the current host, preventing unnecessary decryption.

Registered Secrets

Layer 5: Network Security

Firewall

Enabled by default with reverse path filtering:

networking.firewall = {
  enable = true;
  checkReversePath = "loose";  # Needed for Tailscale/WireGuard
};

LAN-scoped rules: Services like Spotify Connect, Steam Remote Play, and printers use the shared lib/firewall.nix helper to accept connections from RFC 1918 private ranges + Tailscale CGNAT, with cleanup on firewall reload:

# lib/firewall.nix — standard trusted source ranges
lanSources = [
  "192.168.0.0/16"  # RFC 1918 class C — home routers
  "10.0.0.0/8"      # RFC 1918 class A — corporate
  "172.16.0.0/12"   # RFC 1918 class B — mid-size, Docker
  "100.64.0.0/10"   # Tailscale CGNAT
];

# Usage in a service module:
networking.firewall = mkFirewallFor {
  tcp = ["57621"];
  comment = "spotify-connect";
};
# → generates 4 ACCEPT rules (one per source) + 4 cleanup rules

The helper adds -m comment tags for audit visibility and extraStopCommands to clean up on firewall reload. Services that bind beyond localhost use mkRestrictedPort instead, which inserts ACCEPT rules at the top of the nixos-fw chain followed by a terminal DROP.

Tailscale / WireGuard

Tailscale ranges are whitelisted in fail2ban to prevent self-lockout.

Nginx HTTP Protection (modules/services/web/nginx.nix + lib/nginx.nix)

All HTTP virtual hosts (20+ services) inherit a global protection baseline via genNginxVhostBase in lib/nginx.nix. No per-service configuration needed.

Rate Limiting

Two limit_req_zone pools shared across all vhosts:

Connection Limiting

A single IP cannot open more than 20 concurrent HTTP connections to any service. Legitimate browsers use 6-8 connections per domain.

Slow-Loris / Slow-Read Protection

client_header_timeout 10s;  # was 60s (nginx default)
client_body_timeout   30s;  # was 60s (nginx default)

Attackers sending headers or body 1 byte at a time are disconnected after 10s/30s of inactivity.

AI Scraper / Bot Blocking

A User-Agent map blocks 34 known AI scrapers, data-harvesting bots, and SEO crawlers by returning HTTP 444 (drop connection, no response body):

GPTBot, Claude-Web, ClaudeBot, CCBot, anthropic-ai, Bytespider,
PerplexityBot, cohere, Google-Extended, Applebot-Extended,
AhrefsBot, SemrushBot, MJ12bot, DotBot, PetalBot, BLEXBot, ...

Bots are identified by $http_user_agent pattern matching. The if ($block_bot) { return 444; } directive runs at the server level, before any location processing — zero overhead for legitimate traffic. The list is maintained in commonHttpConfig in modules/services/web/nginx.nix.

Note: This blocks AI crawlers that respect robots.txt AND those that don’t. It targets the HTTP request itself, not the crawl policy. If you use these bots (e.g., for search indexing), remove the corresponding entry from the map.

fail2ban

Two jails protect the HTTP surface:

Tailscale IPs (100.64.0.0/10) are excluded to prevent self-lockout.

Stalwart Email (modules/services/net/stalwart.nix)

When publicEmail.enable = true, the mail server has four-layer abuse protection:

SMTP AUTH brute-force protection is handled at the nginx connection limit layer rather than via fail2ban, because the TCP stream proxy obscures the real client IP from Stalwart. See email.md for the full security design, tuning guidance, and residual risks.

Proxy (modules/services/net/sing-box.nix)

Sing-box handles traffic routing with protocol-level isolation. Workstation TUN configs should use encrypted DNS, strict routing, and explicit local-network exclusions:

• Remote DoH goes through the proxy path; China/private DNS goes through local DoH.
• TUN uses auto_route, auto_redirect, and strict_route on Linux.
• Tailscale, LAN, container bridge, libvirt, Waydroid, and MicroVM private ranges stay local.
• Public internet traffic from browsers, host services, and containers can still route through Sing-box.
• VLESS/REALITY multiplex stays disabled by default to reduce correlation and avoid server-compatibility surprises.

See docs/networking-proxy.md for the detailed Sing-box policy.

Layer 6: Mandatory Access Control (AppArmor)

security.apparmor.enable = true;
security.apparmor.killUnconfinedConfinables = true;

AppArmor confines processes to predefined profiles. killUnconfinedConfinables terminates any process that isn’t confined by an AppArmor profile — a strict policy that ensures all running processes are under MAC control.

Layer 7: Application Sandbox (Nixpak)

What is Nixpak?

Nixpak wraps GUI applications in bubblewrap-based sandboxes using a Flatpak-like model. It provides per-app DBus filtering via xdg-dbus-proxy, GPU passthrough, filesystem isolation, and Flatpak-compatible metadata — all built from native Nix derivations without ostree or runtimes.

Our integration lives in modules/sandbox.nix, which defines three shared nixpak modules (gui-base, network, common) and a mkNixPakApp helper. Per-app configs in the apps attrset import the shared modules and add app-specific bind mounts, sockets, and environment variables.

Sandbox Architecture

                    ┌──────────────────────────┐
                    │  wrapperScript            │
                    │  HOME=$XDG_FAKE_HOME      │
                    │  XAUTHORITY fallback       │
                    └──────────┬───────────────┘
                               │ exec
                    ┌──────────▼───────────────┐
                    │  nixpak launch script     │
                    │  exports BWRAP_EXE,       │
                    │  NIXPAK_APP_EXE, etc.     │
                    └──────────┬───────────────┘
                               │ exec
                    ┌──────────▼───────────────┐
                    │  nixpak launcher (Go)     │
                    │  Reads bwrap-args.json    │
                    │  Starts xdg-dbus-proxy    │
                    │  Creates .var dirs         │
                    └──────────┬───────────────┘
                               │ exec
                    ┌──────────▼───────────────┐
                    │  bubblewrap (bwrap)       │
                    │  --unshare-user/pid/net   │
                    │  --ro-bind /nix/store     │
                    │  --bind .var/app/<appId>  │
                    │  --setenv HOME ...        │
                    └──────────┬───────────────┘
                               │
                    ┌──────────▼───────────────┐
                    │  Sandboxed Application    │
                    └──────────────────────────┘

What the Sandbox Provides

Data Isolation Model

Nixpak apps follow a Flatpak-style data layout under $HOME/.var/app/<appId>/:

~/.local/user/.var/app/<appId>/
├── data/      → bind-mounted to $XDG_DATA_HOME
├── config/    → bind-mounted to $XDG_CONFIG_HOME
└── cache/     → bind-mounted to $XDG_CACHE_HOME

Because our wrapper sets HOME=~/.local/user before the nixpak launcher runs, the entire .var tree lives under the fake home. The configuration bridge (~/.local/user/.config → ~/.config) still applies for host config files exposed via bind.ro.

Shared Nixpak Modules

Three reusable modules in modules/sandbox.nix define the sandbox baseline:

Per-app configs import these via sharedModules and add app-specific bind.rw, sockets, and env.

Sandboxed Applications

Note: QQ (com.qq.QQ) works via nixpak after clearing stale config data. WeChat (com.tencent.WeChat) uses a dedicated bubblewrap wrapper in packages/wechat/ — nixpak’s --unshare-user conflicts with CEF’s internal GPU sandbox, so the custom wrapper provides PID/IPC/cgroup/tmpfs isolation without unsharing the user namespace. Data is stored under $XDG_FAKE_HOME/WeChat_Data/. Both are disabled by default; set modules.desktop.apps.messaging.{qq,wechat}.enable = true to enable.

Non-sandboxed messaging apps (Element, Fractal) are installed directly without wrapping since they’re open source.

How to Sandbox a New App

1. Add a nixpak definition in modules/sandbox.nix:

myapp = mkNixPakApp {
  package = pkgs.myapp;
  binPath = "bin/myapp";       # binary inside the package
  appId = "com.example.MyApp";  # Flatpak-style app ID
  extraConfig = {sloth, ...}: {
    bubblewrap = {
      bind.rw = [
        sloth.xdgDocumentsDir
        sloth.xdgDownloadDir
      ];
      sockets = {
        x11 = false;
        wayland = true;
        pipewire = true;
      };
    };
  };
};

2. Add a toggle in the consuming module (e.g. modules/desktop/apps/myapp.nix):

options.modules.desktop.apps.myapp = {
  enable = mkBoolOpt false;
  sandbox = mkBoolOpt true;
};
config = mkIf cfg.enable {
  user.packages = [
    (if cfg.sandbox
     then pkgs.nixpaks.myapp
     else wrapFakeHome pkgs.myapp "myapp")
  ];
};

The overlay at the bottom of modules/sandbox.nix automatically exposes all apps as pkgs.nixpaks.*.

When NOT to Use Nixpak

Verifying the Sandbox

# Check if an app is running inside bwrap:
ps aux | grep bwrap

# Verify HOME isolation:
# Inside the sandbox:
echo $HOME  # Should be ~/.local/user, not ~
ls ~/.var/app/  # Should show app data dirs

# Check DBus proxy:
ls /run/user/$(id -u)/nixpak-bus*  # nixpak DBus sockets

# Verify process isolation:
# Inside the sandbox:
ls /proc/  # Only sandbox processes visible (unshare-pid)

App-by-App Security Posture

Browsers

Recommendation: Firefox is the most hardened browser. Use it for untrusted browsing. Chromium for sites requiring Chrome compatibility.

Messaging

Meeting

Editors

Security Checklist for New Hosts

When commissioning a new host, verify:

• LUKS2 encryption enabled on root partition
• FIDO2 enrolled: sudo systemd-cryptenroll --fido2-device=auto /dev/...
• YubiKey PAM configured: modules.profiles.hardware = ["yubikey"]
• SSH key-only auth: modules.services.net.ssh.enable = true
• Firewall enabled: networking.firewall.enable = true
• Fail2Ban active: modules.services.monitoring.fail2ban.enable = true
• Agenix secrets scoped: check nodes attribute in secrets.nix
• USBGuard configured: populate device ID rules
• AppArmor active: security.apparmor.enable = true
• Secure boot (workstations): modules.profiles.boot = "lanzaboote"
• Mutable users disabled: users.mutableUsers = false

Future Improvements

These items are planned but not yet implemented. Ordered by impact/effort ratio.

Fail2Ban Jail Expansion

Current jails: SSH, Forgejo, Vaultwarden. Should be extended to cover:

Implementation: Add filter definitions to modules/services/monitoring/fail2ban.nix and reference from each service module, or centralize via services.fail2ban.jails in the host config.

Intrusion Detection (aide)

File integrity monitoring to detect trojaned binaries, tampered configs, or unexpected persistent state changes:

services.aide = {
  enable = true;
  config = ''
    /etc           p+i+u+g+s+md5
    /persist       p+i+u+g+md5
    /nix/store     p+i+u+g
  '';
};

Runs nightly via systemd timer. Sends diffs to Gotify or email on change. Minimal overhead (~5MB rules DB). Catches post-intrusion tampering.

Future escalation path: wazuh agent → centralized log analysis + MITRE ATT&CK mapping, or osquery + falco for continuous system introspection and kernel-level anomaly detection.

Service Health Dashboard (gatus)

Declarative endpoint monitoring with config-as-code:

services.gatus = {
  enable = true;
  settings = {
    endpoints = [
      {
        name = "Forgejo";
        url = "https://git.alienzj.org";
        interval = "5m";
        conditions = ["[STATUS] == 200"];
        alerts = [{ type = "gotify"; }];
      }
      # ... repeat for each service
    ];
  };
};

A single JSON/YAML file defines all health checks. No database, no agents. Exposes a simple status page. Paired with Gotify for push alerts on failure. Fits the declarative NixOS model.

Hardware Monitoring & Alerting

Prevent silent hardware degradation by monitoring temperature, fan, memory, and disk, with alerts routed to Gotify/email/Telegram.

Stack

Alert thresholds (lab-matrix)

Crash-loop detection

Today’s Mealie+Affine incident demonstrated that crash-looping services cause sustained high CPU temperatures without visible load in top. A dedicated alert rule catches this:

# Prometheus alert rule
- alert: ServiceCrashLooping
  expr: rate(systemd_unit_state{state="failed"}[5m]) > 0.1
  for: 2m
  annotations:
    summary: "{{ $labels.name }} crash-looping"

Implementation priority

1. Quick win: prometheus-node-exporter + Gotify — zero-config alerting on CPU temp and fan via a simple shell cron job reading sensors output
2. Medium: Full Prometheus stack with alertmanager rules for all thresholds above
3. Full: VictoriaMetrics (lighter than Prometheus) + Grafana dashboard for historical trends

The quick-win script (run every 60s via systemd timer):

#!/usr/bin/env bash
# /etc/scripts/check-temp.sh — quick thermal watchdog
TEMP=$(sensors -j 2>/dev/null | jq '.["k10temp-pci-00c3"]["Tctl"]["temp1_input"]')
if [ "${TEMP%.*}" -gt 85 ]; then
  curl -s "https://push.alienzj.org/message?token=<token>" \
    -F "title=CPU HOT" -F "message=lab-matrix Tctl: ${TEMP}°C" -F "priority=5"
fi

Service Network Isolation (4.1)

Currently all 18 lab-matrix services share one network namespace and localhost. A compromised service can reach PostgreSQL, Redis, other services, and internal APIs. Isolation limits lateral movement.

Architecture

Today:                         Target:
┌──────────────────────┐       ┌──────────────────────────┐
│ lab-matrix (host)    │       │ lab-matrix (host)        │
│                      │       │                          │
│ PostgreSQL :5432     │       │  ┌─────────────────┐     │
│ Redis :6379          │       │  │ dmz-net (bridge) │     │
│ Kanidm :8443         │       │  │  nginx :80       │     │
│ Immich :2283         │       │  │  gatus :9091     │     │
│ Vaultwarden :8222    │       │  └─────────────────┘     │
│ Forgejo :3000        │       │                          │
│ ...15 more services  │       │  ┌─────────────────┐     │
│                      │       │  │ app-net (bridge) │     │
│ All share localhost  │       │  │  Immich :2283    │     │
└──────────────────────┘       │  │  Forgejo :3000   │     │
                               │  │  ...            │     │
                               │  └─────────────────┘     │
                               │                          │
                               │  ┌─────────────────┐     │
                               │  │ vault-net       │     │
                               │  │  Vaultwarden    │     │
                               │  │  PostgreSQL     │     │
                               │  └─────────────────┘     │
                               └──────────────────────────┘

Implementation plan (3 phases)

Step 1: Split network zones (podman networks)

# Create isolated bridge networks
podman network create --subnet 10.89.0.0/24 dmz-net    # nginx, gatus, gotify
podman network create --subnet 10.89.1.0/24 app-net    # Immich, Forgejo, etc.
podman network create --subnet 10.89.2.0/24 vault-net  # Vaultwarden, PostgreSQL

Each service runs in its podman container with --network=<zone>. Services in the same zone can reach each other on their container-private IPs. Cross-zone communication goes through the host’s nginx or is explicitly denied.

Step 2: Gradual migration

Step 3: Remove host-level services

After all services are containerized, remove NixOS services.<name>.enable entries. The host runs only podman + nginx + tailscale + sing-box. Everything else is a container.

Cross-zone access control

# Podman network policy (iptables/nftables rules)
# vault-net → no outbound (except PostgreSQL port to app-net)
# app-net → no access to vault-net
# dmz-net → can reach app-net:service-ports only
# All zones → can reach host nginx for reverse proxy

Trade-offs

MicroVM Perimeters for High-Risk Services (4.2)

For the most sensitive services (Vaultwarden, Kanidm), containers may not be enough — a kernel exploit in one container compromises the host. MicroVMs run a separate kernel per VM, providing hardware-level isolation.

When to use MicroVM vs container

Implementation (NixOS native)

# hosts/lab-matrix/modules/virt.nix
microvm = {
  vaultwarden-vm = {
    enable = true;
    memory = 512;   # MB
    cpu = 2;
    shares = [ "/persist/vaultwarden-vm/var" ];
    modules = [
      ({ ... }: {
        services.vaultwarden.enable = true;
        # Vaultwarden runs inside the VM with its own PostgreSQL
        # or connects to host PostgreSQL via virtio-vsock
      })
    ];
  };
};

MicroVM candidates (priority order)

Centralized Log Analysis (4.4)

Today, journald on each host stores logs locally. No aggregation, no alerting on patterns, no retention beyond rotation. For a multi-service homelab, centralized logging enables:

• Correlation: See that a nginx 401 spike correlates with a Kanidm restart
• Retention: Keep service logs beyond journald rotation (1 week → 6 months)
• Alerting: Trigger Gotify alerts on log patterns (e.g., “failed auth from new IP”)
• Compliance: Audit trail for who accessed what and when

Architecture options

Recommended: Loki + Promtail (simplest fit for NixOS)

┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│ lab-matrix   │    │ vps-pacman   │    │ id3-eniac    │
│              │    │              │    │              │
│ promtail ────┼────┼── promtail ──┼────┼── promtail ──┼───┐
│ (journald)   │    │ (journald)   │    │ (journald)   │   │
└──────────────┘    └──────────────┘    └──────────────┘   │
                                                           │
                                                    ┌──────▼──────┐
                                                    │ lab-matrix   │
                                                    │              │
                                                    │ Loki :3100   │
                                                    │ (S3 backend  │
                                                    │  or local    │
                                                    │  filesystem) │
                                                    └──────┬──────┘
                                                           │
                                                    ┌──────▼──────┐
                                                    │ Grafana      │
                                                    │ (query logs  │
                                                    │  + metrics   │
                                                    │  in one UI)  │
                                                    └──────────────┘

Implementation

# modules/services/monitoring/loki.nix (future module)
services.loki = {
  enable = true;
  config = {
    server.http_listen_port = 3100;
    ingester.wal.dir = "/var/lib/loki/wal";
    storage_config.filesystem.directory = "/var/lib/loki/data";
    schema_config.configs = [{
      from = "2025-01-01";
      store = "tsdb";
      object_store = "filesystem";
      schema = "v13";
      index = { prefix = "index_"; period = "24h"; };
    }];
  };
};

services.promtail = {
  enable = true;
  config = {
    server.http_listen_port = 9080;
    clients = [{ url = "http://127.0.0.1:3100/loki/api/v1/push"; }];
    scrape_configs = [{
      job_name = "journal";
      journal = { };
      relabel_configs = [
        { source_labels = ["__journal__systemd_unit"];
          target_label = "unit"; }
      ];
    }];
  };
};

Alert rules (LogQL → Gotify)

# Alert on SSH brute force (5+ failed auths in 5min)
- alert: SSHButeForce
  expr: |
    sum(rate({job="journal", unit="sshd.service"}
      |~ "Failed password"[5m])) > 5
  annotations:
    summary: "SSH brute force on {{ $labels.host }}"

Priority deployment

Service data on lab-matrix should be encrypted at rest in backups. Current persistence covers /persist state on the host itself, but off-site backups need separate encryption.

Strategy

Primary: restic + rclone to remote storage (Backblaze B2, S3, or another Tailscale host).

# Initialize a restic repository (one-time)
restic init --repo rclone:b2:alienzj-homelab-backup

# Nightly backup of all persisted service data
restic backup /persist/var/lib \
  --exclude '*.tmp' --exclude 'cache' \
  --repo rclone:b2:alienzj-homelab-backup \
  --password-file /run/agenix/restic-password

# Prune old snapshots (keep 7 daily, 4 weekly, 6 monthly)
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 \
  --repo rclone:b2:alienzj-homelab-backup \
  --password-file /run/agenix/restic-password

Secondary: btrfs send to an external disk attached to lab-matrix.

Age secrets needed

# Restic repository password (raw)
echo -n 'strong-password' | agenix -e restic-password.age

# Rclone config with cloud credentials (KEY=VALUE)
cat > /tmp/rclone-env.txt << 'SECRET'
B2_ACCOUNT_ID=xxx
B2_ACCOUNT_KEY=xxx
SECRET
agenix -e rclone-env.age < /tmp/rclone-env.txt

What to back up

Recovery test

Test restoration quarterly by restoring a single service to a temp directory:

restic restore latest --target /tmp/restore-test --include /persist/var/lib/vaultwarden

idm_admin Hardening

idm_admin is a built-in system account in Kanidm — it doesn’t support WebAuthn enrollment via the UI. To enforce YubiKey MFA for admin operations:

1. Create a regular person account (e.g., admin-alienzj)
2. Enroll YubiKey via Kanidm web UI (Profile → security key)
3. Add to idm_admins group: kanidm group add-members idm_admins admin-alienzj
4. Use this account for admin operations instead of idm_admin
5. Keep idm_admin as break-glass recovery (requires root shell on lab-matrix)

Current Attack Surface (vps-pacman)

SSH is well-protected. Sing-box ports appear as random HTTPS to scanners — REALITY rejects unauthenticated clients.

Related Documentation