Networking Proxy

Networking: Proxying, Tunnels & SSH

Advanced traffic routing, transparent proxying, and secure shell access.

Overview

While our Tailscale/Headscale mesh provides the primary layer of connectivity between nodes, we utilize Sing-box and SSH for granular traffic routing, egress proxying, and secure port forwarding.

🚀 Sing-box (Universal Proxy Platform)

Sing-box is deployed via modules/services/net/sing-box.nix as our primary egress router and proxy client.

Key Features

• TUN Mode: Transparently intercepts and routes traffic at the kernel level (tun0 interface). All internet-bound traffic is automatically proxied — no per-app proxy configuration needed.
• Dual-stack routing: TUN configs carry both IPv4 and IPv6 prefixes and route both address families unless an explicit direct/local rule applies.
• Privacy-first DoH: Remote DNS uses DoH through the proxy path; China/private domains resolve through local DoH.
• Web Panel: Integrates metacubexd as a local UI. If webPanel.enable = true, you can manage connections visually.
• Robust Execution:
  • The Systemd service is configured with LimitNOFILE = 1048576 for high-throughput connections.
  • Cleans up stale cache.db from manual sudo runs (DynamicUser can’t remove root-owned files). Non-fatal — service starts even if cleanup fails.
  • powerManagement.resumeCommands restarts sing-box after system sleep/wake (see below).
  • Required Linux capabilities (CAP_NET_ADMIN, CAP_NET_RAW, etc.) are explicitly granted to the dynamic user.
• Encrypted Config: config.json is never stored in plaintext; it is injected securely via Agenix into Systemd’s LoadCredential directory.

TUN Mode vs Explicit Proxy

With TUN mode enabled, sing-box transparently intercepts all internet-bound traffic. Do not configure applications to use the SOCKS5 proxy (127.0.0.1:2080) unless you have a specific reason. TUN already handles everything:

App → system routing → TUN (tun0) → sing-box → proxy outbounds → internet
App → 100.64.0.0/10   → TUN excludes → tailscale0 → Tailnet

Adding an explicit SOCKS5 proxy (network.proxy.type = 1) on top of TUN causes double-proxying and breaks Tailscale access:

App → SOCKS5 :2080 → sing-box mixed-in → direct outbound
→ strict_route fwmark → DROPPED (can't reach Tailscale IPs)

If an app MUST use explicit proxy (e.g., Firefox proxy profile), ensure its proxy bypass list includes all local and Tailscale ranges:

no_proxies_on = "127.0.0.1,localhost,
  100.64.0.0/10,fd7a:115c:a1e0::/48,   ← Tailscale
  192.168.0.0/16,10.0.0.0/8,172.16.0.0/12"   ← LAN

This is configured in modules/desktop/browsers/firefox.nix under proxyConfigPacman.

Sleep / Wake Resilience

sing-box stops before system sleep (Conflicts=sleep.target) because TUN/routing state does not survive suspend. powerManagement.resumeCommands restarts it on wake — a direct post-resume hook that runs after the kernel resumes, independent of systemd target deactivation ordering.

Going to sleep:  sleep.target activates → Conflicts= stops sing-box
Waking up:       kernel resumes → powerManagement.resumeCommands fires
                   → systemctl restart sing-box

The old sing-box-resume companion service (wantedBy + bindsTo=sleep.target with ExecStopPost) was unreliable — sleep.target doesn’t always deactivate on resume, so ExecStopPost never fires and sing-box stays dead.

To verify after wake:

systemctl is-active sing-box tailscaled systemd-networkd
tailscale ping -c 2 nas-nasa
ip link show tun0 | grep -q UP && echo "tun0 OK"

Privacy-First DNS Policy

Host sing-box secrets should follow this split DNS model:

• dns-remote: DoH to a privacy-oriented resolver, with detour = "proxy".
• dns-local: DoH to a local/regional resolver, with detour = "direct".
• dns-tailscale: UDP to 100.100.100.100, with detour = "direct", only for .ts.net and .tailscale.net. The detour is mandatory — without it, sing-box’s internal routing would send the DNS queries themselves through the proxy outbounds, and 100.100.100.100 (a CGNAT address) would be unreachable.
• dns-fakeip: fake-IP answers for proxied domains so TUN routing can carry domain intent into route rules.

China/private domains resolve through dns-local and route direct. Non-China/GFW-sensitive domains resolve through dns-fakeip and route proxy. The default resolver for ordinary dial fields should be dns-local unless the upstream endpoint is a domain that must be resolved through another path.

Do not keep a dns-bootstrap resolver just because older configs had one. Bootstrap DNS is only needed when sing-box must resolve a hostname before it can reach an upstream, such as a proxy server named proxy.example.com or a DoH server named dns.example.com. If proxy and DoH servers are IP literals, bootstrap DNS is unnecessary and adds another plaintext/UDP path.

DNS and route rules should block known-malware, phishing, cryptomining, ads, tracker, and explicitly unwanted site categories before proxy/direct routing. Rejecting HTTPS, SVCB, and ANY query types is intentional: it avoids some resolver-side endpoint discovery and ECH-related surprises at the cost of disabling some newer DNS behaviors.

TUN Routing Policy

Recommended Linux TUN settings:

{
  "auto_route": true,
  "auto_redirect": true,
  "strict_route": true,
  "stack": "system",
  "mtu": 1400
}

auto_redirect is important on Linux. It improves routing performance and avoids common Docker bridge conflicts. Keep strict_route = true for leak resistance.

Use explicit route exclusions for local-only destinations:

• Loopback: 127.0.0.1/32, ::1/128
• Link-local and multicast: 169.254.0.0/16, fe80::/10, 224.0.0.0/4, ff00::/8
• LAN and container/VM bridge ranges: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, host-specific MicroVM ULA prefixes
• Tailscale/Headscale: 100.64.0.0/10, fd7a:115c:a1e0::/48
• TUN self-ranges and proxy/DNS endpoint IPs

These exclusions protect local reachability; they do not mean containers bypass the proxy for internet egress. Docker, Podman, libvirt, Waydroid, and MicroVM traffic to public internet destinations can still enter TUN. Only traffic to private bridge/LAN/Tailscale destinations stays local.

Important: route_exclude_address only prevents the TUN from intercepting traffic to those IPs. The mixed inbound (SOCKS5 on :2080) does not respect exclusions — it routes all connections through sing-box’s own route rules. For Tailscale IPs to work through the mixed inbound, the host’s sing-box config MUST include an explicit route rule:

{
    "ip_cidr": ["100.64.0.0/10", "fd7a:115c:a1e0::/48"],
    "outbound": "direct"
}

Without this rule, 100.64.0.0/10 is not ip_is_private (it’s CGNAT, not RFC1918) and falls through to the catch-all proxy outbounds, which can’t reach Tailscale IPs.

MTU and Multiplex

Keep TUN MTU at 1400 by default. It is a conservative balance for IPv6, Wi-Fi, PPPoE-like links, and GFW-sensitive paths. Lower only when large transfers or TLS sessions stall; raise only after testing on the exact host/network.

Keep VLESS multiplex disabled by default when using REALITY plus xtls-rprx-vision. Multiplex can reduce connection count, but it creates long-lived high-activity proxy connections, can increase traffic correlation, and may hurt latency-sensitive or mixed workloads. If needed, test multiplex as a separate outbound rather than enabling it on all production outbounds.

Integration with SSH Tunnels

Sing-box works seamlessly with our automated SSH tunneling system (see below). If sshTunnel.enable = true, Sing-box routes its outbound proxy traffic through a local SOCKS5 port generated by an SSH dynamic tunnel.

Verification

For the full operational guide — config lifecycle, validation, deployment, debugging, DNS architecture, and common error reference — see docs/sing-box.md.

Quick daily checks:

# Service health
systemctl status sing-box --no-pager
systemctl is-active sing-box

# Recent errors (filter expected scanner noise)
journalctl -u sing-box --since '10min ago' --no-pager |
  grep -v 'invalid connection\|mux connection closed'

# Config syntax (on target host)
sing-box check -c /path/to/config.json

Common Proxy Failures

🔒 Secure Shell (SSH)

The SSH configuration (modules/services/net/ssh.nix) is highly hardened and acts as both an administrative backdoor and a secure transport layer for proxying.

Hardening Policies

• No Passwords: PasswordAuthentication and KbdInteractiveAuthentication are strictly disabled. Only cryptographic keys (primarily ed25519 and YubiKey-backed ed25519-sk) are accepted.
• Root Login: Set to prohibit-password.
• Moduli Filtering: Automatically filters /etc/ssh/moduli to remove weak Diffie-Hellman parameters (drops anything < 3071 bits).
• Timeouts: Aggressive ClientAliveInterval settings to drop dead connections quickly.
• Extensibility: You can inject host-specific or client-specific configurations via extraConfig and extraClientConfig.

Automated SSH Tunneling

The sing-box.nix module provides two powerful Systemd user services for automated SSH tunnels:

1. SOCKS SSH Tunnel (ssh-socks-tunnel)

   • What it does: Creates a local SOCKS5 proxy (e.g., 127.0.0.1:10800) by executing ssh -D 10800 -N host.
   • Use Case: Used by Sing-box as a secure, encrypted outbound gateway.

2. Reverse SSH Tunnel (ssh-reverse-tunnel)

   • What it does: Forwards remote ports from a jump host back to the local network using ssh -R.
   • Use Case: Bypassing strict NATs by opening ports on a VPS (e.g., 20022 on the VPS forwards to port 22 locally).

Note: Both services are configured with strict ServerAliveInterval and ExitOnForwardFailure = yes to ensure they automatically restart if the network drops.

🌐 The “Big Picture” Traffic Flow

How all our networking layers interact:

1. Ingress (Nginx + Fail2Ban): Public traffic hits the VPS. Nginx terminates the SSL, checks for Kanidm SSO authorization, and Fail2Ban monitors the logs.
2. Backbone (Tailscale/Headscale): Nginx forwards the clean traffic over the encrypted WireGuard mesh to a homelab node.
3. Egress (Sing-box + SSH): If a homelab node needs to access the internet (or bypass regional firewalls), Sing-box intercepts the traffic via TUN, routes it through the SSH SOCKS tunnel, and sends it out securely.