Sing Box

Sing-box Operations

Config lifecycle, validation, deployment, debugging, and DNS architecture for sing-box across the fleet.

Architecture Overview

The fleet uses sing-box in two distinct roles:

Client intercepts local traffic via TUN and routes it through VLESS outbounds.
Server accepts VLESS+REALITY connections and forwards them to the internet via direct.

Client (eniac)                        Server (pacman/ultraman)
┌──────────────┐                      ┌──────────────────┐
│ App → TUN    │                      │ VLESS+REALITY    │
│   → sing-box │── VLESS+REALITY ──→  │   → direct       │── Internet
│   → pacman-1 │   (encrypted)       │   → Cloudflare    │
└──────────────┘                      └──────────────────┘

Both roles share identical DNS tag naming conventions (see DNS Tags).

Config File Locations

All sing-box configs are agenix-encrypted secrets stored in the secrets repo:

nix-secrets/secrets/config/sing-box/
├── id3-eniac.singboxConfigFile.json     # Client (workstation)
├── vps-pacman.singboxConfigFile.json    # Server (L1 gateway: +nginx, +oauth2-proxy)
├── vps-ultraman.singboxConfigFile.json  # Server (pure proxy node)
├── id3-yoga.singboxConfigFile.json      # Client (laptop)
├── lab-matrix.singboxConfigFile.json    # Client (lab)
└── ...

The Nix module (modules/services/net/sing-box.nix) wires these into systemd:

age.secrets."${hostName}.singboxConfigFile".path
→ LoadCredential = "config.json:${secretPath}"
→ ExecStart = "sing-box run -c ${CREDENTIALS_DIRECTORY}/config.json"

Per-host options are set in hosts/<name>/modules/modules.nix:

sing-box = {
  enable = true;
  tun.enable = false;       # false = server mode, true = client mode
  webPanel.enable = true;   # metacubexd UI (Clash API port set in encrypted config, default :9090)
  ports = [10443 20443 30443];
};

Config Lifecycle

1. Edit

Edit the plaintext JSON in nix-secrets/secrets/config/sing-box/<host>.singboxConfigFile.json.

Safety rules:

• Never commit plaintext secrets to the dotfiles repo — these files live in the separate nix-secrets repo
• Run all changes against a live sing-box before committing
• Keep the original file as backup until verified
• Do not change UUIDs, private keys, or short_ids unless you know the corresponding client/server outbounds

2. Validate (Python + sing-box)

Always validate in two steps — strict JSON first, then sing-box semantic check:

# Step 1: Strict JSON validity
python3 -c "import json; json.load(open('config.json'))" && echo "valid"

# Step 2: sing-box semantic check (must pass on target host's sing-box version)
ssh <host>_root 'cat > /tmp/check.json && /nix/store/*/bin/sing-box check -c /tmp/check.json' < config.json

Always validate on the target host — sing-box versions differ across hosts and flag compatibility changes between releases (e.g., domain_strategy deprecated in 1.12, sniff on inbounds removed in 1.13).

Common validation failures:

3. Deploy

Write to the agenix secret path then restart:

# Upload (separate SSH call — stdin is consumed!)
ssh <host>_root 'cat > /run/agenix/<host>.singboxConfigFile' < config.json

# Verify file size (must be > 100 bytes)
ssh <host>_root 'wc -c /run/agenix/<host>.singboxConfigFile'

# Restart
ssh <host>_root 'systemctl restart sing-box'

Critical: Always upload and restart in separate SSH calls. Chaining cat > file && restart inside a single ssh "..." causes the second command to receive EOF on stdin.

For the local host, use sudo:

sudo cp config.json /run/agenix/$(hostname).singboxConfigFile
sudo systemctl restart sing-box

4. Verify

# Check service health
ssh <host>_root 'systemctl is-active sing-box'
# active

# Check for errors (filter out expected REALITY scanner noise)
ssh <host>_root "journalctl -u sing-box --since '2min ago' --no-pager" |
  grep -v 'invalid connection\|mux connection closed\|Scanner'

Healthy output: zero FATAL lines, zero lookup.*EOF lines.
Expected noise: REALITY: processed invalid connection from random IPs (internet scanners probing REALITY ports — this is normal and means REALITY is working).

Debugging

Service Status

systemctl status sing-box --no-pager
systemctl is-active sing-box
journalctl -u sing-box -n 100 --no-pager

Common Error Patterns

DNS Chain Testing (Client)

# Test local DoH (AliDNS for China domains)
curl -s --max-time 3 --doh-url https://223.5.5.5/dns-query example.com

# Test remote DoH through proxy
curl -sk --max-time 5 --socks5 127.0.0.1:2080 \
  'https://dns.quad9.net/dns-query?name=example.com' -o /dev/null -w "HTTP %{http_code}\n"

# Test Tailscale MagicDNS (should respond)
ping -c1 -W2 100.100.100.100

DNS Chain Testing (Server)

# Test each DoH resolver
curl -s --max-time 3 --doh-url https://1.1.1.1/dns-query example.com | head -3
curl -s --max-time 3 --doh-url https://9.9.9.9/dns-query example.com | head -3
curl -s --max-time 3 --doh-url https://8.8.8.8/dns-query example.com | head -3

# Test Tailscale DNS via MagicDNS
dig @100.100.100.100 <host>.ts.net +short

Proxy Chain Testing (Client)

# Test internet through SOCKS5 proxy
curl -sk --max-time 5 --socks5 127.0.0.1:2080 https://www.google.com \
  -o /dev/null -w "HTTP %{http_code}\n"

# Test Tailscale reachability through proxy
curl -sk --max-time 5 --socks5 127.0.0.1:2080 http://<tailscale-ip>:5001 \
  -o /dev/null -w "HTTP %{http_code}\n"

# Test via TUN (traffic automatically intercepted)
curl -sk --max-time 5 https://www.google.com -o /dev/null -w "HTTP %{http_code}\n"

TUN Interface Verification (Client)

# TUN must be UP
ip link show tun0 | grep -q UP && echo "tun0 OK"

# Default route through TUN
ip route show default | grep tun0

# Tailscale MUST bypass TUN
ip route get 100.64.243.70 | grep tailscale0  # should show tailscale0

# Public internet MUST go through TUN
ip route get 8.8.8.8 | grep tun0              # should show tun0

Proxy Target Verification (Server)

# Verify sing-box is listening on expected ports
ss -tlnp | grep sing-box
# Expected: :10443 :20443 :30443 (or host-specific ports)

# Verify no port conflicts with nginx/other services
ss -tlnp | grep -E ':80 |:443 '

DNS Tag Reference

All configs use a unified naming convention across client and server:

Server DNS flow:

Query → [ad/tracker rules? → reject]
      → [.ts.net / .tailscale.net? → dns-tailscale (100.100.100.100)]
      → [HTTPS/SVCB/ANY? → reject]
      → final: dns-cloudflare (1.1.1.1)

Client DNS flow:

Query → [ad/tracker rules? → reject]
      → [.ts.net? → dns-tailscale (100.100.100.100)]
      → [China/private? → dns-local (223.5.5.5)]
      → [!cn domains? → dns-fakeip (FakeIP)]
      → [A/AAAA catch-all? → dns-fakeip]
      → final: dns-cloudflare (1.1.1.1 via proxy)

default_domain_resolver setting:

• Server: dns-cloudflare (for direct outbound when it needs to resolve domains)
• Client: dns-local (for direct outbound — China/local domains resolve via AliDNS)

Route Rule Reference

Server Rules (priority order)

Client Rules (priority order)

Tailscale Integration

Both server and client configs are designed to coexist with Tailscale:

DNS:

• .ts.net and .tailscale.net domains resolve via dns-tailscale (100.100.100.100) — Tailscale’s virtual MagicDNS server, always reachable when tailscaled is running
• disable_cache: true on Tailscale DNS rules prevents stale MagicDNS entries
• dns-tailscale does NOT use detour — MagicDNS (100.100.100.100) is a virtual IP handled by the tailscale daemon, not a real network endpoint

Routing:

• 100.64.0.0/10 (Tailscale CGNAT) and fd7a:115c:a1e0::/48 (Tailscale IPv6 ULA) route direct
• Server: placed BEFORE the private-IP reject rule (so Tailscale IPs are allowed)
• Client: placed before the catch-all proxy rule (so tailnet traffic stays local)
• Client TUN: these ranges are in route_exclude_address so they never enter TUN

No TUN on server: The server doesn’t use TUN mode. Local processes (nginx, oauth2-proxy, sshd) use the host’s DNS and networking directly — sing-box does not intercept their traffic.

TUN Coexistence with Tailscale (Client)

On client hosts, sing-box’s TUN (tun0) and Tailscale’s WireGuard (tailscale0) operate side by side. Making them coexist requires three mechanisms working together:

1. Route Exclusion (route_exclude_address)

The TUN inbound explicitly excludes Tailscale IP ranges from interception:

"route_exclude_address": [
  ...
  "100.64.0.0/10",        // Tailscale IPv4 mesh
  "fd7a:115c:a1e0::/48"   // Tailscale IPv6 ULA
]

This means packets destined for any 100.64.x.x address are not captured by tun0 — they follow the host’s normal routing table instead.

2. Route Rules (direct priority)

The route rule chain places Tailscale before the catch-all proxy:

Rule priority (top to bottom):
  ...
  → 100.64.0.0/10       → direct    ← Tailscale IPs go direct
  → fd7a:115c:a1e0::/48 → direct    ← Tailscale IPv6 ULA direct
  → ip_is_private: true  → direct    ← LAN (192.168.x.x, 10.x, 172.16.x)
  → 198.18.0.0/15       → proxy     ← FakeIP → proxy
  → geosite-!cn         → proxy     ← Foreign domains → proxy
  → 0.0.0.0/0           → proxy     ← Everything else → proxy

This ensures that even if a process explicitly sends traffic through sing-box’s SOCKS5 proxy (:2080) or the TUN doesn’t exclude the range, the route rule still sends it direct — no proxy hop.

3. Verification:

# Tailscale traffic must NOT enter tun0
ip route get 100.64.243.70
# Expected: 100.64.243.70 dev tailscale0 table 52

# Public internet MUST enter tun0
ip route get 8.8.8.8
# Expected: 8.8.8.8 dev tun0 ...

Failure mode: If route_exclude_address is missing 100.64.0.0/10, Tailscale traffic enters tun0 → sing-box processes it → if the route rule sends it to proxy instead of direct, WireGuard packets get forwarded to a VLESS outbound → Tailscale can’t establish P2P → forced DERP relay.

DNS Interaction with Tailscale MagicDNS

Sing-box’s DNS module and Tailscale MagicDNS operate at different layers:

Application DNS query
  → intercepted by sing-box TUN (protocol: dns → hijack-dns)
  → sing-box DNS rules:
      ├─ .ts.net / .tailscale.net? → dns-tailscale (100.100.100.100:53)
      │    └─ Tailscale daemon resolves to 100.64.x.x mesh IP
      ├─ geosite-private?          → dns-local (223.5.5.5)
      └─ final                    → dns-cloudflare (1.1.1.1 via proxy)

Key details:

• MagicDNS (100.100.100.100) is a virtual address handled by tailscaled — it never leaves the host, so it does NOT need a detour in sing-box
• .ts.net queries are routed to dns-tailscale with disable_cache: true to prevent stale MagicDNS cache entries
• Tailscale hostnames resolve to 100.64.x.x addresses, which the route rules send direct via tailscale0 — no proxy involvement
• Non-Tailscale DNS follows the normal sing-box flow (domains → direct or proxy depending on geosite rules)

DNS leak check: Verify that Tailscale hostnames resolve correctly:

dig @100.100.100.100 nas-nasa.ts.net +short
# Should return a 100.64.x.x address

Application-Layer Impact of 100.64.0.0/10

Even with perfect sing-box/Tailscale routing, applications may misclassify Tailscale mesh IPs because 100.64.0.0/10 is not RFC 1918 private:

This means any application that determines “local vs remote” via ipaddress.is_private / is_global will treat your Tailscale mesh as the public internet.

Consequences:

• Streaming apps (jellyfin-mpv-shim, Plex, Emby): apply remote_kbps throttle (often ~10 Mbps) instead of full local_kbps
• File sync (Syncthing, Resilio): force relay-only mode for “non-LAN” peers
• Any Python app doing ipaddress.is_private checks: misclassifies your NAS

Mitigation:

1. Always prefer LAN IPs (192.168.x.x) for services on your local network. Use Tailscale IPs only for remote access.
2. Check app configs for remote_kbps/local_kbps or similar local-vs-remote split settings.
3. For apps that support it, explicitly mark 100.64.0.0/10 as a local network range (some apps allow custom LAN subnet configuration).

See also networking-vpn.md and networking-topology.md for extended discussion and real-world examples.

Sleep/Resume (Client Hosts)

On client hosts that can suspend (workstations, laptops), sing-box is stopped before sleep (Conflicts=sleep.target) and automatically restarted on wake via powerManagement.resumeCommands. The TUN interface is torn down on suspend and recreated when sing-box restarts.

Previously this used a bindsTo=sleep.target companion service with ExecStopPost, but sleep.target doesn’t reliably deactivate on resume across all hardware/kernel combinations. The powerManagement.resumeCommands hook runs directly after the kernel resumes, independent of systemd target state.

To verify the resume hook is active:

grep -r "sing-box" /run/current-system/sw/etc/systemd/system/

If sing-box is dead after resume, check:

journalctl -u sing-box --since "10min ago" --no-pager | grep -E "stop|start|Deactivat"

Pre-Deploy Checklist

Before deploying a config change:

1. python3 -c "import json; json.load(open('config.json'))" passes
2. sing-box check -c config.json passes (on target host)
3. No domain_strategy on individual DNS servers
4. default_domain_resolver matches a defined DNS tag
5. dns-tailscale tag points to 100.100.100.100:53
6. .ts.net domains use leading dot (subdomain matching)
7. 100.64.0.0/10 routes to direct (not blocked)
8. All rule_set tags are defined in route.rule_set
9. No trailing commas in strict JSON (though sing-box accepts them)
10. Original config backed up before deploy

Recovering from Bad Config

If sing-box fails to start after a config change:

# 1. Check what's wrong
journalctl -u sing-box --since '1min ago' --no-pager | grep FATAL

# 2. Restore the previous config (if backed up)
ssh <host>_root 'cp /run/agenix/<host>.singboxConfigFile.bak /run/agenix/<host>.singboxConfigFile'

# 3. Or revert to the nix-store config from the last successful build
ssh <host>_root 'cp /nix/store/*-source/config/sing-box/<host>.singboxConfigFile.json /run/agenix/<host>.singboxConfigFile'

# 4. Restart
ssh <host>_root 'systemctl restart sing-box'