Ssh

SSH Architecture

How SSH config layers, agent keys, agenix host keys, and IdentitiesOnly interact — and the MaxAuthTries pitfall that can lock you out.

Config Layers

SSH merges config from multiple sources, in priority order:

Host * blocks merge across layers. A system Host * and a user Host * both apply. User values override system ones for the same key, but IdentityFile is additive — all matching IdentityFile lines from all layers are collected, not replaced.

System config (agenix)

modules/agenix.nix injects two host keys for ALL outbound connections:

Host *
  IdentityFile /persist/etc/ssh/ssh_host_ed25519_key
  IdentityFile /persist/etc/ssh/global_ed25519

These are needed so agenix can decrypt secrets using the host’s SSH key. They are offered on every SSH connection — including to remote hosts where they don’t match any authorized_keys.

User config

~/.ssh/config defines per-host IdentityFile entries for each target:

Host vps_ultraman_root
  HostName 100.67.58.96
  User root
  IdentityFile ~/.ssh/keys.homelab/vps/ssh_host_ed25519_key_ultraman

Resolved example

For ssh vps_ultraman_root, the merged identity files are:

identityfile ~/.ssh/keys.homelab/vps/ssh_host_ed25519_key_ultraman  (user)
identityfile /persist/etc/ssh/ssh_host_ed25519_key                   (system)
identityfile /persist/etc/ssh/global_ed25519                         (system)

User files come first. All three are tried.

SSH Agent (ssh-agent)

The agent holds decrypted private keys in memory. Keys are added via ssh-add and listed with ssh-add -l.

$ ssh-add -l
256 SHA256:... alienzj@id3-eniac (ED25519)
256 SHA256:... [email protected] 20250423 (ED25519)
256 SHA256:... lab-matrix.local (ED25519)
...

By default, SSH offers every agent key in addition to config IdentityFile entries. If you have 7 agent keys + 3 config files = 10 keys offered per connection attempt.

MaxAuthTries Pitfall

OpenSSH server defaults to MaxAuthTries 6. After 6 failed key attempts, the server disconnects with:

Received disconnect: Too many authentication failures

If your agent has many keys, the correct key may never be reached.

The sequence without IdentitiesOnly

Client offers:  agent_key_1  →  reject
                agent_key_2  →  reject
                agent_key_3  →  reject
                agent_key_4  →  reject
                agent_key_5  →  reject
                agent_key_6  →  reject
                                💥 MaxAuthTries=6, disconnect
                (correct key never reached)

The fix: IdentitiesOnly yes

Setting IdentitiesOnly yes in Host * tells SSH: skip the agent, only use IdentityFile directives from config.

Client offers:  ~/.ssh/.../ssh_host_ed25519_key_ultraman  →  accept ✅

3 config keys, the correct one is first. Well within MaxAuthTries=6.

Agenix Interaction

modules/agenix.nix adds system-wide IdentityFile entries so agenix can decrypt secrets with the host’s SSH key. This is intentional and required — without these keys, agenix cannot read age.secrets.*.

Key path strategy

Paths are conditionally set based on whether persistence is enabled:

This is derived from config.modules.persist.enable at build time.

Conditional IdentityFile injection

programs.ssh.extraConfig uses pathExists at build time to only include IdentityFile entries for keys that actually exist. Missing keys are silently omitted from the generated /etc/ssh/ssh_config.

An assertion enforces that hostKey exists when age.secrets are configured — the host key is required for agenix decryption.

The trade-off: these keys (when present) are offered on every outbound SSH connection. With IdentitiesOnly yes, only 2 extra keys are offered (well within limits). Without it, they combine with all agent keys, inflating the count.

TERM and Terminfo

SSH forwards TERM through the PTY protocol automatically — no SetEnv or SendEnv needed. The remote host must have the matching terminfo entry for terminal apps (vim, tmux, htop) to work correctly.

Strategy

• modules/services/net/ssh.nix: Installs foot.terminfo on all hosts with sshd enabled, so they can handle incoming TERM=foot.
• modules/desktop/term/foot.nix: Installs foot.terminfo locally, sets tmux term to foot. Does NOT override SSH TERM globally.
• modules/shell/git.nix: Adds SetEnv TERM=xterm-256color to git Host blocks — GitHub/AUR servers lack foot terminfo.

File-level summary

Key Hygiene Best Practices

1. IdentitiesOnly yes in Host * — prevents agent keys from leaking to hosts that don’t need them. Every host block should explicitly list its IdentityFile. Must be placed LAST in the config — SSH uses first-match for each parameter, so Host localhost overrides must come BEFORE Host *.

2. Host localhost must precede Host * — if Host * sets IdentitiesOnly yes, add this ABOVE it:

   Host localhost
       IdentitiesOnly no

   Without this, ssh localhost (used by --build-host localhost) fails because no IdentityFile matches the local user’s authorized_keys.

3. Per-host identity files — each host gets a unique key. Never reuse the same key across hosts.

4. Root access uses per-host keys — authorizedKeys in alienzj.nix selects the correct key per host via config.networking.hostName.

5. Agenix host keys stay system-level — these are managed by agenix.nix, not the user config. Don’t duplicate them.

6. Debug with ssh -G — shows the resolved config for a host:

   ssh -G vps_ultraman_root | grep identityfile

7. Debug with ssh -vvv — shows which keys are offered and which is accepted:

   ssh -vvv root@<host> 2>&1 | grep -E 'Offering|Accepted|authenticated'

Related Docs

• Deployment Workflows — hey ops deploy and the --boot vs --switch distinction
• Security Hardening — SSH hardening, fail2ban
• Nix Expressions — mkAliasDefinitions bridge between options.user and users.users