Ops

Deployment Workflows

hey sync, hey install, hey ops bootstrap, hey ops deploy — four wrappers around the standard NixOS toolchain.

How hey fits into the toolchain

The hey CLI is a thin wrapper over the standard NixOS deployment tools. It:

1. Picks the right backend for the job: nixos-rebuild, nh, nixos-install, nixos-anywhere, or disko.
2. Adds sensible defaults (--show-trace, --build-host localhost for remote targets, etc.).
3. Resolves the flake from the current directory or --flake flag.

The flake is fully hermetic — no --impure, no environment variables needed for evaluation. Host identity comes from the flake attribute name (hosts/<name>/).

Quick Reference

Local Workflows

hey sync — Rebuild Current System

Wraps nh os <cmd> (preferred) or nixos-rebuild <cmd> with --show-trace.

# Rebuild and set as default (activates on next boot):
hey sync
# → nh os boot --hostname id3-eniac -- --show-trace
# → sudo nixos-rebuild --show-trace --flake /path/to/flake#id3-eniac boot

# Rebuild and activate immediately:
hey sync switch
# → nh os switch --hostname id3-eniac -- --show-trace

# Build only (don't activate):
hey sync build
# → nixos-rebuild --show-trace --flake /path/to/flake#id3-eniac build

# Rebuild for a different host:
hey sync --host vps-ultraman
# → nh os boot --hostname vps-ultraman -- --show-trace

This is the daily driver — use hey sync (boot by default) for routine changes, hey sync switch when you want instant activation.

hey install — Fresh Local Install

Wraps nixos-install. Used from a NixOS installer ISO after partitioning disks.

hey install --host id3-eniac --root /mnt
# → sudo nixos-install --show-trace \
#       --root /mnt --flake /path/to/flake#id3-eniac

• When: First-time NixOS installation on a local machine.
• Requires: Disks already partitioned and formatted (via hey disko format or manually).

Remote Workflows

hey ops bootstrap — Fresh Remote Install

Wraps nix run github:nix-community/nixos-anywhere. Installs NixOS onto a bare remote machine — kexecs into an installer, partitions the disk via disko, installs, and reboots.

hey ops bootstrap vps-ultraman vps_ultraman_root
hey ops bootstrap vps-ultraman vps_ultraman_root --no-disko-deps  # low-RAM VPS
# → nix run github:nix-community/nixos-anywhere -- \
#       --flake .#vps-ultraman \
#       [--no-reboot] [--no-disko-deps] [--debug] \
#       vps_ultraman_root

• When: First-time NixOS on a remote VPS running any Linux (Ubuntu, Arch, rescue image).
• Requires: target accessible via SSH as root, with your public key authorized. flake-host defined in hosts/.
• Options:
  • --no-reboot: Don’t reboot after bootstrap.
  • --no-disko-deps: Use disko tools from kexec image (critical for <1GB RAM VPS).
  • --debug: Verbose nixos-anywhere output.

hey ops deploy — Remote Update

Wraps nixos-rebuild switch (or boot) with --target-host and --build-host localhost. Builds locally, copies closure to target via SSH, activates in-place — no compilation on the VPS.

# Live activation (restarts changed services immediately):
hey ops deploy vps-ultraman vps_ultraman_root
# → nixos-rebuild switch --show-trace \
#       --flake .#vps-ultraman \
#       --target-host root@vps_ultraman_root \
#       --build-host localhost

# Safer: only update the bootloader entry, activate on next reboot:
hey ops deploy vps-ultraman vps_ultraman_root --boot
# → nixos-rebuild boot --show-trace \
#       --flake .#vps-ultraman \
#       --target-host root@vps_ultraman_root \
#       --build-host localhost

# Target defaults to flake-host if omitted:
hey ops deploy vps-pacman
# → nixos-rebuild switch --show-trace \
#       --flake .#vps-pacman \
#       --target-host root@vps-pacman \
#       --build-host localhost

• When: Updating an existing NixOS host that was previously bootstrapped or installed.
• Options:
  • --fast: Skip some checks for a faster deploy.
  • --boot: Use nixos-rebuild boot instead of switch — safer for remote hosts where a bad config could lock you out.
  • --debug: Verbose nixos-rebuild output.

hey ops push-keys — Provision SSH Keys

Pushes host and global identity keys to a remote host so agenix can decrypt secrets during bootstrap.

# Push keys to /etc/ssh/ (standard):
hey ops push-keys vps-ultraman vps_ultraman_root
# → scp ~/.ssh/keys.homelab/vps/ssh_host_ed25519_key_vps-ultraman vps_ultraman_root:/etc/ssh/ssh_host_ed25519_key
# → scp /persist/etc/ssh/global_ed25519 vps_ultraman_root:/etc/ssh/global_ed25519

# Push with auto-generation of missing host key:
hey ops push-keys vps-ultraman vps_ultraman_root --generate
# → ssh-keygen -t ed25519 -f ~/.ssh/keys.homelab/vps/ssh_host_ed25519_key_vps-ultraman -N '' -C 'vps-ultraman.local'

# Push to persist path (impermanent hosts):
hey ops push-keys vps-ultraman vps_ultraman_root --persist
# → Keys go to /persist/etc/ssh/ instead of /etc/ssh/

# Skip global key (host key only):
hey ops push-keys vps-ultraman vps_ultraman_root --no-global

• When: Before hey ops bootstrap on a new host — ensures agenix has access to decryption keys.
• Options:
  • --persist: Push to /persist/etc/ssh/ (for impermanent root hosts).
  • --no-global: Skip the global agenix key, push host key only.
  • --generate: Generate a new ed25519 host key if none found locally.
• Key sources: Host key is searched in ~/.ssh/keys.homelab/{vps,pc,server}/. Global key is read from /persist/etc/ssh/global_ed25519 or /etc/ssh/global_ed25519.

hey ops ssh — Remote Shell

hey ops ssh vps_ultraman_root
# → ssh root@vps_ultraman_root

hey ops sync — Alias

Alias for hey ops deploy. Identical behavior.

Typical Lifecycle

# 1. First time: push keys so agenix can decrypt secrets
hey ops push-keys vps-ultraman vps_ultraman_root --generate

# 2. Bootstrap a bare VPS
hey ops bootstrap vps-ultraman vps_ultraman_root

# 3. Ongoing: push config updates to remote
hey ops deploy vps-ultraman vps_ultraman_root

# 4. Safer updates for critical hosts (verify, then reboot)
hey ops deploy vps-ultraman vps_ultraman_root --boot

# 5. Rebuild your local workstation (daily driver)
hey sync

🛠️ Implementation Details

Backend Selection

Runtime DOTFILES_HOME

DOTFILES_HOME is set by /etc/zshenv (generated by modules/hey.nix) and points to the flake store path (/nix/store/xxx-source). It is a runtime convenience for Janet CLI tools and shell scripts — not a build-time input.

The flake uses toString self (pure) for all build-time path resolution. No --impure flag, no getEnv, no environment variables needed.

Build & Deploy Topology

nixos-rebuild can delegate build and deploy to different hosts via SSH:

┌─────────────────────┐     ┌──────────────────┐     ┌─────────────────────┐
│   where you type    │     │  where Nix builds │     │  where system runs  │
│   nixos-rebuild     │     │  (--build-host)   │     │  (--target-host)    │
└─────────────────────┘     └──────────────────┘     └─────────────────────┘

Default: When --build-host is omitted, the build happens on the machine running nixos-rebuild. For hey ops deploy vps-ultraman, that’s your workstation — the VPS never compiles anything.

Nix store copy: After the build, the closure is copied from the build host to the target host via SSH (nix copy). Only changed paths are transferred.

Self-deploy: nixos-rebuild --target-host root@localhost builds and deploys to the same machine through SSH — useful when sudo isn’t available. Requires sshd running and ssh localhost working.

SSH config: If Host * has IdentitiesOnly yes, add BEFORE it:

Host localhost
    IdentitiesOnly no

SSH uses first-match — the more specific block must precede the wildcard.

Resource Efficiency

For low-RAM VPS nodes (e.g., 512MB-1GB RAM):

• Building locally (default) ensures no compilation happens on the target.
• Btrfs subvolumes and Zstd compression (configured in storage.nix) save disk space.
• --no-disko-deps with nixos-anywhere avoids copying disko deps to tmpfs during bootstrap.

⚠️ Safety & Security

• SSH Lockout: Be careful when modifying services.openssh.extraConfig. Ensure root is allowed from your deployment IPs, or you will lose the ability to use hey ops deploy.
• Tailscale: Remote hosts are typically accessed via Tailscale IPs for added security.
• --boot vs switch: On remote hosts, prefer --boot for risky changes — you can SSH in and reboot when ready. If switch breaks SSH, you’re locked out.