Ops

Deployment Workflows

hey sync, hey install, hey ops bootstrap, hey ops deploy — four wrappers around the standard NixOS toolchain.

How hey fits into the toolchain

The hey CLI is a thin wrapper over the standard NixOS deployment tools. It:

1. Picks the right backend for the job: nixos-rebuild, nh, nixos-install, nixos-anywhere, or disko.
2. Adds sensible defaults (--show-trace, --build-host localhost for remote targets, etc.).
3. Resolves the flake from the current directory or --flake flag.

The flake is fully hermetic — no --impure, no environment variables needed for evaluation. Host identity comes from the flake attribute name (hosts/<name>/).

Quick Reference

Local Workflows

hey sync — Rebuild Current System

Wraps nh os <cmd> (preferred) or nixos-rebuild <cmd> with --show-trace.

# Rebuild and set as default (activates on next boot):
hey sync
# → nh os boot --hostname id3-eniac -- --show-trace
# → sudo nixos-rebuild --show-trace --flake /path/to/flake#id3-eniac boot

# Rebuild and activate immediately:
hey sync switch
# → nh os switch --hostname id3-eniac -- --show-trace

# Build only (don't activate):
hey sync build
# → nixos-rebuild --show-trace --flake /path/to/flake#id3-eniac build

# Rebuild for a different host:
hey sync --host vps-ultraman
# → nh os boot --hostname vps-ultraman -- --show-trace

This is the daily driver — use hey sync (boot by default) for routine changes, hey sync switch when you want instant activation.

hey install — Fresh Local Install

Wraps nixos-install. Used from a NixOS installer ISO after partitioning disks.

hey install --host id3-eniac --root /mnt
# → sudo nixos-install --show-trace \
#       --root /mnt --flake /path/to/flake#id3-eniac

• When: First-time NixOS installation on a local machine.
• Requires: Disks already partitioned and formatted (via hey disko format or manually).

Remote Workflows

hey ops bootstrap — Fresh Remote Install

Wraps nix run github:nix-community/nixos-anywhere. Installs NixOS onto a bare remote machine — kexecs into an installer, partitions the disk via disko, installs, and reboots.

hey ops bootstrap vps-ultraman vps_ultraman_root
hey ops bootstrap vps-ultraman vps_ultraman_root --no-disko-deps  # low-RAM VPS
# → nix run github:nix-community/nixos-anywhere -- \
#       --flake .#vps-ultraman \
#       [--no-reboot] [--no-disko-deps] [--debug] \
#       vps_ultraman_root

• When: First-time NixOS on a remote VPS running any Linux (Ubuntu, Arch, rescue image).
• Requires: target accessible via SSH as root, with your public key authorized. flake-host defined in hosts/.
• Options:
  • --no-reboot: Don’t reboot after bootstrap.
  • --no-disko-deps: Use disko tools from kexec image (critical for <1GB RAM VPS).
  • --debug: Verbose nixos-anywhere output.
  • --phases: Comma-separated list of phases to run (e.g. disko,install). Specify --phases install on an already partitioned system with target volumes manually mounted under /mnt to repair/re-install a configuration without formatting the disk.

hey ops deploy — Remote Update

Wraps nixos-rebuild switch (or boot) with --target-host and --build-host localhost. Builds locally, copies closure to target via SSH, activates in-place — no compilation on the VPS.

# Live activation (restarts changed services immediately):
hey ops deploy vps-ultraman vps_ultraman_root
# → nixos-rebuild switch --show-trace \
#       --flake .#vps-ultraman \
#       --target-host root@vps_ultraman_root \
#       --build-host localhost

# Safer: only update the bootloader entry, activate on next reboot:
hey ops deploy vps-ultraman vps_ultraman_root --boot
# → nixos-rebuild boot --show-trace \
#       --flake .#vps-ultraman \
#       --target-host root@vps_ultraman_root \
#       --build-host localhost

# Target defaults to flake-host if omitted:
hey ops deploy vps-pacman
# → nixos-rebuild switch --show-trace \
#       --flake .#vps-pacman \
#       --target-host root@vps-pacman \
#       --build-host localhost

• When: Updating an existing NixOS host that was previously bootstrapped or installed.
• Options:
  • --fast: Skip some checks for a faster deploy.
  • --boot: Use nixos-rebuild boot instead of switch. Registers the new configuration as the default choice in the bootloader without activating it immediately. The new system only takes effect upon the next reboot. Highly recommended for testing risky system, firewall, or SSH settings.
  • --debug: Verbose nixos-rebuild output.

Advanced Deploy: Build Location vs. Activation Mode

You can pass additional flags directly to nixos-rebuild by adding them after a double dash (--):

1. Workstation Build (Default): Compiles the closure locally on your workstation and copies it over SSH to the target. For different target architectures (e.g. building for aarch64 on an x86_64 workstation), this uses slow QEMU emulation.

2. Target Native Build (--build-host): Delegates compilation directly to the target machine itself, avoiding local emulation overhead. Combining it with --use-substitutes allows the target to download pre-built packages from binary caches.

   For standard hosts (e.g., VPS nodes):

   hey ops deploy vps-ultraman vps_ultraman_root -- --build-host root@vps_ultraman_root --use-substitutes

   [!WARNING] Do not use the --build-host flag directly on target hosts that use a cross-compiled kernel (like sbc-opi5p). Doing so forces the workstation-specific cross-compiler package to evaluate and run on the target ARM board, leading to a platform mismatch error. For sbc-opi5p, stick to the Workstation Build (Default) or configure a Distributed Build (Scenario B).

3. Combining Native Build with --boot: You can combine these flags to build natively on the target but delay activation until the next reboot (for standard hosts):

   hey ops deploy vps-ultraman vps_ultraman_root --boot -- --build-host root@vps_ultraman_root --use-substitutes

hey ops push-keys — Provision SSH Keys

Pushes host and global identity keys to a remote host so agenix can decrypt secrets during bootstrap.

# Push keys to /etc/ssh/ (standard):
hey ops push-keys vps-ultraman vps_ultraman_root
# → scp ~/.ssh/keys.homelab/vps/ssh_host_ed25519_key_vps-ultraman vps_ultraman_root:/etc/ssh/ssh_host_ed25519_key
# → scp /persist/etc/ssh/global_ed25519 vps_ultraman_root:/etc/ssh/global_ed25519

# Push with auto-generation of missing host key:
hey ops push-keys vps-ultraman vps_ultraman_root --generate
# → ssh-keygen -t ed25519 -f ~/.ssh/keys.homelab/vps/ssh_host_ed25519_key_vps-ultraman -N '' -C 'vps-ultraman.local'

# Push to persist path (impermanent hosts):
hey ops push-keys vps-ultraman vps_ultraman_root --persist
# → Keys go to /persist/etc/ssh/ instead of /etc/ssh/

# Skip global key (host key only):
hey ops push-keys vps-ultraman vps_ultraman_root --no-global

• When: Before hey ops bootstrap on a new host — ensures agenix has access to decryption keys.
• Options:
  • --persist: Push to /persist/etc/ssh/ (for impermanent root hosts).
  • --no-global: Skip the global agenix key, push host key only.
  • --generate: Generate a new ed25519 host key if none found locally.
• Key sources: Host key is searched in ~/.ssh/keys.homelab/{vps,pc,server}/. Global key is read from /persist/etc/ssh/global_ed25519 or /etc/ssh/global_ed25519.

hey ops ssh — Remote Shell

hey ops ssh vps_ultraman_root
# → ssh root@vps_ultraman_root

hey ops sync — Alias

Alias for hey ops deploy. Identical behavior.

Typical Lifecycle

# 1. First time: push keys so agenix can decrypt secrets
hey ops push-keys vps-ultraman vps_ultraman_root --generate

# 2. Bootstrap a bare VPS
hey ops bootstrap vps-ultraman vps_ultraman_root

# 3. Ongoing: push config updates to remote
hey ops deploy vps-ultraman vps_ultraman_root

# 4. Safer updates for critical hosts (verify, then reboot)
hey ops deploy vps-ultraman vps_ultraman_root --boot

# 5. Rebuild your local workstation (daily driver)
hey sync

🛠️ Implementation Details

Backend Selection

Runtime DOTFILES_HOME

DOTFILES_HOME is set by /etc/zshenv (generated by modules/hey.nix) and points to the flake store path (/nix/store/xxx-source). It is a runtime convenience for Janet CLI tools and shell scripts — not a build-time input.

No --impure flag, no getEnv, no environment variables needed.

Cross-Compilation vs Emulated Native Builds

When deploying to a host with a different CPU architecture (e.g. your workstation is x86_64-linux and the target Orange Pi 5 Plus is aarch64-linux), you have three targets available:

1. Hybrid Target Build (sbc-opi5p-hybrid target, Recommended / Default)

This is the recommended deployment configuration when building on your x86_64-linux workstation.

• Cache Behavior: Nix will always query standard binary caches (like cache.nixos.org or cache.numtide.com) first. If a package (e.g., sing-box, bash) is already cached for the target architecture (aarch64-linux), Nix will download the pre-compiled binary from the cache and skip compilation entirely. QEMU emulation is only used as a fallback to compile small uncached packages from source.

• Hybrid Kernel Cross-Compilation: The heavy custom Rockchip vendor kernel is configured via pkgsKernel in lib/nixos.nix to be cross-compiled natively on the x86_64 host toolchain, completely bypassing slow QEMU emulation for the largest compilation task.

  [!WARNING] Passing --build-host root@opi5p_root to nixos-rebuild will force all build jobs (including the x86_64-linux cross-compiled kernel) to run on the Orange Pi, causing a platform mismatch error. To split the build (compile the kernel on the workstation and build standard packages on the Orange Pi), you must configure the Orange Pi as a remote builder in /etc/nix/machines on your workstation, and deploy without the --build-host flag.

• Pros: Extremely fast deployment, no cross-compilation toolchain bugs, and high binary cache hit rate.

• Cons: Any uncached package built from source runs under slower QEMU emulation (but custom packages in this repo are small, and the large kernel is cross-compiled).

2. Pure Cross-compiled Build (sbc-opi5p-cross target)

Uses target-specific cross-compilers for all packages (x86_64-linux compiling for aarch64-linux directly, no QEMU).

• Cache Misses: Derivation hashes for cross-compiled packages differ from standard native packages. Nix cannot use the public binary cache, forcing everything (bash, openssl, git) to compile from source.
• Pros: Compiling from source runs at native host CPU speeds.
• Cons: Very slow first build. Highly susceptible to cross-compilation toolchain failures (Go -m64 flags, Rust target issues, Haskell/GHC cross-compiling, Zig linkers, Guile patches).
• Usage: Ideal as a debugging/test target to find and fix packages that lack cross-compilation support by using stdenv.hostPlatform == stdenv.buildPlatform conditional gates.

3. Pure Native Build (sbc-opi5p target)

Evaluates the system as a native aarch64-linux target for all packages, including the kernel (no cross-compilation is configured).

• Usage: This is the default target configured when running native builds directly on the Orange Pi itself (hey sync). If built on an x86_64 workstation, it will attempt to compile the kernel under slow QEMU emulation.

3. Build Scenario Matrix (Standard vs. Cross Target)

Here is how Nix schedules compilation in the three typical deployment setups:

4. Compilation Performance Comparison (Fastest to Slowest)

When compiling packages from source, the performance characteristics of the strategies differ dramatically:

1. Cross-compilation on Workstation (Fastest): 🚀
   • How it works: Compiles natively using your workstation CPU’s full raw speed using a target cross-compiler (e.g., GCC cross-compiling x86_64 -> aarch64).
   • Speed: Highly optimized. Bypasses emulation entirely. Takes ~15 minutes for the Linux kernel.
2. Native ARM compilation on SBC (Medium): ⚖️
   • How it works: Compiles natively on the target ARM CPU (e.g. RK3588) without instruction translation.
   • Speed: Limited by the slower mobile-class CPU, slower memory speed, and thermal throttling constraints of SBC hardware.
3. QEMU Emulation on Workstation (Slowest): 🐌
   • How it works: Compiles on the workstation, but every aarch64 instruction is dynamically translated to x86_64 at runtime via QEMU.
   • Speed: Extreme overhead. The translation layer adds a massive performance tax, often making it 2x to 5x slower than even native compilation on the SBC.

Rationale for the hybrid build: Because of these constraints, the standard sbc-opi5p workstation deployment (Scenario A) routes the heavy kernel compilation to Strategy 1 (cross-compiling natively on the workstation’s x86_64 CPU), while relying on standard binary caches to fetch pre-compiled aarch64 packages, falling back to Strategy 3 (QEMU emulation) only for small uncached packages. For advanced users who want to avoid QEMU entirely for uncached packages, configuring a distributed build (Scenario B) delegates standard package compilation to the target board itself (Strategy 2).

ARM Builder Server Alternatives

Do you need an expensive Apple Silicon Mac (M2 Ultra) to build ARM packages? No. Here are three free alternatives:

1. Local QEMU Emulation (Default): Leverages binfmt_misc + QEMU on your workstation to pull cached binaries, combined with cross-compiling the heavy kernel.
2. SBC as a Remote Builder: Once bootstrapped, you can add the Orange Pi 5 Plus as a remote builder in /etc/nix/machines on your workstation. Nix will transparently SSH in and compile native packages on the SBC, copying outputs back.
3. Oracle Cloud Free Tier VM: Oracle offers a free tier VM with up to 4 Ampere ARM cores and 24 GB of RAM. This makes an excellent, dedicated ARM builder server for $0.

Build & Deploy Topology

nixos-rebuild can delegate build and deploy to different hosts via SSH:

┌─────────────────────┐     ┌──────────────────┐     ┌─────────────────────┐
│   where you type    │     │  where Nix builds │     │  where system runs  │
│   nixos-rebuild     │     │  (--build-host)   │     │  (--target-host)    │
└─────────────────────┘     └──────────────────┘     └─────────────────────┘

Default: When --build-host is omitted, the build happens on the machine running nixos-rebuild. For hey ops deploy vps-ultraman, that’s your workstation — the VPS never compiles anything.

Nix store copy: After the build, the closure is copied from the build host to the target host via SSH (nix copy). Only changed paths are transferred.

Self-deploy: nixos-rebuild --target-host root@localhost builds and deploys to the same machine through SSH — useful when sudo isn’t available. Requires sshd running and ssh localhost working.

SSH config: If Host * has IdentitiesOnly yes, add BEFORE it:

Host localhost
    IdentitiesOnly no

SSH uses first-match — the more specific block must precede the wildcard.

Resource Efficiency

For low-RAM VPS nodes (e.g., 512MB-1GB RAM):

• Building locally (default) ensures no compilation happens on the target.
• Btrfs subvolumes and Zstd compression (configured in storage.nix) save disk space.
• --no-disko-deps with nixos-anywhere avoids copying disko deps to tmpfs during bootstrap.

⚠️ Safety & Security

• SSH Lockout: Be careful when modifying services.openssh.extraConfig. Ensure root is allowed from your deployment IPs, or you will lose the ability to use hey ops deploy.
• Tailscale: Remote hosts are typically accessed via Tailscale IPs for added security.
• --boot vs switch: On remote hosts, prefer --boot for risky changes — you can SSH in and reboot when ready. If switch breaks SSH, you’re locked out.
• SSH Password / Key Login (password.mode):
  • On a new host, set modules.security.password.mode = "bootstrap"; which temporarily enables password login over SSH (with password "nixos") as well as key login, avoiding immediate lockout.
  • CRITICAL: Before transitioning a host to modules.security.password.mode = "deploy"; (which disables password logins over SSH entirely), remember to add the host’s new public key to the mapping inside modules/profiles/user/alienzj.nix or the corresponding user’s Nix file. If you switch to "deploy" without authorized keys mapped, you will be permanently locked out of SSH access.