Containers Virt

Containers & Virtualization Strategy

This repository employs a multi-layered virtualization strategy to ensure reproducibility, hardware access, and service isolation.

1. Technology Matrix

2. Host Usage Table

3. Operational Guides

Arch Linux on NixOS (via Distrobox)

We use Distrobox to run Arch Linux for tools like ROS Rolling.

• Protection: Arch containers use an isolated home at ~/distrobox/arch to prevent conflicts with Nix-managed symlinks.
• Persistence: The ~/distrobox directory is automatically persisted, ensuring containers and installed packages survive reboots.
• Entry: Run hey .arch to enter the environment.

Sing-box TUN Interaction

Workstation hosts that enable Sing-box TUN should preserve local bridge reachability while still allowing container internet egress through the proxy path.

• Docker, Podman, libvirt, Waydroid, and MicroVM private bridge CIDRs should be present in Sing-box route_exclude_address.
• Those exclusions keep host-to-container, host-to-VM, LAN, and Tailscale traffic local.
• They do not exclude container public internet traffic; public destinations still enter TUN and follow Sing-box route rules.
• Do not exclude whole bridge interfaces unless the goal is to bypass Sing-box for all traffic from that interface.

For the detailed routing policy, see docs/networking-proxy.md.

Android Apps (via Waydroid)

Waydroid provides a containerized Android environment using LineageOS inside an LXC container.

Quick Start

# First-time initialization (downloads LineageOS images)
sudo waydroid init -s GAPPS -f

# Session lifecycle
hey .waydroid start     # or: waydroid session start
hey .waydroid show      # or: waydroid show-full-ui
hey .waydroid stop      # or: waydroid session stop
hey .waydroid status    # or: waydroid status

# Google Play certification
hey .waydroid register  # prints Android ID retrieval + registration URL

id3-eniac Setup Guide (NVIDIA Pascal 1080 Ti)

The host id3-eniac runs linuxPackages_latest (7.x) with the NVIDIA proprietary driver (legacy 580.x). Two fixes are applied in modules/services/virt/waydroid.nix:

Networking fix — Kernel 7.x omits the legacy ip_tables module required by waydroid-net.sh. The module uses pkgs.waydroid-nftables (sets USE_NFTABLES=1, swaps iptables for nftables in the wrapper PATH) instead of the default pkgs.waydroid.

GPU fix — The NVIDIA proprietary driver’s userspace libraries are compiled against glibc, but the Android container uses bionic libc. The driver cannot load inside Waydroid. The workaround is software rendering via SwiftShader:

# Already set in /var/lib/waydroid/waydroid_base.prop:
ro.hardware.gralloc=default
ro.hardware.egl=swiftshader

Make it persistent across image upgrades:

sudo waydroid prop set persist.waydroid.software_renderer true

Performance expectations with software rendering:

Alternative for GPU acceleration: Switching to Nouveau + NVK (Mesa ≥ 25.2) enables hardware-accelerated OpenGL ES via Zink-on-Vulkan. This is not recommended for id3-eniac because it cripples native Linux gaming on the proprietary driver. If you only need one or two FOSS Android apps, consider Android Translation Layer (ATL) as a future option once packaged for NixOS.

After hey sync (first time or after kernel/nixpkgs updates):

sudo systemctl restart waydroid-container
waydroid session start
waydroid show-full-ui

Debugging:

systemctl status waydroid-container
waydroid status
waydroid log                    # or: tail -f /var/lib/waydroid/waydroid.log
journalctl -u waydroid-container --no-pager -n 50

4. Future Modularization Plan

Our roadmap focuses on moving from “enabled services” to a “Unified Virt-Provider” model.

Phase 1: Unified OCI Module

Currently, affine and home-assistant hardcode their OCI definitions. We will create a modules.services.virt.oci wrapper that:

• Allows defining a container once.
• Allows switching the backend (podman vs docker) via a single option.
• Handles standard storage persistence automatically.

Phase 2: MicroVM for Security

Move critical services (like vaultwarden and postgresql) from host-level systemd units to MicroVMs. This provides kernel-level isolation while maintaining NixOS’s declarative configuration.

Phase 3: Proxmox-NixOS Integration

Standardizing the deployment of NixOS nodes onto Proxmox via the proxmox-nixos input, allowing lab-matrix to orchestrate new VM nodes declaratively.

Phase 4: Standardized Passthrough

Create a shared DSL for hardware passthrough:

virt.passthrough = {
  gpu = "nvidia";
  usb = ["yubikey" "webcam"];
  target = "distrobox-arch";
};

This will automatically generate the correct flags for Podman, XML for Libvirt, or parameters for Waydroid.