Backup

Backup Strategy & Architecture

This document describes the multi-tiered backup strategy designed for hosts in this flake (specifically focused on workstation hosts like id3-eniac and lab-matrix) to safeguard user home folders against sync propagation issues, hardware failure, and catastrophic disasters.

The Core Strategy: Multi-Tiered Redundancy

A single synchronization tool (like Syncthing) is not a backup system because deletions, overrides, or corruption on one device propagate to all others in seconds. We enforce a four-tier hybrid backup shield:

[User Directories] (documents, projects, pictures, toolkits)
       │
       ├──► Tier 0: Syncthing Shield (Near-instant) ──► peer device `.stversions/`
       │
       ├──► Tier 1: SSD Snapshots (Hourly/Daily) ─────► Local `/snapshots`
       │
       ├──► Tier 2: HDD Redundancy (Incremental) ─────► Local SATA HDD `/mnt/store/backup` (btrbk)
       │
       └──► Tier 3: Offsite/NAS Archive (Deduplicated) ─► QNAP NAS via REST Server (restic)

Tier 0: The Syncthing Shield

  • Purpose: Near-instant protection against accidental file deletion or revision edits propagated by peer devices.
  • Mechanism: Syncthing Staggered File Versioning configured via the Syncthing Web UI (127.0.0.1:8384).
  • Behavior: Instead of deleting or overwriting files, Syncthing moves old versions to a hidden .stversions directory, keeping them for a customizable period (e.g. 30 days) or until disk space is low.

Tier 1: Fast Recovery (SSD Snapshots)

  • Purpose: Instant, read-only local restore points to recover from user mistakes, typos, or software issues on the active host.
  • Mechanism: Btrfs copy-on-write snapshots taken using btrbk.
  • Schedule: Frequent (hourly/daily) snapshots taken on the main fast NVMe system drive (disk.lexar -> @snapshots subvolume).
  • Properties: Taking a snapshot is instantaneous and consumes 0 bytes of disk space initially.

Tier 2: Hardware Redundancy (HDD Replication)

  • Purpose: Protection against system drive (disk.lexar NVMe SSD) failure.
  • Mechanism: btrbk incremental send/receive replication.
  • Target: A separate physical disk (local LUKS-encrypted Btrfs HDD, e.g., disk.toshiba -> @store-backup subvolume).
  • Properties: Relies on Btrfs block-level transfer streams, sending only the modified data blocks between snapshots.

Tier 3: Disaster Recovery (Network NAS Archives)

  • Purpose: Redundancy against host theft, power surges (destroying all local disks), or physical disasters.
  • Mechanism: Restic client-side encrypted, deduplicated backups.
  • Target: QNAP NAS via restic-server (HTTP).
  • Why Restic over REST Server: Instead of using local SMB mounts or SFTP (which has higher CPU/connection overhead), Restic connects directly via the high-performance restic-server HTTP protocol. This achieves wire-speed transfers, supports append-only mode for ransomware protection, and provides a fully isolated API interface for backups, keeping the credentials secure.

NixOS Configuration Guides

1. Local Btrfs Snapshots & Replication (btrbk)

Btrbk is configured in modules/services/sharing/btrbk.nix and enabled on hosts via modules.services.sharing.btrbk.enable = true.

To ensure snapshots are frequent enough to catch sync propagation gaps, configure the following settings:

  services.btrbk.instances.btrbk = {
    # Run btrbk every 4 hours to catch deletions quickly
    onCalendar = "*-*-* 00,04,08,12,16,20:00:00"; 
    
    settings = {
      # Retain short-term SSD snapshots locally
      snapshot_preserve = "24h 7d";
      snapshot_preserve_min = "latest";
      
      # Retain long-term Btrfs backups on the HDD target
      target_preserve = "14d 8w 6m";
      target_preserve_min = "no";

      snapshot_dir = "@snapshots";

      volume."/btr_pool" = {
        subvolume."@persist".snapshot_create = "always";
        target = cfg.target;
      };
    };
  };

2. Encrypted NAS Backups (restic)

Configure Restic on your host to periodically archive your persisted directories onto the QNAP NAS directly via the high-performance restic-server HTTP protocol. Restic is natively supported in NixOS via services.restic.

[!NOTE] Before configuring the client, the restic-server application must be running on your QNAP NAS. For the server-side installation and container setup guide, see hosts/nas-nasa.md.

A. Initialize Secrets

You must configure two secrets using Agenix:

  1. restic-password.age: The client-side password used to encrypt/decrypt the actual backup data.

    echo -n "strong-encryption-password" | agenix -e restic-password.age

  2. restic-env.age: Environment variables containing the HTTP basic authentication credentials for the REST server.

    # File contents:
RESTIC_REST_USERNAME="restic-id3-eniac"
RESTIC_REST_PASSWORD="your_rest_server_password"

    [!IMPORTANT] How these variables align with the server configuration:

    • RESTIC_REST_USERNAME: This must match the exact username defined in the .htpasswd file on the NAS (e.g., restic-id3-eniac). When the REST server runs with --private-repos, it jails the client’s access to a directory named after this username (e.g., /data/restic-id3-eniac/), securing it from other clients.
    • RESTIC_REST_PASSWORD: This must be the plaintext password you set when generating the .htpasswd file on the server. The Restic client transmits this password in plaintext HTTP Basic Authentication, and the server validates it against the bcrypt hash stored in the .htpasswd file.

    Encrypt it:

    agenix -e restic-env.age

B. NixOS Declaration

Declare the restic backup service in your host configuration:

  services.restic.backups.nas-backup = {
    # Run as the local user
    user = "alienzj";
    initialize = true;
    passwordFile = config.age.secrets.restic-password.path;
    environmentFile = config.age.secrets.restic-env.path;
    
    # Backup user home data inside the @persist subvolume
    paths = [
      "/persist/home/alienzj/documents"
      "/persist/home/alienzj/projects"
      "/persist/home/alienzj/pictures"
      "/persist/home/alienzj/toolkits"
    ];
    
    # Exclude ephemeral build artifacts and caches
    exclude = [
      "**/node_modules"
      "**/.cache"
      "**/.direnv"
      "**/*.iso"
      "**/*.qcow2"
    ];

    # QNAP NAS REST server path
    repository = "rest:http://192.168.31.12:8000/restic-eniac";
    
    # Run nightly at 4:30 AM (after local btrbk completes)
    timerConfig = {
      OnCalendar = "*-*-* 04:30:00";
      Persistent = true;
    };
    
    # Pruning policy to maintain long-term archives
    pruneOpts = [
      "--keep-daily 7"
      "--keep-weekly 4"
      "--keep-monthly 12"
      "--keep-yearly 1"
    ];
  };

Restoring Files

From Local Btrfs Snapshots (Tier 1/2)

Since Btrfs snapshots are exposed as read-only directories, you can browse and copy files directly:

# Browse SSD snapshots
ls -la /snapshots/
cp -r /snapshots/@persist.YYYYMMDDTHHMM/home/alienzj/projects/myproject ~/projects/

# Browse HDD backups
ls -la /mnt/store/backup/alienzj/id3-eniac/btrbk/

From Restic NAS Backups (Tier 3)

To browse and restore files from the encrypted Restic repository:

# Export the HTTP auth variables for the REST server session
export RESTIC_REST_USERNAME="restic-id3-eniac"
export RESTIC_REST_PASSWORD="your_rest_server_password"

# List all snapshots on the NAS
restic -r rest:http://192.168.31.12:8000/restic-eniac -p /run/agenix/restic-password snapshots

# Mount the repository as a filesystem to browse and copy files
mkdir /tmp/restic-mount
restic -r rest:http://192.168.31.12:8000/restic-eniac -p /run/agenix/restic-password mount /tmp/restic-mount

# Or restore a specific folder directly
restic -r rest:http://192.168.31.12:8000/restic-eniac -p /run/agenix/restic-password restore latest --target /tmp/restore-test --include /persist/home/alienzj/projects

restic-server Security & Storage Engineering

The restic-server instance on the QNAP NAS runs with the --append-only and --private-repos flags to ensure robust security and filesystem isolation.

1. Security Parameters Explained

A. Private Repositories (--private-repos)

  • Mechanism: The REST server isolates data based on the HTTP Basic Authentication username.
  • Usecase: When id3-eniac authenticates as restic-id3-eniac, the server jails its access to /data/restic-id3-eniac/. It has no visibility into repositories belonging to other workstations (e.g., restic-lab-matrix).
  • Benefit: Multi-node isolation. A security breach on one workstation cannot lead to data theft or inspection of backups from other workstations.

B. Append-Only Mode (--append-only)

  • Mechanism: The REST server rejects any HTTP requests from the client that attempt to delete, overwrite, or modify existing files/snapshots.
  • Usecase: Protects against client-side ransomware or accidental execution of destructive commands (like restic prune or restic forget).
  • Benefit: Even if an attacker gains root access to your workstation, they cannot delete your historical backups on the NAS.

2. Pruning Backups to Save Space (5TB Limit)

Because your NAS storage allocation for Restic backups is capped at 5TB, you must prune old snapshots regularly to free up space. However, because the client is restricted to --append-only mode, running restic prune or restic forget from your workstation will result in an access error.

To safely prune old backups, you can run the pruning task directly on the NAS server (bypassing the REST server API constraints) using a Docker container.

The Pruning Command (Execute on QNAP NAS):

SSH into your QNAP NAS and run the pruning command using the Restic Docker image. This mounts your local backup directory directly and performs the cleanup locally:

# 1. SSH into the NAS
ssh nasnasa

# 2. Run the prune command locally using Docker
docker run --rm \
  -v /share/CACHEDEV2_DATA/alienzj/backup:/data \
  restic/rest-server:latest \
  restic -r /data/restic-id3-eniac \
  -p /data/restic-password \
  forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune

Automation:

This repository includes a declarative automated NixOS systemd timer (restic-nas-prune.timer) in restic.nix that can connect via SFTP to forget/prune old snapshots.

[!WARNING] Ransomware Security Boundary Hardening By default, the prune service is disabled (nasBackup.prune.enable = false) on workstations. If the workstation stores the SSH private key used for SFTP pruning, a compromised workstation could bypass the REST server’s append-only restriction and delete all historical backups.

If you choose to use it, you must explicitly enable it via modules.services.sharing.restic.nasBackup.prune.enable = true; and provision the required SFTP keys. However, for maximum security, running the pruning task directly on the QNAP NAS (bypassing any client keys) is strongly recommended.