Noter

Note Taking and Knowledge Management (Noter)

The noter module (modules/desktop/apps/noter.nix) packages note-taking, sketching, and academic reference management tools:

  • Zotero: Academic reference and PDF manager.
  • Obsidian: Markdown-based knowledge base.
  • Logseq: Outliner and local-first knowledge base.
  • Xournal++ / Rnote / Lorien / Styluslabs Write: Hand-drawn sketching, PDF annotation, and whiteboarding tools.

Zotero Hybrid Sync Architecture

We employ a hybrid syncing architecture for Zotero PDF attachments across workstations and mobile devices to achieve both LAN-speed file access and mobile convenience:

  1. Workstations (Syncthing-connected):

    • Zotero’s actual attachment storage is relocated to the Syncthing-synced folder ~/documents/doraemon/papers/zotero/storage.
    • Zotero’s standard storage directory (~/documents/zotero/storage) is replaced by a symlink pointing to this Syncthing-synced location.
    • When Zotero writes or reads a PDF, it is transparently handled by the local filesystem, and Syncthing syncs it to all other workstations instantly.
    • To prevent per-machine reader states from creating sync conflicts, a .stignore rule ignores papers/zotero/storage/**/.zotero-reader-state.

  2. Mobile Devices (Android / iPad):

    • Zotero mobile clients do not run Syncthing. Instead, they access PDF attachments via WebDAV over the dav.alienzj.org/documents endpoint.
    • Since the backend WebDAV directories map to the same file locations, modifications sync cleanly across both transports.

  3. Collection-Based Symlink Mirror:

    • Because Zotero’s internal stored file directory structure is obfuscated (e.g. storage/XXXXXXXX/), a python-based background mirror script (zotero-mirror) runs periodically on the workstation.
    • This script queries Zotero’s SQLite database to generate a nested, human-readable directory hierarchy in ~/documents/doraemon/papers/references/ containing symlinks back to the obfuscated stored PDFs. The location of this directory is configured via the zotero.mirror.referencesDir option. Zotero’s native “Linked Attachment Base Directory” setting remains unset to avoid side-effects.

The File Layout & Directory Tree

Below is a visual representation of how files are physically stored and how the mirror script exposes them in a human-readable structure:

~/documents/
├── doraemon/                            # Syncthing Synced Root
│   └── papers/
│       ├── zotero/
│       │   └── storage/                 # Physical Storage (obfuscated keys)
│       │       ├── ABCDEFGH/
│       │       │   └── nature_paper.pdf
│       │       └── IJKLMNOP/
│       │           └── cell_paper.pdf
│       └── references/                  # Human-Readable Symlink Mirror
│           ├── Bioinformatics/
│           │   └── nature_paper.pdf     # Symlink -> ../zotero/storage/ABCDEFGH/nature_paper.pdf
│           ├── Genomics/
│           │   └── cell_paper.pdf       # Symlink -> ../zotero/storage/IJKLMNOP/cell_paper.pdf
│           └── _Unsorted/               # Symlink for papers not in any collection
│
└── zotero/                              # Local Zotero Data Directory (Not Synced)
    ├── zotero.sqlite                    # Local Zotero database
    └── storage                          # Symlink -> ../doraemon/papers/zotero/storage

How the Mirror Script Handles Library Updates

The mirror script (zotero-mirror) runs periodically via a systemd user timer. It is fully state-aware and dynamically syncs the references/ directory with your Zotero collections:

  • Recursive Folder Matching: The script queries Zotero’s database, resolving nested collection hierarchies (sub-collections) of arbitrary depth, and recreates them exactly as nested directories under references/.
  • Multi-Category Mapping: If a PDF is placed in multiple collections inside Zotero, the script creates relative symlinks in all corresponding folders. Because symlinks use virtually zero disk space, this provides multiple paths to the same single-sourced PDF.
  • Dynamic Renaming & Moves: If a Zotero collection is renamed, or an item is moved to a different folder:
    1. The script creates the symlinks in the new location.
    2. Any symlinks in the old folder that are no longer referenced in the database are recognized as orphans and deleted (unlink).
    3. The now-empty old directory is deleted automatically (rmdir).
  • Safety First: The cleanup loop only targets symlinks (fp.is_symlink()). Real files or manually placed items inside the references/ directory are never touched or deleted, preventing any accidental data loss.

Why This Architecture Works (Design Decisions)

1. Stored Files + Syncthing = Seamless Portability

Instead of using Zotero’s “Linked Files” mode, this design relies on Zotero’s default Stored Files mode combined with local symlinking:

  • Unique ID Namespaces: Zotero isolates every Stored File inside a unique, 8-character alphanumeric folder (e.g., storage/XXXXXXXX/). Because of this, files with identical names (e.g., paper.pdf or main.pdf) never conflict or overwrite each other during synchronization.
  • No Path Fragmentation: Stored Files use relative paths in Zotero’s SQLite database (relative to the active Data Directory). Since our Nix configuration forces the Data Directory to ~/documents/zotero on every computer, Zotero dynamically resolves the absolute paths locally. It will never fail to find files due to different user home folder structures or system platforms.
  • Separation of Concerns: Zotero manages the metadata and database relationships (synced via Zotero’s unlimited free cloud database), while Syncthing handles the heavy file transfers over LAN/Tailscale direct links at gigabit speeds. Neither system blocks or interferes with the other.

2. Why We Keep Zotero’s “Linked Attachment Base Directory” Empty

We deliberately do not set extensions.zotero.baseAttachmentPath (the Linked Attachment Base Directory setting) inside Zotero’s user.js preferences:

  • The Native Purpose: Zotero’s base path setting is designed only to convert absolute linked file paths to relative paths so that Zotero can resolve linked attachments across multiple computers.
  • The Conflict: Because we converted all attachments to Stored Files, Zotero does not have any linked attachments to resolve.
  • Keeping it Clean: Leaving Zotero’s native base directory setting unconfigured keeps Zotero’s internal file resolver in its clean default state. The path where we want to build our readable references tree is managed solely by our custom background mirror script (zotero.mirror.referencesDir), ensuring complete separation from Zotero’s native linked file resolver.

Sync Propagation & Cooperative Workflow

When you annotate a PDF on one workstation, Syncthing and Zotero cooperate seamlessly to update other devices without creating duplicate files or causing sync loops:

  • Local Save: You annotate a PDF and save it in Zotero on Workstation A. The file is modified at ~/documents/zotero/storage/XXXXXXXX/paper.pdf.
  • LAN/Tailscale Propagation (Syncthing): Syncthing instantly detects the change in the doraemon directory and syncs the updated PDF file directly to Workstation B (usually under 2 seconds).
  • WebDAV Upload (Zotero): Zotero on Workstation A performs its next sync, packages the modified directory into XXXXXXXX.zip, and uploads it to the WebDAV server (dav.alienzj.org/documents/zotero/), overwriting the old archive.
  • Cooperative Resolution on Workstation B: When Zotero on Workstation B syncs, it learns from Zotero’s cloud that the attachment XXXXXXXX has changed. It checks the local path storage/XXXXXXXX/paper.pdf. Since Syncthing has already updated this file, Zotero verifies its checksum/timestamp and skips the WebDAV download entirely.
  • Mobile Fetch: Mobile devices (Android/iOS) download the updated XXXXXXXX.zip from WebDAV and overwrite their sandboxed local copy when the user accesses the paper.

Because both transports map to the exact same folder structure and use Zotero’s unique item keys (XXXXXXXX), files are always overwritten in-place, preventing duplicate files or conflict loops.

Workstation Client Configuration

To transition to this hybrid sync architecture:

  1. Enable Zotero module and mirror (in hosts/<host>/modules/modules.nix): 
    modules.desktop.apps.noter = {
  enable = true;
  zotero.mirror.enable = true; # Enable on primary workstation
};
  2. Set up the Storage Symlink: Create the target directories and the symbolic link in your user space: 
    mkdir -p ~/documents/zotero ~/documents/doraemon/papers/zotero/storage
# If you have existing files in ~/documents/zotero/storage, migrate them first:
# rsync -a ~/documents/zotero/storage/ ~/documents/doraemon/papers/zotero/storage/ && rm -rf ~/documents/zotero/storage
ln -s ~/documents/doraemon/papers/zotero/storage ~/documents/zotero/storage
  3. Convert Linked Files to Stored Files (if migrating from legacy attachments):
    • In Zotero Desktop, select all items in your library.
    • Go to Tools → Manage Attachments → Convert Linked Files to Stored Files.
    • This moves all PDFs from the legacy references directory into the Zotero storage/ directory (which now symlinks to the Syncthing-synced directory).
  4. Configure Zotero Sync Preferences:
    • Open Zotero and navigate to Settings/Preferences → Sync.
    • Log into your Zotero account for Data Syncing (Metadata/SQLite syncing is free and unlimited on Zotero’s cloud).
    • Under the File Syncing section, select:
      • Sync attachment files in My Library using: WebDAV
      • URL: Set protocol to https:// and enter dav.alienzj.org/documents
      • Username: alienzj
      • Password: Your decrypted agenix password.
    • Click Verify Server. Zotero will verify the endpoint and authenticate.