Email

Email System — Self-Hosted Mail on alienzj.org

Production email for @alienzj.org: send/receive to Gmail and other providers without third-party relays. Built on Stalwart with nginx TCP stream proxying through the L1 gateway.

Architecture

The same L1→L2 proxy pattern that serves 20 HTTP services extends to email via nginx stream (TCP) proxying:

Internet
  │  SMTP :25, SMTPS :465, Submission :587, IMAP :143, IMAPS :993
  ▼
vps-pacman (L1, public IP — Cloudflare DNS)
  │
  ├── nginx stream{} TCP proxy (raw bytes, no TLS termination)
  │     :25  → lab-matrix.<tailnet>:25
  │     :465 → lab-matrix.<tailnet>:465
  │     :587 → lab-matrix.<tailnet>:587
  │     :143 → lab-matrix.<tailnet>:143
  │     :993 → lab-matrix.<tailnet>:993
  │
  ├── Firewall: TCP 25, 465, 587, 143, 993 open
  │
  └── [Optional] OpenSMTPD :10026 — outbound relay for lab-matrix
        (routes mail through VPS's clean IP)

  │ Tailscale WireGuard │
  ▼
lab-matrix (L2, no public IP)
  └── Stalwart
        ├── TLS: Stalwart ACME (DNS-01 Cloudflare) for mail.alienzj.org
        ├── DKIM: ed25519 signer, private key via agenix
        ├── IMAP/JMAP: Thunderbird connects to mail.alienzj.org:993
        ├── Webadmin: https://mail.alienzj.org (OAuth2 SSO)
        └── Outbound: → vps-pacman:10026 (relay) → Internet MX

Key design decisions:

• TLS is end-to-end: nginx stream{} proxies raw TCP bytes. TLS negotiation (STARTTLS on :25, implicit TLS on :465/:993) happens between the remote server and Stalwart directly on lab-matrix. The nginx proxy never sees the encrypted content.
• Stalwart ACME: Stalwart on lab-matrix runs its own ACME client (DNS-01 via Cloudflare) to get a specific certificate for mail.alienzj.org. This is independent from the wildcard *.alienzj.org cert on vps-pacman — no conflict, no shared private keys.
• Outbound relay: lab-matrix’s home IP is likely on residential blocklists. Outbound mail routes through vps-pacman’s clean VPS IP via a lightweight OpenSMTPD relay. This is self-hosted, not a third-party service.
• Inbound path (Internet → you): External server → vps-pacman:25 → nginx stream → Stalwart on lab-matrix → encrypted-at-rest (PGP) → stored.
• Outbound path (you → Internet): Thunderbird → vps-pacman:587 → nginx stream → Stalwart on lab-matrix → DKIM signing → vps-pacman OpenSMTPD relay → Gmail MX.

Prerequisites

These two steps are mandatory. Without them, no external mail server will deliver to or accept mail from @alienzj.org.

VPS Provider: Port 25 & PTR Record

1. Diagnose outbound port 25

Determine vps-pacman’s public IP and test connectivity:

# Get vps-pacman's public IP
ssh vps_pacman_root "curl -s4 ifconfig.me"

# Test OUTBOUND port 25 — can vps-pacman reach Gmail's mail server?
ssh vps_pacman_root "nc -zw3 gmail-smtp-in.l.google.com 25 && echo 'outbound 25: OPEN' || echo 'outbound 25: BLOCKED'"

# Test INBOUND port 25 — is vps-pacman reachable from the internet?
# (Run AFTER nginx stream proxy is deployed — port 25 won't respond before that)
nc -zw3 mail.alienzj.org 25 && echo "inbound 25: OPEN" || echo "inbound 25: BLOCKED (may be pre-deploy)"

If outbound is blocked, you need to request unblocking from Vultr (see below). If inbound is blocked after deployment, check the nginx stream proxy and firewall config on vps-pacman.

2. Diagnose PTR record

# Replace with vps-pacman's actual public IP
VPS_IP=$(ssh vps_pacman_root "curl -s4 ifconfig.me")
echo "vps-pacman public IP: $VPS_IP"

# Check current reverse DNS
dig -x $VPS_IP +short

The output is typically a generic hostname like vps12345.vultr.com. It must return mail.alienzj.org. If you see dig: 'x' is not a legal name, replace with the actual IP.

3. Vultr: Request port 25 unblock {#vultr-request-port-25-unblock}

Vultr blocks outbound SMTP on port 25 by default for new accounts. To unblock:

1. Log into my.vultr.com
2. Navigate to Support → Open Ticket
3. Select category: Network / Firewall
4. Use this template:

Subject: Port 25 Unblock Request — Personal Mail Server

Hello,

I'm running a personal mail server on my VPS instance and need outbound
SMTP port 25 unblocked. I agree to Vultr's terms regarding spam and
email abuse.

Instance details:
- IP address: <YOUR_VPS_PUBLIC_IP>
- Hostname: vps-pacman
- Domain: mail.alienzj.org

My use case:
- Personal email for @alienzj.org (single user)
- No bulk email, no newsletters, no marketing
- Low volume (< 50 outbound emails/day)
- SPF, DKIM, DMARC all configured
- Reverse DNS (PTR) will be set to mail.alienzj.org

Thank you!

Vultr typically responds within 24 hours. They may ask you to confirm you agree to their anti-spam policy — reply affirmatively.

Note: Vultr’s automated system may flag new unblock requests. If denied, reply to the ticket explaining it’s a personal mail server (not transactional/bulk), with SPF/DKIM/DMARC configured. Manual review usually approves legitimate personal use.

4. Vultr: Set PTR (reverse DNS)

1. Log into my.vultr.com
2. Navigate to Products → Instances → vps-pacman
3. Click the instance → Settings tab → Reverse DNS
4. For IPv4, enter: mail.alienzj.org
5. For IPv6 (if assigned), also set to: mail.alienzj.org
6. Click Save

Wait 5-15 minutes for DNS propagation, then verify:

dig -x <VPS_IP> +short
# Must return: mail.alienzj.org.

# Also verify forward DNS round-trips correctly
dig mail.alienzj.org +short
# Must return your VPS IP

Both forward (A → IP) and reverse (IP → PTR) must match. Gmail checks this.

Cloudflare DNS

• The alienzj.org zone must be managed in Cloudflare (already the case for ACME wildcard certs).
• Cloudflare API token with Zone:DNS:Edit permission on alienzj.org (already exists for ACME).

ISP Check (lab-matrix, optional)

Your home ISP may block outbound port 25 from lab-matrix. This check is purely informational — the architecture routes all outbound mail through vps-pacman’s clean VPS IP regardless. Even if your ISP blocks port 25, email works fine through the relay.

If you want to test anyway (e.g., to try direct delivery without the relay), sing-box TUN mode on lab-matrix intercepts all traffic, so a plain nc goes through the proxy — not your raw ISP. Bypass the TUN by binding to the physical NIC:

# Find your LAN IP (replace eno1 with your physical interface)
LAN_IP=$(ip -4 addr show eno1 | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
echo "LAN IP: $LAN_IP"

# Test raw ISP path (bypasses sing-box TUN)
nc -s $LAN_IP -zw3 gmail-smtp-in.l.google.com 25 \
  && echo 'ISP port 25: OPEN' || echo 'ISP port 25: BLOCKED'

If blocked (very common on residential connections), it confirms why the relay is necessary. If open, direct delivery from lab-matrix is possible but still not recommended — residential IPs are on Spamhaus PBL by design, so Gmail will still reject mail sent directly.

DNS Records

All records go in the Cloudflare DNS zone for alienzj.org. Add them before or immediately after deploying the Nix changes.

Required records

The mail A record — why grey cloud is mandatory

You almost certainly already have a wildcard *.alienzj.org record proxied through Cloudflare (orange cloud) that handles all subdomains for your HTTP services. This is correct for web traffic but breaks email completely:

*.alienzj.org  →  A  →  192.248.190.229  (orange cloud / proxied)
                            │
                            ▼
mail.alienzj.org resolves to 104.21.63.9, 172.67.142.12  (Cloudflare edge IPs)
                            │
                            ▼
SMTP/IMAP connections hit Cloudflare's HTTP-only edge → DROPPED

Cloudflare’s proxy (orange cloud) only forwards HTTP/HTTPS — not SMTP, IMAP, or any other TCP protocol. A remote mail server connecting to mail.alienzj.org:25 hits Cloudflare’s edge, which has no idea what to do with SMTP traffic and drops the connection.

The fix: add an explicit A record for mail with grey cloud (DNS only). Explicit records override wildcards. All other subdomains (git, bw, immich, etc.) keep their orange-cloud protection through the wildcard.

GUI method

1. Log into Cloudflare Dashboard → select alienzj.org
2. DNS → Records → Add record
3. Type: A, Name: mail, IPv4: 192.248.190.229
4. Proxy status: toggle OFF (grey cloud — “DNS only”)
5. TTL: Auto
6. Save

CLI method (Cloudflare API)

# Extract token and zone ID
CF_TOKEN=$(ssh vps_pacman_root 'cat /run/agenix/cloudflare-api-token' | sed 's/.*=//' | tr -d '\n')
ZONE_ID=$(curl -s -H "Authorization: Bearer $CF_TOKEN" \
  "https://api.cloudflare.com/client/v4/zones?name=alienzj.org" | jq -r '.result[0].id')

# Create the grey-clouded A record
curl -s -X POST \
  -H "Authorization: Bearer $CF_TOKEN" \
  -H "Content-Type: application/json" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
  -d "{\"type\":\"A\",\"name\":\"mail.alienzj.org\",\"content\":\"$VPS_IP\",\"proxied\":false,\"ttl\":1}" | \
  jq '{success, name: .result.name, content: .result.content, proxied: .result.proxied}'

# Verify
dig mail.alienzj.org +short
# Must return your VPS IP, not Cloudflare proxy IPs (104.21.x.x / 172.67.x.x)

DNS record coexistence — no conflicts

The zone contains a mix of record types at apex and subdomains. Here’s how they coexist:

No conflicts. The key rule: explicit records override wildcards, and grey-cloud ensures Cloudflare stays out of the mail path.

SPF: v=spf1 mx ip4:<VPS_IP> -all is recommended — expresses intent clearly and avoids the extra DNS lookup if the MX resolves through the same IP anyway.

DMARC policy stages

DMARC tells receiving servers what to do when an email from your domain fails authentication. There are three policies, and you progress through them as you gain confidence:

Stage 1: p=none (monitor, no enforcement)

v=DMARC1; p=none; rua=mailto:[email protected]

• All mail is delivered normally, even if it fails DKIM/SPF.
• You receive XML aggregate reports (rua) showing who is sending mail from your domain and whether it passes authentication.
• Goal: Confirm that all legitimate mail (your server, your services) passes DKIM + SPF. If an internal service is misconfigured, the reports tell you without any mail being blocked.

Stage 2: p=quarantine (spam-folder failures)

v=DMARC1; p=quarantine; pct=100; rua=mailto:[email protected]

• Mail that fails DKIM and SPF alignment goes to the recipient’s spam folder instead of inbox.
• pct=100 means apply to 100% of failing mail. Start with pct=25 for a partial rollout.
• Fully-authenticated mail is unaffected — your legitimate mail still lands in inbox.
• Goal: Run for 1-2 weeks. Review reports. If only spam/phishing is being quarantined and your own mail is clean, you’re ready for the next stage.

Stage 3: p=reject (full enforcement)

v=DMARC1; p=reject; rua=mailto:[email protected]

• Mail that fails both DKIM and SPF is outright rejected — it never reaches the recipient, not even spam.
• This prevents anyone from spoofing @alienzj.org in phishing attacks.
• Prerequisite: Weeks of clean reports from p=quarantine showing zero false positives.

How to verify alignment before upgrading:

# 1. Send a test email to Gmail and check the headers
swaks --to [email protected] --from [email protected] \
      --server mail.alienzj.org --port 587 --tls \
      --auth-user [email protected] --auth-password '<password>'

# 2. In Gmail, open the email → "Show original" → look for:
#    Authentication-Results: ... dkim=pass ... spf=pass ... dmarc=pass

# 3. Also send to a DMARC test address that replies with your report:
#    https://www.mail-tester.com  (gives you a score, checks alignment)
#    https://dkimvalidator.com     (shows DKIM/SPF pass/fail detail)

Typical timeline: Week 1-2 on p=none, Week 3-4 on p=quarantine, then p=reject permanently. For a single-user domain with one mail server, you can often skip from none to reject after confirming 2-3 test emails pass DKIM/SPF — there are no other legitimate senders to worry about.

Optional advanced records

MTA-STS enforces TLS for inbound delivery. TLSRPT provides delivery failure visibility. Both require additional web endpoints and are optional for initial setup.

DKIM Key Generation

Generate the private key once, encrypt it with agenix for lab-matrix, and publish the public key in DNS.

1. Generate Ed25519 keypair

# OpenSSL 3.x genpkey outputs PKCS#8 natively — no conversion needed.
# Stalwart reads this format directly via pkcs8.private-key.
openssl genpkey -algorithm ed25519 -out /tmp/dkim-ed25519.pem

# Extract public key (base64-encoded, for DNS)
openssl pkey -in /tmp/dkim-ed25519.pem -pubout 2>/dev/null |
  awk '!/^-/' | tr -d '\n'

# The output is the value for 'p=' in the DKIM DNS record.

2. (Optional) Generate RSA 2048-bit keypair

Some older MTAs only verify RSA-signed DKIM. Generate a second key for compatibility:

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out /tmp/dkim-rsa.pem

# Extract public key for DNS
openssl pkey -in /tmp/dkim-rsa.pem -pubout |
  awk '!/^-/' | tr -d '\n'

3. Encrypt with agenix (lab-matrix only)

# Encrypt the private key — OpenSSL 3.x genpkey output is already PKCS#8,
# so no format conversion needed.
agenix -e lab-matrix.stalwart-dkim-private-key.age < /tmp/dkim-ed25519.pem

# For dual RSA + Ed25519, encrypt the second key too:
agenix -e lab-matrix.stalwart-dkim-rsa-private-key.age < /tmp/dkim-rsa.pem

# Place the .age files in nix-secrets/secrets/
# Add to nix-secrets/secrets/secrets.nix:
#   "lab-matrix.stalwart-dkim-private-key.age" = { nodes = ["lab-matrix"]; };

# Securely delete temp files
shred -u /tmp/dkim-ed25519.pem /tmp/dkim-rsa.pem

4. Publish DNS record

Name:  mail._domainkey.alienzj.org
Type:  TXT
Value: "v=DKIM1; k=ed25519; p=<base64-public-key-from-step-1>"

For RSA dual-signing, add a second selector (e.g., mail-rsa._domainkey.alienzj.org).

5. Verify DKIM DNS

dig TXT mail._domainkey.alienzj.org +short
# Should show: "v=DKIM1; k=ed25519; p=..."

DKIM vs PGP: Two Keys, Two Purposes

Stalwart uses two entirely independent cryptographic keys. They look similar (both generate keypairs) but serve opposite ends of the mail pipeline. Do not confuse them.

The two keys at a glance

How they work

DKIM (openssl key) — authenticates OUTBOUND mail

You compose an email in Thunderbird
    │
    ▼
Stalwart receives it on port 587
    │
    ▼
Stalwart signs the email body + selected headers
with the DKIM PRIVATE key (from lab-matrix.stalwart-dkim-private-key.age)
    │
    ▼
Stalwart sends to Gmail's MX
    │
    ▼
Gmail looks up mail._domainkey.alienzj.org in DNS
    │
    ▼
Gmail verifies the signature with the DKIM PUBLIC key from that TXT record
    │
    ├── Signature matches → dkim=pass  → this is really from @alienzj.org
    └── Signature fails   → dkim=fail  → possibly spoofed, deliver to spam

The DKIM private key config in modules/services/net/stalwart.nix:

signer.dkim = {
  domain = "alienzj.org";
  selector = "mail";
  pkcs8.private-key = "%{file:/run/agenix/stalwart-dkim-private-key}%";
};

The DKIM public key is published as a DNS TXT record. Anyone on the internet can read it — public keys can only verify signatures, not create them. That’s the asymmetry that makes DKIM useful: publish freely, keep the private key locked down.

OpenPGP (gpg key) — encrypts INBOUND mail at rest

Someone sends an email to [email protected]
    │
    ▼
Stalwart receives it on port 25
    │
    ▼
Stalwart encrypts the email body with your PGP PUBLIC key
(uploaded to Stalwart webadmin → Account → Public Keys)
    │
    ▼
Stalwart writes the encrypted blob to disk (PostgreSQL)
    │
    ▼
Later, you open Thunderbird
    │
    ▼
Thunderbird fetches the encrypted blob via IMAP
    │
    ▼
Thunderbird decrypts it with your PGP PRIVATE key
(from ~/.config/gnupg/ on your desktop)
    │
    ▼
You read the plaintext email

The PGP private key never touches the server. Only the public key is uploaded to Stalwart. Decryption happens entirely on your desktop, in Thunderbird. Even if an attacker gains root on lab-matrix and steals the PostgreSQL database, all they get is encrypted blobs — unreadable without the PGP private key.

For step-by-step PGP setup, see the encryption-at-rest guide in web-services.md.

Can I reuse the same keypair for both?

No. They are fundamentally incompatible formats:

• DKIM uses PKCS#8 (X.509-style) keys, 256-bit Ed25519 or 2048-bit RSA
• PGP uses OpenPGP keyblocks with user ID, self-signature, and subkey packets

Even if you could convert between formats, you should not reuse keys across purposes. Key compromise in one system would compromise the other. Keep them separate — generate a DKIM key with openssl and a PGP key with gpg.

Cloudflare API Token for lab-matrix

Stalwart’s ACME client on lab-matrix needs the Cloudflare API token for DNS-01 challenges. Update the agenix secret to include lab-matrix:

# In nix-secrets/secrets/secrets.nix, update:
"cloudflare-api-token.age" = {
  nodes = ["vps-pacman" "lab-matrix"];  # Add lab-matrix
};

If you prefer separate tokens for security isolation, create a dedicated token in Cloudflare (Zone:DNS:Edit on alienzj.org only) and encrypt it as a new secret for lab-matrix.

Pre-Flight Checklist

Before deploying, verify every prerequisite is met:

# Set this once
VPS_IP="<vps-pacman-public-ip>"

# 1. Port 25 outbound from vps-pacman
ssh vps_pacman_root "nc -zw3 gmail-smtp-in.l.google.com 25" \
  && echo "[OK]" || echo "[FAIL] Port 25 blocked — request unblock from Vultr"

# 2. PTR record matches forward DNS
PTR=$(dig -x $VPS_IP +short)
FWD=$(dig mail.alienzj.org +short)
[ "$PTR" = "mail.alienzj.org." ] && echo "[OK] PTR: $PTR" \
  || echo "[FAIL] PTR is '$PTR', expected 'mail.alienzj.org.'"
[ "$FWD" = "$VPS_IP" ] && echo "[OK] Forward: mail.alienzj.org → $FWD" \
  || echo "[FAIL] Forward DNS mismatch"

# 3. DKIM keypair generated and encrypted
#    (These files must exist in nix-secrets)
ls nix-secrets/secrets/lab-matrix.stalwart-dkim-private-key.age >/dev/null 2>&1 \
  && echo "[OK]" || echo "[FAIL] DKIM key not found"

# 4. Cloudflare API token available for lab-matrix
#    Check nix-secrets/secrets/secrets.nix includes lab-matrix in nodes list
grep -A2 'cloudflare-api-token' nix-secrets/secrets/secrets.nix | grep -q lab-matrix \
  && echo "[OK]" || echo "[FAIL] Cloudflare token not scoped to lab-matrix"

# 5. DNS records in place (MX, SPF minimum — DKIM/DMARC after deploy is fine)
echo -n "MX:  "; dig MX alienzj.org +short | grep -q mail && echo "[OK]" || echo "[MISSING]"
echo -n "SPF: "; dig TXT alienzj.org +short | grep -q spf && echo "[OK]" || echo "[MISSING]"
echo -n "DKIM:"; dig TXT mail._domainkey.alienzj.org +short | grep -q DKIM && echo "[OK]" || echo "[MISSING]"
echo -n "DMARC:"; dig TXT _dmarc.alienzj.org +short | grep -q DMARC && echo "[OK]" || echo "[MISSING]"

All five items marked [OK] → proceed to deployment. Any [FAIL] → fix before continuing.

Deployment

Step 1: Enable on lab-matrix

In hosts/lab-matrix/modules/modules.nix, update the stalwart block:

net.stalwart = {
  enable = true;
  nginx = {
    enable = true;
    ssl = false;
  };
  oauth2 = {
    enable = true;
    allowedGroups = ["[email protected]"];
  };
  # NEW: enable public email
  publicEmail.enable = true;
};

Step 2: Enable on vps-pacman

In hosts/vps-pacman/modules/modules.nix, add:

services.web.nginx = {
  # ... existing config ...
  stalwartStreamProxy = true;
};

This activates:

• nginx stream{} proxy for SMTP/IMAP on ports 25, 465, 587, 143, 993
• Firewall rules for those ports
• Optional OpenSMTPD outbound relay (if configured in the module)

Step 3: Rebuild both hosts

# On dev machine
hey ops deploy lab-matrix lab_matrix_root --boot
hey ops deploy vps-pacman vps_pacman_root --boot

# Reboot to activate
ssh lab_matrix_root "reboot"
ssh vps_pacman_root "reboot"

Post-Deploy: Stalwart Setup

1. Verify Stalwart and ACME

After reboot, check that Stalwart has obtained its TLS certificate:

ssh lab_matrix_root "systemctl status stalwart --no-pager"
ssh lab_matrix_root "journalctl -u stalwart --since '5min ago' --no-pager | grep -i 'acme\|certificate\|tls'"

Stalwart’s ACME client runs on first startup. It may take up to 5 minutes (configured wait = "5m") for the DNS-01 challenge to propagate and the certificate to be issued.

2. Verify TLS on port 25

# From any machine with internet access
openssl s_client -connect mail.alienzj.org:25 -starttls smtp < /dev/null 2>&1 | head -20

Look for: verify return code: 0 (ok) and a valid certificate for mail.alienzj.org.

3. Add domain in Stalwart webadmin

1. Visit https://mail.alienzj.org — authenticate via Kanidm SSO
2. Management → Directory → Domains → Add alienzj.org
3. Management → Directory → Accounts → Create your personal account (e.g., [email protected])
4. Set a password for IMAP/SMTP auth (used by Thunderbird and other mail clients)

4. Create service accounts

Create accounts for transactional email (the same ones your services already use):

• [email protected]
• [email protected]
• [email protected]

Set passwords for each. Update the corresponding agenix SMTP password secrets if the passwords changed.

5. (Optional) Enable encryption-at-rest

For your personal account, follow the OpenPGP guide in web-services.md to encrypt incoming mail with your PGP public key.

Client Setup (Thunderbird)

Two paths to reach the mail server:

Tailscale Direct (recommended when on Tailscale)

Bypasses the public path entirely. Traffic stays inside WireGuard:

Thunderbird will warn the certificate is not trusted — the cert is issued for mail.alienzj.org, not the IP. Accept the permanent security exception. The connection is inside WireGuard encryption regardless.

Public (any internet connection)

The public path requires the mail A record to be grey-clouded in Cloudflare (DNS only — no HTTP proxy). See DNS Records.

Thunderbird & OpenPGP Key Import

If you enabled Encryption-at-Rest (OpenPGP) in Stalwart, Thunderbird needs your PGP private key to decrypt incoming mail. Unlike DKIM (server-side), PGP decryption happens entirely on the client:

Gmail (TLS on the wire → plaintext storage → plaintext IMAP → readable)
Stalwart + OpenPGP (TLS on the wire → encrypt with public key → encrypted storage → encrypted IMAP → needs private key)

Thunderbird uses its own OpenPGP library (RNP), not the system GPG keyring. Even if the key exists in ~/.config/gnupg/, Thunderbird won’t find it automatically — you must import it:

# Export the private key (will prompt for passphrase — needed to decrypt
# the private key material, which is passphrase-protected on disk)
gpg --export-secret-keys --armor [email protected] > /tmp/alienzj-pgp-priv.asc

# Securely delete after importing
shred -u /tmp/alienzj-pgp-priv.asc

Then in Thunderbird: Account Settings → End-to-End Encryption → Add Key → Import from file → /tmp/alienzj-pgp-priv.asc. Enter the key’s passphrase when prompted.

Why export asks for a passphrase but listing keys doesn’t: --list-keys reads only the public keyring (no secret material). --export-secret-keys must decrypt the private key, which is passphrase-protected on disk. This is the same passphrase you set when running gpg --full-generate-key.

Verifying the key matches: Check that Thunderbird imported the right key by matching key IDs:

# In terminal: show your key's encryption subkey ID (last 16 chars)
gpg --list-keys --with-colons [email protected] | grep '^sub' | cut -d: -f5

# In Thunderbird's OpenPGP debug log (enable via OpenPGP Key Manager →
#   View → Show Debug Log), look for "key id: 0x...". The hex ID after 0x
#   should match your subkey ID. Case doesn't matter.

If Thunderbird shows “OpenPGP blocked” and the key IDs match, the key import likely failed or the passphrase wasn’t provided. Try re-importing.

Disabling encryption-at-rest (if you don’t need it): Go to Stalwart webadmin → Directory → Accounts → [email protected] → Settings → Encryption at rest → disable. New emails arrive unencrypted and are immediately readable. Existing encrypted emails stay encrypted.

Testing

Inbound (receive)

# Test SMTP port is reachable
nc -zw3 mail.alienzj.org 25 && echo "SMTP open" || echo "SMTP blocked"

# Test STARTTLS
openssl s_client -connect mail.alienzj.org:25 -starttls smtp < /dev/null 2>&1 |
  grep -E "Verify return code|subject=|issuer="

# Test SMTPS (implicit TLS)
openssl s_client -connect mail.alienzj.org:465 < /dev/null 2>&1 |
  grep -E "Verify return code|subject="

# Test IMAPS
openssl s_client -connect mail.alienzj.org:993 < /dev/null 2>&1 |
  grep -E "Verify return code|subject="

# Send a test email from Gmail to [email protected]
# Check if it arrives in Stalwart webadmin → Status → Message Queue

Outbound (send)

# Install swaks (Swiss Army Knife SMTP)
# nix shell nixpkgs#swaks

# Test submission (port 587 with STARTTLS + auth)
swaks --to [email protected] \
      --from [email protected] \
      --server mail.alienzj.org --port 587 --tls \
      --auth-user [email protected] --auth-password '<your-password>'

# Check your Gmail inbox. View original → verify headers:
#   Authentication-Results: ... dkim=pass ... spf=pass ... dmarc=pass

DNS verification

dig MX alienzj.org +short
# → 10 mail.alienzj.org.

dig TXT alienzj.org +short | grep spf
# → "v=spf1 mx -all"

dig TXT mail._domainkey.alienzj.org +short
# → "v=DKIM1; k=ed25519; p=..."

dig TXT _dmarc.alienzj.org +short
# → "v=DMARC1; p=quarantine; ..."

# PTR (from any machine)
dig -x <vps-pacman-ip> +short
# → mail.alienzj.org.

Online testing tools

• Mail Tester: https://www.mail-tester.com — sends a test email and scores deliverability
• DKIMValidator: https://dkimvalidator.com — verifies DKIM/SPF alignment
• MXToolbox: https://mxtoolbox.com — checks MX, SPF, DKIM, DMARC, and blacklists

Outbound Relay (OpenSMTPD)

The outbound relay on vps-pacman routes lab-matrix’s outgoing mail through the VPS’s clean IP. This is optional — try direct delivery from lab-matrix first.

When you need it

• lab-matrix’s home IP is on residential blocklists (UCEPROTECT-L3, Spamhaus PBL)
• Your ISP blocks outbound port 25
• Gmail rejects or spam-folders mail sent directly from lab-matrix’s IP

How it works

Stalwart (lab-matrix)
  │ outbound relay to vps-pacman:10026 (via Tailscale)
  ▼
OpenSMTPD (vps-pacman)
  │ SMTP client with clean VPS IP, proper PTR
  ▼
Internet MX (Gmail, Outlook, etc.)

Enable the relay

The vps-pacman module includes an optional OpenSMTPD service. When enabled:

• OpenSMTPD listens on vps-pacman’s Tailscale IP:10026
• Accepts relay from the Tailscale CGNAT range (100.64.0.0/10)
• Delivers to destination MX servers from vps-pacman’s clean public IP

On lab-matrix, Stalwart is configured to relay all outbound mail through this endpoint.

Verify relay

# On vps-pacman: check OpenSMTPD is running
ssh vps_pacman_root "systemctl status opensmtpd --no-pager"

# On vps-pacman: check port 10026 is listening (Tailscale IP only)
ssh vps_pacman_root "ss -tlnp | grep 10026"

# From lab-matrix: test connectivity to relay
ssh lab_matrix_root "nc -zw3 <vps-pacman-tailscale-ip> 10026 && echo 'OK'"

Maintenance

Certificate renewal (automatic)

Stalwart’s ACME client auto-renews the mail.alienzj.org certificate before expiry. No manual intervention needed. The certificate is stored in /var/lib/stalwart-mail/ (must be in persistence config).

Verify the current cert:

ssh lab_matrix_root "echo | openssl s_client -connect 127.0.0.1:25 -starttls smtp 2>/dev/null |
  openssl x509 -noout -dates -subject"

DKIM key rotation

Rotate DKIM keys every 6-12 months:

1. Generate a new keypair (same commands as initial setup)
2. Add the new public key to DNS as a second selector (e.g., mail2026._domainkey)
3. Update Stalwart config to sign with the new selector
4. Wait 48 hours (DNS propagation + mail in flight)
5. Remove the old DNS record
6. Remove the old key from Stalwart config

Queue monitoring

Check for stuck outbound mail:

# Stalwart queue status (via CLI or API)
ssh lab_matrix_root "stalwart-cli queue status"

# OpenSMTPD queue
ssh vps_pacman_root "smtpctl show queue"

DMARC reports

DMARC aggregate reports arrive at [email protected]. Set up an account for this address and periodically review the XML reports for authentication failures and spoofing attempts. Tools like parsedmarc can visualize these.

Security Protections

The email system uses a layered defense against DDoS, spam relaying, and abuse. All protections are active by default when publicEmail.enable = true.

Layer 1: Edge Connection Limiting (nginx stream proxy on vps-pacman)

Before traffic reaches Stalwart, the nginx stream proxy enforces per-IP connection limits:

nginx stream (vps-pacman)
  limit_conn_zone $binary_remote_addr zone=smtp_conn:10m  ← 10MB shared memory
  limit_conn_zone $binary_remote_addr zone=imap_conn:10m

  SMTP servers (25/465/587): limit_conn smtp_conn 10    ← 10 concurrent per IP
  IMAP servers  (143/993):   limit_conn imap_conn 10    ← 10 concurrent per IP

A legitimate remote MTA (Gmail, Outlook) or mail client (Thunderbird) uses 1-3 connections. Any IP exceeding 10 concurrent connections is rejected at the edge, before consuming Stalwart resources. This blocks connection-flood DDoS attacks regardless of protocol-level behavior.

Layer 2: Session Throttling (Stalwart)

Once a connection reaches Stalwart, per-session rate limits prevent a single authenticated or trusted session from flooding the server with outbound mail:

These limits apply to both authenticated users (Thunderbird, service accounts) and trusted-network connections (Tailscale). A compromised service account cannot send more than 50 emails per hour through a single connection — well below any spam threshold.

Layer 3: Relay Restrictions

Outbound mail relaying (sending to external domains) is strictly gated by session.rcpt.relay rules:

session.rcpt.relay = [
  { "if" = "!is_empty(authenticated_as)";  "then" = true;   }  # Must be logged in
  { "if" = "ip_in(remote_ip, '100.64.0.0/10', ...)"; "then" = true; }  # Or on Tailscale
  { "else" = false; }                                          # Everyone else: DENY
];

Unencrypted, unauthenticated connections on port 25 are treated as inbound delivery only — they can deliver mail TO @alienzj.org but cannot relay mail to external domains. Open-relay scanners see a hard rejection. This is the standard MTA security model.

Layer 4: Fail2Ban (HTTP surface)

The existing fail2ban jails protect the HTTP attack surface:

Tailscale IPs (100.64.0.0/10) are excluded from bans to prevent self-inflicted lockouts.

What is NOT protected

• SMTP AUTH brute force: fail2ban cannot see the real attacker IP through the TCP stream proxy (Stalwart sees vps-pacman’s Tailscale IP). Connection limits at the nginx edge mitigate this by capping attempts per IP, but a distributed attack could still probe credentials. Mitigation: use strong per-account passwords.
• Volumetric DDoS beyond connection limits: A botnet with thousands of IPs could still exhaust the 200-session concurrency limit. Mitigation: Cloudflare Spectrum or a DDoS-protection reverse proxy in front of vps-pacman if this becomes a real threat.
• SMTP protocol-level attacks: Malformed headers, line-length bombs, etc. Mitigation: Stalwart is written in Rust (memory-safe) and actively maintained; protocol parsing bugs are low-risk.

Tuning for higher volume

If you send bulk transactional mail (newsletters, notifications to many users):

# In the host config, override the defaults:
modules.services.net.stalwart.publicEmail.enable = true;
services.stalwart.settings = {
  session.mail.throttle = "500/1h";
  session.rcpt.throttle = "1000/1h";
  server.concurrent-sessions = 500;
};

Adjust upward in small increments. Monitor Stalwart logs for throttling rejection messages before raising limits.

Firewall Reference

vps-pacman public ports

vps-pacman internal ports

lab-matrix ports (reachable via Tailscale)

Same TCP ports as above, already open from existing Stalwart config. No changes needed.

Known Limitations

• No catch-all alias: Stalwart does not support wildcard/catch-all addresses by default. Each alias must be manually created in the webadmin.
• No sieve filters via Nix: Mail filtering rules (sieve scripts) are managed through Stalwart’s webadmin, not declaratively in Nix.
• Outbound IP reputation is manual: Warming up a new sending IP takes weeks. Start with low-volume, high-reputation mail (personal correspondence) before sending bulk notifications.
• Single MX, no backup: Currently one MX host. If vps-pacman is down, inbound mail is deferred by senders (retry for 24-48 hours typically). Consider a secondary MX (e.g., a free tier from Migadu) for high-availability.

Troubleshooting

Outbound port 25 blocked on vps-pacman

Symptoms:

• ssh vps_pacman_root "nc -zw3 gmail-smtp-in.l.google.com 25" returns “blocked”
• Outbound mail sits in Stalwart queue, never delivers
• journalctl -u stalwart | grep -i "connection refused\|timeout"

Fix: Vultr blocks port 25 by default. Submit a support ticket as described in Vultr: Request port 25 unblock. After unblocking, verify:

ssh vps_pacman_root "nc -zw3 gmail-smtp-in.l.google.com 25 && echo 'open' || echo 'still blocked'"

If the ticket is denied, reply explaining it’s a personal mail server with SPF/DKIM/DMARC configured. Vultr’s default denial is automated — manual review typically approves personal use.

Stalwart ACME fails: “DNS-01 challenge timeout”

Cloudflare DNS propagation can be slow. Stalwart retries for wait = "5m" (configured in the module). If it still fails:

1. Verify the Cloudflare API token is correct and has Zone:DNS:Edit permission
2. Check the agenix-secret is decrypted: ssh lab_matrix_root "cat /run/agenix/cloudflare-api-token"
3. Check Stalwart logs: ssh lab_matrix_root "journalctl -u stalwart --since '10min ago' | grep -i acme"
4. Try manually: curl -H "Authorization: Bearer <token>" https://api.cloudflare.com/client/v4/user/tokens/verify

nginx stream proxy fails: “host not found”

nginx stream module needs to resolve lab-matrix’s Tailscale hostname at runtime. Verify:

ssh vps_pacman_root "dig +short lab-matrix.tailf0ad7b.ts.net @100.100.100.100"
# Should return lab-matrix's Tailscale IP

If resolution fails, the Tailscale MagicDNS suffix may have changed. Run tailscale dns status on vps-pacman and update tailnetSuffix in hosts/vps-pacman/modules/nginx.nix.

Gmail rejects mail: “not permitted to send as this domain”

SPF check failed. This means the sending IP is not authorized. Verify:

1. dig TXT alienzj.org — confirm SPF record exists and includes the sending IP
2. Check the Return-Path and Authentication-Results headers in a test email
3. If using the outbound relay, verify mail is actually routing through vps-pacman

Gmail rejects mail: “suspicious activity from your IP”

IP reputation issue. The sending IP (vps-pacman or lab-matrix) is on a blocklist:

1. Check blocklists: https://mxtoolbox.com/blacklists.aspx
2. If vps-pacman’s IP is listed, request delisting (most lists auto-delist after 1-7 days)
3. If lab-matrix’s residential IP is listed (Spamhaus PBL), you MUST use the outbound relay
4. Sign up for Google Postmaster Tools to monitor domain/IP reputation

Outbound port 25 blocked by VPS provider

If your VPS provider blocks outbound port 25:

1. Open a support ticket requesting port 25 unblock (most providers do this)
2. Some providers (AWS) require a form explaining mail use case
3. As a last resort, use a smarthost relay (SendGrid free tier) ONLY for outbound — this is still self-hosted for inbound