Backup and restore

Why the backup is a SQLite snapshot and not a file copy, the script that takes it, how to restore from it, and how to prove the restored installation is the one you backed up.

Everything an installation owns lives in one directory — the one that holds registry.db. So a backup is that directory, and a restore is putting it back. The two paragraphs that follow are the reason it is not cp -r.

Why not `cp`

The four databases are opened the same way:

Database	`journal_mode`
`registry.db`, `nexus-audit.db`, `nexus-log.db`, `nexus-ui.db`	`DELETE`, with `synchronous=FULL` — chosen so the directory can sit on a shared filesystem

A cp over a database with the server running can catch it in the middle of a transaction, with its rollback journal still beside it: a file that looks intact and is not. Nothing fails at restore time. The last writes are simply wrong or missing, and nothing says so.

.backup is consistent whatever the journal mode, which is why the procedure below is written with it, and why sqlite3 is worth having on the machine.

The backup

#!/bin/sh
set -e
umask 077
NEXUS_DIR=/var/lib/nexus
DEST=/backup/nexus/$(date +%Y%m%dT%H%M%S)
mkdir -p "$DEST"

# The databases: `.backup` takes a consistent snapshot, live server included.
for db in registry nexus-audit nexus-log nexus-ui; do
  sqlite3 "$NEXUS_DIR/$db.db" ".backup '$DEST/$db.db'"
done

# The queues: files, not a database. See the note below.
tar -C "$NEXUS_DIR" -cf "$DEST/queues.tar" queues

sqlite3 "$DEST/registry.db" "PRAGMA integrity_check;"
sqlite3 "$DEST/nexus-audit.db" "PRAGMA integrity_check;"

The note that matters: .backup is consistent per database, not across them, and queues/ is not a database at all. A backup taken with the server running can catch a message already pulled from the queue and not yet recorded in the log, or the reverse. A tar of queues/ can also catch a queue in the middle of opening a new segment file; the copy restores to a consistent queue (an empty last segment is read and appended to, a torn tail is cut) but holds only what had reached the disk when tar read each file. Stop the server first when you need a copy you can account for.

For a mutually consistent set — the only kind you restore a whole installation from, as opposed to reading one database out of — stop the server:

systemctl stop nexus && /backup/nexus-backup.sh && systemctl start nexus

The live backup keeps its job as the daily net. The one you restore from is taken with the server stopped.

The archive is in the clear. Nothing in the state directory is encrypted at rest: not any of the four databases, not the queue files. Encryption at rest is blocked on key management the platform does not have, and the backup is where that is felt hardest: an archive of the directory carries away everything written in the clear inside it. Store and move it as you would the data itself.

The script sets umask 077 before it creates anything, so the destination is 0700 and every copy in it 0600, readable only by the account that took the backup; the copies keep those modes wherever they are moved.

In a container

Same procedure, with the destination mounted and with -u root:

docker stop nexus
docker run --rm -u root -v nexus-data:/data -v /backup/nexus:/backup \
  --entrypoint sh <the image you were given> -c '
    set -e
    umask 077
    DEST=/backup/$(date +%Y%m%dT%H%M%S); mkdir -p "$DEST"
    for db in registry nexus-audit nexus-log nexus-ui; do
      sqlite3 "/data/$db.db" ".backup \"$DEST/$db.db\""
    done
    tar -C /data -cf "$DEST/queues.tar" queues
    sqlite3 "$DEST/registry.db" "PRAGMA integrity_check;"
    sqlite3 "$DEST/nexus-audit.db" "PRAGMA integrity_check;"'
docker start nexus

-u root is not convenience. The process in the image runs as uid 10001, and a freshly created volume or host directory belongs to root — without the flag, .backup stops with cannot open before writing anything. If the destination already belongs to 10001, drop the flag; what does not work is the default form over a new destination.

The restore

systemctl stop nexus
mv /var/lib/nexus /var/lib/nexus.broken          # nothing is deleted until the new one works
install -d -o nexus -g nexus -m 0750 /var/lib/nexus
cp /backup/nexus/<timestamp>/*.db /var/lib/nexus/
tar -C /var/lib/nexus -xf /backup/nexus/<timestamp>/queues.tar
chown -R nexus:nexus /var/lib/nexus
systemctl start nexus

The -wal and -shm files are never copied from a backup. .backup produces complete databases, and a foreign -wal sitting next to a restored database is the most effective way to corrupt exactly what was just recovered.

Then verify, in this order:

curl -s localhost:9090/health                       # {"ok":true}
nexus audit --tenant acme --verify --db /var/lib/nexus/nexus-audit.db
#   -> chain integrity: OK (N entries checked)
nexus tenant list --db /var/lib/nexus/registry.db
nexus keys list --tenant acme --db /var/lib/nexus/registry.db
nexus state list --tenant acme --db /var/lib/nexus/registry.db

nexus audit --verify on an empty chain answers (no audit entries for tenant …) and exits successfully. That is not a check that passed; it is a chain that is not there. The entry count in the success message is the number to compare with what was there before.

A chain restored from an older backup is shorter than the one that was lost, and its own links hold. What tells you is a checkpoint kept elsewhere: run nexus audit verify --tenant acme --anchors <the log files shipped off the host> --db /var/lib/nexus/nexus-audit.db and a restore that went back in time answers tail truncated with the number of entries missing. See Anchoring the tail.

What a restore does not bring back

A restore is a restart with older files, so everything a restart loses is lost here too. None of it is in any backup, because none of it is on disk.

Rate-limit buckets are in memory. After a restart every key and every OIDC subject gets a new bucket holding one second’s worth — rps tokens, the capacity being twice that. So a key limited to 10 requests per second that was throttled just before a restart can pass 10 requests at once just after it.
Cron triggers that fell inside the stop window are not caught up. The scheduler computes the next tick from the moment it starts.
The gRPC routing table is rebuilt at boot from the latest artifacts, so a flow deployed after the restore is routed only after the next restart.
Queue workers start for queues that hold an undelivered message, at start and every 5 seconds after, and on the first POST /enqueue for a pair; an empty queue has none.

And what survives, which is most of it: the retry heap is rebuilt from segment.log.retry with absolute deadlines; queue position comes from segment.log.ckpt, so an unacked message is redelivered — at-least-once, so a flow without a dedup_key: can run that delivery twice; UI sessions are in nexus-ui.db and stay valid; deduplication memory and open correlations are in registry.db and come back with it.

Those last two come back only if the state key does. Deduplication fingerprints and correlation keys are stored keyed by NEXUS_SECRET_NEXUS_STATE_KEY, which deliberately lives outside the databases — whoever holds the file with the fingerprints would otherwise hold the key that made them. registry.db records a check value that names the key the rows were written under, without revealing it, so a restore started under a different key while rows are stored is refused: the start stops before any listener and says how many deduplication rows and correlations would stop matching. Two ways forward: start with the key from the backup’s secrets, and everything matches again; or start once with nexus serve --accept-new-state-key, which records the new key, writes state.key_replaced to the audit chain, and costs one retention window in which every redelivered message runs again. The next start drops the flag; left in place, it is refused. So the key travels with the backup, in whatever holds the installation’s other secrets, and it is the first thing to check before the restore rather than after. See the state key at install time.

Upgrades

An upgrade of the platform migrates the databases in place, on the first start of the new binary, and there is no down migration. So the backup taken with the server stopped — the mutually consistent one — is what makes an upgrade reversible: you go back by putting the old binary and those files back. The procedure is under What an upgrade of the platform does.