Running more than one engine

An installation scales by starting the same binary again. There is no orchestrator to install, no node registry to keep, no consensus protocol, no heartbeat and no leader-election service. Engines do not know about each other; they coordinate through the state directory they share, using POSIX advisory record locks. Add an engine and the work redistributes; stop one and the rest keep serving.

This page describes that shape, what each engine keeps to itself, and what the filesystem underneath has to support for it to hold.

The shape

Every engine is started the same way, pointed at the same state directory:

$ nexus serve --port 9090 --db /srv/nexus/registry.db     # engine 1
$ nexus serve --port 9091 --db /srv/nexus/registry.db     # engine 2
$ nexus serve --port 9092 --db /srv/nexus/registry.db     # engine 3

--db is the only path flag. Its parent directory is the state directory, and everything else is placed relative to it — see Server and configuration. Put a reverse proxy in front of the HTTP ports, terminate TLS there, and health-check each engine on GET /health, which needs no credential and answers {"ok":true}.

Each engine boots identically and says on its own console which roles it took:

INFO scheduler: acquired singleton lock — this node is the scheduler

The engines that did not win it say nothing at info level: they ask again every 5 seconds, and the line “another node holds the scheduler lock” is written at debug. When one of them takes the seat later, after the holder is gone, it writes the same acquired singleton lock line then.

What is elected, and what is merely serialised

Three things in the state directory must not happen twice at once. Each is answered by a lock file, and the difference between them is whether the losing engine waits or walks away.

Concern	Lock file	Behaviour
One queue worker per `(tenant, flow)`	`queues/{tenant}/{flow}/worker.lock`	Non-blocking, asked again every 5 seconds while the queue has an undelivered message. The engine that holds it runs the worker; the others do not.
One cron scheduler per installation	`queues/scheduler.lock`	Non-blocking, asked again every 5 seconds until taken. The engine that takes it runs the scheduled flows and the correlation deadline sweep; the others do not, and take the seat if the holder goes.
Appends to one queue’s segments (sealing the active one when it reaches `queue_segment_max_bytes` included)	`queues/{tenant}/{flow}/segment.log.lock`	Polled. Every engine appends; an engine that finds the lock held retries after a pause of 1 to 20 ms, and the lock is held across open, seek, write and fsync, so two engines submitting to the same queue interleave messages rather than corrupting the file. The segment is opened by name for every append, so an append always lands in the file the name designates now, also after another engine has compacted the queue.

The scheduler and the correlation sweep share one lock on purpose. Both answer the same question — “exactly one sweeper per installation” — and a second lock would be a second answer to it.

Everything else that crosses engines is settled by SQLite rather than by a lock file. The deduplication row and the correlation lease in flow_inbox are taken with a single atomic statement, so two engines racing on the same key produce one winner and one refusal, not two executions. The audit chain resolves the same race by retrying: two engines can read the same tail and try to write the same sequence number, one loses on the UNIQUE constraint, re-reads the tail with the new previous hash and appends behind the winner. The chain stays verifiable.

What the filesystem has to support

POSIX fcntl record locks. Not flock() — its semantics vary by kernel and by network filesystem implementation, which is why the platform never uses it. If fcntl locking does not work on the mount, nothing above holds.

The journal mode of the four databases. They are all the same:

File	Journal mode	Settings
`registry.db` (the `--db` file)	`DELETE`	`synchronous=FULL`, `busy_timeout=5000`
`nexus-audit.db`	`DELETE`	same, plus the chain retry above
`nexus-log.db`	`DELETE`	same
`nexus-ui.db`	`DELETE`	same

DELETE is chosen deliberately: WAL needs SQLite’s shared-memory index, the -shm file, which a network filesystem cannot share between processes on different hosts. Because no database uses WAL, the whole state directory, the four databases, queues/ and logs/, can sit on one NFSv4 export shared by engines on different hosts. There is one state directory and one flag, so the four cannot be placed on different filesystems, and they do not need to be.

Upgrading is not rolling. The first engine of this version that opens a database an older engine left in WAL converts it to DELETE, and that needs the file to have no other connection. If another engine still holds it, the engine refuses to start and says so: stop every engine of the installation, then start them again. An older engine started against a converted directory would put the file back in WAL. So stop all engines, upgrade all, start all.

The mount and the export. One export holds the whole state directory. Mount it NFSv4.1 or later with hard and vers=4.1, and without nolock or soft. The server exports it with sync: with async an acknowledged write is not stored, and an fsync on the engine means nothing. Keep the clocks of the engines within one second of each other with NTP, because a management-UI session expires on the wall clock. The run that proves two engines over such a mount is in docs/ha-two-engines-run.md of the source tree.

What the mount costs. On a network filesystem every file lock and every read of a database that another host also uses is a round trip to the server, so a statement takes milliseconds, not microseconds. The platform reads the registry for each request through a small pool of read-only connections, one transaction per request, and caches nothing: a key revoked on one engine is refused by the next request on the other. Over an NFSv4.1 export, two engines each sustained 100 requests per second for 60 seconds on one flow with no event lost. A message submitted to one queue from two hosts at once is slower, in the order of 10 per second per host, because each append is a lock, a write and a flush on the server. A queue worker takes the next message at once while the queue has any, and polls only when it is empty: after 500 ms, then 1 s, then every 2 s while it stays empty; on the shared mount an idle queue costs about 0.6 RPC per second per host (measured: 0.60 and 0.63), and that grows with the number of idle flows. If the journal is busy for longer than five seconds, a batch of events waits and is tried again until it is stored; it is not dropped, and the log says so.

A queue is a set of segment files, and a rotation made on one host reaches the other after the NFS directory cache. When a queue’s active segment reaches queue_segment_max_bytes, the writing engine creates the next segment.<offset>.log. The worker on the other host lists the directory again at the end of the last segment it knows, and an idle worker checks at most every 2 seconds at the default poll_interval_ms. So a message enqueued on one host right after a rotation can wait up to 2 seconds more on the other host’s worker. It happens once per queue_segment_max_bytes of traffic and no message is lost. See Queues.

Sessions of the management UI are shared. They live in nexus-ui.db, so a login on one engine is valid on the others, and the log page of any engine shows the rows every engine wrote, because there is one flow_log. The proxy needs no sticky routing; see Gateway boundary.

The daily log files. Each engine writes its own file, logs/nexus-log-YYYY-MM-DD.<host>-<pid>.jsonl, where <host> is the host name and <pid> the process id, so no file has two writers and nothing interleaves. A restart opens a new file the same day. Files written before this version, nexus-log-YYYY-MM-DD.jsonl, are still recognised and are removed by the same retention. A log shipper reads the glob nexus-log-*.jsonl and orders by the ts field of each line. The rows in nexus-log.db remain the ordered record — see Logging and audit.

What each engine keeps to itself

Some state is per process by construction, and an operator sizing an installation has to count it per engine rather than per installation:

Rate limit buckets. The token bucket lives in memory, keyed by API key. With N engines, a key configured at r requests per second can spend up to N × r if the proxy spreads its requests. Size the ceiling for the engine, not for the installation.
The execution budgets of ADR-037 — the residency slots that decide how many queue workers one engine will start, and the in-flight ceiling. Each engine has its own.
The gRPC routing table. Built at boot from the deployed artifacts, on each engine independently. A flow published afterwards is routed by an engine only after that engine restarts.
The OAuth2 token cache and the circuit breakers of http_egress and grpc_egress. Each engine holds its own, so a breaker opened by one engine does not open on the others.
The in-memory retry heap of a queue worker, which belongs to the engine holding that queue’s worker lock and is persisted next to the segment.

The maintenance loop — log retention, queue compaction, the certificate sweep, the flow-state purge — runs on every engine rather than on the elected one. Its work is deletion of things past a window, so repeating it is harmless; a race is a file already gone, reported as a warning. Queue compaction is the one that could destroy data, and it asks for the worker lock before deleting anything, so it cannot delete the files under another engine’s worker.

Losing an engine

Requests in flight on that engine are lost to their callers the way any request is when the process behind a proxy goes away. A synchronous /run has no record; a message already written by /enqueue is durable and is drained by whichever engine holds that queue’s worker.

Queues keep draining, with no one asked to do anything. Every engine, every 5 seconds, lists the queues that hold an undelivered message and asks for each one’s worker.lock. The engine that gets it runs the worker; the others try again at the next tick. So when the engine holding a queue’s worker goes away, a surviving engine takes the queue over by itself, with no new submission and no restart. The same pass runs at boot, so a backlog left on disk is drained by the engine that starts, including a single engine after a restart. How long it takes depends on how the engine went away:

What happened	A surviving engine starts delivering within
The process was killed (including `kill -9`)	10 seconds. The kernel releases the lock when the process exits; the next tick takes it.
The engine was stopped with `SIGTERM` (`systemctl stop`, `docker stop`, a rolling restart)	10 seconds from the moment it exits. The engine stops taking requests, lets the delivery in progress finish, exits with status 0 and releases its locks at exit, usually in under a second; it waits at most 60 seconds for a delivery that does not end, then exits anyway and the message is delivered again by the survivor.
The host was lost (power, kernel panic, network)	The NFS lease time plus 10 seconds. The server releases a dead client’s locks only when its lease expires. The Linux server default is 90 seconds, so 100 seconds; with the lease set to 30 seconds, 40 seconds.
The NFS server restarted	Its grace period, then 10 seconds.

The bounds are counted from the moment the lock is free. They were measured on three machines (two engines and an NFS server) with a backlog of 100 messages on the queue: with the process killed, the survivor delivered its first message after 2.7 to 3.2 seconds and had drained the backlog after 8.4 to 9.4 seconds, in three runs; with the host stopped without a shutdown and a lease of 30 seconds, the first delivery came after 19 to 29 seconds and the backlog was drained after 25 to 35 seconds. How long the whole backlog takes after that is the queue’s own rate, which on a shared mount is bounded by the mount, not by the takeover. To make the host-loss bound shorter, lower the lease time on the NFS server (/proc/fs/nfsd/nfsv4leasetime); the engine has no setting for it.

Delivery stays at-least-once. A message in the middle of delivery when the engine dies is delivered again by the next worker, and that is at most one duplicate per death. Two workers are never running on one queue at once. A message waiting for a retry keeps its schedule: the retry heap is rebuilt from segment.log.retry, so its attempts continue where the dead engine left them, and the engine that adopts the queue moves it to the DLQ on the attempt that exhausts the limit.

Scheduled flows move too, and so does the correlation deadline sweep. Until an engine holds scheduler.lock, every engine asks for it at each tick. When the holder exits, the next tick on a surviving engine takes the seat, reads the scheduled flows published at that moment (including those published since boot) and starts them, within the same bounds as the queues. Cron ticks that fall in the gap are missed, not replayed, exactly as with a single engine that is down. A flow that must not lose a tick should not depend on the clock alone: let a queue or a correlation carry the work.

A lock that cannot be read is not a lock taken. If the worker lock file cannot be opened — an I/O error on the mount, not a lock held elsewhere — the engine does not start the worker. It logs a warning naming the tenant and the flow, on every tick for as long as the error lasts, and tries again at the next one. Two workers on one queue are what the lock exists to prevent, and a mount that cannot answer a lock request is the mount where that would happen. The same holds for the scheduler lock: an engine that cannot read it schedules nothing and says so. See Queues and delivery.

Putting a new version live

The procedure is the one in Server and configuration — deploy the new flow version, then let it resolve — and it does not change with more engines, because an artifact is immutable and resolved per request. What does change is the engine restart: take one engine out of the proxy at a time, restart it, put it back. If the engine you restart held the scheduler or a queue’s worker, a surviving engine takes it over within the bounds under Losing an engine; no order of restarts is needed.

Planning an installation

Two properties of this shape decide how an installation is sized and operated. Each is described above; collected here because they are what an operator plans around.

Rate limits are counted per engine. A key configured at r requests per second can spend up to N × r across N engines, depending on how the proxy spreads its requests. Size the ceiling for the engine.
The cron scheduler and each queue’s worker are taken over by a survivor, not instantly. Plan for 10 seconds when a process dies and for the NFS lease plus 10 seconds when a host does; cron ticks inside that gap are missed.