Operating

An engine is the binary plus the state directory you point it at. A second engine is the same binary, started again over the same directory. There is no node orchestrator to install, no node registry to keep, no consensus protocol and no heartbeat: engines do not know about each other, they coordinate through POSIX locks over the directory they share. One takes the scheduling lock, each queue gets exactly one worker, and appends to the same segment serialise on its lock.

That shape asks something of the filesystem, and the requirement is not a formality: real fcntl record locks — never flock(), whose semantics vary from one network mount to the next. The registry and the audit chain are opened deliberately with a rollback journal instead of WAL and with a sync on every write, precisely so they can be shared. The L1 journal and the management-UI database stayed in WAL, which needs SQLite’s shared-memory index; since there is one state directory and one path flag, it has to support both.

Three things are planned around in this shape, and they are said here rather than at three in the morning. Failover between engines is not automatic: the cron scheduler is elected once, at boot, so if the engine holding it stops, the schedules resume at the next restart. Rate limiting is counted per engine, so size a ceiling for the engine, not for the installation. And the worker lock is deliberately permissive on an I/O error: it prefers a duplicated delivery to a stalled queue, which stays inside the at-least-once contract. The manual writes all of it, in the operations chapter on running more than one engine.

• The flow endpoints, the WSDL, the health check, the queue workers, the schedule runner and the management UI all run in the same process. The UI is not a second binary and has no port of its own.
• The gRPC door listens on a separate port, in the same process, and only when asked for at start-up.
• One runtime owner per process, and a platform at capacity refuses loudly, leaving a trace in the journal.
• An installation comes up knowing only the operations manual: the first flow runs with the gate closed and configuration from the environment, inside the shipped image.

TLS is not terminated in the process. It is terminated in front of it, at the reverse proxy, and the gRPC door speaks in the clear behind that.

Back it up as a unit. A database taken without its queues describes an installation that never existed.

The audit chain in nexus-audit.db can be verified at any time — nexus audit --tenant <t> --verify, or the button on the audit page. A break verdict names the sequence number of the entry that was touched. A chain nobody asked about is not verified on its own, and a listing says nothing about integrity. The command takes one tenant, so an installation with several is verified one at a time.

Nothing in that directory is encrypted at rest: not any of the four databases, not the queue files. Encryption at rest is blocked on key management the platform does not have, and the backup is where that is felt hardest — an archive of the directory carries away everything written in the clear inside it.

A flow declares the environment configuration it needs, and a declared key the installation does not supply stops the boot, naming the key. At start-up, not at the first message.

The same question can be asked without starting anything: nexus config check gives the same verdict as start-up and exits non-zero when a key is missing. One verdict, two moments — not a second implementation that would go stale in exactly the copy the operator runs.

And a boot scan cannot stay silent: a registry that cannot be listed stops the boot rather than letting it look checked.

• Environment configuration is captured once, at start-up. Changing it requires a restart — if the values were re-read along the way, the gate at start-up would have checked values other than the ones the platform is running on.
• Adding, changing or removing a schedule requires a restart. Changing the steps of an already scheduled flow does not: the next tick picks up the new version.
• The gRPC routing table is built at start-up. A flow published after that is not routed until a restart, and a withdrawn one answers like a flow that does not exist.
• Rate ceilings live in process memory. A restart empties them, and two engines keep two counters: a key configured at *r* requests per second can spend up to *N × r* across N engines. Size the ceiling for the engine, not for the installation.

At-least-once. That is the whole guarantee, and it is written here because the rest of the page depends on it: whoever builds on the queue builds idempotence at the destination rather than declaring it on the platform. What the platform offers is declarative dedup, which is a different thing.

• The queue has its own write-ahead log: an accepted message survives a crash, and a scheduled retry survives a restart.
• Order is kept on first delivery only. A retry takes its place in the order its turn came up in, not the order the message arrived in.
• The worker ceiling is not a rate: it is the number of distinct queues that can drain on this server at all. Count your queues, not your traffic.
• After a restart, a queue’s unprocessed messages sit on disk until something is submitted to that flow again. The worker starts on the first submission, not on process start.
• A drained queue reclaims its bytes while the worker keeps consuming, and a dead letter discarded is really gone from disk.

A message body sits in the queue file as plaintext JSON inside a checksummed frame. The checksum is integrity, not a cipher: it says whether the bytes rotted, it does not hide them from anyone.

Masking does not reach there. It is applied at the journal sink, on JSON object keys, and the body written to a queue never sees it.

Acknowledging deletes nothing. An acknowledged message moves a byte marker forward; everything before it stays on disk, in the clear. Those bytes go in one of two moments: the queue reaches a fully drained state and the worker truncates its file, or the queue is abandoned and housekeeping deletes its files. There is no time limit, no size limit and no expiry. On a queue with continuous traffic that moment may never come.

And there is no erasure of one specific message on request. The only per-message deletion is discarding a dead letter: it acts on the dead-letter copy, not on the original in the queue, and takes effect at the next drain.

A flow can carry a six-field schedule, and a tick leaves the same pair in the journal — a receipt plus a verdict — as a request through the front door, on the cron channel. A schedule that failed is not invisible.

And in the same breath, because an operator has to learn this beforehand rather than at three in the morning: the scheduling lock makes a single engine carry the schedules, so nothing fires twice — but the election runs once, at boot. If the engine holding the lock stops, the schedules go silent until the next engine restart. Plan the restart; there is no automatic takeover.

The journal has two sinks — rows in a database and files on disk. One retention window governs both, in a single pass, wired into the serving process.

That window does not reach beyond the journal: it does not touch the queues, the registry, the audit chain or flow state. Planning an installation’s disk means planning four things, not one.

There is no /metrics endpoint and no journal export to an external collector — syslog, OTLP, statsd, webhook, fluentd, Kafka: there are exactly two destinations, both local.

• ObservabilityWhat stays written after a delivery, and for how long.