Connectors

Two connectors ship with the platform: one consumes from a message broker over STOMP, the other watches a directory. Both are separate binaries with their own process and their own configuration. Neither runs inside the platform: each presents a credential at the queue door and posts the message there, exactly like any other caller.

That is worth knowing when a tender is being written: everything the gateway page says about authentication, throughput ceilings and refusals applies to a connector too — nothing extra, and nothing special for it.

A credential refusal stops the process instead of letting it spin. The credential belongs to the connector, not to the message: if it is wrong, the next thousand messages get the same refusal, and a process retrying forever fills the journal and buries the cause. The platform never answers 401 when the gate could not decide — there it answers 503, which is transient and really is retried.

What is deliberately not terminal: 400, 404 and 422 are still retried. A connector started before the flow it feeds gets a legitimate 404, and stopping on it would be a false defect. Worth planning for: a body the flow refuses is retried until it is removed from the source, so that one message holds the queue in place.

A connector reports into the same journal as the rest of the platform: every message picked up leaves an event, under a correlation of its own that also appears in the delivery’s trace. That reporting does not spend the credential’s throughput ceiling. The ceiling is per credential, so a connector sharing a key with synchronous callers would pay twice per message and quietly lower everyone else’s ceiling.

The message connector takes identity from the identifier the broker hands it; the file connector builds one from path, modification time and size. The queue’s contract stays at-least-once in both cases.

The defence against a duplicate is written in the flow, not in the connector: a deduplication key declared in the frontmatter, and a second delivery of the same key does not run the flow again — it answers from the record, or it stops, as the flow asks. That is the mechanism a business process that must not run twice rests on.

A practical consequence of an identity made of path, time and size: the resolution is one second. A file replaced within the same second, at the same path and with the same size, is not read a second time.

A message’s ceiling is resolved per access point, and a body announced over the ceiling is refused without a single byte being read. The connectors read the same ceiling from the frontmatter of the flow they feed, so there is no second ceiling to drift apart from the first.

Above the ceiling there is no opaque content and no reference into the content-addressed store: the body is still materialised in memory and parsed. Hence the rule we state before any architecture discussion — large files do not travel through a flow. A point cloud, a video recording, a photogrammetric survey moves directly between systems; what goes through the platform is the metadata: where the file is, what it covers, what its fingerprint is, what changed.

And in the other direction: nothing in the platform writes a file on a flow’s behalf. The file connector reads a directory and posts; there is no step that puts a file anywhere.

A flow has two guards with two meanings, and choosing between them is the author’s call: one silently filters out messages it does not care about, the other refuses explicitly, at the first rule that is false, with the message the author wrote. An incomplete, insufficiently aggregated or non-anonymised set is rejected by the second, not dropped by the first.

A flow can also carry a fault sequence. A handler that succeeds does not answer 200: the status stays the error’s natural one, so a refusal is never read as an acceptance.

What does not exist: schema validation. No XSD or JSON Schema is loaded, and the message is not checked against one. The rules are expressions written in the flow, over the message that arrived.

The file connector polls the directory at a configured interval, oldest first. It does not subscribe to filesystem notifications and does not descend into subdirectories. The message connector sits on a broker queue and consumes from there.

Separately from the connectors, a flow can carry a six-field schedule, and a tick leaves the same pair of journal events as a request does, on the cron channel.

What does not exist: a push subscription to a third-party product. If the source has to announce itself to the platform, it does so by calling the door — and then no connector is needed.

• A message accepted at the queue door is written to the queue’s own write-ahead log and survives a crash; a scheduled retry survives a restart.
• The message body sits in that log in the clear. Masking applies at the journal sink, only to JSON, and only on object keys — the queue frame does not get it.
• Acknowledging a message does not delete the bytes. There is neither a time nor a size limit on how long a body stays on a queue: the bytes go when the queue drains completely, or when an abandoned queue is compacted.
• The retention window governs both journal sinks — rows and files — in a single pass. It does not touch the queues, the registry, the audit chain or flow state.

• The file connector reads a local directory. It does not speak SFTP, FTP or SMB, and it does not read an object store.
• The message connector speaks STOMP over plain TCP, on queues. No TLS to the broker, no broker credentials, no topics, and broker headers never reach the flow.
• There is no extension by custom code: no JavaScript runs in a flow, and no modules are loaded. The extension point is the flow itself — written in FlowMarkdown, drawn in the visual editor in VS Code, and checked by the language-server extension as you write it.
• There is no time-series store and no indicator store. A series is stored in the system that owns it, and the flow carries it there.
• No foreign WSDL is consumed, and the WSDL generated for an exposed service is loosely typed.

• ObservabilityWhat stays written after a delivery, and for how long.
• OperatingWho starts a connector, with what configuration, and what shows when it stops.