Connectors
A connector brings messages into the platform from outside the program that runs your flows: one consumes from a broker, the other watches a directory. Both present a credential at the queue door, like any other caller, with an API key and that key’s ceiling — so what the gateway page says about API keys applies to them too.
What a connector is
Two connectors ship with the platform: one consumes from a message broker over STOMP, the other watches a directory. Neither is part of the program that runs your flows: each presents a credential at the queue door and posts the message there, exactly like any other caller.
What the gateway page says about the API key, the throughput ceiling and refusals applies to a connector too: nothing extra, and nothing special for it.
What a connector does with a refusal
| The door’s answer | What the connector does | What happens to the message |
|---|---|---|
| 401, 403 | Stops, naming the reason and the repair | Stays recoverable — not lost, and not retried forever |
| 400, 413 | Moves on to the next message | Set aside and kept, to be corrected and sent again |
| 429 | Waits as long as the answer says, with a five-minute ceiling of its own | Retried |
| 5xx, network failure | Retries, with a growing back-off | Retried |
A credential refusal stops the process instead of letting it spin. The credential belongs to the connector, not to the message: if it is wrong, the next thousand messages get the same refusal, and a process retrying forever fills the journal and buries the cause. The platform never answers 401 when the gate could not decide — there it answers 503, which is transient and really is retried.
What is deliberately not terminal: a 404 is still retried. A connector started before the flow it feeds gets a legitimate 404, and stopping on it would be a false defect. A body the queue door cannot read gets a 400 and is set aside like one over the ceiling: the connector moves on to the next message, so one unreadable message does not hold up the ones behind it.
Connector telemetry does not spend the caller’s ceiling
A connector reports into the same journal as the rest of the platform: every message picked up leaves an event, under a correlation of its own that also appears in the delivery’s trace. That reporting does not spend the credential’s throughput ceiling. The ceiling is per credential, so a connector sharing a key with synchronous callers would pay twice per message and quietly lower everyone else’s ceiling.
How the platform recognises a message it has already seen
The message connector takes identity from the identifier the broker hands it; the file connector builds one from path, modification time and size. The queue’s contract stays at-least-once in both cases.
The defence against a duplicate is written in the flow, not in the connector: a deduplication key declared in the frontmatter, and a second delivery of the same key does not run the flow again — it answers from the record, or it stops, as the flow asks. That is the mechanism a business process that must not run twice rests on.
A practical consequence of an identity made of path, time and size: the resolution is one second. A file replaced within the same second, at the same path and with the same size, is not read a second time.
The size ceiling, and what does not travel through a flow
A message’s ceiling is resolved per access point, and a body announced over the ceiling is refused without a single byte being read. The connectors read the same ceiling from the frontmatter of the flow they feed, so there is no second ceiling to drift apart from the first.
Above the ceiling there is no opaque content and no reference into the content-addressed store: the body is still materialised in memory and parsed. Hence the rule we state before any architecture discussion — large files do not travel through a flow. A point cloud, a video recording, a photogrammetric survey moves directly between systems; what goes through the platform is the metadata: where the file is, what it covers, what its fingerprint is, what changed.
Incomplete or damaged data
A flow has two guards with two meanings, and choosing between them is the author’s call: one silently filters out messages it does not care about, the other refuses explicitly, at the first rule that is false, with the message the author wrote. An incomplete, insufficiently aggregated or non-anonymised set is rejected by the second, not dropped by the first.
A flow can also carry a fault sequence. A handler that succeeds does not answer 200: the status stays the error’s natural one, so a refusal is never read as an acceptance.
What starts on its own: polling and a schedule, not a subscription
The file connector polls the directory at a configured interval, oldest first. It does not subscribe to filesystem notifications and does not descend into subdirectories. The message connector sits on a broker queue and consumes from there.
Separately from the connectors, a flow can carry a six-field schedule, and a tick leaves the same pair of journal events as a request does, on the cron channel.
What stays on disk, and for how long
- A message accepted at the queue door is written to the queue’s own write-ahead log and survives a crash; a scheduled retry survives a restart.
- The message body sits in that log in the clear. Masking applies at the journal sink, only to JSON, and only on object keys — the queue frame does not get it.
- Acknowledging a message does not delete the bytes. There is neither a time nor a size limit on how long a body stays on a queue: the bytes go when the queue drains completely, or when an abandoned queue is compacted.
- The retention window governs both journal sinks — rows and files — in a single pass. It does not touch the queues, the registry, the audit chain or flow state.