Skip to content
NexusFabric

Tracing

This page covers two traces. The payload trace is kept in the engine and read in the management UI. Spans over OTLP leave the engine for the collector you run, and carry no payload.

The payload trace is a second, independent trace for one purpose: a support operator investigating a specific delivery needs to see the payload at every step, not just what log_level happened to keep.

log_level is the flow author’s decision, made at deploy time, about what to retain long-term. It is the wrong lever for “what happened to this one message yesterday” — the flow would have had to be deployed with full logging before the incident, on the chance it would need it. Tracing is the other kind of switch: turned on by an operator, on a running flow, and it stays on — retroactively useful the next time something goes wrong, not just the next request.

Turning it on

Aspect Detail
Where Registry page, per flow. A toggle next to the flow’s versions.
Who sysadmin only. support reads a trace; only sysadmin decides one gets collected.
Stored flow_trace_config in the registry database — not the flow’s frontmatter. Turning it on or off does not require a redeploy.
Scope One (tenant, flow) pair. Turning it on for one flow does not affect any other.
Duration Until explicitly turned off. Not “trace the next request” — every delivery is traced from the moment it is enabled.

Both actions are audited (trace_enabled / trace_disabled), attributed to the sysadmin who did it, on that tenant’s chain. Turning it on twice, or off when it is already off, writes nothing — same rule as revoking an already-revoked API key.

The correlation_id on a traced event is the one the caller received in the X-Correlation-ID response header, so a trace is found from the id a caller quotes (where the caller finds it).

What it adds

Two event kinds, on top of the ones in Logging and audit:

Kind Emitted when
step_traced After a step finishes. Carries ctx.input as it stood at that point — masked, the same 25 global fields plus the flow’s own log_extra_mask_fields.
egress_traced After one http_egress call, successful or not. Carries the response status, headers, and body — as one record, since a support investigation needs all three together, not three rows to reassemble.

Both are independent of log_level. A flow deployed with log_level: event_only still produces a full trace once tracing is turned on — the two settings answer different questions and neither implies the other.

step_traced

{
  "ts": "2026-08-20T09:14:02.118Z",
  "correlation_id": "3f8b1c42-9e70-4a51-b8d2-1c0e7a4f9b33",
  "tenant": "acme",
  "flow": "forward-order",
  "step": "build-order",
  "event": "step_traced",
  "payload": { "orderId": "A-1", "apiKey": "***" }
}

One row per step, in execution order. payload is ctx.input after that step ran, so a value that changes at step three and breaks at step five is visible as the change, not inferred from the before-and-after of the whole flow.

A plain http_egress step — no transform of its own — does not also get a step_traced row: egress_traced already reports the same value, and one step producing two identical rows would be noise. The exception is restore_body: if the step restores a snapshot into ctx.input, that value can differ from what egress_traced saw (the backend’s raw response), so both rows are kept.

egress_traced

{
  "ts": "2026-08-20T09:14:02.240Z",
  "correlation_id": "3f8b1c42-9e70-4a51-b8d2-1c0e7a4f9b33",
  "tenant": "acme",
  "flow": "forward-order",
  "step": "call-backend",
  "event": "egress_traced",
  "http_status": 502,
  "payload": {
    "headers": { "content-type": "application/json" },
    "body": { "error": "downstream unavailable" }
  }
}

Written whether the call succeeded or not — a failed backend call is exactly the case a trace exists for. headers has the same credential denylist applied as everywhere else on this platform (authorization, set-cookie, cookie, www-authenticate, proxy-authorization are stripped before this is ever recorded).

body is null in two different situations, and the trace does not collapse them:

  • the response genuinely had an empty body;
  • the response was abandoned before it was read, because it announced a size over max_message_bytes (see Limits and constraints). The call still shows up, with its status and headers — a rejected-for-size response is not silently missing from the trace.

Reading it

Trace rows are visible on the Logs page to support only. This is enforced where the rows are selected, not filtered out afterward in the UI — a developer or sysadmin who explicitly asks for step_traced or egress_traced by name gets zero rows, not a page that quietly omits them. The event filter dropdown and the step column only appear for support, for the same reason: offering a filter nobody with that role can ever get a result from would just be confusing.

The entry row

The very first traced row of a delivery has no step name yet — nothing has run. Instead of a blank step, it carries where the delivery came from:

step on the entry row Delivery arrived via
received POST /run
enqueued POST /enqueue, what was written to the queue
dequeued A queue worker, what it read back out

This is what lets two entry rows on the same correlation_id be told apart — what was written to the queue and what the worker later read can differ in principle, and both are worth keeping.

This label lives only on step_traced rows. The ordinary boundary events — received, completed, queue_enqueued, queue_processed, queue_filtered, queue_failed, dlq_moved — never carry anything in their own step field. event already says what they are; a step column duplicating it would suggest an intermediate stage that does not exist.

Fan-out and re-entry

A foreach, a split, a fork branch, and the ## Fault: sequence each start their own inner run of the executor — and each one writes its own entry row, the same way a delivery does. These are labelled too, distinctly from the top-level ones and from each other:

step on the entry row Written by
foreach[i] Iteration i of a foreach, in order.
split[i] Branch i of a split — the index is stable even though branches run concurrently and can land in the log in any relative order.
fork[i] Branch i of a fork.
fault The ## Fault: sequence. Not indexed — it never recurses into itself, so there is at most one per delivery. Its payload is not the original message but the synthetic { fault_step, fault_error, fault_correlation_id } object it runs with.

So a trace of a flow with a ten-item foreach shows ten entry rows, foreach[0] through foreach[9], each followed by that iteration’s own step_traced rows — the index is what lets you follow one item’s path through the fan-out instead of guessing from the payload alone.

XML input

Masking works on JSON object keys. An XML value has none, so a step whose ctx.input is XML at the moment it is traced does not carry the document — it carries a marker instead:

{ "_unrepresentable": "XML value (root <Order>) — masking is JSON-only, not traced" }

This is deliberate, not a gap to be filled by converting XML to JSON first: a conversion built for field access does not guarantee that every attribute lands under a name the mask list recognizes. An XML body queued on /enqueue is traced the same way, as the marker. Running one here would risk a masked-looking payload that quietly leaked a value the conversion mapped somewhere unexpected — worse than admitting the document cannot be traced. See also the equivalent note for XML output in Logging and audit.

Text input

A step whose ctx.input is a string when it is traced (a CSV queued as text/csv, a bare JSON string) does not carry the text. Its step_traced payload is the string’s size and digest:

{ "_text": { "bytes": 45, "blake3": "9dc753d7d62b83a964419dd8f113b7a17b63d4831bf53b51f242a6dd930f2fda" } }

The entry row of a text body queued on /enqueue is the same marker. The rule, and what it covers, is in A string with no field name.

Volume

Trace payloads are masked but never truncated — unlike an ordinary completed or queue_processed payload at payload_trimmed, which is capped at log_payload_max_bytes. This is deliberate, not an oversight: a support trace exists precisely because that cap was not enough to answer “what happened to this message” in the first place. Applying the same cap to the tool built to get past it would just recreate the shortfall inside it.

The trade-off is storage, not correctness: a flow whose payloads are large writes large trace rows, for as long as tracing stays on for it. Turn it off once the investigation is done, the same way you would stop a tcpdump.

Cost

The enabled/disabled check runs on every /run, /enqueue, and dequeue, uncached, so the question of whether that matters was measured rather than assumed: it costs on the order of 10µs, regardless of how many flows have tracing on. Leaving it on for a flow you are actively debugging is not a capacity decision — the cost that matters is the one above, storage, not this one.

Spans over OTLP

With an endpoint set, the engine exports one span per accepted delivery, and one child span per effect call that delivery makes, to an OpenTelemetry collector. The spans join the caller’s trace when the caller sends traceparent, and every egress sends the trace on. Without an endpoint none of this happens: no span is made, no traceparent or tracestate is read or sent, no queued message carries one, and nothing connects to a collector.

Turning it on

The engine reads the standard OpenTelemetry exporter variables itself; it does not load an OpenTelemetry SDK. Each variable has a traces form and a generic form, and the traces form wins when both are set. An empty value counts as unset.

Variable Value
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_ENDPOINT The collector’s URL, http:// or https://. The traces form is used as given; the generic form gets /v1/traces appended. Announced at start as NexusFabric OTLP trace export to <url>.
OTEL_EXPORTER_OTLP_TRACES_HEADERS, OTEL_EXPORTER_OTLP_HEADERS Comma-separated key=value pairs sent on every export request, values percent-encoded: Authorization=Bearer%20<token>. Secret: the values are never printed, not at start, not in an error.
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, OTEL_EXPORTER_OTLP_PROTOCOL http/protobuf, the default and the only value accepted.
OTEL_EXPORTER_OTLP_TRACES_TIMEOUT, OTEL_EXPORTER_OTLP_TIMEOUT Milliseconds per export request, 1 or more. Default 10000.
OTEL_EXPORTER_OTLP_TRACES_COMPRESSION, OTEL_EXPORTER_OTLP_COMPRESSION gzip or none, the default.
OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE, OTEL_EXPORTER_OTLP_CERTIFICATE A PEM file of CA certificates trusted for the collector only, for a collector under a private CA.
OTEL_EXPORTER_OTLP_TRACES_CLIENT_CERTIFICATE, OTEL_EXPORTER_OTLP_TRACES_CLIENT_KEY, and the generic pair A PEM certificate chain and its PEM private key, both or neither, for a collector that asks for a client certificate. The key file is secret.
OTEL_SERVICE_NAME The service.name of the spans. Default nexus-fabric.

The protocol is OTLP/HTTP with protobuf bodies, sent to the endpoint with POST. OTLP over gRPC and OTLP/HTTP with JSON bodies are not supported; a collector listens for OTLP/HTTP on port 4318 by default.

Use https:// when the headers carry a credential. Over http:// the token crosses the network in clear on every export request.

A value the engine does not accept (an endpoint that is not an http or https URL, another protocol, a timeout that is not a positive number, a malformed header list, a certificate file that cannot be read, a client certificate without its key) stops nexus serve before it opens any port, and nexus config check fails on it. The message names the variable, never its value. Setting one of these variables without an endpoint is refused the same way, since it would have no effect. Without an endpoint, nexus serve and nexus config check print nothing about trace export.

Any other OTEL_* variable (OTEL_TRACES_SAMPLER, OTEL_RESOURCE_ATTRIBUTES, and the rest) is not obeyed. Each one present is named at start in a warn line, OTEL_* variable not read, without its value.

What a span holds

Span Kind Name Attributes
A delivery SERVER on the HTTP and gRPC doors, CONSUMER for a queue attempt, INTERNAL for a scheduled run deliver <flow> nexus.tenant, nexus.flow, nexus.channel, nexus.correlation_id, nexus.outcome
An effect call CLIENT for http_egress (SOAP included) and grpc_egress, PRODUCER for queue_publish, INTERNAL for the others <effect> <step> nexus.effect, nexus.step

The resource carries service.name and service.version. nexus.channel is http, grpc, queue or cron; nexus.outcome is completed, fault, dedup_replayed or dedup_held_back. A span whose delivery or call ended in an error has status ERROR.

These are the only attributes. No span holds the payload, a header, a secret, an API key, a URL, a host name or an error text: what went wrong is in the structured log, found by the nexus.correlation_id on the span.

A request the access gate refuses (401, 403, 429) makes no span.

traceparent and tracestate

Every door reads the caller’s W3C Trace Context headers: the HTTP door from the request headers, the gRPC door from the call’s metadata. A traceparent that is not valid under W3C Trace Context Level 1 is dropped whole, and the delivery starts a new trace. tracestate is kept only beside a valid traceparent; several header lines are joined, and a value over 512 bytes, or with a byte outside the allowed set, is dropped whole. A dropped value is not logged, not stored, and not sent on.

Each http_egress, SOAP and grpc_egress call sends traceparent with the effect span as parent, and tracestate as received. A message put on a queue carries both in its frame: from queue_publish, the effect span as parent; through /enqueue, the caller’s traceparent as received. The queue worker’s delivery continues that trace on every attempt, and the frame keeps it through the dead-letter queue and a replay.

A caller that sends the sampled flag 00 is followed: nothing is exported for that delivery, and the egress calls send traceparent with 00 on. A delivery with no caller trace is sampled.

The trace id and X-Correlation-ID are independent. Neither is derived from the other; the correlation id is on the delivery span as nexus.correlation_id, which is how a trace leads to the log rows of the same delivery.

A flow cannot write either header: traceparent and tracestate are refused at publish in headers:, gRPC metadata: and a step’s response_header_<name>:.

When the collector is slow or down

Spans wait in a queue of 2048 and leave in batches of up to 512, or every 5 seconds. A delivery never waits for the collector. A span that does not fit in the queue, a batch the collector does not take, or spans it reports as rejected are lost and counted. One warn line reports the loss:

trace export: spans dropped dropped=<since the last line> dropped_total=<since start>

at most once a minute while spans are being lost, and once more at shutdown. The same counts are on /metrics as nexus_trace_spans_dropped_total with reason queue_full, export_failed or rejected (see Metrics). At a shutdown the engine sends what the queue holds within one export timeout and counts the rest as queue_full. Spans held by a process that is killed are lost and not counted.