Components

This document provides detailed documentation for all implemented sources, stages, and sinks in Tram.

Available Sources

Name	Type	Description
generator	generator	Emits empty events at a fixed interval, for testing pipelines
nats	nats	Consumes events published by Tram's nats sink from NATS JetStream
otlp	otlp	Receives protobuf-encoded OpenTelemetry Protocol (OTLP) data via HTTP

Available Stages

Name	Type	Description
batch	batch	Merges concurrently arriving events into larger batches
cel_filter	cel_filter	Keeps only the records that match a Common Expression Language (CEL) expression
cel_validator	cel_validator	Accepts, drops or rejects whole events using Common Expression Language (CEL)
delay	delay	Holds every event back for a fixed duration
noop	noop	Pass-through stage that does no processing (useful for testing)

Available Sinks

Name	Type	Description
debug	debug	Writes events to the log (stderr) for debugging and development
nats	nats	Publishes OTLP-encoded events to NATS JetStream with dynamic subject mapping
otlp	otlp	Sends OTLP-encoded events via HTTP to an OpenTelemetry endpoint

Notes

• Component Type vs Name: The type field in configuration refers to the component implementation type (e.g., generator, cel_validator), while the name field is a user-defined identifier for the component instance within a stream.
• Connections: The from field in stages and sinks specifies which component they receive data from.
• Tree Structure: Pipelines form a tree with the source as root, stages as intermediate nodes, and sinks as leaves.
• Delivery Guarantee: Delivery is at-least-once, and Tram does not deduplicate. A sink can receive the same data again, e.g. when a client retries after a 503 although some sinks had already accepted the data, when a NATS message is redelivered after one branch of its stream failed, or when a publish succeeded but its acknowledgement was lost. Deduplicate at the destination where duplicates matter.

Hold Times

A source like OTLP waits for a delivery only for a limited time, its delivery_timeout, and then answers with 503. batch stages can hold an event back for up to their timeout, delay stages for their delay. When a stream is created or updated, Tram adds up these hold times on the way from the source to each sink and rejects the stream if the sum reaches the source's limit: every event that does not fill its batch would be cancelled, so the stream would fail as soon as traffic gets low. The error names the stages:

stream spec is invalid: on the way to sink openobserve-traces, batch-traces (10s) can hold an event for 10s, but source otlp cancels deliveries after 8s

An update that is rejected leaves the running stream unchanged. Shorten the timeout or delay, or raise the source's delivery_timeout, and keep that below the time the clients wait. Sources without such a limit, like NATS, are not checked.