Streams¶
Pipeline Structure¶
Streams follow a tree structure with these rules:
- Each pipeline has exactly one source (root)
- Stages can be connected to the source or other stages
- Sinks must be connected to either the source or a stage
- All leaves must be sinks (no dangling stages)
flowchart LR
subgraph stream["Stream"]
source(["Source"]) --> stage["Stage"]
stage --> sink(["Sink"])
source --> sink
stage --> stage
end
Data Flow¶
Arrows represent the direction of data flow, from source to sinks.
Available Components¶
| Type | Components |
|---|---|
| Sources | broker, generator, otlp |
| Stages | batch, delay, filter, noop, validation, value_map |
| Sinks | broker, debug, otlp |
For detailed documentation on each component and its configuration, see COMPONENTS.md.
flowchart LR
otlp_src(["otlp source<br>OTLP/HTTP"]) --> tram(["TRAM"])
broker_src(["broker source"]) --> tram
generator(["generator source"]) --> tram
tram --> otlp_sink(["otlp sink"])
tram --> broker_sink(["broker sink"])
tram --> debug(["debug sink"])
Fan-out Pattern¶
Route data from a single source to multiple sinks. Every sink receives every event:
curl -X PUT http://localhost:8080/v1/streams/fanout-example \
-H "Content-Type: application/json" \
--data-binary @- <<'EOF'
{
"stream": {
"name": "fanout-example",
"source": {
"name": "otlp-in",
"type": "otlp",
"spec": { "http_port": 4318 }
},
"sinks": [
{
"name": "console",
"type": "debug",
"from": "otlp-in"
},
{
"name": "archive",
"type": "broker",
"from": "otlp-in",
"spec": {
"driver": "nats",
"nats": { "url": "nats://localhost:4222" },
"destination": {
"logs": "'logs.' + logs.resource_logs[0].resource.attributes.get_attr('service.name')"
}
}
}
]
}
}
EOF
The broker sink with driver: nats needs a NATS server with JetStream enabled, e.g. docker compose -f examples/nats/docker-compose.yaml up -d, and a stream for its subjects, which TRAM does not create: nats stream add telemetry --subjects 'logs.>' --defaults.
Chained Stages¶
Process data through multiple stages before reaching sinks:
curl -X PUT http://localhost:8080/v1/streams/chained-pipeline \
-H "Content-Type: application/json" \
-d '{
"stream": {
"name": "chained-pipeline",
"source": {
"name": "source",
"type": "generator",
"spec": { "interval": "1s" }
},
"stages": [
{
"name": "stage1",
"type": "noop",
"from": "source"
},
{
"name": "stage2",
"type": "noop",
"from": "stage1"
}
],
"sinks": [
{
"name": "output",
"type": "debug",
"from": "stage2"
}
]
}
}'
Example Setup¶
Two streams read a NATS topic and forward the data, one to an OTLP endpoint and one to the debug log; a third stream receives OTLP data and publishes it to NATS:
flowchart LR
nats(["NATS"]) --> src1["broker source<br>topic 1"]
nats --> src2["broker source<br>topic 2"]
otlp_in(["otlp source"])
src1 --> val1["validation"]
src2 --> val2["validation"]
otlp_in --> val3["validation"]
val1 --> sink1(["otlp sink"])
val2 --> sink2(["debug sink"])
val3 --> sink3(["broker sink"])
sink3 --> nats
subgraph tram["TRAM"]
src1
val1
sink1
src2
val2
sink2
otlp_in
val3
sink3
end
Updating a Stream¶
A PUT to an existing stream changes as little as possible:
| Change | Effect |
|---|---|
| None | Nothing happens; the running stream is kept |
Stages, sinks or metrics only |
The new stages and sinks are created, events still in flight finish in the old ones, and the source switches over. The source keeps running, e.g. an OTLP source keeps its port open, and no event is lost |
| Source | The stream is stopped and created again. An OTLP source closes its port in between, so clients briefly cannot connect |
If an update fails, e.g. because of an invalid component spec, the request returns an error and the previous version of the stream keeps running. Delete and create a stream to restart it completely.
Dry Run¶
PUT /v1/streams/{name} with "dry_run": true in the request body validates the stream and reports what applying it would do, without changing the running stream. The flag is a field of the request body, not a query parameter, because the PUT body maps to the whole request message. The response holds a plan:
{
"stream": { "...": "the requested stream" },
"plan": {
"action": "STREAM_PLAN_ACTION_HOT_SWAP",
"sourceReplaced": false,
"createdNodes": ["enrich"],
"stoppedNodes": ["old-sink"]
}
}
action is STREAM_PLAN_ACTION_NOOP (nothing would change), STREAM_PLAN_ACTION_HOT_SWAP (the source keeps running and its stages and sinks are replaced below it) or STREAM_PLAN_ACTION_REBUILD (the stream is new, stopped, or its source changed: it would be stopped and created again). An invalid spec is rejected with 400 as usual; nothing is applied either way.
If the source of a stream stops on its own, e.g. because the NATS stream of a broker source does not exist yet, TRAM restarts the stream after a delay that grows from 1 second to 1 minute while it keeps failing. A PUT of the unchanged stream restarts it right away.
Testing a Stream¶
A stream can carry test cases in tests. They are never run by loading the stream; only an explicit test run executes them:
The file holds either a bare stream object or a file sync file that wraps it in {"stream": {...}}, like the examples in examples/tests/; see examples/tests/otlp_with_filters.json for a stream with test cases.
The stream is loaded into its own in-memory tree: the source is replaced by a test source that emits the test inputs, and every sink by a capture sink, so no external system is contacted. The stages are the real stages and process the events.
Each test has a name, an input event and the expected outcome per sink in expect:
"out": <event>— sinkoutmust receive exactly this event. Events are OTLP JSON objects with one of the keyslogs,metricsortraces"out": "drop"— sinkoutmust receive nothing"out": "reject"— the event must be rejected by a stage- a sink without an entry in
expectmust receive nothing
"tests": [
{
"name": "keeps real logs",
"input": {"logs": {"resourceLogs": [{"scopeLogs": [{"logRecords": [{"body": {"stringValue": "hello"}}]}]}]}},
"expect": {"out": {"logs": {"resourceLogs": [{"scopeLogs": [{"logRecords": [{"body": {"stringValue": "hello"}}]}]}]}}}
},
{
"name": "drops noise",
"input": {"logs": {"resourceLogs": [{"scopeLogs": [{"logRecords": [{"body": {"stringValue": "noise"}}]}]}]}},
"expect": {"out": "drop"}
}
]
--secrets reads a JSON file mapping "<name>/<key>" to values and resolves secretKeyRef references in stage specs. A reference that cannot be resolved is an error naming the reference, never its value. Source and sink specs are not used by a test run. The exit code is 0 when all tests pass and 1 otherwise.