Skip to content

Streams

Pipeline Structure

Streams follow a tree structure with these rules:

  1. Each pipeline has exactly one source (root)
  2. Stages can be connected to the source or other stages
  3. Sinks must be connected to either the source or a stage
  4. All leaves must be sinks (no dangling stages)
flowchart LR
    subgraph stream["Stream"]
        source(["Source"]) --> stage["Stage"]
        stage --> sink(["Sink"])
        source --> sink
        stage --> stage
    end

Data Flow

Source → [Stage] → Sink

Arrows represent the direction of data flow, from source to sinks.

Available Components

Type Components
Sources broker, generator, otlp
Stages batch, delay, filter, noop, validation, value_map
Sinks broker, debug, otlp

For detailed documentation on each component and its configuration, see COMPONENTS.md.

flowchart LR
    otlp_src(["otlp source<br>OTLP/HTTP"]) --> tram(["TRAM"])
    broker_src(["broker source"]) --> tram
    generator(["generator source"]) --> tram
    tram --> otlp_sink(["otlp sink"])
    tram --> broker_sink(["broker sink"])
    tram --> debug(["debug sink"])

Fan-out Pattern

Route data from a single source to multiple sinks. Every sink receives every event:

curl -X PUT http://localhost:8080/v1/streams/fanout-example \
  -H "Content-Type: application/json" \
  --data-binary @- <<'EOF'
{
  "stream": {
    "name": "fanout-example",
    "source": {
      "name": "otlp-in",
      "type": "otlp",
      "spec": { "http_port": 4318 }
    },
    "sinks": [
      {
        "name": "console",
        "type": "debug",
        "from": "otlp-in"
      },
      {
        "name": "archive",
        "type": "broker",
        "from": "otlp-in",
        "spec": {
          "driver": "nats",
          "nats": { "url": "nats://localhost:4222" },
          "destination": {
            "logs": "'logs.' + logs.resource_logs[0].resource.attributes.get_attr('service.name')"
          }
        }
      }
    ]
  }
}
EOF

The broker sink with driver: nats needs a NATS server with JetStream enabled, e.g. docker compose -f examples/nats/docker-compose.yaml up -d, and a stream for its subjects, which TRAM does not create: nats stream add telemetry --subjects 'logs.>' --defaults.

Chained Stages

Process data through multiple stages before reaching sinks:

curl -X PUT http://localhost:8080/v1/streams/chained-pipeline \
  -H "Content-Type: application/json" \
  -d '{
    "stream": {
      "name": "chained-pipeline",
      "source": {
        "name": "source",
        "type": "generator",
        "spec": { "interval": "1s" }
      },
      "stages": [
        {
          "name": "stage1",
          "type": "noop",
          "from": "source"
        },
        {
          "name": "stage2",
          "type": "noop",
          "from": "stage1"
        }
      ],
      "sinks": [
        {
          "name": "output",
          "type": "debug",
          "from": "stage2"
        }
      ]
    }
  }'

Example Setup

Two streams read a NATS topic and forward the data, one to an OTLP endpoint and one to the debug log; a third stream receives OTLP data and publishes it to NATS:

flowchart LR
    nats(["NATS"]) --> src1["broker source<br>topic 1"]
    nats --> src2["broker source<br>topic 2"]
    otlp_in(["otlp source"])

    src1 --> val1["validation"]
    src2 --> val2["validation"]
    otlp_in --> val3["validation"]

    val1 --> sink1(["otlp sink"])
    val2 --> sink2(["debug sink"])
    val3 --> sink3(["broker sink"])
    sink3 --> nats

    subgraph tram["TRAM"]
        src1
        val1
        sink1
        src2
        val2
        sink2
        otlp_in
        val3
        sink3
    end

Updating a Stream

A PUT to an existing stream changes as little as possible:

Change Effect
None Nothing happens; the running stream is kept
Stages, sinks or metrics only The new stages and sinks are created, events still in flight finish in the old ones, and the source switches over. The source keeps running, e.g. an OTLP source keeps its port open, and no event is lost
Source The stream is stopped and created again. An OTLP source closes its port in between, so clients briefly cannot connect

If an update fails, e.g. because of an invalid component spec, the request returns an error and the previous version of the stream keeps running. Delete and create a stream to restart it completely.

Dry Run

PUT /v1/streams/{name} with "dry_run": true in the request body validates the stream and reports what applying it would do, without changing the running stream. The flag is a field of the request body, not a query parameter, because the PUT body maps to the whole request message. The response holds a plan:

{
  "stream": { "...": "the requested stream" },
  "plan": {
    "action": "STREAM_PLAN_ACTION_HOT_SWAP",
    "sourceReplaced": false,
    "createdNodes": ["enrich"],
    "stoppedNodes": ["old-sink"]
  }
}

action is STREAM_PLAN_ACTION_NOOP (nothing would change), STREAM_PLAN_ACTION_HOT_SWAP (the source keeps running and its stages and sinks are replaced below it) or STREAM_PLAN_ACTION_REBUILD (the stream is new, stopped, or its source changed: it would be stopped and created again). An invalid spec is rejected with 400 as usual; nothing is applied either way.

If the source of a stream stops on its own, e.g. because the NATS stream of a broker source does not exist yet, TRAM restarts the stream after a delay that grows from 1 second to 1 minute while it keeps failing. A PUT of the unchanged stream restarts it right away.

Testing a Stream

A stream can carry test cases in tests. They are never run by loading the stream; only an explicit test run executes them:

tram test stream.json [--secrets secrets.json]

The file holds either a bare stream object or a file sync file that wraps it in {"stream": {...}}, like the examples in examples/tests/; see examples/tests/otlp_with_filters.json for a stream with test cases.

The stream is loaded into its own in-memory tree: the source is replaced by a test source that emits the test inputs, and every sink by a capture sink, so no external system is contacted. The stages are the real stages and process the events.

Each test has a name, an input event and the expected outcome per sink in expect:

  • "out": <event> — sink out must receive exactly this event. Events are OTLP JSON objects with one of the keys logs, metrics or traces
  • "out": "drop" — sink out must receive nothing
  • "out": "reject" — the event must be rejected by a stage
  • a sink without an entry in expect must receive nothing
"tests": [
  {
    "name": "keeps real logs",
    "input":  {"logs": {"resourceLogs": [{"scopeLogs": [{"logRecords": [{"body": {"stringValue": "hello"}}]}]}]}},
    "expect": {"out": {"logs": {"resourceLogs": [{"scopeLogs": [{"logRecords": [{"body": {"stringValue": "hello"}}]}]}]}}}
  },
  {
    "name": "drops noise",
    "input":  {"logs": {"resourceLogs": [{"scopeLogs": [{"logRecords": [{"body": {"stringValue": "noise"}}]}]}]}},
    "expect": {"out": "drop"}
  }
]

--secrets reads a JSON file mapping "<name>/<key>" to values and resolves secretKeyRef references in stage specs. A reference that cannot be resolved is an error naming the reference, never its value. Source and sink specs are not used by a test run. The exit code is 0 when all tests pass and 1 otherwise.