Skip to content

Benchmarks

This page shows what Tram's pipelines and components cost, to help with sizing deployments and choosing configurations. The tables are generated by tools/benchreport from the benchmarks in the repository; see Regenerating.

Absolute numbers depend on the machine, its load and the network to NATS, so treat them as an indication for the environment below. Costs per record and the ratios between configurations carry over much better than throughput figures.

How to Read the Numbers

  • Record: a log record, metric data point or span. Event: one OTLP request or one NATS message, carrying one or more records.
  • CPU per record: CPU time (user and system) of the whole process, divided by the records processed. It includes garbage collection. For the OTLP scenarios it also includes the HTTP client that generates the load in the same process, so it is an upper bound. NATS runs in a container, and its CPU time is not included.
  • Small events overstate CPU. With 1 record per event, the process mostly waits for HTTP and NATS round trips, and the Go runtime uses CPU between those short requests as well. These rows therefore show far more CPU per record than the work itself costs (compare the per-event costs under Where the Time Goes). Use the rows with 100 or 1,000 records for sizing; treat single-record rows as a pessimistic bound
  • Records/s per core: how many records one fully used CPU core can process, derived from the CPU per record. This is the figure for sizing, as Tram is stateless and scales horizontally.
  • Records/s: the throughput actually measured on this machine. It is also limited by the number of callers and by NATS.
  • Latency: time from sending an OTLP request until Tram answered it. Tram answers once all sinks accepted the data, so it includes publishing to NATS.
  • Spread: range of the runs relative to the median. Values above 10% are flagged with ⚠️ and are less reliable.

Environment
Commit c4c1bab (with uncommitted changes)
Date 2026-10-15
CPU 13th Gen Intel(R) Core(TM) i7-13700K
Go go1.27.2
NATS nats:2.11 in a local container
Runs 3 per benchmark, medians shown; scenarios -benchtime=1s, streams -benchtime=1x, components -benchtime=300ms

Pipeline Scenarios

Complete streams, built by the core from the real components, as tram server does. OTLP data is sent over HTTP; NATS runs in a local container. Each scenario runs with 1, 4 and 8 concurrent callers (for the NATS source: workers) and with events of 1, 100 and 1,000 records.

Configuration used in all scenarios:

  • NATS sink: one subject per service ('<stream>.' + ...get_attr('service.name')), so events with 10 services are published as 10 messages
  • batch: max_records: 1000, timeout: "10ms"
  • Split: two cel_filter stages, team == 'team-0' and its negation, each with its own NATS sink; the payload alternates between two teams
  • NATS source: durable consumer, workers equal to the number of callers; the messages are in the stream before the measurement starts

otlp-nats: OTLP → NATS

Callers Records per event Records/s Events/s CPU per record Records/s per core p50 latency p99 latency Allocs per record Spread
1 1 (1 service) 12 k 12 k 87.10 µs 11 k 0.08 ms 0.20 ms 205 ±4%
1 100 (1 service) 188 k 1.9 k 7.67 µs 130 k 0.45 ms 0.97 ms 54 ±3%
1 1000 (10 services) 247 k 247 6.09 µs 164 k 3.57 ms 5.87 ms 53 ±3%
4 1 (1 service) 35 k 35 k 87.08 µs 11 k 0.11 ms 0.31 ms 205 ±3%
4 100 (1 service) 602 k 6.0 k 7.87 µs 127 k 0.65 ms 1.13 ms 54 ±2%
4 1000 (10 services) 859 k 859 6.04 µs 166 k 4.37 ms 6.53 ms 53 ±0%
8 1 (1 service) 52 k 52 k 97.71 µs 10 k 0.14 ms 0.38 ms 205 ±1%
8 100 (1 service) 909 k 9.1 k 8.46 µs 118 k 0.89 ms 1.38 ms 54 ±1%
8 1000 (10 services) 1.48 M 1.5 k 6.19 µs 162 k 5.21 ms 7.40 ms 53 ±0%

otlp-batch-nats: OTLP → batch → NATS

Callers Records per event Records/s Events/s CPU per record Records/s per core p50 latency p99 latency Allocs per record Spread
1 1 (1 service) 96 96 318.91 µs 3.1 k 10.41 ms 10.88 ms 220 ±1%
1 100 (1 service) 9.0 k 90 11.56 µs 87 k 11.03 ms 11.73 ms 54 ±1%
1 1000 (10 services) 241 k 240 6.26 µs 160 k 3.74 ms 6.00 ms 53 ±3%
4 1 (1 service) 377 377 192.67 µs 5.2 k 10.53 ms 10.87 ms 188 ±0%
4 100 (1 service) 34 k 341 9.52 µs 105 k 11.56 ms 12.67 ms 54 ±0%
4 1000 (10 services) 859 k 859 6.05 µs 165 k 4.38 ms 6.46 ms 53 ±2%
8 1 (1 service) 754 754 173.22 µs 5.8 k 10.57 ms 11.00 ms 183 ±1%
8 100 (1 service) 62 k 621 9.43 µs 106 k 12.70 ms 14.11 ms 54 ±1%
8 1000 (10 services) 1.47 M 1.5 k 6.25 µs 160 k 5.25 ms 7.42 ms 53 ±1%

otlp-split-nats: OTLP → 2 × cel_filter → 2 × NATS

Callers Records per event Records/s Events/s CPU per record Records/s per core p50 latency p99 latency Allocs per record Spread
1 1 (1 service) 8.8 k 8.8 k 125.17 µs 8.0 k 0.11 ms 0.23 ms 250 ±3%
1 100 (1 service) 141 k 1.4 k 13.50 µs 74 k 0.70 ms 1.15 ms 88 ±1%
1 1000 (10 services) 173 k 173 11.75 µs 85 k 6.12 ms 6.95 ms 86 ±1%
4 1 (1 service) 28 k 28 k 115.64 µs 8.6 k 0.13 ms 0.35 ms 251 ±6%
4 100 (1 service) 442 k 4.4 k 14.62 µs 68 k 0.93 ms 1.38 ms 88 ±1%
4 1000 (10 services) 644 k 644 11.28 µs 89 k 6.17 ms 7.83 ms 86 ±0%
8 1 (1 service) 44 k 44 k 126.18 µs 7.9 k 0.16 ms 0.46 ms 251 ±2%
8 100 (1 service) 649 k 6.5 k 16.26 µs 62 k 1.25 ms 1.84 ms 88 ±0%
8 1000 (10 services) 1.03 M 1.0 k 12.67 µs 79 k 7.76 ms 9.61 ms 86 ±1%

nats-filter-nats: NATS → cel_filter → NATS

Callers Records per event Records/s Events/s CPU per record Records/s per core p50 latency p99 latency Allocs per record Spread
1 1 (1 service) 7.6 k 7.6 k 107.37 µs 9.3 k – – 158 ±4%
1 100 (1 service) 164 k 1.6 k 7.93 µs 126 k – – 61 ±4%
1 1000 (10 services) 194 k 194 7.33 µs 136 k – – 61 ±2%
4 1 (1 service) 28 k 28 k 78.05 µs 13 k – – 154 ±4%
4 100 (1 service) 600 k 6.0 k 7.83 µs 128 k – – 61 ±3%
4 1000 (10 services) 757 k 757 6.71 µs 149 k – – 61 ±1%
8 1 (1 service) 45 k 45 k 70.94 µs 14 k – – 153 ±2%
8 100 (1 service) 966 k 9.7 k 8.40 µs 119 k – – 61 ±1%
8 1000 (10 services) 1.34 M 1.3 k 6.86 µs 146 k – – 61 ±1%

Sizing

CPU needed for pure processing, from the CPU per record of the scenarios above. The unbatched row is a pessimistic bound, see How to Read the Numbers; the point it makes holds regardless: unbatched single-record events cost many times more than batched ones. Plan headroom on top, for load peaks, garbage collection and the network. Allocation rates are high (see Allocs per record), so pods also need memory headroom; set GOMEMLIMIT and resource requests accordingly.

Pipeline CPU per record Cores for 100 k records/s Cores for 1 M records/s
Batched: OTLP → batch → NATS, 1,000 records per request, 8 callers 6.25 µs 0.6 6.3
Split: OTLP → 2 × cel_filter → 2 × NATS, 1,000 records per request, 8 callers 12.67 µs 1.3 12.7
Forwarding: NATS → cel_filter → NATS, 1,000 records per message, 8 workers 6.86 µs 0.7 6.9
Unbatched: OTLP → NATS, 1 record per request, 8 callers 97.71 µs 9.8 97.7

Scaling out works because Tram keeps no state between events, with two conditions:

  • NATS source: set durable, so the pods share one consumer and split the messages. Without it, every pod receives every message
  • NATS server: all pods publish to the same JetStream streams. A stream is written by one NATS node, so at very high load NATS, not Tram, becomes the limit; spread the data over several streams

Streams per Instance

The cost of processing data is shared by the replicas, but the cost of a stream itself is not: every replica that runs a stream holds its connections, goroutines and memory, and its NATS sources keep asking the NATS server for messages even when there are none. This section measures that cost, so you can tell when to shard streams over several deployments with --watch-namespace or --stream-selector (see the operator documentation) instead of running every stream on every replica.

BenchmarkStreams builds 10, 100 and 1,000 streams of one shape in one Tram core and observes them for 10 seconds without data. Values are per stream, measured against the process before the streams were built:

  • Build: time to create the stream through the core, including connecting to NATS
  • Heap: live heap after a garbage collection. RSS is higher; see Sizing
  • NATS connections: client connections the NATS server sees. Every NATS source and sink has its own
  • NATS messages/s, idle: messages the NATS server receives from the idle stream, mostly the pull requests of the NATS source, which renews its request every 5 seconds
  • Idle CPU: CPU the process uses for the idle stream, as a share of one core

Idle CPU is shown for 1,000 streams only; for fewer streams it is below the noise of the process.

Stream Streams Build Heap Goroutines NATS connections NATS messages/s, idle Idle CPU of a core
NATS (1 worker) → cel_filter → NATS 10 1.6 ms 344 KiB 11.0 2.0 0.00 –
NATS (1 worker) → cel_filter → NATS 100 1.4 ms 337 KiB 11.0 2.0 0.00 –
NATS (1 worker) → cel_filter → NATS 1,000 1.4 ms 336 KiB 11.0 2.0 0.00 0.0026 %
NATS (4 workers) → cel_filter → NATS 10 1.4 ms 339 KiB 11.0 2.0 0.00 –
NATS (4 workers) → cel_filter → NATS 100 1.4 ms 336 KiB 11.0 2.0 0.00 –
NATS (4 workers) → cel_filter → NATS 1,000 1.3 ms 336 KiB 11.0 2.0 0.00 0.0026 %
OTLP → batch → NATS 10 933.10 µs 211 KiB 5.0 1.0 0.00 –
OTLP → batch → NATS 100 847.80 µs 208 KiB 5.0 1.0 0.00 –
OTLP → batch → NATS 1,000 903.60 µs 207 KiB 5.0 1.0 0.00 0.0014 %

What a number of streams costs when every replica runs all of them, projected from the 1,000-stream rows. Heap and CPU are per replica; connections and messages add up on the NATS server over all replicas:

Stream Streams Replicas Heap per replica Idle CPU per replica, of a core NATS connections NATS messages/s, idle
NATS (1 worker) → cel_filter → NATS 100 1 33 MiB 0.26 % 200 0
NATS (1 worker) → cel_filter → NATS 100 10 33 MiB 0.26 % 2.0 k 0
NATS (1 worker) → cel_filter → NATS 1,000 1 329 MiB 2.6 % 2.0 k 0
NATS (1 worker) → cel_filter → NATS 1,000 3 329 MiB 2.6 % 6.0 k 0
NATS (1 worker) → cel_filter → NATS 1,000 10 329 MiB 2.6 % 20 k 0
NATS (4 workers) → cel_filter → NATS 100 1 33 MiB 0.26 % 200 0
NATS (4 workers) → cel_filter → NATS 100 10 33 MiB 0.26 % 2.0 k 0
NATS (4 workers) → cel_filter → NATS 1,000 1 329 MiB 2.6 % 2.0 k 0
NATS (4 workers) → cel_filter → NATS 1,000 3 329 MiB 2.6 % 6.0 k 0
NATS (4 workers) → cel_filter → NATS 1,000 10 329 MiB 2.6 % 20 k 0
OTLP → batch → NATS 100 1 20 MiB 0.14 % 100 0
OTLP → batch → NATS 100 10 20 MiB 0.14 % 1.0 k 0
OTLP → batch → NATS 1,000 1 202 MiB 1.4 % 1.0 k 0
OTLP → batch → NATS 1,000 3 202 MiB 1.4 % 3.0 k 0
OTLP → batch → NATS 1,000 10 202 MiB 1.4 % 10 k 0

How to read it:

  • Memory limits the streams per replica first. Most of a stream's heap are the buffers of its NATS connections. Plan the memory request of a replica from the streams it runs, not only from its data rate
  • Replicas multiply the load on NATS. Doubling the replicas of a deployment that runs every stream doubles the connections and idle requests the NATS server handles, while each replica still holds all streams. Shard the streams when the per-replica heap or the NATS totals get too high, and scale replicas within a shard for the data rate
  • Workers cost only while messages are in flight. A NATS source has one pending pull request however many workers it has, and runs a goroutine per message being processed

Where the Time Goes

Costs of single components, measured in isolation. Per-record costs are measured with events of 1,000 records from 10 services.

Component Per record Allocs per record Records/s Spread
NATS sink: group and publish, 10 subjects 3.22 µs 31.2 310 k ±4%
NATS sink: group and publish, 1 subject 3.82 µs 30.1 262 k ±9%
NATS sink: grouping by subject only 2.48 µs 30.1 403 k ±13% ⚠️
Serializing for NATS and back (logs) 2.28 µs 34.2 438 k ±9%
NATS source, 1 worker 1.68 µs 34.2 594 k ±7%
NATS source, 8 workers 269.0 ns 34.2 3.72 M ±0%
OTLP source: decoding a request 1.65 µs 34.3 606 k ±16% ⚠️
cel_filter, keeping half of the records 1.90 µs 16.6 527 k ±14% ⚠️
cel_filter, keeping all records 869.1 ns 8.0 1.15 M ±3%
cel_validator, record-level expression 963.0 ns 9.1 1.04 M ±4%
cel_validator, resource-level expression 14.1 ns 0.1 70.97 M ±28% ⚠️
Record filter helper without CEL 9.2 ns 0.0 108.58 M ±10% ⚠️
batch, 8 concurrent callers 0.5 ns 0.0 2.21 G ±3%
noop stage 1.1 ns 0.0 931.97 M ±3%

Fixed costs per event, independent of its size. They dominate when events are small:

Component Per event Allocs per event Spread
NATS sink: one record (waits for the acknowledgement) 37.76 µs 67 ±2%
OTLP source: request with one record 6.67 µs 85 ±4%
batch, one caller 1.38 µs 15 ±7%
Passing an event to 1 child 374.3 ns 6 ±3%
Passing an event to 8 children 1.63 µs 20 ±2%
Each additional noop stage 455.0 ns 7 ±4%

Tuning

  • Batch before the NATS sink. The sink waits for JetStream to acknowledge the messages of an event. With small events that round trip dominates; batching spreads it over many records. Compare otlp-nats with 1 and with 1,000 records per event
  • Batching needs concurrent callers. Each caller of a batch stage waits for its batch to be delivered. With a single caller, batches never fill and every event waits for timeout; see otlp-batch-nats with 1 caller
  • Keep the number of subjects per event low. The NATS sink publishes one message per subject. It sends them all at once and waits for their acknowledgements together, so many subjects cost little time, but every message still costs CPU on Tram and the NATS server
  • Give a NATS source enough workers. workers is the number of messages processed at the same time, so a batch stage after the source holds at most that many messages. The source acknowledges a message once its batch is delivered, so a source with a batch stage handles at most workers / timeout messages per second: 20 workers and a timeout of 1 s cap it at 20 messages/s per replica, however fast the sink is
  • Prefer resource-level CEL expressions. They are evaluated once per resource instead of once per record when used in cel_validator; per record, CEL costs about as much as decoding the record
  • Many consumers filtering one stream make NATS the limit. Each filtered consumer searches the stream for its next matching message. With hundreds of them on one stream, NATS spends its CPU on that search and, on slow storage, on reading blocks from disk; see Load Test on Kubernetes
  • Prefer retention: interest for streams that only pass data on. NATS removes a message once every consumer acknowledged it, so the stream stays small while consumers keep up and still holds the backlog of one that falls behind, up to max_age

Load Test on Kubernetes

Unlike the tables above, this is a manual end-to-end test of one topology, not generated by tools/benchreport. It ran the tenant routing topology with production-shaped traffic on a managed Kubernetes cluster to see what one Tram pod handles and where a complete system hits its limit.
Date 2026-10-11
Cluster STACKIT SKE, Kubernetes 1.35, nodes with 16 vCPUs
Tram operator mode, 553 streams on every replica, 1 replica, CPU request 1 core without limit, GOMEMLIMIT=1600MiB
NATS 2.15, one server with 4 cores, JetStream on block storage premium-perf29 (60,000 IOPS)
Topology OTLP → batch → NATS stream telemetry; 500 links, one per tenant, each a consumer filtering telemetry and publishing to the stream router; 50 routers delivering to OpenObserve; 2 consumers delivering audit logs and metrics. All batch stages 200 ms; both streams retention: interest
Load gzip OTLP requests of ~27 KB (~115 KB decoded), 5 records and 3 of 500 tenants each; 55% logs, 30% traces, 15% metrics; sent through the ingress (Traefik, TLS) by a generator in the same cluster
Requests/s Tram CPU Tram RSS Client p95 NATS CPU NATS publish, mean
207 0.73 850 MiB – 0.57 36 ms
311 1.04 900 MiB – 0.89 63 ms
414 1.37 870 MiB 259 ms 1.0 65 ms
517 1.66 900 MiB 291 ms 1.3 96 ms
620 1.92 950 MiB 364 ms 1.63 140 ms
713 (800 sent) 2.2 1.5 GiB 2.5 s 2.05 1.7 s

What it shows:

  • Tram needs about 0.32 cores per 100 requests/s, and its memory hardly grows with load. About 450 MB of the heap are the 553 streams alone (see Streams per Instance); load adds a few hundred MB. Scale Tram on CPU
  • NATS sets the limit of the system, at about 650 to 700 requests/s. NATS used only 2 of its 4 cores: half of its CPU went to the 500 link consumers searching telemetry for their next message, 15% to writing consumer state on acknowledgements. Publishes compete with that, so their latency jumps from 140 ms to 1.7 s. More Tram replicas do not move this limit, as they share the same consumers
  • NATS needs fast storage when many consumers filter one stream. On block storage with 500 IOPS, the same setup collapsed at 100 requests/s: NATS kept re-reading blocks from disk for the filtered consumers, publishes took up to 7 s, and NATS grew to its 8 GB memory limit. With 60,000 IOPS, 100 requests/s took 47 ms per publish
  • retention: interest made NATS' work lighter. At 100 requests/s, the streams stayed at a few MB instead of 2.6 GB with limits and 5 minutes of retention; NATS publishes took 7 to 9 ms instead of 47 ms
  • The batch timeout limited the routers. With 20 workers and a timeout of 1 s, each router handled at most 20 messages/s per replica, and a backlog built up at 400 requests/s; with 200 ms it did not

Known Limits

  • NATS ran in a container on the same machine. Over a real network, every acknowledgement takes longer, which affects small events and many subjects most
  • The OTLP scenarios generate load from the same process and machine, which competes with Tram for CPU
  • The generated benchmarks measure one Tram instance; they do not cover several pods sharing a NATS server. The fleet table under Streams per Instance multiplies the per-stream costs instead, assuming the NATS server's cost grows linearly with connections and requests. Load Test on Kubernetes measured one replica against a shared NATS server; scaling out to several replicas under load is not measured yet
  • BenchmarkStreams measures idle streams. It does not measure how idle streams slow down a busy one, beyond the idle CPU they use

Regenerating

The scenario and NATS benchmarks need Docker or Podman; without them they are skipped and their rows show n/a. With Podman on macOS, point Testcontainers to the socket inside the Podman VM first:

export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/run/podman/podman.sock
export TESTCONTAINERS_RYUK_CONTAINER_PRIVILEGED=true

Then run all benchmarks and update this page (takes about 25 minutes, of which the streams benchmarks take about 9):

go run ./tools/benchreport -raw bench.txt

The benchmarks run one package at a time, as parallel packages would distort each other. Keep the machine otherwise idle. To change only the text around the tables, regenerate from the saved output without measuring again:

go run ./tools/benchreport -input bench.txt

Flags: -count (runs per benchmark, default 3), -scenario-benchtime (default 1s), -component-benchtime (default 300ms), -out (default BENCHMARK.md).