Metrics

GET /metrics on the HTTP API serves Tram's metrics in the Prometheus format, for Prometheus or any other scraper. By default the source and the sinks of every stream are measured, whatever their type, and the stages are not; see Choosing the metrics of a stream:

Metric	Type	Labels	Description
tram_node_records_total	Counter	stream, node, kind, signal	Records the node received: log records, metric data points and spans. kind is source, stage or sink, signal is logs, metrics or traces
tram_node_records_out_total	Counter	stream, node, kind, signal	Records a source or stage passed on: log records, metric data points and spans. With several children, each record counts once, not once per child. Sinks pass nothing on
tram_node_duration_seconds	Histogram	stream, node, kind, outcome	Time the node took to handle an event. outcome is ok, rejected (the event was refused for good, e.g. by a cel_validator), cancelled (the caller gave up, e.g. an OTLP client that closed its connection) or error. An expired delivery_timeout counts as error, since then Tram took too long
tram_source_request_size_bytes	Histogram	stream, node, kind, encoding, body	Size of the requests a source decoded events from, such as the OTLP source. body is received (as sent, e.g. gzip-compressed) or decoded (after undoing the content encoding), encoding is gzip or identity. Sources that consume messages, like NATS, report nothing

The endpoint also serves the Go runtime (go_*) and process (process_*) metrics.

For a source, the records are what it accepted and passed on, each counted once however many branches the stream has, and the outcome is the result its client gets: an event a single branch fails fails for the client. Events a source refuses before passing them on, such as an OTLP request that cannot be decoded, are not counted.

Events pass through a stream synchronously, so the duration of a source or stage includes the stages and sinks below it, and for a batch stage the time an event waits for its batch to be flushed. The duration of a source is the time its client waits for Tram. Only the duration of a sink is its own.

So the time a stream's stages take, including the wait of a batch stage, is the duration of its source minus that of its sinks, without measuring the stages themselves.

A stage's input minus its output is what it dropped, so the share each stage drops, for streams that measure the records of their stages, is:

1 - sum by (stream, node) (rate(tram_node_records_out_total{kind="stage"}[5m]))
  / sum by (stream, node) (rate(tram_node_records_total{kind="stage"}[5m]))

The average size of the requests a source received, compressed as sent and decoded, and the share of requests above 1 MB, half of a common 2 MB ingress limit:

sum by (stream, body) (rate(tram_source_request_size_bytes_sum[5m]))
  / sum by (stream, body) (rate(tram_source_request_size_bytes_count[5m]))

1 - sum by (stream) (rate(tram_source_request_size_bytes_bucket{body="received",le="1.048576e+06"}[5m]))
  / sum by (stream) (rate(tram_source_request_size_bytes_count{body="received"}[5m]))

A stage that drops every record of a signal, such as a cel_filter with "false" for it, has no output series for that signal, so drop the signal label or use or vector(0) when comparing per signal.

Choosing the metrics of a stream

Every node is a series of its own per metric, so many streams with several stages each add up to many series. The optional metrics field of a stream turns each metric on or off per kind of node; fields left out keep their defaults:

Field	Metrics	Defaults
records	tram_node_records_total and tram_node_records_out_total	source: true, stages: false, sinks: true
duration	tram_node_duration_seconds	source: true, stages: false, sinks: true
request_size	tram_source_request_size_bytes	true

To see what each stage of a stream drops and how long it takes:

"metrics": {
  "records": {"stages": true},
  "duration": {"stages": true}
}

The field is part of the stream in the API, in the files of the file sync and in the Stream resource of the Kubernetes operator (spec.metrics). Changing it rewires the stream's stages and sinks but keeps its source running. Series of a node that stops being measured stay at their last value until Tram restarts.