The Telemetry Pipeline

You have the signals (taxonomy), the sources (protocols), and the join keys (correlation). This page is the plumbing that carries all of it — from a counter on a switch ASIC to a panel on an on-call engineer's screen — without falling over at 10k-GPU scale.

The shape is always the same five stages. The engineering is entirely in the two constraints that decide whether it survives: cardinality and retention.

After this page, you'll be able to

1. Draw the five-stage pipeline — collect → transport → store → visualize → alert — and place each exporter.
2. Choose the collectors per layer (gNMI, sFlow, WJH, node_exporter, dcgm-exporter, NCCL profiler).
3. Do the cardinality math and apply the aggregate-at-the-exporter rule so RoCE per-QP metrics don't melt the TSDB.
4. Design tiered retention — high-resolution recent, downsampled long-term — and use recording rules for expensive rollups.
5. Know where OpenTelemetry fits and why you must monitor the pipeline itself.

---

The five-stage pipeline

Prefer streaming telemetry (gNMI subscribe in STREAM/SAMPLE mode) over SNMP polling for the fabric — sub-second congestion events are invisible at 30 s SNMP intervals.

---

Collectors & exporters

Layer	Collector / exporter	Protocol	Emits
Switch	gNMI (OpenConfig streaming)	gNMI/gRPC	queue stats, PFC/ECN, watermarks, interface counters
Switch	sFlow + WJH	sFlow / event stream	sampled flows + event-based drop reasons
Host NIC	node_exporter (+ RDMA/perfquery collector)	HTTP :9100	hw_counters, per-priority pause/ECN, port errors
GPU	dcgm-exporter	HTTP :9400	all DCGM_FI_* fields, labelled by GPU UUID
Job	NCCL profiler plugin	push / OTel	per-collective timing, busbw

The collector tier for switches is usually gNMIc or Telegraf subscribing to the streams and re-emitting to the TSDB; large operators fan through Kafka first so multiple consumers (TSDB, anomaly detection, long-term archive) can each read the firehose independently.

---

Storage: Prometheus and where it stops scaling

Prometheus is the default and the right starting point: pull-based scraping, PromQL, a huge exporter ecosystem. A single Prometheus handles a rack or a pod comfortably. At fabric scale you outgrow one instance and move to a horizontally-scalable, long-retention backend that still speaks PromQL:

Backend	Model	Why you'd pick it
Prometheus	single-node scrape + local TSDB	start here; per-pod / per-rack
VictoriaMetrics	clustered, high-cardinality tolerant	when cardinality is the pain point
Thanos / Mimir	Prometheus + object storage, global query	long retention + a single global view

The migration trigger is almost always cardinality, not query load — which is the constraint below.

Language Bridge

TSDB (time-series database)→The store built for metrics — millions of (timestamp, value) points keyed by a metric name plus labels. Prometheus is one; VictoriaMetrics/Mimir/Thanos scale the same idea out. The label set you choose is what makes or breaks it.

---

Cardinality & retention — the real design constraint

This is where naïve observability designs die. RoCE metrics are per-queue-pair, and a large fabric has thousands of QPs per NIC × hundreds of NICs — scrape all of that per-QP into Prometheus and you'll melt the TSDB.

The rules that keep it alive:

• Aggregate at the exporter. Emit per-port and per-priority by default; expose per-QP only on demand (rdma statistic show qp during an incident), not as a standing scrape.
• Recording rules for the expensive rollups (per-rack ECN rate, per-leaf drop rate) so dashboards query pre-computed series instead of re-aggregating raw ones on every refresh.
• Tiered retention: high-resolution (1–5 s) for a short window, downsampled for long-term trend and baselining.
• Label discipline: the join-key labels are worth their cardinality; free-form or per-flow labels are not. Every label multiplies series count — spend them deliberately.

Language Bridge

Recording rule→A Prometheus rule that pre-computes an expensive query on a schedule and stores the result as a new series. Dashboards and alerts read the cheap pre-computed series instead of re-aggregating millions of raw points every refresh.

A back-of-envelope

Take 1,000 NICs. Emit 20 useful counters each, sliced per-port × 8 priorities — that's 1000 × 20 × 8 = 160,000 series. Bounded and totally fine. Now add a naïve per-QP label at 2,000 QPs/NIC: 1000 × 20 × 2000 = 40,000,000 series — a 250× explosion that no single TSDB survives. Same counters, one label, two wildly different outcomes. That is the entire cardinality lesson in one line: bound the dimension or sample it; never stream the unbounded one.

---

Tiered retention

Not every resolution is worth keeping forever:

Tier	Resolution	Retention	Purpose
Hot	1–5 s	hours–days	incident forensics, the walk-down
Warm	30 s–1 min	weeks	dashboards, recent trend
Cold	5 min (downsampled)	months–years	baselining, seasonality, capacity

Baselining and seasonal alerting (next page) depend on the cold tier — you can't alert on "10× the 7-day median" if you only kept 48 hours.

---

Where OpenTelemetry fits

OpenTelemetry (OTel) is the converging standard for application-emitted telemetry — the NCCL profiler and framework instrumentation naturally speak it (metrics, traces, and the collective "spans" that are your best approximation of tracing). The practical pattern:

• Fabric/host counters → stay on the Prometheus/gNMI path (that ecosystem is mature and counter-shaped).
• Job/collective signals → OTel SDK → OTel Collector, which can fan out to the same TSDB and to a trace backend.
• The OTel Collector is also a convenient single point to attach the join-key labels consistently before data lands in storage.

Don't rip out Prometheus for OTel; let OTel own the application/job pillar and Prometheus own the infrastructure counters, unified at the storage/query layer.

---

Monitor the pipeline itself

If the pipeline is down, you're blind — and worse, you may not know it. The observability stack is Tier-0 infrastructure and must watch itself:

• up / scrape success for every exporter — a silent dead dcgm-exporter means that node is invisible, not healthy.
• Ingestion rate and active series count — your early warning that a cardinality bomb just got deployed.
• Retention headroom / disk — the classic 3 AM "why is there no data from last night" is a full TSDB.
• Alert-path liveness — a synthetic heartbeat alert that must fire; if it stops, Alertmanager/PagerDuty is broken.

Language Bridge

Watching the watcher→Instrumenting the monitoring system with the monitoring system. A dead exporter looks exactly like a healthy-but-quiet node — so you alert on scrape failures and a heartbeat, or you'll trust a dashboard that's silently stale.

---

💡 What you should remember

#		Concept	Why it matters
1	🔟	Five stages	collect → transport → store → visualize → alert — every design is this shape.
2	🧮	Aggregate at the exporter	Per-port × per-priority is bounded; per-QP is a 250× cardinality bomb.
3	🗜️	Recording rules + tiered retention	Pre-compute rollups; keep hot data fine-grained and cold data downsampled.
4	📶	Stream, don't poll, the fabric	30 s SNMP averages away the microsecond congestion events.
5	🔭	OTel owns the job pillar	Let OTel carry collective telemetry; keep Prometheus for infra counters.
6	👁️	Watch the watcher	Alert on scrape failures, series growth, and a heartbeat — a dead exporter looks like a quiet node.

---

Next: SLIs, SLOs & Alerting → — turning these signals into a pager that fires when the job is actually hurting, not when a counter is merely non-zero.