Telemetry Sources & Protocols

The previous page said what to monitor. This one is how you get it off the box — the protocols on the switch side, the counter interfaces on the host side, and the fundamental choice that shapes your whole pipeline: push vs. pull, streamed vs. sampled.

Get the sources right and the pipeline (next page) is plumbing. Get them wrong — poll at the wrong cadence, sample the wrong thing — and no amount of Grafana saves you, because the microburst that caused the stall was averaged out of existence before it ever left the switch.

After this page, you'll be able to

1. Trace the switch-telemetry shift — from legacy SNMP polling to gNMI/gRPC streaming (OpenConfig), with sFlow for flows and in-band telemetry for per-hop truth — and know why streaming is the default.
2. Name the host/NIC sources — ethtool -S, the RDMA hardware counters, and vendor exporters — and read the key counters.
3. Pull GPU and collective telemetry — DCGM/dcgm-exporter, and the NCCL debug/profiler path — and know why you need both.
4. Choose push vs. pull and streaming vs. sampling deliberately, and put the collectors on an out-of-band management network.

---

The switch side: the move to streaming telemetry

Switch telemetry has moved decisively from polling to streaming. The old world polled SNMP MIBs every few minutes; the modern fabric streams counters over gNMI, samples flows with sFlow, and reaches for in-band telemetry on the hard latency questions. SNMP is the legacy predecessor — you support it only for gear too old to do better, and you never build congestion visibility on it.

SNMP — the legacy predecessor, not a plan

Pull-based polling of MIB counters. Universally supported, and structurally wrong for AI fabrics: poll intervals of 30 s to 5 min average away the microsecond-scale events that matter, and polling thousands of ports doesn't scale. Treat it as legacy: keep it only for devices that speak nothing better, and never rely on it for congestion signals. The default for everything that matters is streaming telemetry.

sFlow / IPFIX — sampled flow visibility

Samples 1-in-N packets and ships headers to a collector. Cheap, scalable, and the right tool for the flow questions: which 5-tuples exist, which ECMP path they took, who the top talkers are. It's sampled, so it's statistical — great for "is traffic balanced across ECMP members," useless for "did this specific packet get dropped." Pairs with streaming counters rather than replacing them.

gNMI / gRPC streaming — the modern standard

The one to build on. The switch pushes counters to a collector over gRPC as they change (or on a sub-second cadence), modeled with OpenConfig YANG so the data schema is vendor-neutral. This is model-driven streaming telemetry: subscribe once to the paths you want, receive a stream.

Language Bridge

gNMI + OpenConfig→gNMI is the transport (gRPC push instead of SNMP poll). OpenConfig is the vendor-neutral data model (same path name for PFC counters whether it's Arista, Nvidia, or SONiC). Together: subscribe once, get a stream, don't relearn per vendor.

# Subscribe to PFC pause + queue counters at 1s, streamed (gnmic)
gnmic -a leaf-14:57400 -u admin -p '***' --skip-verify \
  subscribe --mode stream --stream-mode sample --sample-interval 1s \
  --path "/interfaces/interface[name=et-0/0/9]/openconfig-if-ethernet:ethernet/state/counters" \
  --path "/qos/interfaces/interface/output/queues/queue/state"

Every modern platform exposes it: Arista via EOS/CloudVision, NVIDIA Spectrum via the NOS + DOCA, Broadcom Tomahawk/Trident white boxes via SONiC (built in, no vendor agent), Cisco/Juniper via their model-driven telemetry stacks. Cadence sub-second where it matters (PFC/ECN/queues), slower for stable inventory.

In-band network telemetry — per-packet, per-hop truth

The frontier: the data packets themselves carry telemetry, so you learn the exact per-hop latency and queue depth each packet experienced. This is the only way to answer "which hop added the delay" definitively. It's powerful and costly; it gets its own treatment in advanced fabric visibility.

Protocol	Model	Cadence	Best for
SNMP	Pull	30 s–5 min	Legacy inventory only
sFlow / IPFIX	Push (sampled)	Continuous sample	Flow mix, ECMP balance, top talkers
gNMI / OpenConfig	Push (stream)	Sub-second	Counters, PFC/ECN, queues — your backbone
INT / postcard	In-band	Per-packet	Per-hop latency, drop attribution (on demand)

---

The host & NIC side

The switch tells you about the fabric; the host tells you about its edge — and the edge sees losslessness fail first.

RDMA NIC counters — vendor-neutral gold

The single richest source. Read them with ethtool -S (driver counters) and the RDMA HCA counters exposed under /sys/class/infiniband/*/ports/*/counters and via rdma tooling. Crucially, the standard RDMA counter set (the OFED / rdma-core counters) reads the same whether the NIC is NVIDIA/Mellanox, Broadcom, or Intel — build one exporter, run it everywhere.

# The counters that actually predict a stall
ethtool -S enp1s0f0 | grep -E 'prio3_pause|out_of_buffer|out_of_sequence|cnp|discard'

# Standard RDMA HW counters (hardware, vendor-neutral)
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/{out_of_sequence,packet_seq_err,np_cnp_sent,rp_cnp_handled}

# RDMA link + QP state
rdma link show ; rdma statistic show link mlx5_0/1

Export these to Prometheus with node_exporter (textfile collector) or a vendor exporter. Aggregate per-NIC — never emit per-QP as steady metrics (that's the cardinality trap from the last page). NVIDIA additionally exposes deeper detail via DOCA Telemetry Service; the standard counters are enough for the golden signals.

Optics diagnostics

Transceiver DDM/DOM (Rx/Tx power, temperature) and the FEC counters (pre-/post-FEC BER) come off the same interfaces — ethtool -m for module diagnostics on the host side, and the switch's optical/FEC telemetry via gNMI. Pre-FEC BER is the highest-value early-warning signal you'll collect, so make sure your gNMI subscription includes the FEC paths.

GPU telemetry — DCGM

GPUs report through DCGM (Data Center GPU Manager); dcgm-exporter turns it into Prometheus metrics. This is where SM utilization, HBM bandwidth, ECC, thermals, PCIe replay, and — critically — Xid errors come from. AMD's equivalent is rocm-smi / the ROCm exporter. You need this layer because half of "the network is slow" is really a GPU: an idle GPU (low SM util) waiting on a collective, or an Xid 79 that just knocked a rank out.

# What DCGM watches (fields dcgm-exporter scrapes)
dcgmi dmon -e 203,254,1009,1010,155   # SM util, mem util, PCIe replay, NVLink, power
dcgmi health -c                        # rolls up Xid / ECC / thermal health

Collective telemetry — the NCCL path

The fabric and NIC layers can be spotless while the job is slow, so you need the collective's own view. NCCL exposes two things:

• NCCL_DEBUG=INFO (plus NCCL_DEBUG_SUBSYS=INIT,NET,COLL) logs the ring/tree topology it built, which transport it chose per pair, and per-collective timings — the raw material for finding the straggler rank.
• The NCCL profiler plugin / NVTX ranges feed per-step, per-collective timings into a metrics endpoint or a profiler (Nsight, PyTorch/Kineto). Production jobs wrap this so per-rank step times land in the same TSDB as the fabric counters — which is what makes correlation possible.

This layer is usually the training team's responsibility, but you must insist it's plumbed in — without it you can detect a slow fabric but you can't prove the fabric is (or isn't) the cause.

---

The choice that shapes everything: push vs. pull, stream vs. sample

Two orthogonal decisions define your data plane:

• Pull (poll) — collector asks on a schedule (SNMP, Prometheus scrape). Simple, but the cadence is a floor on your blindness: a 15 s scrape can't see a 200 µs burst.

• Push (stream) — device sends on change or sub-second (gNMI, sFlow). Catches fast events; needs the device to support it.

• Streaming every value — complete but heavy; reserve for bounded, high-value counters (PFC/ECN/queues).

• Sampling — 1-in-N (sFlow) or on-demand (INT). The only sane approach for unbounded dimensions (flows, per-packet).

The design rule mirrors the cardinality rule: stream the bounded high-value counters, sample the unbounded ones, poll only the legacy stragglers.

Language Bridge

Out-of-band (OOB) management network→A separate physical/logical network for telemetry and management, isolated from the RoCE data plane. So your monitoring never competes with training traffic — and still works when the data plane is the thing that's broken.

And put every collector on an out-of-band management network. Telemetry that rides the data plane both steals bandwidth from training and goes blind exactly when you need it most — during a data-plane incident. The management plane must be independent.

---

💡 What you should remember

#		Concept	Why it matters
1	📡	gNMI + OpenConfig is the backbone	Sub-second streaming, vendor-neutral models — subscribe once, get a stream.
2	🐌	SNMP is legacy — stream instead	Polling at 30 s–5 min averages away the microsecond events that cause stalls; gNMI streaming is the default.
3	🧪	sFlow/IPFIX for flows, streaming for counters	Sampled flow visibility answers ECMP balance; it doesn't replace streamed counters.
4	🔌	RDMA counters are vendor-neutral	ethtool -S + standard HW counters read the same across NIC vendors — one exporter.
5	🎮	DCGM + NCCL are non-optional	Rule out the GPU (Xid, idle SM) and see the straggler rank — the fabric is often innocent.
6	🔀	Stream bounded, sample unbounded, on OOB net	Matches the cardinality rule and keeps monitoring alive during data-plane incidents.

---

Next: RoCEv2 Fabric Telemetry → — reading the congestion-control loop (PFC → ECN → CNP → DCQCN) from real counters, per vendor.