What to Monitor — The Signal Taxonomy
You can't monitor everything, and you shouldn't try. Modern RDMA NICs expose hundreds of counters each; a switch exposes thousands; multiply by every port in a 10k-GPU fabric and naive collection melts your time-series database before it finds a single straggler.
This page is the inventory and the priority order — the whole territory, so nothing falls through the cracks. It starts with the mental model (symptom at the top, cause at the bottom), lays out the ten monitorable domains rated by how badly you need each, then narrows to the golden signals you keep on screen, the two gap-check methods, and the cardinality rule that keeps the pipeline alive.
- State the symptom-at-top model — a problem at any layer is felt as slowness at the top, so you monitor every layer to descend from symptom to cause.
- Walk the ten monitorable domains and rate each signal Must / Should / Nice using blast radius × likelihood × how blind you are without it.
- List the golden signals for an AI fabric and what good vs. bad looks like for each.
- Apply RED and USE as two lenses that ensure you didn't leave a gap — and know why saturation, not utilization, predicts stalls.
- Avoid the cardinality traps — per-QP, per-VF, and per-flow explosions that quietly bankrupt your metrics pipeline.
The model: symptom at the top, cause at the bottom
Everything rests on one asymmetry. An AI fabric is a layered stack, and a problem at any layer is felt as the same symptom at the top — a training step got slower. A step is only as fast as its slowest collective; a collective only as fast as its slowest rank; a rank only as fast as its NIC's slowest queue-pair; and that queue-pair is at the mercy of one congested switch buffer or one marginal optic.
The golden rule: a healthy top layer proves nothing about the bottom. A job can hit its step-time target while a NIC quietly burns 3% of its bandwidth on retransmits — headroom that vanishes the day you scale the job. That's why you instrument the lower layers even when nothing is "wrong." Observability is the discipline of walking that chain downward — from the symptom you noticed to the layer that caused it.
The ten monitorable domains
The five-layer model is the mental map. In practice an AI cluster has ten domains that can stall a GPU, each with its own signals and its own tools. Rate each by a simple principle:
Must (Tier 0) — lose it and a cluster-wide problem goes undetected or un-diagnosable. Should (Tier 1) — you can run without it, but root-cause gets slow and stragglers hide. Nice (Tier 2) — deep-diagnostic; high cost, specialist value, reached for during an incident.
That's just blast radius × likelihood × how blind you are without it, made concrete.
1 · Compute / GPU health
The thing you're protecting. A dead or throttled GPU stalls its entire collective.
| Signal | Tools | Tier |
|---|---|---|
| Temp, power, clocks, throttle reasons | DCGM, nvidia-smi, NVML, Redfish/BMC | Must |
| ECC / XID errors, row-remap, "fell off bus" (XID 79) | DCGM (DCGM_FI_DEV_XID_ERRORS), dmesg | Must |
| SM active / tensor active / DRAM active, mem used | DCGM profiling fields, Nsight (deep) | Must |
| NVLink/NVSwitch BW + CRC/flit errors + recovery | DCGM NVLink fields, nvidia-smi nvlink | Must |
| PCIe replay, gen/width, GPU↔NIC affinity | DCGM, nvidia-smi topo -m, PCIe AER | Should |
2 · Collective comms / NCCL
The layer that drives the wire — where "the network is slow" first becomes visible.
| Signal | Tools | Tier |
|---|---|---|
| Collective duration, busbw vs algobw, hangs/timeouts | NCCL profiler plugin, NCCL_DEBUG=INFO | Must |
| Network-attributed time (comms % of step) | NCCL profiler + DCGM correlation | Must |
| Ring/tree construction, topology detection | NCCL_DEBUG=INFO logs | Should |
| Baseline bandwidth (regression test) | all_reduce_perf / nccl-tests | Should |
3 · RDMA / NIC — the RoCEv2 endpoint
Where the fabric's health shows up on the host. Expanded in 19.4.
| Signal | Tools | Tier |
|---|---|---|
| Congestion loop: ECN marks, CNP sent/handled/ignored | rdma statistic, /sys/class/infiniband, node_exporter | Must |
Retransmit/loss: out_of_sequence, packet_seq_err, local_ack_timeout_err, rnr_nak_retry_err, out_of_buffer | rdma statistic, ethtool -S | Must |
| Per-priority PFC pause (TC3) | ethtool -S … prio3 | Must |
| Per-port / per-QP throughput, link state | node_exporter, rdma statistic show qp (on-demand) | Should |
4 · Fabric / Switch
Where the root cause usually lives. Expanded in 19.4.
| Signal | Tools | Tier |
|---|---|---|
| PFC (per priority), ECN/WRED marks | gNMI/OpenConfig streaming, SONiC counters DB | Must |
| Drops with reason (buffer / ACL / L3) | WJH, mirror-on-drop, sFlow drop-monitoring | Must |
| Queue watermarks / buffer occupancy (leading indicator) | gNMI, SONiC counters DB, vendor buffer telemetry | Must |
| Interface errors / discards, link flaps, utilization | gNMI streaming (SNMP only for legacy gear) | Must |
| PFC-storm / deadlock detection | SONiC PFC watchdog (show pfcwd stats) | Must |
| ECMP load-balance distribution, hash polarization | sFlow/IPFIX, per-port utilization spread | Should |
| Microburst detection (sub-second) | gNMI high-frequency, INT | Nice |
5 · Routing / control plane
Rare, but total blast radius when it goes.
| Signal | Tools | Tier |
|---|---|---|
| BGP/underlay sessions, adjacency flaps, route churn | gNMI, BGP exporter, syslog | Must |
| Config drift, image / firmware / driver versions | config mgmt (Ansible/NetBox), gNMI | Should |
6 · Optical / physical
The "marginal optic" that quietly degrades a link before it flaps.
| Signal | Tools | Tier |
|---|---|---|
| FEC: pre-FEC BER, corrected/uncorrected codewords, symbol errors | ethtool FEC stats, gNMI, transceiver show | Must |
| Optical Rx/Tx power (DOM), transceiver temp, bias current | ethtool -m, DOM via gNMI | Should |
| Lane errors, cable / connector faults | switch transceiver telemetry | Should |
7 · Host / system
The server around the GPU — and where clock sync lives.
| Signal | Tools | Tier |
|---|---|---|
| Time sync (PTP/NTP) — required for cross-layer correlation | chrony / ptp4l metrics, node_exporter | Must |
| CPU, memory, NUMA balance, huge pages | node_exporter | Should |
Kernel: dmesg, MCE, soft-lockups, PCIe AER | journald, node_exporter textfile collector | Should |
| Local NVMe / disk health | node_exporter, SMART | Should |
8 · Storage / data pipeline
A slow checkpoint or a starved dataloader stalls every rank at once.
| Signal | Tools | Tier |
|---|---|---|
| Checkpoint read/write duration (can block all ranks) | framework timers, FS exporter | Must |
| Parallel FS health + throughput/latency (Lustre/GPFS/WEKA) | vendor exporter, node_exporter diskstats | Should |
| Dataloader stalls / input starvation | framework metrics, PCIe RX bytes | Should |
9 · Power & cooling / facility
The layer everyone forgets until a rack browns out and GPUs power-cap.
| Signal | Tools | Tier |
|---|---|---|
| Rack / PDU power draw, power-cap events (→ throttling) | Redfish/IPMI, PDU telemetry, DCIM | Should |
| Cooling: inlet temp, liquid-cooling CDU, humidity | BMS / DCIM, BMC | Should |
10 · The telemetry pipeline itself
If this is down, you're blind — and you don't know it.
| Signal | Tools | Tier |
|---|---|---|
| Prometheus up, scrape success, exporter liveness | Prometheus self-monitoring, up metric | Must |
| TSDB cardinality / ingestion rate / retention headroom | Prometheus internal metrics | Should |
The Must / Should / Nice rollup
| Tier | When you need it | Domains |
|---|---|---|
| Must (Tier 0) | Before a single production job — lose these and problems are invisible or un-diagnosable | GPU HW + XID · NCCL timing · RDMA congestion + retransmits · per-priority PFC/ECN · switch drops-with-reason + watermarks + PFC watchdog · interface errors · BGP/underlay · FEC/pre-FEC BER · time sync · checkpoint duration · job step-time + per-rank variance · pipeline self-health |
| Should (Tier 1) | Needed at scale and for fast root-cause; runnable without, but RCA slows and stragglers hide | optical DOM · buffer-occupancy trending · ECMP balance · parallel-FS telemetry · power/thermal facility · host NUMA/kernel · config/version drift |
| Nice (Tier 2) | Deep-diagnostic, reach-for-during-incident — high cost, specialist value | in-band network telemetry (INT) · per-QP tracing · Nsight profiling · sFlow flow analytics · DCIM correlation |
None of these rows, on its own, is observability — they're monitoring. What turns the inventory into observability is two things: the join keys (rank ↔ GPU ↔ NIC ↔ port ↔ TC) stapled onto every metric, so a Tier 0 GPU stall can be walked down to a Tier 1 optic; and time sync (domain 7), because without a common clock the signals from five layers can't be lined up on one timeline.
The golden signals
The ten domains are the complete inventory. You don't stare at all of it. Borrowing from SRE's golden-signals idea, here is the short list every AI-fabric on-call keeps on one screen — chosen because together they detect ~all real incidents and each points at a different layer:
- AllReduce step time — the symptom. If this is flat, you're fine. If it moves, the other five tell you where.
- PFC pauses per port·priority — buffers filling; the fabric fighting congestion.
- ECN marks per port·priority — the signal DCQCN reacts to; proves the control loop is alive.
- RDMA NIC errors — retransmits / CQE errors / out-of-sequence; losslessness failing at the edge.
- ECMP imbalance — hash polarization; one link doing everyone's work.
- Pre-FEC BER — the earliest physical-degradation warning, hours before a link-down.
Set alerts on anomalies, not absolutes — a sudden change from each signal's own baseline. Absolute thresholds either miss slow degradations or cry wolf; more on that in SLIs, SLOs & alerting.
Two lenses that catch what you missed: RED and USE
Golden signals are the curated list. RED and USE are methods — checklists that ensure you didn't leave a hole. Run every service and every resource through both.
-
RED (for request-driven things — inference endpoints, gateways, the collective itself):
- Rate — requests/collectives per second
- Errors — failed ones (timeouts, CQE errors, NCCL aborts)
- Duration — latency distribution, especially the tail (p99/p999)
-
USE (for every resource — a link, a buffer, a NIC queue, a GPU):
- Utilization — % busy / % full
- Saturation — the queued/waiting overflow beyond 100% (
port_xmit_wait, buffer at watermark) - Errors — discards, retransmits, CRC
The AI-specific insight: saturation, not utilization, predicts stalls. A link at 60% average utilization that's microbursting into PFC every few milliseconds is saturated even though it looks half-idle. USE forces you to watch the saturation signals — pause frames, buffer watermarks, port_xmit_wait — that averaged utilization hides.
The cardinality trap
Every signal above can be sliced by dimensions — port, priority, queue, QP, VF, flow, job, rank. Cardinality is the number of unique time series you create, and it's the product of those dimensions. It is the thing that kills observability pipelines.
The explosive dimensions to respect:
| Dimension | Scale on a 10k-GPU fabric | Rule |
|---|---|---|
| Per-QP (queue pair) | Thousands per NIC | Never export per-QP as steady metrics. Aggregate on the host; keep per-QP for on-demand deep dives. |
| Per-VF (SR-IOV virtual function) | 8–32 per NIC × thousands of NICs | Roll up to per-NIC or per-pod for dashboards. |
| Per-flow / 5-tuple | Effectively unbounded | Sample (sFlow) or use INT on-demand — don't store every flow as a metric. |
| Per-port × per-priority | Ports × 8 | Keep — bounded and essential. Slice to the RoCE prio. |
The design rule: bounded dimensions become always-on metrics; unbounded dimensions become sampled telemetry or logs you can query on demand. Getting this wrong is the difference between a pipeline that costs a rounding error and one that costs more than the incidents it catches. The pipeline mechanics for handling this live in the telemetry pipeline architecture.
💡 What you should remember
| # | Concept | Why it matters | |
|---|---|---|---|
| 1 | ⬇️ | Symptom at top, cause at bottom | A healthy top layer proves nothing about the bottom — instrument every layer to descend. |
| 2 | 🗂️ | Ten domains, rated Must/Should/Nice | Blast radius × likelihood × blindness tells you what to deploy before job one. |
| 3 | ✨ | Six golden signals | Step time + PFC + ECN + NIC errors + ECMP balance + pre-FEC BER cover ~all real incidents. |
| 4 | 🔭 | RED + USE as gap-checks | Watch saturation, not just utilization — a half-idle link can still be stalling on microbursts. |
| 5 | ⏱️ | Time sync is a Tier-0 prerequisite | Without a common clock, five layers' signals can't be lined up on one timeline. |
| 6 | 💥 | Cardinality kills pipelines | Never export per-QP/per-VF/per-flow as always-on metrics; bound it or sample it. |
Next: Telemetry Sources & Protocols → — where each of these signals actually comes from, and how to get it off the box.