Why AI Fabrics Need Their Own Observability
There's no single way to build an AI cluster. You can run it on Kubernetes or on bare metal. InfiniBand or Ethernet. RoCEv2 or UEC or Falcon. DCQCN or HPCC or Swift. Each layer of the stack has a design space, and the right answer depends on your scale, your vendors, and what you've already deployed.
This curriculum walks every one of those design spaces — what exists, who's running what, what this course picks, and why. Each section ends with one combination so you can build it; the comparison stays on the page so you can swap when your stack differs.
This section is about seeing inside the fabric you built. Every other section taught you how the machine, the fabric, and the transport work. This one teaches you how to know they're working — what to measure, where the data comes from, how to pipe it, and how to turn ten thousand counters into one answer to the only question that matters: is the job healthy, and if not, which link do I go touch?
Two interactive companions make this whole section concrete. Open them first — everything below is teaching you to read them.
The dashboard is a Grafana-style control room for a 4,096-GPU cluster — flip between a healthy fleet, fabric congestion, a straggler rank, and an XID storm, then click any job to walk it down to a host / network / switch verdict. The map is the whole section on one screen.
The one fact that changes everything
A distributed training job is bulk-synchronous. Every few hundred milliseconds every GPU stops computing, exchanges gradients in a collective (AllReduce, AllGather, ReduceScatter), waits for all peers to finish, then takes the next step. No rank moves to step N+1 until the slowest rank finishes step N.
That single property is why AI observability is a different discipline:
A training job runs at the speed of its slowest participant.
One NIC dropping to a lower retransmit rate, one optic degrading, one switch buffer running hot on one priority queue — any of them slows one rank, and one slow rank slows all of them. A 1,024-GPU job doesn't degrade gracefully by 0.1%. It degrades by however much the single worst link degrades, multiplied across every GPU that's now sitting idle waiting.
The economics — why anyone pays for this
Put numbers on it. A modest cluster:
- 1,024 H100 GPUs, 128 nodes × 8 GPUs
- Rented/amortized at ~$3/GPU-hour (all-in: hardware, power, cooling, DC)
- That's ~$3,072/hour, ~$73,700/day, ~$26.9M/year of GPU time
Now one rail's optics degrade and add 10% to average step time. The job is synchronous, so every GPU inherits that 10%:
| Value | |
|---|---|
| Effective GPU-hours wasted | 10% of 1,024 = ~102 GPU-hours per wall-clock hour |
| Cost of the waste | ~$307/hour, ~$7,370/day |
| If it takes 3 days to notice and fix | ~$22,000 — from one bad optic |
The optic itself costs a few hundred dollars. The time-to-detect is the expense. Observability is the discipline of collapsing that time-to-detect from days (someone eventually notices training is slow) to minutes (an alert names the rail before the training team files a ticket). That is the entire business case.
- State the core thesis — a synchronous job runs at the speed of its slowest rank, so a single degraded link taxes the entire job, and time-to-detect is the real cost.
- Explain why traditional NMS is necessary but blind — link-up, BGP-up, and 5-minute SNMP utilization can all be green while a job crawls.
- Distinguish monitoring from observability — dashboards for known failure modes vs. the ability to ask new questions of a running system without shipping new instrumentation.
- Name the observability loop — detect → localize → attribute → prevent — and the four pillars (metrics, logs, traces, events) plus the two AI-specific ones (topology/flow telemetry and collective profiling).
Why your existing NMS won't save you
Traditional network monitoring answers "is the network up?" An AI fabric needs "is the network fast, lossless, and balanced — right now, on this priority, for this job?" Those are different questions, and the classic tools are structurally blind to the second one:
| Traditional NMS shows | An AI fabric can be dying while it shows |
|---|---|
| All links up, all BGP sessions established | A single optic at high pre-FEC BER is silently corrupting frames |
| Interface utilization 40% avg over a 5-min SNMP poll | A microburst filled the buffer for 200 µs and triggered PFC — invisible at 5-min averaging |
| No interface errors/discards in the classic counters | out_of_sequence and port_xmit_wait on the RDMA priority are climbing |
| Ping/latency nominal | AllReduce tail latency (p99) doubled while the median held |
Three structural gaps:
- Wrong cadence. Congestion on an AI fabric happens in microseconds to milliseconds. A 30-second or 5-minute poll averages the burst out of existence. You need streaming telemetry and high-frequency counters.
- Wrong signals. Up/down and utilization don't capture losslessness. The signals that predict a stall — PFC pause frames, ECN marks, per-queue buffer occupancy, NIC retransmits, per-QP completion errors — aren't in the classic MIB dashboards.
- Wrong subject. NMS watches devices. AI observability watches jobs. The unit of health is "the AllReduce for job 4417," which spans dozens of NICs, hundreds of links, and a software collective — no single device knows it's slow.
Monitoring vs. observability
The two words get used interchangeably; the distinction is load-bearing here.
- Monitoring = watching for known failure modes. You know PFC storms happen, so you build a PFC-pause dashboard and an alert. Monitoring answers known-unknowns: "has the thing I worried about happened yet?"
- Observability = the property that lets you interrogate the system for unknown failure modes. A job is slow, no existing alert fired, and you can still pivot through metrics/logs/traces/topology and find the cause — because you collected rich, high-cardinality, correlatable data. Observability answers unknown-unknowns: "something is wrong that I never anticipated — can I still explain it?"
You need both. Monitoring catches the failures you've seen before at 3 AM without waking anyone up to think. Observability is what you fall back on when the failure is new — and on a fabric with tens of thousands of links, novel failures are a weekly event.
The observability loop
Every mature operation runs the same four-step loop. Design your observability stack to serve each step, in order:
- Detect — a top-line indicator moved. Usually AllReduce step time or fabric goodput. This is the symptom, and it's the only thing the customer feels.
- Localize — narrow from "the job is slow" to "this link / NIC / GPU." This is the hard part on a big fabric and the reason topology-aware correlation exists.
- Attribute — find the cause at the localized spot: pre-FEC BER on an optic, a PFC storm, an out-of-sequence flood, a driver-version mismatch, a GPU Xid error.
- Prevent — remediate, then close the loop: add the signal or alert that would have caught it earlier, so the next occurrence is a page, not an outage.
Each page in this section maps onto this loop. Knowing which step you're serving keeps you from the classic trap: a beautiful dashboard that detects brilliantly but can't localize, so every incident still becomes an all-hands hunt.
The pillars — and the two AI adds
Classic observability has three (sometimes four) pillars. AI fabrics need two more that general-purpose stacks don't emphasize:
| Pillar | What it is | Primary use in the loop |
|---|---|---|
| Metrics | Numeric time series (counters, gauges): PFC pauses, ECN marks, goodput, step time | Detect, and trend/baseline |
| Logs | Discrete text events: syslog, dmesg, NCCL debug output, driver messages | Attribute |
| Traces | Causally-linked spans across a request/collective | Localize latency in inference paths |
| Events | State changes: link flap, BGP transition, config change, cordon/drain | Attribute + timeline correlation |
| ➕ Topology / flow telemetry | The fabric graph + which 5-tuple/flow took which ECMP path | Localize — turn "slow job" into "slow link" |
| ➕ Collective profiling | Per-rank, per-step NCCL/RCCL timings and comm patterns | Detect + localize the straggler rank |
The two additions are why you can't just point a stock Prometheus/Grafana at an AI cluster and call it done. Without the topology graph you can detect and alert but you can't localize. Without collective profiling you're blind to which rank is the straggler and whether the fault is even in the network (it's often a GPU or a host, not the fabric).
Three lenses, one truth
An AI fabric has three audiences who each need a different view of the same underlying data:
- Network / fabric engineer — PFC, ECN, buffers, optics, ECMP balance, BGP. "Is the fabric lossless and balanced?"
- Platform / host engineer — NIC counters, RDMA state, GPU health (DCGM/Xid), drivers, SR-IOV/VF mapping. "Is the host feeding the fabric correctly?"
- Training / ML engineer — step time, goodput, MFU, per-rank variance. "Is my job healthy?"
The failure of most observability efforts is that these three build separate, unjoinable stacks. The whole art — the subject of correlation & topology — is making them one dataset so you can jump from "job 4417 is slow" (training lens) to "rank 812 is the straggler" (collective lens) to "worker-102's NIC is retransmitting" (host lens) to "leaf-14 port et-0/0/9 optic is at high BER" (fabric lens) in seconds, not meetings.
💡 What you should remember
| # | Concept | Why it matters | |
|---|---|---|---|
| 1 | ⛓️ | Slowest-rank-wins | A synchronous job runs at the speed of its worst link; one straggler taxes every GPU. |
| 2 | 💸 | Time-to-detect is the cost | The bad optic is cheap; the days of wasted GPU-hours before you find it are not. |
| 3 | 🙈 | Traditional NMS is blind here | Up/down + 5-min utilization can be all-green while a job crawls. |
| 4 | 🔬 | Monitoring ≠ observability | Monitoring catches known failures; observability lets you explain new ones. |
| 5 | 🔁 | Detect → localize → attribute → prevent | Design every signal to serve a step of this loop — localization is the hard one. |
| 6 | ➕ | Two AI-specific pillars | Topology/flow telemetry and collective profiling are what stock stacks lack. |
Next: The Signal Taxonomy → — the full inventory of what to monitor, layer by layer, from the collective down to the optic.