Section 19 · Curriculum Map

Seeing inside an AI fabric

Ten pages, one mental model. Read a slow training step all the way down to a single marginal optic — then build the pipeline, SLOs and alerting that make it repeatable. Start anywhere; the arc below is the recommended path.

Start reading → ◐ Open the live dashboard Section index
01

The mental model

symptom at the top · root cause at the bottom
SYMPTOM ↓ CAUSE
Job / Training step p99 · MFU · tokens/s · JCTwhere you feel it
GPU / Compute SM_ACTIVE · XID · temp · powerbusy, or waiting?
Collective / NCCL busbw vs algobw · coll. time · ring/treethe comms layer
RDMA / NIC hw_counters · CNP · retransmits · GIDthe host's edge
Fabric / Switch PFC · ECN · watermarks · dropsthe shared medium
Optical / Physical pre-FEC BER · Rx power · flapswhere truth lives
02

The ten pages, as a path

orient → collect → correlate → master
1Orientwhy & what
19.1
Why AI Fabrics Need Observability
One slow link stalls thousands of GPUs. Why classic NMS is blind, and the detect → localize → attribute → prevent loop.
read →
19.2
The Signal Taxonomy
What to monitor: the five-layer model, ten domains rated Must / Should / Nice, golden signals, and cardinality traps.
read →
2Collectget the signals
19.3
Telemetry Sources & Protocols
How to get signals off the box: gNMI / OpenConfig streaming, in-band telemetry, DCGM, the NCCL profiler, optics diagnostics.
read →
19.4
RoCEv2 Fabric Telemetry
Read the congestion loop from real counters: ECN → CNP → DCQCN → PFC, per-vendor names, WJH, watermarks, retransmits.
read →
19.5
GPU & Job Telemetry
SM_ACTIVE not GPU_UTIL, the XID table, NCCL busbw vs algobw, per-rank stragglers, and MFU as the north-star.
read →
3Correlatemake it answer
19.6
Correlation & the Walk-Down
The join-key identity chain, the time-sync prerequisite, and the six-query walk from a slow step to one marginal optic.
read →
19.7
The Telemetry Pipeline
Collect → transport → store → visualize → alert, plus the cardinality and retention math that decides if it scales.
read →
19.8
SLIs, SLOs & Alerting
A pager that fires when the job is hurting — anomaly-relative alerts, burn-rate SLOs, probes, three-tier dashboards.
read →
4Mastergo deeper
19.9
Advanced Fabric Visibility
When per-port counters aren't enough: in-band INT, microbursts, deep buffers, mirror-on-drop, and the vendor platforms.
read →
19.10
Reference Architecture & Maturity
The capstone: one end-to-end blueprint, the instrumentation checklist, the anti-patterns, and a five-level maturity model.
read →
03

What you'll be able to do — the walk-down

a slow step, in six queries
1
Step p99 is up
2
Find the straggler
3
Resolve identity
4
Cross to the fabric
5
Read the port
ROOT CAUSE
6
Down to the optic