19.1 Why AI Fabrics Need Observability
A training job runs at the speed of its slowest participant, so one degraded link stalls thousands of GPUs. Why traditional NMS is blind to it, monitoring vs observability, and the detect → localize → attribute → prevent loop.
19.2 The Signal Taxonomy
The full inventory of what to monitor on an AI fabric — the five-layer symptom-at-top model, the ten monitorable domains rated Must/Should/Nice, the golden signals, RED and USE applied to a fabric, and the cardinality traps that melt your TSDB.
19.3 Telemetry Sources & Protocols
Where the signals come from and how to get them off the box — SNMP to sFlow to gNMI/OpenConfig streaming to in-band telemetry on the switch, plus ethtool/RDMA counters, DCGM, the NCCL profiler, and optics diagnostics on the host.
19.4 RoCEv2 Fabric Telemetry
Reading the RoCEv2 congestion-control loop from real counters — ECN → CNP → DCQCN → PFC — with per-vendor counter names for NVIDIA/Mellanox, Broadcom/SONiC, and vendor-neutral gNMI, plus drop visibility (WJH, PFC watchdog), the NIC retransmit family, and buffer/watermark telemetry.
19.5 GPU & Job Telemetry
The GPU and job signals a network engineer cares about — DCGM core metrics, why GPU_UTIL is a trap and SM_ACTIVE is the truth, the GPU-stall "waiting on comms" signal, the XID errors that point at the interconnect, NCCL busbw vs algobw, per-rank straggler detection, and MFU as the north-star.
19.6 Correlation & the Walk-Down
The single most important idea in AI-fabric observability — the join-key identity chain (rank ↔ GPU ↔ NIC ↔ switch port ↔ TC) that lets you correlate five layers, the time-sync prerequisite, and the six-query walk-down from a slow training step to a single marginal optic.
19.7 The Telemetry Pipeline
The reference pipeline that turns raw signals into answers — collectors and exporters per layer, the collect → transport → store → visualize → alert flow, the cardinality and retention math that decides whether it scales, OpenTelemetry convergence, and monitoring the pipeline itself.
19.8 SLIs, SLOs & Alerting
Turning telemetry into a pager that fires when the job is hurting and the fabric is the reason — anomaly-relative alerting instead of absolute thresholds, SLIs and SLOs for a training fabric, error budgets and multi-window burn-rate, synthetic probes, and a three-tier dashboard hierarchy.
19.9 Advanced Fabric Visibility
When per-port counters aren't enough — in-band network telemetry (INT/postcard), microburst and watermark deep-dive, deep-buffer analytics, mirror-on-drop packet capture, and the vendor deep-telemetry platforms (WJH, BroadView, CloudVision, UFM) with a capability matrix and a clear-eyed view of when the cost is worth it.
19.10 Reference Architecture & Maturity
The capstone — one end-to-end observability reference architecture, the instrumentation checklist of what to deploy where, the anti-patterns that quietly collapse observability back into monitoring, an OSS-vs-vendor view, a five-level maturity model to grade yourself against, and the closed-loop caveats.