AI Fabric Observability — Single Pane of Glass

Job Completion Time — ETA

–

MFU

–

Step p99

–

AllReduce busbw

–

of 400

Job avail (SLO 99.5)

–

RoCE loss

–

no-drop contract

PFC pause (TC3)

–

tx_prio3_pause

▚ Job health — click any row to drill down– jobs

Job / podPhaseGPU%°CbusbwQ3 bufPortSM_ACTIVE

◈ Root-cause verdict — the selected job

⌘ Remediation — copy-ready querieshost/switch CLI + PromQL

▦ Switch port × metric — leaf-14 (SONiC)gNMI stream

✧ GPU_UTIL trap — selected nodeDCGM :9400

GPU_UTIL

–

SM_ACTIVE

–

◉ RoCEv2 congestion loop — TC3node_exporter · gNMI · WJH

ECN marked /s

–

CNP /s

–

packet_seq_err /s

–

FEC pre-BER

–

ECN marks vs CNP (60s) — ECN-first, PFC-rarely

Watermark → xoff · et33

⚑ Drops with reason — WJHevent-based

⑆ The Walk-Down — rank → portidle

🔔 Alerts — anomaly, not absolutes