Reference Architecture & Maturity

This is the capstone. Nine pages built the pieces — signals, sources, RoCE and GPU telemetry, correlation, the pipeline, alerting, and advanced visibility. This page assembles them into one design you can point at, one checklist you can deploy from, and one maturity ladder you can grade yourself on.

After this page, you'll be able to

1. Draw the end-to-end reference architecture from ASIC counter to pager.
2. Deploy from the instrumentation checklist — what goes where, and the irreducible minimum.
3. Name the anti-patterns that collapse observability back into monitoring.
4. Choose OSS vs vendor per layer with eyes open.
5. Grade your fabric on the L0–L4 maturity model and know the closed-loop caveats.

---

The reference architecture

Every arrow into the store passes through the join-key labeller — that is the load-bearing element. Without it you have beautiful, uncorrelatable dashboards; with it, a GPU stall walks down to a single switch port.

◐ See this architecture live → — the single-pane dashboard is this diagram's output rendered for a 4,096-GPU cluster. The DASH tier (fleet / fabric / job) with the join-key labeller wired in: pick a fault scenario, click a job, and watch it resolve to a host / network / switch verdict.

---

Instrumentation checklist — what to deploy where

Where	Deploy	Gives you	Priority
Every switch	gNMI streaming (OpenConfig)	queue stats, PFC/ECN, watermarks, interface counters	Must
Every switch	WJH / drop-reason telemetry	why packets dropped, per port/queue	Must
Every switch	PFC watchdog (SONiC) + stats	deadlock detection on lossless queues	Must
Every host	node_exporter + RDMA counter collector	NIC hw_counters, per-priority pause/ECN	Must
Every GPU node	dcgm-exporter (:9400)	SM_ACTIVE, tensor active, XID, NVLink/PCIe	Must
Every job	NCCL profiler + NCCL_DEBUG → central logs	per-collective timing, busbw, ring/tree	Should
Central	join-key label map (rank↔GPU↔NIC↔port)	cross-layer correlation	Must
Central	Prometheus + Grafana + Alertmanager, recording rules	storage, dashboards, anomaly alerts	Must
Fabric	sFlow / INT sampling	flow-level hotspot attribution	Nice

The irreducible minimum

If you do nothing else: gNMI + WJH on the switch, DCGM + RDMA counters on the host, and the join-key labels that connect them. That minimum set is what makes the walk-down possible — everything else is refinement.

---

Anti-patterns

Each of these is a real, common way teams spend money on telemetry and still stay blind. They are the negative image of the whole section:

• No join keys. Separate GPU and fabric dashboards that can't be correlated — the most common and most expensive mistake. Every incident becomes manual detective work while the cluster burns money.
• Alerting on absolutes. PFC > 0 and ECN > 0 page on a healthy fabric; you'll train the team to ignore the pager. Alert on anomaly instead.
• Per-QP everything. Scraping every queue-pair as a standing series is a cardinality bomb that takes down the TSDB you rely on during incidents.
• Watching aggregates, not per-TC. RoCE lives on one priority; port-level totals hide the signal on TC3.
• Trusting GPU_UTIL. It reads ~100% while GPUs spin idle on collectives — use SM_ACTIVE.
• No drop visibility. Interface discard counters without WJH/drop-reasons leave you knowing a "lossless" fabric lost packets but never where or why.
• GPU and fabric owned by different teams with different tools. Nobody sees the stall→port link that connects them — which is the whole point of observability.

Language Bridge

Monitoring vs observability→Monitoring = many dashboards, each showing one layer's health. Observability = those signals joined so you can ask a new question — 'why is rank 47 slow?' — and walk it to a root cause. Every anti-pattern above is a way to have expensive monitoring and zero observability.

---

OSS vs vendor

You will mix these; the split usually falls along the same seams:

Layer	Open-source	Vendor / commercial	Typical choice
Store + query	Prometheus, VictoriaMetrics, Mimir/Thanos	Chronosphere, Grafana Cloud, Datadog	OSS core, commercial for scale/managed
Dashboards	Grafana	vendor NMS UIs	Grafana almost always
Switch deep telemetry	gNMI/OpenConfig, SONiC counters	WJH, BroadView, CloudVision, UFM	gNMI baseline + vendor for drop reasons
GPU/job	dcgm-exporter, NCCL profiler, OTel	vendor observability suites	OSS — the ecosystem is strong here
Alerting	Alertmanager	PagerDuty, Opsgenie	Alertmanager → commercial pager

The durable pattern: open-source, gNMI/OpenConfig-based core for portability, vendor deep-telemetry only where it uniquely adds value (event-based drop reasons, microburst capture, fabric-manager analytics).

---

The maturity model

Grade your fabric honestly. Most organizations are lower than they think — usually because they stalled at L2 with great dashboards and no join keys.

Level	Name	You have	You still can't
L0	Blind	SNMP up/down, maybe link utilization	tell a slow job from a fast one
L1	Monitored	per-layer dashboards (GPU, fabric, host) separately	correlate across layers
L2	Correlated	join keys + time sync; can walk a stall to a port	do it without a human driving
L3	Proactive	anomaly/burn-rate alerts, synthetic probes, baselining	act before impact automatically
L4	Closed-loop	automated response — drain a marginal optic, reroute, quarantine a node	trust it blindly (see caveats)

The jump that matters most is L1 → L2: it's not more data, it's the join keys. That single investment is what converts monitoring into observability.

---

Closed-loop caveats

L4 automation is the aspiration, not the default. Treat automated remediation with respect:

• Confidence gating. Only automate actions on high-confidence, multi-signal conditions — the same correlate-before-paging bar, raised. Auto-draining a node on a single lifetime counter is how you take down a healthy fabric.
• Blast-radius limits. Rate-limit and cap automated actions; a feedback loop reacting to its own disruption can cascade.
• Human-in-the-loop for the ambiguous. Automate the unambiguous (a node throwing XID 74 storms → cordon it); page a human for anything the signals don't make certain.
• Observability for the automation. The closed loop is itself infrastructure that must be observed — log every automated action and alert when it fires abnormally often.

Aim for L3 done well before reaching for L4. A proactive, well-instrumented fabric with a human making the final call beats a brittle closed loop that occasionally amputates a healthy rack.

---

💡 What you should remember

#		Concept	Why it matters
1	🏗️	Everything flows through the labeller	Join keys + time sync are the one load-bearing element of the architecture.
2	✅	The irreducible minimum	gNMI + WJH + DCGM + RDMA counters + join keys — deploy that first.
3	⚠️	Anti-patterns collapse observability	No join keys, absolute alerts, per-QP scraping, and GPU_UTIL are how you stay blind.
4	🪜	L1 → L2 is the real jump	The value isn't more data — it's the join keys that make correlation possible.
5	🔁	L3 before L4	Earn proactive alerting before automating remediation; gate the loop on multi-signal confidence.

---

Where to go next

You've built the observability system. Now put it to work in operations:

• 18.2 Common Failure Modes → — the failure library the walk-down maps to.
• 18.3 Incident Response Playbooks → — step-by-step response when the pager fires.
• 18.4 MFU & Diagnosis → — the north-star metric this whole section defends.
• 18.5 RoCEv2 Operator Cheatsheet → — the counter commands, at your fingertips.

Next: 20. Cluster Build Guide → — designing and standing up the fabric you now know how to observe.