SLIs, SLOs & Alerting

You now collect the right signals and can join them (pipeline). The last mile is judgment: which combination of signals should wake a human at 3 AM, and which is just the control loop doing its job?

Get this wrong in the obvious direction and you drown in pages. On a healthy, congestion-controlled RoCE fabric, PFC > 0 and "any ECN mark" fire constantly — ECN marking is the DCQCN loop working, not breaking. Absolute thresholds on a self-regulating system are a noise generator.

After this page, you'll be able to

1. Reject absolute thresholds for self-regulating signals and alert on anomaly instead.
2. Build alerts three ways — baseline-relative, rate-of-change, and multi-signal correlate-before-paging.
3. Define SLIs and SLOs for a training fabric and run an error budget.
4. Use multi-window burn-rate alerting to be both fast on real fires and quiet on blips.
5. Deploy synthetic probes and a three-tier dashboard hierarchy.

---

Alert on anomalies, not absolutes

Three patterns turn a noisy counter into a signal worth paging on.

1. Baseline-relative

Compare to the metric's own recent history, not to zero.

"TC3 PFC on this port is 10× its 7-day median" — not "PFC > 0."

This is why the cold retention tier exists: you can't say "10× the 7-day median" if you only kept 48 hours. Recording rules pre-compute the baseline so the alert query stays cheap.

2. Rate-of-change

Many fabric and NIC error counters are lifetime totals — a non-zero packet_seq_err might be from a cable event three weeks ago. Alert on the derivative, not the level:

packet_seq_err is increasing right now — not packet_seq_err > 0.

Language Bridge

Lifetime counter→A number that only ever goes up since the NIC booted. Its absolute value is almost meaningless for alerting; what matters is the slope — rate() in PromQL — because a rising slope means the error is happening now.

3. Multi-signal — correlate before paging

The highest-quality alerts fire on a combination that can only mean one thing. Any single signal below is noise; together they are an actionable fabric problem:

step-time p99 ↑ AND SM_ACTIVE ↓ AND fabric ECN/PFC ↑ — together.

This is the entire point of the join keys and the walk-down: the pager should fire when the job is measurably hurting and the fabric is demonstrably the reason — not on either fact alone.

---

SLIs and SLOs for a training fabric

An SLI is a measured indicator of health; an SLO is the target you hold it to. Borrow SRE discipline, but pick indicators that map to training goodput, not raw plumbing.

SLI (what you measure)	Example SLO (the target)
Effective training throughput (tokens/s or step-time p99)	within X% of the known-good baseline
MFU (model FLOPs utilization)	≥ your hardware's realistic floor
Job availability (uptime not lost to fabric faults)	≥ 99.x% of scheduled GPU-hours
Collective health (busbw vs line-rate; straggler variance)	busbw ≥ target; max/median rank time ≤ 1.x
Fabric drop-freeness (no-drop is the RoCE contract)	out-of-sequence / drops ≈ 0 sustained

Language Bridge

SLI / SLO→SLI = the number you watch (e.g. step-time p99). SLO = the promise you make about it (e.g. within 5% of baseline, 99.9% of the time). For a training fabric the north-star SLI is goodput/MFU, not link utilization — the job's health, not the wire's.

The north-star remains MFU — see MFU & diagnosis. Everything else is a leading indicator that MFU is about to move.

---

Error budgets & burn-rate alerting

An SLO of 99.9% grants a 0.1% error budget. That budget converts a fuzzy "is it bad?" into a rate question: how fast are we spending it?

Multi-window, multi-burn-rate alerting is the standard that keeps you both responsive and quiet:

Burn rate	Windows (short and long)	Meaning	Action
Fast (~14×)	5 min and 1 h both breaching	budget gone in hours	🔴 page now
Slow (~3×)	30 min and 6 h both breaching	steady erosion	🟡 ticket

Requiring a short and a long window together is what filters blips: a 90-second microburst trips the short window but not the long one, so it never pages.

Language Bridge

Burn-rate alerting→Instead of 'error rate > X', you alert on how fast you're consuming the error budget, measured over two windows at once. Fast burn on both a short and long window = real fire, page. This is what stops single microbursts from paging while still catching sustained degradation quickly.

---

Synthetic probes — don't wait for a real job to fail

Passive telemetry tells you about jobs that are running. Active probes catch problems on idle links before the next big job lands on them:

• perftest (ib_write_bw / ib_send_lat) between rank pairs — verifies line-rate bandwidth and latency on the actual QP path.
• nccl-tests (all-reduce / all-to-all) on a spare slice — the closest synthetic to real collective behaviour; catches a degraded optic or a mis-tuned queue that raw bandwidth tests miss.
• Schedule them on idle capacity and alert on regression vs baseline, the same anomaly-relative way.

A link that quietly fell to half-bandwidth is invisible until a job hits it — a periodic probe turns that into a page while the rack is still drained.

---

Dashboards: three tiers, matched to who's looking

Dashboards are for exploration and confirmation, not detection — alerts detect, dashboards explain. Structure them top-down so the walk-down has somewhere to land:

Tier	Audience	Answers
① Fleet / SLO	leadership, capacity	Are SLOs green? How much budget is left? Overall MFU trend.
② Fabric / pod	network on-call	Per-leaf ECN/PFC, drop hotspots, top-talker ports, watermark heat.
③ Job / node drill-down	ML + platform on-call	Per-rank step-time, DCGM per-GPU, NCCL busbw, the walk-down view.

The tiers mirror the five-layer symptom model: you start at Tier 1 (the job is slow) and descend to Tier 3 to find the single optic.

---

💡 What you should remember

#		Concept	Why it matters
1	🚫	Anomaly, not absolute	PFC > 0 is the loop working; alert on 10× baseline instead.
2	📈	Rate-of-change	Error counters are lifetime totals — alert on the slope, not the level.
3	🔗	Correlate before paging	Page on step-time↑ + SM_ACTIVE↓ + fabric↑ together; any one alone is noise.
4	🎯	SLO on goodput, not utilization	Hold the fabric to MFU/step-time targets, not link %.
5	🔥	Multi-window burn-rate	Short+long windows together = fast on fires, silent on blips.
6	🛰️	Synthetic probes	perftest/nccl-tests catch a half-bandwidth link before a job does.

---

Next: Advanced Fabric Visibility → — INT, microburst capture, and vendor deep-telemetry for when per-port counters aren't enough.