Advanced Fabric Visibility

Everything so far — RoCEv2 counters, DCGM, the walk-down — is built on per-port, per-priority counters streamed at 1–5 s. That baseline solves the overwhelming majority of incidents. This page is the tier you reach for when it doesn't: when the counters say "TC3 buffer overran on et33" but you need to know which flow, on what sub-second timescale, and exactly which packet the switch dropped and why.

These are Nice-tier / reach-for-during-incident tools (see the inventory): high cost, specialist value. Deploy the baseline first; add these where the blast radius justifies the spend.

After this page, you'll be able to

1. Explain in-band network telemetry (INT) and the postcard variant, and what they add over counters.
2. Read microburst and watermark telemetry to catch congestion that 5 s counters average away.
3. Use mirror-on-drop and deep-buffer analytics to see the exact dropped packet and why.
4. Map the vendor deep-telemetry platforms — WJH, BroadView, CloudVision, UFM — to what they expose.
5. Decide when advanced visibility is worth it and when per-port counters are enough.

---

In-band network telemetry (INT)

Counters tell you a queue was deep. INT tells you the path and the per-hop delay of an individual packet. As a packet traverses the fabric, each switch appends metadata — ingress/egress port, queue occupancy, hop latency, timestamp — so the collector reconstructs the exact route and where the time went.

Two flavours:

• Classic INT — metadata is carried in the packet and stripped at the last hop. Highest fidelity, adds header overhead.
• Postcard / INT-MD (deferred) — each hop emits a small separate "postcard" report to the collector instead of growing the packet. Lower data-plane overhead, easier to deploy incrementally.

What it buys you: per-hop queueing delay and exact ECMP path for a specific flow — the answers per-port counters cannot give, because a port counter is the sum over every flow through it.

Language Bridge

INT (In-band Network Telemetry)→The switch data plane annotates real packets (or emits postcards) with per-hop port, queue depth, and latency. Instead of 'this port was congested on average', you get 'this flow took 40 µs at the spine because that specific queue was 800 KB deep' — flow-level, hop-level truth.

---

Microburst & watermark deep-dive

The single most important thing 1–5 s counters hide is the microburst: incast that fills and drains a buffer in sub-millisecond time. Averaged over a 5 s scrape it looks like nothing; on the wire it caused a PFC pause or a tail-drop.

Two ways to see through the averaging:

• High-frequency gNMI on the queue depth / watermark leaves — sample fast enough to catch the spike, accepting the cardinality cost on a targeted set of ports.
• Hardware watermark counters — the ASIC records the peak occupancy between reads, so even a 5 s scrape captures that a burst touched the ceiling.

The interpretation from the RoCE telemetry page is the tell:

Rising peak watermark with a flat average = bursty incast — the classic all-to-all / AllReduce signature. Peaks approaching xoff mean PFC is about to fire.

This is why peak and average watermarks are different signals: average is sustained load, peak is the microburst the average erased.

---

Deep-buffer analytics & mirror-on-drop

When a lossless fabric drops a packet, the contract is broken and you need the why, not just the count.

• Mirror-on-drop — the switch copies the dropped packet (or its headers) to a collector, tagged with the drop reason (buffer/WRED, ACL, L2/L3 miss). You get the actual 5-tuple that got tail-dropped, not just an incremented discard counter.
• Deep-buffer occupancy analytics — for switches with large shared buffers, per-port/per-priority-group occupancy over time shows which traffic class is consuming headroom and how close it runs to the xoff threshold before PFC engages.

Mirror-on-drop is the packet-capture-grade version of what event-based drop telemetry (below) already gives you as structured events. Reach for the mirror when you need the payload/headers; the event stream is usually enough to place blame.

---

Vendor deep-telemetry platforms

Each major fabric vendor ships an event-based, richer-than-counters telemetry system. They mostly answer the same question — "what just happened to this packet?" — in vendor-specific form.

Platform	Vendor / stack	What it exposes
WJH (What Just Happened)	NVIDIA/Mellanox Spectrum	event-based drop visibility with a reason code per dropped packet (buffer/WRED, ACL, L2/L3)
BroadView	Broadcom ASIC	buffer statistics, congestion drop counters, microburst detection off the ASIC
CloudVision	Arista (EOS)	streaming state, LANZ microburst/queue-depth capture, historical fabric telemetry
UFM	NVIDIA InfiniBand / fabric mgmt	fabric-wide health, congestion, and topology analytics (IB-centric; RoCE fabric mgmt overlaps)
gNMI + OpenConfig	vendor-neutral	streaming queue/PFC/ECN/watermark state — the portable baseline under all of the above

The one that earns its keep first is WJH (or its per-vendor equivalent): it turns "a packet was lost somewhere" into "port et33 tail-dropped a TC3 packet due to buffer exhaustion at 14:03:07." That single line is, per the draft, the most useful line in a RoCE incident — and it's what the walk-down's final step depends on.

note

Prefer the vendor-neutral gNMI/OpenConfig streams for anything you can get portably (queue stats, PFC/ECN, watermarks), and use the vendor platform for the event-based drop reasons and microburst capture it uniquely provides. That keeps most of your pipeline vendor-independent while still getting the deep signals.

---

When is advanced visibility worth it?

These tools cost engineering time, data-plane overhead, and pipeline cardinality. Apply the same prioritization principle from the signal taxonomy — blast radius × likelihood × blindness — and add them deliberately:

Situation	Reach for
Recurring drops you can't localize with counters + WJH	INT / postcard for per-hop path + delay
Intermittent PFC with flat average utilization	Microburst / peak-watermark high-freq capture
"Lossless" fabric losing packets, need the exact packet	Mirror-on-drop
Need the drop reason per packet (almost always)	WJH / BroadView / CloudVision event stream
Everyday operations	Per-port counters — you already have enough

The honest default: per-port, per-priority counters plus WJH-class drop reasons solve the vast majority of incidents. INT, mirror-on-drop, and high-frequency capture are the specialist instruments you switch on for the residual hard cases — not an always-on layer.

---

💡 What you should remember

#		Concept	Why it matters
1	🛰️	INT / postcard	Per-hop delay and exact ECMP path for one flow — what port counters can't give.
2	⚡	Microbursts hide in averages	Sub-ms incast is invisible at 5 s; peak watermark & high-freq gNMI reveal it.
3	🪞	Mirror-on-drop	Copies the dropped packet + reason — packet-grade proof on a lossless fabric.
4	🏷️	WJH-class drop reasons	"port et33 tail-dropped TC3 at 14:03:07" — the most useful line in a RoCE incident.
5	⚖️	Specialist, not always-on	Counters + drop reasons solve most incidents; add INT/capture by blast radius.

---

Next: Reference Architecture & Maturity → — assembling everything into one end-to-end design, an instrumentation checklist, the anti-patterns to avoid, and a maturity model to grade yourself against.