AI Fabric Switches — An Architect's Guide
Start with the network technology, not the switch brand. GPU scale-out networks have two major choices:
| Question | InfiniBand | AI Ethernet |
|---|---|---|
| What carries GPU traffic? | Native InfiniBand RDMA | Usually RoCEv2 over Ethernet/IP |
| Example NVIDIA switch family | Quantum | Spectrum |
| Fabric control plane | Subnet Manager (SM); optional UFM management | Commonly eBGP or other Ethernet routing; depends on design |
| Congestion and flow control | InfiniBand credit-based link flow control and fabric congestion mechanisms | ECN, endpoint rate control, sometimes PFC; platform-dependent |
| Typical architectural reason to choose | Integrated HPC/AI fabric and InfiniBand ecosystem | Ethernet interoperability, operational familiarity and vendor choices |
400G or 800G is a port rate. Every capacity table below uses one-direction aggregate port bandwidth: 64 × 800G = 51.2 Tb/s. Some vendors double this when advertising full-duplex switching capacity (102.4 Tb/s for the same ports). Do not compare one-direction and full-duplex figures as though they were equivalent. A physical OSFP cage can expose more than one logical port. Never size a fabric from the number of cages alone.
1. NVIDIA portfolio: InfiniBand Quantum versus Ethernet Spectrum
NVIDIA InfiniBand models
| Generation | Representative switch | Logical port configuration | Nominal aggregate port bandwidth | Architect's takeaway |
|---|---|---|---|---|
| HDR / Quantum | QM8700 / QM8790 | 40 × 200G | 8 Tb/s | Earlier-generation InfiniBand |
| NDR / Quantum-2 | QM9700 / QM9790 | 64 × 400G via 32 OSFP cages | 25.6 Tb/s | NDR fabric; verify management variant |
| XDR / Quantum-3 | Q3200-RA | Two independent 36 × 800G switches in one chassis | 28.8 Tb/s per independent switch | Do not design it as a single 72-port forwarding domain |
| XDR / Quantum-3 | Q3400-RA | 144 × 800G via 72 OSFP cages | 115.2 Tb/s | High-radix XDR system |
What Quantum brings: native InfiniBand, virtual lanes, credit-based flow control, fabric-managed routing, and supported adaptive routing and collective-offload capabilities. The Subnet Manager is fundamental. Validate UFM/SM redundancy, SHARP support, firmware, cables and NIC generations before building the BOM.
References: NVIDIA Quantum-3 system guide · Q3200 dual-switch architecture.
NVIDIA Ethernet models
| Silicon generation | Representative switch | Maximum nominal configuration | Aggregate port bandwidth | Packet buffer | Architect's takeaway |
|---|---|---|---|---|---|
| Spectrum-3 | SN4700 | 32 × 400GbE | 12.8 Tb/s | 64 MB | Earlier 400G Ethernet generation |
| Spectrum-4 | SN5400 | 64 × 400GbE | 25.6 Tb/s | 160 MB | 400G leaf/spine candidate |
| Spectrum-4 | SN5600 | 64 × 800GbE | 51.2 Tb/s | 160 MB | 800G leaf/spine; up to 128 × 400G via supported breakout |
| Spectrum-6 | SN6600-LD | 128 × 800GbE through 64 OSFP cages | 102.4 Tb/s | 256 MB | Liquid-cooled 2RU, pluggable optics |
| Spectrum-6 | SN6600 | 128 × 800GbE through 64 OSFP cages | 102.4 Tb/s | Model manual: check revision | Air-cooled 3RU, pluggable optics |
| Spectrum-6 | SN6810-LD | 128 × 800GbE via MMC-16 interfaces | 102.4 Tb/s | 256 MB | Liquid-cooled single-ASIC co-packaged optics |
| Spectrum-6 | SN6800-LD | Up to 2,048 × 200GbE | 409.6 Tb/s (four ASICs) | 4 × 256 MB | Liquid-cooled 5RU multi-ASIC co-packaged optics |
The SN5400 has 64 QSFP-DD cages; the SN5600 has 64 OSFP cages. They have different breakout possibilities. The figures above come from NVIDIA's SN5000 datasheet. NVIDIA's Spectrum-6 hardware manual documents these models, including physical configurations and ordering SKUs. 102.4 Tb/s is the capacity of one Spectrum-6 ASIC; SN6800-LD combines four ASICs. Spectrum-6 offers both pluggable and co-packaged optical designs and is associated with the ConnectX-9 generation. Published documentation and ordering SKUs do not establish that every configuration is immediately available in every region; confirm lead times.
A switch alone is not the entire Spectrum-X platform: its performance features depend on compatible NICs, transport and software.
How Spectrum-X packet spraying works
Think of two large RDMA flows hashing to one congested uplink, while other equal-cost uplinks are underused.
- Traditional ECMP: each flow hashes to a selected next hop. A collision can persist even with idle capacity elsewhere.
- Spectrum-X adaptive routing: in supported configurations, the Spectrum switch can select a less-congested eligible egress path packet by packet.
- Packet-ordering challenge: different path delays can cause packets from one flow to arrive out of order.
- SuperNIC coordination: the supported receiving NVIDIA SuperNIC/transport places out-of-order data correctly, making the reordering transparent to the application.
- Remaining bottleneck: when all paths converge on one saturated receiver, adaptive routing cannot fix incast. Congestion control must regulate sending rates.
Packet spraying is not a universal Ethernet feature. Other vendors may offer dynamic ECMP, flowlet switching or different adaptive load-balancing mechanisms. Those are not automatically equivalent to per-packet same-flow spraying with endpoint reorder support.
NVIDIA describes the switch/SuperNIC coordination in its Spectrum-X technical explanation and platform overview.
2. AI Ethernet: other vendors, real models and capacity
The same 51.2 Tb/s switch class can be sold by different vendors with different software, silicon, buffers and support models. Compare the actual SKU, not just the vendor name. All bandwidth figures follow the one-direction convention explained at the top.
| Vendor | Model | Maximum ports | Port bandwidth | Packet buffer | OS / silicon | Architectural role |
|---|---|---|---|---|---|---|
| Arista | 7060X6-32PE | 32 × 800GbE | 25.6 Tb/s | 84 MB | EOS; merchant silicon | Compact 800G leaf/spine |
| Arista | 7060X6-64PE / -64PE-B | 64 × 800GbE or 128 × 400GbE | 51.2 Tb/s | 165 MB | EOS; merchant silicon | High-radix fixed leaf/spine |
| Arista | 7700R4C-38PE | 18 × 800G host + 20 × 800G fabric ports | 14.4 Tb/s host-facing | 16 GB | EOS; distributed Etherlink | Deep-buffer distributed leaf; fabric ports are not extra host ports |
| Arista | 7720R4-128PE | 128 × 800G fabric ports | 102.4 Tb/s fabric | Not specified as a host buffer | EOS; distributed Etherlink | Distributed spine, not a conventional 128-host-port ToR |
| Arista | 7800R4 with 7800R4C-36PE line cards | 36 × 800GbE per line card | 28.8 Tb/s per card | 32 GB per card | EOS; deep-buffer modular | Modular AI spine; chassis totals depend on installed cards |
| Cisco | N9K-C9332D-GX2B | 32 × 400GbE | 12.8 Tb/s | 120 MB | NX-OS; Cloud Scale | 400G leaf/spine |
| Cisco | N9K-C9364D-GX2A | 64 × 400GbE | 25.6 Tb/s | 120 MB | NX-OS; Cloud Scale | Dense 400G leaf/spine |
| Cisco | N9364E-SG2-O / -Q | 64 × 800GbE or 128 × 400GbE | 51.2 Tb/s | 256 MB | NX-OS; Silicon One G200 | 800G leaf/spine; OSFP (-O) or QSFP-DD (-Q) |
| Juniper / HPE | QFX5240-64OD / -64QD | 64 × 800GbE or 128 × 400GbE | 51.2 Tb/s | 165 MB | Junos OS Evolved; Broadcom Tomahawk 5 | Fixed high-radix AI leaf/spine; OSFP or QSFP-DD |
| Juniper / HPE | QFX5241-32OD | 32 × 800GbE or 64 × 400GbE | 25.6 Tb/s | 83 MB | Junos OS Evolved; model-dependent | Smaller fixed 800G leaf/spine |
| Juniper / HPE | QFX5250-64OE-L | 64 × 1.6TbE | 102.4 Tb/s | See model datasheet | Junos OS Evolved; Broadcom Tomahawk 6 | Liquid-cooled next-generation high-radix switch |
| Edgecore | AIS800-64D | 64 × 800GbE or 128 × 400GbE | 51.2 Tb/s | Not publicly specified | Open-networking / qualified NOS | Disaggregated 800G design; confirm SONiC and SAI feature matrix |
Next-generation comparison: NVIDIA Spectrum-6 and Juniper QFX5250 represent 102.4 Tb/s per-switch platforms; Arista's 7720R4-128PE is a 102.4 Tb/s distributed-fabric spine, not an equivalent standalone front-panel leaf. Other vendors' 102.4T products should be added only when a specific published model and configuration can be substantiated. See Juniper QFX5250 hardware overview.
Read this table carefully: 7700R4 distributed-fabric ports, 7800R4 line-card ports, and conventional 7060X6 front-panel ports serve different purposes. Adding them together as if they were all GPU downlinks produces a bad BOM. Similarly, the buffer number for one line card is not the buffer of every switch in that vendor's catalog.
What differs beyond bandwidth?
| Feature | What architects must check | Important distinction |
|---|---|---|
| ECMP | Hash inputs, ECMP width, resilient hashing and polarization | Supported across major routed Ethernet families; does not guarantee elephant-flow balance |
| Dynamic load balancing | Per-flow versus flowlet versus packet-level granularity, congestion signals and software support | Arista, Cisco and Juniper have load-balancing features; they are not automatically identical to Spectrum-X |
| Packet spraying | Whether the system sprays packets within a single RDMA flow, and how receiver ordering is handled | Do not equate Arista distributed fabric traffic spraying with NVIDIA Spectrum-X same-flow adaptive packet spraying |
| Congestion control | ECN thresholds, PFC headroom/watchdog, DCQCN and NIC compatibility | Buffer depth cannot substitute for sender-rate control |
| Telemetry | Queue occupancy, microbursts, ECN marks, PFC duration, FEC errors and per-path utilization | Marketing labels such as “AI-ready” do not prove observability granularity |
| Buffer architecture | On-chip shared memory versus distributed/VOQ/deep-buffer design | Arista has both shallow 7060X6 and deep-buffer 7700R4/7800R4; other vendors also vary by SKU |
Manufacturer specifications: Arista 7060X6 · Arista 7700R4 · Arista 7800R4 · Cisco Nexus 9364E-SG2 · Cisco 400G AI POD models · Juniper QFX5240/QFX5241 · Edgecore AIS800-64D · Cisco GX2 buffer specifications.
3. Deep buffers versus shallow buffers
A switch buffer is a waiting room, not additional link bandwidth. When traffic arrives at an egress port faster than it can leave, a queue builds.
Example: 800 Gb/s offered to a 400 Gb/s output for 100 microseconds produces about 5 MB of excess data. A buffer can absorb that burst, but if the mismatch persists, the queue eventually fills.
| Design choice | Typical examples | Strength | Limitation |
|---|---|---|---|
| Relatively shallow, high-speed on-chip buffering | NVIDIA Spectrum-4 SN5600 (160 MB); many Broadcom Tomahawk-based platforms | Low latency at high radix | Less room for sustained queue growth; depends on fast congestion response and good traffic distribution |
| Deeper buffering | Selected Broadcom Jericho-based modular/routed platforms offered by multiple vendors | More burst absorption for traffic with variable fan-in or longer congestion feedback | More memory does not eliminate persistent oversubscription; can increase queueing delay |
Do not say “NVIDIA is shallow, everyone else is deep.” Arista, Cisco, Juniper and open-networking vendors sell different buffer architectures by model. Nor does a deeper buffer automatically outperform a well-tuned Spectrum-X design on synchronized GPU collectives. The relevant test is job completion time and tail behavior, not memory size alone.
4. How to choose: an architect's workflow
- Define workload and transport: training or inference, largest simultaneous job, GPUs per server, number and speed of network ports, and whether InfiniBand or Ethernet/RoCE is required.
- Set capacity target: calculate server-facing bandwidth and desired oversubscription. For a 1:1 leaf, the aggregate uplink bandwidth must equal aggregate downlink bandwidth.
- Build topology: choose rails/planes, leaf radix, spine count, fault domains, ECMP groups and cabling. A switch with 64 ports cannot simultaneously dedicate all 64 to servers and all 64 to spines.
- Qualify transport behavior: test ECMP collisions, adaptive-routing granularity, reordering tolerance, incast, ECN/PFC where applicable, link failures and fail-slow behavior.
- Validate physical BOM: port breakout, OSFP/QSFP-DD, optics reach, FEC, fiber count, rack power, cooling, spares and lead times.
- Prove operability: upgrade/rollback, telemetry, per-queue counters, automated configuration, fabric-management redundancy and vendor support ownership.
Worked two-tier design: 4,096 GPUs at 400G
Assumptions: one logical 400G fabric port per GPU, 64-cage × 800G leaf/spine switches, supported 2 × 400G breakout on leaf downlinks, 1:1 bandwidth, and one physical 800G link per leaf–spine pair. This is a logical GPU-port model, not a complete eight-GPU server/NIC or rail-based BOM.
| Item | Calculation | Result |
|---|---|---|
| GPU-facing bandwidth | 4,096 × 400G | 1.6384 Pb/s |
| Per leaf: downlinks | 32 OSFP cages × (2 × 400G) | 64 × 400G = 25.6 Tb/s |
| Per leaf: uplinks | 32 OSFP cages × 800G | 32 × 800G = 25.6 Tb/s |
| Leaf count | 4,096 ÷ 64 | 64 leaves |
| Spine count | 32 uplinks/leaf, one to each spine | 32 spines |
| Spine port use | One 800G link from each of 64 leaves | 64 × 800G = 51.2 Tb/s per spine |
| Total fabric links | 64 leaves × 32 spines | 2,048 × 800G |
Why this works: each leaf uses all 64 physical cages: 32 broken out to GPU-facing 400G links, and 32 to the spines at 800G. Every leaf reaches all 32 spines with one link each. Every spine uses its 64 ports to reach all 64 leaves. The result is a symmetric two-tier Clos with 32 equal-cost paths for a given leaf-to-leaf destination, assuming equal routing policy.
What this does not prove: 4,096 GPUs is not a universal maximum for a non-blocking two-tier fabric; it is the maximum for this specific full-mesh port allocation and link-speed assumption. A different switch radix, link speed, oversubscription target, or spine grouping changes the bound. In particular, changing those 800G spine-facing links to 400G does not double the 1:1 GPU ceiling: with the same 32 uplink cages it halves each leaf's uplink bandwidth to 12.8 Tb/s and creates 2:1 oversubscription against its 25.6 Tb/s of downlinks. Recalculate all port and bandwidth budgets before claiming an 8,192-GPU design.
Physical validation: confirm supported OSFP breakout and transceivers, NIC-to-GPU mapping, rail placement, optics reach, power, cooling and fault domains. A 32-way ECMP topology does not guarantee perfectly even traffic distribution under a small number of large RDMA flows.
5. What to demand in a vendor POC
| Test | Why it matters | What to measure |
|---|---|---|
| ECMP hash collision | Reveals stranded capacity | Per-uplink throughput, queue depth, job throughput |
| Adaptive routing / DLB | Tests actual feature granularity | Path selection, fairness, packet reordering |
| Incast | Separates path selection from rate control | ECN, PFC if enabled, queue occupancy, recovery |
| Fail-slow optical link | A link can be UP but degraded | Pre-FEC BER, post-FEC errors, effective throughput |
| Link or spine failure | Tests blast radius and convergence | Recovery time, retransmissions, collective completion |
| Sustained NCCL workload | Validates the system, not a single ASIC | AllReduce/AllGather bandwidth and tail job time |
Architect's rule: shortlist by exact switch SKU + ASIC + NOS release + NIC + firmware + transport, then test the complete stack. Advertised switch throughput, buffer depth and adaptive-routing claims are inputs to the design—not proof of application performance.
Next: Switch Silicon — understand the ASIC choices beneath these platforms.