11.1 What a Communication Library Is
The software layer that moves tensors between GPUs — what it is, where it sits in the stack, and why you can't just run gradient sync over a TCP socket.
11.2 The Library Landscape
The four kinds of communication library — GPU collective, MPI, transport-abstraction, framework-native — and the whole AI comms stack mapped onto the OSI layers you already know.
11.3 Collective Operations
Collectives are the core API of every communication library — AllReduce, Broadcast, AllGather, ReduceScatter, Scatter, Gather, Barrier — each mapped to a routing protocol you already run.
11.4 The Collective That Runs Every Step
AllReduce on the wire — ring mechanics, why it's bandwidth-optimal, the simultaneous-flow pattern your fabric has to handle, and the other collectives you'll meet at scale.
11.5 Inside the Libraries
A closer look at the three libraries you'll actually tune — NCCL's topology detection and env-var knobs, UCX's transport auto-selection, and SHARP's in-network AllReduce.
11.6 Topology Awareness
How NCCL detects the interconnect hierarchy — NVLink, PCIe, InfiniBand — and treats it like a routing table with administrative distances, then builds rings that respect the fat-tree.
11.7 Parallelism Strategies
Data, tensor, pipeline, expert, sequence and ZeRO parallelism — how a job is split decides the collective it runs and the traffic matrix your fabric sees.