DESIGN 04 / 05 — BARE METAL + MPI + RoCE (HIGH-LEVEL HPC ARCHITECTURE)
Best for: HPC researchers, scientific computing, legacy MPI workloads [cite: Overview]
4.ML/AI APPLICATIONS
& TRAINING[cite: L9, L12-L14, L15]
🐍
[cite: L9]
MPI Training Code
PyTorch MPI backend
or pure mpi4py
→
Native MPI Process
GPU
GPU
→
🔄
[cite: L14]
MPI_Allreduce
Gradient sync across
all nodes via RDMA
→
🔍
[cite: L12]
MPI Rank Discovery
COMM_WORLD.Get_rank()
MPI assigns 0 to N-1
→
⚡
[cite: L13]
CUDA-Aware MPI Transfer
GPU memory direct
No CPU copy needed
→
✅
[cite: L15]
Training Completes
All MPI processes exit
Manual cleanup + save
⬇
3.MPI JOB
LAUNCHER[cite: L4, L10, L11, L12, L13]
🔑
[cite: L4]
SSH Passwordless Setup
mpirun uses SSH to
launch on remote nodes
→
📋
[cite: L10]
Create MPI Hostfile
node01 slots=8
node02 slots=8
→
🚀
[cite: L11]
mpirun Launch Command
mpirun -np 32
--hostfile hostfile
--mca pml ucx
python train.py
→
🌐
[cite: L12]
MPI COMM_WORLD Init
Each process gets rank
World size = total GPUs
→
📡
[cite: L13]
UCX Transport Selection
Auto-selects RDMA
over RoCE for large msgs
⬇
2.NETWORKING LAYER
(RoCE FABRIC)[cite: L6, L7, L13, L14]
⚙️
[cite: L7]
Install UCX Library
Unified Communication X
Enables RDMA for MPI
→
🔀
[cite: L6]
Ethernet Switches
Leaf-Spine
PFC + ECN + QoS
⟶
🖧
[cite: L5]
Physical NIC Paths
IPs + /etc/hosts entries
for all node hostnames
⟶
🔗
[cite: L13]
UCX Selects RDMA via RoCE
UCX_TLS=rc,ud,sm
Physical 800G NICs
⟶
800 Gbps
800 Gbps
RDMA
⚡ MPI + UCX + RDMA PATH
⬇
1.INFRASTRUCTURE
& BARE METAL[cite: L1-L5, L8]
🐧
[cite: L1]
Install Linux OS
Ubuntu / RHEL
Bare metal, no containers
→
B300 Servers
→
🎮
[cite: L2]
Install NVIDIA Drivers & CUDA
System-wide CUDA
8 GPUs per server
→
📦
[cite: L3]
Install OpenMPI / IntelMPI
apt install openmpi-bin
mpirun --version
→
🔌
[cite: L5]
Configure Physical NICs
800G NICs + IPs
No SR-IOV, no VFs
STORAGE NETWORK
[cite: L8]
🗂 Shared Storage
NFS / Lustre
Checkpoints & Datasets
DESIGN 04 / 05 · HPC RoCE Design Patterns