DESIGN 02 / 05 — BARE METAL + SLURM + RoCE (HIGH-LEVEL HPC ARCHITECTURE)
Best for: Dedicated research clusters, traditional HPC workloads [cite: Overview]
4.ML/AI APPLICATIONS
& TRAINING[cite: L7, L12-L13, L15]
💻
[cite: L7]
Load Training Code
System-wide NCCL library
No containers — native OS
→
Native Training Code
GPU
GPU
→
🧮
[cite: L12]
Training Code Calls NCCL
Backward pass triggers
NCCL all-reduce
→
🔍
[cite: L12.5]
NCCL Rank Discovery
Via SLURM_PROCID
and SLURM_NTASKS
→
🌐
[cite: L13]
Initialize Comm Topology
Ring or Tree all-reduce
topology across nodes
→
✅
[cite: L15]
Training Job Completes
Code exits, resource
freed, checkpoint saved
⬇
3.SLURM JOB
SCHEDULER[cite: L6, L6.5, L9, L10]
📦
[cite: L6]
Install Slurm
slurmctld + slurmd
via packages or source
→
📊
[cite: L6.5]
Configure Slurm GRES
GPU resource tracking
Gres=gpu:a100:8
→
🖥️
[cite: L6]
Central Controller
(slurmctld Master)
slurmd
slurmd
→
📋
[cite: L9]
Create SBATCH Job Scripts
Resource requirements
GPU count, nodes, time
→
📌
[cite: L10]
Scheduler Allocates Nodes
Exclusive GPU + node
allocation per job
⬇
2.NETWORKING LAYER
(RoCE FABRIC)[cite: L5, L6, L11, L13.5, L14]
🖧
[cite: L5]
Configure Physical NIC IPs
800G NICs with IPs
No SR-IOV needed
→
🔀
[cite: L6]
Ethernet Switches
Leaf-Spine Topology
PFC + ECN + QoS
⟶
🔀
[cite: L11]
Allocate Physical Paths
Direct paths between
allocated nodes & GPUs
⟶
🔗
[cite: L13.5]
NCCL Creates RDMA Queue Pairs
Direct between
physical NICs
⟶
800 Gbps
800 Gbps
RDMA
⚡ DIRECT RDMA PATH
⬇
1.INFRASTRUCTURE
& BARE METAL[cite: L1-L3, L8]
🐧
[cite: L1]
Install Linux OS
Ubuntu 22.04 LTS
Directly on hardware
→
B300 Servers
NVIDIA
B300
→
🎮
[cite: L3]
Install NVIDIA Drivers & CUDA
System-wide CUDA
All 8 GPUs per node
→
🔌
[cite: L2]
Enable Physical NICs via BIOS
Single large physical NIC
No virtual functions
STORAGE NETWORK
[cite: L8]
🗂 Shared Storage
NFS / Lustre
Object Storage
Checkpoints & Datasets
DESIGN 02 / 05 · BARE METAL + SLURM + RoCE · HPC RoCE Design Patterns