Setup:

8 GPUs in a logical ring. Sequence of 1M tokens split into 8 chunks of 125K each.

GPU i holds Q_i (125K × d), K_i (125K × d), V_i (125K × d) — its local chunk.

Goal: each Q_i must attend to all 8 K, V chunks, eventually producing attention output for its 125K tokens.

Round 0 (local):

GPU i computes Q_i · K_i^T (its local diagonal block of the attention matrix).

Apply softmax (with running statistics — the online softmax from §12.3, since later rounds will contribute more).

Compute Q_i · K_i^T → weights → weights · V_i → partial output O_i.

Maintain (m, ℓ, O) running state per query row.

Rounds 1 through 7:

Step 1: GPU i sends its K_i, V_i to GPU (i+1) mod 8. Receives K_(i-1), V_(i-1) from previous GPU.

Step 2: Compute Q_i · K_(i-1)^T → partial scores. Update the running (m, ℓ, O) with this contribution (using the §12.3 online softmax recurrence).

Repeat: rotate K, V chunks around the ring. After 7 rotations, every GPU has computed attention from its Q to every K, V chunk in the sequence.

Final:

Each GPU i has O_i = attention output for its 125K queries. Divide by ℓ to normalise.

Output Y_i = O_i / ℓ_i. Each GPU holds its slice of the output sequence.

Memory per GPU during computation:

• Own Q, K, V chunks: 3 × 125K × d × 2 bytes = ~6 GB (for d=8192).
• One additional K, V pair being rotated: 2 × 125K × d × 2 = ~4 GB.
• Running state (m, ℓ, O): 125K × d × 2 bytes ≈ 2 GB.
• Total: ~12 GB per GPU for attention. Plus model weights, etc.

This is much less than the 164 GB per layer that would be needed without CP.

Communication cost:

Per layer per direction: ~125K × d × 2 bytes × 8 rotations = ~16 GB of K, V traffic.

Per layer (forward + backward): ~32 GB.

At NVLink 900 GB/s: ~35 ms per layer. Per step (80 layers): ~3 seconds.

That’s significant overhead. Modern implementations overlap K, V transfer with compute, reducing effective overhead to ~30%.

For 1M-context training, this is the price paid. CP is the only way to fit; communication is the price.

The challenge:

Each GPU sees only 1/CP of the K, V tensors at a time. It needs to compute attention scores and apply softmax — but softmax requires the maximum and sum across the FULL sequence of scores.

Naive approach: gather all K, V to one GPU, compute softmax, scatter back. Memory infeasible (defeats CP).

Alternative: each GPU computes softmax just over its local block. But then attention is no longer correct (uses local max instead of global).

The online softmax fix:

Each GPU maintains running (m, ℓ, O) per query row across the sequence of K, V chunks it processes:

• m: running max of scores seen so far for this row.
• ℓ: running sum-of-exp normalized to m.
• O: running weighted sum of V.

When a new K, V chunk arrives from the ring:

1. Compute partial scores Q_local · K_chunk^T.

2. Get the local max m_chunk and local sum ℓ_chunk over the chunk.

3. Apply the online recurrence:

m_new = max(m, m_chunk)

α = exp(m - m_new) (rescale factor for old state)

β = exp(m_chunk - m_new) (rescale factor for new contribution)

ℓ_new = α · ℓ + β · ℓ_chunk

O_new = α · O + β · (P_chunk · V_chunk)

4. Update m, ℓ, O.

After processing all CP_size chunks: the running (m, ℓ, O) is mathematically identical to what you’d compute with all K, V in one place.

Final attention output: Y = O / ℓ.

What would fail without online softmax:

You’d compute softmax over each K, V chunk in isolation. The weights would be normalised to the LOCAL maximum of that chunk, not the global maximum. Combining them after the fact would require careful re-normalisation that’s mathematically equivalent to the online recurrence.

Without the online recurrence, you’d need to:

• Either materialise ALL scores then softmax (memory infeasible).
• Or use approximations (correctness lost).

The online softmax is the bridge that makes “compute softmax in pieces” mathematically equivalent to “compute softmax over the whole.” Without it, Ring Attention can’t be exact.

The full assembly:

Ring Attention is: Context Parallelism (sequence split) + Ring rotation (K, V transfer pattern) + Online Softmax (mathematical correctness across chunks) + FlashAttention (efficient SRAM-staged computation within each chunk).

Each piece comes from a different paper, but they compose cleanly because the online softmax recurrence is associative (Ch.12 §3). The Ch.12 work — what felt like a small efficiency trick — became the structural enabler of training models with 1M+ token contexts.

1. Hardware budget:

• 100B model in mixed-precision needs ~1.6 TB total state.
• At Chinchilla-optimal D = 20·N = 2T tokens → ~1.2·10²³ FLOPs.
• On H100 at 50% MFU: ~50 PFLOPs/s sustained → ~14 weeks on 1000 H100s.
• Budget: ~$10M of compute. + engineering team + data pipeline + storage.

2. Cluster topology:

• 1024 H100s = 128 nodes × 8 GPUs/node.
• NVLink within node (900 GB/s); InfiniBand between (50 GB/s).
• Need to map parallelism to bandwidth hierarchy.

3. Parallelism configuration:

• TP = 8 (intra-node, splits each matmul). Communication frequent → needs NVLink.
• PP = 4 (split layers across 4 node-groups). Communication per-microbatch → can use InfiniBand.
• DP = 1024 / (8 · 4) = 32 (data parallel replicas). Per-step all-reduce → InfiniBand acceptable.
• Microbatches K = 64-128 for good PP efficiency.
• Effective global batch: 32 × 64 microbatches × 32K tokens = 64M tokens per step.

4. Memory optimisation:

• FSDP / ZeRO Stage 1 within each TP × PP unit (saves Adam state).
• Activation checkpointing (recompute during backward instead of storing).
• bf16 training with fp32 master copy.

5. Data pipeline:

• 2T tokens of high-quality data — FineWeb + curated additions (Ch.16).
• Tokenization, dedup (MinHash), quality classification.
• Streaming dataloader: shards across DP replicas; no full data ever loaded.

6. Optimization recipe:

• AdamW with β1=0.9, β2=0.95, weight decay 0.1.
• Linear warmup (1K steps) → cosine decay over remaining steps.
• Peak LR ~1.5e-4 for 100B model.
• Gradient clipping at 1.0.

7. Monitoring and recovery:

• Checkpoint every ~30-60 minutes (state too big to checkpoint more often).
• Monitor loss curves, gradient norms, learning rate.
• Have plan for OOM / NCCL hangs / GPU failure (~5-10 GPU failures per week at this scale).
• Use a tool like W&B or Tensorboard for distributed metrics.

8. Hyperparameters at scale:

• Maximum learning rate often empirically derived from smaller-scale runs (Hoffmann scaling).
• Warmup length proportional to total steps.
• RoPE base scaling for long contexts.

9. Inference plan:

• At end of training, you have a 100B fp16 model = 200 GB.
• For serving: quantize to int8 or int4 for cheaper deployment.
• Continuous batching, PagedAttention, speculative decoding.
• Or distil to a smaller model for cheaper inference.

10. The whole picture:

From the math up through the systems: every layer of the stack matters. The 100B-token training is bottlenecked by communication primitives (Ch.21), memory hierarchy (Ch.21), framework efficiency (Ch.22), inference economics (Ch.23), parallelism choices (Ch.24), and quantization for deployment (Ch.25).

The trajectory of LLM scale (1B → 7B → 70B → 671B) has been enabled by systems-engineering breakthroughs as much as by ML breakthroughs. FlashAttention enabled long context; ZeRO enabled large models; PagedAttention enabled serving economics; speculative decoding enabled latency; quantization enabled deployment. Each was an enabler that unlocked the next scale.

This is the world a senior systems engineer is moving INTO. The math wasn’t optional; it was the foundation. The systems engineering is where the work happens.