The Bottleneck of Tightly Coupled RL

Standard Reinforcement Learning (RL) post-training loops require the trainer and the rollout fleet to reside in the same cluster to maintain high-speed weight synchronization via RDMA. This creates a "cathedral" architecture where the rollout fleet is constrained by the trainer's physical location and GPU availability. Because RL requires four things simultaneously—sufficient GPU count, regional proximity, fast fabric, and immediate availability—it is notoriously difficult to scale. The core problem is that full model checkpoints (often ~500 GB) are treated as the unit of synchronization, making cross-datacenter updates impossible due to latency.

The "Adam Absorption" Mechanism

The key insight is that while master weights in FP32 are dense and constantly changing, the weights visible to the rollout engine (typically in BF16, FP8, or INT4) remain remarkably stable. This occurs due to the interaction between the Adam optimizer and finite precision:

  • The Floor: In BF16, the rounding boundary (the distance between representable values) is roughly $\theta/256$.
  • The Push: The Adam update step is typically on the order of the learning rate, which at post-training scales is often 1,000x smaller than the rounding boundary.

Because the "push" (update) is smaller than the "floor" (rounding boundary), the rollout engine's view of the weights does not change for over 99% of parameters. This is not gradient sparsity—gradients are dense—but rather "Adam absorption," where small updates are effectively swallowed by the precision limits of the serving format.

Implementing Stitch for Global Elasticity

By treating the weight update as a lossless patch (a diff of changed weights and metadata) rather than a full checkpoint, the synchronization payload shrinks from ~500 GB to ~500 MB. This allows for a "bulletin board" architecture:

  1. Trainer: Publishes immutable weight versions to a shared store.
  2. Sidecar: A sidecar process on the rollout engines makes them "version-aware." It checks if the engine is up-to-date, applies missing patches if behind, or returns a "not ready" status if the gap is too large.
  3. Elasticity: Rollout fleets can now be scattered across different regions and cloud providers, using whatever capacity is available.

Modal’s implementation of this, called Stitch, enables this framework-agnostic, async-first approach. It transforms scattered inference capacity into a single, elastic rollout fleet, decoupling the training compute from the rollout compute without sacrificing bitwise accuracy in the served model.