The Synthetic Data Pipeline: Modular and Goal-Oriented

Poolside views synthetic data not as a replacement for organic data, but as a mechanism to extract implicit rationale and structure. Their approach relies on a modular architecture composed of six components: seeds, primary inputs, metadata, secondary inputs, generator functions (LLMs/agents), and validation filters.

Key strategies include:

  • Task Decomposition: If a task is too complex, the model will fail to learn or collapse into bias. They break tasks into multi-stage workflows (e.g., generating a novel by first defining settings and characters before writing chapters).
  • Cross-Domain Porting: Translating data across modes, such as converting math problems into code, to expose the model to new reasoning patterns.
  • Multi-Turn Iteration: Utilizing agentic loops where a judge and an evolver iterate on a task to improve quality.
  • Orchestration: Using a "Hive" infrastructure to manage queues of agents, allowing for dynamic instruction tuning and global supervision to police generations.

Scaling Challenges: Trust Nothing

As models scale to hundreds of billions of parameters, the team assumes all training infrastructure is fallible. They implement strict verification to catch silent failures that standard monitoring misses:

  • Replica Hashing: They run multiple replicas of the same model on identical data. If the weight hashes do not match, the training is immediately killed. This has surfaced critical issues like silent data corruption from broken GPUs.
  • Numerical Precision: During the training of Laguna M.1, the model stopped converging due to activations growing too large for BF16 precision during tensor parallel accumulation. Moving this accumulation to FP32 restored convergence.
  • Race Conditions: When adopting FP8 kernels, they encountered silent gradient corruption where ~0.5% of gradients were replaced by random values. This was invisible to standard metrics, highlighting the need for redundant verification even in non-replicated training runs.

Results and Future Direction

By applying these rigorous data and infrastructure improvements, Poolside developed the 118B parameter model, Laguna S. Early evaluations show it outperforming their previous models and competitors like GLM 4.5 Air on agentic coding benchmarks (e.g., SWE-bench Agentless). The team notes that while they currently lag in general knowledge benchmarks like MMLU Pro, this is a deliberate trade-off to prioritize agentic coding performance.