The Shift from Web Mirror to Reasoning Prior

The traditional concept of a base model—a massive, uncurated scrape of the internet meant to reflect human knowledge—is obsolete. In the GPT-3 era, pre-training on raw web text (like Common Crawl) accounted for ~85% of the data mix, with reinforcement learning (RL) serving merely as a final "cherry on top." Today, the base model's primary function is to provide a robust prior for downstream RL. As models shift toward reasoning and agentic behaviors, the data mix has pivoted heavily toward code, STEM-focused content, and synthetic reasoning traces.

Integrating Post-Training Data Early

Modern training recipes, such as those seen in Nemotron-3 Ultra, demonstrate a trend of pulling post-training data (like SFT chat templates) into the pre-training phase. This "mid-training" approach helps the model learn the structure of downstream tasks earlier, which is critical for solving stability issues in Mixture-of-Experts (MoE) architectures. By exposing the model to the expected distribution of RL and long-context agentic traces during pre-training, developers can avoid the massive load-balancing imbalances that occur when a model encounters a radically different data distribution during post-training.

The Role of Synthetic Data and RL

Synthetic data has moved from a niche experiment to a core component of high-performance training. Techniques like rephrasing seed data allow models to see information in multiple ways, effectively cleaning the data and shaping the model's representations toward desired task formats.

As RL compute budgets grow—sometimes rivaling or exceeding supervised learning budgets—the goal of supervised learning has changed. It is no longer about building a general-purpose knowledge base, but about providing the "atomic skills" the model needs to compose complex behaviors during RL. The base model must now be "warmed up" to the specific shapes of reasoning traces and test-time compute schemes, ensuring the model is prepared to explore and extrapolate effectively once it enters the RL phase.