The Limitation of Public Data

Modern AI models are defined by three scaling axes: more data, more compute, and larger models. While this has driven the deep learning revolution, it relies entirely on public data (Wikipedia, GitHub, arXiv). Consequently, models lack 'depth' in private domains—they cannot learn your specific emails, company meetings, or niche technical skills (like AMD kernels) because these are not part of the public corpus. Training from scratch on private data is impractical, and simply performing next-token prediction on a private corpus leads to 'collapsed generation,' where the model memorizes the data perfectly but loses its ability to generalize or reason.

The Failure of Current Approaches

Researchers have attempted several methods to bridge this gap, but each hits a ceiling:

• KV Compaction: Attempts to compress long contexts into a succinct representation. This is limited by the initial context window and fails to leverage the power of gradient updates.
• On-Policy Distillation: Uses the model to generate synthetic data (often Q&A pairs) to simulate pretraining. While effective, it eventually hits a 'synthetic data wall.' Once the model absorbs the synthetic dataset, adding more compute yields diminishing returns rather than the depth observed in original pretraining.
• Continued Pretraining: Fine-tuning on synthetic data conditioned on the private corpus. This often overwrites previous pretraining knowledge and is difficult to scale, especially when starting from a post-trained model rather than a raw base model.

The Path to Recursive Self-Improvement

The goal is to move from static training to a process that mimics the success of AlphaGo, where the model's improvement makes its own training tasks progressively harder. Instead of a fixed dataset, the system must recursively generate better data as the model's internal representation of the domain (the 'value function') deepens. This approach treats the data budget as dynamic rather than fixed, allowing for the application of compute to deepen the model's understanding of a specific context over time, rather than simply memorizing it.