The Data Gap in AI Engineering

Modern AI agents excel at application-layer tasks but struggle with infrastructure-level reasoning. Current benchmarks (e.g., SWE-bench) rely on isolated codebases where an agent performs a task over 50–100 turns to produce a code diff. This approach ignores the reality of software engineering: managing distributed nodes, handling network partitions, dealing with clock skew, and navigating organizational context like tickets and post-mortems. The core argument is that model capability is limited by the quality and fidelity of the training data; if the environment is a sterile, single-node sandbox, the agent will never learn to handle the messy, high-stakes reality of production systems.

Moving Beyond Single-Node Sandboxes

To bridge this gap, Emulated advocates for "full-fidelity" simulation. A production-grade task is not just a code change; it involves:

  • Resource Provisioning: Managing VPCs, subnets, and security groups.
  • Operational Constraints: Meeting strict bars for cost, latency, and deployment safety.
  • Blast Radius Management: Performing rolling deployments and rollbacks while serving live traffic.
  • System Complexity: Handling failures in distributed clusters (e.g., etcd consensus issues) and managing external dependencies.

While single-node containers are useful for basic testing, they fail to represent the multi-node, multi-service architecture of companies like AWS or Vercel. Emulated proposes "cloud-in-a-box" environments that provision real infrastructure, allowing agents to learn how to operate systems that are living, evolving, and prone to unforeseen failures.

Why Infrastructure is the Starting Point

Emulated focuses on infrastructure for two primary reasons:

  1. Domain Expertise: High-quality data requires deep domain knowledge. By starting with infrastructure—a field with clear, well-defined problem statements—the team can create higher-fidelity training data than they could for more abstract or ambiguous domains.
  2. Vertical to Horizontal Scaling: Infrastructure provides a concrete foundation for testing agent autonomy. Lessons learned in managing distributed systems and cloud resources provide a blueprint for eventually simulating entire companies, which can then be applied to other horizontal domains.