Background Execution via Accessibility APIs

Traditional "Computer Use 1.0" agents rely on taking over the physical mouse and keyboard, which forces a serial, human-in-the-loop workflow that is prone to interference. Cua Driver shifts this paradigm by communicating directly with operating system accessibility layers (UI Automation on Windows, AT-SPI on Linux, and AX on macOS).

By leveraging these undocumented APIs, Cua allows multiple agents to perform tasks in the background simultaneously without stealing focus from each other or the human user. The driver observes the environment by combining accessibility tree data with window-specific screenshots. If an accessibility-based action fails, the system falls back to pixel-based clicking, providing a robust abstraction layer that works across platforms.

Benchmarking and Trust in Agent Performance

To quantify agent reliability, the team developed CUABench, a framework containing over 130 verifiable tasks across five platforms. A task in this framework consists of three components: a setup function, an oracle (golden trajectory), and an evaluator. To prevent reward hacking, the team subjects every task to a matrix of agents designed to break the environment before it is included in the dataset.

Performance results remain humbling: when tested on professional electrical engineering software (via Snorkel AI), the best agent achieved only a 24% pass rate (6 of 25 tasks), all of which involved editing existing schematics. Starting from a blank schematic resulted in a 0% success rate. However, when switching from standard computer-use tools to Cua Driver, pass rates on a 4K benchmark improved from 62% to 80%, while token consumption dropped by 34% because the agent focuses on specific windows rather than the entire desktop.

Optimizing Infrastructure for RL Training

Training agents via Reinforcement Learning (RL) often leads to significant GPU waste due to idle time spent waiting for sandbox environments to spin up or reset. To mitigate this, the team suggests using a demand-based autoscaler to manage a warm pool of sandboxes. By decoupling sandbox startup time from GPU execution, infrastructure costs can be reduced by 2x to 4x. This approach ensures that GPU workers maintain high utilization, as the overhead of environment initialization is shifted to the infrastructure layer rather than the expensive compute layer.