The Limitations of Static Analysis for AI-Generated Code

Coding agents frequently produce code that passes syntax checks but contains latent security vulnerabilities. Traditional static analysis tools often fail to catch these issues because they lack context regarding the agent's intent or the specific runtime environment. The authors argue that security testing for AI must shift from static inspection to execution-grounded validation, where the generated code is actually run and observed within a controlled, sandboxed environment.

The Execution-Grounded Testing Framework

The proposed framework integrates security testing directly into the software engineering pipeline. Instead of relying solely on prompt engineering to 'ask' an agent to be secure, the system treats the agent's output as a candidate for dynamic analysis. Key components include:

• Sandboxed Execution: Running agent-generated code in isolated containers to prevent malicious or accidental system impact.
• Dynamic Security Probing: Actively testing the code against common attack vectors (e.g., injection, insecure API usage) during the execution phase.
• Feedback Loops: Using the results of these runtime tests to provide immediate, actionable feedback to the agent, allowing it to iterate and remediate vulnerabilities before the code is merged into the main codebase.

By grounding security testing in actual execution, the framework reduces the reliance on 'security-by-prompting' and provides a more robust, empirical layer of defense against the inherent unpredictability of LLM-based coding agents.