The Anatomy of Agentic Risk

PostHog's "Wizard" is an agentic CLI that automates SDK installation and event instrumentation. Because it possesses the ability to execute shell commands, it functions similarly to a "malware starter pack." The primary security challenge is that agentic systems are complex, and while individual components (like prompts or allow-lists) may appear secure, vulnerabilities often arise from the composition of innocent parts. Code review processes typically examine diffs in isolation, failing to account for how different system components interact to create security holes.

Deterministic Security vs. Probabilistic Judgment

Sanders argues that security must be deterministic and mechanical. At PostHog, they developed "The Warlock," a standalone scanner that uses YARA rules—a deterministic, battle-tested pattern-matching engine—to audit content. Crucially, the Warlock only reports findings; it does not take action.

To manage noise and false positives, the team added an LLM-based "triage" layer. This layer acts strictly as an adviser, not a bouncer. The security gate remains deterministic: if a YARA rule matches, the system blocks the action immediately, regardless of the LLM's opinion. The LLM is only invoked to categorize or filter noise after the deterministic gate has already performed its function. If the LLM layer fails, the system defaults to a "fail-closed" state.

Securing the Supply Chain

Security is not just about user input; it is about the entire context provided to the agent. PostHog feeds the Wizard context from documentation, example apps, and prompts. If an attacker poisons this content (e.g., via a malicious pull request to an open-source repo), the agent might execute a prompt injection payload signed by the company itself. To mitigate this, PostHog scans content at two stages:

  1. At the source: When a skill bundle is built.
  2. At the point of use: When the Wizard loads the context at runtime.

Practical Rules for Agentic Safety

  • Prompts are not security: They are for steering, not for enforcing boundaries.
  • Scan your supply chain: Treat your own documentation and example code as potential attack vectors.
  • Design for real-world impact: Avoid overly broad rules that trigger on common developer tasks (like rm -rf in build folders), as this leads to developers disabling security tools. Use negative tests to prevent false positives and ensure rules are specific to the agent's actual environment.