The AI Skill as a Supply Chain Risk

As AI tools become embedded in developer workflows, "AI skills"—instructions, context, and capabilities bundled for models or agents—behave like software dependencies. In a regulated environment like a bank, these skills introduce significant risks, including unauthorized credential access, destructive shell command execution, and excessive permission scoping. Because these skills are often shared across teams, they represent a critical supply chain vulnerability that requires the same rigor as traditional package management.

The Skill Vector Pipeline

Nubank developed "Skill Vector," a security gate that sits between skill creation and the internal marketplace. The system enforces a two-stage scanning process:

  1. Deterministic Checks: Uses regular expressions to catch known risky patterns, such as hardcoded credentials or dangerous shell commands.
  2. LLM Review: Analyzes the context and behavior of the skill to identify risks that deterministic checks miss, such as over-broad permissions or unsafe instructions.

This hybrid approach is essential because LLMs provide the necessary behavioral context, while deterministic checks provide a baseline of reliability. The system integrates directly into the developer workflow by providing feedback within Pull Requests (PRs) and generating SARIF reports, allowing security teams to ingest findings into existing vulnerability management programs.

Lessons from Scaling Vetting

After scanning over 2,000 skills, the team identified over 1,500 risks, leading to over 1,000 remediations before deployment. Key operational takeaways include:

  • Shift Left: Enabling developers to run the same security checks locally during iteration significantly improves compliance and reduces friction.
  • Human-in-the-Loop Validation: A major risk is "AI-looping," where a model asks itself for confirmation to execute a dangerous action. Security policies must enforce genuine human approval gates for high-risk operations.
  • Context-Aware Policies: Not all warnings are equal. The team learned to differentiate between harmless local behaviors and those that become dangerous when deployed to production.
  • Unified Governance: The goal is a canonical, trusted marketplace. Whether a skill is created internally or sourced from a third party, it must pass through the same vetting pipeline before being made available to the organization.