The Problem of Open-World Tool Hallucination

LLM agents often operate in an 'open-world' assumption, where they are encouraged to be creative and flexible. However, this flexibility frequently leads to 'tool hallucination'—where an agent attempts to invoke functions, APIs, or tools that do not exist or are not part of the provided toolset. This behavior introduces significant security risks and reliability failures in production systems, as the agent may attempt to execute arbitrary code or interact with non-existent endpoints based on its internal probabilistic predictions rather than actual system capabilities.

Implementing Closed-World Resolution

The proposed solution shifts the agent's operating paradigm to a 'closed-world' model. Instead of allowing the LLM to freely generate tool calls, the system enforces a strict resolution layer that acts as a gatekeeper. This approach involves:

  • Schema-Constrained Generation: Forcing the model to output structured data (e.g., JSON) that must strictly adhere to a predefined OpenAPI or function-calling schema.
  • Verification Layer: Before any tool is executed, an intermediary validation step checks the agent's requested function name and parameters against a hard-coded, immutable registry of available tools.
  • Resolution Logic: If the agent requests a tool that is not in the registry, the system triggers a fallback mechanism—such as providing an error message to the agent to correct its trajectory or terminating the request—rather than attempting to execute the hallucinated function.

Impact on Agent Reliability

By moving from an open-world to a closed-world framework, developers can significantly reduce the surface area for injection attacks and runtime errors. This architecture ensures that the agent's 'knowledge' of its environment is limited to what is explicitly provided, effectively turning the LLM into a deterministic controller for a defined set of capabilities. This is essential for building robust AI agents that interact with sensitive APIs or internal business logic, where unpredictable behavior is not acceptable.