The Problem: Disconnected Tool Usage in VLMs

Agentic Vision-Language Models often struggle with the 'tool-evidence gap.' When tasked with complex visual reasoning, models frequently trigger external tools (like object detectors or OCR engines) without a clear logical connection to the visual input. This leads to redundant API calls, increased latency, and hallucinations where the model relies on tool output that contradicts or ignores the actual image content.

The Solution: Necessary Tool-Evidence Path Rewards

The authors propose a training framework that introduces a specific reward signal during the reinforcement learning phase. Instead of rewarding only the final correct answer, the model receives a 'Necessary Tool-Evidence Path' (NTEP) reward. This mechanism evaluates the causal chain between the visual query, the specific tool selected, and the evidence extracted from that tool.

Key components of this approach include:

• Evidence Verification: A constraint that forces the model to justify why a specific tool is necessary for a given visual region.
• Path Alignment: A reward penalty for tool calls that do not contribute to the final reasoning chain or that provide information already present in the visual context.
• Efficiency Optimization: By penalizing unnecessary tool calls, the model learns to prioritize direct visual inference, reserving tool usage for high-uncertainty or high-complexity tasks.

Impact on Agentic Performance

By implementing these path-based rewards, the researchers demonstrate that models become more selective and accurate. The approach effectively reduces the 'noise' in agentic workflows, ensuring that every tool call serves as a verifiable step in the reasoning process rather than a speculative attempt to gather data. This results in faster execution times and higher reliability in multi-step visual reasoning tasks.