The Case Against Runtime VLM Deployment

Developers often default to using Vision Language Models (VLMs) for real-time computer vision tasks, but this approach is rarely optimal. VLMs are computationally expensive, failing to meet real-time requirements (30-40 FPS) on edge hardware. Furthermore, many popular object detection models (like YOLO) carry restrictive licenses (AGPL 3.0), creating significant legal risk for commercial deployment. The solution is to shift from runtime VLM inference to training task-specific, Apache 2.0-licensed models that offer superior robustness and speed.

The 'Vibe Training' Pipeline

To bridge the gap between raw data and a production-ready model, Noyan proposes a toolkit that treats coding agents as "clueless computer vision engineers" and automates the annotation process:

  1. Auto-Labeling: Use a capable VLM (e.g., Qwen 2.5 7B) to generate initial bounding box annotations for an unlabeled dataset.
  2. Ensemble Judging: Pass the overlaid images to two smaller VLM judges (e.g., Gemma 2 8B and Llama 3.2 11B). Because judges often exhibit bias, use a "minimum agreement" strategy—if at least one judge approves the annotation, the example is kept. This prevents the over-filtering that occurs with strict consensus.
  3. Training: Feed the curated dataset into a lightweight, high-performance architecture like RF-DETR.

This pipeline is cost-effective, typically costing $3–$4 per full run on Hugging Face infrastructure. It generalizes well; in document parsing tests, the trained model successfully identified signatures that the original labeling VLM missed.

Managing Agentic Limitations

Coding agents are powerful but lack domain-specific common sense. Noyan notes that agents frequently suggest harmful data augmentations, such as horizontally flipping traffic signs or jittering the color of traffic lights, which corrupt training data. Developers must maintain a human-in-the-loop approach to approve prompts and verify the agent's proposed augmentation strategies. Future iterations of this workflow will focus on "image-guided detection"—using visual references rather than natural language descriptions to identify non-describable industrial parts—and refining the merger logic using Intersection over Union (IoU) metrics rather than binary acceptance.