The Multi-Agent Architecture

To solve the complex problems of modality misalignment (e.g., political content inserted into sports clips) and unoriginal content (AI-generated duplicates), Meta employs a three-agent system rather than a single monolithic model. This decomposition allows for specialized handling of retrieval, perception, and reasoning:

  • Perceiver Agent: A specialized VLM that performs temporal decomposition. Instead of fixed frame rates, it identifies points of temporal change, compresses redundant frames, and extracts clip-level embeddings, OCR, and natural language descriptions.
  • Retriever Agent: Indexes metadata, embeddings, and entities into inverted, vector, and graph databases. It performs offline clustering to identify similar content and online retrieval to provide context for the reviewer.
  • Reviewer Agent: The central orchestrator. It analyzes the temporal JSON from the perceiver, integrates live user signals (comments, sentiment, reports), and evaluates whether the content violates policies. It can trigger re-retrieval if more context is needed.

Specialized Model Development

Because the system operates on messy, user-generated content rather than clean web data, off-the-shelf frontier models are insufficient. The team uses smaller, specialized VLMs optimized for specific tasks:

  • Pre-training & Fine-tuning: Models are pre-trained on in-house image tokens to handle the unique distribution of user-generated video. Instruction tuning is performed using a strict JSON schema to ensure consistent chain-of-thought reasoning.
  • Continuous DPO Loop: The system uses production samples to identify failures. An in-house LLM judge, backed by a human-in-the-loop queue, traces failures to specific nodes (e.g., tool call vs. retrieval). This data is used for Direct Preference Optimization (DPO) to continuously refine model behavior.
  • Distillation & Quantization: To maintain scale, the team distills and quantizes models to reduce inference costs. They maintain a performance-to-cost table to balance reasoning budgets against token costs.

Holistic Evaluation and Efficiency

Evaluation goes beyond standard precision/recall to include system-level metrics like per-node latency, reasoning budgets, and judge drift. To keep the pipeline performant at 100M+ scale, three primary optimizations are used:

  1. Spatial-Temporal Reduction: Compressing similar frames to minimize processing requirements.
  2. Caching: Storing verdicts for viral content to avoid re-processing identical uploads.
  3. Metadata Pruning: Using creator reputation and authenticity scores to bypass the pipeline entirely for trusted content, significantly reducing total compute load.