The Failure of Absolute Scoring

Traditional metrics like CLIP score or LPIPS are insufficient for video because they evaluate individual frames or pixel-level drift rather than the holistic storytelling quality. Maor Bril notes that absolute scoring (e.g., 1-10 scales) is inherently subjective and prone to "vibe-based" bias, where a model might reward a visually glossy but temporally broken video (e.g., a character frozen for four seconds) with a high score. These metrics fail to capture temporal consistency, physics, pacing, and narrative coherence.

Pairwise Preference as a Solution

To move beyond the limitations of absolute scoring, the team at Character.ai shifted to a pairwise preference approach. Humans are significantly more consistent when asked to choose which of two videos tells a better story than when asked to assign an arbitrary score to one. By training a small Vision Language Model (VLM)—specifically Qwen3-VL—using Bradley-Terry loss on pairs of real and deliberately corrupted footage, they created a judge capable of identifying specific failure modes like extra limbs, hovering physics, and audio-visual desync.

Integrating Evals into the Generation Loop

To maintain quality at scale, evaluation must be moved "left" in the development pipeline. The team treats video generation as an agentic workflow where the judge acts as a regression gate in CI. This allows the system to:

  • Catch drift early: By validating starting frames and short clips before they are combined into long-form content, the system reduces the cost of regeneration.
  • Self-verify: The agent can identify why a clip is "slop" (e.g., physics violation) and trigger a regeneration.
  • Optimize for speed: A smaller VLM was chosen over larger frontier models because the latency benefits for high-volume generation outweighed the marginal gains in accuracy from larger models.

Calibrating Against Human Taste

To prevent the model from becoming a simple "AI detector" (which would overfit to artifacts rather than quality), the team uses consistent encoding and annotation methods across both human-generated and AI-generated training pairs. Human feedback is collected periodically through 10-15 minute annotation sessions, which are then used to calibrate the AI judge and refine the dataset for future iterations. This creates a feedback loop that evolves alongside user expectations.