The Failure of Current Video AI
Most video AI systems treat video as a "bag of frames" or a sequence of text tokens. This approach fails because it discards the continuity and spatial-temporal relationships inherent in video. Current systems are stateless—they answer queries from scratch, lack durable memory, and cannot reason over motion or causality across large archives. A larger context window does not solve this; the problem is the lack of a persistent, queryable representation of the video content.
The Context Graph: A Durable Memory Layer
To move beyond simple clip retrieval, video intelligence requires a "context graph" that acts as a navigable structure over the entire video volume. This graph organizes data into a hierarchy:
- Time-bounded moments: The base evidence units (scenes/shots).
- Appearances: Where and when entities (people, brands, objects) appear.
- Entities: The subjects themselves.
- Relationships: Co-occurrences, sequences, and interactions between entities.
- Corpus-level themes: High-level patterns and coverage across the entire dataset.
By storing these primitives rather than pre-computed answers, developers can traverse the graph to answer complex, multi-hop questions (e.g., "Track Messi across the entire World Cup and describe the camera framing") that a standard vector search cannot handle.
Five Principles for Video Memory Architecture
To build a robust video memory layer, James Le proposes five design rules:
- Ingest once, reason many times: Move the cost of interpretation to the ingestion phase. Do not re-process the source for every query.
- Store primitives, not answers: Focus on moments, entities, and relationships. This allows the system to support diverse downstream tasks like editing, analytics, and compliance.
- Ground every claim: Every output must cite a specific timestamp to ensure the system is verifiable and trustworthy.
- Let intent shape memory: Different workflows (e.g., brand safety vs. sports highlights) require different primitives. The memory layer must be configurable based on what the developer needs to track.
- Keep it composable: Expose the memory layer via APIs so it can serve as infrastructure for various application-level tools.
From Static Models to Video Workers
Moving from a static model to a "video worker" requires a deterministic harness. A video worker should be capable of task planning (deciding whether to search, summarize, or reason), retrieval (selecting the right evidence), and utilizing expert tools (e.g., zooming, comparing frames). Crucially, these workers must operate within an "operating envelope" that defines time, cost, and depth constraints, while providing an output contract that includes structured data and references.