Fairies: AI Agents as Canvas Collaborators
Embed AI agents as draggable 'fairies' on tldraw's infinite canvas to draw diagrams, coordinate tasks via leader delegation, and execute code directly in a local desktop app for full interactivity.
Visual Agent Embedding Solves Opacity and Coordination
Place AI agents directly on the tldraw canvas as animated 'fairies'—draggable sprites showing real-time thinking, actions, and outputs—to make agentic loops transparent. Unlike sidebar agents, canvas fairies reveal spatial context: users see multiple agents working simultaneously, observing each other's changes (e.g., one draws a cat's hat while another adds a neck). This addresses blindness in agents by giving them shared canvas state and relative positioning, preventing overlaps. For structured drawing, use text-to-shape tools (circles, arrows) via structured outputs, overcoming vision model limits like conflicting Y-axis conventions (math up vs. web down) through prompt engineering. Outcome: agents produce editable diagrams (e.g., butterfly lifecycle) that users iterate on collaboratively, feeling like peers rather than distant tools.
In the one-month Fairydraw experiment (December 2025), three fairies handled large tasks: summon via prompt, select all for group mode where one elects as leader to scout canvas, generate to-do list, delegate subtasks, observe progress, and judge completion. Play at fairies.tldraw.com to test wireframing an ebook app from description—agents build functional prototypes in ~10 minutes.
Leader-Follower Orchestration Scales Multi-Agent Work
Implement agent swarms with a leader-follower pattern: leader scans canvas context, breaks tasks into subtasks (e.g., 'draw more animals'), assigns to followers, monitors without drawing itself, and iterates until satisfied. This mirrors early 2025 agent conventions (rejection sampling, thinking indicators) but adds visual coordination, solving overlap and state-sharing issues. Agents reference each other's work dynamically—prompt one for a cat, another for accessories—and handle ambiguities like 'blow out candle' by inferring shapes despite no prior canvas definition. Trade-off: fun for exploration (D&D sheets, homework tutors) but one-shot limits depth; agent loops enable refinement. Used in apps like Lovelace.dev and MagicPath for design/education.
Evolution from 2023's Make Real (draw UI → generate HTML/JS prototype via vision models) shows progression: early one-shots → iterative agents → spatial multi-agents. Canvas as React components enables hackability, powering Replit/Luma AI canvases.
Desktop Runtime Unlocks Code Execution for True Agency
Bypass web safety limits with a local Electron desktop app exposing an HTTP endpoint for agents to POST and execute raw JavaScript against the tldraw runtime. Agents script inject to add interactivity (on-hover sliders, clicks) despite lacking primitives, generate screenshots/DOM for vision, or even modify external apps (e.g., rip podcasts from Spotify bundle). Prompt: draw UI with 'leg length' slider and 't-shirt color' picker → agent writes event handlers, potentially blinking elements or creating auxiliary HTML. Trade-offs: high risk (API key leaks, unintended changes) but maximizes agency in file-based, offline context—'hand sharp tools to users.' Enables bi-directional flows: canvas diagram → update code, or code → visualize. Future: release as local-first canvas motivator, evolving 'file-over-app' ideals into practical scripting.