The Shift from Coding to Discovery

With AI handling code generation, the primary bottleneck in software development has moved upstream. The challenge is no longer how to build, but what to build. Because LLMs are optimized to provide the most common, average answer, they cannot replace the human capacity to read a room, identify unique business problems, and avoid the "faster horse" trap. In a recent internal hackathon, 17 out of 21 agent ideas were abandoned because they lacked clear business value or data access, proving that technical feasibility is no longer the primary hurdle.

The Analyst Toolkit for AI Engineering

To move from demo-ware to production-grade agents, engineers must adopt traditional product management frameworks. These tools provide the context necessary for LLMs to generate high-quality, relevant outputs:

  • Story Mapping: Use this to capture the process backbone. By mapping user activities (e.g., contact, triage, resolve, close), you can define a clear MVP and organize the backlog. AI excels at processing structured user stories (Persona + Need + Why), so providing this structure leads to more coherent system specifications.
  • The 4-Question Value Framework: Before building, answer these four questions to validate the agent's purpose:
    1. Whose problem is this?
    2. What does winning look like?
    3. What would cause the user to refuse the tool?
    4. What specific decision does this change?
  • VAD Thinking Path (Value, Architecture, Design): Always start by defining the value, then map the supporting process/architecture, and only then proceed to design. This ensures the system is built to support a specific outcome rather than just showcasing a feature.

Redefining Success Metrics

Shipping features is no longer a proxy for progress. To avoid the "demo trap" where systems are built but never used, teams should shift their KPIs:

  • Stop measuring: Number of features shipped or time spent on site.
  • Start measuring: Number of features used more than twice.

By involving subject matter experts in the decision-making process and performing mapping sessions before writing a single prompt, teams can ensure they are building solutions that solve real problems rather than just automating existing inefficiencies.