Forum AI Scales Elite Experts for LLM Evaluation
Forum AI deploys world-class experts (e.g., Niall Ferguson, Fareed Zakaria) to build custom rubrics, annotate data, and create training packs for AI models in high-stakes domains like news, ethics, and mental health.
Targeting High-Stakes Domains Requiring Expert Oversight
Forum AI focuses on AI use cases where nuanced judgment is critical, including News & Current Events, Mental Health Advice, Culture & Society, Ethics & Safety, Education & Guidance, and Finance & Economics. Their approach counters AI's vulnerability to losing trust, as noted by historian Sir Niall Ferguson: new info techs risk credibility without human intelligence, especially for non-human-generated content. They partner with institutions like Carnegie Endowment for International Peace, Atlantic Council, Foundation for Defense of Democracies, Manhattan Institute, Mount Sinai, and Hudson Institute to ensure reliable oversight.
Advisors include Avik Roy (former Marco Rubio advisor), Kevin McCarthy (former House Speaker), Salena Zito (author/journalist), Hon. Ivan Duque Marquez (former Colombian President), Jackie Reses (Lead Bank CEO), Fareed Zakaria (CNN host/author), Dr. Jordan Shlain (physician), Scott Jennings (GOP strategist), Kristen Soltis Anderson (pollster), Elizabeth Economy (Hoover Institute), Vuk Jeremic (former UNGA President), and Emmanuel Acho (ex-NFL player/author).
Expert-in-the-Loop Services for Model Improvement
Services scale limited expert time via 'expert-in-the-loop' systems:
- Evaluation: Custom rubrics, evaluators, and prompt sets for frontier performance; standardized benchmarks with third-party certifications; expert evaluation reports with recommendations; expert-trained LLM judges via API for auto-evals and reward modeling.
- Data Annotation: Labels for training datasets; retrieval source annotation to enhance LLM prioritization of real-time sources; integrates into search/retrieval stacks.
- Data Production: Licensed retrieval packs for news/topics coverage; SFT data packs of expert-designed prompt-response pairs targeting specific gaps.
Teams get bespoke support from evaluation to production, including prompt sets and rubrics for internal use.
Related Insights from Forum AI Blog
Team posts expand on the approach: 'Why We Built Forum AI' (Campbell Brown & Robbie Goldfarb), 'When AI Needs Judgment, Not Just Data' (Robbie Goldfarb), 'The Disappearing Expert' (Campbell Brown), 'Expert-in-the-Loop: Strategies for Scaling the World's Best Human Knowledge' (Robbie Goldfarb), 'No-Bias, No-Bull AI' (Campbell Brown). These emphasize scaling diverse human expertise to address AI biases and judgment gaps.