Quantifying Media Bias with Structured LLM Analysis

PressLens moves beyond simple bias detection by asking what facts survive across conflicting narratives. The system evaluates news outlets across six dimensions: emotional tone, framing, source diversity, loaded language, political stance, and factual density. By enforcing a strict JSON schema for LLM outputs, the tool enables reliable data visualization, such as spider charts and comparative tables, which allow users to see how different outlets frame the same event.

System Architecture and Performance

The backend is built with FastAPI and uses asyncio.gather() to perform concurrent LLM calls, significantly reducing latency. To manage costs and improve performance, the system employs a two-stage reasoning pipeline:

  1. Bias Scoring: Individual outlets are analyzed against the six dimensions using structured outputs.
  2. Neutral Synthesis: A separate LLM pass consumes the structured summaries from stage one to extract consensus facts and identify the sharpest narrative disagreements.

This separation ensures that the synthesis layer performs meta-analysis on structured data rather than raw text, improving reliability and cost efficiency. The system also includes a SQLite TTL cache to prevent redundant API calls for trending topics.

The Challenge of Consensus and Neutrality

The core philosophy of PressLens is that averaging bias scores is misleading; instead, the tool prioritizes extracting consensus while explicitly flagging outliers. For example, when analyzing the Ukraine-Russia war, the system identifies RT as an outlier due to fundamentally different framing, rather than attempting to manufacture false common ground.

However, the author notes critical limitations:

  • Western Bias: LLMs are trained primarily on Western internet text, which inherently skews their definition of "neutral" toward Western editorial norms.
  • Stochastic Variance: Due to temperature settings, bias scores can fluctuate by ±1 point across different runs.
  • Simplification: The six chosen dimensions are optimized for UI legibility rather than exhaustive editorial analysis.