The Limitation of Routing Weights in MoE Interpretability

In standard Mixture-of-Experts (MoE) architectures, interpretability is often limited to analyzing routing weights—the coefficients that determine how much input is allocated to each expert. However, the authors argue that these weights are insufficient for understanding reward models. Routing weights only describe the flow of information, not the functional contribution of an expert to the final scalar reward. Relying on them leads to a superficial understanding that fails to capture how experts interact or how they specifically shape the model's preference judgments.

Contribution Contrast: A Response-Level Approach

The researchers propose 'Contribution Contrast,' a method designed to provide a faithful, response-level interpretation of MoE reward models. Instead of looking at the routing mechanism in isolation, this technique measures the actual impact of an expert by contrasting the model's output when specific experts are active versus when they are ablated or modified. This allows researchers to isolate the causal influence of individual experts on the final reward score. By focusing on the response level, the method provides a granular view of how specific experts contribute to the model's evaluation of text, revealing which experts are responsible for identifying specific quality markers (e.g., factual accuracy, tone, or coherence).

Implications for Reward Model Transparency

This approach addresses a critical gap in AI alignment: the 'black box' nature of reward models. By moving beyond routing weights, developers can verify if a reward model is relying on the intended experts for specific tasks. For instance, if a reward model is intended to prioritize safety, Contribution Contrast can verify whether the 'safety-aligned' experts are actually driving the reward signal or if the model is relying on spurious correlations in other experts. This technique provides a path toward more robust auditing of reward models, ensuring that the components responsible for preference learning are behaving according to design specifications rather than just routing data efficiently.