The Failure of Aggregate Compression Scores

Traditional LLM pruning techniques rely heavily on global compression scores—metrics that evaluate how much a model's performance degrades on a general dataset after removing parameters. However, these aggregate scores often mask significant performance drops in specific subgroups (e.g., minority dialects, niche technical domains, or specific demographic representations). When compression scores cannot distinguish between a model that is uniformly accurate and one that is highly accurate on majority data but failing on minority groups, the resulting pruned model inherits these hidden biases.

Establishing Information-Theoretic Boundaries

This research introduces a framework for 'group-robust' pruning by defining information boundaries. Instead of optimizing for global perplexity or accuracy, the authors propose constraining the pruning process to maintain the mutual information between the model's internal representations and specific, critical subgroups. By establishing these boundaries, engineers can ensure that the pruning process does not discard parameters essential for maintaining the model's performance on underrepresented data, even if those parameters appear redundant from a global perspective.

Practical Implications for Model Efficiency

Moving beyond simple magnitude-based or Hessian-based pruning, this approach treats model robustness as a constraint on the information bottleneck. The key takeaway for builders is that pruning is not a neutral compression task; it is a selection process that inherently prioritizes certain data distributions over others. By incorporating group-robust boundaries, developers can create smaller, faster models that maintain performance parity across diverse inputs, preventing the 'accuracy tax' that often hits marginalized subgroups when models are aggressively compressed.