Rethinking Perplexity with Transferable Semantic Reweighting

Perplexity provides an efficient likelihood-based signal for assessing how strongly a language model supports a target generation, but it uniformly aggregates token likelihoods despite large differences in their predictability and semantic importance. Consequently, variation driven by lexical frequency, surface realization, or model-specific regularities can obscure likelihood changes associated with meaningful contextual evidence. We introduce semantic reweighting of contextual token likelihoods, which learns non-negative weights over token-level negative log-likelihoods from contextual representations, allowing the resulting score to emphasize more informative parts of generations. Across nine model–dataset settings, our method reduces average disagreement with document relevance annotations from 15.69% for standard perplexity to 3.55%. Moreover, weighting functions transferred across models and datasets retain 91.72% of the improvement achieved by weighting, suggesting that the informativeness of weighting exhibits transferable structure. These results show that a pre-trained contextual reweighting model can make perplexity more sensitive to semantic evidence while retaining its computational efficiency, with strong cross-domain generalization and simple plug-and-play deployment.