- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how to improve perplexity-based evaluation of document with reference answers. The authors observe that standard perplexity score can favor an irrelevant document over a relevant one. It proposes a simple method that learns context-dependent token weights from labeled document pairs and uses them to reweight token-level negative log-likelihoods. The authors evaluate its effectiveness and cross-model and cross-dataset transferability across three QA datasets and multiple LLMs.
Critical strengths
[S1] The paper reports an interesting finding: standard perplexity can favor an irrelevant document over a relevant one.
[S2] The method is simple and easy to understand: it learns token weights from labeled document pairs and uses them to reweight token-level negative log-likelihoods for document scoring.
Critical weaknesses
[W1] Since the method requires labeled data for training, please compare it with supervised baselines trained on the same data. Without such comparisons, it is difficult to indicate the contribution of token reweighting.
[W2] It remains unclear whether the method can be transferred to closed-source models, as their APIs may not provide the token log-probabilities required to compute perplexity.
Questions for authors
[Q1] The evaluation relies on open-sourced LLMs, such as Gemma-2-9B and Llama-3.1-8B. Please include closed-sourced APIs.
[Q2] The evaluation only consider simple question answering. It remains unclear whether these findings generalize to code generation, mathematical reasoning, and agent tasks.
[Q3] The context budgets differ across the proposed method (18,000 tokens) and baselines (2,048-token). Please use same budgets to ensure a fair comparison.
[Q4] The evaluation measures document-pair ranking given a gold answer. It remains unclear how the method could be applied when the answer is unknown.
[Q5] It remains unclear whether improved ranking translates into higher answer accuracy on downstream tasks such as NQ and HotpotQA.
3 comments
Thank you for your rebuttal, it has addressed the bulk of my concerns.
Dear Reviewer bY4Y,
Thank you for your follow up and for taking the time to consider our rebuttal. We are glad that our response has addressed most of your concerns. We appreciate your constructive feedback and would be happy to provide any further information while time remains.
Thank you again for your time and consideration.
Supervised Baselines, Matched Context Budgets, and Clarifications of Closed-Source Transfer and Downstream Applicability
We thank the reviewer for the thoughtful summary and encouraging comments, and are glad that they found the finding interesting and the method is easy to understand.
We thank the reviewer for this suggestion. We added token2score, a trained discriminative ranking head using Qwen3-8B backbone, NQ training splits, and LoRA configuration as our method in Appendix A.4. Token2score achieves lower disagreement than weighted PPL on 2 datasets: 0.97% vs. 2.34% on NQ, and 1.13% vs. 2.02% on MS MARCO, higher on HotpotQA: 8.40% vs. 7.27%. See VJ1y W1/Q2 for the complete baseline and ablation results, and further discussion.
We appreciate all reviewers' suggestions to build a more comprehensive baseline. However, as emphasized in Section 4.4 and the conclusion, our central objective is to improve likelihood-based assessment through transferable, context-sensitive token weighting, rather than to outperform dedicated document ranking models or discriminative ranking head. None likelihood-based methods can’t be applied to different models, nor other uncertainty estimation tasks.
We thank the reviewer for highlighting this limitation. The biggest challenge is that these closed-sourced APIs like gpt-4o-mini don’t provide last hidden layer embedding but only top N result log probability. If a required gold answer token is absent from the returned probabilities, its contribution to the score cannot be computed exactly. We investigated gpt-4o-mini’s Chat Completions API, which exposes output token log probabilities and at most 20 alternatives per position. We send 10 query, documents and gold answer pairs per each dataset, only 2 of 30 pairs (1 NQ and 1 HotpotQA) were successfully evaluated. We therefore do not claim empirical validation of closed-source model transfer. We clarified that its applicability depends on whether the target API exposes the conditional token probabilities required by the scoring protocol in Appendix B.2.
Q2 We thank the reviewer for this suggestion.We have demonstrated an experiment in Section 4.5 shows that WeightedPPL better distinguishes correct from incorrect answers among multiple Monte Carlo samples, achieving a higher mean within-question AUROC than BasePPL (see our response to Reviewer EsPH, W1/Q1 for more detail). Although this extends our evaluation to generate answer correctness, it remains within question answering. Evaluating generalization to code generation, mathematical reasoning, and agent tasks is beyond the scope of the present study, but we consider it an important direction for future work.
Q3 We thank the reviewer for highlighting this potential confound. We reevaluated Base PPL and Weighted PPL using a shared 2,048 token document budget, matching the document cap used by SAR and CSL. Both scorers receive identical truncated documents and are evaluated on the same saved held out pairs, using the existing Qwen3-8B checkpoints without retraining.
The updated disagreement rates for WeightedPPL are 2.31%, 7.27%, 2.01% for NQ, HotpotQA and MSMARCO respectively, the difference is small compared to what we report in paper, 2.34%, 7.27%, 2.02%.
Q4 We thank the reviewer for raising this point. The experiment described in our response to Reviewer VJ1y, W3/Q3 uses WeightedPPL scores computed from gold answers to supervise a student document reranker through a recent self distillation idea. The student learns to predict these scores using only the query and document, enabling document reranking without gold answers at inference time.
Besides, in a more capable model, the knowledge needed to answer a query may already be encoded in their parameters. This motivates applying WeightedPPL directly to their generated answers, potentially improving uncertainty estimation without requiring gold answers at inference. We consider this as a promising direction.
Q5 We thank the reviewer for raising this point. Notice that our method could only be used as a more accurate uncertainty proxy to evaluate generated content, but not directly improve any generation accuracy itself.
We added a generated answer evaluation experiment in Section 4.5 using 1000 heldout NQ questions that Qwen3-8B answered wrong during an initial closed-book screening.We sample 10 closed-book answers at temperature 1.0 and score these answers using BasePPL and our WeightedPPL, conditioning both scorers on the same question, positive document, and gold reference answer. An external gpt-5.4-mini judge labels each answer’s correctness. We compute AUROC on 10 sampled answers within each question and average equally over all questions.
The results show improvement without additional training in all 3 datasets, suggesting that the benefits of learned token weighting extend from document ranking can translate to assessment of generated-answer correctness.