- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper audits whether correctness scores derived from internal model signals can support answer selection. Using KV24, a compact summary of KV-cache statistics, it distinguishes pooled correctness discrimination, within-question ranking, and actual answer-choice accuracy. The central finding is that high pooled AUC can largely survive the removal of within-question ordering, while downstream selection gains remain inconsistent.
Critical strengths
- The paper clearly demonstrates an important evaluation pitfall: strong pooled correctness discrimination does not necessarily translate into better answer selection.
- The empirical audit is careful and transparent, including representation controls, frozen transfer tests, same-pool voting, and explicit reporting of negative results.
Critical weaknesses
- Limited incremental contribution. Question-level confounding is already recognized in prior work. The paper offers a detailed diagnostic study, but its additional methodological insight remains modest, and no effective corrective method is demonstrated.
- Insufficient evidence of generality. The investigation centers on KV24 and restricted readout families. Small mixed-question samples and model-dependent response eligibility further limit the evidence. These findings do not establish how broadly the observed failures apply to stronger answer-selection systems.
Questions for authors
I suggest evaluating a broader range of verifiers to establish whether the identified limitations are widespread. An analysis restricted to a few verifier families provides insufficient evidence for the broader significance needed for a strong contribution.