- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies the discrepancy between an LLM's ability to generate a correct answer and its ability to select the correct answer. The paper decomposes the factual question-answering process into three stages: whether the model generates a correct answer, how it ranks candidate answers, and how it selects the final answer. The paper especially compares the effects of selecting the final answer based on generation likelihood versus explicit verification. Through empirical evaluations across multiple frontier model families, the paper demonstrates that explicit verification methods improve ranking and plurality accuracy over standard mean log-likelihood and chat-template likelihood baselines. The paper also explores how evaluation metrics and entity masking affect the performance gains.
Critical strengths
- The paper addresses an interesting diagnostic research question: why LLMs have difficulty directly selecting the correct answer even if their generated candidates cover it. By examining candidate coverage, candidate ranking, and final answer selection, the paper provides a useful perspective for revisiting this problem.
- The paper provides extensive and comprehensive experiments to study the raised problem. The analysis covers multiple model families across Gemma, Llama, and Qwen, which makes the study generalizable. Additional analyses also help in understanding when verification improves ranking and whether those improvements translate into better final answers.
Critical weaknesses
- The presentation of this paper is disorganized and hard to follow. After a brief motivation, the introduction quickly shifts to detailed empirical results without fully explaining the background, research gap, or experimental setup. The discussion also moves between several baselines and correctness criteria, making it difficult to follow which comparisons support which conclusions. Overall, the paper's writing and expression do not meet this venue's standards.
- The paper is mostly analytical and observational, but does not provide
- The paper lacks an identifiable novel contribution. Gekhman et al. [1] already compare likelihood with and discuss why better pairwise ranking need not improve final selection. Although the paper discusses this in related work and adds more controlled experiments and observations, it does not clearly reveal a novel scientific insight from these additions.
References: [1] Gekhman, Zorik, et al. "Inside-out: Hidden factual knowledge in llms." COLM 2025.
Questions for authors
- Can the authors clarify the conceptual structure of the paper and the connection between its experiments? The paper is disorganized, making it difficult for a human reader to follow. For example, the paper presents empirical results in the introduction without fully discussing the background and motivation. What is the purpose of presenting it that way? Can the authors also provide a more comprehensive introduction to the limitations of existing work?
- How should the improvement from same-model verification be interpreted, and what are its main limitations? It would also be useful to explain cases where the verifier strongly favors an incorrect candidate even when a correct candidate is available. This would help readers understand when self-verification is useful and when its judgments remain unreliable.
- Given the closely related work by Gekhman et al. [1], can the authors clarify what principal conclusions this paper summarizes, and what novel contributions this paper clearly makes? Although the paper is analytical and informative, it does not clearly state, guide, or propose how the findings can contribute to the development of future methods for factual answer generation and selection.
References: [1] Gekhman, Zorik, et al. "Inside-out: Hidden factual knowledge in llms." COLM 2025.