- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper evaluates whether LLMs can judge better than they generate. It is studied in a gold-answer setting, not in rubric-based, pairwise, or subjective judging settings. The paper subtracts generation accuracy from self-evaluation accuracy and uses it as a main metric, delta. The paper finds inconsistent outcomes regarding the sign of the delta (Table 3). Other ablations, such as a closed-book setting, are performed.
Critical strengths
- LLM judge fidelity is important in both practical and academic settings.
Critical weaknesses
- The paper doesn't test "rubric-based, pairwise, and subjective judging" and leaves them to future work. Then, what exactly is the paper testing? Those 3 are the most commonly used LLM-as-a-Judge use cases, and "gold-answer" generation is a limited use case where LLM-as-a-Judge is often not necessary. The title is seriously over-claiming, and the contribution does not fit the standards of ICLR.
- The abstract contains too much irrelevant information, that are hard to understand the significance. "One mechanism underlies this variation. The evaluator does not re-read the passage." Why does detailed experimental setup belong in the abstract?
- The contributions, which should be clearly readable and understandable and summarize the paper, are filled with run-on sentences that make it hard to discern which findings are most important and well-supported (is it "last-token attention" or "anchored on the candidate"?)
- Similarly, it is clear that the paper is written entirely by LLMs, specifically Claude (use of "load-bearing," etc.).
Questions for authors
N/A