- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors study a multi-model LLM pipeline in which a primary model answers, two reviewer models score the answer, and a judge model grades the result. They focus on a gate that skips the synthesis step when reviewers reject the primary answer, and ask whether the automatic judge can be trusted to evaluate this gate. On two 100-question TruthfulQA slices the gate changes 24 answers. Human annotation of these 24 questions changes the estimated effect from −0.075 (automatic judge) to between −0.060 and −0.050 (adjudicated), after an AI-assisted first pass gave −0.035 to −0.025.
Critical strengths
The audit is careful. Because only 24 answers change, the authors can label all of them by hand, and the paper keeps label ambiguity separate from sampling error. The main statistics check out, and the supplement reproduces the automatic −0.075.
Critical weaknesses
- The main finding is narrower than the framing. In the supplement the gate weights are constants (0.66 for the primary vs. 0.41 and 0.385 for the reviewers), so the gate simply keeps the primary answer whenever the mean reviewer score is below 0.5.
- The result then mostly says that reviewer rejections are informative, on 24 questions from one model trio and one benchmark. This is a case study, and the title and conclusion claim much more.
- The human labels are less independent than described. The introduction says "two independent human annotators", but annotator 1 had a verdict suggested by AI for 22 of 54 records in l.256. At the same time, the AI Disclosure says AI was used "only to improve the language and grammar".
Questions for authors
see weaknesses