- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents Reasoning Jury, which uses multiple LLMs to identify errors in reasoning traces and combines their findings through consolidation or moderated discussion. Each finding includes the affected steps, a diagnosis, severity, and evidence. Experiments evaluate error localization on Hard2Verify and DeltaBench, token-priced cost, and the usefulness of the feedback for retrying AIME2026 problems.
Critical strengths
The paper addresses a useful problem, and the gains from combining inexpensive judges are promising. The comparisons with simple aggregation help explain where those gains come from. The AIME experiment also provides useful evidence: rich feedback improves retry accuracy from 71.2% to 76.2% over step locations alone, with an analysis that accounts for dependence among samples from the same problem.
Critical weaknesses
1. The methodological contribution is limited
Multi-model judging and multi-agent debate are already established [1, 2]. Here, the main change is to apply them to step-level reasoning defects with structured reports. I do not see a substantial methodological advance, particularly when simple union captures most of the localization gains and slightly outperforms deliberation on DeltaBench. The paper shows that collecting more judgments helps, but does not make a convincing case for its particular consensus mechanism.
2. The richer judgments are not adequately validated
The paper's case for going beyond simple aggregation rests on better diagnoses, severity ratings, and supporting evidence. Yet the main benchmarks only score error locations. The retry experiment is useful, but some feedback supplies explicit corrections, and better final answers do not establish that the underlying diagnoses are reliable. Without directly checking these judgments or comparing equally rich feedback from simpler alternatives, I cannot tell whether consensus actually produces a better evaluator.
3. The numerical inconsistencies weaken my trust in the results
Appendix L acknowledges unresolved inconsistencies in the Hard2Verify metrics and conflicting entries across tables. I appreciate the disclosure, but these are still the results used to support the paper's conclusions. Until they are reconciled, I cannot confidently rely on the reported comparisons. This materially affects my assessment of the paper's credibility.
[1] Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models.
[2] Improving Factuality and Reasoning in Language Models through Multiagent Debate.
Questions for authors
My main questions follow directly from the concerns above: what does the consensus mechanism add over simpler aggregation, and can the authors reconcile the reported results and establish the reliability of the resulting judgments?