- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Reasoning Arena, an adaptive reward framework for reinforcement learning with verifiable rewards (RLVR). The method targets rollout groups in which all sampled traces receive the same binary verifier reward and consequently have zero group-relative advantages. Instead of discarding these groups or resampling them, the method routes them to an LLM-based trace tournament, where reasoning trajectories are compared pairwise. A live opponent-selection strategy compares each newly generated trace against selected anchors representing the current best, worst, and median traces. The resulting incomplete comparison graph is aggregated using a Bradley–Terry model to obtain scalar rewards. Groups with diverse verifier outcomes continue to use the original verifiable rewards.The authors evaluate the approach using Ministral-3-8B-Instruct-2512 on competition mathematics and scientific reasoning benchmarks, with LiveCodeBench used for out-of-distribution evaluation. They report an average score of 53.9 for Reasoning Arena-Live, compared with 46.3 for the RLVR baseline, along with approximately 50% fewer generations per training step and reduced estimated total training cost.
Critical strengths
1.The paper identifies a concrete inefficiency in group-relative RLVR: groups with identical binary rewards produce zero advantages despite potentially substantial differences in the quality of their reasoning traces. Recovering useful supervision from these already-generated samples is a worthwhile research direction.
2.The live opponent-selection strategy and Bradley–Terry aggregation offer a practical way to reduce pairwise judging costs. The method is also designed to accommodate asynchronous rollout generation, making its systems motivation relevant to large-scale RL training.
3.The experiments include verifier-only RLVR, pointwise judge rewards, entropy-based advantage shaping, and tournament-only reward training. These comparisons help investigate the contributions of adaptive routing and pairwise trace evaluation separately.
Critical weaknesses
1.The central assumption is that pairwise LLM judgments can reliably distinguish reasoning quality when the verifier assigns identical outcomes. However, direct validation of the judge's reasoning-quality assessments is limited to three selected qualitative examples in Appendix D, which the authors explicitly acknowledge do not establish overall judge reliability. The downstream gains and alternative-judge experiments provide useful but indirect evidence. The study lacks a systematic evaluation against independently verified reasoning-quality annotations, human preferences, or a sufficiently large set of automatically checkable intermediate derivations. For all-incorrect groups, the concern is not that the preferred trace remains incorrect, but whether the judge consistently prefers more valid intermediate reasoning over superficial plausibility.
2.The comparison between Reasoning Arena and Reasoning Arena-Live changes both the comparison graph and reward aggregation, from full-round-robin win rates to live comparisons with Bradley–Terry fitting. Consequently, the downstream results do not isolate the effects of opponent selection and aggregation. On a fixed set of rollout groups, comparisons against full-round-robin Bradley–Terry scores and equal-budget random-opponent comparisons, together with sensitivity to arrival order, would help establish whether the live strategy recovers stable relative rewards.
3.The main results are reported as point estimates, without a clear account of the number of independent RL training runs or uncertainty across runs. Although the paper uses multiple evaluation samples to reduce evaluation variance, this does not quantify variability across independently trained models. Consequently, the robustness of the reported improvements—particularly the smaller differences between Reasoning Arena-Live and the full-tournament variant—remains unclear. Reporting the number of training runs and appropriate uncertainty estimates, with repeated runs for the key comparisons where feasible, would strengthen the empirical conclusions.
Questions for authors
1.Does the judge reliably prefer better reasoning rather than responses that merely sound more convincing? Have the authors checked its preferences against expert assessments or independently verified reasoning steps? How often do they agree, and does this agreement differ between all-correct and all-incorrect groups?
2.After reducing the number of pairwise comparisons, how consistent are the resulting rewards with those obtained from full pairwise comparisons, using Bradley–Terry aggregation in both cases? For the same sampled rollout groups, how sensitive are the rewards to trace completion order? Under the same comparison budget, does the proposed opponent-selection strategy preserve agreement with the full-comparison rewards better than random selection?
3.How many independent RL training runs were conducted for each method in Table 1? How much do the scores vary across training runs? Is the reported advantage of Reasoning Arena-Live over the full-tournament variant consistent across runs?