acceptodds
Under review as a conference paper at ICLR 2027

Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a leading paradigm for improving the reasoning ability of large language models through outcome-based supervision. However, verifiable rewards frequently become uninformative at the group level: when all sampled traces of a given prompt receive identical rewards, group-relative advantage estimation provides no gradient signal, even though the traces may differ substantially in reasoning quality. We propose Reasoning Arena, an adaptive training framework that routes such non-diverse reward groups to a judge system instead of discarding them. Beyond examining the final answer, Reasoning Arena constructs trace tournaments, where reasoning traces are compared head-to-head to expose finer-grained preferences within the group, converting reasoning quality into rich relative reward signals. To make reward estimation efficient, rather than exhaustively comparing every pair, each new trace is evaluated against a small, dynamically updated pool of previously generated traces as anchors to efficiently establish a relative ranking. We then fit a Bradley-Terry model on the incomplete comparison graph, enabling scalable RL integration without quadratic pairwise comparisons. Empirical results demonstrate that Reasoning Arena consistently outperforms the RLVR baseline by 7.6% on average in competition mathematics and coding benchmarks. By converting otherwise wasted zero-advantage samples into useful gradient updates, our method accelerates training by 27% to 41%, saving nearly 50% of generation compute, and substantially improves overall reasoning performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Trace Tournament

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_EgRPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reasoning Arena, an adaptive reward framework for reinforcement learning with verifiable rewards (RLVR). The method targets rollout groups in which all sampled traces receive the same binary verifier reward and consequently have zero group-relative advantages. Instead of discarding these groups or resampling them, the method routes them to an LLM-based trace tournament, where reasoning trajectories are compared pairwise. A live opponent-selection strategy compares each newly generated trace against selected anchors representing the current best, worst, and median traces. The resulting incomplete comparison graph is aggregated using a Bradley–Terry model to obtain scalar rewards. Groups with diverse verifier outcomes continue to use the original verifiable rewards.The authors evaluate the approach using Ministral-3-8B-Instruct-2512 on competition mathematics and scientific reasoning benchmarks, with LiveCodeBench used for out-of-distribution evaluation. They report an average score of 53.9 for Reasoning Arena-Live, compared with 46.3 for the RLVR baseline, along with approximately 50% fewer generations per training step and reduced estimated total training cost.

Critical strengths

1.The paper identifies a concrete inefficiency in group-relative RLVR: groups with identical binary rewards produce zero advantages despite potentially substantial differences in the quality of their reasoning traces. Recovering useful supervision from these already-generated samples is a worthwhile research direction.

2.The live opponent-selection strategy and Bradley–Terry aggregation offer a practical way to reduce pairwise judging costs. The method is also designed to accommodate asynchronous rollout generation, making its systems motivation relevant to large-scale RL training.

3.The experiments include verifier-only RLVR, pointwise judge rewards, entropy-based advantage shaping, and tournament-only reward training. These comparisons help investigate the contributions of adaptive routing and pairwise trace evaluation separately.

Critical weaknesses

1.The central assumption is that pairwise LLM judgments can reliably distinguish reasoning quality when the verifier assigns identical outcomes. However, direct validation of the judge's reasoning-quality assessments is limited to three selected qualitative examples in Appendix D, which the authors explicitly acknowledge do not establish overall judge reliability. The downstream gains and alternative-judge experiments provide useful but indirect evidence. The study lacks a systematic evaluation against independently verified reasoning-quality annotations, human preferences, or a sufficiently large set of automatically checkable intermediate derivations. For all-incorrect groups, the concern is not that the preferred trace remains incorrect, but whether the judge consistently prefers more valid intermediate reasoning over superficial plausibility.

2.The comparison between Reasoning Arena and Reasoning Arena-Live changes both the comparison graph and reward aggregation, from full-round-robin win rates to live comparisons with Bradley–Terry fitting. Consequently, the downstream results do not isolate the effects of opponent selection and aggregation. On a fixed set of rollout groups, comparisons against full-round-robin Bradley–Terry scores and equal-budget random-opponent comparisons, together with sensitivity to arrival order, would help establish whether the live strategy recovers stable relative rewards.

3.The main results are reported as point estimates, without a clear account of the number of independent RL training runs or uncertainty across runs. Although the paper uses multiple evaluation samples to reduce evaluation variance, this does not quantify variability across independently trained models. Consequently, the robustness of the reported improvements—particularly the smaller differences between Reasoning Arena-Live and the full-tournament variant—remains unclear. Reporting the number of training runs and appropriate uncertainty estimates, with repeated runs for the key comparisons where feasible, would strengthen the empirical conclusions.

Questions for authors

1.Does the judge reliably prefer better reasoning rather than responses that merely sound more convincing? Have the authors checked its preferences against expert assessments or independently verified reasoning steps? How often do they agree, and does this agreement differ between all-correct and all-incorrect groups?

2.After reducing the number of pairwise comparisons, how consistent are the resulting rewards with those obtained from full pairwise comparisons, using Bradley–Terry aggregation in both cases? For the same sampled rollout groups, how sensitive are the rewards to trace completion order? Under the same comparison budget, does the proposed opponent-selection strategy preserve agreement with the full-comparison rewards better than random selection?

3.How many independent RL training runs were conducted for each method in Table 1? How much do the scores vary across training runs? Is the reported advantage of Reasoning Arena-Live over the full-tournament variant consistent across runs?

Reviewer_3GBqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses a critical failure mode in reinforcement learning with verifiable rewards: when all rollouts for a prompt receive identical verifier scores, group-relative advantages collapse to zero and expensive samples provide no learning signal. The authors propose reasoning arena, an adaptive framework that detects such non-diverse groups online and routes them to an LLM-judged trace tournament that compares reasoning trajectories pairwise, aggregates preferences via a Bradley–Terry model over an incomplete comparison graph constructed with a live anchor opponent strategy, and feeds the resulting per-trace rewards back into a standard RLVR update. Experiments on math and code benchmarks report consistent gains over RLVR and judge-only baselines, along with improved training efficiency and reduced wasted generations.

Critical strengths
  • Introducing trace-level tournaments within a verifiable-reward setting is a meaningful innovation, distinct from prior tournament-only reward training in non-verifiable domains.

  • The live tournament plus Bradley-Terry aggregation is a sensible way to make pairwise judging compatible with asynchronous rollouts, and the constant-rate comparison schedule is a practical contribution.

  • The study evaluates across multiple math, scientific, and coding benchmarks, reporting consistent improvements versus baselines.

Critical weaknesses
  • The whole approach rests on the assumption that the judge's pairwise preferences reflect genuine reasoning quality, but the paper provides no quantitative evidence for this. If the judge itself carries biases, for example favoring more verbose answers or confidently written but flawed derivations, the policy will learn these biases rather than actual reasoning ability.

  • The paper does not demonstrate that randomizing the presentation order actually mitigates position bias.

  • The soft margin γ\gamma is not ablated. γ\gamma determines how strongly the judge's verdicts are injected into the policy gradient. With γ\gamma fixed at 0.75 and no sensitivity analysis, it is hard to tell whether the reported gains are robust to this choice.

Questions for authors
  • Could the authors characterize the judge’s behavior on the two types of non-diverse groups separately, perhaps with a small sampled evaluation? For all-correct groups, it would help to know whether the judge reliably prefers rigorously derived solutions over ones that reach the same answer through flawed steps, for which a rough error rate on known fallacious cases would suffice. For all-incorrect groups, the question is how often the judge favors the trace with more valid intermediate reasoning when both responses are wrong. These two capabilities are precisely what the tournament reward is assumed to provide, so even a modest measurement here would go a long way toward supporting the central claim.

  • Would the authors consider reporting a swap-consistency diagnostic for the judge, namely the fraction of pairs judged twice in opposite order that yield the same verdict? Since judge preferences serve directly as the RL reward, such a measurement could help clarify whether the policy is genuinely improving reasoning or partly exploiting the judge’s surface preferences.

  • Did the authors check during training whether the traces preferred by the judge drift in length, formatting, etc.?

  • When the same pair of traces is presented in both orders, what fraction of verdicts agree?

  • Are the results sensitive to the choice of γ\gamma? For instance, do the conclusions still hold under different values of γ\gamma?

Reviewer_rocGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reasoning Arena, an adaptive routing strategy that addresses zero advantage groups in RLVR with binary rewards. Groups with homogeneous rewards are routed to a pairwise tournament judged by a much larger LLM that has access to the reference answer. The tournament is either full round-robin or has a live variant that compares each new trace against the current best, worst, and median, then fits a Bradley-Terry model with soft verdict labels.

Experiments on an 8B model show a 7.6 point average gain over RLVR, 27-41% lower training time, approximately half as many generated responses, and 10-14% estimated total cost reductions.

Critical strengths

The problem is real and the proposed approach is simple. In these experiments, RLVR requires roughly twice as many generations per step because homogeneous groups are discarded. The paper shows evidence that external pairwise judging can recover learning signals from otherwise discarded groups. The authors conducted different ablations to decompose the effects of reward type and routing, and they also report evaluation curves rather than only final numbers. It is interesting that judge-only baselines also outperform RLVR, suggesting there is value in richer feedback beyond binary pass/fail. The gains are consistent across multiple judges and there is less OOD regression than with RLVR.

Critical weaknesses

Most of the gains seem to come from combining verifier rewards with a fallback pairwise LLM judge, and the additional gains from the live tournament and Bradley-Terry model are unclear. The effectiveness of best/worst/median anchors and BT aggregation needs to be demonstrated separately (full vs live changes both simultaneously). The paper doesn't report ablations against simpler approaches such as circular ring comparison with N=8 judge calls (helps also balance position bias). Overall, the live variant makes fewer judge calls but consumes more local GPU hours and is more expensive overall (using the authors estimated total cost) than the full variant, for a small performance improvement that may not be significant. The number of effective calls in Table 2 looks misleading: in Algorithm 1 judging starts while the partially completed group remains homogeneous and those judge results could be discarded later.

Other important alternatives haven't been tested. There is no reward profiling or curriculum baseline despite the focus on training efficiency. Profiling takes time and compute but that overhead should be measured as part of the trade-off. It'd be very beneficial to report evaluation numbers for the judges themselves, along with ablations using much smaller judges, no reference answer, fine-grained judging rubric, etc.

The last checkpoint might not be the best comparison, as the final RLVR checkpoint shows a large dip on AIME 2025, AIME 2026, and GPQA Diamond (still, evaluation curves clearly show RLVR underperforms). We also need the scores of the starting checkpoint to support the claim that there is no regression.

Questions for authors

Could you clarify what a "reasoning trace" is? Does the judge receive raw thinking traces or only the response with the final answer? Could you also clarify "concise reasoning traces natively generated within the policy model's response trunk"? Figure 4 shows response lengths around 30-40k, and the paper reports a maximum sequence length of 82k, while the judge has a 32k context window and must receive two responses. How are these responses / reasoning traces extracted?

Why are generations per step for Reasoning Arena below the minibatch size (480 sampled rollouts)?

Why does Reasoning Arena Live produce longer responses than Reasoning Arena?

Could you clarify how a synthetic reverse judgment helps? This contributes to the same loss, and seems equivalent to just halving the regularization.

Why gamma=0.75 and how does it compare to gamma=1?

Why do Adaptive Pointwise and RLAIF use different judging prompts?

You said "We rerun both forms of reward source, and our experiments show that the gain from REASONING ARENA is dominated by the reward signals learned from all-incorrect groups". How is this measured?

Have you measured pointwise and pairwise judging accuracy before downstream RL?

Suggestions for future work: you could try N=16 for potentially more diverse responses, and blend judge and verifier rewards as judge feedback could also help with heterogeneous groups.

Reviewer_s1fhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses zero-variance reward groups in group-relative RL by routing them to an external pairwise judge while retaining verifiable rewards for diverse groups. Its live variant uses asynchronous anchor comparisons and Bradley–Terry aggregation. On one 8B policy, the reported six-benchmark average improves from 46.3 to 53.9, alongside fewer generated samples and shorter training steps.

Critical strengths

The paper addresses a practical limitation of group-relative RL with a simple routing rule that preserves verifiable supervision whenever it provides within-group contrast.

Improvements over RLVR are reported across all six benchmarks. The verifier-only, judge-only, entropy-based, and adaptive-pointwise baselines provide useful evidence for the overall approach.

Critical weaknesses

The reliability of the recovered supervision is insufficiently established. The qualitative examples show that reasoning-sensitive feedback is possible, but do not quantify judge accuracy across training. Equal verifier rewards do not guarantee distinguishable reasoning quality, and increased nonzero-advantage rates cannot distinguish useful feedback from judgment noise. The treatment of weak or unreliable preferences needs clarification.

The incremental contribution is not cleanly isolated. All experiments use one 8B policy; changing judges does not establish generality across policy families or scales. Live changes both opponent selection and aggregation, leaving their individual contributions unclear. Training variability is unreported, particularly relevant to its 0.4-point advantage over the full tournament. These limitations weaken the evidence for the contribution beyond combining existing components.

Questions for authors

Can you provide representative evidence of judge reliability on all-correct and all-incorrect groups, and clarify how tied or unreliable comparisons are handled?

With the same judge, comparison budget, and BT aggregation, how does the live anchor rule compare with random anchors? Please also provide numerical results for the all-correct/all-incorrect ablations discussed in Appendix C.

Does the cost estimate include speculative comparisons discarded when a group later becomes reward-diverse? Please clarify the accounting assumptions and distinguish generation-count savings from compute savings; raw logs are unnecessary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.