- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes RankBuffer for efficient ranking-based rewards in reinforcement learning. It reuses an ordered, query-specific buffer of historical responses, combines coarse insertion with local fine ranking, and dynamically updates the buffer.
Critical strengths
The motivation is clear, and the coarse-to-fine ranking framework is straightforward to understand and implement.
Critical weaknesses
-
The quality scale depends on a fixed judge and lacks independent validation. Sec. 3.3 treats the stored anchor order as an established quality scale, but its consistency across judges or with human preferences is not evaluated. Moreover, Sec. 4.1 uses the same Qwen model for both training rewards and all four benchmark evaluations. The reported improvements therefore lack independent confirmation beyond the training judge’s preferences. It is also unclear how the buffer would be recalibrated if the judge changes or historical comparisons come from heterogeneous evaluators (like LM Arena).
-
The reported cost reductions do not establish practical latency gains. In Sec. 3.3, local fine ranking depends on completing coarse insertion for the corresponding group, creating two sequential stages despite parallelism within each stage. Figures 4–5 and Table 8 report tokens and successfully parsed calls, but provide no latency judging or training time. Consequently, the practical efficiency of this scheduling design remains insufficiently demonstrated. If the latency introduced by multi-stage comparisons has a negligible impact in this scenario, does Elo-style mechanism that reuses historical ratings and adaptively compares responses with nearby ratings also work well?
Questions for authors
- How does RankBuffer compare with directly ranking all eight rollouts in one listwise call, using the prompt already employed for bootstrap initialization in Sec. 3.2?
- How concentrated are rollouts across anchor intervals, and what fraction requires fine ranking throughout training? Does dynamic maintenance improve these quantities enough to offset longer anchor contexts? Why does dynamic maintenance perform worse and consume more tokens than the static buffer under bootstrap initialization in Table 2?
- How often do coarse insertion errors produce incorrect cross-interval orders? Can such errors persist after the affected responses become anchors?
5 comments
Q1. Comparison with direct listwise ranking
We added a listwise baseline that ranks all rollouts in each group in a single judge call.
| Method | AlpacaEval 2 | Arena-Hard v2 | WildBench v2 | WritingBench | Average |
|---|---|---|---|---|---|
| Listwise | 67.18 | 34.90 | 55.84 | 68.13 | 56.51 |
| RankBuffer | 79.19 | 36.80 | 59.25 | 71.86 | 61.78 |
RankBuffer outperforms direct listwise ranking on all four benchmarks, with an average improvement of 5.27 points. For open-ended long-form generation, putting all responses into a single prompt creates a long context, which can make it harder for the judge to distinguish differences in response quality. Moreover, without anchor responses as references, the judge must infer a quality scale from the candidates alone, and its ranking may be sensitive to their presentation order. In contrast, RankBuffer uses reusable anchors for coarse insertion and limits fine ranking to rollouts within each anchor interval, reducing the scope of each ranking decision.
Q2. Rollout distribution and dynamic buffer maintenance
Q2.1 Where rollouts concentrate and how often fine ranking is used
The evolution of the buffer in Sec. 4.4 (Fig. 6) gives an indication of where rollouts concentrate. Local refinement adds an anchor only when an interior interval contains at least three rollouts, and these refinement anchors accumulate mainly in higher-quality interior regions. At the same time, the share of anchors in the highest rank bin added through upper-boundary expansion rises from 71% after the first epoch to 82% after the last. Together, these patterns suggest that rollouts frequently cluster in upper intervals and reach beyond the existing upper boundary as training progresses. Figure 6 measures this pattern indirectly through changes in the retained anchors.
Across the training epochs reported in Table 8, at least 30.4% of rollouts received fine ranking. The original paper already includes a coarse-only ablation (Table 5). Removing fine ranking reduced performance on all four benchmarks, showing that resolving ties within anchor intervals improves downstream performance.
Q2.2 Dynamic maintenance versus a static buffer
Table 2 directly tests the score–cost trade-off of continuing buffer updates. With Nectar initialization, dynamic maintenance raised the average benchmark score from 55.01 to 57.27, with judge tokens increasing from 68.0M to 88.8M. Figure 6 helps explain where this benefit may come from. The buffer adds anchors above the initial quality range and refines higher-quality interior intervals where rollouts tend to cluster. Table 2 captures the net effect on benchmark scores and judging cost.
With bootstrap initialization, the initial anchors already span the first rollout group, from its lowest- to highest-ranked response. A plausible explanation is that these policy-generated anchors remained useful throughout the four-epoch run, leaving less room for continued updates to improve the ranking. The static buffer scored 56.54 versus 55.34 for the dynamic buffer. Judge calls rose only from 28.1K to 28.4K with dynamic maintenance, while tokens rose from 60.7M to 76.5M. The increase therefore came mainly from higher token use per call, consistent with the added anchor context.
Q3. Coarse insertion errors and error propagation
Judgment errors are not unique to RankBuffer but are also inherent in pointwise, listwise, and other ranking-based methods. As stated in Sec. 4.1, all methods use the same judge model under controlled experimental settings. Therefore, potential judgment errors are a shared limitation rather than an issue exclusive to RankBuffer. Under this consistent judge setting, RankBuffer outperforms all pointwise baselines and achieves competitive performance against the strongest ranking-based methods while substantially reducing judging costs. These results demonstrate the effectiveness of RankBuffer despite the potential judgment errors shared across methods.
Thank you for the detailed response and the additional experiments. The independent evaluation and wall-clock measurements largely address my main concerns about the evidence supporting the paper’s claims. In light of these results, I will consider raising my score. I encourage the authors to incorporate the new results and the corresponding experimental details into the revised manuscript to make the evidence for the main claims more complete. If a revised manuscript is made available, I will assess it and update my review accordingly.
Thank you for your review. We regret that our submission did not fully convince you. We respectfully disagree with some of the weaknesses and address your concerns point by point below and in the follow-up comment. If our response resolves your concerns, we would be grateful if you would reconsider your overall score.
W1. Judge dependence and validation
W1.1 Evaluation with a different judge
We kept Qwen3-30B-A3B as the training judge and used GPT-6 Luna to evaluate the trained policies on all four benchmarks. Consistent with the findings in our paper, RankBuffer outperformed the pointwise baselines and achieved competitive performance relative to other ranking-based methods at a lower judging cost.
W1.2 Agreement across judges
We sampled 200 queries from the training set and measured agreement between judges on the rollout rankings using Kendall’s τ.
Qwen3's agreement with each alternative judge (0.78 and 0.81) was close to the agreement between the two alternative judges (0.84). This suggests that the three judges broadly agree on the relative ordering of the sampled rollouts.
W1.3 Human preference validation
The benchmarks used in our study incorporate human preferences to varying degrees in their design or validation. Therefore, improvements on these benchmarks provide, to some extent, indirect evidence of alignment with human preferences.
W1.4 Recalibrating the buffer
We understand the reviewer’s question about a change of judge as referring to a switch within a single training run, such as using GPT for the first 100 steps and DeepSeek for the next 100. In practice, the reward judge is typically held fixed throughout training to provide a consistent reward signal. We respectfully disagree that the reward judge needs to be changed within a single training run.
W2. Practical efficiency
W2.1 End-to-end training time
We measured end-to-end wall-clock training time and the results are shown below. RankBuffer completed training in 7h 10m, 33.5% less time than ArenaRL, the fastest baseline in this comparison.
As the reviewer notes, RankBuffer has a sequential dependency. However, other ranking-based baselines also require sequential stages. Tournament-GRPO executes its bracket rounds in order, and ArenaRL completes seed comparisons before starting its elimination rounds. All five ranking-based baselines took longer to train than RankBuffer in our experiments.
W2.2 Adaptive Elo
RRC shares a similar underlying idea with the suggested Elo-style approach by evaluating new responses through pairwise comparisons against reference responses. We have already compared RankBuffer with RRC in the original paper (Table 8). RankBuffer achieves a higher average benchmark score (57.27 vs. 51.61) while reducing training judge tokens by 55.8%. Although adaptive Elo-style methods may reduce the number of comparisons, they still rely on iterative pairwise judgments, potentially introducing sequential dependencies. In contrast, RankBuffer directly determines the insertion interval among multiple anchors in a single, parallelizable judgment, avoiding such iterative comparisons.