acceptodds
Under review as a conference paper at ICLR 2027

RankBuffer: Efficient Ranking-Based Rewards for Open-Ended Generation

Abstract

Open-ended generation lacks canonical answers, making pointwise rewards difficult to calibrate for group-based reinforcement learning. Directly ranking same-query rollouts provides a more suitable relative reward signal, but existing ranking-based reward methods can incur substantial judging cost. We introduce RankBuffer, which maintains an ordered, query-specific buffer of previously judged responses as a reusable quality scale. Each rollout is first inserted into an anchor interval through an independent coarse judgment, after which only rollouts assigned to the same interval undergo local fine ranking. The resulting complete order is converted into bounded rank rewards, while boundary expansion, local refinement, and inactive-anchor pruning adapt the buffer as the policy evolves. Across four open-ended benchmarks, RankBuffer consistently outperforms all pointwise baselines. It also achieves nearly on-par performance with the strongest ranking-based reward baseline while substantially reducing judging cost. Ablations demonstrate the importance of both local fine ranking and anchor response content, while buffer analyses show that rollout-derived anchors progressively extend and refine the covered quality scale. These results establish response reuse as an effective approach to efficient relative reward construction.

Primary area:
reinforcement learning
Keywords:
Open-Ended Generation, Reinforcement Learning, LLM-as-a-Judge, Ranking-Based Rewards

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_N4NBOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The paper proposes RankBuffer for efficient ranking-based rewards in reinforcement learning. It reuses an ordered, query-specific buffer of historical responses, combines coarse insertion with local fine ranking, and dynamically updates the buffer.

Critical strengths

The motivation is clear, and the coarse-to-fine ranking framework is straightforward to understand and implement.

Critical weaknesses
  • The quality scale depends on a fixed judge and lacks independent validation. Sec. 3.3 treats the stored anchor order as an established quality scale, but its consistency across judges or with human preferences is not evaluated. Moreover, Sec. 4.1 uses the same Qwen model for both training rewards and all four benchmark evaluations. The reported improvements therefore lack independent confirmation beyond the training judge’s preferences. It is also unclear how the buffer would be recalibrated if the judge changes or historical comparisons come from heterogeneous evaluators (like LM Arena).

  • The reported cost reductions do not establish practical latency gains. In Sec. 3.3, local fine ranking depends on completing coarse insertion for the corresponding group, creating two sequential stages despite parallelism within each stage. Figures 4–5 and Table 8 report tokens and successfully parsed calls, but provide no latency judging or training time. Consequently, the practical efficiency of this scheduling design remains insufficiently demonstrated. If the latency introduced by multi-stage comparisons has a negligible impact in this scenario, does Elo-style mechanism that reuses historical ratings and adaptively compares responses with nearby ratings also work well?

Questions for authors
  1. How does RankBuffer compare with directly ranking all eight rollouts in one listwise call, using the prompt already employed for bootstrap initialization in Sec. 3.2?
  2. How concentrated are rollouts across anchor intervals, and what fraction requires fine ranking throughout training? Does dynamic maintenance improve these quantities enough to offset longer anchor contexts? Why does dynamic maintenance perform worse and consume more tokens than the static buffer under bootstrap initialization in Table 2?
  3. How often do coarse insertion errors produce incorrect cross-interval orders? Can such errors persist after the affected responses become anchors?
5 comments

Thank you for your review. We regret that our submission did not fully convince you. We respectfully disagree with some of the weaknesses and address your concerns point by point below and in the follow-up comment. If our response resolves your concerns, we would be grateful if you would reconsider your overall score.

W1. Judge dependence and validation

W1.1 Evaluation with a different judge

We kept Qwen3-30B-A3B as the training judge and used GPT-6 Luna to evaluate the trained policies on all four benchmarks. Consistent with the findings in our paper, RankBuffer outperformed the pointwise baselines and achieved competitive performance relative to other ranking-based methods at a lower judging cost.

MethodAlpacaEval 2Arena-Hard v2WildBench v2WritingBenchAverage
Base model24.7811.5039.3859.2333.72
GRPO62.4330.4051.4466.3152.65
GDPO59.3129.1053.0965.2851.70
Dr. GRPO53.8132.3050.9867.4051.12
DAPO57.4928.6049.4668.6351.05
GPG54.9227.7048.2164.4248.81
RankBuffer79.1936.8059.2571.8661.78
MethodAverageTraining judge tokensTraining judge calls
Tournament-GRPO61.07147,039,27485,134
RRC-AGR55.04217,401,740237,470
RRC-SCR53.22204,710,918149,392
Pairwise Rank58.79198,810,740129,493
ArenaRL60.31176,496,401132,841
RankBuffer61.7886,747,07233,909

W1.2 Agreement across judges

We sampled 200 queries from the training set and measured agreement between judges on the rollout rankings using Kendall’s τ.

Judge pairKendall's τ
Qwen3 vs. GPT-6 Luna0.81
Qwen3 vs. DeepSeek-V4.1-Flash0.78
GPT-6 Luna vs. DeepSeek-V4.1-Flash0.84

Qwen3's agreement with each alternative judge (0.78 and 0.81) was close to the agreement between the two alternative judges (0.84). This suggests that the three judges broadly agree on the relative ordering of the sampled rollouts.

W1.3 Human preference validation

The benchmarks used in our study incorporate human preferences to varying degrees in their design or validation. Therefore, improvements on these benchmarks provide, to some extent, indirect evidence of alignment with human preferences.

W1.4 Recalibrating the buffer

We understand the reviewer’s question about a change of judge as referring to a switch within a single training run, such as using GPT for the first 100 steps and DeepSeek for the next 100. In practice, the reward judge is typically held fixed throughout training to provide a consistent reward signal. We respectfully disagree that the reward judge needs to be changed within a single training run.

W2. Practical efficiency

W2.1 End-to-end training time

We measured end-to-end wall-clock training time and the results are shown below. RankBuffer completed training in 7h 10m, 33.5% less time than ArenaRL, the fastest baseline in this comparison.

MethodEnd-to-end training time
Tournament-GRPO14h 31m
RRC-SCR12h 58m
RRC-AGR11h 02m
Pairwise Rank11h 29m
ArenaRL10h 47m
RankBuffer7h 10m

As the reviewer notes, RankBuffer has a sequential dependency. However, other ranking-based baselines also require sequential stages. Tournament-GRPO executes its bracket rounds in order, and ArenaRL completes seed comparisons before starting its elimination rounds. All five ranking-based baselines took longer to train than RankBuffer in our experiments.

W2.2 Adaptive Elo

RRC shares a similar underlying idea with the suggested Elo-style approach by evaluating new responses through pairwise comparisons against reference responses. We have already compared RankBuffer with RRC in the original paper (Table 8). RankBuffer achieves a higher average benchmark score (57.27 vs. 51.61) while reducing training judge tokens by 55.8%. Although adaptive Elo-style methods may reduce the number of comparisons, they still rely on iterative pairwise judgments, potentially introducing sequential dependencies. In contrast, RankBuffer directly determines the insertion interval among multiple anchors in a single, parallelizable judgment, avoiding such iterative comparisons.

Q1. Comparison with direct listwise ranking

We added a listwise baseline that ranks all rollouts in each group in a single judge call.

MethodAlpacaEval 2Arena-Hard v2WildBench v2WritingBenchAverage
Listwise67.1834.9055.8468.1356.51
RankBuffer79.1936.8059.2571.8661.78

RankBuffer outperforms direct listwise ranking on all four benchmarks, with an average improvement of 5.27 points. For open-ended long-form generation, putting all responses into a single prompt creates a long context, which can make it harder for the judge to distinguish differences in response quality. Moreover, without anchor responses as references, the judge must infer a quality scale from the candidates alone, and its ranking may be sensitive to their presentation order. In contrast, RankBuffer uses reusable anchors for coarse insertion and limits fine ranking to rollouts within each anchor interval, reducing the scope of each ranking decision.

Q2. Rollout distribution and dynamic buffer maintenance

Q2.1 Where rollouts concentrate and how often fine ranking is used

The evolution of the buffer in Sec. 4.4 (Fig. 6) gives an indication of where rollouts concentrate. Local refinement adds an anchor only when an interior interval contains at least three rollouts, and these refinement anchors accumulate mainly in higher-quality interior regions. At the same time, the share of anchors in the highest rank bin added through upper-boundary expansion rises from 71% after the first epoch to 82% after the last. Together, these patterns suggest that rollouts frequently cluster in upper intervals and reach beyond the existing upper boundary as training progresses. Figure 6 measures this pattern indirectly through changes in the retained anchors.

Across the training epochs reported in Table 8, at least 30.4% of rollouts received fine ranking. The original paper already includes a coarse-only ablation (Table 5). Removing fine ranking reduced performance on all four benchmarks, showing that resolving ties within anchor intervals improves downstream performance.

Q2.2 Dynamic maintenance versus a static buffer

Table 2 directly tests the score–cost trade-off of continuing buffer updates. With Nectar initialization, dynamic maintenance raised the average benchmark score from 55.01 to 57.27, with judge tokens increasing from 68.0M to 88.8M. Figure 6 helps explain where this benefit may come from. The buffer adds anchors above the initial quality range and refines higher-quality interior intervals where rollouts tend to cluster. Table 2 captures the net effect on benchmark scores and judging cost.

With bootstrap initialization, the initial anchors already span the first rollout group, from its lowest- to highest-ranked response. A plausible explanation is that these policy-generated anchors remained useful throughout the four-epoch run, leaving less room for continued updates to improve the ranking. The static buffer scored 56.54 versus 55.34 for the dynamic buffer. Judge calls rose only from 28.1K to 28.4K with dynamic maintenance, while tokens rose from 60.7M to 76.5M. The increase therefore came mainly from higher token use per call, consistent with the added anchor context.

Q3. Coarse insertion errors and error propagation

Judgment errors are not unique to RankBuffer but are also inherent in pointwise, listwise, and other ranking-based methods. As stated in Sec. 4.1, all methods use the same judge model under controlled experimental settings. Therefore, potential judgment errors are a shared limitation rather than an issue exclusive to RankBuffer. Under this consistent judge setting, RankBuffer outperforms all pointwise baselines and achieves competitive performance against the strongest ranking-based methods while substantially reducing judging costs. These results demonstrate the effectiveness of RankBuffer despite the potential judgment errors shared across methods.

Thank you for the detailed response and the additional experiments. The independent evaluation and wall-clock measurements largely address my main concerns about the evidence supporting the paper’s claims. In light of these results, I will consider raising my score. I encourage the authors to incorporate the new results and the corresponding experimental details into the revised manuscript to make the evidence for the main claims more complete. If a revised manuscript is made available, I will assess it and update my review accordingly.

Reviewer_uDsCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a reward construction problem in group-based RL for open-ended generation: pointwise rubric rewards are hard to calibrate without canonical answers, and independently assigned scores often cluster within a rollout group, producing vanishing or noisy GRPO advantages. Besides, existing ranking-based reward methods fix the signal but incur substantial judging cost because every rollout group is re-compared from scratch. To address this problem, the paper designs RankBuffer, a buffer-assisted coarse-to-fine ranking framework that maintains an ordered, query-specific buffer of previously judged responses as a reusable quality scale: each new rollout is inserted into an anchor interval through a single independent, parallelizable judgment, only rollouts landing in the same interval undergo local fine ranking, the concatenated complete order is mapped to bounded rank rewards, and rule-based maintenance (boundary expansion, local refinement, inactive-anchor pruning) adapts the buffer as the policy evolves. Results on four open-ended benchmarks show that RankBuffer outperforms all pointwise baselines, achieves nearly on-par performance with the strongest ranking-based baseline RRC-SCR while using less than half of its judge tokens, and ablations confirm that both local fine ranking and anchor response content are necessary components, with buffer analyses showing that rollout-derived anchors progressively extend the covered quality scale during training.

Critical strengths

S1: The reward design removes score-scale instability by anchor-based reward construction

The rank-to-reward mapping makes rewards invariant to any monotone change of the judge's scores. The classic point-wise failure mode, clustered within-group scores and vanishing advantages, is removed by locating the response relative to concrete anchors.

Critical weaknesses

W1: The reward construction does not consider the judge's confidence, so an incorrect ranking yields the same set of advantage magnitudes as a correct one. Under a strict total order, the rank mapping (Eq. 9) and group normalization (Eq. 11) fix the advantages to a predetermined set of values. The judge only decides which rollout receives which rank. The paper claims that independent insertion plus local fine ranking produces reliable relative rewards, but it does not validate how reliable the judge is. Because incorrect and correct rankings induce the same advantage magnitudes, judge errors are not down-weighted.

W2: A list-wise ranking baseline is missing from the comparisons. The simplest alternative is to rank all rollouts in one list-wise judgment at each step, and to compare both quality and cost against RankBuffer.

Questions for authors

Q1: Could you detail how ranking fidelity and stability are measured? In particular, (1) agreement between RankBuffer's full ordering and an independent re-ranking, (2) the consistency of insertion decisions under re-judging, and (3) the sensitivity of both to anchor presentation order.

3 comments

Thank you for your thoughtful review. To address your concerns, we report additional experimental results below.

W1 / Q1. Judge confidence and ranking stability

First, with the same training judge, RankBuffer outperformed the baselines on all four benchmarks. These downstream gains provide indirect evidence that its ranking-based rewards can guide policy optimization effectively even when individual judgments may be imperfect.

Second, judgment errors are a shared limitation of methods that rely on LLM judges to construct rewards, including pointwise, pairwise, and other ranking-based methods. All methods in our paper use the same judge model, which holds the underlying judging capabilities constant and controls for differences attributable to the choice of judge. Using a shared judge provides a fair basis for comparing the methods with respect to the choice of judge.

Agreement across judges

We sampled 200 queries from the training set and measured agreement between judges on the corresponding rollout rankings using Kendall’s τ. Qwen3’s agreement with each alternative judge was close to the agreement between the two alternative judges. This suggests that the relative ordering of the sampled rollouts was broadly consistent across judges.

Judge pairKendall’s τ
Qwen3 vs. GPT-6 Luna0.81
Qwen3 vs. DeepSeek-V4.1-Flash0.78
GPT-6 Luna vs. DeepSeek-V4.1-Flash0.84

Sensitivity to anchor presentation order

We tested sensitivity to anchor presentation order by reversing the order of anchors in the coarse insertion prompt. The original RankBuffer presents anchors from worst to best, and the new variant presents them from best to worst.

Anchor presentation orderAlpacaEval 2Arena-Hard v2WildBench v2WritingBenchAverage
Best-to-worst78.4537.2059.9172.0361.90
Worst-to-best (original)79.1936.8059.2571.8661.78

Reversing the anchor presentation order changed the average score by only 0.12 points. The absolute difference was below 0.75 points on every benchmark, indicating similar downstream performance under the two presentation orders tested.

W2. Comparison with direct listwise ranking

We added a direct listwise baseline that uses the bootstrap initialization prompt (Appendix B.2) throughout training to rank all the rollouts in each group in a single judge call. We held GRPO and the training judge fixed and used GPT-6 Luna for benchmark evaluation.

MethodAlpacaEval 2Arena-Hard v2WildBench v2WritingBenchAverage
Direct Listwise67.1834.9055.8468.1356.51
RankBuffer79.1936.8059.2571.8661.78

RankBuffer outperformed Direct Listwise on all four benchmarks, with a 5.27-point gap in the reported averages. For open-ended long-form generation, putting all responses into a single prompt creates a long context, which can make it harder for the judge to distinguish differences in response quality. Moreover, without anchor responses as references, the judge must infer a quality scale from the candidates alone, and its ranking may be sensitive to their presentation order. In contrast, RankBuffer uses reusable anchors for coarse insertion and limits fine ranking to rollouts within each anchor interval, reducing the scope of each ranking decision.

Thank you for the helpful response and additional experiments. The direct list-wise baseline addresses the missing comparison, and the cross-judge agreement and anchor-order experiments strengthen the empirical support for RankBuffer. However, results of W1/Q1 do not directly establish the fidelity of RankBuffer’s complete ordering. Considering the additional evidence and this remaining concern, I will maintain my original score.

Dear Reviewer,

Thank you for your follow-up and for considering our additional experiments.

Regarding your concern that “results of W1/Q1 do not directly establish the fidelity of RankBuffer’s complete ordering,” we would like to offer one final clarification.

The primary goal of RankBuffer’s ranking mechanism is to construct reward signals that effectively guide policy optimization. Its practical effectiveness can therefore be assessed through the trained policy’s performance on downstream tasks. Our results across four benchmarks show that RankBuffer consistently improves model performance. Comparisons with the baselines in both the original paper and the rebuttal experiments further demonstrate RankBuffer’s advantages and support the effectiveness of its reward signals for policy optimization.

We acknowledge that these results do not directly establish the accuracy of the complete ordering. We nevertheless believe that they provide strong empirical support for the practical effectiveness of RankBuffer’s ranking mechanism in reinforcement learning.

Thank you again for your constructive feedback and consideration.

Reviewer_nqEVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RankBuffer constructs relative rewards for open-ended generation by maintaining a persistent, query-specific buffer of ordered anchor responses. Each new rollout is independently assigned to an anchor interval, and only rollouts sharing an interval undergo local listwise ranking. The resulting order is mapped to bounded rank rewards for GRPO. The buffer can be initialized from Nectar responses or first-epoch policy rollouts and maintained through boundary expansion, local refinement, and inactive-anchor pruning. Experiments with Qwen3-8B cover four benchmarks evaluated by the same model used for training judgments. RankBuffer improves the strongest reported pointwise baseline by 3.83 average points and approaches RRC-SCR performance while using 55.8% fewer judge tokens counted over successfully parsed requests. The contribution is primarily a practical cost–quality improvement.

Critical strengths
  1. The measured cost–quality trade-off is useful and supported by multiple competitive baselines. RankBuffer improves all four benchmark scores over the reported pointwise methods. Its average score is 57.27 versus 57.71 for RRC-SCR, while successfully counted judge tokens decrease from 200.82M to 88.82M and requests from 104.03K to 36.86K. The comparisons with ArenaRL and Tournament-GRPO also support the value of the approach. This practical benefit matters even though the method combines familiar ranking components. [pp. 6–7, 18, Tables 1 and 8]

  2. The design is understandable and accompanied by informative analyses. The paper tests initialization sources, static and dynamic buffers, LLM-based maintenance, anchor counts, and removal of local fine ranking or anchor content. Fine ranking improves all four scores over coarse-only ranking. The authors also report that static bootstrap buffers outperform their dynamic counterpart, providing a useful boundary on the maintenance claim. Pseudocode and judging prompts make the main procedure clear. [pp. 7–9, 13–19]

Critical weaknesses
  1. Training and all four evaluations use the same Qwen3-30B-A3B judge. This supports a consistent comparison across methods, but does not independently establish that the improvements generalize beyond that evaluator’s preferences. The PDF also leaves evaluation metrics, reference responses, and length-control details insufficiently specified, and reports no run-to-run uncertainty. These limitations qualify the quality claim rather than demonstrate reward hacking or invalidate the observed improvements. [p. 6, §4.1; pp. 15–18]

  2. A natural efficiency comparator is missing: repeatedly ranking the full rollout group listwise, as already done during bootstrap initialization. Its quality cannot be assumed, but the comparison would clarify the benefit specifically attributable to persistent anchors. “Near-pointwise” cost also requires qualification: RankBuffer uses approximately 1.18 times DAPO’s requests but 3.69 times its judge tokens. Reported totals exclude failed or unparseable requests and do not establish end-to-end or wall-clock savings. These issues limit the scope of the efficiency conclusion without erasing the measured savings against the tested ranking baselines. [pp. 4, 6, 15, 18]

Questions for authors
  1. Could you specify the benchmark metrics, evaluation prompts, reference responses, and length-control settings? If available, would a modest independent-judge or human comparison preserve the principal ranking of methods? This would help distinguish improvement under the shared evaluator from more general response-quality improvement; a large new evaluation suite is not necessary to address the concern. [p. 6, §4.1; p. 18, Table 8]

  2. How does RankBuffer compare with using the existing full-group listwise bootstrap judge throughout training under matched settings? Please also clarify retry-inclusive judging costs, or explicitly limit the savings claim to successfully parsed requests. This comparison would locate the additional value of persistent anchors without presuming that the simpler alternative has equal quality. [pp. 4, 15, 17–18]

1 comment

Thank you for the constructive review. To address your concerns, we conducted several additional experiments and report the results below.

W1 / Q1. Evaluation protocol and independent validation

W1.1 Benchmark evaluation protocol

We followed the published benchmark protocols. For each benchmark, all reported checkpoints used the same metric, judge prompt, reference responses (where applicable), and length-control settings.

BenchmarkReported metricJudge prompt and referenceLength handling
AlpacaEval 2Length-controlled win rate (%)Pairwise preference against the released GPT-4 Turbo response, with randomized response order.Regression estimates win rate at equal response length.
Arena-Hard v2Estimated win rate (%) against o3-miniThe judge drafts an answer, then compares the candidate and released o3-mini response in both orders.Style control adjusts for length and Markdown.
WildBench v2WB-Score = 20 × (mean score − 5)Pointwise 1–10 rating using the query, conversation history, and task checklist.The optional length margin applies to pairwise WB-Reward.
WritingBenchMean criterion score × 10Pointwise 1–10 ratings on five query-specific criteria.Length adherence is scored where specified.

W1.2 Evaluation with a separate judge

We kept Qwen3-30B-A3B as the training judge and used GPT-6 Luna for all four benchmarks. The main findings held under the separate judge. RankBuffer outperformed the pointwise baselines and remained competitive with the ranking-based baselines at lower judging cost.

MethodAlpacaEval 2Arena-Hard v2WildBench v2WritingBenchAverage
Base model24.7811.5039.3859.2333.72
GRPO62.4330.4051.4466.3152.65
GDPO59.3129.1053.0965.2851.70
Dr. GRPO53.8132.3050.9867.4051.12
DAPO57.4928.6049.4668.6351.05
GPG54.9227.7048.2164.4248.81
RankBuffer79.1936.8059.2571.8661.78
MethodAverageTraining judge tokensTraining judge calls
Tournament-GRPO61.07147,039,27485,134
RRC-AGR55.04217,401,740237,470
RRC-SCR53.22204,710,918149,392
Pairwise Rank58.79198,810,740129,493
ArenaRL60.31176,496,401132,841
RankBuffer61.7886,747,07233,909

W2 / Q2. Full-group listwise ranking and judging efficiency

W2.1 Direct listwise comparison

Under the same GRPO training setup, we added a direct listwise baseline (using the bootstrap initialization prompt) and several pointwise reward baselines. For each benchmark, we report the mean and standard deviation across multiple runs. GPT-6 Luna served as the benchmark judge. Benchmark results are reported as mean ± standard deviation.

MethodAlpacaEval 2Arena-Hard v2WildBench v2WritingBenchAverage
Base model24.78 ± 0.4211.5 ± 0.339.38 ± 0.7559.23 ± 0.3633.72
Pointwise
Direct Pointwise62.43 ± 0.5630.4 ± 0.251.44 ± 0.6266.31 ± 0.4452.65
Rubric-wise Weighted60.77 ± 0.5530.6 ± 0.351.66 ± 0.5464.65 ± 0.5151.92
Rubric-wise Binary57.31 ± 0.3828.9 ± 0.450.31 ± 0.3264.13 ± 0.6350.16
G-Eval-style63.49 ± 0.3931.6 ± 0.548.24 ± 0.6766.20 ± 0.4252.38
Listwise
Direct Listwise67.18 ± 0.2134.9 ± 0.355.84 ± 0.6468.13 ± 0.2956.51
Our method
RankBuffer79.19 ± 0.3436.8 ± 0.559.25 ± 0.2071.86 ± 0.6761.78

RankBuffer outperformed Direct Listwise on all four benchmarks, with a 5.27-point gap in the reported averages. For open-ended long-form generation, placing all responses in a single prompt creates a long context that can make subtle differences in quality harder to discern. Without reference anchors, the judge must infer a quality scale from the candidates alone, and its rankings may be sensitive to the order in which the responses are presented.

W2.2 Judging costs and retries

All judging costs reported in the revised paper include retries in both the judge-call and token totals.

W2.3 End-to-end training time

We measured the end-to-end wall-clock training time for RankBuffer and five ranking-based baselines.

MethodWall-clock training time
Tournament-GRPO14h 31m
RRC-SCR12h 58m
RRC-AGR11h 02m
Pairwise Rank11h 29m
ArenaRL10h 47m
RankBuffer7h 10m

Tournament-GRPO and ArenaRL require multiple sequential judging rounds, which can add latency. RankBuffer parallelizes coarse insertion and uses at most one follow-up round for fine ranking. It took 7h 10m, 33.5% less time than ArenaRL.

Reviewer_pvdwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces RankBuffer to reduce judge model's cost of group-based RL training in open-ended generation task. RankBuffer continuously updates a "buffer" storing previous responses and corresponding ranking for the same query, and uses it for RL training's advantage calculation.

Critical strengths

The problem formulation is clear, paper and figures are well written. The efficiency of methods are clear in terms of token used and judge calls.

Critical weaknesses

Key weaknesses

  1. It is not entirely clear to me whether the training scenarios that utilize RankBuffer are realistic. Since RankBuffer requires the model to revisit the same queries during RL training, necessitating multiple epochs, it seems more beneficial to incorporate additional data instances rather than repeatedly cycling through the same data.

  2. It seems the experimental setup for the pointwise reward comparison is incorrect. In section 4.2 of the experiments, the focus is on the pointwise reward method, and the section compares various RL algorithms. In my view, while authors should stick with GRPO as the training algorithm, authors ought to compare it against other existing pointwise methods. In this regard, one of the most naive approaches, which should be considered for comparison, is to provide all the responses to a judge model and have it rank them all at once.

Minor weaknesses

  1. There is no distinction between the judge model used during training and the one used for evaluation (both are Qwen3-30B). This could lead to the model simply overfitting to the judge model.

  2. In my opinion, 200 steps of RL training is too little, and the variance is likely to be high. It would be worth examining the learning curve and running tests across multiple random seeds.

  3. As the buffer is updated, the degree of "off-policyness" could become excessively high. In particular, if the buffer is initialized using the Nectar approach, there is likely to be a significant discrepancy between the old logprobs and the current logprobs. I suspect this might impact the RL training.

Questions for authors
  1. Are there any scenarios in RL tarining where it would be practical or necessary to go thorugh multiple epochs? Since generating queries is relatively easy. For instance, one could use an LLM to perform data augmentation in a fairly naive way. Wouldn't it be better to simply generate more quries to train the policy?

  2. Are there any other pointwise reward models available? Is it possible to compare them?

It would be good to also look at the minor points I've noted. (But not critical to me.)

Additionally, here are some minor points I noticed in the paper.

  1. In section 2.1 (Related Works), it would be benefical to more clearly articulate the differences from RankBuffer.

  2. In section 4.1, it would be helpful to define the abbreviations used for each evaluation dataset (e.g., AlpacaEval2, Arena-Hard v2, etc.).

  3. It would be useful to compare the complexity of this approach against other ranking-based reward methods. For instance, by using big O notation.

7 comments

Thank you for your thoughtful review. To address your concerns, we conducted several additional experiments.

Minor Weakness 1. Evaluation with a separate judge

We kept Qwen3-30B-A3B as the training judge and used GPT-6 Luna as the benchmark judge. RankBuffer outperformed the pointwise baselines and achieved competitive performance relative to ranking-based methods at a lower judging cost.

MethodAlpacaEval 2Arena-Hard v2WildBench v2WritingBenchAverage
Base model24.7811.5039.3859.2333.72
GRPO62.4330.4051.4466.3152.65
GDPO59.3129.1053.0965.2851.70
Dr. GRPO53.8132.3050.9867.4051.12
DAPO57.4928.6049.4668.6351.05
GPG54.9227.7048.2164.4248.81
RankBuffer79.1936.8059.2571.8661.78
MethodAverageTraining judge tokensTraining judge calls
Tournament-GRPO61.07147,039,27485,134
RRC-AGR55.04217,401,740237,470
RRC-SCR53.22204,710,918149,392
Pairwise Rank58.79198,810,740129,493
ArenaRL60.31176,496,401132,841
RankBuffer61.7886,747,07233,909

Key Weakness 1 / Q1. Revisiting queries versus adding more data

We compared the original RankBuffer setup (1,000 queries for four epochs) with single-pass runs over 1,000 to 4,000 distinct queries. GPT-6 Luna served as the benchmark judge.

Training configurationAlpacaEval 2Arena-Hard v2WildBench v2WritingBenchAverage
Base model24.7811.539.3859.2333.72
1,000 queries × 1 epoch37.0219.043.4161.5040.23
2,000 queries × 1 epoch45.1323.548.6465.5745.71
3,000 queries × 1 epoch50.1630.450.5268.4249.88
4,000 queries × 1 epoch52.6433.854.8069.9052.79
RankBuffer (1,000 queries × 4 epochs)79.1936.859.2571.8661.78

At the same total of 4,000 query presentations, the single-pass run scored below RankBuffer on all four benchmarks. Expanding the query set also adds preparation costs, especially when collecting vetted prompts or real user requests. Even with synthetic augmentation, generating new queries and their judging rubrics consumes additional tokens.

Key Weakness 2 / Q2, Minor Weakness 2. Pointwise and listwise baselines under GRPO (variation across runs)

As suggested by the reviewer, we kept GRPO fixed as the training algorithm and added pointwise and listwise reward baselines. The pointwise baselines use weighted rubric-wise scoring, binary rubric-wise scoring, and G-Eval-style scoring. The listwise baseline ranks all rollouts in each group in a single judge call using the prompt (Appendix B.2).

MethodAlpacaEval 2Arena-Hard v2WildBench v2WritingBenchAverage
Base model24.78 ± 0.4211.5 ± 0.339.38 ± 0.7559.23 ± 0.3633.72
Pointwise
Direct Pointwise62.43 ± 0.5630.4 ± 0.251.44 ± 0.6266.31 ± 0.4452.65
Rubric-wise Weighted60.77 ± 0.5530.6 ± 0.351.66 ± 0.5464.65 ± 0.5151.92
Rubric-wise Binary57.31 ± 0.3828.9 ± 0.450.31 ± 0.3264.13 ± 0.6350.16
G-Eval-style63.49 ± 0.3931.6 ± 0.548.24 ± 0.6766.20 ± 0.4252.38
Listwise
Direct Listwise67.18 ± 0.2134.9 ± 0.355.84 ± 0.6468.13 ± 0.2956.51
Our method
RankBuffer79.19 ± 0.3436.8 ± 0.559.25 ± 0.2071.86 ± 0.6761.78

For open-ended long-form generation, putting all responses into a single prompt creates a long context, which can make it harder for the judge to distinguish differences in response quality. Moreover, without anchor responses as references, the judge must infer a quality scale from the candidates alone, and its ranking may be sensitive to their presentation order.

Minor Weakness 3. Historical anchors and off-policy training

RankBuffer computes the GRPO policy loss exclusively from fresh rollouts sampled for the current iteration (Eq. 1 and Sec. 3.4). The old and current policy log-probabilities in the GRPO ratio are both evaluated on those fresh rollouts. Historical anchors serve only as references for the judge when assigning rewards.

For open-ended queries, LLM judges can use rubrics or teacher-generated responses to assess rollouts. Historical anchors serve the same role in RankBuffer, informing rewards while only freshly sampled rollouts serve as training data. RankBuffer thus preserves on-policy sampling for GRPO.

For these three minor points, we commit to making the relevant additions and revisions in the camera-ready version if the paper is accepted. Thank you again for your valuable feedback.

Thank you for your response. My concern about off-policyness has been resolved. However, a few questions remain, which I list below.

  • In "Minor Weakness 1. Evaluation with a separate judge," only the evaluation judge was changed, yet the reported training judge calls differ considerably from those in the paper (Table 8). Could you explain why?
  • Regarding "Key Weakness 1 / Q1. Revisiting queries versus adding more data," thank you for the new experiment. What I wanted to understand, however, is the practical scenario in which one would use RankBuffer. For example, is it simply because its performance is substantially better? If so, why? Could you also describe the experimental setup in more detail, and explain why the performance gap is so large?
  • For "Key Weakness 2 / Q2, Minor Weakness 2," what are the judging costs of each method behind these results?
  • Could you also explain how the results should be interpreted overall? At present, the response mostly presents results without much interpretation.

1. Why the training costs differ from Table 8

The additional benchmark evaluations used newly trained checkpoints. Since some of the original checkpoints used in Table 8 were lost, we reran the training experiments using Qwen3-30B-A3B as the training judge and GPT-6 Luna as the benchmark judge. The reported training costs therefore correspond to these new runs.

2. Query reuse, experimental setup, and practical applications

2.1 Purpose of the comparison

We added this experiment to examine your suggestion that training on additional queries may be preferable to revisiting existing ones. The main comparison is 1,000 queries × 4 epochs versus 4,000 distinct queries × 1 epoch.

2.2 Experimental setup

The original setup uses 1,000 queries sampled from Nectar and trains for four epochs, as described under “Training data and buffer initialization” in Sec. 4.1.

For the single-pass settings, we retained these 1,000 queries and randomly sampled 1,000, 2,000, or 3,000 additional distinct queries from the remaining Nectar queries. This produced datasets of 2,000, 3,000, and 4,000 queries, each used for one epoch. The 1,000-query, one-epoch result is the first-epoch checkpoint of the repeated-query setting.

All experiments used the same base-policy initialization, GRPO optimizer, judge model, number of rollouts per query, rubric-construction procedure, buffer-update rules, and other training hyperparameters. Only the number of distinct queries and epochs varied. GPT-6 Luna evaluated all resulting policies.

2.3 Results

Training configurationAlpacaEval 2Arena-Hard v2WildBench v2WritingBenchAverage
Base model24.7811.539.3859.2333.72
1,000 queries × 1 epoch37.0219.043.4161.5040.23
2,000 queries × 1 epoch45.1323.548.6465.5745.71
3,000 queries × 1 epoch50.1630.450.5268.4249.88
4,000 queries × 1 epoch52.6433.854.8069.9052.79
RankBuffer (1,000 queries × 4 epochs)79.1936.859.2571.8661.78

The intermediate single-pass settings show how performance changes as query coverage increases. Their average rises from 40.23 to 52.79. Compared with training on 4,000 queries × 1 epoch, RankBuffer (1,000 queries × 4 epochs) scores higher on all four benchmarks.

2.4 Interpretation and the role of the buffer (why rankbuffer better)

RankBuffer retains selected earlier rollouts and the judge’s assessment of their relative quality through the stored anchor order. These responses provide references for subsequent training on the same query. As the policy evolves, boundary expansion extends the covered quality range, while local refinement adds anchors where rollouts concentrate. The judge can therefore assess fresh rollouts against a reference scale that adapts throughout training.

The 4,000 queries × 1 epoch and RankBuffer (1,000 queries × 4 epochs) settings have the same total number of query presentations. In the latter, later rollout groups benefit from earlier judgments for the same query. With 4,000 queries × 1 epoch, each query produces only one training rollout group, so its buffer updates cannot inform subsequent groups. This helps explain RankBuffer’s higher scores: By retaining earlier rollouts and their relative quality judgments in the buffer, training across multiple epochs allows RankBuffer to accumulate and reuse query-specific comparison information. These references help the judge more accurately rank the rollouts within the current group, and the resulting ranking is converted into rewards for GRPO.

2.5 Practical scenario

A practical application is training an AI search assistant to write reports, synthesize information, or produce travel plans. To preserve anonymity during double-blind review, we omit the name of the specific product in which this training approach has been applied. These tasks admit multiple valid long-form responses, so training requires rewards that distinguish their relative quality.

Collecting and curating high-quality real user queries can be costly. Synthetic augmentation can expand the dataset, with additional token costs for generating queries and their judging rubrics. RankBuffer is useful when the available set of curated queries is limited. Even when additional queries are available, it remains a useful training approach, as our experiments show that it can outperform single-pass training on a larger query set.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Key Contributions and Rebuttal Findings

Dear Area Chair and Reviewers,

We thank the reviewers for their thoughtful feedback and recognition of the following strengths:

  • Richer relative reward signals. For open-ended generation, pointwise scores are difficult to calibrate and can cluster within a rollout group. RankBuffer uses rankings to produce differentiated advantages for GRPO.
  • Reuse of historical judgments. Ordered anchors preserve earlier responses and quality judgments, reducing repeated comparisons and judging cost. The buffer provides a reusable yardstick for ranking rollouts within each group and assigning rewards.
  • Consistent benchmark gains. RankBuffer demonstrates improvements over the baselines across four benchmarks.

During the rebuttal, we primarily provided the following additional evidence:

  • Additional baselines. We added pointwise and listwise baselines under the same GRPO training setup for more comprehensive comparisons.
  • Independent evaluation. We used different judge models for reward assignment during training and for benchmark evaluation.
  • Practical efficiency. In addition to the judge token consumption and call counts reported in the original paper, we provided end-to-end training-time measurements.

We sincerely appreciate the Area Chair’s and reviewers’ time and consideration throughout the review process. We believe that academic progress is driven by the exchange of diverse perspectives and constructive discussions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.