- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses the difficulty of maintaining reliable reward scores as the policy evolves during RLHF. It proposes BoNVoyage, which trains scalar reward models by increasing the likelihood of preferred responses under the optimal policy induced by the reward model with KL regularization. The training gradient contrasts preferred demonstrations with responses sampled from the reward-tilted base-model distribution. The authors approximate this sampling process using short independent Metropolis chains initialized at preferred responses. Pre-generated candidate pools are reused during training to reduce computational cost. The method is evaluated through downstream RL and best-of-N selection on mathematics and science tasks using Qwen-based models. The reported results improve over on-policy and off-policy Bradley–Terry baselines across several benchmarks. The mathematics experiments also assess robustness to reward over-optimization during RL.
Critical strengths
-
A novel approach to a well-motivated reward-learning problem. I find the motivation convincing: good pairwise ranking on a fixed response distribution does not ensure reliable rewards as the policy evolves during RLHF. The paper connects RM training to the response distribution induced by downstream optimization. Its likelihood-based approach emphasizes responses favored by the current RM and provides a practical training procedure without an inner RL loop.
-
Encouraging evidence from downstream use. The evaluation includes both RL and best-of-N selection, two important uses of reward models. Compared with the on-policy BT baseline, downstream RL accuracy improves from 63.4% to 67.7% on MATH-500 and from 82.2% to 84.7% on GSM8K. Improvements are also reported across the science benchmarks. The mathematics training curves provide additional evidence that the proposed RM remains useful under continued policy optimization for longer than the BT baselines in the studied setting.
Critical weaknesses
-
The implemented estimator is not sufficiently validated against its stated target. The likelihood-gradient derivation in Eq. (4) is consistent with the stated objective. However, training uses short CD chains and a fixed pool of 32 responses per prompt. Appendix A assesses convergence through benchmark accuracy stabilization. This does not establish distributional convergence or the accuracy of the negative-phase gradient. The diagnostic also uses already-trained RMs, leaving estimator quality during training unclear. The connection to downstream RL needs clarification: RM training uses β = 1, while RL uses a KL coefficient of 10⁻⁴ and standardized rewards. Reward scaling must be considered when relating these settings. Appendix F also reports temperature-0.6 candidate generation in the non-verifiable setting, without clearly relating this proposal distribution to Eqs. (3)–(5). These gaps weaken the proposed distributional explanation of the method.
-
The comparisons do not fully isolate the source or establish the robustness of the gains. Using shared candidate pools is a useful control. However, BT samples preferred/dispreferred pairs randomly, while BoNVoyage uses reward-dependent selection and substantially more scoring computation. A focused comparison with score-informed hard-negative BT would help separate the benefit of the objective from candidate selection. The compute-matched comparison also needs clearer reporting. Section 4.5 and Appendix I describe it differently, and its configuration and numerical results are not clearly identified. Finally, the meaning of the error bars and the number of independent RM and RL training runs are unspecified. This limits assessment of the gains’ attribution and reproducibility.
-
Generality across backbones and sensitivity to β remain insufficiently established. All experiments use Qwen-family models. Section 4.4 varies RM size, but does not test the method on another model family like Llama or evaluate transfer to a different base policy. Appendix E reports exploratory β tuning and notes a tradeoff between BoN@16 and BoN@32. However, it provides no quantitative sensitivity results for downstream RL and leaves systematic analysis to future work. Since β directly controls reward-dependent sampling, the current evidence does not establish how consistently the gains persist across backbones and β settings. This limits confidence in their generality beyond the reported configurations.
Questions for authors
-
What distribution does the implemented sampler target? I would appreciate clarification on how proposals are selected from the fixed candidate pools. Are they sampled with replacement, without replacement, or in another way? How does temperature-0.6 candidate generation in the non-verifiable setting relate to pLM in Eqs. (3) and (5)? Please also explain the relationship between β = 1 during RM training and the downstream KL coefficient, taking reward standardization into account.
-
How accurate is the short-chain gradient estimate? Could you compare the negative-phase estimate with a reference estimate at a few RM checkpoints, including early in training? With uniformly sampled proposals from a finite pool, normalized exp(r/β) weights provide a direct reference distribution within that pool. This comparison would help me assess the effects of CD initialization and chain truncation. I would also like to know how the initial responses for the CD curves in Figure 4 were obtained. Benchmark accuracy stabilization alone does not establish distributional convergence.
-
What explains the gains, and how reproducible are they? Please clarify the compute-matched BT experiment, since §4.5 and Appendix I describe it differently, and report its configuration and results. Could you also test BT with score-informed hard negatives from the same labeled candidate pools? This would help distinguish the benefit of the proposed objective from candidate selection. Please define the ± values in Tables 1–2 and report the number of independent RM and RL training runs. I would like to distinguish training variability from evaluation uncertainty.
-
Do the gains persist across backbones and β settings? Could you add a limited comparison against BT using another model family, such as Llama, on one of the existing tasks? I would also appreciate a small sweep of the RM-training β, with downstream RL performance reported on a representative task. Appendix E mentions exploratory tuning and a BoN@16/BoN@32 tradeoff, but does not quantify the effect on RL. These experiments would help establish whether the gains depend strongly on the chosen backbone or β.
If the authors address these questions satisfactorily, I would consider raising my rating to 3.
2 comments
Continuing on Q3
Could you also test BT with score-informed hard negatives from the same labeled candidate pools?
Thank you for suggesting this baseline. We trained a Qwen3-4b-base BT RM using the same candidate pool as BoNVoyage and selected the highest-reward candidate at each step for each prompt. On math, this variant achieved approximately 60% RM pairwise validation accuracy, below the usual 78%–88% range for BT and BoNVoyage. When used as a reward signal to train a Qwen3-1.7B-Base policy, it reached 58.05% on MATH-500, well below the 63.4% BT baseline.
One possible explanation is that reward-adaptive selection changes the training pairs as the RM evolves and may select good responses as dispreferred, especially early in training when reward scores are unreliable. This can introduce noisy and inconsistent supervision ( which is harmful for BT). In BoNVoyage, by contrast, reward-dependent sampling arises from the objective’s negative phase, and sampled responses are not assumed to be labeled dispreferred.
Question 4.
appreciate a small sweep of the RM-training β, with downstream RL performance reported on a representative task.
Thank you for suggesting these ablations.
We trained pool-size ablations of Qwen3-1.7B-base RMs on the same math data, then used each RM to train a Qwen3-1.7B-base policy for 600 steps and evaluated it on MATH-500.
Pool size: With , BoNVoyage reaches 63.75% with N=8, 64.80% with N=16 and 66.88% with N=32, compared with 62.30% for on-policy BT. The smaller pool remains competitive at a lower cost.
For the KL coefficient experiments. We take the same setting but for N=16 and change the value of .
| 0.1 | 0.2 | 0.5 | 1.0 | 2.0 | |
|---|---|---|---|---|---|
| MATH-500 | 65.35% | 65.88% | 64.90% | 64.80% | 65.15% |
BoNVoyage still outperforms BT across the settings tested. We will add these results and discuss the pool-size accuracy–compute tradeoff in the revision.
Could you add a limited comparison against BT using another model family, such as Llama, on one of the existing tasks
We repeated the Science experiments on Gemma-3-1B-IT. With the same hparams for RM training and RL as in the Qwen experiments, except 400 RL steps.
Legend: Accuracy (%), averaged over eight evaluation seeds.Higher is better.
| Model | ARC-Challenge | ARC-Easy | MMLU science | SciKnowEval | SciQ |
|---|---|---|---|---|---|
| Gemma-3-1B-IT | 46.59% | 59.26% | 34.09% | 42.86% | 49.90% |
| + BonVoyage | 53.24% | 75.59% | 38.82% | 54.53% | 81.90% |
| + BT | 51.28% | 72.31% | 38.27% | 53.57% | 78.80% |
These results extend our conclusions to Gemma model families.
Thank you for outlining what would help resolve your concerns. If our responses address them, we hope you will consider raising your score.
Thank you for taking the time to carefully read our paper and give constructive feedback, especially the suggestion and approach for analysing the gradient estimates. We will focus our rebuttal on directly responding to the questions that you raised.
Question 1
For a chain of T steps, we use a pool of T responses drawn from the base model. We select proposals without replacement, so each response is proposed once.
We used temperature 0.6 because the small models we use generated significantly lower-quality responses at temperature 1.0. If we use the tokenwise power distribution at temperature 0.6 as the base distribution in Eq. (3), it matches the proposal, so Eq. (5) needs no correction. If we instead use the original temperature-1 base model in Eq. (3) while still proposing at temperature 0.6, we must adjust the acceptance ratio to account for the difference between the two distributions (i.e., add the ratio of the product of power distributions).
Before RL, we standardize each RM’s scores using division by max-min from the first RL batch. The scaling factor ranges from roughly 2 to 20 across RMs, so dividing by it increases the relative weight of the KL penalty by the same factor. This motivates using a smaller downstream coefficient than 1. However, in practice, these are hyperparameters that should be tuned separately using validation data at each stage.
Question2
Thank you for suggesting this analysis. We estimated the gradient of the log-likelihood of a preferred response y+:
∇ϕlogπϕ(y+∣x)= β1(∇ϕrϕ(x,y+)- Ey∼πϕ(⋅∣x)[∇ϕrϕ(x,y)]).
We use a Monte Carlo estimate with 256 samples as our reference gradient. We evaluated the reference gradient at RM checkpoints 5, 100, and 195. At each checkpoint, we start the contrastive divergence (CD) chain at the preferred response and measured how closely the CD gradient estimate approached the reference as we increased the chain length from 1 to 32 MCMC steps.
We measured the relative error of the full gradient update as ∥u−v∥2/∥v∥2, where u is the CD estimate and v is the full-pool estimate. We averaged the errors across eight MATH-500 prompts.
Legend: Relative error (%) of the full gradient update at β = 1, averaged across eight MATH-500 prompts. The reference gradient was estimated using 256 examples. Columns give the number of MCMC steps.Lower is better.
At 32 steps, relative error averaged 0.3%, 0.4%, and 0.3% across the eight prompts at checkpoints 5, 100, and 195, respectively. This analysis conclusively shows our method produces effective estimates of the gradient. We expand Appendix A to include this analysis.
For Figure 4, we took only the correct responses sampled by the base model as the preferred demonstration that starts the chain
Question3
Section 4.5 and Appendix I describe the same experimental setup. At each update, BoNVoyage scores the candidate pool and selects a response using test-time alignment with CD. BT uses one labeled pair per prompt, so its updates are cheaper. We let BT revisit prompts and draw fresh pairs from the same pools. This allows BT to use the additional compute for more updates, but validation performance stops improving before its training time matches BoNVoyage’s.
On the science setup, training the BT and BoNVoyage RMs took 62 and 293 minutes, respectively, on four H200 GPUs. For context, training with reinforcement learning took about 12 h (around 3.25x more time) on four H200 GPUs.
We will improve the language of those sections to make it clear for the reader what we intend to communicate.
Tables 1 and 2 report mean accuracy ± standard error across five evaluations of the same final checkpoint using different evaluation seeds. We trained one RM and one RL run for each setting. The error bars therefore quantify evaluation uncertainty. We will add this clarification to the tables.