acceptodds
Under review as a conference paper at ICLR 2027

BoNVoyage: Learning Better Rewards without Ranking

Abstract

Reward models (RMs) play a central role in reinforcement learning from human feedback (RLHF), yet they are typically trained with a Bradley-Terry (BT) objective that ranks response pairs under a fixed text distribution. This objective does not match how RMs are used in RLHF: as training progresses, the policy increasingly concentrates on responses that receive high reward under the RM itself. But what matters is not whether the RM ranks fixed pairs correctly, but whether it remains reliable on the responses that its own signal makes more likely under the policy. We develop BoNVoyage, a method for training RMs directly for this downstream role. Instead of training a pairwise ranker, BoNVoyage maximizes the likelihood that the RM-induced RLHF-optimal policy assigns to preferred responses. Estimating the gradient of this objective requires samples from the optimal policy induced by the learning RM. We obtain these samples through test-time alignment: given the base LM and the RM, we use Markov chain Monte Carlo to sample from the induced policy. To make this practical, we use the base LM as an independent proposal distribution, reuse candidate responses across optimization steps, and use contrastive divergence with short chains initialized at preferred responses. We evaluate BoNVoyage by training 4B RMs for Qwen3-1.7B-Base in mathematics and our supervised finetuned model Qwen3-1.7B-Science in science. Compared with BT baselines, BoNVoyage yields stronger downstream policies under RLHF and stronger rerankers under best-of- sampling across MATH-500, GSM8K, SciQ, SciKnowEval, ARC-Challenge, and MMLU-Science. On verifiable math tasks, BoNVoyage also shows substantially more robustness to reward over-optimization, preserving alignment between RM scores and ground-truth correctness later into RLHF training. On publication, we will release our code, trained models, and datasets.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Reward Modeling, Test-Time Alignment, Markov Chain Monte Carlo, Contrastive Divergence, Reward Over-optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zsBKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the difficulty of maintaining reliable reward scores as the policy evolves during RLHF. It proposes BoNVoyage, which trains scalar reward models by increasing the likelihood of preferred responses under the optimal policy induced by the reward model with KL regularization. The training gradient contrasts preferred demonstrations with responses sampled from the reward-tilted base-model distribution. The authors approximate this sampling process using short independent Metropolis chains initialized at preferred responses. Pre-generated candidate pools are reused during training to reduce computational cost. The method is evaluated through downstream RL and best-of-N selection on mathematics and science tasks using Qwen-based models. The reported results improve over on-policy and off-policy Bradley–Terry baselines across several benchmarks. The mathematics experiments also assess robustness to reward over-optimization during RL.

Critical strengths
  1. A novel approach to a well-motivated reward-learning problem. I find the motivation convincing: good pairwise ranking on a fixed response distribution does not ensure reliable rewards as the policy evolves during RLHF. The paper connects RM training to the response distribution induced by downstream optimization. Its likelihood-based approach emphasizes responses favored by the current RM and provides a practical training procedure without an inner RL loop.

  2. Encouraging evidence from downstream use. The evaluation includes both RL and best-of-N selection, two important uses of reward models. Compared with the on-policy BT baseline, downstream RL accuracy improves from 63.4% to 67.7% on MATH-500 and from 82.2% to 84.7% on GSM8K. Improvements are also reported across the science benchmarks. The mathematics training curves provide additional evidence that the proposed RM remains useful under continued policy optimization for longer than the BT baselines in the studied setting.

Critical weaknesses
  1. The implemented estimator is not sufficiently validated against its stated target. The likelihood-gradient derivation in Eq. (4) is consistent with the stated objective. However, training uses short CD chains and a fixed pool of 32 responses per prompt. Appendix A assesses convergence through benchmark accuracy stabilization. This does not establish distributional convergence or the accuracy of the negative-phase gradient. The diagnostic also uses already-trained RMs, leaving estimator quality during training unclear. The connection to downstream RL needs clarification: RM training uses β = 1, while RL uses a KL coefficient of 10⁻⁴ and standardized rewards. Reward scaling must be considered when relating these settings. Appendix F also reports temperature-0.6 candidate generation in the non-verifiable setting, without clearly relating this proposal distribution to Eqs. (3)–(5). These gaps weaken the proposed distributional explanation of the method.

  2. The comparisons do not fully isolate the source or establish the robustness of the gains. Using shared candidate pools is a useful control. However, BT samples preferred/dispreferred pairs randomly, while BoNVoyage uses reward-dependent selection and substantially more scoring computation. A focused comparison with score-informed hard-negative BT would help separate the benefit of the objective from candidate selection. The compute-matched comparison also needs clearer reporting. Section 4.5 and Appendix I describe it differently, and its configuration and numerical results are not clearly identified. Finally, the meaning of the error bars and the number of independent RM and RL training runs are unspecified. This limits assessment of the gains’ attribution and reproducibility.

  3. Generality across backbones and sensitivity to β remain insufficiently established. All experiments use Qwen-family models. Section 4.4 varies RM size, but does not test the method on another model family like Llama or evaluate transfer to a different base policy. Appendix E reports exploratory β tuning and notes a tradeoff between BoN@16 and BoN@32. However, it provides no quantitative sensitivity results for downstream RL and leaves systematic analysis to future work. Since β directly controls reward-dependent sampling, the current evidence does not establish how consistently the gains persist across backbones and β settings. This limits confidence in their generality beyond the reported configurations.

Questions for authors
  1. What distribution does the implemented sampler target? I would appreciate clarification on how proposals are selected from the fixed candidate pools. Are they sampled with replacement, without replacement, or in another way? How does temperature-0.6 candidate generation in the non-verifiable setting relate to pLM in Eqs. (3) and (5)? Please also explain the relationship between β = 1 during RM training and the downstream KL coefficient, taking reward standardization into account.

  2. How accurate is the short-chain gradient estimate? Could you compare the negative-phase estimate with a reference estimate at a few RM checkpoints, including early in training? With uniformly sampled proposals from a finite pool, normalized exp(r/β) weights provide a direct reference distribution within that pool. This comparison would help me assess the effects of CD initialization and chain truncation. I would also like to know how the initial responses for the CD curves in Figure 4 were obtained. Benchmark accuracy stabilization alone does not establish distributional convergence.

  3. What explains the gains, and how reproducible are they? Please clarify the compute-matched BT experiment, since §4.5 and Appendix I describe it differently, and report its configuration and results. Could you also test BT with score-informed hard negatives from the same labeled candidate pools? This would help distinguish the benefit of the proposed objective from candidate selection. Please define the ± values in Tables 1–2 and report the number of independent RM and RL training runs. I would like to distinguish training variability from evaluation uncertainty.

  4. Do the gains persist across backbones and β settings? Could you add a limited comparison against BT using another model family, such as Llama, on one of the existing tasks? I would also appreciate a small sweep of the RM-training β, with downstream RL performance reported on a representative task. Appendix E mentions exploratory tuning and a BoN@16/BoN@32 tradeoff, but does not quantify the effect on RL. These experiments would help establish whether the gains depend strongly on the chosen backbone or β.

If the authors address these questions satisfactorily, I would consider raising my rating to 3.

2 comments

Thank you for taking the time to carefully read our paper and give constructive feedback, especially the suggestion and approach for analysing the gradient estimates. We will focus our rebuttal on directly responding to the questions that you raised.

Question 1

How are proposals selected from the pool?

For a chain of T steps, we use a pool of T responses drawn from the base model. We select proposals without replacement, so each response is proposed once.

How does temperature-0.6 generation relate to pLM in Eqs. (3) and (5)?

We used temperature 0.6 because the small models we use generated significantly lower-quality responses at temperature 1.0. If we use the tokenwise power distribution at temperature 0.6 as the base distribution in Eq. (3), it matches the proposal, so Eq. (5) needs no correction. If we instead use the original temperature-1 base model in Eq. (3) while still proposing at temperature 0.6, we must adjust the acceptance ratio to account for the difference between the two distributions (i.e., add the ratio of the product of power distributions).

Please also explain the relationship between β = 1 during RM training and the downstream KL coefficient, taking reward standardization into account.

Before RL, we standardize each RM’s scores using division by max-min from the first RL batch. The scaling factor ranges from roughly 2 to 20 across RMs, so dividing by it increases the relative weight of the KL penalty by the same factor. This motivates using a smaller downstream coefficient than 1. However, in practice, these are hyperparameters that should be tuned separately using validation data at each stage.

Question2

Could you compare the negative-phase estimate with a reference estimate at a few RM checkpoints, including early in training

Thank you for suggesting this analysis. We estimated the gradient of the log-likelihood of a preferred response y+y_+:

∇ϕlog⁡πϕ(y+∣x)\nabla_\phi \log \pi_\phi(y_+ \mid x)= 1β(∇ϕrϕ(x,y+)\frac{1}{\beta}(\nabla_\phi r_\phi(x,y_+)- Ey∼πϕ(⋅∣x)[∇ϕrϕ(x,y)]).E_{y \sim \pi_\phi(\cdot \mid x)}[\nabla_\phi r_\phi(x,y)]).

We use a Monte Carlo estimate with 256 samples as our reference gradient. We evaluated the reference gradient at RM checkpoints 5, 100, and 195. At each checkpoint, we start the contrastive divergence (CD) chain at the preferred response and measured how closely the CD gradient estimate approached the reference as we increased the chain length from 1 to 32 MCMC steps.

We measured the relative error of the full gradient update as ∥u−v∥2/∥v∥2\lVert u-v\rVert_2 / \lVert v\rVert_2, where uu is the CD estimate and vv is the full-pool estimate. We averaged the errors across eight MATH-500 prompts.

Legend: Relative error (%) of the full gradient update at β = 1, averaged across eight MATH-500 prompts. The reference gradient was estimated using 256 examples. Columns give the number of MCMC steps.Lower is better.

RM checkpoint12481632
546.426.39.42.10.70.3
10052.830.712.93.40.60.4
19557.136.117.45.51.10.3

At 32 steps, relative error averaged 0.3%, 0.4%, and 0.3% across the eight prompts at checkpoints 5, 100, and 195, respectively. This analysis conclusively shows our method produces effective estimates of the gradient. We expand Appendix A to include this analysis.

I would also like to know how the initial responses for the CD curves in Figure 4 were obtained

For Figure 4, we took only the correct responses sampled by the base model as the preferred demonstration that starts the chain

Question3

Please clarify the compute-matched BT experiment, since §4.5 and Appendix I describe it differently

Section 4.5 and Appendix I describe the same experimental setup. At each update, BoNVoyage scores the candidate pool and selects a response using test-time alignment with CD. BT uses one labeled pair per prompt, so its updates are cheaper. We let BT revisit prompts and draw fresh pairs from the same pools. This allows BT to use the additional compute for more updates, but validation performance stops improving before its training time matches BoNVoyage’s.

On the science setup, training the BT and BoNVoyage RMs took 62 and 293 minutes, respectively, on four H200 GPUs. For context, training with reinforcement learning took about 12 h (around 3.25x more time) on four H200 GPUs.

We will improve the language of those sections to make it clear for the reader what we intend to communicate.

Please define the ± values in Tables 1–2 and report the number of independent RM and RL training runs

Tables 1 and 2 report mean accuracy ± standard error across five evaluations of the same final checkpoint using different evaluation seeds. We trained one RM and one RL run for each setting. The error bars therefore quantify evaluation uncertainty. We will add this clarification to the tables.

Continuing on Q3

Could you also test BT with score-informed hard negatives from the same labeled candidate pools?

Thank you for suggesting this baseline. We trained a Qwen3-4b-base BT RM using the same candidate pool as BoNVoyage and selected the highest-reward candidate at each step for each prompt. On math, this variant achieved approximately 60% RM pairwise validation accuracy, below the usual 78%–88% range for BT and BoNVoyage. When used as a reward signal to train a Qwen3-1.7B-Base policy, it reached 58.05% on MATH-500, well below the 63.4% BT baseline.

One possible explanation is that reward-adaptive selection changes the training pairs as the RM evolves and may select good responses as dispreferred, especially early in training when reward scores are unreliable. This can introduce noisy and inconsistent supervision ( which is harmful for BT). In BoNVoyage, by contrast, reward-dependent sampling arises from the objective’s negative phase, and sampled responses are not assumed to be labeled dispreferred.

Question 4.

appreciate a small sweep of the RM-training β, with downstream RL performance reported on a representative task.

Thank you for suggesting these ablations.

We trained pool-size ablations of Qwen3-1.7B-base RMs on the same math data, then used each RM to train a Qwen3-1.7B-base policy for 600 steps and evaluated it on MATH-500.

Pool size: With β=1.0\beta=1.0, BoNVoyage reaches 63.75% with N=8, 64.80% with N=16 and 66.88% with N=32, compared with 62.30% for on-policy BT. The smaller pool remains competitive at a lower cost.

For the KL coefficient experiments. We take the same setting but for N=16 and change the value of β\beta.

β\beta0.10.20.51.02.0
MATH-50065.35%65.88%64.90%64.80%65.15%

BoNVoyage still outperforms BT across the settings tested. We will add these results and discuss the pool-size accuracy–compute tradeoff in the revision.

Could you add a limited comparison against BT using another model family, such as Llama, on one of the existing tasks

We repeated the Science experiments on Gemma-3-1B-IT. With the same hparams for RM training and RL as in the Qwen experiments, except 400 RL steps.

Legend: Accuracy (%), averaged over eight evaluation seeds.Higher is better.

ModelARC-ChallengeARC-EasyMMLU scienceSciKnowEvalSciQ
Gemma-3-1B-IT46.59%59.26%34.09%42.86%49.90%
+ BonVoyage53.24%75.59%38.82%54.53%81.90%
+ BT51.28%72.31%38.27%53.57%78.80%

These results extend our conclusions to Gemma model families.


Thank you for outlining what would help resolve your concerns. If our responses address them, we hope you will consider raising your score.

Reviewer_TMwzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Instead of the classical BT reward modeling objective, this paper proposes BoNVoyage and maximizes the likelihood of preferred responses under the policy induced by the reward model, using contrastive divergence and MCMC sampling to approximate the training objective. Experiments on mathematical reasoning and science tasks show improvements in downstream policy performance and suggest reduced reward overoptimization compared with conventional reward modeling.

Critical strengths
  1. Strong pairwise preference discrimination does not necessarily translate into effective downstream policy optimization, especially when the policy increasingly exploits imperfections in the reward model. The paper provides an interesting perspective by explicitly considering the distribution induced by reward optimization.

  2. The proposed objective is reasonably well motivated. Deriving reward-model training from the likelihood of preferred responses under a KL-constrained optimal policy provides a direct connection between reward learning and policy optimization. The use of contrastive divergence also offers a practical approximation without requiring an expensive inner RL training loop.

Critical weaknesses

My main concern is that the contribution of the proposed objective is not sufficiently isolated from the negative-sampling strategy. Compared with standard BT training, BoNVoyage changes both the learning objective and how negative responses are selected. It would be helpful to compare against stronger baselines, such as BT training with the same reward-adaptive hard negatives. In addition, the paper lacks sensitivity analyses for two important hyperparameters, the KL coefficient β\beta and candidate pool size NN, which directly influence the target distribution and the quality of the MCMC approximation. Without these experiments, it is difficult to understand how robust the proposed method is and which components are responsible for the improvements. It would also be interesting to examine whether the proposed reward model maintains competitive pairwise preference discrimination on standard RM benchmarks, which could help clarify the relationship between conventional RM accuracy and downstream policy utility.

Questions for authors

Please refer to the weaknesses.

1 comment

Thank you for taking the time to carefully read our paper and give constructive feedback. Below, we discuss the weaknesses you raised:

Weakness 1.

My main concern is that the contribution of the proposed objective is not sufficiently isolated from the negative-sampling strategy.

Thank you for suggesting this baseline. We trained a Qwen3-4b-base BT RM using the same candidate pool as BoNVoyage and selected the highest-reward candidate at each step for each prompt. On math, this variant achieved approximately 60% RM pairwise validation accuracy, below the usual 78%–88% range for BT and BoNVoyage. When used as a reward signal to train a Qwen3-1.7B-Base policy, it reached 58.05% on MATH-500, well below the 63.4% BT baseline.

One possible explanation is that reward-adaptive selection changes the training pairs as the RM evolves and may select good responses as dispreferred, especially early in training when reward scores are unreliable. This can introduce noisy and inconsistent supervision (which is harmful for BT). In BoNVoyage, by contrast, reward-dependent sampling arises from the objective’s negative phase, and sampled responses are not assumed to be labeled dispreferred.

Weakness 2.

.. the paper lacks sensitivity analyses for two important hyperparameters, the KL coefficient (\beta) and candidate pool size (N) ..

Thank you for suggesting these ablations.

We trained pool-size ablations of Qwen3-1.7B-base RMs on the same math data, then used each RM to train a Qwen3-1.7B-base policy for 600 steps and evaluated it on MATH-500.

Pool size: With β=1.0\beta=1.0, BoNVoyage reaches 63.75% with N=8, 64.80% with N=16 and 66.88% with N=32, compared with 62.30% for on-policy BT. The smaller pool remains competitive at a lower cost.

For the KL coefficient experiments. We take the same setting but for N=16 and change the value of β\beta.

β\beta0.10.20.51.02.0
MATH-50065.35%65.88%64.90%64.80%65.15%

BoNVoyage still outperforms BT across the settings tested. We will add these results and discuss the pool-size accuracy–compute tradeoff in the revision.

Weakness 3.

It would also be interesting to examine whether the proposed reward model maintains competitive pairwise preference discrimination on standard RM benchmarks

Thank you for suggesting this evaluation. We agree it can strengthen our argument. On the RewardBench 2 Math and Science subsets, BoNVoyage and on-policy BT perform similarly, while the off-policy Skywork 4B achieves higher benchmark scores.

Table legend. RB2 score is the RewardBench 2 score for the subset. Pairwise accuracy is the percentage of response pairs the RM ranks correctly. Higher is better for both metrics.

RewardBench 2 ( Math)

Reward modelRB2 scorePairwise accuracy
Skywork 4B73.22%85.79%
Math BoNVoyage 4B65.57%79.96%
Math BT 4B65.57%80.33%
Skywork 1B60.11%77.60%

RewardBench 2 ( Science )

Reward modelRB2 scorePairwise accuracy
Skywork 4B69.84%84.13%
Science BoNVoyage 4B57.14%78.31%
Science BT 4B55.56%76.72%
Skywork 1B42.86%69.84%

These RewardBench2 results, contrasted with downstream RL performance (off-policy BT is worse than on-policy BT and BonVoyage), support our view that static pairwise accuracy alone does not fully capture an RM’s utility for RL. We will add these results to the paper.


Thank you for suggesting these comparisons. If the hard-negative baseline, sensitivity analyses, and RewardBench 2 results address your concerns, we would appreciate your consideration of a higher score.

Reviewer_sMgzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BoNVoyage proposes a new reward modeling method, which optimizes reward models for downstream usage, rather than simply for ranking response pairs. Specifically, it maximizes the likelihood of preferred responses under the RLHF-optimal policy induced by the reward model, and approximates the required gradient using MCMC sampling and contrastive divergence. Experiments show that BoNVoyage consistently outperforms BT baselines in downstream RL and best-of-n reranking.

Critical strengths
  1. The paper identifies the mismatch between pairwise ranking objectives and the actual role of reward models during RL, where policy optimization can induce distribution shift and reward hacking.
  2. The idea and design of this paper is novel and interesting. It explicitly trains the reward model against the responses that RL would exploit, before RL happens. This method can potentially be more robust to reward over-optimization and less likely to have reward hacking.
Critical weaknesses
  1. The method pre-generates candidate responses from the base LM and reuses them during RM training. This improves efficiency, but it may limit the diversity of negative samples and the method's ability to capture responses that emerge as the policy evolves during RL. Actual RL can go far beyond these pre-generated responses.
  2. The method relies on contrastive divergence with short MCMC chains to approximate the gradient of the proposed objective. However, the quality of this approximation is not sufficiently investigated.
  3. RL runs on the same prompts the RM was trained on. This makes the results less impressive as the RM has already seen the downstream tasks. Generalization of the RM to unseen prompts is not tested.
  4. BoNVoyage requires substantially more computation cost than standard Bradley-Terry training, with approximately 4.7× higher wall-clock training time in the reported science experiment. The performance-efficiency trade-off could be investigated more thoroughly.
  5. The experiments focus on two task domains (mathematics and science), and a single model family (Qwen) with small scale. It remains unclear whether the method generalizes to larger models, different model architectures, or broader open-ended tasks where reward hacking is harder to quantify.
Questions for authors
  1. Can the authors compare BoNVoyage with highest reward selection (hard-negative) instead of soft version of BoN sampling to show whether the framing matters more or the choice of negatives?
  2. Can the authors provide results of RL training on different prompts from RM training?
1 comment

Thank you for the careful review. We address your points below:

Question 1

Can the authors compare BoNVoyage with highest reward selection (hard-negative) instead of soft version of BoN sampling to show whether the framing matters more or the choice of negatives?

Thank you for suggesting this baseline. We trained a Qwen3-4b-base BT RM using the same candidate pool as BoNVoyage and selected the highest-reward candidate at each step for each prompt. On math, this variant achieved approximately 60% RM pairwise validation accuracy, below the usual 78%–88% range for BT and BoNVoyage. When used as a reward signal to train a Qwen3-1.7B-Base policy, it reached 58.05% on MATH-500, well below the 63.4% BT baseline. One possible explanation is that reward-adaptive selection changes the training pairs as the RM evolves and may select good responses as dispreferred, especially early in training, when reward scores are unreliable. This can introduce noisy and inconsistent supervision (which is harmful for BT). In BoNVoyage, by contrast, reward-dependent sampling arises from the objective’s negative phase, and sampled responses are not assumed to be labeled dispreferred.

Question 2

Can the authors provide results of RL training on different prompts from RM training?

Thank you for giving us the opportunity to report these results. We trained Qwen3-1.7B-Base for 800 RL steps on DeepMath, a new distribution of math prompts, using 4B reward models trained with BT or BoNVoyage (β=1\beta=1 for BoNVoyage). At the end of training, MATH-500 accuracy was 62.50% for BT and 65.575% for BoNVoyage. When we selected checkpoints based on validation accuracy, MATH-500 accuracy was 64.25% for BT and 66.18% for BoNVoyage. All results are averages over eight evaluation seeds. The results of our paper stand under this setting.

Weakness 4.

BoNVoyage requires substantially more computation cost than standard Bradley-Terry training, with approximately 4.7× higher wall-clock training time in the reported science experiment.

We have two comments on this point.

(1) At each update, BoNVoyage scores the candidate pool and selects a response using test-time alignment with CD. BT uses one labeled pair per prompt, so its updates are cheaper. We let BT revisit prompts and draw fresh pairs from the same pools. This allows BT to use the additional compute for more updates, but validation performance stops improving before its training time matches BoNVoyage’s. On the science setup, training the BT and BoNVoyage RMs took 62 and 293 minutes, respectively, on four H200 GPUs. For context, training with reinforcement learning took about 12 h (around 3.25x more time) on four H200 GPUs.

(2) We want to show results using a smaller pool size N, which greatly decreases the wall-clock time. We trained Qwen3-1.7B RMs on the same math data, then used each RM to train a Qwen3-1.7B-Base policy for 600 steps and evaluated it on MATH-500.

With =1.0, BoNVoyage reaches 63.75% with N=8 and 66.88% with N=32, compared with 62.30% for on-policy BT. The smaller pool remains competitive at a lower cost.

BoNVoyage still outperforms BT across the settings tested. We will add these results and discuss the pool-size accuracy–compute tradeoff in the revision.

Weakness 2.

.. the quality of this approximation is not sufficiently investigated.

In Appendix A, we study how quickly benchmark accuracy approaches its final value as we increase the number of MCMC steps in our test-time alignment procedure with contrastive divergence. During the rebuttal period, we complemented this analysis with an additional comparison suggested by reviewer zsBK. We describe the results in our response to zsBK and invite the reviewer to consult it for details.

--

We hope these clarifications and additional comparisons address your concerns. If so, we would appreciate your consideration of a higher score.

Comments on the paper

1 comment

Thank you for taking the time to review our work and give substantial feedback. All three reviewers (sMgz, TMwz, and zsBK) recognize the value of our motivation and framing and the novelty of our approach. Furthermore, TMwz and zsBK highlighted the practicality of BonVoyage, zsBK emphasized the downstream evaluation results , and sMgz and zsBK recognized its potential for greater robustness to reward over-optimization, which is an important concern for the ICLR community and machine learning research community more broadly.

The reviewers raised concerns about the short-chain gradient approximation, the contributions of BoNVoyage’s individual components, generalization beyond Qwen, hyperparameter sensitivity, and the tradeoff between compute cost and performance. We addressed these concerns through additional empirical results and analyses. Which we encourage the reviewers to take a look.

We believe that with the experiments and analysis we added, we have considerably strengthened our empirical findings. For this reason, we encourage you to take a look at our rebutall and, if it addresses your concerns sufficiently, to consider raising your scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.