acceptodds
Under review as a conference paper at ICLR 2027

Hard Preference Calibration for On-Policy Distillation

Abstract

On-policy distillation (OPD) has emerged as an effective paradigm for post-training large language models, yet teacher preferences can conflict with verifier outcomes. In practice, a teacher may prefer an incorrect trajectory over a correct one, making token-level guidance unreliable on critical examples. We thus propose Hard Preference Calibration (HPC), which augments OPD with an adaptive contrastive objective that calibrates verifier-derived hard preferences using the teacher's pairwise reliability. For each correct-incorrect trajectory pair, HPC applies stronger correction as the teacher's preference becomes less trustworthy, while preserving OPD's dense supervision. Our theoretical analysis further shows that this calibration yields a strictly tighter generation-error bound than OPD and improves Pass@k. Across multiple mathematical reasoning and code generation benchmarks, it outperforms OPD and related variants across student scales and teacher regimes, with absolute Pass@16 gains of up to 5.3% on HMMT25 and gains particularly evident when teacher preferences are less well aligned with verifier outcomes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
machine learning, on policy distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ydtgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission proposes Hard Preference Calibration (HPC), which augments OPD with verifier-guided preference optimization. HPC constructs trajectory pairs using verifier outcome and introduces a weighted contrastive loss, so student prefer correct trajectories. Experiments on math and coding datasets demonstrate improvements over OPD and SCOPE across student sizes and teacher configurations.

Critical strengths
  1. A relatively simple approach to address a meaningful limitation of OPD. Using verifier-based preferences is an intuitive mechanism,

  2. Promising empirical improvements across several configurations (math and coding, two student sizes, two teacher settings). The ablation experiments also suggest that the reliability gate provides additional benefits.

Critical weaknesses
  1. Limited methodological novelty and insufficient comparison with alternatives. The method can essentially be understood as combining OPD with an additional verifier-guided pairwise preference objective, where the teacher's relative sequence likelihood determines the weight assigned to each pair. While this is reasonable, the paper does not sufficiently establish why the particular calibration mechanism represents a substantial contribution over combinations of existing objectives. E.g., OPD augmented with a standard pairwise preference loss, and other verifier-guided OPD approaches such as reward-gated OPD.

  2. Insufficient empirical validation. The experiments do not include rejection sampling followed by supervised fine-tuning, stronger verifier-guided OPD variants, or more direct combinations of OPD and standard preference optimization. For metrics, some individual results are comparable to or worse than simpler baselines. Without results across multiple random seeds or uncertainty estimates, it is difficult to establish the reliability of the smaller gains. No executable code or supplementary implementation is provided.

Questions for authors

See comments above around comparison with standard preference optimization and other verifier-guided OPD methods, reliability of empirical results

1 comment
AuthorsOpenReview ↗

Response to Reviewer ydtg

Limited methodological novelty and insufficient comparison with alternatives. The method can essentially be understood as combining OPD with an additional verifier-guided pairwise preference objective, where the teacher's relative sequence likelihood determines the weight assigned to each pair. While this is reasonable, the paper does not sufficiently establish why the particular calibration mechanism represents a substantial contribution over combinations of existing objectives. E.g., OPD augmented with a standard pairwise preference loss, and other verifier-guided OPD approaches such as reward-gated OPD.

Thank you for this helpful comment. In our method, verifier correctness determines which response should be preferred, while the teacher’s relative sequence scores determine how strongly each pair is corrected. Unlike uniform pairwise correction, HPC focuses on pairs where the teacher’s preference is weak or incorrect. In our ablation, uniform pairwise correction improves the average from 38.36% for OPD to 39.41%, while HPC reaches 40.54%, roughly doubling the improvement over OPD.

To compare HPC directly with a standard preference objective that weights all verifier-labeled response pairs equally, we have added a theoretical result in Appendix B.3. At the OPD reference point, let δi,p,n\delta_{i,p,n} be the teacher margin deficit for a correct and incorrect response pair, and wi,p,nw_{i,p,n} its HPC reliability weight. Both increase as the teacher’s relative score for the correct response decreases. Their covariance over pairs is therefore nonnegative, which gives

δˉical=∑(p,n)wi,p,nδi,p,n∑(p,n)wi,p,n≥1∣Ei∣∑(p,n)δi,p,n=δˉiunif.\bar{\delta} _ i^{\mathrm{cal}} = \frac{\sum _ {(p,n)}w _ {i,p,n}\delta _ {i,p,n}} {\sum _ {(p,n)}w _ {i,p,n}} \geq \frac{1}{|\mathcal{E} _ i|} \sum _ {(p,n)}\delta _ {i,p,n} = \bar{\delta} _ i^{\mathrm{unif}}.

Each deficit measures how much correction is needed for that pair at the OPD reference point. Therefore, under the same normalized correction budget, HPC assigns more correction to pairs with larger teacher deficits than uniform pairwise preference optimization. These reliability-weighted pairs contribute to the correction certificate in our existing analysis, which gives a lower generation error bound than OPD and implies an improvement in Pass@kk.

Thank you again for this insightful comment. We have incorporated the discussion and theoretical analysis into Appendix B.3.

Insufficient empirical validation. The experiments do not include rejection sampling followed by supervised fine-tuning, stronger verifier-guided OPD variants, or more direct combinations of OPD and standard preference optimization. For metrics, some individual results are comparable to or worse than simpler baselines. Without results across multiple random seeds or uncertainty estimates, it is difficult to establish the reliability of the smaller gains. No executable code or supplementary implementation is provided.

Thank you for this detailed feedback. We have expanded the comparison to include an outcome-guided margin calibration method like Uni-OPD and trajectory-level reward gating like OPDVR. The table reports results on four mathematics benchmarks. Scores are percentages (%).

BenchmarkMetricOPDOPDVRUni-OPDSCOPEHPC
AIME24Pass@1664.661.363.661.967.7
AIME24Pass@859.256.359.157.260.7
AIME24Pass@453.250.953.151.353.1
AIME25Pass@1649.957.150.752.552.8
AIME25Pass@845.351.046.547.747.5
AIME25Pass@440.644.441.342.441.7
HMMT25 Feb.Pass@1634.029.734.734.636.2
HMMT25 Feb.Pass@829.427.128.929.529.7
HMMT25 Feb.Pass@424.323.423.824.724.6
HMMT25 Nov.Pass@1634.235.938.338.440.6
HMMT25 Nov.Pass@830.131.033.033.536.4
HMMT25 Nov.Pass@424.825.526.727.530.6
Overall average40.841.141.641.843.5

As shown in the table, HPC achieves the strongest average performance among the compared methods. The main experiments show similarly consistent average gains across student and teacher settings. For individual Pass@k results, HPC exceeds OPD in 93.8% of the mathematics results and 95.8% of the code results, and outperforms the other methods on most benchmark-level results. Post-training requires substantial computational resources, and we have included means and standard deviations across multiple runs for some experiments in Appendix C.2. We also provide detailed configurations in Appendix C and core implementation in the supplementary material. Thank you again for this valuable feedback and for encouraging a broader evaluation.

Reviewer_X6pxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hard Preference Calibration (HPC), a method that augments on-policy distillation (OPD) with verifier-guided pairwise preference optimization. The main motivation is that teacher preferences may conflict with verifier outcomes, potentially reinforcing incorrect trajectories during distillation. HPC introduces an adaptive contrastive objective that emphasizes correct–incorrect trajectory pairs when teacher preferences are unreliable. The paper also provides theoretical analysis and reports improvements over OPD and related baselines on mathematical reasoning and code generation benchmarks.

Critical strengths

Addressing the inconsistency between teacher supervision and response correctness is an important and practically relevant research problem.

HPC offers a simple and intuitive approach to combining dense token-level supervision with verifier-guided preference correction.

Experiments across multiple student sizes, teacher configurations, and reasoning benchmarks demonstrate consistent improvements over OPD.

Ablation studies and theoretical analysis provide additional insights into the proposed calibration mechanism.

Critical weaknesses

HPC distinguishes correct and incorrect trajectories solely through outcome-based verification, without considering the correctness of intermediate reasoning steps. A trajectory may produce the correct final answer despite flawed reasoning, potentially introducing misleading preference supervision. Although the paper considers random verifier noise, it does not sufficiently investigate systematic verification errors or the mismatch between outcome correctness and reasoning quality.

Since HPC already relies on multiple on-policy rollouts and verifier rewards, its computational requirements overlap with those of RLVR methods such as GRPO. Although GRPO is included in some experiments, the comparisons do not sufficiently establish performance under matched computational budgets. More comprehensive comparisons are needed to demonstrate the practical advantages of HPC over directly optimizing verifier rewards.

HPC essentially combines OPD with an additional verifier-guided pairwise contrastive loss. The ablation results suggest that a substantial portion of the improvement comes from the contrastive objective itself, while the additional contribution of the proposed reliability calibration is less clear.

The theoretical analysis relies on stationary solutions over the complete trajectory distribution, whereas practical LLM training involves finite sampling and non-convex optimization. The connection between these theoretical results and the observed empirical improvements remains insufficiently established.

Questions for authors

1.How does HPC handle trajectories with correct final answers but incorrect intermediate reasoning? Could such cases introduce misleading preference supervision?

2.Can the authors provide compute-matched comparisons with GRPO or other verifier-based RL methods, accounting for rollout generation, verifier calls, and total training cost?

3.How much of HPC's improvement comes specifically from the adaptive reliability calibration rather than simply adding a verifier-guided contrastive objective to OPD?

2 comments
AuthorsOpenReview ↗

Response to Reviewer X6px (1/2)

HPC distinguishes correct and incorrect trajectories solely through outcome-based verification, without considering the correctness of intermediate reasoning steps. A trajectory may produce the correct final answer despite flawed reasoning, potentially introducing misleading preference supervision. Although the paper considers random verifier noise, it does not sufficiently investigate systematic verification errors or the mismatch between outcome correctness and reasoning quality. How does HPC handle trajectories with correct final answers but incorrect intermediate reasoning? Could such cases introduce misleading preference supervision?

Thank you for this constructive comment. HPC preserves OPD’s guidance throughout generation and uses feedback on final results to calibrate trajectory preferences. The teacher’s conditional probabilities continue to provide supervision at each token, including intermediate reasoning. Although this cannot rule out flawed reasoning that happens to reach a correct answer, it preserves a complementary source of guidance beyond the correctness of the final answer.

Distinguishing genuine errors from legitimate variations in reasoning paths, levels of detail, and solution strategies can be difficult, making reliable verification of intermediate reasoning a common challenge. Misjudging these variations can introduce noise into training, so additional process supervision requires careful assessment of its reliability. HPC remains open to methods that assess reasoning quality and could complement its teacher guidance and outcome feedback. We view this as a potential direction for future work, and we sincerely appreciate your suggestion.

Since HPC already relies on multiple on-policy rollouts and verifier rewards, its computational requirements overlap with those of RLVR methods such as GRPO. Although GRPO is included in some experiments, the comparisons do not sufficiently establish performance under matched computational budgets. More comprehensive comparisons are needed to demonstrate the practical advantages of HPC over directly optimizing verifier rewards. Can the authors provide compute-matched comparisons with GRPO or other verifier-based RL methods, accounting for rollout generation, verifier calls, and total training cost?

Thank you for this constructive suggestion. Both methods use 128 prompts per step and eight rollouts per prompt, resulting in the same numbers of rollouts and verifier calls for a given number of training steps. We have added a comparison under training budgets of 30 and 160 GPU hours using a Qwen3-4B student, with Qwen3-30B-A3B as the teacher for HPC.

DatasetGRPO Pass@4HPC Pass@4GRPO Pass@8HPC Pass@8
AIME2457.0272.2462.0177.10
AIME2552.8363.8155.4669.53
HMMT25 Feb.29.6743.2134.4248.66
HMMT25 Nov.38.7151.3143.7557.50
Average44.5657.6448.9163.20

Table 1. Performance under a training budget of 30 GPU hours. Scores are percentages (%).

DatasetGRPO Pass@4HPC Pass@4GRPO Pass@8HPC Pass@8
AIME2458.6476.6963.7880.73
AIME2555.7865.6361.6072.50
HMMT25 Feb.39.5845.0845.3752.23
HMMT25 Nov.43.3353.2047.9058.86
Average49.3360.1554.6666.08

Table 2. Performance under a training budget of 160 GPU hours. Scores are percentages (%).

As shown in the tables, HPC outperforms GRPO across all benchmarks at both budgets. At the larger budget of 160 GPU hours, its average gains over GRPO reach 10.82% in Pass@4 and 11.42% in Pass@8. These results support HPC’s computational efficiency. Thank you again for this helpful suggestion. We have included these results in Appendix C.1.

AuthorsOpenReview ↗

Response to Reviewer X6px (2/2)

HPC essentially combines OPD with an additional verifier-guided pairwise contrastive loss. The ablation results suggest that a substantial portion of the improvement comes from the contrastive objective itself, while the additional contribution of the proposed reliability calibration is less clear. How much of HPC's improvement comes specifically from the adaptive reliability calibration rather than simply adding a verifier-guided contrastive objective to OPD?

Thank you for this constructive comment. In our method, the contrastive objective implements correction, while adaptive reliability calibration selects and weights response pairs based on teacher reliability. Our existing ablation shows that adding the contrastive objective alone improves OPD by 1.05%, while reliability calibration contributes a further 1.13%, accounting for approximately 52% of the total gain in this comparison.

To further address your concern, we have included an ablation that replaces our contrastive correction with a direct margin adjustment to OPD advantages. For each prompt, if the mean advantage of correct responses does not exceed that of incorrect responses by a prescribed margin, we increase the former and decrease the latter equally until the margin is reached. The results below use the Qwen3-1.7B student and Qwen3-30B-A3B-Instruct-2507 teacher. OPD+MC denotes OPD with margin calibration. All values are percentages; the overall average covers the twelve benchmark metrics.

BenchmarkMetricOPDOPD+MCHPC
AIME24Pass@1664.663.067.7
AIME24Pass@859.258.460.7
AIME24Pass@453.253.153.1
AIME25Pass@1649.951.252.8
AIME25Pass@845.346.847.5
AIME25Pass@440.641.541.7
HMMT25 Feb.Pass@1634.032.536.2
HMMT25 Feb.Pass@829.428.129.7
HMMT25 Feb.Pass@424.323.624.6
HMMT25 Nov.Pass@1634.238.040.6
HMMT25 Nov.Pass@830.133.836.4
HMMT25 Nov.Pass@424.828.130.6
Overall average40.841.543.5

As shown in the table, margin calibration improves the overall average from 40.8% to 41.5%, while HPC reaches 43.5%. Together with the existing gate ablation, these results support the additional contribution of adaptive reliability calibration and suggest that directly adjusting OPD advantages does not fully recover the benefit of correction guided by teacher reliability at the level of individual response pairs. Thank you again for this helpful suggestion; we have included this additional ablation and discussion in Appendix C.3.

The theoretical analysis relies on stationary solutions over the complete trajectory distribution, whereas practical LLM training involves finite sampling and non-convex optimization. The connection between these theoretical results and the observed empirical improvements remains insufficiently established.

Thank you for this insightful comment. Our theorem gives a target for the full response distribution, whereas LLM training uses sampled rollouts and repeated updates to a neural network. We clarify how the two are connected.

With KK independent rollouts and correctness probability αi\alpha_i per response, both correct and incorrect responses appear with probability 1−αiK−(1−αi)K1-\alpha_i^K-(1-\alpha_i)^K. This gives a direct probabilistic link between the rollouts collected during training and the availability of an HPC correction pair.

We have also added an analysis of the parameter update produced by a rollout batch. Let gDg_D and gHg_H denote the OPD and correction loss gradients. The combined gradient is gD+λgHg_D+\lambda g_H, and under local smoothness the correction loss satisfies

=-\eta\lambda\lVert g_H\rVert^2+O(\eta^2).$$ Thus, for a sufficiently small step, the added term moves the student toward the desired ordering to leading order. This connects the distribution result to the updates used in training without requiring global convergence of non-convex optimization. We also show that the improvement in the generation error bound over OPD can persist under approximate stationarity. We have included these analyses in Appendix B.5. Thank you again for this helpful comment.
Reviewer_JYSLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a basic issue in on-policy distillation: imitating the teacher's generation distribution does not necessarily improve answer correctness. The authors propose Hard Preference Calibration (HPC), which retains token-level teacher supervision, uses verifier-based preferences between correct and incorrect responses for correction, and strengthens the correction when the teacher gives less support to correct trajectories. Experiments on math and code, together with gating ablations, support the effectiveness of this combination. The main value of the work is a targeted way to combine supervision signals. However, the current experiments do not establish whether this method outperforms other approaches that combine OPD with verifiable rewards; the method also has more requirements than other approaches, as it needs both correct and incorrect responses to be generated during rollouts to provide a calibration signal.

Critical strengths
  1. The research problem has practical value. The teacher's generation distribution is not equivalent to task correctness. The paper uses a verifier to try to correct conflicts between the two, which directly relates to the goals of reasoning and code training.
  2. The method preserves the complementary roles of the two types of supervision. It retains dense token-level distillation while adding targeted correction for correct trajectories that receive insufficient teacher support, making the design simple and easy to understand.
  3. There is some evidence for the incremental value of gating. Ablations with no gating, binary gating, and continuous gating show that, in the tested settings, the gains come not only from adding correctness labels but also from selective correction itself (Table 3).
Critical weaknesses
  1. An advantage over other joint training approaches has not been established. Prior work also combines OPD with outcome correctness feedback: Uni-OPD uses outcome-guided margin calibration to address ranking issues in teacher supervision; OPDVR gates distillation rewards using per-trajectory correctness; RSTG combines GRPO with selective teacher guidance. The main tables mainly compare OPD, SCOPE, and GRPO, without performance comparisons against these directly related approaches. Therefore, the current evidence is insufficient to determine whether HPC's additional design outperforms other ways of combining these signals.

  2. The additional correction is limited by within-group sampling conditions. The HPC correction term produces gradients only when both correct and incorrect trajectories are sampled for the same problem and the gating and student-margin conditions are met. This term is zero when all responses are correct or all are incorrect. Compared with other methods such as OPDVR, HPC additionally requires positive and negative samples to be paired within the same group; the method would be more valuable if an analysis, together with the comparisons discussed above, showed that these additional requirements lead to meaningful performance gains.

Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer JYSL

An advantage over other joint training approaches has not been established. Prior work also combines OPD with outcome correctness feedback: Uni-OPD uses outcome-guided margin calibration to address ranking issues in teacher supervision; OPDVR gates distillation rewards using per-trajectory correctness; RSTG combines GRPO with selective teacher guidance. The main tables mainly compare OPD, SCOPE, and GRPO, without performance comparisons against these directly related approaches. Therefore, the current evidence is insufficient to determine whether HPC's additional design outperforms other ways of combining these signals.

Thank you for this important suggestion. To evaluate our calibration against alternative ways of combining teacher supervision with correctness feedback, we have added results for an outcome-guided margin calibration method like Uni-OPD and trajectory-level reward gating like OPDVR. Uni-OPD uses a fixed target margin, which may not sufficiently adapt correction strength to teacher preference reliability. OPDVR’s correctness gating can discard useful teacher guidance within incorrect responses, while HPC retains this guidance and adds targeted pairwise correction. The table reports Pass@16, Pass@8, and Pass@4 on four mathematics benchmarks. All values are percentages (%).

BenchmarkMetricOPDOPDVRUni-OPDSCOPEHPC
AIME24Pass@1664.661.363.661.967.7
AIME24Pass@859.256.359.157.260.7
AIME24Pass@453.250.953.151.353.1
AIME25Pass@1649.957.150.752.552.8
AIME25Pass@845.351.046.547.747.5
AIME25Pass@440.644.441.342.441.7
HMMT25 Feb.Pass@1634.029.734.734.636.2
HMMT25 Feb.Pass@829.427.128.929.529.7
HMMT25 Feb.Pass@424.323.423.824.724.6
HMMT25 Nov.Pass@1634.235.938.338.440.6
HMMT25 Nov.Pass@830.131.033.033.536.4
HMMT25 Nov.Pass@424.825.526.727.530.6
Overall average40.841.141.641.843.5

Both additional approaches improve the overall average over OPD. HPC achieves the highest overall average, showing a consistent aggregate improvement over the compared methods. This supports the effectiveness of using the teacher’s relative sequence scores to calibrate verifier-guided pairwise correction. Our emphasis on teacher preference reliability differs from RSTG’s primary focus. We have added the comparison and discussion in Appendix C.4 and included these approaches in the Related Work section. Thank you again for this suggestion.

The additional correction is limited by within-group sampling conditions. The HPC correction term produces gradients only when both correct and incorrect trajectories are sampled for the same problem and the gating and student-margin conditions are met. This term is zero when all responses are correct or all are incorrect. Compared with other methods such as OPDVR, HPC additionally requires positive and negative samples to be paired within the same group; the method would be more valuable if an analysis, together with the comparisons discussed above, showed that these additional requirements lead to meaningful performance gains.

Thank you for this constructive comment. The mixed response requirement applies only to HPC’s additional correction; OPD remains the underlying training signal. If all sampled responses are correct, there is no incorrect response to correct; if all are incorrect, there is no verified correct response to serve as a target. OPD supervision remains active in both cases.

Student models commonly begin distillation with basic reasoning ability, making this a relatively minor concern. For a prompt with correctness probability α\alpha per response, the probability that all eight independently sampled responses are incorrect is (1−α)8(1-\alpha)^8. This is only 5.8% at α=0.3\alpha=0.3 and 1.7% at α=0.4\alpha=0.4, illustrating that moderate correctness levels can already provide frequent opportunities for pairwise correction.

The additional comparisons above show that HPC achieves higher average performance than the other methods. HPC provides finer control over where correction is needed than Uni-OPD’s fixed margin target allows, while retaining teacher signals that OPDVR may gate out. It concentrates correction on pairs with weak teacher support and an insufficient student preference gap. These results show the value of using verified response pairs for targeted correction. Thank you again for this valuable feedback.

Thank you for your detailed response and additional experiments. The added comparisons with related methods have addressed some of my concerns, but I would still like to better understand two points.

  1. Some baselines perform noticeably worse than OPD on certain tasks, especially OPDVR. These drops seem larger than the occasional decreases reported in its original paper. I understand that the experimental settings and evaluation metrics differ, but I would appreciate an explanation of this pattern and a brief description of how these baselines were implemented and how their hyperparameters were chosen.

  2. HPC requires multiple samples per question for its additional correction, while methods such as OPDVR can use one sample per question and spend the same budget on more different questions. Could the authors compare these two ways of allocating the same total budget of generated tokens to assess whether HPC's sampling requirement is worthwhile?

Thank you for your thoughtful response and careful consideration. We would like to further address your concerns below:

Some baselines perform noticeably worse than OPD on certain tasks, especially OPDVR. These drops seem larger than the occasional decreases reported in its original paper. I understand that the experimental settings and evaluation metrics differ, but I would appreciate an explanation of this pattern and a brief description of how these baselines were implemented and how their hyperparameters were chosen.

Thank you for your thoughtful response and valuable insights. We implemented both methods within HPC’s training framework following their official repositories, using the same student, teacher, training data, learning rate, and evaluation protocol. For OPDVR, we followed the default hyperparameter settings in the official implementation, with the correctness threshold set to 0 and the correctness gating mode set to default.

OPDVR’s lower performance than OPD on some cases may stem from its correctness-based reward filtering. It retains only positive OPD rewards within correct responses and negative rewards within incorrect responses, potentially discarding useful teacher guidance. In practice, during our training, we found that this filtering removed OPD rewards for 42.63% of tokens on average across the entire training dataset. And we also measured the change in the strength of teacher supervision in OPDVR using the absolute difference between the teacher’s and student’s token log probabilities. Across the entire training dataset, the average magnitude of this signal decreased from 0.437 before filtering to 0.205 afterward. This substantial reduction in teacher supervision may hinder the learning and help explain the observed performance drops in some datasets.

HPC requires multiple samples per question for its additional correction, while methods such as OPDVR can use one sample per question and spend the same budget on more different questions. Could the authors compare these two ways of allocating the same total budget of generated tokens to assess whether HPC's sampling requirement is worthwhile?

Thank you for this thoughtful suggestion. We would like to clarify that, in the experiments above, HPC and OPDVR use the same training dataset and matched response generation budgets in our experiments. Specifically, we configure OPDVR to sample eight responses for each of 128 prompts, matching HPC’s sampling setup and producing 1,024 responses per step. We also use identical maximum response lengths and sampling settings. This provides a fair basis for comparison with matched prompt coverage and generation budgets. These results therefore demonstrate that HPC outperforms OPDVR under the same generation budgets.

We hope these details help address your concerns. Thank you again for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.