acceptodds
Under review as a conference paper at ICLR 2027

Weak-to-Strong On-Policy Distillation

Abstract

On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, has become an effective paradigm for transferring capabilities across large language models (LLMs). Prevailing approaches assume a teacher at least as capable as the student, and either distill a larger model into a smaller one, which fails at the frontier when no larger teacher exists, or train multiple domain experts from a shared base and consolidate them into one student, which requires costly training at the student's scale. To tackle these challenges, we introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a simple yet effective OPD framework that improves the strong student by distilling from multiple weak models. Specifically, W2S-OPD constructs a proxy teacher in logit space from a contrast pair of a positive and a negative model, both smaller than the student and cheap to obtain. Their logit difference isolates the capability direction, which is then added to the student's own base model. The resulting proxy teacher thus couples this direction while staying distributionally adjacent to the student. The student then distills it by minimizing the per-token reverse KL on its own rollouts. We instantiate the contrast pair as i) a post-RL expert against its pre-RL initialization, isolating the skill RL instills, ii) a larger against a smaller base model, isolating the capability from scale, and iii) a small base model with correct and wrong hints, isolating the instance-level direction toward the solution. Across four math and three code benchmarks, W2S-OPD consistently outperforms OPD and even enables the student to surpass the domain teacher and continues to improve the student when every supervision source is weaker, with gains holding from small dense models (e.g., 8B) to large MoE models (e.g., 30B). Further analysis shows that different contrasts yield distinct learning signals: the post-RL and hint contrasts emphasize reasoning frameworks, while the scale contrast emphasizes the solving procedure.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reasoning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_bcfYOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes Weak to Strong OPD method, with the goal to reduce computational cost of doing OPD at frontier scale models, they show in this work using Qwen3 Family models that Larget student model (8B) can get good learning signals from smaller teacher models from the same Family of models ( 4B, 0.6B). They aim to use directional update instead of imitating teacher and propose three ways to create proxy teacher with smaller model contrast pairs, namely : using expert RL, stronger base and Hints. They showed that their method improves performance on math/coding (non-agentic) benchmarks compared to baseline of plain OPD with smaller teacher and SFT.

Critical strengths
  • The proposed method is simple and promising to improve student model with less computational resources, specifically extracting capability directions using smaller teacher model instead of training the same student with larger computational resources (expert RL).
  • Performance Gains: From the Tables 2/3/4 , it's clear that the proposed method using all 3 different proxy teacher initialization works good compared to the baseline in the paper.
Critical weaknesses
  • One of my strongest concern regarding this method is that the paper only considers Qwen3 Family models, and lacks the evidence for cross family generalization of the method, because I believe its not practical to assume that everyone will have different sizes model from the same family. I suspect that the cross family experiments may not work this well, because "exact" directional capability extraction may not hold with models having different pertaining and SFT pipeline distributions, these differences can lead to contrast methods to encode capabilities that are not transferrable.

  • The work doesn't have any stronger baseline, like the exact OPD/MOPD setup, take student model do expert RL and then distill using the stronger Expert RL model to student. This brings out the question of training compute vs maximum performance gains, which to choose when sufficient compute is available?

Questions for authors

Following from my previous text:

  • I would strongly encourage authors to provide any experiments with cross family models using their method.
  • A stronger baseline with expert RL on same student not the weaker model of the same family.

I am happy to increase my scores if authors address these concerns.

3 comments
AuthorsOpenReview ↗

Response to Reviewer bcfY

We sincerely thank you for your valuable feedback and helpful suggestions. We appreciate your recognition of our method as simple and promising with good performance gains. Below, we provide detailed responses to each of your comments and hope to address any further considerations you may have.

Q1: The paper only considers Qwen3 models and lacks evidence of cross-family generalization. Differences in pretraining and SFT distributions may cause the contrast to encode capabilities that do not transfer to the student. Cross-family experiments are needed to assess whether the method remains effective beyond different sizes of the same model family.

A1: Thank you for your insightful comments and suggestion. We extend our evaluation to a cross-family setting using OLMo-3-7B-Instruct as the student, which differs from Qwen3 in both training background and vocabulary. We reuse the same contrast pairs from our main experiments: the math- and code-specific Qwen3-4B-RL experts serve as the positive models, and their pre-RL Qwen3-4B initialization serves as the negative model. To accommodate the tokenizer mismatch, we follow the cross-tokenizer alignment procedure in [1], applying supervision at strictly aligned token positions over the shared vocabulary. The results are presented below:

MethodAIME24AIME25HMMT25(Feb)HMMT25(Nov)Math Avg.HumanEval+MBPP+LCBCode Avg.
Neg. Model (4B)21.619.910.08.214.979.963.620.954.8
Pos. Model (4B-RL)62.057.934.840.448.886.371.027.161.5
Student Base (OLMo-3-7B)50.040.020.225.734.078.264.421.054.5
OPD56.740.825.831.238.680.266.226.157.5
W2S-OPD62.550.028.335.844.283.869.028.660.5

The results show that W2S-OPD consistently outperforms OPD on both math and coding benchmarks, providing evidence that the contrast extracted from Qwen3 models remains useful for an OLMo student despite differences in training background and vocabulary. These findings support the applicability of W2S-OPD beyond same-family teacher–student settings. We added these results and discussions to Section 4.4 in the main text.

Q2: The paper lacks a stronger OPD/MOPD baseline that first performs expert RL on the student-sized model and then distills the resulting stronger expert into the student. How does W2S-OPD compare with this baseline in terms of training compute and achievable performance, particularly when sufficient compute is available?

A2: Thank you for your helpful suggestion. We compare two pipelines: (i) performing expert RL on Qwen3-8B and distilling the resulting expert into the original Qwen3-8B model via OPD; and (ii) using the pre-/post-RL contrast from Qwen3-4B and Qwen3-4B-RL used in our main experiments to train Qwen3-8B via W2S-OPD. Both students start from the same Qwen3-8B initialization. We report math reasoning performance below.

MethodAIME24AIME25HMMT25 (Feb.)HMMT25 (Nov.)Math Avg.
Neg. Model (4B)21.619.910.08.214.9
Pos. Model (4B-RL)62.057.934.840.448.8
Student Base (8B)26.121.111.69.017.0
Expert (8B-RL)70.259.235.047.553.0
OPD w/ 4B-RL62.154.830.638.346.5
OPD w/ 8B-RL67.156.831.740.249.0
W2S-OPD68.960.135.143.151.8

The results show that W2S-OPD outperforms OPD using the student-sized 8B-RL expert on all four math benchmarks, achieving a 2.8 percentage-point gain in average accuracy. This suggests that training a student-sized RL expert does not necessarily yield a better distilled student. We hypothesize that a teacher’s standalone performance does not fully determine the effectiveness of its supervision. Although the 4B-RL expert is weaker than the 8B-RL expert, its pre-/post-RL contrast, when combined with the student’s initial distribution, may provide more effective guidance than directly distilling from the 8B-RL expert even though sufficient compute is available. We added the above results and discussion to the Appendix E.6.

[1] Hou, Bingxi, et al. "Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability." arXiv preprint arXiv:2610.08448 (2026).

Thank you for providing the experiments to establish cross-family generalization and OPD baseline, I have increased my score to reflect the positive assessment of this work. Few things still missing, like end-to-end GPU-hours or training costs for the two pipelines, compute-efficiency gap still remains and for multi-domain setup an actual MOPD baseline.

AuthorsOpenReview ↗

Response to Reviewer bcfY

Thank you so much for your encouraging and positive assessment of our work! We further address your comments as follows:

Q1: End-to-end GPU-hours or training costs for the two pipelines, and compute-efficiency gap.

A1: Thank you for your helpful suggestions. We report the total GPU-hours for both training pipelines on math reasoning, including expert RL and subsequent distillation. The GPU-hours are measured on NVIDIA A100 GPUs.

Training pipelineExpert RL (GPU-hours)Distillation (GPU-hours)Total (GPU-hours)Math Avg.
OPD with an 8B-RL expert1816.466.51882.949.0
W2S-OPD with a 4B pre-/post-RL contrast1288.856.01344.851.8

W2S-OPD reduces end-to-end training GPU-hours by 28.6% while improving average math accuracy from 49.0% to 51.8%. These results demonstrate that W2S-OPD is able to achieve higher performance with lower training cost, when sufficient compute is available to train a student-sized RL expert. We added these results to Appendix E.7.

Q2: An actual MOPD baseline for the multi-domain setting.

A2: Thank you for your suggestion. We added a multi-teacher OPD (MOPD) baseline that jointly distills two Qwen3-8B-RL experts, trained separately on math and coding, into a single Qwen3-8B student. We compare this baseline with W2S-OPD using the corresponding Qwen3-4B pre-/post-RL contrasts. The results are shown below:

MethodAIME24AIME25HMMT25 (Feb.)HMMT25 (Nov.)Math Avg.HumanEval+MBPP+LCBCode Avg.
Neg. Model (4B)21.619.910.08.214.979.963.620.954.8
Pos. Experts (4B-RL)62.057.934.840.448.886.371.027.161.5
Student Base (8B)26.121.111.69.017.081.270.021.757.6
Experts (8B-RL)70.259.235.047.553.087.072.028.662.5
MOPD w/ 4B-RL experts64.354.929.637.446.583.469.525.659.5
MOPD w/ 8B-RL experts66.258.032.542.949.984.370.126.460.3
W2S-OPD67.461.034.745.252.185.271.027.161.1

W2S-OPD outperforms MOPD with student-sized 8B-RL experts on all seven benchmarks, improving average accuracy by 2.2 percentage points on math benchmarks and 0.8 percentage points on coding benchmarks. These results show that the performance advantage of W2S-OPD also extends to jointly transferring multiple domain capabilities into a single student. We added these results and discussions to Appendix E.6.

Thank you again for your helpful feedback and positive assessment of our work! We hope these additional results address your remaining concerns.

Reviewer_1nu5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes W2S-OPD. It builds a proxy teacher by adding the logit difference between a positive and a negative small model to the student's initial model, and then distills this teacher into the student with reverse KL on the student's own rollouts. Three contrast sources are studied (pre- vs. post-RL, larger vs. smaller models, and correct vs. wrong hints). Evaluation covers math, code, and several out-of-domain tasks.

Critical strengths
  1. The paper addresses a practically important problem: improving stronger models using weaker and cheaper supervision, especially when a stronger teacher is unavailable.
  2. The experiments are consistent with the proposed method and relatively comprehensive, covering different model sizes, three types of contrast pairs, and both math reasoning and code generation. The additional analyses and case studies help explain the method’s behavior.
  3. The method is clearly formulated, with clear mathematical notation and informative figures. The paper is generally easy to read.
Critical weaknesses
  1. The methodological novelty is somewhat limited. The proxy teacher combines an existing expert/anti-expert logit composition with on-policy distillation. The paper could better explain the conceptual insights beyond applying this combination to weak-to-strong learning.
  2. Although the experiments cover different model sizes, all models belong to the Qwen3 family. The generality of the method beyond this family remains unclear, particularly when the contrast pair and student have different training backgrounds or vocabularies.
  3. The paper briefly discusses future directions, but provides limited discussion of the method’s applicability conditions and potential failure cases.
Questions for authors
  1. What are the main limitations of the proposed method, and under what conditions might weak-model supervision fail to improve the stronger student?
  2. What broader insights does your method provide beyond combining expert/anti-expert logit composition with on-policy distillation?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1nu5

We sincerely thank you for your positive evaluation of our work. We really appreciate you recognize that we address a practically important problem with comprehensive and consistent experiments, and our paper is easy to read. Below, we provide detailed responses to each of your comments and hope to address any further considerations you may have.

Q1: The methodological novelty is somewhat limited, as the proxy teacher combines existing expert/anti-expert logit composition with on-policy distillation. What broader conceptual insights does the method provide beyond applying this combination to weak-to-strong learning?

A1: Thank you for your comment. We respond to your comments point-to-point as follows:

(i) Technical Novelty: As highlighted in the paper, on-policy distillation faces two key limitations: (1) a stronger teacher may be unavailable for frontier models, and (2) training multiple domain experts at the student’s scale and consolidating their capabilities into a single model is computationally expensive. To address these limitations, our key technical contribution is a simple yet effective method that constructs a proxy teacher by combining the student’s initial distribution with a directional signal extracted from the contrast between weaker models. This allows a stronger student to learn from weaker supervision while reducing the training cost.

(ii) Broader conceptual insight: Our core insight is that a model’s standalone performance does not fully determine its value as a teacher or supervisor. Even when both teacher models are weaker than the student, their relative differences can provide useful guidance when applied to the student’s own base distribution. This suggests that weak-to-strong transfer can benefit from transferring relative behavioral preferences rather than imitating a weak model’s behavior.

We added these discussions to the Discussion paragraph in Section 5 Conclusion.

Q2: All evaluated models belong to the Qwen3 family. The generality of the method beyond this family remains unclear, particularly when the contrast pair and student have different training backgrounds or vocabularies.

A2: Thank you for your helpful suggestion. We extend our evaluation to a cross-family setting using OLMo-3-7B-Instruct as the student, which differs from Qwen3 in both training background and vocabulary. We reuse the same contrast pairs from our main experiments: the math- and code-specific Qwen3-4B-RL experts serve as the positive models, and their pre-RL Qwen3-4B initialization serves as the negative model. To accommodate the tokenizer mismatch, we follow the cross-tokenizer alignment procedure in [1], applying supervision at strictly aligned token positions over the shared vocabulary. The results are presented below:

MethodAIME24AIME25HMMT25(Feb)HMMT25(Nov)Math Avg.HumanEval+MBPP+LCBCode Avg.
Neg. Model (4B)21.619.910.08.214.979.963.620.954.8
Pos. Model (4B-RL)62.057.934.840.448.886.371.027.161.5
Student Base (OLMo-3-7B)50.040.020.225.734.078.264.421.054.5
OPD56.740.825.831.238.680.266.226.157.5
W2S-OPD62.550.028.335.844.283.869.028.660.5

The results show that W2S-OPD consistently outperforms OPD on both math and coding benchmarks, providing evidence that the contrast extracted from Qwen3 models remains useful for an OLMo student despite differences in training background and vocabulary. These findings support the applicability of W2S-OPD beyond same-family teacher–student settings. We added these results and discussions to Section 4.4 in the main text.

Q3: The paper provides limited discussion of the method's applicability conditions and potential failure cases. What are the main limitations, and under what conditions might weak-model supervision fail to improve the stronger student?

A3: Thank you for your question. We respond point-by-point as follows:

(i) Regarding applicability conditions and the main limitations: W2S-OPD requires access to both the teacher and student models' token-level probability distributions, which may restrict its applicability in scenarios where output logits from closed-source models are unavailable.

(ii) Regarding potential failure cases of weak-model supervision: W2S-OPD may provide limited gains when the positive and negative models produce nearly identical distributions on the given tasks, yielding little additional guidance. A contrast dominated by task-irrelevant differences, such as formatting preferences, may also be less useful for improving the target capability.

We have added these discussions to Section 5 (Conclusion).

Reference:

[1] Hou, Bingxi, et al. "Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability." arXiv preprint arXiv:2610.08448 (2026).

Thank you for your response. It addresses most of my concerns, so I’ve raised my Informativeness score to 3 and left my overall score at 3.Good luck to the authors!

AuthorsOpenReview ↗

Thank you!

Thank you for recognizing that we address your concerns and for raising your Informativeness score. We sincerely appreciate your positive assessment and constructive feedback!

Reviewer_SR7QOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes W2S-OPD, an on-policy distillation method that constructs a proxy teacher by combining a frozen student-base model’s logits with a scaled logit contrast between smaller models. The contrasts are derived from RL training, model scale, or correct/incorrect hints. Experiments on math and code demonstrate improvements over standard OPD in the RL-expert setting and gains over initialization even when both supervision sources are weaker than the student, with additional evidence of transfer to a 30B-A3B MoE student.

Critical strengths
  1. Convincing evidence of improvement from weaker supervision. The strongest evidence goes beyond transferring from a smaller but more accurate expert. In Table 3, the contrast between untuned 4B and 0.6B models improves the 8B student’s average math accuracy from 17.0 to 23.0, although both sources have lower average accuracy than the student.

  2. A practical recipe with consistent benefits across several settings. Constructing a proxy teacher from a frozen student base and a small-model contrast is straightforward, and distillation leaves a single student for deployment. In the main RL-expert setting, W2S-OPD improves over OPD by 5.3 percentage points on math and 2.6 points on code, on average. The approach also transfers to the 30B-A3B MoE student, gaining 3.7 points over OPD on math. The scale-based and hint-based constructions further broaden its applicability beyond RL-trained experts. Together with the approximately 20% per-step overhead reported for the pre-/post-RL setting, these results make the method a practically useful contribution.

Critical weaknesses
  1. The contribution of the individual components is not fully isolated. W2S-OPD combines anchoring to the student’s base distribution with subtracting a negative model, whereas ordinary OPD lacks both components. The main comparison therefore establishes the effectiveness of the combined recipe but does not determine how much of the gain specifically requires the negative source. An OPD baseline with a tuned KL penalty toward the student’s initialization would sharpen this attribution.

  2. The top-k implementation needs a more careful theoretical characterization. Equation 7 restricts both distributions to the teacher’s top-32 tokens and renormalizes them. This conditional KL is insensitive to the student’s total probability mass outside the selected support when its relative probabilities within that support remain unchanged. Consequently, small omitted teacher probability mass alone does not justify Appendix B’s claim of an almost lossless approximation to full reverse KL. Because OPD and W2S-OPD use different teacher-dependent supports, differences in student–teacher support overlap could also affect the comparison. The reported improvements remain valuable, but the connection between the implemented loss and Appendix C’s full-distribution objective should be qualified or supported by additional validation.

Questions for authors

How were the amplification coefficients selected, and was this selection performed on a held-out validation set? How well do the selected values transfer between the pre-/post-RL, scale-based, and hint-based settings?

4 comments
AuthorsOpenReview ↗

Response to Reviewer SR7Q (Part I)

We sincerely thank you for your very positive evaluation of our work. We appreciate your recognition of the convincing evidence of improvement from weaker supervision, the practical and useful contribution of our method, and its consistent benefits across several settings. Below, we address each comment and hope to address any further considerations you may have.

Q1: An OPD baseline with a tuned KL penalty toward the student's initialization would help determine how much of the gain specifically requires the negative source.

A1: Thank you for your helpful question. We add an OPD baseline with a KL penalty toward the student’s initialization πS,0\pi_{S,0} :

LOPD+Penalty(πS)=Ex∼D,  y∼πS(⋅∣x)[∑t=1∣y∣(DKL(πS(⋅∣st) ∥ πT(⋅∣st))+βDKL(πS(⋅∣st) ∥ πS,0(⋅∣st)))],\mathcal{L}_{\mathrm{OPD+Penalty}}(\pi_S) =\mathbb{E}_{x \sim \mathcal{D},\; y \sim \pi_S(\cdot \mid x)} \left[ \sum_{t=1}^{|y|} \left( D_{\mathrm{KL}}\bigl( \pi_S(\cdot \mid s_t) \,\big\|\, \pi_T(\cdot \mid s_t) \bigr) + \beta D_{\mathrm{KL}}\bigl( \pi_S(\cdot \mid s_t) \,\big\|\, \pi_{S,0}(\cdot \mid s_t) \bigr) \right) \right],

and evaluate it in the pre-RL/post-RL setting. We sweep the regularization coefficient over β∈{0.25,0.5,0.75,1.0,1.25,1.5}\beta \in \lbrace 0.25, 0.5, 0.75, 1.0, 1.25, 1.5 \rbrace and report the best one selected on held-out validation. The results are shown below:

MethodAIME24AIME25HMMT25 (Feb.)HMMT25 (Nov.)Math Avg.
Neg. Model (4B)21.619.910.08.214.9
Pos. Model (4B-RL)62.057.934.840.448.8
Student Base (8B)26.121.111.69.017.0
OPD62.154.830.638.346.5
OPD w/ KL penalty55.650.726.335.242.0
W2S-OPD68.960.135.143.151.8

The results show that OPD with the tuned KL penalty underperforms standard OPD, while W2S-OPD outperforms both baselines. One possible explanation is that regularization toward the initialization can also constrain beneficial changes needed to acquire the expert’s capabilities, as it does not explicitly distinguish the expert’s acquired capabilities from its underlying base-model behavior. W2S-OPD instead uses the pre-/post-RL contrast to emphasize changes induced by expert training and incorporates this signal into the student’s initial distribution. These results suggest that regularization toward the initialization alone does not explain the gains of W2S-OPD and support the additional value of the negative-model contrast. We added these results and discussions to the Appendix E.5.

Q2: Eq. 7’s renormalized top-32 KL ignores the student’s probability mass outside the teacher’s top-32 support, so a small omitted teacher mass alone doesn’t support Appendix B’s “almost lossless” claim. Also, OPD and W2S-OPD use different teacher supports, which may affect the comparison. The connection between the implemented loss and Appendix C's full-distribution objective should be qualified or supported by additional validation.

A2: Thank you for your helpful comments. We clarify below the approximation error of the top-kk objective and provide additional empirical evidence supporting its use.

(i) Regarding the theoretical characterization: Let s=πS(K)s=\pi_S(\mathcal K) and t=πT(K)t=\pi_T(\mathcal K) denote the student’s and teacher’s probability masses within the teacher’s top-kk support K\mathcal K. Let DKD_{\mathcal K} and DtailD_{\mathrm{tail}} denote the reverse KL divergences between their renormalized conditional distributions within K\mathcal K and its complement, respectively. The full reverse KL admits the exact decomposition:

Dfull=sDK+DKL(Bern⁡(s)∣∣Bern⁡(t))+(1−s)Dtail.D_{\mathrm{full}} = sD_{\mathcal K} + D_{\mathrm{KL}}\bigl(\operatorname{Bern}(s) || \operatorname{Bern}(t)\bigr) + (1-s)D_{\mathrm{tail}}.

Since Eq. 7 computes DKD_{\mathcal K} at each token position, the approximation error satisfies:

∣Dfull−DK∣≤DKL(Bern⁡(s)∣∣Bern⁡(t))+(1−s)(DK+Dtail).\left|D_{\mathrm{full}}-D_{\mathcal K}\right| \le D_{\mathrm{KL}}\bigl(\operatorname{Bern}(s) || \operatorname{Bern}(t)\bigr) + (1-s)\left(D_{\mathcal K}+D_{\mathrm{tail}}\right).

When these terms are small, the conditional top-kk objective closely approximates full reverse KL. We replaced the “almost lossless” characterization with this explicit bound and clarify its connection to the full-distribution objective in Appendix B. The empirical analysis below complements this characterization by measuring student probability coverage on that support.

(ii) Regarding empirical validation: We randomly sampled 64 problems from the DeepMath training set and generate rollouts from the student covering 106,422 response-token positions. At each position, we selected the teacher’s top-3232 tokens and summed their probabilities under the student’s full-vocabulary distribution before renormalization. The resulting average in-support mass is 99.72% for OPD and 99.79% for W2S-OPD, corresponding to average omitted student masses of 0.28% and 0.21%, respectively. These measurements indicate that the selected support captures nearly all student probability mass on average, with similar coverage across the two methods.

AuthorsOpenReview ↗

Response to Reviewer SR7Q (Part II)

Q3: How were the amplification coefficients selected, and was this selection performed on a held-out validation set? How well do the selected values transfer between the pre-/post-RL, scale-based, and hint-based settings?

A3: Thank you for your question. We respond point-by-point as follows:

(i) Selection of amplification coefficients: To select the optimal coefficient α\alpha, we held out a validation set of 256 examples from each of the math and coding training splits. We conducted a grid search over α∈{0.25,0.5,0.75,1.0,1.25,1.5}\alpha \in \lbrace 0.25, 0.5, 0.75, 1.0, 1.25, 1.5 \rbrace and selected the value that yielded the best validation performance.

(ii) Transferability across settings (pre-/post-RL, scale-based, and hint-based): We tuned α\alpha separately for math and coding solely in the pre-/post-RL setting. We then reused the selected coefficient for each domain across the scale-based and hint-based settings without further tuning. The consistent performance gains across all three settings support the transferability of these domain-specific coefficients.

I appreciate the author's effort on the additional experiments. My concerns are addressed. I will maintain my score.

AuthorsOpenReview ↗

Thank you!

Thank you for acknowledging that we have addressed your concerns and for your positive assessment of our work. We sincerely appreciate your thoughtful feedback!

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and effort in reviewing this paper. Please take a moment to read the authors’ rebuttal and the other reviewers’ comments, and participate in the discussion. It would be helpful to indicate which concerns have been addressed and which remain unresolved, and to follow up with the authors where further clarification is needed. Please also update your review and score if warranted, or briefly explain why your assessment remains unchanged. Your engagement will be valuable in reaching a well-informed decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.