acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Relative Policy Differences Transfer of On-Policy Distillation

Abstract

On-policy distillation (OPD) trains a target policy on its own rollouts using token-level supervision from a stronger teacher, but typically relies on the teacher being substantially more capable than the target. Recent work relaxes this requirement by transferring the relative policy difference between an expert and a reference policy rather than the expert's absolute distribution. Yet it remains unclear which differences provide useful supervision and which targets can benefit from them. We develop a unified view in which the log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization. This formulation induces an on-policy update for the target, recovers conventional OPD as a special case, and enables any-to-any transfer across models that differ in scale, training stage or generation. Our systematic experiments yield three findings. First, a stronger expert is not necessarily a better source of supervision: differences induced by task-specific training, including reinforcement learning and supervised fine-tuning, transfer consistently, whereas those arising from model scale alone transfer weakly or not at all. Second, the expert and reference serve complementary roles: the expert provides the capability to be transferred, while the reference filters out shared behavior and isolates the relevant residual. Third, even a useful difference does not transfer to every target: success requires the induced update to move the target toward the expert and generalize beyond sampled rollouts. Raw policy distance does not predict transferability, whereas an appropriate reference can rescue otherwise unsuccessful transfer. These results characterize when relative policy differences provide effective supervision for OPD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Policy Transfer, Relative Policy Differences, Large Language Model Post-Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Nge2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper makes analyses on on-policy distillation to answer several questions such as stronger reference/model scale impacts. Based on unified any-to-any formulation to cover several OPD variants, various empirical analyses are provided. For example, it was demonstrated that model scale alone as a reference model does not always help, whereas task-specific models are consistently beneficial.

Critical strengths

This paper first formulate several OPD methods and analyzes any-to-any OPD with different teacher/student models. Several empirical findings include: model scale as a reference model does not always help; task-specific models can consistently benefit distillation; the transfer role of expert and filter role of reference; matching reference benefits. Relatively rich benchmarks are considered across math and coding tasks.

Critical weaknesses

Fig. 2 overlap ratio shows comparison of task-specific or agnostic models, whereas the scale-only impact should compare the case with smaller target model like 1.7B-RL as in Fig. 1(a). All experiments consider alpha=beta and one learning rate, fixed step. However, different proxy pairs have significantly different log-ratio probability. So, we should sweep those hyperparameters to have a sufficiently converged performance to compare. Experiments use a limited token coverage within top 16 tokens. The paper does not provide sensitivity experiments for the token cutoff, the fraction of retained tokens for each source–target pair, or the resulting approximation errors in policy ratios and KL diagnostics. The authors associate successful transfer with increasing expert overlap and generalization to validation examples. However, both quantities are observed during or after target optimization. The paper does not provide a measurable criterion available before training that predicts whether a particular proxy pair will transfer to a given target. The target sweep covers Qwen3 and Qwen2.5 variants, without testing independently developed model families. Most experiments focus on mathematical reasoning; the code results cover a small selection of representative configurations rather than the complete matrix. No training-seed variability, confidence intervals, or significance tests are reported for the transfer comparisons. There are also numerical inconsistencies: the Qwen3-14B/Qwen3-4B source gain is reported as +1.11 in Table 2 but +1.72 in Table 3.

Questions for authors

The state distribution dπθd_{\pi_\theta} depends on the target policy, but Eq. (8) ignores its gradient. Under what assumptions does this surrogate optimization approximate the original objective, and how might this approximation affect transferability? The authors claim that matching the reference model's generation improves transfer. However, changing the reference also changes the transferred policy difference Δϕ\Delta_\phi. How can the effects of generation compatibility and changes in the transferred learning signal be disentangled? Task-trained proxy pairs generally have larger task-performance gains than scale-only pairs. Could the observed transfer differences primarily reflect the proxy performance gap GPG_P, rather than the source of the policy difference? Can the authors provide comparisons controlling for GPG_P? All experiments use a fixed transfer strength α/β=1\alpha/\beta=1. Could be evalluated by tuning this ratio or normalizing the transferred log-probability differences? The initial proxy–target distance KBaseK_{\mathrm{Base}} is measured using trajectories generated by the optimized target, making it dependent on the training outcome. Would the conclusions regarding the relationship between policy distance and transferability remain unchanged if distances were evaluated on a shared, fixed set of states before training?

11 comments
AuthorsOpenReview ↗

Response to Reviewer Nge2 [1/3]

We thank the reviewer for the constructive assessment. The requested sensitivity analyses are below and will be added to the revision.

Response to Weakness 1

Fig. 2 compares two pairs at the same expert scale, chosen because their induced updates are of comparable magnitude: the task-trained pair Qwen3-4B-RL / Qwen3-1.7B and the scale-only pair Qwen3-4B / Qwen3-1.7B (KInducedK_{\mathrm{Induced}} 0.3531 vs. 0.4181, Table 2). The target's overlap with the task-trained expert rises from 0.75 to about 0.81 and overtakes the scale-only curve within 20 steps, while the scale-only overlap drifts down to 0.78.

The 1.7B-scale case has no non-trivial scale-only counterpart, because Qwen3-1.7B against itself is a zero difference. What that scale does provide is the task-trained pair itself, Qwen3-1.7B-RL / Qwen3-1.7B → Qwen3-8B, already reported in Table 2 (first row): 18.43 → 28.21 (+9.78), KInduced=0.0773K_{\mathrm{Induced}} = 0.0773. Task-trained differences therefore transfer at the smaller scale as well, and the scale-only control has to come from a larger scale, which is why Fig. 2 uses the 4B/1.7B pair.

Response to Weakness 2 & Question 4

We swept λ ∈ {0.5, 1, 1.5, 2} on two successful and two failing configurations, with additional λ ∈ {0.75, 0.95, 1.1} for the first pair.

Proxy pairλ=0.5λ=1.0λ=1.5λ=2.0
Qwen3-4B-RL / Qwen3-4B → Qwen3-8B30.8545.2738.74–
Qwen3-1.7B-RL / Qwen3-1.7B → Qwen3-8B0.0028.2137.1933.61
Qwen3-4B-RL / Qwen3-4B → Qwen2.5-7B0.390.050.050.08
Qwen3-4B-RL / Qwen3-4B → Qwen2.5-7B-Math2.872.212.533.29

The trend is consistent with prior work[1][2]: λ materially changes the final score, and its best value depends on the proxy pair — 1.0 for Qwen3-4B-RL / Qwen3-4B and 1.5 for Qwen3-1.7B / Qwen3-1.7B, the latter collapsing to 0.00 at λ = 0.5. The success/failure split, however, is unchanged across the sweep: the failing pairs stay at or below 0.39 (pre-distillation 1.79) and 3.29 (pre-distillation 0.77) at every λ tested.

[1] Fu D, Wu R, Yang Y, et al. Proxy OPD: On-Policy Distillation with Transferable Relative Proxy Update[J]. arXiv preprint arXiv:2607.11505, 2026.

[2] Feng S, Gao H, Chi H, et al. Weak-to-Strong Generalization via Direct On-Policy Distillation[J]. arXiv preprint arXiv:2607.05394, 2026.

Response to Weakness 3

We swept the top-k cutoff over k ∈ {0, 4, 16, 32, 64} on Qwen3-4B-RL / Qwen3-4B → Qwen3-8B.

k04163264
Mean@1646.3246.3046.5747.1347.40
OPD preparation (min/step)8.28.79.911.615.0
Total (min/step)38.138.840.043.146.4

The score spans 1.08 over the whole range, and the curve is flat up to k = 16: k = 16 costs 40.0 min/step and scores 0.83 below k = 64, which costs 46.4 min/step. Beyond k = 16 the remaining gain is therefore bought at a rising price, as OPD preparation time grows from 9.9 to 15.0 min/step.

We also measured how much probability mass the student's top-16 set retains, on 500 randomly drawn training-set samples, for four representative pairs.

PairExpert massReference massStudent-base mass
Qwen3-4B-RL / Qwen3-4B → Qwen3-8B99.85%99.38%99.39%
Qwen3-4B-RL / Qwen3-1.7B → Qwen3-8B99.78%98.70%99.30%
Qwen3-4B-RL / Qwen3-4B → Qwen2.5-7B96.42%96.77%97.39%
Qwen3-4B-RL / Qwen3-4B → Qwen2.5-Math-7B96.09%95.73%96.66%

Across these four pairs, the student's top-16 set already retains at least 95.7% of each frozen model's probability mass, so top-16 is sufficient for computing the reward.

Response to Weakness 4

We tested a broad set of candidate a-priori criteria and none of them separates successful from failing pairs: KBaseK_{\mathrm{Base}}, KInducedK_{\mathrm{Induced}}, KTargetK_{\mathrm{Target}}, the pre-training expert–target overlap, and combinations of these. All three distances are computed for every configuration and reported in Tables 7–10, with the evaluation protocol in Appendix C.

Two failures of these criteria illustrate this. The failing cross-generation pair Qwen3-4B-RL / Qwen3-1.7B → Qwen2.5-7B (Table 10, row 22) carries the largest induced distance in our grid (KInduced=0.6416K_{\mathrm{Induced}} = 0.6416) together with a small target-side distance (KTarget=0.0397K_{\mathrm{Target}} = 0.0397), so it is extreme in one coordinate and unremarkable in the other. Conversely, the two pairs with similar KInducedK_{\mathrm{Induced}} (0.3531 vs. 0.4181, Table 2) differ by more than 22 points in target gain (+25.85 vs. +3.42).

AuthorsOpenReview ↗

Response to Reviewer Nge2 [2/3]

Response to Weakness 5

We agree that both families tested are Qwen. During the rebuttal period our compute was prioritised for the sensitivity analyses the reviewers requested, so an additional independently developed family is not included here; we will add one in the revision.

Response to Weakness 6

We agree that the code results are a targeted check rather than a full matrix. Their purpose is to test whether the findings obtained on math survive a change of domain, not to re-run the grid, and the configurations we did run reproduce the ordering observed on math. Extending the code matrix is a scope we acknowledge as future work.

Response to Weakness 7

We now report bootstrap 95% confidence intervals (CI), computed by resampling each benchmark's evaluation set with replacement 1000 times over per-sample Mean@16, so that the reported comparisons carry an explicit measure of evaluation uncertainty.

PairMean@1695% CIOutcome
Qwen3-4B-RL / Qwen3-4B → Qwen3-8B45.27[40.06, 50.04]success
Qwen3-4B-RL / Qwen2.5-3B → Qwen3-8B47.52[42.34, 52.11]success after reference swap
Qwen3-4B-RL / Qwen3-4B → Qwen2.5-7B0.05[0.00, 0.13]failure
Qwen3-4B-RL / Qwen2.5-3B → Qwen2.5-7B8.53[5.96, 11.15]recovered by reference swap
Qwen3-1.7B-RL / Qwen3-1.7B → Qwen3-8B28.21[23.69, 32.60]success
Qwen3-1.7B-RL / Qwen3-1.7B → Qwen2.5-7B0.15[0.05, 0.27]failure

The intervals for successful and failing configurations are disjoint by a wide margin.

Response to Weakness 8

The reviewer is right that the two tables disagree. The correct value is +1.72: Table 3, row Qwen3-14B / Qwen3-4B-Base, reports 18.43 → 19.54 (+1.72), and Table 6 gives the same two scores. Table 2 mistakenly used the Qwen3-8B value 18.43 as the Qwen3-4B baseline in that row. Table 3 is correct, and we will fix Table 2.

Response to Question 1

Eq. (8) differentiates the state-wise objective: for a state xx drawn from πθ\pi_\theta, the loss is the per-state divergence at xx and the gradient is taken with xx held fixed. The term ∇θdπθ\nabla_\theta d_{\pi_\theta} does not appear because dπθd_{\pi_\theta} is not part of that objective — the surrogate is exact for it, not an approximation of it.

Removing the term does not move the optimum. The state-wise objective is zero at every state the target visits exactly when πθ\pi_\theta matches the teacher's distribution on those states, and that same condition is the global minimum of the trajectory-level objective, where the omitted term vanishes. The term therefore affects the update direction away from the optimum, i.e. the optimization path, but not the optimum. The state distribution is updated implicitly rather than through the gradient: states are re-sampled from the current target at every step, so each iteration re-solves the per-state problem on the new state distribution — the same treatment used in PPO and GRPO. Appendix B.3 derives both gradients explicitly.

AuthorsOpenReview ↗

Response to Reviewer Nge2 [3/3]

Response to Question 2

We agree, and the sentence in §4.4 is incorrect as written. Since Δϕ=log⁡πϕ+−log⁡πϕ\Delta_\phi = \log \pi^+_\phi - \log \pi_\phi, replacing the proxy reference changes the transferred log-ratio by construction, so generation compatibility cannot be separated from the learning signal. Fixed in this comparison: the proxy expert (the same Qwen3-4B-RL checkpoint), the target and its initialization, the data, the steps, and α/β\alpha/\beta. Not fixed: the reference, and with it both the magnitude and the direction of Δϕ\Delta_\phi.

We will replace the sentence "To test whether the family split is causal, we hold the difference fixed and change only the reference's family" with:

"We fix the proxy expert and vary only the proxy reference. Because the reference enters Δϕ\Delta_\phi, this changes the transferred log-ratio as well as the reference's model generation; the comparison thus reflects their combined effect on transfer."

Response to Question 3

This is controlled in Table 3. In the block labelled "Two RL increments with similar proxy gains", GPG_P is +4.80 and +5.39 while GTG_T is +9.62 and +25.03: proxy gains are nearly identical and target gains differ by a factor of 2.6. We agree that GPG_P is strongly associated with GTG_T across Table 2 (Pearson r = 0.96, Fig. 1b) and we state this in §4.1; the controlled comparison shows it is not sufficient.

Response to Question 5

On Question 5: KBaseK_{\mathrm{Base}} is measured on target-visited states because distillation is on-policy — Δϕ\Delta_\phi is evaluated only on states the target itself generates during training. A distance measured on a fixed shared state set would score each pair on states the training signal never reaches, so it could rank pairs differently from the states where transfer actually acts. KBaseK_{\mathrm{Base}} is used in the paper as a diagnostic of what happened, not as a pre-training predictor, and the claim it supports is negative: even measured on exactly those states, distance does not separate successful from failing pairs.

We will add these analyses to the revision.

Reviewer_a1dwOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes using a DPO-style implicit reward as the supervision signal for on-policy distillation. The authors derive the corresponding optimization objective and establish its relationship with conventional OPD. Experiments are conducted on mathematical reasoning, code generation, and other tasks. The experimental results demonstrate that the proposed method is effective to some extent.

Critical strengths
  1. The idea is simple and attempts to address the problem of insufficiently strong teachers using techniques inspired by contrastive learning. This is at least intuitively reasonable.
  2. The theoretical derivations are mostly correct, particularly the relationship between log-probability differences and implicit rewards.
  3. Interpreting OPD from a reinforcement learning perspective is insightful and may motivate future research.
Critical weaknesses
  1. Insufficient novelty and contributions. Implicit rewards have been widely known since DPO, and using implicit rewards to train models is a fairly intuitive idea. Therefore, the novelty and contributions should primarily lie in how the method is implemented and in theoretical guarantees regarding the performance of different implementations. However, the proposed implementation is extremely simple, and the paper does not offer substantial theoretical contributions. Most of the derivations are essentially standard results.

  2. Questionable effectiveness of the method. The accuracy of the implicit reward depends on two factors: (1) whether the expert model was actually optimized using standard reinforcement learning with KL regularization under a BT preference model. However, most open-source models use variants of RL (especially Qwen3), where the KL coefficient is often set to zero, or the reference policy is not the other proxy model used in this paper. (2) Even if condition (1) is satisfied, the extent to which the expert model accurately represents the implicit reward must still be considered, which the paper does not investigate. Consequently, the implicit rewards used in this paper may contain substantial noise. Directly using such rewards for distillation may be unstable and is also theoretically questionable.

  3. Insufficient baseline comparisons. Treating a DPO-trained model as an implicit reward model is already a well-established idea. Therefore, the proposed method is at least closely related to directly performing policy optimization with a reward model. The paper should, at a minimum, include a baseline that performs direct reinforcement learning using a reward model with a parameter count comparable to that of the teacher model.

  4. High computational overhead. The proposed method requires multiple proxy-model probability evaluations during distillation, making it less computationally efficient than conventional OPD.

  5. Biased gradient estimation. When deriving the gradient of the objective function, the authors do not account for the gradient of the distribution $ p_\theta $ appearing inside the expectation.

Questions for authors
  1. How accurate are the implicit rewards used in the proposed method?
  2. How stable is the proposed method across different proxy reference models, particularly when the two proxy models belong to different model families?
8 comments
AuthorsOpenReview ↗

Response to Reviewer a1dw [1/2]

We thank the reviewer for the detailed comments. We address each point below, and we disagree with the assessment on novelty and on the reward-model baseline, for the reasons given.

Response to Weakness 1

We respectfully disagree that the contribution reduces to applying a known implicit reward. The implicit reward is the vehicle, not the contribution; the contribution is a characterisation of when relative policy differences transfer, which prior work does not provide. Specifically, we (i) place relative-difference OPD and conventional OPD in one objective (Eq. 7), so that conventional OPD is recovered as the case where the reference equals the target's initial policy, and the roles of expert, reference and target become explicit; (ii) derive the closed-form optimum (Eq. 19) and its induced-reward form (Eq. 21), from which the effect of the reference on the induced objective can be read off; and (iii) use the formulation to run an intervention grid over expert source, reference and target (Tables 2–4), which yields the paper's three findings: task-training differences transfer better than scale differences, reference selection substantially affects transfer and the expert's original base model is not necessarily the best reference, and the same difference can succeed on one target and fail on another while changing the reference may recover it.

Response to Weakness 2 & Question 1

The stated condition is a sufficient condition for the closed-form identification result, not a requirement for the method to be useful, and we do not assume it holds exactly. Distillation signals are inherently imperfect: a conventional OPD teacher is likewise not the optimum of an explicit KL-regularised objective under a Bradley–Terry model, and its per-token log-probability target is likewise only a proxy for task reward. The question this paper asks is empirical — which pairs yield usable signal — and that is what the experiments measure.

The formulation uses Δϕ\Delta_\phi as a direction, not as a calibrated reward, and the transfer results test that direction end to end. At comparable induced KL, the row Qwen3-4B-RL / Qwen3-4B → Qwen3-8B (KInduced=0.35K_{\mathrm{Induced}} = 0.35) gains +25.85, whereas the scale-only row Qwen3-14B / Qwen3-4B → Qwen3-8B (KInduced=0.42K_{\mathrm{Induced}} = 0.42) gains +0.02 (Table 2).The pair that perturbs the target more transfers essentially nothing while the pair that perturbs it less transfers a great deal, so the magnitude of the perturbation does not determine the outcome; what the difference encodes does.

Response to Weakness 3

We believe the suggested baseline optimises the same objective as Eq. (7). Eq. (7) is KL-regularised policy optimisation with r=Δϕr = \Delta_\phi and a KL anchor to the target's initial policy; the difference from a reward-model RL pipeline lies in the estimator, not the formulation — we use the token-level log-ratio directly as a dense signal under teacher forcing, rather than estimating sequence-level advantages from sampled rewards. A separately trained reward model would replace Δϕ\Delta_\phi with a different reward and would require preference labels or a verifier, which is the requirement this formulation removes. We will make the equivalence between Eq. (7) and reward-model RL explicit in the revision.

AuthorsOpenReview ↗

Response to Reviewer a1dw [2/2]

Response to Weakness 4

We agree that evaluating multiple frozen models introduces additional cost during distillation, and we report the breakdown in the revision.

Per stepA²-OPD: Qwen3-4B-RL / Qwen3-4B → Qwen3-8BOPD: Qwen3-4B-RL → Qwen3-8B
Response length (tokens)82856800
Rollout663 s604 s
Probability evaluation327 s169 s
Total990 s773 s
Probability evaluation / rollout49.3%28.0%
Probability evaluation per 1k tokens39.5 s24.9 s

The two runs differ in response length, so we also report the length-normalised cost; the evaluation-to-rollout ratio is computed within each run and is unaffected by this difference. End to end, per-step time increases by 28% (990 s vs. 773 s), and rollout remains the larger component in both settings.

The evaluation cost is an implementation-level cost with a bound that does not grow with the target: the frozen proxy models are of fixed size, so their evaluation cost stays roughly constant while rollout grows with the target. In conventional OPD the teacher must be of the same size as the student, so this relative overhead decreases as the target grows. The frozen-model evaluations are also mutually independent and could be merged into a single batched pass or executed concurrently given sufficient memory and compute; this is not used in the current implementation and is left to future work.

The overall trade-off also includes expert construction: methods within A²-OPD that transfer relative policy differences can reuse smaller or existing proxy models, which reduces the need to train a large expert specifically for distillation and can substantially reduce expert-construction cost.

Response to Weakness 5

Eq. (8) differentiates the state-wise objective: for a state xx drawn from πθ\pi_\theta, the loss is the per-state divergence at xx and the gradient is taken with xx held fixed. The term ∇θdπθ\nabla_\theta d_{\pi_\theta} does not appear because dπθd_{\pi_\theta} is not part of that objective — the surrogate is exact for it, not an approximation of it.

Removing the term does not move the optimum. The state-wise objective is zero at every state the target visits exactly when πθ\pi_\theta matches the teacher's distribution on those states, and that same condition is the global minimum of the trajectory-level objective, where the omitted term vanishes. The term therefore affects the update direction away from the optimum, i.e. the optimization path, but not the optimum. The state distribution is updated implicitly rather than through the gradient: states are re-sampled from the current target at every step, so each iteration re-solves the per-state problem on the new state distribution — the same treatment used in PPO and GRPO. Appendix B.3 derives both gradients explicitly.

Response to Question 2

This is tested in Table 4, which varies the reference across families with the expert fixed. The cross-family reference is not only stable but can rescue a transfer that fails with a same-family reference: Qwen2.5-7B obtains 0.05 with the Qwen3-4B reference and 8.53 with the Qwen2.5-3B reference. No cross-family configuration collapsed or diverged.

Dear Reviewer a1dw,

Thank you again for your time and thoughtful review of our paper. Your comments and suggestions have been very helpful in improving our work. We hope our responses have addressed your concerns. If you have any further questions or would like us to clarify anything, please feel free to let us know. We would be happy to provide more details.

Best regards, The Authors of A2A^2-OPD

Reviewer_oxDrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when relative policy differences provide effective supervision for on-policy distillation. Building on existing relative transfer methods, it presents A²-OPD, a unified KL-regularized formulation that uses the log-ratio between a proxy expert and reference as an implicit reward, while anchoring the target to its own initial policy. The formulation recovers conventional OPD as a special case and clarifies the roles of the expert, reference, and target.

The validation combines state-wise theoretical analysis with experiments using Qwen3 and Qwen2.5 models on eight math and three code benchmarks. The experiments compare differences arising from reinforcement learning, supervised fine-tuning, and model scaling, and separately vary expert checkpoints, references, and targets.

The results suggest three main findings in the tested settings. First, differences from task training generally produce larger gains than the selected scale differences. Second, reference selection substantially affects transfer, and the expert’s original base model is not necessarily the best reference. Third, the same difference can help some targets but fail on others; changing the reference can recover unsuccessful transfers. Analyses of KL distances, token overlap, and training versus validation performance further investigate transfer behavior, suggesting that simple policy distance alone does not adequately explain the observed successes and failures.

Critical strengths
  1. Reference selection can recover failed transfer. With the Qwen3-4B-RL expert fixed, replacing Qwen3-4B with Qwen2.5-3B as the reference raises the reported Qwen2.5-7B target score from 0.05 to 8.53. This is a consequential finding: the reference materially affects whether an existing expert provides useful supervision, making its selection an important consideration when applying relative policy transfer.
  2. Informative comparisons of what makes a difference transferable. The comparisons across RL, SFT, and model scaling help characterize which source pairs provide useful supervision. With the same target and reference, the pair using the RL expert achieves a reported gain of +25.85 versus +3.42 for the corresponding base-model pair, despite comparable induced KL changes. This supports the paper’s explanatory value by showing that update magnitude alone is insufficient to account for transfer gains.
Critical weaknesses
  1. The central explanation of transferability is not sufficiently established. Section 3.4 claims to hold the policy difference fixed while changing the reference’s model generation. However, changing the reference necessarily changes the log-ratio learning signal, so this intervention cannot isolate generation compatibility from changes in the signal’s direction or magnitude. The overlap and generalization diagnostics also establish associations rather than necessary conditions for successful transfer.

  2. The conclusions about transfer failure lack a sensitivity analysis of transfer strength. All experiments fix α/β = 1 (§3.1), although this coefficient controls the balance between following the source signal and preserving the target’s initial policy. The paper does not test whether unsuccessful configurations remain unsuccessful under reasonable coefficient changes.

Questions for authors
  1. Clarification question regarding the reference intervention. Section 3.4 describes the policy difference as fixed while changing the reference’s model generation. Given that replacing the reference also changes the transferred log-ratio, what precisely is held fixed? What evidence distinguishes an effect of generation compatibility from changes in the learning signal?
  2. Robustness evidence for the current conclusions. Do representative failed configurations remain unsuccessful when α/β is varied around 1? Any available sensitivity results, or clarify whether the conclusions about transferability and target compatibility are intended to apply only to the fixed transfer strength.
5 comments
AuthorsOpenReview ↗

Response to Reviewer oxDr

We sincerely thank the reviewer for the constructive comments. Both weaknesses are valid, and we address them together with the corresponding questions below.

Response to Weakness 1 & Question 1

We agree, and the sentence in §3.4 is incorrect as written. Since Δϕ=log⁡πϕ+−log⁡πϕ\Delta_\phi = \log \pi^+_\phi - \log \pi_\phi, replacing the proxy reference changes the transferred log-ratio by construction, so the difference cannot be held fixed. Fixed in this intervention: the proxy expert (the same Qwen3-4B-RL checkpoint), the target and its initialization, the data, the steps, and α/β\alpha/\beta. Not fixed: the reference, and with it both the magnitude and the direction of Δϕ\Delta_\phi.

We will replace the sentence "To test whether the family split is causal, we hold the difference fixed and change only the reference’s family" with:

"We fix the proxy expert and vary only the proxy reference. Because the reference enters Δϕ\Delta_\phi, this changes the transferred log-ratio as well as the reference's model generation; the comparison thus reflects their combined effect on transfer."

We also agree that the overlap and generalization curves are correlates observed in successful and failing runs, not necessary conditions, and we will describe them as diagnostics.

Response to Weakness 2 & Question 2

We swept λ≡α/β\lambda \equiv \alpha/\beta on the two representative configurations that failed to benefit, with the pair, target, data and steps fixed (Mean@16; Qwen2.5-7B and Qwen2.5-Math-7B score 1.79 and 0.77 before distillation).

Proxy expert / reference → target0.51.01.52.0
Qwen3-4B-RL / Qwen3-4B → Qwen2.5-7B0.390.050.050.08
Qwen3-4B-RL / Qwen3-4B → Qwen2.5-Math-7B2.872.212.533.29

Failures do not recover. Qwen2.5-7B stays at 0.05–0.39 across a 4× range of λ\lambda and never reaches its 1.79 pre-distillation score, so at every λ\lambda tested the transfer is harmful rather than merely weak. Qwen2.5-Math-7B stays in the same low band (2.2–3.3, at most +2.5 over its 0.77 baseline, against +26.84 for the successful pairs), i.e. essentially no transfer.

We will add these results and the corrected sentence to the revision.

Dear Reviewer oxDr,

Thank you again for your time and thoughtful review of our paper. Your comments and suggestions have been very helpful in improving our work. We hope our responses have addressed your concerns. If you have any further questions or would like us to clarify anything, please feel free to let us know. We would be happy to provide more details.

Best regards, The Authors of A2A^2-OPD

Thanks for the additional experiments and for narrowing the claims. These additions help address my original concerns, but I still find the explanatory depth of the empirical analysis limited. My overall assessment of the contribution remains unchanged

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.