acceptodds
Under review as a conference paper at ICLR 2027

Can We Trust the Teacher? Decoupled Credit Direction-Magnitude for Self-Distillation

Abstract

RLVR provides reliable trajectory-level credit, while OPSD offers dense supervision for token-level credit. This exposes a fundamental coupling when updating step-level credit direction and magnitude with teacher supervision, preventing steps from receiving reliable credit directions and contribution magnitudes, while making both vulnerable to teacher judgment errors and preference variance, as supported by our theoretical analysis. To separate credit direction from its contribution magnitude, we introduce Decoupled Credit Self-Distillation (DCSD), which theoretically decouples credit direction and magnitude into two reliable signals and uses them to calibrate privileged teacher supervision. Specifically, we design belief-margin probing to determine credit direction and marginal information gain to quantify credit magnitude, enabling step-to-token credit assignment for policy optimization. Across 11 benchmarks, DCSD achieves the best overall scores against GRPO, OPSD, RLSD, and RLCSD. Compared with base models, DCSD improves the overall score by 8.45 points on mathematical reasoning and 7.01 points on multimodal reasoning, while correcting the credit direction for 6% of tokens and yielding a 1.5 reduction in token credit magnitude.

Primary area:
reinforcement learning
Keywords:
On Policy Self Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vFcJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Decoupled Credit Self-Distillation to improve local supervision by separating update direction from strength. It combines a student answer-belief probe, information-based step weighting, and constrained teacher allocation across tokens. Evaluation on several benchmarks shows improved aggregate accuracy and shorter reasoning outputs relative to related baselines.

Critical strengths
  • Separating direction and magnitude is an interesting approach to the weaknesses of direct self-distillation.
  • The method explicitly limits teacher influence, and its fallback to trajectory-level direction is a reasonable design choice.
  • Results span mathematical and multimodal reasoning, with useful reporting of generation length and computational overhead.
Critical weaknesses
  • Stronger evidence is needed to connect the conditional theory to the implementation. The probe error budget is not measured, and the assumed correspondence between Gaussian evidence and policy success probabilities is not validated.
  • Ablations should isolate direction probing, information weighting, teacher allocation, and overall credit shrinkage. Current diagnostics do not directly measure whether local credit becomes more accurate.
  • Additional variance across multiple seeds and controlled comparisons would strengthen the empirical conclusions.
Questions for authors

Please refer to the weakness

7 comments
AuthorsOpenReview ↗

Response(1/4)

Thank you for recognising that decoupling credit direction from credit magnitude addresses weaknesses in direct self-distillation, and for highlighting the explicit limit on teacher influence and the fallback to outcome-based credit.

To address your three concerns, we (1) clarify where the theory relies on modelling assumptions and restate the two conditional results under weaker, directly measurable conditions that connect the theory to the implementation, (2) use continuation sampling to check whether these conditions hold and directly assess the accuracy of the assigned local credit, and (3) add controlled ablations, including a baseline that uniformly shrinks credit, and report controlled comparisons across multiple random seeds.

W1. Connecting the conditional theory to the implementation

We first clarify which parts of Theorems 2 and 3 involve modeling assumptions. Both results need only conditions on quantities defined by the student's own policy, and these conditions can be measured directly. The oracle Ak⋆A^\star_k is defined on the student's continuation policy, so continuations sampled from a boundary state with the training decoding give unbiased estimates of VπV^\pi, and no surrogate model is needed.

Continuation study. We estimate the oracle of Eq. (1) as A^k⋆=V^(Sk+1)−V^(Sk)\hat A^\star_k=\hat V(S_{k+1})-\hat V(S_k), where V^(Sk)\hat V(S_k) is the success rate of continuations sampled from SkS_k with the training decoding (temperature 1.0, 16k tokens). We used the initial Qwen3-4B. From 480 DAPO problems (disjoint from the 5,120 training prompts of the ablations in W2), each sampled 8 times, we kept the 100 problems with 2–6 correct rollouts and took one correct and one incorrect response per problem (200 responses). Steps follow DCSD's own boundaries. We evaluated all 461 probe-selected steps (32 continuations per boundary; 460 used for sign agreement) together with randomly sampled steps (16 continuations per boundary): 1,661 steps, 2,862 boundaries and 58,880 continuations in total. We call a step's oracle credit significant when ∣A^k⋆∣\lvert\hat A^\star_k\rvert is at least 1.96 standard errors.

(a) Direction (Theorem 2). The proof of Theorem 2 uses only the realized margin error; Ξk\Xi_k is an analytical upper bound on it, built from readout calibration and candidate coverage. In the revision we state the result directly in terms of this error, with a common readout temperature γ 0\gamma\>0 (Theorem 2′):

Let ek=ΔMk−γDk⋆e_k=\Delta M_k-\gamma D^\star_k for some γ 0\gamma\>0. If ∣ek∣<min⁡τM+,−τM−\lvert e_k\rvert<\min\\{\tau_M^+,-\tau_M^-\\}, every probe-selected step satisfies σk=sign⁡(Ak⋆)\sigma_k=\operatorname{sign}(A^\star_k).

The original Theorem 2 is the case γ=1\gamma=1 with ∣ek∣≤Ξk\lvert e_k\rvert\le\Xi_k. The temperature accounts for a teacher-forced readout being uniformly sharper or flatter than free continuation, which rescales margins without changing their sign.

Theorem 2′ also implies an assumption-free statement. Restrict to steps with Ak⋆≠0A^\star_k\neq0, let Uc\mathcal U_c be the steps whose direction DCSD corrects (σk≠sign⁡(Aτ)\sigma_k\neq\operatorname{sign}(A_\tau)), and let pp be the fraction of corrections that agree with sign⁡(Ak⋆)\operatorname{sign}(A^\star_k). Then

∣EDCSD∣−∣Eout∣=∣Uc∣,(1−2p).|\mathcal E_{\mathrm{DCSD}}|-|\mathcal E_{\mathrm{out}}|=|\mathcal U_c|\\,(1-2p).

So DCSD makes fewer direction errors than any outcome-anchored RL method (GRPO, RLSD) whenever more than half of its corrections are right, with no modeling assumption.

Table R1.

QuantityValue
correction precision pp on significant steps (95% CI)61.5% (42.5–77.6), n = 26
direction errors on significant steps: outcome sign → DCSD46 → 40 (−13.0%)
fitted readout temperature γ\gamma0.773
probe-selected steps with ∣ek∣<min⁡τM+,−τM−\lvert e_k\rvert<\min\\{\tau_M^+,-\tau_M^-\\}50.2%
direction precision on the steps satisfying this condition99.13%

The probe error budget is therefore directly measurable. With γ=0.773\gamma=0.773 (the readout is slightly flatter than free continuation), the condition of Theorem 2′ holds on half of the probe-selected steps, and on those steps the probe direction agrees with the continuation estimate in 99.13% of cases, as the theorem predicts. Correction precision also rises with the reliability of the oracle estimate (53.0% → 56.5% → 61.5%; Table R4 in W2), and by the identity above, DCSD makes fewer direction errors than the outcome sign (46 → 40 on significant steps).

AuthorsOpenReview ↗

Response(2/4)

(b) Magnitude (Theorem 3). We first clarify the role of the Gaussian construction. (E1) is a construction, not an assumption about the model. It describes a virtual Bayesian observer that treats the realized hidden states as fixed measurement directions and receives one noisy measurement per token. The identity "ΔFk\Delta F_k = this observer's information gain" holds for any representations, and no actual quantity (hidden states, rewards or values) is assumed to be Gaussian. The link to the policy's success probability is the correspondence in (E3) that the reviewer points to. We show that Theorem 3 needs only a much weaker condition.

Let κk=KL(Ber(Vπ(Sk+1)),∣,Ber(Vπ(Sk)))\kappa_k=\mathrm{KL}\big(\mathrm{Ber}(V^\pi(S_{k+1}))\\,\\|\\,\mathrm{Ber}(V^\pi(S_k))\big) be the information that step kk carries about eventual success; E[κk∣Sk]=I(R;Ck∣Sk)\mathbb E[\kappa_k\mid S_k]=I(R;C_k\mid S_k). The following hold for the actual policy without any evidence model:

  • E[Ak⋆∣Sk]=0\mathbb E[A^\star_k\mid S_k]=0 (the Bellman identity);
  • (Ak⋆)2≤12κk(A^\star_k)^2\le\tfrac12\kappa_k (Pinsker's inequality);
  • E[(Ak⋆)2∣Sk]≤14\mathbb E[(A^\star_k)^2\mid S_k]\le\tfrac14;
  • E[Zk⋆∣R=1,Sk]=KL(π+∣π)≤E[κk∣Sk]/Vπ(Sk)\mathbb E[Z^\star_k\mid R=1,S_k]=\mathrm{KL}(\pi^+\\|\pi)\le\mathbb E[\kappa_k\mid S_k]/V^\pi(S_k), where π+\pi^+ is the policy conditioned on success;
  • Ak⋆=0  ⟺  κk=0A^\star_k=0\iff\kappa_k=0.

The evidence model enters only through one inequality linking κk\kappa_k to information gain:

(B)κk≤c,ΔFkfor a constant c 0.\text{(B)}\qquad \kappa_k\le c\\,\Delta F_k \quad\text{for a constant } c\>0.

In the revision, we restate Theorem 3 as Proposition 3′ with (B) as its only assumption. Under (B):

  • E[(Ak⋆)2]≤min⁡c2ΔFk,14\mathbb E[(A^\star_k)^2]\le\min\\{\tfrac c2\Delta F_k,\tfrac14\\};
  • E[Zk⋆∣R=1,Sk]≤c,E[ΔFk∣Sk]/Vπ(Sk)\mathbb E[Z^\star_k\mid R=1,S_k]\le c\\,\mathbb E[\Delta F_k\mid S_k]/V^\pi(S_k);
  • ΔFk=0⇒Ak⋆=0\Delta F_k=0\Rightarrow A^\star_k=0.

(E1)–(E3) is one model under which (B) holds with c=1c=1, by data processing; the sharpness of the envelope is kept as a remark within that model. Since αk\alpha_k depends only on ratios of ΔF\Delta F, the constant cc does not affect the method.

We then test (B) and its predictions directly on the continuation data:

  • (B) itself. The ratio κk/ΔFk\kappa_k/\Delta F_k is small and light-tailed: its 90/95/99% quantiles are 0.0036/0.0081/0.032 and its maximum is 0.332. Hence (B) holds on every evaluated step with the constant c=1c=1 implied by (E1)–(E3), and on 95% of steps with the much tighter c=0.0081c=0.0081, which we use below.
  • Monotonicity and envelope (Table R2):

Table R2.

ΔFk\Delta F_k quintile12345
mean ΔFk\Delta F_k (nats)14.236.266.2113.6261.8
mean ∣A^k⋆∣\lvert\hat A^\star_k\rvert0.0930.0980.1030.1130.115
noise-corrected E[(A^k⋆)2]\mathbb E[(\hat A^\star_k)^2]0.01150.01170.01280.01960.0273
envelope min⁡c2ΔFk,14\min\\{\tfrac c2\Delta F_k,\tfrac14\\}, c=0.0081c=0.00810.0570.1460.2500.2500.250

The noise-corrected second moment increases monotonically with ΔFk\Delta F_k (2.4× from the lowest to the highest quintile), the lowest-ΔF\Delta F quintile has the smallest oracle credit, and every quintile stays below the envelope. As stated in the manuscript, the envelope is an upper bound rather than a pointwise prediction, and it is conservative here.

  • Discrimination. For identifying the 129 steps with significant oracle credit, αk\alpha_k gives AUC 0.645 (raw ΔFk\Delta F_k: 0.596), compared with 0.497 for step length and 0.494 for teacher magnitude.

We will add Theorem 2′, the corollary above, Proposition 3′ and these measurements to the revised manuscript as a new appendix section.

AuthorsOpenReview ↗

Response(3/4)

W2. Ablations including overall credit shrinkage, and direct accuracy of local credit

Ablations. We ran a controlled suite. Each variant changes one component of DCSD, and all other conditions are identical: initial model (Qwen3-4B), the same 20 training batches (the first 5,120 DAPO-17K prompts, 256 prompts × 4 rollouts), 16k response limit, seed, and hardware/software stack.

  • direction probing: σk=sign(Aτ)\sigma_k=\mathrm{sign}(A_\tau) for all steps;
  • information weighting: αk≡1\alpha_k\equiv1 (uniform), and αk\alpha_k computed from the step length ∣Ck∣\lvert C_k\rvert with the same normalization;
  • teacher allocation: ϵw=0\epsilon_w=0, i.e. uniform allocation within each step;
  • overall credit shrinkage: αk\alpha_k randomly permuted within each response. This keeps the same set of step weights in every response, and hence the same distribution of step magnitudes and an overall attenuation of the same order as DCSD; only which step receives which weight changes.

Table R3. avg@32 on AIME24/25/26 and HMMT25, avg@16 on AMC23 and MATH500 (temperature 0.6, top-p 0.95, top-k 20, 16k tokens). Avg. is the unweighted mean of the six benchmarks. Because the decoding, sampling budget and training horizon differ, absolute numbers are not directly comparable to the mean@4 results in Table 1. Best result per column in bold.

VariantAIME24AIME25AIME26HMMT25AMC23MATH500Avg.Δ vs. DCSD
DCSD67.457.357.616.693.492.764.17–
w/o teacher allocation67.556.855.618.192.892.263.83−0.33
Shuffled (overall shrinkage)66.954.557.216.593.392.863.53−0.63
Uniform weighting66.655.055.916.292.792.963.22−0.95
w/o direction probing66.255.754.216.990.692.862.73−1.43
Step-length weighting63.651.956.216.892.392.662.23−1.93
Qwen3-4B (before training)63.050.251.516.690.991.860.67−3.50

Removing direction probing, information weighting or teacher allocation reduces average accuracy by 1.43, 0.95 and 0.33 points, all larger than the seed standard deviation of 0.13 (Table R5). Step-length weighting is the worst variant (−1.93). The shuffled variant loses 0.63 points despite using the same weights, while uniform weighting (no shrinkage) loses 0.95. Overall shrinkage alone therefore does not explain the gains; the allocation across steps does.

Direct accuracy of local credit. Using the continuation study described in W1, we measure local-credit accuracy directly against the oracle of Eq. (1). All methods' credits are computed on the same responses and steps.

Table R4. Agreement with the sign of A^k⋆\hat A^\star_k (%), by how reliably the oracle is estimated.

all A^k⋆≠0\hat A^\star_k\neq0 (n = 1,249)≥ 1 s.e. (n = 645)significant, ≥ 1.96 s.e. (n = 129)
outcome sign (GRPO / RLSD)53.255.564.3
teacher sign (OPSD, ∑t∈Ckδt\sum_{t\in C_k}\delta_t)50.151.049.6
DCSD σk\sigma_k54.057.269.0
On steps whose direction DCSD corrects:
DCSD correct (correction precision pp)53.0 (n = 151)56.5 (n = 85)61.5 (n = 26)
— positive corrections inside failed responses51.052.858.8
— negative corrections inside successful responses56.462.566.7
teacher sign correct on the same steps47.748.253.8
  • Direction agreement. On steps with significant oracle credit: outcome sign (GRPO/RLSD) 64.3%, teacher sign (OPSD) 49.6%, DCSD 69.0%.
  • Correction precision. Where DCSD reverses the outcome direction, the continuation estimate agrees with DCSD in 61.5% of cases (58.8% for positive and 66.7% for negative corrections). The teacher's sign is right on only 53.8% of the same steps. Combined with the identity in W1, DCSD makes 13% fewer direction errors than the outcome sign on these steps.
  • Magnitude. Within-response Spearman correlation between ∣creditk∣\lvert\mathrm{credit}_k\rvert and ∣A^k⋆∣\lvert\hat A^\star_k\rvert: DCSD 0.065, teacher ∣ZkT∣\lvert Z^T_k\rvert 0.032, step length 0.021, shuffled weights −0.003. The absolute values are small because single-step oracle estimates are noisy (the standard error of A^k⋆\hat A^\star_k is comparable to its typical magnitude of about 0.1), but the ordering matches the ablation above.
Reviewer_uWxTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed a Decoupled Credit Self-Distillation (DCSD) method for Self-Distillation, which calibrates teacher supervision by separately determining credit direction and contribution magnitude. Experiments show it works better than GRPO, OPSD, RLSD, and RLCSD. The approach is viable in theory, worth testing experimentally, and shows some novelty.

Critical strengths

The experimental evaluation is fairly comprehensive. The approach is theoretically viable. It decouples credit direction and magnitude.

Critical weaknesses

No obvious issues.

Questions for authors

nil

3 comments
AuthorsOpenReview ↗

Response to Reviewer uWxT

Thank you for your positive assessment, for finding the evaluation comprehensive, and for recognising the theoretical rationale for decoupling credit direction from credit magnitude. In light of your comment that the approach merits further experimental testing, we briefly summarise the additional empirical evidence below.

  • Direct test against the oracle credit. We estimated each step's local advantage (the oracle of Eq. (1)) by continuation sampling. On steps with significant oracle credit, the teacher's step sign agrees with the oracle only at chance level (49.6%), whereas DCSD's direction agrees in 69.0% of cases, and its direction corrections are right in 61.5%.
  • Component ablations. Removing direction probing, information-gain weighting or teacher allocation lowers the six-benchmark average by 1.43, 0.95 and 0.33 points. Step-length weighting (−1.93) and within-response shuffled weights (−0.63) are both worse than DCSD, which rules out step-length and overall-shrinkage explanations.
  • Theory–implementation link. The probe's error condition can be measured directly: it holds on half of the probe-selected steps, which are then correctly signed in 99.13% of cases. Oracle credit increases monotonically with information gain and stays below the information-gain envelope.
  • Robustness. The standard deviation across three seeds is 0.13 points, smaller than every ablation gap. We also trained DCSD on Phi-4-reasoning (14B), a larger model from a different family, where it improves the average by 0.81 points compared with GRPO.

We will include all of these results in the appendix of the revised manuscript. We hope the additional evidence strengthens your assessment of our work.

AuthorsOpenReview ↗

Follow-Up on Our Response Ahead of the November 18 Discussion Deadline

Dear Reviewer uWxT,

We have uploaded a revised manuscript PDF (see blue colour) that incorporates your feedback. We would appreciate it if you could review the revised manuscript and our responses above. Please let us know if you have any further questions.

Thank you once again for your careful review and constructive comments.

Best wishes,

DCSD Authors

Dear Reviewer uWxT,

Thank you again for your positive assessment of our work. As the discussion period draws to a close, we wanted to briefly follow up on the additional evidence presented in our response, including component ablations, continuation-based evaluation of local credit, results across multiple seeds, and experiments on Phi-4-reasoning (14B). These additions are included in the revised manuscript, with revisions highlighted in blue.

We hope these results further strengthen your assessment of our work. We would be grateful if you could consider revisiting your assessment and score in light of this additional evidence, and we would welcome any further questions or suggestions.

Thank you once again for your time and careful review.

Best wishes,

DCSD Authors

Reviewer_1VDVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Decoupled Credit Self-Distillation (DCSD), a method for improving credit assignment in LLM on-policy self-distillation. The authors argue that existing methods couple credit direction and magnitude through single direct teacher supervision, making it vulnerable to teacher judgment errors. To address this issue, DCSD calibrates the teacher signal by separately determining credit direction through belief-margin probing and contribution magnitude through marginal information gain. The paper also provides theoretical analysis of the proposed framework and evaluates its effectiveness across mathematical and multimodal reasoning benchmarks. Experimental results demonstrate promising improvements over existing methods.

Critical strengths
  1. The idea of separating credit direction from magnitude provides a useful and well-motivated perspective on the limitations of existing self-distillation methods.

  2. The experimental results demonstrate promising improvements across mathematical and multimodal reasoning benchmarks, with additional analyses of credit direction, magnitude, and reasoning efficiency.

Critical weaknesses
  1. The paper lacks ablation studies to isolate the contributions of belief-margin probing, marginal information gain, and teacher-guided token allocation. In particular, the effectiveness of information-gain-based credit weighting remains insufficiently justified without comparisons to simpler alternatives, such as uniform or step-length-based weighting. Such comparisons would help clarify whether the improvements stem from information-aware credit assignment rather than differences in step length or overall credit magnitude.

  2. The paper reports corrections to credit direction and improved agreement with a stronger reference model, but these results do not directly establish whether the assigned credit reflects the actual contributions of intermediate reasoning steps. It would be helpful to conduct a comparison with local advantages estimated through continuation sampling.

  3. The experiments are conducted on Qwen3-4B for mathematical reasoning and Qwen3-VL-8B for multimodal reasoning. It remains unclear whether the proposed method can consistently improve performance across different model sizes and families. As a general method for credit assignment, more experiments with larger models and different model architectures would help demonstrate the effectiveness of DCSD.

Questions for authors

See weaknesses.

7 comments
AuthorsOpenReview ↗

Response(1/4)

Thank you for recognising the importance of decoupling credit direction from credit magnitude. Our empirical analyses examine both components and assess reasoning efficiency.

To address your three concerns, we added (1) controlled ablations against simpler alternatives, (2) an offline continuation-sampling study to test whether assigned credit reflects each step’s contribution to final-answer correctness, and (3) experiments on Phi-4-reasoning (14B) to evaluate whether our findings extend beyond Qwen3-4B and Qwen3-VL-8B. We describe each addition below.

W1. Ablations isolating the three components, and simpler magnitude baselines

We ran a controlled ablation suite. Each variant changes exactly one component of DCSD, and all other conditions are identical: initial model (Qwen3-4B), the same 20 training batches (the first 5,120 DAPO-17K prompts, 256 prompts × 4 rollouts), 16k response limit, seed, and hardware/software stack. The three magnitude variants replace only the relative step magnitude αk\alpha_k in Eq. (7), leaving σk\sigma_k and qk,tq_{k,t} unchanged.

VariantChange relative to DCSDIsolates
w/o belief-margin probingσk=sign(Aτ)\sigma_k=\mathrm{sign}(A_\tau) for all stepsdirection probing
Uniform weightingαk≡1\alpha_k\equiv1 for all stepsinformation-aware magnitude
Step-length weightingαk\alpha_k computed from the step length ∣Ck∣\lvert C_k\rvert instead of ΔFk\Delta F_k, with the same normalizationstep-length confound
Shuffled information gainαk\alpha_k randomly permuted among the steps of each responseoverall credit magnitude (the same set of step weights in every response; only the step-to-weight assignment changes)
w/o teacher allocationuniform within-step allocation (ϵw=0\epsilon_w=0, i.e. qk,t=1/∣Ck∣q_{k,t}=1/\lvert C_k\rvert)teacher-guided token allocation

Table R1. avg@32 on AIME24/25/26 and HMMT25, avg@16 on AMC23 and MATH500 (temperature 0.6, top-p 0.95, top-k 20, 16k tokens). Avg. is the unweighted mean of the six benchmarks. Because the decoding, sampling budget and training horizon differ, absolute numbers are not directly comparable to the mean@4 results in Table 1. Best result per column in bold.

VariantAIME24AIME25AIME26HMMT25AMC23MATH500Avg.Δ vs. DCSD
DCSD67.457.357.616.693.492.764.17–
w/o teacher allocation67.556.855.618.192.892.263.83−0.33
Shuffled information gain66.954.557.216.593.392.863.53−0.63
Uniform weighting66.655.055.916.292.792.963.22−0.95
w/o belief-margin probing66.255.754.216.990.692.862.73−1.43
Step-length weighting63.651.956.216.892.392.662.23−1.93
Qwen3-4B (before training)63.050.251.516.690.991.860.67−3.50

To calibrate these gaps, we also trained DCSD with three seeds under the same protocol: Avg. is 64.17, 64.25 and 64.00 (standard deviation 0.13 points).

AuthorsOpenReview ↗

Response(2/4)

Findings:

  1. Each component contributes, and direction probing contributes most. Removing belief-margin probing, information-gain weighting (uniform), or teacher allocation lowers the average by 1.43, 0.95 and 0.33 points, respectively; all three gaps exceed the seed standard deviation. Direction correction matters more than magnitude allocation, consistent with our central claim that the main failure of direct teacher supervision is a wrong direction. Teacher allocation has the smallest effect, as expected: by Theorem 4 it cannot change a step's direction or total credit, only how that credit is spread over the step's tokens, and its role is to keep teacher influence bounded.
  2. Not a step-length effect. Information gain is correlated with step length (Spearman ρ = 0.72 on training rollouts), yet length-based weights with the same normalization are the worst variant: 1.93 points below DCSD and 0.98 points below uniform weighting, so allocating credit by length actively hurts. In the continuation study (W2), information gain also tracks the measured step contributions better than length does: within-response Spearman correlation with ∣A^k⋆∣\lvert\hat A^\star_k\rvert of 0.065 vs. 0.021, and AUC for identifying high-contribution steps of 0.645 vs. 0.497.
  3. Not an overall-magnitude effect. Shuffled weights are exactly DCSD's set of step weights in every response, so the distribution of step magnitudes is identical, yet they are 0.63 points worse. Uniform weighting, which applies no attenuation, is 0.95 points worse. The gain therefore comes from where credit is placed, not from shrinking it. This is consistent with the optimizer: with PPO ratio clipping, Adam and gradient-norm clipping and no KL term, a global rescaling of the advantage has little effect.

These are short-horizon (20-step) runs under a single protocol.

We will include Table R1 in the revised appendix and add full-horizon versions in the final version.

AuthorsOpenReview ↗

Response(3/4)

W2. Does the credit reflect actual step contributions? Comparison with continuation-sampled local advantages

Following your suggestion, we complemented Table 2 and Fig. 3(c–d) with a more direct test of step-level credit. We used continuation sampling to estimate the local advantage defined in Eq. (1), A^k⋆=V^(Sk+1)−V^(Sk)\hat A^\star_k=\hat V(S_{k+1})-\hat V(S_k). Here, V^(Sk)\hat V(S_k) is the fraction of successful continuations sampled from the boundary state SkS_k using the student’s training-time decoding settings (temperature 1.0 and a 16k-token limit). Sampling directly from the student’s policy gives an unbiased Monte Carlo estimate of Vπ(Sk)V^\pi(S_k) under these decoding settings and requires no surrogate model.

Setup. We used the initial Qwen3-4B. From 480 DAPO problems (disjoint from the 5,120 ablation prompts), each sampled 8 times, we kept the 100 problems with 2–6 correct rollouts and took one correct and one incorrect response per problem (200 responses). Steps follow DCSD's own boundaries. We evaluated all 461 probe-selected steps (32 continuations per boundary; 460 used for sign agreement) together with randomly sampled steps (16 continuations per boundary): 1,661 steps, 2,862 boundaries and 58,880 continuations in total. All methods' credits are computed on the same responses and steps. We call a step's oracle credit significant when ∣A^k⋆∣\lvert\hat A^\star_k\rvert is at least 1.96 standard errors.

Table R2. Agreement with the sign of A^k⋆\hat A^\star_k (%), by how reliably the oracle is estimated.

all A^k⋆≠0\hat A^\star_k\neq0 (n = 1,249)≥ 1 s.e. (n = 645)significant, ≥ 1.96 s.e. (n = 129)
outcome sign (GRPO / RLSD)53.255.564.3
teacher sign (OPSD, ∑t∈Ckδt\sum_{t\in C_k}\delta_t)50.151.049.6
DCSD σk\sigma_k54.057.269.0
On steps whose direction DCSD corrects:
DCSD correct (correction precision pp)53.0 (n = 151)56.5 (n = 85)61.5 (n = 26)
— positive corrections inside failed responses51.052.858.8
— negative corrections inside successful responses56.462.566.7
teacher sign correct on the same steps47.748.253.8
  • The teacher's sign is unreliable. On significant steps the teacher's step sign agrees with the oracle in 49.6% of cases (95% CI 41.1–58.1), i.e. at chance, whereas DCSD reaches 69.0% (60.6–76.3). This directly addresses your question, and supports using the teacher only for within-step allocation.
  • DCSD's corrections are accurate. Among significant steps where DCSD reverses the outcome direction, the continuation estimate agrees with DCSD in 61.5% of cases (58.8% for positive corrections inside failed responses, 66.7% for negative corrections inside successful ones). This is above 50%, so on these steps the outcome direction used by GRPO/RLSD is wrong more often than correct; on the same steps the teacher's sign is correct in only 53.8%. Agreement with DCSD's corrections rises as the oracle estimate becomes more reliable (53.0% → 56.5% → 61.5%), whereas the teacher stays near 50% under every filter. Since DCSD and the outcome sign differ only on corrected steps, DCSD makes 13% fewer direction errors than the outcome sign on significant steps.
  • Threshold. Probe precision (agreement of sign(ΔMk)\mathrm{sign}(\Delta M_k) with sign(A^k⋆)\mathrm{sign}(\hat A^\star_k) on probe-selected steps) generally increases with the probe threshold while coverage shrinks (Table R3). The default threshold selects 4.36% of steps. Consistent with Theorem 2, on the probe-selected steps whose measured margin error is within the threshold tolerance (50.2% of them), the probe direction agrees with the continuation estimate in 99.13% of cases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.