- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Decoupled Credit Self-Distillation to improve local supervision by separating update direction from strength. It combines a student answer-belief probe, information-based step weighting, and constrained teacher allocation across tokens. Evaluation on several benchmarks shows improved aggregate accuracy and shorter reasoning outputs relative to related baselines.
Critical strengths
- Separating direction and magnitude is an interesting approach to the weaknesses of direct self-distillation.
- The method explicitly limits teacher influence, and its fallback to trajectory-level direction is a reasonable design choice.
- Results span mathematical and multimodal reasoning, with useful reporting of generation length and computational overhead.
Critical weaknesses
- Stronger evidence is needed to connect the conditional theory to the implementation. The probe error budget is not measured, and the assumed correspondence between Gaussian evidence and policy success probabilities is not validated.
- Ablations should isolate direction probing, information weighting, teacher allocation, and overall credit shrinkage. Current diagnostics do not directly measure whether local credit becomes more accurate.
- Additional variance across multiple seeds and controlled comparisons would strengthen the empirical conclusions.
Questions for authors
Please refer to the weakness
7 comments
Response(2/4)
(b) Magnitude (Theorem 3). We first clarify the role of the Gaussian construction. (E1) is a construction, not an assumption about the model. It describes a virtual Bayesian observer that treats the realized hidden states as fixed measurement directions and receives one noisy measurement per token. The identity " = this observer's information gain" holds for any representations, and no actual quantity (hidden states, rewards or values) is assumed to be Gaussian. The link to the policy's success probability is the correspondence in (E3) that the reviewer points to. We show that Theorem 3 needs only a much weaker condition.
Let be the information that step carries about eventual success; . The following hold for the actual policy without any evidence model:
- (the Bellman identity);
- (Pinsker's inequality);
- ;
- , where is the policy conditioned on success;
- .
The evidence model enters only through one inequality linking to information gain:
In the revision, we restate Theorem 3 as Proposition 3′ with (B) as its only assumption. Under (B):
- ;
- ;
- .
(E1)–(E3) is one model under which (B) holds with , by data processing; the sharpness of the envelope is kept as a remark within that model. Since depends only on ratios of , the constant does not affect the method.
We then test (B) and its predictions directly on the continuation data:
- (B) itself. The ratio is small and light-tailed: its 90/95/99% quantiles are 0.0036/0.0081/0.032 and its maximum is 0.332. Hence (B) holds on every evaluated step with the constant implied by (E1)–(E3), and on 95% of steps with the much tighter , which we use below.
- Monotonicity and envelope (Table R2):
Table R2.
| quintile | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| mean (nats) | 14.2 | 36.2 | 66.2 | 113.6 | 261.8 |
| mean | 0.093 | 0.098 | 0.103 | 0.113 | 0.115 |
| noise-corrected | 0.0115 | 0.0117 | 0.0128 | 0.0196 | 0.0273 |
| envelope , | 0.057 | 0.146 | 0.250 | 0.250 | 0.250 |
The noise-corrected second moment increases monotonically with (2.4× from the lowest to the highest quintile), the lowest- quintile has the smallest oracle credit, and every quintile stays below the envelope. As stated in the manuscript, the envelope is an upper bound rather than a pointwise prediction, and it is conservative here.
- Discrimination. For identifying the 129 steps with significant oracle credit, gives AUC 0.645 (raw : 0.596), compared with 0.497 for step length and 0.494 for teacher magnitude.
We will add Theorem 2′, the corollary above, Proposition 3′ and these measurements to the revised manuscript as a new appendix section.
Response(3/4)
W2. Ablations including overall credit shrinkage, and direct accuracy of local credit
Ablations. We ran a controlled suite. Each variant changes one component of DCSD, and all other conditions are identical: initial model (Qwen3-4B), the same 20 training batches (the first 5,120 DAPO-17K prompts, 256 prompts × 4 rollouts), 16k response limit, seed, and hardware/software stack.
- direction probing: for all steps;
- information weighting: (uniform), and computed from the step length with the same normalization;
- teacher allocation: , i.e. uniform allocation within each step;
- overall credit shrinkage: randomly permuted within each response. This keeps the same set of step weights in every response, and hence the same distribution of step magnitudes and an overall attenuation of the same order as DCSD; only which step receives which weight changes.
Table R3. avg@32 on AIME24/25/26 and HMMT25, avg@16 on AMC23 and MATH500 (temperature 0.6, top-p 0.95, top-k 20, 16k tokens). Avg. is the unweighted mean of the six benchmarks. Because the decoding, sampling budget and training horizon differ, absolute numbers are not directly comparable to the mean@4 results in Table 1. Best result per column in bold.
| Variant | AIME24 | AIME25 | AIME26 | HMMT25 | AMC23 | MATH500 | Avg. | Δ vs. DCSD |
|---|---|---|---|---|---|---|---|---|
| DCSD | 67.4 | 57.3 | 57.6 | 16.6 | 93.4 | 92.7 | 64.17 | – |
| w/o teacher allocation | 67.5 | 56.8 | 55.6 | 18.1 | 92.8 | 92.2 | 63.83 | −0.33 |
| Shuffled (overall shrinkage) | 66.9 | 54.5 | 57.2 | 16.5 | 93.3 | 92.8 | 63.53 | −0.63 |
| Uniform weighting | 66.6 | 55.0 | 55.9 | 16.2 | 92.7 | 92.9 | 63.22 | −0.95 |
| w/o direction probing | 66.2 | 55.7 | 54.2 | 16.9 | 90.6 | 92.8 | 62.73 | −1.43 |
| Step-length weighting | 63.6 | 51.9 | 56.2 | 16.8 | 92.3 | 92.6 | 62.23 | −1.93 |
| Qwen3-4B (before training) | 63.0 | 50.2 | 51.5 | 16.6 | 90.9 | 91.8 | 60.67 | −3.50 |
Removing direction probing, information weighting or teacher allocation reduces average accuracy by 1.43, 0.95 and 0.33 points, all larger than the seed standard deviation of 0.13 (Table R5). Step-length weighting is the worst variant (−1.93). The shuffled variant loses 0.63 points despite using the same weights, while uniform weighting (no shrinkage) loses 0.95. Overall shrinkage alone therefore does not explain the gains; the allocation across steps does.
Direct accuracy of local credit. Using the continuation study described in W1, we measure local-credit accuracy directly against the oracle of Eq. (1). All methods' credits are computed on the same responses and steps.
Table R4. Agreement with the sign of (%), by how reliably the oracle is estimated.
| all (n = 1,249) | ≥ 1 s.e. (n = 645) | significant, ≥ 1.96 s.e. (n = 129) | |
|---|---|---|---|
| outcome sign (GRPO / RLSD) | 53.2 | 55.5 | 64.3 |
| teacher sign (OPSD, ) | 50.1 | 51.0 | 49.6 |
| DCSD | 54.0 | 57.2 | 69.0 |
| On steps whose direction DCSD corrects: | |||
| DCSD correct (correction precision ) | 53.0 (n = 151) | 56.5 (n = 85) | 61.5 (n = 26) |
| — positive corrections inside failed responses | 51.0 | 52.8 | 58.8 |
| — negative corrections inside successful responses | 56.4 | 62.5 | 66.7 |
| teacher sign correct on the same steps | 47.7 | 48.2 | 53.8 |
- Direction agreement. On steps with significant oracle credit: outcome sign (GRPO/RLSD) 64.3%, teacher sign (OPSD) 49.6%, DCSD 69.0%.
- Correction precision. Where DCSD reverses the outcome direction, the continuation estimate agrees with DCSD in 61.5% of cases (58.8% for positive and 66.7% for negative corrections). The teacher's sign is right on only 53.8% of the same steps. Combined with the identity in W1, DCSD makes 13% fewer direction errors than the outcome sign on these steps.
- Magnitude. Within-response Spearman correlation between and : DCSD 0.065, teacher 0.032, step length 0.021, shuffled weights −0.003. The absolute values are small because single-step oracle estimates are noisy (the standard error of is comparable to its typical magnitude of about 0.1), but the ordering matches the ablation above.
Response(1/4)
Thank you for recognising that decoupling credit direction from credit magnitude addresses weaknesses in direct self-distillation, and for highlighting the explicit limit on teacher influence and the fallback to outcome-based credit.
To address your three concerns, we (1) clarify where the theory relies on modelling assumptions and restate the two conditional results under weaker, directly measurable conditions that connect the theory to the implementation, (2) use continuation sampling to check whether these conditions hold and directly assess the accuracy of the assigned local credit, and (3) add controlled ablations, including a baseline that uniformly shrinks credit, and report controlled comparisons across multiple random seeds.
W1. Connecting the conditional theory to the implementation
We first clarify which parts of Theorems 2 and 3 involve modeling assumptions. Both results need only conditions on quantities defined by the student's own policy, and these conditions can be measured directly. The oracle Ak⋆ is defined on the student's continuation policy, so continuations sampled from a boundary state with the training decoding give unbiased estimates of Vπ, and no surrogate model is needed.
Continuation study. We estimate the oracle of Eq. (1) as A^k⋆=V^(Sk+1)−V^(Sk), where V^(Sk) is the success rate of continuations sampled from Sk with the training decoding (temperature 1.0, 16k tokens). We used the initial Qwen3-4B. From 480 DAPO problems (disjoint from the 5,120 training prompts of the ablations in W2), each sampled 8 times, we kept the 100 problems with 2–6 correct rollouts and took one correct and one incorrect response per problem (200 responses). Steps follow DCSD's own boundaries. We evaluated all 461 probe-selected steps (32 continuations per boundary; 460 used for sign agreement) together with randomly sampled steps (16 continuations per boundary): 1,661 steps, 2,862 boundaries and 58,880 continuations in total. We call a step's oracle credit significant when ∣A^k⋆∣ is at least 1.96 standard errors.
(a) Direction (Theorem 2). The proof of Theorem 2 uses only the realized margin error; Ξk is an analytical upper bound on it, built from readout calibration and candidate coverage. In the revision we state the result directly in terms of this error, with a common readout temperature γ0 (Theorem 2′):
The original Theorem 2 is the case γ=1 with ∣ek∣≤Ξk. The temperature accounts for a teacher-forced readout being uniformly sharper or flatter than free continuation, which rescales margins without changing their sign.
Theorem 2′ also implies an assumption-free statement. Restrict to steps with Ak⋆=0, let Uc be the steps whose direction DCSD corrects (σk=sign(Aτ)), and let p be the fraction of corrections that agree with sign(Ak⋆). Then
∣EDCSD∣−∣Eout∣=∣Uc∣,(1−2p).
So DCSD makes fewer direction errors than any outcome-anchored RL method (GRPO, RLSD) whenever more than half of its corrections are right, with no modeling assumption.
Table R1.
The probe error budget is therefore directly measurable. With γ=0.773 (the readout is slightly flatter than free continuation), the condition of Theorem 2′ holds on half of the probe-selected steps, and on those steps the probe direction agrees with the continuation estimate in 99.13% of cases, as the theorem predicts. Correction precision also rises with the reliability of the oracle estimate (53.0% → 56.5% → 61.5%; Table R4 in W2), and by the identity above, DCSD makes fewer direction errors than the outcome sign (46 → 40 on significant steps).