- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies skill-conditioned on-policy self-distillation (building on SGSD), where retrieved natural-language "skills" serve as privileged context for the teacher. It observes that only a small fraction of skill retrievals pass SGSD's confidence gate, and proposes SGUID, which runs one full-bank distillation, counts each skill's verifier-aligned positive signals in early/late phases, filters skills by threshold rules (Eq. 11), and re-distills from the same initialization using only the top-K (K=6, then K=3) skills. Repeating this curate–select–distill loop is termed "model–skill co-evolution". Experiments use Olmo-3-7B and Qwen3-1.7B/4B/8B trained on DAPO-Math-17K with LoRA, evaluated on AIME24/25 and HMMT25 (avg@12).
Critical strengths
The observation that most skill retrievals provide no active distillation signal (Fig. 2) is a useful diagnostic for skill-conditioned distillation, and selecting a small bank (6 vs. 30–71 skills) can match full-bank SGSD at lower teacher cost.
Critical weaknesses
-
The empirical evidence does not reliably support the claims. (a) Checkpoints are selected by average performance on the test benchmarks themselves (Table 1 caption: "best checkpoint according to the average performance on AIME24, AIME25, and HMMT25 across 200 training steps"); there is no validation set. (b) All results are single runs without variance, on benchmarks of 30 problems each. (c) Under these conditions, the gains are within noise: R1 vs. SGSD differs by −0.7 to +0.9 mean points, and on Qwen3-8B SGUID-R1 (64.3) is below answer-only OPSD (65.3). (d) The headline 8B improvement appears only in Round 2, which uses roughly 4× the training budget of the 200-step baselines (a selection run plus a distillation run per round). Compute-matched baselines (OPSD/GRPO/SGSD trained longer) are not reported, and the "w/o selection" co-evolution comparison (Fig. 3) is given only for Qwen3-4B. (e) In Table 4, top-6 (42.3) is close to random-6 (41.2±0.9) and worst-6 (41.5), and all three are below full-bank SGSD (43.0). This weakens the claim that the selection criterion identifies the right skills.
-
Limited novelty. SGUID is a threshold-based filtering heuristic (with hand-set h, τ and early/late split) applied on top of SGSD's existing Verify·Gate coefficient, followed by a restart. "Co-evolution" amounts to iterating this pipeline twice; skills are regenerated rather than evolved, and there is no analysis beyond two rounds. The "skills" themselves (Appendix B) are generic math hints such as "Simplify Expressions" and "Validate Solutions", so the framing (skill bank, internalization, co-evolution) overstates what is essentially selecting prompt hints for a privileged teacher.
Questions for authors
Clarifications:
- Can you report results with checkpoints selected on a held-out validation set, and with multiple seeds / confidence intervals?
- What are the results of OPSD, GRPO and SGSD trained with the same total number of steps/compute as SGUID-R2, and of full-bank (no selection) R2 on all four models?
- Table 4 states that the selection baselines were trained for 100 steps, yet the top-6 row matches the 200-step R1 result in Table 1. Were the variants trained with the same budget?
- The abstract says "fewer than 25% of retrieved skills provide useful signals", but Fig. 2 counts retrieval events passing the gate, not distinct skills. How many distinct skills are ever active, and how sensitive is this fraction to κ? Would materially affect my assessment:
- The paper's own data suggest that skill benefits shrink with scale (6.4% active retrievals on 8B; SGSD below OPSD on 8B). Does SGUID still help on larger models (e.g., 14B/32B) or with full fine-tuning rather than LoRA?
- Section 2.3 states that skill-guided teachers provide better signals than answer-guided ones, but Table 1 shows OPSD ≥ SGSD on Qwen3-4B/8B. How do you reconcile this?
6 comments
Author Rebuttal (Part 2/5)
1(b, c). Compute-matched comparisons with variances reported
All results are single runs without variance ... the gains are within noise.
We retrained GRPO, OPSD, and SGSD for 400 steps over three seeds, matching SGUID R1's budget (200 selection + 200 distillation steps):
| Model | Method | AIME24 | AIME25 | HMMT25 | Mean |
|---|---|---|---|---|---|
| Qwen3-1.7B | GRPO | ||||
| Qwen3-1.7B | OPSD | ||||
| Qwen3-1.7B | SGSD | ||||
| Qwen3-1.7B | SGUID R1 | ||||
| Qwen3-1.7B | SGUID R2 | ||||
| Qwen3-8B | GRPO | ||||
| Qwen3-8B | OPSD | ||||
| Qwen3-8B | SGSD | ||||
| Qwen3-8B | SGUID R1 | ||||
| Qwen3-8B | SGUID R2 |
n = 3 seeds; ± is the standard deviation. Bold / underline: best / second-best. Stars: SGUID R2 vs. SGUID R1 within the same model, one-sided paired t-test (each R2 run continues from the matching R1 run; df = 2; , , ).
- We find the conclusions are consistent with Table 1. R1 matches SGSD with a compact bank (42.59 vs. 42.41 on Qwen3-1.7B; 64.14 vs. 64.35 on Qwen3-8B) while distilling 6 instead of 34 and 71 skills.
- R2 gains are significant: R2 improves over R1 (p = 0.0018 and 0.017) and over SGSD (Welch p = 0.018 and 0.012) on Qwen3-1.7B and Qwen3-8B.
We will add the revised paper results for all four models.
1(d). Round-2 budget and compute-matched baselines (Q2)
The headline 8B improvement appears only in Round 2, which uses roughly 4× the training budget of the 200-step baselines.
R1 + R2 totals 800 steps, four times the 200-step baselines in Table 1. However:
- R1 is step-matched (400 steps, table above) and matches SGSD; we do not present R2 as a compute-matched comparison.
- The baselines have plateaued: going from 200 steps (Table 1) to 400 steps changes their means by at most 0.6 points (e.g., Qwen3-8B OPSD 65.3 → 65.28).
- R1 + R2 costs about the same as 800-step SGSD in GPU-hours (22.96 vs. ≈ 23.2 on Qwen3-4B), since compact-bank distillation needs fewer teacher passes.
We will add 800-step baselines on all four models and the GPU-hour breakdown (Appendix A) in the revision. The requested full-bank R2 is in Part 3.
Author Rebuttal (Part 3/5)
Co-evolution ablations on Qwen3-8B (1d, Q2)
The "w/o selection" co-evolution comparison (Fig. 3) is given only for Qwen3-4B.
We ran it on Qwen3-8B over three seeds, plus an ablation that continues to reuse the R1 bank for another round:
| Qwen3-8B | Round | AIME24 | AIME25 | HMMT25 | Mean |
|---|---|---|---|---|---|
| Base (Table 1) | – | ||||
| w/o selection (full bank) | R1 | ||||
| w/o selection (full bank) | R2 | ||||
| w/ R1 bank | R2 | ||||
| w/ selection (SGUID) | R1 | ||||
| w/ selection (SGUID) | R2 |
Stars: SGUID R2 vs. SGUID R1 (same test as above).
All second rounds use the same budget. Consistent with our claims in the paper, re-curating and re-selecting skills is crucial: full-bank R2 (64.57) and R1-bank R2 (64.32) give no gain, while SGUID R2 beats both (Welch p = 0.029 and 0.014), so the 8B gain is not explained by extra budget. We will add these results to Figure 3.
1(e). Clarifications on Table 4, and evidence that the criterion finds the right skills (Q3)
Top-6 (42.3) is close to random-6 (41.2 ± 0.9) and worst-6 (41.5), and all three are below full-bank SGSD (43.0). ... Were the variants trained with the same budget?
Budget. All K = 6 variants are trained for 100 steps; top-6 equals Table 1's R1 because that run peaks at step 100. The SGSD reference of 43.0 is its best over 200 steps to show an upper bound. Within the same 100 steps, SGSD's performance is 41.57:
| Qwen3-1.7B, 100 steps | AIME24 | AIME25 | HMMT25 | Mean |
|---|---|---|---|---|
| Base | ||||
| SGSD (full bank, 34 skills) | ||||
| top-6 (ours) | ||||
| random-6 (3 selection seeds) | ||||
| worst-6 |
Bold: best among the three K = 6 variants.
Criterion. As the margins on Qwen3-1.7B are small, we additionally compare Qwen3-4B, which further supports the effectiveness of our skill selection. The reviewer might be interested to see additional frequency-6 (the most-retrieved skills) and fast-decay-6 (the fastest-decaying skills) baselines we added. All rows use 100 steps:
| Qwen3-4B, 100 steps | AIME24 | AIME25 | HMMT25 | Mean |
|---|---|---|---|---|
| Base | ||||
| SGSD (full bank, 30 skills) | ||||
| top-6 (ours) | ||||
| random-6 (3 selection seeds) | ||||
| frequency-6 | ||||
| worst-6 | ||||
| fast-decay-6 |
Top-6 is the only 6-skill bank that outperforms the full bank (64.2 vs. 63.52); all alternatives are 1.3–2.0 points lower, and frequency-6 is no better than worst-6. With the persistence ablation in Table 3 (65.1 → 62.3), this supports Reviewer MgrQ's observation that these controls "show that the choice of skills matters, not just the number of skills." We will add this table to Section 5 and the 100-step SGSD row to Table 4.
Author Rebuttal (Part 1/5)
We are delighted that the reviewer finds the absence of active distillation signal in most skill retrievals is a useful diagnostic for skill-conditioned distillation, and that a small bank of 6 instead of 30–71 skills can match full-bank SGSD at lower teacher cost. We also value the reviewer's specific concerns about the reliability of our empirical evidence and the scope of our claims, which led to the following new results: final-checkpoint results with no checkpoint selection, compute-matched comparisons with significance tests, co-evolution ablations on Qwen3-8B, additional skill-selection ablations, and sensitive analysis over threshold κ. We hope the evidence below, including our answer to Q4 in Part 4, resolves the reviewer's main concerns.
1(a). Checkpoints selected on the test benchmarks (Q1)
For comparability, we follow the evaluation protocol of [1] and [2]: checkpoints every 25 steps (step 0 excluded), selected by the mean over AIME24, AIME25, and HMMT25, with the same rule for all methods. Baselines are selected over 400 steps and each SGUID stage over 200, so SGUID gets no extra selection opportunities.
We additionally report the final checkpoint after 400 steps (mean avg@12, three seeds):
SGUID R1 has the best mean on all three models, including over OPSD on Qwen3-8B, and leads SGSD by +1.36, +0.83, and +0.65. We will add this table in the revised paper.
References
[1] Zhao, Siyan, et al. "Self-distilled reasoner: On-policy self-distillation for large language models." arXiv preprint arXiv:2601.18734 (2026).
[2] Huang, Jiazhen, et al. "Skill-Conditioned Gated Self-Distillation for LLM Reasoning." arXiv preprint arXiv:2605.28791 (2026).