acceptodds
Under review as a conference paper at ICLR 2027

Continual Learning with Success-Conditioned Self-Distillation

Abstract

Reinforcement learning with binary verifiable rewards provides only one bit of feedback per rollout, making credit assignment across thousands of tokens difficult. We argue that the success-conditioned policy provides a natural dense learning target for both single-task and continual learning by restricting the reference model to its own successful responses, which is the least disruptive way to succeed. Computing this target, however, requires a token-level value function that many verifiable-reward methods avoid learning. Prompting a frozen model with one of its own verified successes offers a full-vocabulary surrogate without an explicit critic. Yet this surrogate conditions on a particular successful trajectory rather than on success itself, introducing trajectory-specific nuisance information. We propose Success-Conditioned Self-Distillation (SCSD), which addresses this mismatch by searching for a textual transformation of a successful rollout that brings the prompted policy closer to the success-conditioned target. The resulting teacher is then distilled into a standalone student policy through on-policy distillation on the student's own rollouts. Across multiple tasks, SCSD achieves state-of-the-art performance in both single-task and continual learning settings.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual Learning, self-distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zZxhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Success-Conditioned Self-Distillation (SCSD) for continual learning. It searches for rewriting rules that turn successful model-generated solutions into teacher prompts, aiming to preserve teacher success while limiting drift from a reference model. The resulting teacher provides token-level supervision on student trajectories. The approach is motivated by a characterization of the minimum-KL successful policy and evaluated across mathematics, science, and tool use.

Critical strengths
  1. The paper highlights how successful trajectories can contain incidental details that induce unnecessary behavioral changes. Optimizing the information provided to the teacher is a meaningful extension to existing self-distillation methods.
  2. The characterization of the success-conditioned distribution provides a clear rationale for balancing task success with limited distributional drift.
  3. SCSD shows a good empirical results, which substantially mitigates the mathematical degradation observed with the evaluated SDPO baseline and improves several final-task scores in sequential learning.
Critical weaknesses
  1. Comparisons against fixed compression, length-matched summaries, and search without the KL objective would help establish whether the gains come from simpler teacher context or the proposed search procedure.
  2. It would be better to conduct one more experiments on continual learning.
Questions for authors
  1. How much of the improvement comes from the proposed optimization beyond simply compressing the teacher context? Could the authors compare against the fixed initial rewriting instruction and search without the KL objective, while keeping other settings, including teacher feedback, identical?
1 comment

We thank the reviewer for recognizing that optimizing the information given to the teacher is a meaningful extension of self-distillation, and that the drift floor gives a clear rationale for balancing success against drift. We address the concerns below.

W1 & Q1. How much of the improvement comes from the search beyond compressing the teacher's context? This is exactly the right question, and we answer it end to end in G1, where all runs share the teacher, the wrapper and every other setting. More specifically, we compare SCSD against (i) the fixed seed instruction g0g_0 without search, (ii) a length-matched generic summary, (iii) the raw success, and (iv) a search with the reward-only objective. Compression alone does not help: g0g_0 and the summary stay at the accuracy and drift of the raw teacher on math. The reward-only search raises accuracy but has the highest drift of all arms. SCSD improves over g0g_0 by +0.168 / +0.221 / +0.059 on math / science / tool use. G1 shows that the gain carries over to the trained student, and that it comes from the minimum-drift objective rather than from a shorter context.

W2. One more continual-learning experiment. We add three seeds, two task orders, a second model (Qwen3-4B) and three continual-learning comparators (G2). Across the three orders, SCSD forgets less than SDPO on 8 of the 9 (order, task) pairs, and on Qwen3-4B it has the largest on-task gain at every stage and ends above both GRPO and SDPO on every task.

We hope these new results address the reviewer's concerns. If so, we would greatly appreciate it if the reviewer could consider raising the score. We are happy to answer any further questions.

Reviewer_XGR6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Success-Conditioned Self-Distillation (SCSD) to reduce policy drift and forgetting in continual LLM post-training. The key idea is that conditioning a teacher on a full successful trajectory introduces unnecessary information and causes excessive drift. SCSD instead searches for a more compact representation of the successful trajectory that preserves its success rate while minimizing KL divergence from the reference policy. Experiments on math, science, and tool-use tasks show improvements over standard self-distillation (SDPO), particularly in reducing policy drift and mitigating forgetting.

Critical strengths
  1. Interesting motivation and formulation. The paper provides a useful perspective on why self-distillation may cause excessive policy drift. Optimizing the teacher's conditioning context rather than directly regularizing the student is an interesting idea. The theoretical analysis also provides a reasonable motivation for the proposed objective.

  2. Promising empirical results. SCSD substantially reduces KL divergence compared with SDPO and avoids its performance collapse on math. The continual-learning results also show a better balance between learning and forgetting compared with SDPO.

Critical weaknesses
  1. Insufficient ablations to isolate the proposed objective. It is unclear how much of the improvement comes from minimum-drift search rather than simply compressing successful trajectories. While Section 6.1 shows improvements in teacher-level metrics, an end-to-end comparison with the initial rewriting prompt without search is missing. Also, SDPO includes environment feedback in its teacher context while SCSD does not, introducing another difference beyond the proposed transformation. Finally, SCSD takes 2.5–5× longer than SDPO, but the experiments only compare methods at matched gradient updates.

  2. Limited evidence for continual learning. The evaluation uses only one model and a single three-task sequence, without results across multiple training seeds or task orders. SCSD also still suffers considerable forgetting: math accuracy drops from 0.546 to 0.346 after subsequent tasks, below the base model's 0.521. The GPQA results further suggest that reducing policy drift does not necessarily preserve generalization. These limitations make it difficult to assess how robust the reported continual-learning benefits are.

Questions for authors
  1. Could the authors provide an end-to-end ablation comparing SCSD with the initial rewriting prompt without search, as well as a reward-only search objective? It would also be helpful to control for the difference in environment feedback between SDPO and SCSD.

  2. How robust are the continual-learning results across different seeds and task orders? Could the authors also provide compute-matched comparisons, given the additional cost of searching for the teacher context?

2 comments

We thank the reviewer for finding the motivation and formulation interesting, and for recognizing that SCSD substantially reduces drift and avoids SDPO's collapse on math. We address the concerns below.

W1 & Q1. End-to-end ablation of the initial rewriting prompt without search, a reward-only objective, and environment feedback. Please see G1, which trains the student with the fixed initial instruction g0g_0, with the reward-only search, and with the raw success, all without environment feedback and with an identical wrapper. The fixed g0g_0 does not improve over the raw success on math and loses on science; the reward-only search improves accuracy but has the highest drift of all; SCSD is best on every task. The reviewer is right that Table 5 gives SDPO's teacher environment feedback; G1 removes it. As G5 shows, removing the feedback lowers SDPO's result (0.406 → 0.368), so the paper compares SCSD against a stronger SDPO variant.

W1 & Q2. SCSD is 2.5–5× slower, but methods are compared at matched updates. Please see the compute-matched comparison in G3: at SCSD's wall-clock budget, GRPO reaches 0.523 on AIME24 and SDPO 0.496 with its tool use collapsing to 0.222, both below SCSD (0.556).

W2 & Q2. Robustness across seeds and task orders. Please see G2. With three seeds, the paper-order results hold and SCSD has the largest on-task gain at every stage. Across the three orders, SCSD forgets less than SDPO on 8 of the 9 (order, task) pairs and has a positive on-task gain in all 9 stages, against 4 of 9 for SDPO.

W2. SCSD still forgets math (0.559 → 0.343 in the revised Table 2, below the base model). We agree that SCSD does not eliminate forgetting. SCSD forgets less math than SDPO in all three orders (−0.171 vs. −0.293, +0.023 vs. −0.267, −0.056 vs. −0.391), and on Qwen3-4B it ends +0.165 above the base model on math. Regularized baselines do forget somewhat less math (SDPO + KL to the previous stage: −0.129 / −0.133; GRPO with learning rate 3×10−63\times10^{-6}: −0.146), but they buy this retention by learning less: their total on-task gain over the three stages is at most 0.147, against 0.329 for SCSD, and both KL variants end below the base model on tool use. Since SCSD only changes the teacher's context and adds no replay, regularizer or parameters, it is orthogonal to these remedies and can be combined with them.

W2. GPQA suggests that reducing drift does not necessarily preserve generalization. On GPQA, SCSD is above SDPO on mean@8 (0.426 vs. 0.415) and below it on pass@8 (0.609 vs. 0.665), and the GPQA loss grows with the science drift across all three methods. We agree with the reviewer's reading: our claim is that SCSD reduces how far the policy moves for a given in-domain gain, not that lower drift by itself guarantees out-of-domain generalization, and we have revised the text of Appendix B.1 to state this.

We hope these new results address the reviewer's concerns. If so, we would greatly appreciate it if the reviewer could consider raising the score. We are happy to answer any further questions.

We sincerely hope the reviewer could consider raising the score if no major concerns are left, that means a lot to us. We hope you all the best!

Reviewer_HtW5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how the context given to an on-policy self-distillation teacher affects learning and forgetting. It argues that conditioning the teacher on one successful response y⋆y^\star (as in SDPO) carries trajectory-specific detail and moves the teacher further from πref\pi_{\text{ref}} than success requires; Props. 1–2 formalize this with a drift floor dKL(u ∥ px)d_{\mathrm{KL}}(u \,\|\, p_x) attained by the success-conditioned policy πS\pi_S. SCSD searches in text space for a rewriting instruction gg such that the teacher πref(⋅∣s,g(y⋆))\pi_{\text{ref}}(\cdot \mid s, g(y^\star)) keeps the raw teacher's pass rate while minimizing its KL to πref\pi_{\text{ref}}, then distills this frozen teacher on student rollouts. Experiments on Qwen3-8B (math, science, tool use; single-task and a three-stage sequence) compare SCSD with GRPO and SDPO.

Critical strengths
  1. A clear formulation. The teacher's context becomes the optimization variable, with "least drift subject to success" as the objective and the drift floor as its scale; on top of SDPO only the context changes. Under the reported protocol the intended trade-off appears consistently (lower divergence than SDPO everywhere, on-task gains at least SDPO's).
Critical weaknesses
  1. The headline claims are broader than the evidence supports. The abstract claims state-of-the-art performance in both single-task and continual settings, but:

    • The comparison covers two baselines implemented by the authors, one model, one task order, and no dedicated continual-learning baseline.
    • The evaluation sets are small (30 / 50 / 68 prompts): the science gap to SDPO has CI [−0.05, +0.25], Table 2 reports no intervals, and the base model's AIME24 score differs between Tables 1 and 2 (0.579 vs. 0.521) under the same budget (Table 6, App. C.5).
    • Some results point the other way: SDPO is strongest on Math500 (Table 3), SCSD loses more GPQA pass@8 than SDPO (0.731→0.609 vs. →0.665), and in Table 2 SCSD is behind GRPO on math retention and no better than SDPO on science retention.

    The evidence supports a favorable trade-off under the reported protocol; the broader claims would need larger evaluation sets, intervals for Table 2, and a continual-learning comparator.

  2. The link from teacher context to reduced drift to reduced forgetting is not yet isolated.

    • GRPO uses the SDPO reference settings (LR 1e-6, minibatch 8; Table 5), apparently selected under a different budget; at 50 updates it barely moves (0.001–0.003 nats/token from the base), so the comparison says little about RL at this budget.
    • Table 5 gives SDPO's teacher environment feedback but not SCSD's, so the arms also differ in which prompts contribute and what the teacher sees; the reduction to SDPO at g = id (L181–182) holds only approximately.
    • The search is the only drift-reducing alternative compared; the seed instruction alone, an answer-only context, or SDPO with a lower learning rate or KL penalty would show what the search contributes. Search gains are shown for the teacher, not for the trained student.
    • In Table 2, drift and forgetting do not move together: in the science stage SDPO drifts more yet loses less math than SCSD.
Questions for authors

Clarification requests:

  1. Were the GRPO settings carried over from the SDPO setup? What does "50 updates" count for each method, and how many optimizer steps and prompts did each method consume?
  2. What are the success rate and drift (Eq. 6) of the identity transformation on the search set, and what are typical response lengths, so that the response-level drifts (e.g., 736 → 540) can be related to the per-token KLs in Table 1? Could the excess drift Δ of Eq. 3 be reported for the teachers?
  3. Which models serve as the rewriter and the reflector? Do the results hold when Qwen3-8B plays both roles, or with an answer-masked g?
  4. Does the SDPO–SCSD gap persist when environment feedback is removed from SDPO's teacher, and when SDPO's drift is matched (lower learning rate or KL penalty)?

Suggestions:

  1. Report CIs for Table 2; evaluate math on Math500 (or AIME24+25) with more rollouts per prompt; report truncation / no-answer rates under the 8k and 14k budgets; and measure each stage checkpoint's KL on all three tasks' prompts.
  2. Relate Props. 1–2 to prior characterizations of success conditioning (e.g., Korbak et al., 2022; Russo, 2026) and to rejection-sampling fine-tuning; clarify the relation to Tiwari et al. (FST); soften the abstract; and list in App. C.7 the three implementation deviations mentioned at L214–215.
6 comments
AuthorsOpenReview ↗

Response to Reviewer HtW5 (1/2)

We thank the reviewer for the very detailed review and for summarizing the formulation precisely: the teacher's context is the optimization variable, "least drift subject to success" is the objective, and the drift floor is its scale. The suggestions have substantially improved the paper.

W1. The headline claims are broader than the evidence. We have revised the claims to match the evidence. (1) Baselines, models, orders, continual comparators: see G2 and G3. (2) Small evaluation sets and intervals: all new results use 32 rollouts per prompt and report mean ± standard deviation over three seeds (G4). (3) Base AIME24 in Tables 1 and 2: the two numbers came from two independent evaluations; under the shared evaluation the base model scores 0.514 ± 0.008, and SCSD lies +0.042 above it (G4). (4) Results that point the other way: we slightly revised the abstract to remove this confusion.

W2(a) & Q1. GRPO settings and what "50 updates" counts. G3 adds GRPO at SCSD's wall-clock budget and with re-tuned learning rates: compute-matched GRPO still falls behind SCSD, and larger learning rates do not help (collapse at 1×10−51\times10^{-5}).

W2(b) & Q4. Environment feedback; g=idg=\mathrm{id} reduces to SDPO only approximately. See G1 and G5. SDPO's teacher differs from SCSD's at g=idg=\mathrm{id} in the environment feedback and the prompt wrapper; G1's raw-y⋆y^\star run removes both, and is therefore the exact g=idg=\mathrm{id} special case of SCSD. This variant scores 0.368, below SDPO with feedback (0.406), and SCSD improves over it on math / science / tool use. We will state in L181–182 that the reduction holds for this variant.

W2(c) & Q3. Other drift-reducing alternatives; search gains shown only for the teacher. G1 adds the seed instruction alone and a length-matched summary, and G3 adds SDPO with a lower learning rate and with a KL penalty, all evaluated on the trained student. None of them reproduces SCSD: the fixed contexts keep the drift of the raw teacher and its math collapse, the lower learning rate foesn't help, and the KL penalty recovers part of the collapse but stays below SCSD.

W2(d). In the science stage, SDPO drifts more yet loses less math than SCSD. Δstage\Delta_{\text{stage}} is measured on the probe of the stage's own domain, so it measures how far the policy moves on science prompts, not how much math it retains. SCSD does not lose less math per stage than SDPO: over three seeds on Qwen3-8B, the science stage costs SCSD −0.225 and SDPO −0.203 on math. What differs is the level from which the science stage starts. SCSD does not collapse the task it trains on: in the math stage it reaches 0.559 where SDPO falls to 0.417, below the base model, and its on-task gain is positive in all 9 stages across the three orders, against 4 of 9 for SDPO. As a result, SCSD stays above SDPO on math after the science stage (0.334 vs. 0.214) and ends every task order above SDPO on math (e.g., 0.343 vs. 0.221 in the paper order); on Qwen3-4B, the science stage costs SCSD almost no math (0.625 → 0.624, against 0.434 → 0.261 for SDPO). We have revised Section 5.2 accordingly: it no longer uses Δstage\Delta_{\text{stage}} to explain retention.

AuthorsOpenReview ↗

Response to Reviewer HtW5 (2/2)

Q2&Q3. Success and drift of the identity transformation; response lengths; excess drift. At the teacher level, Section 6.1 and Appendix B.2 report the success and response-level drift along each search: on math, drift falls from 736 to 540 at a success of 0.97–0.98; on science, from 845 to 228 at a success of 0.99; on tool use, from 210 to 104 while the success rises from 0.59 to 0.77; and in the continual science stage, g0g_0 reaches a success of 0.73 at a drift of 411, against 0.82–0.89 at 203–349 for the accepted instructions. At the student level, G1 reports the per-token KL from the base model for the raw (g=idg=\mathrm{id}) and seed (g0g_0) teachers: 0.127 / 0.144 on math, 0.282 / 0.271 on science, and 0.346 / 0.324 on tool use. On answer exposure: the raw-success and g0g_0 teachers in G1 both carry the final answer (g0g_0 asks for it explicitly, Figure 4), and SCSD improves over them by +0.188 and +0.168 on math, so access to the answer alone does not explain the gain of SCSD.

Q5. Suggestions on evaluation. See G4.

Q6. Related work and presentation. (1) Korbak et al. (2022a,b) read KL-regularized fine-tuning as distribution matching toward the pretrained model restricted to a constraint, and fit it with sequence-level samples; SCSD targets the same kind of object, πS\pi_S, but approximates it with a prompted teacher instead of a sequence-level objective. (2) Rejection-sampling fine-tuning trains on samples from πS\pi_S at the sequence level, whereas SCSD uses a prompted teacher to supply token-level, full-vocabulary targets toward πS\pi_S on the student's own prefixes. (3) Tiwari et al. (FST) also optimize text with reflective search, but keep it as fast weights at inference and optimize it for task reward; SCSD optimizes it for minimum drift subject to success and uses it only inside the teacher. (4) The abstract is softened (W1), and Appendix C.7 lists the three implementation deviations.

We thank the reviewer again for the insightful questions, which have improved both the framing and the analysis of the paper. We hope our response addresses the concerns, and we would greatly appreciate it if the reviewer could consider raising the score.

Thank you for the thorough rebuttal and the substantial new experiments. They resolve several of my concerns.

  • W2(c), contribution of the search. G1 shows at the student level that shortening the context alone does not help, and that the minimum-drift objective adds further gains over a reward-only search. I consider this addressed at the single-task level.
  • W2(b) / Q4, environment feedback. The controlled raw-y* run and the corrected L181–182 settle this for the single-task comparison.
  • W1, scope and evaluation. The additional seeds, task orders, second model, unified protocol, and softened abstract address most of this point.

My remaining concern is that the link between reduced drift and reduced forgetting (W2(d)) is still not established. This is the central claim of a paper framed around continual learning. Three observations lead me to this view.

  1. Retention. On Qwen3-8B, the first-trained task drops by at least as much under SCSD as under SDPO in all three orders (Δ − Gain 1: −0.216 vs −0.196, −0.051 vs −0.019, −0.174 vs −0.114). SCSD ends higher because it gains more and avoids collapse, not because it retains more, yet Section 5.2 still concludes that the search improves stability.
  2. No drift in the continual setting. Section 5.2, Table 2, and Figure 3 report accuracy only. The cross-task KL of the stage checkpoints (suggestion 5) and the excess drift of Eq. 3 (Q2) are still missing.
  3. GRPO. SCSD learns more per stage and ends higher on Qwen3-4B, but on Qwen3-8B default GRPO ends at or above SCSD on 7 of the 9 (order, task) pairs, and it is absent from the table behind the "only positive net change" statement.

Two clarifications I am still missing: which models act as the rewriter and the reflector (Q3), and the default learning rates of SDPO and GRPO. If I have overlooked them, please point me to where they are stated.

For these reasons I keep my current score. Aligning the continual-learning claim in Section 5.2 with this evidence would address much of my concern. I understand that the remaining time is short, and I will read any further response.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We thank all reviewers for their constructive feedback. We are glad that the reviewers find the formulation "clear" with the drift floor as "a natural scale" (HtW5), the idea of optimizing the teacher's context rather than regularizing the student "interesting" (XGR6) and "a meaningful extension of self-distillation" (zZxh), and the removal of SDPO's math collapse "promising" (zZxh, XGR6). The shared concerns are about what the search contributes beyond a shorter context, the robustness of the continual-learning results, and the strength of the baselines. We address them below with new experiments, and we have revised the paper accordingly (changes marked in blue).

G1. What the search contributes beyond a shorter context (zZxh, XGR6, HtW5). We train the student end to end with the five teachers in the table below, which differ only in the content of g(y⋆)g(y^\star). All runs share the wrapper, the student settings, the budgets and the frozen teacher, and none receives environment feedback. Three results stand out.

  1. A shorter context alone does not help. The fixed g0g_0 and the length-matched summary stay close to the raw y⋆y^\star. The fixed g0g_0 even makes science results worse.
  2. Searching for reward alone raises accuracy but not proximity. The reward-only search improves over every fixed context, yet it has the highest drift of all runs on every task, which is exactly the over-specification predicted in Section 3.3.
  3. The minimum-drift objective is what makes the difference. SCSD gets much better results over the reward-only search and g0g_0.

This shows that the gain of SCSD is not a by-product of compressing the context: it comes from the objective of the search.

Teacher context (mean@32)MathScienceTool useKL from base (M / S / T)
Raw y⋆y^\star (g=idg=\mathrm{id})0.3680.3920.5560.127 / 0.282 / 0.346
Length-matched summary of y⋆y^\star0.3780.4050.5430.139 / 0.252 / 0.318
Seed instruction g0g_0 (no search)0.3880.2750.5530.144 / 0.271 / 0.324
Searched gg, reward only (max⁡Jsucc\max J_{\mathrm{succ}})0.4580.4400.5760.151 / 0.289 / 0.365
SCSD (searched gg, min-drift, Eq. 7)0.5560.4960.6120.039 / 0.111 / 0.268

G2. Robustness of the continual-learning results (zZxh, XGR6, HtW5). We extend the continual setting along the four axes requested: (1) three seeds of the original sequence; (2) two additional task orders; (3) a second model, Qwen3-4B; and (4) three continual-learning comparators (SDPO + replay, SDPO + KL to the previous stage, and re-tuned GRPO).

Three seeds (M → S → T; M = math, S = science, T = tool use), in the format of Table 2 (mean@32 ± std over 3 seeds). With three seeds, SCSD has the largest on-task gain at every stage.

MethodStageMathScienceTool use
Base model—0.514 ± 0.0080.279 ± 0.0120.561 ± 0.017
GRPO1 (math)0.534 ± 0.0060.283 ± 0.0080.558 ± 0.011
2 (science)0.489 ± 0.0190.323 ± 0.0180.569 ± 0.010
3 (tool use)0.540 ± 0.0050.319 ± 0.0060.582 ± 0.009
SDPO1 (math)0.417 ± 0.0130.354 ± 0.0240.152 ± 0.020
2 (science)0.214 ± 0.0230.438 ± 0.0110.372 ± 0.013
3 (tool use)0.221 ± 0.0130.450 ± 0.0130.478 ± 0.014
SCSD1 (math)0.559 ± 0.0160.349 ± 0.0070.569 ± 0.014
2 (science)0.334 ± 0.0120.481 ± 0.0110.468 ± 0.012
3 (tool use)0.343 ± 0.0080.459 ± 0.0080.620 ± 0.014

Task orders. See the updated Figure 3: Gain kk = on-task gain of stage kk (plasticity); Δ = change of each task vs. the base model after all stages (stability). Means over 3 seeds, mean@32. Across the three orders, SCSD forgets less than SDPO on 8 of the 9 (order, task) pairs and has a positive on-task gain in all 9 stages, against 4 of 9 for SDPO.

OrderMethodGain 1Gain 2Gain 3Δ MathΔ SciΔ Tool
M → S → T (paper)GRPO+0.020+0.040+0.013+0.026+0.040+0.021
SDPO−0.097+0.084+0.106−0.293+0.171−0.083
SCSD+0.045+0.132+0.152−0.171+0.180+0.059
T → S → MGRPO+0.007+0.126+0.057+0.034+0.111−0.002
SDPO−0.025+0.198−0.175−0.267+0.184−0.044
SCSD+0.036+0.231+0.133+0.023+0.062−0.015
S → M → TGRPO+0.081+0.042+0.009+0.044+0.089+0.005
SDPO+0.053−0.079−0.105−0.391−0.061−0.386
SCSD+0.137+0.045+0.052−0.056−0.037−0.084
AuthorsOpenReview ↗

General Response to All Reviewers (2/2)

G2 (continued). A second model (Qwen3-4B, M → S → T), in the format of Table 2 (mean@32 ± std over 3 seeds). On Qwen3-4B, SCSD has the largest on-task gain at every stage and ends above both GRPO and SDPO on every task.

MethodStageMathScienceTool use
Base model—0.439 ± 0.0120.207 ± 0.0100.480 ± 0.008
GRPO1 (math)0.560 ± 0.0070.320 ± 0.0110.576 ± 0.010
2 (science)0.573 ± 0.0090.426 ± 0.0090.566 ± 0.018
3 (tool use)0.583 ± 0.0070.431 ± 0.0240.579 ± 0.006
SDPO1 (math)0.434 ± 0.0130.207 ± 0.0280.425 ± 0.008
2 (science)0.261 ± 0.0180.320 ± 0.0260.479 ± 0.009
3 (tool use)0.230 ± 0.0180.306 ± 0.0190.428 ± 0.009
SCSD1 (math)0.625 ± 0.0150.298 ± 0.0110.548 ± 0.016
2 (science)0.624 ± 0.0050.549 ± 0.0160.560 ± 0.017
3 (tool use)0.604 ± 0.0190.554 ± 0.0140.589 ± 0.016

Continual-learning comparators (Qwen3-8B, M → S → T), same quantities as the task-order table. Standard continual-learning remedies trade learning for retention. The KL-regularized variants and the re-tuned GRPO forget less math than SCSD (−0.129 / −0.133 / −0.146 vs. −0.171), but they learn much less: their total on-task gain over the three stages is at most 0.147, against 0.329 for SCSD, and both KL variants end below the base model on tool use (−0.015 / −0.359). SCSD is the only comparator whose net change over the three tasks is positive (+0.068, against −0.072 to −0.419).

MethodGain 1Gain 2Gain 3Δ MathΔ SciΔ Tool
GRPO, tuned LR (3×10−63\times10^{-6})+0.030−0.005+0.037−0.146+0.065+0.009
SDPO + KL to prev. stage, β=0.1\beta=0.1+0.026+0.075+0.046−0.133+0.073−0.359
SDPO + KL to prev. stage, β=1.0\beta=1.0+0.034+0.080−0.013−0.129−0.028−0.015
SDPO + replay−0.113+0.132+0.085−0.328+0.184−0.146
SDPO (reference)−0.097+0.084+0.106−0.293+0.171−0.083
SCSD (reference)+0.045+0.132+0.152−0.171+0.180+0.059

G3. Stronger baselines: smaller steps, more compute, re-tuned GRPO (XGR6, HtW5). To separate "a better target" from "a smaller step", we compare against SDPO with a lower learning rate and SDPO with a KL penalty to πref\pi_{\mathrm{ref}} at two strengths. To account for the cost of the search, we train SDPO and GRPO for the same wall-clock time as SCSD.

SCSD reaches the highest math accuracy of all runs: shrinking SDPO's step either lowers math further (lower learning rate) or recovers only part of the collapse (KL penalty), and neither more compute nor a re-tuned learning rate lets GRPO or SDPO catch up. This is the conceptual difference: a lower learning rate or a KL penalty reduces drift by shrinking the step, which also shrinks what is learned, while SCSD reduces it by changing the target.

Single-task training on Qwen3-8B (mean@32, ± = standard deviation over 3 seeds).

MethodMathScienceTool use
Base model0.514 ± 0.0080.279 ± 0.0120.561 ± 0.017
GRPO, compute-matched0.523 ± 0.0120.357 ± 0.0150.568 ± 0.010
GRPO re-tuned (lr 1×10−51\times10^{-5})0.425 ± 0.0130.322 ± 0.0120.395 ± 0.011
GRPO re-tuned (lr 3×10−63\times10^{-6})0.502 ± 0.0130.335 ± 0.0150.564 ± 0.007
SDPO (reference)0.406 ± 0.0130.418 ± 0.0090.452 ± 0.017
SDPO, lr 3×10−63\times10^{-6}0.358 ± 0.0060.347 ± 0.0100.574 ± 0.010
SDPO + KL to πref\pi_{\mathrm{ref}}, β=0.1\beta=0.10.504 ± 0.0140.280 ± 0.0180.373 ± 0.007
SDPO + KL to πref\pi_{\mathrm{ref}}, β=1.0\beta=1.00.483 ± 0.0060.310 ± 0.0070.585 ± 0.009
SDPO, compute-matched0.496 ± 0.0110.247 ± 0.0070.222 ± 0.013
SCSD (reference)0.556 ± 0.0160.496 ± 0.0090.612 ± 0.010

G4. Evaluation protocol and statistics (HtW5). All checkpoints, including the base model, are re-evaluated under one protocol: 32 rollouts per prompt, temperature 0.6, top-pp 0.95, and a generation cap of 14336 tokens for every benchmark, and new tables report mean ± standard deviation over three seeds. This also removes the discrepancy in the base model's score between Tables 1 and 2 (0.579 and 0.521, which came from two independent 8-rollout evaluations).

G5. Clarifications of the setup (XGR6, HtW5). Following the official SDPO configuration, our SDPO baseline receives environment feedback; it also uses a different prompt wrapper from SCSD ("Correct solution:" / "Correctly solve the original question." versus "From a verified correct solution:" / "Now solve the problem."). The raw-y⋆y^\star run of G1 removes both differences. Removing the feedback lowers SDPO's result from 0.406 to 0.368, and SCSD's advantage is explained by neither difference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.