acceptodds
Under review as a conference paper at ICLR 2027

ReDraft, Don't Just Distill: Reference-Driven Revision for Continual MLLM Post-Training

Abstract

Continual post-training of large multimodal models should add new capabilities while preserving those from pre-training, and the two goals pull in opposite directions. SFT gives explicit target supervision that learns a task from near-zero accuracy, but its off-policy targets move the model far enough to cause forgetting; on-policy methods such as RLVR and self-distillation preserve policy proximity yet supply little signal when the policy cannot yet solve the task. We introduce ReDraft (Reference-Driven Revision and Fine-Tuning), which obtains both from the model's own failures: using an expert response only as a reference, it has the model revise its own incorrect rollout, keeps the revision only if a verifier accepts it, and fine-tunes on what survives. Each retained target is therefore explicit, yet still close to the current policy. Across Counting, Clock Reading, and Jigsaw on Qwen2.5-VL-3B/7B, two of them with near-zero accuracy, ReDraft gains 56.9 points on the target task against SFT's 52.9 while cutting prior-task loss from 16.6 to 1.5 points (11.3x less forgetting), and improves on OPSD along both axes (19.3 gain, 6.2 loss). Data- and parameter-space analyses match the design: revised targets are more probable under the base model, and the updates they induce stay compact and follow SFT's direction more closely than OPSD's. Together, these results show that revising the model's own rollout rather than directly imitating an expert trajectory can reconcile cold-start acquisition with prior-capability retention.

Primary area:
interpretability and explainable AI
Keywords:
continual learning, mllm

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XYBbOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies continual post-training of multimodal large language models, which aims to acquire new capabilities while preserving those learned during pre-training. It identifies explicit target supervision as essential for cold-start learning and policy proximity as important for limiting model changes that can cause forgetting. The proposed method, ReDraft, combines these properties by having the model revise its own incorrect responses using expert answers as references, verifying the revisions, and fine-tuning on accepted responses. Extensive experiments demonstrate its effectiveness in learning new capabilities while substantially reducing forgetting. Further analyses investigate the sources of this effectiveness, providing insights into how different post-training approaches learn new capabilities and affect forgetting.

Critical strengths
  1. The paper provides extensive experiments that demonstrate the method’s effectiveness and investigate the sources of its gains, establishing a solid empirical foundation for its claims. The analyses also offer valuable insights into the behavior of different post-training approaches and could inform future work. Together, the thorough validation and insightful analyses lead to my positive overall assessment.
  2. The paper is well motivated: its analysis of the strengths and limitations of existing training approaches naturally leads to ReDraft, which combines explicit target supervision with policy proximity. The resulting method is simple and effective, making it a practical approach to continual post-training.
Critical weaknesses

Although the related work section discusses several relevant methods, the distinctions from closely related self-training and expert-revision approaches, such as STaR, are not sufficiently clear. This makes it difficult to assess ReDraft’s core contribution and novelty. The experiments also lack comparisons with these methods, leaving it unclear whether ReDraft offers meaningful advantages over existing approaches. Together, these omissions limit my confidence in the distinctiveness and added value of the proposed method.

Questions for authors
  1. How does ReDraft differ from existing self-training and expert-revision approaches, such as STaR? Under the experimental setup in Figure 4, how do these methods compare with ReDraft in target-task acquisition and retention of prior capabilities?
  2. Given that ReDraft appears applicable to LLMs as well, and its design does not seem to rely on any MLLM-specific mechanisms, what motivates the paper's exclusive focus on MLLMs? Is this because ReDraft offers more pronounced advantages in the multimodal cold-start settings considered in the paper, while its benefits may be less significant in other settings? Experimental evidence or further discussion would help clarify the generalizability and limitations of the proposed method.
5 comments
AuthorsOpenReview ↗

Response (1/2) — Weakness

We thank the reviewer for the thoughtful review. Below, Δprior is the signed change of the 8-benchmark prior average relative to the base model.

W1. Distinction from self-training and expert-revision methods, and empirical comparison

(a) How ReDraft differs. ReDraft's core contribution is turning the model's own failed rollouts into explicit, verified targets that stay close to the policy, so that tasks with near-zero base accuracy can be learned with little forgetting. Related methods differ in where the target comes from:

  • Self-regeneration (STaR) keeps the model's own correct samples or, as in STaR, regenerates a new solution from the answer hint. Neither uses the failed attempt. At cold start, correct samples are scarce, and hint-based regeneration tends to rationalize (evidence invented to fit the answer).
  • Expert revision has an oracle edit the learner's failed response (e.g., SPoT, trained with a contrastive objective, a reference model, and an overlap filter) or relabels the learner's states with expert continuations (Lauffer et al., 2025).
  • ReDraft has the policy revise its own failed draft against an expert reference, keeps only verifier-accepted revisions, and fine-tunes with plain SFT. The target is thus explicit, verified, and written by the policy itself (answer-only PPL 1.62 vs. 5.33 for the expert trajectory, Sec. 6).

We further show that self-revision ability scales with model size (Sec. 7) and introduce ReDraft*, our further contribution for small models, in which the expert revises the learner's draft instead of providing a trajectory to imitate. Sec. 3.3 grounds the design in the KL-regularized optimal target, of which verified revision is a surrogate.

We will make these distinctions explicit in Sec. 2.

(b) Comparison under the Figure 4 setup. We add STaR and report the expert-revision variant ReDraft* (3B; Table 4 checkpoints, with STaR at a comparable step budget; target / Δprior):

3BJigsawClock
SFT (expert trajectory)72.5 / −2.676.4 / −3.7
OPSD (self-distillation)6.1 / −0.243.1 / −0.5
STaR (self-regeneration)18.9 / −3.081.1 / −0.2
ReDraft* (expert revision)79.3 / −0.585.8 / −3.1
ReDraft (self-revision)81.3 / −1.184.9 / −0.8

Rejection sampling, which needs correct samples, applies only to Counting, where the base model already scores 44–52%; there it gains +1.0 / +1.3 (3B / 7B) versus +12.8 / +11.4 for ReDraft, with similar retention (Table 4).

Only ReDraft learns both cold-start tasks (above 80%) while forgetting about one point or less. STaR is competitive on Clock (81.1 / −0.2) but fails on Jigsaw (18.9): without the draft, far fewer of its targets are correct and faithful (40.9% vs. 92.4% on Jigsaw and 14.8% vs. 27.6% on Clock, over 500 prompts judged by Claude Opus 5.5 Max). Expert revision (ReDraft*) beats SFT on both axes for both tasks, and self-revision matches it in acquisition without extra expert calls; expert revision is the better choice when the model is too small to revise itself (Sec. 7).

AuthorsOpenReview ↗

Response (2/2) — Questions

Q1. Self-training methods such as STaR keep correct samples or regenerate a solution from the answer hint, without using the failed attempt; expert-revision methods have an oracle edit the learner's response. ReDraft has the policy revise its own failed draft against an expert reference and keeps only verified revisions, with ReDraft* as the expert-revision variant for small models (W1a). Under the Figure 4 setup, only ReDraft learns both cold-start tasks while forgetting about one point or less; STaR fails on Jigsaw (18.9 vs. 81.3), and rejection sampling barely improves Counting (+1.0 / +1.3 vs. +12.8 / +11.4) (W1b).

Q2. ReDraft is not MLLM-specific. We focused on MLLMs because they offer natural cold-start tasks with verifiable answers: analog clock reading and jigsaw ordering start at or near chance even at 7B (0.1% and 5.3%), so RLVR and rejection sampling receive no signal, which is the regime ReDraft targets. To test generality, we add a text-only science Q&A task built from SciKnowEval (as it in SDFT), with Qwen2.5-3B/7B-Instruct; prior tasks are MATH-500 and MMLU-Pro, and checkpoints are aligned on target gain. The Base row is absolute; other rows are changes from Base:

3B target3B MATH-5003B MMLU-Pro7B target7B MATH-5007B MMLU-Pro
Base31.468.244.836.476.056.9
SFT+26.3−7.8−8.4+24.1−7.0−10.7
OPSD+20.9−4.2−10.4+26.1−15.4−10.6
ReDraft+15.3−1.4−0.1+25.6−3.6−4.7
ReDraft*+24.1−3.4−2.5+26.9−5.6−5.0

The retention advantage is not tied to modality or to cold start. At comparable gains on 7B (+24 to +27), ReDraft and ReDraft* forget much less than SFT and OPSD, most visibly on MMLU-Pro; it also holds on Counting, which the base model partly solves (Table 4).

The acquisition advantage over on-policy methods, by contrast, is largest at cold start: +79.4 for ReDraft vs. +26.0 for OPSD on Jigsaw and Clock, against +12.1 vs. +5.9 on Counting. The main limitation is revision competence: at 3B, self-revision yields fewer targets, so ReDraft gains only +15.3 on the text task, while ReDraft* recovers most of SFT's gain with far less forgetting, consistent with Sec. 7. ReDraft also requires an automatic verifier. We will add this experiment and discussion to the paper.

I thank the authors for their detailed response and additional experiments. Upon further reflection, I noticed two additional issues that I had overlooked in my previous review.

Concern 1:

The paper emphasizes preserving capabilities learned during pretraining and repeatedly uses the term "base model," which suggests post-training from a pretrained model. However, the experiments use instruct models that have already undergone post-training. Since previous post-training may have already caused forgetting, how does preserving the remaining capabilities of an Instruct model support the original motivation?

Concern 2:

SSL4RL (ICML 2026) studies a similar 2×2 Jigsaw task with Qwen2.5-VL-3B-Instruct and shows that GRPO can learn despite near-zero initial accuracy. I have also reproduced it under the same 3B setting, using COCO images and GRPO, and observed that the reward increases substantially during training rather than remaining near zero, confirming that the model can learn this task.

In contrast, this paper reports near-zero initial accuracy on Jigsaw and finds that GRPO fails to learn in the 3B setting (Appendix C.2, Figure 11). Could the authors clarify this discrepancy?

Reference

Guo, X., et al. (2026). SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026).

Reviewer_VKk9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReDraft addresses continual multimodal post-training by converting failed model rollouts into verified training targets. The model receives its original response and an expert reference, revises the response, and retains the revision only if verification succeeds. Correct initial rollouts are kept unchanged. The evaluated implementation constructs this corpus once before supervised fine-tuning. Experiments on Counting, Clock Reading, and Jigsaw with Qwen2.5-VL-3B/7B report improved average acquisition–retention trade-offs, with additional analyses of sequential training, target perplexity, and parameter changes. 9629_ReDraft_Don_t_Just_Distil(…

Critical strengths
  1. Sections 1–3 clearly motivate combining verified target supervision with proximity to the learner’s generation distribution. Appendix H provides task-specific revision prompts.
  2. Table 4 covers six task–model configurations. ReDraft retains more prior-task performance than SFT in every configuration and achieves higher target accuracy in five of six. Section 5.3 additionally evaluates mixed and sequential training.
  3. Table 3 measures sampling success, repair rate, and retained-target yield. Figure 7 compares responses on paired prompts and finds lower base-model perplexity for revisions than expert targets, providing empirical support for the proposed policy-proximity argument.
Critical weaknesses
  1. Appendix B.2 disables OPSD’s pointwise clipping. However, the revised OPSD paper includes this mechanism and reports improved training stability in its evaluated setting (Zhao et al., 2026, arXiv:2601.18734v3, Section 4.3.3). Thus, the experiments establish an advantage over the evaluated unclipped variant, but leave the comparison with clipped OPSD unresolved. The paper also does not evaluate SDFT separately.
  2. The contribution specifically attributable to preserving the failed draft remains insufficiently isolated. There is no matched reference-conditioned regeneration control without the failed draft. STaR already uses answer-conditioned rationale generation for failed problems, while SPoT performs minimal-edit rectification and explicitly permits the policy itself to serve as the oracle in Appendix B. Section 2’s external-oracle distinction therefore needs qualification.
  3. Section 2 states that the corpus is regenerated as training proceeds, whereas Limitations states that it is constructed once. The claim of dominance over OPSD across all settings also needs qualification: Table 4 reports slightly lower prior-task scores for ReDraft on 3B Jigsaw and Clock Reading, despite substantially higher acquisition.
  4. Appendix B describes training-corpus sizes and decoding, but the target evaluation sizes and split construction underlying Table 4 are not comparably specified. Table 4 also uses selected checkpoints with different training durations without clearly documenting a common selection protocol.
Questions for authors
  1. Why was OPSD clipping disabled, and are clipped results available?
  2. Is there existing evidence comparing revision with reference-conditioned regeneration under matched budgets?
  3. Please specify evaluation splits, sample sizes, and checkpoint-selection criteria.
  4. Please reconcile the corpus-regeneration descriptions and qualify the dominance claim.
12 comments
AuthorsOpenReview ↗

Response (1/3) — Weakness 1&2

We thank the reviewer for the careful review. Besides the requested baselines and controls, we add a text-only LLM task showing that ReDraft transfers beyond VLMs. Below, Δprior is the signed change of the 8-benchmark prior average relative to the base model.

W1. Clipped OPSD and SDFT

Appendix B.2 disabled clipping so that OPSD optimizes the exact forward-KL objective of Eq. (3) analyzed in Sec. 3.3. We now add both missing baselines: using the official OPSD script, we change only jsd_token_clip from 0 to its default 0.05 and rerun all six settings, and we run SDFT with its official code. Target / Δprior at the same checkpoints as OPSD in Table 4:

SettingOPSD, no clip (paper)OPSD, clip 0.05SDFTReDraft
Counting 3B51.4 / −0.325.4 / −2.049.1 / −0.756.8 / +0.5
Counting 7B56.6 / −20.746.4 / −31.255.9 / −34.763.6 / −0.6
Jigsaw 3B6.1 / −0.20.0 / −19.35.9 / +0.381.3 / −1.1
Jigsaw 7B19.5 / −2.60.6 / −19.99.1 / −2.276.9 / −2.0
Clock 3B43.1 / −0.50.0 / −25.431.5 / −0.284.9 / −0.8
Clock 7B44.5 / −13.10.0 / −27.87.9 / −8.883.6 / −4.8
Mean gain / forgetting+19.3 / 6.2−5.5 / 20.9+9.0 / 7.7+56.9 / 1.5

Clipping does not help new-task learning in our setting. It caps every vocabulary entry whose divergence exceeds 0.05, so at cold start the signal that needs to be learned can be cut off, consistent with target accuracy even falling below the base model. The stability gain reported by Zhao et al. (Sec. 4.3.3) is for math reasoning, where OPSD converges within about a hundred steps; clipping is designed to suppress over-weighted tokens and cannot supply the explicit target that is missing at cold start.

SDFT likewise gains little on the cold-start tasks (mean +9.0). The unclipped variant is therefore the stronger OPSD baseline here. We will add these results and update the Limitations.

W2. Isolating the failed draft; relation to STaR and SPoT

(a) Draft-free controls. We compare revision with draft-free generation for both kinds of reviser. (i) The learner without the draft: the standard STaR control (Zelikman et al., 2022), where the 3B model is given the ground-truth answer for each failed prompt and regenerates a full solution, i.e., one generation per failed prompt, as in one revision. (ii) The expert without the draft: SFT on GPT-5.5's own trajectory, versus ReDraft*, where the same expert revises the learner's draft; both use the same 10k prompts and 626 training steps (3 seeds). On 500 random prompts per task, the share of correct and reasoning-faithful targets (judged by Claude Opus 5.5 Max):

3B, 500 prompts/taskJigsaw: correctJigsaw: correct & faithfulClock: correctClock: correct & faithful
STaR (answer hint, no draft)70.140.927.014.8
ReDraft (self-revision)97.892.455.827.6
ReDraft* (expert revision)10010010097.6

Training results (3B, target / Δprior):

Draft-free vs. draft-basedJigsawClock
STaR18.9 / −3.081.1 / −0.2
ReDraft81.3 / −1.184.9 / −0.8
SFT on expert trajectory78.5 / −4.487.6 / −6.4
ReDraft*79.1 / −0.384.6 / −3.3

For the learner, conditioning on the draft yields 1.9–2.3× more faithful targets and is what makes Jigsaw learnable. For the expert, revising the learner's draft instead of writing a new trajectory cuts forgetting from −4.4 to −0.3 (Jigsaw) and from −6.4 to −3.3 (Clock) at similar accuracy, with the same data and budget. Preserving the draft is thus the key factor that keeps targets close to the learner.

(b) Relation to STaR and SPoT. STaR rationalizes from the answer without a draft, which is exactly control (i) above. For SPoT, we agree that the "external oracle" distinction needs qualification: SPoT also rectifies failed responses with minimal edits and permits the policy itself to serve as the oracle (its App. B). We will revise Sec. 2 to state the actual differences:

  • ReDraft targets cold-start acquisition (near-zero base accuracy): the learner revises its draft against an expert reference trajectory and keeps it only if verified, whereas SPoT's oracle sees only the failed response and optionally the answer.
  • ReDraft uses plain SFT on verifier-accepted revisions, without SPoT's contrastive objective, reference model, or overlap filter
  • we show that self-revision ability scales with model size (Sec. 7) and introduce ReDraft*, our further contribution, in which an expert revises the learner's draft instead of providing a trajectory to imitate.
AuthorsOpenReview ↗

Response (2/3) — Weaknesses 3&4

W3. Corpus construction and the dominance claim

(a) Corpus construction. In all experiments (Secs. 5.1–5.3, including mixed and sequential training), the revised corpus is built once, offline, from π0, as stated in the Limitations; Line 153 and the Fig. 2 caption ("each round") are wrong, and we will correct them.

We also tried an online variant that refreshes revisions with the current policy: since the answer is visible during revision and the verifier checks only the final answer, the policy gradually learned to hack the verifier, producing revisions that state the answer with no reasoning. We therefore position ReDraft as an offline method for cold-start acquisition.

(b) Dominance over OPSD. We agree and will qualify the claim. In Table 4, ReDraft is better on both axes in 4 of 6 settings. On 3B Clock, OPSD's prior score is 0.3 higher, but ReDraft's first-epoch checkpoint (77.0 / −0.2) already beats OPSD's Table 4 checkpoint (43.1 / −0.5) on both axes, and its extended-training checkpoint as well (43.6 / −0.5, Table 5). On 3B Jigsaw, OPSD forgets 1.0 point less but gains only +2.4 (+4.0 with extended training) versus +77.5 for ReDraft, i.e., it barely moves from the base model. On the text-only LLM task below, ReDraft* beats OPSD on both axes at both scales.

W4. Evaluation protocol and checkpoint selection

(a) Evaluation sizes and splits. Each target task is evaluated on 100 held-out prompts that are disjoint from the training prompts, with 8 samples per prompt at temperature 0.7 (avg@8). Train / test sizes are 2,000 / 100 (Counting) and 10,000 / 100 (Clock Reading, Jigsaw). Prior benchmarks use their full official evaluation splits. Across 3 seeds, the target-task std is 0.5–5.0 points (our response to Reviewer Zsw4, W4d). We will add these details to App. B.

(b) Checkpoint selection. Because the methods learn at different speeds, Table 4 makes a Pareto-style comparison: forgetting is compared at checkpoints of comparable or higher target accuracy (ReDraft exceeds SFT in five of six settings), so step counts differ; OPSD is additionally trained longer in Table 5. To remove the step difference, we extend training to 3 epochs and recompute the six-setting means at equal epochs and at each method's best-target checkpoint:

Checkpoint ruleSFT: gain / forg.ReDraft: gain / forg.Forg. ratio (all / w/o 7B Count.)
Table 452.9 / 16.656.9 / 1.511.3× / 4.6×
Same epoch: 152.0 / 17.449.0 / 1.511.5× / 4.5×
Same epoch: 260.9 / 19.556.9 / 1.513.2× / 6.7×
Same epoch: 361.5 / 18.059.5 / 1.117.1× / 8.0×
Best target (epochs 1–3)61.9 / 18.159.6 / 1.116.0× / 7.5×

At every matched epoch, ReDraft forgets less than SFT, with comparable acquisition. Even excluding 7B Counting, ReDraft forgets 4.5–8× less (11–17× including it).

Generalization beyond VLMs

To test whether ReDraft transfers beyond multimodal tasks, we add a text-only science Q&A task built from SciKnowEval(as it in SDFT), with Qwen2.5-3B/7B-Instruct. Prior tasks are MATH-500 and MMLU-Pro. The Base row is absolute; other rows are changes from Base:

3B target3B MATH-5003B MMLU-Pro7B target7B MATH-5007B MMLU-Pro
Base (absolute)31.468.244.836.476.056.9
SFT+26.3−7.8−8.4+24.1−7.0−10.7
OPSD+20.9−4.2−10.4+26.1−15.4−10.6
ReDraft+15.3−1.4−0.1+25.6−3.6−4.7
ReDraft*+24.1−3.4−2.5+26.9−5.6−5.0

The pattern carries over. At comparable target gains on 7B (+24 to +27), ReDraft and ReDraft* retain prior tasks much better than SFT and OPSD, most visibly on MMLU-Pro, the prior task most similar to the target. At 3B, self-revision yields fewer targets and ReDraft gains only +15.3; ReDraft* then comes close to SFT's gain (+24.1 vs. +26.3) with far less forgetting, consistent with Sec. 7, and beats OPSD on both axes.

AuthorsOpenReview ↗

Response (3/3) — Questions

Q1. Clipping was disabled so that OPSD optimizes the exact forward KL of Eq. (3). Clipped results with the official default (0.05) are now available for all six settings: clipping lowers OPSD's mean gain from +19.3 to −5.5 and raises forgetting from 6.2 to 20.9, and SDFT also gains little (+9.0). Details in W1.

Q2. Yes, for both kinds of reviser. Given the same answer and generation budget, STaR without the draft yields about half as many faithful targets and barely learns Jigsaw (18.9 vs. 81.3). With the same expert, data, and steps, revising the learner's draft instead of training on the expert's own trajectory reduces forgetting from −4.4 to −0.3 (Jigsaw) and from −6.4 to −3.3 (Clock). Details in W2(a).

Q3. Target tasks: 100 held-out prompts × 8 samples (avg@8); train / test = 2,000 / 100 (Counting) and 10,000 / 100 (Clock Reading, Jigsaw). Prior benchmarks: full official splits. Checkpoints: a Pareto-style comparison in Table 4, plus same-epoch and best-target rules, under which ReDraft forgets 4.5–8× less than SFT excluding 7B Counting. Details in W4.

Q4. The corpus is built once, offline, from π0 in all experiments; we will correct Line 153 and Fig. 2. We will qualify the dominance claim: ReDraft is better on both axes in 4 of 6 settings, its first checkpoint also dominates OPSD on 3B Clock, and on 3B Jigsaw OPSD forgets 1.0 point less but gains only +2.4 vs. +77.5. Details in W3.

Reviewer_Zsw4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the tension between acquiring new capabilities and retaining prior knowledge during continual post-training of multimodal large language models. The authors propose ReDraft (Reference-Driven Revision and Fine-Tuning), which samples the learner’s own responses, revises failed responses using an expert reference, the ground-truth answer, and revision instructions, and performs supervised fine-tuning on verifier-accepted targets. For smaller models that struggle to self-revise under simplified prompts, the authors introduce ReDraft*, where an external expert performs the revision. Experiments cover Counting, Clock Reading, and Jigsaw on Qwen2.5-VL-3B/7B, eight prior-capability benchmarks, and mixed and sequential training. ReDraft reports a favorable average acquisition–retention trade-off relative to the evaluated SFT and OPSD configurations, supported by target-perplexity and parameter-space analyses.

Critical strengths
  1. The paper addresses a practically meaningful problem: enabling multimodal models to acquire capabilities from near-zero initial performance while preserving existing capabilities. Constructing revised training targets from the learner’s own responses is intuitive and potentially useful.

  2. The evaluation measures both acquisition and retention across three tasks, two model sizes, and mixed and sequential training. Target-perplexity measurements, parameter-update analyses, and revision experiments under simplified prompts provide useful descriptive observations. The appendices also include detailed prompts, repair rates, and generated examples that help readers inspect the method’s behavior.

Critical weaknesses
  1. The main experiments do not sufficiently isolate the contribution of revising a failed draft, nor does verifier acceptance guarantee correctly repaired reasoning. The revision prompts in Appendix H provide the exact final answer alongside an expert reference and extensive task-specific constraints. While 3B repair rates reach 56.84% and 92.25% on Clock Reading and Jigsaw under this main prompt (Table 3), they drop to 0.6% and 2.5% under the simplified prompts in Section 7 (17.2% and 18.8% at 7B). Because the simplified prompts simultaneously remove the explicit ground-truth answer and detailed task rules, this comparison does not isolate the effect of each ingredient. Moreover, two key baselines are missing: (i) generating a new response from the same reference, ground-truth answer, and instructions without conditioning on the failed draft; and (ii) unrestricted rewriting without the minimal-edit constraint. Consequently, the evidence does not separate draft revision from answer-conditioned generation in the learner’s style.

    The verifier’s guarantees are also unclear. In Appendix G.1, the accepted Clock Reading revision retains erroneous phrases such as “at 3 and 30 seconds” and “at 6 and 30 seconds,” alongside a contradiction between the minute-hand description and the final time (03:30:38). While a single example does not establish the prevalence of faulty revisions, it directly shows that verifier acceptance does not guarantee a correctly repaired reasoning trajectory. The paper should specify what the verifier checks and evaluate reasoning correctness beyond the final boxed answer.

  2. The theoretical and empirical analyses do not establish the claimed distributional proximity of the revised targets or its causal link to reduced forgetting. Equation (5) defines the current policy conditioned on correctness, but being the closest valid distribution does not imply a small distance from the original policy: Equation (5) directly yields DKL(π∗∥πθ)=−log⁡Zθ(x)D_{\mathrm{KL}}(\pi^*\Vert\pi_\theta)=-\log Z_\theta(x), which is large in the cold-start regime (Zθ(x)≈0Z_\theta(x)\approx 0). Furthermore, the implemented target in Equations (7)–(8) depends on a reference-conditioned revision kernel RθR_\theta. A positive repair rate (rθ(x)>0r_\theta(x)>0) guarantees more accepted targets, but does not ensure preservation of πθ\pi_\theta’s relative probabilities among valid responses. Appendix A.4 acknowledges that the minimal-edit instruction is not a formal constraint, that the construction is a biased surrogate, and that no discrepancy bound is provided.

    Empirically, lower perplexity under the frozen base model (Section 6) shows that revised responses are easier to predict, but under minimal editing this may largely reflect copied unchanged text rather than the proximity of the edited spans. Separately measuring perplexity on edited versus unchanged spans—and comparing against draft-free regeneration—is needed. Parameter displacement, effective rank, and update-direction cosines are likewise descriptive correlations rather than proof of the causal chain from draft revision to retention.

  3. The OPSD comparison and token-credit analysis do not adequately support the broader claims about self-distillation’s limitations. Appendix B.2 sets OPSD’s token_clip to zero, disabling the published pointwise clipping mechanism specifically designed to prevent stylistic tokens from dominating the loss. Attributing OPSD’s weak performance to stylistic tokens—and extending training under this unclipped configuration (Section 5.2)—does not separate a baseline configuration choice from an inherent limitation of the objective.

    Furthermore, Appendix F (Figure 14) diagnoses OPSD using sampled-token log-probability ratios log⁡q(y^n)−log⁡p(y^n)\log q(\hat y_n)-\log p(\hat y_n), whereas Equation (3) optimizes full-vocabulary forward KL, whose logit gradient is proportional to p−qp-q across the entire vocabulary. A small log ratio on a sampled incorrect token does not preclude a strong corrective gradient toward an unsampled correct token. Analyzing full-vocabulary KL, probabilities assigned to corrective tokens, and answer-position gradients across more examples is needed to substantiate this claim.

  4. Prompt filtering, checkpoint selection, and an offline–online discrepancy complicate attribution and the headline forgetting ratio. Table 3 shows that only 56.85% (3B) and 65.20% (7B) of Clock Reading prompts yield accepted targets. Comparing ReDraft with full-dataset SFT therefore conflates target revision with prompt selection; an SFT baseline trained on the identical retained prompt subset (SθS_\theta) is needed. In Table 4, methods are also evaluated at different step counts without a clearly specified checkpoint-selection protocol. Notably, SFT on 7B Counting drops by 61.87 points on prior tasks (to 18.92%), single-handedly driving the headline 11.3× forgetting ratio; excluding this outlier reduces ReDraft’s average forgetting advantage to ~4.6×. Whether this collapse persists under learning-rate tuning or earlier stopping should be clarified, alongside seed counts, variance, and per-benchmark scores.

    Finally, Line 153 states that the revised corpus is regenerated as training proceeds, whereas the Limitations section states that it is constructed only once offline from π0\pi_0. Because proximity to π0\pi_0 does not guarantee maintained proximity to the evolving policy πθ\pi_\theta, the actual data-collection protocol across all experiments should be clarified and reconciled with the main text.

Questions for authors
  1. Under matched information and output constraints, how does ReDraft compare against (a) answer/reference-conditioned generation without the failed draft, and (b) unrestricted rewriting without the minimal-edit rule? Can explicit answer presentation and instruction detail also be ablated separately to explain the drop in Section 7?

  2. What exact properties does the verifier check, and why is the internally inconsistent Clock Reading revision in Appendix G.1 accepted?

  3. Does ReDraft’s advantage over OPSD persist when OPSD’s pointwise clipping (token_clip > 0) is enabled and properly tuned? Can Appendix F be updated with full-vocabulary KL and answer-position gradient measurements?

  4. How does SFT perform when trained on the exact same retained prompt subset (SθS_\theta) as ReDraft? Please also clarify the checkpoint-selection rule, test-set sizes, seed variance, and whether learning-rate tuning or early stopping prevents the 61.87-point SFT collapse on 7B Counting.

  5. Are revised corpora generated once from π0\pi_0 or refreshed during training in Sections 5.1 and 5.3? Additionally, do edited and unchanged spans exhibit different perplexity patterns under the base model?

6 comments
AuthorsOpenReview ↗

Response (1/4) — Weakness 1

We thank the reviewer for the detailed and constructive review. We add the following new results: a STaR baseline, a faithfulness analysis, span-level PPL statistics, an OPSD baseline with clipping, an SDFT baseline, an SFT baseline on ReDraft's retained subset, a checkpoint-robustness analysis, 3-seed experiments, and an LLM generalization task. Below, Δprior is the signed change of the 8-benchmark prior average relative to the base model.

W1. Does revising the draft matter, and what does the verifier guarantee?

(a) STaR without the draft. We add the standard STaR baseline (Zelikman et al., 2022): for incorrectly answered prompts, the 3B model is given the answer as a hint and regenerates a full solution, as in the original STaR. This directly tests the alternative raised by the reviewer: answer-conditioned generation in own style. On 500 randomly sampled prompts from each of the two new tasks, we report the share with the correct answer, and the share whose reasoning has no claim contradicting its final answer (consistent, scored by a rule-based checker designed with Claude Opus 5.5 Max; Table 13 in App. C.11 adds the Claude Opus 5.5 Max judge and all criteria):

3B, 500 prompts/taskJigsaw: correctJigsaw: correct & consistentClock: correctClock: correct & consistent
Expert trajectory100100100100
STaR (answer hint, no draft)70.140.927.014.8
ReDraft (self-revision)97.892.455.827.6
ReDraft* (expert revision)10010010097.6

Training on these targets (3B):

Jigsaw: target / ΔpriorClock: target / Δprior
SFT (Table 6)72.5 / −2.676.4 / −3.7
STaR18.9 / −3.076.8 / −0.3
ReDraft81.3 / −1.184.9 / −0.8

STaR is reported within the same three-epoch budget as the other methods (Table 3); our original response listed its fourth-epoch Clock result (81.1 / −0.2).

Given the answer, conditioning on the draft (ReDraft) yields a 1.9–2.3× higher share of consistent targets; without the draft (STaR), Jigsaw is barely learned (below 19% at every epoch, Table 14). Because the draft is written in the hint-free task context, its preserved parts stay on-task, whereas STaR generates from the hint and invites rationalization (evidence invented to fit the given answer), which is what this gap shows.

Moreover, ReDraft*, our further contribution beyond ReDraft (Sec. 7), lets the GPT-5.5 expert, rather than the model itself, revise the draft, raising consistency to 97.6–100% (Claude judge: 93.0% / 99.2% faithful, Jigsaw / Clock): the expert corrects the reasoning while the learner's own trajectory is kept. STaR has no draft to edit, so it cannot use an external expert this way; letting the expert write the trajectory instead reduces to off-policy imitation, i.e., SFT.

(b) Unrestricted rewriting. 3B Jigsaw ReDraft is a natural instance: its accepted self-revisions are essentially full rewrites of the draft. These policy-written rewrites still have about half the expert's PPL (4.00 vs. 9.32, Fig. 12) and keep the retention advantage (−1.1 vs. −2.6 for SFT). The key to ReDraft is thus that the policy writes the target starting from its own draft; minimal editing matters most when an external model revises (ReDraft*), where it confines off-policy tokens to the edited spans (W2c).

(c) Section 7 changes two factors at once. We agree. Sec. 7 tests whether the model can revise under weaker instruction guidance, which motivates ReDraft*; it is not meant as an ablation of the main prompt. How much comes from simply knowing the answer is isolated by (a): with the answer but no draft, 3B reaches 27.0 / 70.1 correct (Clock / Jigsaw), versus 55.8 / 97.8 with the draft. The revision states this in Sec. 7.

(d) What the verifier checks. Only whether the final boxed answer matches the ground truth. Intermediate steps are not checked, which is why the G.1 revision was accepted and why unfaithful reasoning can enter the targets. The audit in (a) quantifies this: on 3B Clock, 72.4% of the accepted self-revisions contain a claim that contradicts their final answer, while ReDraft still has 1.9× STaR's share of consistent targets (27.6 vs. 14.8). This reflects outcome-only verification together with the limited revision ability of the 3B model, not revision itself: our ReDraft* resolves it (97.6–100% consistent under both judges; 93.0% / 99.2% faithful and 93.4% / 96.6% grounded under Claude) while retaining most of ReDraft's advantage over SFT (Sec. 7). The revision adds this discussion (App. C.11), annotates the errors in G.1, and states the limitation explicitly.

AuthorsOpenReview ↗

Response (2/4) — Weakness 2&3

W2. Distributional proximity and its link to forgetting

(a) "Closest" is not "close" in absolute terms. We agree, and will present π* as an optimum. For any target q supported on C(x):

KL(q‖πθ) = −log Zθ(x) + KL(q‖π*)

So −log Zθ(x) is a floor paid by every correct target, including SFT's point mass on y*; objectives differ only in the excess KL(q‖π*). We will rephrase Sec. 3.3 accordingly: π* is the minimum-KL correct target, and "proximity" refers to the excess term KL(q‖π*).

(b) No analytical bound for ReDraft. We agree, and will present Eqs. (7)–(8) as a design surrogate rather than a guarantee. What can be measured is an upper bound: for any target distribution q, KL(q‖π0) ≤ E[−log π0(y|x)] with y drawn from q (since entropy ≥ 0), with equality for SFT's point mass. The answer-only PPL in Sec. 6 is exactly the per-token form of the right-hand side. We will make this explicit.

(c) Edited vs. unchanged spans. As suggested, we split answer-only PPL (3B, 500 prompts/task) using a token-level diff against the draft: shared ≥4-token blocks are unchanged and the rest are edited, and revisions with <30% unchanged tokens are treated as rewrites and excluded. Median PPL:

3BKeptDraftUnchangedEditedExpert trajectory
Clock · ReDraft269/2791.341.351.895.20
Clock · ReDraft*500/5001.331.393.815.26
Jigsaw · ReDraft*476/5001.792.049.1312.56
  • Unchanged spans stay at the draft's level, as expected.
  • For ReDraft, self-written edits are themselves near-policy: 1.89 vs. 5.20 for the expert on Clock. On Jigsaw, 3B self-revisions are full rewrites (hence no row), yet still have about half the expert's PPL (W1b).
  • For ReDraft*, expert-written edits are farther from the policy but still below the expert trajectory, while the unchanged spans keep the draft's PPL. This is the intended effect of minimal editing: off-policy supervision is confined to the edited spans.

(d) Weight-space statistics are descriptive. We agree that displacement, effective rank, and update cosines only show correlation, and will soften "attribute" to "are consistent with". Two new controlled comparisons each change a single factor:

  • SFT vs. ReDraft on the retained subset: identical prompts; only the target differs. ReDraft still forgets much less (W4a).
  • SFT vs. ReDraft*: the same GPT-5.5 expert, which either writes the full trajectory or edits the model's draft. The latter learns more and forgets less (Sec. 7).

W3. OPSD clipping and token-level credit

(a) OPSD with clipping enabled, plus SDFT. Using the official OPSD script, we change only jsd_token_clip from 0 to its default 0.05 and rerun all experiments; we also add SDFT, run with its official code. Target / Δprior at the same checkpoints as OPSD in Table 4:

SettingOPSD, no clip (paper)OPSD, clip 0.05SDFTReDraft
Counting 3B51.4 / −0.325.4 / −2.049.1 / −0.756.8 / +0.5
Counting 7B56.6 / −20.746.4 / −31.255.9 / −34.763.6 / −0.6
Jigsaw 3B6.1 / −0.20.0 / −19.35.9 / +0.381.3 / −1.1
Jigsaw 7B19.5 / −2.60.6 / −19.99.1 / −2.276.9 / −2.0
Clock 3B43.1 / −0.50.0 / −25.431.5 / −0.284.9 / −0.8
Clock 7B44.5 / −13.10.0 / −27.87.9 / −8.883.6 / −4.8
Mean gain / forgetting+19.3 / 6.2−5.5 / 20.9+9.0 / 7.7+56.9 / 1.5

Clipping does not help new-task learning. It caps every vocabulary entry whose divergence exceeds 0.05, so the signal that needs to be learned can be cut off, consistent with target accuracy even falling below the base model. Clipping is designed to suppress over-weighted tokens; it cannot remedy OPSD's lack of an explicit target at cold start. SDFT likewise gains little on cold-start tasks (mean +9.0). We will add these results to the paper.

(b) Fig. 14: log ratios do not measure gradient magnitude. We agree. OPSD's loss (Eq. 3) compares the teacher and student distributions over the entire vocabulary at each position, so its gradient acts on all tokens, whereas Fig. 14 only plots the log ratio of the sampled token; it therefore cannot show where the update concentrates. The sign of this ratio, however, determines whether the sampled token is reinforced or suppressed (the gradient on its logit is p − q). We will revise App. F and Sec. 6 to keep only this directional claim.

AuthorsOpenReview ↗

Response (3/4) — Weakness 4

W4. Prompt filtering, checkpoints, the 7B-Counting outlier, and the offline protocol

(a) SFT on the retained subset (the prompts for which ReDraft obtains an accepted target; 3B, same expert responses and steps as ReDraft in Table 4):

3BJigsaw: target / ΔpriorClock: target / Δprior
SFT, full set (Table 4)72.5 / −2.676.4 / −3.7
SFT, retained subset77.0 / −3.781.8 / −6.0
ReDraft (retained subset)81.3 / −1.184.9 / −0.8

Restricting SFT to the retained subset does not reduce forgetting, so data selection does not explain the gap.

(b) Checkpoint selection and robustness. Because the methods learn at different speeds, Table 4 makes a Pareto-style comparison: forgetting is compared at checkpoints of comparable or higher target accuracy (ReDraft exceeds SFT in five of six settings), so step counts differ. To remove this difference, we extend training to 3 epochs and recompute the six-setting means at equal epochs and at each method's best-target checkpoint:

Checkpoint ruleSFT: gain / forg.ReDraft: gain / forg.Forg. ratio (all / w/o 7B Count.)
Table 452.9 / 16.656.9 / 1.511.3× / 4.6×
Same epoch: 152.0 / 17.449.0 / 1.511.5× / 4.5×
Same epoch: 260.9 / 19.556.9 / 1.513.2× / 6.7×
Same epoch: 361.5 / 18.059.5 / 1.117.1× / 8.0×
Best target (epochs 1–3)61.9 / 18.159.6 / 1.116.0× / 7.5×

At every matched epoch, ReDraft forgets less than SFT, with comparable acquisition. Even excluding 7B Counting, ReDraft forgets 4.5–8× less (11–17× including it).

(c) The 7B-Counting collapse is not an over-training or tuning artifact. It appears before the task is sufficiently learned (SFT at 63 steps: +6.9 target, −66.3 prior), so early stopping cannot avoid it. In contrast, at the same learning rate, ReDraft (−0.6) and rejection sampling (−0.9), which train on the model's own outputs, do not collapse, while OPSD (−20.7) and SDFT (−34.7), whose teachers see the same expert responses, collapse to a lesser degree.

(d) Seeds and test sets. 3 seeds on 3B (aligned with Table 4). Target is mean ± std; for Δprior, ± is an upper bound on the std derived from the per-benchmark stds. Target tasks use 100 held-out prompts × 8 samples (avg@8); prior benchmarks use their full official splits. Per-benchmark scores will be added to the appendix.

3B, 3 seedsJigsaw targetJigsaw ΔpriorClock targetClock Δprior
SFT74.5 ± 1.8−2.9 ± 0.480.2 ± 4.0−3.9 ± 0.6
OPSD5.6 ± 0.9−0.2 ± 0.538.4 ± 5.0−0.7 ± 0.5
ReDraft82.7 ± 1.3−0.8 ± 0.586.3 ± 2.9−0.8 ± 0.7
ReDraft*79.1 ± 0.5−0.3 ± 0.584.6 ± 1.8−3.3 ± 0.9

(e) Generalization. We add a text-only science Q&A task using the Chemistry L-3 subset of SciKnowEval (as it in SDFT), with Qwen2.5-3B/7B-Instruct; prior tasks are MATH-500 and MMLU-Pro, and checkpoints are aligned on target gain. The Base row is absolute; other rows are changes from Base:

3B target3B MATH-5003B MMLU-Pro7B target7B MATH-5007B MMLU-Pro
Base (absolute)31.468.244.836.476.056.9
SFT+26.3−7.8−8.4+24.1−7.0−10.7
OPSD+20.9−4.2−10.4+26.1−15.4−10.6
ReDraft+15.3−1.4−0.1+25.6−3.6−4.7
ReDraft*+24.1−3.4−2.5+26.9−5.6−5.0

At comparable target gains on 7B, ReDraft and ReDraft* retain prior tasks much better, most visibly on MMLU-Pro, the prior task most similar to the target. At 3B, self-revision yields fewer targets and ReDraft gains only +15.3; ReDraft* then comes close to SFT's gain (+24.1 vs. +26.3) with far less forgetting, consistent with Sec. 7. We also find that SFT forgets early: after 10 steps on 3B (+2.0 target), MMLU-Pro already drops by 5.5.

(f) Offline protocol. In all experiments (Secs. 5.1–5.3, including mixed and sequential training), the revised corpus is built once, offline, from π0. Line 153 and the Fig. 2 caption ("each round") are wrong; we will correct them.

We also tried an online variant that refreshes revisions with the current policy: since the answer is visible during revision and the verifier checks only the final answer, the policy gradually learned to hack the verifier, producing revisions that state the answer with no reasoning, which the verifier still accepts. We therefore position ReDraft as an offline method for cold-start acquisition and will state this in the paper.

Comments on the paper

2 comments

Dear Reviewers,

As the discussion phase comes to an end, we'd appreciate it if you could take a moment to review the authors' responses. If you have any remaining concerns or questions, please raise them with the authors before the discussion closes.

Best,

Your AC

AuthorsOpenReview ↗

General Response

Dear Reviewers, AC, SAC, and PC members,

We thank all reviewers for their constructive feedback. Reviewers agreed that balancing new-skill acquisition against retention in continual (M)LLM post-training is an important problem, and that building explicit targets from the model's own failed rollouts is a well-motivated idea. Reviewer XYBb confirmed that their concerns have been adequately addressed, and Reviewer VKk9 noted that the added baselines and draft-free controls help address theirs. Below we summarize the shared concerns, how we addressed them, and the contributions of this work.

Shared Concerns

  • Distinction from self-training and expert revision, and the role of the draft (all reviewers). We added STaR (answer only), a draft-free control given the full expert trajectory, and a matched-step comparison between SFT on the expert trajectory and ReDraft*. With the reference held fixed, revising the draft rather than writing a new solution gives a better acquisition–retention trade-off for both the learner and the expert, and answer-only STaR yields far less faithful targets and barely learns Jigsaw.
  • Reasoning faithfulness (Reviewers Zsw4, VKk9). We audited the targets with a rule-based checker and an LLM judge (prompts provided), reporting consistency, faithfulness (consistent and complete) and visual grounding separately. ReDraft* targets are 93.0% / 99.2% faithful and 93.4% / 96.6% grounded (Jigsaw / Clock), the closest to the expert trajectory among all variants.
  • Self-distillation baselines (Reviewers Zsw4, VKk9). We added OPSD with its default clipping and SDFT on all six settings. Neither closes the gap to ReDraft, and clipping does not help in the cold-start regime.
  • Protocol and robustness (Reviewers Zsw4, VKk9). We specified each method's training steps and checkpoint rule (matched-step vs. Pareto-optimal selection), and added SFT on the same retained prompt subset, checkpoint-selection robustness and 3-seed runs. ReDraft's retention advantage holds under all of them, including when the 7B-Counting outlier is excluded.

Other Concerns

  • Reviewer Zsw4: We reformulated the proximity argument (minimum-KL correct target, with PPL as an upper bound), added span-level PPL showing that edited spans also stay close to the policy, restricted the token-credit analysis to directional claims, and clarified the offline corpus construction.
  • Reviewer VKk9: We clarified that the corpus is built once offline, qualified the dominance claim over OPSD, and specified the evaluation splits and sizes.
  • Reviewer XYBb: We added a text-only LLM task, where ReDraft's retention advantage carries over, clarified that the base model is the instruction-tuned starting checkpoint, and showed that our GRPO results agree with SSL4RL once its partial-credit reward is distinguished from our exact-match verifier.

Contributions of the original submission

  1. ReDraft turns the model's own failed rollouts into explicit, verified targets that stay close to the policy, enabling cold-start acquisition with little forgetting.
  2. A unified analysis of SFT, self-distillation, and revision through the KL-regularized optimal target, supported by data- and parameter-space evidence.
  3. ReDraft*: we show that self-revision ability scales with model size, and extend ReDraft to small models by using the expert as a reviser rather than a demonstrator.
  4. Consistent gains across three tasks, two model scales, and mixed and sequential training, against SFT, RLVR, rejection sampling, and OPSD.

New contributions from this rebuttal

  1. Isolating the draft: with the reference held fixed, revising the draft gives a better acquisition–retention trade-off than writing a new solution, for both the learner and the expert; compared with answer-only STaR, it also yields more faithful targets (less rationalization) and better acquisition.
  2. Faithfulness of ReDraft*: expert revision yields targets that are 93.0% / 99.2% faithful and 93.4% / 96.6% visually grounded, close to the expert trajectory, addressing the main limitation of outcome-only verification.
  3. Stronger baselines: OPSD with its default clipping and SDFT, neither of which closes the gap to ReDraft.
  4. Generalization beyond MLLMs: on a text-only LLM task, ReDraft's retention advantage carries over.

All new results and clarifications are included in the revised PDF (highlighted in purple). We hope our responses address the reviewers' concerns.

Best regards,

Authors of Submission 9629

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.