- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies continual post-training of multimodal large language models, which aims to acquire new capabilities while preserving those learned during pre-training. It identifies explicit target supervision as essential for cold-start learning and policy proximity as important for limiting model changes that can cause forgetting. The proposed method, ReDraft, combines these properties by having the model revise its own incorrect responses using expert answers as references, verifying the revisions, and fine-tuning on accepted responses. Extensive experiments demonstrate its effectiveness in learning new capabilities while substantially reducing forgetting. Further analyses investigate the sources of this effectiveness, providing insights into how different post-training approaches learn new capabilities and affect forgetting.
Critical strengths
- The paper provides extensive experiments that demonstrate the method’s effectiveness and investigate the sources of its gains, establishing a solid empirical foundation for its claims. The analyses also offer valuable insights into the behavior of different post-training approaches and could inform future work. Together, the thorough validation and insightful analyses lead to my positive overall assessment.
- The paper is well motivated: its analysis of the strengths and limitations of existing training approaches naturally leads to ReDraft, which combines explicit target supervision with policy proximity. The resulting method is simple and effective, making it a practical approach to continual post-training.
Critical weaknesses
Although the related work section discusses several relevant methods, the distinctions from closely related self-training and expert-revision approaches, such as STaR, are not sufficiently clear. This makes it difficult to assess ReDraft’s core contribution and novelty. The experiments also lack comparisons with these methods, leaving it unclear whether ReDraft offers meaningful advantages over existing approaches. Together, these omissions limit my confidence in the distinctiveness and added value of the proposed method.
Questions for authors
- How does ReDraft differ from existing self-training and expert-revision approaches, such as STaR? Under the experimental setup in Figure 4, how do these methods compare with ReDraft in target-task acquisition and retention of prior capabilities?
- Given that ReDraft appears applicable to LLMs as well, and its design does not seem to rely on any MLLM-specific mechanisms, what motivates the paper's exclusive focus on MLLMs? Is this because ReDraft offers more pronounced advantages in the multimodal cold-start settings considered in the paper, while its benefits may be less significant in other settings? Experimental evidence or further discussion would help clarify the generalizability and limitations of the proposed method.
5 comments
Response (2/2) — Questions
Q1. Self-training methods such as STaR keep correct samples or regenerate a solution from the answer hint, without using the failed attempt; expert-revision methods have an oracle edit the learner's response. ReDraft has the policy revise its own failed draft against an expert reference and keeps only verified revisions, with ReDraft* as the expert-revision variant for small models (W1a). Under the Figure 4 setup, only ReDraft learns both cold-start tasks while forgetting about one point or less; STaR fails on Jigsaw (18.9 vs. 81.3), and rejection sampling barely improves Counting (+1.0 / +1.3 vs. +12.8 / +11.4) (W1b).
Q2. ReDraft is not MLLM-specific. We focused on MLLMs because they offer natural cold-start tasks with verifiable answers: analog clock reading and jigsaw ordering start at or near chance even at 7B (0.1% and 5.3%), so RLVR and rejection sampling receive no signal, which is the regime ReDraft targets. To test generality, we add a text-only science Q&A task built from SciKnowEval (as it in SDFT), with Qwen2.5-3B/7B-Instruct; prior tasks are MATH-500 and MMLU-Pro, and checkpoints are aligned on target gain. The Base row is absolute; other rows are changes from Base:
| 3B target | 3B MATH-500 | 3B MMLU-Pro | 7B target | 7B MATH-500 | 7B MMLU-Pro | |
|---|---|---|---|---|---|---|
| Base | 31.4 | 68.2 | 44.8 | 36.4 | 76.0 | 56.9 |
| SFT | +26.3 | −7.8 | −8.4 | +24.1 | −7.0 | −10.7 |
| OPSD | +20.9 | −4.2 | −10.4 | +26.1 | −15.4 | −10.6 |
| ReDraft | +15.3 | −1.4 | −0.1 | +25.6 | −3.6 | −4.7 |
| ReDraft* | +24.1 | −3.4 | −2.5 | +26.9 | −5.6 | −5.0 |
The retention advantage is not tied to modality or to cold start. At comparable gains on 7B (+24 to +27), ReDraft and ReDraft* forget much less than SFT and OPSD, most visibly on MMLU-Pro; it also holds on Counting, which the base model partly solves (Table 4).
The acquisition advantage over on-policy methods, by contrast, is largest at cold start: +79.4 for ReDraft vs. +26.0 for OPSD on Jigsaw and Clock, against +12.1 vs. +5.9 on Counting. The main limitation is revision competence: at 3B, self-revision yields fewer targets, so ReDraft gains only +15.3 on the text task, while ReDraft* recovers most of SFT's gain with far less forgetting, consistent with Sec. 7. ReDraft also requires an automatic verifier. We will add this experiment and discussion to the paper.
I thank the authors for their detailed response and additional experiments. Upon further reflection, I noticed two additional issues that I had overlooked in my previous review.
Concern 1:
The paper emphasizes preserving capabilities learned during pretraining and repeatedly uses the term "base model," which suggests post-training from a pretrained model. However, the experiments use instruct models that have already undergone post-training. Since previous post-training may have already caused forgetting, how does preserving the remaining capabilities of an Instruct model support the original motivation?
Concern 2:
SSL4RL (ICML 2026) studies a similar 2×2 Jigsaw task with Qwen2.5-VL-3B-Instruct and shows that GRPO can learn despite near-zero initial accuracy. I have also reproduced it under the same 3B setting, using COCO images and GRPO, and observed that the reward increases substantially during training rather than remaining near zero, confirming that the model can learn this task.
In contrast, this paper reports near-zero initial accuracy on Jigsaw and finds that GRPO fails to learn in the 3B setting (Appendix C.2, Figure 11). Could the authors clarify this discrepancy?
Reference
Guo, X., et al. (2026). SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026).
Response (1/2) — Weakness
We thank the reviewer for the thoughtful review. Below, Δprior is the signed change of the 8-benchmark prior average relative to the base model.
W1. Distinction from self-training and expert-revision methods, and empirical comparison
(a) How ReDraft differs. ReDraft's core contribution is turning the model's own failed rollouts into explicit, verified targets that stay close to the policy, so that tasks with near-zero base accuracy can be learned with little forgetting. Related methods differ in where the target comes from:
We further show that self-revision ability scales with model size (Sec. 7) and introduce ReDraft*, our further contribution for small models, in which the expert revises the learner's draft instead of providing a trajectory to imitate. Sec. 3.3 grounds the design in the KL-regularized optimal target, of which verified revision is a surrogate.
We will make these distinctions explicit in Sec. 2.
(b) Comparison under the Figure 4 setup. We add STaR and report the expert-revision variant ReDraft* (3B; Table 4 checkpoints, with STaR at a comparable step budget; target / Δprior):
Rejection sampling, which needs correct samples, applies only to Counting, where the base model already scores 44–52%; there it gains +1.0 / +1.3 (3B / 7B) versus +12.8 / +11.4 for ReDraft, with similar retention (Table 4).
Only ReDraft learns both cold-start tasks (above 80%) while forgetting about one point or less. STaR is competitive on Clock (81.1 / −0.2) but fails on Jigsaw (18.9): without the draft, far fewer of its targets are correct and faithful (40.9% vs. 92.4% on Jigsaw and 14.8% vs. 27.6% on Clock, over 500 prompts judged by Claude Opus 5.5 Max). Expert revision (ReDraft*) beats SFT on both axes for both tasks, and self-revision matches it in acquisition without extra expert calls; expert revision is the better choice when the model is too small to revise itself (Sec. 7).