- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper compares sequential SFT and GRPO-style RL across four reasoning tasks. It reports greater forgetting with SFT, studies parameter-update geometry and gradient inner products as possible explanations, and proposes Parallel-RL, which merges independently trained task-specific RL updates with optional adaptation. In the reported setting, RL updates merge more successfully than SFT updates; the generality of the explanation and the method's incremental benefit remain open.
Critical strengths
-
The paper tackles an important multi-task post-training question and examines it through sequential training, single-task transfer, and four-task merging. These complementary experiments make the observed behavior relevant to practical skill composition.
-
A theoretical account of how SFT and RL differ is difficult and valuable to the community. The advantage-centering argument in Lemma 4.3 offers a concrete starting point for such an account.
Critical weaknesses
- The incremental contribution is unclear: [1] already studies SFT/RL model merging and update conflicts, while [2] compares their forgetting in continual post-training. Naive Parallel-RL trails sequential RL in Table 4; adaptation exceeds it by only 0.5 percentage points after extra training, without an equal-budget comparison.
- The proposed explanation is not yet established. Figure 2 reports three SFT update cosines (+0.968, −0.973, +0.968) that cannot belong to the same three vectors. Theorem 4.5 bounds gradient inner products under assumptions but does not show lower actual RL interference or forgetting, and its link to the design of Parallel-RL is indirect.
- The SFT/RL comparison does not isolate the training paradigm. Appendix A uses different datasets, learning rates, and frameworks. SFT uses teacher CoT, but its serialized targets, reasoning tags/prefixes, and answer-parsing failures are not specified; Appendix G also shows different training and evaluation prompts. Data and response format remain plausible contributors to the measured gap.
References:
[1] Zixuan Ren et al. “Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs.” ICLR 2026.
[2] Song Lai et al. “Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training.” ICML 2026.
Questions for authors
- What is the specific advance over [1, 2], and how does adapted Parallel-RL compare with sequential RL under the same additional data and compute budget?
- Please provide the original Figure 2 cosine matrix. After correction, what evidence connects Theorem 4.5's bound to actual forgetting or to the design of Parallel-RL?
- Does the SFT/RL trend hold in a limited same-prompt, comparable-data/token control? Please specify the SFT CoT target format, reasoning tags/prefixes,.
- While this does not constitute a specific weakness or a point for questioning, I believe the Qwen 1.5B/7B models distilled by DeepSeek used in the paper are already outdated. I would like to see more recent models, though this is not a requirement.
3 comments
Response 2
Weakness 2 & Question 2: Figure 2 Values and Connection Between Theorem 4.5 and Forgetting
(1) Verification and correction of the sign typo in Figure 2: Thank you for catching this mathematical inconsistency. After checking our raw data and plotting scripts, we confirmed this was a sign typo introduced when manually annotating the Figure 2 heatmap. The corrected true cosine similarity values are .
(2) Evidence chain linking Theorem 4.5 to empirical forgetting and Parallel-RL design: Our study connects theory and practice across four linked layers:
- Gradient structure (Lemma 4.3 & Theorem 4.5): Under explicit assumptions, advantage centering bounds RL's cross-task gradient inner product by the centered residual variance , whereas SFT is bounded by the uncentered norm product .
- Training statistics (Section 4.4 / Table 3): Direct measurements of and during real LLM training confirm a two-order-of-magnitude gap between these quantities, while Figure 2 confirms near-zero cross-task cosine and small update norms for RL.
- Forgetting behavior (Section 3 / Table 2): These weakly coupled, small-magnitude updates correspond directly to RL's strong retention of prior capabilities in sequential training and single-task transfer.
- Guidance for Parallel-RL (Section 5 & Appendix D.1): Weak cross-task update coupling enables direct parameter merging without severe interference, motivating Section 5. Conversely, when two tasks exhibit strong score-function residual overlap (violating low coupling), merging fails, which directly motivates our score-function-based prior task compatibility screening in Appendix D.1.
Response 3
Weakness 3 & Question 3: Training Data, Prompts, and Evaluation Formats for SFT and RL
(1) Teacher CoT supervision format for SFT and answer parsing (ruling out parsing failure): Thank you for raising this important question on experimental rigor. We fully agree that training and evaluation interfaces should be completely transparent so that format mismatches or answer parsing failures never penalize either method. We detail our setup below and will include the full specification in Appendix A & Appendix G:
- Serialized SFT targets and reasoning tags: To stay strictly aligned with the native distribution of
DeepSeek-R1-Distill-Qwen-1.5B/7B, our SFT teacher data uses the exact same reasoning-tag format as the base model:<think>\n{Teacher Reasoning Chain}\n</think>\n{Response Prefix & Final Answer}(with final answers following each benchmark's standard extraction format, such as\boxed{}for math/logic multiple-choice questions and standard code blocks for coding tasks). - Evidence ruling out format degradation or parsing failure as the cause of SFT's drop:
- During evaluation, both SFT and RL models use the exact same evaluation parser on every benchmark.
- More directly, single-task SFT results in Table 1 & Table 2 show that training on a single target task with this exact SFT format and hyperparameter setup reliably improves or maintains high accuracy on that task relative to the base model (with normal parsing behavior,
<2.1%failure rate). If the SFT format or prompt caused parsing failures, single-task SFT would collapse in the first stage. - Sharp performance drops occur specifically after cross-task transfer (off-diagonals of Table 2), multi-stage sequential training (Section 3), and multi-model parameter merging (Table 4), where large, coupled SFT gradient updates overwrite reasoning parameters from earlier tasks.
- Training vs. evaluation prompts in Appendix G: Some tasks use the concise dataset-native instructions during training, while evaluation uniformly uses the standard zero-shot evaluation prompt templates from open-source evaluation frameworks. Crucially, these evaluation templates are completely symmetric and identical across Base, SFT, and RL models. We will add a task-by-task mapping table of training prompts, evaluation prompts, and parsing rules to Appendix A / G in the revision.
(2) Controlled comparison with identical prompts and matched data/token budgets: As discussed above, SFT (offline token-level cross-entropy over long teacher CoT traces) and RL (online multi-rollout sampling from prompts only, updated via outcome rewards) differ fundamentally in supervision density and token consumption. RL must sample exploratory rollouts per prompt to compute relative advantages, whereas repeating the same fixed teacher trace times in SFT quickly causes severe overfitting (memorization). Our main experiments therefore adopt the mature, standard configurations for each paradigm (Appendix A). We agree that a small-scale diagnostic on identical prompt subsets would further isolate data effects; given the compute cost of redefining cross-paradigm token-budget matching and re-running multi-stage pipelines, we will explicitly discuss this comparison boundary and note it as a valuable fine-grained diagnostic direction in the revision.
Question 4: Transferability to Newer Models
Thank you for your understanding and helpful suggestion. We chose DeepSeek-R1-Distill-Qwen-1.5B and 7B because this family is widely used as a reproducible benchmark in the open-source reasoning community and provides two parameter scales under a unified tokenizer and architecture for our full suite of single-task, sequential, gradient, and merging comparisons. Moreover, our core finding that intra-group zero-sum advantage centering reduces cross-task gradient interference is a fundamental property of the loss function and optimization dynamics, rather than a quirk of a specific model generation. We will include validation on newer model families such as Qwen3 in our discussion of future work.
Thank you again for your constructive review. Clarifying our overlap and incremental advances over prior work, correcting Figure 2, and fully documenting our training and evaluation interfaces will make our paper's value as a systematic mechanistic study much clearer.
Response 1
Dear Reviewer 9tsW,
Thank you for recognizing the practical relevance of our problem, the complementarity of our sequential training, single-task transfer, and four-task merging experiments, and the potential value of our advantage-centering analysis. Your comments focus on differentiation from prior work, the theory-experiment connection, and the boundaries of our comparison setup. To clarify, our main contribution is neither claiming to be the first to observe that RL forgets less than SFT nor proposing a new parameter-merging operator; rather, it is building a cohesive chain of empirical observation, mechanistic explanation, conditional derivation, quantitative verification, and application exploration around this phenomenon.
Weakness 1 & Question 1: Overlap with Related Work and Significance of Parallel-RL
(1) Specific advances over recent/concurrent works [1] (Ren et al., ICLR 2026) and [2] (Lai et al., ICML 2026): Thank you for pointing out these two important recent works. We will add a detailed comparison in the revision. Their focus differs from our core contributions as follows:
(2) Interpreting the comparison between Parallel-RL and Sequential RL in Table 4: We appreciate your careful reading of Table 4 and would like to clarify our intended framing: