acceptodds
Under review as a conference paper at ICLR 2027

SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning Paradigms for Large Language Models

Abstract

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models. Our preliminary experiments revealed a phenomenon: SFT suffers from task conflicts under multi-stage training, whereas RL enables coexistence across diverse tasks. Empirically, we trace this to parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, improving efficiency and flexibility.

Primary area:
interpretability and explainable AI
Keywords:
Multi-Task Learning, Reasoning Enhancement, Reinforcement Learning, Supervised Fine-Tuning, Gradient Interference, Parallel Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9tsWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares sequential SFT and GRPO-style RL across four reasoning tasks. It reports greater forgetting with SFT, studies parameter-update geometry and gradient inner products as possible explanations, and proposes Parallel-RL, which merges independently trained task-specific RL updates with optional adaptation. In the reported setting, RL updates merge more successfully than SFT updates; the generality of the explanation and the method's incremental benefit remain open.

Critical strengths
  1. The paper tackles an important multi-task post-training question and examines it through sequential training, single-task transfer, and four-task merging. These complementary experiments make the observed behavior relevant to practical skill composition.

  2. A theoretical account of how SFT and RL differ is difficult and valuable to the community. The advantage-centering argument in Lemma 4.3 offers a concrete starting point for such an account.

Critical weaknesses
  1. The incremental contribution is unclear: [1] already studies SFT/RL model merging and update conflicts, while [2] compares their forgetting in continual post-training. Naive Parallel-RL trails sequential RL in Table 4; adaptation exceeds it by only 0.5 percentage points after extra training, without an equal-budget comparison.
  2. The proposed explanation is not yet established. Figure 2 reports three SFT update cosines (+0.968, −0.973, +0.968) that cannot belong to the same three vectors. Theorem 4.5 bounds gradient inner products under assumptions but does not show lower actual RL interference or forgetting, and its link to the design of Parallel-RL is indirect.
  3. The SFT/RL comparison does not isolate the training paradigm. Appendix A uses different datasets, learning rates, and frameworks. SFT uses teacher CoT, but its serialized targets, reasoning tags/prefixes, and answer-parsing failures are not specified; Appendix G also shows different training and evaluation prompts. Data and response format remain plausible contributors to the measured gap.

References:

[1] Zixuan Ren et al. “Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs.” ICLR 2026.

[2] Song Lai et al. “Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training.” ICML 2026.

Questions for authors
  1. What is the specific advance over [1, 2], and how does adapted Parallel-RL compare with sequential RL under the same additional data and compute budget?
  2. Please provide the original Figure 2 cosine matrix. After correction, what evidence connects Theorem 4.5's bound to actual forgetting or to the design of Parallel-RL?
  3. Does the SFT/RL trend hold in a limited same-prompt, comparable-data/token control? Please specify the SFT CoT target format, reasoning tags/prefixes,.
  4. While this does not constitute a specific weakness or a point for questioning, I believe the Qwen 1.5B/7B models distilled by DeepSeek used in the paper are already outdated. I would like to see more recent models, though this is not a requirement.
3 comments
AuthorsOpenReview ↗

Response 1

Dear Reviewer 9tsW,

Thank you for recognizing the practical relevance of our problem, the complementarity of our sequential training, single-task transfer, and four-task merging experiments, and the potential value of our advantage-centering analysis. Your comments focus on differentiation from prior work, the theory-experiment connection, and the boundaries of our comparison setup. To clarify, our main contribution is neither claiming to be the first to observe that RL forgets less than SFT nor proposing a new parameter-merging operator; rather, it is building a cohesive chain of empirical observation, mechanistic explanation, conditional derivation, quantitative verification, and application exploration around this phenomenon.

Weakness 1 & Question 1: Overlap with Related Work and Significance of Parallel-RL

(1) Specific advances over recent/concurrent works [1] (Ren et al., ICLR 2026) and [2] (Lai et al., ICML 2026): Thank you for pointing out these two important recent works. We will add a detailed comparison in the revision. Their focus differs from our core contributions as follows:

  • Comparison with Ren et al. [1]: [1] focuses primarily on post-training model merging conflicts, explaining empirically why RL is more amenable to model merging from the perspectives of on-policy data distributions, reduced update magnitudes, and positive/negative sample optimization.
  • Comparison with Lai et al. [2]: [2] focuses on catastrophic forgetting and knowledge retention in continual post-training (primarily on multimodal tasks), emphasizing selective update and sample filtering strategies.
  • Our distinct incremental contributions:
    1. A unified cross-scenario mechanistic chain: Targeting multi-task LLM reasoning post-training, we connect single-task cross-domain transfer (Table 2), multi-stage sequential training (Section 3), and parallel multi-model merging (Section 5) within one framework, forming a complete evidence chain from empirical phenomena and parameter geometry (Figure 2) to gradient decomposition (Lemma 4.3), conditional interference bounds (Theorem 4.5), and real-training magnitude verification (Section 4.4 / Table 3).
    2. Showing how intra-group zero-sum advantage centering alters the controlling quantity of cross-task coherence: Going beyond attributing RL's stability solely to on-policy sampling, Lemma 4.3 and Theorem 4.5 show the synergy between zero-sum advantage centering in GRPO and on-policy shared-context sampling. Centering cancels the large shared semantic/format component μ\mu across tasks, shifting the cross-task gradient inner-product bound from SFT's Norm-Limited (MiMjM_i M_j) regime to RL's Variance-Limited (ViVjV_i V_j) regime.
    3. Verifiable prior criterion for task coexistence and boundary analysis (Appendix D): Building on our score-function analysis, Appendix D.1–D.2 answers when RL tasks cannot coexist, proposing a few-sample score-function geometric overlap criterion for prior task screening along with RL-specific tuning insights, which are not covered in [1, 2].

(2) Interpreting the comparison between Parallel-RL and Sequential RL in Table 4: We appreciate your careful reading of Table 4 and would like to clarify our intended framing:

  • First, in our study, Sequential RL is not a weak negative baseline; on the contrary, the fact that Sequential RL stably retains and improves performance across all four stages is key positive evidence supporting our core thesis that RL coexists.
  • Second, Parallel-RL is a simple, direct application exploration of our mechanistic analysis, rather than the sole algorithmic claim of the paper. The practical bottleneck of Sequential RL is long multi-stage wall-clock time. Section 5 therefore tests whether weak cross-task update coupling allows us to decompose sequential training into fully independent parallel runs and recombine task capabilities via parameter merging.
  • As Table 4 shows, SFT collapses after parallel merging, whereas Naive Parallel-RL already retains most multi-task gains with zero additional training. Adding lightweight calibration on just 5% of the original training sets allows Adapted Parallel-RL to match and slightly exceed (+0.5%) Sequential RL.
  • We fully accept your rigorous point: without an iso-budget control that gives Sequential RL the same 5% extra training steps, we will not use a 0.5% margin to claim that Adapted Parallel-RL unconditionally outperforms Sequential RL in accuracy. What this experiment actually demonstrates is that independent parallel RL training, parameter merging, and minimal calibration can match the multi-task performance of lengthy multi-stage sequential RL. We will tighten our wording in the revision to reflect this feasibility conclusion accurately.
AuthorsOpenReview ↗

Response 2

Weakness 2 & Question 2: Figure 2 Values and Connection Between Theorem 4.5 and Forgetting

(1) Verification and correction of the sign typo in Figure 2: Thank you for catching this mathematical inconsistency. After checking our raw data and plotting scripts, we confirmed this was a sign typo introduced when manually annotating the Figure 2 heatmap. The corrected true cosine similarity values are (+0.968,+0.973,+0.968)(+0.968, +0.973, +0.968).

(2) Evidence chain linking Theorem 4.5 to empirical forgetting and Parallel-RL design: Our study connects theory and practice across four linked layers:

  1. Gradient structure (Lemma 4.3 & Theorem 4.5): Under explicit assumptions, advantage centering bounds RL's cross-task gradient inner product by the centered residual variance ViVjV_i V_j, whereas SFT is bounded by the uncentered norm product MiMjM_i M_j.
  2. Training statistics (Section 4.4 / Table 3): Direct measurements of ∥S∥\|S\| and ∥δS∥\|\delta S\| during real LLM training confirm a two-order-of-magnitude gap between these quantities, while Figure 2 confirms near-zero cross-task cosine and small update norms for RL.
  3. Forgetting behavior (Section 3 / Table 2): These weakly coupled, small-magnitude updates correspond directly to RL's strong retention of prior capabilities in sequential training and single-task transfer.
  4. Guidance for Parallel-RL (Section 5 & Appendix D.1): Weak cross-task update coupling enables direct parameter merging without severe interference, motivating Section 5. Conversely, when two tasks exhibit strong score-function residual overlap (violating low coupling), merging fails, which directly motivates our score-function-based prior task compatibility screening in Appendix D.1.
AuthorsOpenReview ↗

Response 3

Weakness 3 & Question 3: Training Data, Prompts, and Evaluation Formats for SFT and RL

(1) Teacher CoT supervision format for SFT and answer parsing (ruling out parsing failure): Thank you for raising this important question on experimental rigor. We fully agree that training and evaluation interfaces should be completely transparent so that format mismatches or answer parsing failures never penalize either method. We detail our setup below and will include the full specification in Appendix A & Appendix G:

  • Serialized SFT targets and reasoning tags: To stay strictly aligned with the native distribution of DeepSeek-R1-Distill-Qwen-1.5B/7B, our SFT teacher data uses the exact same reasoning-tag format as the base model: <think>\n{Teacher Reasoning Chain}\n</think>\n{Response Prefix & Final Answer} (with final answers following each benchmark's standard extraction format, such as \boxed{} for math/logic multiple-choice questions and standard code blocks for coding tasks).
  • Evidence ruling out format degradation or parsing failure as the cause of SFT's drop:
    1. During evaluation, both SFT and RL models use the exact same evaluation parser on every benchmark.
    2. More directly, single-task SFT results in Table 1 & Table 2 show that training on a single target task with this exact SFT format and hyperparameter setup reliably improves or maintains high accuracy on that task relative to the base model (with normal parsing behavior, <2.1% failure rate). If the SFT format or prompt caused parsing failures, single-task SFT would collapse in the first stage.
    3. Sharp performance drops occur specifically after cross-task transfer (off-diagonals of Table 2), multi-stage sequential training (Section 3), and multi-model parameter merging (Table 4), where large, coupled SFT gradient updates overwrite reasoning parameters from earlier tasks.
  • Training vs. evaluation prompts in Appendix G: Some tasks use the concise dataset-native instructions during training, while evaluation uniformly uses the standard zero-shot evaluation prompt templates from open-source evaluation frameworks. Crucially, these evaluation templates are completely symmetric and identical across Base, SFT, and RL models. We will add a task-by-task mapping table of training prompts, evaluation prompts, and parsing rules to Appendix A / G in the revision.

(2) Controlled comparison with identical prompts and matched data/token budgets: As discussed above, SFT (offline token-level cross-entropy over long teacher CoT traces) and RL (online multi-rollout sampling from prompts only, updated via outcome rewards) differ fundamentally in supervision density and token consumption. RL must sample GG exploratory rollouts per prompt to compute relative advantages, whereas repeating the same fixed teacher trace GG times in SFT quickly causes severe overfitting (memorization). Our main experiments therefore adopt the mature, standard configurations for each paradigm (Appendix A). We agree that a small-scale diagnostic on identical prompt subsets would further isolate data effects; given the compute cost of redefining cross-paradigm token-budget matching and re-running multi-stage pipelines, we will explicitly discuss this comparison boundary and note it as a valuable fine-grained diagnostic direction in the revision.

Question 4: Transferability to Newer Models

Thank you for your understanding and helpful suggestion. We chose DeepSeek-R1-Distill-Qwen-1.5B and 7B because this family is widely used as a reproducible benchmark in the open-source reasoning community and provides two parameter scales under a unified tokenizer and architecture for our full suite of single-task, sequential, gradient, and merging comparisons. Moreover, our core finding that intra-group zero-sum advantage centering reduces cross-task gradient interference is a fundamental property of the loss function and optimization dynamics, rather than a quirk of a specific model generation. We will include validation on newer model families such as Qwen3 in our discussion of future work.

Thank you again for your constructive review. Clarifying our overlap and incremental advances over prior work, correcting Figure 2, and fully documenting our training and evaluation interfaces will make our paper's value as a systematic mechanistic study much clearer.

Reviewer_ynm1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyses the differences of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) in enhancing multi-task reasoning for large language models. The authors analyze the reasons behind SFT's poor performance and RL's success in multi-stage training. They conduct an empirical study and show that the differences can be traced to the parameter level, in particular to the geometry of gradient updates. To support the experiments, the authors provide theoretical grounding. They introduce the "interference" term to characterize the influence of training on one task on the other tasks. Further, upper bounds of interference are provided for both SFT and RL, revealing that interference in SFT is norm-limited, whereas interference in RL is variance-limited.

Based on the findings about RL gradient update behavior, the paper suggests a new algorithm, Parallel-RL, and its modification Adapted Parallel-RL. The main idea is to train the model in parallel on tasks with low interference and then merge the parameter updates. The authors report that this makes the training process more efficient while maintaining quality, and that Adapted Parallel-RL improves performance compared to the other methods considered in the paper.

Critical strengths
  1. The paper explores a relevant direction: optimization of post-training. The authors highlight that while SFT collapses under multi-stage training, RL post-training techniques are able to benefit from it. Previous literature has already mentioned that SFT tends to memorize information while RL generalizes, and that SFT requires training on a mixed dataset containing different tasks to avoid forgetting skills. Nevertheless, the authors provide novel explanations of the performance differences between SFT and RL in post-training, supporting the conclusions from both empirical and theoretical sides.

  2. The empirical study provided in the paper is very detailed. Within the chosen setup, the experiments contain multiple scenarios and analyses: direct accuracy comparison, cosine-similarity and L2-norm analysis of gradient updates, pass@K analysis, and ablations of the merging process. The paper also includes geometrical intuition and visualizations that are valuable for understanding the idea.

Critical weaknesses
  1. The theoretical part of the work relies on several assumptions: Assumption 4.4 , the assumption of task independence, and the assumption that M_i is large and V_i is small. Their feasibility is not analyzed in the paper. Moreover, they are not stated with the same rigor: Assumption 4.4 is formalized explicitly, but task independence is used as a fact in Section 4.2 without being formalized as an assumption, and the claim that M_i is large and V_i is small is presented as a fact although it is based on empirical results for a particular model family and task set. Since Theorem 4.5 holds only under Assumption 4.4, and the qualitative analysis of Section 4.2 additionally relies on task independence, the lack of discussion of when these premises hold makes it unclear when the theoretical conclusions apply. Moreover, in Appendix the authors highlight that not all tasks are independent, violating the task independence assumption, and this violation is shown to deteriorate Parallel-RL performance. Finally, based on the two inequalities in Theorem 4.5, the authors suggest that SFT "depends" on M_i and RL is "controlled" by V_i, which does not directly follow from the inequalities, as they are only upper bounds. So the intuition about SFT interference being large and RL interference being small is not fully confirmed from the theoretical side.

  2. The central observation that RL parameter updates from different tasks are nearly orthogonal is not established as a property of the tasks. A small value alone does not show that the tasks learned separate directions. As the updates are extremely small and sparse: the cosine can be near zero automatically. The paper itself argues that sparse high-dimensional vectors are near-orthogonal in general, which makes it difficult to use orthogonality as evidence of task-specific separation. A simple check is missing: train the same task twice with different random seeds and report the cosine between the two resulting updates. If it is much higher than the cross-task value, the finding is genuinely task-specific, if it is also very small then any two RL updates look orthogonal to each other and the observation says no much about tasks.

Questions for authors
  1. In your theoretical reasoning you rely on several assumptions (see Weakness 1). Could you please provide a discussion of their fulfillment: how often do Assumption 4.4 and task independence actually hold in real-world LLM training, and what happens if they are violated?

  2. Could you please provide a detailed step-by-step technical description of both Parallel-RL and Adapted Parallel-RL? For Parallel-RL, details on task selection are unclear, while for the Adapted version a full description (adaptation data, learning rate, number of steps) would be beneficial.

  3. The RL baseline simply pools all tasks and trains on a uniform mixture. Authors themselves note that this can cause gradient imbalance, i.e., some tasks dominate the training signal while others end up under-optimized. Recent methods such as https://arxiv.org/abs/2602.05547 address exactly this problem within a single training run. Parallel-RL instead trains N models independently and merges them, so a comparison in terms of both accuracy and total training compute would clarify whether its advantage comes from the parallel training scheme itself, rather than from the weakness of the uniform-mixing baseline. Could you provide such a comparison?

4 comments
AuthorsOpenReview ↗

Response 1

Dear Reviewer ynm1,

Thank you for your thorough review and positive assessment. We are glad you recognize the importance of this research direction, our empirical and theoretical explanation of the multi-task performance gap between SFT and RL in post-training, and the breadth of our experiments (accuracy comparisons, gradient cosine similarity and L2 norm analysis, Pass@K evaluation, merging ablations, and geometric visualizations). We would like to emphasize that our paper does not deduce theory from idealized assumptions alone; instead, it follows a complete pipeline covering empirical phenomena, mechanistic analysis, conditional derivations, empirical measurements, and failure-case / task-selection analysis. Below, we respond to your specific questions.

Weakness 1 & Question 1: Theoretical Assumptions, Validity Conditions, and Behavior When Violated

This is an important theoretical boundary to clarify. First, Assumption 4.4, relative task independence, and the empirical magnitudes of MiM_i and ViV_i are three distinct levels of statements and should not be conflated as unconditional facts:

  1. Role of each premise and why the upper bound is explanatory:

    • Assumption 4.4 (bounded moments): This is a standard mathematical regularity condition needed to derive Theorem 4.5 (full proof in Appendix C), characterizing the structural difference between the two gradient inner-product upper bounds (SFT bounded by the uncentered second moment MiMjM_i M_j, and RL bounded by the intra-group centered residual variance ViVjV_i V_j).
    • Large MiM_i vs. small ViV_i is a measurable empirical property: As you noted, an abstract inequality alone cannot establish actual interference magnitudes. We therefore do not treat this as an axiom; in Section 4.4 / Table 3, we directly measure the raw score-function norm ∥S∥\|S\| (corresponding to MiM_i) and the intra-group centered residual ∥δS∥\|\delta S\| (corresponding to ViV_i) during real LLM training, confirming a two-order-of-magnitude gap.
    • Added scale-invariant cosine bounds: To address the limitation of relying solely on inner-product bounds, we also derived expectation bounds on the scale-invariant absolute cosine similarity (see our response to Reviewer 1, Weakness 2): SFT has a high-coupling lower bound near 11, E[∣cos⁡∣]≥1−O(η+δ)\mathbb{E}[|\cos|] \ge 1 - \mathcal{O}(\eta+\delta), whereas RL's upper bound is controlled by the inter-task residual coupling γi,j\gamma_{i,j}, namely E[∣cos⁡∣]≤O(γi,jG)\mathbb{E}[|\cos|] \le \mathcal{O}(\gamma_{i,j} G). We will revise our wording in the main text to avoid presenting theoretical upper bounds as unconditional sufficient conditions, and instead emphasize the joint explanatory chain formed by the structural bounds and the empirical magnitudes in Table 3.
  2. When do these conditions hold in practice, and what happens when they are violated?

    • When different reasoning tasks (such as Math, Science, Logic, and Code in the main text) have relatively decoupled task-specific reasoning paths and output formats (small residual coupling γi,j\gamma_{i,j}), or distinct score-function distributions (as in Figure 3), task independence holds approximately, and RL exhibits strong coexistence and mergeability.
    • Behavior when violated and consistency with Appendix D.1: As you observed, not all tasks satisfy independence in real training, which is precisely why we designed and reported a failure case in Appendix D.1. When two tasks have direct, severe conflicts in output format or optimization direction, task independence breaks down, cross-task cosine is no longer near 0, and naive Parallel-RL merging degrades performance.
    • Practical value of defining the boundary: Thus, Appendix D.1 does not contradict the main theory; rather, it defines the operating boundary of both the theory and Parallel-RL. This directly motivates our prior task-selection mechanism based on few-sample score-function geometric overlap (Appendix D.1; comparing Figures 3, 4, and 6 identifies and filters out conflicting tasks in advance). Appendix D.2 further analyzes how single-task training settings affect merging behavior. We will highlight these boundary analyses more prominently in the main text.
AuthorsOpenReview ↗

Response 2

Weakness 2: Interpretation of Same-Task vs. Cross-Task Cosine

Your suggestion of training the same task with different random seeds and comparing same-task vs. cross-task update cosine similarity is a valuable perspective. We offer two clarifications:

  1. Distinguishing observed low cross-task coupling from claiming exclusive task-specific semantic subspaces: We agree with your rigorous point that in high-dimensional parameter space, observing near-zero cross-task cosine is jointly influenced by high-dimensional geometry, small update magnitudes, and sparsity, and cannot by itself prove that each task occupies a unique, mutually exclusive semantic subspace. However, from the perspective of multi-task post-training and parameter merging, what physically determines whether task A's parameter update ΔWA\Delta W_A disrupts task B during merging or sequential training is the actual inner product and geometric angle between the two update vectors in parameter space. Within the exact same high-dimensional model space, SFT's cross-task updates are highly collinear (cos⁡>0.9\cos > 0.9, Figure 2) and cause severe forgetting, whereas RL's updates, constrained by zero-sum advantage centering and variance bounds, maintain minimal directional overlap (cos⁡≈0\cos \approx 0) and conservative norms, physically avoiding destructive weight interference.
  2. Intra-task feature clustering and planned revision: In Section 4.4 (Figures 3, 4, and 6) and Appendix D.1, we also visualize score-function distributions across sample subsets within each task. Gradient features from the same task cluster in coherent geometric regions (and single-task training reliably boosts performance on that task in Table 1 & Table 2, confirming that updates carry genuine task-specific optimization signal rather than random noise), while compatible distinct tasks separate clearly. We welcome your suggestion and will revise our text to clearly distinguish between observing near-orthogonal / weakly coupled cross-task updates and claiming strictly disjoint task-specific subspaces, making our geometric intuition more precise.

Question 2: Complete Algorithm and Reproducibility Details for Parallel-RL and Adapted Parallel-RL

Thank you for this concrete suggestion. While these details appear across Section 5.1–5.2, Appendix A.2, and Appendix D.1–D.2, we summarize the full pipeline and hyperparameters below to ensure full reproducibility:

  1. Step 1: Prior task compatibility screening (Task Selection; Section 5.1 & Appendix D.1): For each candidate task, compute score-function features using the initial base model W0W_0 on a small randomly sampled subset, and inspect cross-task overlap via cosine similarity and low-dimensional projections (Figures 3, 4, and 6) to filter out tasks with clear output-format or gradient-direction conflicts (such as SynLogic in Appendix D.1).

  2. Step 2: Independent parallel RL training (Independent RL; Section 5.1 & Appendix A): For the NN selected compatible tasks (Math, MMLU, KK, and LiveCodeBench), initialize from the same base model W0W_0 and run standard GRPO independently in parallel on each task's training set to obtain task-specific parameter updates ΔWi=Wi−W0\Delta W_i = W_i - W_0.

  3. Step 3: Parameter merging (Naive Parallel-RL; Section 5.2 & Appendix A.2): Merge the task updates back into the base model via a merging operator M(⋅)\mathcal{M}(\cdot): Wmerged=W0+M(ΔW1,…,ΔWN)W_{\text{merged}} = W_0 + \mathcal{M}(\Delta W_1, \dots, \Delta W_N). In Table 4, we compare four standard operators: direct summation (Sum), averaging (Mean), TIES-Merging (using the original paper's default density trimming and sign-election hyperparameters; Appendix A.2), and SVD-based low-rank merging (retaining only the Rank-1 leading singular vector for reconstruction; Section 5.2).

  4. Step 4: Post-merge adaption (Adapted Parallel-RL; Section 5.2):

    • Data source, size, and sampling: No external dataset is used. We draw 5% of each of the NN tasks' original training sets via uniform random sampling and combine them into a mixed calibration set.
    • Choice of the 5% ratio: In preliminary experiments, we compared 2.5%, 5%, and 10% sampling ratios on the 1.5B model. On MATH500 / MMLU / KK / LCB, Naive Sum scores 85.2 / 46.5 / 40.0 / 16.9; 2.5% adaptation reaches 85.6 / 46.9 / 42.0 / 17.1; 5% adaptation improves markedly to 86.6 / 48.3 / 45.0 / 17.3; and doubling to 10% yields 86.8 / 48.9 / 46.0 / 17.3, where gains saturate. Thus, 5% offers the best trade-off between calibration quality and minimal compute.
    • Adaptation hyperparameters: Starting from WmergedW_{\text{merged}}, we reuse the exact same RL training hyperparameters as the single-task stage (the same GRPO optimizer, learning rate, batch size, group size GG, and epoch setup; see Appendix A.1) on this 5% mixed subset, so the adaptation step count is only 5% of full mixed-task training.
AuthorsOpenReview ↗

Response 3

Question 3: Mixed-Task RL Baselines, MT-GRPO, and Compute Budgets

We agree that methods such as MT-GRPO mitigate gradient imbalance in mixed-task RL through dynamic task weighting and sampling. Here, it is helpful to distinguish two research questions: MT-GRPO studies task balancing within a single mixed-training run, whereas Parallel-RL studies whether independently trained task policies can be effectively combined afterward. MT-GRPO focuses on balancing task gradients inside a single centralized mixed-training process, while our paper focuses on revealing the fundamental mechanistic difference in cross-task interference between SFT and RL and using Parallel-RL to test whether RL updates trained independently with zero mid-training interaction can coexist via post-hoc parameter merging. Accordingly, Sections 3 and 5 use standard Uniform-Mixture RL and Sequential RL as clean baselines to observe the intrinsic properties of each paradigm.

In Section 5, we evaluate multiple merging operators, adaptation, and task-removal ablations to test the feasibility of post-hoc capability integration. The two approaches are also potentially complementary: dynamic task weighting could conceptually be applied during post-merge joint adaptation, though we did not test that combination here. We acknowledge that a strict iso-budget comparison with MT-GRPO would further illuminate performance-compute trade-offs across training organizations, but it would require re-matching multi-task data, training steps, and total GPU-hours, which is a substantial separate study. We will add a discussion of MT-GRPO and this limitation, and clearly distinguish wall-clock time from total compute to avoid equating parallel training with automatic compute savings. Our main contribution does not hinge on outperforming all existing multi-task RL optimizers.

Thank you for your specific suggestions. Incorporating these conditions, failure cases, and reproducibility details more explicitly into the main text will make the scope and mechanistic contributions of our work much clearer.

Thank you for the response, which fully answers Q2. However, the same-task, different-seed control (Weakness 2) was not run, and the reported RL cosines (about 1e-5) are what random vectors of this size would give. The theory’s assumptions (Weakness 1) also remain informally justified. I therefore keep my score.

Reviewer_mpaQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares supervised fine-tuning (SFT) and reinforcement learning (RL) for multi-task post-training of large language models. Sequential SFT is found to induce cross-task performance degradation, while RL retains better overall task performance. The authors analyze parameter update scales and cross-task similarities, offering a theoretical analysis via advantage normalization, on-policy optimization and gradient interference bounds. Building on these insights, the authors further propose Parallel-RL, which runs independent RL procedure for each task and merges the resulting total updates into a whole after training. Experiments confirm its competitive multi-task performance and analyse its task interference.

Critical strengths
  1. The paper considers a practically important perspective of cross-task interference in multi-task LLM post-training, which facilitates a better understanding of how SFT and RL affect previously acquired capabilities.

  2. It investigates task interference from performance, parameter geometry, and gradients. The advantage-centering decomposition provides a useful analytical perspective on how RL gradients differ from supervised gradients.

  3. The proposed framework explores independent task-specific RL training and post-hoc model merging, offering potential in multi-task learning.

  4. The experiments include multiple merging strategies, additional training configurations, and task-removal ablations.

Critical weaknesses
  1. The central claim that SFT causes task conflicts while RL enables task coexistence is not sufficiently established as a general phenomenon. The comparisons involve different training datasets, learning rates, and substantially different parameter update magnitudes, making it difficult to attribute the observed differences to the intrinsic properties of SFT and RL. More controlled comparisons across training configurations, task orders, and model families are necessary to establish the robustness and scope of this claim. Moreover, in the prior work RL's Razor: Why Online Reinforcement Learning Forgets Less, the discussion on the sparsity of parameter updates attributes the sparsity of RL to the "bfloat16" numerical scale. In other words, changing "bfloat16" into "float32" hurts the sparsity of parameter updates in RL multi-task post-training.

  2. The main theorem merely bounds unnormalized gradient inner products and does not prove near-orthogonality or reduced forgetting. Crucially, the assumption that RL produces small score-function residuals require justification. Moreover, no rigorous connection is established between these gradient-level bounds and the preservation of task performance after sequential training or parameter merging.

  3. The claimed parallelization benefits are not convincingly distinguished from standard distributed multi-task RL. Besides, the paper also lacks sufficient wall-clock time and computational cost evaluations to substantiate its claimed efficiency advantages.

Questions for authors

See above.

4 comments
AuthorsOpenReview ↗

Response 1

Dear Reviewer mpaQ,

Thank you for your careful review and for recognizing the importance of multi-task post-training, our systematic analysis across performance, parameter geometry, and gradients, as well as the practical potential of Parallel-RL and our ablation studies. To clarify, our main contribution is to explain the differences between the two post-training paradigms in the studied multi-task reasoning setting through a complete chain: empirical phenomena, mechanistic analysis, conditional theoretical derivations, quantitative validation, and an initial application. Below, we address each of your points.

Weakness 1: Experimental Controls, Training Order, and Numerical Precision

(1) Different training data, learning rates, and frameworks for SFT and RL. We would like to distinguish between making the two paradigms fairly comparable on the same tasks and forcing them to use identical training recipes. SFT fits teacher-provided CoT traces via supervised learning, whereas RL samples from the current policy and optimizes with verifiable rewards. The two paradigms naturally differ in supervision objectives, data distributions, learning rates, and training stability. Using the same learning rate does not ensure fairness and can instead prevent one method from training effectively. Moreover, differences in parameter update magnitudes are not merely confounders to eliminate; they are part of the optimization mechanism this paper seeks to explain. Accordingly, in Appendix A, we follow well-established, standard configurations validated by the community for each paradigm (including datasets, learning rates, training frameworks, and evaluation benchmarks) to compare how two genuinely effective post-training paradigms behave in multi-task settings.

(2) Further controls on training orders, models, and configurations. Our main text and appendices already build a comprehensive experimental suite: across two model scales (1.5B and 7B) and four core reasoning tasks (Math: MATH500, Science: MMLU, Logic: KK, Code: LiveCodeBench), we systematically evaluate single-task cross-domain transfer (Table 2), multi-stage sequential training (Section 3), mixed-task training comparisons, parameter geometry and gradient analysis (Figure 2 & Section 4.4), four parameter merging strategies (Sum, Mean, TIES, and SVD; Section 5 & Table 4), and task-removal ablations (Appendix D). Our conclusions do not rest on a single isolated curve. With four tasks, enumerating all training orders alone requires 4!=244! = 24 full multi-stage runs; running both SFT and RL would require at least 48 full training pipelines, and adding more model families or scales would multiply this cost further, which exceeds the time and compute available during the rebuttal period.

(3) The bf16 sparsity issue in RL's Razor. This is an important distinction. That work notes that exact zero parameter updates observed under bf16 may stem from rounding precision; switching to float32 removes the same exact sparsity while leaving task performance largely unchanged. Accordingly, we do not treat exact zeros from bf16 as a necessary mechanism for capability retention in RL. Our primary theoretical basis is the advantage-weighted centering structure in Lemma 4.3 and the gradient inner-product upper bound in Theorem 4.5, paired with the quantitative analysis of score-function norms and centered residuals in Section 4.4 / Table 3. None of these arguments assume that parameter updates are strictly zero. Due to the zero-sum centering of group-relative advantages, RL gradient updates are bounded by the small intra-group residual variance (Variance-Limited), whereas SFT gradients are dominated by the uncentered full score-function norm (Norm-Limited). The quantitative measurements in Section 4.4 / Table 3 confirm a two-order-of-magnitude gap between the score-function norm ∥S∥\|S\| and the centered residual ∥δS∥\|\delta S\| during actual training, and this conclusion still holds under float32.

AuthorsOpenReview ↗

Response 2

Weakness 2: Relationship Between Theoretical Upper Bounds and Empirical Performance

We fully agree with your rigorous assessment: an unnormalized inner-product upper bound alone neither equates directly to directional orthogonality nor constitutes a global closed-form theorem guaranteeing zero forgetting in deep non-convex multi-stage training. However, Theorem 4.5 is not meant to carry that burden in isolation. Below, we address your concern by adding a scale-invariant directional cosine derivation and clarifying our three-layer evidence chain.

(1) Scale-invariant derivation of absolute cosine similarity ∣cos⁡(gi,gj)∣|\cos(g_i, g_j)| (directly showing directional decoupling): In the main text, we analyze the inner product ∣I(i,j)∣=∣⟨gi,gj⟩∣|I(i,j)| = |\langle g_i, g_j \rangle| because actual cross-task interference during parameter updates depends on both direction (angle) and step size (magnitude). To show that our conclusion stems not only from RL's smaller gradient norm but also from directional decoupling (near-orthogonality), we add an expectation bound on the scale-invariant absolute cosine similarity (the full derivation will be included in the revision):

  • In high-dimensional parameter space, decompose the score function into a large shared semantic/format component μ\mu across tasks (e.g., general syntax and CoT structure) and a task- and sample-specific high-dimensional residual ϵ\epsilon, i.e., S(x)=μ+ϵS(x) = \mu + \epsilon. Assume that with high probability (1−δ)(1-\delta), ∥ϵ∥≤σ=η∥μ∥\|\epsilon\| \le \sigma = \eta \|\mu\| (η≪1\eta \ll 1), and define the upper bound on residual coupling between tasks ii and jj as max⁡k,m∣⟨ϵi,k,ϵj,m⟩∣σ2≤γi,j\max_{k,m} \frac{|\langle \epsilon_{i,k}, \epsilon_{j,m} \rangle|}{\sigma^2} \le \gamma_{i,j}.
  • For SFT (high-interference lower bound): E[∣cos⁡(giSFT,gjSFT)∣]≥1−2η−η2(1+η)2(1−δ)=1−O(η+δ)\mathbb{E}\left[\left|\cos(g_i^{\text{SFT}}, g_j^{\text{SFT}})\right|\right] \ge \frac{1 - 2\eta - \eta^2}{(1+\eta)^2}(1-\delta) = 1 - \mathcal{O}(\eta + \delta) Off-policy SFT cannot cancel the shared main direction μ\mu. Regardless of how distinct two tasks are (how small γi,j\gamma_{i,j} is), their gradient directions remain strongly pulled toward μ\mu, leading to severe cross-task conflict.
  • For RL (low-interference upper bound): E[∣cos⁡(giRL,gjRL)∣]≤O(γi,jG)\mathbb{E}\left[\left|\cos(g_i^{\text{RL}}, g_j^{\text{RL}})\right|\right] \le \mathcal{O}(\gamma_{i,j} G) By the zero-sum property of advantages in Lemma 4.3, the shared component μ\mu cancels exactly (∑kAi,kμ=0\sum_k A_{i,k}\mu = 0), leaving RL's directional interference strictly bounded by the intrinsic residual coupling γi,j\gamma_{i,j}. When two tasks are semantically distinct (small γi,j\gamma_{i,j}), RL updates become nearly orthogonal in direction.

(2) Basis for small RL residuals (ViV_i) and the three-layer evidence chain from gradient bounds to performance retention: For billion-parameter deep non-convex LLMs, deriving a necessary-and-sufficient theorem that maps local gradient bounds directly to final multi-step accuracy (as in classical convex optimization) is not realistic. Instead, we construct a three-layer closed-loop evidence chain:

  1. Mechanism and conditional derivation (Section 4.2–4.3 & Appendix C): Lemma 4.3 and Theorem 4.5 reveal the structural mathematical difference between the two objectives. Intra-group zero-sum advantage centering and on-policy shared-context sampling shift the controlling quantity of cross-task interference from the uncentered norm product MiMjM_i M_j to the centered residual variance ViVjV_i V_j.
  2. Quantitative validation during actual training (Section 4.4 & Table 3): We do not treat small centered residual ViV_i as an unverified axiom. In Section 4.4 / Table 3, we directly measure the raw score-function norm ∥S∥\|S\| and the intra-group centered residual ∥δS∥\|\delta S\| during real LLM training. The measurements confirm that because GG rollouts sampled from the same prompt share context, the centered residual is indeed two orders of magnitude smaller than the raw gradient norm.
  3. Post-training empirical corroboration (Section 3 & Section 5): The theory explains the structural mechanism, Table 3 & Figure 2 verify the gradient and parameter-geometry magnitudes during training, and Table 2 & Table 4 show final performance retention after sequential training and model merging. Together, these three layers mutually reinforce one another. In the revision, we will tighten our wording to clearly distinguish structural theoretical upper bounds from empirical performance observations.
AuthorsOpenReview ↗

Response 3

Weakness 3: Parallel-RL vs. Distributed Multi-Task RL and Efficiency Evidence

We would like to clarify that the two differ in training organization. Standard distributed mixed-task RL aggregates gradients from different tasks into a single policy optimization trajectory, which still requires handling task sampling and gradient imbalance. In contrast, Parallel-RL starts from the same base model, runs independent RL training on each task, and combines task capabilities via parameter merging followed by lightweight post-merge adaptation (Section 5.1–5.2, Appendix D). It is a direct application of our low cross-task coherence finding and allows independent hyperparameter tuning, task selection, and training. Specifically, each task can use its own rollout length, reward verifier, and compute allocation without coordinating heterogeneous environments within a single mixed-task batch. Under suitable deployment conditions, this task-level decoupling can help mitigate cross-task synchronization overhead and straggler delays from slow verifiers, though we will not claim these system overheads are completely eliminated without runtime measurements.

Regarding efficiency, we distinguish between wall-clock latency and total compute. When tasks run concurrently on separate devices, ideal completion time drops from approximately ∑iTi\sum_i T_i to max⁡iTi+Tmerge+Tadapt\max_i T_i + T_{\mathrm{merge}} + T_{\mathrm{adapt}}. We do not claim that GPU-hours are necessarily reduced, nor do we present Parallel-RL as universally more compute-efficient under all hardware budgets. Its main potential is shortening the multi-task post-training cycle when parallel resources are available, enabling faster iteration and multi-task adaptation. We will tighten our efficiency claims in the paper to frame this strictly as a potential wall-clock advantage from parallelization rather than an established speedup factor. Our core contribution remains the mechanistic analysis above, with Parallel-RL serving as a feasibility validation and application.

Thank you again for your comments. Clarifying the scope of our experimental comparisons, the distinction between theoretical bounds and empirical performance, and the different efficiency metrics will make the paper's main conclusions and contributions much clearer.

Thank you for the detailed rebuttal and for clarifying the scope of the contributions. I appreciate the authors' efforts to address the concerns, particularly the additional cosine similarity derivation, the clarification regarding numerical precision, and the more careful discussion of Parallel-RL's efficiency claims. However, several of my main concerns remain insufficiently resolved.

1. Experimental controls and causal attribution.

I agree that SFT and RL should not necessarily use identical learning rates or training recipes, as they involve fundamentally different optimization objectives. However, the main concern is not whether the two paradigms should share identical configurations, but whether the observed differences can be reliably attributed to the proposed mechanisms. Differences in training data, optimization dynamics, and gradient scales may all influence the results. Comparing two effective training recipes is reasonable for evaluating practical performance, but it is insufficient to isolate the causal mechanisms underlying their different forgetting behaviors. I appreciate the clarification regarding bf16 precision and agree that exact zero updates should not be treated as the primary explanation.

2. The theoretical explanation remains insufficiently established.

The additional scale-invariant cosine derivation is a meaningful attempt to address my previous concern. Nevertheless, the proposed bounds rely on strong assumptions about a dominant shared score-function component and weak cross-task residual coupling. It remains unclear whether these assumptions are adequately supported by the reported measurements. Moreover, obtaining a small upper bound on residual inner products does not automatically imply small cosine similarity without controlling the norms of the resulting gradients. In particular, the role and boundedness of the factor GG should be clarified.

More importantly, even if the cosine bounds hold, their implications for cross-task interference and forgetting remain unclear. Highly aligned gradients do not necessarily indicate harmful interference; positively aligned gradients can instead facilitate joint optimization. Similarly, low absolute cosine similarity does not directly establish improved capability retention. Thus, the connection between the proposed gradient geometry and the observed differences in final task performance remains largely empirical rather than theoretically established.

3. Parallel-RL's distinction and efficiency advantages.

The rebuttal clarifies the difference between independent task-wise RL training followed by parameter merging and distributed mixed-task RL with shared policy optimization. I acknowledge that independent training offers flexibility in task-specific configurations and may reduce coordination overhead. However, distributed multi-task RL can also parallelize rollout collection across multiple devices. Therefore, the comparison between sequential execution and independent parallel execution does not directly establish an advantage over a resource-matched distributed multi-task RL baseline. I appreciate the authors' clarification that the method does not necessarily reduce GPU-hours and that actual wall-clock improvements remain unverified.

Overall, the rebuttal provides useful clarifications and appropriately narrows several claims. However, my primary concern remains that the theoretical analysis does not yet convincingly establish the proposed explanation for the observed differences in cross-task interference and capability retention between SFT and RL. Since the authors identify this mechanistic explanation as the paper's central contribution, I consider this limitation particularly important. Consequently, while I appreciate the additional discussion, my overall assessment remains largely unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.