acceptodds
Withdrawn from ICLR 2027

Decouple before Integration: Post-Hoc Synthesis of SFT and RLVR Task Vectors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

SFT and RLVR improve large language models through complementary learning signals, but existing integration methods rely on training-time choices such as stage lengths, loss weights, or data-mixing ratios. Revisiting this balance after training generally requires another costly optimization run. We therefore ask whether the contributions of SFT and RLVR can instead be calibrated post hoc. Through a task-vector analysis, we find that their updates differ substantially in scale, sign overlap, and module-wise distribution, making direct composition difficult. Motivated by these differences, we propose Decoupled Synthesis (\ours), which independently sparsifies the two task vectors, restores their original norms, and calibrates their relative contributions by searching only two scalar coefficients with Bayesian optimization. We further analyze when such composition can outperform either individually rescaled source. Across seven mathematical reasoning benchmarks, \ours matches or exceeds training-based SFT–RLVR integration methods, while requiring only about of other SFT–RLVR integration baselines' computation when source checkpoints are available. The learned coefficients also transfer to out-of-domain QA and coding benchmarks without re-tuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
SFT & RLVR Intergration, Task Vector, Large lange Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nM8GOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DoST, a post-hoc method for combining independently trained SFT and RLVR checkpoints from a shared base model. It applies magnitude pruning, restores each update’s original norm, and searches two merging coefficients using verifier accuracy. The evaluation covers seven math benchmarks, OOD QA and coding, and three backbones. A stylized theoretical construction illustrates when complementary source updates can outperform either individually rescaled source.

Critical strengths

Post-hoc calibration is practically useful when source checkpoints are already available. On Qwen2.5-Math-7B, DoST achieves an average accuracy of 46.8, compared with 44.3 and 40.9 for the SFT and on-policy RL source models, respectively (Table 1), with a reported search cost of 20 GPU-hours. The OOD results obtained without retuning the coefficients (Table 3) provide further evidence that the selected mixture can generalize beyond the calibration domain.

Critical weaknesses

W1. The comparisons do not isolate DoST’s contribution beyond established model merging. Appendix A.4 reports default settings for TIES and DARE, whereas DoST receives 100 accuracy-guided search trials. Their poor performance therefore does not establish that their failures stem specifically from SFT–RLVR geometry. The main SFT/on-policy RL comparison also lacks a dense-merging baseline with the same search budget. For ExGRPO/ReLIFT, the available dense-search control already achieves 46.6, compared with 47.1 for DoST (Table 4). The empirical evaluation also omits single-source controls with individually optimized coefficients, despite their inclusion in the theoretical comparator. Budget-matched merging baselines and optimized single-source controls are needed to distinguish the benefit of the proposed synthesis procedure from that of coefficient tuning. This distinction matters because pruning and learned merging coefficients are already established in TIES-Merging and AdaMerging, respectively, both of which the paper cites. W2. The proposed conflict-control mechanism has limited direct validation. Figure 1 uses global top-10% selection, whereas the algorithm retains 30% of the coordinates within each parameter tensor. Opposite-sign overlap can decrease simply because pruning reduces the support, so this statistic alone does not establish that harmful interference has been removed. The pruning ablation shows an average gain of only 0.5 percentage points, with lower scores on four of the seven benchmarks (Table 4). The paper provides neither a comparison against random masks at the same retention rate nor an analysis of how accuracy varies with the retention fraction. The theory does not resolve this gap: Appendix D.4.1 removes coordinates that do not affect correctness and establishes preservation, rather than improvement over optimal dense merging. The central design rationale therefore remains insufficiently supported. Controls that use the algorithm’s actual pruning operator are needed to establish why this processing is beneficial.

Questions for authors

For W1, what scores do budget-matched dense, TIES/DARE, and individually rescaled source baselines achieve on the same source pairs and calibration set? For W2, does magnitude pruning outperform matched-retention random pruning across retention fractions under the same coefficient-search budget? These comparisons would materially affect my assessment of the contribution.

Reviewer_N3vJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on studying how to combine SFT with RLVR after training. Existing methods integrate the SFT and RLVR stages using a sequential pipeline, but the subsequent RLVR stage may weaken capabilities learned during the SFT phase. Balancing the two stages is also expensive, since it usually requires another training run. The authors propose another strategy: Decoupled Synthesis (DoST). The idea is to train SFT and RLVR models separately from the same base model and then apply model merging. Specifically, DoST first sparsifies the task vectors, restores their original norms, and applies Bayesian optimization to select the merging coefficients. Experiments demonstrate that DoST performs slightly better than the baseline methods on reasoning benchmarks and on OOD QA and coding tasks.

Critical strengths

Good motivation. The paper clearly motivates the limitations of the sequential paradigm that integrates RL and SFT, and proposes DoST as an alternative solution that can adjust the contributions of the two stages independently without requiring further gradient-based optimization. The idea of merging independently trained RL and SFT models is interesting and useful when both checkpoints are already available.

Critical weaknesses
  1. Lack of evidence for the benefit of the proposed sparsification mechanism:

In Figure 1(b), the authors motivate sparsification as a way to reduce interference. However, retaining only 10% of the SFT task vector necessarily reduces or leaves unchanged the measured interference ratio, since the pruned entries become zero. I do not understand how this supports the authors’ motivation, especially as they acknowledge that “The reduction mainly comes from shrinking the overlap between the two supports.” For example, suppose the top-10% RLVR entries consist of 10 positive entries, and the corresponding entries in the full SFT task vector include 5 positive and 5 negative entries. The interference ratio is then 50%. If sparsification retains only 2 negative SFT entries among these coordinates and sets the others to zero, the interference ratio drops to 2/10 = 20%. However, all retained SFT entries in this overlap still have signs opposite to the RLVR entries. Thus, the decrease shown in this figure alone does not establish that sparsification reduces harmful interference. Moreover, in Table 4, DoST achieves an average score of 47.1 across the mathematical reasoning benchmarks, compared with 46.6 without sparsification. Without uncertainty estimates, it is unclear whether this small improvement exceeds evaluation noise. The variant without sparsification also achieves better scores on AIME 24, AIME 25, and AIME 26.

  1. Unclear sources and consistency of baseline results:

Appendix A.4 states that the authors download the released baseline checkpoints and re-evaluate them under their evaluation setting. However, some groups of results exactly match the original papers. LUFFY’s three QA scores in Table 3 match its original Table 1 (ARC-C, GPQA, MMLU-Pro), and its six shared mathematical benchmark scores on LLaMA3.1-8B in Table 5 (AIME 24, AIME25, AMC, MATH 500, Minerva, Olympiad) match its original Appendix E.2, Table 3. ExGRPO’s three QA scores (ARC-C, GPQA, MMLU-Pro) in Table 3 also match its original Table 1. These matches do not necessarily mean that the baselines were not re-evaluated, but they raise questions about the sources of these results. The authors should clarify which results were independently re-evaluated and which, if any, were taken from prior work, and provide the evaluation configurations and outputs to confirm that the comparisons use the same settings.

Questions for authors
  1. Figure 1(b) is confusing to me. The caption states that “the two dense vectors disagree in sign on 44.91% of coordinates,” but this value is not shown in the figure. I also could not find a clear explanation of how it was calculated. Please clarify the calculation, including which coordinates are included and how zero-valued entries are handled.

  2. Regarding Critical Weakness 2, please recheck the sources and evaluation settings of the baseline results, clarify which results were independently re-evaluated, and re-evaluate any baselines whose settings differ from those used for DoST.

  3. Regarding Critical Weakness 1, the small average improvement from sparsification may be within evaluation noise. It would be helpful to repeat the experiments with and without sparsification three times using different random seeds, and report the mean and standard deviation for each benchmark and the overall average.

Reviewer_yvYzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

DoST combines independently trained SFT and RLVR checkpoints through task-vector arithmetic. It prunes each update, restores its original norm, and searches two mixing coefficients using verifier accuracy. The paper evaluates this procedure on mathematical reasoning and transfers the selected coefficients to QA and coding benchmarks without retuning.

Critical strengths
  • Adjusting the SFT–RLVR balance after training is a useful option when the source checkpoints already exist, and the procedure is easy to implement.
  • The OOD evaluation with fixed math-selected coefficients is a useful check. Including dense task vectors with coefficient search also helps identify what sparsification actually adds.
Critical weaknesses
  • The experiments mainly establish effectiveness within math-focused post-training. The OOD evaluation in Table 3 covers only ExGRPO + ReLIFT, with coefficients still selected on math problems. This is limited evidence for the method's usefulness beyond the mathematical reasoning setting
  • The TIES/DARE comparison is not persuasive as it stands. Appendix A.4 uses fixed settings and a 10% drop rate, whereas DoST gets 100 coefficient-search trials. Their poor results could reflect inadequate calibration rather than a fundamental problem with merging SFT and RLVR updates.
  • Eq. (6) gives each source its best individual scaling, but the experiments compare against the original checkpoints. Without the corresponding single-source controls, it is hard to tell how much of the gain actually requires combining both sources.
  • The sparsification result most needs replication: Table 4 improves from 46.6 with dense-vector search to 47.1 with DoST. There are no repeated calibration sets or search seeds, and no sensitivity analysis for p=0.3p=0.3, so I am unsure how reliable this increment is.
Questions for authors
  • How do TIES and DARE perform with the same 256 calibration questions and 100-trial budget, allowing coefficient tuning and testing comparable sparsity settings?
  • Following Eq. (6), what test scores are obtained by optimizing λRLVR=0\lambda_{\mathrm{RLVR}}=0 and λSFT=0\lambda_{\mathrm{SFT}}=0 separately, for both dense and processed vectors?
  • Can you repeat Tables 3–4 across independently sampled calibration sets and search seeds, and test an additional source pair from code or general instruction tuning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.