acceptodds
Under review as a conference paper at ICLR 2027

Leveraging Instruction Tuning and Merging for Reasoning Model Adaptation

Abstract

Reasoning language models (RLMs) demonstrate impressive performance by leveraging test-time compute in the form of reasoning tokens. However, this behavior makes adapting RLMs to new domains challenging and expensive. The reason is that further training can disturb the learned behavior and degrade model performance. This makes it difficult to leverage supervised fine-tuning data with human-written solutions: although it contains high-quality annotations, it lacks reasoning tokens. In this work, we show how, despite this challenge, such data can be used efficiently for RLM adaptation. For this, we first use standard instruction tuning. Next, we leverage model merging to combine the instruction-tuned model with the original RLM, picking the merging ratio such that the resulting model's reasoning behavior on the target domain is recovered. We evaluate our method across four RLMs on coding and text summarization tasks, where it improves target-task performance by up to 11.0% while preserving reasoning behavior and limiting the out-of-distribution score degradation to on average 0.7%. Importantly, our adaptations are efficient and economical, costing less than USD $10 per model.

Primary area:
generative models
Keywords:
reasoning model, finetuning, model merging, task adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6yUGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a lightweight framework that combines instruction fine-tuning (IFT) and model merging to address the high computational cost of training reasoning models.

To validate the effectiveness and efficiency of the proposed approach, they conduct experiments on Rust coding and text summarization tasks, comparing it against standard reinforcement learning (RL) baseline and an IFT baseline that employs on-policy distillation.

Critical strengths
  1. The use of model merging to mitigate the high training costs of existing RL-based methods is a novel aspect of this work.
Critical weaknesses
  1. As shown in Table 1, the performance gains of the proposed method vary across models and tasks. In particular, comparisons with standard IFT (e.g., Olmo3 7B) suggest that its advantages in effectiveness and cost efficiency may be limited.
  2. The proposed method remains subject to a trade-off between OOD and ID performance as α\alpha increases. Given that improved generalization through enhanced reasoning capabilities is a key motivation for RL-based methods, stronger empirical evidence would be helpful to substantiate the paper’s claim that the proposed method preserves reasoning behavior.
Questions for authors
  1. Is the proposed method also effective on tasks that evaluate the model’s reasoning capabilities, such as complex reasoning benchmarks?
2 comments

We thank the reviewer for recognizing the novelty of our approach in training reasoning models using SFT. We address their concern about adapting to reasoning in the global response, and the concern about the apparent lack of change in reasoning behavior on Olmo in the global response.

I appreciate the authors’ responses to my review and have carefully read their rebuttal.

However, I still feel that my concerns have not been fully addressed.

I therefore maintain my original rating.

Reviewer_UBTCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a simple approach to adapting reasoning language models using instruction tuning and model merging. The authors first fine-tune a reasoning model on task-specific data without reasoning traces, then merge it with the original model to restore reasoning behavior. The merging ratio is selected based on the proportion of calibration examples that produce non-empty reasoning traces. Experiments across four models and two tasks show improvements in target-task performance while largely preserving out-of-distribution performance at low training cost.

Critical strengths
  1. The paper addresses a practical problem in adapting reasoning models, especially when training data does not include reasoning traces.

  2. The proposed method is simple, computationally inexpensive, and achieves promising improvements while largely preserving general model performance.

  3. The experiments cover multiple reasoning models and include comparisons with alternative training approaches, ablations, and cost analysis.

Critical weaknesses
  1. The methodological novelty is limited, as combining fine-tuned and original model weights is an established technique. The main contribution is the selection of the merging ratio based on reasoning behavior.

  2. The paper measures reasoning preservation primarily through the presence of non-empty reasoning traces. This does not necessarily indicate that the model retains its original reasoning capabilities.

  3. The evaluation is limited to Rust coding and text summarization. The authors also report limited improvement in preliminary mathematical proof experiments, raising questions about generalization to reasoning-intensive tasks.

Questions for authors

How well does the method generalize to other tasks, especially those requiring more complex reasoning?

1 comment

We thank the reviewer for recognizing our practical significance and extensive experimental results. We address their concern about adapting to reasoning in the global response, and individual remarks below.

Q1: Do you measure the preservation of reasoning quality?

In our experiments, we measure model performance on in-distribution and out-of-distribution tasks. We argue that this is as close to measuring preservation of reasoning quality as possible, since reasoning shape or form itself are not inherently meaningful: The reasoning behavior is trained via RL, with the main goal of improving task performance. As such, we argue that the best estimate for preserved behavior is this performance itself. In App. A.6, we show that reasoning traces become only slightly shorter as the merge ratio increases (Fig. 5), and that trace length is not significantly correlated with target-task performance (Table 6). Qualitatively, we do not find a significant change in the reasoning traces either, as shown in the example in Fig. 2 and the full case study in App. B (Figs. 6-8), where the merged model reasons like the unadapted model.

Reviewer_5SCEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a practically significant problem. The authors propose a two-stage method: first, performing standard Instruction Fine-Tuning (IFT) on the target task data; second, merging the parameters of the fine-tuned model with the original model. The core design involves using the proportion of non-empty reasoning trajectories generated on a calibration set as a constraint to select the maximum merging coefficient that satisfies the reasoning rate requirement. Additionally, the authors conduct experiments on four reasoning language models across two tasks—Rust code generation and text summarization—to demonstrate the potential value of the proposed method.

Critical strengths
  1. The research problem is practically significant, and the proposed solution is simple with low computational cost. The task adaptation of reasoning models is indeed a worthy area of study. After existing models acquire strong reasoning capabilities through post-training, supervised data for new domains typically provides only task inputs and final answers, without including reasoning traces. Compared to existing methods, this paper employs standard IFT combined with post-training parameter merging, requiring no architectural modifications. This achieves the goal of low implementation cost and high efficiency. Particularly for tasks lacking reliable reward verifiers, this paper provides a valuable alternative approach.

  2. The experimental design is relatively comprehensive. The paper compares the original model with direct IFT and introduces control methods such as RL, OPD, Early Stopping, and forced reasoning. It evaluates target task performance, out-of-distribution (OOD) performance, reasoning rate, and training costs. Furthermore, it supplements the analysis with findings on the forgetting and recovery of the original model's capabilities. Results from different merging algorithms, LoRA vs. full-parameter fine-tuning, and hyperparameter ablations all support the practicality of the method.

Critical weaknesses
  1. scoring scales in the text summarization experiments. In Appendix A.2, four metrics are defined on a 1-5 scale. However, Figure 12 in Appendix C sets 'relevance' on a 1-100 scale, while the RLAIF reward formula in Appendix A.4 still assumes all metrics are on a 1-5 scale. The paper does not clearly state whether a unified scale conversion was performed. Mixing different scoring scales may affect the reliability of the results.

  2. The novelty of the merging ratio selection strategy requires further discussion. There is existing research on IFT, model parameter interpolation, and using model merging to mitigate capability forgetting. Therefore, the main innovation of this paper lies in using the reasoning rate as a constraint to select the maximum merging ratio that satisfies the condition. However, the reasoning rate only measures whether a non-empty reasoning trajectory is generated; it does not reflect actual reasoning capability. The current results show a trade-off between target performance and capability retention, which is insufficient to demonstrate that selecting the maximum merging ratio constrained by the non-empty reasoning rate is superior to fixed ratios or other low-cost selection strategies. In particular, Olmo3 still exhibits OOD performance degradation even when the reasoning rate is fully preserved, indicating that the applicability boundaries of this signal need further explanation.

Questions for authors
  1. Regarding summarization evaluation and reward calculation: Could you clarify whether 'Relevance' was normalized in the actual evaluation code and RLAIF training code, and if so, what the specific conversion rules were? Please also confirm whether the summarization scores in Table 1 and Table 5 are based on a consistent scoring scale. If there are implementation inconsistencies, please explain their impact on the main results.

  2. Regarding the rationale of using reasoning rate as a model selection metric: Why choose the maximum merging ratio that satisfies the Reasoning Rate constraint, rather than other simple, low-cost selection strategies? Given that some models show OOD degradation even when the reasoning rate remains unchanged, and some models lose significant target task gains after merging, could you further explain the conditions of applicability and limitations of this metric?

  3. Regarding the scope of the method: In your experiments, Olmo3 did not exhibit a loss of reasoning behavior under the main training settings, and preliminary experiments on mathematical proof tasks did not yield significant improvements. Which models or tasks are more likely to benefit from this method? Are there specific model characteristics or task attributes that can explain these differences?

1 comment

We thank the reviewer for recognizing our practical significance and extensive experimental results. We address their concern about adapting to reasoning and OLMo in the global response, and individual remarks below.

Q1: Why is the merging ratio selection optimal? Could you provide some baselines?

Our method is designed to achieve two goals: First, we do not want to disturb the reasoning behavior of the trained model. Second, we aim to preserve in-distribution and out-of-distribution behavior.

We compare against two simple alternatives on our existing extensive merging ratio ablation: fixing the merge ratio at 0.5, and choosing the with the highest target-dataset score, which could be cheaply estimated using the calibration dataset. RR is the target-task reasoning rate (% of test examples with a non-empty reasoning trace).

TaskModelα=0.5\alpha=0.5: ΔTargetΔOODRRBest-target α\alpha: ΔTargetΔOODRROurs: ΔTargetΔOODRR
RustOpenThinker 7B+0.9+0.50.0+3.8+0.698.6+3.8+0.698.6
RustQwen 7B+11.9−0.7100.0+19.5−2.463.5+11.0−0.2100.0
RustOlmo3 7B+3.8+0.0100.0+3.9+1.1100.0+3.8−0.5100.0
RustApriel 15B+4.0−0.677.2+4.2−0.768.8+3.7+2.5100.0
Summ.OpenThinker 7B+10.1−5.00.0+10.1−5.00.0+4.4−2.0100.0
Summ.Qwen 7B+0.9−1.399.8+6.4−1.150.1+1.0−2.099.9
Summ.Olmo3 7B+2.2−0.7100.0+3.5−2.0100.0+3.5−3.0100.0
Summ.Apriel 15B+2.2−0.882.5+2.2−0.882.5+1.4−0.798.9
Mean+4.5−1.169.9+6.7−1.370.4+4.1−0.799.7

Fixed alpha. α=0.5\alpha=0.5** is not robust. The best ratio varies widely across models and tasks (our selected ratios range from 0.23 to 1.0). Where a fixed ratio fits badly, it fails completely. On OpenThinker it removes reasoning on both tasks (0.0% RR), and it also loses on target or OOD: +0.9 vs. +3.8 target on Rust, and −5.0 vs. −2.0 OOD on summarization. Its slightly higher mean target gain (+4.5 vs. +4.1) comes entirely from that single setting, where the model no longer reasons at all. Excluding it, our rule is ahead on target, and it degrades OOD performance less overall (−0.7 vs. −1.1).

Best-target α\alpha. Selecting the ratio based on the target task needs a reliable target-task metric for every candidate ratio. Our method works without this assumption. For this experiment, we optimistically selected the ideal performance on the evaluation data itself. Even so, it causes the largest OOD degradation (−1.3). It also removes reasoning behavior in 4 of 8 settings, dropping to 63.5% and 50.1% for Qwen and 0.0% for OpenThinker summarization. Its larger target gains come mostly from these settings, where the model has partly turned into a non-reasoning model. Our rule needs only a cheap calibration signal and comes within one point of its target score in 5 of 8 settings.

Both baselines break the first goal. Baselines that are not calibrated by the reasoning behavior often remove reasoning from the model: mean reasoning rates are 69.9% for fixed α=0.5\alpha=0.5 and 70.4% when calibrating by target performance, compared with 99.7% for our rule. Our rule keeps reasoning at 98.6% or higher in every setting.

We are happy to evaluate other cheap baselines that the reviewer suggests.

Q2: Was 'Relevance' normalized in the evaluation and RLAIF code, and what were the conversion rules? Are the summarization scores in Table 1 and Table 5 on a consistent scale?

Yes. Relevance is normalized to the same 1-5 scale as the other three dimensions, in the same way in both the evaluation and the RLAIF reward. Tables 1 and 5 use the same scale.

Conversion rule. As shown in the relevance rubric prompt, the judge rates relevance from 1 to 100. Fluency, consistency, and coherence are rated from 1 to 5. Every raw score ss is mapped to an integer 1-5 score before averaging:

  • Relevance: score=clip(⌈s/20⌉,1,5)\text{score} = \text{clip}(\lceil s/20 \rceil, 1, 5).
  • Fluency, consistency, coherence: score=clip(s,1,5)\text{score} = \text{clip}(s, 1, 5).

Evaluation (Gemini judge). The four 1-5 scores are averaged per summary. We report every SummEval number as 20×score20 \times \text{score}, so a rating of 5 corresponds to 100%.

RLAIF reward. The frozen base-model judge uses the same prompts, the same score parser, and the same conversion function as the evaluation. The reward is R=(Sˉ−1)/4∈[0,1]R = (\bar{S} - 1)/4 \in [0,1]. A summary that cannot be extracted receives R=0R = 0.

Table 1 vs. Table 5. Both tables are computed from the same per-run results with the same factor of 20. The Table 1 summarization score is therefore exactly the mean of the four Table 5 SummEval columns.

We will add the relevance conversion rule to Appendix A.2 to make this explicit.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank the reviewers for their constructive feedback and are glad they recognize the practical significance (5SCE, UBTC) and strength of our experimental evaluation. We address their shared concern below and answer individual questions in direct replies.

Q1 (5SCE, UBTC, 6yUG): Which tasks more likely benefit from this method? Does it transfer to reasoning-intensive tasks?

As we outline in our manuscript, our method is designed to improve models that are already capable at reasoning for tasks that require additional knowledge, in particular knowledge about facts or output shape. Our two tasks demonstrate this benefit on both the reasoning-intensive task Rust coding and more text-intuitive task of text summarization. We refer in particular to the extensive reasoning required to answer our showcased example in Fig. 2, which we provide unabbreviated in App. B.

Our method is not designed to improve the reasoning itself, and we neither claim nor expect such an effect. In fact, we transparently report the absence of this effect on datasets of mathematical reasoning in our limitations section.

Q2 (5SCE, 6yUG): Does the fact that our method does not improve over IFT for Olmo indicate its limited benefit?

No. For most model setups, IFT significantly hurts model performance. In those cases, our method provides significant benefit over IFT. In the case of Olmo, our method converges to IFT at minimal additional cost. We argue that this rather showcases how generally applicable our method is: Even when model performance is increased in standard IFT, our method preserves this improvement. This is not the case for RLVR or OPD.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.