- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a lightweight framework that combines instruction fine-tuning (IFT) and model merging to address the high computational cost of training reasoning models.
To validate the effectiveness and efficiency of the proposed approach, they conduct experiments on Rust coding and text summarization tasks, comparing it against standard reinforcement learning (RL) baseline and an IFT baseline that employs on-policy distillation.
Critical strengths
- The use of model merging to mitigate the high training costs of existing RL-based methods is a novel aspect of this work.
Critical weaknesses
- As shown in Table 1, the performance gains of the proposed method vary across models and tasks. In particular, comparisons with standard IFT (e.g., Olmo3 7B) suggest that its advantages in effectiveness and cost efficiency may be limited.
- The proposed method remains subject to a trade-off between OOD and ID performance as increases. Given that improved generalization through enhanced reasoning capabilities is a key motivation for RL-based methods, stronger empirical evidence would be helpful to substantiate the paper’s claim that the proposed method preserves reasoning behavior.
Questions for authors
- Is the proposed method also effective on tasks that evaluate the model’s reasoning capabilities, such as complex reasoning benchmarks?
2 comments
I appreciate the authors’ responses to my review and have carefully read their rebuttal.
However, I still feel that my concerns have not been fully addressed.
I therefore maintain my original rating.
We thank the reviewer for recognizing the novelty of our approach in training reasoning models using SFT. We address their concern about adapting to reasoning in the global response, and the concern about the apparent lack of change in reasoning behavior on Olmo in the global response.