acceptodds
Under review as a conference paper at ICLR 2027

Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoning

Abstract

Tool-integrated reasoning (TIR) offers a direct way to extend thinking models beyond the limits of text-only reasoning. Paradoxically, we observe that tool-enabled evaluation can degrade reasoning performance even when the strong thinking models make almost no actual tool calls. In this paper, we investigate how to inject natural tool-use behavior into a strong thinking model without sacrificing its no-tool reasoning ability, and present a comprehensive TIR recipe. We highlight that (i) the effectiveness of TIR supervised fine-tuning (SFT) hinges on the learnability of teacher trajectories, which should prioritize problems inherently suited for tool-augmented solutions; (ii) controlling the proportion of tool-use trajectories could mitigate the catastrophic forgetting of text-only reasoning capacity; (iii) optimizing for pass@k and response length instead of training loss could maximize TIR SFT gains while preserving headroom for reinforcement learning (RL) exploration; (iv) a stable RL with verifiable rewards (RLVR) stage, built upon suitable SFT initialization and explicit safeguards against mode collapse, provides a simple yet remarkably effective solution. When applied to Qwen3 thinking models at 4B and 30B scales, our recipe yields models that achieve state-of-the-art performance in a wide range of benchmarks among open-source models, such as 96.7% and 99.2% on AIME 2025 for 4B and 30B, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tool-Integrated Reasoning, Large Reasoning Models, Reinforcement Learning.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CdABOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a post-training recipe for equipping text-only reasoning models with stateful Python tool execution while preserving their native no-tool reasoning performance. The methodology consists of three stages: data engineering (selecting teacher models with lightweight incremental tool-use patterns, filtering for tool-advantaged problems, mixing 2:1 tool and text-only trajectories, and filtering trajectories exceeding 16K tokens), stage coordination between SFT and RL (characterizing SFT progression into form, substance, and noise stages, and selecting RL initialization checkpoints based on pass@k and rollout length rather than pass@1), and RLVR training (applying strictly on-policy updates and rollout routing replay on 4,325 competition-level math problems without explicit tool-use rewards). Evaluated on Qwen3-4B-Thinking, Qwen3-30B-A3B-Thinking, and GLM-4.7-Flash, the resulting TRICE models achieve strong gains across competition math benchmarks and generalize to scientific and coding tasks.

Critical strengths

The empirical diagnosis of cross-mode negative transfer—showing that TIR-only SFT causes the model in no-tool mode to emit plain-text Python code and hallucinate execution outputs in an endless loop—clearly illustrates the interference between tool-integrated and native text-only reasoning modes.

The investigation into SFT-to-RL transition dynamics demonstrates that selecting late-stage SFT checkpoints solely by highest pass@1 can lead to early RL collapse, providing practical guidance on monitoring pass@k and generation length during SFT checkpoint selection.

The experimental evaluation is comprehensive across multiple model scales and benchmarks, including out-of-domain tasks (FrontierScience, GPQA-Diamond, LiveCodeBench, HLE) and post-cutoff 2026 math competitions accompanied by a data contamination audit.

Critical weaknesses

The overall problem formulation and training pipeline—distilling interleaved code-execution trajectories from a teacher model, filtering for correctness, mixing text-only and tool-integrated data during SFT, and running outcome-rewarded RL—closely follow established paradigms in tool-integrated reasoning (such as ToRA, MathCoder, OpenMathInstruct, Qwen2.5-Math-TIR, ReTool, and ToRL). The constituent techniques in the recipe, including replaying text-only trajectories to prevent forgetting, length filtering, strictly on-policy updates, and Rollout Routing Replay, are combinations of known post-training practices rather than new learning algorithms or training objectives.

The mechanistic claim that late-stage SFT checkpoints (Stage 3, Epoch 16) collapse during RL due to a fourfold increase in train-inference probability mismatch from overfitting teacher noise (Figure 4) is only demonstrated on the sparse MoE model (Qwen3-30B-A3B). In sparse MoE architectures, train-inference log-probability divergence under bfloat16 is highly sensitive to router logit sharpening and expert-routing flips between inference and training engines—an issue reflected by the fact that MoE router freezing and Rollout Routing Replay are enabled exclusively for the 30B model in Table 10. Presenting this phenomenon as a general law of SFT over-training confounds general post-training dynamics (such as entropy collapse) with MoE-specific routing instabilities.

In the teacher selection comparison (Table 1), the 38-point student performance gap between GPT-OSS-120B and MiniMax-M2.7 is attributed to lightweight incremental versus heavyweight monolithic tool-calling styles. However, Table 1 reports that MiniMax-M2.7 trajectories average 30.5K tokens, whereas Table 9 specifies that full-parameter SFT uses a maximum sequence length of 16,384 tokens. Truncating 30.5K-token trajectories at 16K during SFT introduces a severe confounding factor that directly explains why the student model failed to learn complete reasoning chains and exhibited a 46.2% truncation rate.

The RL stage introduces a newly curated dataset of 4,325 high-difficulty competition problems extracted from books and online sources, which differs from the Nemotron-Math-v2 pool used in SFT. The experiments do not isolate how much of the RL improvement (+4.4 points at 4B and +7.6 points at 30B) comes from exposure to this new problem distribution versus the RL optimization process itself.

Questions for authors

Is the fourfold increase in train-inference KL mismatch at Stage 3 (Figure 4) also observed in the dense Qwen3-4B model, or is it primarily driven by router sharpening and expert-switching discrepancies in the sparse Qwen3-30B-A3B MoE model?

Given that SFT uses a maximum sequence length of 16,384 tokens (Table 9) while MiniMax-M2.7 teacher trajectories average 30.5K tokens (Table 1), were MiniMax-M2.7 trajectories truncated during the teacher comparison experiment? How do the two teachers compare when both are restricted to complete, non-truncated trajectories under 16K tokens?

To disentangle the effect of the new 4,325 RL problem set from RL optimization, how does RLVR compare against rejection-sampled SFT or iterative distillation on the exact same 4,325 problems?

Reviewer_MFq6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to teach strong thinking models to use tools while preserving their text-only reasoning ability. The proposed recipe combines teacher and data selection, mixed tool-use and text-only SFT, checkpoint selection, and RL with outcome-based rewards. The resulting TRICE models interleave reasoning with Python execution. Experiments show substantial gains on math benchmarks at both 4B and 30B scales, along with improved average performance without tools. The authors also evaluate transfer to other domains and another model family.

Critical strengths
  1. The final models show strong improvements at both scales. Under the same tool-enabled evaluation, the 30B model’s average accuracy rises from 59.7 to 81.9. Average accuracy without tools also improves over the base model, making the recipe useful for adding tool use to an existing thinking model.

  2. The paper provides useful observations about training failures, including fabricated execution outputs when tools are unavailable and unstable RL from different SFT checkpoints. These examples, together with the data ablations, give practical guidance on what to watch for when training similar models.

Critical weaknesses
  1. The SFT checkpoint selection rule is unclear. The 16-epoch checkpoint has higher accuracy and shorter responses than the 8-epoch checkpoint, yet it collapses during RL. The proposed diagnostics do not clearly explain how to identify this risk before starting RL, making the recipe difficult to apply to a new model.

  2. The data-selection recommendations mainly rely on 4-epoch SFT experiments with the 4B model, while the final model uses 12 SFT epochs followed by RL. Whether the advantages of teacher choice, data mixing, and length filtering persist through the full pipeline remains unclear. This limits the evidence linking these choices to the final gains.

  3. The cross-domain comparisons give TRICE access to Python while evaluating the base models without tools. This combines the effects of training and tool access, making it difficult to tell how much of the improvement comes from the training recipe. Comparisons with matched tool access would help establish the transfer gains.

Questions for authors
  1. How did you select the 8-epoch SFT checkpoint before running RL? What in pass@k or response length distinguishes it from the 16-epoch checkpoint?

  2. Do you have results showing whether the data-selection trends persist beyond four SFT epochs, particularly at the selected SFT checkpoint or after RL?

  3. Do you have cross-domain results with matched tool access for TRICE and the base models? How much of the reported improvement remains under that comparison?

1 comment

Thank you for the additional experiments. The epoch-12 ablations and RL control address my concern about the short SFT runs, and the matched-tool comparisons make the cross-domain gains more convincing. I would still appreciate a clearer rule for choosing an SFT checkpoint before running RL on a new model. I’m keeping my rating at 3.

Reviewer_8LuLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how to add tool-integrated reasoning (TIR) to a strong thinking model without hurting its text-only reasoning, and proposes a recipe that covers the whole pipeline. On the data side, the teacher is chosen by how learnable its tool-use pattern is, prompts are restricted to tool-advantaged problems, TIR and text-only trajectories are mixed, and overly long trajectories are filtered out. The RL starting checkpoint is then chosen using pass@k and response length, based on a "form–substance–noise" progression observed during SFT. Finally, RL uses outcome-only rewards on a hard, self-built dataset, with on-policy updates and rollout routing replay for stability. Applied to Qwen3-Thinking-2507, the resulting TRICE-4B and TRICE-30B reach 96.7 and 99.2 on AIME25 with tools, and the recipe also improves GLM-4.7-Flash, a model that already uses tools natively.

Critical strengths
  1. The problem is practical and well motivated. The observation that strong thinking models get worse once tools are enabled (Table 11) is useful in itself, and it makes clear why simply giving a thinking model a code executor is not enough.

  2. The results are strong and fairly convincing. TRICE outperforms open-source models of similar size and some much larger text-only models. The contamination audit helps here, and so does the tool-vs-tool comparison on AIME/HMMT 2026, which were held after the training data was frozen.

Critical weaknesses
  1. The positioning against prior work could be clearer. [1] START (Li et al., 2025) already adds tool use to a strong reasoning model (QwQ-32B), but it is not cited. [2] CoRT already describes the delayed-code pattern used as motivation here, and it trains R1-Distill thinking models with a similar SFT/RFT/RL pipeline. [3] ASTER studies how interaction-dense cold-start data shapes later RL, which is close to Takeaways 1 and 5. The paper would be stronger if it stated explicitly what is new relative to these works.

  2. The claim about preserving no-tool reasoning is somewhat stronger than the evidence. After SFT, no-tool accuracy drops noticeably: the no-tool average in Table 13 is around 45–48, compared with 58.2 for the base model. Most of the recovery appears to come from RL, and Appendix A.2 says the RL data also includes text-only instances, so the recovery may come from text-only RL rather than from the mixing strategy. For GLM-4.7-Flash, no-tool HMMT25 also drops from 88.3 to 79.6. I would suggest softening the wording in the abstract and conclusion, or adding evidence that isolates where the recovery comes from.

[1] Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Xiang Wang, Bowen Yu, Binyuan Hui, Junyang Lin, and Dayiheng Liu. START: Self-taught reasoner with tools. arXiv preprint arXiv:2503.04625, 2025a.

[2] Chengpeng Li, Zhengyang Tang, Ziniu Li, Mingfeng Xue, Keqin Bao, Tian Ding, Ruoyu Sun, Benyou Wang, Xiang Wang, Junyang Lin, and Dayiheng Liu. CoRT: Code-integrated reasoning within thinking. arXiv preprint arXiv:2506.09820, 2025b.

[3] Xuqin Zhang, Quan He, Zhenrui Zheng, Zongzhang Zhang, Xu He, and Dong Li. ASTER: Agentic scaling with tool-integrated extended reasoning. arXiv preprint arXiv:2602.01204, 2026.

Questions for authors
  1. Where does the no-tool recovery come from? What fraction of RL prompts are in the text-only format? Have you tried text-only RLVR on the same 4.3K problems, starting from the base model, with a similar budget? This comparison would show how much of the no-tool improvement, and of the gains over larger text-only models in Table 5, is specific to TIR training rather than to RLVR on hard data.

  2. Can you complete Table 6 as a 2×2 comparison? Table 6 currently compares TRICE with tools against the base model without tools. Adding the base model with tools and TRICE without tools would separate the effect of training from the effect of tool access. This matters especially for LiveCodeBench, where a sandbox lets the model run its own code.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.