- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a post-training recipe for equipping text-only reasoning models with stateful Python tool execution while preserving their native no-tool reasoning performance. The methodology consists of three stages: data engineering (selecting teacher models with lightweight incremental tool-use patterns, filtering for tool-advantaged problems, mixing 2:1 tool and text-only trajectories, and filtering trajectories exceeding 16K tokens), stage coordination between SFT and RL (characterizing SFT progression into form, substance, and noise stages, and selecting RL initialization checkpoints based on pass@k and rollout length rather than pass@1), and RLVR training (applying strictly on-policy updates and rollout routing replay on 4,325 competition-level math problems without explicit tool-use rewards). Evaluated on Qwen3-4B-Thinking, Qwen3-30B-A3B-Thinking, and GLM-4.7-Flash, the resulting TRICE models achieve strong gains across competition math benchmarks and generalize to scientific and coding tasks.
Critical strengths
The empirical diagnosis of cross-mode negative transfer—showing that TIR-only SFT causes the model in no-tool mode to emit plain-text Python code and hallucinate execution outputs in an endless loop—clearly illustrates the interference between tool-integrated and native text-only reasoning modes.
The investigation into SFT-to-RL transition dynamics demonstrates that selecting late-stage SFT checkpoints solely by highest pass@1 can lead to early RL collapse, providing practical guidance on monitoring pass@k and generation length during SFT checkpoint selection.
The experimental evaluation is comprehensive across multiple model scales and benchmarks, including out-of-domain tasks (FrontierScience, GPQA-Diamond, LiveCodeBench, HLE) and post-cutoff 2026 math competitions accompanied by a data contamination audit.
Critical weaknesses
The overall problem formulation and training pipeline—distilling interleaved code-execution trajectories from a teacher model, filtering for correctness, mixing text-only and tool-integrated data during SFT, and running outcome-rewarded RL—closely follow established paradigms in tool-integrated reasoning (such as ToRA, MathCoder, OpenMathInstruct, Qwen2.5-Math-TIR, ReTool, and ToRL). The constituent techniques in the recipe, including replaying text-only trajectories to prevent forgetting, length filtering, strictly on-policy updates, and Rollout Routing Replay, are combinations of known post-training practices rather than new learning algorithms or training objectives.
The mechanistic claim that late-stage SFT checkpoints (Stage 3, Epoch 16) collapse during RL due to a fourfold increase in train-inference probability mismatch from overfitting teacher noise (Figure 4) is only demonstrated on the sparse MoE model (Qwen3-30B-A3B). In sparse MoE architectures, train-inference log-probability divergence under bfloat16 is highly sensitive to router logit sharpening and expert-routing flips between inference and training engines—an issue reflected by the fact that MoE router freezing and Rollout Routing Replay are enabled exclusively for the 30B model in Table 10. Presenting this phenomenon as a general law of SFT over-training confounds general post-training dynamics (such as entropy collapse) with MoE-specific routing instabilities.
In the teacher selection comparison (Table 1), the 38-point student performance gap between GPT-OSS-120B and MiniMax-M2.7 is attributed to lightweight incremental versus heavyweight monolithic tool-calling styles. However, Table 1 reports that MiniMax-M2.7 trajectories average 30.5K tokens, whereas Table 9 specifies that full-parameter SFT uses a maximum sequence length of 16,384 tokens. Truncating 30.5K-token trajectories at 16K during SFT introduces a severe confounding factor that directly explains why the student model failed to learn complete reasoning chains and exhibited a 46.2% truncation rate.
The RL stage introduces a newly curated dataset of 4,325 high-difficulty competition problems extracted from books and online sources, which differs from the Nemotron-Math-v2 pool used in SFT. The experiments do not isolate how much of the RL improvement (+4.4 points at 4B and +7.6 points at 30B) comes from exposure to this new problem distribution versus the RL optimization process itself.
Questions for authors
Is the fourfold increase in train-inference KL mismatch at Stage 3 (Figure 4) also observed in the dense Qwen3-4B model, or is it primarily driven by router sharpening and expert-switching discrepancies in the sparse Qwen3-30B-A3B MoE model?
Given that SFT uses a maximum sequence length of 16,384 tokens (Table 9) while MiniMax-M2.7 teacher trajectories average 30.5K tokens (Table 1), were MiniMax-M2.7 trajectories truncated during the teacher comparison experiment? How do the two teachers compare when both are restricted to complete, non-truncated trajectories under 16K tokens?
To disentangle the effect of the new 4,325 RL problem set from RL optimization, how does RLVR compare against rejection-sampled SFT or iterative distillation on the exact same 4,325 problems?
Thank you for the additional experiments. The epoch-12 ablations and RL control address my concern about the short SFT runs, and the matched-tool comparisons make the cross-domain gains more convincing. I would still appreciate a clearer rule for choosing an SFT checkpoint before running RL on a new model. I’m keeping my rating at 3.