- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes I-Flow, an intent-revealing flow optimization framework for multi-turn LLM reasoning, aiming to address hierarchy collapse and reasoning simplification under terminal-reward reinforcement learning. The method introduces turn-level intents, a Bootstrap-Free Subtrajectory Balance objective, and Intent Structure Replay to preserve effective reasoning patterns. Experiments on mathematical reasoning and code generation benchmarks demonstrate consistent improvements over existing methods.
Critical strengths
- This paper identifies two important failure modes in multi-round LLM reasoning: hierarchy collapse and reasoning simplification. The formulation of these problems provides a compelling motivation for the proposed framework.
- This paper establishes a connection between intent-conditioned continuous flows and KL-regularized optimal policies. The theoretical analysis interprets intermediate flow values in terms of downstream reasoning success and derives bounds relating local flow-balance errors to the proposed training objective, providing useful theoretical justification for the method.
- Although I-Flow is trained on mathematical reasoning data, it also demonstrates improvements on code generation benchmarks. The learned intent-based reasoning behavior is not limited to mathematical problem-solving and may transfer to other tasks requiring structured reasoning and iterative refinement.
Critical weaknesses
- BF SubTB constructs multiple subtrajectory constraints anchored to the same terminal reward. Although this avoids bootstrapping from learned intermediate endpoint flows, the supervision remains highly correlated. The method does not identify which intermediate decisions contribute to success or failure, leaving credit assignment for individual intents unclear.
- The proposed framework relies on six predefined intent categories to represent high-level reasoning decisions. However, complex reasoning processes may involve intents that cannot be adequately captured by this fixed taxonomy, raising concerns about the adaptability of intent representations, particularly for fine-grained tasks.
Questions for authors
- How does I-Flow ensure that the generated high-level guidance and low-level reasoning remain semantically consistent with the selected intent?
- All BF SubTB residuals are anchored to the same terminal reward. How does the method distinguish intermediate intents that contribute to successful reasoning from ineffective ones? How does the proposed objective achieve more informative credit assignment than conventional trajectory-level supervision?
- Since I-Flow incurs a higher overall training cost than ReMA, it remains unclear whether its performance improvements primarily stem from the proposed optimization mechanism or the additional computational resources.
4 comments
Q1 (Semantic Consistency):
We thank the reviewer for asking whether the selected intent, high-level guidance, and low-level reasoning remain semantically consistent. We directly assessed semantic consistency on I-Flow’s MATH500 rollouts and added the analysis to RQ2 in Section 5.3, with judging prompts in Appendix C.5. GPT-4o-mini evaluates whether (1) the high-level guidance matches the selected intent and (2) the low-level reasoning follows that guidance. As shown in Table R.4 below, Both consistency rates exceed 92% across all six intents, providing empirical evidence that the intended relationships among intent, guidance, and reasoning are maintained. This likely reflects how BF-SubTB supports exploration of diverse successful intent structures and ISR re-executes those structures with freshly generated guidance and reasoning. Repeated learning from successful trajectories under these structures may help maintain semantic alignment among the selected intent, high-level guidance, and low-level reasoning.
Table R.4. Semantic consistency (%) of I-Flow by intent type, reproduced from Table 5 in Section 5.3.
| Intent | (1) Intent–Guidance | (2) Guidance–Reasoning |
|---|---|---|
| EXP | 92.24 | 96.75 |
| VER | 98.55 | 94.24 |
| BCK | 95.80 | 92.43 |
| CON | 94.88 | 93.24 |
| FIN | 96.09 | 93.16 |
| IDL | 93.13 | 95.99 |
Q3 (Explaining Performance Gains):
We thank the reviewer for asking whether I-Flow’s performance gains can be explained by additional training compute alone. To examine this, we compare I-Flow and ReMA under approximately matched total training times in Appendix E.1 and analyze the corresponding trajectory lengths and hierarchy-collapse rates in Section 5.4. Under identical maximum token and turn limits, I-Flow takes approximately 54% longer per training step than ReMA, while flow-network training itself accounts for only about 0.11%. The additional time primarily reflects generating and optimizing longer trajectories that sustain hierarchical multi-turn reasoning, whereas ReMA’s trajectories shorten alongside hierarchy collapse and reasoning simplification.
To assess whether additional training can compensate for this difference, we extended ReMA from 250 to 400 steps, approximately matching I-Flow’s total training time at 250 steps. Even with this extended training, ReMA’s turn and token counts continued to decrease, and its hierarchy collapse rate remained around 60%, compared with 3.21% for I-Flow (Section 5.4). I-Flow also achieved higher scores than ReMA’s 300- and 400-step checkpoints on all seven mathematical reasoning benchmarks (Appendix E.1). These results suggest that additional training alone has limited effects on the observed hierarchy collapse and reasoning simplification. Importantly, preserving distinct high-level guidance and low-level execution enables verification, revision, and adaptive reasoning across turns, which are central benefits of hierarchical multi-turn reasoning. Thus, I-Flow’s additional computational cost largely reflects its ability to sustain these capabilities throughout training.
The theoretical clarification and empirical analyses support the usefulness of learned flows as intent-level learning signals and show high semantic consistency between selected intents, guidance, and execution in the evaluated outputs. We sincerely thank the reviewer for their valuable feedback, which helped us clarify the learning mechanism and strengthen the evidence for its effectiveness.
Thank you for the detailed response and additional experimental analyses. The authors' clarifications have addressed most of my concerns. I will maintain my original score of 3 (Weak Acceptance).
Thank you for your prompt response and for taking the time to consider our rebuttal. We appreciate your thoughtful feedback.
Authors
We sincerely thank the reviewer for their detailed comments and questions about I-Flow’s learning mechanism. In particular, we clarify how learned flows support intent-level credit assignment and how the intent taxonomy can be adapted, and provide semantic-consistency evaluations and extended-training comparisons to further examine the effectiveness of the proposed approach.
W1/Q2 (Intent-Level Credit Assignment):
We thank the reviewer for this important question. I-Flow distinguishes intents not by the terminal reward alone, but by the flow Fϕ(st) learned at every post-intent state, which serves as an intermediate signal, much like learning a value function in RL instead of relying on the final return. This follows from Eq. (13), πI∗(ut∣ct)∝πrefI(ut∣ct)F∗(ct,ut): since the optimal intent policy is determined by the intent-conditioned flow, an intent with higher flow at a given state is a better one, and an intent with lower flow is a worse one. Proposition 1 guarantees that the BF-SubTB conditions identify Fϕ=F∗ at every post-intent state, so the learned flow serves as this criterion.
In contrast, the terminal-reward-only TB variant also evaluated in Section 5.4 imposes only a single constraint from the initial state to the terminal state, so no intermediate flow is learned and no such criterion is available. While TB constrains only the sum of local balance residuals, where positive and negative errors can cancel, I-Flow bounds their sum of squares by its training loss up to a constant factor (Proposition 2), thereby providing finer intermediate supervision with the same terminal reward. Empirically, Figure. 9 in Appendix D.4 shows that, from the same reasoning prefix, selecting the highest-flow intent yields a 64.2% success rate, compared with 55.8% for random and 46.1% for lowest-flow selection, confirming that the flow indeed distinguishes intents that contribute to success. We thank the reviewer again for this insightful question.
W2 (Adaptability of the Intent Taxonomy):
We thank the reviewer for raising this concern. Our taxonomy consists of five core reasoning intents (
EXP,VER,BCK,CON, andFIN) reflecting reasoning operations identified in prior work on hierarchical reasoning for mathematical and coding tasks [R.1, R.2]. Alongside these five core intents, the taxonomy already includesIDLfor general progress, accommodating steps outside them. For different domains, categories can be added, merged, or refined without modifying I-Flow’s optimization objective, with corresponding updates to SFT intent annotations. For example, in document-based question answering, behaviors assigned toIDLcould be refined into SELECT and SYNTHESIZE for evidence identification and integration, respectively. Thus, the intent taxonomy can be adapted to different reasoning requirements while preserving I-Flow's core optimization framework. We have clarified the taxonomy's rationale in Section 5.1 and expanded Appendix A with this adaptation example and a discussion of its scope.[R.1] Wan, Ziyu, et al. “ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning.” NeurIPS, 2025.
[R.2] Shahariar, G. M., et al. “Modeling Hierarchical Thinking in Large Reasoning Models.” ICML, 2026.