- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposed rethink for long-horizon agents. The paper formulate self-correction as an explicit tool, RETHINK, where a helper makes the verification.
Critical strengths
None.
Critical weaknesses
- Discussions with related works may not be complete, for example, the following work is not discussed or compared. https://arxiv.org/abs/2609.11061
Fork Where the Model Changes Its Mind: Belief-Shift Branching for Tree-Structured Reinforcement Learning
-
There is no theoretical or analytical analysis
-
Numerical results were done only for small-sized models.
-
The novelty of this paper is limited: it is very natural to "rethink" with a help of RL algorithms. This paper did not clearly explain what is the biggest nontrivial innovation point in the proposed idea.
-
The proposed approach appears to provide OK results in Table 2, but the comparisons with other stronger approaches are missing.
-
Some notations are not well defined, for example, o and an are not defined in the explanations, just assuming everybody will see it as action and observations.
What is the difference between x and x_t?
In equation (8), can you better explain u, v and U_{z}, and their relationship?
Questions for authors
-
Could you identify the fundamental innovative ideas that make this paper different from others?
From its current form, it is not clear what is.
-
Could you provide some analytical results on the proposed approach, such as sample complexity of the RL algorithms, theoretically where the performance gain comes from.
-
Could you extend the numerical results to larger models and more challenging reasoning tasks.
-
In this paper, there is only one main agent. What if there are multiple agents which can explore different branches of reasoning paths?
4 comments
Reply to Xz9B Part.2
Question 1: Fundamental innovative ideas
The contribution is the joint design of self-triggered consultation and credit assignment for learning to use it. The main agent requests targeted advice from a helper sharing its parameters but operating on selected evidence in a fresh context. Direct-execution controls, alternative advice, and multiple continuations provide empirical relative-value estimates for invocation, advice generation, and execution, alongside root credit. This structure distinguishes learning when to consult from learning what to advise and how to act afterward. The Tree-Score ablation investigates the value of this decomposition.
Question 2: Analytical justification and sample complexity
An estimator-level analysis explains the sampling design. Fix the policy, prefix , and advice , and let be the conditional success probability. For conditionally independent binary continuation returns, satisfies . Multiple continuations reduce execution noise when evaluating advice, while alternative advice samples permit comparisons between suggestions. This is a local estimation result, not a sample-complexity guarantee for the complete RL algorithm.
TreeRL [2] uses uncertainty-guided branching, while Wang et al. [3] identify high-entropy tokens as consequential reasoning decisions. These provide complementary motivation for concentrating computation on informative decisions. Our method constructs comparisons around consultation points, with its overall effectiveness evaluated through the sampling studies, frozen-agent analysis, and new ablation. The absence of a general convergence theorem does not negate this empirical and estimator-level support.
Question 3: Larger models and more challenging tasks
Table 1 already evaluates the RETHINK interface on Apex Agents and τ³ using the additional models listed above. These experiments complement the joint RL results on AppWorld and BFCL. Extending joint RL beyond Qwen3-8B is a future scaling direction, and WebShop experiments are underway to broaden the evaluation.
Question 4: Multiple agents exploring different reasoning paths
Our training already samples alternative helper responses and multiple continuations; ordinary inference returns one helper response per call. Multiple independently acting agents would additionally require coordination and proposal selection. Sec. 4.3 and Table 3 examine related questions by varying helper capacity and training the helper with a fixed main agent. These results motivate improving both advice and its use: a recommendation must still be translated into effective actions. Additional agents could complement our framework, but would not remove the invocation, advice, and execution credit-assignment problem.
Closing remarks
We therefore partially agree with Reviewer Xz9B on the need for clearer exposition and broader experimental coverage. The clarifications and additional controls strengthen the evidence for learning when to consult and how to use advice. The methodological contribution and existing model coverage should be assessed in light of these distinctions and results.
References
[1] Lei et al. Fork Where the Model Changes Its Mind: Belief-Shift Branching for Tree-Structured Reinforcement Learning. arXiv:2609.11061, 2026. Paper.
[2] Hou et al. TreeRL: LLM Reinforcement Learning with On-Policy Tree Search. ACL, 2025. Paper.
[3] Wang et al. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. NeurIPS, 2025. Paper.
The authors have addressed the notations clarity issue, but I still think the fork idea is straightforward and limited in novelty.
Combined with limited analysis, I would maintain my current score.
We thank the reviewer for the follow-up and are glad the notation issues are resolved. We would like to briefly clarify three points.
On novelty. We respectfully disagree that our contribution lacks novelty. "Straightforward" describes how a method reads in hindsight, not whether it is novel. The ICLR 2027 Reviewer Guide asks whether a submission presents "novel findings (e.g. theoretically, algorithmically, etc.)" and "new, relevant, impactful knowledge (incl., empirical, theoretical, for practitioners, etc)". Our paper makes both an algorithmic and an empirical contribution: Algorithmic. A consultation request is an optional action, not a recurring state, so state-grouped credit such as GiGPO cannot evaluate it. Our counterfactual branches compare calling with not calling, alternative pieces of advice, and continuations under the same advice, and give separate credit to each behavior. Reviewer LWtV highlights this point. Empirical. The straightforward versions of the idea fail. Adding the RETHINK interface through SFT alone lowers performance on BFCL (mean@8 41.38 → 37.00) and WebShop (mean@2 55.47 → 53.13). Removing our credit decomposition loses 5.49 (AppWorld) and 3.00 (BFCL) mean@8 points. If the idea were as straightforward as suggested, these simpler versions should already work. Moreover, the review has not identified any prior work that learns self-triggered rethinking with this credit structure. The only work it cites is an arXiv-only preprint posted on 10 September 2026, and the Reviewer Guide states that the lack of comparisons to such work "cannot be a basis for rejection".
On the focus of the paper. Branching is a common tool, and several works study where to branch, including the paper the reviewer cited. Our question is different: can an agent learn, through post-training, when to rethink and how to use the resulting advice? The fork is the instrument we use to answer this question, not the claim itself. Our answer is positive. With a comparable or larger sampling budget for the baselines (group size 32), RETHINK outperforms GRPO and GiGPO on AppWorld and BFCL, and it also ranks first on WebShop.
On analysis. Beyond the main results, the paper includes: a matched-prefix analysis (Appendix B), where rethinking beats direct action from the same prefix by 5.99 pp (95% CI [3.14, 9.10]); a fork-position study (Appendix C); a frozen-agent study (Section 4.3). The rebuttal adds matched-budget baselines, the Tree-Score ablation, an estimator-level variance analysis, and the WebShop results.
We respect the reviewer's judgment and hope these clarifications are helpful.
Reply to Xz9B Part.1
We partially agree with Reviewer Xz9B that the presentation, notation, and experimental coverage can be improved. We address these points through clearer definitions and additional controls. However, we disagree with several broader judgments about novelty, model coverage, and the treatment of concurrent work.
Weakness 1: Relationship to contemporaneous work
The cited preprint [1] appeared on 10 September 2026, approximately two weeks before the ICLR 2027 full-paper deadline of 25 September 2026 (AoE). The ICLR 2027 Reviewer Guidelines state that comparisons with contemporaneous work or unpublished arXiv papers are not required and that their absence cannot justify rejection. We will add the citation, but disagree with treating the missing comparison as grounds for rejection. The methodological distinction also matters: [1] selects training-time fork locations, whereas our method learns an optional consultation and assigns separate credit to invocation, advice, and execution. Branching alone is not our novelty claim.
Weaknesses 2–4: Novelty, analysis, and model scale
Our contribution goes beyond simply applying RL to “rethink.” Terminal rewards entangle the consultation decision, advice quality, and subsequent execution; Tree-Score assigns separate learning signals to these components. The paper also includes empirical sampling analyses in Appendices B–C and a frozen-main-agent study in Sec. 4.3. These analyses should be distinguished from a general convergence theorem, which we do not claim.
The statement that all numerical results concern small models is inaccurate. Table 1 evaluates Kimi K2.5, DeepSeek-V4.1-Flash, and Qwen3.6-Flash, while the appendix includes Qwen3.8-27B. Our joint RL experiments use Qwen3-8B; the largest backbone trained with RL in the cited work [1] is 9B, a comparable parameter scale. Its 31B/32B models serve as pre-RL probes. The scope of RL training should therefore be distinguished from a paper's overall model coverage.
Weakness 5: Comparisons with stronger approaches
The review does not specify which stronger approaches are missing. Our goal is to demonstrate the benefit of learning when and how to rethink, rather than to develop a new state-of-the-art RL algorithm. Our comparisons use the same Qwen3-8B backbone and task-specific SFT initialization. We additionally evaluate GRPO and GiGPO with group size 32 and RETHINK without Tree-Score. RETHINK achieves mean@8 of 73.25 versus 71.71 for GiGPO on AppWorld and 66.88 versus 60.25 on BFCL. Removing Tree-Score reduces mean@8 by 5.49 and 3.00 percentage points, respectively. These controls examine increased root-level sampling and the contribution of Tree-Score within RETHINK, supporting the claimed benefit of learning selective consultation.
Weakness 6: Notation and Eq. (8)
We will make the notation more explicit. Here, an is the main agent's action and on its resulting observation; ht=(x,a0,o0,…,at−1,ot−1) records completed interactions before decision t. The variable x denotes the fixed benchmark task, whereas xt denotes the active user request, as defined after Eq. (1). In multi-turn BFCL, xt can change within one task.
In Eq. (8), u is a sampled token position, z is a credit channel, and v∈Uz is a unit in that channel: a root trajectory, valid natural RETHINK call, helper response, or continuation. Each unit has an advantage Avz and a trainable token span. The indicator 1{u∈v} selects units covering token u; their advantages are normalized by ∣Uz∣ and summed across channels. Empty channels contribute zero, and copied context and tool observations are excluded from trainable spans.