acceptodds
Under review as a conference paper at ICLR 2027

Fork of Thought: Eliciting Self-Triggered Rethinking in LLM Agents via Branch-Sampled Reinforcement Learning

Abstract

LLM agents can benefit from rethinking during multi-step interaction, but effective rethinking requires learning both when to intervene and how to provide useful guidance. Jointly training these behaviors is challenging: task-level rewards provide indirect credit for individual interventions, and advice can become less effective as the acting policy evolves. We introduce Fork of Thought, a reinforcement learning framework for learning self-triggered rethinking within a single LLM. At selected decision points, the framework branches a shared interaction prefix into alternative advice-conditioned continuations and a no-intervention control. These branches provide comparative feedback for learning when to invoke rethinking and how to generate useful advice, while task-level rewards train the agent’s overall behavior. We further adopt a two-stage training procedure that first focuses on rethinking under a constrained acting policy, then jointly optimizes acting and rethinking. The framework aims to align rethinking with the evolving agent policy and make it an adaptive component of sequential decision-making.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic RL, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Xz9BOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposed rethink for long-horizon agents. The paper formulate self-correction as an explicit tool, RETHINK, where a helper makes the verification.

Critical strengths

None.

Critical weaknesses
  1. Discussions with related works may not be complete, for example, the following work is not discussed or compared. https://arxiv.org/abs/2609.11061

Fork Where the Model Changes Its Mind: Belief-Shift Branching for Tree-Structured Reinforcement Learning

  1. There is no theoretical or analytical analysis

  2. Numerical results were done only for small-sized models.

  3. The novelty of this paper is limited: it is very natural to "rethink" with a help of RL algorithms. This paper did not clearly explain what is the biggest nontrivial innovation point in the proposed idea.

  4. The proposed approach appears to provide OK results in Table 2, but the comparisons with other stronger approaches are missing.

  5. Some notations are not well defined, for example, o and an are not defined in the explanations, just assuming everybody will see it as action and observations.

What is the difference between x and x_t?

In equation (8), can you better explain u, v and U_{z}, and their relationship?

Questions for authors
  1. Could you identify the fundamental innovative ideas that make this paper different from others?

    From its current form, it is not clear what is.

  2. Could you provide some analytical results on the proposed approach, such as sample complexity of the RL algorithms, theoretically where the performance gain comes from.

  3. Could you extend the numerical results to larger models and more challenging reasoning tasks.

  4. In this paper, there is only one main agent. What if there are multiple agents which can explore different branches of reasoning paths?

4 comments
AuthorsOpenReview ↗

Reply to Xz9B Part.1

We partially agree with Reviewer Xz9B that the presentation, notation, and experimental coverage can be improved. We address these points through clearer definitions and additional controls. However, we disagree with several broader judgments about novelty, model coverage, and the treatment of concurrent work.

Weakness 1: Relationship to contemporaneous work

The cited preprint [1] appeared on 10 September 2026, approximately two weeks before the ICLR 2027 full-paper deadline of 25 September 2026 (AoE). The ICLR 2027 Reviewer Guidelines state that comparisons with contemporaneous work or unpublished arXiv papers are not required and that their absence cannot justify rejection. We will add the citation, but disagree with treating the missing comparison as grounds for rejection. The methodological distinction also matters: [1] selects training-time fork locations, whereas our method learns an optional consultation and assigns separate credit to invocation, advice, and execution. Branching alone is not our novelty claim.

Weaknesses 2–4: Novelty, analysis, and model scale

Our contribution goes beyond simply applying RL to “rethink.” Terminal rewards entangle the consultation decision, advice quality, and subsequent execution; Tree-Score assigns separate learning signals to these components. The paper also includes empirical sampling analyses in Appendices B–C and a frozen-main-agent study in Sec. 4.3. These analyses should be distinguished from a general convergence theorem, which we do not claim.

The statement that all numerical results concern small models is inaccurate. Table 1 evaluates Kimi K2.5, DeepSeek-V4.1-Flash, and Qwen3.6-Flash, while the appendix includes Qwen3.8-27B. Our joint RL experiments use Qwen3-8B; the largest backbone trained with RL in the cited work [1] is 9B, a comparable parameter scale. Its 31B/32B models serve as pre-RL probes. The scope of RL training should therefore be distinguished from a paper's overall model coverage.

Weakness 5: Comparisons with stronger approaches

The review does not specify which stronger approaches are missing. Our goal is to demonstrate the benefit of learning when and how to rethink, rather than to develop a new state-of-the-art RL algorithm. Our comparisons use the same Qwen3-8B backbone and task-specific SFT initialization. We additionally evaluate GRPO and GiGPO with group size 32 and RETHINK without Tree-Score. RETHINK achieves mean@8 of 73.25 versus 71.71 for GiGPO on AppWorld and 66.88 versus 60.25 on BFCL. Removing Tree-Score reduces mean@8 by 5.49 and 3.00 percentage points, respectively. These controls examine increased root-level sampling and the contribution of Tree-Score within RETHINK, supporting the claimed benefit of learning selective consultation.

Weakness 6: Notation and Eq. (8)

We will make the notation more explicit. Here, ana_n is the main agent's action and ono_n its resulting observation; ht=(x,a0,o0,…,at−1,ot−1)h_t=(x,a_0,o_0,\ldots,a_{t-1},o_{t-1}) records completed interactions before decision tt. The variable xx denotes the fixed benchmark task, whereas xtx_t denotes the active user request, as defined after Eq. (1). In multi-turn BFCL, xtx_t can change within one task.

In Eq. (8), uu is a sampled token position, zz is a credit channel, and v∈Uzv\in\mathcal U_z is a unit in that channel: a root trajectory, valid natural RETHINK call, helper response, or continuation. Each unit has an advantage AvzA_v^z and a trainable token span. The indicator 1{u∈v}\mathbf{1}\{u\in v\} selects units covering token uu; their advantages are normalized by ∣Uz∣|\mathcal U_z| and summed across channels. Empty channels contribute zero, and copied context and tool observations are excluded from trainable spans.

AuthorsOpenReview ↗

Reply to Xz9B Part.2

Question 1: Fundamental innovative ideas

The contribution is the joint design of self-triggered consultation and credit assignment for learning to use it. The main agent requests targeted advice from a helper sharing its parameters but operating on selected evidence in a fresh context. Direct-execution controls, alternative advice, and multiple continuations provide empirical relative-value estimates for invocation, advice generation, and execution, alongside root credit. This structure distinguishes learning when to consult from learning what to advise and how to act afterward. The Tree-Score ablation investigates the value of this decomposition.

Question 2: Analytical justification and sample complexity

An estimator-level analysis explains the sampling design. Fix the policy, prefix hh, and advice gg, and let qgq_g be the conditional success probability. For MM conditionally independent binary continuation returns, q^g=M−1∑m=1MRm\widehat q_g=M^{-1}\sum_{m=1}^M R_m satisfies Var⁡(q^g∣h,g)=qg(1−qg)/M≤1/(4M)\operatorname{Var}(\widehat q_g\mid h,g)=q_g(1-q_g)/M\leq1/(4M). Multiple continuations reduce execution noise when evaluating advice, while alternative advice samples permit comparisons between suggestions. This is a local estimation result, not a sample-complexity guarantee for the complete RL algorithm.

TreeRL [2] uses uncertainty-guided branching, while Wang et al. [3] identify high-entropy tokens as consequential reasoning decisions. These provide complementary motivation for concentrating computation on informative decisions. Our method constructs comparisons around consultation points, with its overall effectiveness evaluated through the sampling studies, frozen-agent analysis, and new ablation. The absence of a general convergence theorem does not negate this empirical and estimator-level support.

Question 3: Larger models and more challenging tasks

Table 1 already evaluates the RETHINK interface on Apex Agents and τ³ using the additional models listed above. These experiments complement the joint RL results on AppWorld and BFCL. Extending joint RL beyond Qwen3-8B is a future scaling direction, and WebShop experiments are underway to broaden the evaluation.

Question 4: Multiple agents exploring different reasoning paths

Our training already samples alternative helper responses and multiple continuations; ordinary inference returns one helper response per call. Multiple independently acting agents would additionally require coordination and proposal selection. Sec. 4.3 and Table 3 examine related questions by varying helper capacity and training the helper with a fixed main agent. These results motivate improving both advice and its use: a recommendation must still be translated into effective actions. Additional agents could complement our framework, but would not remove the invocation, advice, and execution credit-assignment problem.

Closing remarks

We therefore partially agree with Reviewer Xz9B on the need for clearer exposition and broader experimental coverage. The clarifications and additional controls strengthen the evidence for learning when to consult and how to use advice. The methodological contribution and existing model coverage should be assessed in light of these distinctions and results.

References

[1] Lei et al. Fork Where the Model Changes Its Mind: Belief-Shift Branching for Tree-Structured Reinforcement Learning. arXiv:2609.11061, 2026. Paper.

[2] Hou et al. TreeRL: LLM Reinforcement Learning with On-Policy Tree Search. ACL, 2025. Paper.

[3] Wang et al. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. NeurIPS, 2025. Paper.

The authors have addressed the notations clarity issue, but I still think the fork idea is straightforward and limited in novelty.

Combined with limited analysis, I would maintain my current score.

We thank the reviewer for the follow-up and are glad the notation issues are resolved. We would like to briefly clarify three points.

On novelty. We respectfully disagree that our contribution lacks novelty. "Straightforward" describes how a method reads in hindsight, not whether it is novel. The ICLR 2027 Reviewer Guide asks whether a submission presents "novel findings (e.g. theoretically, algorithmically, etc.)" and "new, relevant, impactful knowledge (incl., empirical, theoretical, for practitioners, etc)". Our paper makes both an algorithmic and an empirical contribution: Algorithmic. A consultation request is an optional action, not a recurring state, so state-grouped credit such as GiGPO cannot evaluate it. Our counterfactual branches compare calling with not calling, alternative pieces of advice, and continuations under the same advice, and give separate credit to each behavior. Reviewer LWtV highlights this point. Empirical. The straightforward versions of the idea fail. Adding the RETHINK interface through SFT alone lowers performance on BFCL (mean@8 41.38 → 37.00) and WebShop (mean@2 55.47 → 53.13). Removing our credit decomposition loses 5.49 (AppWorld) and 3.00 (BFCL) mean@8 points. If the idea were as straightforward as suggested, these simpler versions should already work. Moreover, the review has not identified any prior work that learns self-triggered rethinking with this credit structure. The only work it cites is an arXiv-only preprint posted on 10 September 2026, and the Reviewer Guide states that the lack of comparisons to such work "cannot be a basis for rejection".

On the focus of the paper. Branching is a common tool, and several works study where to branch, including the paper the reviewer cited. Our question is different: can an agent learn, through post-training, when to rethink and how to use the resulting advice? The fork is the instrument we use to answer this question, not the claim itself. Our answer is positive. With a comparable or larger sampling budget for the baselines (group size 32), RETHINK outperforms GRPO and GiGPO on AppWorld and BFCL, and it also ranks first on WebShop.

On analysis. Beyond the main results, the paper includes: a matched-prefix analysis (Appendix B), where rethinking beats direct action from the same prefix by 5.99 pp (95% CI [3.14, 9.10]); a fork-position study (Appendix C); a frozen-agent study (Section 4.3). The rebuttal adds matched-budget baselines, the Tree-Score ablation, an estimator-level variance analysis, and the WebShop results.

We respect the reviewer's judgment and hope these clarifications are helpful.

Reviewer_vfCrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Fork of Thought, which enables LLM agents to reconsider previous decisions through a RETHINK tool operating in a fresh context. The authors further propose a counterfactual fork-based RL framework that assigns separate credit to RETHINK invocation, helper advice, and subsequent actions. Experiments across multiple models and agent benchmarks demonstrate performance improvements over standard baselines.

Critical strengths
  1. Meaningful method design.

The paper addresses an important challenge in long-horizon agents, where earlier mistakes can affect subsequent decisions. The combination of fresh-context rethinking and counterfactual credit assignment provides a reasonable and useful approach to learning when and how agents should correct themselves.

  1. Solid experimental evaluation.

RETHINK shows overall improvements across multiple strong models and benchmarks. The joint RL method also outperforms GRPO and GiGPO on both AppWorld and BFCL. Additional analyses of frozen-main agents, fork sampling, and task difficulty provide further insights into the method's effectiveness and limitations.

Critical weaknesses

Insufficient ablation of the proposed credit assignment.

The GRPO and GiGPO baselines do not use RETHINK, making it difficult to separate the benefits of the consultation mechanism, additional training rollouts, and the proposed credit assignment. A comparison with standard GRPO using the same RETHINK scaffold and a matched training budget is missing. As a result, the necessity of the fine-grained credit assignment is not fully established.

Questions for authors

Can the authors compare the proposed method with standard GRPO using the same RETHINK scaffold and a matched training budget? This would help clarify the independent contribution of the fine-grained credit assignment.

1 comment

Thank you for your constructive review comments and insightful questions! We have provided responses below to address each of your concerns.

Weakness & Question: Ablation of the proposed credit assignment

Insufficient ablation of the proposed credit assignment. The GRPO and GiGPO baselines do not use RETHINK, making it difficult to separate the benefits of the consultation mechanism, additional training rollouts, and the proposed credit assignment. A comparison with standard GRPO using the same RETHINK scaffold and a matched training budget is missing.

Our RETHINK w/o Tree-Score ablation retains the same RETHINK scaffold, including the agent’s ability to invoke the helper, and removes the Tree-Score credit decomposition. With Tree-Score, mean@8 improves by 5.49 percentage points on AppWorld and 3.00 points on BFCL. To test whether increasing root-level sampling closes the gap, we also trained GRPO and GiGPO with 32 rollouts per task. This group size is close to the higher of the two observed RETHINK sampling counts (32.8 samples per task on BFCL) and exceeds the lower one (13.4 on AppWorld). RETHINK achieves higher mean@8 and pass@8 than both larger-group baselines on both benchmarks.

Table R.1: Performance comparison with the Tree-Score ablation and baselines trained with larger sampling groups.

MethodAppWorld mean@8AppWorld pass@8BFCL mean@8BFCL pass@8
GRPO (group=32)69.52 ± 1.2087.72 ± 1.7562.50 ± 0.9066.00 ± 1.00
GiGPO (group=32)71.71 ± 1.3087.72 ± 1.7560.25 ± 1.0063.67 ± 1.00
RETHINK w/o Tree-Score67.76 ± 2.6084.21 ± 3.5163.88 ± 2.0068.00 ± 3.00
RETHINK (ours)73.25 ± 1.5089.47 ± 1.7566.88 ± 1.1078.33 ± 1.53
Reviewer_LWtVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses self-correction in long-horizon LLM agents. Errors introduced by early decisions often manifest only after many subsequent steps: conventional continuation corrects the most recent action but cannot revisit the earlier decision that caused the problem, whereas self-reflection conditioned on the full interaction history tends to be anchored by the model's own earlier mistaken conclusions, reducing self-correction to self-confirmation. The core idea of this paper is to realize the reconsideration of earlier decisions as an explicit tool. At moments of its own choosing, the main agent emits a targeted question; a fixed rule selects fragments of environmental feedback from the trajectory, which are passed to a rethinking helper, an instance of the same model with shared parameters operating in a fresh, uncontaminated context. The helper may issue read-only queries to the environment and returns concrete advice rather than a verdict on correctness; the main agent then incorporates the advice and continues execution. Training proceeds in two stages. First, supervised fine-tuning on teacher-model demonstrations equips the policy with the invocation format and covers both the acting and advising roles. Reinforcement learning is then applied, in which the rollout at each invocation point is expanded counterfactually into control branches and multiple advice-conditioned branches, each followed by sampled continuations; the terminal reward is thereby decomposed into distinct credit channels, jointly optimizing the agent's complete consultation policy under budget constraints.

Critical strengths
  1. This work is insightful in its problem positioning. In agentic settings, grounding reflection in environmental feedback is a well-motivated line of attack: the failure of single-turn intrinsic self-correction stems from information poverty, and in long-horizon agents every tool call yields an external observation, which naturally removes this limitation. More notably, the authors offer a two-level diagnosis of the difficulty: first, the consequences of early decisions often manifest only many steps later, and ordinary continuation corrects only the most recent action without revisiting the earlier decision that caused the problem; second, the agent's own earlier interpretations accumulate and solidify in the context, causing self-reflection over the full history to degenerate into self-confirmation. This diagnosis distinguishes the presence of feedback from its usability, and motivates isolating earlier erroneous conclusions behind a clean context and selectively chosen raw evidence. The matched-prefix experiments in Appendix B support this positioning.

  2. The authors consciously address the credit assignment between tool-invocation behavior and reflection behavior. A successful trajectory may owe to well-timed consultation, useful advice, or effective execution after the advice; the standard trajectory-level advantage cannot distinguish these contributions, and methods such as GiGPO rely on repeated visits to identical environment states for grouping, whereas requesting advice does not constitute a recurring state. This paper therefore constructs counterfactual forks at each invocation point and decomposes the terminal reward into independent credit channels, applied respectively to the tokens that initiate the call, the tokens that generate the advice, and the action tokens that follow the advice. The effectiveness of this decomposition is supported experimentally: SFT alone with the interface degrades performance on BFCL, whereas the full joint learning consistently outperforms GRPO and GiGPO across both environments — indicating that per-channel credit assignment, rather than the interface itself, is the active ingredient of the gains.

Critical weaknesses
  1. The method builds upon prior work (GiGPO's grouped relative advantages, SCoRE's RL-learned self-correction, ReGRPO's incorporation of reflection into RL, and CRITIC's tool-grounded correction), and its increment lies in constructing comparison groups around an optional tool call and assigning separate credit to invocation, advice, and execution. However, this mechanism-level increment has not been isolated experimentally: the margin over GiGPO in the main table is small, and the fork construction in training substantially increases the number of samples per task, so the gains may stem primarily from the larger sampling scale rather than from the credit decomposition itself. In the absence of a controlled comparison that distinguishes the two, the necessity and effectiveness of this core mechanism cannot be adequately assessed.

  2. The paper's central hypothesis, namely that re-reasoning conditioned on the full history reinforces earlier mistaken assumptions, is intuitively plausible but currently serves mainly as a design motivation and has not been experimentally tested: the paper provides no comparison between a helper conditioned on the full history and one conditioned on selected evidence, nor does it specify under what conditions, and why, the hypothesis holds. The matched-prefix analysis in Appendix B offers correlational evidence, but it cannot substitute for an isolation test of the hypothesis itself.

  3. The paper does not adequately establish that the performance margins in the main tables exceed training noise. The validation curves in Figure 2 show mean@8 fluctuating between 53.73 and 59.21 across checkpoints, a range of about 5.5 percentage points; the joint-training margin over GiGPO is only +1.76 (AppWorld mean@8), which amounts to roughly one task out of the 57 development tasks. Since the paper reports neither multi-seed variance nor error bars, a margin of this magnitude is difficult to distinguish from training randomness.

Questions for authors
  1. Checkpoints are selected by mean@8 on the AppWorld dev set, while Table 2 reports results on that same dev set. Does this constitute model selection on the reporting set, and could it introduce a systematic optimistic bias? In addition, please clarify the basis for checkpoint selection: the phrase "among its available evaluations" in Appendix A does not specify whether the mean@8 used for selection is computed on AppWorld dev only, or also incorporates the validation evaluations on the held-out BFCL tasks.

  2. Does the gain stem primarily from more samples per task, or from the credit decomposition mechanism itself? If the budget consumed by the fork construction in training (two control branches, four advice samples, and two continuations per advice at each invocation point) were instead spent on additional GiGPO-style root trajectories, would the method still maintain its lead over GiGPO?

  3. The validation curves in Figure 2 show mean@8 fluctuating between 53.73 and 59.21 across checkpoints, a range of about 5.5 percentage points; the joint-training margin over GiGPO is only +1.76 (AppWorld mean@8) Since the paper reports neither multi-seed variance nor error bars, how can the readers distinguish a margin of this magnitude from training randomness?

4 comments
AuthorsOpenReview ↗

Reply to LWtV Part.1

We greatly appreciate your thoughtful review and valuable questions. Below, we have provided detailed responses to address each of your comments and concerns.

Weakness 1: Credit assignment vs. sampling scale

Thank you for highlighting the need to distinguish the contribution of our credit assignment mechanism from the effect of increased sampling. We measured the average number of samples per task in the two RETHINK RL experiments, obtaining 32.8 and 13.4. To examine whether increasing the sampling group alone explains the gains, we additionally trained GRPO and GiGPO with a group size of 32. As shown below, RETHINK outperforms both baselines on both metrics across the two benchmarks. We also identified and corrected a bug in BFCL environment cloning: function-name rewriting inadvertently modified string literals, introducing namespaces from other environment instances. The corrected BFCL results are reported below and will replace the corresponding entries in the main table.

Tabel R.1: Performance Comparision

MethodAppWorld mean@8AppWorld pass@8BFCL mean@8BFCL pass@8
GRPO (group=32)69.52 ± 1.2087.72 ± 1.7562.50 ± 0.9066.00 ± 1.00
GiGPO (group=32)71.71 ± 1.3087.72 ± 1.7560.25 ± 1.0063.67 ± 1.00
RETHINK w/o Tree-Score67.76 ± 2.6084.21 ± 3.5163.88 ± 2.0068.00 ± 3.00
RETHINK (ours)73.25 ± 1.5089.47 ± 1.7566.88 ± 1.1078.33 ± 1.53

The Tree-Score ablation further addresses the mechanism-level concern: removing Tree-Score reduces mean@8 by 5.49 percentage points on AppWorld and 3.00 percentage points on BFCL, supporting the contribution of the proposed credit assignment. We would also like to clarify the scope of our contribution. Tree-Score is designed specifically to teach agents when and how to rethink, through separate credit assignment for invocation, advice generation, and subsequent execution. Our broader objective is to establish the value of self-triggered reflection in long-horizon agent tasks and investigate when it can be more beneficial than reflection in conventional question-answering settings. The additional baseline comparisons and ablation strengthen the empirical support for this targeted training mechanism.

Weakness 2: Untested full-history hypothesis

Thank you for raising this important point. Our hypothesis is motivated primarily by empirical observations in long-horizon agent settings. On Apex Agents, we observed that a helper using a truncated history together with read-only inspection tools outperformed a helper conditioned on the full history, as summarized in Table R.2. These results support the practical effectiveness of combining selective context with environmental verification. We agree, however, that evidence for this configuration should be distinguished from an experiment that isolates the proposed mechanism, and that the matched-prefix analysis in Appendix B does not independently establish why the difference occurs.

Table R.2 Matched Fork Analysis

Matched-call fork analysisWindow pass@1Full-history pass@1Window mean rubricFull-history mean rubric
All scored pairs15.04%14.16%30.56%28.51%

To isolate the effect of helper context, we run a matched-call intervention on Apex Agents using Kimi K2.5 as both the main agent and the RETHINK helper. Two Kimi K2.5 helpers answer the same question in parallel, receiving either the full observed history or the recent window requested by the agent. We then resume two independent branches from the saved state, each with its assigned advice, the same remaining action budget, and the same policy for later RETHINK calls. Both outcomes are scored with the official Apex Agents rubric using DeepSeek Flash 1017 as judge.

Our intuition is that generating another continuation from the same history may preserve an earlier mistaken assumption, particularly when an unsuccessful tool call has led the agent to an unsupported conclusion about the environment. Read-only inspection allows the helper to check that conclusion against the environment. For example, a Kimi K2.5 agent searched for a key file using incorrect capitalization, failed to locate it, and consequently concluded that the file did not exist. A RETHINK helper can inspect the relevant environment information and identify the capitalization mismatch. This example illustrates a setting in which we expect rethinking to help: an earlier interpretation is incorrect, and verifiable environmental evidence is available to challenge it.

AuthorsOpenReview ↗

Reply to LWtV Part.2

Weakness 3: Small gains amid training noise

Thank you for raising this important concern. Our evaluation and checkpoint-selection protocol follows the settings used in AgentEvolver[1] and Elastic Horizon[2]: we evaluate every 10 training steps, yielding 11 evaluations over each 100-step training run, including the initial evaluation. Within each run, we select the checkpoint with the highest validation mean@8. We provide multi-seed varience in the previous table.

We acknowledge that performance on the AppWorld development set fluctuates substantially across checkpoints, and that three-seed averages alone do not fully characterize training variability. The updated results provide complementary evidence on BFCL, where RETHINK achieves a mean@8 of 66.88, compared with 62.50 for GRPO and 60.25 for GiGPO, both trained with a group size of 32. These larger margins offer additional support for the effectiveness of RETHINK, while the smaller AppWorld margin should be interpreted cautiously. To broaden the empirical evaluation, we are also testing RETHINK on WebShop and will report these results in the supplementary material.

Question 1: Checkpoint selection criteria and bias

Thank you for pointing out this ambiguity. Our RL experimental setup was informed by AgentEvolver [1] and Elastic Horizon (EMNLP 2026) [2]. During training, we evaluate the model every 10 steps and record both mean@8 and pass@8. Checkpoint selection is performed independently for each benchmark and each training run: AppWorld checkpoints are selected by the highest mean@8 on AppWorld dev, whereas BFCL checkpoints are selected by the highest mean@8 on the held-out BFCL validation tasks (100–199). Scores are not combined across the two benchmarks. Table 2 reports both mean@8 and pass@8 from the same selected checkpoint; pass@8 is not used for checkpoint selection. The phrase "among its available evaluations" refers to the scheduled evaluations within the corresponding training run.

We acknowledge that this constitutes model selection on the reporting set and can introduce an optimistic bias. Given the limited number of tasks available in these agent benchmarks, we adopted validation-based reporting, which also appears in related agent RL work; for example, CoEvolve (ACL 2026) [3] reports BFCL results on its provided validation set. Accordingly, Table 2 should be interpreted as the validation performance of the selected checkpoints. We will make this reporting protocol and its potential selection bias explicit in the revised manuscript and clarify the checkpoint-selection criterion in Appendix A.

Question 2: GRPO/GiGPO comparison at matched sampling budget

Thank you for raising this important question. As shown in Table R.1, increasing the sampling group size of GRPO and GiGPO to 32 does not close the performance gap: RETHINK with Tree-Score continues to outperform both baselines on both benchmarks and both metrics. In particular, RETHINK achieves mean@8 scores of 73.25 on AppWorld and 66.88 on BFCL, compared with 71.71 and 60.25, respectively, for GiGPO with a group size of 32. Thus, RETHINK maintains its lead when GiGPO is given more root trajectories, and we observe only modest benefits from increasing root-level sampling in these experiments.

The Tree-Score ablation provides complementary evidence for the contribution of credit decomposition: removing Tree-Score reduces mean@8 by 5.49 percentage points on AppWorld and 3.00 percentage points on BFCL. Together, these comparisons support the effectiveness of the proposed credit assignment and suggest that the gains are not explained by increased sampling alone.

Question 3: AppWorld gain vs. seed variability

Thank you for raising this concern. We have updated Table R.1 to report variability across random seeds for all compared methods, making run-to-run variation explicit. We would also like to clarify that Figure 2 shows variation across checkpoints in the helper-only training experiment with a frozen main agent, whereas the main table concerns joint training. We are also evaluating RETHINK on WebShop and plan to include these results in the camera-ready version to further assess its effectiveness across agent benchmarks.

Thank you for your constructive feedback, and we would be happy to answer any follow-up questions.

References

[1] Zhai et al. AgentEvolver: Towards Efficient Self-Evolving Agent System. arXiv:2511.10395, 2025. https://arxiv.org/abs/2511.10395

[2] Zhang et al. Elastic Horizon: Discovering the Effective Interaction Frontier in Agentic Reinforcement Learning. EMNLP, 2026. https://aclanthology.org/2026.emnlp-main.1693/

[3] Yang et al. CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution. ACL, 2026. https://aclanthology.org/2026.acl-long.1055/

AuthorsOpenReview ↗

Web Shop Results

To further assess generality, we evaluated RETHINK on WebShop using the same training recipe as for AppWorld and BFCL, which is SFT warm-up followed by RL. Our setup follows verl-agent [1]. At each evaluation, we sample 256 tasks from the 500 held-out tasks. RETHINK again outperforms both RL baselines on mean@2 and pass@2. This shows that its gains carry over to a third agent benchmark.

MethodWebShop mean@2WebShop pass@2
SFT w/o RETHINK55.47%64.06%
SFT w/ RETHINK53.13%61.72%
GRPO80.86%85.16%
GiGPO82.03%85.93%
RETHINK (ours)83.20%87.50%

[1] Feng et al. Group-in-Group Policy Optimization for LLM Agent Training. arXiv:2505.10978, 2025. https://arxiv.org/abs/2505.10978

Thank you for the detailed rebuttal and the additional experiments. The GRPO and GiGPO baselines with group size 32, together with the RETHINK w/o Tree-Score ablation, address my main concern about separating the credit decomposition from the effect of larger sampling scale. My concerns have been addressed, and I will maintain my positive score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion of Submission 25771

Dear Area Chair,

Thank you for handling our submission. As the author–reviewer discussion closes, we summarize where it stands.

  1. Two reviewers recommend acceptance. Reviewers LWtV and vfCr both rate the paper positively. LWtV finds the work "insightful in its problem positioning". The review notes that our credit decomposition addresses a case that GiGPO-style state grouping cannot handle, and concludes that "per-channel credit assignment, rather than the interface itself, is the active ingredient of the gains". vfCr highlights a "meaningful method design" and a "solid experimental evaluation" across "multiple strong models and benchmarks", and finds the paper "clear and easy to follow".

  2. Their main concern has been resolved with new experiments. Both reviewers raised the same question: are the gains due to credit decomposition or to a larger sampling scale? We added two controlled comparisons (three seeds each) and a third environment; GRPO and GiGPO with group size 32. This is comparable to or larger than RETHINK's per-task sample count. RETHINK still leads on all metrics, for example BFCL mean@8 66.88 vs. 60.25 (GiGPO). RETHINK without Tree-Score. This is the same scaffold without the decomposition, and it loses 5.49 (AppWorld) and 3.00 (BFCL) mean@8 points. A third environment, WebShop. RETHINK ranks first with mean@2 83.20 vs. 82.03 (GiGPO). LWtV has confirmed that these experiments "address my main concern about separating the credit decomposition from the effect of larger sampling scale. My concerns have been addressed, and I will maintain my positive score." vfCr has not yet replied. However, vfCr's question (the same scaffold under a matched budget) is answered by the same experiments.

  3. On Reviewer Xz9B. We respectfully ask you to weigh this review in light of the ICLR 2027 policy on review quality. We list only verifiable facts:

  • No specific basis for the novelty judgment. The review lists no strengths ("Critical Strengths: None") and calls the novelty limited without identifying any prior work that does what we do. The only paper it cites is an arXiv-only preprint, and the Reviewer Guide states that the lack of such comparisons "cannot be a basis for rejection".
  • A factual error about model scale. The review states that results cover "only small-sized models". Table 1 evaluates DeepSeek-V4.1-Flash, Qwen3.6-Flash and Kimi K2.5.
  • Notation questions about symbols the paper already defines. x is defined at line 188 ("Let x be a task") and x_t at line 197 ("x_t is the user request currently being handled"). The symbols u, v and U_z in Eq. (8) are defined at lines 320 and 324.
  • No engagement with the new evidence. The follow-up keeps the score, calling the idea "straightforward" with "limited analysis". It does not engage with the new experiments above or with the existing analyses in Appendices B–C and Section 4.3.

Both reviewers who discussed the technical content in detail recommend acceptance, and LWtV has confirmed that their concerns are resolved. We would be grateful if you could take this into account, and consider vfCr's assessment during the reviewer–AC discussion.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.