acceptodds
Under review as a conference paper at ICLR 2027

From Trajectory Evidence to Control: Latent Control-State Inference for Long-Horizon Language Agents

Abstract

Language agents often fail in long-horizon tasks despite making locally reasonable decisions. A recurring difficulty is that accumulated trajectory evidence may call for a change in how the agent should proceed, while next-action generation continues along the current course. We formulate this missing intermediate decision as latent control-state inference and introduce an Explore–Exploit–Recover () controller. At each step, infers a trajectory-conditioned control state consisting of a control mode and a trajectory-specific target, and uses it to regulate the proposed action: Explore resolves missing task-relevant evidence, Exploit advances a supported course of action, and Recover breaks from one rendered ineffective by feedback. We evaluate on ALFWorld, WebArena-Lite, and ToolSandbox. Across environments, consistently improves end-to-end task performance and performance accumulated over the interaction budget. Behavioral analysis shows that acts as a sparse, mode-conditioned control layer, preserving most proposals while concentrating explicit corrections under particular inferred control states. Controlled interventions further indicate that both control-state content and its trajectory-aligned enactment contribute to the observed gains. These results suggest that effective long-horizon agency benefits from an explicit control layer between trajectory understanding and next-action generation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
language Agents, Long-Horizon Decision Making, Adaptive Control, Trajectory Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CzqFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper summary: This paper proposes E2RE^2R, a framework that injects a Controlling state into the rollout of an RL agent. More specifically, E2RE^2R injects controlling states after the agent proposes the next possible action and determines whether that action is appropriate for the current trajectory. The experiments evaluate the performance of E2RE^2R with other settings, including ReAct and PlanAndExecute, which illustrate the importance of the proposed controlling states and action assessment in RL agent rollout.

Contributions: The paper's main contribution is the introduction of controlling states before deciding to follow the action for the inference of an RL agent. The controlling states include: EXPLORE, EXPLOIT, and RECOVER. The experiments illustrate that by equipping the agent with the Controlling states, the RL agent can achieve more terminal efficiency compared with other baselines.

Critical strengths

There are several strengths of the paper, listed below:

  1. Important problem statement: The problem is important, and the paper points out the framework like ReAct can repeat failed actions; therefore, a verification framework is needed to inject and avoid repetition.
  2. Simple plug-and-play method: The proposed method is simple and does not require any training or fine-tuning. Prompting is enough to inject the controlling state into the agent.
  3. Experimental results: Compared with baselines, e.g., ReAct and Vanilla, E2RE^2R achieves better success and termination score.
Critical weaknesses

Several weaknesses should be included:

  1. Novelty and clarity: It seems that the improvement of E2RE^2R comes from the feedback it receives from the environment and from previous episodes. However, I think this proposal is not new. Previous methods such as AdaPlanner [1] and SwiftSage [2] already do quite similar work. This paper should include a highlight to illustrate the difference between E2RE^2R and other previous methods (comparison in the experiments)
  2. Need to include more baselines in the experiments: The baselines reported in Table 1, Table 2 should include AdaPlanner [1], Self Refine [3], and Reflecxion [4] at each step. These baselines already use trajectory feedback to revise the behavior of the agent, therefore, a quantitative comparison with them is necessary.
  3. Clarity in notation: What can be the value of budget BB for PB-AUC in Equation 13? And in Appendix B, do the Vanilla and Plan&Execute baselines receive the demonstration similar to ReAct?
  4. Different backbone consideration: The method only runs on a single backbone (DeepSeek). The prompting can be different with different backbones; therefore, can the authors run it on different LLM families, including Qwen3-8B or LLaMA-8B?
  5. Motivating failures are not supported in the main experiment: In Figure 1, the paper reports 36.8% of ReAct failures due to repetition failures. In Table 4, what are the percentages of Matching Reasoning and E2RE^2R for this statistic?
  6. What are the average termination steps of ReAct compared with E2RE^2R? Also, can the paper report the token usage of ReAct compared with E2RE^2R?

References: [1] Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. In Advances in Neural Information Processing Systems, volume 36, 2023.

[2] Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. In Advances in Neural Information Processing Systems, volume 36, 2023.

[3] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, volume 36, 2023.

[4] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, 2023.

Questions for authors

See the weaknesses section

Reviewer_Wpy5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Explore–Exploit–Recover (E2R), a framework for adapting long-horizon language-agent behavior through explicit control-state inference. Based on accumulated trajectory evidence, it identifies a control mode and a trajectory-specific target, preserving compatible action proposals and redirecting incompatible ones. Its main contribution is an explicit intermediate representation between trajectory understanding and action selection. Experiments on ALFWorld, WebArena-Lite, and ToolSandbox report improvements in task performance and interaction efficiency.

Critical strengths
  • Clear and interpretable formulation. The Explore–Exploit–Recover modes provide an intuitive way to describe how an agent should respond to trajectory evidence. This offers a useful perspective on agent adaptation.
  • Some effort to isolate the source of improvements. The controlled comparisons with additional reasoning, mode feedback, and perturbed control strengthen the empirical analysis, although they do not fully resolve the concerns about generality.
Critical weaknesses

Major concerns:

  1. Weak motivation. It is difficult to believe that this issue is widespread in current LLMs, particularly for DeepSeek-Flash, a model with hundreds of billions of parameters. The subsequent results further suggest that the issue occurs infrequently. Moreover, a simple intervention may suffice: explicitly instructing the model in the prompt not to repeatedly attempt the same failed action.

  2. Misleading presentation of the method and limited novelty. The appendix describes two variants, Decoupled E2R and Fused E2R, and reports comparable performance on ALFWorld. Since Fused E2R requires only one LLM call, the authors use it for all main benchmark experiments. However, this implementation choice is not mentioned in the main text.

    If the experiments indeed use Fused E2R, as stated in the appendix, the novelty is quite limited. It appears to be a prompting scheme with no implementation-level difference from ReAct beyond the required output structure. Consequently, the proposed components—including the Controller, compatibility function, and Mode-conditioned Operator—do not appear to be necessary as separate components, or may not actually exist as such in the implementation.

    Decoupled E2R more closely matches the description in the main text, but it also raises questions. The authors claim that it requires only two LLM calls, one for π_ReAct and one for F_ctrl. How, then, is the compatibility function implemented? Furthermore, this implementation would appear to require at least twice the token consumption, since the controller must also process the full trajectory. Are these additional costs included in the reported budget?

  3. Insufficient experimental support for the conclusions. Apart from ALFWorld, the improvements on the other two benchmarks are modest. Furthermore, the evaluation uses only DeepSeek Flash v4.1, leaving it unclear whether the findings generalize to other models, such as GPT, Claude, GLM, or Kimi.

Other issues:

  • The citation format is unusual and difficult to distinguish from ordinary numbering.
  • Lines 44–45 and 47–48 repeat the same sentence: “In these formulations, however, the manner of continuation is generally embedded within the tasklevel decision process.”
  • Line 174 contains a duplicated word: “selectively selectively.”
Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Wpy5[Part 1]

W1: Motivation and Significance of Trajectory-conditioned Control

We thank the reviewer for raising this important question. We would like to further clarify the motivation behind E2RE^2R and provide additional experimental evidence addressing the suggested simple intervention.

Motivation. Long-horizon interaction requires agents to continuously determine how accumulated trajectory evidence should guide subsequent behavior. This involves three distinct control requirements: (i) acquiring missing evidence needed for further progress, (ii) advancing an evidence-supported continuation, and (iii) recovering from a continuation rendered ineffective by observed feedback. These requirements motivate our trajectory-conditioned control state zt=(mt,ℓt)z_t=(m_t,\ell_t), where the mode specifies how to proceed and the target grounds this decision in the current trajectory.

Empirical evidence. Figure 1 illustrates a concrete manifestation of inappropriate continuation: an agent repeatedly attempts an ineffective action despite explicit negative feedback. Among 38 ReAct episodes containing the specified negative-feedback event, 14 (36.8%) exhibit exact action persistence. This example highlights one observable control failure, while the broader formulation also addresses information acquisition and evidence-supported execution.

Sparse intervention. The low redirect rates in Table 3 reflect E2RE^2R's selective regulation mechanism. The controller infers a control state at each step and modifies a proposal only when it conflicts with the inferred control objective. Accordingly, the reported redirect rates (0.4–6.8% of steps) and intervention coverage (5.2–23.6% of episodes) characterize the frequency of explicit action modification, rather than the frequency of control-state inference or all possible control failures.

Comparison with the suggested simple intervention. To directly evaluate the reviewer's suggestion, we additionally tested ReAct with an explicit instruction to avoid repeating failed actions:

"Do not repeatedly attempt the same failed action. If an action fails (for example, the environment responds 'Nothing happens.'), choose a different action instead of repeating it, unless the relevant state has changed so that the action can now succeed."

The results are summarized below:

MethodCalls/StepSR (%) ↑PB-AUC ↑
ReAct170.900.5369
ReAct + No-Repeat175.370.5581
E²R-Fused188.810.6538
E²R-Decoupled288.810.6544

The No-Repeat instruction improves ReAct by 4.47 percentage points in SR and 0.0212 in PB-AUC, confirming that local failure avoidance is beneficial. E²R achieves a further 13.44 percentage-point improvement in SR and 0.0963 in PB-AUC over this baseline. These results indicate that the benefits of E²R extend beyond those obtained from the suggested failure-avoidance instruction.

AuthorsOpenReview ↗

Response to Reviewer Wpy5[Part 2]

W2: Method Formulation, Implementation, and Novelty

We appreciate the reviewer's careful examination of the implementation details. We would like to clarify the relationship between the two realizations of E²R, their experimental purposes, and the evidence supporting the proposed control mechanism.

Consistency between the two realizations. We first highlight the comparison in Appendix Table 6, conducted on the full 134-task ALFWorld evaluation:

MethodLLM Calls/StepSR (%)PB-AUC
ReAct170.900.5369
E²R-Decoupled288.810.6544
E²R-Fused188.810.6538

Both realizations achieve identical success rates of 88.81% and nearly identical PB-AUC values (0.6544 vs. 0.6538), demonstrating closely matched performance under the same backbone, task set, and environment interface.

Design rationale. The two realizations serve complementary experimental purposes.

  • Decoupled E²R supports controlled mechanism analysis. It obtains action proposals from the unchanged standalone ReAct policy before applying trajectory-conditioned regulation. This separation allows us to investigate the effect of control on independently generated proposals and compare alternative control mechanisms under a common proposal-generation procedure.
  • Fused E²R supports inference-efficient end-to-end evaluation. It serializes proposal generation, control-state inference, and action regulation within a single structured completion, requiring only one LLM invocation per environment step.

The closely matched performance also shows that the control procedure remains effective when the initial proposal is generated independently of E²R instructions. Accordingly, we use Fused E²R for the main benchmark experiments (Table 1) and Decoupled E²R for controlled mechanism studies (Section 4.4).

Implementation details. E²R follows three logically ordered operations: control-state inference, proposal compatibility assessment, and mode-conditioned regulation.

In Decoupled E²R, the first LLM invocation generates the standalone ReAct proposal. The second invocation jointly implements the three control operations: it infers the mode and trajectory-specific target, evaluates whether the fixed proposal satisfies the inferred control objective, and preserves or redirects the proposal accordingly. The compatibility function (K) and mode-conditioned operators (O_m) are implemented through these conditional instructions within the controller invocation, requiring no additional LLM calls.

Fused E²R further incorporates proposal generation into the same structured completion. We will make the correspondence between the logical operations and their concrete implementations explicit in the main text.

Evidence for structured control. The decoupled realization enables us to investigate the contributions of trajectory reasoning, structured control information, and its explicit enactment. In Section 4.4, Matched Reasoning achieves 82.8% SR, Mode Feedback achieves 83.6%, and E²R further improves performance to 88.8%. The corresponding PB-AUC values are 0.596, 0.630, and 0.654.

These comparisons provide evidence that trajectory-level reconsideration contributes substantially to performance, while structured control information and mode-conditioned enactment provide additional benefits. They support the central design of E²R: inferring how an agent should proceed from accumulated trajectory evidence and using this control decision to regulate task-level actions.

AuthorsOpenReview ↗

Response to Reviewer Wpy5[Part 3]

W3: Evidence for Structured Control Beyond Additional Reasoning

We thank the reviewer for raising the important question of whether the proposed control structure provides benefits beyond simpler forms of trajectory reconsideration. We designed the controlled experiments in Section 4.4 and additional ablations to investigate three aspects: the value of structured control, the contribution of its representation, and the effect of explicit control enactment.

All experiments are conducted on the complete 134-task ALFWorld evaluation using the decoupled implementation, with action proposals generated by the same standalone ReAct policy. This setting allows us to compare alternative control procedures while keeping the underlying proposal policy fixed.

The results are summarized below:

MethodSR (%) ↑PB-AUC ↑
ReAct70.90.5369
Matched Reasoning82.80.5960
Mode Feedback83.60.6300
Perturbed Control83.60.6270
Target-only Control83.60.6185
Continue/Recover Control85.80.6466
E²R88.80.6544

1. Structured control beyond generic reconsideration. Matched Reasoning improves SR from 70.9% to 82.8%, confirming that additional trajectory-level assessment contributes substantially to performance. E²R further improves SR to 88.8% and PB-AUC from 0.5960 to 0.6544, providing evidence for the additional value of structured control.

2. Contribution of the control-state representation. We additionally evaluate two simplified controllers to examine the role of explicit control modes.

  • Target-only Control retains the trajectory-specific control target while removing the explicit mode representation. E²R improves upon this variant by 5.2 percentage points in SR and 0.0359 in PB-AUC.
  • Continue/Recover Control merges EXPLORE and EXPLOIT into a single CONTINUE mode while retaining RECOVER and the trajectory-specific target. E²R achieves a further improvement of 3.0 percentage points in SR and 0.0078 in PB-AUC.

These results suggest that trajectory-specific targets provide useful control information, while explicitly distinguishing the control requirements associated with information acquisition, supported execution, and recovery offers additional benefits.

3. Explicit enactment and trajectory alignment. We further examine how the inferred control state influences action regulation. Mode Feedback provides the inferred mode and target as textual guidance, achieving 83.6% SR and 0.6300 PB-AUC. E²R explicitly enacts the inferred state through selective regulation, improving performance to 88.8% SR and 0.6544 PB-AUC. Perturbed Control, which alters the enacted control state, obtains 83.6% SR and 0.6270 PB-AUC, providing further evidence that alignment between the inferred control state and trajectory evidence contributes to effective regulation.

Together, these experiments provide complementary evidence for the proposed control formulation. Generic trajectory reconsideration accounts for a substantial portion of the improvement, while trajectory-specific control targets, explicit mode distinctions, and mode-conditioned enactment contribute additional benefits. These findings support trajectory-conditioned control as a useful intermediate decision process for long-horizon language agents.

Reviewer_K84gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces E²R (Explore–Exploit–Recover) to help language agents avoid repeated failures and adapt during multi-step tasks. Using interaction history and feedback, it identifies a control mode and a specific objective, then preserves or revises the proposed action accordingly. E²R improves performance across three benchmarks, raising ALFWorld success from 70.90% to 88.81% over ReAct.

Critical strengths
  1. Interpretability: Each control mode is paired with a concrete objective, making behavioral adjustments explicit.
  2. Evaluation: Experiments assess task performance, interaction efficiency, and the contribution of structured control through controlled comparisons.
  3. Practicality: E²R requires no model retraining, supports a single-call implementation, and complements Reflexion.
Critical weaknesses
  1. Necessity of the three modes. The ablations do not isolate the contribution of mode labels from that of concrete control targets. Could the authors compare E²R with a target-only controller or a simpler continue/recover controller?

  2. Interpretation of sparse intervention. In the fused implementation, E²R instructions may already influence the initial proposal. Could the authors report intervention statistics for the decoupled implementation to better support the claim of selective correction?

  3. Reliability of the improvements. Some gains, particularly on WebArena-Lite, involve relatively few additional successful tasks. Could the authors provide repeated runs or paired confidence intervals to establish their robustness?

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer K84g[Part 1]

We sincerely thank the reviewer for the careful and constructive evaluation of our work. We found the comments very helpful in sharing both the motivation and the technical presentation of the paper. Below we address each point in turn and have incorporated the clarifications into the revised manuscript.

W1: Necessity of the three control modes.

Reviewer concern: The ablations do not isolate the contribution of mode labels from that of concrete control targets. Could the authors compare E²R with a target-only controller or a simpler continue/recover controller?

R1: Thanks for raising this interesting question. We would also like to clarify the rationale behind our formulation again, which we believe deserves a more explicit explanation.

Specifically, our goal is to represent the agent's control relation to accumulated trajectory evidence. This design can be motivated by two sequential questions (i) Has observed feedback invalidated the current continuation? If so, recovery is required. Otherwise, (ii) does further progress depend on acquiring missing evidence, or can the agent advance an already supported commitment? These questions motivate the current three-mode design. Our hypothesis is that explicitly representing the control type and coupling it with mode-conditioned regulation provides a useful structural inductive bias for translating trajectory evidence into action.

We agree that this motivation should be supported by more direct experimental evidence. Accordingly, we evaluate two additional controlled comparisons:

  1. Target-only control: Infer a trajectory-specific target directly from the interaction history without assigning an Explore/Exploit/Recover mode, and use a generic compatibility and regulation procedure.
  2. Continue/Recover control: Merge Explore and Exploit into a single Continue mode while retaining Recover and trajectory-specific control targets.

Both comparisons use the same independently generated ReAct proposals and the same ALFWorld evaluation tasks as the decoupled E²R study, with matched model and evaluation configurations.

MethodSR(%) ↑\uparrowPB-AUC ↑\uparrow
Target-only control83.60.6185
Continue/Recover control85.80.6466
E2RE^2R88.80.6544

These results show that the full E2RE^2R controller outperforms both simplified alternatives. Compared with target-only control, E2RE^2R improves SR by 5.25.2 percentage points and PB-AUC by 0.03590.0359, suggesting that the full structured mode-and-target control mechanism provides benefits over a target-only controller with generic regulation. Compared with Continue/Recover, E2RE^2R further improves SR by 3.03.0 percentage points and PB-AUC by 0.00780.0078, suggesting a potential benefit from distinguishing evidence acquisition from evidence-supported execution.

We have also revised the manuscript accordingly to make both the design rationale and the scope of our claims clearer.

AuthorsOpenReview ↗

Response to Reviewer K84g[Part 2]

W2: Interpretation of sparse intervention.

Reviewer concern: In the fused implementation, E²R instructions may already influence the initial proposal. Could the authors report intervention statistics for the decoupled implementation to better support the claim of selective correction?

R2: We thank the reviewer for this thoughtful observation. We would like to clarify our motivation for using the fused implementation in the main experiments and provide additional evidence from the decoupled implementation.

Why the fused implementation? Our main experiments adopt the fused implementation primarily to maintain the same number of LM calls per environment step as the single-call baselines, ensuring a more comparable evaluation setting. Importantly, as already reported in Appendix A.4 (Table 6), the fused and decoupled implementations achieve nearly identical performance on ALFWorld:

MethodLM calls / stepSR (%) ↑\uparrowPB-AUC ↑\uparrow
ReAct170.900.5369
E2RE^2R-Decoupled288.810.6544
E2RE^2R-Fused188.810.6538

These results show that E²R's performance gains persist when the initial proposals are generated independently by the standalone ReAct policy, without exposure to E²R instructions.

Additional intervention analysis. We agree with the reviewer that, in the fused implementation, the E2RE^2R instructions may influence proposal generation. Consequently, the decoupled implementation provides a cleaner setting for examining selective regulation of independently generated proposals.

To address this concern, we additionally report intervention statistics for both implementations, using the same definitions as in Section 4.3.

MethodRedirect Rate (%)Intervention Coverage (%)Redirects/Episode
E2RE^2R-Decoupled1.8314.920.32
E2RE^2R-Fused1.0015.700.31

The decoupled implementation exhibits a low redirect rate of 1.83%1.83\%, with an average of 0.320.32 redirects per episode. These results indicate that sparse explicit action modification is also observed when proposals are generated independently of E²R instructions. While the fused and decoupled implementations differ in their proposal-generation processes, the additional results support the selectivity of the decoupled regulation mechanism.

We have clarified this distinction in the revised manuscript: the fused implementation is primarily used for inference-efficient end-to-end evaluation, while the decoupled implementation provides a controlled setting for analyzing the proposed regulation mechanism.

We thank the reviewer for this suggestion, which helps us distinguish the intervention behavior of the two implementations more precisely.

AuthorsOpenReview ↗

Response to Reviewer K84g[Part 3]

W3: Reliability of performance improvements.

Reviewer concern: Some gains, particularly on WebArena-Lite, involve relatively few additional successful tasks. Could the authors provide repeated runs or paired confidence intervals to establish their robustness?

R3: Thank you for highlighting this. To quantify the statistical reliability, we calculate task-level paired confidence intervals for the performance differences. Since both methods are evaluated on the same benchmark tasks, we use paired bootstrap resampling over task IDs, preserving the correspondence between methods. We report 95% confidence intervals for both terminal task performance and PB-AUC. Improvements are computed against the strongest baseline for each reported metric (ReAct for ALFWorld and WebArena-Lite, and Vanilla for ToolSandbox).

EnvironmentMetricImprovement95% Paired CI
ALFWorldSR+17.91 pp[+10.45, +25.37]
PB-AUC+0.12[+0.07, +0.17]
WebArena-LiteSR+4.24 pp[+1.21, +7.27]
PB-AUC+0.03[+0.01, +0.04]
ToolSandboxScore+0.0202[+0.01, +0.03]
PB-AUC+0.0346[+0.02, +0.05]

As shown in the results, all six paired confidence intervals have positive lower bounds, providing statistical support for the observed improvements across the three environments under the paired evaluation.

In particular, we acknowledge that the WebArena-Lite improvement corresponds to a net gain of only seven successful tasks (47/16547/165 versus 40/16540/165). Nevertheless, its paired confidence interval remains positive, with a success-rate improvement of +4.24 percentage points (95% CI: [+1.21,+7.27][+1.21, +7.27]). The corresponding PB-AUC improvement is also positive (+0.0267+0.0267, 95% CI: [+0.01,+0.04][+0.01, +0.04]). The larger gains on ALFWorld and the improvements on ToolSandbox are likewise supported by positive paired confidence intervals.

We have incorporated these uncertainty estimates into the revised manuscript and clarified their statistical interpretation. We appreciate the reviewer's suggestion, which helps strengthen the empirical evidence and present the reliability of the results more rigorously.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.