acceptodds
Under review as a conference paper at ICLR 2027

Fragile Follow-Through: Uncovering How User Revisions Disrupt Long-Horizon Task Execution in LLM Agents

Abstract

LLM agents are increasingly entrusted with complex, long-horizon tasks, where fulfilling a user's request requires coordinating decisions and actions across many interdependent steps. Existing work typically evaluates agents against requirements specified at the outset, although recent studies have begun to consider user interruptions and revisions during execution. However, current evaluations do not adequately capture revisions prompted by task progress or explain how they disrupt long-horizon task completion. To bridge this gap, we adapt long-horizon tasks by altering their initial instructions and introducing mid-execution revisions that restore the original requirements. This yields ReQuest, an application-grounded benchmark designed to evaluate task completion and analyze agent behavior under dynamic user revisions. Revisions are triggered following observable progress under the altered requirements, with the agent's history and environment state preserved. Extensive experiments show that user revisions reduce average task success rates by approximately 17–63% relative to the original tasks across models. Process analyses help explain these losses: agents retain incompatible prior decisions, lose valid progress, or leave unchanged requirements unmet. Answers and guidance targeting these problems yield uneven benefits, pointing to mixed difficulties behind fragile follow-through in long-horizon tasks. Our benchmark, code, and evaluation protocols will be publicly available to support reproducible evaluation.

Primary area:
datasets and benchmarks
Keywords:
long-horizon LLM agents, user-initiated task revisions, agent evaluation and behavioral analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_43uiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper focus on the evaluation of multiple user questions within one session. Besides, the authors identify that current evaluation process only introduce additional question based on a fixed process, which may be inconsistent with real world scenarios. Based on this, the authors propose a brand new dataset and analysis current models. Besides, they also give several behavior analysis on the failure pattern.

Critical strengths
  1. The authors identify that current multi-question evaluation benchmarks ignore that users usually give new prompts when some milestone s appear during the task.
  2. The authors conduct broad experiments and analysis of their dataset.
Critical weaknesses

The evaluation of multi-question within a single session is valuable. But I think the additional value of this benchmark against existing multi-question benchmark is unclear, which significantly limits the novelty. The authors argue existing benchmarks is inconsistent with real world, but actually, users usually not react directly when some actions are token, cause they cannot monitoring the agent all the time.

Questions for authors

Could the authors clarify the position of this paper? At least I don't think this paper bring something useful to the research community. It looks like a simple and toy experiment, which is only a slight change of current works.

Reviewer_FjaTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ReQuest, a benchmark of 359 paired tasks adapted from BrowseComp, WebArena, and DeepPlanning. Initial requirements are changed and then restored through 962 progress-triggered revision events, while conversation history and environment state are preserved. The original and revised versions share final requirements and outcome evaluators. Ten backbones exhibit approximately 17-63% relative reductions in the reported cross-domain success rate. Independent subproblem runs, human-reviewed trajectory attribution, requirement-level checks, and answer/restate/continuation interventions provide evidence that failures involve both solving revised problems and integrating them with still-valid work.

Critical strengths
  1. ReQuest covers 359 paired tasks across search, web interaction, and planning, restoring identical final requirements and evaluators after revisions. This makes a practical agent failure mode measurable across diverse settings.
  2. Independent subproblem runs, 400 human-reviewed trajectories, and cart/itinerary checks provide concrete evidence of integration failures. They show how correct local updates can coexist with lost valid work or an incomplete objective.
Critical weaknesses
  1. Figure 8's task-review checklist includes detection rules, but rollout-level precision, recall, and timing remain unaudited. Coverage-qualified scores therefore partly depend on delivery reliability: deep-search endpoint success is 40.2%, versus 32.0% with the coverage requirement (Table 6).
  2. The support controls retain requirement changes, leaving conflicting revisions confounded with additional provisional work and retained history. Without a matched no-op or history/state control, the diagnostics establish integration failures more clearly than their causal share of the performance loss.
Questions for authors
  1. Can you report a stratified human audit of the revision detector, including false positives, missed eligible triggers, timing errors, and model/domain breakdowns? How do the main success-rate comparisons change when delivery is corrected by this audit and what are the per-model endpoint-only drops, complementing the model-averaged results already in Table 6?
  2. Can you provide a matched control that preserves interruption timing and extra history/work while avoiding a requirement conflict, or otherwise separate these sources of difficulty? Please also give the sample sizes for Figure 5a and explain whether repeated original/revised runs change the separate-success finding. These answers would determine how strongly the integration explanation can be supported beyond the observed failure witnesses.
Reviewer_ecH5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how LLM agents perform when users revise the request in the course of “long-horizon” tasks. The authors build a benchmark called REQUEST generated based on existing datasets and evaluate how task completion ratio drops with revision prompts and study agent behavior under such cases. On experiments from 10 different models, the authors find that revision prompts result in 17–63% drop in task completions.

Critical strengths

Overall the paper is well written and the authors conduct a wide range of experiments on different models. The results and findings mostly align with related work.

Critical weaknesses

While it’s an important and exciting problem to measure LLM’s ability to collaborate with users and handle user revisions, this problem has been extensively studied recently. To name a few,

  • Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration by Shao et al.
  • Completion ≠ Collaboration: Scaling Collaborative Effort with Agents by Shen et al.
  • PPP-Agent: Training Proactive and Personalized LLM Agents, by Sun et al.
  • CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks by Wang et al.
  • Tom-swe: User mental modeling for software engineering agents by Zhou et al.
  • Align AI to Dynamic Human-AI Workflows by Chen et al. In my view, what’s proposed in this paper represents a specific setup that is already covered in the prior work. The conclusion in this paper seems to be aligned with the previous work. And I do not think this paper reconciles with this previous literature properly. I think this is an important limitation that the authors should address in the rebuttal.

I think there might be some important limitations in the experiment design. As indicated in figure 2, it seems the authors have a prescribed model of how the user can intervene in the execution, which I do not think is a reasonable model. As shown in the previous work, there are rich patterns where a user can follow up a message and issues requests and followups. A common pattern is to invoke a user model (e.g., ODYSSIM, Building Foundation Models for Human Behavior Simulation) rather than using a “fixed” function for the follow ups.

Questions for authors
  • How is "long-horizon" defined in this paper? Is it based on number of steps in tasks? Or is it based on the actual wall clock hours?

Comments on the paper

2 comments
AuthorsOpenReview ↗

Request to Facilitate Reviewer Discussion

Dear Program Chairs, Senior Area Chairs, and Area Chairs,

Thank you for your time and efforts in coordinating the review process. We are writing to kindly ask for your help in facilitating discussion of our paper.

We submitted our rebuttal several days ago, with detailed responses addressing the reviewers' concerns and important misunderstandings about our work, but have not yet received any replies. To address the reviewers' concerns, we have also revised the manuscript with additional experiments and clarifications of the task formulation, related work, and evaluation protocol.

We would appreciate your help in encouraging the reviewers to engage with our responses and clarify any remaining concerns. We hope to resolve these misunderstandings through discussion before the discussion period ends. Thank you for your time and assistance.

Best regards, Submission252 Authors

AuthorsOpenReview ↗

Follow-up on Reviewer Discussion

Dear Program Chairs, Senior Area Chairs, and Area Chairs,

Following our earlier message, we have not yet received any reviewer responses. Some reviews reflect important misunderstandings that we have addressed in detail in our rebuttal. As the discussion period draws to a close, we would greatly appreciate your help in encouraging the reviewers to engage with our clarifications and reconsider their assessments accordingly.

Best regards, Submission252 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.