acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Task Drift in Long-Horizon GUI Agents

Abstract

While coding agents have advanced rapidly, progress in GUI agent harnesses has lagged, especially for long-horizon tasks. In such tasks, a GUI agent can carry an incorrect or outdated completion judgment into later decisions and overlook unmet user requirements. We define task drift as this persistent mismatch between the agent’s believed progress and the actual satisfaction of the original task requirements. The root cause is that progress judgments detach from the original requirements and lack requirement-specific environmental evidence. To address this, we introduce a framework that separates persistent user requirements from revisable plans and progress judgments, and grounds each judgment in concrete environmental evidence. Its controller distinguishes missing evidence from evidence of an unmet requirement: the former triggers inspection, the latter triggers correction, and invalidated support triggers revalidation only for affected requirements. On OSWorld, our framework improves the mean task score from 73.3 to 82.1 with Qwen3.7-Plus as the backbone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI Agent GUI harness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wbpjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses task drift in long-horizon GUI tasks, where an agent’s assessment of task completion may diverge from the user’s requirements, or previously satisfied requirements may be invalidated by subsequent actions. The authors propose a verification framework centered on a GUI actor. The framework links the original requirements and their supporting evidence to the environmental states on which they depend, re-evaluates the requirements when relevant states change, and provides targeted feedback based on the findings. Evaluations on OSWorld, including ablation studies using the same backbone model, show that the full method improves the task success rate from 73.3% to 82.1%.

Critical strengths
  1. The research problem is practically significant. The paper addresses both incorrect assessments of task completion and cases in which a requirement is initially satisfied but subsequently invalidated. The latter is directly relevant to the need for continuous state maintenance in long-horizon GUI tasks.

  2. The methodological workflow is clear and provides differentiated feedback. The framework distinguishes between cases in which existing evidence indicates that a requirement is unmet and cases in which the evidence is insufficient to support a judgment, providing specific repair or inspection feedback accordingly. Comparisons of feedback types suggest that explicitly identifying discrepancies and the scope of necessary repairs helps correct errors while preserving progress already achieved.

Critical weaknesses
  1. The substantive innovation relative to existing verification frameworks remains somewhat unclear. VLAA-GUI [1], for example, already incorporates success-condition checks, independent completion verification, and feedback following failed verification. Adding checks and corrective feedback is therefore insufficient, by itself, to establish a distinct contribution. The paper’s most notable design element—the continuous maintenance of dependencies among requirements, evidence, and versioned environmental states—is not yet fully justified. It remains unclear which specific challenges unaddressed by existing frameworks this mechanism resolves, or why standard re-verification processes in existing frameworks could not achieve similar outcomes. The current evidence makes it difficult to determine whether this is a novel methodological contribution with independent value or a refinement of existing verification workflows.

  2. The effectiveness of dependency maintenance, arguably the paper’s most innovative component, lacks independent validation. Table 2 introduces the entire verification framework as a single module and then adds a requirement prior. Although these results support the framework’s overall effectiveness, they do not distinguish whether the gains arise from additional state reads, more frequent verification opportunities, more granular feedback, or evidence-dependency maintenance itself. The system-level comparison with VLAA-GUI in the main table cannot resolve this question either, because the methods use different backbone models. I suggest a controlled experiment that holds the actor, available tools, and verification budget constant while comparing three approaches: (1) no dependency tracking, with re-verification only upon completion; (2) full re-verification after every state change; and (3) the proposed dependency-triggered partial re-verification. The first comparison would help assess the necessity of continuous maintenance, while the second would reveal whether dependency information itself offers distinct benefits in accuracy or cost-efficiency.

  3. The AI Use Statement has not been substantively completed. Although the section appears on page 10, it contains template placeholders, including "[tasks with required disclosure]" and "[Elaborate…]", rather than an actual declaration. The ICLR 2027 Author Guidelines [2] require an AI Use Statement; the authors should provide one.

References

[1] Qijun Han et al. VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation. arXiv:2604.21375, 2026.

[2] ICLR. ICLR 2027 Author Guidelines, “Required: AI use statement.” Accessed October 17, 2026.

Questions for authors
  1. What specific capabilities does dependency maintenance add beyond the success-condition checks and completion verification already present in VLAA-GUI? Could the authors provide a concrete execution trace illustrating task drift that existing checks might miss but that the proposed mechanism can successfully handle?

  2. With the backbone, tool permissions, and verification budget held constant, how does removing evidence-dependency maintenance affect the success rate, the incidence of erroneous completion judgments, and verification costs? What specific advantages does partial re-verification offer over full re-verification?

  3. How many failures in the empirical evaluation involved previously satisfied requirements being invalidated by subsequent actions? Of these, how many were detected through dependency invalidation and ultimately corrected? This analysis would help determine whether the mechanism effectively addresses the core problem the paper claims to solve.

1 comment

Thank you for recognizing the importance of preserving requirements that were satisfied earlier in a task. We agree that Table 2 evaluates several components together and that the comparison with VLAA-GUI across different backbones cannot isolate the contribution of dependency maintenance. We therefore examined the question at three levels: what triggers a new check, whether its scope affects task outcomes under a common budget, and how often we can trace a natural failure through detection and repair.

Q1. What does dependency maintenance add beyond completion verification?

VLAA-GUI already checks a proposed finish against observable success conditions and returns feedback when the evidence is inadequate. A full check at completion can also discover a regression if it inspects the relevant property. Our contribution is to record which observation supported each requirement, on which object and version. When that object changes, the controller can reopen the affected requirement before the next completion claim. A changed version calls for a fresh inspection. A repair is requested only when that inspection finds a discrepancy. VLAA-GUI, §3.2

A controlled three-file execution illustrates the distinction. A read established that report.txt contained the required value ALPHA. An injected change then replaced it with BROKEN. The scoped controller reopened the requirement linked to report.txt, read the changed value, and the actor restored ALPHA. Inspection of the three final files confirmed that all three requirements held. Full re-verification also succeeded on this case. The capability demonstrated here is identifying which earlier judgment has lost its evidential basis and directing the next read to that file.

Q2. What changes when the actor, tools, and verification budget are held constant?

Our new online experiment used the same Qwen3.8-Flash actor, tool permissions, global execution limits, and native completion check across eight configurations. We report the observed portion of the planned 40-task, two-repeat study. The table averages available official scores within each task and then weights tasks equally. These scores include partial credit.

ConfigurationOfficial score (%)ConfigurationOfficial score (%)
A · Baseline74.73E · Independent summary74.51
B · Chronological facts79.39F · Source locator73.14
C · Object-organized facts74.37G · Full checks after changes76.67
D · Exploration prompts77.89H · Dependency-scoped checks84.79

G versus H is the most direct comparison. Both extend F, use the same rule for eligible state changes, and allow at most three additional checks per episode. G checks every requirement at each such opportunity. H checks the requirements linked to the change and falls back to a full check when the link is unclear. On the 39 tasks with matched task-and-repeat scores, H scored 84.40% versus 76.67% for G, a paired difference of +7.73 percentage points.

Q3. How many previously satisfied requirements were invalidated, detected, and repaired?

We also ran the reviewer’s three requested schedules directly in an earlier, separate implementation, with the same actor, checker, tools, and global limits across 24 tasks and two repeats. This comparison matters because A in the new experiment retains the native recovery system.

Verification scheduleFull-credit episodes / scoredAccepted completions below full credit / acceptedMean reads per episode
Full check only at completion29/482/1512.02
Full check after relevant changes31/473/1756.40
Dependency-scoped recheck29/482/1543.13

This experiment, addresses the related question of preserving correct content during repair. In a separate controlled file experiment, protected-content violations occurred in 4/18 sessions with a neutral record, 5/18 with a restatement of the prior judgment, 3/18 with a forced reread, and 1/18 when the restatement identified the object, version, and observation supporting it. The restatement and provenance conditions both averaged 2.06 reads. In a paired Writer example, both actors restored the missing line break in paragraph 5. The restatement actor then changed paragraph 33, outside the specified repair scope. The provenance-conditioned actor stopped after the first edit, and all 46 protected paragraphs passed inspection when the file was reopened. This gives concrete evidence about repair scope under a controlled change.

Reviewer_sjpzOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies task drift in long-horizon GUI agents. It proposes a wrapper that preserves the original requirements, binds progress judgments to host-observed evidence and its dependencies, and routes unsupported requirements to inspection, contradicted requirements to repair, and invalidated judgments to scoped revalidation. On OSWorld, the reported Qwen3.7-Plus score improves from 73.3 to 77.6 with the harness and to 82.1 with an additional requirement prior. Controlled studies compare evidence bundles and repair feedback.

Critical strengths
  1. Distinguishing missing evidence from a measured violation is a useful design principle. The controller avoids treating an uncertain state as something that necessarily needs changing, while dependency tracking offers a concrete way to reconsider stale completion beliefs.

  2. The controlled repair study supports the value of localized feedback.

Critical weaknesses
  1. Baseline comparability is unclear. Table 1 does not establish that external baselines use the same subset, benchmark revision, or auxiliary verification resources. A shared 100-action limit alone does not establish this.

  2. Task counts are not reconciled. Section 4.1 describes 361 canonical tasks and a fresh run over 359 tasks, but also excludes nine Google Sheets tasks and two unreachable tasks. The public OSWorld description instead distinguishes 369 tasks from a 361-task setting excluding eight Google Drive tasks. The exact manifest, scored denominator, and infrastructure exclusions are needed, especially because only naturally terminating, officially scored runs enter the results.

  3. The distinction from prior work needs sharpening. VLAA-GUI already uses observable success criteria and completion gating, while VAGEN probes the environment for verification. The potentially distinctive contribution is persistent, dependency-scoped requirement maintenance, not environment-grounded verification alone.

  4. Independent reproducibility is weak. The submission provides neither a code attachment nor an accessible repository for this method.

Questions for authors

See Critical Weaknesses.

1 comment

Thank you for recognizing the distinction between missing evidence and a measured violation. With Qwen3.7-Plus as the actor and a 100-action limit, the mean OSWorld score in Table 2 rises from 73.3 to 77.6 with the harness alone, and to 82.1 with the requirement prior. The harness therefore accounts for a 4.3-point gain before the prior is introduced. The design addresses a problem that arises even after a correct verification: a later action can change the file or object that supported an earlier judgment. The agent then needs to know which requirement to check again and which parts of the work to preserve.

1–2. Evaluation protocol and baseline comparison. We evaluated our harness twice on the 361-task OSWorld set. All runs terminated naturally. Two tasks could not be scored because the websites required for evaluation, Macy’s and IMDb, were inaccessible. The mean scores over the 359 scored tasks were 80.5 and 82.1 in the two evaluations. Table 1 provides context from published systems. Table 2 compares configurations with the same actor and action limit.

  1. Distinction from VLAA-GUI and VAGEN. VLAA-GUI checks observable success criteria against the current interface and verifies the agent’s completion claim. VAGEN actively probes the environment to assess the outcome of a trajectory. Our controller addresses a further question during execution: after the agent changes a document, which earlier judgments can it still rely on? It records the object and property that supplied the evidence for each requirement. If a later edit may have affected that evidence, the controller reopens the corresponding requirement for inspection. A measured discrepancy then identifies where a repair is needed, while requirements supported by unaffected evidence remain satisfied. VLAA-GUI · VAGEN

We tested the choice of what to recheck. Both policies allowed at most three additional checks per run.

ConfigurationOfficial score (%)ConfigurationOfficial score (%)
A · Baseline74.73E · Independent summary74.51
B · Chronological facts79.39F · Source locator73.14
C · Object-organized facts74.37G · Full checks after changes76.67
D · Exploration prompts77.89H · Dependency-scoped checks84.79

One checked all requirements after a change. The other selected requirements whose dependencies may have changed, falling back to a full check when the affected requirements were unclear. Across 39 tasks with matched runs, the latter policy scored 7.73 points higher. Mean trajectory lengths were similar, at 19.90 steps for scoped checks and 20.79 for full checks. The scoped policy thus obtained the higher observed score with a slightly shorter average trajectory. This is an interim comparison, and the paired 95% interval includes zero.

The controlled experiment illustrates why the scope of a repair matters. Both branches read the same source presentation and target document. Both correctly restored a missing line break between a name and a job title . With an ordinary summary, the agent then copied another line break from the source into paragraph 33. The source did contain that break, so the edit had a plausible rationale. It nevertheless changed a paragraph the repair instructions required it to preserve. When the same summary identified the measurement and document version it referred to, the agent stopped after fixing paragraph 5. Reopening the two files confirmed that both repaired the target, but only the second preserved all 46 other paragraphs. Across 18 controlled file cases per policy, 11 of 18 final files passed both checks with the ordinary summary, compared with 14 of 18 when the summary identified its source. Both policies averaged 2.06 reads. Checking only whether the missing line break had been restored would have counted both Writer outputs as successes and missed the unintended edit.

Reviewer_T1d1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper separates persistent task requirements from mutable plans and uses an evidence ledger to inspect, repair, and selectively revalidate completion claims in GUI agents. It reports improved OSWorld task scores through same-actor ablations and studies discrepancy feedback and preservation of correct states in controlled experiments.

Critical strengths
  1. The controller distinguishes missing evidence from evidence of an unmet requirement and routes them to inspection and repair, respectively. Binding support to state dependencies also gives a concrete way to revisit completion claims after environmental changes. This is a useful design for long-horizon agents.
  2. The same-actor ablation provides evidence of a practical benefit, and the controlled feedback experiment checks whether repairs preserve already-correct states. These are relevant checks for a method intended to improve completion reliability.
Critical weaknesses
  1. The eval set is insufficiently specified for the main comparison. Section 4.1 lists 361 canonical tasks, then a run over all 359 tasks, while also reporting the exclusion of nine Google Sheets tasks and two additional tasks. These counts are not reconciled, and only naturally terminated runs with official scores enter the results. Without per-system denominators and a common task manifest, I cannot determine whether Table 1 establishes a lead on the same evaluation set. This materially limits the evidence supporting the headline result.
  2. A major source of gain is not reproducibly defined. In Table 2, the requirement prior adds 4.5 points, compared with 4.3 from the harness, but its construction is not described. Section 3.2 also uses frozen conditions, candidate differences, and a fallible prior as checker inputs. Their provenance and generation procedures are needed to establish what information the agent receives and how much of the gain is attributable to the proposed method. I do not infer evaluator leakage; the missing specification prevents that assessment.
Questions for authors
  1. Can you reconcile the task counts and provide the task manifests, scoring denominators, and handling of infrastructure failures or non-natural terminations for the systems in Table 1? Please identify any existing comparison on a common task set.
  2. How are the requirement prior, frozen conditions, and source-bound discrepancies generated? Templates or pseudocode, together with their separation from official evaluators and test annotations, would materially affect my assessment of Table 2.
1 comment

Thank you for highlighting the distinction between missing evidence and evidence that a requirement is unmet. This distinction matters in practice. In Table 2, with the same Qwen3.7-Plus actor and the same 100 action limit, the mean OSWorld score rose from 73.3 to 77.6 when we added the evidence controller, and to 82.1 when we also enabled the requirement prior. The controller therefore contributed 4.3 points before the prior was introduced. The additional 4.5 points measure the benefit of the prior with the controller in place.

  1. Task coverage and fairness. The two tasks in question required access to Macy’s and IMDb. Both sites were inaccessible from our evaluation environment, so we excluded these tasks for network availability. The original analysis averaged official scores from naturally terminated runs. We will clarify the separate reference to nine Google Sheets tasks in Section 4.1 and identify the evaluation set reported for each external result in Table 1. We also compared eight configurations using the same actor and v19 backbone. A is the shared baseline. Each of B through H is a separate variant of A, and the rows do not represent cumulative additions. Scores are official task scores, averaged first across available repeats within each task and then across tasks. | Configuration | Score (%) | |:---|---:| | A Baseline | 74.73 | | B Chronological facts | 79.39 | | C Object-organized facts | 74.37 | | D Exploration prompts | 77.89 | | E Independent summary | 74.51 | | F Source locator | 73.14 | | G Full checks after changes | 76.67 | | H Dependency-scoped checks | 84.79 |

H achieved the highest mean score, 10.06 percentage points above A and 5.40 points above B, the strongest alternative. D encouraged further exploration. G repeated all checks after a change. H revisited the requirements whose supporting evidence could have been affected by that change. H scored 6.90 points above D and 8.12 points above G, while using slightly fewer steps on average (19.90 for H, 20.46 for D, and 20.79 for G). Thus, the highest scoring configuration also had the shortest mean trajectory among these three. The differences reported here are differences between configuration means. G had official scores for 39 tasks, and each other configuration had scores for 40.

  1. How the checker inputs are generated. In the implementation we audited, the model generates the prior before its first action from the user’s instruction, the initial source material, and the initial observation. The prior suggests what to check and what to preserve. At initialization, each frozen condition retains the passage of the instruction that supports it. During execution, the checker compares the expected property with a reading from the relevant object and version. For example, the instruction specifies Total in cell A12, while the workbook contains TotalTotal. That observation supports a repair to A12. If the cell has not been read, the controller requests an inspection. The official evaluator scores the outcome separately. In Figure 4, we constructed discrepancies from the controlled experiment’s specification to isolate the effect of the feedback given to the actor.

We tested whether a conflicting prior could override the user’s wording. The same 123 word document was rejected under “exactly 120 words” and accepted under “115 to 125 words,” with either a strict or a permissive prior. Across six file properties, the checker made 68 correct decisions in 72 checks and abstained on the other four. One case shows where the prior can still interfere. A workbook contained 20 locations and satisfied the instruction “at least 20,” but a prior suggesting “exactly 21” led the checker to abstain. We will include the generation template and checking pseudocode so readers can trace each decision from the user’s wording to the observed value.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.