acceptodds
Withdrawn from ICLR 2027

The World Doesn't Wait: Windowed Action Commitment for Real-Time GUI Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-time GUI agents choose actions from screenshots while the interface continues to change. An action that is correct for the observed state can therefore be invalid when executed. Preparing several actions reduces repeated inference, but leaves later actions tied to increasingly old observations. We study how to preserve useful decisions within such a plan. Our central insight is that task choices can outlive their original execution contexts. We introduce windowed action commitment, which binds a bounded action sequence to observable targets. The executor uses fresh visual feedback to retain compatible actions, skip expired independent prefixes, and rebind moved targets, replanning when the remaining commitments cannot be used. Compared with single-step execution in the faster settings of two instrumented Android tasks, successor reuse raises correctness from 12% to 32% at equal mean model-call count; target rebinding raises correctness from 53.0% to 71.7% while reducing calls by 53.1%. Cross-model diagnostics distinguish constructing a usable window from executing it successfully. These findings identify selective reuse of prepared decisions as a mechanism for maintaining task progress while interfaces change.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI agents, real-time interaction, action chunking, vision-language models, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_WDeBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the problem that a GUI agent's action can be correct for the observed screenshot yet fail when it reaches the device, because the interface continues to evolve during model inference. The central insight is that a task choice can outlive its execution context: an answer computed from a preview remains usable when its question becomes current, and a moved target may need only a new tap location. The authors formalize this as windowed action commitment, a bounded, revocable execution contract that couples a model-selected action sequence (K ≤ 2) with visible target descriptions, which the runtime retains, rebinds, or revokes against fresh screenshots without additional model calls. On two clock-controlled Android tasks, successor reuse raises TurboQuiz correctness from 3.6 to 9.6/30 at equal model-call count, and rebinding raises Ticket Queue correctness from 53.0% to 71.7% while reducing calls by 53.1%. Action-level attribution shows the gains come from reuse rather than checking; cross-model diagnostics identify usable window construction as the prerequisite; and a matched Tetris control rules out release timing as an alternative explanation.

Critical strengths
  1. Clean factorial decomposition with action-level attribution. The 2×2 design (window × feedback) separates preparation from feedback, and the key contrast—checking without a window (C) cancels all 188 actions and scores zero, while feedback with a window (D) recovers 141 successor injections with 96 correct—demonstrates that "reuse, not checking, produces gains." The source-correctness / target-correspondence / task-credit distinction across all 1,679 injections, the formalization of action age and target availability, and the principle of evaluating feedback by preserved work are durable contributions.

  2. Honest reporting of boundary conditions. The paper reports that rebinding at the slower 7.5 s refresh hurts by 4.7 points, and that four of five models produce zero two-action plans, making the mechanism inapplicable. These negative results strengthen the credibility of the positive findings.

  3. The Tetris release-timing control is a clean contrast. Earlier injection (56.95% latency reduction) yields 115/120 zero-score episodes, ruling out release timing as an explanation and motivating the commitment mechanism itself.

Critical weaknesses
  1. The tasks and mechanism carry only the "skin" of GUI, not its "core." None of the three tasks involves GUI semantics—timed quiz answering, single-rule classification, real-time control—each replaceable by an API without changing its intelligence demands. The mechanism likewise reduces every GUI-understanding component to task-specific deterministic rules: OCR text matching, a fixed-layout coordinate transform, a geometric index-to-grid mapping—the reused action bypasses the model's visual grounding entirely. The counterfactual confirms this: swapping screenshots for structured state input and taps for API calls would leave the headline results unchanged—GUI is the vehicle of measurement, not the object of study. A controllable clock cannot explain the missing GUI semantics—the same clocked environment could host tasks requiring navigation, state tracking, or re-grounding—and with all evidence confined to these two self-built tasks, the paper's practical value to the GUI agent community is difficult to assess.

  2. The cross-model diagnostic undermines the generality claim. Four of five models produce 0/3,997 two-action plans, leaving the mechanism inapplicable—and actively harmful: policy D scores 3.5 versus 19.7 out of 30 for single-step A on Qwen3-VL-8B at the 2.5 s deadline. The paper attributes this to prompt design and shows a fix restores two-action output, yet the formal matrix is never re-run with the fixed prompt, and the Appendix D.3 substitute uses a single development seed without estimating a cross-model gain. The full factorial results thus hold for Qwen3.8-27B alone; whether the gains transfer—or even remain non-harmful—remains unknown.

Questions for authors
  1. Can the authors provide even a small-scale live-clock result (e.g., 3–5 seeds on at least one additional model) with the structured two-answer prompt, to establish that the reuse gains are not specific to Qwen3.8-27B?

  2. The target-matching mechanism relies on task-specific rules: OCR text comparison for TurboQuiz and fixed-layout coordinate transform for Ticket Queue. In a real GUI with diverse widget types, dynamic layouts, and semantic state changes (e.g., a button changing from "Submit" to "Processing"), how would the compatibility test m(o, ẑ) be implemented without falling back to a model call—and if a model call is needed, does the cost-benefit balance that motivates this work still hold?

  3. The reuse scenario illustrated in Figure 1—a previewed future target becoming actionable under a fixed preview-to-grid mapping, with independent subtasks—has few counterparts in real GUI workflows, where future targets are generally not visible in advance. What is the concrete correspondence between this mechanism and real GUI agent tasks?

  4. The comparison to prior chunking work (ACT, Diffusion Policy, π₀, RTC) stops at a one-sentence capability claim—what can be revoked—with no analysis of when revocation is beneficial or at what cost, and no discussion of why RTC's execution-conditioned chunk overlap does not transfer to the GUI setting. Is the claimed novelty—skipping an independent expired prefix and rebinding a moved target while preserving its selected operation—a general mechanism, or an artifact of the tasks' specific structure (independent questions, fixed card layout)?

  5. The paper contains no explicit limitations section. Given the constrained scope (two self-built tasks, K ≤ 2, one primary model, task-specific matching), can the authors add a discussion of scope and transferability?

Reviewer_Cd9QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how GUI agents can reuse decisions when the interface changes during inference. A window couples up to two actions with visible target descriptions. Fresh screenshot checks can retain an action, skip an expired independent prefix, or rebind a selected point to a moved target. Paired experiments on two instrumented Android interfaces separate preparation, checking, and rebinding. Additional model-interface diagnostics examine whether requested windows actually contain multiple actions, while a separate Tetris study distinguishes faster action release from task improvement.

Critical strengths
  1. The paired ablations provide a convincing account of the mechanism within the tested settings.

  2. The 19-page appendix (Appendices A-J, pages 13-31) reports complete matrices, timing costs, target-attribution coverage, prompt diagnostics, and evidence-provenance limitations.

Critical weaknesses

1, Cross-model evidence remains limited. All 3,997 valid B/D plans from the four additional models are single-action outputs under the original interface. The corrected named-window episodes use one development seed per model and a different serving route; Appendix D appropriately states that they do not estimate a cross-model score gain.

  1. AAPT already routes observations to guarded, preauthorized actions, and ActionEngine separates program preparation from execution and recovery. Section 6 acknowledges these connections, but the conceptual distinction and applicability beyond the two task-specific readers need sharper substantiation. This is a novelty and scope concern, not evidence of plagiarism.

  2. The submission provides neither a code attachment nor an accessible repository for this method. Local script paths and archive hashes in Appendix J do not provide access to the interfaces, prompts, readers, scoring implementation, or logs.

  3. The tasks are custom instruments rather than established public benchmarks, and their generators and records are not linked. No private industrial evaluation is reported, so industrial value is not established by a public/private combination.

Questions for authors

See Critical Weaknesses

Reviewer_SMqxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses GUI agents whose interface change during inference. The model returns up to two actions. At the same time, an OCR tool reads the same screenshot and records what each action is aimed at, such as the question and its options. Before each tap, the system takes a new screenshot and compares it with these records. It then taps as planned, skips a first action whose target is already gone, or asks the model for a new plan. The method is tested on two Android tasks, TurboQuiz and Dynamic Ticket Queue, with ten paired seeds, mainly with Qwen3.8-27B.

Critical strengths

The problem is real and under-studied: it is one of the main corner cases that current GUI agents ignore and can fail on.

Critical weaknesses
  • The evaluation tasks are purpose-built toys that cover very few real-world scenarios. Each task is built around one leaking signal, where real apps rarely offer. For example, TurboQuiz shows a miniature preview of the next question, and Ticket Queue gives every ticket a unique ID on a fixed-layout card.

  • The screen checks are custom-made for each task. In Quiz, the OCR reads "fixed Quiz regions" of the screen (Sec. 4.2). In Ticket Queue, it looks for "the same identifier with unchanged issue and OPEN state", and it relies on "the internal card layout" being fixed (Sec. 3.2, App. C). The paper does not explain how the agent would know what to check in a general app, where no such regions or layouts are known.

  • skipping an expired action only works when the actions don't depend on each other. The paper says this rule "relies on the independence of Quiz questions" (Sec. 3.2), and only Quiz uses it. Ticket Queue has "no successor preview or prefix skipping" (Sec. 3.2). In many real tasks the steps do depend on each other, such as tapping a field, typing, then submitting, so skipping one breaks the task. The paper does not propose a way for the agent to tell which actions are safe to skip.

  • The tests are small in scope. Windows are capped at two actions (Sec. 3.1). Of the five models in Table 1, only Qwen3.8-27B improves. Both main tasks are "instrumented interfaces on Android emulators" (Sec. 4.1). The paper cites existing dynamic-GUI benchmarks such as DynamicGUIBench and LivingScreen (Sec. 1 and 6) but never uses them.

Questions for authors

Refer to weakness.

Reviewer_jgTHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies GUI agents whose screens keep changing while the model is thinking. It proposes letting the model plan a short window of actions, each tied to a visible target. Before each tap, the runtime checks a fresh screenshot to continue, skip an expired action, relocate a moved target, or replan. Evaluation uses custom Android tasks. The problem is real, but the gains appear only in a narrow, self-constructed regime and do not survive faster models or realistic settings.

Critical strengths
  • Real and under-studied problem. The gap between the screen a model reasons over and the screen its action lands on is a real failure mode for live GUIs. Separating the task decision from its execution context is a clean way to frame it.
  • Careful paired design. Same seeds and device per block, rotated arm order, fixed denominators that keep expired questions, a designated primary contrast, leave-one-seed-out checks. The experimental bookkeeping is more rigorous than most agent papers.
  • Negative results are reported, not hidden. The 7.5 s null, the Tetris null, the collapse of checking-only policies, and the cross-model failure all appear in the paper. Action-level attribution separating source-correct answers from credited answers is a useful diagnostic.
  • Full cost accounting. Calls, wall time, per-check latency and tokens per correct answer are reported for every arm, so the trade-off between saved inference and added checking is visible.
Critical weaknesses
  • The headline gain exists only in a regime the authors built. This is the biggest problem of this paper. It limits the generalization of proposed method. At 2.5 s TTL, a ~2.4 s model call plus the 700 ms gap outlives the question, so single-step A never hits the question it saw (0/146). At 7.5 s, D−A is +0.3. The baseline fails by construction.

  • A faster model beats the method outright. At the same fast TTL, four 8–9B models with ~0.8 s calls score 15.4–20.0/30 using plain single-step A, versus 9.6 for Qwen3.8-27B with the full method. Even with the fixed prompt (Table 18), their windowed D scores 5–11, below their own A.

  • In Quiz the mechanism reduces to "answer the preview". All 141 fast-D injections are second actions after a skipped first action: the current question always expires. A trivial baseline that only answers NEXT and taps when it appears is missing, and would likely match D without any OCR routing.

  • Noise > improvements Single-step rebinding E reaches 21.0 versus F's 21.5 (F−E +1.7, CI [−11.3, 14.7]). At the 7.5 s refresh, F is 4.7 points below A and E is 14.0 below. Windowing cuts calls; the correctness gain comes from relocation alone.

  • No real benchmark, no real app. Evidence comes from two author-built instruments plus Tetris, where nothing improves (115/120 episodes score zero). DynamicGUIBench and LivingScreen are cited in Related Work but not used. The title promises real-time GUI agents; the evaluation is three custom toys.

  • Hard to read The entire document is filled with self-created indicators and terminology.

Questions for authors
  1. Please report a preview-only Quiz baseline (always answer NEXT, tap when it becomes current) and fast single-step baselines at matched wall time. If D does not beat these, what remains of the claim that selective reuse is the source of the gain?
  2. Rerun the four-model formal matrix with the corrected CURRENT/NEXT prompt on all ten seeds, through the same direct-vLLM serving path. Does D beat A for any model whose call latency is below the TTL?
  3. Can the method work without task-specific OCR regions and unique visible IDs? Results on DynamicGUIBench or LivingScreen, or on an unmodified Android app using a generic grounder for target matching, would address this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.