acceptodds
Under review as a conference paper at ICLR 2027

Learning When to Observe: Visual Readiness for GUI Agents

Abstract

GUI agents must decide when to observe an interface again after taking an action. Existing systems often use fixed delays or global visual stability, conflating observation timing with action execution timing. We show that these are distinct control factors. Using identical-seed counterfactual replay with a frozen vision-language actor, we independently vary observation maturity and execution delay. Across two MiniWoB task families, early observations produce incorrect coordinates that cannot be repaired by delaying execution, while mature observations still require sufficient execution readiness. Matched no-delay controls isolate these timing effects from grounding failures. We further construct static-pending and moving-ready interfaces, where global pixel stability releases too early or waits unnecessarily. Motivated by these findings, we formulate post-action control as instruction-conditioned visual readiness and train a binary controller from counterfactual frozen-actor outcomes. On 80 unseen episodes, the controller matches a validation-locked 0.5-second fixed wait at 83.75% success, while reducing mean observation time from 0.500 to 0.132 seconds and mean actual execution time from 1.448 to 1.066 seconds. These results establish post-action observation timing as a distinct control problem and show that learning when to observe can preserve task success while reducing latency under a fixed actor-query budget.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI agents, vision-language models, multimodal agents, visual grounding, interactive decision making, agent efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_AB66OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that GUI agent needs to make a decision on what and when to act. The current practice, where agents typically wait for a fixed interval or for global image differences to vanish, conflate two events as one. Instead, the paper propsoed to separate two conditions.

  • Observation maturity: the screenshot contains the evidence the actor needs;
  • Execution readiness: the interface can accept the action when it arrives.

The authors measure the two with identical-seed counterfactual replay in 2 task faimily from MiniWoB. Two synthetic interfaces show that a whole-screen pixel-stability rule can fire too early or too late. The authors then train an MLP with one hidden layer on the actor's hidden states, pixel change and elapsed time. It predicts whether querying the actor on the current frame will succeed, and it commits at the first frame whose score passes a threshold chosen on validation.

In an offline evaluation, the controller matches a validation-selected 0.5 s fixed wait at 83.75% success. An online evaluation gives equal success but no reduction in observation time, and completion is slower once feature extraction is counted.

Critical strengths

The paper names a decision that current GUI agents make implicitly, i.e., when to look again. It separates two causes of failure that fixed sleeps treat as one.

Critical weaknesses

The storyline is easy to follow, but too many details are ommited, making it extremely difficult to understand the paper. Just name a few to make the point.

  • The equations in Section 3 are mostly also hard to parse. For example, Eq. (1), Ys(o,e)Y_s(o, e)=⊮[replaying aoa_o at time ee succeeds], introduces its symbols without defining them. The ss might be State of Seed (for the simulation). Plus, Eq. (1) is never used. In Section 3.2, timing symbols are inconsistent. Section 3.1 treats ee as an absolute time (I guess?). Then Section 3.2 switches to tet_e and tlt_l for observation times and to "delays". ded_e and dld_l for execution.

  • Fig. 2B is malformed. Its y-axis ticks read 0, −0.5, 0, and the negative differences are drawn as upward bars whose heights do not follow their labels: the −0.518 bar is shorter than the −0.493 bar, and the −0.236 bar is taller than the −0.279 bar. Terms such as "validation-locked", "frozen episodes" and "mechanism intervention" are never defined.

The experiments are also weak as the scale is small. All evidence comes from two MiniWoB tasks with sub-second transitions, where a 3B actor is used to make a single decision after one opening action. There is no realistic environment (WebArena/VisualWebArena, OSWorld, AndroidWorld), no multi-step task and no second actor. The claims about GUI agents in general are not supported.

Questions for authors
  • How were the animated and no-delay variants built, and how long do the transitions last?
  • How were the two synthetic interfaces built and timed?
  • What are the actor's prompt, action space, and generation configurations?
  • Which hidden state serves as hth_t?
  • How does the learned controller perform on the static-pending and moving-ready interfaces? What do Δpix, Δt, and ht−ht−1h_t-h_{t-1} achieve in equation (2)?
Reviewer_53dxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the problem of observation timing in GUI agents. The authors first introduce a counterfactual replay protocol to distinguish between observation maturity and execution readiness. They then train an auxiliary classifier using frozen VLM features, elapsed time, and pixel differences between consecutive screenshots to determine whether the agent should proceed with the current observation or wait for another one. Experiments on two MiniWoB task families show reduced latency in counterfactual evaluation while maintaining comparable success rates. However, the proposed approach offers limited methodological novelty, and the experimental evaluation remains narrow in scope. More importantly, the reported efficiency gains do not translate to the online setting, where the proposed method incurs higher end-to-end latency than a simple fixed-wait baseline. These limitations raise concerns about the practical effectiveness and generalizability of the approach.

Critical strengths
  1. The paper addressed a relatively meaningful and less-discussed issue. Separating "observation maturity" and "execution readiness" provides a clear perspective, and the counterfactual replay also effectively demonstrates that the two are indeed not the same problem.
  2. I also quite like the static-pending and moving-ready controlled examples in the paper. They clearly illustrate that global pixel stability does not equal true task readiness.
Critical weaknesses
  1. My main concern is that the claimed efficiency improvement in the paper disappears in the actual online setting. After placing feature extraction on the critical path, both the visual controller and the 0.5 s fixed wait achieved 95% success rate, but the completion time increased from 1.560 s to 1.732 s, and the observation time was almost the same. Therefore, the actual efficiency advantage of this learned controller is currently unclear.
  2. The experimental scope is also relatively narrow. The controller was only tested on two click-based MiniWoB task families and one frozen Qwen2.5-VL actor. Therefore, it is difficult to determine whether it can be transferred to longer workflows, different transition dynamics, or other actors. Additionally, the paper did not compare the frozen VLM representation with a simple learned baseline that only uses elapsed time/pixel difference, so it is unclear how much the frozen VLM representation contributes.
Questions for authors
  1. After incorporating both feature extraction and controller inference, can a visual controller be demonstrated to have an online setting that is truly faster than the best fixed-wait baseline? If not, how should the practical efficiency claim in the paper be understood?
  2. What are the results of the controller if only elapsed time, only pixel difference, or a combination of the two are used? Is the frozen VLM feature really necessary?
  3. Can the test be conducted on a more realistic multi-step GUI task or on another frozen actor? This would be more helpful in demonstrating whether the learned timing policy can surpass the current two MiniWoB task families.
Reviewer_K72hOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies when a GUI agent should capture the next screenshot after an action. The authors argue that fixed delays and global pixel stability conflate two distinct events: observation maturity (whether the screenshot contains enough task-relevant evidence) and execution readiness (whether the interface can accept the action). Using identical-seed counterfactual replay with a frozen Qwen2.5-VL-3B actor on two MiniWoB tasks, they show that actions chosen from premature observations fail regardless of execution timing. They then train an MLP readiness controller on the actor’s hidden states and evaluate on 80 frozen test episodes.

Critical strengths

Strengths

  • The problem setting is quite interesting. Separating when to observe from when to execute identifies an under-explored decision in the GUI-agent loop.
Critical weaknesses

[W1] Artificial settings, small scale, and limited ablation. All experiments use synthetic environments: two click-based MiniWoB tasks and two interfaces constructed to expose the hypothesized failures. The scale is very small, with a single 3B actor, 80 test episodes, and 40 online episodes. It is unclear whether the findings transfer to realistic benchmarks (e.g., WebArena, OSWorld), longer workflows, or other actors. No ablation compares the controller against simpler signals, such as elapsed time, pixel change, or a tuned shorter fixed wait.

[W2] Conflicting results that do not support the main claim. The headline reductions come only from a frozen evaluation with precomputed features. In the online evaluation (App. A.4), observation time is unchanged (0.632 vs. 0.635 s) and completion time increases (1.732 vs. 1.560 s). Global pixel stability, which the paper presents as unreliable, achieves the same online success as both the fixed wait and the proposed controller (38/40). Moreover, every fixed wait of at least 0.5 s also reaches this success rate. The claim that visual readiness enables earlier observation at matched success is therefore not supported in actual deployment.

Questions for authors

[Q1] Do the findings hold on realistic GUI benchmarks and with other actor families or scales?

[Q2] How does the controller compare with simpler baselines, such as a tuned shorter fixed wait, elapsed-time or pixel-change thresholds, or a lightweight target-presence detector?

[Q3] Why does the online evaluation show no reduction in observation time, unlike the frozen evaluation?

Reviewer_YDLoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when a GUI agent should capture its next screenshot after executing an action, distinguishing between observation maturity and execution readiness as two separate factors. Experiments on two MiniWoB task families show that, compared with a validation-selected fixed-wait baseline, the proposed controller reduces observation time and next-step completion time while maintaining the same success rate. However, the paper still raises concerns regarding methodological novelty, the soundness of its design, and its generalization capability.

Critical strengths
  1. The distinction between observation maturity and execution readiness is a useful aspect of this work. The paper points out that having sufficient visual evidence to determine the next action does not necessarily mean that the interface is ready for execution. This provides a more informative perspective on GUI timing than simply relying on global visual stability to decide when to observe or act.

  2. The identical-seed counterfactual replay protocol is another interesting aspect of the paper. By varying observation time and execution time independently, the authors attempt to disentangle failures caused by insufficient visual evidence from those resulting from premature action execution. This provides a useful way to analyze the two sources of timing-related failures.

Critical weaknesses
  1. The paper does not sufficiently clarify how the proposed approach differs from existing work on adaptive waiting and GUI readiness prediction. Prior GUI testing methods have already explored using visual changes, widget states, and rendering-related signals to determine whether an interface is ready, with the goal of avoiding unnecessary waiting. The distinction between these approaches and the proposed method needs to be made clearer.

  2. The counterfactual analysis relies on the assumption that episodes with identical seeds have sufficiently comparable environment states at corresponding replay times. However, using the same seed does not necessarily guarantee identical execution trajectories, particularly in interfaces involving asynchronous rendering, animations, timers, or other stateful interactions. These factors may introduce differences between replayed episodes and make it difficult to attribute the observed outcomes solely to observation and execution timing.

  3. The experimental evaluation is relatively limited in scope. It covers only two MiniWoB task families and a single frozen actor, while the main controller comparison is conducted on just 80 unseen episodes. As a result, it remains unclear how well the proposed approach generalizes to more diverse GUI environments and different actor models.

Questions for authors
  1. Could the authors clarify how the proposed visual readiness controller fundamentally differs from existing adaptive waiting and GUI readiness prediction methods?

  2. How is the execution time determined when constructing the controller's training labels? Since the success label depends on both the observation time and the subsequent execution time, how do the authors ensure that the controller learns observation maturity itself, rather than the probability of success under a particular observation-and-execution protocol?

  3. How do the authors verify that episodes with identical seeds have comparable environment states at the corresponding replay times?

  4. Could the authors evaluate the proposed controller on additional GUI environments or with different frozen actors to provide further evidence of its generalization capability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.