- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper argues that GUI agent needs to make a decision on what and when to act. The current practice, where agents typically wait for a fixed interval or for global image differences to vanish, conflate two events as one. Instead, the paper propsoed to separate two conditions.
- Observation maturity: the screenshot contains the evidence the actor needs;
- Execution readiness: the interface can accept the action when it arrives.
The authors measure the two with identical-seed counterfactual replay in 2 task faimily from MiniWoB. Two synthetic interfaces show that a whole-screen pixel-stability rule can fire too early or too late. The authors then train an MLP with one hidden layer on the actor's hidden states, pixel change and elapsed time. It predicts whether querying the actor on the current frame will succeed, and it commits at the first frame whose score passes a threshold chosen on validation.
In an offline evaluation, the controller matches a validation-selected 0.5 s fixed wait at 83.75% success. An online evaluation gives equal success but no reduction in observation time, and completion is slower once feature extraction is counted.
Critical strengths
The paper names a decision that current GUI agents make implicitly, i.e., when to look again. It separates two causes of failure that fixed sleeps treat as one.
Critical weaknesses
The storyline is easy to follow, but too many details are ommited, making it extremely difficult to understand the paper. Just name a few to make the point.
-
The equations in Section 3 are mostly also hard to parse. For example, Eq. (1), =
⊮[replaying at time succeeds], introduces its symbols without defining them. The might be State of Seed (for the simulation). Plus, Eq. (1) is never used. In Section 3.2, timing symbols are inconsistent. Section 3.1 treats as an absolute time (I guess?). Then Section 3.2 switches to and for observation times and to "delays". and for execution. -
Fig. 2B is malformed. Its y-axis ticks read 0, −0.5, 0, and the negative differences are drawn as upward bars whose heights do not follow their labels: the −0.518 bar is shorter than the −0.493 bar, and the −0.236 bar is taller than the −0.279 bar. Terms such as "validation-locked", "frozen episodes" and "mechanism intervention" are never defined.
The experiments are also weak as the scale is small. All evidence comes from two MiniWoB tasks with sub-second transitions, where a 3B actor is used to make a single decision after one opening action. There is no realistic environment (WebArena/VisualWebArena, OSWorld, AndroidWorld), no multi-step task and no second actor. The claims about GUI agents in general are not supported.
Questions for authors
- How were the animated and no-delay variants built, and how long do the transitions last?
- How were the two synthetic interfaces built and timed?
- What are the actor's prompt, action space, and generation configurations?
- Which hidden state serves as ?
- How does the learned controller perform on the static-pending and moving-ready interfaces? What do
Δpix,Δt, and achieve in equation (2)?