acceptodds
Under review as a conference paper at ICLR 2027

One Step, Four Clocks: How GUI Agents Fall Out of Step

Abstract

GUI agents can choose the right action and still fail because the interface changes while they think. We model the timing of agent responses through four events: an environment change, observation availability, decision commitment, and action receipt. These events distinguish failures in which relevant evidence is missed, a decision uses outdated evidence, an action becomes invalid after commitment, and an action arrives after the window of opportunity closes. We introduce ChronotopeGym to make these timing conditions experimentally controllable. We evaluate agent behaviors in the same task under three conditions: the interface waits; observations change while the response remains valid; or the interface continues changing while the agent thinks. Agents that succeed when the response remains valid can fail when it expires before their action arrives. But response speed alone does not explain the failures. On a moving-target task, we keep the model call and the action’s commitment and arrival times unchanged, changing only which observation supplies the click coordinate. Using the newer observation raises success from 0/100 to 100/100: the original click targets where the object was, while the revised click targets where it has moved. Guided by the framework, Chronotope allows new evidence to revise a decision before commitment, preserving useful reasoning while updating action variables that depend on the changing interface.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Computer-Using Agents, GUI Agents

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_ErHSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper models the lifecycle of a single GUI-agent response as four events—environment change (E), observation availability (O), decision commitment (D), and action receipt (A)—and distinguishes four timing failures accordingly: Missed, Stale (outdated at commitment), Invalidated, and Late. The authors build ChronotopeGym, a benchmark of five dynamic tasks under three execution conditions (Paused / Replay / Live) that makes these temporal relations experimentally controllable, and propose Chronotope, a controller that continues observing during inference, reconciles unfinished decisions before commitment (preserve / revise / late-bind / re-decide), and keeps the latency-sensitive control path resident on the GPU accelerator. The paper advances three core claims: (1) configurations that achieve perfect scores under Paused/Replay can fail completely under Live; (2) with the model call and the commitment/receipt times held fixed, changing only the observation that supplies the click coordinate flips success from 0/100 to 100/100, showing that response speed is not the sole determinant of success; (3) lightweight pre-commitment argument updates recover success without additional model calls, whereas full model restarts fail in 100/100 trials on continuously moving targets.

Critical strengths
  1. The EODA decomposition answers a question that a single latency number obscures: with the same O→A = 400 ms, different splits into (O→D, D→A) yield two distinct failures—invalidated-after-commitment versus missed-evidence (Fig 4). The four failure modes are mutually exclusive, and each calls for a different remedy (the intervention-selectivity validation in Table 3 and Fig 12 is elegant: updating state repairs only pre-D changes, earlier receipt repairs only deadline-type failures, and continuous drift requires both).

  2. The experimental protocol is rigorous and reproducible. Qualification gating (Paused ≥ 8/10) separates timing failures from ordinary task incompetence (Table 16 honestly reports all configurations that failed the gate and excludes them from every claim); held-out seeds are disjoint from development seeds; Wilson intervals (which remain informative at 0/n and n/n), exact McNemar tests, and permutation tests are used throughout; the scorer is validated on hand-authored traces and recomputed by a second, independent minimal scorer; manifests are hash-frozen; and oracle results are verified cell-by-cell against closed-form predictions written independently of the harness. Code and seeds are open-sourced.

Critical weaknesses
  1. The evaluation is confined to synthetic toy tasks, so the transferability of the conclusions to real GUI scenarios is unverified. All five tasks are synthetic, single-step, small-canvas tasks. Real GUI tasks are multi-step, semantically complex, and causally coupled across actions, which limits the transferability of the conclusions to real-world deployment. The core increment of this paper is the EODA four-event decomposition, which refines "failure" into four timing-failure modes. However, the diagnostic value of this decomposition is demonstrated mainly through oracle validation in a controlled synthetic environment.

  2. Most analytical experiments verify the self-consistency of the EODA framework itself, rather than discovering properties of real agent behavior. The experimental design is rigorous, but most of the analytical experiments employ scripted controllers to verify the self-consistency of the EODA framework and its intervention strategies—i.e., that each intervention repairs only its corresponding failure mode (Table 3). These experiments are in essence correctness checks of the harness itself rather than discovery-oriented analyses of real agent behavior.

Questions for authors
  1. Can the findings of this paper guide GUI agents in handling dynamic environments in real-world scenarios?

  2. Do the findings still hold when transferred to real-world scenarios?

  3. The paper's discussion of pre-commitment decision revision (late binding vs. restart) coincides with the classical replanning/plan-commitment problem studied in anytime planning, deliberation scheduling, and receding-horizon control, yet none of this literature is cited or compared against. How does the paper position its contribution and novelty relative to these classical results?

8 comments
AuthorsOpenReview ↗

Official Response to Reviewer ErHS (Part 1 of 3)

Dear Reviewer ErHS,

Thank you for your constructive feedback. We have substantially revised the manuscript with new evaluations, architectural ablations, and clearer experimental distinctions. We address your comments below.

1. (W1 & Q1 & Q2) “The evaluation is confined to synthetic toy tasks, so the transferability of the conclusions to real GUI scenarios is unverified... However, the diagnostic value of this decomposition is demonstrated mainly through oracle validation in a controlled synthetic environment.” / “Can the findings of this paper guide GUI agents in handling dynamic environments in real-world scenarios?” / “Do the findings still hold when transferred to real-world scenarios?”

We thank the reviewer for raising questions about the transferability and practical relevance of Chronotope's findings. We extend our evaluation to DynaCU-Bench, featuring complex, multi-step GUI interactions, and introduce a fully live War Thunder application study, featuring continuously evolving states and causally coupled actions. Chronotope achieves 67.5% versus 62.3% success on DynaCU-Bench relative to Agent S3 and 10.9% versus 0% on War Thunder. These results demonstrate benefits beyond the original controlled tasks.

Table R4a. Matched ablations of history representation and meta-control policies.

Meta-control policyHistoryChronotopeGymDynaCU-BenchWar Thunder
Always KEEPEODA/EDM57.0%64.1%7.6%
Always REDECIDEEODA/EDM51.4%67.5%6.8%
Always UPDATEEODA/EDM72.3%67.3%10.4%
Selective KEEP/UPDATE/REDECIDENone72.5%28.1%7.9%
Selective KEEP/UPDATE/REDECIDEChronological text74.1%67.0%9.3%
Selective KEEP/UPDATE/REDECIDEEODA/EDM75.6%67.5%10.9%

All configurations use the same Qwen3.8-27B planner, with history capacity N=8N=8 where applicable.

Matched ablations isolate the contributions of interaction history and selective meta-control. On DynaCU-Bench, retaining chronological text history increases success from 28.1% to 67.0%, while EODA/EDM further improves success to 67.5%. On War Thunder, EODA/EDM improves success by 17.2% relative to chronological text. With EODA/EDM fixed, selective meta-control improves success over Always KEEP by 32.6%, 5.3%, and 43.4% across ChronotopeGym, DynaCU-Bench, and War Thunder, respectively. It also outperforms Always REDECIDE on ChronotopeGym and War Thunder while matching it on DynaCU-Bench. These comparisons establish the distinct roles of retaining history, structuring event dependencies, and selecting an appropriate control policy.

Beyond controller performance, our experiments reveal generalizable properties of model-based agents under temporal constraints. The controlled interventions isolate distinct failure mechanisms without relying on evaluator-only information during agent execution. Our performance-envelope experiments complement these diagnostics by measuring end-to-end latency, response deadlines, and Live task success. Using the same Qwen3.8-27B (NVFP4) planner, Chronotope reduces median end-to-end latency (E→AE\rightarrow A) by 85.5% relative to Agent S3. At 8-second and 2-second deadlines, Chronotope achieves 100% and 81% Live task success, compared with 0% for Agent S3 at both deadlines. Neither succeeds at a 1-second deadline (Figure 7). Meanwhile, Moving Balloon and Reversing Trading Signal demonstrate that agents can fail despite meeting deadlines when environmental changes invalidate their actions. Thus, faster control expands the performance envelope, but maintaining decision validity requires temporal adaptation.

The new evaluations provide two practical findings for dynamic GUI agent design. First, multi-step tasks benefit substantially from interaction history. On DynaCU-Bench, retaining chronological history improves success by 138.4% under identical selective meta-control, demonstrating the importance of previous interactions beyond the latest observation. Second, latency-sensitive applications benefit from concise, causally structured histories. At N=8N=8, EODA reduces token consumption by 45.2% relative to chronological text, reduces median latency by 3.6%, and improves War Thunder success by 5.3%. In War Thunder, preserving dependencies between firing decisions, action receipts, and subsequent observations helps the agent recognize ineffective actions rather than repeatedly firing at obscured targets.

Together, these evaluations support three practical principles for dynamic GUI agents: distinguish latency failures from outdated decisions, preserve causally relevant interaction history, and selectively KEEP, UPDATE, or REDECIDE before commitment. Although the current updaters remain task-specific, these results demonstrate the relevance of Chronotope's temporal control principles beyond synthetic environments.

AuthorsOpenReview ↗

Official Response to Reviewer ErHS (Part 2 of 3)

**2. (W2) "Most analytical experiments verify the self-consistency of the EODA framework itself, rather than discovering properties of real agent behavior...correctness checks of the harness itself rather than discovery-oriented analyses of real agent behavior."

We thank the reviewer for raising the questions of whether Chronotope's findings transfer to realistic GUI applications and whether its experiments reveal properties of agent behavior beyond controlled validation.

To investigate Chronotope's real-world benefits, we extend our evaluation to DynaCU-Bench and a new War Thunder live application study. DynaCU-Bench evaluates semantically complex, multi-step GUI interactions, while War Thunder introduces continuously changing environments with stateful, causally coupled actions. The full Chronotope controller achieves 67.5% success on DynaCU-Bench and 10.9% on War Thunder, compared with 62.3% and 0% for Agent S3, respectively (Table R4a). These evaluations extend beyond the original controlled, single-step tasks.

Table R4a. Realistic task performance and matched ablations.

ControllerMeta-controlHistoryDynaCU-Bench SRWar Thunder SR
Agent S3NativeNone24.0%0%
Agent S3NativeNative (Chronological text)62.3%0%
ChronotopeAlways KEEPEODA/EDM64.1%7.6%
ChronotopeAlways REDECIDEEODA/EDM67.5%6.8%
ChronotopeAlways UPDATEEODA/EDM67.3%10.4%
ChronotopeSelectiveNone28.1%7.9%
ChronotopeSelectiveChronological text67.0%9.3%
ChronotopeSelectiveEODA/EDM67.5%10.9%

All Chronotope configurations use the same Qwen3.8-27B planner, with N=8N=8 for retained histories. Selective denotes KEEP/UPDATE/REDECIDE meta-control.

Matched ablations further isolate the contributions of interaction history and meta-control. On DynaCU-Bench, retaining chronological history increases success from 28.1% to 67.0%, while EODA/EDM further improves success to 67.5%. On War Thunder, EODA/EDM improves success by 17.2% relative to chronological history. With EODA/EDM fixed, selective meta-control improves success over Always KEEP by 5.3% and 43.4% on DynaCU-Bench and War Thunder, respectively. It matches the strongest fixed policy on DynaCU-Bench and outperforms all fixed policies on War Thunder. These results establish the complementary benefits of interaction history and selective temporal adaptation.

Beyond validating the EODA framework, our experiments reveal empirical properties of model-based agents under temporal constraints. While controlled interventions isolate individual failure mechanisms, our performance-envelope experiments measure end-to-end latency, response deadlines, and task success. Using the same Qwen3.8-27B (NVFP4) planner, Chronotope reduces median end-to-end latency (E→AE\rightarrow A) by 85.5% relative to Agent S3. At 8-second and 2-second deadlines, Chronotope achieves 100% and 81% Live task success, respectively, compared with 0% for Agent S3. Neither succeeds at a 1-second deadline (Figure 7). Moreover, Moving Balloon and Reversing Trading Signal demonstrate that agents can fail despite meeting deadlines when environmental changes invalidate their actions. Thus, faster control expands the performance envelope, while maintaining decision validity requires temporal adaptation.

Our evaluations further yield two practical findings for dynamic GUI agent design. First, multi-step dynamic GUI tasks benefit substantially from interaction history. The 28.1% to 67.0% improvement on DynaCU-Bench demonstrates the importance of retaining previous interactions rather than relying solely on the latest observation and pending-action state. Second, latency-sensitive applications benefit from concise, causally structured histories. At N=8N=8, EODA/EDM reduces token consumption by 45.2% relative to chronological text, reduces median latency by 3.6%, and improves War Thunder success by 17.2%. Preserving dependencies among previous actions, receipts, and subsequent observations helps agents recognize ineffective actions rather than repeatedly acting on outdated assumptions.

Together, these results provide concrete guidance for dynamic GUI agents: preserve causally relevant interaction history, distinguish latency failures from outdated decisions, and selectively KEEP, UPDATE, or REDECIDE before commitment. Although the current updaters remain task-specific, the findings demonstrate the practical relevance of Chronotope's temporal control principles beyond synthetic environments.

AuthorsOpenReview ↗

Official Response to Reviewer ErHS (Part 3 of 3)

3. (Q3) "The paper's discussion of pre-commitment decision revision (late binding vs. restart) coincides with the classical replanning/plan-commitment problem studied in anytime planning, deliberation scheduling, and receding-horizon control, yet none of this literature is cited or compared against. How does the paper position its contribution and novelty relative to these classical results?

We thank the reviewer for highlighting these connections to classical planning and meta-reasoning. We clarify the conceptual distinctions below. We have also revised our manuscript to include this discussion in the Extended Related Works section in Appendix A.3.

Anytime planning. Anytime planning produces a usable plan quickly, then continues searching for a better plan if more time is available (Dean and Boddy, 1988, Zilberstein, 1996). Its central concern is the tradeoff between computation time and the quality of the available solution. In contrast, Chronotope studies the temporal coordination of observation, decision-making, and action execution in continuously changing environments. An action can meet its deadline yet fail because the environment has changed, independently of the computation-quality tradeoff addressed by anytime planning.

Deliberation scheduling. Deliberation scheduling determines which reasoning processes to run and for how long based on their expected benefits (Russell and Wefald, 1991, Boddy and Dean, 1994). Its central problem is allocating limited computation among competing reasoning processes to maximize decision utility. In contrast, dynamic, latency-sensitive GUI agents must act in changing environments without knowing in advance how long their intended actions will remain valid. Even under identical computation schedules, actions can succeed or fail depending on whether environmental changes invalidate them before execution.

Receding-horizon control. Receding-horizon control repeatedly optimizes future control actions using updated system states and executes the next action (García et al., 1989, Mayne et al., 2000). Its central problem is future-trajectory optimization, typically using a predictive model of how the system evolves. In contrast, dynamic, latency-sensitive GUI agents operate without necessarily knowing the environment's dynamics, future states, or action-validity windows. The key challenge in our setting is not predicting an optimal future trajectory, but ensuring that an immediate action remains valid as the environment changes unpredictably.

We thank the reviewer again for the thoughtful feedback and constructive suggestions. We hope that our new experiments, additional comparisons, and expanded discussion of related work address your concerns about Chronotope's generalizability, practical benefits, and novelty. We would be happy to provide any further clarification. If these additions have sufficiently addressed your concerns, we would be grateful if you would consider revisiting your assessment.

Reviewer_J6TLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the timing problems in GUI agents that work with changing interfaces, where an action may appear to be correct for a screenshot but not for the current interface. The authors model this process using four events – environment updates, observation arrival, action commitment, and action arrival at the application. They present ChronotopeGym, which uses three classes of failures, including paused interfaces, replays of observations, and interface updates. The method, Chronotope, is based on fresh observations to update pending actions just before commitment, for instance, keeping the target of a click while updating its coordinates. The results demonstrate that refreshing the visual information used to select an action can have an excellent effect, even when the model’s call and action arrival times are fixed.

Critical strengths
  1. The paper clarifies the general issue of “agent is too slow” more precisely and the experiments back up this claim. I find the experiment in which the model call, action commitment time and arrival time is held fixed, changing only the observation used to inform the click coordinates the most convincing. For the moving-target task, this alone improves performance from 0/100 to 100/100. This shows failures depend on not only inference latency, but also the staleness of the information used to make the final action. That distinction is valuable for building GUI agents for dynamic interfaces.

  2. The controlled experiments are thorough and include situations where the method isn’t useful. The authors test out stationary targets, irrelevant changes, differing movement speeds, one-time position changes, and continued changes following action commitment. They also demonstrate that refreshing observations can’t correct failures caused purely by deadlines. The UI-TARS results are especially enlightening: updating the coordinates restores performance on dynamic tasks to 69%, just about their 70% on static tasks, without correcting the underlying localization errors in the model. Therefore, the experiments back up their conclusion more convincingly.

Critical weaknesses
  1. The methods’ current effectiveness relies too heavily on task-specific updaters; a decent amount of distance to general GUI control remains. The appendix describes position updaters using image template matching around the previous location and CAPTCHA updaters using a hard-coded decoder. The implementations both represent updates to the same decision with different parameters, but real-world interfaces also require updating the target appearance, dealing with similar controls, and avoiding accidental presses when the right one is temporarily unavailable. The authors are aware of this limitation, so my concern is related to evidence: what I see is a good update design principle, but the results do not demonstrate that the described control method is broadly applicable. In particular, it would be informative to see the comparison task in which the model simply re-localizes the target after planning because it will show how much value is in the described event-based update and decision-update loop.

  2. The claims about practical latency and GPU control path benefits are not supported by the evidence as well, or it is not presented in the paper. Appendix C.7 clearly states that all the experiments used the software track; even the browser task with the moving button only alternated between two pre-rendered pages. The main model experiments only set the action latency to 20 ms. The paper does feature analyses of latency injections and temporal jitter, but they do not concern the practical end-to-end latency of the screenshot capture, model processing, controller update, and application reaction. Most importantly, the improvement cited in the main text (9/100 to 35/100) is actually the result of using a color detector and fixed wait time, not the model-based control loop. The results effectively stage a demonstration of the method but only provide weak evidence for its real-world benefits.

Questions for authors
  1. Under the same observation frequency and compute budget, what is the Chronotope's advantage over simply replanning the target location after observing and maintaining only its latest location and dependencies? What results clearly demonstrate what improvements are being made by maintaining this extra information?

  2. If the target changes, or disappears behind other objects for a brief period, or otherwise ceases to be a target, how does the current updater decide between continuing to update the coordinates or abandoning it in favor of another target? Would the authors please consider adding results under these conditions to demonstrate their method's applicability?

8 comments
AuthorsOpenReview ↗

Official Response to Reviewer J6TL (Part 1 of 4)

Dear Reviewer J6TL,

Thank you for your constructive feedback. We have substantially revised the manuscript with new evaluations, architectural ablations, and clearer experimental distinctions. We address your comments below.

1. (W1a & Q2) “The methods’ current effectiveness relies too heavily on task-specific updaters…but the results do not demonstrate that the described control method is broadly applicable.” / “If the target changes, or disappears behind other objects for a brief period, or otherwise ceases to be a target, how does the current updater decide between continuing to update the coordinates or abandoning it in favor of another target?”

We thank the reviewer for raising these questions about generalizability and the handling of changes that invalidate an existing decision. Chronotope's central contribution is meta-control over KEEP, UPDATE, and REDECIDE, rather than any particular task-specific updater. We clarify the distinction between updating action arguments and reconsidering the underlying decision, and provide new experimental evidence.

Handling changing or invalid targets. Chronotope separates argument updates from decision revision. When a target moves but the original decision remains valid, UPDATE can adjust the pending action's coordinates. When the target disappears, becomes ambiguous, or ceases to be appropriate, REDECIDE allows the planner to reconsider the underlying decision rather than continuing to update coordinates. The updater itself is not responsible for deciding whether the original goal remains valid. This distinction is important for the reviewer's examples of changing appearance, similar controls, and temporarily unavailable targets, which cannot always be resolved through coordinate updates alone.

For example, in ChronotopeGym's Reversing Trading Signal, a signal reversal invalidates the original trading decision, requiring reconsideration rather than an argument update. In our new War Thunder live application experiment, an enemy vehicle may disappear behind a hill after the planner issues a firing decision. Continuing to fire at its previous location wastes ammunition and incurs reloading time without advancing the objective. This requires reconsidering the decision rather than simply re-localizing the target. To isolate the contribution of meta-control, we compare four policies while holding EODA/EDM, the planner, updater, and execution conditions fixed.

Table R3a. Live task success under different meta-control policies.

Meta-control policyChronotopeGymDynaCU-BenchWar Thunder
Always KEEP57.0%64.1%7.6%
Always REDECIDE51.4%67.5%6.8%
Always UPDATE72.3%67.3%10.4%
Selective KEEP/UPDATE/REDECIDE75.6%67.5%10.9%

Selective meta-control improves success over Always KEEP by 32.6%, 5.3%, and 43.4% on ChronotopeGym, DynaCU-Bench, and War Thunder, respectively. It also matches or exceeds the fixed UPDATE and REDECIDE policies. These results demonstrate the value of selecting among control operations rather than applying a single response to every environmental change. We have revised our manuscript to include this ablation in Table 1 in Section 5.3.

AuthorsOpenReview ↗

Official Response to Reviewer J6TL (Part 2 of 4)

2. (W1b & Q1) “In particular, it would be informative to see the comparison task in which the model simply re-localizes the target after planning because it will show how much value is in the described event-based update and decision-update loop.” / “Under the same observation frequency and compute budget, what is the Chronotope's advantage over simply replanning the target location after observing and maintaining only its latest location and dependencies?”

We thank the reviewer for suggesting this comparison to clarify the contribution of Chronotope beyond simple target re-localization. We address this by isolating the contribution of interaction history under matched controller and execution conditions.

Comparison against a latest-observation controller. A controller retaining the latest observation and conventional pending-action state can update action arguments based on the current target location. However, this does not provide the planner with earlier interactions and their outcomes. Chronotope's EODA/EDM instead organizes dependencies among environment events, observations, decisions, and action receipts. This information supports both the grounder, which updates pending-action arguments, and the planner, which uses previous outcomes to inform subsequent decisions.

We conduct a matched ablation comparing three history configurations: no history, retaining only the latest observation and conventional pending-action state; chronological text history; and EODA/EDM history. The two history-based configurations retain up to N=8N=8 changed observations. All configurations use the same Qwen3.8-27B planner, observation schedule, updater, selective KEEP/UPDATE/REDECIDE policy, execution budget, task instances, and hardware. We report actual token consumption and latency separately to quantify the costs of each representation.

Table R3b. Interaction-history ablation under matched controller conditions.

HistoryChronotopeGym SRDynaCU-Bench SRWar Thunder SRTokens/episodeMedian latency
None72.5%28.1%7.9%3,9684.7 s
Chronological text74.1%67.0%9.3%15,5045.5 s
EODA/EDM75.6%67.5%10.9%8,5025.3 s

Contribution of retained interaction history. Relative to no history, EODA improves success by 4.3% on ChronotopeGym, 140.2% on DynaCU-Bench, and 38.0% on War Thunder. Chronological text history recovers much of the improvement on DynaCU-Bench, demonstrating that retaining previous interactions is important independently of their representation.

Contribution of dependency-structured history. Relative to chronological text, EODA further improves success by 2.0%, 0.7%, and 17.2%, respectively, while reducing token consumption by 45.2% (and median latency by 3.6%. Thus, EODA provides a more compact representation of previous interactions, with a particularly clear success improvement on War Thunder. Increasing EODA history capacity from N=8N=8 to N=16N=16 improves DynaCU-Bench success by 2.2%, but increases token consumption by 43.5%, supporting N=8N=8 as a practical balance. We have revised our manuscript to include these comparisons and the history-capacity analysis in Appendix D.1.

Beyond target re-localization. In War Thunder, for example, an enemy vehicle may disappear behind a hill after a firing decision. Re-localizing the target alone cannot determine whether continuing to fire is appropriate. EODA preserves dependencies among previous firing decisions, action receipts, and subsequent observations, allowing the planner to consider whether earlier actions were ineffective and whether the decision should change. Since the same selective meta-control policy is used across these configurations, the ablation isolates the contribution of interaction history rather than changes to the decision-update policy.

AuthorsOpenReview ↗

Official Response to Reviewer J6TL (Part 3 of 4)

3. (W2a) “The claims about the practical latency and GPU control path benefits are not supported by the evidence as well, or it is not presented in the paper.”

We thank the reviewer for highlighting the need to distinguish Chronotope's overall architectural advantage from the incremental contribution of accelerator residency. We address this through new performance-envelope experiments with VLM planners and matched CPU-versus-GPU comparisons within Chronotope.

Full-controller performance envelope. We compare Chronotope against Agent S3 using the same planner model, measuring Live task success under different response deadlines and end-to-end latency from environment event to action receipt (E→AE\rightarrow A). The measurements capture the combined effects of observation, decision-making, and action delivery, rather than only isolated updater execution.

Table R3c. End-to-end latency and Live task success with Qwen3.8-27B (NVFP4).

ControllerE→AE\rightarrow A p50 (ms)SR 16sSR 8sSR 4sSR 2sSR 1s
Agent S312,84088%0%0%0%0%
Chronotope CPU1,910100%100%97%75%0%
Chronotope GPU1,864100%100%98%81%0%

With Qwen3.8-27B, Chronotope GPU reduces median end-to-end latency by 85.5% relative to Agent S3. At 8-s and 4-s response windows, Chronotope achieves 100% and 98% Live success, respectively, compared with 0% for Agent S3. At 2 s, Chronotope maintains 81% success, while both controllers fail at 1 s.

Table R3d. End-to-end latency and Live task success with Qwen3.5-0.8B.

ControllerE→AE\rightarrow A p50 (ms)SR 16sSR 8sSR 4sSR 2sSR 1s
Agent S33,39199%96%96%0%0%
Chronotope CPU31099%97%97%95%90%
Chronotope GPU26799%97%97%96%93%

With Qwen3.5-0.8B, Chronotope GPU reduces median latency by 92.1% relative to Agent S3 and achieves 93% versus 0% success at a 1-s deadline. Together, these results demonstrate that Chronotope's architectural improvements expand the achievable latency-deadline-success-rate envelope across model sizes.

Incremental benefit of GPU residency. To isolate accelerator residency from the broader architecture, we compare CPU and GPU implementations within Chronotope, holding the planner, EODA/EDM history, and control policy fixed. With Qwen3.8-27B, GPU residency further reduces median latency by 2.4% and improves Live success at 2 s from 75% to 81%. With Qwen3.5-0.8B, GPU residency reduces median latency by 13.9% and improves success at 1 s by 3.3%. These results distinguish two contributions. The comparison with Agent S3 establishes the erformance advantage of Chronotope's complete control architecture, while the matched CPU-versus-GPU comparison quantifies the smaller incremental benefit attributable to accelerator residency. We have revised our manuscript in Section 5.1, Table 2, and Figure 7 to present these performance-envelope results and clarify the separate contributions of the full controller and accelerator residency.

Reviewer_BHvTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that latency alone cannot explain why GUI agents fail on changing interfaces. It splits each response into four events (environment change, observation, commitment, receipt) and names four failure types. ChronotopeGym runs synthetic tasks under paused, replayed and live clocks. Chronotope updates a pending action's coordinate or value from the newest frame before committing it. The diagnostic framing is useful, but the experiments are deterministic toy cases and the fix is a task-specific tracker.

Critical strengths
  • Useful diagnostic framing. Separating missed evidence, stale decisions, post-commit invalidation and late arrival is a clean vocabulary. Paused versus Replay versus Live neatly isolates "observations change" from "the task itself expires". The benchmark survey of timing semantics (Table 4) is useful on its own.
  • Careful causal design. Same-decision counterfactuals hold the model call, commitment and receipt fixed and change one factor. Negative controls (static targets, irrelevant motion), selectivity checks across interventions (Figure 12), Wilson intervals and exact McNemar tests are all done properly.
  • Some non-obvious findings. Restart-on-update can be worse than doing nothing (2 px/s: 0/100 vs 100/100), and a wasted early decision costs 1.5 s that success rate hides but return reveals. These are the kind of results the framework is good at exposing.
Critical weaknesses
  • The tasks do not need a VLM. In Table 10 a plain color detector scores 10/10 on the card and slow-balloon tasks, while Qwen3.8-27B without the binder scores 5/10 and 4/10. When a hard-coded detector solves the benchmark, it is measuring latency, not GUI agent competence.
  • Toy environment only. Every task is a single click or typed value on a synthetic canvas. The "Chromium page" is two pre-rendered frames, not a live browser. The physical capture path is described but never measured. DynamicGUIBench, LivingScreen, DynaCU-Bench and the AAPT testbed are surveyed but not used.
  • No comparison to the closest methods. AOI (continuous observation), AAPT (precompiled guarded policies) and MobileDreamer (state prediction) all target the same failures and appear in Related Work. None is run. The only baselines are the authors' frozen coordinate and restart controllers.
  • The accelerator-resident contribution is marginal. It is tested with a color detector plus a fixed sleep, not a model, on one boundary cell at 200 px/s (9/100 vs 35/100). The two binders differ by about 1 ms. That is a tuned edge case, not a design principle.
  • The linked code cannot be accessed. ChronotopeGym is a headline contribution, and the reproducibility statement points to an anonymous repository. During the review period the repository shows only "This content can't be shown". The prompts are deferred to that code, so neither testbed nor results can be checked.
  • Presentation is unfinished. Nine broken "§??" cross-references, prompts deferred to code ("will be reproduced verbatim in the released code"), and a 10-episode pilot table that is very hard to parse. An Ethics Statement lists first- and second-author contributions and cites U.S. securities regulations for a synthetic chart task.
  • Please differentiate /cite and /citep.
Questions for authors

Overall, the paper is interesting, but the writing is very very poor (see last weakness and Q2)

  1. Please evaluate on an existing dynamic GUI benchmark (DynamicGUIBench, LivingScreen, or DynaCU-Bench) or on real multi-step applications. Use a general binder such as re-querying a grounding model on the newest frame, not NCC or a glyph decoder. Does the improvement survive?
  2. (Clarification) Fix the broken references and include all prompts in the appendix. What does the GPU-resident path buy when the planner is a real VLM, rather than a color detector with a fixed sleep?
  3. The anonymous repository is not accessible ("This content can't be shown"). Please restore access, or attach the task suite, evaluators, controller, all prompts and the per-trial traces behind Tables 2 and 8–10 as supplementary material. For a testbed paper, this is necessary.
9 comments
AuthorsOpenReview ↗

Official Response to Reviewer BHvT (Part 1 of 7)

Dear Reviewer BHvT,

Thank you for your careful reading and constructive suggestions. We have substantially revised the manuscript with new evaluations, architectural ablations, and clearer experimental distinctions. We address your concerns below.

1. (W1 & Q1b) “The tasks do not need a VLM...When a hard-coded detector solves the benchmark, it is measuring latency, not GUI agent competence.” / “Use a general binder such as re-querying a grounding model on the newest frame, not NCC or a glyph decoder. Does the improvement survive?”

We thank the reviewer for highlighting the distinction between semantic GUI competence and temporal correctness. ChronotopeGym deliberately uses controlled tasks to isolate failures that arise when an agent selects a correct action but the interface changes before that action is received. Each agent configuration must first pass a Paused qualification check, allowing us to distinguish failures caused by temporal changes from failures in basic task competence. The original Table 10 compared VLM-driven agents with a color-detector baseline on simple tasks. The detector's performance demonstrates that these tasks admit inexpensive specialized solutions. To establish how well the proposed controller handles more general GUI interactions, have therefore removed the original pilot table from the main argument and refocused the evaluation on temporal correctness, Live task success, and the contributions of Chronotope's control mechanisms.

We additionally evaluate general-purpose grounding models as pending-action updaters. Rather than relying exclusively on task-specific computer vision routines such as normalized cross-correlation or fixed-font decoding, we use Qwen3.5-0.8B and UI-TARS-1.5 to update action arguments from the newest observation. The Qwen3.8-27B planner and Chronotope controller remain unchanged across updater choices.

Table R2a. Live task success with task-specific and general-purpose updaters.

UpdaterChronotopeGymDynaCU-BenchWar Thunder
Task-specific CV75.6%67.5%10.9%
Qwen3.5-0.8B70.3%58.1%8.2%
UI-TARS-1.573.3%63.8%9.0%

The results show that general-purpose grounding models preserve much of Chronotope's performance without specialized visual routines. For example, UI-TARS-1.5 achieves 73.3% versus 75.6% on ChronotopeGym and 63.8% versus 67.5% on DynaCU-Bench. The specialized updater remains more effective and computationally inexpensive, but the controller can operate with general-purpose grounding. These findings distinguish the proposed control architecture from any particular argument-updating implementation. The contribution is not that every GUI task requires complex visual reasoning, but that actions selected by VLM agents must remain valid as interfaces change during inference.

AuthorsOpenReview ↗

Official Response to Reviewer BHvT (Part 2 of 7)

2. (W2 & Q1) “Toy environment only.” / “Please evaluate on an existing dynamic GUI benchmark (DynamicGUIBench, LivingScreen, or DynaCU-Bench) or on real multi-step applications.”

We thank the reviewer for emphasizing the importance of evaluation beyond controlled synthetic tasks. ChronotopeGym was designed to isolate temporal failure mechanisms under Paused, Replay, and Live execution, rather than to establish general GUI-agent competence. We retain these experiments because they distinguish missed deadlines, outdated observations, and invalidated actions under controlled timing. We have also clarified that the original Chromium example uses pre-rendered interface states and that describing a physical capture path does not establish its measured performance.

We substantially extend the evaluation to DynaCU-Bench and the live War Thunder application. These environments introduce interaction demands beyond the original single-step diagnostic tasks. DynaCU-Bench includes multi-step dynamic GUI interactions that depend on earlier actions and their outcomes. War Thunder adds continuously evolving, causally coupled interactions in which previous actions affect subsequent decisions. For example, repeatedly firing at an enemy obscured by terrain is ineffective, requiring the agent to use prior interaction outcomes rather than simply re-localizing a target in the latest frame.

To evaluate the resulting architectural contributions, we introduce matched ablations of interaction history and decision meta-control. The no-history baseline retains the latest observation and conventional pending-action state, while Always UPDATE applies the same argument updater without selective KEEP/UPDATE/REDECIDE control.

Table R2. Selected matched ablations of interaction history and meta-control.

ConfigurationChronotopeGymDynaCU-BenchWar Thunder
No history, Always UPDATE68.0%28.1%7.1%
No history, Selective72.5%28.1%7.9%
Chronological text, Selective74.1%67.0%9.3%
EODA/EDM, Always UPDATE72.3%67.3%10.4%
EODA/EDM, Selective (Chronotope)75.6%67.5%10.9%

Relative to the no-history Always UPDATE baseline, full Chronotope improves success by 11.2%, 140.2%, and 53.5% on ChronotopeGym, DynaCU-Bench, and War Thunder, respectively. The largest improvement on DynaCU-Bench arises from retaining interaction history, which allows the planner to account for earlier actions and their outcomes. With EODA/EDM fixed, selective meta-control provides further improvements over unconditional updating, although these are smaller on DynaCU-Bench and War Thunder.

The revised results distinguish the roles of controlled diagnosis and practical evaluation. ChronotopeGym establishes where and why temporal failures occur. DynaCU-Bench and War Thunder demonstrate that maintaining interaction history and controlling pending actions can also benefit multi-step and continuously changing applications. These experiments are incorporated into Section 5.3 and the additional ablations in Appendix D.

AuthorsOpenReview ↗

Official Response to Reviewer BHvT (Part 3 of 7)

3. (W3) "No comparison to the closest methods. AOI (continuous observation), AAPT (precompiled guarded policies) and MobileDreamer (state prediction) all target the same failures and appear in Related Work. None is run. The only baselines are the authors' frozen coordinate and restart controllers."

We thank the reviewer for highlighting the importance of empirical comparisons with AOI, AAPT, and MobileDreamer. As discussed in Section 2 and Appendix A.3, these approaches address different aspects of dynamic GUI interaction through continuous observation, anticipatory policy execution, and learned state prediction, respectively. We provide additional method-level comparisons with Agent S3, AOI, and AAPT across ChronotopeGym, DynaCU-Bench Standard, and the new War Thunder live task. These evaluations compare approaches with distinct mechanisms across controlled dynamic tasks, multi-step GUI interactions, and a continuously changing application.

ControllerChronotopeGym SRDynaCU-Bench Standard SRWar Thunder SR
Full Chronotope75.6%67.5%10.9%
Agent S351.0%62.3%0%
AOI61.1%55.0%0%
AAPT45.6%24.2%1.1%

Agent S3 provides a strong conventional baseline through reflection and chronological interaction history. Its 62.3% success on standard DynaCU-Bench demonstrates that conventional agent architectures remain effective on multi-step GUI tasks. However, its success decreases to 51.0% on ChronotopeGym and 0% on War Thunder, compared with 75.6% and 10.9% for Chronotope. These results suggest that conventional reflection alone does not address all the temporal challenges encountered in continuously changing environments.

AOI introduces a model-agnostic observation interface layered on top of an existing GUI agent, rather than an integrated action controller. While this improves temporal awareness, the additional interface layer introduces processing and coordination overhead that can increase per-step latency, particularly under tight response windows. AOI achieves 61.1%, 55.0%, and 0% success across the three benchmarks, compared with 75.6%, 67.5%, and 10.9% for Chronotope. In contrast, Chronotope integrates observation handling, interaction history, and KEEP/UPDATE/REDECIDE meta-control into a latency-sensitive execution path, allowing pending actions to be revised without introducing a separate observation-management layer.

AAPT anticipates future events by preparing conditional action policies that can be executed when corresponding guard conditions are satisfied. This approach can reduce response latency when plausible events can be anticipated and policies prepared ahead of time. However, continuously changing GUI tasks do not necessarily provide sufficient preparation windows. When model inference already occupies the accelerator, policy preparation competes for the same computational resources rather than occurring without additional cost. AAPT achieves 45.6%, 24.2%, and 1.1% success, respectively. Chronotope instead preserves or revises pending decisions as new observations arrive, without requiring a precompiled policy tree.

MobileDreamer introduces a world-model-based lookahead framework that augments an existing GUI agent rather than replacing its controller. Given candidate actions proposed by the agent, its Textual Sketch World Model (TSWM) predicts their resulting GUI states and constructs multi-step prediction trees to inform action selection. This is complementary to Chronotope's focus on reconciling pending decisions with actual observations before commitment. However, MobileDreamer's world model is trained on Android GUI transitions represented as structured textual sketches and cannot be assumed to generalize to ChronotopeGym, DynaCU-Bench, and War Thunder. A meaningful comparison would require constructing suitable structured state representations, collecting environment-specific action-transition trajectories, and training or substantially fine-tuning a new world model to predict the dynamics of these environments. This would constitute a separate model-development and training effort rather than a direct reproduction of the existing method. We therefore discuss MobileDreamer as a complementary predictive approach but leave its numerical results blank rather than presenting an unsupported cross-domain comparison.

Reviewer_uUnZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper analyzes cua timing failures through environment changes, observation availability, decision commitment, and action receipt. ChronotopeGym controls these events in synthetic tasks, while Chronotope updates pending actions using newer observations and places parts of the control path on the GPU. Matched experiments show that refreshing an action argument can recover success without changing the model call or response time. I see a useful diagnostic demonstration, but the evidence does not establish the additional value of the full control architecture or its claimed end-to-end execution benefit.

Critical strengths
  1. The authors keep the model call, commitment time, and receipt time fixed while changing the observation used to locate the target. This supports the conclusion that better outcomes can come from fresher evidence rather than faster inference.
    1. The UI-TARS 1.5 result is informative: updated clicks recover moving-target successes against stationary targets, rather than correcting the model's original grounding errors indiscriminately.
  2. I appreciate the static-target and irrelevant-motion checks, the cases where change occurs after the action arrives, and the sensitivity analysis for restarting after movement.
Critical weaknesses
  1. The full controller's research contribution is not established. IMO, Chronotope is presented as a system that maintains event dependencies and decides whether to preserve, revise, delay, or recompute an unfinished action. Yet the strongest recovery results come from updating a specified coordinate with an image-template matcher or rereading a code with a fixed-font decoder. The comparisons mainly use a frozen action, buffering for the next decision, or restarting the full VLM. They do not establish what the event-history representation and meta-control add over keeping the original operation and applying the same cheap updater immediately before execution. The supplementary pending-action experiments show that remembering an unfinished effect can help, but do not isolate the need for the proposed event anchoring. IMO, a matched comparison with a simple latest-frame updater and ordinary pending-action state is needed to establish the architectural contribution.
  2. The GPU comparison does not establish the claimed benefit with a real model workload. The paper highlights an improvement from 9→35 SR when moving late binding from the host to the accelerator. The experiment producing those numbers uses a color detector followed by an artificial wait, rather than model inference. The two binders differ by about 1 ms, and the tested high-speed condition sits near the hit boundary; at a lower tested speed both succeed in every trial. This is evidence that small delays can matter in that constructed condition, but not that the full GPU-resident pipeline materially improves realistic agent execution. Real in-process planners are included in a separate small pilot, but that does not supply the missing matched host/GPU comparison with the same model, updater, and task workload.
  3. The timing used for eval excludes a delay. the paper states in the same accelerator study that the device's actual commitment occurs roughly one second after the commitment time recorded by the harness. Frames due during the artificial wait are rendered afterwards, and this work is excluded from the harness's controller time. This distinction is substantial when outcomes can change with approximately 1ms of additional delay?
    1. A software clock can support controlled timing interventions, but success scored on that clock does not directly establish timely execution along the real path. The paper needs a consistent account of frame availability, rendering, controller work, and action receipt, with all incurred delays charged when claiming an end-to-end systems benefit.
  4. The eval leaves the practical scope of the update mechanism largely untested. The recovery tasks use simple moving targets, a known-font four-digit code, and a prescribed button displacement. These are useful probes, but they mainly test whether a task-specific visual routine can refresh a predefined field. They provide little evidence about choosing the relevant update when appearance, target identity, or the required operation changes in a less structured interface.
Questions for authors
  1. What benefit remains when the full controller is compared with the same latest-frame updater and a conventional pending-action, observations, and execution budget fixed?
  2. Which reported recovery results use live applications rather than prerendered states, and how robust are the task-specific updaters to changes in appearance, identity, or required operation?
  3. Can the authors identify the exact controller and execution configuration behind each main intervention?
6 comments
AuthorsOpenReview ↗

Official Response to Reviewer uUnZ (Part 1 of 4)

Dear Reviewer uUnZ,

Thank you for your careful reading and constructive suggestions. We respond to your concerns and questions below.

1. (W1 & Q1) "The full controller's research contribution is not established... ; What benefit remains when the full controller is compared with the same latest-frame updater and a conventional pending-action, observations, and execution budget fixed?"

We thank the reviewer for this constructive suggestion. Chronotope introduces a clock-decoupled architecture integrating three components. First, EODA/EDM represents dependencies among environment changes, observations, decisions, and action receipts, providing structured interaction history for subsequent decisions. Second, decision meta-control selectively keeps valid actions, updates outdated arguments, or redecides when the intended operation must change. Third, accelerator-resident execution reduces the overhead of interaction-state maintenance and decision reconciliation before commitment.

To address Q1, we add a matched 2×4 factorial ablation that varies EODA history capacity (N=0N=0 or N=8N=8) and decision policy (Always KEEP, Always REDECIDE, Always UPDATE, or selective meta-control). All configurations use the same Qwen3.8-27B planner, observation schedule, argument updater, task instances, hardware, and execution-budget limits. Accelerator-resident execution is held fixed. At N=0N=0, the controller retains the latest observation and conventional pending-action state without historical observations. Always UPDATE at N=0N=0 therefore provides the requested simple latest-frame updater baseline, preserving the original operation while updating its arguments before execution. Table R1a. Matched factorial ablation of EODA history and decision meta-control.

Meta-control policyChronotopeGym SRDynaCU-Bench SRWar Thunder SR
N = 0 (No EODA history)
Always KEEP54.0%25.1%5.3%
Always REDECIDE48.5%28.0%4.7%
Always UPDATE68.0%28.1%7.1%
Selective KEEP/REDECIDE/UPDATE72.5%28.1%7.9%
N = 8 (EODA history)
Always KEEP57.0%64.1%7.6%
Always REDECIDE51.4%67.5%6.8%
Always UPDATE72.3%67.3%10.4%
Selective KEEP/REDECIDE/UPDATE75.6%67.5%10.9%

Full controller versus latest-frame updating. Relative to Always UPDATE at N=0N=0, full Chronotope (N=8N=8, selective meta-control) improves success by 11.2%, 140.2%, and 53.5% on ChronotopeGym, DynaCU-Bench, and War Thunder, respectively. These gains combine retaining history, representing dependencies, and selective meta-control. We have revised our manuscript to include this matched comparison in Section 5.3 (Table 1).

Contribution of interaction history. With Always UPDATE fixed, increasing EODA history from N=0N=0 to N=8N=8 improves success by 6.3%, 139.5%, and 46.5%, respectively. With selective meta-control fixed, the gains are 4.3%, 140.2%, and 38.0%. History benefits all four policies, particularly on DynaCU-Bench, where multi-step tasks depend on earlier observations and action outcomes. Conventional chronological text history alone, with selective meta-control fixed, improves success over no history by 2.2%, 138.4%, and 17.7%, showing that retaining past interactions accounts for much of the improvement. We have revised our manuscript to include these history ablations in Appendix D.1 (Tables 3 and 4).

Contribution of dependency-structured representation. Holding history capacity (N=8N=8) and selective meta-control fixed, EODA/EDM improves success over conventional chronological text history by an additional 2.0%, 0.7%, and 17.2%, respectively, while reducing token consumption by 45.2% and median latency by 3.6%. Thus, conventional history provides substantial benefits, while explicitly representing event dependencies yields further improvements in success and efficiency.

Contribution of selective meta-control. Holding EODA history at N=8N=8, selective meta-control improves success over Always UPDATE by 4.6%, 0.3%, and 4.8%, respectively. Relative to Always KEEP, the improvements are 32.6%, 5.3%, and 43.4%. The smaller incremental gain over Always UPDATE on DynaCU-Bench reflects its relaxed deadlines, where unconditional updating or re-decision remains competitive. ChronotopeGym and War Thunder benefit more from distinguishing changes that require preserving, revising, or replacing pending actions.

Together, these results distinguish the benefits of retaining interaction history, structuring its dependencies, and selectively adapting pending actions. Interaction history provides temporal context, EODA represents its dependencies efficiently, and selective meta-control determines how to respond to changing conditions. We evaluate accelerator-resident execution separately in Response 2.

AuthorsOpenReview ↗

Official Response to Reviewer uUnZ (Part 2 of 4)

2. (W2) “The GPU comparison does not establish the claimed benefit with a real model workload…Real in-process planners are included in a separate small pilot, but that does not supply the missing matched host/GPU comparison with the same model, updater, and task workload.”

We thank the reviewer for this constructive comment and the opportunity to clarify the distinction between our controlled execution study and full-system performance. The original Table 9 was narrowly designed to isolate the effect of the late-binding updater's device placement within Chronotope under exceptionally tight deadlines. To establish the performance benefit of the complete controller, we have added two complementary evaluations to distinguish the contribution of accelerator residency from full-controller improvements.

Matched CPU/GPU Chronotope comparison. To isolate accelerator residency, we compare CPU- and GPU-resident Chronotope controllers using identical VLMs, EODA representations, meta-control policies, argument updaters, observations, and execution settings. VLM inference remains on the GPU in both configurations. As shown in Table R1d, accelerator residency reduces total control-path latency from 67.2 ms to 19.9 ms (70.39%), including a 66.1% reduction in EODA maintenance and elimination of measured host-device transfer and synchronization overhead within the control path. This breakdown excludes VLM inference and action delivery, isolating the contribution of accelerator-resident controller execution. We have revised our manuscript to include this analysis in Appendix D.2 and Table 5. The corresponding end-to-end comparisons in Tables R1b–R1c show that accelerator residency reduces median latency by 2.4% for Qwen3.8-27B and 13.9% for Qwen3.5-0.8B. Under a 2 s deadline, it further improves the 27B model's Live task success by 8.0%.

Table R1d. Matched component-level latency breakdown.

Controller componentChronotope CPU (ms)Chronotope GPU (ms)
EODA/EDM maintenance29.810.1
Decision meta-control11.09.8
Transfers and synchronization26.40
Total control-path cost67.219.9

Table R1b. Performance comparison with Qwen3.8-27B.

ControllerMedian E→A (ms)SR 16sSR 8sSR 4sSR 2sSR 1s
Agent S312,84088%0%0%0%0%
Chronotope CPU1,910100%100%97%75%0%
Chronotope GPU1,864100%100%98%81%0%

Table R1c. Performance comparison with Qwen3.5-0.8B.

ControllerMedian E→A (ms)SR 16sSR 8sSR 4sSR 2sSR 1s
Agent S33,39199%96%96%0%0%
Chronotope CPU31099%97%97%95%90%
Chronotope GPU26799%97%97%96%93%

Full-controller performance envelope. Separately, we evaluate Chronotope against the host-orchestrated Agent S3 controller across varying response deadlines. As shown in Tables R1b–R1c, Chronotope achieves 85.5% lower median end-to-end latency in the 27B comparison and ** 92.1%** in the 0.8B comparison. With Qwen3.8-27B, Chronotope maintains 100% Live task success at 8 s and 98% at 4 s, whereas Agent S3 achieves 0% at both deadlines. With Qwen3.5-0.8B, Chronotope maintains 96% success at 2 s, compared with 0% for Agent S3. We have revised our manuscript to characterize how Chronotope expands the performance envelope of GUI agents under tight deadlines, in Section 5.3 and Figure 7.

AuthorsOpenReview ↗

Official Response to Reviewer uUnZ (Part 3 of 4)

3. (W3 & Q3) “The timing used for eval excludes a delay… Can the authors identify the exact controller and execution configuration behind each main intervention?”

We thank the reviewer for highlighting the need to clarify timing and agent configurations used for our studies. While our main studies measure end-to-end latency with VLM-driven GUI agents, the former Table 9 narrowly isolates host-versus-accelerator late-binding overhead, where we replaced a typical VLM with a domain-specific algorithm and fixed deliberation interval to eliminate inference-time variability, and defer rendering during this interval to avoid potential interference. Before commitment, both configurations process the deferred frames, ensuring equally updated observations. This rendering takes approximately one second, which harness timing excludes because it is unrelated to late-binding computation, while device timing includes it to reflect actual commitment. Identical deliberation, rendering, observation, and timing conditions ensure a fair comparison of late-binding placement, independent of full-system overhead. The revised manuscript distinguishes controlled temporal interventions, full-controller comparisons, and matched component-level ablations. We identify the exact controller and execution configuration behind each evaluation below:

Table R3. Controller and execution configurations across the revised evaluation.

ExperimentController / interventionExecution configuration
5.2 Claim 1: Paused / Replay / LiveAgent S3 controller, VLM planner under three temporal conditionsFive ChronotopeGym tasks. Paused freezes task progression, Replay provides changing observations while preserving response validity, and Live evaluates action validity at receipt
5.2 Claim 1: Controlled timing experimentsAgent S3 controller, VLM planner using newer observations or changing when actions arriveOn ChronotopeGym, Qwen3.8-27B-BF16 uses a newer screenshot to update click coordinates without changing response time.
5.3 Claim 2: History and meta-controlFixed KEEP, REDECIDE, UPDATE, versus selective meta-control, with N=0N=0 or N=8N=8 EODA historyMatched Qwen3.8-27B planner across ChronotopeGym, DynaCU-Bench, and live War Thunder, varying history availability and meta-control policy
5.4 Claim 3: Performance envelopeHost-orchestrated Agent S3 versus accelerator-resident ChronotopeVLM-based controller comparisons using Qwen3.8-27B and Qwen3.5-0.8B across response deadlines, measuring Live success and end-to-end latency
Appendix D.1 Claim 4: History representationNo history versus chronological text versus EODA/EDMQwen3.8-27B with selective meta-control fixed. Chronological text and EODA/EDM use N=8N=8, evaluated across ChronotopeGym, DynaCU-Bench, and War Thunder
Appendix D. 1 Claim 4: History capacityChronological text versus EODAacross N∈{1,2,4,8,16}N\in\{1,2,4,8,16\}, with N=0N=0 as referenceSame Qwen3.8-27B planner and selective meta-control, measuring task success, token consumption, and latency
Appendix D.2 Accelerator residencyCPU-resident versus GPU-resident ChronotopeIdentical VLMs, EODA/EDM, meta-control, updaters, observations, and execution settings. VLM inference remains on the GPU in both. Control-path timing excludes inference and action delivery

Comments on the paper

7 comments
AuthorsOpenReview ↗

General Response

Dear AC and Reviewers,

We thank the reviewers for their careful reading and constructive comments. We have made the following updates in response to your feedback:

  • Added comparisons of KEEP, UPDATE, REDECIDE, and selective control.
  • Added an evaluation on a 16-task subset of DynaCU-Bench with history and policy ablations.
  • Evaluated general-purpose grounders, a latest-frame updater, and adapted temporal-control baselines.
  • Expanded the performance-envelope evaluation to two planner models.
  • Added matched CPU/GPU placement and updater-cost measurements.
  • Added history-capacity experiments measuring success, token consumption, and step latency.
  • Expanded related work on classical planning and control.
  • Clarified implementation differences, evaluation protocols, and timing metrics.
  • Narrowed the claims to match the measured evidence.
  • Revised figures, captions, cross-references, and experimental discussions.
  • Corrected typos, standardized citation formatting, and revised explanations for clarity and conciseness.

Thank you again for the helpful feedback. We look forward to engaging in discussions with the reviewers.

AuthorsOpenReview ↗

Follow-up Request for Reviewer Engagement

Dear Area Chair,

Thank you again for your time and effort in overseeing the review process for our submission.

We are writing to respectfully follow up on our previous request for reviewer engagement. Since our earlier message, we have sent another round of individual reminders to the reviewers, expressing our willingness to clarify any remaining concerns. However, we have still not received any follow-up responses from reviewers uUnZ, BHvT, and J6TL. Reviewer ErHS initially engaged in the discussion but has not responded further to our subsequent clarifications.

Throughout the rebuttal period, we have made substantial efforts to address the reviewers' concerns through extensive additional experiments, detailed analyses, and point-by-point responses. We believe these new results directly address the major concerns raised in the original reviews. However, without further reviewer engagement, we have had limited opportunity to discuss whether any concerns remain unresolved or whether our additional evidence has adequately addressed them.

As the discussion period is now drawing to a close, we would be deeply grateful for any assistance in encouraging the reviewers to engage with our responses. We also sincerely hope that the substantial additional experimental evidence and clarifications provided during the rebuttal period can be carefully considered in the final evaluation, particularly if further reviewer discussion is not possible.

Thank you very much for your continued time, attention, and consideration. We sincerely appreciate your efforts in ensuring a thorough and fair review process.

Best regards,

The Authors

AuthorsOpenReview ↗

Official Response to Further Response from uUnZ (Part 1 of 5)

Dear Reviewer uUnZ,

We sincerely appreciate your follow-up questions on the specific contribution and the interpretation of the reported gains and scientific contribution. We clarify them below:

1. (W1a) "Most of the improvement on DynaCU is already obtained with conventional history...Please provide the sample sizes and paired uncertainty estimates for these comparisons, and clarify which findings specifically support these components rather than retaining history in general." for EODA

We thank the reviewer for inviting us to clarify the exact contribution of the EODA component inside our Chronotope architecture.

Experimental isolation. We explicitly isolate the contribution of EODA/EDM's compactly represented interaction history beyond retaining textual interaction history itself by comparing it against conventional chronological text history at matched history capacities. We hold the planner, meta-control policy, updater, and execution conditions fixed, and repeat the comparison across varying history capacities to distinguish representational benefits from those of retaining more information. We observe up to 20.2% additional success-rate benefit of structured dependencies in latency-sensitive tasks. Each improvement is based on 2,000 paired experiments, using Wald 95% CI and two-sided McNemar test.

Table R1-5. EODA/EDM vs. Chronological Text History Across Tasks and History Capacities

NChronotopeGym SR95% CI (pp) / pDynaCU-Bench SR95% CI (pp) / pWar Thunder SR95% CI (pp) / p
174.5% → 75.0% (+0.7%)[+0.08, +0.92] / 0.030946.0% → 46.0% (0.0%)[−0.88, +0.88] / 1.0008.8% → 9.0% (+2.3%)[+0.004, +0.396] / 0.1250
275.3% → 76.2% (+1.2%)[+0.13, +1.67] / 0.030054.3% → 54.5% (+0.4%)[−0.04, +0.44] / 0.21889.2% → 9.7% (+5.4%)[+0.16, +0.84] / 0.00635
475.0% → 76.0% (+1.3%)[+0.15, +1.85] / 0.028661.9% → 62.2% (+0.5%)[+0.06, +0.54] / 0.031259.4% → 10.4% (+10.6%)[+0.27, +1.73] / 0.01045
874.1% → 75.6% (+2.0%)[+0.19, +2.81] / 0.030467.0% → 67.5% (+0.7%)[+0.16, +0.84] / 0.006359.3% → 10.9% (+17.2%)[+0.48, +2.72] / 0.00675
1672.4% → 75.3% (+4.0%)[+0.39, +5.41] / 0.026268.6% → 69.0% (+0.6%)[+0.09, +0.71] / 0.021488.9% → 10.7% (+20.2%)[+0.58, +3.02] / 0.00491

Interpretation. These results reveal two distinct benefits of EODA/EDM. First, EODA/EDM provides a more efficient representation of interaction history, with its token and latency advantages increasing as history capacity grows. Second, EODA/EDM success-rate benefits depend on the environment's temporal demands. As our paper discusses in Appendix A.3, existing timing-aware GUI agent benchmarks like DynaCU-Bench do not rigorously isolate latency-induced failures. DynaCU-Bench targets discrete multi-step (up to 15 steps) and transient observations, but permits extraordinarily large latencies up to 240 seconds. Unsurprisingly, most slower agents can succeed as long as it is provided with a short history such as Chronological Text and enough observations, leaving limited additional benefit from explicit dependency structuring, despite Chronological Text's outsized impact on latency. In contrast, a non-synthetic, latency-sensitive task such as War Thunder involves continuously evolving states and causally coupled interactions over continuous, potentially infinite interaction horizons where subsequent decisions depend on previous observations, actions and their observed outcomes. EODA/EDM's larger improvements on War Thunder are consistent with explicit temporal dependencies helping the planner recover and use this information at a much lower impact on latency. The history-capacity results further suggest that EODA/EDM better preserves task performance as longer histories introduce additional required context.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.