acceptodds
Under review as a conference paper at ICLR 2027

When Motion Logs Go Missing: Selective Visual Recovery for Persistent GUI State Tracking

Abstract

Persistent GUI state trackers propagate spatial memory using both rendered screens and recorded actions, implicitly assuming that the action stream is complete. When a motion record is explicitly missing, the tracker applies an incorrect zero update and the resulting coordinate error can persist across subsequent observations. We study this failure mode and formulate recovery as selective substitution: observed actions remain unchanged, while only known motion erasures may be replaced using visual evidence. Our method estimates translation-like motion from consecutive screenshots with linear phase correlation and substitutes the estimate only on missing-action transitions, preserving exact behavior on complete logs. Across five independently trained checkpoints, visual recovery reduces hidden-state and recovery-state error on three frozen stress sets, with all six predeclared comparisons remaining significant after Holm correction. The benefit grows with missing-action severity and remains positive under fixed-budget bursts of one to six consecutive erasures. On held-out GUI-Odyssey trajectories with artificially erased motion logs, linear phase recovery reduces cumulative trajectory-log error from to and endpoint error from to at 20% erasure, with positive improvements from 5% through 40% erasure. Under the same trajectory protocol, RAFT-Small provides smaller gains, while a missing-only true-action oracle reveals substantial remaining headroom. On Android-in-the-Wild, linear phase yields the largest gesture-referenced displacement-error reduction among the evaluated estimators. These results show that repairing missing motion inputs can improve persistent GUI state without altering valid action logs. The method targets explicit translation-like motion erasures and does not address arbitrary nonzero action errors or end-to-end GUI-agent control.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI agents, persistent state tracking, missing action recovery, visual motion estimation, phase correlation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_vkSQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses phase correlation between consecutive screenshots to estimate motion when a GUI action log has a missing entry. Recorded actions are left unchanged. It reports lower tracking errors on synthetic sequences across five checkpoints, better trajectory-log reconstruction on GUI-Odyssey, and lower gesture-referenced displacement error on AITW.

Critical strengths
  1. The method changes only entries marked as missing. When no entry is marked missing, it follows exactly the same updates as the baseline. This is useful for repairing logs without changing recorded actions.
  2. The comparisons use the same checkpoints and missing-action masks. The burst tests keep the number of erased actions fixed, and the oracle replaces only missing actions. These controls make the comparisons easier to interpret. The authors also clearly state the limits of screening and the real-data metrics.
Critical weaknesses
  1. The new part of the method seems limited: it uses standard phase correlation and inserts the estimate only when a record is missing. The paper does not show how often this failure occurs in practice or explain how valid zero-motion records are kept separate from missing ones in Equation (1). Figure 3 and Appendix E measure errors against gesture logs. They do not directly show whether target positions are more accurate on real GUIs.
  2. The description is missing details needed to reproduce the experiments. Appendix A does not describe the trained frontend, how visual observations update the memory, the training settings, or data generation. Appendix B does not give exact error formulas, averaging rules, bootstrap resampling details, or Holm-adjusted p-values. Appendix E also omits crop and image-size settings, how RAFT flow is reduced to a displacement, and parameters for the registration baselines. These gaps make the reported results difficult to check.
Questions for authors
  1. Can you provide the tracker and data-generation settings, exact error formulas and recovery window, and sequence-level results for each checkpoint, together with the bootstrap and Holm calculations?
  2. How do you distinguish missing records from valid zero-motion records? Please give the image preprocessing and baseline settings, and explain what evidence links lower gesture-log error to more accurate viewport or target tracking.
Reviewer_JaASOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work studied a persistent-state failure caused by explicitly missing GUI motion logs. Rather than modifying observed actions, we formulate recovery as selective substitution: consecutive screenshots provide a visual motion estimate only when the corresponding translation record is known to be missing. Observed actions remain authoritative, yielding exact clean-log invariance without additional learned recovery parameters.

Critical strengths
  • The authors provide a detailed experimental analysis in the appendix; however, why was this not included in the main text, given that the current main text has not yet reached the length limit for ICLR 2027?
Critical weaknesses
  • Poor writing and bad organization. During the review process, I have had to invest significant time and effort to understand the authors' motivations and the problems they address; therefore, I do not believe this manuscript is currently suitable for publication at ICLR 2027.

  • The organization and comparison of the experiments in the paper are also confusing.

  • The section on "Image registration and visual motion estimation" in the "Related Work" chapter cites many early papers, which do not appear to be highly relevant to the research presented in this paper.

Questions for authors
  • Please refer to the weaknesses.
Reviewer_z6rfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies persistent GUI state tracking when some translation records are known to be missing. The method estimates displacement between consecutive screenshots using phase correlation and substitutes the estimate only for missing records before updating the stored target coordinates. The authors report reduced hidden- and recovery-state errors on a controlled benchmark, together with improvements in trajectory-log reconstruction and gesture-referenced displacement estimation on GUI-Odyssey and AITW, respectively.

Critical strengths

Controlled experimental evidence: The experiments compare recovery and zero update using the same checkpoints, sequences and missing-action masks, which helps isolate the effect of replacing the missing input. The reported reductions in hidden- and recovery-state error support visual compensation under the evaluated conditions.

Critical weaknesses
  1. Unclear problem motivation: The paper assumes that missing motion records are identifiable while adjacent screenshots remain available. However, it does not provide a concrete real-world example explaining how such records become unavailable or how substantially this affects tracking. Artificially erasing records demonstrates the effect of recovery under the assumption, but does not establish the practical importance of the problem.

  2. Limited methodological contribution and incomplete validation: The method mainly combines established phase correlation with selective substitution, making the evidence for its tracking benefit particularly important. However, the real-data metrics measure agreement with gesture logs rather than target-coordinate accuracy. The controlled benchmark does evaluate target state, but the tracker and how visual observations are combined with coordinate memory are not sufficiently specified. These limitations make the main results difficult to reproduce and leave the benefit for persistent target tracking on real GUI data unclear.

Questions for authors

What real-world GUI failure motivates this missing-log setting, and what evidence shows that it materially affects persistent tracking?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.