acceptodds
Under review as a conference paper at ICLR 2027

Keeping Recorded GUI Workflows Reliable When Interfaces Change

Abstract

Conventional GUI record and replay is fast and cheap, but it breaks under interface reflow, restyling, distractors, and renamed controls. Vision-language computer-use agents adapt to such changes, yet re-planning an already demonstrated workflow on every run adds latency, cost, and error. We study the middle ground: given one successful demonstration, keep that workflow reliable under interface perturbations without generating a new plan. Our method preserves the recorded action sequence and re-ground each target from pixels on the live interface. Our resolver ranks OCR, template, and geometry candidates with 82 scene-relative features and uses calibrated confidence to blend learned evidence with the Hybrid prior and to trigger bounded recovery. We trained on a synthetic corpus of 38 designed workflows rendered under identity with 9 perturbation classes and evaluated on three benchmarks covering 464 replayable workflows from MiniWoB++, WebArena/WONDERBREAD, and WebShop under identity and six task-preserving perturbation classes, yielding 2,670 admitted task–perturbation cells and 16,020 method episodes. Averaged over the six perturbation classes, our method obtains 88.4% MiniWoB++ success versus 67.2% for Hybrid, and 83.9% WebArena success versus 55.2%. On WebShop it reaches 97.6 reward points against 96.3. Code is available at: https://anonymous.4open.science/r/ANONYMIZED.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reliable AI, Efficient AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rZVmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors proposed a novel GUI agents' trajectory generation method, which preserve the action sequence and re-ground the targets from pixels on the live interface. They found training on a few workflows could achieve a high success rate on benchmarks such as MiniWoB++ and WebArena.

Critical strengths
  1. The overall logic is clear and the problem is important and sound.
  2. The performance seems strong based on the experiments.
Critical weaknesses
  1. The writing needed to be improved.

The sentence in Figure 1 has grammatical errors “ A computer-use agent completes the task with expensive and delay”.

“A agent treats both as unknown” L060 -> “An agent …”

“Procedure failures are a planning problem and belong upstream” -> “belong to the upstream”

The current citation style in l065-073 is wierd, it should be like (Leotta et al. 2016, Choudhary et al. 2011, Leotta et al. 2015, Nass et al. 2023) instead of the current form.

“gaming where user does not have access for DOM become challenge for above solutions” -> “become a challenge” / “become challenging”

These sentences have grammatical errors “However, different task with different GUI will not depend on same setting of hard-coded design. Implement heavy weight on text (OCR) over GUI without text will under perform the performance while same setting could performed well under rich text screenshot”.

I highly recommend the authors to go through the paper and proofread the sentences. The current paper contains a lot of grammatical errors and hinder reading. A few more: L151 “They do not measures end-to-end task success on public benchmarks …”

This sentence is difficult to understand “The conclusion is a boundary on the claim rather than a caveat about it”

  1. The authors need justification for their methodological choices.

How the number 9 is determined in L184-185, and similarly, how intersection-over-union of 0.5, and how centers lies within 12 pixels are determined in L217-219.

Similarly, how the exact coefficients was determined in L243-248. The underlying rationale is clear, but how the exact number is determined is less disclosed.

Why the removal of webshop font size tasks are fair? This needs more explanation.

The authors need to specify what the error bar represents in Figure 3.

I think the p value in Table 2 should be compared to the strongest baseline for each task, rather than compared in a fixed manner with hybrid. A more formal way should be to detect the main effects of techniques, and then perform post-hoc comparisons with appropriate adjustments.

I think the authors could better explain “the frozen resolver transfers to unseen websites, unseen layouts, and unseen tasks within a represented perturbation vocabulary. It does not learn a universally perturbation-invariant notion of a target”, especially around how the current results could support these implications.

Questions for authors
  1. How they determined the methodological choices for their methods and experiments?
Reviewer_bo48OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies replaying a successful GUI demonstration after interface changes while preserving the recorded action sequence. It trains a candidate ranker using OCR, template, geometry, and contextual features, combined with confidence-based blending and bounded recovery. Evaluation on 464 admitted workflows from MiniWoB++, WebArena/WONDERBREAD, and WebShop shows substantial gains over fixed matching rules on the first two benchmarks, with no statistically significant improvement over Text or Hybrid on WebShop.

Critical strengths

S1. The paper addresses a useful automation problem with a focused evaluation. Separating a known procedure from changing target locations is a practical way to study repeated GUI workflows. Shared demonstrations, task-preserving perturbations, and native end-to-end evaluators make the evaluation relevant to successful execution.

S2. The gains transfer beyond the synthetic training workflows. Training is confined to 38 designed workflows, yet the frozen resolver substantially improves over Hybrid on MiniWoB++ and WebArena (Table 10). The perturbation holdouts also provide useful evidence about where this transfer breaks down.

Critical weaknesses

W1. The comparisons do not isolate the contribution of the proposed design. Hybrid uses hand-set weights, while Learned adds a richer feature representation, fitted ranking, confidence blending, and bounded recovery. Without a training-tuned Hybrid baseline and component ablations, it remains unclear how much improvement comes from adaptive ranking versus additional evidence or recovery behavior.

W2. The practical efficiency advantage is not established. The paper reports analytic FLOP estimates rather than measured execution time, despite efficiency being a central motivation. A modern grounding model could also follow the supplied action sequence, so a comparison under this same contract is needed to establish the trade-off between robustness and runtime.

Questions for authors

Q1: Validation of the proposed components. How does Hybrid perform when its weights are tuned on the same training corpus, and how much improvement remains when contextual/relative features, confidence blending, and bounded recovery are removed separately?

Q2: Validation of the efficiency claim. Can you compare with a representative visual grounder, such as GroundNext[1] or UGround[2], while keeping the recorded action sequence and available reference information matched? Reporting native task success and measured end-to-end latency would clarify the practical advantage.

[1] Grounding Computer Use Agents on Human Demonstrations. ICLR 2026. [2] Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents. ICLR 2025.

Reviewer_PRzVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reliable GUI workflow replay under interface changes. Given a successful demonstration, the method preserves the recorded action sequence and relocates each target using the current screenshot.

The proposed resolver combines OCR, template matching, and relative geometry through an 82-feature gradient-boosted ranker. It also uses confidence-based blending with a fixed Hybrid baseline and bounded recovery operations.

The method is trained on 38 designed workflows and evaluated on 464 workflows from MiniWoB++, WebArena, and WebShop under controlled interface perturbations. It achieves 88.4% success on MiniWoB++ and 83.9% on WebArena, compared with 67.2% and 55.2% for Hybrid. The improvement on WebShop is small.

Critical strengths

S1. Practical and well-defined problem. Recorded workflows often fail when interface elements move or change appearance. The paper clearly defines the target relocation problem and develops a lightweight method that preserves the original action sequence. S2. Strong experimental results. The method substantially outperforms Hybrid on MiniWoB++ and WebArena. The evaluation covers multiple perturbation types, different workflow lengths, and held-out perturbation classes. The paired statistical analysis supports the reported improvements.

Critical weaknesses

W1. Insufficient comparison with learned baselines. The main baselines use individual matching signals or fixed combinations of text, template, and geometry. The proposed method instead uses a trained 82-feature ranker. The Text baseline is implemented using OCR label similarity, although the cited work by Coppola et al. (2025) studies machine learning and learning-to-rank for element localization. The paper also reports only 40.8% success for Text on unperturbed WebArena workflows, compared with 98.9% for Exact replay, without analyzing the causes of these failures. The experiments establish an advantage over fixed matching rules, but do not isolate the benefit of the proposed feature design from that of supervised learning. A simple learned pixel-based baseline using the same training data and matching signals is needed.

W2. Missing ablations of the core method components. The method combines scene-relative features, gradient-boosted ranking, confidence calibration, Hybrid blending, and bounded recovery. The existing perturbation-transfer and OCR-versus-DOM experiments do not isolate these components. The paper does not report how performance changes when the scene-relative features, confidence-based blending, or recovery mechanism are removed. This leaves the source of the reported gains insufficiently established.

W3. Limited evidence for runtime efficiency. The paper emphasizes fast, low-cost replay, including the latency claim in Figure 1. However, the efficiency analysis relies mainly on FLOP estimates, without measured execution times. The evaluation also focuses on controlled web-interface perturbations, leaving performance across real application revisions untested.

Questions for authors

Q1. Why does the Text baseline achieve only 40.8% success on unperturbed WebArena workflows? Please analyze its main failure causes and compare the proposed method with a simple learned pixel-based baseline trained on the same data and basic matching signals.

Q2. What are the individual contributions of scene-relative features, confidence-based blending, and bounded recovery? Please report component ablations, including a comparison between a simple learned ranker and the full 82-feature model.

Q3. Figure 1 suggests low replay latency, but the paper mainly reports FLOP estimates. What are the measured per-step or end-to-end execution times of Hybrid and the proposed method under identical hardware conditions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.