acceptodds
Under review as a conference paper at ICLR 2027

VersionShift: Evaluating and Adapting Prior Experience Across Software Versions for GUI Agents

Abstract

GUI agents interact with graphical user interfaces (GUIs) to perform tasks in real-world software environments, yet these interfaces continuously evolve through software updates, changing layouts, element locations, and task workflows. Such evolution makes **cross-version experience adaptation** essential: prior experience may remain useful after an update, but can also become outdated or misleading. However, selective adaptation of prior experience across software versions remains underexplored, and existing benchmarks provide limited support for controlled evaluation of such experience transfer. To systematically study this problem, we introduce **VersionShift**, a benchmark that holds task intent fixed while pairing executions across chronological software versions. Through semantic alignment of paired trajectories, VersionShift characterizes source-experience validity at each target step using **Reuse, Reground, and Replan** annotations, distinguishing directly executable source actions, applicable semantics that require new grounding, and target steps with no applicable source action. VersionShift contains 312 tasks and 1,248 trajectories across 53 applications and websites spanning Mobile, Desktop, and Web. Evaluation on VersionShift shows that historical trajectories can improve action prediction while also introducing substantial negative transfer, with beneficial and harmful effects coexisting within the same model. To retain useful experience while limiting negative transfer, we propose **VAEA**, a validity-aware experience adaptation approach that learns to select applicable source steps and estimate direct-reuse validity. VAEA uses the predicted applicable source steps and reuse-validity score to conditionally reuse source actions, reground selected semantics, or replan when no source step is applicable, without requiring cross-version annotations at inference. Experiments show that VAEA improves action prediction over controlled experience-use baselines and consistently reduces history-induced negative transfer, while remaining competitive with other memory methods. We will release the benchmark and code to facilitate future research.

Primary area:
datasets and benchmarks
Keywords:
GUI Agents, GUI Benchmarks, Experience Adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rQQaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how GUI agents can adapt prior interaction experience across different software versions. It introduces VersionShift, which pairs same-task trajectories across four chronological versions and provides step-level annotations for 312 tasks across 53 applications and websites. The paper further proposes VAEA to selectively reuse or adapt historical experience, achieving improved action prediction and reduced negative transfer under the proposed evaluation protocol.

Critical strengths
  1. The paper introduces a meaningful problem, as prior GUI experience can remain useful after software updates but may also become partially outdated or misleading.

  2. VersionShift provides a relatively comprehensive dataset covering Mobile, Web, and Desktop applications. Its controlled task construction, paired trajectories, and step-level validity annotations form a reasonably systematic evaluation setting.

  3. The paper also proposes VAEA, which achieves measurable improvements in experience adaptation and reduces negative transfer under the current protocol.

Critical weaknesses
  1. This is primarily a dataset paper, but the research problem itself is not particularly new. VersionShift mainly extends current data through paired multi-step trajectories, finer-grained validity annotations, and a more controlled task formulation.

  2. From the perspective of dataset construction, the improvements over existing benchmarks also seem incremental. As reflected in Table 1, the most distinctive differences are the broader platform coverage and the availability of paired trajectories. Meanwhile, the task selection is relatively constrained because the intent, initial state, and success criterion must remain reproducible across all four versions, and the trajectories are generally short, averaging only 3.36 executable actions and 1.93 actions for Web tasks.

  3. The dataset would benefit from stronger quantitative validation of its annotations. All REUSE, REGROUND, and REPLAN labels were produced and verified by one trained annotator, without an inter-annotator agreement study or independent relabeling of a subset.

Questions for authors
  1. What do the authors consider the single most important contribution that clearly separates VersionShift from existing cross-version benchmarks such as TransBench and TimeWarp, and what is the most convincing empirical evidence that this distinction is substantive rather than primarily an incremental improvement in pairing and annotation granularity?
  2. Please refer to the weaknesses.
4 comments
AuthorsOpenReview ↗

Response to Reviewer rQQa: Weakness 1 & Question 1

We sincerely thank the reviewer for the constructive feedback. We address the concerns regarding novelty, task complexity, and annotation reliability below.

W1 & Q1: Distinction from Existing Cross-Version Benchmarks

We agree that cross-version GUI evaluation is not new. However, the key contribution of VersionShift is not merely finer-grained trajectory annotation, but a distinct evaluation problem: determining whether and how prior execution experience remains applicable in a new software version. This requires evaluating the validity of individual historical actions and their adaptation requirements, rather than only measuring performance under interface changes.

TransBench focuses on single-step GUI grounding transfer, while TimeWarp is limited to Web environments and primarily evaluates robustness and generalization across interface versions. In contrast, to our knowledge, VersionShift is the first cross-platform benchmark to explicitly evaluate the step-level validity of prior execution experience across software versions. By pairing multi-step trajectories across Mobile, Web, and Desktop and introducing REUSE, REGROUND, and REPLAN annotations, VersionShift enables fine-grained analysis of experience adaptation and positive/negative transfer, rather than relying solely on aggregate grounding accuracy or task success.

Our mode-wise analysis provides direct empirical evidence of the diagnostic value of VersionShift. As shown in Table 1, negative transfer varies substantially across adaptation modes, with REPLAN exhibiting the highest NTR despite representing less than one-fifth of evaluated steps. VAEA reduces NTR across all three adaptation modes and improves net transfer on REPLAN steps to +4.78 percentage points. These findings show that step-level experience validity provides a meaningful basis for diagnosing negative transfer and evaluating selective experience adaptation, rather than merely increasing annotation granularity.

Table 1. Mode-wise positive and negative transfer on the VersionShift held-out test split (Qwen3-VL-8B). PTR and NTR are measured relative to Target-only, with each adaptation mode's total steps as the denominator.

Adaptation ModeSamplesRaw History PTR ↑Raw History NTR ↓VAEA PTR ↑VAEA NTR ↓
REUSE640 / 1,16415.632.8115.781.41
REGROUND294 / 1,16416.339.8615.654.08
REPLAN230 / 1,16416.5218.2612.177.39
Overall1,164 / 1,16415.987.6515.033.26

The practical significance is further supported by closed-loop evaluation, where VAEA achieves 52.12% task success, compared with 40.61% for Raw History and 46.06% for Supervised Selection + Actor (Table 18 in the manuscript). Together, these findings demonstrate that VersionShift enables systematic diagnosis of distinct experience failure modes and adaptation requirements, establishing step-level experience validity as a meaningful evaluation target across software versions.

AuthorsOpenReview ↗

Response to Reviewer rQQa: Weakness 2 & Weakness 3

W2: Constrained Task Design and Short Trajectories

We sincerely thank the reviewer for highlighting these limitations. We acknowledge that requiring consistent task intents, initial states, and success criteria across four versions constrains task diversity. Nevertheless, this controlled design helps isolate the effects of software evolution on experience validity, preventing differences in task objectives and execution conditions from confounding source–target comparisons.

We also acknowledge the relatively short trajectories (3.36 actions on average and 1.93 for Web). However, short trajectories do not necessarily imply simple cross-version adaptation. On the held-out test split, 57.76% of steps from 1–2-action trajectories require REGROUND (37.93%) or REPLAN (19.83%). Furthermore, Table 2 shows that VAEA improves Step accuracy and reduces negative transfer across all trajectory-length groups, including trajectories with at least six actions. These results indicate that experience adaptation remains relevant across different trajectory lengths, rather than being confined to short workflows.

Table 2. Step accuracy and negative transfer across trajectory lengths on the VersionShift held-out test split (Qwen3-VL-8B). All accuracy and NTR values are percentages (%). Trajectories are grouped by the number of target-version reference actions. NTR is measured relative to Target-only. ΔStep denotes the improvement of VAEA over Raw History.

Trajectory LengthTraj. PairsStepsREGROUND+ REPLAN(%)Raw History Step↑VAEA Step ↑ΔStep (pp) ↑Raw History NTR ↓VAEA NTR ↓
1–2 actions13223257.7683.1986.21+3.025.602.59
3–5 actions16063042.2278.2581.43+3.177.303.17
≥6 actions3830241.0670.8675.17+4.309.933.97
Overall3301,16445.0277.3280.76+3.447.653.26

We recognize that these results do not establish generalization to long-horizon planning or recovery from accumulated execution errors. We clarify these limitations in the revised manuscript and plan to expand VersionShift with longer and more diverse trajectories in future work, supported by the release of our annotation and validation tools.

W3: Annotation Reliability

We sincerely thank the reviewer for this valuable suggestion. To address this concern, we conduct an independent reannotation study involving three additional annotators, each independently labeling the same 900 samples without access to the original annotations. The samples are balanced across the three validity categories and cover all platforms and version pairings.

Table 3. Inter-annotator agreement on independently reannotated source-conditioned target steps. Samples reports the number of reannotated instances relative to all annotated instances for each validity category. A1–A3 report exact agreement with the original annotations. Fleiss' κ measures agreement among the three independent annotators.

Validity LabelSamples / TotalA1 (%) ↑A2 (%) ↑A3 (%) ↑Fleiss' κ
REUSE300 / 3,24797.797.098.0—
REGROUND300 / 1,52695.796.095.3—
REPLAN300 / 1,56997.096.396.7—
Overall900 / 6,34296.896.496.70.91

As shown in Table 3, the three independent annotators achieve overall agreement rates of 96.8%, 96.4%, and 96.7% with the original annotations, respectively, with consistently high agreement across all three validity categories. The Fleiss' κ of 0.91 further indicates strong agreement among independent annotators. These results provide additional evidence of the reliability and consistency of VersionShift's step-level validity annotations.

Thank you for the thoughtful and detailed response, which addresses most of my concerns. I will therefore maintain my current score, which is already positive.

Thank you for your thoughtful and constructive feedback, and for maintaining your positive assessment of our work.

We are glad that our responses and additional experiments have addressed most of your concerns. We have incorporated the corresponding experimental results, clarifications, and discussions into the revised manuscript.

Thank you again for your time and valuable suggestions, which have greatly contributed to improving our work!

Reviewer_m917OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the domain of GUI agents. The authors first observe a practical reality: app versions are continuously updated, and each update may bring changes to both UI layouts and functionalities. This observation leads them to raise several key questions: "Can currently trained agents maintain strong generalization capabilities in the face of such new changes? Can performance be improved by feeding execution experiences from older versions into the new model? And how can optimization strategies be adjusted to enable the model to better adapt to new versions?" To address these questions, the authors construct a benchmark called VERSIONSHIFT, which encompasses task trajectories across multiple apps at different time points, as well as the discrepancies in trajectories between different versions. Furthermore, they propose a method named VAVE, designed to either reuse prior experiences or select new planning strategies, which demonstrates superior performance compared to several baseline models.

Critical strengths
  1. The identified problems are accurate and well-founded. The issues raised by the authors indeed reflect real-world challenges. Most existing GUI agents focus solely on their capabilities with current app versions at a specific point in time, while overlooking their adaptability to newer versions over an extended temporal span. Furthermore, task trajectories between old and new versions share similarities but also exhibit differences. Determining which prior experiences can be effectively reused and which cannot is undoubtedly a valuable and insightful research question. 2.The dataset construction is novel. Within the GUI agent community, datasets containing trajectories across different app versions are rare, and even fewer provide annotations indicating which actions are reusable and which are not. This work therefore represents a significant contribution to the community.
  2. The evaluation and experiments are relatively comprehensive. The authors not only demonstrate that directly feeding old experiences leads to performance degradation, but also conduct ablation studies on the various modules proposed in their method.
Critical weaknesses
  1. The scale of the benchmark is relatively small, which may undermine the persuasiveness of the conclusions. For instance, the dataset includes only 53 apps and a total of 312 tasks, with merely one trajectory per task per version. By the standards of the GUI agent domain, this data scale is comparatively limited. Furthermore, all annotations were completed solely by the authors. While I acknowledge the inherent difficulties of data annotation in academic research, more robust evaluation data and consequently more reliable conclusions are arguably of greater importance for a scholarly publication.
  2. The evaluation is still conducted exclusively on an offline test set, without testing in a real-world GUI runtime environment. Consequently, it remains unclear how much performance improvement the accumulated experiences and the VAVE method can actually yield for all current new apps in practice.
  3. The experiments are conducted solely on a 7B VLM base model. It remains to be further demonstrated whether the proposed approach can be integrated with state-of-the-art agents in the community (e.g., Qwen-UI-Agent) and whether it would also yield performance benefits for them.
Questions for authors

Please refer to the 3 weakness.

I believe this paper has an excellent entry point. The constructed benchmark, the proposed method, and the alignment with the initial motivation form a coherent closed loop, making it an acceptable academic contribution. However, GUI agents represent a highly application-oriented field that is also rapidly evolving in industry. Whether the problems identified in the paper hold true in real-world environments, and whether the proposed method can be effectively deployed in practice, still require rigorous validation.

2 comments
AuthorsOpenReview ↗

Response to Reviewer m917: Weakness 1

We sincerely thank the reviewer for the thoughtful assessment and for recognizing the practical importance of our work. We address the concerns regarding benchmark scale and annotation reliability, closed-loop evaluation, and integration with stronger GUI agents through clarifications, additional analysis, and supplementary results below.

W1: Limited Benchmark Scale and Annotation Reliability

Benchmark scale and validity. We acknowledge that broader application coverage and multiple successful trajectories per task would further strengthen the generalizability of VersionShift. Nevertheless, its controlled cross-version design enables systematic evaluation of experience transfer under matched task objectives and execution conditions.

Table 1. Comparison with representative cross-version GUI benchmarks. Tasks / Queries denotes task goals or grounding queries. Step Validity indicates explicit annotations of historical action validity, while PTR/NTR indicates evaluation of positive and negative transfer.

BenchmarkApps / SitesTasks / QueriesPlatformsEvaluationStep ValidityPTR/NTR
TransBench8122K+M/WGrounding✗✗
TimeWarp3231WMulti-step Task✗✗
VersionShift53312M/D/WMulti-step Task✓✓

As shown in Table 1, VersionShift covers 53 applications and 312 multi-step tasks across three platforms, with broader application and task coverage than TimeWarp. Although TransBench is larger in scale, it primarily focuses on grounding queries rather than executable multi-step tasks. VersionShift further provides paired cross-version trajectories and step-level validity annotations, enabling systematic evaluation of positive and negative experience transfer.

We recognize that a single successful trajectory per task and version cannot exhaust alternative execution paths, and clarify this limitation in the revised manuscript. In future work, we plan to expand VersionShift with broader application and task coverage and multiple successful trajectories per task–version instance, while preserving its controlled cross-version evaluation protocol. To facilitate this expansion and independent replication, we will release our annotation tools, labeling guidelines, and validation workflow.

Annotation reliability. To address concerns about annotation bias, we conduct an independent reannotation study involving three additional annotators, each independently labeling the same 900 stratified samples without access to the original annotations. The samples cover all three platforms, adaptation modes, and version pairings. We assess label reliability through class-wise agreement with the original REUSE, REGROUND, and REPLAN labels and overall inter-annotator agreement.

Table 2. Inter-annotator agreement on independently reannotated source-conditioned target steps. Samples denotes the number of reannotated instances relative to all annotated instances for each validity category. A1–A3 report exact agreement with the original annotations. Fleiss' κ measures agreement among the three independent annotators.

Validity LabelSamples / TotalA1 (%) ↑A2 (%) ↑A3 (%) ↑Fleiss' κ
REUSE300 / 3,24797.797.098.0—
REGROUND300 / 1,52695.796.095.3—
REPLAN300 / 1,56997.096.396.7—
Overall900 / 6,34296.896.496.70.91

As shown in Table 2, the three independent annotators achieve overall agreement rates of 96.8%, 96.4%, and 96.7% with the original annotations, respectively, with consistently high agreement across REUSE, REGROUND, and REPLAN labels. The Fleiss' κ of 0.91 further indicates strong agreement among independent annotators. These results provide further evidence of the reliability and consistency of our step-level validity annotations.

AuthorsOpenReview ↗

Response to Reviewer m917: Weakness 2 & Weakness 3

W2: Lack of Real-World Online Evaluation

We agree that offline evaluation alone provides limited evidence of VAEA's practical effectiveness. To address this concern, we conducted additional closed-loop evaluations on the full held-out test split in VersionShift. Specifically, we execute model-predicted actions using pywinauto for Desktop, playwright for Web, and uiautomator2 for Mobile. After each action, agents receive updated observations from the environment to determine their next actions. We incorporate these results into the revised manuscript and provide the execution protocol and implementation details in the appendix.

Table 3. Offline and closed-loop evaluation on the VersionShift held-out test split. All values are reported as percentages (%). Offline evaluation (Columns 3–6): Step and Traj. denote step-level and trajectory-level accuracy, respectively, while PTR and NTR measure positive and negative transfer at the step level. Closed-loop evaluation (Columns 7–9): Task denotes end-to-end task success rate, while Task-PTR and Task-NTR measure positive and negative transfer at the task level.

BackboneMethodStep ↑Traj. ↑PTR ↑NTR ↓Task ↑Task-PTR ↑Task-NTR ↓
Qwen3-VL-4BTarget-only63.5737.88——36.36——
Raw History76.1248.7917.965.4128.486.0613.94
Supervised Selection + Actor78.0953.9418.303.7835.157.278.48
VAEA79.2154.5518.903.2640.618.794.55
Qwen3-VL-8BTarget-only68.9943.94——45.45——
Raw History77.3252.1215.987.6540.617.8812.73
Supervised Selection + Actor79.7354.5514.523.7846.068.487.88
VAEA80.7656.0615.033.2652.1212.736.06

As shown in Table 3, VAEA improves closed-loop task success by 5.46 and 6.06 percentage points over Supervised Selection + Actor on Qwen3-VL-4B and 8B, respectively, while reducing Task-NTR from 8.48% to 4.55% (4B) and from 7.88% to 6.06% (8B). Notably, Raw History performs worse than Target-only under closed-loop execution, highlighting the practical risks of negative transfer. These results provide evidence that VAEA improves end-to-end task completion and mitigates negative transfer under real-world GUI runtime environment.

W3: Limited Backbone Diversity and Integration with Stronger GUI Agents

We agree that evaluating VAEA with GUI-specialized agents would provide further evidence of its applicability. As Qwen-UI-Agent checkpoints were not publicly available, we extend our evaluation to its open-weight predecessor MAI-UI-8B. Specifically, we replace the original Qwen3-VL backbone with MAI-UI-8B and apply the same training protocol. We incorporate these results into the revised manuscript and provide implementation details in the appendix.

Table 4. VAEA integration with MAI-UI- 8B on the VersionShift held-out test split. All values are percentages (%). Step and Traj. denote offline step and trajectory accuracy, respectively. PTR and NTR measure positive and negative transfer relative to Target-only.

BackboneMethodStep ↑Traj. ↑PTR ↑NTR ↓
MAI-UI-8BTarget-only59.1934.85——
Raw History57.1336.677.479.54
Supervised Selection + Actor63.6642.426.532.06
VAEA65.7245.768.421.89

As shown in Table 4, VAEA improves Step and trajectory accuracy over Supervised Selection + Actor by 2.06 and 3.34 percentage points, respectively, while achieving higher PTR and lower NTR. These results provide further evidence that validity-aware experience adaptation can benefit GUI-specialized agents.

Reviewer_XeqoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

VersionShift pairs the same tasks across four software versions and labels whether each target step permits reuse, requires regrounding, or needs replanning. VAEA uses this supervision to improve offline action prediction while reducing harmful transfer from older trajectories.

Critical strengths
  1. Holding task intent fixed across versions makes experience validity directly testable. Coverage of 53 applications across three platforms and application-disjoint splits add value (§3; §5.1).
  2. Reporting positive and negative transfer separately exposes errors hidden by average gains. Raw history improves step accuracy for only 5 of 13 evaluated agents (Table 2).
  3. Two-pass and supervised-selection controls make the method comparison informative. The stale-locator ablation also supports removing old coordinates when regrounding (Tables 3 and 5).
Critical weaknesses
  1. Step and trajectory accuracy use recorded target observations and reference histories. With trajectories averaging 3.36 actions, the results provide limited evidence about recovery and task completion during live interaction (§5.1; Table 6).
  2. VAEA exceeds Supervised Selection + Actor by only 1.12 and 1.03 percentage points in step accuracy. These gains use one training seed and nine test applications, without uncertainty estimates (Tables 3, 9 and 10).
Questions for authors
  1. On a runnable subset, do the gains persist when agents execute their own actions and encounter the resulting states?
  2. Does VAEA consistently beat Supervised Selection + Actor across training seeds, with confidence intervals clustered by task or application?
  3. The related-work discussion could include MemGUI-Bench, MobileWorld, and OSWorld 2.0, clarifying the distinct focus on cross-version experience transfer.
1 comment
AuthorsOpenReview ↗

Response to Reviewer Xeqo

We sincerely thank the reviewer for the constructive feedback. We address each concern through additional evaluations, statistical analyses, and revisions to the manuscript.

W1 & Q1: Closed-Loop Evaluation of End-to-End Task Success

We agree that offline evaluation based on recorded observations provides limited evidence of agent behavior during live interaction. To address this concern, we conduct additional closed-loop evaluations on the full held-out test split. Specifically, we execute model-predicted actions using pywinauto for Desktop, playwright for Web, and uiautomator2 for Mobile. After each action, the agent observes the resulting interface state and selects its next action based on its own interaction history rather than recorded target observations. We report end-to-end task success and task-level positive/negative transfer (Task-PTR/NTR), with the execution protocol and implementation details provided in the appendix.

Table 1. Offline and closed-loop evaluation on the VersionShift held-out test split. All values are reported as percentages (%). Offline evaluation (Columns 3–6): Step and Traj. denote step-level and trajectory-level accuracy, while PTR and NTR measure positive and negative transfer at the step level. Closed-loop evaluation (Columns 7–9): Task denotes end-to-end task success under live interaction, while Task-PTR and Task-NTR measure task-level positive and negative transfer.

BackboneMethodStep ↑Traj. ↑PTR ↑NTR ↓Task ↑Task-PTR ↑Task-NTR ↓
Qwen3-VL-4BTarget-only63.5737.88——36.36——
Raw History76.1248.7917.965.4128.486.0613.94
Supervised Selection + Actor78.0953.9418.303.7835.157.278.48
VAEA79.2154.5518.903.2640.618.794.55
Qwen3-VL-8BTarget-only68.9943.94——45.45——
Raw History77.3252.1215.987.6540.617.8812.73
Supervised Selection + Actor79.7354.5514.523.7846.068.487.88
VAEA80.7656.0615.033.2652.1212.736.06

As shown in Table 1, VAEA achieves closed-loop task success rates of 40.61% and 52.12% on Qwen3-VL-4B and 8B, outperforming Supervised Selection + Actor by 5.46 and 6.06 percentage points, respectively. Meanwhile, Task-NTR decreases from 8.48% to 4.55% (4B) and from 7.88% to 6.06% (8B). These results demonstrate that VAEA improves end-to-end task success under closed-loop execution while mitigating negative transfer during live interaction.

W2 & Q2: Limited Statistical Robustness of the Improvements

We agree that VAEA's improvements require statistical validation. To address this concern, we evaluate both Supervised Selection + Actor and VAEA across three training seeds (0, 1, and 2), reporting mean accuracy, standard deviation, and task-clustered 95% confidence intervals for their performance differences. We incorporate these results in the revised manuscript and provide experimental details in the appendix.

Table 2. Training-seed robustness on the VersionShift held-out test split. Accuracy values are percentages (%). S0–S2 denote three training seeds. Step and Traj. Mean ± Std report accuracy across seeds. ΔStep (pp) denotes the mean improvement over Supervised Selection + Actor, with task-clustered 95% confidence intervals.

BackboneMethodStep S0Step S1Step S2Step Mean ± StdTraj. Mean ± StdΔStep (pp)Task-clustered 95% CI (pp)
Qwen3-VL-4BSupervised Selection + Actor78.0977.4178.5278.01 ± 0.5653.84 ± 0.76——
VAEA79.2178.7879.5579.18 ± 0.3954.65 ± 0.76+1.17[-0.42, +2.78]
Qwen3-VL-8BSupervised Selection + Actor79.7380.4179.3079.81 ± 0.5654.55 ± 0.91——
VAEA80.7681.5380.4180.90 ± 0.5755.96 ± 0.76+1.09[+0.11, +2.03]

As shown in Table 2, VAEA consistently outperforms Supervised Selection + Actor across all three evaluated seeds, with stronger statistical evidence for the improvement on Qwen3-VL-8B. These results provide evidence that VAEA's gains are consistent across the evaluated training seeds.

Q3: Additional Related Work

We agree that including these benchmarks would help clarify VersionShift's distinct focus on cross-version experience transfer. Following the reviewer's suggestion, we incorporate MemGUI-Bench, MobileWorld, and OSWorld 2.0 into the revised Related Work section. The revised paragraph is provided below, with additions highlighted in bold:

Existing GUI benchmarks span Web, Mobile, and Desktop environments, including Mind2Web, WebArena, AndroidWorld, and OSWorld, while OpenCUA and TongUI extend evaluation across multiple platform families. Other benchmarks investigate complementary challenges: MemGUI-Bench evaluates memory capabilities in mobile GUI agents, MobileWorld focuses on long-horizon and cross-application mobile tasks, and OSWorld 2.0 examines complex, long-horizon computer-use workflows.

Comments on the paper

1 comment

We are grateful to all reviewers for their thorough and constructive feedback. Below is a summary of our work and how each concern has been addressed.

Summary of the Work

VersionShift is a cross-platform benchmark for evaluating how GUI agents transfer and adapt historical execution experience across software versions. It covers 53 applications and websites, 312 tasks, and four chronological versions across Mobile, Web, and Desktop. By pairing multi-step trajectories across software versions and introducing step-level experience validity annotations (REUSE, REGROUND, and REPLAN), VersionShift enables systematic analysis of whether historical actions remain applicable and how they should be adapted.

Evaluations on VersionShift reveal that historical experience can introduce negative transfer in most evaluated GUI agents. To address this, we propose VAEA, which explicitly models experience validity to selectively reuse, reground, or replan prior experience, effectively reducing negative transfer while improving action accuracy.

The central contribution of this work is establishing step-level experience validity as an explicit evaluation target for cross-version GUI agents, together with a corresponding adaptation framework that demonstrates the practical value of this formulation.

Reviewer Responses

  • Response to Reviewer Xeqo
    • W1 & Q1 (Lack of closed-loop evaluation). We conducted closed-loop evaluations on the full held-out test split. VAEA improves task success and reduces negative transfer on both 4B and 8B backbones, demonstrating its effectiveness in live GUI interaction.
    • W2 & Q2 (Limited statistical robustness). We evaluated both methods across three independent training seeds and reported task-clustered 95% confidence intervals. VAEA consistently improves Step accuracy across the evaluated seeds, with stronger statistical evidence on the 8B backbone.
    • Q3 (Additional related work). We expanded the Related Work section to discuss MemGUI-Bench, MobileWorld, and OSWorld 2.0, further clarifying the distinct focus of VersionShift.
  • Response to Reviewer m917
    • W1 (Benchmark scale and annotation reliability). We provided a comparison with TransBench and TimeWarp to clarify VersionShift's distinctive evaluation scope. We also conducted an independent reannotation study involving three additional annotators, achieving 96.4%–96.8% agreement with the original annotations, supporting annotation reliability.
    • W2 (Lack of real-world online evaluation). Please refer to Reviewer Xeqo, W1 & Q1.
    • W3 (Limited backbone diversity). We extended our evaluation to the GUI-specialized MAI-UI-8B model. VAEA outperforms baselines in step accuracy and trajectory accuracy, demonstrating its applicability on GUI-specialized models.
  • Response to Reviewer rQQa (concerns fully resolved)
    • W1 & Q1 (Limited novelty and distinction from existing benchmarks). We clarified that VersionShift introduces a distinct evaluation problem rather than merely extending annotation granularity. Mode-wise analysis reveals distinct negative-transfer patterns across REUSE, REGROUND, and REPLAN, demonstrating the diagnostic value of step-level validity annotations.
    • W2 (Constrained tasks and short trajectories). We acknowledged the limitations of controlled task construction and analyzed adaptation difficulty across trajectory lengths. Even among 1–2-action trajectories, 57.76% of steps require REGROUND or REPLAN. VAEA improves accuracy and reduces negative transfer across all trajectory-length groups.
    • W3 (Annotation reliability). Please refer to Reviewer m917, W1.

We have incorporated the additional experiments, statistical analyses, and clarifications into the revised manuscript and appendix. We sincerely thank the Program Chairs, Senior Area Chairs, Area Chairs, and reviewers for their time and constructive feedback, which has been invaluable in improving the quality and rigor of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.