- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how GUI agents can adapt prior interaction experience across different software versions. It introduces VersionShift, which pairs same-task trajectories across four chronological versions and provides step-level annotations for 312 tasks across 53 applications and websites. The paper further proposes VAEA to selectively reuse or adapt historical experience, achieving improved action prediction and reduced negative transfer under the proposed evaluation protocol.
Critical strengths
-
The paper introduces a meaningful problem, as prior GUI experience can remain useful after software updates but may also become partially outdated or misleading.
-
VersionShift provides a relatively comprehensive dataset covering Mobile, Web, and Desktop applications. Its controlled task construction, paired trajectories, and step-level validity annotations form a reasonably systematic evaluation setting.
-
The paper also proposes VAEA, which achieves measurable improvements in experience adaptation and reduces negative transfer under the current protocol.
Critical weaknesses
-
This is primarily a dataset paper, but the research problem itself is not particularly new. VersionShift mainly extends current data through paired multi-step trajectories, finer-grained validity annotations, and a more controlled task formulation.
-
From the perspective of dataset construction, the improvements over existing benchmarks also seem incremental. As reflected in Table 1, the most distinctive differences are the broader platform coverage and the availability of paired trajectories. Meanwhile, the task selection is relatively constrained because the intent, initial state, and success criterion must remain reproducible across all four versions, and the trajectories are generally short, averaging only 3.36 executable actions and 1.93 actions for Web tasks.
-
The dataset would benefit from stronger quantitative validation of its annotations. All REUSE, REGROUND, and REPLAN labels were produced and verified by one trained annotator, without an inter-annotator agreement study or independent relabeling of a subset.
Questions for authors
- What do the authors consider the single most important contribution that clearly separates VersionShift from existing cross-version benchmarks such as TransBench and TimeWarp, and what is the most convincing empirical evidence that this distinction is substantive rather than primarily an incremental improvement in pairing and annotation granularity?
- Please refer to the weaknesses.
4 comments
Response to Reviewer rQQa: Weakness 2 & Weakness 3
W2: Constrained Task Design and Short Trajectories
We sincerely thank the reviewer for highlighting these limitations. We acknowledge that requiring consistent task intents, initial states, and success criteria across four versions constrains task diversity. Nevertheless, this controlled design helps isolate the effects of software evolution on experience validity, preventing differences in task objectives and execution conditions from confounding source–target comparisons.
We also acknowledge the relatively short trajectories (3.36 actions on average and 1.93 for Web). However, short trajectories do not necessarily imply simple cross-version adaptation. On the held-out test split, 57.76% of steps from 1–2-action trajectories require REGROUND (37.93%) or REPLAN (19.83%). Furthermore, Table 2 shows that VAEA improves Step accuracy and reduces negative transfer across all trajectory-length groups, including trajectories with at least six actions. These results indicate that experience adaptation remains relevant across different trajectory lengths, rather than being confined to short workflows.
Table 2. Step accuracy and negative transfer across trajectory lengths on the VersionShift held-out test split (Qwen3-VL-8B). All accuracy and NTR values are percentages (%). Trajectories are grouped by the number of target-version reference actions. NTR is measured relative to Target-only. ΔStep denotes the improvement of VAEA over Raw History.
| Trajectory Length | Traj. Pairs | Steps | REGROUND+ REPLAN(%) | Raw History Step↑ | VAEA Step ↑ | ΔStep (pp) ↑ | Raw History NTR ↓ | VAEA NTR ↓ |
|---|---|---|---|---|---|---|---|---|
| 1–2 actions | 132 | 232 | 57.76 | 83.19 | 86.21 | +3.02 | 5.60 | 2.59 |
| 3–5 actions | 160 | 630 | 42.22 | 78.25 | 81.43 | +3.17 | 7.30 | 3.17 |
| ≥6 actions | 38 | 302 | 41.06 | 70.86 | 75.17 | +4.30 | 9.93 | 3.97 |
| Overall | 330 | 1,164 | 45.02 | 77.32 | 80.76 | +3.44 | 7.65 | 3.26 |
We recognize that these results do not establish generalization to long-horizon planning or recovery from accumulated execution errors. We clarify these limitations in the revised manuscript and plan to expand VersionShift with longer and more diverse trajectories in future work, supported by the release of our annotation and validation tools.
W3: Annotation Reliability
We sincerely thank the reviewer for this valuable suggestion. To address this concern, we conduct an independent reannotation study involving three additional annotators, each independently labeling the same 900 samples without access to the original annotations. The samples are balanced across the three validity categories and cover all platforms and version pairings.
Table 3. Inter-annotator agreement on independently reannotated source-conditioned target steps. Samples reports the number of reannotated instances relative to all annotated instances for each validity category. A1–A3 report exact agreement with the original annotations. Fleiss' κ measures agreement among the three independent annotators.
| Validity Label | Samples / Total | A1 (%) ↑ | A2 (%) ↑ | A3 (%) ↑ | Fleiss' κ |
|---|---|---|---|---|---|
| REUSE | 300 / 3,247 | 97.7 | 97.0 | 98.0 | — |
| REGROUND | 300 / 1,526 | 95.7 | 96.0 | 95.3 | — |
| REPLAN | 300 / 1,569 | 97.0 | 96.3 | 96.7 | — |
| Overall | 900 / 6,342 | 96.8 | 96.4 | 96.7 | 0.91 |
As shown in Table 3, the three independent annotators achieve overall agreement rates of 96.8%, 96.4%, and 96.7% with the original annotations, respectively, with consistently high agreement across all three validity categories. The Fleiss' κ of 0.91 further indicates strong agreement among independent annotators. These results provide additional evidence of the reliability and consistency of VersionShift's step-level validity annotations.
Thank you for the thoughtful and detailed response, which addresses most of my concerns. I will therefore maintain my current score, which is already positive.
Thank you for your thoughtful and constructive feedback, and for maintaining your positive assessment of our work.
We are glad that our responses and additional experiments have addressed most of your concerns. We have incorporated the corresponding experimental results, clarifications, and discussions into the revised manuscript.
Thank you again for your time and valuable suggestions, which have greatly contributed to improving our work!
Response to Reviewer rQQa: Weakness 1 & Question 1
We sincerely thank the reviewer for the constructive feedback. We address the concerns regarding novelty, task complexity, and annotation reliability below.
W1 & Q1: Distinction from Existing Cross-Version Benchmarks
We agree that cross-version GUI evaluation is not new. However, the key contribution of VersionShift is not merely finer-grained trajectory annotation, but a distinct evaluation problem: determining whether and how prior execution experience remains applicable in a new software version. This requires evaluating the validity of individual historical actions and their adaptation requirements, rather than only measuring performance under interface changes.
TransBench focuses on single-step GUI grounding transfer, while TimeWarp is limited to Web environments and primarily evaluates robustness and generalization across interface versions. In contrast, to our knowledge, VersionShift is the first cross-platform benchmark to explicitly evaluate the step-level validity of prior execution experience across software versions. By pairing multi-step trajectories across Mobile, Web, and Desktop and introducing REUSE, REGROUND, and REPLAN annotations, VersionShift enables fine-grained analysis of experience adaptation and positive/negative transfer, rather than relying solely on aggregate grounding accuracy or task success.
Our mode-wise analysis provides direct empirical evidence of the diagnostic value of VersionShift. As shown in Table 1, negative transfer varies substantially across adaptation modes, with REPLAN exhibiting the highest NTR despite representing less than one-fifth of evaluated steps. VAEA reduces NTR across all three adaptation modes and improves net transfer on REPLAN steps to +4.78 percentage points. These findings show that step-level experience validity provides a meaningful basis for diagnosing negative transfer and evaluating selective experience adaptation, rather than merely increasing annotation granularity.
Table 1. Mode-wise positive and negative transfer on the VersionShift held-out test split (Qwen3-VL-8B). PTR and NTR are measured relative to Target-only, with each adaptation mode's total steps as the denominator.
The practical significance is further supported by closed-loop evaluation, where VAEA achieves 52.12% task success, compared with 40.61% for Raw History and 46.06% for Supervised Selection + Actor (Table 18 in the manuscript). Together, these findings demonstrate that VersionShift enables systematic diagnosis of distinct experience failure modes and adaptation requirements, establishing step-level experience validity as a meaningful evaluation target across software versions.