- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how GUI-agent performance varies across personalized interfaces. PAIR instantiates common and personalized targets with explicit and implicit instructions across synthetic personas, complementing a descriptive analysis of FingerTip interactions. RePAIR uses cross-user rollout groups and a normalized reward to improve a fine-tuned GUI agent. The paper reports substantial variation across personas and improved average and all-user success after training.
Critical strengths
-
Cross-user reliability is an important evaluation axis that aggregate task success can conceal. In particular, holding the common target and instruction fixed while varying surrounding personalized content provides a useful controlled test. Reporting all-user success alongside average success makes the reliability issue concrete.
-
The benchmark exposes several distinct sources of difficulty, including personalized targets and implicit instructions, and the paper connects these measurements to an intervention rather than stopping at a disparity report. The appendices candidly disclose important limitations of the task selection and training protocol.
Critical weaknesses
-
The reported RePAIR evaluation is not an untouched test of unseen-user adaptation. Appendix D states that training-task selection used baseline outcomes on evaluation users p0-p9 to identify mixed-success tasks. Although those users' trajectories do not enter the gradient updates, their outcomes inform the training curriculum. The appendix appropriately calls this exploratory, but this qualification materially limits the main improvement claim. In addition, the benchmark includes unresolved task flags and incomplete semantic/browser review of some persona instances, so intended-target validity, especially for implicit instructions, is not yet sufficiently established.
-
The evidence does not cleanly attribute gains to the cross-user reward formulation. RePAIR adds RL and substantial implicit-instruction exposure to an SFT baseline trained on explicit demonstrations; the user-pool ablation also changes the diversity of available users. A fixed-pool, fixed-rollout comparison with ordinary terminal-reward RL or alternative grouping is needed. This is particularly important because the Appendix C reward checks are permissive: route checks omit query parameters, click matching is not necessarily temporally ordered, and substring matching can accept partial text matches. These are potential false-positive channels, not evidence that exploitation actually occurred, but their frequency needs auditing before treating reward gains as reliable task completion.
Questions for authors
-
Can the main comparison be repeated with training-task selection fixed using only training/validation personas, followed by evaluation on personas whose outcomes were never used in selection?
-
With the same eight-user training pool, instructions, rollout count, and optimizer budget, how does RePAIR compare with conventional terminal-reward RL and with groups that do not normalize across users?
-
What fraction of flagged/implicit benchmark instances and automatically successful rollouts pass a manual check of intended target, argument values, and final app state? Please show whether the conclusions persist on the validated subset.
5 comments
Author Response [2/4]
[W1-2, Q3-1] Benchmark validity and implicit-instruction ambiguity
W1-2) In addition, the benchmark includes unresolved task flags and incomplete semantic/browser review of some persona instances, so intended-target validity, especially for implicit instructions, is not yet sufficiently established. Q3) What fraction of flagged/implicit benchmark instances and automatically successful rollouts pass a manual check of intended target, argument values, and final app state? Please show whether the conclusions persist on the validated subset.
Thank you for your question. Before running the benchmark in our study, to assess the validity of implicit instructions, we manually verified all 14 “unresolved task flags” flagged by automated review (1.2% of the 1,200 benchmark instances). For each instance, a human annotator examined the instruction and rendered UI to determine whether the intended target was uniquely identifiable. Manual verification confirmed that all 14 instances uniquely identified their intended targets.
We also assessed whether these flagged instances influenced our conclusions by excluding all 14 instances. Importantly, excluding these instances leaves our main findings unchanged. We have additionally evaluated the benchmark after excluding the 14 instances. The best–worst persona gap in Task SR remained nearly identical (9.32 vs. 9.31 pp on the full benchmark), as did the mean user-conditioned SAR drop (11.9 vs. 12.1 pp). Personal targets remained more difficult than common targets (4.4 pp lower SAR), and incorrect-item selections remained more frequent before target exposure for both target types.
In our revised manuscript, we have clarified training-user-only task selection in Section 6.1, updated the main RePAIR results in Section 6.2, and detailed the selection procedure in Appendix D.
Author Response [3/4]
[W2-1, Q2] Matched RL baselines and cross-user reward contribution
W2-1) The evidence does not cleanly attribute gains to the cross-user reward formulation. RePAIR adds RL and substantial implicit-instruction exposure to an SFT baseline trained on explicit demonstrations; the user-pool ablation also changes the diversity of available users. A fixed-pool, fixed-rollout comparison with ordinary terminal-reward RL or alternative grouping is needed. Q2) With the same eight-user training pool, instructions, rollout count, and optimizer budget, how does RePAIR compare with conventional terminal-reward RL and with groups that do not normalize across users?
Thank you for suggesting these controls. We have conducted matched RL comparisons with a fixed user pool, instruction exposure, rollout count, and optimizer budget to distinguish the contribution of cross-user reward normalization from those of additional RL training and subgoal supervision. Specifically, we compare RePAIR with (i) GRPO using binary task-completion rewards and within-user normalization and (ii) GRPO using the same subgoal rewards as RePAIR but with within-user normalization.
All RL methods use the same training-user-only task selection (31 tasks, p10–p17), SFT initialization, and training budget (155 updates, 620 rollouts), with evaluation on p0–p9 at iteration 5. We fix the two-user rollout configuration to control user-state exposure, varying only the reward formulation and normalization scheme.
| Method | Reward | Cross-user normalization | Task SR (%) | All-success (%) | UA SAR (%) | UC SAR (%) |
|---|---|---|---|---|---|---|
| SFT | — | — | 55.1 | 20.0 | 92.3 | 71.9 |
| (i) GRPO | Task completion | ✗ | 58.4 | 25.8 | 94.5 | 73.2 |
| (ii) GRPO | Subgoal | ✗ | 60.9 | 23.3 | 90.9 | 76.7 |
| RePAIR | Subgoal | ✓ | 64.0 | 28.3 | 91.0 | 78.8 |
✗: Within-user normalization only. ✓: Cross-user normalization.
RePAIR outperforms conventional task-completion-reward GRPO (i) by 5.6 percentage points in Task SR, 2.5 points in All-success, and 5.6 points in UC SAR. Subgoal rewards and cross-user normalization contribute differently to these gains. Introducing subgoal rewards under within-user normalization improves Task SR by 2.5 points and UC SAR by 3.5 points, but decreases All-success by 2.5 points. This suggests that finer-grained supervision helps the model learn user-conditioned behavior, but alone does not translate into more consistent success across users.
Compared with control (ii), which uses the same subgoal rewards without cross-user normalization, RePAIR further improves Task SR by 3.1 points, All-success by 5.0 points, and UC SAR by 2.1 points. This supports the role of cross-user comparisons in translating subgoal-level improvements into broader success across users. Although task-reward GRPO achieves higher UA SAR, it decreases when introducing subgoal rewards (94.5% to 90.9%); adding cross-user normalization leaves UA SAR nearly unchanged (90.9% to 91.0%). This pattern suggests that the gains reflect better fulfillment of user-specific requirements rather than greater achievement of user-agnostic subgoals, and that cross-user normalization does not cause the lower UA SAR. Together, these results suggest complementary roles: subgoal rewards provide finer-grained supervision for user-conditioned behavior, while cross-user normalization helps convert that supervision into more consistent task success across users.
We have added the matched RL baselines to Table 2, analyzed their results in Section 6.2, and documented the matched training setup in Appendix D in the revised manuscript.
Author Response [1/4]
Dear Reviewer 9h7v,
Thank you for your careful review. We appreciate your recognition that cross-user reliability reveals limitations hidden by average task success, that our common-target setting provides a useful controlled test, and that reporting all-user success makes these reliability differences concrete. Below, we address your concerns about evaluation independence, validation, and the evidence for RePAIR's improvements. We have also uploaded a revised manuscript reflecting our rebuttal, with all changes highlighted in blue.
[W1-1, Q1] Training-task selection using evaluation-user outcomes
We thank the reviewer for raising the concern about potential selection bias. We originally selected tasks with mixed success and failure outcomes across users to highlight cross-user performance differences. To address the reviewer's concern, we additionally evaluated RePAIR using training tasks selected solely from training-user outcomes (p10–p17), without using evaluation-user outcomes.
Both variants use the same eight training users, SFT initialization, and training budget (31 task specifications, 155 updates), with evaluation on p0–p9 at iteration 5.
RePAIR maintains comparable performance without using evaluation-user outcomes for task selection, achieving similar Task SR (64.0% vs. 64.5%) and slightly higher UC SAR (78.8% vs. 77.7%) and All-success (28.3% vs. 27.5%).
These results suggest that RePAIR's improvements do not rely on evaluation-user outcomes to select training tasks. We have added these results and clarified the task-selection procedure in
Section 6.2of the revised paper.