acceptodds
Under review as a conference paper at ICLR 2027

Can Agent Work for Everyone? Cross-User Reliability for Mobile GUI Agents in Personalized User Interfaces

Abstract

Mobile GUI agents increasingly operate on interfaces influenced by users' histories and preferences, but their reliability across different users remains underexplored. We introduce PAIR (Personalized Application-state Instantiation and Rendering), a pipeline for constructing user-conditioned application states that enables controlled evaluation of the same task across different users. We further introduce RePAIR (Reinforcement learning with Personalization-Aware Interaction Rewards), a training approach that learns from cross-user differences in subgoal outcomes to improve reliability across user-conditioned mobile environments. Across six agents, we find substantial variation in task success across users and consistently lower subgoal achievement in user-conditioned UI contexts (6.98 to 15.4 pp). This gap further increases for personal targets drawn from each user’s own content (8.77 to 22.0 pp). Failures in these contexts frequently involve selecting another item instead of the intended target, particularly before target exposure. Finally, RePAIR improves user-conditioned SAR (+5.87 pp), all-success (+7.50 pp), and overall Task SR (+9.42 pp) over its supervised fine-tunning parent on unseen users, providing initial evidence that explicitly learning from cross-user variation can improve GUI-agent reliability.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Mobile GUI Agent Personalization, Personalized User Interfaces, Cross-User Reliability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9h7vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how GUI-agent performance varies across personalized interfaces. PAIR instantiates common and personalized targets with explicit and implicit instructions across synthetic personas, complementing a descriptive analysis of FingerTip interactions. RePAIR uses cross-user rollout groups and a normalized reward to improve a fine-tuned GUI agent. The paper reports substantial variation across personas and improved average and all-user success after training.

Critical strengths
  1. Cross-user reliability is an important evaluation axis that aggregate task success can conceal. In particular, holding the common target and instruction fixed while varying surrounding personalized content provides a useful controlled test. Reporting all-user success alongside average success makes the reliability issue concrete.

  2. The benchmark exposes several distinct sources of difficulty, including personalized targets and implicit instructions, and the paper connects these measurements to an intervention rather than stopping at a disparity report. The appendices candidly disclose important limitations of the task selection and training protocol.

Critical weaknesses
  1. The reported RePAIR evaluation is not an untouched test of unseen-user adaptation. Appendix D states that training-task selection used baseline outcomes on evaluation users p0-p9 to identify mixed-success tasks. Although those users' trajectories do not enter the gradient updates, their outcomes inform the training curriculum. The appendix appropriately calls this exploratory, but this qualification materially limits the main improvement claim. In addition, the benchmark includes unresolved task flags and incomplete semantic/browser review of some persona instances, so intended-target validity, especially for implicit instructions, is not yet sufficiently established.

  2. The evidence does not cleanly attribute gains to the cross-user reward formulation. RePAIR adds RL and substantial implicit-instruction exposure to an SFT baseline trained on explicit demonstrations; the user-pool ablation also changes the diversity of available users. A fixed-pool, fixed-rollout comparison with ordinary terminal-reward RL or alternative grouping is needed. This is particularly important because the Appendix C reward checks are permissive: route checks omit query parameters, click matching is not necessarily temporally ordered, and substring matching can accept partial text matches. These are potential false-positive channels, not evidence that exploitation actually occurred, but their frequency needs auditing before treating reward gains as reliable task completion.

Questions for authors
  1. Can the main comparison be repeated with training-task selection fixed using only training/validation personas, followed by evaluation on personas whose outcomes were never used in selection?

  2. With the same eight-user training pool, instructions, rollout count, and optimizer budget, how does RePAIR compare with conventional terminal-reward RL and with groups that do not normalize across users?

  3. What fraction of flagged/implicit benchmark instances and automatically successful rollouts pass a manual check of intended target, argument values, and final app state? Please show whether the conclusions persist on the validated subset.

5 comments
AuthorsOpenReview ↗

Author Response [1/4]

Dear Reviewer 9h7v,

Thank you for your careful review. We appreciate your recognition that cross-user reliability reveals limitations hidden by average task success, that our common-target setting provides a useful controlled test, and that reporting all-user success makes these reliability differences concrete. Below, we address your concerns about evaluation independence, validation, and the evidence for RePAIR's improvements. We have also uploaded a revised manuscript reflecting our rebuttal, with all changes highlighted in blue.

[W1-1, Q1] Training-task selection using evaluation-user outcomes

W1-1) The reported RePAIR evaluation is not an untouched test of unseen-user adaptation. Appendix D states that training-task selection used baseline outcomes on evaluation users p0-p9 to identify mixed-success tasks. Although those users' trajectories do not enter the gradient updates, their outcomes inform the training curriculum. The appendix appropriately calls this exploratory, but this qualification materially limits the main improvement claim. Q1) Can the main comparison be repeated with training-task selection fixed using only training/validation personas, followed by evaluation on personas whose outcomes were never used in selection?

We thank the reviewer for raising the concern about potential selection bias. We originally selected tasks with mixed success and failure outcomes across users to highlight cross-user performance differences. To address the reviewer's concern, we additionally evaluated RePAIR using training tasks selected solely from training-user outcomes (p10–p17), without using evaluation-user outcomes.

Both variants use the same eight training users, SFT initialization, and training budget (31 task specifications, 155 updates), with evaluation on p0–p9 at iteration 5.

Task selectionTask SR (%)UC SAR (%)All-success (%)
Original64.577.727.5
Training-user only64.078.828.3

RePAIR maintains comparable performance without using evaluation-user outcomes for task selection, achieving similar Task SR (64.0% vs. 64.5%) and slightly higher UC SAR (78.8% vs. 77.7%) and All-success (28.3% vs. 27.5%).

These results suggest that RePAIR's improvements do not rely on evaluation-user outcomes to select training tasks. We have added these results and clarified the task-selection procedure in Section 6.2 of the revised paper.


AuthorsOpenReview ↗

Author Response [2/4]

[W1-2, Q3-1] Benchmark validity and implicit-instruction ambiguity

W1-2) In addition, the benchmark includes unresolved task flags and incomplete semantic/browser review of some persona instances, so intended-target validity, especially for implicit instructions, is not yet sufficiently established. Q3) What fraction of flagged/implicit benchmark instances and automatically successful rollouts pass a manual check of intended target, argument values, and final app state? Please show whether the conclusions persist on the validated subset.

Thank you for your question. Before running the benchmark in our study, to assess the validity of implicit instructions, we manually verified all 14 “unresolved task flags” flagged by automated review (1.2% of the 1,200 benchmark instances). For each instance, a human annotator examined the instruction and rendered UI to determine whether the intended target was uniquely identifiable. Manual verification confirmed that all 14 instances uniquely identified their intended targets.

We also assessed whether these flagged instances influenced our conclusions by excluding all 14 instances. Importantly, excluding these instances leaves our main findings unchanged. We have additionally evaluated the benchmark after excluding the 14 instances. The best–worst persona gap in Task SR remained nearly identical (9.32 vs. 9.31 pp on the full benchmark), as did the mean user-conditioned SAR drop (11.9 vs. 12.1 pp). Personal targets remained more difficult than common targets (4.4 pp lower SAR), and incorrect-item selections remained more frequent before target exposure for both target types.

In our revised manuscript, we have clarified training-user-only task selection in Section 6.1, updated the main RePAIR results in Section 6.2, and detailed the selection procedure in Appendix D.


AuthorsOpenReview ↗

Author Response [3/4]

[W2-1, Q2] Matched RL baselines and cross-user reward contribution

W2-1) The evidence does not cleanly attribute gains to the cross-user reward formulation. RePAIR adds RL and substantial implicit-instruction exposure to an SFT baseline trained on explicit demonstrations; the user-pool ablation also changes the diversity of available users. A fixed-pool, fixed-rollout comparison with ordinary terminal-reward RL or alternative grouping is needed. Q2) With the same eight-user training pool, instructions, rollout count, and optimizer budget, how does RePAIR compare with conventional terminal-reward RL and with groups that do not normalize across users?

Thank you for suggesting these controls. We have conducted matched RL comparisons with a fixed user pool, instruction exposure, rollout count, and optimizer budget to distinguish the contribution of cross-user reward normalization from those of additional RL training and subgoal supervision. Specifically, we compare RePAIR with (i) GRPO using binary task-completion rewards and within-user normalization and (ii) GRPO using the same subgoal rewards as RePAIR but with within-user normalization.

All RL methods use the same training-user-only task selection (31 tasks, p10–p17), SFT initialization, and training budget (155 updates, 620 rollouts), with evaluation on p0–p9 at iteration 5. We fix the two-user rollout configuration to control user-state exposure, varying only the reward formulation and normalization scheme.

MethodRewardCross-user normalizationTask SR (%)All-success (%)UA SAR (%)UC SAR (%)
SFT——55.120.092.371.9
(i) GRPOTask completion✗58.425.894.573.2
(ii) GRPOSubgoal✗60.923.390.976.7
RePAIRSubgoal✓64.028.391.078.8

✗: Within-user normalization only. ✓: Cross-user normalization.

RePAIR outperforms conventional task-completion-reward GRPO (i) by 5.6 percentage points in Task SR, 2.5 points in All-success, and 5.6 points in UC SAR. Subgoal rewards and cross-user normalization contribute differently to these gains. Introducing subgoal rewards under within-user normalization improves Task SR by 2.5 points and UC SAR by 3.5 points, but decreases All-success by 2.5 points. This suggests that finer-grained supervision helps the model learn user-conditioned behavior, but alone does not translate into more consistent success across users.

Compared with control (ii), which uses the same subgoal rewards without cross-user normalization, RePAIR further improves Task SR by 3.1 points, All-success by 5.0 points, and UC SAR by 2.1 points. This supports the role of cross-user comparisons in translating subgoal-level improvements into broader success across users. Although task-reward GRPO achieves higher UA SAR, it decreases when introducing subgoal rewards (94.5% to 90.9%); adding cross-user normalization leaves UA SAR nearly unchanged (90.9% to 91.0%). This pattern suggests that the gains reflect better fulfillment of user-specific requirements rather than greater achievement of user-agnostic subgoals, and that cross-user normalization does not cause the lower UA SAR. Together, these results suggest complementary roles: subgoal rewards provide finer-grained supervision for user-conditioned behavior, while cross-user normalization helps convert that supervision into more consistent task success across users.

We have added the matched RL baselines to Table 2, analyzed their results in Section 6.2, and documented the matched training setup in Appendix D in the revised manuscript.


Reviewer_NNbjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether mobile GUI agents complete the same request when app content differs across users. PAIR constructs app states from user profiles and supports comparisons with shared or personal targets and explicit or implicit instructions. RePAIR trains an agent with GRPO using subgoal rewards normalized across rollouts from different users. The authors evaluate existing agents, analyze failures during target search and selection, and report improvements over supervised fine-tuning after RePAIR training.

Critical strengths

I found the problem quite interesting (and timely). Personalized screens are a common source of failure for deployed agents, and holding the app structure, instruction, and target fixed while changing only user content is a useful way to study it. The common target condition in particular compares the same request across users, and the PAIR pipeline could be reused for other agents and apps.

The analysis on trajectory level also provides more insights that i found interesting. Breaking tasks into subgoals, and separating wrong item selections before and after the target becomes visible, shows at which step agents fail and what kind of mistake they make. That points to concrete fixes (e.g., searching further before committing to an item). I also appreciate that the appendix documents limitations openly, including how the RePAIR training tasks were selected.

Critical weaknesses

The improvement over SFT does not show whether comparisons across users are responsible. RePAIR adds RL, subgoal rewards, and implicit instructions that SFT did not use, but Table 2 has no RL control with the same data and budget. Since Eq. 4 pools rewards from different initial states, the advantage also depends on the difficulty of the paired instance. I would compare against GRPO that trains across the same users but forms each group from one user's task instance. The training tasks were also selected using baseline outcomes on the evaluation users (Appendix D). Although the authors disclose that choice, the test outcomes have already influenced where the model receives additional training. Task selection should use only training or validation users before the comparison is repeated.

I also had trouble interpreting the SAR gap as evidence about the effect of personalized content. Many shared contexts involve fixed navigation controls, while personalized contexts require a search for the target, so the comparison includes different operations (Figure 5 / Appendix B.3). SAR also counts only subgoals the agent reaches, which means the cases included in each average depend on earlier failures (Appendix E). The shuffled lists may put the same target at different depths for different profiles as well (Appendix B.1). The results demonstrate variation across app states, but I could not tell which differences account for it. A comparison of the same operation at a fixed target position, with the number of reached and successful subgoals reported separately, would help explain the gap.

Questions for authors

(1) Do the RePAIR gains remain when task selection uses only training or validation users? On that split, how does RePAIR compare with GRPO using the same subgoal rewards and users, but groups drawn from one user's instance? A control that rewards only task completion would also help identify what the subgoal supervision contributes. (please keep the rollout budget fixed)

(2) How much does target position explain the performance differences across profiles? Could the authors compare the same subgoal with the target initially visible versus requiring scrolling, and report how often each agent reaches and completes that subgoal?

(3) How often does the reward scorer credit an incorrect nonterminal subgoal? Route matching removes query parameters and does not enforce visit order (Appendix C). If item identity is stored in a query parameter, could the route condition in Eq. 7 award success for the wrong item even when the target click does not match? Please check a sample against the intended item IDs and state changes, and clarify whether evaluation SAR uses the same scorer.

(4) How many implicit instructions retain unresolved review flags, and do the conclusions hold after those cases are checked for ambiguity (Appendix B.2)? A valid interpretation of an ambiguous request should not count as a failure to identify the target.

7 comments
AuthorsOpenReview ↗

Author Response [1/6]

Dear Reviewer NNbj,

Thank you for your thoughtful review. We appreciate your recognition of the importance and timeliness of cross-user reliability, PAIR’s controlled comparison of the same request and target across personalized interfaces, and our trajectory analyses that identify where failures occur and suggest concrete improvements. Below, we address your concerns and clarify the evaluation and training protocols. We have also uploaded a revised manuscript reflecting our rebuttal, with all changes highlighted in blue.

[W1, Q1] Training-task selection and matched RL comparisons

W1) The improvement over SFT does not show whether comparisons across users are responsible. RePAIR adds RL, subgoal rewards, and implicit instructions that SFT did not use, but Table 2 has no RL control with the same data and budget. Since Eq. 4 pools rewards from different initial states, the advantage also depends on the difficulty of the paired instance. I would compare against GRPO that trains across the same users but forms each group from one user's task instance. The training tasks were also selected using baseline outcomes on the evaluation users (Appendix D). Although the authors disclose that choice, the test outcomes have already influenced where the model receives additional training. Task selection should use only training or validation users before the comparison is repeated. Q1) Do the RePAIR gains remain when task selection uses only training or validation users? On that split, how does RePAIR compare with GRPO using the same subgoal rewards and users, but groups drawn from one user's instance? A control that rewards only task completion would also help identify what the subgoal supervision contributes. (please keep the rollout budget fixed)

We thank the reviewer for raising the concern about potential selection bias. We originally selected tasks with mixed success and failure outcomes across users to highlight cross-user performance differences. To address the reviewer's concern, we additionally evaluated RePAIR using training tasks selected solely from training-user outcomes (p10–p17), without using evaluation-user outcomes.

Both variants use the same eight training users, SFT initialization, and training budget (31 task specifications, 155 updates), with evaluation on p0–p9 at iteration 5.

Task selectionTask SR (%)All-success (%)User-agnostic SAR (%)User-conditioned SAR (%)
Evaluation-persona-informed64.527.594.377.7
Training-user only64.028.391.078.8

RePAIR maintains comparable performance without using evaluation-user outcomes for task selection, achieving similar Task SR (64.0% vs. 64.5%) and slightly higher UC SAR (78.8% vs. 77.7%) and All-success (28.3% vs. 27.5%). These results indicate that RePAIR's improvements do not rely on evaluation-user outcomes to select training tasks.

In addition, following the reviewer's suggestion, we compare RePAIR with (i) GRPO using task-completion rewards with within-user normalization and (ii) GRPO using the same subgoal rewards but applying within-user normalization. To evaluate the effect of cross-user normalization under matched user-state exposure, we use the same two-user rollout configuration across all methods and vary only the reward formulation and normalization scheme. All methods use the same training-user-only task selection (p10 to p17), SFT initialization, and training budget (155 updates, 620 rollouts).

MethodTask SR (%)All-success (%)UA SAR (%)UC SAR (%)
SFT55.120.092.371.9
(i) GRPO (task rewards, within-user normalization)58.425.894.573.2
(ii) GRPO (subgoal rewards, within-user normalization)60.923.390.976.7
RePAIR (subgoal rewards, cross-user normalization)64.028.391.078.8

(Continued in the next comment)

AuthorsOpenReview ↗

Author Response [2/6]

These matched comparisons clarify the contributions of subgoal rewards and cross-user normalization beyond additional RL training. Subgoal rewards improve Task SR by 2.5 percentage points and UC SAR by 3.5 points over task-reward GRPO, supporting the benefit of finer-grained supervision for user-conditioned behavior. However, All-success decreases by 2.5 points, indicating that subgoal supervision alone does not improve success consistently across users. The direct comparison between RePAIR and control (ii) isolates the contribution of cross-user normalization: with the same subgoal rewards, training users, user-state exposure, and total rollout budget, RePAIR improves Task SR by 3.1 points, All-success by 5.0 points, and UC SAR by 2.1 points, while UA SAR remains nearly unchanged. These results support the benefit of cross-user reward comparisons beyond additional RL training, subgoal supervision, or greater user-state exposure under this matched setup.

We have added the matched RL baselines to Table 2, analyzed their results in Section 6.2, and documented the matched training setup in Appendix D in the revised manuscript.


AuthorsOpenReview ↗

Author Response [3/6]

[W2, Q2] Target position and performance differences across profiles

W2) I also had trouble interpreting the SAR gap as evidence about the effect of personalized content. Many shared contexts involve fixed navigation controls, while personalized contexts require a search for the target, so the comparison includes different operations (Figure 5 / Appendix B.3). SAR also counts only subgoals the agent reaches, which means the cases included in each average depend on earlier failures (Appendix E). The shuffled lists may put the same target at different depths for different profiles as well (Appendix B.1). The results demonstrate variation across app states, but I could not tell which differences account for it. A comparison of the same operation at a fixed target position, with the number of reached and successful subgoals reported separately, would help explain the gap. Q2) How much does target position explain the performance differences across profiles? Could the authors compare the same subgoal with the target initially visible versus when it requires scrolling, and report how often each agent reaches and completes that subgoal?

Thank you for the question. We clarify that UI personalization in PAIR refers to user-dependent variations in application states, encompassing not only the semantic content of UI elements but also their arrangement, visibility, and navigation requirements. These variations are consistent with the naturally personalized environments described in prior studies [r1, r2], which PAIR is designed to evaluate. Accordingly, the UA-UC SAR gap reflects the challenges associated with these varying UI states, rather than the isolated effect of content semantics.

Regarding the SAR denominator, since GUI tasks consist of sequential subgoals, counting unreached subgoals as failures would incorrectly attribute earlier failures to subgoal-level performance. We therefore compute SAR over reached subgoals, whereas Task SR measures end-to-end task success.

As for target position, Finding 3 in Section 6.2 already identifies target visibility as an important factor contributing to failures in the user-conditioned context, showing more incorrect-item selections before the intended target becomes visible.

In addition, following the reviewer's suggestion, we further examine target position through two analyses.

(A) Target position across user profiles in PAIR

We examine how target position explains performance differences across profiles in PAIR. Specifically, for each reached subgoal included in SAR, we measure the ground-truth number of swipes required to reveal its target and report the distribution across 0, 1, and 2+ swipes. All results are aggregated across the six evaluated GUI agents.

ProfileReached User-agnostic ContextReached User-conditioned ContextUA−UC SAR (pp)Task SR (%)
NSwipes to Reach Target (%) (0 / 1 / 2+)SAR (%)NSwipes to Reach Target (%) (0 / 1 / 2+)SAR (%)
p0354100 / 0 / 081.691162.2 / 26.9 / 10.967.813.849.9
p1357100 / 0 / 079.389461.3 / 30.0 / 8.769.89.551.7
p2362100 / 0 / 081.591359.7 / 24.0 / 16.371.110.453.1
p3367100 / 0 / 082.391356.7 / 26.4 / 16.970.312.052.4
p4357100 / 0 / 082.989254.6 / 29.9 / 15.565.117.8 (Max)47.9
p5355100 / 0 / 081.791057.5 / 28.1 / 14.468.613.150.7
p6357100 / 0 / 080.190155.3 / 31.3 / 13.466.513.647.5
p7349100 / 0 / 079.488351.5 / 27.2 / 21.362.916.543.8
p8362100 / 0 / 079.091357.2 / 27.2 / 15.771.77.3 (Min)52.8
p9356100 / 0 / 077.890560.0 / 24.6 / 15.470.17.850.7
Overall357.6 ± 5.0100 / 0 / 080.6 ± 1.7903.5 ± 10.657.6 / 27.6 / 14.968.4 ± 2.812.2 ± 3.550.0 ± 2.9

The results show that UC SAR varies more across profiles than UA SAR (8.8 vs. 5.1 pp), with UC SAR consistently lower across all profiles. While UC targets often require more scrolling, their depth alone does not fully explain the observed SAR gaps. For example, p4 has the largest UA–UC SAR gap (17.8 pp), although only 15.5% of its targets require two or more swipes, compared with 21.3% for p7. Meanwhile, p8 has the smallest gap (7.3 pp), despite 15.7% of its targets requiring two or more swipes. These results suggest that multiple aspects of user-dependent UI states jointly shape task difficulty, consistent with PAIR's evaluation objective.

(Continued in the next comment)

Reviewer_JgRDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the reliability of mobile GUI agents across user-specific application states. PAIR constructs executable states from user profiles and evaluates common or personal targets under explicit or implicit instructions. The paper also proposes RePAIR, which uses GRPO with subgoal rewards normalized across rollouts from different users.

Critical strengths
  1. The benchmark provides a useful setting for evaluating GUI tasks across different user-specific application states.
  2. The subgoal and target-exposure analyses provide useful insights into where agents fail.
Critical weaknesses
  1. Some benchmark tasks retain unresolved review flags, and implicit instructions may not uniquely identify the target (Appendix B.2). The extent of this issue is not quantified, so some reported failures may reflect task ambiguity rather than agent unreliability.
  2. RePAIR is mainly compared with SFT, while the persona-pool ablation also changes data diversity. Without a matched RL baseline, the benefit of cross-user rewards cannot be separated from that of additional RL training.
Questions for authors
  1. How many test instances have unresolved review flags or ambiguous target descriptions? Do the main findings hold on a subset whose instructions and targets have been validated against the rendered interfaces?
  2. On the same multi-user training data and at a matched training budget, how does RePAIR compare with task-reward GRPO and with subgoal rewards normalized within each user?
  3. Appendix D states that baseline outcomes on the evaluation personas informed training-task selection. Can the improvements be checked on users whose outcomes did not inform this selection?
6 comments
AuthorsOpenReview ↗

Author Response [1/3]

Dear Reviewer JgRD,

Thank you for your constructive review. We appreciate your recognition of PAIR's value in evaluating GUI agents across user-specific application states, and of our subgoal and target-exposure analyses as delivering insights into where agents fail. Below, we address your concerns about benchmark validity, matched RL baselines, and evaluation independence. We have also uploaded a revised manuscript reflecting our rebuttal, with all changes highlighted in blue.


[W1, Q1] Benchmark validity and implicit-instruction ambiguity

W1) Some benchmark tasks retain unresolved review flags, and implicit instructions may not uniquely identify the target (Appendix B.2). The extent of this issue is not quantified, so some reported failures may reflect task ambiguity rather than agent unreliability. Q1) How many test instances have unresolved review flags or ambiguous target descriptions? Do the main findings hold on a subset whose instructions and targets have been validated against the rendered interfaces?

We apologize for the unclear description in Appendix B.2, which may have led to a misunderstanding about the status of the review flags. Our statement that “some benchmark tasks retain unresolved review flags, and implicit instructions may not uniquely identify the target” referred to potential issues identified by automated review, not confirmed ambiguities in the benchmark. Before running the benchmark, a human annotator manually reviewed all flagged instructions. Using the instructions and the rendered interfaces, we confirmed that each instruction uniquely identified its intended target.

Specifically, the automated review flagged 14 implicit instructions (1.2% of 1,200 instances), but manual verification confirmed that all 14 uniquely identified their intended targets. The submitted Appendix B.2 reported the automated flags as “unresolved” and omitted our manual verification results.

Importantly, excluding these instances leaves our main findings unchanged. We have additionally evaluated the benchmark after excluding the 14 instances. The best-worst persona gap in Task SR remained nearly identical (9.32 vs. 9.31 pp on the full benchmark), as did the mean user-conditioned SAR drop (11.9 vs. 12.1 pp). Personal targets remained more difficult than common targets (4.4 pp lower SAR), and incorrect-item selections remained more frequent before target exposure for both target types.

We have revised Appendix B.2 to document the manual verification of all 14 flagged instructions and the findings after excluding those instances in the revised manuscript.


AuthorsOpenReview ↗

Author Response [2/3]

[Q3] Training-task selection using evaluation-user outcomes

Q3) Appendix D states that baseline outcomes on the evaluation personas informed training-task selection. Can the improvements be checked on users whose outcomes did not inform this selection?

We thank the reviewer for raising the concern about potential selection bias. We originally selected tasks with mixed success and failure outcomes across users to highlight cross-user performance differences.

To address the reviewer's concern, we additionally evaluated RePAIR using training tasks selected solely from training-user outcomes (p10–p17), without using evaluation-user outcomes. Both variants use the same eight training users, SFT initialization, and training budget (31 task specifications, 155 updates), with evaluation on p0–p9 at iteration 5.

Task selectionTask SR (%)All-success (%)User-agnostic SAR (%)User-conditioned SAR (%)
SFT55.120.092.371.9
Evaluation-persona-informed64.527.594.377.7
Training-user only64.028.391.078.8

RePAIR maintains comparable performance without using evaluation-user outcomes for task selection, achieving similar Task SR (64.0% vs. 64.5%) and slightly higher UC SAR (78.8% vs. 77.7%) and All-success (28.3% vs. 27.5%).

These results suggest that RePAIR's improvements do not rely on evaluation-user outcomes to select training tasks. In the revised manuscript, we clarify training-user-only task selection in Section 6.1, update the main RePAIR results in Section 6.2, and detail the selection procedure in Appendix D.


AuthorsOpenReview ↗

Author Response [3/3]

[W2, Q2] Matched RL baselines and cross-user reward contribution

W2) RePAIR is mainly compared with SFT, while the persona-pool ablation also changes data diversity. Without a matched RL baseline, the benefit of cross-user rewards cannot be separated from that of additional RL training. Q2) On the same multi-user training data and at a matched training budget, how does RePAIR compare with task-reward GRPO and with subgoal rewards normalized within each user.

Thank you for suggesting these additional comparisons. Following the reviewer's suggestion, we have conducted matched RL comparisons to isolate the contributions of subgoal rewards and cross-user reward normalization. Specifically, we start with GRPO using binary task-success rewards and within-user normalization, then incrementally introduce each RePAIR component.

For these comparisons, we adopt the training-user-only task selection described in our response to [Q3], ensuring that evaluation-user outcomes do not influence task selection. All RL methods use the same eight training users, 31 tasks, SFT initialization, and training budget, with evaluation on p0 to p9 at iteration 5. To ensure a fair comparison, we also keep user-state exposure identical across methods by using the same two-user rollout configuration (two rollouts per user), such that only the reward formulation and normalization scheme differ.

MethodTask SR (%)All-success (%)User-agnostic SAR (%)User-conditioned SAR (%)
SFT55.120.092.371.9
GRPO (task rewards, within-user normalization)58.425.894.573.2
+ subgoal rewards60.923.390.976.7
+ Cross-user normalization (RePAIR)64.028.391.078.8

These results suggest complementary roles for subgoal rewards and cross-user normalization. Subgoal rewards provide finer-grained feedback for learning user-conditioned behavior, improving Task SR by 2.5 percentage points and user-conditioned SAR by 3.5 points over task-reward GRPO. However, All-success decreases by 2.5 points, suggesting that these improvements alone do not translate into more consistent success across users. Cross-user normalization helps translate subgoal-level improvements into broader success across users: with the subgoal reward formulation held fixed, it increases Task SR by 3.1 points, All-success by 5.0 points, and user-conditioned SAR by 2.1 points, while leaving user-agnostic SAR nearly unchanged. Thus, subgoal rewards improve the learning signal for user-conditioned subgoal achievement, while cross-user normalization contributes additional gains in both user-conditioned performance and success across users under the matched training setup.

We have added the matched RL baselines to Table 2, analyzed their results in Section 6.2, and documented the matched training setup in Appendix D in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.