acceptodds
Under review as a conference paper at ICLR 2027

SAIL: Learning What to Personalize from Sparse GUI Interaction History

Abstract

Sparse interaction history is an ambiguous basis for personalization because observed actions may reflect the task, interface, or temporary context rather than transferable user tendencies. We introduce SAIL, a population relative framework for cold start GUI agents that compares historical behavior with plausible alternatives in the same context. SAIL aggregates these deviations into a compact probabilistic user state and applies query conditioned residual corrections to Population predictions. On FingerTip 20K, SAIL improves User Macro Sim1 by 7.48 points on average over the strongest prior history baselines and retains a 1.61 point average gain under a strict control matched for observable support, active capacity, and training budget. It also reaches 43.25% GUI execution success and retains personalization gains on held out users and Persona2Web. Controlled behavioral and adverse history tests show that task relevance alone does not determine history value: transferable user conditioned patterns help, while forced, misleading, or wrong user histories can still cause negative transfer. Overall, population relative modeling provides an effective way to extract transferable user information from sparse GUI interaction history.

Primary area:
foundation or frontier models, including LLMs
Keywords:
GUI agents, cold-start personalization, probabilistic user modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nHqHOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  • This work studies the predictive value of historical behavior beyond its original task context in cold-start intent prediction.
  • SAIL combines an observation-conditioned Population reference, Gaussian user-state aggregation, and query-conditioned residual adaptation.
  • This work assesses end-to-end gains, matched-control differences, negative transfer, and generalization to held-out users and Persona2Web.
Critical strengths
  • The motivation is good; the GUI interface can indeed differ depending on the customized content.
  • Provided a detailed experimental setup
Critical weaknesses
  • The method is too difficult to understand. The article contains an excessive number of symbols and formulas, which makes it difficult for readers to grasp the author's core contributions. After reading the methodology section, I still do not understand what the inputs and outputs of SILA are, nor how it differs from a standard UI agent.

  • I am also unclear about how the metrics in Table 1 are calculated. I have examined the original content of the FingerTip 20K dataset, and the calculations used in this paper do not appear to align with those.

  • Minor weakness:

    • I support that this manuscript is better suited for the “ applications to computer vision, audio, language, and other modalities" area than for the “foundation or frontier models, including LLMs” area.
Questions for authors
  • Please refer to the weaknesses.
5 comments

Response to W1: Clarification of SAIL's Inputs, Outputs, and Methodological Contributions

We thank the reviewer for highlighting the importance of methodological clarity. We agree that the overall framework could be introduced more intuitively. However, we would like to clarify that the inputs, outputs, and modeling objectives of SAIL are explicitly defined in Section 3.1, and that SAIL addresses a different stage from conventional instruction-following GUI agents.

1. Inputs and outputs

SAIL takes two inputs:

  • Current GUI context: The current GUI observation, including available screenshots and contextual information such as time, scenario, and sanitized user profile.
  • Sparse interaction history: At most 20 completed interactions, each containing its original context and the user's observed intent.

The output is a natural-language prediction of the user's next intended task, rather than a GUI action sequence or screen coordinates.

Formally, SAIL predicts the target intent conditioned on the current GUI context and sparse interaction history, as defined in Equation (1) of our paper.

2. Difference from conventional GUI agents

Conventional GUI agents generally receive an explicit task instruction and determine how to execute it through interface interactions. In contrast, SAIL addresses the preceding problem: inferring what the user intends to accomplish before an explicit instruction is provided.

For example, when a user is viewing a product listing, a conventional GUI agent may receive the instruction "open product reviews" and execute the corresponding actions. SAIL instead predicts whether the user intends to inspect reviews, compare products, or perform another operation, based on the current GUI context and historical behavior.

The predicted intent can subsequently be passed to a separate GUI executor, but action execution is not the direct output of SAIL.

3. Core methodological contributions

Our contribution is not simply incorporating historical interactions. Rather, SAIL investigates how to distinguish transferable user-conditioned information from behaviors driven or constrained by the original task and interface context.

Specifically, SAIL combines three components:

  • Population-relative representation: Historical behavior is compared with plausible alternatives generated by a frozen, task-adapted Population predictor under the same historical GUI context.
  • Probabilistic user-state aggregation: Context-relative behavioral deviations are aggregated into a compact user representation through learned Gaussian factors.
  • Query-conditioned residual adaptation: The inferred user state modifies the Population prediction according to the current GUI query through token-level residual correction.

Importantly, the Population predictor is separately fine-tuned for intent prediction. SAIL exactly recovers its predictions when no history is available and introduces history-conditioned adjustments when support interactions are provided.

Therefore, the central contribution is modeling which aspects of historical behavior remain predictive of future user intent beyond their original task context, rather than directly learning GUI execution actions.

We appreciate the reviewer's feedback on presentation and will improve the methodological overview with a concise input-output example and a clearer explanation of the three-stage framework. We hope these clarifications make the research objective and technical contributions easier to assess.

Response to W2: Clarification of Evaluation Metrics and Consistency with FingerTip-20K

We thank the reviewer for examining the original FingerTip-20K benchmark. We would like to clarify that our work does not redefine the primary intent-similarity metric. Instead, we follow the original benchmark's query-level Sim1 definition while adopting user-balanced aggregation for personalized intent prediction.

The corresponding metric definitions and aggregation procedures are already specified in Section 4.1 (Metrics and Evaluation) and Appendix A.1 (Evaluation Protocol, Baselines, and Metrics).

1. Consistency of the Sim1 definition

Following the original FingerTip-20K benchmark (Yang et al., ICLR 2026), query-level Sim1 is defined as the average of two similarity measures:

Sim1 = (S1 + S2) / 2

Here, S1 denotes the MiniLM embedding cosine similarity between the predicted and reference intents, while S2 denotes their normalized Levenshtein similarity.

Our evaluation adopts the same query-level formulation without introducing a different similarity measure. Therefore, the underlying mathematical definition of Sim1 is consistent with the original FingerTip-20K benchmark.

2. User-Macro aggregation and its existing description in the manuscript

It is important to distinguish the definition of query-level Sim1 from the aggregation procedure used to report the final evaluation results.

As specified in Section 4.1 (Metrics and Evaluation), our primary evaluation contains 1,633 queries from 83 users. Since different users may contribute different numbers of evaluation queries, directly averaging all queries would assign greater weight to users with more observations.

We therefore adopt User-Macro Sim1: we first average query-level Sim1 within each user and then average these user-level scores with equal weight. The resulting value is multiplied by 100 to obtain the point scale reported in Table 1.

This procedure is further detailed in Appendix A.1 under "Intent metrics", where we explicitly distinguish User-Macro Sim1 from Query-Micro Sim1. The former assigns equal weight to users, whereas the latter assigns equal weight to individual queries.

Therefore, our evaluation preserves the original FingerTip-20K query-level Sim1 definition while employing User-Macro aggregation to ensure equal representation of users in personalized evaluation. This is a distinction in statistical aggregation rather than a redefinition of the benchmark metric.

We will make these existing explanations more prominent in the revised manuscript to avoid potential ambiguity between query-level scoring and user-level aggregation.

We hope this clarification resolves the concern regarding the consistency of our evaluation metrics with the original FingerTip-20K benchmark.

Response to the Area Recommendation and Overall Assessment

We thank the reviewer for the suggestion regarding the submission area.

We agree that SAIL is closely related to multimodal GUI understanding and personalized agent systems. However, we would like to clarify that our primary research focus is on understanding and modeling agent behavior, with GUI agents serving as a concrete research setting rather than the sole scope of the proposed methodology.

More specifically, our work investigates a general challenge in agent systems: how an agent can infer user intent from sparse interaction history while distinguishing transferable user-specific signals from behaviors driven by task context. This challenge extends beyond GUI interaction and is relevant to broader agent scenarios involving implicit intent understanding, historical context modeling, and adaptive decision-making.

To address this challenge, SAIL develops a population-relative modeling framework combining context-conditioned reference construction, probabilistic user-state inference, and query-conditioned residual adaptation. These components offer a general modeling perspective for interpreting historical interactions and adapting pretrained models to user-specific contexts.

Although our current experiments primarily focus on GUI agents, the underlying formulation is not inherently restricted to particular GUI actions or interface structures. We therefore view GUI intent prediction as a concrete setting for studying a broader problem in agent understanding and adaptation, rather than the sole intended scope of the approach.

This methodological motivation is also an important reason why we selected the "Foundation or Frontier Models, including LLMs" area. Our goal is not merely to develop an application-specific GUI agent system, but to investigate how foundation-model-based agents can represent and interpret sparse interaction histories and incorporate transferable user information into subsequent predictions.

Regarding the overall assessment, we hope that our clarifications of SAIL's inputs, outputs, methodological contributions, and evaluation protocol will help address the reviewer's concerns and make the technical contribution clearer.

We appreciate the feedback on presentation and will improve the manuscript accordingly. We respectfully invite the reviewer to reconsider the overall recommendation in light of these clarifications.

Thank you again for your time and consideration.

Reviewer_4ZTWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies personalization for proactive GUI agents: with no explicit instruction, it predicts what the user wants to do next from only the current screen and a very small interaction history. The core claim is that "what a user did is not the same as their preference"—the method treats the deviation of the user's behavior from a reference model of "what an average person would do" as the user signal, and learns when to enable and when to suppress personalization. On FingerTip-20K, as well as unseen-user and cross-domain transfer settings, the authors report improvements over a range of baselines.

Critical strengths
  1. The problem perspective is valuable. "Historical behavior does not equal preference and should be interpreted relative to the population" is a point often overlooked in personalized agents, and organizing the method around it is a reasonable direction.
  2. Correct derivations and practical engineering properties. The closed-form posterior and incremental-update derivations are sound; with no history the model exactly reduces to the population model, and online cost does not grow with history length.
Critical weaknesses
  1. The "learn what to personalize" mechanism is structurally inert, and the gain under fair controls is only about 1–2 points. The derivations are algebraically correct; the problems lie in the objective and architecture. The support loss is an in-sample self-fitting objective over the same history, whose fitting target is itself produced by learnable networks—shrinking residuals and increasing precision both reduce the loss, so the per-interaction precision is pushed toward its upper bound. Posterior concentration does not depend on whether the residuals across interactions are consistent, so once precision saturates, the global strength is effectively a fixed function of the number of history items (the values in Figures 6 and 15 quantitatively match this). The residual is normalized to a fixed magnitude before being added, so the query gate can only redirect the correction, not weaken it. The model therefore lacks a direct mechanism for attenuating personalization by context, and the corresponding ablation gaps are all small. At the same time, the +7.48 in the abstract comes from comparison with untrained prompting methods that see only textual history; under the authors' own matched controls the gain is only about 1.6 points, and swapping in another user's history costs only 4–5 points—the claim of "learning user-specific information from sparse history" is not supported.

  2. The reported numbers and training process are mutually inconsistent and currently hard to trust. Under the same setting (4 history items, replaced with another user's task-matched history), Table 12 shows about +6.8 points over the population model while Table 26 shows −1.18 points; yet by the model's definition the global strength is computed from the history alone and cannot tell another user's history apart, whereas Table 26 reports a clear drop for it. In Table 1, the general-purpose models' means are all exact multiples of 0.05; the population model's success rate when executing predicted intents is identical to the original benchmark's result when executing ground-truth intents. The appendix training curves present the same set of runs three times, the training losses of the ablation variants nearly coincide—which does not square with the sizable evaluation gaps reported in the main text—and the counts of the data splits do not add up to the described evaluation set. None of this is necessarily intentional error, but the conclusions cannot be verified until the authors provide the underlying data.

Questions for authors

Clarifications needed

  1. Why do Table 12 and Table 26 point in opposite directions under the same setting? Please provide the query sets and per-query results for both.
  2. Please provide the raw per-run scores behind Table 1, and explain why the general-purpose models' means are all multiples of 0.05 and their standard deviations follow an identical pattern.
  3. In the execution experiments, how is "success" judged (relative to the user's true task, or relative to the predicted intent)? By whom? Why does the population model's result match the original benchmark's result with ground-truth intents?
  4. The support loss pushes precision close to its upper bound (Table 21: adding this loss raises mean precision from 31.8 to 88.6, out of 100). What happens to precision, global strength, and results under a variational objective with a KL term or under marginal likelihood? Under the current design, how can the global strength distinguish the user's own history from another user's?
  5. Please provide per-user counts for each data split, and the overlap between the SAIL and population training sets; and add the frozen population model's loss to the training curves as a reference.
  6. If the population model were replaced by one that is not weaker than zero-shot UI-TARS-1.5-7B, how much of SAIL's gain over the matched controls would remain?

Suggestions

  • Discuss direct precedents such as Bayesian Context Aggregation (Volpp et al., ICLR 2021) and DEP (EMNLP 2025), and clarify the novelty of the method components.
  • Report the original benchmark's main metric SR1 to rule out that the similarity score rewards mimicking the user's phrasing.
9 comments

Response to W1: Effectiveness of Population-Relative Personalization

We thank the reviewer for the detailed analysis of our objective and architecture. We would like to clarify how SAIL learns history-dependent personalization strength, applies query-conditioned corrections, and extracts user-specific predictive information. Our theoretical formulation and controlled experiments provide direct evidence for the effectiveness of these mechanisms.

1. Precision learning and history-dependent adjustment.

The support objective encourages the inferred user state to explain historical residuals. Table 21 shows that it increases mean precision from 31.8 to 88.6, reflecting greater concentration of learned Gaussian factors.

Global personalization strength depends on interaction-specific precision and observability:

\begin{equation} \Lambda_{u,j}=\lambda_0+\sum_i\lambda_{i,j}a_{i,j}^{2},\qquad \alpha_u=1-\exp(-I_u). \end{equation}

Since λi\lambda_i and aia_i are learned from historical observations, adjustment strength depends on support content as well as history count.

Table 23 provides direct empirical evidence. With the support budget fixed at S=4S=4, the mean global strengths are 0.82, 0.76, 0.29, and 0.34 across four controlled behavioral conditions. In particular, environment-forced behavior produces a substantially lower adjustment strength than transferable user-choice patterns.

Table 3 also evaluates Count Gate Only, αu=Su/(1+Su)\alpha_u=S_u/(1+S_u). At S=1S=1, SAIL achieves 62.35 Sim1 versus 61.11, a +1.24-point gain (95% CI [+0.64, +1.84], Holm-adjusted p=0.004p=0.004).

These results support history-conditioned adjustment beyond support-count scaling.

2. Query-conditioned correction and suppression of negative transfer.

SAIL separates the global strength αu\alpha_u from the query-conditioned dimension gate gqg_q. The former controls the overall correction scale, while the latter determines how the inferred user representation contributes to the current prediction.

Although RMS calibration normalizes the residual logits, query-conditioned gating and token-level residual interaction still modify the direction and structure of the correction, thereby influencing application, action, and target predictions.

At S=4S=4, Table 21 shows that adaptive scaling improves Sim1 from 61.95 to 62.43 and reduces negative-transfer rate from 18.1% to 14.6% relative to fixed mean strength.

Under task-matched wrong-user history, Table 26 shows that adaptive strength improves the Population-relative difference from −2.76 to −1.18 points and reduces negative transfer from 53.1% to 39.4%.

Together, these results show that history-dependent strength mitigates, though does not eliminate, harmful personalization.

3. Population-relative modeling under strictly matched controls.

We distinguish the system-level improvement from the contribution of the Population-relative mechanism, as already specified in the manuscript.

The reported +7.48-point gain measures end-to-end advantage over prior history-based methods. MHR-ActiveCap instead matches observable support, active parameter count, and training budget.

Under these matched conditions, SAIL gains +2.27, +1.44, and +1.13 Sim1 points at S=1,2,4S=1,2,4, averaging +1.61 points.

Table 19 further examines the role of representation scale. At S=1S=1, Population-relative representation achieves 61.68 Sim1, compared with 60.96 for scale-matched observations and 60.08 for raw observations. This demonstrates that the benefit of context-relative representation persists after matching global input scale.

These experiments provide complementary evidence under controlled adaptation settings.

4. Evidence for learning user-specific predictive information.

Table 12 directly evaluates whether task relevance alone can substitute for the target user's history. Replacing correct-user support with task-matched other-user history reduces Sim1 by 3.91, 4.05, and 4.86 points at S=1,2,4S=1,2,4, respectively.

Table 23 further distinguishes transferable user tendencies from environmental constraints.

SAIL improves over Population by +7.6 points when user rules vary in a fixed environment and +5.5 points when a user rule transfers across environments, versus approximately 0.0 for environment-forced actions.

These results indicate that SAIL captures transferable user-specific information beyond task similarity or generic prediction errors.

In summary, SAIL combines Population-relative representations, Gaussian aggregation, and query-conditioned residual correction to model user-specific patterns from sparse history. Matched controls, adaptive-strength ablations, and behavioral diagnostics jointly support these components.

We will clarify component roles and distinguish system-level from mechanism-level gains in the revised manuscript.

Response to W2: Clarification of Experimental Consistency and Reported Results (1)

We thank the reviewer for the detailed examination of our experimental results. We provide the following clarifications and supporting evidence regarding the reported comparisons, evaluation protocols, and training procedures.

1. Clarification of Tables 12 and 26.

Tables 12 and 26 examine different aspects of history-conditioned personalization. Table 12 evaluates the predictive value of correct-user versus task-matched wrong-user history, reporting 62.29 versus 57.43 User-Macro Sim1 at S=4S=4, a difference of 4.86 points.

The reviewer's inferred +6.82-point gain is obtained by subtracting the full-evaluation Population mean of 50.61 from the wrong-user score of 57.43. However, Table 12 uses the subset with valid task-matched replacements, whereas 50.61 is the Population average over the complete evaluation set. This subtraction therefore does not constitute a matched-subset comparison.

Table 26 instead reports Population-relative changes under diagnostic history-stress conditions, including a −1.18-point change for wrong-user history, using the corresponding Population predictions and query-level aggregation.

We further clarify the experimental construction, evaluation subsets, and per-query comparisons in Response to Q1.

2. History-dependent global adjustment strength.

The global adjustment strength depends on interaction-specific precision and observability, both derived from historical observations, rather than support count alone.

Table 23 provides direct evidence: with the same support budget S=4S=4, four controlled behavioral conditions produce mean adjustment strengths of 0.82, 0.76, 0.29, and 0.34. Table 26 further reports strengths of 0.84 for informative correct-user history and 0.49 for task-matched wrong-user history.

These results demonstrate that the learned adjustment strength responds to historical content. The mechanism and its interpretation are further discussed in Response to Q4.

3. Correction and re-evaluation of Table 1.

After re-examining the original evaluation pipeline, we identified an error in the general-purpose model evaluation script that affected the reported results. We apologize for this oversight.

We have corrected the evaluation procedure and conducted a new evaluation using seven currently accessible models, as several historical API versions are no longer available.

The complete updated results, model replacements, and corresponding evaluation details are provided in Response to Q2. This correction concerns the external general-purpose model comparison and does not alter how the separately reported SAIL–MHR-ActiveCap matched-control differences are calculated.

4. Population model and GUI execution results.

The Population model is a task-specific intent predictor obtained by fine-tuning UI-TARS-7B-SFT on FingerTip-20K, rather than the original off-the-shelf GUI executor.

In our execution framework, each method generates a predicted intent that is passed to a shared fixed executor, whereas the original benchmark supplies ground-truth task instructions directly.

The numerical equality of the two reported 38.50% aggregate success rates does not, by itself, establish identical execution outcomes. The success criterion, evaluation procedure, and task-level evidence are addressed in Response to Q3.

Response to W2: Clarification of Experimental Consistency and Reported Results (2)

5. Training curves and ablation performance.

We would like to clarify that Figures 8–10 are complementary visualizations of the same representative training runs, rather than three independent sets of experiments. Figure 8 presents the total training objective, Figure 9 presents the query-specific objective for the same four variants (SAIL, Composite Mean-State, w/o Dimension Gate, and w/o Population-relative), and Figure 10 overlays their trajectories for direct comparison.

Regarding the apparent similarity of their loss curves, three factors are particularly relevant.

First, the visualization scale makes differences between training trajectories less apparent. The figures cover the complete optimization process over 17,800 microsteps (4,450 optimizer updates). The total-objective plots span approximately 3–7, while the query-objective plots cover approximately 1.5–2.2. Consequently, relatively small differences between variants, particularly during later training stages, are visually compressed when displayed over these ranges. Figure 10 additionally applies a 256-step rolling median, further smoothing short-term fluctuations. Thus, visually overlapping trajectories do not necessarily indicate identical underlying losses.

Second, the training objective is optimized across multiple support budgets, whereas the largest ablation differences occur specifically under sparse history. During training, support sizes are sampled from S∈{0,1,2,4,8,16,20}S\in\{0,1,2,4,8,16,20\}, so the aggregate loss trajectories summarize optimization across different history conditions. In contrast, the largest evaluation differences arise at S=1S=1. For example, removing Population-relative subtraction decreases Sim1 by 6.80 points at S=1S=1, while the corresponding differences are 3.95 and 0.58 points at S=2S=2 and S=4S=4. Therefore, similarities in the overall training trajectories do not directly reflect the performance differences within individual support regimes.

Third, training loss and downstream evaluation measure different properties. The training objective combines query-level negative log-likelihood, support-state supervision, and residual-direction supervision, whereas the evaluation metric measures semantic and edit similarity between freely generated intents and reference user intents. Moreover, all four variants share the same frozen Population backbone and comparable optimization budgets, which naturally leads to similar overall optimization tendencies. Nevertheless, differences in the learned personalization pathways can alter the generated application, action, or target entity, resulting in different Sim1 outcomes even when the aggregate teacher-forced losses remain close.

Accordingly, the training curves characterize optimization behavior, while the ablation evaluations measure generation-level predictive effectiveness. We will clarify this distinction in the revised manuscript and make the visualization scale and relationship between the training diagnostics and evaluation results more explicit.

6. Data partitioning and episode accounting.

Population development and the primary SAIL evaluation use distinct data-partitioning protocols.

The auxiliary Population split contains 14,417 training episodes, 2,439 validation episodes, and 3,144 test episodes. Within validation, 200 earlier interactions are reserved for historical-support construction, leaving 2,239 checkpoint-selection queries. Thus,

20,000=14,417+200+2,239+3,144.20,000=14,417+200+2,239+3,144.

The primary SAIL evaluation separately follows a within-user chronological protocol with 1,633 held-out queries. Appendix A.2, Table 5, reports zero evaluation-episode overlap and zero future-data exposure for the upstream datasets covered by its audit.

The relationship between Population and SAIL training sets, per-user counts, and the frozen Population loss reference is explained further in Response to Q5.

Summary.

These clarifications distinguish the experimental quantities, model roles, and training stages underlying the reported results. We have corrected the identified general-purpose model evaluation issue and provide the corresponding detailed explanations and supporting evidence in our responses to Q1–Q5.

Reviewer_BmgVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAIL predicts user intent for GUI agents from the current context and up to 20 earlier interactions. It compares each historical intent with an observation-conditioned reference built from a frozen Population predictor. Learned Gaussian factors aggregate these differences into a user state, which a query-conditioned residual head uses to adjust Population logits. Posterior concentration controls the overall correction strength. With no history, SAIL recovers Population predictions.

The main evaluation uses 1,633 queries from 83 FingerTip-20K users. At S = 1, 2, 4, SAIL improves User-Macro Sim1 by an average of 7.48 points over the strongest prior history baselines and 1.61 points over MHR-ActiveCap, which matches observable support, active parameter count and training budget. The paper also reports GUI execution, held-out-user evaluation, Persona2Web transfer and history diagnostics.

Critical strengths
  1. The controls support the population-relative reference. A historical intent can reflect task and interface constraints as well as user preferences. Comparing it with context-matched alternatives is a sensible way to account for this. The reference weights use both Population likelihood and compatibility with the observed historical intent. In Table 20, this gives 62.35 Sim1 at S = 1, compared with 61.71 for compatibility alone, the stronger of the two single-cue alternatives.

Table 17 is especially useful: at S = 1, the benefit of subtraction is 6.80 points when the precision network receives the reference and 1.60 points when it does not. The effect therefore depends on the precision configuration. Subtraction also retains an advantage after matching global input RMS (61.68 versus 60.96 at S = 1 in Table 19). Finally, the target user's history outperforms task-matched history from another user by 3.91-4.86 points at S = 1, 2, 4 (Table 12), supporting the use of user-specific information under this matching protocol.

  1. The paper tests when personalization hurts. In Table 26, history-dependent correction strength reduces NTR by 8.7-15.4 percentage points across Population-typical, repeated, conflicting and wrong-user histories, with higher mean Sim1 in all four conditions. This comparison fixes the inferred state, dimension gate and residual head, changing only the global strength. The result also holds on natural queries at S = 4: NTR falls from 18.1% to 14.6% (Table 21). Table 25 fixes the predictions when comparing selection scores, so it tests ranking separately from prediction quality. These are useful checks given the risk of applying misleading history. Harmful corrections remain, and the authors acknowledge that posterior concentration is not calibrated reliability.
Critical weaknesses
  1. Deterministic Residual needs a clearer specification. Section 4.1 and Appendix A.1 describe it only at a high level. I could not determine its history modalities, aggregation rule, correction head or training configuration. This matters because it is only 0.51 Sim1 points behind SAIL at S = 4 and 0.37 points behind at S = 8 (Table 1). Without knowing what is matched, it is difficult to interpret this comparison or reproduce it.

Table 16 does provide trained ablations: Mean Aggregation Only, Count Gate Only and Fixed Variance Input lose 1.47, 1.24 and 0.63 one-shot points, respectively. Composite Mean-State loses 6.30 points, but changes all three together. Appendix B.3 also makes clear that these controls retain learned modules; Composite Mean-State keeps learned precision and observability in its support objective. MHR and MHR-ActiveCap retain Gaussian aggregation. These experiments establish useful component effects, but leave the separate comparison with Deterministic Residual under-specified. Its inputs and training conditions need to be stated before that comparison can tell us what the full state construction adds.

  1. The execution gains need confidence intervals. On the same 400 tasks, SAIL reaches 43.25% SR2, compared with 42.00% for MHR and 40.75% for Retrieve (Tables 2 and 13). These are 173, 168 and 163 successful tasks, respectively. Appendix C.3 describes the paired bootstrap estimator but does not report the resulting intervals. The net gains of five and ten tasks are difficult to assess without the paired outcomes or uncertainty estimates.

Persona2Web provides some additional evidence, but its success difference over MHR is also uncertain: +2.0 percentage points, with a 95% interval of [-1.8, +5.8]. The Pweb and Ppref differences have positive intervals of [+0.3, +2.7] and [+0.2, +3.0] (Appendix C.4). I find the evidence for improved intent prediction and personalization stronger than the evidence for improved task completion.

Two smaller concerns:

(a) The benefit of additional history is less clear than the one-shot result. Sim1 rises from 62.35 at S = 1 to 63.85 at S = 20, a gain of 1.50 points. Figures 12 and 17 show point estimates, and Figure 13 compares methods at fixed budgets; none gives a paired interval for this cross-budget difference. The separate training-run standard deviations do not supply that interval. One interaction already recovers 88.7% of the improvement over Population at S = 20, which is a good cold-start result. The remaining question is how precisely the additional-history gain is estimated.

(b) The headline gain includes a modality difference. SAIL encodes historical screenshots, while the external-adaptation pipeline mainly uses textual fields (Appendix A.1). The 7.48-point system gain therefore cannot be attributed solely to the modeling choices. For the population-relative construction, I give more weight to the 1.61-point comparison with MHR-ActiveCap, which matches observable support, active parameter count and training budget. The paper does not isolate the contribution of historical screenshots.

Questions for authors

(1) What history modalities, aggregation rule, correction head, objectives, trainable parameter count and training budget does Deterministic Residual use? Which are matched to SAIL? Please also report paired 95% confidence intervals for the SAIL-minus-Deterministic differences at S = 1, 2, 4, distinguishing evaluation-user uncertainty from training-run variation.

(2) Please report the paired-task 95% confidence intervals for the SR2 gains over MHR and Retrieve, using Appendix C.3's estimator. For each comparison, how many tasks succeed only with SAIL, and how many succeed only with the baseline?

(3) What is the paired, user-clustered 95% confidence interval for the S = 20 minus S = 1 User-Macro Sim1 difference of 1.50 points? Please preserve query pairing across budgets, state how predictions from the three training runs are combined, and report between-run variation separately.

For future work, I would prioritize comparing the posterior-derived strength with a learned scalar rule, and extending execution evaluation to MHR-ActiveCap and Deterministic Residual. A text-history-only SAIL ablation would also help identify the contribution of historical screenshots. I do not require these extensions for my current recommendation.

5 comments

Response to Q1: Specification and Statistical Comparison of Deterministic Residual

We thank the reviewer for requesting a more explicit specification of Deterministic Residual and its comparison with SAIL. We clarify the baseline construction, matched experimental conditions, and statistical evaluation required to interpret the observed differences.

1. Baseline construction and matched settings.

Deterministic Residual is designed to examine the role of probabilistic user-state aggregation. It retains the population-relative residual representation and downstream residual adaptation pathway, while replacing SAIL's Gaussian posterior aggregation with deterministic mean pooling:

\begin{equation} z_u^{\mathrm{det}}=\frac{1}{S}\sum_{i=1}^{S}\delta_i, \end{equation} where δi\delta_i denotes the residual representation extracted from the ii-th historical interaction.

Both methods use the same chronological support episodes, historical screenshot and text representations, and frozen Population backbone. They also retain the same query-conditioned correction architecture, including the memory projection, dimension gate, and token-level residual prediction head (Eqs. 11–16), as well as the same three training objectives: query prediction loss, support-state supervision, and residual-direction supervision (Eq. 17).

The principal distinction lies in user-state construction: SAIL employs learned precision- and observability-weighted Gaussian aggregation, whereas Deterministic Residual uses arithmetic mean pooling. The downstream correction architecture and corresponding adjustment pathways are retained under the deterministic formulation.

SAIL and Deterministic Residual have 83.11M and 82.30M active trainable parameters, respectively, corresponding to a difference of approximately 0.98%. Both follow the same fixed training budget of 17,800 microsteps and 4,450 optimizer updates, with the Population backbone frozen. Thus, the comparison is conducted under closely matched model capacities, historical inputs, and training conditions.

2. Paired statistical comparison.

Table R1 summarizes the reported User-Macro Sim1 results at S = 1, 2, 4, together with paired 95% confidence intervals for the SAIL-minus-Deterministic differences.

Table R1. SAIL versus Deterministic Residual.

History budgetSAIL (mean ± SD)Deterministic Residual (mean ± SD)DifferencePaired 95% CI
S = 162.35 ± 0.4260.21 ± 0.48+2.14[+0.90, +3.38]
S = 262.05 ± 0.3660.83 ± 0.43+1.22[+0.20, +2.24]
S = 462.43 ± 0.3361.92 ± 0.38+0.51[-0.02, +1.04]

The paired confidence intervals are estimated through user-clustered bootstrap resampling, preserving the same evaluation users and query identities across methods. These intervals characterize uncertainty across evaluation users, whereas the reported standard deviations reflect variation across three independent training runs. The two sources of variability are therefore reported separately.

3. Interpretation.

The reported point estimates consistently favor SAIL across all three history budgets. The paired confidence intervals exclude zero at S = 1 and S = 2, providing stronger evidence of SAIL's advantage under sparse-history conditions. At S = 4, the difference narrows to +0.51 points, with a confidence interval that includes zero, indicating that the advantage is less conclusive as more history becomes available.

These findings support the effectiveness of SAIL's probabilistic user-state construction, particularly in the cold-start regime, while showing that deterministic aggregation becomes increasingly competitive with additional interactions. The separate aggregation, variance-input, and global-strength ablations in Table 16 provide complementary evidence for the contributions of these components.

We will incorporate these baseline specifications and statistical comparisons into the revised manuscript to improve transparency and reproducibility.

Response to Q2: Paired Confidence Intervals for GUI Execution Success

We thank the reviewer for the careful examination of the execution results. We agree that paired-task analysis is important for assessing the uncertainty of the observed SR2 improvements, since aggregate success rates alone do not fully characterize the differences between methods.

1. Paired execution analysis.

As reported in Tables 2 and 13, SAIL achieves 43.25% SR2, compared with 42.00% for MHR and 40.75% for Retrieve, corresponding to improvements of +1.25 and +2.50 percentage points, respectively.

Following the paired bootstrap estimator described in Appendix C.3, we estimate 95% confidence intervals by jointly resampling task-level execution outcomes while preserving the correspondence between methods. The paired execution records also allow us to identify tasks successfully completed by only one of the compared methods.

Table R2. Paired SR2 comparison.

StatisticSAIL vs. MHRSAIL vs. Retrieve
SAIL SR243.25%43.25%
Baseline SR242.00%40.75%
SAIL-only successes510
Baseline-only successes00
SR2 improvement+1.25 pp+2.50 pp
Paired 95% CI[+0.25, +2.50][+1.00, +4.25]

The paired results show that SAIL achieves additional successful executions over both baselines, with no baseline-only successes observed in these comparisons. Both paired confidence intervals exclude zero, providing statistical support for the observed SR2 improvements. Nevertheless, the absolute gains remain modest.

2. Interpretation of execution and personalization results.

We also appreciate the reviewer's distinction between downstream task success and personalization quality. On Persona2Web, SAIL improves the success-rate point estimate over MHR by 2.0 percentage points, with a 95% confidence interval of [-1.8, +5.8]. In contrast, the intervals for Pweb [+0.3, +2.7] and Ppref [+0.2, +3.0] are both positive.

These findings provide stronger and more consistent evidence for personalization improvements across evaluation settings, while the evidence for downstream task completion varies across benchmarks. This distinction is consistent with SAIL's primary objective of extracting user-specific predictive information from limited interaction history.

Task completion additionally depends on action planning, environmental dynamics, and multi-step execution. Accordingly, improvements in personalized intent prediction need not translate proportionally into final execution success.

We will distinguish these levels of evidence more explicitly and incorporate the paired execution statistics into the revised manuscript.

Response to Q3: Paired Evaluation of Additional-History Gains

We thank the reviewer for highlighting the distinction between SAIL's strong one-shot performance and the incremental benefit of additional interaction history. We further clarify the paired cross-budget evaluation protocol, the aggregation of three independent training runs, and the separate estimation of evaluation-user and training-run variability.

1. Cold-start effectiveness.

As reported in Table 1, User-Macro Sim1 increases from 62.35 ± 0.42 at S = 1 to 63.85 ± 0.31 at S = 20, corresponding to an additional improvement of 1.50 points.

Notably, a single historical interaction already recovers 88.7% of the total improvement over Population achieved at S = 20. This demonstrates SAIL's effectiveness in the cold-start setting, where meaningful personalization can be achieved from very limited interaction history.

2. Paired evaluation and aggregation across training runs.

For each independent training run, we preserve identical evaluation users and query identities across S = 1 and S = 20, changing only the amount of available historical support.

We first compute query-level Sim1 scores and aggregate them within each user. The S = 20 minus S = 1 difference is then calculated for each user, maintaining the paired evaluation structure.

Across three independent training runs, aggregation is performed on evaluation scores rather than generated prediction texts. User-Macro Sim1 is computed separately for each run and history budget, and the resulting scores are averaged across runs.

Table R3. Three-run analysis of additional-history gains.

Training runS = 1S = 20Difference
Run 161.8863.52+1.64
Run 262.4863.90+1.42
Run 362.6964.13+1.44
Mean62.3563.85+1.50
Between-run SD0.420.310.12
Paired user-clustered 95% CI——[+1.18, +1.82]

3. Separating evaluation-user uncertainty from training-run variability.

We distinguish two sources of variability. Evaluation-user uncertainty is estimated through paired, user-clustered bootstrap resampling. In each bootstrap replicate, users are sampled with replacement while preserving their paired query observations across history budgets.

The same resampled user indices are applied across all three training runs. For each replicate, the cross-budget difference is computed within each run and subsequently averaged across runs to obtain the bootstrap statistic.

Training-run variability is reported separately using the sample standard deviation of the three independent run-level scores and their corresponding cross-budget differences, as summarized in Table R3.

Accordingly, the paired confidence interval characterizes uncertainty across evaluation users, conditional on the evaluated training runs, whereas the between-run standard deviation measures sensitivity to training randomness. These quantities capture distinct sources of variability and are therefore reported separately.

4. Interpretation of additional-history gains.

SAIL's primary advantage lies in its ability to construct effective personalized representations from very limited interaction history. Additional interactions can further refine these representations, although their marginal contribution becomes smaller once strong one-shot performance has been achieved.

The mean improvement of +1.50 points, together with the paired user-clustered 95% CI of [+1.18, +1.82], supports a positive but relatively modest incremental benefit from additional history. This interval excludes zero, while the separately reported between-run variation provides complementary information about training stability.

We therefore distinguish SAIL's strong cold-start effectiveness from the additional gains achievable with longer histories. We will incorporate the paired evaluation protocol, run-level statistics, and separate uncertainty estimates into the revised manuscript to make this distinction more transparent.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Updates & Reviewer Discussion for the Area Chair

Dear Area Chair,

We sincerely thank you for your time and consideration of our submission. We would like to briefly summarize the key developments during the rebuttal and provide additional context for the overall assessment of our work.

First, we believe SAIL addresses an important and underexplored challenge in personalized agents: historical behavior does not necessarily reflect transferable user preferences, as it may also be shaped by task requirements and environmental constraints. SAIL addresses this challenge through Population-relative modeling, probabilistic user-state inference, and query-conditioned adaptation. Beyond the reported system-level improvements, our matched-control experiments and behavioral diagnostics provide evidence that SAIL captures user-specific predictive information beyond task similarity while mitigating harmful personalization.

Second, we have made substantial efforts to address the reviewers' concerns through additional experiments, methodological explanations, and greater experimental transparency. In response to Reviewer 4ZTW, we identified and corrected an issue affecting the general-purpose model evaluations in Table 1, provided updated results, and made supporting evaluation records available through an anonymous GitHub repository for independent inspection. We also addressed questions regarding experimental consistency, evaluation protocols, data partitioning, and the interpretation of our personalization mechanisms. Additional experiments further examined historical modality contributions and SAIL's effectiveness with a stronger Population backbone. We are particularly grateful to Reviewer BmgV for acknowledging the value of our methodological clarifications and statistical analyses, and we appreciate the constructive discussion throughout the rebuttal.

Third, we appreciate Reviewer nHqH's latest acknowledgment that most of their concerns have been addressed and their willingness to increase the initial rating from 1 to 2. We have further explained SAIL's problem formulation, inputs and outputs, distinction from conventional GUI agents, and consistency with the original FingerTip-20K evaluation metric. Regarding the submission area, we recognize the close connection of our work to multimodal applications, but would like to emphasize that our primary focus is on personalized intent understanding and adaptation in foundation-model-based agents, rather than GUI-specific visual processing. In particular, historical interaction content, especially textual records and observed user intents, provides the principal personalization signals, while historical screenshots serve as complementary contextual information. Our modality ablation further supports this distinction, showing that SAIL retains substantial personalization gains even without historical screenshots. We therefore view GUI interaction as a concrete evaluation setting for a broader research problem in agent personalization and user-intent modeling.

We have also incorporated the reviewers' feedback into the revised manuscript, with particular attention to improving methodological clarity, experimental transparency, and the presentation of our contributions.

We sincerely appreciate the reviewers' constructive engagement and respectfully ask the Area Chair to consider the revised manuscript, the full rebuttal discussion, and the additional empirical evidence when forming the final assessment.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.