- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- This work studies the predictive value of historical behavior beyond its original task context in cold-start intent prediction.
- SAIL combines an observation-conditioned Population reference, Gaussian user-state aggregation, and query-conditioned residual adaptation.
- This work assesses end-to-end gains, matched-control differences, negative transfer, and generalization to held-out users and Persona2Web.
Critical strengths
- The motivation is good; the GUI interface can indeed differ depending on the customized content.
- Provided a detailed experimental setup
Critical weaknesses
-
The method is too difficult to understand. The article contains an excessive number of symbols and formulas, which makes it difficult for readers to grasp the author's core contributions. After reading the methodology section, I still do not understand what the inputs and outputs of SILA are, nor how it differs from a standard UI agent.
-
I am also unclear about how the metrics in Table 1 are calculated. I have examined the original content of the FingerTip 20K dataset, and the calculations used in this paper do not appear to align with those.
-
Minor weakness:
- I support that this manuscript is better suited for the “ applications to computer vision, audio, language, and other modalities" area than for the “foundation or frontier models, including LLMs” area.
Questions for authors
- Please refer to the weaknesses.
5 comments
Response to W2: Clarification of Evaluation Metrics and Consistency with FingerTip-20K
We thank the reviewer for examining the original FingerTip-20K benchmark. We would like to clarify that our work does not redefine the primary intent-similarity metric. Instead, we follow the original benchmark's query-level Sim1 definition while adopting user-balanced aggregation for personalized intent prediction.
The corresponding metric definitions and aggregation procedures are already specified in Section 4.1 (Metrics and Evaluation) and Appendix A.1 (Evaluation Protocol, Baselines, and Metrics).
1. Consistency of the Sim1 definition
Following the original FingerTip-20K benchmark (Yang et al., ICLR 2026), query-level Sim1 is defined as the average of two similarity measures:
Sim1 = (S1 + S2) / 2
Here, S1 denotes the MiniLM embedding cosine similarity between the predicted and reference intents, while S2 denotes their normalized Levenshtein similarity.
Our evaluation adopts the same query-level formulation without introducing a different similarity measure. Therefore, the underlying mathematical definition of Sim1 is consistent with the original FingerTip-20K benchmark.
2. User-Macro aggregation and its existing description in the manuscript
It is important to distinguish the definition of query-level Sim1 from the aggregation procedure used to report the final evaluation results.
As specified in Section 4.1 (Metrics and Evaluation), our primary evaluation contains 1,633 queries from 83 users. Since different users may contribute different numbers of evaluation queries, directly averaging all queries would assign greater weight to users with more observations.
We therefore adopt User-Macro Sim1: we first average query-level Sim1 within each user and then average these user-level scores with equal weight. The resulting value is multiplied by 100 to obtain the point scale reported in Table 1.
This procedure is further detailed in Appendix A.1 under "Intent metrics", where we explicitly distinguish User-Macro Sim1 from Query-Micro Sim1. The former assigns equal weight to users, whereas the latter assigns equal weight to individual queries.
Therefore, our evaluation preserves the original FingerTip-20K query-level Sim1 definition while employing User-Macro aggregation to ensure equal representation of users in personalized evaluation. This is a distinction in statistical aggregation rather than a redefinition of the benchmark metric.
We will make these existing explanations more prominent in the revised manuscript to avoid potential ambiguity between query-level scoring and user-level aggregation.
We hope this clarification resolves the concern regarding the consistency of our evaluation metrics with the original FingerTip-20K benchmark.
Response to the Area Recommendation and Overall Assessment
We thank the reviewer for the suggestion regarding the submission area.
We agree that SAIL is closely related to multimodal GUI understanding and personalized agent systems. However, we would like to clarify that our primary research focus is on understanding and modeling agent behavior, with GUI agents serving as a concrete research setting rather than the sole scope of the proposed methodology.
More specifically, our work investigates a general challenge in agent systems: how an agent can infer user intent from sparse interaction history while distinguishing transferable user-specific signals from behaviors driven by task context. This challenge extends beyond GUI interaction and is relevant to broader agent scenarios involving implicit intent understanding, historical context modeling, and adaptive decision-making.
To address this challenge, SAIL develops a population-relative modeling framework combining context-conditioned reference construction, probabilistic user-state inference, and query-conditioned residual adaptation. These components offer a general modeling perspective for interpreting historical interactions and adapting pretrained models to user-specific contexts.
Although our current experiments primarily focus on GUI agents, the underlying formulation is not inherently restricted to particular GUI actions or interface structures. We therefore view GUI intent prediction as a concrete setting for studying a broader problem in agent understanding and adaptation, rather than the sole intended scope of the approach.
This methodological motivation is also an important reason why we selected the "Foundation or Frontier Models, including LLMs" area. Our goal is not merely to develop an application-specific GUI agent system, but to investigate how foundation-model-based agents can represent and interpret sparse interaction histories and incorporate transferable user information into subsequent predictions.
Regarding the overall assessment, we hope that our clarifications of SAIL's inputs, outputs, methodological contributions, and evaluation protocol will help address the reviewer's concerns and make the technical contribution clearer.
We appreciate the feedback on presentation and will improve the manuscript accordingly. We respectfully invite the reviewer to reconsider the overall recommendation in light of these clarifications.
Thank you again for your time and consideration.
We thank the reviewer for highlighting the importance of methodological clarity. We agree that the overall framework could be introduced more intuitively. However, we would like to clarify that the inputs, outputs, and modeling objectives of SAIL are explicitly defined in Section 3.1, and that SAIL addresses a different stage from conventional instruction-following GUI agents.
1. Inputs and outputs
SAIL takes two inputs:
The output is a natural-language prediction of the user's next intended task, rather than a GUI action sequence or screen coordinates.
Formally, SAIL predicts the target intent conditioned on the current GUI context and sparse interaction history, as defined in Equation (1) of our paper.
2. Difference from conventional GUI agents
Conventional GUI agents generally receive an explicit task instruction and determine how to execute it through interface interactions. In contrast, SAIL addresses the preceding problem: inferring what the user intends to accomplish before an explicit instruction is provided.
For example, when a user is viewing a product listing, a conventional GUI agent may receive the instruction "open product reviews" and execute the corresponding actions. SAIL instead predicts whether the user intends to inspect reviews, compare products, or perform another operation, based on the current GUI context and historical behavior.
The predicted intent can subsequently be passed to a separate GUI executor, but action execution is not the direct output of SAIL.
3. Core methodological contributions
Our contribution is not simply incorporating historical interactions. Rather, SAIL investigates how to distinguish transferable user-conditioned information from behaviors driven or constrained by the original task and interface context.
Specifically, SAIL combines three components:
Importantly, the Population predictor is separately fine-tuned for intent prediction. SAIL exactly recovers its predictions when no history is available and introduces history-conditioned adjustments when support interactions are provided.
Therefore, the central contribution is modeling which aspects of historical behavior remain predictive of future user intent beyond their original task context, rather than directly learning GUI execution actions.
We appreciate the reviewer's feedback on presentation and will improve the methodological overview with a concise input-output example and a clearer explanation of the three-stage framework. We hope these clarifications make the research objective and technical contributions easier to assess.