- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how personal GUI agents can use users’ interaction histories to carry out ambiguous instructions and proactively offer suggestions at appropriate times. The authors argue that task-level relevance alone does not ensure that historical experience applies to the current decision, and propose ExpActivator, a training-free method. For personalized execution, the method first retrieves historical trajectories based on the instruction. At each step, it then uses visual features from a frozen GUI model to match the current screenshot against historical screenshots, serializes the action associated with the most similar historical state, and adds it to the policy model’s prompt. For proactive suggestions, the method selects an intent based on the semantic neighborhoods of historical intents, scene consistency, temporal distance, and a support threshold, or abstains from making a suggestion.
The authors evaluate the method on AndroidIntent using four frozen GUI backbone models, covering 775 execution episodes, 7,915 steps, and 315 states for proactive suggestions. The results show that the method improves offline step prediction and discriminative performance for proactive suggestions while reducing the number of history tokens fed into the policy model. The authors also analyze the contributions of individual design choices through experiments that distinguish the first step from subsequent steps, hold candidate trajectories fixed, vary the reference update frequency, and remove temporal or scene conditions.
Critical strengths
-
The analysis of where history provides benefits has practical value. Table 2 evaluates the initial app-selection steps separately from the subsequent 7,140 steps. On Qwen3-VL-8B, Retrieve achieves a success rate of 18.35% on subsequent steps, close to the 18.43% achieved without history, whereas ExpActivator reaches 26.67%. This makes the paper’s observation more specific by identifying when history helps, providing useful evidence for designing more effective personalized memory mechanisms.
-
The paper includes targeted controlled experiments, and the improvements extend across multiple backbone models. Table 3 holds the same candidate historical trajectory fixed and shows that selecting a single action as a reference increases SSR from 18.47% when the full trajectory is provided to 23.84%, supporting the value of fine-grained selection. The results across four backbone models also indicate some consistency in these gains. The method requires no model parameter updates and provides shorter historical context to the policy model, making it attractive for deployment.
Critical weaknesses
- The methodological novelty over per-step visual RAG has not been sufficiently established, and the conceptual claim of “applicability activation” is stronger than the actual mechanism warrants.
The execution branch centers on text-based retrieval of candidate trajectories, visual nearest-neighbor matching between the current screenshot and historical screenshots, and adding the corresponding action to the frozen policy model’s prompt. Equation 3 always selects the historical step with the highest similarity, without explicitly checking action preconditions or providing a rule for withholding references when confidence is low. Thus, the mechanism directly measures visual state similarity, while action applicability is still judged by the downstream GUI model (Appendix B.6). Visual similarity does not guarantee action applicability, and the paper has not sufficiently established what “applicability activation” adds mechanistically beyond per-step visual RAG.
- The decomposition into first and subsequent steps conflates step position with decision type and is insufficient to support the conclusion that the role of history diminishes as execution progresses.
Appendix F.1 explicitly states that the first step of every episode is to identify the app, whereas subsequent steps involve taking actions on an observed app interface. Moreover, offline evaluation uses recorded interfaces and ground-truth action prefixes (Appendices C.2 and D.1). For example, history may help the model select the correct app at the first step, while subsequent actions may inherently depend less on history once the model is in the correct app. Even if a model without history selects the wrong app at the first step, offline evaluation still provides it with subsequent interfaces from the correct trajectory, thereby obscuring the benefits that history brings to the entire task through early decisions. Thus, the grouped results can characterize per-step gains across different decision types, but they are insufficient to conclude that historical experience ceases to be useful after the first step.
Questions for authors
-
On the added mechanisms and controls: Under the same eight candidate trajectories, text encoder, action-reference format, and prompt, what SSR does per-step nearest-neighbor retrieval achieve using uncentered global visual features? Appendix B.3 provides action-matching results from the calibration stage, but could the authors further clarify the individual contributions of centering, spatial partitioning, and the number of candidate trajectories to the final policy gains? This would help determine whether the improvements come from the paper’s specific designs or primarily from switching to per-step visual retrieval.
-
On the meaning of applicability: Does the execution branch include any low-similarity rejection mechanism, action precondition checks, or applicability validation beyond the equations in the main text? If Equation 3 always outputs a reference, could the authors provide an error analysis of cases where similar pages require different actions or no suitable historical state exists? This determines whether “applicability activation” should be understood as an independent mechanism or as a description of visual matching combined with downstream model judgment.
-
On the first-step conclusion and evaluation protocol: In Appendix F.1, all first steps involve app selection, whereas subsequent steps are already given the current interface. How can the effect of step position be distinguished from the effects of decision type and the amount of input information? Are gains already available broken down by subsequent-step position, action type, or personalization requirements, rather than aggregating all 7,140 steps? Given that four baseline–backbone combinations in Table 12 still show significant positive gains, how strongly do the authors believe the evidence supports the claim that “the first step uses up the benefits of history”? Please also confirm whether all results in the main tables use recorded screens and ground-truth action prefixes, and whether any closed-loop evaluation has been conducted in which model actions drive state transitions.
-
On the contribution of the proactive suggestion branch: The support and threshold rules in Equations 4–6 already determine which states are eligible for suggestions. When only the historical intents selected by these rules are used, without GUI-model generation, how much do BA, MCC, and semantic scores differ from those of the full system? Appendix D.3 already reports some pre-generation metrics; could the authors clarify how these correspond to the results in the main tables and whether the generative model further changes the triggering decisions? This would help distinguish the respective roles of the contextual filtering rules and the downstream model.
11 comments
Response to Reviewer T6RN (2/2)
W2 & Q3: Step position vs. decision type, and the evaluation protocol
The gain from appended history is primarily concentrated at the opening decision, which chooses an application and differs from later interface actions. The breakdowns below show that ExpActivator's later-step gains extend across depth ranges and action types.
Appended history barely helps after the first step. Across all 16 baseline–backbone pairs, the within-trajectory gain is at most 2.4 points, while the same baselines raise opening-step success on Qwen3-VL-8B from 28% to 62%. Activated history, by contrast, keeps helping: ExpActivator's within-trajectory gain over Base is significant on all four backbones and reaches 8.2 points on Qwen3-VL-8B. It is also the best method at the opening step (70.1% vs. 62.3%).
On Qwen3-VL-8B, selecting a step instead of appending the trajectory improves SSR in every group:
| Breakdown | Gain (points) |
|---|---|
| Steps 2–4 / 5–8 / 9+ | +5.0 / +4.3 / +6.5 |
| Click / Scroll / Type | +3.1 / +13.5 / +5.0 |
As a proxy for personalization, we looked at steps whose past and current goals are judged equivalent by two LLM annotators; on these steps, selection beats appending by 6.4 points.
All main-table results use recorded screens and ground-truth action histories, following the official AndroidIntent protocol, and we do not claim a closed-loop evaluation. When each model's own predicted actions replace the ground-truth history, ExpActivator is statistically unchanged and still leads every baseline by more than 7 points.
We report predicted-history results in Sec. 3.4, Table 4(b), and grouped controls in Appendix H.5; Sec. 3.3 clarifies the opening-step interpretation.
Q4: The activation rule alone vs. the full system
The activation rule decides whether to suggest, and the generator mainly phrases the suggestion; it never adds a trigger.
| Backbone | BA | MCC | Withheld by generator | HIM-Agent MCC |
|---|---|---|---|---|
| Rule only (Table 11) | 72.7 | 45.2 | – | – |
| Qwen3-VL-8B | 71.8 | 42.7 | 7 | 20.7 |
| MAI-UI-8B | 72.7 | 45.2 | 0 | 14.1 |
| Qwen3.5-9B | 70.9 | 38.9 | 58 | 22.0 |
| GUI-Owl-1.5-8B | 72.4 | 44.6 | 1 | 19.0 |
Without any generation, the rule already reaches MCC 45.2. The generator never adds a trigger beyond the rule and only occasionally withholds one, mostly on Qwen3.5-9B. With the same suggestion prompt as HIM-Agent, every full system reaches about twice its MCC, so the difference comes from activation.
The rule-only Semantic score (55.4) is computed on the user's own recorded intent, while the full system rewrites the intent as a new instruction. Even so, the full system has the highest Semantic score in the main table on Qwen3-VL-8B and GUI-Owl-1.5-8B.
We clarified the Table 11–Table 1 correspondence in Appendix D.3 and added the generator diagnostics to Appendix H.6.
Thank you for your constructive feedback; we remain open to further discussion and suggestions to improve the clarity and rigor of the work.
Dear Reviewer T6RN,
Thank you for your precise questions. Our response clarifies how experience activation differs from per-step visual retrieval and reports the descriptor ablations, decision breakdowns and rule-versus-generation comparison you asked for. If you have a moment, we would very much welcome your thoughts on whether these address your concerns, and we would be glad to discuss any remaining points.
Best regards,
The Authors
Response to Reviewer T6RN (1/2)
Thank you for your thoughtful review; we clarify the activation mechanism, decision breakdowns, and evaluation protocol below, with the corresponding manuscript revisions highlighted in purple.
W1 & Q1: Novelty over per-step visual RAG, and the contribution of each design choice
The contribution is to make applicability at each decision explicit, beyond retrieving visually similar history. In the memory baselines we compare, task-level retrieval supplies context but does not select which past decision applies to the current situation. Appended history improves the opening step but adds at most 2.4 points afterwards in all 16 baseline–backbone pairs. Experience must therefore be activated by the current situation, not merely retrieved for the task.
ExpActivator turns this diagnosis into one principle that governs both branches. Relevance only narrows history to candidates; the situation at each decision then selects which past decision to reuse. The mechanism has three parts:
The experiments confirm that the gain comes from this principle rather than from a particular retrieval component. On Qwen3-VL-8B, none of the appended-history baselines improves within-trajectory SSR, while ExpActivator adds 8.2 points. The four descriptor variants use the same eight candidate trajectories, prompt and reference format. The K=1 control changes the candidate budget while keeping the centered spatial descriptor (within-trajectory SSR, Qwen3-VL-8B):
Every descriptor reaches about 26.5%, far above the best memory baseline, so the gain does not hinge on how screens are encoded. On GUI-Owl-1.5-8B, our representation still adds a significant 0.6 points over uncentered global features. Selecting across trajectories also matters: going from one to eight candidates adds 2.8 points.
The decisive factor is alignment with the current decision. On the same trajectory, matching the current screen beats replaying the step at the same index (23.8% vs. 22.3%). Keeping the visual matching unchanged but applying it to the previous screen costs 4.0 points, and shifting the reference by one step costs 3.4 points (Tables 9–10). These controls keep the available history fixed while changing the alignment between the current decision and its action reference.
We clarified the mechanism in Secs. 1 and 2.2 and report the controls in Sec. 3.4 and Appendix H.2.
Q2: Is there any rejection or applicability check beyond Eq. 3?
Eq. 3 has no hard rejection threshold. Applicability is judged twice, first by κ_step and then by the policy, which changes the reference in 65.5% of decisions.
κ_step tracks applicability: agreement with the recorded action rises from 20.5% to 51.7% as κ_step increases. The policy is selective, adopting 43.7% of correct references but only 29.3% of incorrect ones. Weak matches still help. In the lowest similarity quartile on Qwen3-VL-8B, ExpActivator reaches 17.8%, against 13.0% for Base and 13.3% for the strongest baseline.
Both failure types you mention appear in the logs. The three pages below are all near-identical to their historical counterparts (κ_step ≈ 0.99):
The policy usually corrects such references, but page similarity alone does not always determine the next action.
We clarified the absence of hard rejection in Sec. 2.2 and added applicability diagnostics and failure cases to Appendix H.3.