acceptodds
Under review as a conference paper at ICLR 2027

Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents

Abstract

Personal Graphical User Interface (GUI) agents rely on interaction history to infer what a user wants from ambiguous instructions and to anticipate recurring routines. Existing approaches retrieve task-relevant history and append it to the policy's context, implicitly assuming that experience relevant to a task remains useful for each decision within it. We find that this help is largely spent at the first decision: retrieved history strongly improves the opening step of an episode, yet provides little sustained benefit over the remaining of steps, and offers weak guidance on whether a proactive suggestion is warranted. A relevant record may tell the agent where to begin, but not which past action applies to the current screen or whether a routine is due now. The underlying issue is that *relevance does not imply applicability*: relevance is determined at the task level, whereas applicability depends on the situation at decision time. We therefore recast personalization as *experience activation* and introduce **ExpActivator**, a training-free framework that activates only the experience applicable to the current situation. During execution, ExpActivator matches each new screen to historical states in the frozen GUI backbone's latent space and supplies the corresponding action as a reference. Before execution, it activates a recurring intent only when the current time and scenario provide sufficient support, and otherwise abstains. Across four GUI backbones, ExpActivator improves within-trajectory step success by on average, achieves the best personalized execution on every backbone while using about one-fifth as many history tokens, and reaches approximately the Matthews correlation coefficient of the strongest proactive baseline. Experience pays where it is activated, not where it is appended.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
GUI Agents, personalized

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T6RNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how personal GUI agents can use users’ interaction histories to carry out ambiguous instructions and proactively offer suggestions at appropriate times. The authors argue that task-level relevance alone does not ensure that historical experience applies to the current decision, and propose ExpActivator, a training-free method. For personalized execution, the method first retrieves historical trajectories based on the instruction. At each step, it then uses visual features from a frozen GUI model to match the current screenshot against historical screenshots, serializes the action associated with the most similar historical state, and adds it to the policy model’s prompt. For proactive suggestions, the method selects an intent based on the semantic neighborhoods of historical intents, scene consistency, temporal distance, and a support threshold, or abstains from making a suggestion.

The authors evaluate the method on AndroidIntent using four frozen GUI backbone models, covering 775 execution episodes, 7,915 steps, and 315 states for proactive suggestions. The results show that the method improves offline step prediction and discriminative performance for proactive suggestions while reducing the number of history tokens fed into the policy model. The authors also analyze the contributions of individual design choices through experiments that distinguish the first step from subsequent steps, hold candidate trajectories fixed, vary the reference update frequency, and remove temporal or scene conditions.

Critical strengths
  1. The analysis of where history provides benefits has practical value. Table 2 evaluates the initial app-selection steps separately from the subsequent 7,140 steps. On Qwen3-VL-8B, Retrieve achieves a success rate of 18.35% on subsequent steps, close to the 18.43% achieved without history, whereas ExpActivator reaches 26.67%. This makes the paper’s observation more specific by identifying when history helps, providing useful evidence for designing more effective personalized memory mechanisms.

  2. The paper includes targeted controlled experiments, and the improvements extend across multiple backbone models. Table 3 holds the same candidate historical trajectory fixed and shows that selecting a single action as a reference increases SSR from 18.47% when the full trajectory is provided to 23.84%, supporting the value of fine-grained selection. The results across four backbone models also indicate some consistency in these gains. The method requires no model parameter updates and provides shorter historical context to the policy model, making it attractive for deployment.

Critical weaknesses
  1. The methodological novelty over per-step visual RAG has not been sufficiently established, and the conceptual claim of “applicability activation” is stronger than the actual mechanism warrants.

The execution branch centers on text-based retrieval of candidate trajectories, visual nearest-neighbor matching between the current screenshot and historical screenshots, and adding the corresponding action to the frozen policy model’s prompt. Equation 3 always selects the historical step with the highest similarity, without explicitly checking action preconditions or providing a rule for withholding references when confidence is low. Thus, the mechanism directly measures visual state similarity, while action applicability is still judged by the downstream GUI model (Appendix B.6). Visual similarity does not guarantee action applicability, and the paper has not sufficiently established what “applicability activation” adds mechanistically beyond per-step visual RAG.

  1. The decomposition into first and subsequent steps conflates step position with decision type and is insufficient to support the conclusion that the role of history diminishes as execution progresses.

Appendix F.1 explicitly states that the first step of every episode is to identify the app, whereas subsequent steps involve taking actions on an observed app interface. Moreover, offline evaluation uses recorded interfaces and ground-truth action prefixes (Appendices C.2 and D.1). For example, history may help the model select the correct app at the first step, while subsequent actions may inherently depend less on history once the model is in the correct app. Even if a model without history selects the wrong app at the first step, offline evaluation still provides it with subsequent interfaces from the correct trajectory, thereby obscuring the benefits that history brings to the entire task through early decisions. Thus, the grouped results can characterize per-step gains across different decision types, but they are insufficient to conclude that historical experience ceases to be useful after the first step.

Questions for authors
  1. On the added mechanisms and controls: Under the same eight candidate trajectories, text encoder, action-reference format, and prompt, what SSR does per-step nearest-neighbor retrieval achieve using uncentered global visual features? Appendix B.3 provides action-matching results from the calibration stage, but could the authors further clarify the individual contributions of centering, spatial partitioning, and the number of candidate trajectories to the final policy gains? This would help determine whether the improvements come from the paper’s specific designs or primarily from switching to per-step visual retrieval.

  2. On the meaning of applicability: Does the execution branch include any low-similarity rejection mechanism, action precondition checks, or applicability validation beyond the equations in the main text? If Equation 3 always outputs a reference, could the authors provide an error analysis of cases where similar pages require different actions or no suitable historical state exists? This determines whether “applicability activation” should be understood as an independent mechanism or as a description of visual matching combined with downstream model judgment.

  3. On the first-step conclusion and evaluation protocol: In Appendix F.1, all first steps involve app selection, whereas subsequent steps are already given the current interface. How can the effect of step position be distinguished from the effects of decision type and the amount of input information? Are gains already available broken down by subsequent-step position, action type, or personalization requirements, rather than aggregating all 7,140 steps? Given that four baseline–backbone combinations in Table 12 still show significant positive gains, how strongly do the authors believe the evidence supports the claim that “the first step uses up the benefits of history”? Please also confirm whether all results in the main tables use recorded screens and ground-truth action prefixes, and whether any closed-loop evaluation has been conducted in which model actions drive state transitions.

  4. On the contribution of the proactive suggestion branch: The support and threshold rules in Equations 4–6 already determine which states are eligible for suggestions. When only the historical intents selected by these rules are used, without GUI-model generation, how much do BA, MCC, and semantic scores differ from those of the full system? Appendix D.3 already reports some pre-generation metrics; could the authors clarify how these correspond to the results in the main tables and whether the generative model further changes the triggering decisions? This would help distinguish the respective roles of the contextual filtering rules and the downstream model.

11 comments
AuthorsOpenReview ↗

Response to Reviewer T6RN (1/2)

Thank you for your thoughtful review; we clarify the activation mechanism, decision breakdowns, and evaluation protocol below, with the corresponding manuscript revisions highlighted in purple.


W1 & Q1: Novelty over per-step visual RAG, and the contribution of each design choice

The contribution is to make applicability at each decision explicit, beyond retrieving visually similar history. In the memory baselines we compare, task-level retrieval supplies context but does not select which past decision applies to the current situation. Appended history improves the opening step but adds at most 2.4 points afterwards in all 16 baseline–backbone pairs. Experience must therefore be activated by the current situation, not merely retrieved for the task.

ExpActivator turns this diagnosis into one principle that governs both branches. Relevance only narrows history to candidates; the situation at each decision then selects which past decision to reuse. The mechanism has three parts:

  1. The unit of reuse is a single past decision, re-selected at every observation from any of the candidate trajectories. In the compared memory baselines, retrieved context is fixed once per task. In 92% of episodes, the references come from two or more trajectories, and the source switches on 55% of consecutive steps (Figure 4).
  2. What reaches the policy is one action reference in its native format (144.6 tokens), not retrieved screenshots or trajectories. The policy can adopt, adapt or reject it.
  3. The same principle decides whether to make a proactive suggestion from time and scenario, without a current execution screen. The rule alone reaches MCC 45.2, more than twice the highest memory-baseline MCC of 22.0.

The experiments confirm that the gain comes from this principle rather than from a particular retrieval component. On Qwen3-VL-8B, none of the appended-history baselines improves within-trajectory SSR, while ExpActivator adds 8.2 points. The four descriptor variants use the same eight candidate trajectories, prompt and reference format. The K=1 control changes the candidate budget while keeping the centered spatial descriptor (within-trajectory SSR, Qwen3-VL-8B):

VariantSSR (%)
Uncentered global (K=8)26.40
Centered global (K=8)26.61
Uncentered spatial (K=8)26.76
Centered spatial (ours, K=8)26.67
Centered spatial (K=1)23.84
Best memory baseline18.36

Every descriptor reaches about 26.5%, far above the best memory baseline, so the gain does not hinge on how screens are encoded. On GUI-Owl-1.5-8B, our representation still adds a significant 0.6 points over uncentered global features. Selecting across trajectories also matters: going from one to eight candidates adds 2.8 points.

The decisive factor is alignment with the current decision. On the same trajectory, matching the current screen beats replaying the step at the same index (23.8% vs. 22.3%). Keeping the visual matching unchanged but applying it to the previous screen costs 4.0 points, and shifting the reference by one step costs 3.4 points (Tables 9–10). These controls keep the available history fixed while changing the alignment between the current decision and its action reference.

We clarified the mechanism in Secs. 1 and 2.2 and report the controls in Sec. 3.4 and Appendix H.2.

Q2: Is there any rejection or applicability check beyond Eq. 3?

Eq. 3 has no hard rejection threshold. Applicability is judged twice, first by κ_step and then by the policy, which changes the reference in 65.5% of decisions.

κ_step tracks applicability: agreement with the recorded action rises from 20.5% to 51.7% as κ_step increases. The policy is selective, adopting 43.7% of correct references but only 29.3% of incorrect ones. Weak matches still help. In the lowest similarity quartile on Qwen3-VL-8B, ExpActivator reaches 17.8%, against 13.0% for Base and 13.3% for the strongest baseline.

Both failure types you mention appear in the logs. The three pages below are all near-identical to their historical counterparts (κ_step ≈ 0.99):

CaseHistorical referenceCurrent targetPolicy outcome
Finance home pageMain-funds shortcutExchange-rate shortcutCorrects the click; succeeds
Bus-stop listDirections actionBus-stop searchCorrects the click; succeeds
Building-search listList-item clickTop search controlCopies the reference; fails

The policy usually corrects such references, but page similarity alone does not always determine the next action.

We clarified the absence of hard rejection in Sec. 2.2 and added applicability diagnostics and failure cases to Appendix H.3.

AuthorsOpenReview ↗

Response to Reviewer T6RN (2/2)

W2 & Q3: Step position vs. decision type, and the evaluation protocol

The gain from appended history is primarily concentrated at the opening decision, which chooses an application and differs from later interface actions. The breakdowns below show that ExpActivator's later-step gains extend across depth ranges and action types.

Appended history barely helps after the first step. Across all 16 baseline–backbone pairs, the within-trajectory gain is at most 2.4 points, while the same baselines raise opening-step success on Qwen3-VL-8B from 28% to 62%. Activated history, by contrast, keeps helping: ExpActivator's within-trajectory gain over Base is significant on all four backbones and reaches 8.2 points on Qwen3-VL-8B. It is also the best method at the opening step (70.1% vs. 62.3%).

On Qwen3-VL-8B, selecting a step instead of appending the trajectory improves SSR in every group:

BreakdownGain (points)
Steps 2–4 / 5–8 / 9++5.0 / +4.3 / +6.5
Click / Scroll / Type+3.1 / +13.5 / +5.0

As a proxy for personalization, we looked at steps whose past and current goals are judged equivalent by two LLM annotators; on these steps, selection beats appending by 6.4 points.

All main-table results use recorded screens and ground-truth action histories, following the official AndroidIntent protocol, and we do not claim a closed-loop evaluation. When each model's own predicted actions replace the ground-truth history, ExpActivator is statistically unchanged and still leads every baseline by more than 7 points.

We report predicted-history results in Sec. 3.4, Table 4(b), and grouped controls in Appendix H.5; Sec. 3.3 clarifies the opening-step interpretation.

Q4: The activation rule alone vs. the full system

The activation rule decides whether to suggest, and the generator mainly phrases the suggestion; it never adds a trigger.

BackboneBAMCCWithheld by generatorHIM-Agent MCC
Rule only (Table 11)72.745.2––
Qwen3-VL-8B71.842.7720.7
MAI-UI-8B72.745.2014.1
Qwen3.5-9B70.938.95822.0
GUI-Owl-1.5-8B72.444.6119.0

Without any generation, the rule already reaches MCC 45.2. The generator never adds a trigger beyond the rule and only occasionally withholds one, mostly on Qwen3.5-9B. With the same suggestion prompt as HIM-Agent, every full system reaches about twice its MCC, so the difference comes from activation.

The rule-only Semantic score (55.4) is computed on the user's own recorded intent, while the full system rewrites the intent as a new instruction. Even so, the full system has the highest Semantic score in the main table on Qwen3-VL-8B and GUI-Owl-1.5-8B.

We clarified the Table 11–Table 1 correspondence in Appendix D.3 and added the generator diagnostics to Appendix H.6.


Thank you for your constructive feedback; we remain open to further discussion and suggestions to improve the clarity and rigor of the work.

Dear Reviewer T6RN,

Thank you for your precise questions. Our response clarifies how experience activation differs from per-step visual retrieval and reports the descriptor ablations, decision breakdowns and rule-versus-generation comparison you asked for. If you have a moment, we would very much welcome your thoughts on whether these address your concerns, and we would be glad to discuss any remaining points.

Best regards,

The Authors

Reviewer_JPGfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at how personal GUI agents should use a user's interaction history. The authors observe that when retrieved trajectories are appended to the context as a whole, nearly all of the benefit shows up at the first step of an episode, and they attribute this to the gap between task-level relevance and decision-level applicability. They propose ExpActivator, a training-free method. During execution it matches the current screenshot against historical screenshots in the visual feature space of the frozen backbone at every step and passes the matched action to the policy as a reference. For proactive suggestion it decides whether to fire based on scenario and time support. On AndroidIntent with four backbones, the method beats the existing memory baselines on both personalized execution and proactive suggestion while using about one fifth of the history tokens.

Critical strengths

The analysis of where task-level retrieval fails is convincing. Tables 2 and 12 cover 16 pairings of four backbones and four memory baselines. Within the trajectory, the gain over no history is at most 2.4 points and 12 of the 16 are indistinguishable from zero, while the same baselines lift first-step success from 28% to 62%. These numbers use the benchmark's own baselines and the same history and do not depend on the proposed method, so they are credible. They also explain why earlier gains in this area tend to show up in CSR, which weights the early steps more heavily.

The method needs no training and is best on all four backbones, which use different visual encoders, while using about a third of the history tokens of HIM-Agent, the cheapest baseline. Tables 3 and 8 compare appending a whole trajectory with selecting a single step from the same trajectory, and add two controls that match against the previous screen or refresh only every four steps. This attributes the gain to selection at decision time rather than to more input, which says more than a higher leaderboard number would.

Critical weaknesses

All conclusions come from step-wise offline evaluation on one benchmark. Each step is given the ground-truth previous actions and screenshot, so errors never compound, yet step-level activation is most likely to break exactly when the current screen has drifted away from the stored trajectories. The test split is also the later 20% of the same users and apps, so current and historical screenshots are very close, with similarities of 0.93 to 0.98 in the case study, which favors nearest-neighbor matching. The paper does not look at UI updates, new apps, changing user habits, or online and closed-loop settings. Since the paper puts forward a general principle about when history should be used, the current validation is too narrow to support it.

The method and the baselines do not receive the same modality of information, and the margin over the strongest baseline on the strongest backbone is small. On GUI-Owl-1.5-8B, ExpActivator beats Retrieve by 0.7 SSR points, the within-trajectory gains are +2.38 versus +1.72 with clearly overlapping intervals, and the average 20% gain claimed in the text is driven mainly by the +37% on Qwen3-VL-8B. Visual matching is the key signal for ExpActivator, while all memory baselines get text-only history without any screenshots. Giving Retrieve or HIM-Agent the retrieved screenshots as well would be needed to separate the effect of step-wise selection from the effect of visual information. The K=1 control in Table 8 partly addresses this, but only on one backbone.

Questions for authors
  1. The appendix calls the six task-level constants declared constants and reports a sensitivity analysis, but it does not say what data or criterion was used to set them, or whether the 315 evaluation states were used even indirectly. Please clarify.
  2. SSR improves substantially even though B.3 reports that the reference action agrees with the true next action only about 35% to 37% of the time during calibration. What is this agreement rate on the evaluation set, and how often does the policy adopt, modify, or reject the reference?
  3. Could you add results in a closed-loop or online setting, even a small one? If that is not possible, an offline simulation where the stored screenshots come from a different UI version than the current screen would also help.
  4. The main text only reports significance against Base. Was the gap to Retrieve on GUI-Owl tested with the paired bootstrap?
  5. On the Qwen3.5-9B backbone, LLM-UM has a recall of 6% on proactive suggestion. Since its user profiles were regenerated by the authors with Qwen3.5-9B, could this be a profile quality issue?
8 comments
AuthorsOpenReview ↗

Response to Reviewer JPGf (1/3)

Thank you for your detailed review; we address the evaluation protocol, comparison fairness, and activation settings below, with the corresponding manuscript revisions highlighted in purple.


W1 & Q3: Offline evaluation, near-duplicate screens, and closed-loop settings

The advantage persists under three stricter tests, one for each of your concerns (Qwen3-VL-8B, within-trajectory SSR, %):

SettingBaseRetrieveHIM-AgentExpActivator
Model's own predicted actions as prefix17.4518.7117.9826.30
Only the oldest 50% of history18.4317.7717.8924.72
Near-identical goals removed18.4318.1918.3926.43
  1. The gain persists with predicted action histories. We replaced the ground-truth action history with each model's own predictions, so earlier mistakes enter every later input. ExpActivator's SSR is statistically unchanged (26.3% vs. 26.7%), because it selects the reference from the current screen rather than from the action history.
  2. Older history still works. Keeping only the oldest half of each user's records raises the average age of the history from 19 to 27 days, and ExpActivator still leads every baseline by more than 6 points.
  3. The gain persists after removing near-identical goals. With near-identical goals removed, ExpActivator still leads every baseline by about 8 points. The 0.93–0.98 similarities in the case study are not typical: the median similarity of the selected reference is 0.67, and fewer than 8% of steps exceed 0.93.

These tests use recorded screens, so a predicted action changes the action history but not the next screen.

We clarified the protocol in Sec. 3.1 and report all three controls in Sec. 3.4, Table 4(b).

AuthorsOpenReview ↗

Response to Reviewer JPGf (2/3)

W2 & Q4: Modality, the GUI-Owl margin, and significance against Retrieve

Giving the baselines screenshots does not close the gap. On GUI-Owl-1.5-8B, ExpActivator's margin over Retrieve is significant in CSR but not in SSR. The table reports within-trajectory SSR (%). Significance uses 20,000 paired user-cluster bootstrap draws and 95% intervals.

ConditionQwen3-VL-8BGUI-Owl-1.5-8B
Base (no history)18.4322.30
Retrieve + past screenshots15.1810.35
Whole trajectory + past screenshots (K=1)11.1610.11
Single selected step + its screenshot (K=1)21.7417.18
ExpActivator (K=8)26.6724.68

With screenshots, the memory baselines fall below Base. With the same trajectory and the same screenshot access, showing only the matched step instead of the whole trajectory gains 10.6 points on Qwen3-VL-8B and 7.1 points on GUI-Owl-1.5-8B. The modality difference therefore does not explain our results.

In overall SSR, ExpActivator significantly outperforms the strongest baseline on Qwen3-VL-8B (+8.3), MAI-UI-8B (+5.0) and Qwen3.5-9B (+3.4). GUI-Owl-1.5-8B is the strongest backbone without history, so it leaves the least room to improve. Against Retrieve on this backbone, ExpActivator gains a significant 4.1 points in CSR, while the 0.7-point SSR difference is not significant. It does so with about one-fifth of the history tokens, and it more than doubles HIM-Agent's proactive MCC (44.6 vs. 19.0).

We report screenshot controls in Sec. 3.4, Table 4(a), and the backbone comparisons in Appendix H.6.

AuthorsOpenReview ↗

Response to Reviewer JPGf (3/3)

Q1: How were the six task-level constants set?

The constants were fixed before evaluation, based on what each quantity means, and none of the 315 evaluation states was used to set them, directly or indirectly.

Each value follows from its role in the activation rule:

  • η = 0.94 counts only near-paraphrases as the same routine.
  • n_min = 12 and n_ctx = 4 require a routine to recur often, both overall and in the current scenario, before it is suggested.
  • σ = 1 hour matches the hour-level regularity of daily routines.
  • γ = 1 is plain frequency normalization.
  • δ = 0.25 sets the activation cutoff.

The sweeps in App. E were run afterwards as robustness checks; every reported result uses the values above. AUROC stays within 75–81% across all 40 (η, n_min) settings in the reported plateau. It changes by less than 0.6 points for σ = 0.25–2 hours; BA changes by less than 1.3 points for δ = 0.05–1.0, and γ leaves trigger decisions unchanged.

We added the rationale for the six constants to Appendix B.1 and clarified the timing of the sweeps in Appendix E.

Q2: Agreement on the evaluation set, and how the policy uses the reference

On the evaluation set, the activated reference matches the recorded next action on 35.5–36.1% of within-trajectory steps across the four backbones. This is close to the calibration estimates (35.1–36.6%). Agreement rises with κ_step, from 20.5% in the lowest similarity bin to 51.7% in the highest.

On Qwen3-VL-8B, the policy adopts 34.5% of references, adapts 29.7% (same action type, new parameters) and overrides 35.8%. It does so selectively: it adopts 43.7% of correct references but only 29.3% of incorrect ones.

Reference vs. recorded actionAdoptAdaptOverride
Reference agrees1,121703742
Reference disagrees1,3421,4161,816

We added evaluation-set reference agreement and policy adoption counts to Appendix H.3.

Q5: Is LLM-UM's 6% recall on Qwen3.5-9B a profile-quality issue?

No. The profiles are shared across backbones, and the outputs show conservative abstention rather than malformed or incomplete responses.

All four backbones use the same sealed profile files. With them, LLM-UM reaches 99.1%, 100% and 70.2% recall on Qwen3-VL-8B, MAI-UI-8B and GUI-Owl-1.5-8B. Qwen3.5-9B is conservative for every method, including those that never see the profiles: Retrieve reaches 34.0% recall on it and HIM-Agent 41.4%. All 315 LLM-UM outputs on Qwen3.5-9B are well-formed and complete, and 294 of them are explicit "False" decisions.

Higher recall would not help LLM-UM much in any case. On the other three backbones, its high recall comes with near-zero MCC, because a profile describes what the user tends to do but not whether a routine is due now. Task-level activation supplies this information and reaches MCC 38.9–45.2 on all four backbones.

We added the shared-profile and output-validity checks to Appendix H.6.


We would be glad to discuss any remaining questions and will use your further feedback to strengthen the evaluation and presentation.

Reviewer_mgyCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ExpActivator, a training-free method for personal GUI agents. After the first step, it finds the most similar screen in related past tasks. It then gives the matching past action to the model as a hint. For proactive suggestions, it uses time and scenario rules to decide when to speak. On AndroidIntent with four backbones, it beats memory baselines on step success and MCC, with fewer history tokens.

Critical strengths

The paper splits each episode into the first step and the later steps. This shows that existing memory methods help little after the first step. The finding holds on all four backbones.

Critical weaknesses
  1. The baselines get past history only as text, with no past screens. So they cannot link a past action to the current screen. The main claim needs a baseline that also sees past screenshots.

  2. Both parts are simple rules: a nearest-neighbor action lookup and fixed time and scenario thresholds. All tests are offline, on one benchmark with 82 users.

Questions for authors
  1. How do the baselines perform when they also see screenshots of past steps?

  2. How much gain remains on tasks with no near-identical past episode?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mgyC

Thank you for your careful review; we address your questions about historical screenshots and transferable experience below, with the corresponding manuscript revisions highlighted in purple.


W1 & Q1: How do the baselines perform when they also see past screenshots?

Past screenshots do not help the baselines; what helps is selecting the step that applies to the current screen. The table reports within-trajectory SSR (%). Each past image is given at the same resolution as the current screen, and K is the number of candidate trajectories.

ConditionQwen3-VL-8BGUI-Owl-1.5-8B
Base (no history)18.4322.30
Retrieve + past screenshots15.1810.35
Whole trajectory + past screenshots (K=1)11.1610.11
Single selected step + its screenshot (K=1)21.7417.18
ExpActivator (K=8)26.6724.68

With past screenshots appended, both baselines fall below the no-history Base. On the same trajectory, showing only the matched step instead of the whole trajectory raises SSR by 10.6 points on Qwen3-VL-8B and by 7.1 points on GUI-Owl-1.5-8B. The match must also be made against the current screen: with everything else fixed, matching against the previous screen lowers SSR by 4.0 points (Table 10).

We report the historical-screenshot controls in Sec. 3.4, Table 4(a), with the full protocol in Appendix H.1.

W2: Simple rules, offline evaluation, one benchmark

We kept the method simple on purpose, because the simplicity is part of the finding. A nearest-neighbour lookup outperforms memories that summarize, cluster or profile the same history, which suggests that the bottleneck is when experience is used rather than how it is stored. ExpActivator is the best method on all four backbones.

Each design choice is checked by a control: refreshing the reference at every step (Table 10), matching states rather than step indices, and conditioning on time and scenario (Table 3). No component is fine-tuned, and the same procedure and constants are used with all four frozen backbones.

The advantage also holds in stricter settings. When the policy sees its own predicted actions instead of the ground-truth ones, ExpActivator reaches 26.30% against 17.45% for Base. When only the oldest half of the history is available, it reaches 24.72% against 18.43%.

We report these protocol controls in Sec. 3.4, Table 4(b), with full details in Appendix H.4.

Q2: How much gain remains without a near-identical past episode?

Almost all of it. ExpActivator reuses individual decisions rather than whole episodes, and the same screens recur across many different tasks.

  1. We removed every past record whose goal is near-identical to the current request (cosine similarity ≥ 0.95, or an exact match). ExpActivator stays at 26.4%, compared with 26.7% before the removal, and still leads every baseline by about 8 points. The removal takes out only about one record per affected episode, and the useful references come from other tasks that pass through the same screens. History is still essential: without it, Base reaches only 18.4%.
  2. In 92% of episodes, the references come from two or more different past trajectories (Figure 4).
  3. Even for users with only 39–111 records, ExpActivator exceeds the strongest baseline by 8.3 points (Figure 3).

We report the goal-overlap control in Sec. 3.4, Table 4(b), with details in Appendix H.4.


We welcome further discussion and any additional suggestions that could help us refine the analysis and improve the paper.

Thanks for the additional clarification and evaluation. I acknowledge the results and will maintain my positive score.

Thank you for confirming that our rebuttal addressed your concerns and for your continued support of acceptance; your suggestions have substantially strengthened the paper.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Summary of Response and Revisions

Dear Area Chair and Reviewers,

We thank the Area Chair and Reviewers mgyC, JPGf and T6RN for their time and constructive feedback. We are encouraged that all three reviewers found our central diagnosis useful: retrieved history mainly helps the opening step and adds little afterwards, consistently across four backbones (mgyC), in a convincing analysis (JPGf) with practical value for designing personalized memory (T6RN). We also appreciate the recognition that the method is training-free and supported by targeted controls (JPGf, T6RN).

We summarize the main concerns and how the revision addresses them.

1. Visual information vs. step-wise selection (mgyC: W1&Q1; JPGf: W2). We gave the memory baselines past screenshots on two backbones. The extra screenshots do not help them. With the same trajectory and the same screenshot access, selecting the applicable step still clearly outperforms appending the whole trajectory, so the gain comes from selection rather than from visual input.

2. Evaluation protocol and generality (mgyC: W2&Q2; JPGf: W1&Q3; T6RN: Q3). We clarified that the main results follow the official AndroidIntent protocol, with recorded screens. The advantage persists in three stricter settings: when each model's own predicted actions replace the ground-truth history, when only the oldest half of the history is available, and when near-identical past goals are removed.

3. What activation adds beyond visual retrieval (T6RN: W1, Q1&Q2; JPGf: Q1&Q2). Descriptor and candidate-set controls show that the gain does not hinge on how screens are encoded; it comes from re-selecting a past decision at every observation, aligned with the current screen. We added evaluation-set diagnostics of how the policy adopts, adapts or overrides references, together with failure cases, and the rationale for fixing the task-level constants before evaluation.

4. Where the gains come from (T6RN: W2, Q3&Q4; JPGf: W2, Q4&Q5). Breakdowns by step depth, action type and goal equivalence show that selection improves later steps throughout the trajectory. We also report paired significance tests against the strongest baseline on every backbone, confirm that LLM-UM profiles are shared across backbones, and show that the activation rule alone decides when to suggest; the generator never adds a trigger.

ConcernRaised byAdditional experiments and clarificationsLocation
Visual information vs. selectionmgyC (W1, Q1); JPGf (W2)Historical-screenshot baselines on two backbonesPages 7–8, 30
Evaluation protocol and generalitymgyC (W2, Q2); JPGf (W1, Q3); T6RN (Q3)Protocol clarification; own predicted histories; oldest-half history; near-identical goals removedPages 5, 8, 31
Activation mechanismT6RN (W1, Q1, Q2); JPGf (Q1, Q2)Descriptor and candidate-set controls; reference agreement and policy decisions; failure cases; rationale for constantsPages 3–4, 15, 22, 30–31
Gain attribution and diagnosticsT6RN (W2, Q3, Q4); JPGf (W2, Q4, Q5)Depth, action-type and goal-equivalence breakdowns; significance tests; shared-profile check; activation rule vs. full systemPages 7, 21, 31–32

All revisions are highlighted in purple in the updated manuscript. We hope this summary is helpful for the remaining discussion, and we would welcome any further questions or comments from the reviewers.

AuthorsOpenReview ↗

Rebuttal Summary for ACs and SACs (Part1)

Dear Area Chairs, Senior Area Chairs, and Reviewers,

We thank you for your time and careful engagement with our submission. To help the Area Chairs follow the discussion, we summarize our contributions, the strengths noted by the reviewers, and how each weakness and question was addressed. The revised manuscript has been uploaded, with all changes highlighted in purple.

Contributions

  1. We show that task-level retrieval, the common way personal GUI agents use history, is largely spent at the first decision. Across all 16 baseline–backbone pairs, it lifts opening-step success from 28% to 62%, yet adds at most 2.4 points over the remaining steps. The reason is that relevance does not imply applicability: relevance is fixed at the task level, while applicability depends on the situation at each decision.
  2. We recast personalization as experience activation and introduce ExpActivator, a training-free framework. Task relevance only narrows history to candidates. At every step, the current screen selects which past decision to reuse. The same principle governs proactive suggestion, where the current time and scenario decide whether a routine is due.
  3. Across four GUI backbones, ExpActivator improves within-trajectory step success by 28% on average and achieves the best personalized execution on every backbone. It uses about one-fifth as many history tokens and reaches about 2.3× the MCC of the strongest proactive baseline.

Strengths

StrengthDetails
Convincing and useful diagnosismgyC: "This shows that existing memory methods help little after the first step. The finding holds on all four backbones." JPGf: "The analysis of where task-level retrieval fails is convincing." T6RN: "The analysis of where history provides benefits has practical value."
Gain attributed to decision-time selectionJPGf: "This attributes the gain to selection at decision time rather than to more input, which says more than a higher leaderboard number would." T6RN: "The paper includes targeted controlled experiments, and the improvements extend across multiple backbone models."
Training-free and deployableJPGf: "The method needs no training and is best on all four backbones, which use different visual encoders." T6RN: "The method requires no model parameter updates and provides shorter historical context to the policy model, making it attractive for deployment."

Summary of the discussion

  • All three reviewers credit the central diagnosis in their written assessments: task-level retrieval helps the opening step but adds little afterwards. JPGf notes that these numbers “do not depend on the proposed method, so they are credible.”
  • Most weaknesses ask for specific additional controls. We ran every requested analysis except a closed-loop evaluation, for which we report three stricter offline tests. The analyses we ran include:
    • screenshot-augmented baselines on a second backbone;
    • near-duplicate removal;
    • paired significance tests;
    • the constant-setting protocol;
    • evaluation-set reference diagnostics;
    • descriptor and candidate-set ablations;
    • an error analysis;
    • step-wise breakdowns;
    • rule-only suggestion metrics.
  • Reviewer mgyC acknowledged the new results and maintains a positive score. In the follow-up, Reviewer T6RN agreed with our description of the mechanism. The remaining point concerns how to characterize the methodological contribution, not the correctness of the results.
AuthorsOpenReview ↗

Rebuttal Summary for ACs and SACs (Part2)

Weaknesses and Questions

ReviewerCommentOur ResponseRating (Confidence)Reviewer Response
mgyCBaselines see history only as text, with no past screens (W1&Q1)We gave the baselines past screenshots on two backbones. Both fall below the no-history Base, while selecting the matched step from the same trajectory gains 10.6 and 7.1 points (Sec. 3.4, Table 4(a); App. H.1).3Resolved; maintains positive score: "I acknowledge the results and will maintain my positive score."
Simple rules; offline, one benchmark (W2)Simplicity is part of the finding: a nearest-neighbour lookup succeeds where memories that summarize, cluster or profile history do not. Each design choice is checked by a control, and the advantage holds with predicted action histories and with the oldest half of the history (Table 4(b)).
Gain without a near-identical past episode (Q2)With near-identical goals removed, ExpActivator stays at 26.4% (from 26.7%) and still leads every baseline by about 8 points. In 92% of episodes, references come from two or more past trajectories (Table 4(b); App. H.4).
JPGfOffline evaluation, near-duplicate screens, closed-loop settings (W1&Q3)The advantage persists in three stricter tests: predicted action histories, oldest-half history, and near-identical goals removed. The 0.93–0.98 similarities in the case study are not typical: the median similarity of the selected reference is 0.67 (Sec. 3.1; Table 4(b); App. H.4).2No response yet
Modality mismatch; GUI-Owl margin; significance against Retrieve (W2&Q4)Past screenshots do not close the gap (Table 4(a)). With paired bootstrap tests, ExpActivator significantly outperforms the strongest baseline on three backbones. On GUI-Owl-1.5-8B, it significantly improves CSR over Retrieve with about one-fifth of the history tokens (App. H.6).
How were the six constants set? (Q1)All six were fixed before evaluation from their role in the rule, and no evaluation state was used. The sweeps in App. E were run afterwards as robustness checks (App. B.1, E).
Reference agreement on the evaluation set; policy behavior (Q2)Agreement is 35.5–36.1%, close to calibration. The policy adopts 43.7% of correct references but only 29.3% of incorrect ones (App. H.3).
LLM-UM's low recall on Qwen3.5-9B (Q5)Profiles are shared across backbones and reach 70–100% recall elsewhere. Qwen3.5-9B is conservative even for methods that use no profile (App. H.6).
AuthorsOpenReview ↗

Rebuttal Summary for ACs and SACs (Part3)

ReviewerCommentOur ResponseRating (Confidence)Reviewer Response
T6RNNovelty over per-step visual RAG (W1&Q1)Appending retrieved screenshots falls below Base, while activating one step from the same trajectory raises SSR from 18.47% to 23.84%. Every descriptor variant reaches about 26.5%, so the gain comes from activation rather than from how screens are encoded (Sec. 3.4; App. H.2).2
Any rejection or applicability check beyond Eq. 3? (Q2)Applicability is judged by κ_step and then by the policy, which changes the reference in 65.5% of decisions. Even in the lowest-similarity quartile, ExpActivator beats Base (17.8% vs. 13.0%). Failure cases are added (Sec. 2.2; App. H.3).
Step position vs. decision type; protocol (W2&Q3)Selection beats appending across depth ranges and action types, and by 6.4 points on goal-equivalent steps. All main results follow the official AndroidIntent protocol (Sec. 3.3; App. H.5).
Activation rule alone vs. full system (Q4)The rule alone reaches MCC 45.2; the generator never adds a trigger and only phrases the suggestion (App. D.3, H.6).
Follow-up: is the method step-by-step RAG within a fixed candidate set?Standard RAG appends retrieved content and fails even with screenshots. The same principle also drives proactive suggestion, where there is nothing to retrieve visually, and more than doubles the MCC of every memory baseline.Final reply posted

We express our sincere gratitude to all reviewers for their constructive feedback. The suggestions and insights provided have been invaluable in improving our paper. We hope that this summary clarifies the questions raised by the reviewers.

We sincerely thank ACs and SACs once more for your valuable time and insightful feedback. Your dedication contributes to making ICLR a more open, inclusive, fair, and equitable research community.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.