acceptodds
Withdrawn from ICLR 2027

Stop Thinking when Muscles Remember: Boosting Mobile GUI Agents with Memoir

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mobile graphical user interface (GUI) agents are gaining wide attention in both academia and industry thanks to their ability to automate everyday tasks. However, they suffer from poor execution efficiency due to the expensive per-step vision-language model (VLM) inference, which prohibits their wide deployment. Existing methods accelerate GUI agents by reducing input information for VLMs, overlooking the high sequential output overhead. In this paper, we identify that the long and redundant thinking tokens in VLM outputs are the key efficiency bottleneck for mobile GUI agents and propose to equip GUI agents with muscle memory to materialize redundant thinking at runtime. Based on the observation, we design Memoir, a Muscle-mEMOry traIning and Recall framework, to accelerate GUI agents. The framework adopts a domain-specific language (DSL), i.e., reduction DSL, to store muscle memory, an offline muscle training pipeline to learn muscle memory, and a dual-gated memory trigger to efficiently and reliably use muscle memory. Our evaluation results on AndroidControl show that Memoir accelerates GUI agents by up to 3.25x over state-of-the-art baselines while raising step accuracy from 65.2% to 69.3%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
GUI, VLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_N6oYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper observes that mobile GUI agents waste most of their latency on redundant per-step chain-of-thought tokens, and proposes to replace that repeated reasoning with "muscle memory": reusable reduction recipes, learned offline from past successful executions, that crop the screen to the few relevant GUI elements and are invoked at runtime by a lightweight dual-gated (OOD-distance + learned scorer) trigger. MEMOIR thereby compresses both the VLM's input and its thinking output, cutting per-step latency substantially on AndroidControl while slightly improving step accuracy.

Critical strengths

Accurate problem framing: identifying output-side thinking tokens as the neglected real bottleneck. Prior pruning- and shortcut-based methods compress only the input (visual tokens, multi-step actions). By measuring the prefill/decode throughput gap and breaking down output tokens, the authors show that thinking tokens make up over 84% of the output and that existing methods barely reduce output tokens — explaining why earlier speedups fall short. The contribution is a shift in perspective: from "letting the VLM see less" to "letting the VLM think less."

Critical weaknesses
  1. "Without sacrificing accuracy" is thinly supported. High-level success moves only 10.8% → 11.3% over 1,543 episodes with no significance test, and the aggressive calibration setting collapses accuracy to 63.5%, so accuracy preservation looks conditionally tuned rather than guaranteed. The claim that cropping reduces "distraction" is also unverified: no controls (cropping with thinking preserved, random crops of equal size, shuffled candidate order) rule out alternative explanations.

  2. Only per-step speedup is demonstrated, not faster task completion. Because evaluation is offline, the agent never executes actions: inputs are recorded screenshots and episode length is fixed to the human trajectory, so retrying and error correction cannot occur. Latency evidence is therefore per-step only, and the sole end-to-end-sounding claim ("cuts end-to-end latency by 3.25×") is drawn directly from that per-step column.

  3. The memory library is built from successful trajectories and validated with ground truth, so in a static dataset the state always lies in-distribution; the reported hit rate is an artifact of the protocol, and the policy-induced distribution shift that threatens this method in real execution is never tested.

Questions for authors

How would the memory hit rate change under interactive execution?

The memory library is built from past successful trajectories, and the appendix states that ground-truth information is used during the offline filtering stage. In a static dataset the current state always lies within the seen distribution, so the reported 92.4% hit rate can be regarded as an artifact of the protocol. Under real execution, where the agent's own actions alter the interface and produce states absent from human trajectories, by how much would the b_full fallback frequency rise, and how far would the speedup degrade?

Reviewer_WUG7OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper identify a bottleneck of GUI Agents, and proposes MEMOIR a memory based framework to accelerate mobile GUI agents by reducing redundant thinking tokens. It introduces a reduction DSL to represent reusable execution patterns, an offline training pipeline to learn them from successful trajectories, and a dual gated mechanism to retrieve suitable memories at runtime.

Critical strengths
  1. The paper addresses a realistic problem. The authors identify a practical runtime efficiency bottleneck in GUI agents and analyze its underlying causes.
  2. The proposed model achieves strong experimental results.
Critical weaknesses

1.More dataset should be studied, not only AndroidControl. Missing Strong Baselines. Lacks comparisons with adaptive thinking methods and heuristic GUI cropping. 2.Unclear Muscle Memory Contribution. The proposed muscle memory primarily stores context reduction recipes and thinking budgets rather than reusable reasoning patterns. It is unclear how this differs fundamentally from conventional adaptive computation and context selection.

Questions for authors

Could the authors clarify what reusable knowledge the proposed muscle memory captures beyond context reduction recipes and adaptive thinking budgets? A comparison with a simple adaptive budget and heuristic cropping baseline would help establish the unique contribution of the memory mechanism.

Reviewer_DwF7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that CoT output tokens, not visual input tokens, dominate the latency of mobile GUI agents. It proposes MEMOIR, which learns "muscle memory" entries offline. Each entry is a DSL recipe plus a budget. The recipe crops the screenshot around selected a11y elements. The budget decides how the VLM is used. At runtime, two gates choose the entry: a cosine OOD filter and a per-entry GBDT scorer. On AndroidControl with three 8B VLMs, MEMOIR reports up to 3.25x speedup. With Qwen3-VL, it also raises step accuracy from 65.2 to 69.3.

Critical strengths

The paper shows that decode dominates latency. Visual-token pruning gives at most 1.06x speedup. This is a useful reframing for work on efficient GUI agents.

Critical weaknesses

Results are offline and on AndroidControl only. They are not split into in-domain and unseen-app/category/task subsets. That split matters most for a memory-based method. There is no online benchmark.

Questions for authors

Please break the test results down into in-domain and unseen-app/category/task subsets. Which data do the memories and the two calibration sets come from?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.