acceptodds
Withdrawn from ICLR 2027

MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent GUI agents have made substantial progress in visual grounding and action prediction, yet they remain brittle in long-horizon tasks that require maintaining task state across many interface transitions. Existing agents typically rely on raw history replay or text-only memory, which either overwhelms the model with redundant screenshots or discards localized visual evidence needed for future decisions. To address these limitations, we introduce MementoGUI, a plug-in agentic memory framework that equips MLLM-based GUI agents with MementoCore, a learned controller for online memory selection, compression, and retrieval. Rather than treating interaction history as a fixed context, MementoGUI formulates long-horizon GUI control as an online memory-control problem: working memory selectively preserves task-relevant interface events with textual summaries and ROI-level visual evidence, while episodic memory retrieves reusable past trajectories through learned relevance selection. MementoCore modularizes memory control into specialized operators for step processing, memory compression, episodic writing, and episodic selection, enabling plug-in memory augmentation without finetuning the GUI agent backbone. We further develop a scalable data curation pipeline that converts computer-use trajectories into memory-controller training data, introduce MementoGUI-Bench for evaluating long-horizon decision-making in GUI agents, and design MLLM-based metrics for semantic action matching, task progress, and memory consistency. Experiments on GUI-Odyssey, MM-Mind2Web, and MementoGUI-Bench show that MementoGUI consistently improves GUI agents over no-history, history-replay, and text-only memory baselines, with larger MementoCore backbones further strengthening memory-augmented GUI control.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Memory, Long-Horizon Control, GUI Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_o9HCOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to retain and retrieve useful history for long-horizon GUI tasks. MementoGUI adds a trainable memory controller to a frozen action model. It maintains working memory through textual summaries and local screenshot crops, and retrieves episodic memory from completed trajectories. The authors automatically construct training data and evaluate on GUI-Odyssey, MM-Mind2Web, and the new offline MementoGUI-Bench. The results show improved action prediction across several backbones, with contributions from local visual evidence and two-stage retrieval.

Critical strengths
  1. The method combines textual summaries, local images, and experience retrieval into a trainable memory layer that works with frozen GUI action models, offering a concrete approach to using interaction history more effectively.
  2. Experiments across backbones and component ablations support the design. Table 14 compares trained and untrained controllers of the same size while fixing the memory pipeline and action model, providing stronger evidence for training gains than simply adding context.
Critical weaknesses
  1. The central goal is better long-horizon control, but experiments mainly use recorded interface states and do not test recovery after incorrect actions, subsequent memory updates, or final task completion. Better offline action matching does not establish long-horizon gains in live interaction. This is the main limitation affecting the overall assessment.
  2. VAM and TPS read memory, potentially conflating memory–action consistency with action correctness. The action-only scores in Table 13 provide additional evidence but lack corresponding human validation; the human comparison in Table 9 also includes only configurations using the proposed memory. More independent evidence is needed to interpret improvements in action quality and task progress.
Questions for authors
  1. Could you provide closed-loop long-horizon results comparing final completion, repeated actions, and forgotten constraints?
  2. In light of Table 15, is the main advantage of the compact trained controller over the prompt-only Gemini controller latency, inference cost, or deployment requirements? Do existing measurements support this trade-off?
Reviewer_HwZbOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

MementoGUI introduces a plug-in multimodal memory system for long-horizon GUI agents while keeping the action backbone frozen. Its MementoCore controller maintains within-task working memory and cross-task episodic memory through four operations: step processing, memory compression, episodic writing, and episodic selection. The system stores concise text together with ROI images and retrieves reusable past experience through coarse retrieval followed by learned filtering. The authors also construct large-scale SFT/DPO supervision from computer-use trajectories and introduce MementoGUI-Bench. Experiments across three benchmarks and multiple GUI backbones generally show improvements over no-history, full-history, and text-summary baselines, with ablations supporting visual memory and two-stage retrieval.

Critical strengths
  1. The combination of textual summaries and ROI images preserves task state without replaying the full screenshot history, and the plug-in design avoids finetuning the GUI action backbone.
  2. The automatic trajectory-to-supervision pipeline provides substantial training data for writing, compression, episodic generation, and selection, reducing dependence on manual memory annotations.
  3. The experiments cover multiple GUI backbones and benchmarks, with useful ablations on visual memory, episodic retrieval, controller training, controller scale, and adapter sharing.
Critical weaknesses
  1. The four role-specific LoRA adapters introduce some additional routing and maintenance overhead. Since the shared-adapter results are already competitive at larger controller scales, it would be useful to discuss whether a unified LoRA with operation identifiers could offer a simpler deployment option.
  2. The paper provides component ablations, but the behavior of the learned write and retrieval decisions could be presented more directly. Simple statistics such as activation frequency, retained-memory length, and performance grouped by the number of writes or retrievals would make the controller easier to understand.
  3. The evaluation is primarily offline, as the authors acknowledge, and therefore does not fully measure how memory affects task completion when the agent's actions change subsequent observations. Adding evaluations on interactive benchmarks such as AndroidWorld and WebArena would substantially strengthen the evidence for real long-horizon GUI control and could raise the overall rating.
Questions for authors
  1. Have the authors considered jointly training the four memory operations in one LoRA using operation identifiers, and how would this affect quality and deployment complexity?
  2. Can the authors report write/retrieval activation frequencies, memory lengths, and performance across different levels of memory usage?
  3. Can the authors evaluate MementoGUI on AndroidWorld and WebArena and report environment-verified task completion under the same action-backbone and memory-ablation settings? Positive results on these interactive benchmarks would materially increase confidence and will raise the rating.
Reviewer_BbMoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MementoGUI, a multimodal memory framework for GUI agents. The authors train a controller to manage working and episodic memory through selective writing, compression, and retrieval of textual summaries and localized visual evidence. The paper also introduces an automatic supervision pipeline and an offline benchmark for memory-dependent GUI decisions.

Critical strengths
  1. A useful training setup for memory control. The paper turns decisions about which GUI evidence to retain and when to retrieve prior episodes into explicit prediction tasks, with supervision derived from recorded interactions.

  2. Moderate, consistent gains across backbones. MementoGUI improves action-matching scores and offline trajectory success rates across all four tested backbones on GUI-Odyssey.

Critical weaknesses
  1. The offline memory-update protocol is unclear. Equations 1 and 3 imply that memory updates use the previous predicted action, but the next recorded screenshot may reflect a different action. The paper does not explain how this mismatch is handled or whether history and memory baselines receive the same action information.

  2. Limited gains relative to inference cost. On GUI-Odyssey, working memory raises offline trajectory success rates by only 0.05-0.16 percentage points for GPT-5.5 and Gemini-3.1-Pro, while increasing latency by about 22-35% and token usage by 26-68%. The practical benefit of these small gains remains unclear given the additional cost.

Questions for authors
  1. How are mismatches between predicted actions and recorded screenshots handled in offline memory updates, and is the same convention used across baselines?

  2. How do the authors justify the additional latency and token usage given the small gains in trajectory success?

Reviewer_cx9qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MementoGUI, adding a new memory controller to GUI agents. During trajectory rollout, the controller takes in the current observations, previous actions, and working memory. Decisions are made on whether to save an event, and as the trajectory progresses and event entries accumulate, older memories are compressed. The controller also works with memory from already completed tasks, with saved summaries and representative crops, and retrieval is done based on relevance. The authors show results on various backbones and settings, while also introducing a new benchmark.

Critical strengths

The motivations for each component seem clear, and using a practical interface that can be used across various action models gives useful results that can be actually used.

Critical weaknesses

My main concern would be in novelty, since most of the memory component, crops, summarization, retrieval, reranking, etc. are familiar ideas already established in the field. A contribution would be the way the authors train the controller from the GUI traces, but I am not convinced that is enough of a scientific contribution to merit a ICLR paper.

The gains given by the method seem a bit uneven. The action matching part show substantial improvement, but the full system's advantage over history replay in trajectory success is marginal at best.

Questions for authors

Can you give better ablations for the relationship between visual updates and memory? Considering an agent could click the correct menu item just because the controller just zoomed in on the button, rather than due to memory, there should be a direct ablation between current-step outputs only versus earlier memory only.

On steps where the current screen omits a needed value or prior choice, how often does the correct prediction depend on retaining the corresponding earlier entry? Even a small annotated subset with that entry removed at inference would be informative.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.