acceptodds
Under review as a conference paper at ICLR 2027

MemGUI-Agent: An End-to-End Long-Horizon Mobile GUI Agent with Proactive Context Management

Abstract

MLLM-based mobile GUI agents have made substantial progress on short-horizon tasks, yet remain unreliable on long-horizon tasks that require retaining intermediate facts across many steps and app transitions. We attribute this limitation to ReAct-style prompting, which passively accumulates per-step records, leading to prompt explosion and dilution of critical cross-app facts. To address this, ***(i)*** we introduce **MemGUI-Agent**, an end-to-end long-horizon mobile GUI agent with proactive context management. MemGUI-Agent is built on text-as-ion (**CONACT**), which casts context management as first-class actions emitted by the same policy that selects UI actions. Instead of passively appending history, CONACT actively maintains three structured context fields: folded action history, folded UI state, and recent step record, enabling the agent to preserve critical UI facts while keeping context compact. To make proactive context management learnable across model scales, ***(ii)*** we construct **MemGUI-3K**, a 2,956-trajectory dataset with full CONACT annotations, enabling supervised training and analysis of proactive context management. ***(iii)*** Training an 8B model on MemGUI-3K produces **MemGUI-8B-SFT**, an 8B MemGUI-Agent that achieves the best open-data 8B performance on MemGUI-Bench and demonstrates strong out-of-distribution performance on MobileWorld. ***An anonymized release of the code, data, and trained models is publicly available at [https://memgui-agent-anonymous.github.io/](https://memgui-agent-anonymous.github.io/).***

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Mobile GUI Agents, Proactive Context Management, UI Memory, Long-Horizon GUI Tasks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_R9sPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MemGUI-Agent, which uses the CONACT protocol to have the same model generate GUI actions and context-management decisions while maintaining history summaries, UI factual memory, and a record of the most recent step. The authors construct MemGUI-3K, a dataset containing 2,956 trajectories, train an 8B model through supervised fine-tuning, and evaluate the method on MemGUI-Bench and MobileWorld.

Critical strengths
  1. Information loss during long-horizon, cross-app operations is a practical problem for mobile GUI agents. With the same 235B model and unchanged weights, CONACT improves performance on both MemGUI-Bench and MobileWorld, providing evidence for the practical value of context management.
  2. MemGUI-3K provides joint supervision for GUI actions, history compression, and factual memory. The 8B model's training results indicate that these behaviors can be learned from demonstrations, giving the dataset research value.
Critical weaknesses
  1. Limited methodological novelty. AgentFold [1], OpenGCUA [4] already implements learnable history folding, MemAct [2] incorporates memory editing into policy actions, and Mobile-Agent-v2 [3] uses cross-app factual memory. The paper mainly integrates these mechanisms and lacks a direct comparison with a GUI-adapted AgentFold augmented with factual memory. A substantive technical advance has therefore not been clearly established.
  2. Insufficient justification for the memory partition. The paper does not adequately explain why the state should be divided into history summaries, UI factual memory, and the most recent step record. The component ablations in Table 5 do not establish the necessity of this partition. Without matched comparisons against a unified memory store or a two-part state, the independent benefit of the three-part structure remains unclear.

[1] Rui Ye et al. AgentFold: Long-Horizon Web Agents with Proactive Context Management. arXiv preprint arXiv:2510.24699, 2025.

[2] Yuxiang Zhang, Jiangming Shu, Ye Ma, Xueyuan Lin, Shangxi Wu, and Jitao Sang. Memory as Action: Autonomous Context Curation for Long-Horizon Agentic Tasks. arXiv preprint arXiv:2510.12635, 2025.

[3] Junyang Wang et al. Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration. arXiv preprint arXiv:2406.01014, 2024.

[4] OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use

Questions for authors
  1. Please clarify CONACT's specific technical advance over AgentFold with a separate factual memory. Are there comparisons using the same backbone and inference budget that support the independent contribution of this advance?
  2. What justifies the three-part state partition? Are there experiments controlling the model, training conditions, and context budget that compare a unified memory store, a two-part state, and the current three-part state, separating the effect of state structure from the amount of recorded information?
5 comments
AuthorsOpenReview ↗

Response to Reviewer R9sP (1/2)

We thank Reviewer R9sP for recognizing the "practical value of context management" shown by our fixed-weight results and the "research value" of MemGUI-3K's joint supervision. We address each concern below.

W1 & Q1: Technical contribution and matched component evidence

Response. Our contribution comprises CONACT's per-step GUI context interface and MemGUI-3K's joint supervision of GUI actions and context management.

(1) Comparison with related work (New: Appendix A, Table 7\color{blue}{\text{Appendix A, Table 7}}).

MethodMobile GUISame policy controls history and actionsJoint mobile GUI/context training data
AgentFold [1]No\color{red}{\text{No}}; Web tool useYes\color{green}{\text{Yes}}; Policy-controlled foldingNo\color{red}{\text{No}}; Web trajectories
MemAct [2]No\color{red}{\text{No}}; Text QA/searchYes\color{green}{\text{Yes}}; Working-history editsNo\color{red}{\text{No}}; QA/search trajectories
Mobile-Agent-v2 [3]Yes\color{green}{\text{Yes}}No\color{red}{\text{No}}; Separate history planner; decision-agent memoryNo\color{red}{\text{No}}; not reported
OpenGCUA [4]Yes\color{green}{\text{Yes}}; Android SDK; desktop evaluationNo\color{red}{\text{No}}; Fixed history retention; model-controlled task ledgerNo\color{red}{\text{No}}; Evaluation trajectories
MemGUI-AgentYes\color{green}{\text{Yes}}Yes\color{green}{\text{Yes}}; Folding, UI-fact memory, and step recordsYes\color{green}{\text{Yes}}; MemGUI-3K: 2,956 trajectories

The policy column concerns control of both interaction history and task actions. The data column requires joint supervision of mobile GUI actions and context updates.

CONACT records task-relevant screen facts in <ui_observation> before a transition and passes them, with the action intent and action/result, to subsequent folding and memory decisions (Section 2\color{blue}{\text{Section 2}}).

OpenGCUA's arXiv v1 was posted on 15 October 2026, after the 25 September ICLR 2027 submission deadline; we include it in the comparison above.

(2) MemGUI-3K provides supervision for context management. Section 3.1 (Table 1)\color{blue}{\text{Section 3.1 (Table 1)}} (original submission, p. 5) reports 12.5% Pass@1 with ReAct and 7.5% with zero-shot CONACT for the 8B model on MemGUI-Bench-40. The tested 2B/4B models also decline, whereas 235B-Thinking improves. These results motivate training context-management decisions.

MemGUI-3K provides 2,956 GUI trajectories jointly supervising actions, folding, memory editing, and self-description. For the 8B model, SFT raises memory-trigger F1 from 19.9% to 48.0% and format compliance from 94.9% to 99.9% in the gold-context offline evaluation (Table 4\color{blue}{\text{Table 4}}, original submission, p. 8). These gains show improved memory-use decisions alongside reliable structured generation.

AuthorsOpenReview ↗

Response to Reviewer R9sP (2/2)

(3) Evidence for the GUI context-update protocol (Enhanced: Table 5\color{blue}{\text{Table 5}}). All eight component controls share unchanged Qwen3-VL-235B-Thinking weights, 40 tasks, temperature 0, matched initial states and task-step limits, and up to three attempts. F/M/SF/M/S denote folding/UI memory/self-description; Table 18\color{blue}{\text{Table 18}} gives the context setups. Token use and latency vary by configuration.

P@1, P@3, and IRR are percentages. Δ1/Δ3\Delta_1/\Delta_3 gives full CONACT minus each row’s P@1/P@3, in percentage points.

VariantFFMMSSP@1P@3MTPRIRRΔ1/Δ3\Delta_1/\Delta_3
ReAct0005.027.50.14319.535.0/35.0
MM only‡01017.542.50.35739.122.5/20.0
FF only†10022.532.50.17925.717.5/30.0
SS only00117.537.50.11032.222.5/25.0
F+MF+M11027.552.50.38037.812.5/10.0
M+SM+S01125.045.00.21433.115.0/17.5
F+SF+S10127.550.00.25034.812.5/12.5
Full CONACT11140.062.50.42951.00.0/0.0

† The folding-only component shares AgentFold's history-folding idea; AgentFold additionally retains Latest Interaction. ‡ The memory-only component edits UI facts; MemAct applies memory-as-action to working history.

The ablations support joint use of folding, persistent UI facts, and recent screen evidence. Each operation improves success with the other two enabled, and the full protocol leads all four metrics. Self-description supplies the screen facts and action intent used by subsequent folding and memory decisions.

Manuscript changes. We clarified the technical contribution in Section 1\color{blue}{\text{Section 1}} (lines 087–090) and strengthened the training motivation in Section 3.1\color{blue}{\text{Section 3.1}} (p. 5; lines 240–244). Appendix A (Table 7)\color{blue}{\text{Appendix A (Table 7)}} now compares mobile GUI support, policy control, and joint supervision, and explains the component controls’ relation to prior methods (pp. 14–15; lines 722–761). Table 5\color{blue}{\text{Table 5}} (p. 8) now shows each ablation’s gap to full CONACT.


W2 & Q2: Three-part state design

Response. The partition reflects different update rules: history (HH) is compressed, UI facts (MM) persist until explicitly edited, and the latest interaction (LL) is refreshed each step. This lets the policy compress past actions while retaining stored facts and recent screen evidence.

Over the past few days, we have expanded Table 5\color{blue}{\text{Table 5}} to report the complete 232^3 component ablation. All variants use the same zero-shot Qwen3-VL-235B-Thinking checkpoint, 40 tasks, and task-dependent step limits.

The distinct update roles are useful together in practice: full CONACT leads all four metrics. Removing folding, UI memory, or self-description lowers P@1 from 40.0% to 25.0%, 27.5%, and 27.5%, respectively.

Manuscript changes. We clarified the different update rules in Section 2\color{blue}{\text{Section 2}} (p. 3; lines 156–158). The expanded Section 4.3 (Table 5)\color{blue}{\text{Section 4.3 (Table 5)}} (p. 8; lines 421–431) reports all eight configurations, with their context setups detailed in Appendix I.1 (Table 18)\color{blue}{\text{Appendix I.1 (Table 18)}} (p. 38; lines 2006–2019).

References

[1] AgentFold: Long-Horizon Web Agents with Proactive Context Folding. ICLR 2026.

[2] Memory as Action: Autonomous Context Curation for Long-Horizon Agentic Tasks. Findings of ACL 2026.

[3] Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration. NeurIPS 2024.

[4] OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use. arXiv preprint, 2026.

Dear Reviewer R9sP,

Thank you again for your suggestions. In our rebuttal, we clarified the differences from prior work and the rationale for the three-part state, and added the complete component ablation.

We would welcome your thoughts on these clarifications and results before the discussion closes.

Best regards,

Authors of MemGUI-Agent

Reviewer_EidvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets long-horizon mobile GUI tasks where ReAct-style agents suffer from growing prompts and loss of cross-app facts. It proposes CONACT, a step protocol in which the same policy that chooses UI actions also emits history-folding directives, memory operations, and self-describing observation and intent fields, so the model maintains its own structured context. Applied zero-shot to Qwen3-VL-235B-Thinking, the protocol improves Pass@1 and Pass@3 on MemGUI-Bench over the same backbone, with gains also shown for Gemini-3.1-Pro-Preview and Seed-2.0-Pro on a 40-task subset. Because smaller models do not benefit zero-shot, the authors build MemGUI-3K, a dataset of about three thousand teacher trajectories on tasks expanded from the 128 MemGUI-Bench tasks themselves, and fine-tune an 8B model that improves over its base on MemGUI-Bench and MobileWorld.

Critical strengths

Backbone-matched zero-shot gains from a protocol change alone. The same Qwen3-VL-235B-Thinking weights gain more than ten Pass@1 points on the full benchmark just by switching to the CONACT protocol. The backbone study shows consistent gains on Gemini and Seed-2.0-Pro as well. This isolates the protocol from model capacity and from any training data, so it is the cleanest and most reusable finding in the paper.

Critical weaknesses

The dataset and 8B claims are evaluated largely in distribution. MemGUI-3K is expanded from the same 128 tasks used for evaluation, through entity substitution and task simplification. All rollouts also run in the same environment and apps. The only decontamination check is exact matching of task identifiers and instructions. The headline open-data 8B claim therefore compares an in-domain model against models that were not trained on this benchmark's tasks or environment. There is no control that fine-tunes the same 8B on the same trajectories in plain ReAct format, so the gain cannot be separated from in-domain exposure. On MobileWorld, the one likely out-of-distribution test, the model essentially ties the best open-data 8B baseline. Two of the three stated contributions rest on these numbers.

The ablation evidence rests on an anomalous baseline and small samples. The Qwen3-VL-235B-Thinking ReAct baseline scores nearly five times higher on the full benchmark than on the 40-task ablation subset, and the 2B and 4B models beat it under the same protocol. This suggests the baseline fails for reasons unrelated to context management. All component ablations, the backbone study, and the failure analysis are measured against it on 40 tasks with three attempts, where a single task is worth 2.5 points. No variance is reported, even though Pass@3 far exceeds Pass@1 at temperature 0. The claim that in-policy curation beats external curation is never tested against an external summarizer or memory module on the same backbone. The direction of the gains is probably real, but their size is hard to trust.

Questions for authors
  1. Why does the Qwen3-VL-235B-Thinking ReAct baseline score so much lower on the 40-task subset than on the full benchmark? Were the full-benchmark baseline numbers produced with the same ReAct prompt, and how were the 40 tasks selected? Please also explain how three attempts differ at temperature 0 and report per-attempt variance for the 40-task experiments.

  2. Since the 128 evaluation tasks are the seed tasks for MemGUI-3K, could you hold out a subset of them from expansion, retrain, and evaluate on those held-out tasks? Could you also add a control that fine-tunes the same 8B on the same trajectories rendered in plain ReAct format? These two results would show whether the dataset teaches context management or in-domain app flows.

  3. The token-cost figure reports only input tokens. What are the per-step output tokens and latency of CONACT versus ReAct, given the extra folding summary, UI observation, and action intent emitted at each step? As a suggestion, a fixed-interval summarizer driven by the same backbone would directly test the in-policy versus external-curation claim.

8 comments
AuthorsOpenReview ↗

Response to Reviewer Eidv (1/3)

We thank Reviewer Eidv for recognizing our backbone-matched zero-shot gains, noting "consistent gains on Gemini and Seed-2.0-Pro", and describing these results as "clean and reusable finding". We appreciate the constructive feedback and address each concern below.

W1 & Q2: Data overlap and 8B generalization

Response. We completed both requested training controls, seed-held-out retraining and same-trajectories ReAct-SFT, and added evaluation on MobileGym.

(1) Instruction overlap. MemGUI-3K expands the 128 seed templates into new task instances, as described in the submitted Section 3.2\color{blue}{\text{Section 3.2}} (original submission, lines 259–269). No exact task-identifier or raw-instruction matches were found against the evaluation tasks (original submission, lines 274–280).

Raw instruction similarity (New: Table 14\color{blue}{\text{Table 14}}). We use Qwen3-Embedding-8B to find each evaluation instruction's nearest neighbor among 2,661 training instructions. Threshold columns give the percentage at or above each cosine-similarity cutoff.

Evaluation setMedian≥0.70\geq 0.70≥0.80\geq 0.80
MemGUI-Bench0.86392.2%73.4%
MobileWorld0.58310.3%1.7%

MemGUI-Bench shares seed templates with distinct task instructions; MobileWorld is substantially less similar to the training instructions. We assess transfer through the external evaluations below.

(2) External transfer. On MobileWorld [1], MemGUI-8B-SFT improves over its base model from 9.4% to 17.9% SR (Table 3\color{blue}{\text{Table 3}}, original submission, p. 7).

We additionally evaluated all 256 MobileGym test tasks [2] at Pass@1 without benchmark-specific training. Task instances and budgets are matched; native context/parser protocols are detailed in Appendix J.2\color{blue}{\text{Appendix J.2}}.

MobileGym Pass@1 (New: Table 20\color{blue}{\text{Table 20}}). SR is in percent, with success counts in parentheses. The overall gain's paired-task bootstrap 95% CI is [2.73, 10.55] pp.

CompositionTasksQwen3-VL-8BMemGUI-8B-SFTΔ\Delta (pp)
Overall2567.81 (20)14.45 (37)+6.64
atomic2240.91 (9)45.45 (10)+4.55
sequential1108.18 (9)21.82 (24)+13.64
deep_dive682.94 (2)4.41 (3)+1.47
transfer560.00 (0)0.00 (0)0.00

The gains on MobileWorld and MobileGym support transfer beyond MemGUI-Bench. MobileGym's strongest improvement is on sequential tasks, which account for 15 of the 17 additional successes.

(3) Requested training controls. Both start from the same base 8B. Seed-held-out training excludes 32 seeds and all descendants. ReAct-SFT uses the same 2,661 source trajectories, preserving GUI supervision and rendering observations/intents as ordinary conclusions; memory-only steps are removed. Conversion details are in Appendix D.3\color{blue}{\text{Appendix D.3}}.

Training controls (New: Table 16\color{blue}{\text{Table 16}}). The two blocks use different evaluation sets.

Evaluation setAgentSuccessesPass@1 (%)
32 held-out seedsBase 8B, ReAct2/326.25
32 held-out seedsConAct-SFT, remaining seeds only6/3218.75
Full 128 tasksBase 8B, ReAct12/1289.4
Full 128 tasksSame-trajectories ReAct-SFT21/12816.4
Full 128 tasksMemGUI-8B-SFT, ConAct30/12823.4

Together, these controls support generalization across seed families and the value of learning and using CONACT. The held-out model improves on excluded families; with the same source trajectories, CONACT exceeds ReAct-SFT by 7.0 pp.

Manuscript changes. We clarified the evaluation scope in Section 3.2\color{blue}{\text{Section 3.2}} (lines 281–284) and Section 4.1\color{blue}{\text{Section 4.1}} (lines 375–380). We report the overlap audit in Appendix C.7 (Table 14)\color{blue}{\text{Appendix C.7 (Table 14)}} (p. 21), MobileGym evaluation in Appendix J.2 (Table 20)\color{blue}{\text{Appendix J.2 (Table 20)}} (p. 40), and training controls in Appendix D.3 (Table 16)\color{blue}{\text{Appendix D.3 (Table 16)}} (p. 23; lines 1206–1215).

AuthorsOpenReview ↗

Response to Reviewer Eidv (2/3)

W2 & Q1: Baseline consistency and the evaluation protocol

Response. Both evaluations use the same ReAct prompt. We adopt the published MemGUI-Bench-40 subset [3] to limit repeated ablation costs.

(1) Task selection and baseline differences. Our task IDs exactly match the benchmark's 40-task list, a subset of its 128-task list. MemGUI-Bench describes the sampling in Appendix A.9.4\color{blue}{\text{Appendix A.9.4}}. Its Tables 2–3\color{blue}{\text{Tables 2–3}} report close full-set/subset Pass@1 for M3A (32.8%/32.5%) and Agent-S2 (27.3%/27.5%), supporting the subset's use for controlled comparisons.

Full-set/subset Pass@1 is 9.4%/12.5% for the base 8B and 24.2%/5.0% for 235B-Thinking. Restricting the full-benchmark 235B logs to these same 40 tasks gives 12.5%, compared with 5.0% in the separate ablation run. Thus, the full/subset gap also includes variation between evaluation runs. All component comparisons use the same tasks, checkpoint, initial-state setup, budgets, and evaluator.

(2) Pass@1 versus Pass@3. Published GUI-agent evaluations also show gains from additional attempts. Table 1\color{blue}{\text{Table 1}} of OpenMobile [4] reports OpenMobile-8B Pass@1/Pass@3 of 64.7%/78.0% on AndroidWorld and 17.7%/24.8% on MobileWorld. At temperature 0, observations and interaction outcomes can vary across complete GUI rollouts.

(3) Retry protocol and observed outcomes. Following MemGUI-Bench [3], each task has up to three attempts, with a reset after failure and stopping after success. Pass@3 is cumulative coverage, and later attempts include only earlier failures.

Retry outcomes (New: Table 8\color{blue}{\text{Table 8}}). Entries show successful / attempted tasks on MemGUI-Bench-40.

AgentAttempt 1Attempt 2Attempt 3Pass@3
ReAct2/406/383/3211/40 (27.5%)
Full ConAct16/405/244/1925/40 (62.5%)

The published task subset, matched evaluation conditions, and explicit retry accounting provide a controlled, transparent basis for the component comparisons.

Manuscript changes. We have clarified the subset and stopping rule in Appendix B.1\color{blue}{\text{Appendix B.1}} (lines 770–772, 790–792). Appendix B.2 (Table 8)\color{blue}{\text{Appendix B.2 (Table 8)}} now explains the baseline comparisons and retry accounting (p. 16; lines 812–835).

AuthorsOpenReview ↗

Response to Reviewer Eidv (3/3)

Q3: End-to-end cost and a fixed-interval summarizer

Response.

(1) End-to-end efficiency and cost. CONACT trades higher generation and time costs for better task success and slower context growth. We analyze all 128 first attempts, including failures, from the fixed-weight 235B runs in Table 2\color{blue}{\text{Table 2}}.

Aggregate cost (New: Table 22\color{blue}{\text{Table 22}}). Token/call means use recorded provider usage; output includes reasoning. ≥ denotes a lower bound on total tokens/task. Wall-clock time/step averages each task’s end-to-end time divided by benchmark steps, including model, agent, and environment time.

MethodInput tokens/callOutput tokens/callRecorded total tokens/taskWall-clock time/stepEnd-to-end time/taskP@1 (%)
ReAct4,528.21124.35212,672.987.58 s324.96 s31/128 (24.2)
Full CONACT5,228.70631.07≥252,885.8419.44 s846.00 s48/128 (37.5)

Cost as trajectories progress (New: Table 23\color{blue}{\text{Table 23}}). We use 3,914 matched (task, turn) pairs with usage recorded for both methods. Ratios denote full CONACT/ReAct.

StepsPaired calls/tasksReAct input tokensCONACT input tokensInput-token ratioTotal-token ratio
1–201,897/1283,954.294,705.471.19×1.29×
21–40993/764,394.565,191.401.18×1.30×
41–60562/384,832.405,489.171.14×1.24×
61–80281/205,327.645,819.851.09×1.19×
81–10099/95,896.256,109.621.04×1.13×
101+82/46,719.916,195.840.92×1.01×

CONACT slows token growth as trajectories lengthen: task-fixed-effect slopes are 25.7% lower for input and 16.9% lower for total tokens. The 101+ interval covers 82 paired calls from 4 tasks.

Success-normalized interaction cost (New: Table 24\color{blue}{\text{Table 24}}). All first-attempt steps and calls, including failures, are divided by the number of solved tasks.

MethodP@1 solvedBenchmark stepsSteps/solved taskModel callsCalls/solved task
ReAct31/128 (24.2%)5,725184.685,851188.74
Full CONACT48/128 (37.5%)5,889122.69 (−33.6%)6,014125.29 (−33.6%)

CONACT uses its interactions more productively: steps and calls per solved task both fall by 33.6%.

(2) Same-backbone fixed-interval summarizer. We add a separate Qwen3-VL-235B-Thinking summary call to ordinary ReAct every 5 steps, capped at 512 tokens, with no CONACT memory actions or self-description. All three methods use unchanged 235B weights, the same 40 tasks, initial states and task-step budgets, temperature 0, and Pass@1. Summary calls consume no environment steps; the full protocol is in Appendix K.2\color{blue}{\text{Appendix K.2}}.

Fixed-interval summarizer comparison (New: Appendix K.2, Table 25\color{blue}{\text{Appendix K.2, Table 25}}). The ReAct and full-CONACT rows use archived evaluations under the same protocol.

MethodSuccesses / tasksPass@1 (%)
ReAct2/405.0
ReAct + fixed-interval summarizer10/4025.0
Full CONACT16/4040.0

The complete CONACT protocol improves on scheduled history summarization, adding 15.0 pp Pass@1 under the same task-step budget. The schemes differ in folding, memory, and step-record design.

Manuscript changes. We clarified the input-context scope in Section 1\color{blue}{\text{Section 1}} (lines 103–104), Section 4.1\color{blue}{\text{Section 4.1}} (lines 368–371), and Figure 1(a)\color{blue}{\text{Figure 1(a)}} (p. 1). Appendix K.1\color{blue}{\text{Appendix K.1}} reports measured costs and timing definitions (lines 2218–2239; Tables 22–23\color{blue}{\text{Tables 22–23}}, p. 42; Table 24\color{blue}{\text{Table 24}}, p. 43). We also added the summarizer comparison in Appendix K.2 (Table 25)\color{blue}{\text{Appendix K.2 (Table 25)}} (p. 43; lines 2280–2300).

References

[1] MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments. ACL 2026.

[2] MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research. EMNLP 2026.

[3] MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments. ACM MM 2026.

[4] OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis. COLM 2026.

Reviewer_wdVMOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses context explosion and cross-app information loss in long-horizon mobile GUI agents. The authors propose CONACT, a protocol where a single VLM policy jointly outputs UI actions and context management operations (history folding, UI memory add/update/delete, and step-level self-description). To make these behaviors learnable for smaller models, they construct the MemGUI-3K dataset (2,956 trajectories, 64,430 step samples) and fine-tune MemGUI-8B-SFT. Experiments on MemGUI-Bench and MobileWorld show consistent gains across both zero-shot (across three VLM backbones) and SFT settings.

Critical strengths
  1. Putting history folding and UI memory operations directly into the model's action space is a very clean and practical design for long-horizon GUI agents. Unlike external memory modules or heuristic summarization pipelines, everything is handled in a single forward pass. The zero-shot results in Tables 2, 3, and 6—especially the consistent gains across Qwen3-VL-235B-Thinking, Gemini-3.1-Pro-Preview, and Seed-2.0-Pro—convincingly show that the protocol works across different model families without relying on training tricks.

  2. MemGUI-3K is a well-constructed dataset with step-level reasonableness filtering, and training an 8B model on it transfers well to the out-of-distribution MobileWorld benchmark (+8.5% SR over the base 8B model). The offline skill evaluation (Table 4) and failure analysis (Figure 7) also give a clear picture of where the improvements come from.

Critical weaknesses
    1. In Table 5, the notation (+ UI memory actions, + history folding, + self-describing step) makes it slightly unclear whether each row is added on top of the ReAct baseline independently or cumulatively, and pairwise combinations among the three components (FF, MM, SS) are not shown. Also, since the model already receives the current screenshot ItI_t and outputs <thinking> at each step, it is not fully clear from Section 2 alone why <ui_observation> and <action_intent> are needed rather than being redundant.
  1. The online evaluation focuses on two long-horizon mobile benchmarks (MemGUI-Bench and MobileWorld); it remains unclear whether zero-shot CONACT also helps on standard mobile benchmarks like AndroidWorld or desktop environments like OSWorld.

Questions for authors
  1. Could you clarify the exact prompt context setup for each row in Table 5 and report the remaining component combinations (i.e., a full 232^3 ablation over folding, memory, and self-description ) ? Walking through a short two-step example of how Lt+1L_{t+1} is used after a screen transition would also help clarify why self-description is not redundant with the screenshot and <thinking>.

  2. Have you evaluated whether zero-shot CONACT improves performance on AndroidWorld (especially its memory-related tasks) or desktop benchmarks such as OSWorld?

  3. A minor detail in Section 1(line 057): the paper states that AndroidWorld has an average of 8.4 steps, whereas the MobileWorld paper reports 14.3 steps for AndroidWorld. Could you clarify where this difference comes from?

4 comments
AuthorsOpenReview ↗

Response to Reviewer wdVM (1/2)

We thank Reviewer wdVM for recognizing CONACT's action-space design as "a very clean and practical design", noting that our zero-shot results "convincingly show that the protocol works across different model families", and describing MemGUI-3K as "a well-constructed dataset". We appreciate the constructive feedback and address each concern below.

W1 & Q1: Component settings, complete ablation, and the role of self-description

Response. The submitted rows in Table 5\color{blue}{\text{Table 5}} (original submission, p. 8) are independent configurations. Let FF, MM, and SS denote history folding, UI memory actions, and self-description; “+ self-describing step” is M+SM+S.

(1) Prompt and context settings (New: Table 18\color{blue}{\text{Table 18}}). All variants receive a task goal and the current screenshot and emit <thinking> and <tool_call>:

  • FF adds <folding> before <tool_call> from step 2 onward and folds history into Folded Action History HH.
  • MM enables memory_add, memory_update, and memory_delete, maintaining Folded UI State MM. Each step executes exactly one UI or memory action; without MM, only UI actions are available.
  • SS replaces <conclusion> with <ui_observation> and <action_intent>. Together with the action and result, these fields form the self-describing step record.

The table lists next-step context beyond the goal and screenshot; “chronological” means appending without folding.

VariantContext supplied at the next step
ReActTask Progress: chronological conclusion records
MM onlyChronological conclusion records + Folded UI State
FF onlyFolded Action History
SS onlyTask Progress: chronological self-description and action-summary records
F+MF+MFolded Action History + Folded UI State
M+SM+SChronological self-description and action-summary records + Folded UI State
F+SF+SFolded Action History + Recent Step Record
Full CONACTFolded Action History + Folded UI State + Recent Step Record

(2) Complete component ablation (Enhanced: Table 5\color{blue}{\text{Table 5}}). All eight variants use the same zero-shot Qwen3-VL-235B-Thinking [1] checkpoint on MemGUI-Bench-40 [2], with up to three attempts per task, temperature 0, identical task-dependent budgets and initial states, and the same MemGUI-Eval protocol.

P@1, P@3, and IRR are percentages. Δ1/Δ3\Delta_1/\Delta_3 gives full CONACT minus each row’s P@1/P@3, in percentage points.

VariantFFMMSSP@1P@3MTPRIRRΔ1/Δ3\Delta_1/\Delta_3
ReAct0005.027.50.14319.535.0/35.0
MM only01017.542.50.35739.122.5/20.0
FF only10022.532.50.17925.717.5/30.0
SS only00117.537.50.11032.222.5/25.0
F+MF+M11027.552.50.38037.812.5/10.0
M+SM+S01125.045.00.21433.115.0/17.5
F+SF+S10127.550.00.25034.812.5/12.5
Full CONACT11140.062.50.42951.00.0/0.0

Each component improves success when the other two remain enabled. With folding and UI memory fixed, self-description adds 12.5/10.0 pp in P@1/P@3, supporting its role in subsequent context updates.

(3) Why self-description helps across a screen transition. The screenshot ItI_t supplies current visual evidence, and <thinking> supports the current decision. Self-description stores the observation and action purpose in Lt+1L_{t+1}, together with the action and result, for subsequent folding and memory decisions. This preserves task-relevant screen facts across transitions. We illustrate two additional steps from the CompareProductSpecs trajectory shown in Figure 6\color{blue}{\text{Figure 6}} (original submission, p. 9):

  • Step 45, Amazon → Home: the agent records the Pixel 8 Pro specifications, already saved in MM at step 44, and its intent to open Joplin, then presses Home. These fields and the action populate L46L_{46}.
  • Step 46, Home → Joplin: the screenshot shows app icons; L46L_{46} retains the specifications and intent. The agent folds step 45 into a summary and clicks Joplin.

Manuscript changes. We updated Section 4.3 (Table 5)\color{blue}{\text{Section 4.3 (Table 5)}} (p. 8; lines 421–431) and clarified self-description in Section 2\color{blue}{\text{Section 2}} (lines 207–211). Appendix I.1 (Table 18)\color{blue}{\text{Appendix I.1 (Table 18)}} (p. 38; lines 2006–2019) now specifies each context setup, and Appendix I.2\color{blue}{\text{Appendix I.2}} (lines 2038–2051) gives the two-step example.

AuthorsOpenReview ↗

Response to Reviewer wdVM (2/2)

W2 & Q2: Zero-shot transfer to AndroidWorld and desktop environments

Response. The zero-shot gains extend to additional mobile and desktop environments. We compare CONACT with ReAct-style prompting on AndroidWorld [3] and OSWorld2.0 [4], fixing the backbone within each benchmark.

Cross-environment results (New: Table 19\color{blue}{\text{Table 19}}). Scores are percentages; Δ\Delta is in percentage points.

Benchmark / splitFixed backboneEvaluation setMetricAction-ThoughtCONACTΔ\Delta
AndroidWorldQwen3-VL-235B-ThinkingAll 116 tasksMean success score47.459.9+12.5
AndroidWorld memory subsetQwen3-VL-235B-ThinkingOur 9 memory-related tasksSuccess rate0.011.1+11.1
OSWorld2.0Qwen 3.7-Plus35 randomly sampled tasks; 500-step capMean task score12.521.3+8.8

AndroidWorld uses one instance per template and the official mean-score aggregation, including partial credit. Our metadata defines the nine-task memory subset.

Manuscript changes. Appendix J.1 (Table 19)\color{blue}{\text{Appendix J.1 (Table 19)}} (p. 39; lines 2061–2069) now reports these results and protocols, with a pointer in Section 4.1\color{blue}{\text{Section 4.1}} (lines 372–373).


Q3: Clarification of the AndroidWorld step count

Response. The two values use different counting conventions. AndroidWorld's 8.4 (original submission, line 057) averages the optimal_steps reference counts in its official metadata [3]: 976 / 116 = 8.4138 over all 116 tasks. MobileWorld's 14.3 is the mean executed action count of GPT-5 + UI-Ins-7B across successful and failed trajectories under a 50-step cap (MobileWorld [5], Section 4.5, Figure 5\color{blue}{\text{Section 4.5, Figure 5}}).

Manuscript changes. We have clarified that 8.4 denotes reference steps in Section 1\color{blue}{\text{Section 1}} (lines 060–063) and explained the two counting conventions in Appendix B.1\color{blue}{\text{Appendix B.1}} (lines 776–780).

References

[1] Qwen3-VL Technical Report. arXiv technical report, 2025.

[2] MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments. ACM MM 2026.

[3] AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. ICLR 2025.

[4] OSWorld2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks. arXiv preprint, 2026.

[5] MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments. ACL 2026.

I have no further questions. I have already given it a relatively high score, and I will maintain that score. Furthermore, I want to emphasize that this paper is indeed one that I consider well-written and with a clear motivation.

Dear Reviewer wdVM,

Thank you for recognizing our work. We have incorporated your suggestions into the revised manuscript. Thank you again for your time and effort throughout the review and discussion!

Best regards,

Authors of MemGUI-Agent

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

Thank you for your careful reading of MemGUI-Agent. We appreciate the strengths you highlighted in the reviews.

  • A practical problem and a clear design [R9sP, wdVM]. R9sP highlighted how easily agents lose information across apps. wdVM recognized the value of letting the same policy edit memory and act on screen.
  • Evidence from fixed-weight comparisons [wdVM, Eidv, R9sP]. All three reviewers recognized gains from changing the context protocol while keeping model weights fixed. Eidv described this as “the cleanest and most reusable finding in the paper.”
  • A dataset for learning what to remember [wdVM, R9sP]. wdVM noted the step-level filtering in MemGUI-3K, and R9sP highlighted its joint supervision of GUI actions, history folding, and factual memory.

Your questions motivated four revisions to clarify what accounts for the gains, how far they transfer, and what they cost.

  1. The three context updates work together [wdVM Q1, R9sP Q1–Q2, Eidv Q1]. After a screen changes, the recent step record preserves what the agent saw and why it acted. This gives the policy evidence to draw on when it folds history or updates a fact. On all 128 tasks, removing any one component cost 6–7 first-attempt successes and also lowered Pass@3. In particular, self-description helped even when folding and factual memory were both enabled. Full CONACT also beat ReAct in both runs on the same 40 tasks, including the run with the higher ReAct baseline (Tables 5 and 7\color{blue}{\text{Tables 5 and 7}} and Appendix L, Table 26\color{blue}{\text{Appendix L, Table 26}}).
  2. CONACT adds value beyond exposure to the source trajectories [Eidv Q2]. MemGUI-Bench supplied the seed templates, so the full-set 8B score is an in-domain result. Given the same training trajectories and starting weights, MemGUI-8B-SFT solved 30/128 tasks, compared with 21/128 for plain ReAct-SFT. Those nine additional successes show the value of training and using CONACT with the same source data. In a separate test, the retrained agent still improved over its base on 32 seed families excluded from training (Table 16\color{blue}{\text{Table 16}}).
  3. The gains carry over to other environments [wdVM Q2, Eidv Q2]. Switching from ReAct to CONACT with the backbone fixed improved scores on AndroidWorld and on 35 sampled OSWorld2.0 desktop tasks. For the trained 8B agent, the evidence for transfer is its improvement over its own base on both MobileWorld and MobileGym. On MobileGym it gained 6.64 pp without benchmark-specific training (Tables 3, 19 and 20\color{blue}{\text{Tables 3, 19 and 20}}).
  4. Managing context improves task success at a time cost [Eidv Q3]. CONACT solved more tasks than ReAct with a fixed-interval summarizer under the same backbone and task-step budgets. Its input context also grew more slowly than ReAct’s, and it used fewer model calls per solved task. Average token use per call was higher, and each task took about 2.6× as long. We revised Figure 1(a)\color{blue}{\text{Figure 1(a)}} to specify input context and report end-to-end cost separately (Tables 22–25\color{blue}{\text{Tables 22–25}}).

We believe these comparisons address the main concerns about CONACT’s contribution and the scope of the 8B results. Thank you for the suggestions that helped us test those claims more directly.

Best regards,

Authors of MemGUI-Agent

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.