- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DeltaReplay, a training-free, step-level memory reuse framework for mobile GUI agents. It represents previously successful interaction trajectories as paths in a read-only UI transition graph and distinguishes task-independent operations from task-specific parameters at reuse time. Given a new instruction and the current screen, it chooses among verified replay (Route A), parameter adaptation/re-grounding (Route B), and discarding the retrieved step in favor of the base agent (Route C). Structural screen/element matching and an LLM-based step selector implement the routing. On AndroidWorld-116 and SPA-Bench-60, it improves task success over same-backbone agents by 7.8–10.3 and 16.6–25.0 percentage points, respectively; it also reduces latency in the high-memory-coverage settings with the Qwen3.8-Flash backbone. Route statistics and ablations provide supporting evidence for the importance of checking applicability before reusing a step.
Critical strengths
- Principled use of partial memories. The distinction between direct replay, parameter adaptation, and fallback addresses a real limitation of trajectory-level reuse. The method checks both whether a recorded action can be mapped to the current screen and whether it still advances the current instruction, rather than forcing an all-or-nothing decision. The read-only design and explicit abstention are especially useful for reducing harmful negative transfer.
- Promising empirical gains with diagnostic analyses. The study evaluates two distinct backbones across AndroidWorld-116 and SPA-Bench-60 and separates cases with identical, parameter-shifted, partially related, and unavailable memories. The same-template AndroidWorld gain with Qwen3.8-Flash (63.6% to 86.4%) and the SPA-Bench same-family gains (from 0% to 33.3%/46.7%) are particularly informative. Per-step route statistics and comparisons against naive replay support the premise that selective reuse is better than unconditional copying.
Critical weaknesses
- Generalizability of the evaluation setting remains uncertain. Retrieval uses benchmark-provided task/template/family identifiers rather than only the natural-language instruction. Moreover, SPA-Bench memory contains successful base-agent runs for 27 of the 60 evaluated tasks, including some runs appearing in the reported base-agent results. This is a legitimate exact-task memory setting, not necessarily leakage, but it makes the aggregate score less informative about transfer to previously unseen goals. Successful-run-only memory construction also correlates coverage with task difficulty, as the authors acknowledge. An independent evaluation emphasizing new goals, paraphrased instructions, and nonmatching app states would better substantiate the method’s practical breadth.
- The evidence for individual design choices and outcome reliability is incomplete. The ablations do not isolate individual mechanisms: “w/o Route C” also removes the screen check, while “w/o semantic matching” removes both the screen check and the selector. Consequently, Table 4 cannot by itself attribute the declines to discarding or semantic selection independently, and a dedicated Route B ablation is missing. Each task is evaluated only once; the reported success metric uses an LLM judge with no human validation, and agreement with available AndroidWorld checkers is 91.5%. These limitations matter when interpreting modest aggregate differences and claiming reliable superiority. Comparisons to closely related executable-memory methods are also mostly conceptual rather than experimental.
Questions for authors
- Could the authors report results when SPA-Bench exact-task trajectories are excluded from memory, and when retrieval is performed from the instruction alone without benchmark task IDs? In particular, how much of the improvement survives on unseen goals with only related records?
- Can Route A, Route B, screen verification, semantic selection, and Route C fallback be removed one at a time while holding other parts fixed? How does the method compare under a matched interface and memory budget to an executable-memory baseline (e.g., AppAgentX or a re-ground-or-discard approach)?
- Could the authors provide repeated-run variability or paired confidence intervals and a small blind human audit of task-success judgments, especially for episodes where the LLM judge disagrees with an AndroidWorld checker?
3 comments
Response to Reviewer nZd3 (Part 2/3): Independent Component Ablations and Executable Memory Comparisons (Q2)
Q2 Can Each Component Be Independently Ablated and Compared with Executable Memory Methods Under Identical Interfaces and Memory Budgets?
Response: Thank you for your suggestion. Regarding the ablation experiments, we clarify and supplement independent controls as follows:
- In the original Table 4,
w/o Route Cremoved both Route C fallback and the screen check (45.5%);w/o semantic matchingremoved both the screen check and the Selector (20.0%). To strictly distinguish the independent contribution of each component, we supplemented independent ablation controls that disable single components separately. - For the comparison between Route B and executable memory methods under identical interfaces and memory budgets, we supplemented two controls:
- w/o Route B: retains the original screen check and Selector, disables parameter adaptation, continues to replay old actions that can be used directly, and hands over actions requiring parameter modification to Route C;
- AppAgentX-style check, w/o Route B: also disables Route B, but replaces the screen check and Selector with a single LLM applicability judgment, selecting directly executable actions based on the current task, page, and the same batch of retrieved records, or rejecting all candidates and handing over to Route C.
- For Route A, we supplemented a control removing only direct replay (
w/o Route A): retaining memory retrieval, screen check, Selector, Route B parameter adaptation, and Route C, and handing over only actions originally executed directly by Route A to Route C.
To this end, we supplemented independent ablations of the screen check and Selector on the original 30 AndroidWorld Same-app tasks using Qwen3.8-Flash; on the original 22 Same-template tasks, we supplemented independent controls for w/o Route A, w/o Route B, AppAgentX-style check, and w/o Route C, keeping task instances, memory banks, execution budgets, and evaluation pipelines fixed. All controls used identical observation, target grounding, and action execution interfaces, as well as the same batch of retrieved records, identical record counts, and input length limits (existing results from the paper are retained in the table below):
| Task Group | Experimental Group | Screen Check | Selector | Tasks | Successes | Success Rate (%) | Model Calls/Step | A/B/C (%) |
|---|---|---|---|---|---|---|---|---|
| Same-app | Full | On | On | 30 | 15 | 50.0 | 2.19 | 0/14/86 |
| Same-app | w/o selector (New) | On | Off | 30 | 8 | 26.7 | 1.25 | 0/40/60 |
| Same-app | w/o screen check (New) | Off | On | 30 | 9 | 30.0 | 2.40 | 0/22/78 |
| Same-app | Joint Removal (Original Table 4 w/o semantic matching) | Off | Off | 30 | 6 | 20.0 | 0.60 | 0/73/27 |
| Same-template | Full | On | On | 22 | 19 | 86.4 | 1.00 | 41/15/44 |
| Same-template | w/o Route A (New) | On | On | 22 | 16 | 72.7 | 1.85 | 0/18/82 |
| Same-template | w/o Route B (New) | On | On | 22 | 17 | 77.3 | 1.60 | 41/0/59 |
| Same-template | AppAgentX-style check, w/o Route B (New) | Replaced with unified LLM judgment | Replaced with unified LLM judgment | 22 | 18 | 81.8 | 1.90 | 45/0/55 |
| Same-template | Original w/o Route C (Original Table 4 Joint Ablation) | Off | On | 22 | 10 | 45.5 | 0.17 | 68/32/0 |
| Same-template | w/o Route C (New Independent Control) | On | On | 22 | 11 | 50.0 | 0.18 | 67/33/0 |
Summary: Removing the Selector or the screen check individually caused the success rate to drop from 50.0% to 26.7% and 30.0%, respectively, and removing both jointly caused it to drop further to 20.0%, indicating that both checks contribute and their joint absence causes a greater loss. After removing the screen check, the number of calls per step increased from 2.19 to 2.40 because the lack of low-overhead structural filtering caused more mismatched pages to directly trigger LLM Selector calls. Removing Route A, Route B, and Route C individually caused the Same-template success rate to drop to 72.7%, 77.3%, and 50.0%, respectively; switching to the AppAgentX-style check achieved 81.8%, still lower than the Full version, with higher call counts. The reason is that the AppAgentX-style control can only directly replay or abandon reuse, whereas Route B can also replace old parameters in the selected action that clearly correspond to parameters of the current task before continuing execution. This reduces cases where usable actions are abandoned due to parameter changes and handed over to the model for re-decision, while also improving task execution speed.
Response to Reviewer nZd3 (Part 3/3): Repeated-Run Variance, Blinded Human Spot-Check, and Full Disagreement Audit (Q3)
Q3 How Large Is the Variance Across Repeated Runs, and Can Task Success Judgments Pass Blinded Human Review?
Response: Thank you for pointing out the potential variance introduced by single runs and baseline grouping construction. To rigorously evaluate statistical significance and result stability, we independently ran both methods 10 times per task on the 15 Same-family tasks and 18 No-family-record tasks of SPA-Bench. The memory bank and task groupings were kept fixed, resetting the environment each time; task-level resampling (task-level Basic Bootstrap, 10,000 resamples, with interval bounds clipped to [0.0%, 100.0%]) was performed separately for each method group to compute the 95% confidence intervals (95% CI) of their respective success rates:
| Model | Method | Same-family (Single) | 10-Run Mean | 95% CI | No-family-record (Single) | 10-Run Mean | 95% CI |
|---|---|---|---|---|---|---|---|
| Qwen3-VL-30B | Base agent | 0.0% | 1.3% | [0.0%, 2.7%] | 0.0% | 1.1% | [0.0%, 2.2%] |
| Qwen3-VL-30B | DeltaReplay | 33.3% | 32.0% | [27.3%, 36.7%] | 11.1% | 12.2% | [9.4%, 15.0%] |
| Qwen3.8-Flash | Base agent | 0.0% | 1.3% | [0.0%, 2.7%] | 5.6% | 6.7% | [3.9%, 9.4%] |
| Qwen3.8-Flash | DeltaReplay | 46.7% | 48.0% | [43.3%, 52.7%] | 16.7% | 15.6% | [12.8%, 18.3%] |
After independent repetitions, the baseline scores showed a slight rebound, yet DeltaReplay still maintained higher average success rates across both groups and both backbones, with completely non-overlapping 95% confidence intervals.
Regarding automatic annotation, we previously compared Qwen, DeepSeek, GPT, and Fable 5.1 in small-scale experiments and found that Fable 5.1 performed best, while Qwen3.8-Flash achieved the fastest annotation speed while maintaining the second-highest accuracy; therefore, the paper adopted the annotation results of these two models.
To verify the reliability of automatic annotation, we supplemented a blinded human spot-check experiment: a total of 30 complete task records were sampled, covering both backbones as well as Base agent and DeltaReplay; additionally, 10 steps were sampled for each route from the original step samples, totaling 30 steps. Annotators viewed the task requirements, action records, and before-and-after states, with the method, backbone, route, and original labels hidden before making their decisions. Task success labels were compared against the original judgments of Qwen3.8-Flash, and step labels were compared against Fable 5.1.
| Evaluated Backbone | Sampled Tasks | Original Auto-Judged Successes | Human-Judged Successes | Agreed / Sampled | Agreement Rate |
|---|---|---|---|---|---|
| Qwen3.8-Flash | 15 | 8 | 8 | 15/15 | 100.0% |
| Qwen3-VL-30B | 15 | 8 | 8 | 15/15 | 100.0% |
| Total | 30 | 16 | 16 | 30/30 | 100.0% |
| Route | Sampled Steps | LLM-Judged Correct | Human-Judged Correct | Label Agreement Rate |
|---|---|---|---|---|
| Route A | 10 | 8 | 8 | 100.0% |
| Route B | 10 | 9 | 10 | 90.0% |
| Route C | 10 | 6 | 6 | 100.0% |
| Total | 30 | 23 | 24 | 96.7% |
Summary: The blinded human agreement rate for task success judgment was 100.0% (30/30), and the step label agreement rate was 96.7% (29/30). Among the 30 sampled steps, a discrepancy occurred in only 1 step (Route B), which was an action replacing historical input text with current task parameters on a form page (human judgment determined that the parameter replacement direction was correct, whereas automatic annotation judged it invalid because the page did not undergo an immediate transition). Overall, automatic annotation is highly consistent with human judgment. Furthermore, regarding the 8.5% (23/271, Appendix D) samples where the AndroidWorld rule-based checker and the LLM judge disagreed, we also conducted a full human review: 19 of them were misjudgments by the rule-based checker due to the external evaluation framework prematurely closing the foreground app or file-naming detail restrictions (the actual tasks were properly completed), and only 4 were misjudgments by the LLM judge on the final state (achieving an overall judgment accuracy of 98.5%).
Response to Reviewer nZd3 (Part 1/3): Trajectory Exclusion, Instruction-Only Retrieval, and Paraphrase Adaptation (Q1)
Q1 Does the Method Remain Effective After Excluding Same-Task Trajectories and Retrieving Solely by Instruction?
Response: Thank you for your suggestion. The original experiments on SPA-Bench retrieved memories by task identifiers. The 15 tasks in Same-family inherently had no trajectories of the exact task itself and could only use records from simpler tasks in the same family; in this group, DeltaReplay achieved success rates of 33.3% and 46.7% under the two backbones, respectively, compared to 0.0% for the corresponding Base agents. These results demonstrate the utility of related task records.
However, we understand your concern: whether the method's improvement relies on same-task trajectories and the retrieval cues provided by task IDs. To this end, we supplemented the following controls: on the original 60 SPA-Bench tasks using Qwen3-VL-30B, we separately tested: excluding only same-task trajectories (
w/o exact-task trajectories), retrieving solely by instruction (Instruction-only retrieval, w/o task IDs), and applying both restrictions simultaneously. Excluding same-task trajectories means that for each test task, all historical execution records of that task are removed from the retrievable memory while retaining records of other tasks; retrieving solely by instruction means searching for relevant records solely based on the natural language description of the current task, without using benchmark task IDs or task-family information derived from IDs for filtering, ranking, or type hints. Task instances, memory size and context budgets, execution budgets, and evaluation pipelines remained consistent, reporting results under the original 27/15/18 task split.When simultaneously excluding same-task trajectories and task identifiers, the success rate on the Same-family group remained 26.7%, still a substantial improvement over the Base agent's 0.0% performance in the paper (retaining 80.0%, or 4/5, of the relative gain). This shows that while same-task trajectories and task identifiers indeed provide assistance, the primary success rate improvement still stems from our method itself.
Furthermore, regarding parameter expressions in real-world instructions, we agree that the current token-level alignment-based parameter adaptation is sensitive to expression forms, especially in paraphrased instructions and instructions with implicit values that require contextual resolution. The current rule-based token-level alignment is designed to isolate and verify the effectiveness of the routing framework itself; in real-world applications, large language models can be directly used to replace the current parameter replacement rules.
To test whether obfuscating parameter expressions would prevent Route B from proceeding during the parameter identification stage, we conducted a small-scale offline module control experiment. We selected 10 types of tasks from real task instances in AndroidWorld, constructing three versions for each—local paraphrasing, full-sentence restructuring, and complex long instructions—totaling 30 rewrites, and used GPT-6 Astra (medium) for fuzzy parameter replacement testing. The overall accuracy was 96.7% (29/30). The test was conducted in batches of 10, with details as follows:
The only failed case involved implicit value reasoning requiring external geographic knowledge:
Add a favorite location marker for 47.2165476, 9.5699984 in the OsmAnd maps app.In OsmAnd, mark Balzers in Liechtenstein as a favorite location.{"id":"M09","replacements":[{"id":"p1","text":null}]}(The model did not return the corresponding replacement value).To keep the framework lightweight and free of additional training, we adopted deterministic rule-based parsing in the main experiments; the above experiments demonstrate that using an LLM for parameter parsing can effectively supplement this simplified component.