acceptodds
Under review as a conference paper at ICLR 2027

DeltaReplay: Task-Relative Memory Reuse for Mobile GUI Agents

Abstract

Memory-augmented mobile GUI agents store successful execution trajectories and reuse them in later tasks, but a stored trajectory rarely matches a new task exactly. The new task may use different parameters, share only some of its steps with a stored trajectory, or have no relevant record in memory. Forcing the agent to use irrelevant memory can mislead it, whereas discarding memory that may still be useful deprives it of guidance from past experience. To address this dilemma, we propose DeltaReplay, a step-level memory reuse framework that decides how to use existing memory without modifying it. We observe that the reusable part of a stored record is determined not by the record itself but by its relation to the new task, mainly through two factors: page-level consistency and action-level generality. We therefore store execution trajectories as paths in a transition graph, whose nodes (pages) and edges (actions between pages) capture these two factors. At reuse time, the action on each edge is split into a task-independent operation and task-specific parameters. DeltaReplay then compares each recorded step with the new task and the current screen, and decides whether to follow it, execute it after replacing its parameters, or leave it to the base agent. On AndroidWorld and SPA-Bench, DeltaReplay improves the task success rate over a base agent with the same backbone by up to 10.3 and 25.0 percentage points, respectively. These results indicate that deciding at each step how to use retrieved memory lets agents benefit even from partially matching trajectories.

Primary area:
applications to robotics, autonomy, planning
Keywords:
GUI agents, mobile agents, agent memory, memory reuse, trajectory replay, step-level decision making

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nZd3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DeltaReplay, a training-free, step-level memory reuse framework for mobile GUI agents. It represents previously successful interaction trajectories as paths in a read-only UI transition graph and distinguishes task-independent operations from task-specific parameters at reuse time. Given a new instruction and the current screen, it chooses among verified replay (Route A), parameter adaptation/re-grounding (Route B), and discarding the retrieved step in favor of the base agent (Route C). Structural screen/element matching and an LLM-based step selector implement the routing. On AndroidWorld-116 and SPA-Bench-60, it improves task success over same-backbone agents by 7.8–10.3 and 16.6–25.0 percentage points, respectively; it also reduces latency in the high-memory-coverage settings with the Qwen3.8-Flash backbone. Route statistics and ablations provide supporting evidence for the importance of checking applicability before reusing a step.

Critical strengths
  1. Principled use of partial memories. The distinction between direct replay, parameter adaptation, and fallback addresses a real limitation of trajectory-level reuse. The method checks both whether a recorded action can be mapped to the current screen and whether it still advances the current instruction, rather than forcing an all-or-nothing decision. The read-only design and explicit abstention are especially useful for reducing harmful negative transfer.
  2. Promising empirical gains with diagnostic analyses. The study evaluates two distinct backbones across AndroidWorld-116 and SPA-Bench-60 and separates cases with identical, parameter-shifted, partially related, and unavailable memories. The same-template AndroidWorld gain with Qwen3.8-Flash (63.6% to 86.4%) and the SPA-Bench same-family gains (from 0% to 33.3%/46.7%) are particularly informative. Per-step route statistics and comparisons against naive replay support the premise that selective reuse is better than unconditional copying.
Critical weaknesses
  1. Generalizability of the evaluation setting remains uncertain. Retrieval uses benchmark-provided task/template/family identifiers rather than only the natural-language instruction. Moreover, SPA-Bench memory contains successful base-agent runs for 27 of the 60 evaluated tasks, including some runs appearing in the reported base-agent results. This is a legitimate exact-task memory setting, not necessarily leakage, but it makes the aggregate score less informative about transfer to previously unseen goals. Successful-run-only memory construction also correlates coverage with task difficulty, as the authors acknowledge. An independent evaluation emphasizing new goals, paraphrased instructions, and nonmatching app states would better substantiate the method’s practical breadth.
  2. The evidence for individual design choices and outcome reliability is incomplete. The ablations do not isolate individual mechanisms: “w/o Route C” also removes the screen check, while “w/o semantic matching” removes both the screen check and the selector. Consequently, Table 4 cannot by itself attribute the declines to discarding or semantic selection independently, and a dedicated Route B ablation is missing. Each task is evaluated only once; the reported success metric uses an LLM judge with no human validation, and agreement with available AndroidWorld checkers is 91.5%. These limitations matter when interpreting modest aggregate differences and claiming reliable superiority. Comparisons to closely related executable-memory methods are also mostly conceptual rather than experimental.
Questions for authors
  1. Could the authors report results when SPA-Bench exact-task trajectories are excluded from memory, and when retrieval is performed from the instruction alone without benchmark task IDs? In particular, how much of the improvement survives on unseen goals with only related records?
  2. Can Route A, Route B, screen verification, semantic selection, and Route C fallback be removed one at a time while holding other parts fixed? How does the method compare under a matched interface and memory budget to an executable-memory baseline (e.g., AppAgentX or a re-ground-or-discard approach)?
  3. Could the authors provide repeated-run variability or paired confidence intervals and a small blind human audit of task-success judgments, especially for episodes where the LLM judge disagrees with an AndroidWorld checker?
3 comments
AuthorsOpenReview ↗

Response to Reviewer nZd3 (Part 1/3): Trajectory Exclusion, Instruction-Only Retrieval, and Paraphrase Adaptation (Q1)

Q1 Does the Method Remain Effective After Excluding Same-Task Trajectories and Retrieving Solely by Instruction?

Response: Thank you for your suggestion. The original experiments on SPA-Bench retrieved memories by task identifiers. The 15 tasks in Same-family inherently had no trajectories of the exact task itself and could only use records from simpler tasks in the same family; in this group, DeltaReplay achieved success rates of 33.3% and 46.7% under the two backbones, respectively, compared to 0.0% for the corresponding Base agents. These results demonstrate the utility of related task records.

However, we understand your concern: whether the method's improvement relies on same-task trajectories and the retrieval cues provided by task IDs. To this end, we supplemented the following controls: on the original 60 SPA-Bench tasks using Qwen3-VL-30B, we separately tested: excluding only same-task trajectories (w/o exact-task trajectories), retrieving solely by instruction (Instruction-only retrieval, w/o task IDs), and applying both restrictions simultaneously. Excluding same-task trajectories means that for each test task, all historical execution records of that task are removed from the retrievable memory while retaining records of other tasks; retrieving solely by instruction means searching for relevant records solely based on the natural language description of the current task, without using benchmark task IDs or task-family information derived from IDs for filtering, ranking, or type hints. Task instances, memory size and context budgets, execution budgets, and evaluation pipelines remained consistent, reporting results under the original 27/15/18 task split.

Method and Setting (Qwen3-VL-30B)Original Same-task (27)Same-family (15)No-family-record (18)
Base agent51.9%0.0%0.0%
DeltaReplay81.5%33.3%11.1%
w/o exact-task trajectories (New)63.0%33.3%11.1%
Instruction-only retrieval, w/o task IDs (New)77.8%26.7%11.1%
w/o exact-task trajectories + Instruction-only retrieval (New)59.3%26.7%11.1%

When simultaneously excluding same-task trajectories and task identifiers, the success rate on the Same-family group remained 26.7%, still a substantial improvement over the Base agent's 0.0% performance in the paper (retaining 80.0%, or 4/5, of the relative gain). This shows that while same-task trajectories and task identifiers indeed provide assistance, the primary success rate improvement still stems from our method itself.

Furthermore, regarding parameter expressions in real-world instructions, we agree that the current token-level alignment-based parameter adaptation is sensitive to expression forms, especially in paraphrased instructions and instructions with implicit values that require contextual resolution. The current rule-based token-level alignment is designed to isolate and verify the effectiveness of the routing framework itself; in real-world applications, large language models can be directly used to replace the current parameter replacement rules.

To test whether obfuscating parameter expressions would prevent Route B from proceeding during the parameter identification stage, we conducted a small-scale offline module control experiment. We selected 10 types of tasks from real task instances in AndroidWorld, constructing three versions for each—local paraphrasing, full-sentence restructuring, and complex long instructions—totaling 30 rewrites, and used GPT-6 Astra (medium) for fuzzy parameter replacement testing. The overall accuracy was 96.7% (29/30). The test was conducted in batches of 10, with details as follows:

Instruction FormCorrect CountAccuracy
Local Paraphrasing10/10100%
Full-Sentence Restructuring9/1090%
Complex Long Instructions10/10100%
Total29/3096.7%

The only failed case involved implicit value reasoning requiring external geographic knowledge:

  • history: Add a favorite location marker for 47.2165476, 9.5699984 in the OsmAnd maps app.
  • now: In OsmAnd, mark Balzers in Liechtenstein as a favorite location.
  • Q: According to the new task, what should the input text in the historical action be replaced with?
  • A: {"id":"M09","replacements":[{"id":"p1","text":null}]} (The model did not return the corresponding replacement value).

To keep the framework lightweight and free of additional training, we adopted deterministic rule-based parsing in the main experiments; the above experiments demonstrate that using an LLM for parameter parsing can effectively supplement this simplified component.

AuthorsOpenReview ↗

Response to Reviewer nZd3 (Part 2/3): Independent Component Ablations and Executable Memory Comparisons (Q2)

Q2 Can Each Component Be Independently Ablated and Compared with Executable Memory Methods Under Identical Interfaces and Memory Budgets?

Response: Thank you for your suggestion. Regarding the ablation experiments, we clarify and supplement independent controls as follows:

  1. In the original Table 4, w/o Route C removed both Route C fallback and the screen check (45.5%); w/o semantic matching removed both the screen check and the Selector (20.0%). To strictly distinguish the independent contribution of each component, we supplemented independent ablation controls that disable single components separately.
  2. For the comparison between Route B and executable memory methods under identical interfaces and memory budgets, we supplemented two controls:
  • w/o Route B: retains the original screen check and Selector, disables parameter adaptation, continues to replay old actions that can be used directly, and hands over actions requiring parameter modification to Route C;
  • AppAgentX-style check, w/o Route B: also disables Route B, but replaces the screen check and Selector with a single LLM applicability judgment, selecting directly executable actions based on the current task, page, and the same batch of retrieved records, or rejecting all candidates and handing over to Route C.
  1. For Route A, we supplemented a control removing only direct replay (w/o Route A): retaining memory retrieval, screen check, Selector, Route B parameter adaptation, and Route C, and handing over only actions originally executed directly by Route A to Route C.

To this end, we supplemented independent ablations of the screen check and Selector on the original 30 AndroidWorld Same-app tasks using Qwen3.8-Flash; on the original 22 Same-template tasks, we supplemented independent controls for w/o Route A, w/o Route B, AppAgentX-style check, and w/o Route C, keeping task instances, memory banks, execution budgets, and evaluation pipelines fixed. All controls used identical observation, target grounding, and action execution interfaces, as well as the same batch of retrieved records, identical record counts, and input length limits (existing results from the paper are retained in the table below):

Task GroupExperimental GroupScreen CheckSelectorTasksSuccessesSuccess Rate (%)Model Calls/StepA/B/C (%)
Same-appFullOnOn301550.02.190/14/86
Same-appw/o selector (New)OnOff30826.71.250/40/60
Same-appw/o screen check (New)OffOn30930.02.400/22/78
Same-appJoint Removal (Original Table 4 w/o semantic matching)OffOff30620.00.600/73/27
Same-templateFullOnOn221986.41.0041/15/44
Same-templatew/o Route A (New)OnOn221672.71.850/18/82
Same-templatew/o Route B (New)OnOn221777.31.6041/0/59
Same-templateAppAgentX-style check, w/o Route B (New)Replaced with unified LLM judgmentReplaced with unified LLM judgment221881.81.9045/0/55
Same-templateOriginal w/o Route C (Original Table 4 Joint Ablation)OffOn221045.50.1768/32/0
Same-templatew/o Route C (New Independent Control)OnOn221150.00.1867/33/0

Summary: Removing the Selector or the screen check individually caused the success rate to drop from 50.0% to 26.7% and 30.0%, respectively, and removing both jointly caused it to drop further to 20.0%, indicating that both checks contribute and their joint absence causes a greater loss. After removing the screen check, the number of calls per step increased from 2.19 to 2.40 because the lack of low-overhead structural filtering caused more mismatched pages to directly trigger LLM Selector calls. Removing Route A, Route B, and Route C individually caused the Same-template success rate to drop to 72.7%, 77.3%, and 50.0%, respectively; switching to the AppAgentX-style check achieved 81.8%, still lower than the Full version, with higher call counts. The reason is that the AppAgentX-style control can only directly replay or abandon reuse, whereas Route B can also replace old parameters in the selected action that clearly correspond to parameters of the current task before continuing execution. This reduces cases where usable actions are abandoned due to parameter changes and handed over to the model for re-decision, while also improving task execution speed.

AuthorsOpenReview ↗

Response to Reviewer nZd3 (Part 3/3): Repeated-Run Variance, Blinded Human Spot-Check, and Full Disagreement Audit (Q3)

Q3 How Large Is the Variance Across Repeated Runs, and Can Task Success Judgments Pass Blinded Human Review?

Response: Thank you for pointing out the potential variance introduced by single runs and baseline grouping construction. To rigorously evaluate statistical significance and result stability, we independently ran both methods 10 times per task on the 15 Same-family tasks and 18 No-family-record tasks of SPA-Bench. The memory bank and task groupings were kept fixed, resetting the environment each time; task-level resampling (task-level Basic Bootstrap, 10,000 resamples, with interval bounds clipped to [0.0%, 100.0%]) was performed separately for each method group to compute the 95% confidence intervals (95% CI) of their respective success rates:

ModelMethodSame-family (Single)10-Run Mean95% CINo-family-record (Single)10-Run Mean95% CI
Qwen3-VL-30BBase agent0.0%1.3%[0.0%, 2.7%]0.0%1.1%[0.0%, 2.2%]
Qwen3-VL-30BDeltaReplay33.3%32.0%[27.3%, 36.7%]11.1%12.2%[9.4%, 15.0%]
Qwen3.8-FlashBase agent0.0%1.3%[0.0%, 2.7%]5.6%6.7%[3.9%, 9.4%]
Qwen3.8-FlashDeltaReplay46.7%48.0%[43.3%, 52.7%]16.7%15.6%[12.8%, 18.3%]

After independent repetitions, the baseline scores showed a slight rebound, yet DeltaReplay still maintained higher average success rates across both groups and both backbones, with completely non-overlapping 95% confidence intervals.

Regarding automatic annotation, we previously compared Qwen, DeepSeek, GPT, and Fable 5.1 in small-scale experiments and found that Fable 5.1 performed best, while Qwen3.8-Flash achieved the fastest annotation speed while maintaining the second-highest accuracy; therefore, the paper adopted the annotation results of these two models.

To verify the reliability of automatic annotation, we supplemented a blinded human spot-check experiment: a total of 30 complete task records were sampled, covering both backbones as well as Base agent and DeltaReplay; additionally, 10 steps were sampled for each route from the original step samples, totaling 30 steps. Annotators viewed the task requirements, action records, and before-and-after states, with the method, backbone, route, and original labels hidden before making their decisions. Task success labels were compared against the original judgments of Qwen3.8-Flash, and step labels were compared against Fable 5.1.

Evaluated BackboneSampled TasksOriginal Auto-Judged SuccessesHuman-Judged SuccessesAgreed / SampledAgreement Rate
Qwen3.8-Flash158815/15100.0%
Qwen3-VL-30B158815/15100.0%
Total30161630/30100.0%
RouteSampled StepsLLM-Judged CorrectHuman-Judged CorrectLabel Agreement Rate
Route A1088100.0%
Route B1091090.0%
Route C1066100.0%
Total30232496.7%

Summary: The blinded human agreement rate for task success judgment was 100.0% (30/30), and the step label agreement rate was 96.7% (29/30). Among the 30 sampled steps, a discrepancy occurred in only 1 step (Route B), which was an action replacing historical input text with current task parameters on a form page (human judgment determined that the parameter replacement direction was correct, whereas automatic annotation judged it invalid because the page did not undergo an immediate transition). Overall, automatic annotation is highly consistent with human judgment. Furthermore, regarding the 8.5% (23/271, Appendix D) samples where the AndroidWorld rule-based checker and the LLM judge disagreed, we also conducted a full human review: 19 of them were misjudgments by the rule-based checker due to the external evaluation framework prematurely closing the foreground app or file-naming detail restrictions (the actual tasks were properly completed), and only 4 were misjudgments by the LLM judge on the final state (achieving an overall judgment accuracy of 98.5%).

Reviewer_vEZ1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DeltaReplay is a training-free method for reusing recorded successful runs in mobile GUI agents without modifying them. Memory is a read-only transition graph, and at each step the method chooses among replaying the recorded step (Route A), re-grounding the parameters of the old task (Route B) and discarding memory so that the base agent acts (Route C), using structural screen matching, an LLM selector that can reject all candidates, and rule-based parameter replacement. On AndroidWorld-116 and SPA-Bench-60 with two backbones, success rises over the base agent by 7.8–10.3 and 16.6–25.0 points, reported by memory-coverage group.

Critical strengths
  1. Decomposition of memory reuse. Executability and applicability give three regimes (Eq. 1) in which a retrieved step should be replayed, re-grounded or ignored, and Table 1 uses them to argue where prompt-only, execute-or-discard and re-ground-or-discard methods should lose. This gives a shared vocabulary for where a memory method fails, which is useful beyond this method.
  2. Evaluation by memory coverage. Results are reported by the closest record in memory (Table 3) rather than only in aggregate, and the paper states that the groups correlate with task difficulty, that each task is run once, and that success and step labels come from LLMs without human checks, which lets a reader see where the gains arise and how much weight to give them. Table 4 shows that removing Route C together with the screen check lowers success below the base agent (45.5% vs. 63.6% on same-template tasks), so checking and discarding are jointly necessary.
Critical weaknesses
  1. The comparison with the memory baselines may not isolate step-level routing. DeltaReplay retrieves records by the benchmark's task identifiers, so the kind of record an episode receives follows its coverage group, and the selector prompt names that kind (Sec. 5.1); RAG-prompting and GUI-explorer add the most similar records, and the main text does not indicate that they are told how these relate to the current task. Their drops when memory does not apply (25.0% and 11.5% for RAG-prompting against 39.6% and 30.8% for the base agent, with GUI-explorer also below it; Table 3) may therefore reflect this difference rather than routing. The comparison that would test the claimed advantage most directly, against the execute-or-discard and re-ground-or-discard methods that Table 1 positions DeltaReplay against (AppAgentX, Darwinian Memory, UI-KOBE), is not run either. This affects the claim about inapplicable memory (failure iii) and is the main reason for my rating; Sec. 6 acknowledges identifier-based retrieval as a limitation, and the missing control should be easy to run with the components the paper already has.
  2. The headline gains rest on single runs in small groups, and the SPA-Bench comparison groups are defined by earlier outcomes of the base agent. Each task is run once per method, and the paper counts a difference of about one task per group as run-to-run variation (Sec. 5.1); with Qwen3.8-Flash on AndroidWorld, the same-app and none groups differ from the base agent by one task each. On SPA-Bench, memory includes the base-agent runs reported in Tables 2 and 3, and 32 of the 33 tasks in the same-family and no-family-record groups were failed by both backbones in those runs (Sec. 5.1), so base-agent success there (0.0–5.6%) is low by construction and is compared with a fresh DeltaReplay run; 7 of the 10 tasks gained with Qwen3.8-Flash are in the same-family group, so most of the SPA-Bench gain comes from the group whose baseline is lowest by construction. This is about how large the headline gains are, not about whether routing helps at all.
Questions for authors
  1. Can RAG-prompting and GUI-explorer be given the same retrieved records and the record kind that the selector prompt names, and how do their results by coverage group change?
  2. For the SPA-Bench same-family and no-family-record groups, can the base agent be re-run on these tasks rather than taken from the runs used to build memory, and can the runs be repeated, with confidence intervals reported?
  3. Table 4 has no variant without Route B, and w/o Route C and w/o semantic matching each remove the screen check as well (the latter also the selector). Can Route B, the screen check, the selector and Route C be removed one at a time, so that over-discarding (failure ii) is tested against a replay-or-discard policy, for example one using an AppAgentX-style applicability check?
  4. Qwen3.8-Flash judges task success and is also one of the evaluated backbones, and Sec. 6 states that neither the success labels nor the step labels were checked by humans. Can a sample of both be checked by human annotators?
3 comments
AuthorsOpenReview ↗

Response to Reviewer vEZ1 (Part 1/3): Retrieval Input Alignment (Q1) and Repeated-Run Stability with Bootstrap CIs (Q2)

Thank you for your summary of the paper and detailed comments. We have refined the control settings, component attribution, and statistical analysis, and supplemented the necessary experiments, addressing your questions point by point.

1. Do Differences in Retrieved Information Explain the Baseline Gap?

Response: Thank you for your suggestion. While the original experiments shared the memory bank, the retrieved content and type hints across different methods were not yet fully aligned. We agree that these input differences should be further controlled.

On the 116 tasks of AndroidWorld using Qwen3.8-Flash, we provided RAG-prompting and GUI-explorer with the exact same retrieved records and type descriptions that DeltaReplay retrieved for each task, keeping task instances, execution budgets, and evaluation pipelines consistent.

Method and SettingSame task (20)Same template (22)Same app (48)None (26)
RAG-prompting80.0%77.3%25.0%11.5%
RAG-prompting + same records & type hints (New)85.0%81.8%27.1%11.5%
GUI-explorer80.0%86.4%29.2%23.1%
GUI-explorer + same records & type hints (New)80.0%86.4%33.3%23.1%
DeltaReplay90.0%86.4%41.7%34.6%

After input alignment, the improvements of the two baselines were mainly concentrated in the Same template or Same app groups. This phenomenon is understandable, as receiving information from tasks strongly correlated with the current task helps the agent better understand the nature of the task; however, across all groups—especially in the weakly correlated Same app (41.7% vs. 27.1%/33.3%) and None (34.6% vs. 11.5%/23.1%) groups—DeltaReplay still maintains a significant advantage. We will incorporate the latest empirical results and group-wise analyses into the paper.

2. How Stable Are the Performance Gains Under Independent Reruns and Repeated Runs?

Response: Thank you for pointing out the potential variance introduced by single runs and baseline grouping construction. To rigorously evaluate statistical significance and result stability, we independently reran both methods 10 times per task on the 15 Same-family tasks and 18 No-family-record tasks of SPA-Bench. Task-level resampling (task-level Basic Bootstrap, 10,000 resamples, with interval bounds clipped to [0.0%, 100.0%]) was performed separately for each method group to compute the 95% confidence intervals (95% CI) of their respective success rates:

ModelMethodSame-family (Single)10-Run Mean95% CINo-family-record (Single)10-Run Mean95% CI
Qwen3-VL-30BBase agent0.0%1.3%[0.0%, 2.7%]0.0%1.1%[0.0%, 2.2%]
Qwen3-VL-30BDeltaReplay33.3%32.0%[27.3%, 36.7%]11.1%12.2%[9.4%, 15.0%]
Qwen3.8-FlashBase agent0.0%1.3%[0.0%, 2.7%]5.6%6.7%[3.9%, 9.4%]
Qwen3.8-FlashDeltaReplay46.7%48.0%[43.3%, 52.7%]16.7%15.6%[12.8%, 18.3%]

After independent repetitions, the baseline scores showed a slight rebound, yet DeltaReplay still maintained higher success rates across both groups and both backbones, with completely non-overlapping 95% confidence intervals.

AuthorsOpenReview ↗

Response to Reviewer vEZ1 (Part 2/3): Independent Component Ablations and Route B vs. AppAgentX-Style Check (Q3)

3. Were Component Contributions and Over-Discarding Independently Verified?

Response: Thank you for your suggestion. Regarding the ablation experiments, we clarify and supplement independent controls as follows:

  1. In the original Table 4, w/o Route C removed both Route C fallback and the screen check (45.5%); w/o semantic matching removed both the screen check and the Selector (20.0%). To strictly distinguish the independent contribution of each component, we supplemented independent ablation controls removing only a single component.
  2. For the comparison between Route B and AppAgentX-style methods, we supplemented two controls:
  • w/o Route B: retains the original screen check and Selector, disables parameter adaptation, continues to replay old actions that can be used directly, and hands over actions requiring parameter modification to Route C;
  • AppAgentX-style check, w/o Route B: also disables Route B, but replaces the screen check and Selector with a single LLM applicability judgment, selecting directly executable actions based on the current task, page, and the same batch of retrieved records, or rejecting all candidates and handing over to Route C.

To this end, we supplemented independent ablations of the screen check and Selector on the original 30 AndroidWorld Same-app tasks using Qwen3.8-Flash; on the original 22 Same-template tasks, we supplemented independent controls for w/o Route B, AppAgentX-style check, and w/o Route C, keeping task instances, memory banks, execution budgets, and evaluation pipelines fixed (existing results from the paper are retained in the table below):

Task GroupExperimental GroupScreen CheckSelectorTasksSuccessesSuccess Rate (%)Model Calls/StepA/B/C (%)
Same-appFullOnOn301550.02.190/14/86
Same-appw/o selector (New)OnOff30826.71.250/40/60
Same-appw/o screen check (New)OffOn30930.02.400/22/78
Same-appJoint Removal (Original Table 4 w/o semantic matching)OffOff30620.00.600/73/27
Same-templateFullOnOn221986.41.0041/15/44
Same-templatew/o Route B (New)OnOn221777.31.6041/0/59
Same-templateAppAgentX-style check, w/o Route B (New)Replaced with unified LLM judgmentReplaced with unified LLM judgment221881.81.9045/0/55
Same-templateOriginal w/o Route C (Original Table 4 Joint Ablation)OffOn221045.50.1768/32/0
Same-templatew/o Route C (New Independent Control)OnOn221150.00.1867/33/0

Summary: Removing the Selector or the screen check individually caused the success rate to drop from 50.0% to 26.7% and 30.0%, respectively, and removing both jointly caused it to drop further to 20.0%, indicating that both checks contribute and their joint absence causes a greater loss. After removing the screen check, the number of calls per step increased from 2.19 to 2.40 because the lack of low-overhead structural filtering caused more mismatched pages to directly trigger LLM Selector calls. After removing only Route B, the success rate dropped from 86.4% to 77.3%; moreover, the number of model calls per step increased from 1.00 to 1.60—this is because unreplaced parameters caused subsequent page states to deviate from the original trajectory, forcing subsequent steps that could originally be directly replayed to frequently trigger the Selector and Base Agent for re-decision; switching to the AppAgentX-style check achieved 81.8%, still lower than the Full version, with higher call counts and latency. The reason is that the AppAgentX-style control can only directly replay or abandon reuse, whereas Route B can also replace old parameters in the selected action that clearly correspond to parameters of the current task before continuing execution. This reduces cases where usable actions are abandoned due to parameter changes and handed over to the model for re-decision, while also improving task execution speed. (If you are interested in the accuracy and coverage of Route B parameter matching, please see our discussion with Reviewer LssG regarding Q1, which lists detailed experimental results.)

AuthorsOpenReview ↗

Response to Reviewer vEZ1 (Part 3/3): Reliability of Automatic Annotation and Blinded Human Spot-Check (Q4)

4. Are Task Success and Step Labels Reliable?

Response: Thank you for your suggestion. We previously compared Qwen, DeepSeek, GPT, and Fable 5.1 in small-scale experiments and found that Fable 5.1 performed best, while Qwen3.8-Flash achieved the fastest annotation speed while maintaining the second-highest accuracy; therefore, the paper adopted the annotation results of these two models.

To verify the reliability of automatic annotation, we supplemented a blinded human spot-check experiment: a total of 30 complete task records were sampled, covering both backbones as well as Base agent and DeltaReplay; additionally, 10 steps were sampled for each route from the original step samples, totaling 30 steps. Annotators viewed the task requirements, action records, and before-and-after states, with the method, backbone, route, and original labels hidden before making their decisions. Task success labels were compared against the original judgments of Qwen3.8-Flash, and step labels were compared against Fable 5.1.

Evaluated BackboneSampled TasksOriginal Auto-Judged SuccessesHuman-Judged SuccessesAgreed / SampledAgreement Rate
Qwen3.8-Flash158815/15100.0%
Qwen3-VL-30B158815/15100.0%
Total30161630/30100.0%
RouteSampled StepsLLM-Judged CorrectHuman-Judged CorrectLabel Agreement Rate
Route A1088100.0%
Route B1091090.0%
Route C1066100.0%
Total30232496.7%

Summary: The blinded human agreement rate for task success judgment was 100.0% (30/30), and the step label agreement rate was 96.7% (29/30). Among the 30 sampled steps, a discrepancy occurred in only 1 step (Route B), which was an action replacing historical input text with current task parameters on a form page (human judgment determined that the parameter replacement direction was correct, whereas automatic annotation judged it invalid because the page did not undergo an immediate transition). Overall, automatic annotation is highly consistent with human judgment.

Reviewer_LssGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DeltaReplay reuses successful GUI trajectories that only partly match a new task. At each step, it chooses verified replay, parameter adaptation, or base-agent fallback. Screen matching and a semantic selector guide this choice. Memory remains fixed during evaluation. Experiments cover AndroidWorld and SPA-Bench with two backbones. Success rates improve on both benchmarks. Decision latency also falls when relevant memory supports replay.

Critical strengths
  • The three reuse options are clear and well motivated. The controller checks whether a recorded step still helps and can run as recorded. It can reuse navigation, change parameters, or fall back to the base agent. No additional training is needed.

  • The results show useful partial transfer. With Qwen3.8-Flash, AndroidWorld same-template success rises from 63.6% to 86.4%. On SPA-Bench, the two backbones solve 5/15 and 7/15 same-family tasks using easier-task records. The original base-agent runs solved none.

  • Verified replay can reduce computation. On AndroidWorld same-task episodes, Qwen3.8-Flash latency falls from 21.7 to 9.3 seconds per step. Same-template model calls decrease from 2.11 to 1.00 per step. The coverage breakdown shows when these savings are available.

Critical weaknesses
  • The retrieval setup uses benchmark metadata. Task, template, and family identifiers guide record selection. The shared-memory baselines provide a useful control. However, performance with imperfect instruction-based retrieval remains untested.

  • The scope of parameter adaptation is still limited. Route B aligns instruction words and replaces explicit parameter values. This suits explicit template changes. Coverage for paraphrases and implicit values is less clear. High precision on accepted steps does not measure missed adaptations.

  • The ablations do not isolate individual components. Removing fallback also removes screen checking. The semantic-matching ablation removes screen checking too. Table 4 shows that unchecked reuse is harmful. The separate benefits of matching, fallback, and parameter adaptation remain less clear.

Questions for authors
  • Section 5.4 reports high precision for accepted parameter adaptations. How often does Route B handle steps that require changed parameters on same-template tasks? What are the main reasons for fallback? A brief summary from existing execution records would help clarify adaptation coverage.

  • Section 5.5 removes screen checking together with fallback or semantic matching. How should Table 4’s results be interpreted given these combined changes? Please clarify whether the existing experiments separate parameter adaptation from verified replay plus fallback. This would help identify which component-level claims are supported.

4 comments
AuthorsOpenReview ↗

Response to Reviewer LssG (Part 1/3): Paraphrases, Coreference, Implicit Values, and Retrieval Generalization (Q1.1)

Thank you for your summary of the paper and detailed comments. We have further clarified the explanations regarding parameter adaptation, ablation attribution, and retrieval generalization, and supplemented the necessary experiments.

Q1.1 Paraphrases, Coreference, and Implicit Values

Response: We completely agree that the current token-level alignment-based parameter adaptation is sensitive to expression forms, especially in paraphrased instructions and instructions with implicit values that require contextual resolution. The current rule-based token-level alignment is designed to isolate and verify the effectiveness of the routing framework itself; in real-world applications, large language models can be directly used to replace the current parameter replacement rules.

To test whether obfuscating parameter expressions would prevent Route B from proceeding during the parameter identification stage, we conducted a small-scale offline module control experiment. We selected 10 types of tasks from real task instances in AndroidWorld, constructing three versions for each—local paraphrasing, full-sentence restructuring, and complex long instructions—totaling 30 rewrites, and used GPT-6 Astra (medium) for fuzzy parameter replacement testing. The overall accuracy was 96.7%96.7\% (29/3029/30). The test was conducted in batches of 10, with details as follows:

Instruction FormCorrect CountAccuracy
Local Paraphrasing10/1010/10100%100\%
Full-Sentence Restructuring9/109/1090%90\%
Complex Long Instructions10/1010/10100%100\%
Total29/3029/3096.7%96.7\%

The only failed case involved implicit value reasoning requiring external geographic knowledge:

  • history: Add a favorite location marker for 47.2165476, 9.5699984 in the OsmAnd maps app.
  • now: In OsmAnd, mark Balzers in Liechtenstein as a favorite location.
  • Q: According to the new task, what should the input text in the historical action be replaced with?
  • A: {"id":"M09","replacements":[{"id":"p1","text":null}]} (The model did not return the corresponding replacement value).

To keep the framework lightweight and free of additional training, we adopted deterministic rule-based parsing in the main experiments; the above experiments demonstrate that using an LLM for parameter parsing can effectively supplement this simplified component, covering more complex natural language paraphrases and implicit parameters.

Furthermore, regarding your concern about the generalization scenario of "detaching from benchmark metadata identifiers and relying solely on imperfect retrieval based on natural language instructions (instruction-based retrieval)," we supplemented an Instruction-only retrieval (w/o task IDs) control experiment on the 60 tasks of SPA-Bench: when completely removing task IDs and task-family cues and retrieving solely based on natural language descriptions, DeltaReplay still maintained success rates of 77.8%77.8\% and 26.7%26.7\% on the Same-task and Same-family groups, respectively (compared to 51.9%51.9\% and 0.0%0.0\% for the Base agent), verifying that the framework remains effective under realistic retrieval conditions without metadata assistance. (If you would like more detailed experimental data, please refer to our response to Reviewer nZd3 Q1.)

AuthorsOpenReview ↗

Response to Reviewer LssG (Part 2/3): Route B Adaptation Coverage and Fallback Categorization (Q1.2)

Q1.2 How Much Adaptation Demand Does Route B Cover, and What Are the Main Fallback Reasons?

Response: We re-examined the 229 execution decisions across the 22 Same-template tasks, among which Route A was executed 95 times, Route B 34 times, and fallback to the Base Agent 100 times. Route B is responsible for replacing parameters and target elements, handing over to the Base Agent if adaptation fails.

Table 1: How Much Adaptation Demand Does Route B Cover

StatisticSteps
Correctly adapted by Route B3232 (86.5%86.5\%)
Required adaptation, but handed over to Base Agent55 (13.5%13.5\%)
Required replacing parameters or target elements (Total)3737

Among the 37 steps requiring replacement of parameters or target elements, Route B correctly completed 32 steps (Route B executed 34 steps in total, with the other 2 steps failing to complete correctly due to element mis-mapping), yielding an adaptation coverage of 86.5%86.5\% (32/3732/37), while the remaining 5 steps (13.5%13.5\%) erroneously fell back to the Base Agent (of which 2 steps were due to parameter adaptation failure, and 3 steps were due to insufficient or inconsistent input information).

Table 2: Categorization of Reasons for the 100 Fallbacks to the Base Agent

Fallback ReasonCountCorrect FallbackErroneous Fallback
Needed to re-evaluate whether the task was completed3333333300
Needed to generate an answer based on current content1212121200
Parameter adaptation code failed to determine new value2323202033
Failed to distinguish elements with the same name997722
Inconsistency between screen and UI tree information111100
No matching historical step found14149955
Candidate rejected or invalid output format returned554411
Unable to locate historical target element on current page332211
Total10010088881212

In summary, categorized by fallback source, among the 100 fallbacks, 88 were correct fallbacks and 12 were erroneous fallbacks, of which 5 were due to parameter adaptation errors, 1 was due to an LLM output format error, and 6 were due to insufficient input information or lack of applicable historical records.

AuthorsOpenReview ↗

Response to Reviewer LssG (Part 3/3): Clarifications and Independent Ablations for Table 4 (Q2)

Q2 How to Interpret the Joint Ablation in Table 4, and Were the Contributions of Individual Components Isolated?

Response: Thank you for your suggestion. Regarding the joint ablation settings in the original Table 4, we clarify and supplement independent controls as follows:

  1. In the original Table 4, w/o Route C removed both Route C fallback and the screen check (45.5%); w/o semantic matching removed both the screen check and the Selector (20.0%), aiming to verify the impact on success rate when directly executing old actions without checking whether they fit the current page and task.
  2. To strictly isolate the independent contribution of each component (screen check, Selector, Route B, Route C), we supplemented individual ablation experiments by disabling single components one by one on the original 30 AndroidWorld Same-app tasks and the original 22 Same-template tasks using Qwen3.8-Flash (existing results from the paper are retained in the table below for comparison):
Task GroupExperimental GroupScreen CheckSelectorTasksSuccessesSuccess Rate (%)Model Calls/StepA/B/C (%)
Same-appFullOnOn301550.02.190/14/86
Same-templatew/o Route A (New)OnOn221672.71.850/18/82
Same-appw/o selector (New)OnOff30826.71.250/40/60
Same-appw/o screen check (New)OffOn30930.02.400/22/78
Same-appJoint Removal (Original Table 4 w/o semantic matching)OffOff30620.00.600/73/27
Same-templateFullOnOn221986.41.0041/15/44
Same-templatew/o Route B (New)OnOn221777.31.6041/0/59
Same-templateOriginal w/o Route C (Original Table 4 Joint Ablation)OffOn221045.50.1768/32/0
Same-templatew/o Route C (New Independent Control)OnOn221150.00.1867/33/0

Summary:

  1. On Same-app, removing the Selector or the screen check individually caused the success rate to drop from 50.0% to 26.7% and 30.0%, respectively, and removing both jointly caused it to drop further to 20.0%, indicating that both checks make independent contributions and their joint absence causes a greater loss. After removing the screen check, the number of calls per step increased from 2.19 to 2.40 because the lack of low-overhead structural filtering caused more mismatched pages to directly trigger LLM Selector calls.
  2. On Same-template, removing only Route B caused the success rate to drop from 86.4% to 77.3% (verifying the independent gain of parameter adaptation on top of verified replay plus fallback); moreover, the number of model calls per step increased from 1.00 to 1.60—this is because unreplaced parameters caused subsequent page states to deviate from the original trajectory, forcing subsequent steps that could originally be directly replayed to frequently trigger the Selector and Base Agent for re-decision; when retaining the screen check and Selector and removing only Route C, the success rate was 50.0%, still significantly lower than the 86.4% of the Full version, showing that falling back to the Base Agent (Route C) is equally indispensable when applicable historical steps are lacking.

Thank you for the detailed rebuttal and the additional experiments. The instruction-only retrieval results, coverage analysis, and separate ablations address several of my concerns. However, the paraphrase experiment tests an LLM-based replacement module offline, leaving the original rule-based method’s performance on varied instructions unclear. I maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.