acceptodds
Withdrawn from ICLR 2027

ECHOES: Seeing Beyond the Current Image via Recursive History Compression and Privileged On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon GUI agents rely on visual history to maintain task states, but retaining historical screenshots increases visual-token usage and inference costs, while textual summaries can discard critical state information, undermining long-horizon reliability. We propose ECHOES, which leverages training-time privileged information to guide textual history compression and strengthen long-horizon reasoning. Specifically, ECHOES adopts a Compression–Think–Answer (CTA) format to recursively update a textual history state and reason. The same model, given privileged multi-image inputs, serves as a teacher that provides dense token-level on-policy distillation on student-generated CTA rollouts. Our Perception–Memory Fidelity (PMF) Gap-guided OPD–RL framework dynamically balances privileged distillation and task-reward optimization, with periodic teacher updates supporting continued improvement. We further introduce ECHOESBMK, a fine-grained diagnostic benchmark derived from real GUI agent failure trajectories to jointly evaluate visual perception and cross-modal state retention. Experiments on four GUI benchmarks and ECHOESBMK demonstrate substantial gains in task performance and memory fidelity. Without retaining historical screenshots at deployment, ECHOES achieves competitive performance with a favorable performance–inference-cost trade-off and exhibits positive zero-shot transfer to unseen visual planning tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
GUI agents, vision-language models, long-horizon reasoning, history compression, privileged on-policy distillation, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PD8tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ECHOES, a long-horizon GUI agent framework that combines recursive textual memory, privileged on-policy distillation, and RL. It also introduces ECHOESBMK to evaluate perception and memory fidelity. Experiments show improvements across several GUI benchmarks.

Critical strengths

1.The problem is practical and important. Reducing historical image overhead while preserving useful task information is a meaningful direction.

2.The experiments cover multiple benchmarks, and the efficiency and ablation results are encouraging.

Critical weaknesses
  1. The framework mainly combines existing techniques, including history summarization, on-policy distillation, and RL. I'm not fully convinced by the novelty of the PMF-guided scheduling. Its improvement over standard linear scheduling is only 3.46 points, and stronger scheduling baselines are missing.

  2. The proposed PMF benchmark mainly evaluates single-screen summaries rather than recursive memory across long trajectories. Also, most GUI results rely on offline action evaluation rather than actual end-to-end task completion. These results do not support the claimed long-horizon reliability.

Minor Comments

1.The privileged teacher only observes up to four previous screenshots, which may limit supervision of longer-term dependencies.

2.The zero-shot transfer experiment contains only 20 Sokoban tasks, making the generalization claim somewhat limited.

3.The main comparison uses different inference-history protocols across baselines.

Questions for authors

1.How does PMF-guided scheduling compare against fixed-weight or other adaptive weighting strategies?

2.Can the authors evaluate memory fidelity over longer recursive trajectories?

3.Are there end-to-end online GUI task completion results?

Reviewer_z1DqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ECHOES targets long-horizon GUI agents that lose task-relevant information when screenshots are replaced by text summaries. It uses a recursive Compression–Think–Action (CTA) format, then trains a deployable single-image policy with privileged on-policy distillation from the same model given recent screenshots, combined with GRPO. A controller adjusts the OPD/RL mixture according to the performance gap between privileged and deployable views, and the paper introduces ECHOESBMK to measure screen perception and visual-to-text state retention. Experiments cover four GUI benchmarks, a zero-shot Sokoban setting, and the diagnostic benchmark.

Critical strengths

1.The training-time privileged-view idea is sensible. Instead of paying to retain screenshots at deployment, the student learns CTA rollouts from its own deployable state, while the teacher scores the same prefixes using recent visual history. This is a clear and practical way to transfer information across observation modes.

  1. The paper includes useful controls beyond headline task success. The CTA interventions in Table 3(b) test shuffled, stale, and absent states, and the training-objective ablations compare the PMF-guided schedule with a linear mixture. ECHOESBMK also attempts to separate direct screen understanding from degradation when visual
Critical weaknesses
  1. The supervision used to create the initial compression targets may leak future information. Appendix D says GPT-5.6 generates target C annotations using the task instruction and the corresponding interaction trajectory as context, but it is not clear whether each C_t is generated from only the prefix available at step t or from the full trajectory, including later actions and outcomes. If future states are visible when constructing intermediate compression targets, CTA-SFT may learn hindsight summaries that cannot be produced online. The paper should specify the annotation prompt and enforce prefix-only targets, or report a controlled comparison.

  2. The proposed PMF Gap is not yet a direct measure of the fidelity of the recursive CTA state. ECHOESBMK’s memory-only protocol first generates a textual screen summary and then answers the same query from that summary; it does not evaluate the recurrent C produced during a multi-step task. In addition, the reported average is computed over the four L2 tasks, excluding element-level and cross-frame tasks. This can be a useful diagnostic, but it supports a narrower claim about screen-summary retention than the paper’s broader claim about long-horizon recursive memory. A benchmark evaluation using actual CTA states from held-out trajectories would better validate the central claim.

  3. The evaluation does not fully isolate the benefits of the proposed training and control strategy. Section 5.1 retains model-specific history protocols, including native or ground-truth history where applicable, while ECHOES receives the current screenshot and its own C. These results are useful as practical comparisons, but do not establish a causal advantage under a matched information budget. The controlled Qwen3-VL-8B history comparison helps, yet a matched-protocol comparison across baselines would make the efficiency claim stronger. Separately, the PMF-guided controller scales the next OPD weight according to the privileged–deployable gap, with clipping. The linear-schedule ablation is useful, but the paper does not show whether this gap is stable across validation subsets and training runs, or whether a fixed weight selected with the same validation budget performs similarly. It is therefore difficult to tell how much of the gain comes from the adaptive controller rather than the choice of training mixture.

Questions for authors
  1. When generating the SFT compression target C_t, does the annotator see only the trajectory prefix up to step t, or the complete trajectory including future actions and outcomes? Can you provide the prompt or a prefix-only control to rule out hindsight leakage?
  2. Can you evaluate PMF using the actual recursive CTA state from held-out multi-step trajectories, rather than a separately generated single-screen summary? How does this change the benchmark’s ranking or the claimed relationship between PMF and task success?
  3. How stable is the PMF-guided OPD weight under different validation subsets and training seeds, and does it outperform a fixed weight selected with the same validation budget?
Reviewer_rxtsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ECHOES, a framework for improving long-horizon GUI agents by preserving task-relevant visual history without retaining previous screenshots during inference. It addresses the trade-off between expensive visual history and information loss from textual compression. ECHOES introduces a Compression–Think–Action (CTA) mechanism that recursively updates textual memory while generating reasoning and actions. Its key contribution is privileged on-policy distillation, where a teacher with access to multiple historical screenshots supervises a student operating on the current screenshot and compressed memory. A PMF Gap-guided OPD–RL framework dynamically balances distillation and reinforcement learning. The paper also introduces ECHOESBMK, a diagnostic benchmark evaluating visual perception and memory fidelity. Experiments across four GUI benchmarks demonstrate substantial improvements. On GUI-Odyssey, exact-match accuracy increases from 48.27% to 68.58%, with reduced inference costs. Ablation studies validate individual components, while additional experiments demonstrate improved memory fidelity, long-horizon robustness, and zero-shot transfer to visual planning tasks.

Critical strengths
  1. Strong performance–efficiency trade-off. The paper addresses a practical challenge in long-horizon GUI agents: preserving historical information without expensive visual context. ECHOES improves GUI-Odyssey exact-match accuracy from 48.27% to 68.58% while requiring substantially fewer input tokens than multi-screenshot approaches. This demonstrates practical value and provides a meaningful argument for acceptance, particularly for resource-constrained deployment.

  2. Comprehensive experimental validation. The paper evaluates ECHOES across four GUI benchmarks and introduces ECHOESBMK to assess perception and memory fidelity. Ablation studies isolate the contributions of recursive compression, privileged distillation, reinforcement learning, and adaptive scheduling. Notably, PMF-guided scheduling improves accuracy by 3.46 percentage points over linear scheduling, with statistical significance. These experiments provide reasonable evidence supporting the proposed design choices.

Critical weaknesses

The first concern is limited methodological novelty. Recursive textual memory, privileged-context distillation, and combining on-policy distillation with reinforcement learning have been explored previously. Although ECHOES integrates these components effectively, the PMF-guided weighting mechanism is relatively straightforward. Stronger comparisons with closely related methods under matched training settings would help establish whether the approach offers substantial methodological advances beyond combining existing techniques.

The second concern is the limited validation of real-world long-horizon execution. Most GUI evaluations rely on offline action prediction and grounding metrics rather than closed-loop task completion, making it difficult to assess robustness under accumulated execution errors. Additionally, ECHOESBMK primarily evaluates memory fidelity through single-screen summary replay, which does not fully capture recursive memory degradation across extended trajectories. While the reported improvements are encouraging, these limitations weaken the evidence supporting broader generalization and practical reliability.

Questions for authors

no

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.