- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TRACE, a training-free visual token compression method for GUI agents. It combines interface layout, instruction relevance, and feature diversity to select tokens, while keeping some tokens for spatial coverage. Historical screenshots are compressed further using nested token sets and cache reuse. The paper evaluates six benchmarks, multiple backbones, and different token budgets, with extensive ablations and latency measurements.
Critical strengths
-
Extensive experiments. The paper evaluates six benchmarks, multiple model sizes, and a different model family. It also includes detailed ablations, sensitivity tests, and failure analysis. This provides substantial evidence that the method helps under tight token budgets.
-
Practical efficiency gains. The paper measures latency and cache size, includes selection overhead, and separates the gains from cache reuse and token selection. It also tests existing selectors on the same cache-reuse path. These comparisons make the efficiency results more convincing.
Critical weaknesses
-
No closed-loop task success. The method is motivated by preserving historical screenshots for future decisions. However, the multi-step benchmarks report Step SR rather than full-task success. A correct action on a given step does not show whether an agent can finish a task when its own mistakes change later observations. Appendix C.1 also states that the evaluated trajectories are short and mostly rely on recent screens. The target-retention tests are useful, but they do not establish that the retained information is sufficient for later decisions. This leaves the main claim of trajectory robustness insufficiently tested.
-
Limited novelty. The method mainly combines layout-based importance, instruction relevance, feature diversity, spatial coverage, and cache reuse. The combined system is useful, but the new algorithmic contribution appears incremental. In particular, Table 8 already identifies existing methods whose selections form nested sets. Thus, nesting itself is not a new property. The paper needs to explain more clearly what new principle TRACE adds beyond adapting and combining these ideas for GUI agents.
Questions for authors
1.Do you have closed-loop results in which the agent executes its predicted actions and receives the resulting observations? Full-task success and total task time, especially for longer tasks, could materially change my assessment.
2.How much of the gain comes specifically from preserving historical evidence? With the current-frame budget fixed, how does TRACE compare with removing visual history or keeping only recent frames on tasks that require information from much earlier screens?
3.Table 8 shows that several existing selectors already support nested sets. What is the main algorithmic novelty beyond adding layout guidance and coverage repair to such selectors?
5 comments
Response to Reviewer pzov (Part 2/3)
Q2. Contribution of historical evidence at a fixed current-frame budget
How much of the gain comes specifically from preserving historical evidence? With the current-frame budget fixed, how does TRACE compare with removing visual history or keeping only recent frames on tasks that require information from much earlier screens?
With current-frame selection fixed, TRACE gains 3.64 points between H0 and H10 on MemGUI and 1.79 points between H0 and H1 on OmniGUI. These are recorded-action accuracy comparisons, while MiniWeb separately evaluates interactive task success.
① Mind2Web Cross-Task history sweep. Only preceding screenshots change, and each entry is action-prediction accuracy in percent.
| Method | H0 | H1 | H2 | H3 | H4 | H5 | H5 minus H0 (pp) |
|---|---|---|---|---|---|---|---|
| Dense | 48.11 | 51.94 | 53.29 | 54.08 | 53.64 | 54.48 | +6.37 |
| TRACE | 41.78 | 41.53 | 41.58 | 41.73 | 41.33 | 42.63 | +0.85 |
Dense gains 6.37 points between H0 and H5, while TRACE gains 0.85 points with a paired interval that crosses zero. OmniGUI and MemGUI below provide the clearer measured history effects for TRACE.
② OmniGUI and AndroidControl short-history comparisons. Each pair uses the same decision context at H0 and H1.
| Dataset / method | H0 | H1 | Gain (pp) |
|---|---|---|---|
| OmniGUI / Dense | 49.07 | 52.57 | +3.50 |
| OmniGUI / TRACE | 47.74 | 49.53 | +1.79 |
| AndroidControl / Dense | 57.38 | 57.67 | +0.29 |
| AndroidControl / TRACE | 52.51 | 52.61 | +0.10 |
On OmniGUI, TRACE's 1.79-point gain has a paired trajectory-bootstrap 95% confidence interval from 0.77 to 2.82 points. On AndroidControl, its H0-to-H1 difference is 0.10 points. The size of the history benefit therefore depends on the evaluated task.
③ MemGUI-3K tests the use of earlier visual observations in mobile GUI trajectories (“MemGUI-Agent: An End-to-End Long-Horizon Mobile GUI Agent with Proactive Context Management,” arXiv:2606.19926). In the evaluated subset, trajectories average 28.12 steps and reach 121, putting older screenshots beyond a short recent window. With current-frame selection fixed, changing only the number of preceding screenshots directly tests whether retained observations help later recorded-action decisions. This recorded-action analysis complements MiniWeb's closed-loop task-completion results. Each entry reports recorded-action accuracy in percent.
| Method | H0 | H5 | H10 | H10 minus H0 (pp) | Paired 95% interval (pp) |
|---|---|---|---|---|---|
| Dense | 50.88 | 56.09 | 56.73 | +5.86 | [4.85, 6.86] |
| TRACE | 47.01 | 49.60 | 50.66 | +3.64 | [2.76, 4.51] |
Across 5,875 decisions common to H0 and H10, TRACE makes 214 more correct predictions at H10, a 3.64-point paired gain computed before display rounding. The further H5-to-H10 gain is 1.06 points, with a paired 95% interval from 0.51 to 1.62 points. When context overflow counts as incorrect over all 6,146 decisions, TRACE improves from 46.10% at H0 to 48.80% at H5. These controlled trajectories give the clearest evidence that access beyond recent frames can improve TRACE's action prediction.
④ MiniWeb TRACE history sweep. Within each method, the visual history window changes across H0, H1, H3, and H6. TRACE retains 50% of the current frame at every window, while Dense repeated encoding provides the corresponding full-token baseline. The table reports Task SR for both methods at each history window.
| Method | H0 | H1 | H3 | H6 |
|---|---|---|---|---|
| Dense, repeated encoding | 36.31 | 39.84 | 41.46 | 43.90 |
| TRACE | 37.70 | 40.11 | 42.28 | 44.44 |
TRACE improves Task SR over Dense repeated encoding by 1.39, 0.27, 0.82, and 0.54 percentage points at H0, H1, H3, and H6, respectively. Across the four matched history windows, TRACE consistently achieves higher Task SR. Within TRACE, the current-frame budget remains fixed while the retained visual history window changes.
Response to Reviewer pzov (Part 3/3)
Q3. Algorithmic contribution beyond nested selection
Table 8 shows that several existing selectors already support nested sets. What is the main algorithmic novelty beyond adding layout guidance and coverage repair to such selectors?
The central contribution is the lifecycle formulation and MKC mechanism that turn one admitted native-token order into reusable, context-consistent multi-step GUI state. Five complementary aspects define this contribution.
① Table 8 establishes pipeline compatibility. It audits whether each official selector can decide a keep before the first LLM prefill, expose nested keeps, and preserve native rows that can later be deleted from the cache. A check mark means that, under its official rule and this deletion-only contract, the selector can feed the same MKC serving path without changing its selection rule. This modular interface enables matched execution-path comparisons across selectors. The remaining TRACE components supply and evaluate trajectory robustness, GUI coverage, and the reusable state transition.
② MKC realizes the reusable state transition. It implements a concrete Prepare–Retire–Append state machine. When a new screenshot arrives, it truncates the cache at the retiring frame, crops that frame's already encoded native embeddings to the smaller history prefix by indexing, and restores those rows together with the previous answer, the fresh frame, and the new instruction in one merged forward pass. Only the fresh screenshot is visually encoded. Historical pixels, detections, and selector computations are not rerun. The unchanged causal prefix remains reusable, while the dependent language/action suffix is replayed against the contracted visual context. This replay is essential: simply deleting visual rows while retaining downstream states computed from the old visual context would leave stale cross-modal state. MKC reuses the retained visual features exactly and recomputes the affected suffix against the contracted context. This context-consistent state transition turns a nested order into one-encode-per-frame execution.
③ The algorithm and the serving lifecycle are co-designed through an explicit interface contract. LIP, NEO, and NCR decide what evidence should survive: they admit task-relevant, layout-aware, diverse, spatially supported native rows before future targets are known. MKC decides how that evidence survives: the current keep becomes a deletion-only history prefix and then persistent session state. Native rows and nesting are therefore the interface contract that makes monotone contraction executable without feature reconstruction. Conversely, MKC is selector-compatible: Table 8 identifies existing selectors that can be ported to this lifecycle, while Table 9 locates TRACE's integration across the surveyed method families by jointly considering multi-step GUI operation, pre-admission, native rows, nested budgets, and no historical re-encoding.
④ The measurements separate the value of MKC from the value of the admitted order. In Fig. 5, MKC alone reduces TTFT from 1116.8 ms to 487.4 ms at unchanged accuracy. Porting DivPrune, VisPruner, and ZOO-Prune to the same MKC path accelerates them by 1.8–2.0×, demonstrating that MKC provides a reusable pipeline contribution across compatible selectors. The complete TRACE selector then reduces TTFT further to 452.7 ms at the mild budget while adding the accuracy margin. In Fig. 5's visual-cache attribution, visual KV falls from 697 MB to 292 MB. The new MiniWeb results confirm the same division of labor in closed-loop execution: methods share the MKC path, while TRACE attains 44.44% Task SR with 13.14% lower mean task-trajectory time than Dense MKC.
⑤ Selection quality complements the systems contribution. Under the same LIP prior, TRACE retains a substantial accuracy margin: at 5% ScreenSpot-v2 retention, the strongest tested same-prior baseline reaches 34.36%, versus 55.19% for TRACE. On the shared MiniWeb MKC path, TRACE reaches 44.44% Task SR, compared with 42.00% for DivPrune + LIP and 42.55% for Uniform. These matched controls separate two complementary contributions: MKC supplies the reusable monotone lifecycle and its dominant serving savings, whereas LIP–NEO–NCR supplies a trajectory-robust admitted order that preserves quality under contraction.
Together, Table 8 establishes a modular serving interface, while the mechanism and matched controls validate the end-to-end coupling of irreversible admission with MKC-based reusable state.
Response to Reviewer pzov (Part 1/3)
We sincerely thank the reviewer for requesting closed-loop evidence and a sharper account of our contribution. We added 44.44% MiniWeb Task SR, a 3.64-point paired MemGUI history gain, and matched serving analyses showing 61.19% lower language prefill. We also clarify MKC as the core reusable-state mechanism and Table 8 as a pipeline-compatibility analysis. We respectfully ask the reviewer to reconsider the score based on this strengthened evidence.
Q1. Closed-loop completion and total task time
MiniWeb evaluates complete interactive tasks with automatic outcome checks (“Back to Basics: A Skill-Centric Approach to Generalist Web Agents,” arXiv:2610.35823). Each action changes the next screenshot, testing compact history over full trajectories. TRACE reaches 44.44% Task SR in 124.29 seconds per task trajectory. All methods use GUI-Owl-1.5-8B-Instruct under the same action protocol, prompt family, and execution limits.
① MiniWeb task outcomes. H is the number of preceding screenshots. H6 selectors share the MKC path and retain 50% of the current frame and 10% of each historical frame. Time is averaged over observed trajectories.
At H6, TRACE leads DivPrune, DivPrune + LIP, and Uniform by 3.52, 2.44, and 1.89 Task-SR points. Its mean task time is 18.60% lower than dense repeated encoding and 13.14% lower than Dense MKC. Dense repeated encoding rises from 36.31% at H0 to 43.90% at H6, confirming the value of history.
② MiniWeb request latency. Timing scopes overlap. For repeated encoding, the visual column combines vision and pruning.
TRACE reduces prefill, TTFT, and request E2E by 61.19%, 52.87%, and 19.14% versus repeated encoding. Versus Dense MKC, it reduces prefill by 51.71% and TTFT by 24.33% despite its selection cost, while attaining higher Task SR.
③ MiniWeb language workload. Values are rounded means per request, with replay counts following observed trajectories. Displayed sums may differ by one token.
TRACE reuses the unchanged causal prefix while reducing the mean replayed suffix from 8,411 to 3,550 tokens, or 57.79% versus Dense MKC. This explains its lower prefill time.
④ MiniWeb full-path GPU memory. Allocated and reserved denote live-allocation and allocator-pool peaks. The sampled device peak has broader scope.
Versus Dense MKC, TRACE reduces peak allocated memory by 9.29%, persistent visual features by 84.25%, and committed cached KV by 20.01%. The allocated peak includes detection, generation, and selection temporaries. Together, these results establish complete-task quality and serving efficiency.