acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Trajectory-robust Admission with Evidence Ordering for Efficient GUI Agents

Abstract

GUI agents accumulate high-resolution screenshots as the trajectory unfolds, increasing inference latency and memory usage. Training-free visual token pruning can reduce this cost, but cache reuse introduces a fundamental constraint. Once tokens are discarded, the corresponding visual evidence cannot be recovered without re-encoding. Pruning therefore becomes an *irreversible admission decision* that must remain useful for unknown future targets while preserving coverage of operable regions under tight budgets. To address these challenges, we propose **TRACE**, a training-free framework for ***T**rajectory-**r**obust **A**dmission and **C**overage-aware **E**vidence ordering*. Specifically, we combine a query-independent layout-derived interaction prior with instruction relevance and feature novelty to rank visual evidence according to both potential future utility and diversity. Then, we reserve part of the budget for native visual tokens distributed across the screen, repairing missing spatial coverage without breaking the ordering. Together, these mechanisms produce a nested token order, allowing retained visual evidence to shrink monotonically across budgets while remaining reusable throughout the trajectory. Finally, Monotone KV Contraction incrementally contracts retired frames into compact session state, avoiding repeated visual encoding or pruning. Extensive experiments across six GUI benchmarks and diverse models verify the effectiveness of our proposed TRACE under tight budgets. The source code will be released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Layout-derived Interaction Prior,Nested Evidence Ordering,Monotone KV Contraction,Efficient GUI Agents,Training Free

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pzovOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRACE, a training-free visual token compression method for GUI agents. It combines interface layout, instruction relevance, and feature diversity to select tokens, while keeping some tokens for spatial coverage. Historical screenshots are compressed further using nested token sets and cache reuse. The paper evaluates six benchmarks, multiple backbones, and different token budgets, with extensive ablations and latency measurements.

Critical strengths
  1. Extensive experiments. The paper evaluates six benchmarks, multiple model sizes, and a different model family. It also includes detailed ablations, sensitivity tests, and failure analysis. This provides substantial evidence that the method helps under tight token budgets.

  2. Practical efficiency gains. The paper measures latency and cache size, includes selection overhead, and separates the gains from cache reuse and token selection. It also tests existing selectors on the same cache-reuse path. These comparisons make the efficiency results more convincing.

Critical weaknesses
  1. No closed-loop task success. The method is motivated by preserving historical screenshots for future decisions. However, the multi-step benchmarks report Step SR rather than full-task success. A correct action on a given step does not show whether an agent can finish a task when its own mistakes change later observations. Appendix C.1 also states that the evaluated trajectories are short and mostly rely on recent screens. The target-retention tests are useful, but they do not establish that the retained information is sufficient for later decisions. This leaves the main claim of trajectory robustness insufficiently tested.

  2. Limited novelty. The method mainly combines layout-based importance, instruction relevance, feature diversity, spatial coverage, and cache reuse. The combined system is useful, but the new algorithmic contribution appears incremental. In particular, Table 8 already identifies existing methods whose selections form nested sets. Thus, nesting itself is not a new property. The paper needs to explain more clearly what new principle TRACE adds beyond adapting and combining these ideas for GUI agents.

Questions for authors

1.Do you have closed-loop results in which the agent executes its predicted actions and receives the resulting observations? Full-task success and total task time, especially for longer tasks, could materially change my assessment.

2.How much of the gain comes specifically from preserving historical evidence? With the current-frame budget fixed, how does TRACE compare with removing visual history or keeping only recent frames on tasks that require information from much earlier screens?

3.Table 8 shows that several existing selectors already support nested sets. What is the main algorithmic novelty beyond adding layout guidance and coverage repair to such selectors?

5 comments
AuthorsOpenReview ↗

Response to Reviewer pzov (Part 1/3)

We sincerely thank the reviewer for requesting closed-loop evidence and a sharper account of our contribution. We added 44.44% MiniWeb Task SR, a 3.64-point paired MemGUI history gain, and matched serving analyses showing 61.19% lower language prefill. We also clarify MKC as the core reusable-state mechanism and Table 8 as a pipeline-compatibility analysis. We respectfully ask the reviewer to reconsider the score based on this strengthened evidence.

Q1. Closed-loop completion and total task time

Do you have closed-loop results in which the agent executes its predicted actions and receives the resulting observations? Full-task success and total task time, especially for longer tasks, could materially change my assessment.

MiniWeb evaluates complete interactive tasks with automatic outcome checks (“Back to Basics: A Skill-Centric Approach to Generalist Web Agents,” arXiv:2610.35823). Each action changes the next screenshot, testing compact history over full trajectories. TRACE reaches 44.44% Task SR in 124.29 seconds per task trajectory. All methods use GUI-Owl-1.5-8B-Instruct under the same action protocol, prompt family, and execution limits.

① MiniWeb task outcomes. H is the number of preceding screenshots. H6 selectors share the MKC path and retain 50% of the current frame and 10% of each historical frame. Time is averaged over observed trajectories.

MethodWindowTask SR (%)Mean elapsed time per task trajectory (s)
Dense, repeated encodingH036.31112.97
Dense, repeated encodingH139.84123.20
Dense, repeated encodingH341.46138.52
Dense, repeated encodingH643.90152.69
Dense MKCH643.90143.10
DivPrune MKCH640.92123.56
DivPrune + LIP MKCH642.00128.98
Uniform MKCH642.55129.83
TRACE MKCH644.44124.29

At H6, TRACE leads DivPrune, DivPrune + LIP, and Uniform by 3.52, 2.44, and 1.89 Task-SR points. Its mean task time is 18.60% lower than dense repeated encoding and 13.14% lower than Dense MKC. Dense repeated encoding rises from 36.31% at H0 to 43.90% at H6, confirming the value of history.

② MiniWeb request latency. Timing scopes overlap. For repeated encoding, the visual column combines vision and pruning.

Method, H6Processor (ms)Vision + detection + selection (ms)LLM prefill (ms)TTFT (ms)Request E2E (s)
Dense, repeated encoding—284.131,145.532,065.205.946
Dense MKC290.4568.57920.711,286.425.252
DivPrune MKC280.99108.29435.03830.074.562
DivPrune + LIP MKC289.86193.28439.19928.364.631
Uniform MKC292.4871.27444.59814.394.672
TRACE MKC292.79229.82444.59973.374.808

TRACE reduces prefill, TTFT, and request E2E by 61.19%, 52.87%, and 19.14% versus repeated encoding. Versus Dense MKC, it reduces prefill by 51.71% and TTFT by 24.33% despite its selection cost, while attaining higher Task SR.

③ MiniWeb language workload. Values are rounded means per request, with replay counts following observed trajectories. Displayed sums may differ by one token.

Method, H6Input tokensReused prefix tokensReplayed suffix tokensPrefix reuse (%)
Dense, repeated encoding12,279012,2790.00
Dense MKC12,3873,9768,41132.10
DivPrune MKC7,3743,9503,42353.57
DivPrune + LIP MKC7,4293,9743,45553.50
Uniform MKC7,4863,9823,50453.20
TRACE MKC7,5203,9693,55052.78

TRACE reuses the unchanged causal prefix while reducing the mean replayed suffix from 8,411 to 3,550 tokens, or 57.79% versus Dense MKC. This explains its lower prefill time.

④ MiniWeb full-path GPU memory. Allocated and reserved denote live-allocation and allocator-pool peaks. The sampled device peak has broader scope.

Method, H6Peak allocated (GiB)Peak reserved (GiB)Sampled device peak (GiB)Persistent visual features (MiB)Committed cached KV peak (MiB)Detector weights / buffers (MiB)
Dense MKC27.6543.1943.70204.754,254.190.00
DivPrune MKC24.6230.7031.2132.253,180.800.00
DivPrune + LIP MKC25.5934.2434.7632.253,600.5638.21
Uniform MKC25.3231.8232.3332.253,470.910.00
TRACE MKC25.0832.7733.2932.253,402.8438.21

Versus Dense MKC, TRACE reduces peak allocated memory by 9.29%, persistent visual features by 84.25%, and committed cached KV by 20.01%. The allocated peak includes detection, generation, and selection temporaries. Together, these results establish complete-task quality and serving efficiency.

AuthorsOpenReview ↗

Response to Reviewer pzov (Part 2/3)

Q2. Contribution of historical evidence at a fixed current-frame budget

How much of the gain comes specifically from preserving historical evidence? With the current-frame budget fixed, how does TRACE compare with removing visual history or keeping only recent frames on tasks that require information from much earlier screens?

With current-frame selection fixed, TRACE gains 3.64 points between H0 and H10 on MemGUI and 1.79 points between H0 and H1 on OmniGUI. These are recorded-action accuracy comparisons, while MiniWeb separately evaluates interactive task success.

① Mind2Web Cross-Task history sweep. Only preceding screenshots change, and each entry is action-prediction accuracy in percent.

MethodH0H1H2H3H4H5H5 minus H0 (pp)
Dense48.1151.9453.2954.0853.6454.48+6.37
TRACE41.7841.5341.5841.7341.3342.63+0.85

Dense gains 6.37 points between H0 and H5, while TRACE gains 0.85 points with a paired interval that crosses zero. OmniGUI and MemGUI below provide the clearer measured history effects for TRACE.

② OmniGUI and AndroidControl short-history comparisons. Each pair uses the same decision context at H0 and H1.

Dataset / methodH0H1Gain (pp)
OmniGUI / Dense49.0752.57+3.50
OmniGUI / TRACE47.7449.53+1.79
AndroidControl / Dense57.3857.67+0.29
AndroidControl / TRACE52.5152.61+0.10

On OmniGUI, TRACE's 1.79-point gain has a paired trajectory-bootstrap 95% confidence interval from 0.77 to 2.82 points. On AndroidControl, its H0-to-H1 difference is 0.10 points. The size of the history benefit therefore depends on the evaluated task.

③ MemGUI-3K tests the use of earlier visual observations in mobile GUI trajectories (“MemGUI-Agent: An End-to-End Long-Horizon Mobile GUI Agent with Proactive Context Management,” arXiv:2606.19926). In the evaluated subset, trajectories average 28.12 steps and reach 121, putting older screenshots beyond a short recent window. With current-frame selection fixed, changing only the number of preceding screenshots directly tests whether retained observations help later recorded-action decisions. This recorded-action analysis complements MiniWeb's closed-loop task-completion results. Each entry reports recorded-action accuracy in percent.

MethodH0H5H10H10 minus H0 (pp)Paired 95% interval (pp)
Dense50.8856.0956.73+5.86[4.85, 6.86]
TRACE47.0149.6050.66+3.64[2.76, 4.51]

Across 5,875 decisions common to H0 and H10, TRACE makes 214 more correct predictions at H10, a 3.64-point paired gain computed before display rounding. The further H5-to-H10 gain is 1.06 points, with a paired 95% interval from 0.51 to 1.62 points. When context overflow counts as incorrect over all 6,146 decisions, TRACE improves from 46.10% at H0 to 48.80% at H5. These controlled trajectories give the clearest evidence that access beyond recent frames can improve TRACE's action prediction.

④ MiniWeb TRACE history sweep. Within each method, the visual history window changes across H0, H1, H3, and H6. TRACE retains 50% of the current frame at every window, while Dense repeated encoding provides the corresponding full-token baseline. The table reports Task SR for both methods at each history window.

MethodH0H1H3H6
Dense, repeated encoding36.3139.8441.4643.90
TRACE37.7040.1142.2844.44

TRACE improves Task SR over Dense repeated encoding by 1.39, 0.27, 0.82, and 0.54 percentage points at H0, H1, H3, and H6, respectively. Across the four matched history windows, TRACE consistently achieves higher Task SR. Within TRACE, the current-frame budget remains fixed while the retained visual history window changes.

AuthorsOpenReview ↗

Response to Reviewer pzov (Part 3/3)

Q3. Algorithmic contribution beyond nested selection

Table 8 shows that several existing selectors already support nested sets. What is the main algorithmic novelty beyond adding layout guidance and coverage repair to such selectors?

The central contribution is the lifecycle formulation and MKC mechanism that turn one admitted native-token order into reusable, context-consistent multi-step GUI state. Five complementary aspects define this contribution.

① Table 8 establishes pipeline compatibility. It audits whether each official selector can decide a keep before the first LLM prefill, expose nested keeps, and preserve native rows that can later be deleted from the cache. A check mark means that, under its official rule and this deletion-only contract, the selector can feed the same MKC serving path without changing its selection rule. This modular interface enables matched execution-path comparisons across selectors. The remaining TRACE components supply and evaluate trajectory robustness, GUI coverage, and the reusable state transition.

② MKC realizes the reusable state transition. It implements a concrete Prepare–Retire–Append state machine. When a new screenshot arrives, it truncates the cache at the retiring frame, crops that frame's already encoded native embeddings to the smaller history prefix by indexing, and restores those rows together with the previous answer, the fresh frame, and the new instruction in one merged forward pass. Only the fresh screenshot is visually encoded. Historical pixels, detections, and selector computations are not rerun. The unchanged causal prefix remains reusable, while the dependent language/action suffix is replayed against the contracted visual context. This replay is essential: simply deleting visual rows while retaining downstream states computed from the old visual context would leave stale cross-modal state. MKC reuses the retained visual features exactly and recomputes the affected suffix against the contracted context. This context-consistent state transition turns a nested order into one-encode-per-frame execution.

③ The algorithm and the serving lifecycle are co-designed through an explicit interface contract. LIP, NEO, and NCR decide what evidence should survive: they admit task-relevant, layout-aware, diverse, spatially supported native rows before future targets are known. MKC decides how that evidence survives: the current keep becomes a deletion-only history prefix and then persistent session state. Native rows and nesting are therefore the interface contract that makes monotone contraction executable without feature reconstruction. Conversely, MKC is selector-compatible: Table 8 identifies existing selectors that can be ported to this lifecycle, while Table 9 locates TRACE's integration across the surveyed method families by jointly considering multi-step GUI operation, pre-admission, native rows, nested budgets, and no historical re-encoding.

④ The measurements separate the value of MKC from the value of the admitted order. In Fig. 5, MKC alone reduces TTFT from 1116.8 ms to 487.4 ms at unchanged accuracy. Porting DivPrune, VisPruner, and ZOO-Prune to the same MKC path accelerates them by 1.8–2.0×, demonstrating that MKC provides a reusable pipeline contribution across compatible selectors. The complete TRACE selector then reduces TTFT further to 452.7 ms at the mild budget while adding the accuracy margin. In Fig. 5's visual-cache attribution, visual KV falls from 697 MB to 292 MB. The new MiniWeb results confirm the same division of labor in closed-loop execution: methods share the MKC path, while TRACE attains 44.44% Task SR with 13.14% lower mean task-trajectory time than Dense MKC.

⑤ Selection quality complements the systems contribution. Under the same LIP prior, TRACE retains a substantial accuracy margin: at 5% ScreenSpot-v2 retention, the strongest tested same-prior baseline reaches 34.36%, versus 55.19% for TRACE. On the shared MiniWeb MKC path, TRACE reaches 44.44% Task SR, compared with 42.00% for DivPrune + LIP and 42.55% for Uniform. These matched controls separate two complementary contributions: MKC supplies the reusable monotone lifecycle and its dominant serving savings, whereas LIP–NEO–NCR supplies a trajectory-robust admitted order that preserves quality under contraction.

Together, Table 8 establishes a modular serving interface, while the mechanism and matched controls validate the end-to-end coupling of irreversible admission with MKC-based reusable state.

Reviewer_5mwGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies visual token pruning for GUI agents that reuse screenshot history and propose TRACE and a KV contraction mechanism. TRACE selects visual tokens once and keeps a smaller subset when a screenshot becomes history. It combines a layout prior from OmniParser, instruction relevance, feature diversity, and spatial coverage repair to construct nested token sets. Its Monotone KV Contraction mechanism reuses stored visual embeddings and recomputes the affected language-model context when a frame is shortened. Experiments show improved accuracy under limited token budgets and reduced latency and visual KV storage.

Critical strengths
  1. The paper identifies a meaningful problem: tokens removed from a screenshot may be needed by later actions. Combining current relevance with layout information and spatial coverage is a reasonable approach.
  2. The experiments provide useful evidence for the individual components. TRACE improves substantially on several single-step benchmarks, and comparisons using the same layout prior or MKC serving path help separate its selection benefits from additional detector information and cache reuse.
Critical weaknesses
  1. TRACE fixes the historical subset when a screenshot first appears, using the available instruction, layout, and visual features. This selection cannot adapt when later actions require different information. The reported results leave uncertainty about this trade-off when later actions depend on information outside the initially selected regions.
  2. The history repair chooses regional medoids based on feature similarity. This favors representative content, which may differ from task-critical content such as an unusual error message or a specific value. Although other retained tokens may preserve these details, the paper does not clearly explain how the combined selection addresses this potential mismatch.
Questions for authors
  1. The paper shows efficiency benefits and a small accuracy loss from nested selection. How would this trade-off change if full encoder features were retained for later re-selection, including storage and downstream LLM recomputation costs?
  2. How does the combined selection preserve task-critical details, such as unusual error messages or specific values, that may not be captured by spatial coverage or regional medoids?
  3. What motivates using the same retention ratio across historical frames despite their potentially different importance to future actions?
  4. What is the main methodological difference between NEO and existing relevance-and-diversity selection methods, beyond adding the layout prior?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 5mwG (Part 1/3)

We sincerely thank the reviewer for highlighting the central trade-off between compact reusable state and later access to visual evidence. We took these concerns seriously. We added matched full-pool controls, complete storage and replay measurements, component-level evidence for rare and spatially distributed details, and age-aware history-allocation experiments. These results quantify the recovery–storage trade-off and clarify the contributions of NEO, NCR, and MKC. We respectfully ask the reviewer to reconsider the score based on this expanded evidence and the clearer empirical account of TRACE's design choices.

Q1. Retaining full features for later re-selection

How would this trade-off change if full encoder features were retained for later re-selection, including storage and downstream LLM recomputation costs?

Keeping full encoder features permits later re-selection, but compact TRACE uses 83.34% less persistent visual-feature storage on OmniGUI. The comparison holds the current and historical budgets fixed and measures feature storage alongside quality, latency, and recomputation.

① Full-pool and compact-state serving comparison. The table reports the complete OmniGUI evaluation set under the same serving configurations. Step SR, timing, memory, and replay measurements are collected together for the corresponding configurations.

Feature storage and language pathStep SR (%)Mean E2E (s)Mean TTFT (s)Persistent features CPU / GPU (MiB)Peak allocated GPU (GiB)
Re-encode all frames, fresh prefill48.914.3590.9200 / 021.88
Full CPU pool, fresh prefill48.914.5180.804591.38 / 020.47
Full GPU pool, fresh prefill48.914.4130.7890 / 591.3821.15
Full GPU pool, unchanged-prefix reuse48.914.2050.7500 / 591.3822.07
Compact TRACE with MKC48.914.2160.7340 / 98.5021.40

At 48.91% Step SR across all five serving paths, compact TRACE uses 83.34% less persistent visual-feature storage than the full GPU pool. Its mean end-to-end time is 4.216 seconds, close to the full GPU pool with prefix reuse at 4.205 seconds, while stored GPU features fall from 591.38 to 98.50 MiB. The full pool retains the ability to re-select previously discarded evidence. TRACE instead commits a compact state and preserves prefix reuse without maintaining the full feature pool.

All five serving paths achieve the same 48.91% Step SR on the complete OmniGUI evaluation set. They differ in feature-storage and language-model execution paths, allowing the timing, storage, and replay measurements to isolate the corresponding serving trade-offs on that same evaluation set.

② Frame encoding and language replay. The counts cover the complete OmniGUI decision set.

DeploymentFrame encodingsReplayed language tokens
Re-encode, fresh prefill6,7035,034,856
Full CPU pool, fresh prefill2,5725,034,856
Full GPU pool, fresh prefill2,5725,034,856
Full GPU pool, prefix reuse2,5723,039,486
Compact TRACE with MKC2,5723,043,288

Caching cuts frame encodings from 6,703 to 2,572, a reduction of 61.63%. Without prefix reuse, cached variants still replay 5,034,856 language tokens. Prefix reuse reduces the count to about 3.04 million for both the full-pool and compact variants. When the retained visual rows change, the unchanged causal prefix remains reusable and only the dependent suffix is recomputed. Compact admission adds the storage saving to those encoding and language-computation benefits.

AuthorsOpenReview ↗

Response to Reviewer 5mwG (Part 2/3)

Q2. Rare task-critical details and regional medoids

How does the combined selection preserve task-critical details, such as unusual error messages or specific values, that may not be captured by spatial coverage or regional medoids?

NEO prioritizes likely task evidence, while NCR preserves spatial support for locating it. NEO combines instruction relevance, a soft interaction prior, and conditional novelty to order native visual tokens. The prior favors potentially operable regions without masking others. NCR then uses the unprotected portion of the budget to restore distributed native-token support without displacing NEO's protected evidence prefix.

① Component ablation. Entries report prediction accuracy in percent at the indicated current-frame and historical retention budgets.

Selection componentsScreenSpot-v2, 10%OmniGUI, 50%/10%Mind2Web pooled, 25%/5%
Prior only32.6340.9425.43
NEO without prior or repair36.4845.8814.31
Prior + NEO56.4547.4327.79
Prior + NEO + NCR73.9048.9134.20

Adding NCR to the prior plus NEO increases ScreenSpot-v2 accuracy from 56.45% to 73.90% at 10% retention. The full combination reaches 48.91% on OmniGUI and 34.20% on pooled Mind2Web. Table 2 measures the contribution of complete modules. Table 3 complements it by removing the instruction-relevance factor alone from the full NEO ordering at the tight budget, reducing accuracy by 15.57 points on ScreenSpot-v2 at 5% retention and 4.12 points on OmniGUI at 25%/5%. Together, the two ablations isolate distinct contributions from task relevance and spatial repair.

② Spatial grounding mechanism. The spatial failure mode is concrete. Li et al. report an example in which pruning shifts an object shown on the right side of a screenshot to the upper-left in the model's response (“Where and How to Prune,” arXiv:2603.26041v4). Such an error can misplace a GUI action even if the object is identified. NEO helps protect task-relevant details, while NCR maintains the visual support needed to ground their location. The full-pool control complements this mechanism analysis by quantifying the storage cost of retaining recovery access to all evidence.

Q3. Equal retention ratios across historical frames

What motivates using the same retention ratio across historical frames despite their potentially different importance to future actions?

Uniform retention offers a simple, reuse-friendly history policy, while age-dependent allocation raises action accuracy with additional language replay. We compare policies under a fixed current-frame budget and the same total historical-token budget.

① History allocation and accuracy on OmniGUI. Replay change is relative to uniform fixed admission.

Historical policyStep SR (%)Gain over uniform (pp)Replayed language tokensReplay change (%)
Uniform fixed admission48.91—3,043,288—
Inverse-age allocation49.65+0.743,487,254+14.59
Exponential allocation49.73+0.823,489,580+14.66
Two-level age allocation49.38+0.473,304,604+8.59

Exponential allocation reaches 49.73% Step SR, improving by 0.82 points over uniform retention. Inverse-age and two-level allocation improve by 0.74 and 0.47 points. The corresponding replay counts show the computational cost of reallocating past screenshots as the trajectory evolves.

② History allocation and causal prefix reuse. Every policy has the same 5,034,856 total carrier tokens.

Historical policyReused prefix tokensPrefix reuse (%)Additional replay tokensDecisions with changed historical selection
Uniform fixed admission1,991,56839.560—
Inverse-age allocation1,547,60230.74443,9661,158
Exponential allocation1,545,27630.69446,2921,158
Two-level age allocation1,730,25234.37261,316607

Exponential allocation reduces prefix reuse from 39.56% to 30.69% and adds 446,292 replayed tokens. Two-level allocation retains 34.37% prefix reuse and adds 261,316 replayed tokens. This additional work arises because changing historical selections invalidates part of the causal suffix even though the total token budget is fixed. In the matched full-GPU-pool control above, prefix reuse reduces mean end-to-end time from 4.413 to 4.205 seconds. Replay overhead is therefore the direct computational comparison across the age policies.

AuthorsOpenReview ↗

Response to Reviewer 5mwG (Part 3/3)

Q4. NEO beyond adding a layout prior

What is the main methodological difference between NEO and existing relevance-and-diversity selection methods, beyond adding the layout prior?

① Ordering beyond the prior. NEO constructs a task- and layout-conditioned evidence order for irreversible admission when future GUI targets are not yet known. For a selected set S, it chooses the next native visual row by combining instruction relevance with conditional residual novelty. The layout prior scales candidate novelty while leaving every token eligible. Because residual novelty is recomputed as the selected set grows, each choice reflects task relevance, interaction support, and redundancy relative to the evidence already selected.

② Lifecycle integration. The resulting order supports the complete TRACE lifecycle. Tighter budgets use prefixes of the same order, NCR restores native spatial support outside the protected evidence prefix, and MKC encodes each arriving screenshot once while contracting retired frames into compact state. The language model reuses the unchanged causal prefix and recomputes the affected suffix. NEO determines what visual evidence should survive, and MKC makes that evidence reusable throughout the trajectory.

③ Matched evidence beyond prior access. Matched comparisons isolate the gains beyond prior access. At 5% retention on ScreenSpot-v2, the strongest tested same-prior baseline reaches 34.36%, while TRACE reaches 55.19%. On the shared MiniWeb MKC path, TRACE reaches 44.44% Task SR, compared with 42.00% for DivPrune plus LIP, which uses the same layout prior and budget, and 42.55% for Uniform. These results show that TRACE's evidence ordering and its integration with the reusable cache lifecycle add measurable value beyond the prior itself.

Reviewer_nWHKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TRACE, a training-free framework for reducing visual-token costs in GUI agents. It combines layout-derived interaction priors, relevance- and diversity-based token ordering, spatial coverage repair, and a cache mechanism that contracts historical frames while reusing visual features. Experiments across six benchmarks and two model families cover single-step grounding and offline multi-step action prediction. The results demonstrate improved accuracy under restrictive token budgets and reduced time to first token and visual KV storage relative to dense re-prefill.

Critical strengths
  1. Addresses an important practical problem. High-resolution screenshots and accumulated visual history make GUI-agent inference expensive. The paper studies token selection together with cache reuse, providing a useful perspective on preserving visual evidence for later interactions while controlling computation and storage.

  2. The selector design is supported by substantial component analysis. The interaction prior, diversity term, and coverage repair address different causes of information loss. Module ablations, prior-transfer experiments, and target-retention analyses support their complementary roles and show benefits beyond using the detector or an importance ranking alone.

  3. Strong results under restrictive token budgets. The evaluation covers six benchmarks, multiple model scales, and a second model family. On ScreenSpot-v2 with GUI-Owl-1.5-8B at 5% retention, TRACE achieves 55.19% accuracy versus 34.20% for the strongest reported baseline. Comparisons on the shared MKC path also support the selector’s accuracy advantage, while serving measurements demonstrate substantial reductions in TTFT and visual KV storage.

Critical weaknesses
  1. Trajectory robustness is not sufficiently tested in interactive tasks. The multi-step experiments measure action prediction on recorded trajectories. In an interactive task, an incorrect action can produce a different screen and affect subsequent decisions. The current evaluation therefore leaves unclear whether pruning errors accumulate or prevent recovery. The target-retention and cache analyses show which evidence survives, but do not establish its effect on complete-task success. Appendix C.1 also notes that the evaluated tasks rarely require information from much earlier screenshots. Interactive task evaluation, together with history ablations on tasks that require earlier observations, would better support the trajectory-robustness claim.

  2. It is unclear how the final parameter settings were selected. Appendix A.4 uses different settings for each benchmark and token budget, but does not clearly identify the data used to choose them. These choices can affect the comparison: on tight-budget AndroidControl, the selected configuration improves performance by 4.09 percentage points over the fixed configuration, while TRACE’s reported lead over the strongest baseline is only 1.30 points. The transfer protocol also needs clarification. Appendix C states that the GUI-Owl settings are transferred to UI-TARS without retuning, whereas Table 26 presents selected UI-TARS configurations. The authors should specify the data used for parameter selection and explain whether Table 26 reports a sensitivity check or actual configuration changes.

  3. Total memory requirements are not fully reported. The timing measurements account for detector latency, but the reported memory savings concern visual KV storage. Total peak GPU memory should also include the OmniParser detection branch, stored visual features, and temporary buffers. Reporting these costs alongside dense MKC and representative selectors would clarify the memory savings of the complete inference pipeline.

Questions for authors
  1. Can you report complete-task success in an interactive environment, particularly on tasks requiring earlier visual observations? Beyond the reported window-truncation comparison, can history ablations isolate the benefit of retained historical evidence?

  2. Which data were used to select the settings for each benchmark and budget? Were these choices made independently of the reported evaluation results? Please also clarify whether Table 26 is a sensitivity check of unchanged transferred settings or involves retuning for UI-TARS.

  3. What are the total peak GPU-memory requirements of dense MKC, TRACE, and representative selectors on the same caching path, including the detection branch, stored features, and temporary buffers?

3 comments
AuthorsOpenReview ↗

Response to Reviewer nWHK (Part 1/3)

We sincerely thank the reviewer for highlighting the need for complete-task and full-pipeline evidence. In response, we added closed-loop MiniWeb evaluation, controlled history studies, configuration-selection details, and a full-pipeline memory audit. TRACE reaches 44.44% Task SR on MiniWeb with 18.60% lower mean time than dense repeated encoding. We respectfully ask the reviewer to reconsider the score in light of this substantially strengthened evidence.

Q1. Complete-task success and the contribution of visual history

Can you report complete-task success in an interactive environment, particularly on tasks requiring earlier visual observations? Beyond the reported window-truncation comparison, can history ablations isolate the benefit of retained historical evidence?

TRACE achieves 44.44% complete-task success on MiniWeb. Its 369 human-written tasks span 65 local websites and 37 interaction primitives, with outcome-based verification. Because each action determines the next screenshot, MiniWeb tests compressed visual evidence over complete trajectories. All methods use GUI-Owl-1.5-8B-Instruct under the same action protocol, prompt family, and execution limits (“Back to Basics: A Skill-Centric Approach to Generalist Web Agents,” arXiv:2610.35823).

① MiniWeb task success and time. H denotes the number of preceding screenshots. H6 includes six historical screenshots plus the current screenshot. Dense repeated encoding re-encodes the visual window at each step. The other methods share the MKC path. Selectors retain 50% of the current frame and 10% of each historical frame. Mean elapsed time is reported over the observed task trajectories.

MethodWindowTask SR (%)Mean elapsed time per task trajectory (s)
Dense, repeated encodingH036.31112.97
Dense, repeated encodingH139.84123.20
Dense, repeated encodingH341.46138.52
Dense, repeated encodingH643.90152.69
Dense MKCH643.90143.10
DivPrune MKCH640.92123.56
DivPrune + LIP MKCH642.00128.98
Uniform MKCH642.55129.83
TRACE MKCH644.44124.29

Visual history raises dense repeated encoding from 36.31% at H0 to 43.90% at H6. At H6, TRACE achieves the highest Task SR of 44.44%, exceeding DivPrune, DivPrune + LIP, and Uniform by 3.52, 2.44, and 1.89 points. It also reduces mean task-trajectory time by 18.60% versus dense repeated encoding and 13.14% versus Dense MKC. DivPrune is only 0.73 s faster but has 3.52 points lower Task SR.

We next isolate historical evidence by fixing current-frame selection and varying only the preceding screenshots in recorded-action evaluations.

② Mind2Web Cross-Task history sweep. Each entry reports action-prediction accuracy in percent.

MethodH0H1H2H3H4H5H5 minus H0 (pp)
Dense48.1151.9453.2954.0853.6454.48+6.37
TRACE41.7841.5341.5841.7341.3342.63+0.85

Dense improves by 6.37 points from H0 to H5. TRACE improves by 0.85 points, with a paired interval that crosses zero. The clearer measured history effects appear on OmniGUI and MemGUI below.

③ OmniGUI and AndroidControl short-history controls. Both comparisons use the same decision context at H0 and H1.

Dataset / methodH0H1Gain (pp)
OmniGUI / Dense49.0752.57+3.50
OmniGUI / TRACE47.7449.53+1.79
AndroidControl / Dense57.3857.67+0.29
AndroidControl / TRACE52.5152.61+0.10

On OmniGUI, TRACE gains 1.79 points, with a paired trajectory-bootstrap 95% confidence interval from 0.77 to 2.82 points. AndroidControl shows a smaller H0-to-H1 difference of 0.10 points, demonstrating that the measured value of history depends on the task and trajectory.

④ MemGUI-3K evaluates historical evidence over long, screenshot-grounded mobile trajectories (“MemGUI-Agent: An End-to-End Long-Horizon Mobile GUI Agent with Proactive Context Management,” arXiv:2606.19926). The evaluated trajectories average 28.12 steps and reach 121 steps. With current-frame selection fixed, extending the history window isolates its contribution to recorded-action prediction.

MethodH0H5H10H10 minus H0 (pp)Paired 95% interval (pp)
Dense50.8856.0956.73+5.86[4.85, 6.86]
TRACE47.0149.6050.66+3.64[2.76, 4.51]

Across 5,875 paired decisions, TRACE gains 3.64 points from H0 to H10, or 214 additional correct predictions. H5 to H10 adds 1.06 points, with a paired 95% interval of 0.51–1.62. Counting overflow as incorrect across all 6,146 decisions, accuracy rises from 46.10% at H0 to 48.80% at H5. This complements MiniWeb's complete-task results with recorded-action evidence for retained history.

AuthorsOpenReview ↗

Response to Reviewer nWHK (Part 2/3)

Q2. Configuration selection and UI-TARS transfer

Which data were used to select the settings for each benchmark and budget? Were these choices made independently of the reported evaluation results? Please also clarify whether Table 26 reports unchanged transferred settings or actual UI-TARS configuration changes.

① Matched configuration selection. We use the same training-free token-pruning comparison protocol for all methods. Because no independent development split is available, benchmark-specific settings are selected from the evaluated configuration grid. Crucially, this selection is applied symmetrically: for TRACE and for each competing leading method, we report the method's best evaluated configuration under the same benchmark, retention budget, and evaluation protocol. This symmetric selection compares every method at its respective best evaluated configuration under matched conditions. Retention budgets are shared across selectors. Appendix A.4 illustrates the resulting configuration sensitivity: on AndroidControl at a 25% context budget, the selected configuration is 4.09 points better than the fixed reference configuration.

② Fixed transfer versus Oracle-style sweep. Table 25 evaluates transfer to UI-TARS using the GUI-Owl configuration together with UI-TARS's native prompt and action grammar. Table 26 is an Oracle-style configuration sweep for configuration sensitivity: it reports the best evaluated setting for each displayed benchmark and budget over the declared grid. Table 25 supplies the complementary fixed-configuration transfer result. Four of its six selected cells differ from the reference setting with a prior cap of two and a current-frame dose of 0.10: MMBench-GUI L2 at 25% and 50% retention, ScreenSpot-Pro at 50% retention, and ScreenSpot-v2 at 5% retention. Together, Table 25 measures fixed-configuration UI-TARS transfer, while Table 26 measures sensitivity to configuration selection.

③ AndroidControl margin. On AndroidControl, TRACE obtains 4,818 correct decisions out of 8,444, compared with 4,740 for Uniform. This is a 0.92-point margin based on unrounded counts. The 1.30-point margin is against ZOO.

AuthorsOpenReview ↗

Response to Reviewer nWHK (Part 3/3)

Q3. Complete GPU-memory requirements

What are the total peak GPU-memory requirements of dense MKC, TRACE, and representative selectors on the same caching path, including the detection branch, stored features, and temporary buffers?

On the shared MiniWeb caching path, TRACE reduces peak allocated GPU memory from 27.65 GiB for Dense MKC to 25.08 GiB. The measurement includes the active detection branch, stored features, active-generation state, selection temporaries, and other live buffers. Allocated memory counts live allocations, reserved memory counts the allocator pool, and the sampled device peak also includes device allocations outside that pool.

① MiniWeb full-path GPU memory. All rows use the H6 setting and the same MKC caching path.

Method, H6Peak allocated (GiB)Peak reserved (GiB)Sampled device peak (GiB)Persistent visual features (MiB)Committed cached KV peak (MiB)Detector weights / buffers (MiB)
Dense MKC27.6543.1943.70204.754,254.190.00
DivPrune MKC24.6230.7031.2132.253,180.800.00
DivPrune + LIP MKC25.5934.2434.7632.253,600.5638.21
Uniform MKC25.3231.8232.3332.253,470.910.00
TRACE MKC25.0832.7733.2932.253,402.8438.21

From the displayed values, TRACE reduces the peak allocated memory by 9.29%, persistent visual-feature storage by 84.25%, and committed cached KV by 20.01% relative to Dense MKC. Its peak reserved memory is 32.77 GiB, compared with 43.19 GiB for Dense MKC. The detector column reflects the loaded OmniParser icon-detection branch, while OCR and captioning are disabled. A repeated-encoding H6 run has a 23.69 GiB allocator peak under its different execution setting, so the matched MKC rows provide the relevant total-peak comparison.

The OmniGUI serving study separately measures the language process and detector worker, whose peaks occur at different times and in separate processes. We report one cache-specific quantity: Committed cached KV peak, the reusable session KV retained after admission. This directly captures the persistent cache cost relevant to MKC. For Dense repeated encoding, the dash indicates that no reusable committed cache is maintained on that path.

② OmniGUI serving memory breakdown. The detector-worker peak includes its activation and temporary allocation.

MethodLanguage-process peak allocated (GiB)Detector-worker peak allocated (MiB)Committed cached KV peak (MiB)Encoder output temporary peak (MiB)
Dense, repeated encoding22.27——591.38
Dense MKC24.19—2,911.7898.56
DivPrune MKC18.68—693.8498.56
VisPruner MKC19.48—693.8498.56
ZOOPrune MKC20.19—693.8498.56
TRACE MKC18.90296.33693.8498.56

Language-model weights and buffers occupy 16,721.96 MiB across these methods. The OmniGUI table uses committed cached KV as its single cache-specific quantity, while the MiniWeb same-process peak supplies the complete allocated-memory comparison that includes active detection and temporary allocations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.