acceptodds
Under review as a conference paper at ICLR 2027

GUIFlow: Temporal Redundancy-Aware Incremental Inference for GUI Agents

Abstract

GUI agents built on vision-language models interact with continuously evolving interfaces, yet conventional inference repeatedly recomputes representations of largely overlapping observations and histories. This mismatch makes multi-step interaction costly, while changes in context, interface content, and action arguments prevent straightforward cache reuse. We introduce GUIFlow, a framework for full-pipeline incremental inference in multi-image GUI interaction. Our central insight is that advancing a decision need not require rebuilding all supporting representations: each interaction step can instead selectively update computational states maintained across the trajectory. To support new decisions as the context evolves, GUIFlow adapts how past computation is reused: historical KV states are position-aligned and injected into the current context; change signals guide selective updates of current-observation states in the vision encoder and language model; and earlier action responses serve as speculative drafts verified by the target model. Together, these mechanisms enable cross-step reuse throughout visual encoding, prefill, and decoding without additional training, a separate draft model, or changes to the model-visible inputs and action interface. Experiments demonstrate up to geometric-mean speedup across offline multi-image benchmarks and up to in online, closed-loop interaction on OSWorld, while maintaining task performance broadly comparable to the dense baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI,Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jUYoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

GUIFlow accelerates GUI agents through historical KV reuse, selective visual and language-state updates, and speculative decoding from previous actions. Across AITW, Mind2Web, and OSWorld, it reports approximately 3.3–3.5× FLOPs reductions and 1.9–2.6× processing-latency speedups, with performance generally close to dense inference.

Critical strengths
  • The method offers a novel and coherent integration of incremental computation across visual encoding, prefill, and decoding. Source-aware tracking and representation-guided refresh address the challenge of reusing states under evolving contexts without additional training.

  • The approach is effective across multiple backbones and both offline and closed-loop evaluations. Reported processing-latency speedups of approximately 1.9–2.6× demonstrate practical benefits beyond FLOPs reductions.

  • The paper is well written and clearly organized. It explains the problem and motivation effectively, making the sources of temporal redundancy and the rationale for each component easy to follow.

Critical weaknesses
  • All three components of the method have close precedents: historical KV integration resembles CacheBlend (EuroSys 2025) in reusing cached states beyond unchanged prefixes; current-observation refinement resembles VLA-Cache (NeurIPS 2025) in selectively refreshing visual-token KV across successive observations; and trajectory-guided decoding follows the retrieval-based draft-and-verify principle of REST (NAACL 2024), using previous actions instead of an external datastore. While their integration and adaptation to GUI interaction are valuable, the paper could more clearly distinguish the new mechanisms from established techniques and isolate their benefits through targeted comparisons.

  • CodecSight (https://arxiv.org/pdf/2604.06036) uses codec metadata to guide visual computation reduction and selective KV refresh. Could motion vectors or residual signals provide cheaper or more informative change detection than pixel differences in GUIFlow, particularly during scrolling or window movement? A discussion of applicability and trade-offs would be useful.

Questions for authors
  1. What are the main technical advances beyond CacheBlend’s non-prefix KV reuse, VLA-Cache’s cross-frame selective refresh, and REST’s retrieval-based speculative decoding? Could targeted comparisons isolate the benefits of GUIFlow’s new mechanisms from those of their integration?

  2. CodecSight uses codec metadata to guide visual computation reduction and selective KV refresh. Could motion vectors or residual signals offer cheaper or more informative change detection than pixel differences, especially during scrolling or window movement?

  3. Tables 2–3 report 70.85 T FLOPs and 1.50 s per step, whereas Table 5 reports 224.76 T and 3.57 s at the default Lp=6,Mcur=3L_p=6, M_{\mathrm{cur}}=3. Please clarify which components are enabled in the sensitivity study; the shared parameter settings alone do not explain this discrepancy.

  4. In Table 4, adding speculative decoding increases strict success from 24.65% to 26.87%. Under identical inputs, prefill states, and numerical decisions, the greedy verification procedure should preserve ordinary decoding outputs. Please explain whether this difference comes from environment variability, numerical effects, or other experimental changes.

  5. Table A3 describes measurements as paired by “common task-step keys,” but the same step index can correspond to different observations after trajectories diverge. Please clarify whether these comparisons use identical replayed inputs or merely matching task and step indices.

  6. The abstract reports up to 3.49× speedup without explicitly identifying it as a FLOPs ratio. Since measured processing-latency speedups are 1.94–2.55×, the abstract should distinguish computational savings from wall-clock acceleration.

I will consider raise my score if questions are addressed.

5 comments

We thank the reviewer for the careful reading and constructive feedback.

W1 / Q1: Contributions beyond prior work and the value of coordinated pipeline acceleration

We thank the reviewer for the suggestions. We address targeted experiments, specific mechanism differences, and coordinated pipeline design below.

1. Targeted mechanism ablations validate the contributions of cache-source tracking and adaptive refresh.

We add two controls: one compares current representations only with the previous step instead of their actual cache sources; the other replaces residual-budget-driven adaptive refresh with a fixed refresh ratio.

ConfigurationSuccess rateDifference from full GUIFlow
Full GUIFlow28.89%—
Previous-step comparison only20.00%−8.89 pp
Fixed refresh ratio, ρ = 0.6021.11%−7.78 pp

Full GUIFlow achieves the highest success rate among these configurations, outperforming the two controls by 8.89 and 7.78 percentage points, respectively. These results demonstrate the value of tracking each token's actual cache source during successive reuse and adapting the refresh set to representation deviations. They isolate the contributions of specific mechanisms and complement the component ablation in Table 4.

2. We will distinguish established principles from GUIFlow's specific designs.

GUIFlow builds on non-prefix cache reuse, selective cross-frame updates, and draft verification. Its contributions address the specific challenges of applying these mechanisms to multi-image, sequential GUI interaction.

Prior workEstablished ideasGUIFlow's specific designs
CacheBlendNon-prefix KV reuse and selective recomputation guided by KV deviationsHistorical-image matching, multimodal positional alignment, and reuse-lifetime management for sliding multi-image contexts
VLA-CacheCross-frame caching, task-relevant token refresh, and layer-adaptive reuseComparing current representations with each token's actual cache source, then selecting refreshes using a residual-mass budget, pixel changes, and expiration rules
RESTRetrieved text drafts verified by the target modelOrganizing within-task historical drafts by GUI action type and exploiting structured action text for repeated verification and structural re-entry

3. Coordinated acceleration across the full pipeline is itself a substantive challenge.

A module that works effectively in isolation does not guarantee that task quality will be preserved when multiple modules are combined. From the perspective of information preservation and distortion control, approximations to visual representations and historical KV states affect downstream computation, and their errors can propagate through closed-loop interaction. Therefore, the most aggressive configurations selected independently for each module do not necessarily form the best overall configuration.

This challenge also motivates prior joint-optimization research: APQ jointly optimizes architecture, pruning, and quantization [1]; PRANCE jointly optimizes tokens and channels [2]; EfficientVLA and ElegantVLA accelerate VLA models through multi-module compression and dynamic compute scheduling, respectively [3] [4]. For multi-image GUI interaction, GUIFlow coordinates reuse across visual encoding, prefill, and decoding without additional training, controlling approximation errors across stages and interaction steps to expand the overall acceleration achievable while preserving task quality. We will include these contribution comparisons, mechanism ablations, and discussions in the full version.

References

[1] Wang et al. APQ: Joint Search for Network Architecture, Pruning and Quantization Policy. CVPR, 2020.

[2] Li et al. PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inference. IEEE TPAMI, 48(1): 283–298, 2026 (published online in 2025).

[3] Yang et al. EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models. NeurIPS, 2025.

[4] Li et al. ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models. arXiv:2605.29438, 2026.

W2 / Q2: Applicability of motion-vector and residual signals

We thank the reviewer for the suggestion. Inspired by CodecSight, we replace pixel-difference detection in GUIFlow's current-frame visual updates with motion-vector and residual signals, keeping the other components unchanged, and conduct additional experiments with Qwen3-VL-8B-Instruct on OSWorld. This experiment adapts its change-detection idea, with results shown below.

MethodStrict task success rateVLM FLOPs (T/request; speedup)Total processing latency (ms/request; speedup)Detection time (ms)Vision patch recomputation ratio
Dense23.55%247.007 (1.00×)3764.47 (1.00×)——
GUIFlow26.87%70.846 (3.49×)1502.69 (2.51×)0.4453.04%
CodecSight-inspired adaptation24.38%85.385 (2.89×)1832.50 (2.05×)123.8478.92%

1. Motion-vector and residual detection incurs higher overhead without delivering greater computational savings.

Detection time increases from 0.44 ms to 123.84 ms, while the vision patch recomputation ratio increases from 53.04% to 78.92%. Total processing latency consequently increases by approximately 21.9%, reducing the latency speedup from 2.51× to 2.05×; strict task success also decreases by 2.49 percentage points.

2. The value of these signals depends on the input source and task characteristics.

CodecSight targets compressed video streams and uses motion vectors and residuals available during decoding to guide computation. GUIFlow receives raw screenshots, so obtaining and processing these signals introduces additional overhead. Codec-block motion information must also be mapped to the model's patch-update requirements; small motion does not imply unchanged button states or text. In our GUI setting, pixel differences achieve a better quality–efficiency balance at a lower detection cost. We will discuss CodecSight in the related work and explain its differences from GUIFlow in input format and change-detection mechanisms.

Q3: Computational-cost differences between Tables 2–3 and Table 5

We thank the reviewer for pointing this out. Tables 2–3 evaluate full GUIFlow, whereas Table 5 is a single-component sensitivity study with only TokenLease enabled. The original caption did not explicitly list the component settings, creating ambiguity. We will clarify this in the full version.

The configurations are compared below:

Component or metricTables 2–3: Full GUIFlowTable 5: TokenLease sensitivity study
Historical screenshot KV reuseEnabledDisabled; historical context is recomputed
Sparse current-frame vision computationEnabledDisabled; full vision computation is performed
Current-frame LLM TokenLeaseEnabledEnabled
Trajectory-guided speculative decodingEnabledDisabled
FLOPs / step70.85 T224.76 T
Latency / step1.50 s3.57 s

Table 5 disables the other acceleration components to isolate the effects of TokenLease's probe depth and reuse limit on task quality and computational cost. Thus, the shared values of Lp=6L_p=6 and Mcur=3M_{\mathrm{cur}}=3 indicate identical TokenLease parameters, not identical configurations of the entire inference pipeline. Tables 2–3 additionally benefit from historical KV reuse, sparse vision computation, and speculative decoding, explaining their lower FLOPs and latency. We will make these settings explicit in Table 5's caption and the experimental description.

Q4: Speculative decoding and improved closed-loop task success

We thank the reviewer for pointing this out. Given identical inputs, prefill states, and numerical decisions, greedy draft verification preserves ordinary greedy decoding outputs. Table 4 reports task success from actual closed-loop interactions, where the improvement must be understood in the context of interaction timing and state reuse.

1. Faster inference provides denser feedback from intermediate states, improving action prediction.

The Qwen evaluation configuration captures the next screenshot directly after action execution, without an additional wait for the interface to stabilize, while application responses and interface updates in OSWorld take time. For example, when an action enters five words, the next screenshot can capture the interface when only three words have appeared. We directly observed that accelerated interaction captured more such intermediate states, giving the model finer-grained information about task progress. This feedback enabled more timely and accurate subsequent actions and improved closed-loop task success. Thus, decoding equivalence on fixed inputs is consistent with the improved task performance observed in this experiment.

2. Lightweight computation and stable-state reuse can support both efficiency and output quality.

GUIFlow retains stable representations while recomputing changed regions, maintaining continuity across interaction steps with characteristics similar to temporal filtering. Prior work also shows that pruning can improve generalization through regularization [1], while EAZY reduces erroneous outputs by removing a small set of hallucination-inducing visual tokens [2]. These findings show that reducing redundant representations can both lower computational cost and suppress interference, providing a mechanistic explanation for maintaining or improving task performance under acceleration.

We will add the interaction timing, observed behavior, and related discussion to the full version, clarifying how reduced decision latency and denser environment feedback improve task performance in this closed-loop setting.

References

[1] Jin et al. Pruning's Effect on Generalization Through the Lens of Training and Regularization. NeurIPS, 2022.

[2] Che et al. Hallucinatory Image Tokens: A Training-free EAZY Approach to Detecting and Mitigating Object Hallucinations in LVLMs. ICCV, 2025.

Q5: Comparison protocol in Table A3

We thank the reviewer for pointing this out. We run Dense and GUIFlow independently on the same task set and measure model computation and latency along their actual closed-loop trajectories. We do not replay a fixed trajectory or perform step-by-step comparisons on identical inputs. The action and observation sequences can differ, so these results report model processing efficiency along actual closed-loop trajectories.

The lease acceptance and image-token reuse rates in Table A3 are computed directly from the internal records of each run. FLOPs reductions and latency speedups compare the actual execution costs on the corresponding tasks. We will revise the caption in the full version to explicitly describe this as an independent closed-loop comparison on the same tasks, avoiding the implication that "paired" denotes step-by-step timing on identical inputs.

Q6: Explicit identification of the speedup metric in the abstract

We thank the reviewer for pointing this out. The headline 3.49× in the abstract refers to a FLOPs speedup. Table 3 already reports FLOPs and measured latency speedups separately; the measured processing-latency speedups are 1.94–2.55×, excluding environment execution.

We will explicitly label the FLOPs speedup and report the measured latency gains in the abstract and main text of the full version, clearly distinguishing computational savings, model-side latency speedups, and end-to-end runtime speedups. We also provide the timing boundaries and a complete runtime breakdown in our response to Reviewer 2's W2.

Reviewer_iFpoOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents GUIFlow, a training-free framework that accelerates vision-language GUI agents by reusing computation across successive interaction steps. It targets redundancy in screenshot histories, persistent screen regions, and recurring action-response patterns. GUIFlow reuses historical screenshot KV states with positional alignment, selectively updates current-screen representations using pixel changes and early-layer representation differences, and uses previous action responses as speculative drafts verified by the target model. Evaluations on AITW, Mind2Web, and online OSWorld report FLOPs reductions of 3.29-3.49x and processing-latency speedups of 1.94-2.55x, alongside measurements of task performance.

Critical strengths
  • GUIFlow is a novel approach. It exploits three recurring features of GUI interaction: overlapping screenshot histories, small changes between screens, and repetitive action syntax.

  • Experiments cover AITW, Mind2Web, and OSWorld using both general-purpose and GUI-specialized models. The improvement over baselines looks promising.

Critical weaknesses
  • There is inconsistency in result reporting. For example, Table 1 includes baseline methods like STAR-KV, GUI-KV, which are absent from Table 2. Additionally, Table 1 uses Qwen3-VL-4B and UI-TARS-1.5 but Table uses Qwen3-VL-8B and GUI-Owl-1.5. The paper doesn’ t provide justification for either.

  • There is incomplete evidence for the end-to-end runtime gains. The headline 3.49x is a FLOPs ratio, while the corresponding measured latency speedup is 2.51x. Aggregate latency excludes environment execution, and online comparisons involve different trajectories. The paper should provide a more detailed runtime profiling, such as benchmark-wide task-completion times and controlled timing comparisons. This would better establish the practical speedup.

  • Statistical significance may be questionable on OSWorld. OSWorld is an interactive and longer-horizon benchmark and due to the size of this benchmark the variance across runs can be high. The reported numbers would be more solid if the authors can report mean scores and standard deviation across.

  • The paper specifies probe depth and current-token reuse limits, but does not provide concrete defaults for several consequential choices: the historical reuse limit residual-mass budget, pixel-change thresholds, and vision refresh limits. Draft re-entry rules are also underspecified.

Questions for authors

Please see weakness.

4 comments

We thank the reviewer for the careful reading and constructive feedback.

W1: Baseline coverage and model selection across offline and online evaluations

We thank the reviewer for pointing this out. The original manuscript did not adequately explain our baseline coverage and model selection. We clarify both below and will include the additional experiments in the full version.

1. Some baseline combinations were omitted because our reproductions showed substantial quality degradation or little computational benefit.

For example, STaR-KV reduces the mean score of Qwen3-VL-8B from 24.30% to 12.39%, with a FLOPs speedup of only 0.99×. Its compression requires layer-wise indexing and greedy selection, whose overhead exceeds the inference time saved. On GUI-Owl, GUI-KV reduces the mean score from 48.79% to 44.32%, with a FLOPs speedup of approximately 1.00×. These results are nevertheless informative about the methods' applicability, and we will include them in the full version. The expanded comparison is shown below; speedups are Dense/method mean FLOPs ratios.

ModelMethodMean Score (%) ↑Strict SR (%) ↑FLOPs (T) ↓Speedup (×) ↑
Qwen3-VL-8B-InstructDense24.3023.55247.011.00
FastV23.1622.16165.501.49
VisionZip21.2220.22161.481.53
GUI-KV22.8121.61246.541.00
STaR-KV12.3911.63249.780.99
HistPrune20.8219.94194.521.27
DivPrune22.9322.71185.891.33
GUIFlow27.9026.8770.853.49
GUI-Owl-1.5-8B-InstructDense48.7947.09278.121.00
FastV45.8344.32187.361.48
VisionZip44.3642.38183.181.52
GUI-KV44.3242.11277.531.00
STaR-KV30.2828.81274.791.01
HistPrune45.7443.77219.071.27
DivPrune41.7940.17182.841.52
GUIFlow46.3044.6083.813.32

2. Different models broaden our evaluation coverage while accounting for their capabilities in each setting.

Offline evaluation uses Qwen3-VL-4B and UI-TARS-1.5-7B, whereas online evaluation uses Qwen3-VL-8B and GUI-Owl-1.5-8B. This tests GUIFlow across different model sizes, architectures, and training backgrounds. For closed-loop evaluation, we additionally select GUI-Owl-1.5, an open-source policy trained with multi-platform environment reinforcement learning, which achieves 47.09% strict success under our Dense configuration. Including this stronger GUI-specialized policy alongside a general-purpose model allows us to assess acceleration and task-quality preservation across a broader range of baseline capabilities. All acceleration results are compared against Dense inference using the same underlying model.

W2: Measured latency gains and runtime accounting

We thank the reviewer for the suggestion. We clarify the acceleration metrics and provide a breakdown of the complete execution pipeline below.

1. We will explicitly identify the acceleration metric in the abstract and main text.

The headline 3.49× refers to a FLOPs speedup, which we will label explicitly. FLOPs measure arithmetic computation, whereas model-side latency also reflects memory access, execution scheduling, and cache management. End-to-end runtime additionally includes communication, action execution, environment waits, and observation collection, so it cannot be inferred directly from FLOPs speedup.

The original manuscript already specifies the timing boundaries: Table 3 (p. 8) includes per-step processing and inference but excludes environment execution; Table 4 (p. 9) likewise reports cumulative server processing time excluding environment execution. To distinguish model optimization gains from deployment-environment effects, we additionally report model-call latency and complete runtime by stage.

2. GUIFlow translates model-side acceleration into time savings across the complete execution pipeline.

The additional measurements show that cumulative model-call time decreases from 104.39 s to 41.80 s, a 2.50× speedup. Environment interaction accounts for 56.9% and 76.9% of the online interaction time for Dense and GUIFlow, respectively, showing that simulator action execution, waits, and observation collection substantially influence overall acceleration. Even including these costs, GUIFlow achieves a 1.28× online interaction speedup. Including initialization and evaluation cleanup, complete runtime decreases from 409.00 s to 349.45 s, saving 59.55 s (14.6%). We will include the following breakdown in the full version to present both model-side and complete-pipeline gains.

The following measurements use GUI-Owl-1.5-8B-Instruct on OSWorld. All times are in seconds; speedups are relative to Dense.

StageTimed operationDenseGUIFlow
Task preparationEnvironment reset, task configuration, and other initialization71.4066.74
Environment readiness wait60.0060.00
Task preparation subtotalInitialization and environment readiness131.40126.74
Online interactionCumulative agent prediction-call time104.3941.80 (2.50×)
Action execution, post-action waits, and subsequent observation collection142.05150.19
Initial observation collection, recording startup, and interaction-loop management3.273.29
Online interaction subtotalWall-clock time for closed-loop interaction249.71195.28 (1.28×)
Evaluation and cleanupPost-interaction environment stabilization wait20.0020.00
Scoring preparation, task evaluation, result saving, and recording finalization7.907.43
Evaluation and cleanup subtotalPost-interaction waits, evaluation, and cleanup27.9027.43
Complete runtimeTask preparation + online interaction + evaluation and cleanup409.00349.45 (1.17×)

W3: Statistical stability of OSWorld evaluation

We thank the reviewer for the suggestion. We conducted five additional closed-loop evaluations of GUI-Owl-1.5-8B-Instruct on OSWorld under identical configurations and report means and 95% confidence intervals below. These results also address Reviewer 1's Q2.

Evaluation cohortMean score: Dense / GUIFlow (%)Score difference (pp)Requests/task: DenseRequests/task: GUIFlowRequest differenceTotal latency/request speedup (×; 95% CI)FLOPs/request speedupTask FLOPs speedup
Original evaluation48.79 / 46.30−2.4922.9724.61+1.642.55×3.32×3.10×
Additional evaluations48.16 / 46.61−1.55 [−5.62, +2.52]24.39 [22.51, 26.21]25.92 [24.10, 27.74]+1.53 [+0.27, +2.80]2.45× [2.39, 2.50]3.35× [3.31, 3.39]3.16× [2.99, 3.34]

Brackets indicate 95% confidence intervals. The five additional evaluations demonstrate stable acceleration, with 2.45× per-request total processing latency speedup and 3.16× task-level FLOPs speedup.

W4: Default parameters and draft verification rules

We thank the reviewer for the suggestion. Table 5 already reports sensitivity to probe depth and the current-token reuse limit, while Table A3 provides the corresponding reuse statistics. We consolidate the default configuration below and will include these parameters and the draft verification details in the full version.

1. Default configuration

ParameterHistorical reuse limit MhistM_{\mathrm{hist}}Probe depth LpL_pCurrent-token reuse limit McurM_{\mathrm{cur}}Residual-mass budget ε\varepsilonTokenLease selection granularityVisual patch difference thresholdMaximum change fraction for sparse vision updatesMaximum consecutive sparse vision updatesTrajectory draft verificationMinimum re-entry candidate lengthDecoding temperatureDrafts retained per action type
Default2630.01751×1 token0.0530%3Enabled3 tokens0 (greedy)Most recent 1

2. Draft verification and structural re-entry

GUI action outputs are structured text with repeated action names, field names, and formatting. We use the previous round's output as a draft for the current round and keep the latest draft for each action type. The target model checks multiple draft tokens in one forward pass, using its own next-token predictions under the current context. Matching tokens are accepted up to the first mismatch; the remaining draft states are discarded, and the model supplies the next token.

When generation reaches another repeated part of the action format, the model can check the draft again if its next-token prediction matches the draft at that point and at least 3 draft tokens remain. This lets us repeatedly verify and accept several tokens at once, reducing the number of model forward passes during decoding. We will add these details to the full version.

Dear Reviewer,

Thank you again for your time and thoughtful feedback. As the discussion period draws to a close, we would be very grateful if you could kindly consider our responses and additional results, should your schedule permit. We would be happy to address any remaining questions.

Best regards,

The Authors

Reviewer_pkxoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GUIFlow proposes a training-free framework for accelerating screenshot-based GUI agents through temporal reuse. It combines position-aligned historical KV reuse, sparse updates to visual representations, a shallow-layer probe that selects current-image tokens for deeper recomputation, and speculative decoding using historical action responses. The framework is evaluated on AITW, screenshot-based Mind2Web, and closed-loop OSWorld with general-purpose and GUI-specialized models. The reported FLOPs speedups are approximately 3.3–3.5×, while measured per-step processing latency improves by 1.94–2.55×. Task performance remains close to dense inference in the offline evaluations, but the online quality–efficiency trade-off varies across models and applications.

Critical strengths
  1. The framework addresses repeated computation across the full inference pipeline. Historical context reuse, current-observation updates, and action decoding are treated together without retraining the underlying policy. This makes the approach relevant to existing screenshot-based agents.
  2. The evaluation includes meaningful runtime measurements and closed-loop interaction. Beyond FLOPs reductions, Table 3 reports 1.94–2.55× lower processing latency, and OSWorld tests whether the accelerated policy can complete interactive tasks. These are useful results that go beyond isolated token-reduction measurements.
Critical weaknesses

1.The method is not sufficiently general. If the focus is on accelerating long-horizon interactions in document-based GUIs, why not use code to convert the interface into text and use that as input? Most browser interfaces can be efficiently converted into structured text, which would be more efficient and precise. In GUI scenarios such as games or simulation environments, most regions of the image change. When the camera rotates, these changes become particularly frequent, and the method would fail in such cases.

2.The method introduces lossy acceleration. In contrast, StreamingVLM combined with vLLM can already achieve 5–10× lossless acceleration. Compared with this more general, lossless alternative, the proposed method lacks generality and is unlikely to be adopted in real-world applications.

  1. Additional evaluation on GUI-RL models is needed. I believe the method should also be evaluated on models such as GUI-R1 and UI-R1 to better demonstrate its generalization.

  2. The related work section is not sufficiently thorough. There is a substantial body of work on efficient VLMs. Relevant studies covered in Towards Efficient Multimodal Large Language Models: A Survey on Token Compression could be included.

Questions for authors

1.What inference engine, attention kernels, and caching mechanisms are used for Dense and GUIFlow? Can you provide a comparison against optimized dense inference, such as vLLM with applicable exact-prefix caching, under matched hardware, precision, inputs, and generation settings? Reporting GUIFlow’s additional benefit on the same backend would materially clarify its practical contribution.

2.Are the GUI-Owl score decreases and increased request counts consistent across repeated closed-loop evaluations? Please report uncertainty for the paired quality differences and task-level runtime, particularly for the application categories with larger regressions.

3.Can you stratify the existing evaluation by screenshot change magnitude and trajectory length, reporting reuse rates, full-recomputation frequency, latency, and task success? In particular, where does the method cease to provide a runtime advantage, and how much overhead remains when reuse opportunities are limited?

  1. For browser tasks where DOM or accessibility information is available, what motivates the screenshot-only deployment setting? A comparison with a structured-text or hybrid agent would help establish where visual-state reuse provides the most practical value. Such a comparison should account for differences in observation access and task success.

5.Could you apply the same fixed configuration to additional RL-trained policies such as GUI-R1 or UI-R1? Beyond the existing UI-TARS and GUI-Owl results, this would test whether different reasoning lengths and action-generation patterns change cache reuse, draft acceptance, and the quality–latency trade-off.

4 comments

We thank the reviewer for the careful reading and constructive feedback.

W1 / Q4: Research scope, the value of screenshot-based interaction, and applicability to highly dynamic scenes

We thank the reviewer for the suggestions. We address the research scope and adaptive reuse mechanisms below.

1. GUIFlow aims to accelerate inference for existing GUI agents.

We reduce redundant computation across interaction steps while keeping model-visible inputs and the action interface unchanged. Converting interfaces into structured text entails redesigning the observation interface and agent policy, which is beyond this paper's scope. We agree that structured inputs can be more efficient in some settings. However, security, privacy, or access-control requirements may prevent applications from exposing internal data or structured interfaces to third-party agents. Where screen interaction is authorized by the user, screenshot-based GUI agents and their acceleration remain practically valuable. Comparisons across observation paradigms should also account for differences in information access and task success.

2. GUIFlow dynamically adjusts computation to the available reuse opportunities.

Pixel changes and representation deviations trigger recomputation, with visual encoding reverting to full computation when changes are extensive. Historical KV states are recomputed when reuse conditions are not met, and the target model generates tokens individually when action drafts do not match. Thus, substantial changes reduce acceleration opportunities rather than render the method inapplicable. Adaptive recomputation and target-model verification control reuse errors, while the actual speedup depends on the scene's temporal redundancy.

The following results, obtained with GUI-Owl-1.5-8B-Instruct on OSWorld, show FLOPs speedups with comparable accuracy across different image-change rates, including rates above 80%. These results demonstrate that GUIFlow remains applicable and provides FLOPs speedups across all evaluated image-change rates.

Image-change rateFLOPs speedupDense accuracyGUIFlow accuracy
30%3.47×54.17%56.25%
50%2.22×33.33%33.33%
80%2.20×33.33%33.33%

W2 / Q1: Comparison with optimized inference and StreamingVLM

We thank the reviewer for the suggestions. All methods in this comparison use Qwen3-VL-8B-Instruct on OSWorld and are evaluated on NVIDIA RTX 4090 GPUs, with all other experimental conditions following those in the main paper. All latency speedups in the table are measured relative to HF Dense.

MethodSuccess rate ↑Latency speedup ↑
Dense (HF)24.3%1.00×
Dense + vLLM26.1%1.89×
StreamingVLM + vLLM22.4%2.00×
GUIFlow + vLLM27.6%3.91×

1. GUIFlow requires dedicated adaptation to the inference engine.

Both Dense and GUIFlow in the original paper use the Hugging Face Transformers inference backend built on PyTorch. GUIFlow requires positional-encoding corrections for historical image KV states and dynamic recomputation and reuse of current-image tokens, so it cannot be deployed simply by switching to vLLM. After adaptation, GUIFlow achieves a 3.91× latency speedup over HF Dense. Compared with Dense + vLLM, it reduces per-request latency from 1841 ms to 890 ms, providing a further 2.07× speedup, with a success rate of 27.6%, compared with 26.1% for Dense + vLLM.

2. StreamingVLM is not strictly lossless relative to full-context inference.

It evicts historical tokens outside its retained windows, reducing directly accessible history, and uses dedicated fine-tuning to adapt to streaming inference (StreamingVLM, Sec. 2). In our GUI evaluation, StreamingVLM + vLLM achieves 22.4% success, whereas GUIFlow + vLLM reaches 27.6% with a greater latency speedup.

W3 / Q5: RL-trained models and cross-model applicability

We thank the reviewer for the suggestions. We clarify the existing RL coverage and report additional model evaluations below.

1. Both the original and additional evaluations cover RL-trained models.

UI-TARS-1.5 uses reinforcement learning to enhance reasoning, while GUI-Owl-1.5 uses multi-platform environment RL (MRPO). We additionally evaluate Youtu-VL-4B-Instruct and EvoCUA-8B, both of which include RL-based post-training: Youtu-VL uses multi-stage RL, and EvoCUA learns from interaction experience through step-level preference optimization (Youtu-VL, Sec. 4.2; EvoCUA, Sec. 5.3).

2. The additional models match our multi-image interaction setting.

The public inference and evaluation setups of GUI-R1 and UI-R1 use a single current screenshot as visual input, so they do not directly exercise the historical-image KV reuse studied in this paper (GUI-R1 implementation; UI-R1 evaluation setup). We therefore select two other RL-trained models suitable for multi-image, multi-step interaction. As shown below, GUIFlow achieves 2.708× and 1.695× latency speedups on Youtu-VL-4B-Instruct and EvoCUA-8B, respectively, further demonstrating cross-model applicability. EvoCUA's explicit thinking process adds decoding time, resulting in a smaller latency speedup than its FLOPs speedup. We will include these experiments and the models' RL training backgrounds in the full version, and discuss GUI-R1 and UI-R1 in the related work.

ModelMethodMean Score (%) ↑Latency (s/step) ↓Latency speedup ↑FLOPs (T/step) ↓FLOPs speedup ↑
Youtu-VL-4B-InstructDense18.846.7321.000×205.4011.000×
GUIFlow17.742.4862.708×66.6593.081×
EvoCUA-8BDense39.975.7591.000×281.7711.000×
FastV39.484.7931.202×189.0511.490×
VisionZip38.625.0211.147×184.5131.527×
GUI-KV44.235.8580.983×281.4871.001×
STaR-KV31.205.7201.007×283.1510.995×
GUIFlow40.653.3971.695×93.8493.002×

Q2: Quality and acceleration stability across repeated closed-loop evaluations

We thank the reviewer for the suggestion. We conducted five additional closed-loop evaluations under identical configurations. Each evaluation takes approximately 24 hours, with a maximum of 50 interaction steps per task. The results are shown below; brackets indicate 95% confidence intervals.

Evaluation cohortMean score: Dense / GUIFlow (%)Score difference (pp)Requests/task: DenseRequests/task: GUIFlowRequest differenceTotal latency/request speedup (×; 95% CI)FLOPs/request speedupTask FLOPs speedup
Original evaluation48.79 / 46.30−2.4922.9724.61+1.642.55×3.32×3.10×
Additional evaluations48.16 / 46.61−1.55 [−5.62, +2.52]24.39 [22.51, 26.21]25.92 [24.10, 27.74]+1.53 [+0.27, +2.80]2.45× [2.39, 2.50]3.35× [3.31, 3.39]3.16× [2.99, 3.34]

The five additional evaluations demonstrate stable acceleration, with 2.45× per-request total processing latency speedup and 3.16× task-level FLOPs speedup.

Q3: Performance across screenshot change magnitudes and trajectory lengths

We thank the reviewer for the suggestion. We stratify the existing evaluation of GUI-Owl-1.5-8B-Instruct on OSWorld by task-average screenshot change rate and Dense reference trajectory length, reporting success rates, latency speedups, and FLOPs ratios (Dense/GUIFlow).

Screenshot change rate (task average)Dense success rateGUIFlow success rateSuccess-rate changeLatency speedupFLOPs ratio
Small: 0–5%25.00%25.00%0.00 pp2.36×3.44×
Medium: >5% to 30%43.58%46.33%+2.75 pp2.10×3.28×
Large: >30%53.06%57.14%+4.08 pp2.31×3.23×
Dense reference trajectory lengthDense success rateGUIFlow success rateSuccess-rate changeLatency speedupFLOPs ratio
Short: 1–10 steps71.43%70.71%−0.71 pp1.24×3.02×
Medium: 11–25 steps42.86%41.67%−1.19 pp1.39×3.12×
Long: 26–49 steps22.58%19.35%−3.23 pp1.81×3.08×
Reference trajectory reaches the 50-step limit0.99%5.94%+4.95 pp3.09×3.39×

GUIFlow achieves 2.10–2.36× latency speedups across screenshot-change groups and 1.24–3.09× across trajectory-length groups. Even the largest-change group and the shortest-trajectory group retain 2.31× and 1.24× speedups, respectively. GUIFlow achieves an average latency speedup of at least 1.24× in every evaluated group.

W4: Related work

We thank the reviewer for the suggestion. We will expand the related work section in the revised manuscript to include the suggested survey, Towards Efficient Multimodal Large Language Models: A Survey on Token Compression, and relevant studies discussed therein.

Dear Reviewer,

Thank you again for your time and thoughtful feedback. As the discussion period draws to a close, we would be very grateful if you could kindly consider our responses and additional results, should your schedule permit. We would be happy to address any remaining questions.

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for the time and effort devoted to evaluating our work, and for their contributions to scholarly exchange, peer review, and the research community. To aid the final assessment, we summarize our paper's core contributions and the main experimental evidence added during the discussion.


Contributions

We introduce GUIFlow to exploit the distinctive characteristics of GUI agents: successive observations, overlapping historical contexts, and structurally repetitive action outputs. To our knowledge, GUIFlow is the first training-free framework for full-pipeline incremental inference in dynamic, multi-image GUI interaction, spanning visual encoding, prefill, and decoding. It organizes successive decisions as selective updates to existing computational states, offering a new perspective on inference acceleration for GUI agents. Specifically, GUIFlow reuses historical-image KV states through positional alignment as the context window advances, adaptively refreshes current-observation states through cache-source tracking and a residual-mass budget, and exploits structured action text through multi-token draft verification and structural re-entry. Together, these mechanisms reduce redundant computation across inference stages without additional training, a separate draft model, or changes to model-visible inputs and the action interface. Across multiple models and both offline navigation and online closed-loop tasks, GUIFlow achieves up to 3.49× FLOPs speedup and 1.94–2.55× processing-latency speedups, while maintaining task performance broadly comparable to Dense.

Key concerns and revisions

We added the following experiments and analyses to address the reviewers' key questions and further validate GUIFlow's design, practical acceleration, and applicability:

  • Comparison with existing methods and validation of specific designs.
    (Reviewer pkxo: W2 / Q1; Reviewer iFpo: W1; Reviewer jUYo: W1 / Q1)

    We expanded comparisons to include GUI-KV, STaR-KV, and StreamingVLM, and adapted GUIFlow to vLLM. GUIFlow provides a further 2.07× latency speedup over Dense + vLLM, reaching a 3.91× latency speedup relative to the original HF Dense baseline, demonstrating that its algorithmic benefits compound with inference-engine optimizations.

    In additional mechanism ablations on 90 tasks, full GUIFlow improves success rates by 8.89 and 7.78 percentage points over previous-step-only comparison and fixed-ratio refresh, respectively, directly validating the roles of actual cache-source tracking and adaptive refresh.

  • Inference acceleration and task-time savings in actual closed-loop execution.
    (Reviewer iFpo: W2)

    In the GUI-Owl evaluation on OSWorld, GUIFlow achieves a 2.50× model-side speedup and reduces cumulative model-call time by 60.0%, including all model requests along the actual trajectories. The complete runtime breakdown is:

    Dense: 104.39 s for model calls and 304.61 s for environment initialization, interaction, evaluation, and other non-model overhead, totaling 409.00 s.

    GUIFlow: 41.80 s for model calls and 307.65 s for environment initialization, interaction, evaluation, and other non-model overhead, totaling 349.45 s.

    With Dense's non-model overhead held fixed, eliminating its model-call time entirely would yield a complete-runtime speedup of at most 1.34×. In this configuration, GUIFlow saves 59.55 s, translating substantial inference acceleration into nearly one minute of savings across the complete execution pipeline.

  • Stable acceleration and quality–efficiency performance across repeated evaluations.
    (Reviewer pkxo: Q2; Reviewer iFpo: W3)

    We conducted five additional closed-loop evaluations and reported confidence intervals for the relevant metrics. GUIFlow achieves a 2.45× per-request processing-latency speedup (95% CI: [2.39, 2.50]) and, even accounting for the additional requests, a 3.16× task-level FLOPs speedup (95% CI: [2.99, 3.34]). The corresponding mean score difference is −1.55 percentage points (95% CI: [−5.62, +2.52]). Together, these results demonstrate reproducible computational and time savings alongside a modest observed mean score change.

  • Applicability across models and interaction conditions.
    (Reviewer pkxo: W3 / Q5 and Q3; Reviewer iFpo: W1)

    We added evaluations on Youtu-VL-4B-Instruct and EvoCUA-8B, two models with RL-based post-training, achieving 2.708× and 1.695× latency speedups, respectively. EvoCUA's explicit thinking process adds decoding overhead, yielding a smaller latency speedup than its FLOPs speedup while retaining substantial measured latency gains. Stratification by screenshot-change rate and trajectory length further shows that every evaluated group retains an average latency speedup, supporting GUIFlow's effectiveness across models, degrees of interface change, and task lengths.

Together, the mechanism ablations, baseline comparisons, repeated evaluations, and cross-model experiments demonstrate that GUIFlow systematically converts temporal redundancy in GUI interaction into substantial, reproducible inference gains and complements optimized inference engines, providing an efficient and practical framework for full-pipeline incremental acceleration of existing GUI agents.


We sincerely thank the Area Chair and reviewers again for their dedicated efforts and contributions to the research community. We value the opportunity to strengthen our work through this discussion and appreciate your consideration of the paper and the additional evidence.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.