acceptodds
Under review as a conference paper at ICLR 2027

FlashGaze: Training-Free Multi-Scale Patch Pruning for Efficient Video Understanding

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated strong performance in video understanding, yet efficiently processing long, high-resolution videos remains challenging. Such videos often contain substantial spatiotemporal redundancy, and processing redundant visual tokens can incur avoidable computational overhead. Many existing methods prune visual tokens during or after vision transformer (ViT) encoding, leaving much of the encoding cost unaddressed. Some approaches prune patches before encoding but rely on learned auxiliary networks for patch selection, incurring additional training and inference overhead. To address these limitations, we propose FlashGaze, a training-free method that reduces spatiotemporal redundancy before ViT encoding without introducing auxiliary networks. FlashGaze uses pixel-space differences as a proxy for information loss and employs Quadtree Dynamic Programming to jointly optimize patch dropping, merging, and keeping under a fixed budget. Experiments on two MLLM backbones across multiple benchmarks demonstrate substantial efficiency gains while largely preserving accuracy. On Qwen3-VL-8B, FlashGaze retains 98% of the full-input baseline accuracy on LongVideoBench while achieving up to 5.4 and 17 speedups in ViT encoding and MLLM prefill, respectively, and reducing peak GPU memory usage by a factor of 1.8. These efficiency gains enable the model to process videos with more frames and higher resolutions on the same GPU hardware, unlocking video understanding at scales previously out of reach.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Video Understanding, Visual Token Pruning, Efficient Inference, Long Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QfnBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a method for spatial patch selection for video MLLMs, using error of an accumulated "memory" buffer (last selected patches at each location) and the current frame. A simple quadtree DP algorithm and pixel-space error measures are used to select, merge or drop patches at each frame based on trading off their error reduction vs number of selected patches. Notably, this involves only simple tuning and calibration training, with no significant extra neural-net-based model used for selection. Measurements on LongVideoBench, VideoMMMU, and HLVid with Qwen and NVILA show good improvements over baseline systems, even as the algorithm is simpler.

Critical strengths
  • Simple, well explained method. Uses reconstruction in pixel space (with local alignment) for patch selection, and quadtree DP merge to find best cost.

  • Improved performance on video benchmarks, even without needing an additional trained model

  • Ablations on retention ratios vs acc, reconstruction error metric, and resolution vs latency show good profiles of system behavior

Critical weaknesses
  • Some additional method details could be better explained or are currently missing --- see below; in particular:

    • (a) how lambda is selected (there must be some sort of calibration/tuning, that I don't see described)
    • (b) the format of M_t
    • (c) how mid and coarse patches are input to the vit
  • All retention ratios and speedups are given as averages. It would also be good to see the distribution of retained patches for different videos, and if there anyvideos that this method fails for, e.g. large or fast camera motions, scene cuts, busy object movements

  • More analysis on the cost vs downstream accuracy would be good. Accuracy is profiled in terms of retention ratio, but not related to the cost (pixel-space error), both on average and for different videos.

  • Similarly, while accuracy is measured vs retention ratio, there is less analysis on how to increase resolution (spatial and/or temporal) and how this relates to accuracy, using the patches selected by this method. Only one line in table 2 seems to relate to this.

Questions for authors
  • Fig 1 when to prune: This is a good figure and example, but the same figure comparing pre-vit to other placements is also the first figure in autogaze (Shi et al 2026), with similar 12x total reduction arrow in the pre-vit row. Yet it appears like it is showing a novel observation. Instead it probably should add attribution in the caption and/or include more methods as examples for each type.

  • how is M_t stored and updated in the implementation? if local alignment were not used, it appears it is just the selected patches written into a maintained frame buffer at each t. But with local alignment costs, there are patches that overlap in their matched alignment --- how are they stored? Or are the patches always stored at the grid-aligned locations in a frame buffer and just compared allowing local search, for whether to update?

  • For a merged patch: What happens to M_t (and the recon costs) in merge cases? In particular, what happens if M_t contains a merged patch and then a fine-scale patch in one of its locations is kept at t+1 instead? And similarly the other way around: a fine-scale patch is in M_t and then merge is chosen at t+1? If using a framebuffer as described in my point above, would these simply overwrite their patch locations in the M_t+1?

  • min displacement search l.224: What is the size of the local window search, and how large is it relative to the patch size?

  • same for merge: is there a local search (over possibly larger window, at coarser stride) to get coarser level distance?

  • l.288 "selected patches are encoded by the ViT" --- how does the vit run on scales coarser than the finest? I'm assuming the fine scale corresponds to the vit stem stride, in which case the input for finest would be the subsampled stem features. But for mid and coarse level, what is used for the vit?

  • similarly, is the scale encoded in the input to vit and/or mllm for each patch? in particular, if m and c patches are downsampled in pixel space and fed to the vit as smaller rgb patches with pos encoding for their center, they would be indistinguishable from fine-scale patches with the same content at the same location. so presumably the scale is encoded somewhere, or somehow this doesn't matter. either way an experimental ablation on the scale encoding would be interesting.

  • what is the FPS used in the experiments? for example l.314 says "For LongVideoBench and VideoMMMU, the maximum number of sampled frames is set to 768." But I'm not sure if this is the pre-flashgaze rate, or somehow corresponds to something after the sampling algorithm? And not sure what frame rate it's starting from

  • table 1: "vanilla" qwen and nvila are not here, they would be a good point of reference

  • ablation studies: much of this is phrased with retained ratio, but the resulting ratio is determined by the lambda coeff, and empirically on the particular data, how much patch-difference penalty there is vs the lambda value. how is lambda chosen and aligned to the ratios?

    • also, are the ratios average retention over the dataset, and if so, what is the distrib of retention rates for different videos in each dataset?

    • likewise what is the distribution of ttft for different videos?

  • table 4: do the other 3 (non-pixel) comparisons use the same local search window for best alignment for finest scale? if not (or even if so), it would be good to see lines for pixel space both with and without the search alignment.

  • fig 5 right: How is "change across frames" computed, and how does it correspond to the error measures used for patch selection in the flashgaze model? If they correspond in how they are computed, then this would naturally lead to using them more (as it throws away other less-changing regions). Of course, that is kind of the point by construction, making this more of a sanity check for the system behavior.

    • also in fig 5: I think it's confusing to put acc on the right axis with dotted lines, esp since the dotted lines almost look like averages of the attention lines at first glance, even though they are unrelated. IMO it would be clearer to just note accuracy in the caption or legend alone.
17 comments
AuthorsOpenReview ↗

Response to Reviewer QfnB (Attribution and Positioning of Figure 1)

Question 1: Fig 1 when to prune: This is a good figure and example, but the same figure comparing pre-vit to other placements is also the first figure in autogaze (Shi et al 2026), with similar 12x total reduction arrow in the pre-vit row. Yet it appears like it is showing a novel observation. Instead it probably should add attribution in the caption and/or include more methods as examples for each type.

Thank you for pointing out the issue with the attribution and framing of this figure. We apologize for giving the impression that this is a novel observation of ours. In fact, AutoGaze already presents the comparison of different pruning positions in its Figure 1, and we follow this finding. We have made the following two changes in the revised version:

  1. We add "This figure is adapted from Figure 1 of AutoGaze (Shi et al., 2026)." to explicitly state the source of this categorization.
  2. We change "Existing methods prune visual tokens before or inside the MLLM" to "Many methods prune visual tokens before or inside the MLLM".

Thank you again for the correction. These revisions make the source of the figure explicit and the related statements more accurate.

AuthorsOpenReview ↗

Response to Reviewer QfnB (Visual Memory Storage, Local Alignment, and Updates)

Weakness 1: Some additional method details could be better explained or are currently missing --- see below; in particular:

  • (b) the format of M_t

Question 2: how is M_t stored and updated in the implementation? if local alignment were not used, it appears it is just the selected patches written into a maintained frame buffer at each t. But with local alignment costs, there are patches that overlap in their matched alignment --- how are they stored? Or are the patches always stored at the grid-aligned locations in a frame buffer and just compared allowing local search, for whether to update?

Thank you for the suggestion. Our description in the paper was indeed not clear enough, and we apologize for the confusion.

Your latter understanding is correct: MtM_t is always a plain frame buffer. Patches are stored at fixed, grid-aligned locations; each spatial region keeps only the most recently written patch content, and an update overwrites the old content. Local alignment is used only for computing costs and does not change where the memory is stored. At each step, MtM_t is first temporarily aligned to the current frame according to the local displacements, giving a reference copy that is used only to compute the Drop cost, i.e., to decide whether each location needs to be updated. This copy is discarded after use and is never written back to MtM_t. In the actual update, Keep and Merge write the original pixels and the reconstructed pixels, respectively, back to their own grid locations, while Drop locations remain unchanged.

Thank you again for the suggestion. We are happy to address any further questions.

AuthorsOpenReview ↗

Response to Reviewer QfnB (Memory Updates and Cross-Scale Overwriting for Merged Patches)

Question 3: For a merged patch: What happens to M_t (and the recon costs) in merge cases? In particular, what happens if M_t contains a merged patch and then a fine-scale patch in one of its locations is kept at t+1 instead? And similarly the other way around: a fine-scale patch is in M_t and then merge is chosen at t+1? If using a framebuffer as described in my point above, would these simply overwrite their patch locations in the M_t+1?

We apologize for the confusion caused by the original description. Yes, switching between scales is implemented by overwriting the corresponding spatial locations in MtM_t. MtM_t is a pixel canvas of the same size as the frame, and it is updated at each frame by the following rules: Keep locations are written with the original pixels of the current frame; Merge regions are written with the pixels of the current frame downsampled and then upsampled back to the original size; Drop locations keep their existing content. For the two cases you mention:

  • Merge first, then Keep: if a fine-scale location inside a previously merged region is kept at the next frame, only this location is overwritten with the sharp pixels of the current frame; the other locations in the merged region that are not updated remain unchanged.
  • Keep first, then Merge: if a location that previously held fine-scale content is included in a Merge at the next frame, the whole merged region is overwritten with the reconstructed pixels of the current frame at that scale, replacing the fine-scale content it held.

Therefore, cross-scale switching only requires overwriting the corresponding region of the memory. We hope this answers your question.

Reviewer_7vtNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FlashGaze, a training-free video patch pruning method that operates before ViT encoding. It estimates temporal dropping and spatial merging costs using pixel-space differences and employs Quadtree Dynamic Programming to select Drop, Merge, and Keep operations under a patch budget. Experiments on Qwen3-VL-8B and NVILA-8B-Video report inference speedups, reduced memory usage, and improved scalability to longer and higher-resolution videos.

Critical strengths
  1. A practically relevant optimization target. Moving compression before ViT encoding directly reduces the cost of processing redundant visual inputs, rather than merely reducing downstream language-model computation.

  2. Promising efficiency results. The evaluation covers two backbones and reports accuracy, TTFT, and memory usage at different retained ratios, demonstrating practical potential.

Critical weaknesses
  1. The core novelty is insufficiently established. The method primarily combines pixel-space costs with Quadtree DP to jointly perform temporal dropping and spatial merging. Given the related work discussed in the paper, including AutoGaze, FlashVID, and PixelPrune, the authors have not sufficiently established what constitutes a substantive algorithmic advance over existing pre-encoding selection, multi-scale compression, and tree-based optimization methods. Empirical efficiency gains alone do not establish sufficient methodological novelty.

  2. Key assumptions and some experimental claims remain insufficiently validated. Pixel differences are not necessarily equivalent to task-relevant information loss, and local search only addresses some minor motion, leaving robustness in complex dynamic scenes unclear. Moreover, Table 2 simultaneously changes the frame count and resolution, preventing the performance differences from being attributed specifically to FlashGaze. Table 3 also shows that HLVid accuracy drops from 71.6 to 68.2 at a 20% retained ratio, indicating that performance preservation depends substantially on the benchmark.

Questions for authors
  1. What is the core algorithmic novelty of FlashGaze over AutoGaze, FlashVID, PixelPrune, and other related compression methods? What capabilities or gains does the combination of pixel-space costs and Quadtree DP provide that existing methods cannot readily achieve?

  2. Can pixel differences reliably estimate information loss under rapid camera motion, illumination changes, occlusion, scene transitions, and small objects or fine-grained text? Can the authors provide systematic evaluations of these scenarios?

  3. Since Table 2 changes both frame count and resolution, can the authors provide matched-scale comparisons and separately quantify the contributions of increasing frame count, increasing resolution, and applying FlashGaze?

  4. Can the authors provide more complete retained-ratio evaluations on both backbones to characterize the accuracy–TTFT–memory trade-off and establish whether the advantage over baselines persists across different compression levels?

  5. Given the inconsistent accuracy degradation across benchmarks at high compression ratios, how do the authors define the operating range in which FlashGaze remains reliable, and what are its practical limitations relative to full-input inference?

9 comments
AuthorsOpenReview ↗

Response to Reviewer 7vtN (Core Algorithmic Novelty and Resulting Capabilities)

Weakness 1: The core novelty is insufficiently established. The method primarily combines pixel-space costs with Quadtree DP to jointly perform temporal dropping and spatial merging. Given the related work discussed in the paper, including AutoGaze, FlashVID, and PixelPrune, the authors have not sufficiently established what constitutes a substantive algorithmic advance over existing pre-encoding selection, multi-scale compression, and tree-based optimization methods. Empirical efficiency gains alone do not establish sufficient methodological novelty.

Question 1: What is the core algorithmic novelty of FlashGaze over AutoGaze, FlashVID, PixelPrune, and other related compression methods? What capabilities or gains does the combination of pixel-space costs and Quadtree DP provide that existing methods cannot readily achieve?

Thank you for your attention to the core algorithmic contribution. The core novelty of FlashGaze is to formulate pre-ViT spatiotemporal redundancy compression as an information-loss optimization problem in pixel space. As described in Section 3.2 and Section 3.3, the Drop, Merge, and Keep operations for each region of each frame are priced by their pixel-space information loss and decided jointly under a single patch budget; Quadtree DP then exactly solves the defined objective over all valid multi-scale partitions. Combining pixel-space costs with the quadtree structure gives this optimization the following three properties.

First, decisions are made before ViT encoding, directly reducing visual encoding computation. Post-encoding methods such as FlashVID rely on attention or features produced by the ViT for selection, and therefore still pay the visual encoding cost of producing these signals. FlashGaze estimates information loss directly in pixel space and determines the compressed input without running the ViT or an additional selection network, reducing the number of tokens processed by both the ViT and the language model.

Second, temporal and spatial redundancy are removed together and allocated jointly under the same budget. PixelPrune targets document and GUI images: it decides whether to drop each patch based only on its spatial neighbors within the same image, does not model temporal redundancy across video frames, and does not explicitly solve a cross-region budget allocation. FlashGaze places temporal dropping and multi-scale spatial merging in a single objective, and the DP jointly compares the costs of different regions, scales, and operations under a given budget: low-change regions save budget through dropping or coarse-scale merging, while detail-rich regions retain more fine-scale patches.

Third, spatiotemporal compression is formulated as a decomposable structured optimization problem that requires no trained selector. AutoGaze relies on a learned video reconstruction model to measure information loss. There, the contribution of each patch depends on the other patches already selected and cannot be computed independently, so patches can only be selected step by step with an autoregressive policy. In FlashGaze, the operation cost of each region can be computed independently and summed; Quadtree DP handles the cross-scale exclusivity constraints and the cross-region budget allocation together, and deterministically determines the compression operation and representation scale of each region, without any training.

In summary, compared with FlashVID, FlashGaze moves the computation and memory savings forward to the ViT stage; compared with AutoGaze, it replaces the learned selector with Quadtree DP, avoiding the training and inference overhead of the selector itself; compared with PixelPrune, it extends compression from the spatial dimension to the temporal dimension and removes the inter-frame redundancy that PixelPrune does not model. The unification of pre-encoding execution, joint spatiotemporal allocation, and training-free efficient solving constitutes the algorithmic contribution of FlashGaze.

AuthorsOpenReview ↗

Response to Reviewer 7vtN (Reliability of Pixel-Space Costs in Complex Visual Scenes)

Weakness 2: Key assumptions and some experimental claims remain insufficiently validated. Pixel differences are not necessarily equivalent to task-relevant information loss, and local search only addresses some minor motion, leaving robustness in complex dynamic scenes unclear.

Question 2: Can pixel differences reliably estimate information loss under rapid camera motion, illumination changes, occlusion, scene transitions, and small objects or fine-grained text? Can the authors provide systematic evaluations of these scenarios?

Thank you for pointing out these challenging scenarios. Our additional experiments show that pixel differences serve as an effective proxy for information loss in pre-encoding compression, allowing FlashGaze to preserve task-relevant key information in the complex visual scenes we evaluate. Under camera motion, illumination changes, occlusion and scene transitions, and in scenes containing small objects and fine-grained text, FlashGaze maintains video understanding accuracy close to that of the full input.

We select a total of 1,605 evaluation videos from the evaluation benchmarks and compute five scenario metrics for each of them. For each metric, the 321 videos (20%) with the highest scores form the hard group, and the remaining 1,284 videos (80%) form the normal group. The grouping metrics are as follows:

  • Camera motion: the average motion magnitude between adjacent frames, computed with Farneback optical flow.
  • Illumination change: the variance of the per-frame mean brightness.
  • Occlusion and scene transitions: the number of abrupt frame changes, normalized by video duration to a per-minute count.
  • Fine-grained text: we run OCR on uniformly sampled frames and average the number of detected text boxes per frame whose area is below a threshold.
  • Small objects: we run YOLOv8n on uniformly sampled frames and average the number of objects per frame whose area is below 1% of the frame.

With Qwen3-VL-8B, we compare Vanilla, FlashGaze with the pixel cost, and FlashGaze with the patch-embedding cost, under the same retained ratio

ScenarioGroupVanilla Acc.Pixel Acc.Patch-embedding Acc.
Camera motionHard71.873.067.9
Camera motionNormal66.766.565.3
Illumination changeHard65.965.463.7
Illumination changeNormal68.268.466.4
Occlusion and scene transitionsHard64.864.260.3
Occlusion and scene transitionsNormal68.568.767.2
Fine-grained textHard67.969.366.6
Fine-grained textNormal67.767.465.6
Small objectsHard76.677.173.4
Small objectsNormal65.565.563.9

In all five hard groups, Pixel maintains accuracy close to or above that of the full input. In contrast, the patch-embedding cost drops by 1.3 to 4.5 points in the hard groups, with the largest drops under camera motion (−3.9), occlusion and scene transitions (−4.5), and small objects (−3.2). Pixel outperforms the patch-embedding cost in all ten groups. These results show that pixel differences are a reliable proxy for information loss in the scenarios above.

AuthorsOpenReview ↗

Response to Reviewer 7vtN (Contributions of Frame Count, Resolution, and FlashGaze)

Weakness 2: Moreover, Table 2 simultaneously changes the frame count and resolution, preventing the performance differences from being attributed specifically to FlashGaze.

Question 3: Since Table 2 changes both frame count and resolution, can the authors provide matched-scale comparisons and separately quantify the contributions of increasing frame count, increasing resolution, and applying FlashGaze?

We add a comparison under matched visual token budgets on six benchmarks. On HLVid, we further vary the frame rate and the resolution separately, and compare the results with and without FlashGaze at the same input scale. These experiments have been added to Appendix H of the paper.

First, on each benchmark, we match the average number of visual tokens of FlashGaze and Vanilla. The results are as follows:

BenchmarkAvg. visual tokensFlashGaze framesFlashGaze resolutionFlashGaze Acc.Vanilla framesVanilla resolutionVanilla Acc.
VideoMME93K2 fps, max 1536102471.81 fps, max 768102470.2
NExT-QA29K8 fps, max 1536102484.74 fps, max 768102484.3
VideoMMMU103K2 fps, max 1536128067.01 fps, max 768128065.2
LongVideoBench122K2 fps, max 1536128066.71 fps, max 768128063.3
MLVU114K2 fps, max 1536102477.61 fps, max 768102475.3
HLVid290K2 fps, max 1536358473.21 fps, max 768179265.7

On the first five benchmarks, the two methods use the same resolution and the same visual token budget, while FlashGaze supports 2× the sampling frame rate and maximum number of frames; accuracy improves by 1.6, 0.4, 1.8, 3.4, and 2.3 percentage points, respectively. These results show that, with the resolution fixed, FlashGaze improves understanding accuracy within the same budget by supporting more video frames.

To further separate the effects of frame count and resolution, we add the following configurations on HLVid. A is the Vanilla baseline; B and C directly increase the frame rate and the resolution, respectively; D, E, and F use FlashGaze to keep the number of visual tokens of the different input configurations at 290K, the same as A.

RowFramesResolutionFlashGazeVisual tokensAcc.
A1 fps1792✗290K65.7
B2 fps1792✗580KOOM
C1 fps3584✗1.1MOOM
D2 fps179250% retained290K70.1 (+4.4)
E1 fps358427% retained290K73.0 (+7.3)
F2 fps358413% retained290K73.2 (+7.5)

With the resolution fixed at 1792, raising the sampling rate from 1 fps to 2 fps while using FlashGaze to keep the 290K-token budget (A → D) improves accuracy from 65.7 to 70.1, a gain of 4.4 percentage points. With the sampling rate fixed at 1 fps, raising the resolution from 1792 to 3584 while keeping the same budget with FlashGaze (A → E) yields 73.0, 7.3 percentage points above the baseline. Under the same budget, raising both the sampling rate and the resolution to 2 fps and 3584 (A → F) yields 73.2, 7.5 percentage points above the baseline. The uncompressed configurations B and C both run out of memory, whereas with FlashGaze the corresponding configurations D and E complete inference within the baseline budget.

Reviewer_BNeZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes training-free spatiotemporal patch pruning before the vision encoder. It estimates the costs of dropping and merging patches using pixel-space differences, and applies quadtree dynamic programming to select multi-scale patches under a budget. Experiments on Qwen3-VL-8B and NVILA-8B-Video demonstrate reduced time-to-first-token (TTFT) and memory consumption while largely preserving video understanding accuracy.

Critical strengths
  1. FlashGaze removes the need for a learned auxiliary selector while reducing computation in both the vision encoder and the language model. This provides a useful simplification of existing pre-ViT pruning approaches.

  2. The paper provides comprehensive experiments across multiple benchmarks and backbones, demonstrating the effectiveness of FlashGaze in reducing inference latency and memory usage while largely preserving accuracy. It also includes ablation studies on key design choices and analyses of visual representations and attention patterns.

Critical weaknesses
  1. The conceptual novelty is limited. AutoGaze has already explored pre-ViT multi-scale spatiotemporal patch selection. FlashGaze mainly replaces its learned selector with heuristic pixel-space costs and quadtree DP. While this simplification offers practical efficiency benefits, the paper does not sufficiently demonstrate a novel methodological contribution or broader significance beyond existing approaches.

  2. The practical significance and generalizability of the method remain insufficiently demonstrated. Although Table 2 shows that FlashGaze supports more frames and higher resolutions, the accuracy improvements over the original Qwen3-VL are modest. More importantly, the paper does not report the actual visual token counts, latency, or memory consumption at these expanded input scales, making it difficult to assess the cost and practical benefit of such scaling. Furthermore, the evaluation is limited to relatively older backbones, leaving its effectiveness on newer and more efficient architectures, such as Qwen3.5, unclear.

Questions for authors
  1. For the expanded input settings in Table 2, what are the actual visual token counts, TTFT, and peak memory usage? Can the authors demonstrate that processing more frames or higher-resolution inputs improves accuracy under the same overall computational or wall-clock budget?

  2. Since the paper only reports TTFT, can the authors also report end-to-end wall-clock latency, including the full decoding process? Does visual token pruning affect the number of generated tokens or the reasoning required to produce accurate answers?

  3. Can the authors evaluate FlashGaze on a more recent backbone, such as Qwen3.5-9B, to verify whether its accuracy and efficiency benefits generalize to newer model architectures?

6 comments
AuthorsOpenReview ↗

Response to Reviewer BNeZ (Actual Costs of Expanded Inputs and Gains under the Same Budget)

Weakness 2: The practical significance and generalizability of the method remain insufficiently demonstrated. Although Table 2 shows that FlashGaze supports more frames and higher resolutions, the accuracy improvements over the original Qwen3-VL are modest. More importantly, the paper does not report the actual visual token counts, latency, or memory consumption at these expanded input scales, making it difficult to assess the cost and practical benefit of such scaling.

Question 1: For the expanded input settings in Table 2, what are the actual visual token counts, TTFT, and peak memory usage? Can the authors demonstrate that processing more frames or higher-resolution inputs improves accuracy under the same overall computational or wall-clock budget?

We sincerely thank you for this insightful question, which pinpoints a key gap in our evidence and has made the paper stronger. FlashGaze improves accuracy with richer video inputs while keeping the actual inference cost unchanged or lower. To address your questions about the cost of the expanded inputs and the gains under the same budget, we add resource measurements, a comparison under matched token budgets, and an ablation over input configurations.

First, the table below reports the actual visual token counts, TTFT, and peak memory for the Qwen3-VL-8B (Vanilla) and FlashGaze configurations in Table 2.

VideoMMENExT-QAVideoMMMULongVideoBenchMLVUHLVid
VanillaVisual tokens100K30K105K217K120K290K
TTFT (s)11.32.212.239.415.064.7
Peak mem. (GiB)54.136.655.483.559.2101.8
+FlashGazeVisual tokens93K ↓29K ↓103K ↓122K ↓114K ↓290K -
TTFT (s)10.11.911.214.313.564.6
Peak mem. (GiB)53.036.254.359.657.6101.7

With no more visual tokens than the baseline, FlashGaze with the expanded inputs maintains or reduces TTFT and peak memory on all six benchmarks. These measurements directly show the practical benefit of input expansion.

Next, to answer whether more frames and higher resolution improve accuracy under the same budget, we match the average number of visual tokens between Vanilla and FlashGaze. Under this budget, FlashGaze doubles the sampling rate, raises the maximum number of frames from 768 to 1536, and, on HLVid, increases the resolution from 1792 to 3584.

VideoMMENExT-QAVideoMMMULongVideoBenchMLVUHLVid
Avg. visual tokens93K29K103K122K114K290K
VanillaFrames1 fps, max 7684 fps, max 7681 fps, max 7681 fps, max 7681 fps, max 7681 fps, max 768
Resolution102410241280128010241792
Acc.70.284.365.263.375.365.7
FlashGazeFrames2 fps, max 15368 fps, max 15362 fps, max 15362 fps, max 15362 fps, max 15362 fps, max 1536
Resolution102410241280128010243584
Acc.71.884.767.066.777.673.2

These results show that FlashGaze improves accuracy on all six benchmarks under the same token budget. In particular, LongVideoBench improves from 63.3% to 66.7% and HLVid from 65.7% to 73.2%, gains of 3.4 and 7.5 percentage points, respectively. This indicates that the budget freed by removing redundancy can be spent on richer spatiotemporal input and translates into real accuracy gains.

We further increase the sampling rate and the resolution separately on HLVid to analyze their individual contributions.

RowFramesResolutionFlashGazeVisual tokensAcc.
A1 fps1792✗290K65.7
B2 fps1792✗580KOOM
C1 fps3584✗1.1MOOM
D2 fps179250% retained290K70.1 (+4.4)
E1 fps358427% retained290K73.0 (+7.3)
F2 fps358413% retained290K73.2 (+7.5)

Under the same 290K-token budget, compared with 65.7% for the original configuration (A), increasing only the sampling rate (D), only the resolution (E), and both (F) yields 70.1%, 73.0%, and 73.2%, respectively. The corresponding uncompressed expanded configurations (B, C) require 580K and 1.1M tokens; both run out of memory and far exceed the model's context length. Therefore, FlashGaze not only improves accuracy under the same budget, but also makes input configurations that were previously limited by memory usable. These experiments have been added to Appendix H.

AuthorsOpenReview ↗

Response to Reviewer BNeZ (End-to-End Latency, Generation Length, and Reasoning Process)

Question 2: Since the paper only reports TTFT, can the authors also report end-to-end wall-clock latency, including the full decoding process? Does visual token pruning affect the number of generated tokens or the reasoning required to produce accurate answers?

FlashGaze substantially reduces end-to-end latency including the full decoding process, while keeping the generation length and accuracy nearly unchanged and preserving the key reasoning steps in the examples shown. Except for VideoMMMU, all benchmarks are evaluated without thinking/CoT and consist of single-choice questions, for which the model outputs a single option-letter token; TTFT is therefore the latency of producing the complete answer. For VideoMMMU, where thinking/CoT is enabled, we additionally measure the average wall-clock latency covering the full reasoning and answer generation, the number of generated tokens, and the accuracy. The latency includes the processing overhead of FlashGaze. The results are as follows.

MethodRetained ratioGenerated tokens (avg.)Wall-clock latency (s)Acc.
Qwen3-VL-8B100%860113.067.2
+FlashGaze50%85134.367.0

When retaining 50% of the visual tokens, FlashGaze reduces the average end-to-end latency from 113.0 s to 34.3 s, a 3.29× speedup; the average number of generated tokens changes from 860 to 851, a change of about 1%; and the accuracy is 67.0%, essentially the same as the 67.2% of the original model. This shows that FlashGaze also substantially accelerates inference that includes full decoding, with the generation length essentially unchanged.

To further illustrate the effect on the reasoning process, we compare the outputs before and after compression on the following two VideoMMMU examples:

Example 1

Question: Looking at the natural response equation shown on the right side of the video, if V0 = 10V and t = RC, what would be the value of v(t) based on the exact exponential equation written in blue?

Qwen3-VL-8B (100% retained):

The natural response equation shown in the video is:
v(t) = V₀ * e^(-t/RC)
Given:
- V₀ = 10V
- t = RC
Substitute t = RC into the equation:
v(RC) = 10 * e^(-RC / RC) = 10 * e^(-1)
We know that e^(-1) ≈ 0.3679
So,
v(RC) = 10 * 0.3679 = 3.679V
Rounding to two decimal places, this is approximately 3.68V.
---
✅ Therefore, the correct answer is: C. 3.68V

FlashGaze (50% retained):

The natural response equation for the RC circuit is given as:
v(t) = V₀ * e^(-t/RC)
We are told:
- V₀ = 10V
- t = RC
Substitute t = RC into the equation:
v(t) = 10 * e^(-RC/RC) = 10 * e^(-1)
We know that e^(-1) ≈ 0.3679
So,
v(t) = 10 * 0.3679 = 3.679V
Rounding to two decimal places, this is approximately 3.68V
---
### ✅ Correct Answer: C. 3.68V

Example 2

Question: If the gauge length was increased to 300 mm (keeping all other conditions the same), what would be the percentage elongation at fracture?

Qwen3-VL-8B (100% retained):

The percentage elongation is defined as:
Percentage elongation = (Final extension / Original length) × 100
In the original problem, the final extension at fracture was 56 mm, and the original gauge length was 200 mm. This gave a percentage elongation of:
56 / 200 × 100 = 28%
Now, if the gauge length is increased to 300 mm, but the final extension remains the same (56 mm), then:
Percentage elongation = (56 / 300) × 100
Let’s calculate that:
56 ÷ 300 = 0.1867
0.1867 × 100 = 18.67%
---
✅ So, the correct answer is: C. 18.67%
---
Answer: C. 18.67%

FlashGaze (50% retained):

The percentage elongation is defined as:
Percentage elongation = (Final extension / Original length) × 100
In the original problem, the final extension at fracture is 56 mm, and the original gauge length is 200 mm. This gives:
Percentage elongation = (56 / 200) × 100 = 28%
Now, if the gauge length is increased to 300 mm, but the final extension remains the same (56 mm), then:
Percentage elongation = (56 / 300) × 100
Let’s calculate that:
56 ÷ 300 = 0.18666...
0.18666... × 100 = 18.67%
---
✅ So, the correct answer is: C. 18.67%
---
Answer: C. 18.67%

Overall, FlashGaze substantially reduces end-to-end latency while keeping the average generation length essentially unchanged, leaving the reasoning process almost unaffected, and maintaining high accuracy. These experiments have been added to Appendix I.

AuthorsOpenReview ↗

Response to Reviewer BNeZ (Generalization to Newer Model Architectures)

Weakness 2: Furthermore, the evaluation is limited to relatively older backbones, leaving its effectiveness on newer and more efficient architectures, such as Qwen3.5, unclear.

Question 3: Can the authors evaluate FlashGaze on a more recent backbone, such as Qwen3.5-9B, to verify whether its accuracy and efficiency benefits generalize to newer model architectures?

We have added experiments on a newer model architecture, Qwen3.5-9B, in Appendix G. The evaluation in Section 4.1 indeed did not cover the latest model architectures. Your suggestion helps us strengthen the experiments and further examine how FlashGaze generalizes to newer backbones. The results show that, on Qwen3.5-9B, FlashGaze still maintains accuracy close to that of the original model while substantially reducing TTFT and peak memory. Specifically, we evaluate retained ratios of 50%, 40%, 30%, and 20% on LongVideoBench and HLVid, and compare against ToMe, FlashVID, and AutoGaze.

MethodRetained RatioLongVideoBench Acc.Peak GPU Mem. (GiB)TTFT (s)HLVid Acc.Peak GPU Mem. (GiB)TTFT (s)
Qwen3.5-9B100%70.0104.560.766.0103.762.5
+ToMe50%69.379.427.865.279.229.5
+FlashVID50%69.681.828.565.281.531.6
+AutoGaze50%68.969.533.564.662.928.9
+FlashGaze50%69.665.224.765.356.719.3
+ToMe40%69.179.422.365.279.224.0
+FlashVID40%69.281.824.065.281.527.0
+AutoGaze40%68.762.326.864.956.924.2
+FlashGaze40%69.353.917.565.349.114.5
+ToMe30%69.379.418.064.279.219.6
+FlashVID30%69.381.819.165.381.521.9
+AutoGaze30%68.753.120.264.752.420.9
+FlashGaze30%69.545.512.565.343.010.7
+ToMe20%68.479.413.864.979.215.5
+FlashVID20%69.381.815.065.081.517.7
+AutoGaze20%66.548.616.464.348.717.3
+FlashGaze20%69.337.07.665.038.58.1

When retaining only 20% of the visual tokens, FlashGaze reaches 69.3% and 65.0% accuracy on LongVideoBench and HLVid, retaining 99.0% and 98.5% of the full-input accuracy. TTFT drops from 60.7 s to 7.6 s and from 62.5 s to 8.1 s, corresponding to 7.99× and 7.72× speedups. Meanwhile, peak memory drops from 104.5 GiB to 37.0 GiB and from 103.7 GiB to 38.5 GiB, corresponding to 2.82× and 2.69× memory reductions. This shows that, on Qwen3.5-9B, FlashGaze likewise delivers substantial latency and memory gains with accuracy essentially unchanged.

Compared with the other compression methods, at all four retained ratios on both benchmarks, FlashGaze achieves the lowest TTFT and peak memory at the same retained ratio, and the highest or tied-highest accuracy. These results confirm that the accuracy and efficiency advantages of FlashGaze carry over to the newer architecture, demonstrating its adaptability to new-generation model architectures and its potential for broader use in video understanding models.

Comments on the paper

2 comments

We sincerely thank all reviewers for their time and constructive feedback. We appreciate the strengths the reviewers recognized in our work, and their comments have helped us substantially improve the manuscript.

Strengths:

  • Practically relevant design. Pruning before ViT encoding reduces computation in both the vision encoder and the language model, without a learned auxiliary selector.
  • Simple and well-explained method. Patch selection uses pixel-space costs with local alignment, and Quadtree DP finds the lowest-cost assignment.
  • Comprehensive experiments. Results across multiple benchmarks and backbones show reduced latency and memory usage while largely preserving accuracy, covering accuracy, TTFT, and memory at different retained ratios.
  • Informative ablations and analyses. Ablations on retained ratios, cost metrics, and input scale, together with analyses of visual representations and attention patterns, characterize the behavior of the method.

In response to common concerns, we have significantly strengthened the manuscript. First, we clarify the core novelty of our method from algorithm as well as efficiency. Second we conduct comprehensive additional experiments on up-to-date backbones, matched token budgets, end-to-end latency, and challenging visual scenes. They confirm that the efficiency gains of FlashGaze hold across backbones and visual conditions, and that the saved computation can be reinvested in more frames and higher resolution for better accuracy.

In the revisited version of the paper, we added the following new sections that address common questions and weaknesses from reviewers:

Appendix G: Additional Experiment On Different Retained Ratios On Qwen3.5

Appendix H: Additional Experiment On More Frames And Higher Resolution Under The Same Token Budget

Appendix I: Additional Experiment On End-To-End Latency With Full Decoding

Appendix J: Robustness Of The Pixel Cost In Challenging Visual Scenes

Through these revisions, we have addressed the key concerns raised. We believe this revised paper now presents a solid advancement in efficient video understanding.

AuthorsOpenReview ↗

Response on the Methodological Contribution and Broader Significance

We thank all reviewers for their replies.

The purpose of token compression is efficiency, so we believe a method in this area should be judged primarily by whether it advances the accuracy–efficiency trade-off. Across multiple benchmarks, three backbones, and a range of retained ratios, FlashGaze achieves the lowest latency in every setting and the highest accuracy in nearly all of them. FlashGaze formulates token compression as spatiotemporal budget allocation in pixel space, which differs from existing methods and makes it simpler, faster, and more accurate than more complex ones. In our view, this is a meaningful contribution.

We thank you again for your time and for engaging with our responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.