- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a method for spatial patch selection for video MLLMs, using error of an accumulated "memory" buffer (last selected patches at each location) and the current frame. A simple quadtree DP algorithm and pixel-space error measures are used to select, merge or drop patches at each frame based on trading off their error reduction vs number of selected patches. Notably, this involves only simple tuning and calibration training, with no significant extra neural-net-based model used for selection. Measurements on LongVideoBench, VideoMMMU, and HLVid with Qwen and NVILA show good improvements over baseline systems, even as the algorithm is simpler.
Critical strengths
-
Simple, well explained method. Uses reconstruction in pixel space (with local alignment) for patch selection, and quadtree DP merge to find best cost.
-
Improved performance on video benchmarks, even without needing an additional trained model
-
Ablations on retention ratios vs acc, reconstruction error metric, and resolution vs latency show good profiles of system behavior
Critical weaknesses
-
Some additional method details could be better explained or are currently missing --- see below; in particular:
- (a) how lambda is selected (there must be some sort of calibration/tuning, that I don't see described)
- (b) the format of M_t
- (c) how mid and coarse patches are input to the vit
-
All retention ratios and speedups are given as averages. It would also be good to see the distribution of retained patches for different videos, and if there anyvideos that this method fails for, e.g. large or fast camera motions, scene cuts, busy object movements
-
More analysis on the cost vs downstream accuracy would be good. Accuracy is profiled in terms of retention ratio, but not related to the cost (pixel-space error), both on average and for different videos.
-
Similarly, while accuracy is measured vs retention ratio, there is less analysis on how to increase resolution (spatial and/or temporal) and how this relates to accuracy, using the patches selected by this method. Only one line in table 2 seems to relate to this.
Questions for authors
-
Fig 1 when to prune: This is a good figure and example, but the same figure comparing pre-vit to other placements is also the first figure in autogaze (Shi et al 2026), with similar 12x total reduction arrow in the pre-vit row. Yet it appears like it is showing a novel observation. Instead it probably should add attribution in the caption and/or include more methods as examples for each type.
-
how is M_t stored and updated in the implementation? if local alignment were not used, it appears it is just the selected patches written into a maintained frame buffer at each t. But with local alignment costs, there are patches that overlap in their matched alignment --- how are they stored? Or are the patches always stored at the grid-aligned locations in a frame buffer and just compared allowing local search, for whether to update?
-
For a merged patch: What happens to M_t (and the recon costs) in merge cases? In particular, what happens if M_t contains a merged patch and then a fine-scale patch in one of its locations is kept at t+1 instead? And similarly the other way around: a fine-scale patch is in M_t and then merge is chosen at t+1? If using a framebuffer as described in my point above, would these simply overwrite their patch locations in the M_t+1?
-
min displacement search l.224: What is the size of the local window search, and how large is it relative to the patch size?
-
same for merge: is there a local search (over possibly larger window, at coarser stride) to get coarser level distance?
-
l.288 "selected patches are encoded by the ViT" --- how does the vit run on scales coarser than the finest? I'm assuming the fine scale corresponds to the vit stem stride, in which case the input for finest would be the subsampled stem features. But for mid and coarse level, what is used for the vit?
-
similarly, is the scale encoded in the input to vit and/or mllm for each patch? in particular, if m and c patches are downsampled in pixel space and fed to the vit as smaller rgb patches with pos encoding for their center, they would be indistinguishable from fine-scale patches with the same content at the same location. so presumably the scale is encoded somewhere, or somehow this doesn't matter. either way an experimental ablation on the scale encoding would be interesting.
-
what is the FPS used in the experiments? for example l.314 says "For LongVideoBench and VideoMMMU, the maximum number of sampled frames is set to 768." But I'm not sure if this is the pre-flashgaze rate, or somehow corresponds to something after the sampling algorithm? And not sure what frame rate it's starting from
-
table 1: "vanilla" qwen and nvila are not here, they would be a good point of reference
-
ablation studies: much of this is phrased with retained ratio, but the resulting ratio is determined by the lambda coeff, and empirically on the particular data, how much patch-difference penalty there is vs the lambda value. how is lambda chosen and aligned to the ratios?
-
also, are the ratios average retention over the dataset, and if so, what is the distrib of retention rates for different videos in each dataset?
-
likewise what is the distribution of ttft for different videos?
-
-
table 4: do the other 3 (non-pixel) comparisons use the same local search window for best alignment for finest scale? if not (or even if so), it would be good to see lines for pixel space both with and without the search alignment.
-
fig 5 right: How is "change across frames" computed, and how does it correspond to the error measures used for patch selection in the flashgaze model? If they correspond in how they are computed, then this would naturally lead to using them more (as it throws away other less-changing regions). Of course, that is kind of the point by construction, making this more of a sanity check for the system behavior.
- also in fig 5: I think it's confusing to put acc on the right axis with dotted lines, esp since the dotted lines almost look like averages of the attention lines at first glance, even though they are unrelated. IMO it would be clearer to just note accuracy in the caption or legend alone.
17 comments
Response to Reviewer QfnB (Visual Memory Storage, Local Alignment, and Updates)
Weakness 1: Some additional method details could be better explained or are currently missing --- see below; in particular:
- (b) the format of M_t
Question 2: how is M_t stored and updated in the implementation? if local alignment were not used, it appears it is just the selected patches written into a maintained frame buffer at each t. But with local alignment costs, there are patches that overlap in their matched alignment --- how are they stored? Or are the patches always stored at the grid-aligned locations in a frame buffer and just compared allowing local search, for whether to update?
Thank you for the suggestion. Our description in the paper was indeed not clear enough, and we apologize for the confusion.
Your latter understanding is correct: is always a plain frame buffer. Patches are stored at fixed, grid-aligned locations; each spatial region keeps only the most recently written patch content, and an update overwrites the old content. Local alignment is used only for computing costs and does not change where the memory is stored. At each step, is first temporarily aligned to the current frame according to the local displacements, giving a reference copy that is used only to compute the Drop cost, i.e., to decide whether each location needs to be updated. This copy is discarded after use and is never written back to . In the actual update, Keep and Merge write the original pixels and the reconstructed pixels, respectively, back to their own grid locations, while Drop locations remain unchanged.
Thank you again for the suggestion. We are happy to address any further questions.
Response to Reviewer QfnB (Memory Updates and Cross-Scale Overwriting for Merged Patches)
Question 3: For a merged patch: What happens to M_t (and the recon costs) in merge cases? In particular, what happens if M_t contains a merged patch and then a fine-scale patch in one of its locations is kept at t+1 instead? And similarly the other way around: a fine-scale patch is in M_t and then merge is chosen at t+1? If using a framebuffer as described in my point above, would these simply overwrite their patch locations in the M_t+1?
We apologize for the confusion caused by the original description. Yes, switching between scales is implemented by overwriting the corresponding spatial locations in . is a pixel canvas of the same size as the frame, and it is updated at each frame by the following rules: Keep locations are written with the original pixels of the current frame; Merge regions are written with the pixels of the current frame downsampled and then upsampled back to the original size; Drop locations keep their existing content. For the two cases you mention:
- Merge first, then Keep: if a fine-scale location inside a previously merged region is kept at the next frame, only this location is overwritten with the sharp pixels of the current frame; the other locations in the merged region that are not updated remain unchanged.
- Keep first, then Merge: if a location that previously held fine-scale content is included in a Merge at the next frame, the whole merged region is overwritten with the reconstructed pixels of the current frame at that scale, replacing the fine-scale content it held.
Therefore, cross-scale switching only requires overwriting the corresponding region of the memory. We hope this answers your question.
Response to Reviewer QfnB (Attribution and Positioning of Figure 1)
Thank you for pointing out the issue with the attribution and framing of this figure. We apologize for giving the impression that this is a novel observation of ours. In fact, AutoGaze already presents the comparison of different pruning positions in its Figure 1, and we follow this finding. We have made the following two changes in the revised version:
Thank you again for the correction. These revisions make the source of the figure explicit and the related statements more accurate.