- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a video tokens subselection method based on frame selection and query-attention subsampling between keyframes. Frame boundaries are identified using an MLP that runs on EMA accumulated differences between vit features, which is trained using gumbel softmax optimizing full-video "thumbnail" summaries. The resulting system is a 45x token reduction compared to using all frame tokens. Evaluations are performed on MMBench-Video, MVBench, MLVU, and Video-MME, showing roughly similar performance levels to comparable systems using ~5x fewer tokens.
Critical strengths
-
shows good token reduction for MLLM input (45x) while likely improving performance compared to baseline
-
A/B test shows improvement over matched no-subsampling baseline in single setting
-
the motivation of identifying scenes and subsampling constant tokens with each is intuitive
-
good ablations on tuning settings (temporal encoding, vit layers, z reg, annealing)
Critical weaknesses
-
recent system comparisons are hard to place in context and interpret --- although I roughly comparable, nothing is quite apples-to-apples, and all are over a year old. some newer ones are in the appendix, but not the main text
-
A/B test mentioned in the abstract is barely discussed --- shown in Fig 2 but I can't find it described anywhere in the text
-
should compare with simpler vis feature clustering (e.g., clip+temporal with kmeans)
-
tokens come from within-cube subsampling, but cube boundaries are optimized using a full-video pooled feature summary, which is not the same mechanism
-
still requires running frames through the full ViT, limiting gains to just the mllm
Questions for authors
-
l.240 thumbnails: This is a good pooling mechanism, and it indeed enables gradients for z. But the grads seem to now come entirely from the thumbnail, not the cube selection boundaries. The most informative frames for the global thumbnail are what the selection will optimize. This still loosely corresponds to scenes, in that it usually may be best to select one frame per scene; and the momentum construction may make that towards the start of the scene (since middle will have closer to 0 frame-to-frame difference). But it's not directly optimizing the use of the cube sampling itself or the cube boundaries --- these just happen to follow. It would be good to have some discussion or even measurements around this point.
-
Along these lines, sec 5.3 says for "visual lag" that "We posit this mechanism enables the model to retain partial memory of preceding scenes to facilitate current scene understanding". However, I think this could be as easily explained just by the frame sampler for thumbnail pooling choosing intermediate frames. They would always tend to be frames close to a scene cut start due to momentum mlp input for z --- which frame is according to the thumbnail pooling, which can be any frame. I don't think it is necessarily aligned to anything about the cube subsampling as implied in this explanation.
-
another simple comparison/ablation would be to use the selected frames (before temporal pooling into a single thumbnail) as the input to the vlm, removing the cube frames subsampling entirely
-
Table 2 comparison systems are all from 2024, and would be good to show newer models for comparison as well. Especially comparable ones in terms of param count and training data. If newer ones operate in a less-comparable regime, please say so in the text; but I think they would be good to include for context as well.
-
l.228 Resampling subsection: This could use a little more detail in its summary of how resampling is done. Currently understanding it depends on reading Yao et al 2024 --- it would be clearer to read if there was a more stand-alone summary of the method here, with reference only needed for smaller details.
-
fig 4a and l.368 analysis section: this is difficult to interpret without a baseline: is poorer performance on these domains due to problems introduced by this subsampling technique, or was it existing already in the base models?
-
do learned keyframes correspond to scene cuts or other scene cut detecting methods? some anecdotal evidence in fig 5 shows they may not. it would be interesting to see a more quantitative comparison
-
l.409 fig 4 should be fig 5.
-
minor grammatical/agreement issues in several places, e.g.: l.193 "For compatible with online ..." -> "For compatibility with", l.196 "the cubing network calculate" -> "calculates", "A 2-layers MLP with LayerNorm (Ba et al., 2016) is then applied to the mean of visual tokens momentums", many others
Weakness-2
Thank you for this suggestion. We agree that “the first dimension” is ambiguous. Following the binary gating formulation of Herrmann et al. [1], our implementation selects the class-1 probability after Gumbel-Softmax, then applies top-k across candidate frames. Here, class 1 means the second component under zero-based indexing.
The relevant code is:
We will revise the passage in Section 3.1 to explicitly define the categories and selection procedure:
\left[ \operatorname{softmax}\left( \frac{\mathbf{z}_i+\eta_t\mathbf{g}_i}{\tau} \right) \right]_1,$$ > where $\cdot_1$ denotes category index 1, $\mathbf{g}_i$ contains independent standard Gumbel samples, $\eta_t$ is the noise scale, and $\tau$ is the relaxation temperature. We select boundary indices by $$\mathcal{I}= \operatorname{TopK}\left( [q_i]_{i=2}^{N_F},\,N_Q-1 \right),$$ > where TopK returns the temporal indices of the largest scores. The first frame is always included as a boundary. At inference, $\eta_t=0$. This revision makes clear that top-k operates on the class-1 probability scores across time. Thank you for helping us clarify the description. **References** - [1] Herrmann, Charles, Richard Strong Bowen, and Ramin Zabih. "Channel selection using gumbel softmax." ECCV, 2020.