acceptodds
Under review as a conference paper at ICLR 2027

Quicksviewer: An LMM with Early Video Compression using Annealed Gumbel-Softmax

Abstract

Large Multimodal Models (LMMs) uniformly perceive video frames, creating computational inefficiency for videos with inherently varying temporal redundancy. This paper present Quicksviewer, an LMM with new perceiving paradigm that partitions a nonuniform redundant video into varying cubes using Gumbel Softmax, followed by a unified resampling for each cube to achieve efficient video understanding. We further propose a Gumbel-noise annealing strategy that enables stable end-to-end LMM training with the discrete cubing network. This simple and intuitive approach dynamically compress video online based on the momentum of frame differences, significantly reducing spatiotemporal complexity (overall 45 compression rate), while enabling efficient training with large receptive field. We train the model from a language backbone through three progressive stages, each incorporating lengthy videos on average of 420s/1fps thanks to the perceiving efficiency. With only 0.8M total video samples, our model outperforms contemporary LMMs with the same backbone and comparable parameters amount. An experiment in A/B pretraining with same backbone and training data shows that, w/ cubing achieves lower training loss and higher benchmark accuracy (+2.1) than the fixed partitioning baseline, strictly validating the benefits. Moreover, the front-loaded compression in encoding substantially reduces latency in the computationally expensive prefill and decoding, making it suited to mainstream EPD disaggregated inference deployment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Gumbel-softmax, Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_i53QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a video tokens subselection method based on frame selection and query-attention subsampling between keyframes. Frame boundaries are identified using an MLP that runs on EMA accumulated differences between vit features, which is trained using gumbel softmax optimizing full-video "thumbnail" summaries. The resulting system is a 45x token reduction compared to using all frame tokens. Evaluations are performed on MMBench-Video, MVBench, MLVU, and Video-MME, showing roughly similar performance levels to comparable systems using ~5x fewer tokens.

Critical strengths
  • shows good token reduction for MLLM input (45x) while likely improving performance compared to baseline

  • A/B test shows improvement over matched no-subsampling baseline in single setting

  • the motivation of identifying scenes and subsampling constant tokens with each is intuitive

  • good ablations on tuning settings (temporal encoding, vit layers, z reg, annealing)

Critical weaknesses
  • recent system comparisons are hard to place in context and interpret --- although I roughly comparable, nothing is quite apples-to-apples, and all are over a year old. some newer ones are in the appendix, but not the main text

  • A/B test mentioned in the abstract is barely discussed --- shown in Fig 2 but I can't find it described anywhere in the text

  • should compare with simpler vis feature clustering (e.g., clip+temporal with kmeans)

  • tokens come from within-cube subsampling, but cube boundaries are optimized using a full-video pooled feature summary, which is not the same mechanism

  • still requires running frames through the full ViT, limiting gains to just the mllm

Questions for authors
  • l.240 thumbnails: This is a good pooling mechanism, and it indeed enables gradients for z. But the grads seem to now come entirely from the thumbnail, not the cube selection boundaries. The most informative frames for the global thumbnail are what the selection will optimize. This still loosely corresponds to scenes, in that it usually may be best to select one frame per scene; and the momentum construction may make that towards the start of the scene (since middle will have closer to 0 frame-to-frame difference). But it's not directly optimizing the use of the cube sampling itself or the cube boundaries --- these just happen to follow. It would be good to have some discussion or even measurements around this point.

  • Along these lines, sec 5.3 says for "visual lag" that "We posit this mechanism enables the model to retain partial memory of preceding scenes to facilitate current scene understanding". However, I think this could be as easily explained just by the frame sampler for thumbnail pooling choosing intermediate frames. They would always tend to be frames close to a scene cut start due to momentum mlp input for z --- which frame is according to the thumbnail pooling, which can be any frame. I don't think it is necessarily aligned to anything about the cube subsampling as implied in this explanation.

  • another simple comparison/ablation would be to use the selected frames (before temporal pooling into a single thumbnail) as the input to the vlm, removing the cube frames subsampling entirely

  • Table 2 comparison systems are all from 2024, and would be good to show newer models for comparison as well. Especially comparable ones in terms of param count and training data. If newer ones operate in a less-comparable regime, please say so in the text; but I think they would be good to include for context as well.

  • l.228 Resampling subsection: This could use a little more detail in its summary of how resampling is done. Currently understanding it depends on reading Yao et al 2024 --- it would be clearer to read if there was a more stand-alone summary of the method here, with reference only needed for smaller details.

  • fig 4a and l.368 analysis section: this is difficult to interpret without a baseline: is poorer performance on these domains due to problems introduced by this subsampling technique, or was it existing already in the base models?

  • do learned keyframes correspond to scene cuts or other scene cut detecting methods? some anecdotal evidence in fig 5 shows they may not. it would be interesting to see a more quantitative comparison

  • l.409 fig 4 should be fig 5.

  • minor grammatical/agreement issues in several places, e.g.: l.193 "For compatible with online ..." -> "For compatibility with", l.196 "the cubing network calculate" -> "calculates", "A 2-layers MLP with LayerNorm (Ba et al., 2016) is then applied to the mean of visual tokens momentums", many others

Reviewer_CGvFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Quicksviewer, a video understanding framework that dynamically partitions video frames into variable-length segments using a Gumbel-Softmax-based cubing mechanism. A spatiotemporal resampler compresses these segments into compact visual representations. this operation reduces the number of tokens processed by the LLM. The authors also propose a Gumbel-noise annealing strategy to stabilize training. Experiments show competitive video understanding performance with good visual token compression and reduced inference latency.

Critical strengths
  1. Interesting and practically relevant approach to video compression. The proposed mechanism addresses the nonuniform temporal redundancy of videos, allowing the model to allocate visual representations according to content changes. This is a promising direction for efficient long-video understanding.

  2. Promising efficiency and downstream performance. Quicksviewer achieves competitive results on several video understanding benchmarks while substantially reducing the number of visual tokens processed by the LLM. The controlled pretraining comparison against fixed partitioning also provides preliminary evidence supporting the benefits of adaptive cubing.

Critical weaknesses
  1. Insufficient controlled evaluation of adaptive cubing. Although the paper reports competitive performance against existing video LMMs, these comparisons involve different backbones, training datasets, and optimization procedures. The controlled comparison against fixed temporal partitioning is limited to an early training stage, rather than the complete training pipeline. Consequently, it remains unclear how much of the final performance improvement can be directly attributed to the proposed adaptive cubing mechanism.

  2. Unclear online applicability and limited end-to-end efficiency gains. The method claims to support online video processing, yet the proposed global top-k selection of cube boundaries appears to require access to the complete sequence of frame scores. Am I wrong? This would raise concerns about its applicability to strictly causal streaming scenarios. Furthermore, since cubing is performed after visual encoding, the reported 45x compression ratio primarily concerns the visual token sequence rather than the overall computational cost. These issues weaken the practical efficiency claims of the paper.

Questions for authors
  1. Could the authors provide a fully controlled comparison between adaptive cubing and fixed temporal partitioning after the complete training pipeline, including supervised fine-tuning, while keeping the backbone, training data, and visual token budget identical? This would help establish whether the proposed mechanism is responsible for the reported downstream improvements.

  2. How can the global top-k selection of cube boundaries be implemented in a strictly causal streaming setting without accessing future frames? Can the authors clarify whether the reported results correspond to offline or online processing and provide evidence supporting the claimed streaming capability?

  3. Since all sampled frames are processed by the visual encoder before cubing, can the authors provide a detailed breakdown of the overall computational cost and latency, including visual encoding, cubing, resampling, prefill, and decoding? This would help clarify the practical significance of the reported 45x compression ratio.

Reviewer_Y8duOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Quicksviewer employs momentum estimation based on frame feature differences to predict variable-length video cubes; it then uses a resampler—incorporating 3D positional encoding—to compress each video cube into 64 tokens.

Critical strengths
  1. compression operations are applied at the computational bottleneck. Grouping along the temporal dimension and resampling in the spatial domain, performed before the data is fed into the language model, help reduce the subsequent context length.
  2. the paper is well-structured and logically coherent.
Critical weaknesses
  1. there is an inconsistency in the mathematical definition of the core training mechanism. In Equation (3), if tau is a shared temperature coefficient, the expression within the parentheses can be simplified; specifically, any common additive term inside the softmax function would cancel out, rendering the equation incapable of achieving the intended temperature-scaling effect.

  2. in Section 3.1, sigmoid(w_1 - w_0) is initially interpreted as the probability of a keyframe, yet a top-k operation is subsequently applied along the first dimension. The text should clarify whether logits or probability values ​​are being used here, and explicitly define the indexing convention for the categories.

  3. the current experiments regarding the initial training phase are insufficient to account for the performance gains observed in the final model. The fixed-grouping baseline in Figure 2 is evaluated only under the S0 setting, while the setup in Table 3 utilizes an initialization based on an image-pretrained Llama 3.1, which is inconsistent with the Qwen 2.5 backbone and the three-stage video training regimen employed for the main model.

Questions for authors
  1. please provide the actual formulas used in the code for operations such as Gumbel sampling.

  2. given full alignment, what are the performance gains of this learnable cube-based grouping approach compared to fixed uniform grouping or grouping based on simple feature differences?

  3. computing top-k over the entire sequence requires scores from subsequent frames as well as the total number of cubes. How does the model implement this process during online inference?

1 comment

Weakness-2

Thank you for this suggestion. We agree that “the first dimension” is ambiguous. Following the binary gating formulation of Herrmann et al. [1], our implementation selects the class-1 probability after Gumbel-Softmax, then applies top-k across candidate frames. Here, class 1 means the second component under zero-based indexing.

The relevant code is:

# logits: (1, N, 2), where N is the number of candidate frames
probs = gumbel_softmax_sample(logits, temperature, lr_gumbel)
boundary_scores = probs[:, :, 1]  # Class 1: boundary
_, indices = boundary_scores.topk(k=topk, dim=-1)  # Temporal axis

We will revise the passage in Section 3.1 to explicitly define the categories and selection procedure:

For each candidate frame ii, the cubing network outputs two logits zi=[wi,0,wi,1]\mathbf{z}_i=[w_{i,0},w_{i,1}], where class 0 denotes a non-boundary and class 1 denotes a boundary. Before perturbation and temperature scaling, the binary gate probability is pi=σ(wi,1−wi,0)p_i=\sigma(w_{i,1}-w_{i,0}). The score used for boundary selection is

\left[ \operatorname{softmax}\left( \frac{\mathbf{z}_i+\eta_t\mathbf{g}_i}{\tau} \right) \right]_1,$$ > where $\cdot_1$ denotes category index 1, $\mathbf{g}_i$ contains independent standard Gumbel samples, $\eta_t$ is the noise scale, and $\tau$ is the relaxation temperature. We select boundary indices by $$\mathcal{I}= \operatorname{TopK}\left( [q_i]_{i=2}^{N_F},\,N_Q-1 \right),$$ > where TopK returns the temporal indices of the largest scores. The first frame is always included as a boundary. At inference, $\eta_t=0$. This revision makes clear that top-k operates on the class-1 probability scores across time. Thank you for helping us clarify the description. **References** - [1] Herrmann, Charles, Richard Strong Bowen, and Ramin Zabih. "Channel selection using gumbel softmax." ECCV, 2020.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.