- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes MEC (matryoshka evidence-to-content), a training-free long video key frame selection method which uses only one time selection process for different frame budgets, i.e number of frames 8/16/128/256, etc. The main motivation of the paper is budget-independent frame ranking: low-budget models can use query-evidence frames; for higher budgets, they can progressively select context and diversity frames.
Overall, in the paper, MEC has been evaluated on multiple benchmarks, i.e, Video-MME, Video-MME-v2, LongVideoBench, and MLVU. MEC achieves ~3.77% average improvement compared to uniform sampling and 47.37-51.19% lower selection latency.
Critical strengths
The proposed nested frame selection method is intuitive and solves the most critical problem in existing long-video frame selection methods: the need to rerun the model for different frame budgets. MEC run the selector only once and uses it for different frame budgets, motivated by MRL[1] training. The authors also provide a detailed analysis of temporal coverage and visual diversity, and they argue that temporal coverage matters more than any other component. Overall, the reported performance improvements, i.e 54.15% -> 62.45% over uniform sampling on LongVideoBench, prove the effectiveness of the paper.
[1] Aditya Kusupati et al. Matryoshka Representation Learning. URL: https://arxiv.org/abs/2205.13147
Critical weaknesses
Even though the nested ranking looks intuitive, it lacks overall algorithmic novelty. Most of the methods used are already well known, i.e MRL[1], temporal coverage, and evidence ranking. The paper only uses these and positions them in the paper.
Another important weakness is that the paper only performs benchmarks against uniform sampling, AKS and WFS-SB, but there are many other recent works for frame selection, i.e FOCUS[2], AKeyS[3], FrameThinker[3]. Without direct comparison with these methods, it would be difficult to assess whether MEC is SOTA or not at this stage,
[1] Aditya Kusupati et al. Matryoshka Representation Learning. URL: https://arxiv.org/abs/2205.13147
[2] Zirui Zhu et al. FOCUS: Efficient keyframe selection for long video understanding. In International Conference on Learning Representations, 2026. URL: https://arxiv.org/pdf/2510.27280
[3] Sunqi Fan et al. Agentic Keyframe Search for Video Question Answering. URL: https://arxiv.org/abs/2503.16032
[4] Zefeng He et al. FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlighting URL: https://arxiv.org/abs/2509.24304
Questions for authors
Request the authors to provide a detailed assessment and comparison with FOCUS[1], AKeyS[2], FrameThinker[3] or any other recent relevant methods. I understand that all the methods may not use the same backbone, and reproducible code may not be available. Where possible, try to compare your model on the same backbone, i.e Qwen3.5-VL, with the same frame budget and report the performance in terms of accuracy as well as the end-to-end latency. This benchmark is very important to position your paper. if you can provide a detailed comparison report and your method performs well, I will be happy to raise my score.
[1] Zirui Zhu et al. FOCUS: Efficient keyframe selection for long video understanding. In International Conference on Learning Representations, 2026. URL: https://arxiv.org/pdf/2510.27280
[2] Sunqi Fan et al. Agentic Keyframe Search for Video Question Answering. URL: https://arxiv.org/abs/2503.16032
[3] Zefeng He et al. FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlighting URL: https://arxiv.org/abs/2509.24304
6 comments
W1. Algorithmic Novelty of Matryoshka Keyframe Ranking
While Matryoshka Representation Learning (MRL) and individual selection cues (relevance, temporal coverage, diversity) are established building blocks, translating the nested-prefix principle from continuous representation learning to discrete, training-free subset selection introduces distinct structural challenges. Whereas MRL optimizes nested dimensions of a continuous vector during training, long-video frame selection operates at inference time over discrete observations whose utility is inherently set-dependent—a frame's marginal value depends on what is already included in the prefix. Moreover, a single shared ranking creates positional asymmetry across prefixes: early positions are exposed to all downstream budgets and must secure diagnostic query evidence, whereas later positions serve larger budgets by contributing complementary temporal context and visual diversity without displacing earlier selections.
To our knowledge, MEC is the first framework to formulate long-video keyframe selection as a Matryoshka ranking problem. Addressing the challenges above led to our core technical design: a set-aware, position-adaptive evidence-to-context greedy surrogate, supported by a reusable sparse video index and query-conditioned local zooming that avoid costly full-video dense scoring.
Addressed some of my concerns - ablation on performance under the same frame budgets
Thanks to the authors for addressing my concerns in the rebuttal. Based on the accuracy comparison under the same frame budget and using the same selector, MEC demonstrates superior performance over the existing methods. I also appreciate the additional comparison with AKeyS and FrameThinker reported in the rebuttal.
My concern remains around the end-to-end latency. Compared to FOCUS, MEC improves avg accuracy by 2.16 pp on Video-MME and 1.35 pp on LongVideoBench; however, the improvements come at a cost of ~10% higher end-to-end latency and ~20% higher selector latency. I will request the author to provide the following additional ablations
- Please report the end-to-end comparison latency for each of the frame budgets, i.e 8/32/64/128 separately.
- If possible, compare MEC and FOCUS under the same end-to-end latency budget rather than only frame budgets
Thank you very much for taking the time to review our paper, for recognizing the practical value of MEC, and for your encouraging support in offering to raise your score. We try our best to address your questions as follows.
Q & W2. Comparisons with Recent Methods in Accuracy and Latency
Accuracy comparison under a unified protocol. Following your suggestion, we evaluated FOCUS (ICLR 2026), QCA (ECCV 2026), and AdaQ (2026) alongside AKS (CVPR 2025) and WFS-SB (CVPR 2026) under a strictly unified protocol. All methods use the exact same frozen Qwen3.5-9B answerer, evaluation samples, target frame budgets (K∈{8,32,128}), downstream preprocessing, prompts, and deterministic decoding across the full Video-MME test set (2,700 questions) and LongVideoBench validation set (1,337 questions).
Table A. Accuracy (%) across three frame budgets on Video-MME and LongVideoBench.
† For transparency, AdaQ uses its official sampling procedure adapted to fixed K with a predefined seed. At K=128, its candidate pool is smaller than K for 33.22% of Video-MME and 27.00% of LongVideoBench questions.
MEC achieves the highest accuracy at every individual frame budget and the best three-budget average on both Video-MME (68.90%) and LongVideoBench (64.80%). Specifically, MEC improves average accuracy over FOCUS by +2.16 pp on Video-MME (reaching +3.63 pp at K=128) and +1.35 pp on LongVideoBench, while outperforming QCA by +0.59 / +1.45 pp and AdaQ by +1.68 / +2.42 pp on Video-MME / LongVideoBench, respectively.
Selection efficiency and end-to-end latency. We also profiled cumulative wall-clock latency on an identical 18-video workload sampled from Video-MME (one question per video across all three budgets K∈{8,32,128}).
Table B. Cumulative multi-budget workload latency across 18 videos (seconds, lower is better).
Note: Selector Time measures the total wall-clock duration across all 18 videos for generating outputs across all three budgets, including initialization, preprocessing, feature extraction, scoring, selection, and local I/O. Full Workload Time additionally includes three separately initialized downstream QA inference processes.
MEC reduces selector latency by 54.35% vs. QCA and 33.21% vs. AdaQ, cutting full-workload time by 30.09% and 11.37%, respectively. While FOCUS is slightly faster in raw selection time (201.71 s vs. 242.79 s), MEC delivers substantially higher accuracy (+2.16 pp on Video-MME and +1.35 pp on LongVideoBench) with modest additional overhead. Crucially, MEC constructs a single budget-independent ranking whose prefixes directly serve all target budgets, whereas competing selectors—even when caching intermediate features—still require budget-dependent selection for each target K.
Discussion of AKeyS and FrameThinker. We will include the comparisons above and also cite and discuss AKeyS [6] and FrameThinker [7] in the revised manuscript. Because AKeyS relies on caption-assisted agentic tree search and FrameThinker couples iterative multi-turn frame acquisition with an RL-trained policy, we distinguish them from standalone, single-pass visual selectors evaluated under a frozen answerer.
References
[1] Tang et al. Adaptive Keyframe Sampling for Long Video Understanding. CVPR 2025.
[2] Chen et al. Wavelet-based Frame Selection by Detecting Semantic Boundary for Long Video Understanding. CVPR 2026.
[3] Zhu et al. FOCUS: Efficient Keyframe Selection for Long Video Understanding. ICLR 2026.
[4] Peng et al. QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding. ECCV 2026.
[5] Zhang et al. Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Sampling. arXiv:2606.24187, 2026.
[6] Fan et al. Agentic Keyframe Search for Video Question Answering. arXiv:2503.16032, 2025.
[7] He et al. FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlighting. ICLR 2026.