acceptodds
Under review as a conference paper at ICLR 2027

Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs

Abstract

Long-video language models cannot look at every frame: an hour sampled once per second is 3,600 images, and a system keeps only a small fixed slice of that pool. Which frames survive that slice is usually treated as a preprocessing detail; we test whether it should be. Published selectors make the comparison hard because they change the frame scorer, the prompt boundary, the resolution policy, and the answering model all at once. We hold each fixed and vary one decision at a time—selection, spatial compression, and reinvestment of the savings—across six training-free selection rules, three long-video benchmarks, and two answering models. Selection is the largest single lever: on LongVideoBench's hour-long bin, eight query-selected frames beat sixteen uniformly spaced ones by 6.9 points, and Orthogonal Matching Pursuit—an unmodified, decades-old sparse-approximation algorithm—matches or comes within a point of every purpose-built selector we compare it against, across all three benchmarks. Compression is close to free: halving each frame's spatial budget at fixed timestamps costs at most 0.44 points. Reinvestment is where that budget turns back into accuracy: spending the freed tokens on twice as many compressed frames, at a measured cost no higher than the original eight, returns a further two to three points; compression only pays off once its savings are spent this way. Along the way, an implementation bug in our own AKS baseline and a 0.07–3.74-point gap between two harnesses running the same published rules at the same budget show why these comparisons need to happen inside one controlled harness rather than across papers.

Primary area:
datasets and benchmarks
Keywords:
long-video understanding, multimodal large language models, frame selection, visual token allocation, keyframe sampling, video question answering, orthogonal matching pursuit, benchmark reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qpYuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is an empirical study of frame selection, spatial compression, and reinvestment of a fixed answerer visual-token budget. It compares six baselines using a shared scorer. The adjustment of the design choices reveals how the token budget should be spent for better long-video understanding performance. During the study, the authors also discovered that "Orthogonal Matching Pursuit" as a central mechanism of token compression/selection works surprisingly well.

Critical strengths
  • A useful controlled decomposition of selection, compression, and reinvestment, with OMP as a strong, simple reference.
  • The experiments breadth looks OK, covering several base models and benchmarks.
Critical weaknesses
  • The paper has clear signs of pure AI writing. I am not objecting to AI writing, but the defensive wording and broken logic make it hard to read and interpret. For example, the abstract even has things like "an implementation bug in our own AKS baseline and a 0.07–3.74-point gap between two harnesses"

  • From Table 2 and Figure 1, it is unclear what is the main conclusions and whether the results are significant

  • As a crucial experiment to verify that token compression has not abandoned useful information, please also apply this to some captioning benchmarks

  • New scientific insight needs to be clarified. The controlled decomposition and OMP baseline are useful. However, the main findings, e.g., query-aware selection and reducing spatial resolution for more timestamps, are largely existing motivations and observations. The paper should clarify which important uncertainty this study resolves and what researchers should do differently because of its findings.

Questions for authors

See above

3 comments
AuthorsOpenReview ↗

Response to Reviewer qpYu

We thank the reviewer for the constructive comments and for finding the decomposition and the OMP reference useful.

W1. Writing. "The defensive wording and broken logic make it hard to read and interpret. For example, the abstract even has things like 'an implementation bug in our own AKS baseline and a 0.07–3.74-point gap between two harnesses'."

Re: The revised PDF is rewritten with a new structure around one question: how a fixed visual-token budget should be spent. The abstract now states the question and three findings; implementation corrections (the AKS fix, the harness gap) are moved to Appendix G. The revised PDF also expands the AI Use Statement to describe how AI tools were used.

W2. Conclusions and significance. "From Table 2 and Figure 1, it is unclear what is the main conclusions and whether the results are significant."

Re: In the submission, Figure 1 was the protocol diagram and Table 2 the reinvestment table; neither stated a conclusion, and some of Table 2's rows had no paired test. In the revision:

  • The new Figure 1 shows the main result: one-moment and multi-moment accuracy for 8 uniform, 8 chosen and 128 small uniform frames at equal cost. The result is significant in four of four pre-registered tests (p ≤ .032; table in the general response).
  • Every new claim carries a paired test and says whether it was pre-registered. Table 12 (Appendix L) lists every pre-registered test with its outcome, including the ones that failed.
  • The old figure and table moved: the protocol diagram is now Figure 2; the reinvestment table is in Appendix K, and its caption says which rows have no paired test.

W3. Captioning. "As a crucial experiment to verify that token compression has not abandoned useful information, please also apply this to some captioning benchmarks."

Re: We agree this is the right check. We could not run it in this period. Section 8 now states it as a limitation: without a captioning benchmark, "compression is close to free" does not show that no information is lost. We plan to add it in the next version.

W4. New insight. "The main findings, e.g., query-aware selection and reducing spatial resolution for more timestamps, are largely existing motivations and observations. The paper should clarify which important uncertainty this study resolves and what researchers should do differently."

Re: We agree that each idea is known on its own. Prior work shows that spending tokens on more, smaller frames helps; it leaves open whether choosing frames still matters then, and what it is good for. Our answer: choosing matters most when the budget buys few frames, and mostly for questions about one moment. In practice (Section 7 of the revision):

  1. Compare selectors at equal visual tokens, not equal frame counts.
  2. Include many small uniform frames at the same token cost as a baseline; with our Qwen models it scored within one point of eight frames chosen by OMP.
  3. Report one-moment and multi-moment questions separately, because an overall score can hide opposite effects.
  4. For models that can shrink frames freely, spend the budget on many small frames, and choose them if a frame scorer is available. On LVBench, 128 chosen small frames were the best input we tested (+5.4 points over 128 uniform ones, p=.0001); on the LongVideoBench 600 s bin, their lead was +1.7 and not significant.

I appreciate the efforts from the authors, but 1/ the current writing quality is still below the bar of this venue; 2/ lacking the captioning result is crucial to justify whether the method in this paper is generalizable; 3/ what the authors have clarified as new insights was not convincing enough for me. Therefore, I keep my current rating of 2.

AuthorsOpenReview ↗

Response to Reviewer qpYu

Thanks for reading the revision and for clearly stating the remaining concerns.

On captioning: we agree it is the most direct test of whether compression loses information. For the next version, we plan to caption videos using a detailed captioning benchmark at fixed timestamps, with full-resolution and half-resolution frames at a matched cost/token budget, and compare caption scores on a per-video basis. Until then, the paper limits its claims to multiple-choice question answering, as stated in Section 8.

On writing and new insights: we take both points and will continue revising the presentation to ensure the new findings are stated more clearly. Thank you again for the time spent on our paper.

Reviewer_WwiUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents an empirical study of visual-token allocation for long-video question answering. It separates the problem into frame selection, spatial compression at fixed timestamps, and reinvestment of the saved budget into additional frames. Using a shared LongCLIP scorer, the study compares six selection rules, including Orthogonal Matching Pursuit (OMP), across three benchmarks. Additional experiments examine different answering models, an alternative scorer, and implementation sensitivity. The main findings are that OMP is competitive with the tested selectors, moderate spatial compression produces small observed accuracy changes, and allocating the recovered budget to more frames can improve performance.

Critical strengths
  1. Clear separation of allocation effects. I appreciate the effort to disentangle temporal frame selection, spatial compression, and reinvestment of the saved token budget. In particular, the fixed-timestamp controls help distinguish improvements from selecting more informative frames from those arising from different spatial allocation strategies.

  2. A competitive and accessible reference baseline. OMP outperforms uniform sampling by 5.69 to 11.81 percentage points across the three benchmarks while remaining within one percentage point of LDDR-select in Table 1. This is a useful finding, as it demonstrates the competitiveness of a simple subset-selection method and provides a meaningful reference for future selector evaluations.

Critical weaknesses
  1. Limited novelty beyond the closest prior work. My primary concern is the extent of the scientific contribution beyond LDDR. The prior work already compares selectors using a shared LongCLIP encoder and evaluation pipeline, and investigates resolution allocation under a fixed visual-token budget. These analyses overlap substantially with the motivation of the present study. Moreover, Section 5.4 explicitly presents the reinvestment result as a replication of an existing observation. While introducing OMP and conducting a more systematic statistical analysis are valuable additions, I am not yet convinced that they lead to fundamentally new insights into visual-token allocation. The contribution would be more compelling if the analysis established a distinct principle or meaningfully revised an existing conclusion.

  2. Limited experimental support for the proposed allocation guidance. The full selector comparison is primarily conducted with eight frames, while the main reinvestment experiment compares eight full-resolution frames against sixteen compressed frames. The scorer-swap analysis covers only Uniform, Top-k, and OMP, and the cross-model experiments largely focus on OMP versus Uniform. Although these experiments provide useful evidence, they do not sufficiently establish whether the findings generalize across different token budgets, scorers, and answering models. More importantly, the recommendation to prioritize frame selection over spatial allocation is derived from comparisons at different operating points, making its broader applicability uncertain. The four-cell experiment in Appendix H partially addresses this concern, but a more comprehensive matched-budget evaluation would strengthen the practical conclusions.

Questions for authors

My main reservations concern the novelty of the scientific insights and the generalizability of the proposed allocation guidance. Clarifying the following points, ideally with additional experimental evidence, would help me better assess the contribution.

  1. Scientific contribution beyond LDDR. Could the authors clarify which conclusions about visual-token allocation are genuinely new relative to LDDR, and identify the strongest evidence that the introduction of OMP or the statistical analysis leads to a different understanding rather than primarily reinforcing existing findings?

  2. Robustness of the allocation guidance. Could the authors provide matched-budget comparisons across multiple frame counts and spatial resolutions, ideally covering the full selector set and additional scorer or answering model configurations, to examine whether the recommended priority of frame selection over spatial allocation remains consistent?

2 comments
AuthorsOpenReview ↗

Response to Reviewer WwiU

We sincerely appreciate your detailed and constructive review, and especially for the clear request for matched-budget comparisons, which shaped our new results.

W1: "Could the authors clarify which conclusions about visual-token allocation are genuinely new relative to LDDR? The contribution would be more compelling if the analysis established a distinct principle or meaningfully revised an existing conclusion."

Re: We appreciate this sharp and perceptive comment. The matched-budget experiment we ran during the discussion period does both.

  • New relative to LDDR. LDDR finds, for its own selector, a best number of tokens per frame (512 beats 1024 and 256 at one total budget). It does not ask whether choosing frames still helps as frames get smaller and more numerous, or which questions each part of the budget serves.
  • A distinct principle. At a fixed budget, choosing frames helps questions about one moment; many small uniform frames help questions that link moments; and overall accuracy hides both. Confirmed in four of four pre-registered tests (two benchmarks, three answering models; table in the general response).
  • A revised conclusion. Our submitted advice, "prioritize selection over spatial allocation", holds only when the budget buys few frames. On LongVideoBench the gain from choosing falls below 2 points at 128 frames for OMP, BOLT and top-k; on LVBench a pre-registered test confirms that it halves (+12.5 to +5.4 points, p=.0001).

We agree that reinvestment alone (Section 5.4 of the submission, Section 5.2 of the revision) is not new.

W2: "Could the authors provide matched-budget comparisons across multiple frame counts and spatial resolutions, ideally covering the full selector set and additional scorer or answering model configurations […]?"

Re: Done for frame counts, selectors and answering models; not for a second scorer. Every new comparison costs the tokens of 8 full-resolution frames (Section 6, Table 2, Figure 3).

What we ran

  • Frame counts and resolutions: 8, 16, 32, 64 and 128 frames at equal cost (the submission had only the four cells of Appendix H).
  • Selectors: uniform, OMP, BOLT, and top-k at every frame count; AKS from 32 frames on.
  • Answering models: Qwen3-VL-8B, InternVL3-8B, Qwen2.5-VL-7B.
  • Benchmarks: LongVideoBench (10-minute and 1-hour bins), LVBench, Video-MME long.

What we found

  • Choosing frames helps one-moment questions on all three answering models. Many small frames help multi-moment questions on Qwen2.5-VL but not on InternVL3, whose fixed tiles allow only 40–72 squashed frames at this cost; we report this as model-dependent.
  • The gain from choosing shrinks with more, smaller frames, for every selector. On LongVideoBench, the gain over uniform from 8 to 128 frames goes from +8.5 to +1.7 for OMP, +5.6 to +0.2 for top-k and +5.3 to +0.5 for BOLT; AKS goes from +7.3 at 32 frames to −0.5 at 128. Top-k's pre-registered test of the drop narrowly misses significance after correction (Holm p=.07), so on LongVideoBench this is a consistent trend, not a significance-tested result.
  • On LVBench (1,344 questions) the pre-registered test passes. OMP's gain falls from +12.5 to +5.4 points (p=.0001) but stays positive at 128 frames (p=.0001). The drop is on one-moment questions (+11.0, p<.0001), not multi-moment ones (+0.8, p=.82).

What we did not cover. AKS has no valid configuration at 8 or 16 frames under its official depth setting (our port falls back to top-k there; Appendix G). FOCUS⋆ and LDDR-select were not rerun, and a second scorer was not tested. Section 8 states these limits.

We hope these changes address your concerns, and we would be glad to clarify anything further.

Thank the authors for the additional experiments and analysis.

I appreciate the efforts made by the authors to dispel my doubts. The evaluation under the same budget provides valuable insights into the trade-off between time selection and space compression, and the revised conclusion statement is more rigorous and appropriate.

However, my main concern about the novelty of this work beyond LDDR still exists. Although supplementary experiments have strengthened the empirical analysis, I am still not fully convinced that this work can establish sufficiently unique scientific contributions.

Besides, the writing of this paper is too hard to follow. And the authors have now substantially rewritten nearly the entire main text. Therefore, I would prefer that they further reorganize and refine the manuscript in light of the reviewers' feedback before submitting it again.

Overall, I acknowledge the improvements made in the paper, but I will maintain my initial assessment.

Reviewer_zncAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper systematically investigates video frame selection strategies for VideoLLMs under controlled settings, examining the effects of frame selection, spatial compression, and token budget reinvestment. Through experiments across six training-free selection methods, three long-video benchmarks, and two answering models, the paper finds that frame selection has the largest impact on performance, with traditional Orthogonal Matching Pursuit (OMP) performing comparably to specialized methods. Furthermore, spatial compression introduces minimal performance degradation, while reinvesting the saved token budget into additional frames improves accuracy.

Critical strengths

The paper provides comprehensive and well-controlled experimental evaluations across several widely used benchmarks and numerous previously published methods. It offers valuable insights into the trade-offs between spatial compression and frame selection, showing that spatial compression incurs minimal performance degradation, while reinvesting the saved token budget into additional frames improves accuracy.

Critical weaknesses
  1. Novelty concern. The paper does not introduce a new research problem or methodology, but primarily focuses on analyzing the limitations of existing approaches. While the empirical findings are valuable, the contribution is closer to an empirical survey or benchmark study than a methodological research paper. The authors may consider submitting it to a more appropriate track.
  2. Incomplete coverage of related methods. Although AKS and FoCUS are representative video frame selection methods, several other relevant approaches [1–3] are not included in the evaluation. It remains unclear whether the paper's conclusions generalize to these methods. A more comprehensive comparison would strengthen the validity and generalizability of the findings.

[1] Liu, Shuming, et al. "Bolt: Boost large vision-language model without training for long-form video understanding." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025. [2] Zou, Yuanhao, et al. "Air: Enabling adaptive, iterative, and reasoning-based frame selection for video question answering." International Conference on Learning Representations. Vol. 2026. 2026. [3] Sun, Hui, et al. "Mdp^3: a Training-Free Approach for List-Wise Frame Selection in Video-Llms." 2025 IEEE/CVF Internati

Questions for authors

Please see Weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer zncA

We thank the reviewer for the careful review and for recognizing the controlled evaluation. We also took the presentation score seriously: the revised PDF has been rewritten with an improved structure around one path.

W1. Novelty. "The paper does not introduce a new research problem or methodology […] the contribution is closer to an empirical survey or benchmark study than a methodological research paper."

Re: We agree that the paper proposes no new selector. Its contribution is a new question and a method to test it. The closest work either fixes the token budget but samples frames uniformly (LoHi), or chooses frames at a fixed frame count (FrameSelect, HAS). None asks whether choosing frames still helps at a fixed token budget, and for which questions. We test this at matched cost, with every input's token count checked against the model's own count, and with pre-registered tests:

  • Choosing frames helps one-moment questions; many small frames help both kinds. Confirmed in four of four pre-registered tests (two benchmarks, three answering models).
  • Overall accuracy hides this. At equal cost, 8 chosen frames and 128 small uniform frames tie overall but win different questions (pre-registered, p=.0001).
  • The gain from choosing shrinks as the budget buys more, smaller frames. On LVBench, a pre-registered test confirms that it halves (+12.5 to +5.4 points, p=.0001).

These results tell a practitioner when selection is worth its cost, and they narrowed our own submitted advice to budgets that buy few frames.

W2. Coverage of related methods. "Several other relevant approaches [BOLT, AIR, MDP3] are not included in the evaluation. It remains unclear whether the paper's conclusions generalize to these methods."

Re: We added BOLT at matched cost, and our main conclusion holds for it. AIR and MDP3 are not included, for the reasons below.

BOLT. Its result depends strongly on its setting, so we report both versions (8 frames, LongVideoBench 10-minute bin):

BOLT versionAccuracyvs uniformOMP's lead
As released (its CLIP-L scorer, power 2.5)56.55+0.97+7.52 (p=.002)
Our shared LongCLIP scorer, power 2.560.92+5.34+3.16 (p=.18)

On LVBench, OMP leads our BOLT port by 7.9 points (p<.0001). The main new conclusion holds for BOLT: on LongVideoBench its gain over uniform shrinks from +5.3 at 8 frames to +0.5 at 128, as OMP's does. We therefore now describe OMP as a strong, simple reference, not a significantly better selector.

AIR and MDP3. AIR calls a vision-language model inside its selection loop, so it spends compute that a visual-token budget does not count, and a comparison at matched visual tokens would not be fair to either method. MDP3 selects frames with a determinantal point process (DPP) over relevance and diversity. Its closest relative in our study, LDDR-select, is also a DPP selector; it is in Table 1, within about one point of OMP. We did not port MDP3 itself, because no published number at our setting lets us check a port. Section 8 states both as limitations.

We hope these changes address your concerns, and we would be glad to clarify anything further.

Comments on the paper

1 comment

Dear Reviewers, ACs, PCs, and SACs,

We thank all reviewers for their time and careful feedback. We appreciate the recognition of the controlled decomposition of selection, compression, and reinvestment (WwiU, qpYu), the well-controlled evaluation across benchmarks (zncA), and OMP as a strong, simple reference (WwiU, qpYu).

All three reviewers raised one shared concern: the paper does not establish a distinct principle. WwiU added that our guidance was based on comparisons across different operating points. We agree, and we used the discussion period to run the experiment WwiU asked for: comparisons at a matched visual-token budget across frame counts and resolutions. Each run below consumes the tokens from 8 full-resolution frames, verified against the model's own count. The main tests were pre-registered before the runs. Code, pre-registrations, and per-item predictions are in the anonymous repository: https://anonymous.4open.science/r/long-video-token-allocation-6BD6/

What we found.

  1. Choosing frames helps one-moment questions; many small frames help both kinds. Choosing helps questions about one moment (LongVideoBench's perception questions) and adds little to questions that link moments, while many small uniform frames help both. Pre-registered and confirmed on 3 answerers and 2 benchmarks:
Gain from choosing (OMP − uniform, 8 frames)One-momentMulti-momentDifference
LongVideoBench 3600 s bin, Qwen3-VL-8B+14.5+1.5p=.003
same, InternVL3-8B+11.5+2.4p=.032
same, Qwen2.5-VL-7B+16.7−2.1p<.0001
LVBench, Qwen3-VL-8B+16.8+5.5p=.0003
  1. Overall accuracy hides this. At the same cost, 8 chosen frames and 128 small uniform frames tie overall, but the chosen frames win one-moment questions, and the small frames win multi-moment ones (pre-registered on Qwen2.5-VL, p=.0001). Full resolution alone does not explain the gain: on one-moment questions of the 600 s pilot, 8 sharp uniform frames score below 128 small ones (58.5 vs 65.2), while choosing the 8 sharp frames gives 73.9.

  2. This revises our own conclusion. The gain from choosing frames shrinks as the same budget is spread over more, smaller frames. On LongVideoBench, OMP's lead over uniform falls from +8.5 points at 8 frames to about 0 at 64–128 frames, and BOLT and top-k show the same trend; this met the direction rule we fixed before the pilot, without a significance test. On LVBench we pre-registered a test of OMP's drop, and it passes: the lead halves, from +12.5 points to +5.4 (p=.0001), but does not vanish (p=.0001 at 128 frames); there, 128 chosen small frames are the best input we tested. Our submitted advice, "prioritize selection over spatial allocation", holds only when the budget buys few frames.

We also report what did not hold: a mix of 4 chosen full-resolution frames and 64 small frames failed its pre-registered test (it was never significantly worse than the better of the other two inputs on a Qwen model, but beat both only on LVBench, in a secondary analysis), and small frames did not help multi-moment questions more than one-moment ones (pre-registered, failed).

We have uploaded a revised PDF. In particular, we have:

  • Added the matched-budget study as a new Section 6, with Table 2 (three answering models, two benchmarks) and a new Figure 1 showing the main result (zncA, WwiU)
  • Added the matched-budget protocol, question levels, and pre-registration (Section 3.5)
  • Added frame counts from 8 to 128 at equal cost, with more selectors and the pre-registered LVBench test (Section 6.3, Figure 3) (WwiU)
  • Reported the pre-registered results that did not hold (Section 6.4)
  • Positioned the paper against LoHi, LDDR, BOLT, FrameSelect and HAS (Section 2) (zncA)
  • Rewritten the abstract, introduction, discussion, and conclusion around one question(Sections 1, 7, 9) (qpYu)
  • Listed every pre-registered test with its outcome (Appendix L, Table 12) (qpYu)
  • Moved implementation corrections (AKS depth, harness gap) from the abstract to Appendix G (qpYu)
  • Expanded the AI Use Statement (qpYu)

We are happy to clarify anything further. If these results address your reservations, we would be very grateful if you would consider updating your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.