acceptodds
Under review as a conference paper at ICLR 2027

COMPASS: Structured Evidence Selection for Long-Video Understanding

Abstract

Long-video understanding with Multimodal Large Language Models (MLLMs) remains challenging because raw video sequences contain far more visual information than can be accommodated within limited frame or token budgets. Existing question-driven frame selection methods typically rank frames independently based on query-frame similarity or temporal statistics, often selecting redundant frames, overlooking critical transitions, and failing to allocate visual evidence effectively across temporally complex videos. We propose COMPASS (Content-aware Multi-granularity Perception and Adaptive Semantic Selection), a deterministic and training-free framework for question-driven long-video understanding. COMPASS treats keyframe selection as a structured evidence allocation problem by jointly modeling query intent, video content structure, and temporal granularity. It derives the temporal scope and information requirements of a query, identifies coherent and informative regions, and performs set-level selection to balance query relevance, semantic coverage, and temporal continuity. A hierarchical allocation strategy further concentrates frame budgets on salient temporal windows while preserving global context. We evaluate COMPASS on LongVideoBench, Video-MME, LVBench, and MLVU under matched frame budgets. Across diverse visual encoders and downstream MLLMs, COMPASS consistently improves reasoning accuracy over existing selection strategies. On LongVideoBench with Qwen3-VL-8B, it achieves a 10.4% relative improvement over uniform sampling under identical frame budgets, with only 8.6 ms of CPU selection latency. These results demonstrate the effectiveness of structured, query-aware evidence allocation for efficient long-video understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Long-Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_w6egOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents COMPASS, a deterministic, training-free framework for question-driven keyframe selection in long-video understanding under a fixed frame budget. It parses the query into a structured sampling plan, partitions the video into content-coherent scenes, and selects frames through hierarchical window allocation and submodular optimization with a per-window (1 − 1/e) guarantee. Evaluated on four long-video benchmarks with three MLLM backbones, it reports 9.8%–10.4% average relative gains over uniform sampling at 8.6 ms CPU latency. However, its margins over the strongest baselines are modest and partly confounded by mixed embedding models, and the framework relies on a stack of hand-crafted rules and hyperparameters.

Critical strengths

S1. Complete empirical evidence for the core claim. COMPASS is evaluated on 4 benchmarks × 3 MLLM backbones × 11 baselines, with full component ablations and cross-backbone transfer, which supports the central claim that structured, set-level selection beats pointwise heuristics in essentially every configuration (9.8%–10.4% average relative gains).

S2. Practical value with bounded theoretical guarantee. The framework is training-free, deterministic, model-agnostic, and runs at 8.6 ms CPU latency; the (1 − 1/e) submodular guarantee is stated with its scope limits, and the released code matches the paper and runs as described. This makes the method genuinely adoptable as a plug-and-play front-end.

Critical weaknesses

W1. Margins over the strongest baselines are modest, and the main table mixes different embedding models, so the headline numbers are not cleanly attributable to the method. On Qwen3-VL-8B, COMPASS leads the strongest baseline by only around one point on most benchmarks and is slightly below it on LVBench. More importantly, the main table compares methods using different visual encoders (CLIP / LongCLIP / BLIP / BLIP-ITM / SigLIP / MLLM-7B); COMPASS's best rows use LongCLIP and are therefore not feature-parity comparisons against most baselines. The authors mitigate this with a controlled-encoder experiment in the appendix, but the headline relative-improvement narrative is partly attributable to encoder capacity rather than the selection strategy itself. Since the paper's contribution is precisely a better selection strategy, this weakness bears directly on how the contribution should be measured.

W2. The framework is a stack of hand-crafted rules and hyperparameters, and the theoretical guarantee does not cover the full pipeline, which limits elegance and generality. IQP is a manually written keyword/regex lexicon (I verified in the released code: roughly 60 hard-coded word groups, with hybrid as the fallback when nothing matches); scene detection, the richness weights, the three discrete λ mappings, the NMS window, the minimum frame count, etc., are all hand-tuned constants. The submodular guarantee, as the authors state, does not apply to the final set after cross-window pooling, NMS, and backfilling: the formal guarantee covers an internal subroutine, while the actual output carries no guarantee. The system's complexity means each component's failure modes are hard to predict (e.g., an IQP misclassification propagates down the pipeline), which sits in tension with the paper's "single deterministic pass" framing.

Questions for authors
  1. Encoder alignment. The main table places methods with different embedding models side by side. Could you provide a full comparison against RIDGE / AdaQ / FOCUS on all four benchmarks with the embedding model unified (e.g., all LongCLIP or all CLIP)? What exactly are the gaps in the controlled experiment of Appendix D.1? How much of the +9.8%–10.4% in the main table is attributable to the method rather than the encoder?
  2. IQP coverage and failure modes. What is the distribution of IQP-assigned scopes on each of the four benchmarks? What fraction of questions miss every keyword and fall back to the hybrid default? How sensitive is final accuracy to IQP misclassification (e.g., local misrouted to global)? (The code falls back to hybrid by default; I would like to know whether this introduces systematic bias.)
  3. Provenance of hyperparameters. On which benchmark(s) were k_scene, β, the λ mappings, τ_nms, and K_min determined? Are the reported results on all four benchmarks produced with a single frozen set of hyperparameters? If they were tuned per benchmark, please state so explicitly.
  4. Coverage of the released code. The anonymous repository currently contains an end-to-end pipeline only for LongVideoBench + Qwen3-VL-8B (Stage 1 registers only an LVB loader; Stage 3 has only the Qwen3 inference script; eval covers only LVB), and the feature-extraction script provides only CLIP/SigLIP loaders, not the LongCLIP path used for the paper's headline numbers. Do the authors commit to releasing scripts for the remaining three benchmarks, the remaining backbones, and LongCLIP feature extraction in the camera-ready version?
  5. Figure 1(a) text overflow: the label "isolated peak → adjacent frames" exceeds the width of its panel and collides with the panel border (verified by high-resolution rendering; the final "s" of "frames" touches/crosses the rounded-box boundary).
  6. Inconsistent table font sizes: The three main tables use noticeably different body font sizes, with Table 3 small enough to impair readability.
1 comment

Dear Reviewer w6eg,

Thank you again for your review. If you have a chance before the discussion closes, we would appreciate hearing whether the shared-encoder comparisons in Table 2, the clarification of the approximation guarantee in Section 3.4 and Appendix A, and the IQP sensitivity tests in Appendix D.5 address your main concerns. We would be happy to clarify any points that remain open.

Thank you for your time.

Reviewer_9fnuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free keyframe selection framework for long-video understanding. It combines rule-based query planning, scene partitioning, hierarchical budget allocation, and submodular selection to balance relevance and coverage. The paper reports the results across four benchmarks and multiple downstream models, supported by matched-scorer comparisons and component ablations.

Critical strengths

The paper proposed a reasonable approach to reducing redundant selections from frame-wise ranking. The method requires neither training nor repeated downstream model calls, making it easy to integrate into existing video understanding pipelines. Experiments cover multiple benchmarks and models, and the authors provide an anonymized code repository link to support reproducibility.

Critical weaknesses
  1. The figures and text descriptions do not match. Figure 1(b) shows 4 phases, but the description in lines 80-85 of the text refers to different components. Figure 2 shows 3 stages, and there is no corresponding relationship. The presentation of the method is somewhat chaotic, and the core content of the method is not clearly explained.

  2. There are too many textual descriptions in the framework diagram of Figure 2, failing to highlight the key points of the method. The method section from Section 3.2 to Section 3.5 mainly consists of stacking formulas, without elaborating on the motivation and role of each component proposed.

  3. In formula 1, why are all these values hardcoded? There are too many hyperparameters, and it cannot be implemented as dynamic adjustment?

  4. Section 3.2 states that IQP generates a selection plan using keywords and regex rules, but does not clearly explain how C needs affects frame selection or how τ time is set when a question contains no explicit temporal information.

  5. With candidate frames at least one second apart (Section 3.5), reducing the NMS threshold from 0.5 to 0.2 seconds should leave the selection unchanged, but Table 6 reports different accuracies.

  6. The main text highlights an 8.6 ms CPU selection time, but does not specify the video duration, candidate-frame count, or whether visual feature extraction is included.

Questions for authors

Refer to weaknesses.

Reviewer_xcbfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes COMPASS, a deterministic, training-free framework for question-driven frame selection in long-video understanding. It combines rule-based query planning, content-adaptive scene partitioning, hierarchical budget allocation, and submodular selection to balance query relevance, semantic coverage, and temporal coverage. Experiments across four benchmarks and multiple downstream models report competitive frame-selection baselines.

Critical strengths
  1. The method balances coverage of the full video with denser sampling around relevant events.

  2. The evaluation covers multiple models and encoders. Matched-scorer comparisons and component ablations help identify where the gains come from.

Critical weaknesses
  1. Some experimental results need clarification. Under the stated 1 FPS sampling and deduplication procedure, why do NMS thresholds of 0.2 and 0.5 seconds produce different accuracies in Table 6? Also, the uniform-sampling results of 57.5 and 60.1 on the two LVB subsets. Could the authors explain whether these experiments use different configurations or samples?

  2. The distinction from the closest methods remains unclear. MDP³ already models set-level diversity and sequential structure, while RIDGE uses question-aware budgeting. Could the authors explain more precisely what COMPASS adds beyond these methods and which results support that contribution?

If the authors satisfactorily clarify these issues, , I would be willing to raise my score.

Questions for authors

See above

1 comment

Dear Reviewer xcbf,

Thank you again for your review. If you have a chance before the discussion closes, we would appreciate hearing whether our rebuttal and revisions address your concerns about the NMS results, the LongVideoBench evaluation splits, and how COMPASS differs from MDP³ and RIDGE. We would be happy to clarify any points that remain open.

Thank you for your time.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Revisions in Response to Reviewer Comments

We thank all three reviewers for their detailed feedback. We have revised the manuscript, added controlled experiments, corrected the NMS results, and expanded the anonymous code release. Below we summarize the changes and their correspondence to the reviewers' concerns. All section, figure, and table numbers refer to the revised manuscript.

1. Controlled encoder comparisons and methodological distinctions (w6eg W1/Q1; xcbf W2). We added Table 2 and its discussion in Section 4.2.1, comparing eight competing selectors with COMPASS under shared CLIP and LongCLIP encoders across all four benchmarks using Qwen3-VL-8B at K=64. COMPASS achieves 66.5% mean accuracy with CLIP and 67.0% with LongCLIP, exceeding the strongest competing selector by 1.3 and 0.9 percentage points, respectively. Switching COMPASS from CLIP to LongCLIP adds approximately 0.5 points. Appendix D.1 and Table 6 provide complementary matched-scorer comparisons. Sections 3.2–3.5 explain the roles of query-guided windows, scene descriptors, richness-based budgeting, and within-window set selection. Table 3 supports these components through ablations.

2. Consistent architecture and clearer component explanations (9fnu W1/W2; w6eg Q5/Q6). Figures 1(b) and 2, their captions, the Introduction, and the Section 3 overview now consistently describe three stages: Perception and Planning, Topological Structuring, and Selection and Reconciliation. Downstream MLLM reasoning follows the selection pipeline. Figure 2 emphasizes the data passed between stages, and Sections 3.2–3.5 introduce each component's motivation and role before its equations. We also corrected the overflowing label in Figure 1(a) and enlarged the text in Tables 3 and 4.

3. IQP semantics, coverage, and sensitivity (9fnu W3/W4; w6eg W2/Q2). Section 3.2 explains how predicted query scope selects among three predefined weight vectors, how content need informs the detail parameter, and how the temporal constraint defaults to the full timeline when no temporal cue is present. Section 3.5 describes how these fields affect window construction. New Appendix D.5 reports scope distributions and perturbation tests in Tables 10 and 11. The macro-averaged rule-hit and fallback rates are 84.7% and 15.3%. Forced Local-to-Global routing reduces mean accuracy by 1.3 percentage points; assigning every query the static Hybrid plan yields 65.4%, still 4.3 points above Uniform. These experiments quantify sensitivity to forced routing changes; they do not estimate actual scope-classification error rates.

4. Scope of the theoretical guarantee (w6eg W2). Sections 3.1 and 3.4, Theorem 1, Appendix A, and Figure 2 explicitly restrict the (1 - 1/e) approximation guarantee to greedy selection within a fixed window, with fixed candidates, anchors, temporal grid, and quota. We clarify that this bound does not establish an end-to-end guarantee after window allocation, pooling, NMS, and budget reconciliation.

5. Corrected NMS results and clarified evaluation splits (xcbf W1; 9fnu W5; w6eg Q3). Section 3.5 and Appendix D.3 clarify that the 0.2 s and 0.5 s NMS thresholds yield identical final frame selections under the stated sampling protocol. Revised Table 7 corrects both settings to 67.6% on LongVideoBench and 77.2% on MLVU. Appendix D.3 and Table 8 distinguish the video-disjoint LVB-dev and LVB-held-out splits, explaining the Uniform scores of 57.5% and 60.1%. The default settings in Sections 3.2, 3.5, and 4.1 are distinguished from the separate 16-configuration calibration on LVB-dev. The calibrated settings are frozen before evaluation on the held-out split and the other three benchmarks; both configurations round to 67.0% mean accuracy.

6. Selection-latency protocol (9fnu W6). Section 4.1, Appendix E.2, and Figure 7(b) specify that 8.6 ms is the mean CPU selection time per query on LongVideoBench at K=32, starting from precomputed visual features and excluding feature extraction. Appendix E.2 reports mean/median video durations of 146/92 seconds and corresponding candidate-pool sizes of approximately 146/92 frames under 1 FPS sampling. We corrected the budget notation from N=32 to K=32.

7. Expanded code release (w6eg Q4). The anonymous repository now includes feature extraction, selection, Qwen3-VL-8B inference, and evaluation for all four benchmarks, together with LongCLIP extraction and updated reproduction instructions. Scripts for the remaining downstream MLLMs are planned for the camera-ready release. The manuscript links the repository, and the updated README documents the released coverage.

All changes described above are highlighted in blue in the revised manuscript.

AuthorsOpenReview ↗

Locations of Changes in the Revised COMPASS Manuscript

The Point column refers to the numbered items in our revision summary. Blue text marks changes in the revised manuscript.

LocationChangePoint
AbstractClarified that the reported 10.4% relative gain over Uniform is averaged across four benchmarks. Specified that the mean CPU selection latency of 8.6 ms per query was measured on the LongVideoBench validation split with a budget of 32 frames, using precomputed visual features. The revised PDF includes the anonymous repository link.6, 7
§1, Fig. 1Aligned the introduction and figure caption with the three stages of COMPASS. Fixed the overflowing label in Fig. 1(a).2
§3 overview, Fig. 2Clarified the data passed between Perception and Planning, Topological Structuring, and Selection and Reconciliation. Placed downstream MLLM reasoning after the selection pipeline and labeled the approximation guarantee as applying within each window.2, 4
§3.1Clarified that Theorem 1 bounds the surrogate objective for greedy selection within a fixed candidate window.4
§3.2Explained how IQP maps query scope to three predefined weight vectors, how content need informs the detail parameter, and how the temporal constraint defaults to the full timeline when no temporal cue is present.2, 3
§3.3Explained how differences between adjacent frames define scenes and how scene representatives serve as anchors for semantic coverage.1, 2
§3.4, Theorem 1Clarified the relevance, semantic coverage, and temporal coverage terms. Stated the (1−1/e)(1-1/e) greedy approximation guarantee for a window with fixed candidates, anchors, temporal grid, and quota.1, 2, 4
§3.5Explained how the query plan determines candidate windows, how content richness informs frame quotas, and how selections are reconciled across windows. Clarified the effect of NMS under 1 FPS sampling.1, 2, 3, 4, 5
§4.1Documented the default experimental settings. Specified that 8.6 ms is the mean CPU selection time per query on LongVideoBench at K=32K=32, measured from precomputed visual features and excluding feature extraction.5, 6
§4.2.1, Table 2 (new)Compared COMPASS with eight competing selectors under shared CLIP and LongCLIP encoders across four benchmarks using Qwen3-VL-8B at K=64K=64. COMPASS reaches 66.5% and 67.0% mean accuracy, ahead of the strongest competitor by 1.3 and 0.9 percentage points, respectively.1
§4.2.2, Table 3Linked the component ablations to the design choices in §3 and enlarged the table text.1, 2
§4.2.3, Table 4Enlarged the text in the backbone transfer table.2
App. A (new)Added the proof of the greedy approximation bound and clarified that it applies before cross-window pooling, NMS, and budget reconciliation.4
App. D.1, Table 6Clarified how Table 6 complements Table 2 by comparing COMPASS with baselines under the encoders paired with those baselines in the main results.1
App. D.3, Table 7Reported identical results for the 0.2 s and 0.5 s NMS thresholds (67.6% on LongVideoBench and 77.2% on MLVU). Verified that both thresholds yield identical final frame selections for every evaluated query.5
App. D.3, Table 8Clarified the video-disjoint LVB-dev and LVB-held-out splits and their respective Uniform scores of 57.5% and 60.1%. Documented the search over 16 configurations on LVB-dev and evaluation with frozen settings. The default and calibrated configurations both round to 67.0% mean accuracy.5
App. D.5, Tables 10–11 (new)Reported IQP scope assignments and forced scope changes. The macro-averaged rule-hit and fallback rates are 84.7% and 15.3%. Forced Local-to-Global routing reduces mean accuracy by 1.3 percentage points, and assigning every query the static Hybrid plan yields 65.4%, 4.3 points above Uniform.3
App. E.2, Fig. 7(b)Corrected the frame budget notation to K=32K=32 and clarified the timing protocol. Reported mean and median video durations of 146 s and 92 s, corresponding to approximately 146 and 92 candidate frames at 1 FPS.6
Reproducibility Statement, anonymous repositoryReferred readers to the anonymous repository linked in the abstract. Expanded the release to cover feature extraction (including LongCLIP), frame selection, Qwen3-VL-8B inference, and evaluation for all four benchmarks. Updated the README with reproduction instructions and release coverage.7
AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Evidence

Dear Area Chair,

Thank you for your time and for coordinating the discussion. We would like to briefly summarize what COMPASS contributes and what the revision adds.

COMPASS is a training-free frame selector that uses the question and the video's scene structure to allocate a limited frame budget across temporal windows. Within each window, it selects frames jointly to balance question relevance, semantic coverage, and temporal coverage.

The main addition during the rebuttal is the shared-encoder comparison in Table 2. On Qwen3-VL-8B with a 64-frame budget, COMPASS improves mean accuracy across four benchmarks over the strongest competing selector. Table 3 also shows that removing Hierarchical Zoom or replacing richness-based quotas with uniform allocation reduces mean accuracy by 2.5 and 1.2 percentage points, respectively. These results show that the gains persist when the visual encoder is held fixed and quantify the contribution of two design choices within COMPASS.

We also clarified the scope of the fixed-window approximation guarantee in Section 3.4 and Appendix A, corrected the NMS results in Table 7, and updated the abstract to state the conditions under which the latency was measured.

Thank you for considering the revised manuscript and the evidence added during the discussion.

Best regards,
The authors of Submission #9728

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.