- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents COMPASS, a deterministic, training-free framework for question-driven keyframe selection in long-video understanding under a fixed frame budget. It parses the query into a structured sampling plan, partitions the video into content-coherent scenes, and selects frames through hierarchical window allocation and submodular optimization with a per-window (1 − 1/e) guarantee. Evaluated on four long-video benchmarks with three MLLM backbones, it reports 9.8%–10.4% average relative gains over uniform sampling at 8.6 ms CPU latency. However, its margins over the strongest baselines are modest and partly confounded by mixed embedding models, and the framework relies on a stack of hand-crafted rules and hyperparameters.
Critical strengths
S1. Complete empirical evidence for the core claim. COMPASS is evaluated on 4 benchmarks × 3 MLLM backbones × 11 baselines, with full component ablations and cross-backbone transfer, which supports the central claim that structured, set-level selection beats pointwise heuristics in essentially every configuration (9.8%–10.4% average relative gains).
S2. Practical value with bounded theoretical guarantee. The framework is training-free, deterministic, model-agnostic, and runs at 8.6 ms CPU latency; the (1 − 1/e) submodular guarantee is stated with its scope limits, and the released code matches the paper and runs as described. This makes the method genuinely adoptable as a plug-and-play front-end.
Critical weaknesses
W1. Margins over the strongest baselines are modest, and the main table mixes different embedding models, so the headline numbers are not cleanly attributable to the method. On Qwen3-VL-8B, COMPASS leads the strongest baseline by only around one point on most benchmarks and is slightly below it on LVBench. More importantly, the main table compares methods using different visual encoders (CLIP / LongCLIP / BLIP / BLIP-ITM / SigLIP / MLLM-7B); COMPASS's best rows use LongCLIP and are therefore not feature-parity comparisons against most baselines. The authors mitigate this with a controlled-encoder experiment in the appendix, but the headline relative-improvement narrative is partly attributable to encoder capacity rather than the selection strategy itself. Since the paper's contribution is precisely a better selection strategy, this weakness bears directly on how the contribution should be measured.
W2. The framework is a stack of hand-crafted rules and hyperparameters, and the theoretical guarantee does not cover the full pipeline, which limits elegance and generality. IQP is a manually written keyword/regex lexicon (I verified in the released code: roughly 60 hard-coded word groups, with hybrid as the fallback when nothing matches); scene detection, the richness weights, the three discrete λ mappings, the NMS window, the minimum frame count, etc., are all hand-tuned constants. The submodular guarantee, as the authors state, does not apply to the final set after cross-window pooling, NMS, and backfilling: the formal guarantee covers an internal subroutine, while the actual output carries no guarantee. The system's complexity means each component's failure modes are hard to predict (e.g., an IQP misclassification propagates down the pipeline), which sits in tension with the paper's "single deterministic pass" framing.
Questions for authors
- Encoder alignment. The main table places methods with different embedding models side by side. Could you provide a full comparison against RIDGE / AdaQ / FOCUS on all four benchmarks with the embedding model unified (e.g., all LongCLIP or all CLIP)? What exactly are the gaps in the controlled experiment of Appendix D.1? How much of the +9.8%–10.4% in the main table is attributable to the method rather than the encoder?
- IQP coverage and failure modes. What is the distribution of IQP-assigned scopes on each of the four benchmarks? What fraction of questions miss every keyword and fall back to the hybrid default? How sensitive is final accuracy to IQP misclassification (e.g., local misrouted to global)? (The code falls back to hybrid by default; I would like to know whether this introduces systematic bias.)
- Provenance of hyperparameters. On which benchmark(s) were k_scene, β, the λ mappings, τ_nms, and K_min determined? Are the reported results on all four benchmarks produced with a single frozen set of hyperparameters? If they were tuned per benchmark, please state so explicitly.
- Coverage of the released code. The anonymous repository currently contains an end-to-end pipeline only for LongVideoBench + Qwen3-VL-8B (Stage 1 registers only an LVB loader; Stage 3 has only the Qwen3 inference script; eval covers only LVB), and the feature-extraction script provides only CLIP/SigLIP loaders, not the LongCLIP path used for the paper's headline numbers. Do the authors commit to releasing scripts for the remaining three benchmarks, the remaining backbones, and LongCLIP feature extraction in the camera-ready version?
- Figure 1(a) text overflow: the label "isolated peak → adjacent frames" exceeds the width of its panel and collides with the panel border (verified by high-resolution rendering; the final "s" of "frames" touches/crosses the rounded-box boundary).
- Inconsistent table font sizes: The three main tables use noticeably different body font sizes, with Table 3 small enough to impair readability.
Dear Reviewer w6eg,
Thank you again for your review. If you have a chance before the discussion closes, we would appreciate hearing whether the shared-encoder comparisons in Table 2, the clarification of the approximation guarantee in Section 3.4 and Appendix A, and the IQP sensitivity tests in Appendix D.5 address your main concerns. We would be happy to clarify any points that remain open.
Thank you for your time.