acceptodds
Desk rejected at ICLR 2027

From Relevance to Representatives: Equal-Rate Evidence Quantization for Long Video Understanding

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Long video understanding often requires selecting useful visual observations under a limited frame budget. Uniform sampling distributes observations by elapsed time, whereas global relevance ranking can concentrate them in locally redundant high-scoring regions. In this paper, we propose Equal-rate Evidence Quantization (E²Q), a training-free frame selector for long video understanding that separates where to allocate the budget from which observation to retain. E²Q uses normalized frame–question relevance to allocate the frame budget, partitioning the ordered candidates into consecutive intervals with approximately equal relevance mass and retaining the highest-scoring frame from each interval. A budget-dependent projection handles overly concentrated candidate weights before partitioning, enabling relevance-adaptive intervals without an additional relevance–coverage coefficient. We characterize this construction and bound frame-level representation error in terms of within-interval variation and interval-mass imbalance. Across four long-video question answering benchmarks and three models, E²Q achieves the highest accuracy among training-free selectors in ten of the twelve model–benchmark settings. Controlled shared-score comparisons further show consistent gains when the relevance signal is fixed and isolate the respective contributions of relevance-mass allocation and within-interval representative selection to these gains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long Video Understanding, Keyframe Selection, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_m9ECOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes E2Q, a training-free method for selecting frames from long videos to help models answer questions under a fixed frame number constraint. The method segments the video based on the relevance of each frame to the question. It ensures that the sum of relevance scores within each segment is roughly equal. Then it selects the most relevant frame from each segment. To prevent a small number of high score frames from only a few segments, the method adjusts the weights used for the partitioning process. The paper also provides a theoretical analysis, evaluates performance across four video qa benchmarks and three models, and assesses the quality of the selected frames. The authors show E2Q improvements on four benchmarks, three downstream MLLMs, and a frame-selection benchmark.

Critical strengths
  1. The algorithm description is clear and reasonable. Especially, Proposition 2 demonstrates that under the stated conditions, the upper-bound constraint ensures the sequence of nearest cumulative boundaries is strictly increasing and minimizes the boundary error in Eq. (4). And Corollary 1 shows that each interval contains at most one candidate frame that reaches the upper bound. These explain how projection and partition work together.

  2. This paper not only shows the performance on final qa accuray on benchmarks, but also gives the scores on KFS-Bench for frame selection, it is important since it more directly reflects the quality of frame selection, while final qa accuracy also depends on the backbone model. And Table 3 compares different selection rules with the same candidate frames, relevance scores, backbone model and visual preprocessing. These experiments help to show the effects of partition and representative selection separately.

Critical weaknesses
  1. The authors highlight one of their important contributions is "separates where to allocate the budget from which observation to retain" (L17-20, 85-91, 99-101), but there is already a close related work SLICE [1] with the same core design for long video qa. SLICE also divides the video into intervals with approximately equal relevance mass, then selects the frame with the highest raw relevance score in each interval. One main difference between E2Q and SLICE is how to adjust the frame allocation weights before segmentation. E2Q also provides more detailed partition rules and theoretical analysis, but the capped KL projection itself follows previous work (authors acknowledged it in L194-196). Without a discussion and comparison with SLICE, it is unclear what additional value these differences bring. Based on this missing related work, I am concerned about the contribution and novelty of this work.

  2. This paper does not include a latency analysis. The paper only shows the algorithm complexity after scoring (which is O(KN)), but does not have any end-to-end latency analysis that includes scoring the candidate frames. The efficiency-accuracy balance is pretty important for a frame selection paper because a small performance gain may not justify a large increase in latency. Existing frame selection methods such as BOLT, MDP3, FrameOracle [2] all report the latency. Without this analysis or comparison with existing frame selection methods, I do not think the authors provide enough evidence to judge whether the performance gain is worth the inference cost.

  3. Although Table 1 shows that E2Q has the highest average performance on all existing methods across all three backbone models, the improvements over the second best models are small (1.2, 0.7, and 0.3 for Qwen2.5-VL-7B, InternVL3-8B, and LLaVA-OneVision-7B). In Table 7, the authors present a statistical comparison between E2Q and WFS-SB using the Qwen2.5-VL-7B. The results for the Video-MME benchmark show no statistically significant difference (the confidence interval includes zero). Thus, I am concerned that similar non-significant results might also the same for other benchmarks when using the other two backbones (cause the average performance gains are even smaller than Qwen2.5-VL-7B). Weaknesses 2 and 3 reduce my confidence in the significance of this paper.

[1] Han et al. SLICE: An Efficient and Tuning-Free Keyframe Sampling Framework for Long-Form Video Understanding. IEEE Access 2026.

[2] Li et al. FrameOracle: Learning What to See and How Much to See in Videos. ICML 2026.

Questions for authors
  1. Could you clarify what are the main contributions of E2Q over SLICE, and could you provide a direct comparison with SLICE under the same settings (like some new rows in Table 1)?

  2. Could you report the end-to-end latency, including candidate scoring, with uniform sampling and strong competing selectors under the same settings? If possible, also please report the peak memory and FLOPs.

  3. Could you provide the paired confidence intervals as Table 7 for E2Q vs. RIDGE under InternVL3-8B and vs. WFS-SB under LLaVA-OneVision-7B?

  4. How are videos with fewer than KK candidate frames handled? And is the same fallback rule applied to all selectors?

2 comments
AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer m9EC,

Since the discussion period will close soon, we would very much value your feedback on our responses, which address each of the points you raised. (1) On the contribution over SLICE, we clarified how E²Q differs in handling concentrated relevance and in its partition guarantees, added SLICE to the paper, and ran a direct comparison under identical conditions. E²Q outperforms SLICE in every model–benchmark setting, and the gains come mainly from questions where the two methods differ by design. (2) On efficiency, new end-to-end measurements show that E²Q adds no extra latency compared with selectors that use the same scorer, and we also report memory and FLOPs. (3) On significance, we provided the requested paired confidence intervals and a pooled test over all settings, which shows that the gains over the strongest baselines are statistically significant. (4) On videos with fewer than K candidates, we clarified that the same fallback rule is applied to all selectors.

We have revised the paper to include all of these analyses, and we believe they address your concerns and make clear what E²Q contributes beyond SLICE. Should any questions remain, we would be more than happy to address them. We sincerely appreciate your time.

Dear Authors,

Thank you for your detailed responses and the additional experiments. My concerns about inference latency and my question about short videos have been addressed.

Regarding my concern about statistical significance, 1) I understand Appendix D.3 already showed the 95% CI on three benchmarks, but my concern was that these experiments use Qwen2.5-VL-7B as the backbone, and E$^2$Q performed best on this backbone. That is why I asked for the results for the other two backbones. And I appreciate that you provided the results in the rebuttal. 2) I agree that for different benchmarks the strongest baseline is different, but I do not agree with the claim of "comparison with an oracle". I am not picking the best baseline on a benchmark, but on the backbone, and the users can choose the best selector for the backbone they use. That is why these model comparisons are important. For the worst case results, it is an interesting analysis from Table 1 but not relevant to this concern. 3) The CI results on InternVL3 and LLaVA-OV confirmed my concern, so it seems the statistical significance is only partly shown under Qwen2.5-VL.

Regarding my concern about comparison with SLICE, I appreciate that you provided detailed explanations and experiments, and revised the paper accordingly. As I understand, the main theoretical difference from SLICE is the cap and the guarantees it provides for the discrete partition. This makes the function of cap particularly important. The 2.1 point over SLICE on cap-active questions compares two methods that also differ in smoothing and boundary selection. So I think the comparison between full E$^2$Q and its no cap variant can isolate the effect of the cap. I found the 5.8 point on the cap-active subset of LongVideoBench (95% CI [2.1, 9.6]) helpful. Could you report the same accuracy difference on the cap-active subsets of Video-MME, MLVU, and LVBench (from Table 8)? Maybe also report a paired 95% confidence interval for each benchmark. It does not require a new experiment.

Reviewer_tJYDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Equal-rate Evidence Quantization (E2Q), a training-free frame selector for long-video understanding with MLLMs. The method partitions the video frames into intervals such that each interval contains approximately equal frame-question relevance mass, and then selects the frame with the highest relevance score within each interval. The paper also provides theoretical guarantees regarding representative selection and a bound on the approximation error.

Critical strengths
  • The paper formulates fixed-budget frame selection as a combination of equal-rate interval allocation and within-interval representative selection, with a theoretical bound on the approximation error. The approach is simple and well explained. Furthermore, its hyperparameter-free and training-free design makes it highly reproducible and practical.
  • E2Q demonstrates almost consistent performance improvements over training-free frame selection baselines across multiple standard long-video understanding benchmarks and backbone VLMs. Extensive ablation studies covering the individual components, scorer dependency, and the number of selected frames provide a detailed analysis of the proposed method.
Critical weaknesses
  • It is unclear why Inverse-CDF produces an incorrect answer while E2Q produces the correct answer in the case study shown in Figure 4. Both methods select frames from the annotated scenes, while also selecting many frames from non-annotated scenes that could potentially act as distractors. Thus, at a coarse level, both methods appear to operate under similar conditions. The difference in their answers seems to arise from small variations in the exact positions of the selected frames, making it difficult to determine whether the outcome reflects a fundamental difference between the frame selection strategies.
Questions for authors

I would appreciate clarification on the point raised in the Critical Weaknesses section, particularly regarding the underlying reason why Inverse-CDF and E2Q produce different answers in the case study shown in Figure 4.

Reviewer_zZfLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes E²Q, a training-free frame selector that separates budget allocation from representative selection for long-video question answering. It caps normalized relevance weights, partitions candidates into contiguous intervals of approximately equal relevance mass, and selects the highest-scoring frame from each interval. The authors provide a frame-level representation bound and evaluate the method across four benchmarks and three models, with additional controlled comparisons and direct frame-selection metrics.

Critical strengths

The separation of relevance-based allocation from representative selection yields a simple, deterministic method that requires no additional relevance–coverage coefficient once candidates and scores are fixed, making it straightforward to apply to existing models. Shared-score comparisons and component ablations provide useful evidence for the design choices, while results across multiple models and direct scene-coverage evaluations strengthen the empirical case beyond downstream accuracy alone.

Critical weaknesses
  1. The method proposed in this paper is primarily an engineering refinement for frame selection; its impact is limited, and its novelty is modest. The overall approach functions more as a collection of heuristic "tricks" for frame selection. The methodological advance over existing cumulative-relevance sampling is incremental.

  2. The experimental section lacks a critical comparison: the inference time of the proposed method versus other training-free approaches.

  3. In the main experiments, the performance gains over the state-of-the-art WFS-SB are only around 1%, representing a very marginal advantage.

  4. Regarding metrics for frame-selection quality—such as KFR, BSR, and UKSS—what specific capabilities do these metrics actually reflect, and how do they correlate with the accuracy of Video QA?

  5. As shown in Table 3, the ablation study covers only "no cap" and "time center," which makes the proposed method appear quite thin.

  6. Regarding Section 4, the connection between the theoretical analysis and the proposed method is unclear. Line 261 states: "The theorem concerns the frame-level representation rather than the downstream model’s computation or QA accuracy, and it does not favor a particular representative." These conclusions are confusing—especially given that our primary focus is on QA accuracy.

Questions for authors

The authors should provide additional experiments on inference time and address the weaknesses mentioned above.

Comments on the paper

2 comments

Thank you to the authors for the detailed response. I have raised my score accordingly.

Thank you for the response. However, my two concerns remain unresolved.

  • W3: My concern is that the gains are small. Statistical significance does not show that they are practically meaningful. Small gaps between previous methods do not resolve this concern either.
  • W6: The theory supports balanced relevance mass, but does not guarantee better evidence coverage. A long, repetitive segment may receive many slots, while a short but important event may be missed. Thus, the theory does not show that the selected frames contain less redundancy or more useful evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.