acceptodds
Withdrawn from ICLR 2027

Scores High Alone, Ranks Low in Context: Full-Context Knockout Frame Selection with an MLLM for Long-Video Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-video question answering asks a multimodal large language model (MLLM) to answer a question about a video that runs for tens of minutes. To make this long visual input manageable, keyframe selection reduces the video to a few dozen frames. A frame can be scored alone, by what it brings when other frames are absent, or in context, by what is lost when it is removed from the complete candidate pool. At deployment the selected frames enter the model together, motivating the second condition. We find that the highest-scoring frame alone can rank near the bottom in context because other frames already supply its evidence, and the two rankings are almost unrelated. We propose KEYSTONE, which removes one temporal unit, a short run of consecutive frames, and scores the displacement of a frozen answering model's answer distribution. The reference is determined jointly by all leave-one-out distributions. The whole chain runs on the frozen answering model, without training or an external scorer. On the Video-MME, MLVU, LongVideoBench, and LVBench benchmarks, with Qwen2.5-VL, LLaVA-OneVision, LLaVA-Video, and InternVL3.5, our KEYSTONE method outperforms existing training-free frame selectors. Code will be made publicly available upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long video understanding, video question answering, keyframe selection, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_t8T3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce KEYSTONE, a training-free frame selector for long-video QA. It scores temporal units using the KL divergence between the leave-one-out answer distributions and an averaged centroid, and uses cached visual features to reduce computation. Experiments conducted on different models and benchmarks show reasonable gains over existing approaches, supported by a set of ablations and runtime analysis.

Critical strengths
  1. The method is intuitive and simple. It does not require additional training or an external judge. Rather, it directly uses the model’s distribution to provide frame selection signals.

  2. The paper includes extensive ablations, alternative selection strategies, and runtime measurements.

Critical weaknesses
  1. Limited novelty. While the experiments show promising results, the proposed method is an incremental application of removal-based feature attribution.

  2. Unclear practical advantage. The results in Table 8 leave me uncertain about the practical value of KEYSTONE. Specifically, on Qwen2.5-VL, directly using all 128 frames achieves essentially the same accuracy at lower latency. It is unclear when the additional selection step is worthwhile.

  3. The proposed mechanisms need stronger support. Specifically, the centroid reference has a reasonable motivation, but the evidence linking it to reduced reinforcement of incorrect predictions remains limited. This leaves an important part of the method’s rationale insufficiently validated.

Questions for authors
  1. Could the authors provide more insight and evidence on why the centroid reference is preferred? Particularly how it helps avoid reinforcing incorrect predictions?

  2. Section 3.4 mentioned that cached masking approximates actual removal. Does this approximation affect the final accuracy compared with actually removing the frames?

  3. Table 8 shows comparable accuracy and lower latency for full-resolution dense answering on Qwen2.5. Under what practical scenarios would KEYSTONE be preferred?

  4. Although evaluation on the latest models is not strictly required, the evaluated models are relatively outdated. Have the authors tested more recent models to see whether the improvement persists?

Reviewer_hghjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KEYSTONE, a training-free frame selection method for long-video question answering. It partitions uniformly sampled candidate frames into temporal units and scores each unit using the KL divergence between its leave-one-out answer distribution and the centroid of all leave-one-out distributions. The highest-scoring units are selected for answering with the same frozen MLLM. The implementation reuses a visual-prefix KV cache and masks selected visual tokens during text-suffix probes.

Critical strengths
  1. Relevant problem and simple method: Selecting useful evidence under a limited frame budget is an important problem. KEYSTONE requires neither additional training nor an external scoring model.

  2. Broad evaluation within video QA: The method leads in 18 of the 20 aggregate evaluation settings, with one tie and one slightly lower result. The experiments include several competing selectors, multiple answering models, and a 72B model.

Critical weaknesses
  1. Limited generalization beyond video QA: The scoring objective is tied to a specific question and its answer distribution. Its applicability to tasks requiring VLM understanding and generation (e.g. video caption), remains unvalidated.

  2. Potential loss of cumulative evidence: Questions involving total counts or durations (just an example can be more) may require aggregating evidence across the entire video or a specified interval. Selecting a fixed number of high-scoring clips can discard necessary contributions without preserving their aggregate information.

Questions for authors
  1. Can adjacent temporal units partially overlap to retain shared context for events spanning clip boundaries? How would this overlap affect knockout scores and the risk of discarding interchangeable evidence?

  2. Have the authors evaluated isolated scoring after calibrating answer distributions against a text-only baseline? Would this reduce the gap to full-context scoring and clarify whether the gains arise from contextual evidence interactions or reliance on question-and-option priors?

  3. Can the authors disentangle the contributions of contextual evidence interactions, text-only priors, and input size to the gains over isolated scoring, using controlled comparisons with a common reference distribution and prior-calibrated baselines?

Reviewer_THdLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses training‑free keyframe selection for long‑video QA. It identifies a critical observation: frames scoring highly in isolation can become redundant and rank poorly when evaluated within the full candidate pool, since other frames already provide overlapping evidence. The authors propose KEYSTONE, an in‑context frame‑selection method based on leave‑one‑out knockout: it removes each contiguous temporal unit and measures posterior displacement of the frozen answering MLLM. To avoid bias from erroneous full‑pool model predictions, it uses the centroid of all leave‑one‑out posteriors as the shared reference, and leverages cached KV‑cache with attention‑mask probing for efficiency. Evaluated across four long‑video benchmarks and four multimodal models, KEYSTONE outperforms uniform sampling and competing training‑free selectors. Extensive ablations validate in‑context scoring and centroid reference; the paper also diagnoses failure modes for distributed cross‑unit evidence and notes limitations for open‑ended QA.

Critical strengths

Existing training‑free selectors mostly compute importance scores for frames in isolation and apply diversity as a post‑processing step. By contrast, KEYSTONE measures unit importance via answer‑posterior shift when removing each temporal unit from the full candidate pool, directly accounting for evidence redundancy among frames at scoring time rather than only in post‑hoc re‑ranking. This is consequential because it tackles a fundamental flaw of conventional frame scoring, offering a new conceptual perspective for keyframe selection.

Critical weaknesses
  1. The leave‑one‑out scoring logic exhibits a fundamental blind spot: when critical visual evidence is distributed across multiple temporal units, removing any single unit only triggers minor posterior displacement, leading all relevant units to receive low importance scores and get filtered out. While qualitative failure examples are presented in the appendix, the paper provides no algorithmic remedy or quantitative fix for this issue. This is consequential, since this flaw can discard essential supporting evidence under common real‑world video‑evidence configurations, directly hurting the robustness and reliability of the frame‑selection pipeline.

  2. Despite KV‑cache masked‑probe optimization, computational cost scales linearly with the number of temporal units. Finer‑grained units improve selection granularity but incur substantial extra inference overhead. This weakness is consequential, as it raises deployment risks: performance may degrade when hyper‑parameters are migrated to unseen datasets, and users face a difficult manual trade‑off between accuracy and compute consumption.

  3. This work mainly adapts the leave‑one‑out knockout paradigm to video frame‑unit selection and introduces the centroid reference trick. However, the overall high‑level paradigm does not depart far from prior ablation‑based attribution frameworks. Much of the improvement comes from engineering instantiation rather than fundamentally new framing for long‑video frame selection.

Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.