acceptodds
Withdrawn from ICLR 2027

-Observe: A Difference-Aware, Delayed-Commitment, Dual-Route Observation Agent for Efficient Video Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video question answering still lacks a reliable way to identify the visual evidence that distinguishes competing answers under a limited observation budget. Relevance-based selection may capture the question topic yet omit the state, order, count, or relation needed to determine the answer. A separate bottleneck arises when textual intermediates omit visual details available in the original frames. To address these limitations, we present D-Observe, a training-free observation agent that uses the question and all answer options to select visual evidence before committing to an answer. It then passes the retained RGB frames directly to Vision-Language Models (VLMs) in a single call. Additionally, a label-free shared-observation route amortizes visual processing across multiple questions about the same video. Across five benchmarks spanning long and short videos (Video-MME, MLVU, EgoSchema, NExT-QA, and LVBench), D-Observe achieves 64.94% macro accuracy with an average of 10.21 frames sent per question. It improves upon uniform 32-frame inference with Qwen3.5-27B by 2.02 percentage points (95% CI [, ]) while reducing sent frames by 68.1%. By holding frame inputs fixed, we further isolate the effect of the textual bottleneck and show that direct visual answering consistently outperforms text-mediated answering.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Question Answering, Long Video Understanding, Vision-Language Models, Efficient Inference, Adaptive Frame Selection, Training-free Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3borOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes D$^3$-Observe, a training-free VideoQA framework with two routes. The per-question route derives rule-based evidence axes from the question and all answer options, then supplements uniformly sampled frames using lightweight visual cues. The shared route jointly answers several questions about the same long video using one uniformly sampled 32-frame context. Both routes complete observation allocation before answering directly from visuals. The paper evaluates accuracy, amortized input costs, and the effects of replacing visual inputs with textual representations.

Critical strengths
  1. The motivation is somehow relevant. Question relevance does not necessarily identify the visual evidence needed to distinguish answer options. Preserving such evidence under a limited observation budget is a worthwhile objective. Actually, the problem settings have some practical impacts.

  2. The experimental controls and uncertainty estimates are useful. The fixed-input interface study includes a same-backbone textual control, and the video-level bootstrap accounts for dependence among questions from the same video.

Critical weaknesses
  1. The central selection mechanism is insufficiently specified. The manuscript does not explain the complete rules for constructing evidence axes, the operational definitions and normalization of the visual cues, their weights, or the criteria governing adaptive budgets and coverage constraints. Crucially, it remains unclear how semantic distinctions between answer options change the ranking of RGB frames. Appendix C still expresses these essential operations as unexpanded functions. Consequently, the proposed core mechanism cannot be adequately reproduced or assessed.

  2. The claimed methodological novelty is not sufficiently established. “Delayed commitment” operationally means fixing the frame set before a direct visual answering call, which also describes the uniform-frame baseline. The textual-interface experiments assess the consequences of particular intermediate representations; they do not isolate the effect of avoiding premature answer commitment. The shared route implements joint answering over uniform frames. A substantive technical advance therefore depends heavily on the difference-aware selector, whose distinctive mechanism remains underdefined.

  3. The shared-route comparisons do not adequately isolate the source of the gains. On Video-MME and LVBench, the method uses uniform 32-frame contexts without difference-aware selection. Comparing joint question batches against per-question uniform inference simultaneously changes observation amortization and the textual context available to the model. Table 7 provides a related MLVU ablation, but its replacement policy is insufficiently described, and equivalent controlled comparisons are not reported across benchmarks. The reported reductions establish transmitted-frame savings under the chosen workload, while complete computation savings—including local decoding and selection—remain insufficiently characterized.

  4. The ablations leave a consequential interaction unexplained. In Table 7, difference coverage improves accuracy by 4.22 points with shared observation enabled, but its effect becomes −0.05 points when sharing is disabled. Since the shared route itself bypasses difference-aware selection, this interaction requires results stratified by original route eligibility and precise descriptions of all four configurations. The current analysis does not establish when the proposed selector helps or explain the claimed complementarity.

  5. The presentation obscures the algorithm, and the mechanism analysis is inadequate. The method interleaves task definitions, accounting conventions, workload routing, and abstract principles before making the computation clear. Meanwhile, the qualitative examples primarily document inputs and successful answers, without showing how option distinctions caused useful frames to be added. Direct evidence-coverage measurements, controlled option-conditioned selection comparisons, and actual selection traces are needed to substantiate the paper’s central explanation.

Questions for authors

Despite this paper points out a potentially practical problem for Video-VLMs in some industrial settings, its presentation cannot support a well-written paper and the methodology leaves many points unexplained.Therefore I cannot recommend any accept to this submission since it still need lots of technical and writing improvements.

Reviewer_G6XWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

D3-Observe is a training-free observation framework for efficient video QA. Its per-question route derives evidence requirements from the question and all answer options, then combines uniform temporal coverage with lightweight visual cues to select frames before answering. For some long videos with MCQs, a shared path supplies one uniform 32-frame context to a joint answering path.

Critical strengths
  • The motivation distinguishes question relevance from evidence that separates competing answers. This is useful for questions involving state, temporal order, counting, or object relations. The fixed-input comparison between direct visual answering and textual intermediates is also informative because it holds the selected frames constant while examining information loss through textual description.

  • The results cover a useful range of short- and long-video benchmarks and are well presented. Reporting paired confidence intervals with video-level resampling is particularly appreciated, since questions from the same video are not independent. The Pareto gain of 2.02 percentage points over uniform 32-frame inference, alongside a 68.1% reduction in amortized sent frames, is promising.

Critical weaknesses
  • The central contribution of the paper is hard to distinguish and separate. The shared path uses uniform 32-frame contexts without difference-aware selection. For example, approximately 10.69 sent frames per question on Video-MME mainly reflects sharing 32 frames across three questions not answering each question from approximately ten selected frames so it's a hard comparison. Don't get me wrong this is an efficiency benefit, but for a different claim. A uniform 32-frame baseline using identical question batching and the same joint-answer interface would better isolating the contributions. The no-sharing ablation the authors did changes both the acquisition and the answering, so not exactly the isolation needed.

  • The link between answer-option differences and frame selection is unclear and a bit confusing. The planner identifies needs such as actions, states, counts, and order, while the selector uses coverage, novelty, motion, detail, entropy, and temporal cues. These may find informative frames but do not show that the frames distinguish the answer choices. The authors should explain and show how option differences affect selection and whether removing difference coverage changes the evidence given to the answerer.

  • Minor:

    • “CLIP-free” describes an implementation choice, but the measured accuracy–cost benefit is the stronger contribution to emphasize.
    • The abstract should state explicitly that the 2.02-point gain is a five-benchmark macro-average, whereas the 68.1% sent-frame reduction uses sample-weighted.
Questions for authors
  • How does the complete method compare with uniform 32-frame inference using identical shared-question batches and the same joint-answer prompt?

  • Can the authors provide a worked example tracing an option-level distinction through evidence-axis construction, frame scoring, and final selection?

Reviewer_Zn9eOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Existing Video Question Answering (VideoQA) methods often select frames solely based on their relevance to the question or rely on textual captions, which can miss critical visual evidence needed to distinguish among candidate answers under a limited frame budget and may incur the cost of repeated model calls. To address these limitations, this paper proposes D3D^3-Observe, a training-free agent. It symmetrically analyzes the question and all answer choices to derive evidence axes, selects frames using lightweight visual cues, and directly feeds the original RGB frames to a VLM without any intermediate textual representation, enabling the model to produce an answer in a single call. In addition, for long videos of at least 300 seconds with three or more questions, the method shares a single uniformly sampled 32-frame observation across batches of up to five questions and processes them with a single VLM call, thereby substantially amortizing the visual encoding cost.

Critical strengths
  1. Training-free and model-agnostic design: The proposed framework can be seamlessly applied to existing VLMs without requiring separate selector training or additional training costs, and its compatibility across different VLM backbones ensures high practicality and scalability.

  2. Good reproducibility and implementation transparency: The paper provides concrete details regarding the complete inference procedure, routing thresholds, frame budgets, prompt contracts, and cost accounting, making the implementation easy to follow and enhancing reproducibility.

  3. Novel perspective shift from query relevance to difference coverage: Moving beyond conventional approaches that merely retrieve frames similar to the question, the paper symmetrically analyzes all candidate options to first define and acquire the decisive visual evidence (e.g., state, order, count) needed to distinguish the correct answer—a problem formulation that is both intuitive and compelling.

Critical weaknesses
  1. Heavy reliance on manually designed heuristics: The core difference-aware selector depends on predefined evidence categories, rule-first evidence-axis construction, and handcrafted visual cues. This raises concerns about whether the method can generalize robustly beyond the specific question patterns and datasets considered in the paper.

  2. Limited methodological coherence of the dual-route design: The dual-route formulation combines two substantially different mechanisms: option-conditioned difference-aware frame selection for the per-question route and uniform 32-frame sampling with shared batching for the shared route. This makes the overall framework feel less unified than the paper’s presentation suggests.

  3. Dependence on predefined answer candidates: The observation-planning mechanism explicitly relies on comparing the available answer options to identify discriminative evidence. This assumes that candidate answers are known in advance, which limits the direct applicability of the core difference-aware formulation to open-ended VideoQA settings without predefined options.

  4. Bottleneck of the 64-frame candidate pool: On the per-question route, frame selection is restricted to a pre-decoded pool of at most 64 candidate timestamps. For long videos, including those just below the 300-second routing threshold, this pool may still be too sparse to capture brief or fine-grained visual events, meaning that relevant evidence can be lost before the difference-aware selector is applied.

  5. Unverified core mechanism on major long-video benchmarks: According to Table 10, 100% of the queries in Video-MME and LVBench are processed exclusively through the shared route, which uses uniform 32-frame sampling with multi-question batching. As a result, the proposed difference-aware selector is not actually evaluated on these two major long-video benchmarks, and a substantial portion of the reported efficiency gains may instead come from batching-based amortization rather than improved evidence selection.

Questions for authors

Q1. How can the proposed selector generalize to complex, out-of-distribution queries or high-level semantic distinctions given its reliance on predefined rule-first patterns and low-level handcrafted visual cues?

Q2. How do the authors justify framing the per-question difference-aware selector and the uniform-sampling shared route as a single unified methodology rather than a heuristic combination of two orthogonal techniques?

Q3. How can the core difference-aware observation mechanism be adapted to open-ended VideoQA tasks where no predefined answer options are available to derive discriminative evidence axes?

Q4. On the per-question route, how does the method prevent brief, fine-grained visual events from being missed when initial selection is strictly bottlenecked by a sparse 64-frame candidate pool?

Q5. Since 100% of the queries in Video-MME and LVBench bypass the per-question route (Table 10), what is the actual performance and efficiency of the core difference-aware selector on these two benchmarks when shared routing is disabled?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.