acceptodds
Under review as a conference paper at ICLR 2027

BEPA: Agent-Guided Evidence Policy Adaptation for Video Question Answering

Abstract

Under a limited frame budget, question-relevant observations may remain compatible with multiple answers, while shared scene content obscures discriminative cues. We propose BEPA (Agent-Guided Evidence Policy Adaptation) to organize evidence around inter-option distinguishability and signal–background separability with a frozen video answerer. Before answering, CORE (Contrastive Option-guided Refinement of Evidence) suppresses shared option responses and refines a few BOLT fill frames. In an independent post-answer setting, a text agent turns prediction disagreement into visual queries for one additional observation. On 4,703 VUDG questions across 11 domains with VideoLLaMA3-7B, exploratory evaluation yields 75.82% domain-averaged accuracy (D-Avg) for CORE, versus 75.16% for prompt-matched BOLT, using one answer and at most 32 frames. Independent agent-guided acquisition reaches 77.26% D-Avg with the original prompts at 2.52 video answers per question on average.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Video Question Answering, Discriminative Evidence Selection, Option-Contrastive Refinement, Agent-Guided Evidence Acquisition, Budget-Constrained Inference

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_wKZpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes BEPA for evidence selection with a frozen video answerer. It evaluates two independent settings: CORE refines BOLT’s frame selection through option-centered matching scores and a coverage objective, while an agent-guided approach uses prediction disagreement to generate retrieval queries for one additional observation. Experiments are conducted on VUDG using VideoLLaMA3-7B.

Critical strengths
  1. The matched CORE–BOLT comparison fixes the answering model, prompt, and frame budget, and reports a positive paired confidence interval alongside repairs and harms.
  2. The answer-preserving call-omission condition is mathematically justified.
Critical weaknesses
  1. Insufficient independent validation. The evaluation is explicitly development-exposed, and Appendix A states that the existing data splits are not independent validation sets. The reported bootstrap intervals therefore do not address possible selection bias during development. Evidence is also limited to one benchmark and one video answerer.

  2. The distinctive mechanism is insufficiently validated. There are no controlled ablations separating option centering, peak normalization, the coverage objective, anchor protection, and the swap budget. Consequently, the modest improvement over BOLT supports the complete configuration but does not establish which proposed design provides the benefit. The advantage over WFS-CLIP remains inconclusive, with a confidence interval crossing zero.

  3. Essential reproduction details are missing. Appendix A explicitly states that the C1 environment, public planner/CLIP snapshot mappings, and complete processor settings remain undocumented. The full planner prompt and local WFS-CLIP configuration are also insufficiently specified, preventing independent reproduction of the reported results.

  4. The agent contribution is not adequately isolated. Global-coverage and stem-retrieval controls are useful, but the paper does not establish whether the gain requires prediction-disagreement information rather than generic question-and-option rewriting. Table 3 lacks paired uncertainty estimates for the principal agent comparisons, and the reported base-call counts require reconciliation before the execution controls can be considered fully understood.

Questions for authors
  1. Which configurations and decisions were selected using the evaluation data, and what evidence from data not used during development confirms the gains?

  2. Under identical candidate pools and frame budgets, what are the individual contributions of centering, normalization, and option coverage compared with uncentered option-aware retrieval?

  3. Does the agent still improve performance when prediction summaries are removed, while preserving the same trigger population and additional-observation budget?

Generally, the paper's quality cannot meet the requirements of this conference, and many questions are left unexplained. Therefore, I recommend a reject to this paper.

Reviewer_JjaQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

BEPA posits that in VideoQA under a limited frame budget, identifying answer-discriminative evidence that distinguishes between candidate options is more critical than relying solely on question relevance. While preserving BOLT’s primary anchors, CORE suppresses shared responses across options and replaces up to four frames with more discriminative evidence. Furthermore, in Agent-Guided Targeted Evidence Acquisition, when BOLT and Uniform predictions disagree, a text-only Qwen agent generates visual queries to verify the discrepancy and retrieves additional frames. Across the 11 domains of VUDG, CORE improves BOLT’s accuracy from 75.16% to 75.82%, while the agent-guided approach achieves a D-Avg of 77.26%.

Critical strengths
  1. Improved discriminability without training or additional frame budget: With the video answerer kept frozen, CORE improves performance while maintaining the same maximum 32-frame budget by suppressing responses shared across answer options and emphasizing frames that better distinguish among candidates, going beyond simple question relevance.

  2. Selective and efficient disagreement-driven agent exploration: Rather than invoking the agent for every question, the method generates targeted queries with a text-only agent only when predictions from initial observations disagree. It further skips unnecessary inference calls through answer-preserving budget control, achieving a strong accuracy of 77.26% with improved computational efficiency.

Critical weaknesses
  1. Lack of integration between the two main contributions: Although the paper presents CORE (C1), a pre-answer evidence refinement module, and Targeted Acquisition (C2), a post-answer agent-based acquisition module under a unified BEPA framework, the two components are evaluated entirely independently. In particular, C2 uses the original BOLT observations rather than the proposed CORE refinement as its base, and the paper does not evaluate whether combining CORE and C2 provides any additional benefit or synergy.

  2. Incomplete research scope with the C3 analysis left unfinished: The paper and Figure 2 introduce a third component, C3, involving cross-domain analysis and semantic-support checks. However, the manuscript explicitly states that this analysis “remains pending” and that the corresponding checks are “planned” rather than completed findings. This makes the submission appear incomplete, as part of the proposed evaluation framework is presented but not actually carried out.

  3. Insufficient reproducibility documentation: Even in Appendix A, the paper states that several implementation details, including the sealed C1 environment, planner/CLIP snapshot mappings, and complete processor settings, “remain to be documented.” Since these details can materially affect frame selection and model execution, their absence raises concerns about the reproducibility of the reported results.

Because of these issues, particularly the incomplete C3 analysis and missing implementation details, the current manuscript does not provide enough evidence to support deeper technical verification or more detailed assessment of the full BEPA framework.

Questions for authors

I recommend carefully revisiting these weaknesses and strengthening the paper accordingly.

Reviewer_J9iROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces BEPA, a framework for improving evidence selection in VideoQA under a limited frame budget. It proposes two main components: (1) CORE, which refines BOLT's frame selection by prioritizing evidence that distinguishes between candidate answers rather than relying only on question relevance, and (2) an agent-guided acquisition strategy that uses prediction disagreements to generate targeted retrieval queries and acquire additional evidence.

Experiments on VUDG using a frozen VideoLLaMA3-7B show that CORE improves accuracy from 75.16% to 75.82%, while the agent-guided approach achieves 77.26%. However, the two components are evaluated independently, and the proposed cross-domain analysis remains incomplete.

Critical strengths
  1. Answer-discriminative evidence selection. The idea of selecting frames based on their ability to distinguish between candidate answers, rather than simply their relevance to the question, is interesting and well motivated. CORE demonstrates a modest improvement over BOLT under a matched frame budget and frozen answering model, suggesting potential value in this direction.

  2. Selective targeted evidence acquisition. The agent-guided approach uses disagreement between initial predictions to trigger additional evidence retrieval, avoiding unnecessary exploration. This is a promising approach to balancing accuracy and computational cost, and the reported improvements suggest that targeted acquisition can be beneficial.

Critical weaknesses
  1. Insufficient experimental validation. The evaluation is limited to a single benchmark and answering model, with no independent held-out validation. Moreover, the paper lacks controlled ablations isolating the effects of CORE's individual mechanisms and does not establish whether the agent's improvements specifically depend on prediction disagreement. These limitations make it difficult to determine which contributions are effective and whether the improvements generalize beyond the evaluated setting.

  2. Incomplete framework and reproducibility. Although BEPA is presented as a unified framework, CORE and agent-guided acquisition are evaluated independently, without investigating whether they can be combined effectively. The proposed C3 analysis remains unfinished, and several essential implementation details are undocumented. Together, these issues raise concerns about the completeness of the work and the reproducibility of its results.

Questions for authors
  1. Independent validation: Which design choices and hyperparameters were selected using the evaluation data? Can the authors provide results on an independent held-out dataset or another benchmark to establish whether the reported improvements generalize?

  2. Mechanism ablations: What are the individual contributions of option centering, normalization, coverage optimization, and anchor protection in CORE? Does the improvement persist relative to simpler option-aware retrieval baselines under identical budgets?

  3. Agent contribution: How much of the agent-guided improvement is specifically attributable to prediction-disagreement information? Can the authors compare against an agent using only the question and answer options, with the same trigger population and observation budget?

  4. Framework completeness and reproducibility: Can the authors report results combining CORE with agent-guided acquisition, clarify the inference-call accounting, and provide the missing implementation details necessary to reproduce the experiments?

Reviewer_uHxBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes BEPA for evidence selection in video QA with frozen models. CORE refines BOLT frame selection using option-contrastive scores, while a separate agent-guided module uses prediction disagreement to acquire an additional targeted observation.

Critical strengths
  1. The paper distinguishes fixed-budget refinement from additional evidence acquisition.
  2. The question-level utility analysis is more informative than relying only on aggregate accuracy.
Critical weaknesses

The work appears incomplete in its current form. The evaluation is not fully independent, the planned C3 analysis is unfinished, and several implementation details remain undocumented.

Questions for authors
  1. Is C3 a completed component or future work? Please provide the missing results or revise the related claims.
  2. The policy was selected after sequential experiments, and pilot questions remain in the final evaluation. Can the authors report results on a truly held-out set?
  3. Table 2 evaluates CORE as a whole. What is the contribution of its individual components?
  4. Table 3 mixes planning, targeted retrieval, an extra VQA call, and probability fusion. Which component actually drives the gain?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.