acceptodds
Under review as a conference paper at ICLR 2027

Cog-Fovea: Conditional Visual Acquisition and Answer Selection for Video Reasoning

Abstract

Sparse frame sampling can omit brief events essential for video reasoning. Extending textual reasoning may improve the interpretation of observed frames, but does not acquire missing observations. We introduce Cog-Fovea, an inference framework with frozen weights that separates two decisions: when to inspect additional video frames and when to replace an existing answer. An initial global pass at high resolution produces an answer and a confidence score. Queries with low confidence trigger temporal grounding and a hybrid view at lower resolution combining local samples with global anchors. Arbitration based on confidence selects between the original and hybrid answers, retaining the initial answer as a candidate rather than automatically accepting the new prediction. All stages share a frozen backbone, with a cap on visual tokens per pass and total computation accounting for routed calls. A conditional analysis of event coverage characterizes when localized sampling can capture brief events. Experiments on five benchmarks examine accuracy, inference cost, and answer selection behavior. On Video-MMMU, Cog-Fovea achieves 60.2% accuracy, compared with 57.0% using only the global pass. Inference without explanations averages 6.3 seconds per query. Approximately 35% of queries receive additional inspection. Arbitration provides a numerical gain of 0.6 percentage points over accepting every available hybrid answer, but reduces accuracy on a diagnostic subset with narrow temporal windows. These findings support conditional visual acquisition as a strategy for allocating computation at test time and show why acquiring more evidence and accepting a revised answer must be evaluated separately.

Primary area:
foundation or frontier models, including LLMs
Keywords:
video reasoning, conditional visual acquisition, temporal grounding, answer selection, frozen backbone

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cjMgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Cog-Fovea, an inference framework for training-free video reasoning. It first generates an answer from high-resolution, globally sampled frames. Low initial-answer confidence triggers temporal grounding and a second answer from a lower-resolution hybrid view containing global anchors and densely sampled local frames. A confidence comparison then determines whether to retain the initial answer or accept the revised answer. The paper evaluates this framework on five benchmarks and provides information-theoretic motivation, sampling controls, and answer-selection diagnostics.

Critical strengths
  1. The problem motivation is technically decent. Sparse sampling can miss brief events, and additional textual reasoning cannot directly acquire those missing observations. Allocating inference computation to further visual inspection is therefore a sensible direction.

  2. Separating evidence acquisition from answer replacement is a reasonable design choice. A temporally denser view can lose spatial details through downsampling. Retaining the initial high-resolution answer acknowledges this trade-off and avoids automatically treating additional observations as a better answer.

Critical weaknesses
  1. The novelty is partially limited. The framework combines confidence routing [1], conditional frame selection [2], and adaptive visual inspection [3–5]. Its particular combination is reasonable, but the paper does not clearly establish a substantive methodological advance beyond these existing directions.

  2. Many technical hyperparameters, including routing thresholds, frame allocations, resolutions, fallback intervals, and answer replacement, are manually specified. Specifically, the initial-answer likelihood does not directly reflect missing visual evidence, and comparisons across prompts and visual contexts lack sufficient support.

  3. Key design choices lack ablation studies. The routing thresholds and spatial–temporal budget allocation are insufficiently examined, while several controls change multiple factors simultaneously. The proposed method should be examined and analyzed carefully, as the grounding scores, oracle proposals, and correctness counts do not clearly connect newly acquired evidence to improvements in answers. Quantitative evidence-coverage analysis and real prediction traces are needed to explain when local inspection helps and when spatial downsampling or arbitration causes failures.

  4. The confusing and excessive notation should be carefully formulated in the paper. Numerous contexts, frame sets, and auxiliary indicators obscure a relatively simple inference procedure. Figure 2 requires additional notation mapping, while the evaluation identities in Eqs. (8)–(9) interrupt the algorithmic explanation. A concise pseudocode description would improve clarity.

  5. Figure 2 combines too many implementation details and analytical formulae, weakening its effectiveness as an overview. Long captions, extensive footnotes, small text, and mixed evaluation settings make the tables difficult to interpret, also indicating insufficient refinement of the presentation.

References

[1] Shuming Liu et al. VideoAuto-R1: Video Auto Reasoning via Thinking Once, Answering Twice. CVPR, 2026.

[2] Jialuo Li et al. Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding. CVPR, 2026.

[3] Xuchen Li et al. Select Less, Reason More: Prioritizing Evidence Purity for Video Reasoning. CVPR, 2026.

[4] Ziang Yan et al. VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception. NeurIPS, 2025.

[5] Xiaohan Wang et al. VideoAgent: Long-form Video Understanding with Large Language Model as Agent. ECCV, 2024.

Questions for authors
  1. Compared to VideoAuto-R1, DIG, VideoChat-R1.5, EARL, and LOVE-R1, what is the main methodological insight beyond combining confidence routing, localized resampling, and answer retention?

  2. How were the confidence thresholds and the global/local frame and resolution allocations selected? How does the accuracy–computation trade-off change under reasonable alternative settings?

  3. Can localization and routing be evaluated with more controlled comparisons, e.g., duration-matched random windows on the same routed queries, and different routing policies with the same subsequent sampling procedure?

  4. What supports comparing confidence scores obtained under different prompts and visual contexts? Can the analysis distinguish cases where arbitration preserves useful spatial detail from cases where it rejects genuinely improved visual evidence?

Reviewer_EtXeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Cog-Fovea introduces a training-free framework for conditional visual acquisition and answer selection in video reasoning. It addresses missing visual evidence caused by sparse frame sampling. The method first generates an answer from 64 high-resolution global frames. Low-confidence answers trigger temporal grounding and denser sampling within a predicted interval. A hybrid view combines local frames with global anchors at lower resolution. Confidence-based arbitration selects between the initial and revised answers. Experiments on five benchmarks show modest accuracy improvements, including 60.2% on Video-MMMU with an average QA-only latency of 6.3 seconds.

Critical strengths
  1. Clear motivation. The paper identifies a fundamental limitation of text-only test-time scaling: additional reasoning cannot directly recover missing visual observations.

  2. Simple and training-free design. The framework combines conditional routing, temporal resampling, and answer selection using a frozen backbone without additional training.

Critical weaknesses
  1. Limited novelty compared with prior work. VideoAuto-R1 already uses confidence-based routing, while Video-o3 and VideoTemp-o3 also explore adaptive visual evidence acquisition. The paper does not clearly explain its advantages over these methods.

  2. Limited robustness to grounding errors and multiple events. The method relies on a single predicted interval, but its grounding accuracy is only 38.1% mIoU on NExT-GQA. Unlike VideoTemp-o3 and Video-o3, it does not support correcting inaccurate localization or retrieving evidence from multiple distant segments.

  3. Insufficient ablation studies. Several ablations change multiple factors at once, including sampling windows, prompts, and inference strategies. It remains unclear how much improvement comes from better frame selection rather than additional inference or answer selection.

  4. Unreliable confidence-based answer selection. The method compares confidence scores from different visual inputs and prompts without calibration. This strategy improves Video-MMMU accuracy by only 0.6% and even hurts performance on narrow-window queries.

  5. Modest performance gains and limited benchmark coverage. The improvements over VideoAuto-R1 are only 0.8–1.6%, without statistical significance tests. Evaluation on more challenging long-video benchmarks (e.g., LVBench, Video-MME-v2, VideoZeroBench) and dedicated temporal grounding benchmarks (e.g., Timelens-Bench) is missing.

  6. Limited comparison with simple test-time scaling. The self-consistency baseline uses five inference passes and much higher computational cost. The paper lacks fair, cost-matched comparisons with adaptive Best-of-N and other simple scaling methods.

Questions for authors

See weaknesses.

Reviewer_LRHpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Cog-Fovea is a training-free inference procedure for video question answering. A frozen VideoAuto-R1 model first answers from 64 uniformly sampled high-resolution frames. Low answer confidence triggers temporal grounding and a second answer from lower-resolution frames combining global anchors with local samples. The initial answer is replaced only if the second answer has higher token-probability confidence. On Video-MMMU, the full procedure reaches 60.2% accuracy versus 57.0% for the initial pass alone, with approximately 35% of queries routed for further inspection. The paper reports comparisons on additional benchmarks and examines cases where answer selection discards a correct revision.

Critical strengths
  1. Keeping the initial answer available after resampling is sensible. The second pass trades spatial detail for temporal coverage, so it can lose evidence as well as acquire it. The method handles that trade-off without training a separate selector. The reported improvement from 57.0% to 60.2% on Video-MMMU makes this a useful inference configuration to investigate.

  2. The answer-selection diagnostic is more revealing than the aggregate score alone. On the 69 narrow-window queries, the initial, hybrid, and selected answers are correct on 33, 37, and 35 cases. Among rejected hybrid answers, arbitration discards 13 correct revisions and restores 11 correct initial answers. Reporting these counts shows exactly how selection can undo part of the benefit of another visual pass. The paper also clearly distinguishes a per-pass token limit from total computation and answer-only latency from explanation-enabled latency.

Critical weaknesses
  1. The comparisons presented in the main text do not establish how much benefit comes from targeted acquisition rather than the other changes in the pipeline. Changing only the dense baseline's prompt raises its score from 57.8% to 58.9%, leaving a 1.3-point difference from Cog-Fovea. The remaining comparison changes routing, frame allocation, and answer selection together. Given that adaptive frame selection is already established, the important question is whether this particular controller improves on a simpler controller at comparable cost. The cited Appendix B controls may help answer that question, but the main-text numbers alone do not. The 23% latency reduction also uses an always-think timing reference, not a matched implementation of the published auto-think baseline.

  2. The evidence for the answer-selection rule is still limited. It compares token probabilities across different prompts and visual contexts, where a larger score need not indicate a better answer. The full-set improvement over always accepting the hybrid answer is only 0.6 percentage points, while selection loses two correct answers on the narrow-window subset. The authors acknowledge both facts; my concern is whether the small positive aggregate result is reliable enough to support arbitration as a main contribution. Paired uncertainty estimates, the full replacement counts, and sensitivity to the routing thresholds would help distinguish a useful policy from a favorable operating point.

Questions for authors
  1. Do the Appendix B controls hold the answer prompt and selection rule fixed while changing only the temporal sampling policy? In particular, how does predicted-window sampling compare with uniform or random-window sampling on the same routed queries, using the same frame budget and accounting for the grounding call? Please identify the relevant comparisons and their total cost. A same-hardware comparison with the original VideoAuto-R1 auto-think policy would also clarify the efficiency claim.

  2. For the full-set 0.6-point arbitration gain, how many rejected hybrid answers restore a correct initial answer, and how many discard a correct revision? What is the paired confidence interval for that difference? Please also clarify how the 0.70 and 0.65 routing thresholds were chosen and whether the benefit persists at nearby thresholds. These results would affect whether I view answer retention as a reliable part of the method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.