acceptodds
Under review as a conference paper at ICLR 2027

Reliable Reasoning for Video Understanding

Abstract

Multimodal chain-of-thought (CoT) has emerged as a promising paradigm for improving the reasoning capabilities of multimodal large language models (MLLMs): It typically extracts question-relevant cues from a static image and reasons over them to derive an answer. The success of this paradigm partly rests on the assumption that the static image contains sufficient evidence for the question. Videos, however, challenge this assumption because they are temporally structured sequences of frames in which relevant evidence can be distributed unpredictably over time. This introduces two sources of unreliability. First, a single sampled video observation may omit crucial evidence, causing the model to reason from an under-informed view; we refer to this as unreliability arising from under-informed reasoning. Second, the model's understanding and answer may change substantially as richer observations become available; we refer to this as unreliability arising from cross-observation inconsistency. To address both challenges, we introduce Reliable Reasoning with Videos (), which assesses answer reliability at two levels. Within a single observation, RRV uses video information extraction and answer reasoning spans when both are available, falling back to whole-response uncertainty otherwise, and aggregates the reliability-weighted trajectories into an answer distribution. Across observations, RRV checks answer consistency along a nested sequence of increasingly informative frame sets until two adjacent observations propose the same answer. By estimating the reliability of CoT within each observation and filtering answers based on consistency across observations, RRV suppresses the influence of unreliable CoT reasoning on final predictions. Experiments on three video MLLMs and eight video QA benchmarks show that RRV improves accuracy in most evaluated settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Understanding, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CKcTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reliable Reasoning with Videos (RRV), a training-free inference framework designed to improve the reliability of video question answering. The method combines two mechanisms: within-observation reliability estimation, which assigns uncertainty-based weights to sampled reasoning trajectories, and cross-observation stability verification, which progressively expands the observed video frames and checks answer consistency. Adaptive stopping rules are introduced to reduce unnecessary sampling and terminate inference when consecutive observations produce stable predictions.

Critical strengths
  • The paper identifies a relevant limitation of conventional self-consistency methods when reasoning over incomplete video observations.
  • The ablation studies, frame-sensitivity experiments, and accuracy–cost comparisons offer insights into how progressive observations and repeated reasoning affect prediction stability.
Critical weaknesses
  • The proposed framework primarily combines existing techniques, including uncertainty-weighted self-consistency, adaptive trajectory sampling, progressive frame expansion, and answer-agreement-based stopping. Related ideas have already been explored in Self-Consistency (ICLR 2023), Escape Sky-High Cost (ICLR 2024), and Flexible Frame Selection for Efficient Video Reasoning (CVPR 2025). Although applying these techniques jointly to video reasoning is reasonable, the proposed method remains algorithmically straightforward and does not establish a sufficiently distinctive contribution beyond their combination.
  • The theoretical results largely formalize general properties of answer distributions and stopping decisions rather than establishing new guarantees about video reasoning reliability. Lemma 1 states a standard margin-stability condition, while Proposition 1 guarantees preservation of the full-budget voting leader rather than answer correctness. Proposition 2 bounds answer reversal using a distribution-shift term between observations, but this term is not empirically estimated or meaningfully constrained and can dominate the bound. Consequently, the theoretical guarantees do not establish that cross-observation agreement implies sufficient visual evidence or a correct answer. The mathematical analysis appears considerably stronger in presentation than in its actual explanatory contribution.
  • The entropy-based trajectory weights show relatively weak discrimination between correct and incorrect predictions, with within-group AUROC values of only 58.5–60.8% in the reported VideoTemp-o3/MMVU analysis.
  • RRV uses substantially more inference computation than standard self-consistency, with approximately 6.76–8.18 times the output tokens on MMVU while achieving relatively modest accuracy gains. Although adaptive stopping reduces computation relative to the full RRV procedure, the experiments do not establish superiority over simpler approaches at matched inference budgets.
  • The method assumes that agreement between two consecutive expanded observations provides a useful indicator of prediction reliability. However, agreement does not necessarily imply correctness, and the paper itself provides examples of consistently incorrect predictions.
  • Although RRV generally outperforms the reported baselines, many improvements are relatively small, particularly on MLVU and several individual benchmark–backbone combinations.
Questions for authors
  • How do Lemma 1 and Propositions 1–2 provide meaningful guarantees about answer correctness or sufficient visual evidence, rather than simply characterizing voting stability?
  • Could the authors compare RRV against self-consistency, confidence-weighted voting, and progressive frame expansion using identical numbers of generated trajectories, visual tokens, and inference budgets?
  • Could the authors clarify which component constitutes a genuinely new mechanism rather than an adaptation of established inference-time techniques?
Reviewer_6qyUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work introduces a new inference time method to improve reasoning capabilities of Video large language models (Video MLLMs). Compared to standard test-time methods like self-consistency, they argue that for videos, naively adopting them is suboptimal because of two reasons. First, if the question-relevant frames are not sampled, the different reasoning trajectories will all be ill-informed and can agree on a wrong answer. Second, the answer can change substantially when new frames are introduced, therefore it will be not in agreement with the answer produced without using the new frames.

To overcome these challenges, they propose reliable reasoning for videos (RRV), which operates at two levels. Within a fixed set of frames, it runs multiple reasoning trajectories but weights each one based on how confident the model was during generation, measured via token-level entropy, so that uncertain trajectories have less influence on the final answer. Then, it progressively expands the input by adding new frames and re-runs the estimation. The algorithm converges when two adjacent frame sets both predict the same answer. In short, the algorithm behaves similarly to a weighted form of self-consistency applied both within the same frame set and across expanding frame sets.

They test their approach on different video MLLMs and different benchmarks, where they observe overall positive improvements compared to naive inference and baseline test-time inference techniques, though gains vary across models and benchmarks.

Critical strengths

Most video MLLMs operate with a fixed set of input frames which are used for different tasks like video question answering. This work emphasizes this as a limitation as fixed set of input frames can potentially miss the question relevant frames required to make correct predictions, and therefore showcasing that existing inference time methods could be suboptimal as the input signal is ill-informed in the first place, which motivates their self-consistency style algorithm that works along variable frame set inputs. I believe this analysis, and the proposed algorithm would be interesting for the researchers in the community.

Additionally, I find that evaluation conducted on different benchmarks and using different Video MLLMs is broad and reasonable, and provides good coverage.

Critical weaknesses

My major concern is the quality of presentation. The paper is unfortunately very hard to follow, and the mathematical notation is quite heavy relative to what the method actually does. The main contributions are deeply buried under mathematical notation and equations, most of which I believe are unnecessary and can be greatly reduced (e.g., nested indices). Following on this, the writing also assumes the reader is holding a lot of context simultaneously. For example, Section 3.2 references the nested sequence from Section 3.3 before Section 3.3 has been introduced, creating a circular reading experience. This is one of my main reasons for leaning toward rejection.

Secondly, I carefully looked into the results. The proposed RRV method shows modest improvements over different baselines and existing methods. This is not alarming given the clear motivation and algorithm intuition, but when accounting for the extra inference cost, the value of the reported gains seems diminishing (e.g., a token budget ratio of 1 vs 15 for self-consistency vs RRV).

Thirdly, since the method strongly relies on the model which supports variable number of frames as input, this poses an another constraint, as video models often uses a fixed set of frames as input and are trained in that setting. This also raises a concern on whether for the currently used video MLLMs, were they trained on different variable frame counts? A discussion on this aspect should be added.

Finally, it would have been a good analysis to include qualitative analysis on test samples. For instance, whether the motivation that fixed observations cannot always contain question-relevant information holds in practice, and whether for samples where improvement occurs it correlates with this motivation. A discussion on failure cases would also be informative.

Questions for authors

Regarding paper presentation, I also found several keywords and terminologies that are quite uncommon and may bring confusion for readers.

Below I share a few examples (it is a non-exhaustive list):

  • The term “leader” is used multiple times in the manuscript but is quite broad and could be replaced with a more precise term such as “leading answer” or “current best answer.”
  • “Answer presentation” is not clearly defined before it is used.
  • “Certificate fires” is not a common phrasing. It could be replaced with something more standard such as “certification condition is met.”
Reviewer_dd8tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that chain-of-thought reasoning on video is unreliable for a reason that doesn't apply to images: the sampled frames may simply not contain the evidence. Repeated sampling on the same frames can then agree confidently on a wrong answer, and the answer can flip once more frames are added. RRV addresses this at two levels. Within one set of frames, it samples trajectories over several option orderings, weights each by token entropy (taking the more uncertain of the "evidence extraction" and "answer reasoning" spans), and stops early once the leading answer can no longer be overtaken. Across frame sets, it moves through a nested 16/32/64-frame schedule and returns as soon as two adjacent levels agree on a certified answer; otherwise it falls back to the 64-frame result. A short theoretical section shows the early-stopping certificate never changes the full-budget answer and bounds the cost of the nested schedule. Experiments on three 7B-scale video MLLMs and eight multiple-choice benchmarks show average gains of 1.2 to 2.1 points over self-consistency, with adaptive stopping cutting output tokens roughly in half.

Critical strengths
  1. The core observation is a good one, and the paper backs it up. Table 9 and Figure 1(b) show that adding frames flips answers far more often than re-sampling on the same frames, and the gap is largest for answers that were wrong to begin with. That experiment would be worth having even without the method.
  2. RRV needs no training and no access to model internals beyond token probabilities, so it drops onto frozen models that look quite different from each other, including the tool-using VideoTemp-o3.
  3. The experimental setup is thorough. Three backbones, fifteen benchmarks, and a baseline set that includes the recent confidence-weighted SC methods (CISC, DeepConf, RTWI), all run with the same frames, prompts, and decoding.
  4. I appreciated how upfront the authors are. They say in the main text that compute is not matched, that the weighting rule has no clear edge, and that stopping hurts on some benchmarks. That made the paper much easier to assess.
Critical weaknesses
  1. The comparison isn't compute-matched. RRV uses several times the trajectories and tokens of the baselines, and there is no SC variant with a matched budget, matched option orderings, or simple multi-scale pooling to separate the method's effect from the effect of more samples.
  2. The entropy weighting doesn't seem to matter. Uniform weights and the proposed rule are within 0.4 points of each other, and σ has almost no effect, so the stage-wise uncertainty the abstract emphasizes is not what's driving the gains.
  3. Cross-observation stopping hurts on several long-video benchmarks. It lowers accuracy on Video-MME, VideoTemp-Bench, and MLVU for all three backbones, which is at odds with the motivation, and the paper doesn't explain why.
  4. The novelty over RTWI is unclear. The within-observation module borrows RTWI's top-k token entropy and stage-wise scoring (the paper cites Li et al., 2026 for its entropy estimate), and the related-work paragraph only says RRV "stabilizes the answer within each observation" before comparing across frame sets, without spelling out what is actually new.
Questions for authors
  1. Why does stopping consistently lose accuracy on Video-MME, VideoTemp-Bench, and MLVU? And for the questions that return early at 32 frames, what is the accuracy if you force them through to 64?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.