- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This article re models long video Q&A as an iterative hypothesis testing process and proposes an EOR framework. The core idea is to break down each candidate answer into several "ongoing evidence obligations" that need to be confirmed item by item in the source video, and maintain three states of pending/satisfied/refuted for each obligation, which are persistently retained between inference rounds, thus making the question of "whether the evidence is sufficient" explicit, rather than staying at "whether the retrieved fragments are relevant". The framework introduces a lightweight time event graph on the retrieval side to locate candidate intervals, and then returns to the source video to determine the obligation state using validators; And use a hypothesis selection mechanism that covers perception to allocate verification costs within a limited verification budget, and stop after meeting the preset grounding conditions; For open-ended questions, it is also allowed to dynamically evolve the hypothesis space driven by evidence.
The paper attempts to isolate the marginal contribution of the "persistent verification state" through a progressive ablation chain (from pure retrieval, to hypothesis retrieval, to instantaneous verification, and then to the complete framework), and verifies its effectiveness on two types of long video Q&A tasks: multiple-choice and open. Overall, its contribution mainly remains at the level of problem modeling and process design, while the empirical evidence supporting these claims is relatively weak.
Critical strengths
Witness the forefront clearly retaining complementary bottleneck profiles, rather than collapsing progress to a single incumbent; This design choice is highly consistent with the minimum margin structure achieved jointly, and is consistently reflected in the early hit behavior reported. Separate language proposals from posterior risk screening to keep the LLM output at a modifiable decision level and avoid contaminating the surrogate model with pseudo observations.
Critical weaknesses
This paper should be rejected.
W1. Deadliest: There is no positive comparison with the most relevant methods. The entire article repeatedly identifies "Think, Then Verify" as the "closest competitor" and claims that the difference between the two lies in "clue level goals vs. persistent obligations". It is unacceptable for an article to use "validation driven" as a selling point and clearly state that it is the primary benchmark, without any quantitative comparison of the strongest and most relevant baselines. Readers have no way of determining whether the benefits come from the new state of 'persistent obligations' or just from a few extra verification calls. This deficiency directly undermines the entire proposition.
W2. Headline numbers are abnormal and lack comparability evidence. The backbone model used in the framework has a limited scale, and the reported performance has shown an unusually large leap compared to the training free methods of the same backbone, even surpassing systems based on stronger closed source large models. Such a significant increase should have been accompanied by an extremely strict "evidence ledger", but on the contrary: the core selling point of the paper is "comparable video access budget/evidence acquisition efficiency", but it does not report the actual video frame rate, retrieval token, average access frame rate per call, or wall clock time of any method from beginning to end. On the premise that the access cost is completely unquantifiable, it cannot be ruled out that high accuracy is achieved through more aggressive video access, and "comparable budget" has become a verbal statement without data support. In addition, a baseline gear number in the ablation table is identical to an external method number in the main table, suggesting direct appropriation rather than independent reproduction, further weakening the credibility of the baseline.
W3. There is inherent tension and even logical conflict in the core ablation data. Compared to its simplified variants, the complete framework has significantly reduced verification calls, but the coverage of obligations confirmed as "fulfilled" has significantly increased, resulting in a synchronous increase in accuracy. There is less verification cost and more obligation to confirm. In the absence of mechanism level evidence to explain how to confirm more obligations with fewer calls, this is more like a loosely triggered "satisfied" judgment rather than a true coverage improvement. The author did not provide any explanation on this matter. At the same time, there is a lack of consistent reconciliation between the marginal revenue caliber presented by the budget curve and the "extremely few calls reaching high levels" described in the ablation table, and multiple numerical calibers are inconsistent with each other.
W4. There is no reliability assessment for the method foundation (obligation decomposition and support/future relationship labels). The relationship label is specified and cached by the model at once during creation, and does not determine whether the video truly contains this condition. This means that subsequent discriminative power calculations, coverage statistics, landing criteria, and even final ranking are all based on these initial labels; Once the label is incorrect, the entire production line runs on the wrong foundation. The author did not make any manual annotations or quality statistics on whether obligations are truly verifiable, whether they are just restatements of answers, and whether relationship labels are accurate, and did not even report the accuracy of obligation decomposition. This is the first domino of reliability, but it is completely hanging in the air.
W5. Validator and generator share the same origin, essentially being the self sealing of the model. The event clues, obligation decomposition, verification, and final implementation are all completed by the same model, and each event only samples a very small number of frames; The validator reports its confidence level, and the landing threshold is set quite loosely. Allowing a model with limited abilities to disassemble problems, search for evidence, and determine that the "evidence has been satisfied" on its own will inevitably result in extremely high noise levels in its "satisfied" label. The so-called 'high coverage of verification obligations' is likely due to overconfidence and incorrect stamping, rather than truly covering decisive evidence. The paper has zero ablation on the accuracy, precision/recall, and how the results change after switching to a stronger validator, and the entire commitment of "validation driven" is built on an unverified and highly likely unreliable validator.
W6. The landing criteria are too loose, making it almost impossible to intercept erroneous assumptions. The coverage rate and weighted evidence threshold required for landing judgment are both very low, and the scoring is conducted in the low resolution interval, resulting in a very small margin for distinguishing between the two candidate hypotheses; A mistaken assumption only needs to gather a very small number of obligations that have been misjudged as' fulfilled 'to pass. The paper neither provides a false positive rate for ground detection nor proves that this threshold can prevent 'confident incorrect answers'.
W7. Promise only, not fulfilled. The main text explicitly states that the triggering frequency of this fallback mechanism will be reported in the experiment, but there is no fallback frequency number throughout the main text and appendix. The absence of indicators in the commitment report itself is an issue of academic rigor.
W8. Multiple long video benchmarks for motivation naming have not been established, and there is no evidence of generalization. The abstract and introduction are motivated by the "difficulty" of multiple mainstream long video benchmarks, but the actual measurement only covers one multiple-choice dataset (and only its subset) and one open-ended dataset. Motivation and verification are severely disconnected; The sample size is relatively small, and there are no standard deviations/multiple runs/significance tests in the entire text. Without a confidence interval, the extent of improvement is not convincing.
W9. "Efficiency" advocates no ledger, and the active set strategy is not compared with the naive control. The greedy assumption of coverage perception is not compared with simple strategies such as ranking by score, randomness, and polling, which cannot prove that the benefits come from "coverage perception" rather than "leaving a few more assumptions arbitrarily". The so-called evidence acquisition efficiency does not have token/frame/delay ledger, and cannot evaluate the true inference cost.
W10. The evidence for the open-ended question "Hypothesis Evolution" is weak, and the reliability of the evaluator has not been verified. The complete evidence of the evolutionary mechanism consists of only two extremely small differences in numbers, without analysis of the triggering rate of evolution, the accuracy of new candidates, and the quality of the hypothesis pool before and after evolution. More importantly, if an open-ended question is scored by another large model acting as a judge without reporting consistency with manual evaluation, then the credibility of the open-ended question indicator itself is questionable.
W11. Lack of reproducibility and sensitivity. A large number of thresholds are fixed on both datasets, but there is no sensitivity analysis for key thresholds that have a significant impact on the results, and the paper does not promise code/data release. Highly customized systems are almost impossible to independently reproduce.
Questions for authors
(W1, most importantly) Please provide a direct quantitative comparison between EOR and the most relevant competitive methods under the same backbone and video access budget; If there is no such comparison, how can the net gain of "persistent obligation" relative to clue level verification be proven?
(W2) Please provide the actual video access cost ledger for each method (average access frame rate, token retrieval, average frame rate per verification call, end-to-end wall clock time), and indicate whether the headline results were obtained under true "comparable access".
(W3) Please explain the mechanism of "reducing verification calls and increasing coverage of fulfilled obligations" in ablation, and provide a decomposition of "how many obligations are confirmed on average per call" to prove that this is not triggered by excessive relaxation of "fulfilled".
(W4) Please provide a manual evaluation of the quality of obligation decomposition: the proportion of "verifiable/non answer restatement" obligations, and the accuracy of support/future/neutral relationship labels relative to manual annotation.
(W5/W6) Please report the accuracy of the validator's own judgment (accuracy/recall/F1 of support/future/uncertainty), and how the results change after switching to a stronger validator; And provide the false positive rate of landing judgment.
(W7) Please provide the promised but not given rollback triggering frequency and its impact on final performance in the main text.
(W8) Please reproduce the headline conclusion on at least one other long video benchmark and report the variance and significance of multiple runs.
(W10) Please report the consistency between automatic refereeing and manual evaluation, otherwise the open question indicators are unreliable.
(W11) Please explain the basis for the value of the key threshold and provide a sensitivity curve, and clarify whether the code is open source.