- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper compares five signals for accepting or abstaining on video question-answering outputs: generation confidence, sampling self-consistency, a 32B vision-language judge, frame-subset agreement, and temporal perturbation probes. Using 36000 labeled candidate answers from 7B models across four benchmarks, it finds that confidence and self-consistency generally discriminate correct from incorrect answers better than the tested judge, while combining signals yields limited gains. It also shows that naive split-conformal gates can exceed nominal false-accept bounds when scores are tied, and that randomized tie-breaking produces results consistent with those bounds. The contribution is primarily an empirical comparison and an analysis of gating behavior, rather than a new model or verification algorithm.
Critical strengths
-
Broad, cost-aware empirical comparison. The study evaluates five verification-signal families on 36000 labeled candidates from two models and four benchmarks, with uncertainty estimates and supplementary controls. This provides a useful comparative baseline and practical evidence that existing generation signals can reduce the need for expensive verification, giving the paper value beyond a single-dataset result.
-
Identification of a consequential gating failure. The paper shows how tied scores can cause naive threshold-based gates to exceed nominal false-accept bounds and evaluates randomized tie-breaking as a remedy. This matters because discrimination performance alone does not establish reliable acceptance decisions; the analysis adds a practically relevant failure mode and mitigation to the empirical comparison.
Critical weaknesses
Insufficient methodological or conceptual advancement. Although the paper conducts extensive experiments, I find its contribution closer to a technical report or an exploratory blog than a top-tier conference paper. Its central contribution is a performance comparison of existing reliability signals under a shared evaluation setup. This is useful as practical guidance, but the paper does not turn these observations into a substantially better method for assessing answer reliability or a new framework that advances our understanding of the problem.
My concern is not that an empirical paper must necessarily propose a new algorithm. Rather, without such an algorithm, the findings need to establish a sufficiently substantial conceptual advance. Here, the evidence that inexpensive generation signals outperform the tested alternatives, together with the limited gains from combining them, remains primarily a setting-specific comparative result. This gap is the main reason I do not consider the paper sufficient for acceptance.
Questions for authors
Please see the weaknesses.
2 comments
Thank you for the clarification. The additional evidence strengthens the paper's empirical and practical value, and I appreciate the diagnosis of the temporal signing rule and the quantification of tie-related gating failures.
However, my main concern remains the extent of the conceptual advance, rather than the robustness of the measurements. The response does not yet establish that these findings provide sufficiently generalizable insights beyond the evaluated setting. For a future version, I would encourage the authors to develop these diagnostic findings into generalizable principles for verifier design and validate them beyond the current evaluation setting.
Response to Reviewer VN79
Thank you for the review.
The conceptual contribution. The paper turns a collection of signals into a measured answer to one deployment question: which signal should decide whether to act on a video answer, and at what cost. The answer has four parts, each tied to a number.
First, the signals a deployment already holds rank answers best. Confidence and self-consistency are above the 32B judge in all 8 cells by point estimate. Their paired intervals exclude zero in 6 of 8 cells. Scoring every candidate with the judge, on a 20% stratified subset, raises its mean AUROC by 0.021, to 0.665. On that subset the cell means for the free signals are 0.728 and 0.736, and the paired intervals favor the free signal in 7 of 16 contrasts and favor the judge in none.
Second, in 7 of 8 cells every video-specific signal that applies there has a lower AUROC point estimate than confidence, self-consistency and the judge. Combining all five signals adds +0.053 AUROC in the best cell.
Third, the temporal result has a diagnosis. The temporal rule signs agreement by the benchmark's temporal flag. Section 5 shows that the signing rule is the failure. Raw agreement, scored without the sign in an exploratory analysis, gives AUROC 0.513 to 0.625, with an interval above chance in 5 of 6 cells. This yields a test for any verifier: check whether the score orientation holds in the subgroup where a rule changes it.
Fourth, the gating failure is measured and traced to its cause. The naive split-conformal gate exceeds its bound in 109 of 114 single-signal settings (95.6%), by 27.1 points on average among those above the bound. On the stored records, the largest tie share of a signal and the naive excess rise together (Spearman 0.71, 0.79 and 0.82 at alpha 0.05, 0.1 and 0.2, across 38 cell and signal pairs). Under randomized tie-breaking, the mean excess over the 114 single-signal settings is 0.1 points, and the largest over all 162 settings is 2.2.
The free signals stay above chance under a second sampling seed and at 3B and 32B. The paper releases the 36,000-candidate scored pool, so a new method can be compared with these baselines and gates without a GPU.