acceptodds
Under review as a conference paper at ICLR 2027

Free Generation-Time Signals Gate a Video LLM Better Than a 32B Judge

Abstract

Video large language models sample many candidate answers, and a large share of those candidates are wrong, so a deployed system has to decide for each answer whether to act on it. We ask which verification signal should make that decision. Two signals the sampling protocol already produces, the model's own confidence and its self-consistency across samples, separate right answers from wrong ones better than a 32B vision-language judge run once per question: they beat the judge in 6 of 8 model-dataset cells with question-level bootstrap intervals excluding zero, at no inference cost beyond the samples already drawn. The evidence is a scored pool of 36,000 labelled candidates from two open backbones on four benchmarks, run through five families of verification signal (confidence, self-consistency, the judge, frame-evidence agreement, and temporal metamorphic probes), each with a finite-sample false-accept gate and its inference cost recorded. The two video-specific families are the weakest, and combining all five adds little over the best single signal. Gating on any of these signals needs one repair: on tied multiple-choice scores a naive split-conformal gate exceeds its false-accept bound in nearly every single-signal setting, and standard randomized tie-breaking restores validity within seed noise. We release the scored pool so that a new signal or gate can be compared against the free baselines without a GPU.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video language models, selective prediction, conformal prediction, uncertainty estimation, LLM-as-judge, certified false-accept rate

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VN79OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper compares five signals for accepting or abstaining on video question-answering outputs: generation confidence, sampling self-consistency, a 32B vision-language judge, frame-subset agreement, and temporal perturbation probes. Using 36000 labeled candidate answers from 7B models across four benchmarks, it finds that confidence and self-consistency generally discriminate correct from incorrect answers better than the tested judge, while combining signals yields limited gains. It also shows that naive split-conformal gates can exceed nominal false-accept bounds when scores are tied, and that randomized tie-breaking produces results consistent with those bounds. The contribution is primarily an empirical comparison and an analysis of gating behavior, rather than a new model or verification algorithm.

Critical strengths
  1. Broad, cost-aware empirical comparison. The study evaluates five verification-signal families on 36000 labeled candidates from two models and four benchmarks, with uncertainty estimates and supplementary controls. This provides a useful comparative baseline and practical evidence that existing generation signals can reduce the need for expensive verification, giving the paper value beyond a single-dataset result.

  2. Identification of a consequential gating failure. The paper shows how tied scores can cause naive threshold-based gates to exceed nominal false-accept bounds and evaluates randomized tie-breaking as a remedy. This matters because discrimination performance alone does not establish reliable acceptance decisions; the analysis adds a practically relevant failure mode and mitigation to the empirical comparison.

Critical weaknesses

Insufficient methodological or conceptual advancement. Although the paper conducts extensive experiments, I find its contribution closer to a technical report or an exploratory blog than a top-tier conference paper. Its central contribution is a performance comparison of existing reliability signals under a shared evaluation setup. This is useful as practical guidance, but the paper does not turn these observations into a substantially better method for assessing answer reliability or a new framework that advances our understanding of the problem.

My concern is not that an empirical paper must necessarily propose a new algorithm. Rather, without such an algorithm, the findings need to establish a sufficiently substantial conceptual advance. Here, the evidence that inexpensive generation signals outperform the tested alternatives, together with the limited gains from combining them, remains primarily a setting-specific comparative result. This gap is the main reason I do not consider the paper sufficient for acceptance.

Questions for authors

Please see the weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer VN79

Thank you for the review.

The conceptual contribution. The paper turns a collection of signals into a measured answer to one deployment question: which signal should decide whether to act on a video answer, and at what cost. The answer has four parts, each tied to a number.

First, the signals a deployment already holds rank answers best. Confidence and self-consistency are above the 32B judge in all 8 cells by point estimate. Their paired intervals exclude zero in 6 of 8 cells. Scoring every candidate with the judge, on a 20% stratified subset, raises its mean AUROC by 0.021, to 0.665. On that subset the cell means for the free signals are 0.728 and 0.736, and the paired intervals favor the free signal in 7 of 16 contrasts and favor the judge in none.

Second, in 7 of 8 cells every video-specific signal that applies there has a lower AUROC point estimate than confidence, self-consistency and the judge. Combining all five signals adds +0.053 AUROC in the best cell.

Third, the temporal result has a diagnosis. The temporal rule signs agreement by the benchmark's temporal flag. Section 5 shows that the signing rule is the failure. Raw agreement, scored without the sign in an exploratory analysis, gives AUROC 0.513 to 0.625, with an interval above chance in 5 of 6 cells. This yields a test for any verifier: check whether the score orientation holds in the subgroup where a rule changes it.

Fourth, the gating failure is measured and traced to its cause. The naive split-conformal gate exceeds its bound in 109 of 114 single-signal settings (95.6%), by 27.1 points on average among those above the bound. On the stored records, the largest tie share of a signal and the naive excess rise together (Spearman 0.71, 0.79 and 0.82 at alpha 0.05, 0.1 and 0.2, across 38 cell and signal pairs). Under randomized tie-breaking, the mean excess over the 114 single-signal settings is 0.1 points, and the largest over all 162 settings is 2.2.

The free signals stay above chance under a second sampling seed and at 3B and 32B. The paper releases the 36,000-candidate scored pool, so a new method can be compared with these baselines and gates without a GPU.

Thank you for the clarification. The additional evidence strengthens the paper's empirical and practical value, and I appreciate the diagnosis of the temporal signing rule and the quantification of tie-related gating failures.

However, my main concern remains the extent of the conceptual advance, rather than the robustness of the measurements. The response does not yet establish that these findings provide sufficiently generalizable insights beyond the evaluated setting. For a future version, I would encourage the authors to develop these diagnostic findings into generalizable principles for verifier design and validate them beyond the current evaluation setting.

Reviewer_YsUJOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies candidate-answer gating for video question answering: after a model generates an answer, a reliability score determines whether to accept it or abstain. The authors collect 36,000 candidate answers from two 7B video-language models on four benchmarks and compare five signal families—generation confidence, self-consistency, a 32B model judge, frame-subset agreement, and temporal-order perturbations—in terms of discrimination, combination gains, and inference cost. In the main experiment, which generates nine candidates per question, both generation-time signals achieve higher AUROC point estimates than the judge in all eight model–dataset combinations. In six combinations, paired confidence intervals support the advantages of both signals. A control that judges each candidate individually on a subset narrows the gap while preserving the overall trend, and combining all five signals provides limited additional gains. The paper also demonstrates failures of naive thresholding with heavily tied scores and shows that randomized tie-breaking reduces false-accept rates at the cost of lower coverage. The appendix additionally evaluates question-level calibration to address dependence among candidates from the same question.

Critical strengths
  1. The unified comparison helps assess practical utility. Comparing discrimination, combination gains, and additional inference cost on the same candidate pool, while providing candidate scores and labels in the supplementary material, offers a reusable reference for future verification methods.
  2. The controls and error analyses are thorough. The authors examine per-candidate judging, a second sampling seed, randomized tie-breaking, and question-level calibration. They also report the coverage cost of risk control, making the findings easier to assess and verify.
Critical weaknesses
  1. Methodological novelty is limited. Similar approaches to confidence, self-consistency, model-based judging, input perturbations, and handling tied scores have been studied previously, so the paper's value largely depends on whether the comparison provides sufficient new insight. The authors include model-scale and 16-frame checks, but neither re-evaluates all five signal families. The main ranking therefore still comes from eight-frame multiple-choice and yes/no experiments with two 7B backbones. I find the resulting guidance on choosing verification signals across deployment conditions insufficient.
  2. The per-candidate judge control does not fully quantify uncertainty in the central ranking. The main experiment judges only the greedy answer and broadcasts its score to all candidates. Appendix E re-evaluates each candidate on a roughly 20% subset, increasing the judge's mean AUROC by 0.021; each free signal still has higher point estimates than the judge in six of the eight combinations on that subset. However, the confidence intervals in Table 12 concern the difference between per-candidate and broadcast judging, while Table 11 does not report paired difference intervals between the per-candidate judge and the free signals. The results support the overall trend but do not establish which advantages in this direct comparison are robust to sampling variation.
Questions for authors
  1. On the same subset used in Table 11, could the authors report paired AUROC differences and question-level bootstrap confidence intervals between the per-candidate judge and each free signal, to clarify which advantages exceed sampling variation?
  2. Using the existing candidates, does the ranking of confidence, self-consistency, and the judge hold when only the greedy answer from each question is evaluated? It would also help to compare the error rate among accepted answers at fixed coverage and include the additional sampling required for self-consistency in the total cost. Section 4.4 already states this sampling overhead; the requested analysis would connect it to the corresponding performance–cost comparison.
3 comments
AuthorsOpenReview ↗

Response to Reviewer YsUJ

Thank you for the review.

Paired intervals on the Table 11 subset (Question 1). On the stored records, we computed them on the same candidates. Each entry is the AUROC of a free signal minus the AUROC of the per-candidate judge, with a pointwise 95% question-level bootstrap interval (20,000 draws, all candidates of a drawn question kept together). The draws are those that reproduce Table 12. The new table is in the revised appendix, in all 8 cells. The free signal has the higher point estimate in 12 of the 16 contrasts. The interval excludes zero in favor of the free signal in 7 of 16: confidence in 4 cells and self-consistency in 3. No interval excludes zero in favor of the judge.

Greedy answers only (Question 2). On the stored records, we scored the greedy answer of each question with all three signals. This uses all 3,976 greedy answers that carry all three scores (487 to 500 per cell). The mean greedy-answer AUROC over the 8 cells is 0.726 for confidence, 0.682 for self-consistency and 0.645 for the judge. By point estimate, confidence is above the judge in 7 of 8 cells and self-consistency in 6. The paired interval excludes zero in favor of each free signal in 4 cells and in favor of the judge, against both free signals, in 1 cell (LLaVA-OV on VideoHallucer). At 50% coverage, the error among accepted answers is 20.8%, 22.8% and 25.4% on average. By point estimate, confidence is lower than the judge in 5 cells and self-consistency in 6. At 70% coverage the three values are 26.6%, 27.9% and 28.6%, and at 90% they are 32.6%, 32.9% and 33.4%. Confidence is below the judge in 6 and 7 cells at these two levels, and self-consistency in 6 and 5. Cutoff ties are broken uniformly in expectation. The per-cell values and the interval counts are in Appendix L.

Total cost. Confidence adds no model call. The eight extra draws for self-consistency cost 1.64 to 1.83 GPU-seconds per question. The two numbers come from allocating the logged generation time over the 4,000 planned model-question groups or over the 3,588 timed ones (Appendix L). The 32B judge costs 10.4 GPU-seconds per question as measured and 17.8 under its declared protocol. Under the first allocation, 2 and 4 draws cost 0.41 and 0.82 GPU-seconds, with mean greedy-answer AUROC 0.614 and 0.647, against 0.682 with 8 draws (Appendix L). So the sampling for self-consistency costs less than the judge, and confidence has no sampling cost.

Scope and guidance. The paper measures all five families on one pool. The guidance by deployment condition is in Section 6. If the system already draws a pool, confidence and self-consistency cost nothing. If it returns a single answer, self-consistency costs the amount above. If a finite-sample bound on P(accept | wrong) is required, assume exchangeable questions, calibrate on one candidate drawn per question and use the randomized gate on a free signal; the bound then applies to a candidate drawn by the same rule on a new question, up to the plug-in treatment of ties. Combining all five signals adds +0.053 AUROC over the best single signal in the best cell. Under the second sampling seed and at 3B and 32B, the free signals' intervals stay above chance.

Thank you for the response. The additional analyses address my concerns about the paired comparisons with the per-candidate judge and the greedy-only evaluation. However, I remain concerned about the limited novelty of the overall approach and therefore maintain my original rating.

AuthorsOpenReview ↗

What the comparison adds beyond the individual signals

Thank you for checking the new analyses and for confirming that they resolve the two technical concerns.

On novelty: each signal family is known, as the review says. What the paper adds is a set of measured answers that were not available for gating video answers. Each is tied to a number in the revised PDF.

  1. The ordering between the free signals and a large judge. On one pool of 36,000 candidates, confidence and self-consistency rank answers above a 32B judge in all 8 cells by point estimate, and their paired intervals exclude zero in 6 of 8 cells. The two controls you asked for keep this ordering: on the per-candidate subset, the paired intervals favor the free signal in 7 of 16 contrasts and favor the judge in none; on greedy answers alone, the mean AUROC is 0.726 and 0.682 against 0.645 for the judge. On this pool, the signals a deployment already holds rank its answers better than an extra call to a 32B judge.

  2. A failure of the standard certificate that AUROC does not show. The naive split-conformal gate exceeds its bound in 109 of 114 single-signal settings (95.6%). The largest tie share of a signal and the naive excess rise together (Spearman 0.71, 0.79 and 0.82 at alpha 0.05, 0.1 and 0.2, across 38 cell and signal pairs). Under randomized tie-breaking, the mean excess over the 114 settings is 0.1 points. The cause is a property of the score, not of video, so the check applies to any gate on tied or saturated scores.

  3. Why the video-specific signals fail. The temporal rule signs agreement by the benchmark's temporal flag, and Section 5 shows that the signing rule is the failure. Raw agreement, scored without the sign in an exploratory analysis, gives AUROC 0.513 to 0.625, with an interval above chance in 5 of 6 cells. This gives a test for any verifier: check whether the score orientation holds in the subgroup where a rule changes it.

  4. A reusable baseline. The released pool carries the five signal scores and the label of each of the 36,000 candidates, so a new verifier can be compared with these baselines and gates without a GPU.

We hope these four results show what the comparison establishes beyond the individual signals.

Reviewer_nnzAOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to verify Video LLM answers by comparing five verification signals across two models and four benchmarks. It finds that simple confidence and self-consistency outperform a larger VLM judge and video-specific signals, while combining signals brings limited gains. It also shows that score ties can break naive conformal gating, which can be addressed with randomized tie-breaking.

Critical strengths
  1. The paper studies a meaningful problem: how to efficiently verify Video LLM answers. Understanding which verification signals are reliable and whether additional inference cost is justified could be useful for real-world deployment.

  2. The finding that simple confidence and self-consistency outperform the tested, more expensive verification methods is interesting and meaningful. It suggests that more complex verification methods do not necessarily provide better reliability estimates, which has practical implications for efficient Video LLM verification.

Critical weaknesses
  1. Limited representativeness of the evaluated verification methods: The paper's main finding is interesting, but it is unclear whether the tested methods are representative of existing Video LLM verification approaches. In particular, the external judge and video-specific signals rely on relatively simple implementations. This makes it difficult to generalize the conclusion beyond the specific methods evaluated.

  2. Limited novelty and insights: While the empirical findings are useful, the paper provides limited new understanding beyond showing that simple verification signals perform better in the evaluated settings. The conformal gating analysis also mainly demonstrates known issues with score ties and an established solution using randomized tie-breaking.

  3. Presentation and clarity: The presentation and writing are a bit hard to follow, with little connection and flow between sentences, especially in intro and related-work.

Questions for authors
  1. How representative are the selected verification methods of existing Video LLM verification approaches? In particular, the frame-based and temporal signals rely on specific heuristic designs. Could the authors discuss whether the main findings are likely to generalize to stronger or alternative verification methods?
  2. How do the improvements in candidate-level correctness prediction translate into better end-to-end VideoQA performance? For example, how would these verification signals affect the final answer accuracy and selective risk–coverage trade-off when the system needs to return a single answer or abstain?
  3. Since the effect of score ties and randomized tie-breaking are already well-established in conformal prediction, what new insight does the gating analysis provide beyond demonstrating these known effects on VideoQA benchmarks?
1 comment
AuthorsOpenReview ↗

Response to Reviewer nnzA

Thank you for the review.

Representativeness (Question 1). The paper tests five specified signal families on one pool of 36,000 labeled candidates, with the cost of each recorded. The free signals are the ones a deployed system already has. The free-signal results hold under several changes. A second sampling seed leaves all 16 verdicts of the free signals' intervals against chance unchanged, with a largest AUROC shift of 0.040. Confidence and self-consistency stay above chance at 3B, 7B and 32B, in all 24 intervals (Appendix F). A stronger deployment of the judge is also tested: scoring every candidate, at 9 times the calls, on a 20% stratified subset raises its mean AUROC by 0.021. The cell means are 0.665 for the judge, 0.728 for confidence and 0.736 for self-consistency. Section 5 diagnoses the temporal signal: raw agreement, scored without the signing rule in an exploratory analysis, gives AUROC 0.513 to 0.625, with an interval above chance in 5 of 6 cells. The released pool lets a new verifier be compared with the free baselines without a GPU.

Final answer and abstention (Question 2). On the stored records, we tested one fixed greedy answer with abstention. At full coverage every signal returns the same greedy answer. The signals decide which answers to withhold. Appendix L (new table) gives the error among accepted answers at 50%, 70% and 90% coverage in each of the 8 cells, on all 3,976 greedy answers that carry all three scores. At 50% coverage, confidence has lower error than the judge in 5 cells and self-consistency in 6, by point estimate. Averaged over the 8 cells, the error is 20.8% for confidence, 22.8% for self-consistency and 25.4% for the judge. At 70% coverage the three values are 26.6%, 27.9% and 28.6%, and at 90% they are 32.6%, 32.9% and 33.4%. Confidence is below the judge in 6 and 7 cells at these two levels, and self-consistency in 6 and 5. The mean greedy-answer AUROC is 0.726, 0.682 and 0.645. Confidence is above the judge in 7 of 8 cells and self-consistency in 6. Appendix L gives the interval counts. When the system must return one answer, mean final-answer accuracy over the 8 cells on these 3,976 questions is 64.5% for the greedy answer, 64.7% for majority vote and 64.8% for top-confidence selection; neither paired difference from greedy has an interval excluding zero. Confidence needs no extra model call. Self-consistency needs 8 extra draws, which cost 1.64 to 1.83 GPU-seconds per question under the two allocations in Appendix L, which also reports 2 and 4 draws. The 32B judge costs 10.4 GPU-seconds per question as measured (17.8 under its declared protocol).

What the gating analysis adds (Question 3). The paper measures the size of the tie effect on video scores and tests its cause. The naive gate's seed-mean false-accept rate exceeds its bound in 109 of 114 single-signal settings (95.6%). Among the settings above the bound, the mean excess is 27.1 percentage points and the maximum is 68.1. One-sided tests with Bonferroni correction reject the bound in 106 of 138 naive settings and in 0 of 138 randomized settings. Under the randomized gate, the mean excess over the 114 single-signal settings is 0.1 points, and the largest over all 162 settings is 2.2. On the stored records, the excess rises with the share of tied scores: across 38 cell and signal pairs, the Spearman correlation between the largest tie share and the naive excess is 0.71, 0.79 and 0.82 at alpha 0.05, 0.1 and 0.2. Decoding at a higher temperature lowers the tie share, and the naive excess for self-consistency falls with it in all 4 cells tried (Section 5, Appendix F). The conclusion holds on the 3B and 32B pools of the two free signals. Section 4.3 gives the mean coverage of each gate over the 162 settings, and Table 8 gives the randomized gate's coverage at alpha 0.1 in each cell for every signal.

Presentation. The introduction states the four questions in one sentence (RQ1 to RQ4). Figure 1 shows ranking in panel (a) and gate calibration in panel (b). The results sections follow the same order.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Which findings hold beyond the 7B setting

Thank you for the reply and for staying open to further discussion. On whether the findings go beyond a setting-specific comparison: the revised PDF tests the gating findings outside the 7B setting (Appendix F, Tables 13, 14 and 16).

  1. AUROC does not decide whether a gate keeps its bound. At 32B, confidence and self-consistency have AUROCs within 0.04 of each other on all four benchmarks (Table 13). Through the naive split-conformal gate at alpha 0.1, their seed-mean excess over the bound differs by 9.3 to 26.9 points (Table 14): +14.0 to +27.4 for self-consistency, +0.5 to +5.6 for confidence. A verifier chosen by AUROC alone can therefore break its certificate. What decides it is the tie structure of the score, so the check applies to any score with few distinct values.

  2. The tie account is also checked by changing the decoding. Raising the sampling temperature from 0.7 to 1.0 lowers the largest tie share of self-consistency in all 4 cells tried (for example, 63.8% to 49.8%), and the naive excess falls with it (+17.7 to +5.7 points in that cell; Table 16).

  3. The failure does not shrink with a larger backbone. On the 3B and 32B pools, the naive gate exceeds alpha in 15 of 16 (pool, signal) entries, by +6.6 points on average and up to +27.4. For self-consistency, the excess at 32B is larger than at 3B on all four benchmarks. The randomized gate's excess is +0.5 points on average and at most +1.5, within seed noise. Both free signals stay above chance at all three scales, in all 24 intervals.

These are statements about scores and certificates, measured on two backbone families, three scales and four benchmarks. They give two checks to run before deploying any gate: the tie share of the score, and whether the score orientation holds in the subgroup where a rule changes it (Section 5).

Thank you for the careful response, especially the greedy-answer abstention results and the additional checks across model scales and sampling seeds. These make the empirical comparison clearer, but I still see the main contribution as a fairly setting-specific comparison of existing signals, and the conformal tie-breaking result as an application of a known fix, so I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.