acceptodds
Under review as a conference paper at ICLR 2027

Certified Selective Gates for Video-Language Models Exceed Their Error Target Before Genre Drift Begins

Abstract

A selective gate lets a frozen video-language model answer only when its error rate on answered questions is certified below a target α, and the standard repair when the genre mix drifts is to detect the break and recalibrate on fresh labels. We find that on most drift streams there is nothing for the repair to restore. A split-conformal gate certifies the false-accept rate, and selective error equals that rate times the base error rate divided by coverage, so at the certified level the gate is over α wherever coverage falls below the base error rate, before any drift begins. We test this on 5 backbones (3B–32B) and 4 video-QA benchmarks with a per-genre certified gate, a label-free coverage-drift monitor, and drift streams built from benchmark metadata. The frozen gate is already over α before the break on 30 of 35 stream cells. Where α did hold before the break, recalibration on a deployment-sized label budget restores it on none, an oracle that recalibrates at the true break with no delay does no better, and a pre-registered sweep to a larger budget shows that more labels help but do not restore the target; the holds that do occur are thin under question-clustered upper bounds. A rule that certifies selective error directly is the honest alternative, and under a Bonferroni grid it is vacuous in most realizations at either budget, answering a third as often as the conformal rule. The quantity to certify, and the coverage regime in which certifying it is possible, are fixed by the identity before a label is collected.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
video-language models, selective prediction, distribution shift, conformal prediction, coverage monitoring, risk control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4D7JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines whether detecting distribution drift and recalibrating a selective gate restores a target error rate for frozen video-language models. Experiments cover five backbones and four video-QA benchmarks, using metadata-defined shifts, a coverage-based detector, and fresh-label recalibration. The central distinction is between the false-accept rate—the proportion of incorrect answers that the gate accepts—and selective error—the proportion of accepted answers that are incorrect. The monitored conformal gate controls the former, while the deployment target concerns the latter. Many evaluated streams already exceed the selective-error target before drift, and recalibration can continue to miss it even with an oracle change point. A separate Clopper–Pearson rule directly controls selective error but frequently abstains completely under the tested budgets and certification procedure.

Critical strengths
  • The paper makes an important distinction between the quantity a certificate controls and the quantity users care about. Reporting false-accept rate, coverage, base error, and selective error together makes the mismatch concrete. The pre-break results are particularly useful because they show that some apparent deployment failures precede any distribution shift.

  • The oracle and fresh-at-alarm experiments help separate recalibration from detection delay. I also appreciate the reporting of abstention, repeated questions, thin accepted samples, and unsuccessful pre-registered predictions. These details make the negative results easier to assess than an aggregate post-drift error comparison alone would.

Critical weaknesses
  • The main negative result has a narrower interpretation than the title and deployment framing suggest. The principal recalibration procedure renews a false-accept certificate rather than a selective-error certificate, so missing the selective-error target does not demonstrate a failure of its stated guarantee. The paper acknowledges this, but the experiment consequently establishes a limitation of this particular proxy-control procedure more directly than a limitation of detect-then-recalibrate generally. Selective error equals the false-accept rate multiplied by the base error rate and divided by coverage. This identity explains the observed outcomes, but coverage below the base error rate does not, by itself, imply that selective error exceeds the target: the conclusion also depends on the realized false-accept rate. An upper bound on that rate alone is insufficient to establish a violation. The empirical finding is useful; its scope needs to remain explicit.

  • The direct selective-risk baseline does not yet establish that useful certification is generally infeasible at these label budgets. It combines a 20-threshold Bonferroni correction, δ=0.02\delta=0.02, and permanent abstention after a vacuous recalibration. Under these settings, even a candidate with zero calibration errors needs at least 31 accepted calibration examples to certify a 20% error target. This is already restrictive at B=50B=50, before accounting for the score's ability to separate correct and incorrect answers. The paper acknowledges that fixed-sequence testing could be less conservative, but does not evaluate it. Without such a comparison at the same confidence level, the reported vacuity and coverage costs remain specific to the tested procedure, rather than evidence of an intrinsic budget limitation.

Questions for authors
  • How do the conclusions change when recalibration directly targets selective error using a less conservative testing procedure, such as a prespecified fixed-sequence scan, at the same confidence level? Please report certified coverage alongside realized error.

  • Can you distinguish the empirical coverage-versus-base-error observation from a guarantee? In particular, how often is the false-accept rate sufficiently below its nominal level that selective error remains acceptable despite coverage being below the base error rate?

  • Which reported recalibration failures have question-clustered lower confidence bounds above the target error rate? The upper bounds help assess evidence for successful control, but point estimates just above the target do not establish a violation. This matters for statements such as "restores it on none," especially given the acknowledged one-error margin in one cell.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4D7J

Thank you for the review.

Direct selective-error certification at these budgets. The abstract states the vacuity result under a Bonferroni grid, and Section 5 gives the 20-candidate grid at δ=0.02. The budget sweep shows how this rule responds to labels. The share of vacuous realizations falls from 100% at B=10 to 63.7% at B=150. The number of cells holding α rises from 0 to 3 of the 10 common cells (Table 2). Over the three backbones drawn in Figure 3, arms that certify selective error or FDR hold α in all but one of their 24 answering arm-cells, at a coverage price.

Fixed-sequence scan at the same δ. A post-review run replaces the Bonferroni grid by a fixed-sequence scan at the nominal δ=0.02 on the same fresh draws, monitor and 10 common cells (α=0.20, recalibration at the alarm). At B=50 the scan is vacuous in 82.2% of realizations (Bonferroni: 98.7%). Its mean post-recalibration coverage is 4.5% (0.5%) at pooled selective error 6.8% (8.8%), and it holds α in 3 of 10 cells (1). At B=150 it is vacuous in 63.1% (63.7%), with coverage 15.2% (13.4%), pooled selective error 10.9% (10.5%) and 4 of 10 cells holding (3). The familywise level of the scan at this δ is not proved, so the comparison is at the same nominal δ and is exploratory. Appendix N gives every budget and timing at α=0.20 and α=0.10.

Coverage below the base error rate. The identity is exact, and the paper uses it as a predictor, not as a guarantee. On the registered cells the sign rule predicts the verdict in 58 of 69 entries (84%). Of the 50 entries with coverage below the base error rate, 3 hold and 47 fail. Of the 19 entries at or above it, 11 hold; one of the 19 has no answered item. Appendix J gives all five backbones. The realized FAR lands within its recorded mean bound in 62 of 68 cell-arms with a defined FAR, and in 50 of the 54 whose selective error exceeds α (Figure 5c).

Question-clustered bounds. Among the 5 alarmed cells screened from 25 seeds where α held before the break, the question-clustered one-sided 95% lower bound of the post-recalibration selective error exceeds α=0.20 for 1 failure at the alarm (Qwen2.5-VL-7B, NExT-QA to Video-MME, 21.0%). With the oracle it exceeds α for 3: Qwen2.5-VL-7B (21.2%), 3B (22.1%) and 32B-AWQ (24.0%), all NExT-QA to Video-MME. The paper states each verdict as a point estimate, does not infer population risk from it, and by point estimate recalibration holds α on none of the 5 (Section 5, RQ5). Table 16 and Appendix O give the counts for every family, and Table 12 gives the question-clustered upper bounds, under which the holds that do occur are thin.

Scope of the negative result. The claim concerns the frozen gate that certifies the false-accept rate and the detect-then-recalibrate loop of this paper, on the protocols tested. The abstract names the gate. In those runs, renewing the false-accept certificate, even with zero delay, does not restore selective error at the target in the cells where it held before the break, by point estimate. The identity states when a gate misses α.

Reviewer_4D7JOpenReview ↗

Certified Selective Gates for Video-Language Models Exceed the Selective-Error Target Before Genre Drift Begins

Thanks for running the fixed-sequence comparison and reporting the clustered bounds. The scan gives better coverage at 50 labels, but without a proven familywise guarantee at the same level, it’s not yet a like-for-like comparison. The bounds also make clear that only one of the five alarm-time failures is supported beyond the point estimates. I find the result useful for the particular gate tested, and I’m keeping my original rating.

Reviewer_DLfXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits detect-then-recalibrate gates for video QA under genre shifts. Its central result is that a gate certified for false acceptance often exceeds the selective-error target before a shift, while the tested Clopper–Pearson alternative often finds no certifiable threshold and therefore abstains entirely under the evaluated calibration budgets.

Critical strengths
  1. The zero-delay oracle and fresh@alarm controls provide useful isolation. They help separate detection delay from the effect of recalibration itself. This makes the negative result more informative about failure causes and is a main reason to consider acceptance.
  2. The shared candidate pool and label-budget sweep offer a practical comparison. Reporting both error and coverage across models exposes the costs of the tested rules. This gives the paper value as an empirical audit.
Critical weaknesses
  1. Theorem 1 is not established under the stated assumptions. It assumes only within-genre exchangeability. The cited binomial argument in COIN explicitly uses iid samples. The risk's conditioning on the calibration set needs to be specified, with a valid argument for the actual quantile and tie handling.
Questions for authors
  1. Is Theorem 1's risk conditional on the calibration set? Under which sampling assumptions do the actual empirical quantiles and tie handling support the binomial test and threshold-selection guarantee? A valid argument matching the implementation would change my assessment of the static guarantee.
3 comments
AuthorsOpenReview ↗

Response to Reviewer DLfX

Thank you for the review.

Fixed in the revision: Theorem 1 now states the assumptions the proof uses for the candidate grid, with the full proof in Appendix B.

Is the risk conditional on the calibration set? The probability is over the calibration draw. The quantity SelErr_g(t) = P(wrong | score ≤ t, genre g) is a fixed function of the threshold. For one realized calibration set, the bound does not guarantee that SelErr_g of the selected threshold is at most α; it holds in at least a 1−δ/(1−α) share of draws. The revised text states this directly below the theorem.

Which sampling assumptions support the guarantee? The revised theorem assumes i.i.d. calibration and test points within genre, with continuous scores. Under these assumptions, with probability at least 1−δ/(1−α) over the calibration draw, no grid candidate with SelErr_g(t) > α is certified. At δ=0.02 and α=0.20 the failure probability is at most 0.025. The introduction now says i.i.d. as well.

The argument for the actual quantile grid. Fix a grid level of rank k, and let T be the k-th smallest calibration score. Given T=t, the k−1 calibration points below t are i.i.d. from the law given score < t. Their wrong count W′ is Binomial(k−1, SelErr(t)), because scores are continuous. The test counts W ≥ W′ wrong answers among the k accepted points, since the point at t also counts. Let δ′=δ/G and let F_k be the Binomial(k, α) CDF. The Clopper–Pearson upper bound at confidence 1−δ′ is at most α exactly when F_k(W) ≤ δ′, that is, when W ≤ w*, with w* the largest w that satisfies F_k(w) ≤ δ′. If SelErr(t) > α, the candidate is certified with probability at most P(W′ ≤ w*) ≤ F_{k−1}(w*). Also F_k(w) = F_{k−1}(w) − α·P(Binomial(k−1, α) = w) ≥ (1−α)·F_{k−1}(w). So this probability is at most δ′/(1−α), and a union bound over the G candidates gives δ/(1−α). The grid is a quantile of the same calibration scores, and the argument conditions on exactly that quantile.

Tie handling. The theorem's continuity assumption rules out ties, and ties are rare in the data. On the stored records, no grid level has a tie excess in the 875 stream calibration folds (0 of 17,500 levels). In the fresh draws, a threshold meets a tie at 2 of 17,500 levels at B=50 and at 0 of 12,500 levels at B=150.

Threshold conventions. The selective branch accepts score ≤ threshold, and the conformal branch accepts score < threshold. The caption of Figure 2 in the revision states both conventions and the i.i.d. assumption.

I'd like to thank the authors for revised theorem and proof, which address my original concern. I also share Reviewer 4D7J’s concern about the direct selective-error comparison. The fixed-sequence scan’s familywise guarantee hasn't been established and the new experiment doesn't yet provide a comparison under matched confidence guarantee. It leaves me uncertain about the broader significance of reported vacuity. I will maintain my current rating for now, follow the other reviewers’ discussion, and consider carefully whether to revise my score.

AuthorsOpenReview ↗

The scan's level, and a floor at the same δ

Thank you for reading the revision and the other threads.

The scan's level. We agree. The familywise level of the fixed-sequence scan is not proved, and Table 14 in Appendix N states this. The scan sets its candidates and its order from the calibration scores of the fold it tests. The textbook fixed-sequence result assumes an order fixed in advance, so it does not apply as stated. The comparison is therefore at the same nominal δ, not under a matched guarantee.

A floor for any procedure at the same δ. A procedure with a proved familywise level δ rejects each single hypothesis at level at most δ. For one exact binomial test at level 0.02, certifying α = 0.20 needs at least 18 accepted calibration items with no error, 27 with one error, or 35 with two. The 20-candidate Bonferroni grid needs 31 with no error. At B = 50, the floor of 18 is 36% of the fresh fold. So at that budget, a certificate at this δ and α must accept over a third of a small fold with no error, or over half of it with one error, whatever the multiplicity procedure. How often a gate can do this depends on how well its score separates correct from incorrect answers.

What the central result rests on. The central finding does not depend on the CP vacuity. On the 5 alarmed cells where α held before the break, renewing the false-accept certificate restores α on none, by point estimate, and zero-delay oracle recalibration does not remove the failures (Section 4, RQ5). The CP rule is the alternative that targets selective error directly. Its vacuity is the cost of that alternative, and the abstract states it under the Bonferroni grid.

Reviewer_a8EDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on allowing a VLM to selectively answer questions to keep its error rate below a specified tolerance level. When deploying a VLM from an old data distribution to a new one, the previous gate mechanism may no longer be suitable for the new distribution. Even if the error rate criterion was previously satisfied, there is no guarantee that the same gate mechanism will work well on the new distribution. The authors conduct empirical experiments and report a negative finding: in most cases, the selective error criterion is not satisfied even before distribution drift occurs, so there is no previously satisfied target to restore by adapting the gate mechanism to the new distribution.

Critical strengths
  1. The motivation is reasonable. The paper focuses on the reliability of selective prediction under distribution drift, which is an important problem for deploying VLMs in real-world scenarios.
  2. The empirical analysis is comprehensive. The authors conduct experiments across multiple VLM backbones and video-QA benchmarks. They also investigate different factors, including drift detection delay, recalibration budget, and different certification rules, to better understand the negative findings.
Critical weaknesses
  1. There are many long and confusing sentences, even in the abstract. Lines 16–19 try to convey the key finding of this paper, but in a single sentence, the authors discuss four different rates and an undefined symbol (\alpha). As for the contribution paragraph in the introduction, it contains too many numbers and parameter settings, making it read more like an experimental log than a summary of the paper's contributions.
  2. The key objective of this paper is to control the selective error (SelErr) under distribution drift. However, the authors adopt a gate that certifies the false-accept rate (FAR) as their main pipeline. Since FAR and SelErr are two different metrics, certifying FAR does not necessarily guarantee SelErr. Given that the authors are aware of this distinction, it is unclear why they choose to adopt an FAR-certified gate as the main pipeline and then spend much of the paper demonstrating that it cannot guarantee SelErr. This raises concerns about the motivation and justification of the experimental design.
  3. The main contribution is a negative empirical finding that the selective-error target is often violated even before distribution drift occurs. While this finding is appreciated, its explanation is largely based on a known mathematical relationship between FAR and SelErr. The paper mainly combines existing techniques to demonstrate this mismatch, without proposing a practical solution to address the identified challenges.
Questions for authors

see weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer a8ED

Thank you for the review.

Readability. The revision splits the long abstract sentences. The abstract defines the target level α before the main finding. The contribution paragraph now states the findings and drops the budget list, the seed count and the registered-cell counts.

Why the main pipeline uses a false-accept gate. The paper studies a split-conformal gate that certifies the false-accept rate (FAR), and the detect-then-recalibrate loop that renews that certificate. It asks what this loop delivers for the quantity a user cares about, the selective error. The identity selective error = FAR × base error rate / coverage gives exactly the condition under which the error exceeds α. At the certified FAR it predicts a miss when coverage is below the base error rate. The frozen gate is over α=0.20 before the break on 30 of 35 stream cells. The paper also evaluates a rule that certifies selective error directly (Theorem 1, adapted from COIN) on the same candidate pool (Section 5, Figure 6, Table 2).

Known identity, and what the paper delivers. The paper states the identity as the explanation and does not present it as new. The contribution is the measurement the identity organizes. It covers 5 backbones, 4 benchmarks and 35 stream cells. A zero-delay oracle and a fresh-at-alarm arm replay the same streams with and without the detection delay. It adds a budget sweep run to the dated plan in PREREG.md (Appendix J gives the commit record).

In the static protocol, over the three backbones drawn in Figure 3, arms that certify selective error or FDR hold α in all but one of their 24 answering arm-cells, at a coverage price. On the 10 common cells at α=0.20, the conformal rule's alarm-time holds go from 5 at B=50 to 6 at B=150 (Figure 6, Table 2).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Title change in the revision

Title change. The revised PDF is titled "Certified Selective Gates for Video-Language Models Exceed the Selective-Error Target Before Genre Drift Begins". The submitted title was "Certified Selective Gates for Video-Language Models Exceed Their Error Target Before Genre Drift Begins". The change names the target that the gates exceed: the selective-error target of the certificate. The claims and experiments are those described in the paper and in our replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.