- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines whether detecting distribution drift and recalibrating a selective gate restores a target error rate for frozen video-language models. Experiments cover five backbones and four video-QA benchmarks, using metadata-defined shifts, a coverage-based detector, and fresh-label recalibration. The central distinction is between the false-accept rate—the proportion of incorrect answers that the gate accepts—and selective error—the proportion of accepted answers that are incorrect. The monitored conformal gate controls the former, while the deployment target concerns the latter. Many evaluated streams already exceed the selective-error target before drift, and recalibration can continue to miss it even with an oracle change point. A separate Clopper–Pearson rule directly controls selective error but frequently abstains completely under the tested budgets and certification procedure.
Critical strengths
-
The paper makes an important distinction between the quantity a certificate controls and the quantity users care about. Reporting false-accept rate, coverage, base error, and selective error together makes the mismatch concrete. The pre-break results are particularly useful because they show that some apparent deployment failures precede any distribution shift.
-
The oracle and fresh-at-alarm experiments help separate recalibration from detection delay. I also appreciate the reporting of abstention, repeated questions, thin accepted samples, and unsuccessful pre-registered predictions. These details make the negative results easier to assess than an aggregate post-drift error comparison alone would.
Critical weaknesses
-
The main negative result has a narrower interpretation than the title and deployment framing suggest. The principal recalibration procedure renews a false-accept certificate rather than a selective-error certificate, so missing the selective-error target does not demonstrate a failure of its stated guarantee. The paper acknowledges this, but the experiment consequently establishes a limitation of this particular proxy-control procedure more directly than a limitation of detect-then-recalibrate generally. Selective error equals the false-accept rate multiplied by the base error rate and divided by coverage. This identity explains the observed outcomes, but coverage below the base error rate does not, by itself, imply that selective error exceeds the target: the conclusion also depends on the realized false-accept rate. An upper bound on that rate alone is insufficient to establish a violation. The empirical finding is useful; its scope needs to remain explicit.
-
The direct selective-risk baseline does not yet establish that useful certification is generally infeasible at these label budgets. It combines a 20-threshold Bonferroni correction, , and permanent abstention after a vacuous recalibration. Under these settings, even a candidate with zero calibration errors needs at least 31 accepted calibration examples to certify a 20% error target. This is already restrictive at , before accounting for the score's ability to separate correct and incorrect answers. The paper acknowledges that fixed-sequence testing could be less conservative, but does not evaluate it. Without such a comparison at the same confidence level, the reported vacuity and coverage costs remain specific to the tested procedure, rather than evidence of an intrinsic budget limitation.
Questions for authors
-
How do the conclusions change when recalibration directly targets selective error using a less conservative testing procedure, such as a prespecified fixed-sequence scan, at the same confidence level? Please report certified coverage alongside realized error.
-
Can you distinguish the empirical coverage-versus-base-error observation from a guarantee? In particular, how often is the false-accept rate sufficiently below its nominal level that selective error remains acceptable despite coverage being below the base error rate?
-
Which reported recalibration failures have question-clustered lower confidence bounds above the target error rate? The upper bounds help assess evidence for successful control, but point estimates just above the target do not establish a violation. This matters for statements such as "restores it on none," especially given the acknowledged one-error margin in one cell.
2 comments
Certified Selective Gates for Video-Language Models Exceed the Selective-Error Target Before Genre Drift Begins
Thanks for running the fixed-sequence comparison and reporting the clustered bounds. The scan gives better coverage at 50 labels, but without a proven familywise guarantee at the same level, it’s not yet a like-for-like comparison. The bounds also make clear that only one of the five alarm-time failures is supported beyond the point estimates. I find the result useful for the particular gate tested, and I’m keeping my original rating.
Response to Reviewer 4D7J
Thank you for the review.
Direct selective-error certification at these budgets. The abstract states the vacuity result under a Bonferroni grid, and Section 5 gives the 20-candidate grid at δ=0.02. The budget sweep shows how this rule responds to labels. The share of vacuous realizations falls from 100% at B=10 to 63.7% at B=150. The number of cells holding α rises from 0 to 3 of the 10 common cells (Table 2). Over the three backbones drawn in Figure 3, arms that certify selective error or FDR hold α in all but one of their 24 answering arm-cells, at a coverage price.
Fixed-sequence scan at the same δ. A post-review run replaces the Bonferroni grid by a fixed-sequence scan at the nominal δ=0.02 on the same fresh draws, monitor and 10 common cells (α=0.20, recalibration at the alarm). At B=50 the scan is vacuous in 82.2% of realizations (Bonferroni: 98.7%). Its mean post-recalibration coverage is 4.5% (0.5%) at pooled selective error 6.8% (8.8%), and it holds α in 3 of 10 cells (1). At B=150 it is vacuous in 63.1% (63.7%), with coverage 15.2% (13.4%), pooled selective error 10.9% (10.5%) and 4 of 10 cells holding (3). The familywise level of the scan at this δ is not proved, so the comparison is at the same nominal δ and is exploratory. Appendix N gives every budget and timing at α=0.20 and α=0.10.
Coverage below the base error rate. The identity is exact, and the paper uses it as a predictor, not as a guarantee. On the registered cells the sign rule predicts the verdict in 58 of 69 entries (84%). Of the 50 entries with coverage below the base error rate, 3 hold and 47 fail. Of the 19 entries at or above it, 11 hold; one of the 19 has no answered item. Appendix J gives all five backbones. The realized FAR lands within its recorded mean bound in 62 of 68 cell-arms with a defined FAR, and in 50 of the 54 whose selective error exceeds α (Figure 5c).
Question-clustered bounds. Among the 5 alarmed cells screened from 25 seeds where α held before the break, the question-clustered one-sided 95% lower bound of the post-recalibration selective error exceeds α=0.20 for 1 failure at the alarm (Qwen2.5-VL-7B, NExT-QA to Video-MME, 21.0%). With the oracle it exceeds α for 3: Qwen2.5-VL-7B (21.2%), 3B (22.1%) and 32B-AWQ (24.0%), all NExT-QA to Video-MME. The paper states each verdict as a point estimate, does not infer population risk from it, and by point estimate recalibration holds α on none of the 5 (Section 5, RQ5). Table 16 and Appendix O give the counts for every family, and Table 12 gives the question-clustered upper bounds, under which the holds that do occur are thin.
Scope of the negative result. The claim concerns the frozen gate that certifies the false-accept rate and the detect-then-recalibrate loop of this paper, on the protocols tested. The abstract names the gate. In those runs, renewing the false-accept certificate, even with zero delay, does not restore selective error at the target in the cells where it held before the break, by point estimate. The identity states when a gate misses α.