I recommend rejection because the statistical guarantees behind the certificate remain uncertain, and several statements about the results do not agree with the reported tables.
First, I found inconsistencies between the main text and the appendix. Section 8 and Appendix A state that all units become INCONCLUSIVE at ε = 0.01, but Table 2 shows that VAU and CLIP on ISIC-2019 remain FAIL. The paper also states that protocol tightening moves verdicts toward weaker claims, although Table 4 shows MMMU moving from INCONCLUSIVE to SURVIVE. This matters because the direction of these changes is used to support the claim that the tightened procedure produces more conservative decisions. Other numerical differences are harder to interpret. Section 6 reports CLIP's baseline AUC as both 0.523 and 0.525, and gives an ISIC lower confidence bound of 0.064, while the description of Table 1 and Appendix Table 2 report 0.072. These values may come from different calculations, but the paper does not explain the distinction. TAD and Video-MME also have upper bounds reported as 0.020, exactly the declared margin, so the unrounded values are needed to check their verdicts. I do not take these discrepancies as proof that the final decisions were computed incorrectly, but they make it difficult to verify the results of a method whose main contribution is statistical certification.
The inferential procedure raises a more direct concern. Equation (2) assigns FAIL when any member of the attacker family has an individual 95 percent lower confidence bound above the margin, while SURVIVE uses a simultaneous upper bound over the family. Although the independent certification fold separates attacker selection from evaluation, it does not by itself control the probability of a false FAIL when several attackers are tested and any one can trigger the verdict. I could not find a correction for this multiple-testing problem. The two reported FAIL effects are large and may survive a correction, but that has not been shown. For SURVIVE, the authors acknowledge that the bootstrap may be inconsistent when the maximum inflation lies on the boundary, which is precisely the setting of their SURVIVE results. They do not compute the alternative Bonferroni bound mentioned in Section 5. Figure 2 provides a useful check with an injected channel, but it tests one leakage mechanism on MMBench with twelve seeds per strength. It does not establish that the confidence bounds have the intended coverage across the attacker families and cluster structures used in Table 1. The Roundabout result is particularly concerning. Its 74 items contain only three independent clusters, and the paper reports a FAIL probability of about 0.33 when the true inflation is below the margin. The authors disclose this limitation, but the certificate does not specify when such data should be considered unsuitable for a reliable verdict.
The theoretical result also needs a more careful connection to the claimed repair limit. Appendix B derives the maximum ESS for reweightings that equalize the full conditional label distribution across nuisance groups within score strata. This is an interesting and useful identity, but complete distribution balancing is stronger than requiring zero inflation under a particular benchmark metric. For example, suppose two equally sized nuisance groups have positive-label rates of 0.7 and 0.9, while the frozen score is constant. A classifier that maximizes accuracy predicts the positive class in both groups, with or without the nuisance information. The nuisance therefore contains additional information about the label, but it does not improve accuracy. In this case no reweighting is needed to eliminate accuracy inflation, although the identity assigns a cost to achieving full distribution balance. The theorem remains valid for its stated balancing constraint, but it does not necessarily give the maximum ESS among all reweightings that eliminate inflation under the native metric. I would like the authors to distinguish these objectives before using the identity to support broader claims about when scoring-side repair is impossible.
My other concern is that the empirical results support a narrower conclusion than capability identification. The proposed inflation measures what a declared nuisance channel adds beyond the frozen model score, not whether the model already uses that channel. As the authors acknowledge, a model that has absorbed a shortcut can receive SURVIVE because there is no additional information left for the attacker. The certificate can therefore bound residual inflation without establishing that the model possesses the intended capability. The experiments do not resolve this distinction. Six of the seven main units use benchmarks whose nuisance structure was already known to the authors. The external FAIL comes from CLIP on ISIC-2019, where the baseline AUC is close to 0.52 and a nuisance-only predictor reaches 0.636. When CLIP is replaced with BiomedCLIP, whose baseline AUC is 0.686, the inflation decreases from 0.120 to 0.020 and the verdict becomes INCONCLUSIVE. This shows that the certificate depends on the frozen scorer, but does not establish that the stronger model has absorbed the nuisance channel. The explanations offered for the SURVIVE results, such as CircularEval blocking answer-position effects, are also not tested through controlled ablations. I appreciate that the paper acknowledges many of these limitations, but the conclusion still treats the absence of further nuisance inflation as a condition for licensing a capability claim.
I find the problem important and the Repair–ESS analysis interesting, but the current evidence does not establish the reliability or scope of the proposed certificate. If the authors can reconcile the reported results, show that the main verdicts hold under valid family-wise confidence bounds, and clarify the relationship between distribution balancing, metric inflation and capability identification, I would be willing to reconsider my assessment.