acceptodds
Withdrawn from ICLR 2027

What Paired Evaluations Reveal under Visual Perturbations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

To understand the reliability of computer vision models in real-world applications, robustness evaluation must examine diverse visual perturbations, yet benchmarks cover only some real-world conditions and physical testing is costly. Paired evaluations link clean and perturbed predictions for the same image, retaining changes in correctness, confidence, and acceptance that aggregate accuracy does not describe. We investigate how this image correspondence supports two needs in robustness evaluation: interpreting paired evaluation results and prioritizing samples for physical testing. To interpret paired evaluation results, we fix both sets of prediction records and vary their correspondence within each class. We prove that classwise correct–correct counts give the same sharp bounds on lost acceptance and mean true-class probability decrease among retained-correct inputs as any feasible five-state refinement. Distinguishing persistent from changed wrong answers can further constrain accepted-error transitions, while shared correspondence can establish policy orderings left unresolved by separate cost intervals. To prioritize samples for physical testing, we retain each image's synthetic responses and rank clean-correct images by their mean true-class probability under corruption. Across 44 classifiers, testing the highest-risk 20% finds 67% and 45% of failures under mild screen and print recaptures, versus 58% and 36% for clean confidence and 60% and 37% for an equal-size natural-transformation average. With both probability averaging and an A3Rank scoring adaptation, the tested corruption set yields higher mean failure recall than the natural-transform set; differences between scores depend on the source and budget. Together, these findings show that the value of correspondence depends on the evaluation objective: classwise counts suffice for specified reliability bounds, while image-specific synthetic responses improve the allocation of physical tests within the evaluated pool.

Primary area:
datasets and benchmarks
Keywords:
robustness evaluation, paired evaluation, partial identification, failure detection, test prioritization, image classification, computer vision, foundation model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3DP4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes what we can learn from paired evaluations, where the prediction for a clean image is linked to the prediction for its perturbed version. It answers two questions. First, for some robustness questions the full pairing is not needed. Knowing per class how many images stay correct already gives the same bounds as the full breakdown of all changes. Second, on existing screen and print recapture datasets, the responses of an image to synthetic corruptions can help to select the images that should be tested physically. This finds clearly more real failures than ranking by clean confidence.

Critical strengths

The screening result is useful in practice. Using the responses of an image to synthetic corruptions helps to select the images that should be tested physically, and the comparison to test-time augmentation with the same number of inferences is fair. This is the part of the paper that could carry a revised version.

Further strengths:

  • The theory is done carefully and the scope of each result is clearly stated.
  • The experiments are large, with 44 models, and the bounds are checked with a second implementation.
Critical weaknesses
  1. The contribution is too weak for ICLR. The paper is mainly an analysis of an evaluation setting, without a new method or a result that changes how we evaluate robustness. The first part assumes that all predictions of both sides are available, but whoever has them also ran the evaluation and therefore has the pairing anyway. For standard robustness analysis the pairing does not matter, because the accuracy drop does not depend on it. It only matters for the specific transition measures the paper defines, and the paper does not show that anyone needs them. The theory itself is small. The bounds are classical results, and the main theorem follows from a simple symmetry argument.
  2. Only ImageNet classification is used. The synthetic part is a reduced version of ImageNet-C, and the physical data comes from ImageNet-ES and ES-Diverse, which are based on the same images. The first part could easily be run on standard benchmarks like the full ImageNet-C, CIFAR-10-C or ImageNet-3DCC, or on other tasks like detection or segmentation. For the second part, physical recaptures are rare, so the choice is understandable. But the screening evaluation is small, with only a few hundred images and two or three images per class, and under strong perturbations there is no gain. On this data it cannot carry the paper, and it stays open how general the results are.

Further weaknesses (less important for my score):

  • The motivation and the main arguments are not presented well. The story is quite sophisticated, but it takes a long time to understand which practical problem the authors want to solve. I had to dig through the paper and the appendix to find it. The text is dense and full of notation, many important results are only in the very long appendix, and Section 6 is only a pointer to the appendix. This could be fixed in a revision.
  • The two parts of the paper are barely connected. The screening experiment does not use the identification results.
  • Some findings are expected. If the corruption score of one image is given to another image, an image-level score of course gets worse. That synthetic corruptions predict failures under recapture better than geometric augmentations is also not surprising.
  • Reproducibility is limited, because the seeds for the corruptions were not stored.
Questions for authors

Clarification and additional analysis (the answers could change my assessment):

  • Who should use the results of the first part? For example, should benchmark papers report per-class counts of images that stay correct, in addition to accuracy? A convincing use case would address my main concern about the practical value.
  • Is there a concrete case where the transition measures lead to a different decision than the accuracy drop? If yes, this would show that the measures are needed in practice.

Suggestions (not part of my assessment):

  • Run the first part on standard paired benchmarks like the full ImageNet-C or CIFAR-10-C, and check whether the screening result also holds for other tasks than ImageNet classification.
  • Compare the screening score to stronger failure prediction methods, for example a learned failure predictor or MC dropout.
  • Name the practical scenario early in the introduction. At the moment the reader learns it late.
  • Put the key results for each claim in the main text and not only in the appendix. Section 6 should either be filled or removed.
  • Use less notation in the main text. The five states, three information levels and five targets are hard to follow.
Reviewer_JJkXOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates whether responses to synthetic visual perturbations can predict the vulnerability of image classifiers to real-world screen- and print-recapture failures. It evaluates 44 classifiers and proposes ranking clean-correct images by their mean true-class probability under synthetic corruptions to prioritize physical testing. The results show that screening the highest-risk 20% of images can identify a substantial proportion of failures under some conditions, particularly for screen recapture. The paper also develops theoretical bounds for assessing reliability when image-level correspondence is unavailable or partially retained. Overall, it demonstrates the potential of synthetic perturbation responses for physical robustness screening, while highlighting that effectiveness varies across models and capture conditions.

Critical strengths
  1. The paper investigates whether responses to synthetic visual perturbations can help identify images that are vulnerable to real-world screen- and print-recapture failure, which is an important problem.
  2. The study evaluates 44 classifiers across multiple synthetic corruption settings and extensive physical recapture conditions, providing a diverse empirical test bed.
Critical weaknesses
  1. In standard paired evaluation, clean and perturbed images come from the same original image, so the true correspondence is known; the paper instead studies the "what if correspondence were missing" scenario, yet does not establish that real evaluation reports, model audits, or deployment decisions typically face such information loss, nor that the proposed bounds would materially change decisions.

  2. The paper cites McNemar, negative-flip analysis (Yan et al. 2021), error consistency (Geirhos et al. 2020), Bench-C (Sui et al. 2026), VLM-RobustBench (Saxena et al. 2026), PRIMA (Wang et al. 2021), and A3Rank (Wei et al. 2024), but runs no experiment or theoretical comparison against any of them.

Questions for authors

Please see the weakness.

Reviewer_f2zZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper examines what paired evaluations of vision-language models (clean and perturbed prediction records) can establish, and whether image-specific synthetic responses help prioritize physical recapture tests. It has two parts. First, an identification analysis: fixing both sides' prediction records and varying within-class correspondence, it asks which reliability targets (lost acceptance, confidence decrease, accepted-error transitions) are determined at different information levels (marginal records, correctness overlap, five-state counts). It shows that classwise correctness overlap already yields sharp bounds for lost acceptance and the conditional confidence decrease, that five-state counts can further tighten accepted-error transitions, and that comparing policies under a common correspondence can certify orderings that separate sharp intervals miss. Second, a prospective screening study: it evaluates whether corruption-based averaging of true-class probabilities (U) improves failure coverage at a fixed recapture budget over clean confidence and an equal-size natural-transformation average, with identity ablations and several probe and scoring controls on ImageNet across 44 models and two physical recapture datasets (ES and Diverse).

Critical strengths
  • The identification analysis is carefully executed and appropriately scoped. The theorem that classwise correctness overlap suffices for sharp lost-acceptance and conditional-confidence bounds, while five-state counts add nothing for those targets but can tighten accepted-error transitions, is a clean and useful separation. It tells practitioners which additional counts are worth collecting for which target.

  • The screening study is large-scale and carefully instrumented: 44 models, two physical recapture datasets, identity ablations, class-mean baselines, development/evaluation split checks, and equal-inference probe comparisons. The identity ablation in particular supports the claim that the gains come from image-specific responses rather than class-level signal.

Critical weaknesses
  • The paper lacks a unifying thesis. The identification part concerns what fixed records establish about completed evaluations; the screening part concerns how to allocate further physical tests. The authors acknowledge this ("the value of correspondence depends on the judgment"), but the paper never articulates why these two analyses belong together or what the combined contribution is beyond two separate studies. As a result, the contribution feels diffuse, and neither part is developed to the depth of a standalone paper.

  • The identification results are mostly elementary. The core machinery is classwise Fréchet/intersection bounds plus a symmetry argument for correct records. This is correct and carefully verified, but it is not a new methodological contribution. The main new element—comparing policies under a shared correspondence—is a straightforward inequality (the maximum of a difference is at most the difference of maxima) applied to a specific setting. The paper does not establish that this yields practically important policy conclusions beyond the realized-selection results, which are small and mixed.

Questions for authors
  1. What is the single overarching thesis connecting the identification analysis to the screening experiments? If the two are independent, why present them in one paper rather than separately?

  2. The identification analysis conditions on within-class unrestricted reassignment, which is a counterfactual rather than a physical or sampling claim. Can you give a concrete evaluation decision that this analysis would change, beyond the descriptive statement that additional counts tighten certain bounds?

  3. The shared-correspondence comparison certifies additional policy orderings, but the realized-selection results are small and mixed (Table 13, Table 14). What practical policy decision does the certified ordering inform, and how often does it change a decision that a practitioner would actually make?

  4. In the screening experiments, the natural-transformation baseline is a fixed 18-transform average without label-based selection. How do the gains compare against a development-selected natural set or a stronger augmentation-based baseline such as A3Rank with full training? The paper reports one adapted A3Rank comparison but notes its original rejection component is not reproduced.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.