- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper analyzes what we can learn from paired evaluations, where the prediction for a clean image is linked to the prediction for its perturbed version. It answers two questions. First, for some robustness questions the full pairing is not needed. Knowing per class how many images stay correct already gives the same bounds as the full breakdown of all changes. Second, on existing screen and print recapture datasets, the responses of an image to synthetic corruptions can help to select the images that should be tested physically. This finds clearly more real failures than ranking by clean confidence.
Critical strengths
The screening result is useful in practice. Using the responses of an image to synthetic corruptions helps to select the images that should be tested physically, and the comparison to test-time augmentation with the same number of inferences is fair. This is the part of the paper that could carry a revised version.
Further strengths:
- The theory is done carefully and the scope of each result is clearly stated.
- The experiments are large, with 44 models, and the bounds are checked with a second implementation.
Critical weaknesses
- The contribution is too weak for ICLR. The paper is mainly an analysis of an evaluation setting, without a new method or a result that changes how we evaluate robustness. The first part assumes that all predictions of both sides are available, but whoever has them also ran the evaluation and therefore has the pairing anyway. For standard robustness analysis the pairing does not matter, because the accuracy drop does not depend on it. It only matters for the specific transition measures the paper defines, and the paper does not show that anyone needs them. The theory itself is small. The bounds are classical results, and the main theorem follows from a simple symmetry argument.
- Only ImageNet classification is used. The synthetic part is a reduced version of ImageNet-C, and the physical data comes from ImageNet-ES and ES-Diverse, which are based on the same images. The first part could easily be run on standard benchmarks like the full ImageNet-C, CIFAR-10-C or ImageNet-3DCC, or on other tasks like detection or segmentation. For the second part, physical recaptures are rare, so the choice is understandable. But the screening evaluation is small, with only a few hundred images and two or three images per class, and under strong perturbations there is no gain. On this data it cannot carry the paper, and it stays open how general the results are.
Further weaknesses (less important for my score):
- The motivation and the main arguments are not presented well. The story is quite sophisticated, but it takes a long time to understand which practical problem the authors want to solve. I had to dig through the paper and the appendix to find it. The text is dense and full of notation, many important results are only in the very long appendix, and Section 6 is only a pointer to the appendix. This could be fixed in a revision.
- The two parts of the paper are barely connected. The screening experiment does not use the identification results.
- Some findings are expected. If the corruption score of one image is given to another image, an image-level score of course gets worse. That synthetic corruptions predict failures under recapture better than geometric augmentations is also not surprising.
- Reproducibility is limited, because the seeds for the corruptions were not stored.
Questions for authors
Clarification and additional analysis (the answers could change my assessment):
- Who should use the results of the first part? For example, should benchmark papers report per-class counts of images that stay correct, in addition to accuracy? A convincing use case would address my main concern about the practical value.
- Is there a concrete case where the transition measures lead to a different decision than the accuracy drop? If yes, this would show that the measures are needed in practice.
Suggestions (not part of my assessment):
- Run the first part on standard paired benchmarks like the full ImageNet-C or CIFAR-10-C, and check whether the screening result also holds for other tasks than ImageNet classification.
- Compare the screening score to stronger failure prediction methods, for example a learned failure predictor or MC dropout.
- Name the practical scenario early in the introduction. At the moment the reader learns it late.
- Put the key results for each claim in the main text and not only in the appendix. Section 6 should either be filled or removed.
- Use less notation in the main text. The five states, three information levels and five targets are hard to follow.