- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Paper addresses an interesting problem of generalisation of Certified Robust Accuracy (CRA) under distribution shift. The hypothesis under investigation is role of CRA for broader model behaviour like accuracy and if CRA and accuracy relationship is transferable under distribution shift. It shows that the CRA shows agreement from in-distribution (ID) to out-of-distribution (OOD) and interestingly, ID CRA correlates with OOD accuracy on various architectures and datasets.
Critical strengths
Paper addresses an interesting problem.
Critical weaknesses
Paper is generally well written but bit hard to follow due to long sentences. E.g., first contribution sentence is almost a paragraph. Whereas, conclusions statements are extremely summarised, Acronyms are introduced but never explained — authors assume that reader has specialised context already. An example is VNNLIB property set, beta-CROWN, etc). Limitations section reads disconnected from the paper, e.g., it talks about ReLU networks, but this is only introduced at Page 9. Conclusion has no link to the contribution as well the title of the paper. Section 5.3 conclusion, you say IC CRA is informative …accuracy, but it should be …. — this is confusing, either you conclude but it should NOT be, or write, so it should be.
Questions for authors
May main concern of the paper is (in my opinion), its flimsy nature. Asking generalisation of certified robustness is definitely a valid and important question, but it boils down to how you define the drift and OOD. You can change the definition of labels (cats are no more cats but dogs, or definition of cats can change without involving labels). You intuition is that in case of P(x,y) drift, CRA should not generalise, but in case of P(x), it should. My suggestion for improvement is that authors should do an explicit study of various forms of distribution shift, and be clear about conclusion they are reaching, rather then tangling them in mathematical subtleties.
Can you explain findings of Table 2 for me. Results suggest degradation (significant) of CRA for structural OOD and Corruption OOD. How does it back your main result? How does it answer your question of does certified robustness generalises?