- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies procedural adversarial training by combining procedural scene generation with CMA-ES-based sampling. While the experimental results demonstrate promising robustness improvements, I have several concerns regarding the novelty and validation of the proposed approach.
Critical strengths
The paper explores adversarial robustness across different vision tasks by optimizing the parameters of procedural image generation, which offers a more practical and dynamic approach to robustness evaluation than conventional adversarial perturbations.
Critical weaknesses
-
The main algorithm appears to be an application of CMA-ES to adversarial sampling, with limited algorithmic innovation beyond heuristic sampling and restart strategies. There is no evidence that CMA-ES is the optimal optimiser for these tasks.
-
Defence against functional perturbations has been an active research field in recent years. The related work section also insufficiently covers recent advances in adversarial scene generation and robustness evaluation.
-
On the experimental side, I would particularly like to see a query-matched random search baseline and end-to-end runtime comparisons to substantiate the efficiency claims. Meanwhile, it would be better to compare CMA-ES with other optimisation methods, such as DIRECT and Bayesian optimisation.
-
While it is generally impossible to find the worst-case perturbation, adversarial robustness studies often attempt to derive some form of bounds, either theoretical or probabilistic, to estimate the worst-case scenario. Although the paper discusses the asymptotic consistency of worst-case estimation, it remains unclear how closely the reported finite-budget results approximate the true worst case. Further discussion of estimation uncertainty, convergence behaviour, or possible bounds would strengthen the robustness claims.
Questions for authors
-
If I understand correctly, the evaluation worst-case error is defined as the maximum error found across DIRECT-L, CMA-ES, and a uniformly sampled test set. This makes it difficult to assess the effectiveness of different optimisation strategies and understand which method primarily contributes to the reported worst-case performance. I suggest that the authors should at least report the results obtained by DIRECT-L, CMA-ES, and uniform random sampling separately in the appendix.
-
Although the paper describes some constraints on scene generation, it is unclear how the validity of the generated adversarial images is verified. Are there any additional checks to ensure that the generated scenes are physically plausible and that their labels remain valid?