- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors of this paper propose AURORA, a framework for improving adversarials training on datasets with class imbalance. AURORA consists of two components: uncertainty-aware anchoring (URA) and first-passage trajectory refinement (FTR) that address two limitations of existing adversarial training methods. First, the authors recognise that AT techniques that use failure rates don't account for the fact that a failure rate calculated on a low frequency group is statistically less reliable, so URA uses the Wilson-score upper confidence bound to account for how many observations support the estimated failure rate. Second, FTR uses the earliest attack iteration at which an initially correct prediction turns incorrect, capturing differences in adversarial vulnerability that aren't reflected by final failure rates alone. These two signals are then combined into class-wise weights applied to the underlying AT objective, without introducing an additional loss function. The authors evaluate AURORA on three long-tailed image datasets, across multiple architectures and adversarial training methods, comparing against existing long-tailed adversarial techniques. The authors show AURORA improves tail-class robustness, while generally maintaining competitive overall performance.
Critical strengths
-
A relatively simple methodology that appears well motivated by limitations of existing techniques. The motivation is interesting, and AURORA seems straightforward to integrate into existing adversarial training procedures.
-
Fairly extensive evaluations on three datasets, three adversarial training methods, and three different model architectures show AURORA performs well on tail-class metrics, and remains competitive for the overall performance metrics. One small note, the presentation of these results could be improved. Table 2's caption should be consistent with Tables 1 and 3 in explaining the highlighting, and using more distinguishable colours for the two types of highlighting would make the results easier to interpret. The current formatting doesn't quite do the evaluation justice.
Critical weaknesses
-
There is some uncertainty in my head for whether the iteration at which the adversarial attack achieves misclassification is a true measure of class vulnerability. For example, a particular attack might simply be more effective at misclassifying for certain classes due to differences in the loss landscape or gradient directions, rather than those classes actually being more vulnerable. Although the ablation shows that FTR improves performance, I would have liked to see some study into whether this first-passage behaviour is consistnt across different attack configurations, such as step sizes, number of iterations, or random initialisations. This would help justify the use of attack timing as a measure of vulnerability.
-
I don't think that the authors justify why the Wilson-score upper bound was chosen over other approaches. Although Table 4 shows that URA improves over using raw failure rates, it would have been useful to compare it against alternatives- for example the Clopper-Pearson confidence bound. This would help to see whether the improvements are specific to Wilson-score.
Questions for authors
-
Please see W1. It would be useful to see whether this first-passage behaviour is consistnt across different attack configurations, such as step sizes, number of iterations, or random initialisations.
-
Please see W2. Can the authors comment on why the Wilson score was the chosen upper bound for URA? Would other appraoches such as different confidence bounds produce similar results?
-
The authors say that their training setup includes prior-aware logit correction and a class-dependent perturbation schedule. Were these same settings used for the competing methods in Tables 1-3? If not, could some of the improvements be due to differences in training setup, not just from AURORA?
Clear answers to the questions above may allow me to reconsider my recommendation more favourably.