- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
the paper introduces DART, a defense-aware open-world red-teaming framework centered on a Critical Cue Bank (CCB). DART first profiles the target defense and structures the resulting attack-relevant characteristics in the CCB, which guides attack selection, open-world retrieval, and target-specific adaptation.
Critical strengths
- topic is very relevant and the paper is suitable for this conference
- the goal of the paper is very clear, and the writing is clear and effective
- the result of the paper is very strong and convincing
Critical weaknesses
- currently only evaluated in one source of benchmark, not sure about broad applicability
- no error bars shown in the results
Questions for authors
N/A
Response to Reviewer WWgr: Generalization and Evaluation Stability (CW1–2)
Reviewer Concerns Addressed — Critical Weaknesses 1–2
CW1 — Generalization: The reviewer notes that DART was evaluated on a single benchmark and questions whether its effectiveness extends to other defense implementations and datasets.
CW2 — Evaluation Stability: The reviewer highlights the absence of error bars and run-to-run variability estimates, raising concerns about the reliability of the reported results.
Our Response
We thank Reviewer WWgr for the positive assessment and constructive suggestions. We conducted additional experiments on (1) external generalization and cross-dataset adaptation and (2) run-to-run variability and evaluation-set stability. Detailed results are provided in Shared Responses (3/4)–(4/4), supplemented by the cross-dataset experiments below.
1. Generalization Beyond AutoAdvExBench (CW1)
In Shared Response (3/4), we evaluate five external defenses across MNIST, CIFAR-10, GTSRB, and ImageNet. DART completes 5/5 evaluations (zero Failures) and achieves 3/5 Successes (60%):
For LeapNet-1, DART identifies randomized execution and adapts an EOT-based attack, reducing RA from 55.0% to 28.5% (−26.5 pp) and changing Not Broken to Success.
DART also generates executable defense-adapted attacks for GradNorm/Swin-B and RMAggNet, obtaining valid final RAs of 56.5% and 73.0%, respectively. Both remain Not Broken rather than failing during execution.
Cross-Dataset Evaluation with Retrained Defenses
To further examine generalization, we retrained TRADES, kWTA-Activation, and Mixup-Inference on MNIST while preserving their defense mechanisms. We evaluated 200 images per defense using MNIST ε = 0.3 and the original CIFAR-10 attack constraints.
DART uses gradient-based PGD for TRADES, surrogate gradients for discontinuous kWTA activations, and EOT for randomized Mixup-Inference.
All three defenses retain their Success/Not Broken outcomes after dataset change and retraining.
The external evaluations demonstrate applicability beyond AutoAdvExBench, while the retraining experiments provide complementary evidence of mechanism-aware adaptation across datasets. Absolute RA values are not directly comparable across different datasets and perturbation budgets.
2. Error Bars and Evaluation Stability (CW2)
In Shared Response (4/4), we report independent full-agent repetitions and evaluations using larger image sets.
Independent Agent Runs
Using Claude Opus, we independently repeated the complete DART workflow three times on each of five defenses, restarting profiling, attack generation, and final RA scoring.
RA is in percent; sample SD is in percentage points.
All 15/15 independent executions produced valid RA, and all five defenses retained consistent Success/Not Broken outcomes across runs. Even kWTA-Activation, with the largest RA variation (6.0 pp), remained Success in all three runs.
Larger Evaluation Set
We additionally evaluated final attacks on 1,000 instead of 200 images across all 51 defenses.
Both settings yielded 27 Success and 24 Not Broken outcomes, with zero classification changes. The mean absolute RA difference was 1.00 pp, with a median of 0.60 pp and maximum of 6.10 pp.
For the near-threshold defense
obfuscated-gradients/sap, RA changed from 44.0% (200 images) to 44.8% (1,000 images), while remaining Not Broken.These experiments examine distinct sources of stability: independent agent-run variability on five defenses and evaluation-set-size sensitivity across all 51 defenses. The 200-image subset is included in the 1,000-image set.
3. Manuscript Revision
We will incorporate the external and cross-dataset evaluations, per-run RA and sample standard deviations, expanded evaluation results, and near-threshold analysis into the revised manuscript.
Together, these findings address CW1 and CW2 by providing evidence of applicability across heterogeneous defenses and consistent evaluation outcomes under independent agent executions and larger evaluation sets.
We thank Reviewer WWgr again for the constructive suggestions, which helped strengthen DART's experimental validation.