- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies adversarial robustness when only a naturally pretrained model is available, without access to the original training data or a robust teacher. DFAT first learns a generator and a natural student through data-free distillation, then adversarially trains the student using the natural teacher’s predictions on synthetic data. Sample- and class-level generator reweighting target class-wise fairness, while negative-KL regularization against a frozen student copy is intended to promote probability sparsity and synthetic-to-real generalization. An optional DENT-based test-time adaptation stage further improves performance. The paper provides conditional theoretical analyses and evaluates the framework on CIFAR-10, CIFAR-100, and Tiny-ImageNet using several architectures and attacks.
Critical strengths
S1. The reported results demonstrate practically meaningful robustness under restrictive training access. On CIFAR-10 with ResNet-18, DFAT achieves 82.41% clean accuracy and 37.79% AutoAttack accuracy without test-time adaptation. Table 5 also reports an increase from 0.31% AutoAttack accuracy after natural pretraining to 29.89% with self-adversarial training and 37.79% with the complete method. These results suggest that the framework has practical potential beyond preserving natural accuracy, which is the strongest positive consideration in my assessment.
S2. The robust-risk decomposition provides a useful conditional analysis of the setting. Theorem 2 distinguishes teacher error, low-margin teacher predictions, robust anchoring on synthetic samples, and synthetic-to-real distribution discrepancy. This helps clarify the conditions under which a non-robust teacher could supervise a robust student, although the result does not establish that the proposed regularizers improve these quantities.
Critical weaknesses
W1. The contribution over the closest same-setting prior work is not established. DataFreeShield—Lee et al., DataFreeShield: Defending Adversarial Attacks without Training Data, ICML 2024—already studies learning robustness from a non-robust pretrained model without real training data. Its soft-guided objective includes clean-student/clean-teacher and adversarial-student/clean-teacher matching, which directly overlap with the first two terms of Eq. (11). Thus, the basic setting and central supervision strategy are established prior work.
Although DataFreeShield is cited, it is absent from the experimental comparisons. The paper instead primarily compares against methods with robust teachers, and its strongest reported comparisons additionally apply DENT only to DFAT. Meanwhile, nasty regularization is explicitly adapted from NT/NAT. Consequently, the current experiments do not establish the incremental value of the proposed combination over the closest alternative under matched training and inference conditions. A direct DataFreeShield comparison is central to assessing both novelty and significance.
W2. The proposed fairness mechanism has a mathematical interpretation problem, and its supporting ablation is internally inconsistent. Equation (8) minimizes , assigning smaller weights to harder classes with the stated intention of generating more samples from those classes. 11843_Data_free_Adversarial_Tr However, with fixed positive weights, minimizing this term over the probability simplex gives
$p_c^\star=\exp\left(-1-\frac{\nu}{w_c}\right), $
where normalization requires for . Within this optimum, smaller weights therefore imply smaller class probabilities—the opposite of the claimed effect. This calculation isolates the regularizer rather than the complete generator objective, but it directly challenges the explanation for the proposed weighting rule.
The empirical evidence does not resolve this concern: Figures 4 and 7 display the same four sets of plotted values, yet Figure 4 labels them as baseline-versus-DFAT comparisons, whereas Figure 7 labels them as DERD accuracy and confidence measurements. These incompatible descriptions prevent Figure 4 from supporting the claimed fairness improvement. Establishing this contribution requires clarification of the implemented objective and correctly identified per-class results, potentially including revised experiments.
Questions for authors
Q1. Essential comparison. What is the substantive methodological advance over DataFreeShield beyond adding generator reweighting and an NT/NAT-derived regularizer? Please provide a comparison using the same natural teacher, student architecture, attack budget, and comparable synthesis/training resources, with test-time adaptation either absent from both methods or applied to both.
Q2. Implementation and evidence clarification. Is Eq. (8) implemented exactly as written, and are the difficulty-dependent weights detached during generator optimization? How does the implemented update increase the representation of difficult classes despite the isolated regularizer’s opposite preference? Please report the corresponding generated class frequencies and clarify which measurements Figures 4 and 7 actually represent.
Q3. Regularization mechanism. Please specify the discrepancy function, KL direction, and temperature used in Eqs. (11)–(12). If the frozen student copy satisfies and the divergences use matching conventions, the corresponding positive and negative KL terms reduce to a change in the distillation coefficients. What evidence distinguishes the proposed sparsity mechanism from this effect? Comparisons against coefficient-matched SAT and a simple entropy regularizer would materially affect my assessment of the claimed contribution.
Q4. Test-time defense evaluation. Appendix E.3 defines the adaptive objective as . How is this objective connected to the classification loss of the model after its test-time updates? Please clarify whether the attack differentiates through adaptation or uses a justified approximation, how it handles random augmentations and model resets, and whether attack convergence was checked. Adding the adaptation loss to the attack objective alone does not establish that the complete adaptive defense has been adequately attacked. This clarification is important because DENT contributes substantially to the reported gains.