- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes Clean Feature Anchoring (CFA), a method for mitigating the clean accuracy–adversarial robustness trade-off in adversarial training. The central idea is to first train a naturally trained clean self-teacher and then freeze it as a reference. During robust training, the student is initialized from the teacher, retains its classifier, and learns by matching its features on adversarial inputs to the teacher's features on the corresponding clean inputs. The paper further introduces sensitivity-matched perturbation budgets, which assign smaller adversarial perturbation radii to examples with larger local gradients of the anchoring loss and larger radii to less sensitive examples, while maintaining the average perturbation budget.
Critical strengths
(1) The clean accuracy–robustness trade-off remains a central challenge in adversarial learning. Preserving clean knowledge while acquiring robustness is practically relevant.
(2) Unlike robust-teacher distillation, CFA uses a naturally trained model as its reference. This reduces dependence on obtaining a separate adversarially trained teacher.
(3) The study compares teachers trained for different durations and with different schemes, including standard training, label smoothing, and mixup. The results suggest that clean accuracy alone is insufficient for selecting a useful teacher.
Critical weaknesses
(1) The methodological novelty relative to prior natural-teacher-guided adversarial training is not sufficiently established. CFA builds on the general idea of using a naturally trained model to guide robust training, which is also explored by methods such as LBGAT and ARREST. The main distinction is the use of a fixed clean feature target for adversarial inputs, together with a frozen classifier and sensitivity-matched radii. It would be better to highlight the differences from LBGAT and ARREST and the improvements from the unique designs.
(2) The connection between clean feature anchoring and sensitivity-matched perturbation radii is unclear. Section 3 analyzes the fixed clean-target strategy, whereas Section 4.2 introduces sensitivity-matched perturbation radii. These two components appear to be developed largely independently, leaving their underlying connection insufficiently explained. In particular, it is unclear whether sensitivity matching is theoretically motivated by the feature-anchoring objective, whether the two components address complementary aspects of the accuracy–robustness trade-off, or whether their benefits are largely independent. A clearer explanation of their interaction, ideally supported by theoretical analysis or targeted ablation studies, would strengthen the coherence of the proposed framework.
(3) There seem to be only limited robustness improvements regarding the proposed strategies. (I) As shown in Table 5, w/ sensitivity-matched schedule only achieves comparable robustness to w/o it (28.01 vs. 28.04 on CIFAR100). (II) As shown in Table 8, logit regression achieves similar robustness to feature regression (25.69 vs. 25.88). In particular, previous methods like LBGAT have already adopted the logit-regression for adversarial training.
Questions for authors
See Weaknesses.
4 comments
Official Review of Submission32083 by Reviewer mgAm - 2
Additional evidence on CFA's design and component effects
We provide the design comparison, full target-by-radius results, and local sensitivity argument underlying our responses to W1–W3.
W1 — How the clean reference is used
| Component | LBGAT | ARREST | CFA |
|---|---|---|---|
| Natural reference | Updated jointly with the robust model | Frozen pretrained model | Frozen pretrained model |
| Student task objective | Logit matching, with variant-specific terms | Classification loss plus feature regularization | Squared feature regression |
| Inner attack | Student prediction KL in the released TRADES variant | Label cross-entropy | Anchor discrepancy |
| Perturbation control | Uniform radius | Uniform-radius PGD or random noise, selected by a feature-drift threshold | Per-example radius from anchor-loss sensitivity |
LBGAT also applies clean-label supervision to its natural branch. ARREST's feature regularizer uses angular distance, and its noisy replay switches to random perturbations when clean representation drift exceeds a threshold. CFA directly uses the fixed clean feature as the regression target and the reference for constructing adversarial inputs. The shared anchor objective makes its role in both optimization steps explicit. LBGAT §3.2, ARREST §§4–5
W2 — Local motivation for sensitivity allocation
Let , with the teacher's clean target held fixed. Linearizing at an input , the worst-case first-order loss increase for an L-infinity perturbation of radius is
For fixed positive sensitivity scores , choosing
preserves the mean radius and equalizes the products . This motivates assigning smaller radii to inputs with greater anchor-loss sensitivity. The implementation uses an L2 gradient-norm proxy and practical stabilization/clipping, so the equal-product argument describes the idealized allocation. The actual nonlinear loss and finite-step PGD require empirical evaluation.
The anchor therefore supplies both the discrepancy to attack and the sensitivity signal for distributing the perturbation budget. Their empirical effects are measured by the attack-only control in our first response and the factorial comparison below.
W2/W3 — Complete target-by-radius comparison
These results use CIFAR-100/ResNet-18, a common natural teacher, 150 student epochs, WA/AWP, three seeds per cell, and final-checkpoint selection. The mean training radius and AutoAttack evaluation radius are both 8/255. Accuracy entries are mean ± sample SD. NRR is the harmonic mean of clean and AA accuracy, computed per seed and then averaged.
| Target / radius | Clean (%) | AA (%) | NRR |
|---|---|---|---|
| Feature / uniform | 62.63±0.22 | 28.01±0.20 | 38.71 |
| Feature / sensitivity | 64.82±0.07 | 28.04±0.09 | 39.15 |
| Logit / uniform | 62.29±0.13 | 27.67±0.20 | 38.32 |
| Logit / sensitivity | 64.44±0.08 | 27.81±0.06 | 38.85 |
Allocation adds 2.19 clean points with feature targets and 2.15 with logit targets, with AA changes of +0.03 and +0.14. The paired NRR interaction, defined as the allocation gain for features minus the allocation gain for logits, averages −0.10 points and ranges from −0.54 to +0.27. Thus, the observed clean-retention benefit extends to both targets, while feature-specific synergy is weakly resolved.
With sensitivity allocation, the feature target adds 0.38 clean points and 0.23 AA points on CIFAR-100. On CIFAR-10, the corresponding three-seed means are 87.32/51.46 for features and 87.35/51.63 for logits. We will state this dataset dependence explicitly and distinguish the target effect from the larger clean-retention gain due to allocation.
Thank you for the clarifications and additional experiments. Two concerns remain:
(1) Limited novelty over BGAT. BGAT already uses pretrained clean logits as regression targets. Therefore, regression to a fixed clean reference is not itself novel.
(2) The improvement in robustness from the proposed strategies is limited. Although the performance on natural images is enhanced, the paper fails to show that the proposed method can further enhance robustness through a better trade-off strategy.
Considering the above issues, I tend to keep my score.
Author Response to Reviewer mgAm
Thank you for clarifying your remaining concerns. We acknowledge BGAT's fixed-target regression precedent, but ask that CFA be assessed through its teacher analysis, sensitivity-based training design, and repeated accuracy–robustness improvements.
1. Improvements over BGAT on both metrics
The following CIFAR-100/ResNet-18 results share the natural teacher, 150 student epochs, WA/AWP, a nominal or mean training radius of 8/255, and final-checkpoint selection. Entries are means over three seeds. Robust evaluation uses APGD-CE and APGD-T at 8/255.
| Method | Clean (%) | Robust (%) |
|---|---|---|
| BGAT + WA/AWP | 61.94 | 27.81 |
| BGAT + WA/AWP + sensitivity allocation | 63.73 | 27.60 |
| CFA | 64.82 | 28.04 |
CFA gains 2.88 clean points and 0.23 robust points over BGAT. Even after giving BGAT our allocation rule, the gains are 1.09 clean points and 0.44 robust points. Both metrics improve against both controls in every paired seed. The clean gain is therefore not obtained by sacrificing robustness.
For a paper targeting the accuracy–robustness trade-off, recovering clean accuracy while preserving robustness is itself a relevant improvement. Assessing only the robust-accuracy increment omits this part of the objective. These are comparisons at the evaluated settings, not a claim of dominance over every possible tuning of BGAT.
2. The clean advantage can be partly exchanged for higher robustness
This is supported by completed experiments, rather than a hypothetical possibility. We repeated CFA with mean training radius 8.8/255 instead of 8/255, keeping its other settings and the evaluation radius of 8/255 unchanged. The table includes our strengthened ADR reproduction as a reference; all rows have three seeds and WA/AWP.
| Method | Student epochs | Training radius | Clean (%) | Robust (%) |
|---|---|---|---|---|
| ADR, our reproduction | 200 | 8/255 | 59.85 | 27.77 |
| CFA | 150 | mean 8/255 | 64.82 | 28.04 |
| CFA, higher training radius | 150 | mean 8.8/255 | 63.22 | 28.67 |
Relative to this ADR reproduction, the standard CFA setting retains approximately 4.97 points more clean accuracy, with 0.27 points higher robust accuracy. At the higher training radius, CFA still retains 3.37 points more clean accuracy, while its robust advantage increases to 0.89 points. Across the two CFA settings, a 1.60-point clean decrease accompanies a 0.62-point robust increase. The higher-radius CFA result is 63.22±0.14 clean / 28.67±0.10 robust accuracy (mean ± sample SD).
The higher-radius setting retains a sizeable clean advantage while increasing robustness. The approximately five-point clean gain is relative to ADR, not BGAT. Changing the training radius is not a matched-budget ablation; CFA's natural pretraining also adds work beyond the listed student epochs.
A complementary matched-radius comparison with ARREST gives 64.79 clean / 26.35 robust, versus CFA's 64.82 / 28.04, over three seeds with the same teacher initialization, 150 epochs, and WA/AWP. Here robustness improves by approximately 1.70 points at essentially unchanged clean accuracy. This compares complete methods under a common training stack.
3. Contribution beyond fixed-reference regression
The paper asks both which natural representation transfers usefully and how to use it for robust training. Its contribution is not exhausted by replacing logit MSE with feature MSE.
First, higher teacher clean accuracy does not necessarily yield a better student clean–robust balance. The 16-teacher analysis motivates examining representation geometry, with feature scatter associated with student clean accuracy. This is an empirical insight and candidate diagnostic, not a causal explanation or validated selection guarantee.
Second, CFA uses the anchor discrepancy for adversarial example generation and backbone training, and allocates radii from that objective's input sensitivity under a preserved mean-radius budget. Adaptive radii in general are not our novelty claim; the proposed signal is the sensitivity of clean-reference matching. This connects the supervision to perturbation allocation.
Third, the ablations test the design's effects. Holding the logit target, head, initialization, allocation rule, and training stack fixed, anchor-discrepancy attacks improve robust accuracy by 0.58 points over CE attacks across three seeds, with a 0.47-point clean decrease. Allocation yields the larger clean-retention benefit; the feature-versus-logit effect is modest and dataset-dependent.
BGAT's precedent does not by itself address these teacher-transfer findings or establish the effects of this sensitivity-guided design. We respectfully request reconsideration of the contribution and overall recommendation based on these analyses, methodological choices, and repeated joint gains, rather than fixed-reference regression or the default robust increment alone.
Author Response to Reviewer mgAm
Thank you for the review. We clarify the contribution of using an unchanged natural teacher's features directly, and provide new evidence for the attack objective and radius allocation.
W1 — A simple use of the clean representation
CFA uses a naturally pretrained model's clean features directly as fixed regression targets. During robust training, the student minimizes squared feature error, and the inner attack maximizes the same error. This makes the clean representation itself the reference for acquiring robustness.
LBGAT §3.2 questions whether a static natural classifier boundary is suitable for robustness and jointly trains the natural and robust branches to adapt that boundary. CFA retains the pretrained teacher throughout robust training. ARREST also uses a frozen clean teacher, combining angular feature regularization with adversarial classification and replacing PGD perturbations with random noise when feature drift exceeds a threshold. Its PGD attack maximizes label cross-entropy. CFA's task loss is direct squared feature regression, which also defines its attack objective. This provides a simpler way to use the clean representation during robust training. LBGAT §3.2, ARREST §§4–5
Our teacher analysis also examines what makes this fixed reference transferable. Teacher clean accuracy alone does not explain student performance. Across 16 checkpoints and configurations, the within-class to between-class feature scatter ratio correlates with student clean accuracy (Spearman ρ = 0.91). This identifies representation geometry as a candidate for teacher selection. Shared training trajectories among checkpoints make this a descriptive association requiring independent validation.
In three CIFAR-100 seeds, LBGAT + WA/AWP reaches 55.79% clean / 26.27% AutoAttack (AA), compared with CFA's 64.82% / 28.04%. This compares complete recipes with LBGAT's own optimizer and attack. For closer attribution, we also strengthened BGAT, the fixed-teacher regression variant, using the same teacher, 150 student epochs, WA/AWP, mean training radius 8/255, final checkpoint selection, and AA at 8/255.
CFA improves both metrics against both controls in every paired seed, including the control that receives the allocation rule.
W2 — How the anchor determines the attack and radius
The anchor loss specifies which deviations the student should resist. Maximizing it searches for inputs that violate the clean reference, while its input gradient measures how sensitive that reference-matching objective is. A first-order approximation motivates inverse sensitivity allocation: smaller radii for more sensitive examples, with the batch mean preserved. The L-infinity approximation uses the L1 gradient norm, and our implementation uses an L2 proxy.
We tested whether this choice of attack matters even when the outer regression loss stays the same. We fixed the logit target, head, initialization, allocation rule, WA/AWP, and schedule, and changed the PGD objective.
Anchor-based attacks increase AA in all three paired seeds, by 0.77, 0.53, and 0.45 points, averaging +0.58 AA with −0.47 clean. Thus, the same regression loss yields different robustness when the attack explicitly targets its discrepancy. This measures the attack effect under a shared adaptive allocation rule, whose realized radii evolve with training.
The target-by-allocation ablation addresses interaction. Allocation adds +2.19 clean / +0.03 AA with features and +2.15 / +0.14 with logits. Its benefit is similar for both targets. The interaction in their harmonic mean averages −0.10 points (seed range −0.54 to +0.27), providing limited evidence of feature-specific synergy.
W3 — Improved clean retention at comparable robustness
Table 5 shows that allocation preserves AA while improving clean accuracy. Across three seeds, uniform feature anchoring gives 62.63±0.22 clean / 28.01±0.20 AA, and allocation gives 64.82±0.07 / 28.04±0.09. The paired clean gains are 2.35, 2.24, and 1.97 points.
Your observation that logits are competitive holds in the repeated full-recipe comparison. Features gain 0.38 clean / 0.23 AA on CIFAR-100, while logits lead by 0.14 points in the harmonic mean on CIFAR-10. Both implement direct regression to a fixed clean target. The method-level gains remain positive against BGAT with allocation in every paired seed. We will distinguish this overall gain from the small, dataset-dependent target effect.