acceptodds
Under review as a conference paper at ICLR 2027

CLEAN FEATURE ANCHORING: MITIGATING THE ACCURACY–ROBUSTNESS TRADE-OFF BY SELF-DISTILLING A NATURALLY TRAINED MODEL

Abstract

Adversarial training improves robustness but often sacrifices accuracy on clean data, leading to a persistent clean–robust trade-off. We address this trade-off by first learning a high-accuracy clean self-teacher from natural images and then preserving its clean knowledge while adversarially training the student. We propose Clean Feature Anchoring (CFA), which matches the student's representation of an adversarial input to the teacher's fixed representation of the corresponding clean input while retaining the teacher's classifier. We show that controlling this fixed clean anchor simultaneously preserves agreement with the teacher on clean inputs and limits representation variation within the perturbation neighborhood. We also find that teacher clean accuracy alone does not determine how well clean performance is retained after robust training, indicating that transferability depends on properties beyond teacher accuracy. Finally, because the same input-space perturbation can induce substantially different changes in the anchoring objective across examples, we introduce sensitivity-matched perturbation budgets that assign smaller radii to more sensitive examples and larger radii to less sensitive ones while preserving the average perturbation budget. Across CIFAR-10, CIFAR-100, and Tiny-ImageNet, CFA consistently combines high clean accuracy with strong adversarial robustness.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial Training

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_mgAmOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Clean Feature Anchoring (CFA), a method for mitigating the clean accuracy–adversarial robustness trade-off in adversarial training. The central idea is to first train a naturally trained clean self-teacher and then freeze it as a reference. During robust training, the student is initialized from the teacher, retains its classifier, and learns by matching its features on adversarial inputs to the teacher's features on the corresponding clean inputs. The paper further introduces sensitivity-matched perturbation budgets, which assign smaller adversarial perturbation radii to examples with larger local gradients of the anchoring loss and larger radii to less sensitive examples, while maintaining the average perturbation budget.

Critical strengths

(1) The clean accuracy–robustness trade-off remains a central challenge in adversarial learning. Preserving clean knowledge while acquiring robustness is practically relevant.

(2) Unlike robust-teacher distillation, CFA uses a naturally trained model as its reference. This reduces dependence on obtaining a separate adversarially trained teacher.

(3) The study compares teachers trained for different durations and with different schemes, including standard training, label smoothing, and mixup. The results suggest that clean accuracy alone is insufficient for selecting a useful teacher.

Critical weaknesses

(1) The methodological novelty relative to prior natural-teacher-guided adversarial training is not sufficiently established. CFA builds on the general idea of using a naturally trained model to guide robust training, which is also explored by methods such as LBGAT and ARREST. The main distinction is the use of a fixed clean feature target for adversarial inputs, together with a frozen classifier and sensitivity-matched radii. It would be better to highlight the differences from LBGAT and ARREST and the improvements from the unique designs.

(2) The connection between clean feature anchoring and sensitivity-matched perturbation radii is unclear. Section 3 analyzes the fixed clean-target strategy, whereas Section 4.2 introduces sensitivity-matched perturbation radii. These two components appear to be developed largely independently, leaving their underlying connection insufficiently explained. In particular, it is unclear whether sensitivity matching is theoretically motivated by the feature-anchoring objective, whether the two components address complementary aspects of the accuracy–robustness trade-off, or whether their benefits are largely independent. A clearer explanation of their interaction, ideally supported by theoretical analysis or targeted ablation studies, would strengthen the coherence of the proposed framework.

(3) There seem to be only limited robustness improvements regarding the proposed strategies. (I) As shown in Table 5, w/ sensitivity-matched schedule only achieves comparable robustness to w/o it (28.01 vs. 28.04 on CIFAR100). (II) As shown in Table 8, logit regression achieves similar robustness to feature regression (25.69 vs. 25.88). In particular, previous methods like LBGAT have already adopted the logit-regression for adversarial training.

Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer mgAm

Thank you for the review. We clarify the contribution of using an unchanged natural teacher's features directly, and provide new evidence for the attack objective and radius allocation.

W1 — A simple use of the clean representation

CFA uses a naturally pretrained model's clean features directly as fixed regression targets. During robust training, the student minimizes squared feature error, and the inner attack maximizes the same error. This makes the clean representation itself the reference for acquiring robustness.

LBGAT §3.2 questions whether a static natural classifier boundary is suitable for robustness and jointly trains the natural and robust branches to adapt that boundary. CFA retains the pretrained teacher throughout robust training. ARREST also uses a frozen clean teacher, combining angular feature regularization with adversarial classification and replacing PGD perturbations with random noise when feature drift exceeds a threshold. Its PGD attack maximizes label cross-entropy. CFA's task loss is direct squared feature regression, which also defines its attack objective. This provides a simpler way to use the clean representation during robust training. LBGAT §3.2, ARREST §§4–5

Our teacher analysis also examines what makes this fixed reference transferable. Teacher clean accuracy alone does not explain student performance. Across 16 checkpoints and configurations, the within-class to between-class feature scatter ratio correlates with student clean accuracy (Spearman ρ = 0.91). This identifies representation geometry as a candidate for teacher selection. Shared training trajectories among checkpoints make this a descriptive association requiring independent validation.

In three CIFAR-100 seeds, LBGAT + WA/AWP reaches 55.79% clean / 26.27% AutoAttack (AA), compared with CFA's 64.82% / 28.04%. This compares complete recipes with LBGAT's own optimizer and attack. For closer attribution, we also strengthened BGAT, the fixed-teacher regression variant, using the same teacher, 150 student epochs, WA/AWP, mean training radius 8/255, final checkpoint selection, and AA at 8/255.

MethodClean (%)AA (%)
BGAT + WA/AWP61.9427.81
BGAT + WA/AWP + allocation63.7327.60
CFA64.8228.04

CFA improves both metrics against both controls in every paired seed, including the control that receives the allocation rule.

W2 — How the anchor determines the attack and radius

The anchor loss specifies which deviations the student should resist. Maximizing it searches for inputs that violate the clean reference, while its input gradient measures how sensitive that reference-matching objective is. A first-order approximation motivates inverse sensitivity allocation: smaller radii for more sensitive examples, with the batch mean preserved. The L-infinity approximation uses the L1 gradient norm, and our implementation uses an L2 proxy.

We tested whether this choice of attack matters even when the outer regression loss stays the same. We fixed the logit target, head, initialization, allocation rule, WA/AWP, and schedule, and changed the PGD objective.

PGD objectiveClean (%)AA (%)
Label cross-entropy64.9127.23
Anchor discrepancy64.4427.81

Anchor-based attacks increase AA in all three paired seeds, by 0.77, 0.53, and 0.45 points, averaging +0.58 AA with −0.47 clean. Thus, the same regression loss yields different robustness when the attack explicitly targets its discrepancy. This measures the attack effect under a shared adaptive allocation rule, whose realized radii evolve with training.

The target-by-allocation ablation addresses interaction. Allocation adds +2.19 clean / +0.03 AA with features and +2.15 / +0.14 with logits. Its benefit is similar for both targets. The interaction in their harmonic mean averages −0.10 points (seed range −0.54 to +0.27), providing limited evidence of feature-specific synergy.

W3 — Improved clean retention at comparable robustness

Table 5 shows that allocation preserves AA while improving clean accuracy. Across three seeds, uniform feature anchoring gives 62.63±0.22 clean / 28.01±0.20 AA, and allocation gives 64.82±0.07 / 28.04±0.09. The paired clean gains are 2.35, 2.24, and 1.97 points.

Your observation that logits are competitive holds in the repeated full-recipe comparison. Features gain 0.38 clean / 0.23 AA on CIFAR-100, while logits lead by 0.14 points in the harmonic mean on CIFAR-10. Both implement direct regression to a fixed clean target. The method-level gains remain positive against BGAT with allocation in every paired seed. We will distinguish this overall gain from the small, dataset-dependent target effect.

AuthorsOpenReview ↗

Official Review of Submission32083 by Reviewer mgAm - 2

Additional evidence on CFA's design and component effects

We provide the design comparison, full target-by-radius results, and local sensitivity argument underlying our responses to W1–W3.

W1 — How the clean reference is used

ComponentLBGATARRESTCFA
Natural referenceUpdated jointly with the robust modelFrozen pretrained modelFrozen pretrained model
Student task objectiveLogit matching, with variant-specific termsClassification loss plus feature regularizationSquared feature regression
Inner attackStudent prediction KL in the released TRADES variantLabel cross-entropyAnchor discrepancy
Perturbation controlUniform radiusUniform-radius PGD or random noise, selected by a feature-drift thresholdPer-example radius from anchor-loss sensitivity

LBGAT also applies clean-label supervision to its natural branch. ARREST's feature regularizer uses angular distance, and its noisy replay switches to random perturbations when clean representation drift exceeds a threshold. CFA directly uses the fixed clean feature as the regression target and the reference for constructing adversarial inputs. The shared anchor objective makes its role in both optimization steps explicit. LBGAT §3.2, ARREST §§4–5

W2 — Local motivation for sensitivity allocation

Let ℓi(u)=∥Φs(u)−Φt(xi)∥22\ell_i(u)=\|\Phi_s(u)-\Phi_t(x_i)\|_2^2, with the teacher's clean target held fixed. Linearizing at an input uu, the worst-case first-order loss increase for an L-infinity perturbation of radius rir_i is

max⁡∥δ∥∞≤ri∇uℓi(u)⊤δ=ri∥∇uℓi(u)∥1.\max_{\|\delta\|_\infty\le r_i}\nabla_u\ell_i(u)^\top\delta = r_i\|\nabla_u\ell_i(u)\|_1.

For fixed positive sensitivity scores gig_i, choosing

ϵi=Nϵˉgi−1∑jgj−1\epsilon_i=N\bar\epsilon\frac{g_i^{-1}}{\sum_j g_j^{-1}}

preserves the mean radius and equalizes the products ϵigi\epsilon_i g_i. This motivates assigning smaller radii to inputs with greater anchor-loss sensitivity. The implementation uses an L2 gradient-norm proxy and practical stabilization/clipping, so the equal-product argument describes the idealized allocation. The actual nonlinear loss and finite-step PGD require empirical evaluation.

The anchor therefore supplies both the discrepancy to attack and the sensitivity signal for distributing the perturbation budget. Their empirical effects are measured by the attack-only control in our first response and the factorial comparison below.

W2/W3 — Complete target-by-radius comparison

These results use CIFAR-100/ResNet-18, a common natural teacher, 150 student epochs, WA/AWP, three seeds per cell, and final-checkpoint selection. The mean training radius and AutoAttack evaluation radius are both 8/255. Accuracy entries are mean ± sample SD. NRR is the harmonic mean of clean and AA accuracy, computed per seed and then averaged.

Target / radiusClean (%)AA (%)NRR
Feature / uniform62.63±0.2228.01±0.2038.71
Feature / sensitivity64.82±0.0728.04±0.0939.15
Logit / uniform62.29±0.1327.67±0.2038.32
Logit / sensitivity64.44±0.0827.81±0.0638.85

Allocation adds 2.19 clean points with feature targets and 2.15 with logit targets, with AA changes of +0.03 and +0.14. The paired NRR interaction, defined as the allocation gain for features minus the allocation gain for logits, averages −0.10 points and ranges from −0.54 to +0.27. Thus, the observed clean-retention benefit extends to both targets, while feature-specific synergy is weakly resolved.

With sensitivity allocation, the feature target adds 0.38 clean points and 0.23 AA points on CIFAR-100. On CIFAR-10, the corresponding three-seed means are 87.32/51.46 for features and 87.35/51.63 for logits. We will state this dataset dependence explicitly and distinguish the target effect from the larger clean-retention gain due to allocation.

Thank you for the clarifications and additional experiments. Two concerns remain:

(1) Limited novelty over BGAT. BGAT already uses pretrained clean logits as regression targets. Therefore, regression to a fixed clean reference is not itself novel.
(2) The improvement in robustness from the proposed strategies is limited. Although the performance on natural images is enhanced, the paper fails to show that the proposed method can further enhance robustness through a better trade-off strategy.

Considering the above issues, I tend to keep my score.

AuthorsOpenReview ↗

Author Response to Reviewer mgAm

Thank you for clarifying your remaining concerns. We acknowledge BGAT's fixed-target regression precedent, but ask that CFA be assessed through its teacher analysis, sensitivity-based training design, and repeated accuracy–robustness improvements.

1. Improvements over BGAT on both metrics

The following CIFAR-100/ResNet-18 results share the natural teacher, 150 student epochs, WA/AWP, a nominal or mean training radius of 8/255, and final-checkpoint selection. Entries are means over three seeds. Robust evaluation uses APGD-CE and APGD-T at 8/255.

MethodClean (%)Robust (%)
BGAT + WA/AWP61.9427.81
BGAT + WA/AWP + sensitivity allocation63.7327.60
CFA64.8228.04

CFA gains 2.88 clean points and 0.23 robust points over BGAT. Even after giving BGAT our allocation rule, the gains are 1.09 clean points and 0.44 robust points. Both metrics improve against both controls in every paired seed. The clean gain is therefore not obtained by sacrificing robustness.

For a paper targeting the accuracy–robustness trade-off, recovering clean accuracy while preserving robustness is itself a relevant improvement. Assessing only the robust-accuracy increment omits this part of the objective. These are comparisons at the evaluated settings, not a claim of dominance over every possible tuning of BGAT.

2. The clean advantage can be partly exchanged for higher robustness

This is supported by completed experiments, rather than a hypothetical possibility. We repeated CFA with mean training radius 8.8/255 instead of 8/255, keeping its other settings and the evaluation radius of 8/255 unchanged. The table includes our strengthened ADR reproduction as a reference; all rows have three seeds and WA/AWP.

MethodStudent epochsTraining radiusClean (%)Robust (%)
ADR, our reproduction2008/25559.8527.77
CFA150mean 8/25564.8228.04
CFA, higher training radius150mean 8.8/25563.2228.67

Relative to this ADR reproduction, the standard CFA setting retains approximately 4.97 points more clean accuracy, with 0.27 points higher robust accuracy. At the higher training radius, CFA still retains 3.37 points more clean accuracy, while its robust advantage increases to 0.89 points. Across the two CFA settings, a 1.60-point clean decrease accompanies a 0.62-point robust increase. The higher-radius CFA result is 63.22±0.14 clean / 28.67±0.10 robust accuracy (mean ± sample SD).

The higher-radius setting retains a sizeable clean advantage while increasing robustness. The approximately five-point clean gain is relative to ADR, not BGAT. Changing the training radius is not a matched-budget ablation; CFA's natural pretraining also adds work beyond the listed student epochs.

A complementary matched-radius comparison with ARREST gives 64.79 clean / 26.35 robust, versus CFA's 64.82 / 28.04, over three seeds with the same teacher initialization, 150 epochs, and WA/AWP. Here robustness improves by approximately 1.70 points at essentially unchanged clean accuracy. This compares complete methods under a common training stack.

3. Contribution beyond fixed-reference regression

The paper asks both which natural representation transfers usefully and how to use it for robust training. Its contribution is not exhausted by replacing logit MSE with feature MSE.

First, higher teacher clean accuracy does not necessarily yield a better student clean–robust balance. The 16-teacher analysis motivates examining representation geometry, with feature scatter associated with student clean accuracy. This is an empirical insight and candidate diagnostic, not a causal explanation or validated selection guarantee.

Second, CFA uses the anchor discrepancy for adversarial example generation and backbone training, and allocates radii from that objective's input sensitivity under a preserved mean-radius budget. Adaptive radii in general are not our novelty claim; the proposed signal is the sensitivity of clean-reference matching. This connects the supervision to perturbation allocation.

Third, the ablations test the design's effects. Holding the logit target, head, initialization, allocation rule, and training stack fixed, anchor-discrepancy attacks improve robust accuracy by 0.58 points over CE attacks across three seeds, with a 0.47-point clean decrease. Allocation yields the larger clean-retention benefit; the feature-versus-logit effect is modest and dataset-dependent.

BGAT's precedent does not by itself address these teacher-transfer findings or establish the effects of this sensitivity-guided design. We respectfully request reconsideration of the contribution and overall recommendation based on these analyses, methodological choices, and repeated joint gains, rather than fixed-reference regression or the default robust increment alone.

Reviewer_jKrjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a method, CFA to mitigate the accuracy-robustness trade-off during adversarial training by self-distilling a naturally trained model. The student starts from this teacher and learns to make its internal features on attacked images match the teacher’s features on the original images, while keeping the final classifier fixed. The paper also introduces sensitivity-matched perturbation to assign different attack strengths to training examples according to their sensitivity. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet show a promising balance between clean accuracy and robustness.

Critical strengths
  • CFA uses a naturally trained teacher rather than requiring an adversarially trained teacher. This makes it an attractive alternative to methods that depend on an expensive robust teacher or multiple teachers.
  • Promising results across several datasets and two network families. For example, on CIFAR-100 with ResNet-18, it achieves 64.82% clean accuracy and 28.04% AutoAttack accuracy, improving both measures over the ADR baseline reproduced in the paper.
  • Useful ablation experiments that examine why the method works. The paper examines teacher initialization, weight averaging, adversarial weight perturbation, different clean self-teachers, and perturbation-radius allocation. Repeated CIFAR-100 experiments show that sensitivity-matched perturbation radii improve clean accuracy while maintaining AutoAttack accuracy.
Critical weaknesses
  • It is unclear whether matching internal features is better than matching output scores. On CFA page 14, Table 11, the results are very close: feature matching achieves 28.42% robust accuracy versus 28.02% for output-score matching, with almost identical clean accuracy. This comparison uses only one run and excludes CFA’s adaptive attack budgets. What is missing: repeated comparisons using the full method on more than one dataset, to show that feature matching provides a consistent benefit.

  • Comparisons with the closest methods are incomplete. B-MTARD is discussed on CFA page 3, but is absent from the main comparison tables on pages 7–8. Also, ADR main paper reports 28.50% robust accuracy on CIFAR-100, compared with CFA’s 28.04% on CFA page 7, Table 2; CFA has higher clean accuracy. What is missing: a direct B-MTARD comparison and repeated comparisons against the strongest ADR configuration, with comparable training budgets and consistent model-selection rules. This would clarify whether CFA offers a reliably better balance.

Questions for authors
  • To compare CFA directly with B-MTARD using the same attack evaluation and model-selection rules, while reporting the total training cost, including teachers?
  • If possible repeat feature matching versus output-score matching using CFA’s full training setup across multiple runs and datasets? The results on page 14, Table 11, are very close.
  • To compare CFA with the strongest ADR configuration using comparable training budgets and repeated runs? ADR reports 28.50% AutoAttack accuracy on CIFAR-100, versus CFA’s 28.04%, although CFA has higher clean accuracy.
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer jKrj

Thank you for the review. We report the requested repeated feature–logit comparison and clarify the comparisons with ADR and B-MTARD. All AutoAttack (AA) results below use an evaluation radius of 8/255.

W1/Q2 — Feature versus logit matching

We repeated the comparison with CFA’s full training recipe, including sensitivity-based radius allocation, on CIFAR-10 and CIFAR-100. Both variants directly regress to a fixed clean teacher target before softmax: internal features or raw logits. In conventional probability-based distillation, the teacher target is a softmax distribution. Our experiment compares the two direct regression targets within CFA.

DatasetFeature clean / AA (%)Logit clean / AA (%)
CIFAR-10064.82 ± 0.07 / 28.04 ± 0.0964.44 ± 0.08 / 27.81 ± 0.06
CIFAR-1087.32 / 51.4687.35 / 51.63

Each comparison uses three paired seeds. CIFAR-100 entries are mean ± sample SD. Feature matching gives a small gain on CIFAR-100. On CIFAR-10, logit matching leads by 0.14 points in the harmonic mean of clean and AA accuracy. These results show that both targets work within CFA and that the modest target effect varies by dataset.

The complete CFA recipe improves over BGAT with the same natural teacher, WA/AWP, and sensitivity allocation rule on CIFAR-100 (64.82 / 28.04 versus 63.73 / 27.60). Both metrics favor CFA in all three paired seeds.

W2/Q3 — Strongest ADR configuration

CIFAR-100 settingStudent epochsSeedsClean (%)AA (%)
Published AT+WA+AWP+ADR——57.3628.50
Our AT+WA+AWP+ADR runs200359.8527.77
CFA150364.8228.04
CFA (tuned training setting)150163.2328.54

The tuned CFA run gains 5.87 points in clean accuracy at essentially the same AA accuracy as published ADR. This exploratory run uses a training radius of 8.8/255 selected from earlier experiments. Its evaluation radius remains 8/255.

With the same evaluation and final-checkpoint selection rule, CFA improves over our ADR runs by 4.97 clean points and 0.27 AA points on average, with both metrics higher in every paired seed. CFA uses a separately trained 200-epoch natural teacher. ADR uses an EMA teacher during its 200 student epochs. The runs align evaluation and checkpoint selection, while a matched comparison of teacher-inclusive training time remains open.

Q1 — B-MTARD

We compare CFA’s three-seed means with the published ResNet-18 student results. B-MTARD, Table III and Section IV-A

DatasetB-MTARD clean / AA (%)CFA clean / AA (%)
CIFAR-10065.08 / 23.9864.82 / 28.04
CIFAR-1088.20 / 47.4487.32 / 51.46

CFA uses one naturally trained ResNet-18 teacher. B-MTARD uses two pretrained teachers: a clean WideResNet-22-6 and a robust WideResNet-70-16 on CIFAR-100, and a clean ResNet-56 and a robust WideResNet-34-10 on CIFAR-10. With its single same-architecture teacher, CFA achieves 4.06 and 4.02 points higher AA, respectively, with clean accuracy lower by 0.26 and 0.88 points.

B-MTARD trains the student for 300 epochs and selects the checkpoint with the best weighted robust accuracy. CFA reports the final checkpoint after 150 student epochs. Matched checkpoint selection and teacher-inclusive timing remain open.

Reviewer_5i5GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the trade-off between clean accuracy and adversarial robustness. The proposed Clean Feature Anchoring (CFA) method first trains a natural teacher, initializes a student from that teacher, and subsequently freezes both the teacher and the inherited classification head. During robust training, only the student feature extractor is updated. The inner maximization searches for perturbations that increase the discrepancy between the student’s perturbed-input features and the teacher’s clean-input features, while the outer minimization reduces this discrepancy. No separate label cross-entropy objective is used during this robust training stage. The paper analyzes how a fixed anchor controls both clean feature fidelity and representation variation within a perturbation neighborhood. A restricted two-feature model further illustrates that reproducing a teacher’s clean representation need not preserve the fragile feature route used by the teacher. The authors also investigate teacher transferability and find that higher teacher clean accuracy does not necessarily produce a better robust student. Finally, they introduce sensitivity-based, per-example training radii that preserve the mean perturbation radius within a batch. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet show its great balance between clean accuracy and robustness, using ResNet-18 and WideResNet-34-10.

Critical strengths
  1. The approach is simple but produces meaningful results, with 6.69% and 5.91% increase of clean accuracy and robust accuracy.
  2. The validation experiments are very sufficient. The sensitivity-based radius allocation has repeated-run support and a reasonably informative assignment control.
Critical weaknesses
  1. The incremental contribution of feature anchoring over existing fixed clean-target regression remains insufficiently isolated. The paper appropriately identifies LBGAT as the closest direct-regression precedent. The basic idea of using a fixed natural teacher and clean-target regression predates this work.
  2. Teacher configurations and implementation versions are not sufficiently consistent. Section 5.1 states that distillation baselines use the same natural teacher as CFA where applicable. However, the discussion on page 18 states that adversarial-distillation baselines rely on robust teachers and that their teacher-training costs are excluded. Table 16 also lists a teacher cost of 0.39 hours for ARD, RSLAD, and AdaAD. A method-by-method specification of teacher type, architecture, and clean/robust accuracy is needed to make these comparisons interpretable.
Questions for authors
  1. Additional evidence relevant to contribution attribution. Under the final 150-epoch, 8/255 noise strength, could the authors provide a matched comparison of clean-feature versus clean-logit regression, each with uniform versus sensitivity-based radii, with repeated runs for the key comparisons? This would clarify whether the incremental benefit comes from the feature target, the allocation rule, or their interaction.
  2. Clarification of baseline configurations. Please specify the teacher type, architecture, clean accuracy, robust accuracy, and cost accounting for the ARD, RSLAD, AdaAD, and AdaAD+IGDM results in the main tables. Do the key Tiny-ImageNet and WideResNet comparisons remain valid after correcting the LBGAT schedule?
  3. Clarification of innovation points. Building upon LBGAT, please elaborate on the innovations presented in this paper.
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer 5i5G

Thank you for the constructive review. We have completed the requested repeated target-by-radius comparison, audited the teacher specifications, and checked the LBGAT comparisons across architectures.

Q1 — Contribution attribution in the final training setting

We evaluated all four combinations on CIFAR-100/ResNet-18 with three student seeds each, 150 epochs, and a mean training radius of 8/255. The teacher, initialization, frozen head, optimizer, WA/AWP, and final-checkpoint rule are shared. Entries are mean ± sample SD. The local robust evaluations use APGD-CE and APGD-T at 8/255.

Target / radiusClean (%)Robust (%)NRR
Feature / uniform62.63±0.2228.01±0.2038.71
Feature / sensitivity64.82±0.0728.04±0.0939.15
Logit / uniform62.29±0.1327.67±0.2038.32
Logit / sensitivity64.44±0.0827.81±0.0638.85

This separates the observed effects. Sensitivity allocation improves clean accuracy by 2.19 points with features and 2.15 with logits, while robust accuracy changes by +0.03 and +0.14. Feature matching improves robust accuracy by 0.34 points under uniform radii and 0.23 under sensitivity allocation. Its advantage is modest; on CIFAR-10 the repeated full-recipe comparison slightly favors logits.

The mean interaction in NRR is −0.10, with paired-seed values spanning −0.54 to +0.27. We therefore do not claim a positive interaction or statistical independence. The strongest supported attribution is a clean-accuracy benefit from allocation with approximately preserved robustness, plus a smaller, dataset-dependent target effect.

W2/Q2 — Teacher specifications and cost accounting

The reviewer correctly identified an inconsistency. For the audited main CIFAR ResNet-18 comparisons, CFA and the four requested distillation baselines use the same standard naturally trained 200-epoch teacher, not an adversarially trained teacher.

MethodSupervising teacherArchitectureStudent initialization
CFANatural, standard 200 epochsResNet-18Teacher checkpoint
ARDSame natural checkpointResNet-18Random
RSLADSame natural checkpointResNet-18Random
AdaADSame natural checkpointResNet-18Random
AdaAD+IGDMSame natural checkpointResNet-18Random

The shared teacher's measured clean / robust accuracies are 95.33 / 0.00 on CIFAR-10 and 77.65 / 0.00 on CIFAR-100, using the above attack protocol. Its CIFAR-100 training cost is 0.39 GPU-hours on one RTX 5080. A teacher-inclusive cost for each method is this pretraining cost plus that method's student-training time; reuse does not make the teacher free.

We will correct the appendix's robust-teacher sentence and cost caption. These rows evaluate the baselines under a common natural-teacher condition, not their original robust-teacher configurations. We will also correct the mixup description of the main CIFAR ResNet-18 teacher and distinguish it from the separate mixup/label-smoothing transfer analysis. The teacher accuracy measurements above apply to the CIFAR ResNet-18 checkpoints; they are not measurements of the Tiny-ImageNet or WRN teachers.

Q2 — LBGAT schedules on Tiny-ImageNet and WideResNet

On Tiny-ImageNet, the corrected LBGAT rerun obtains 50.73 clean / 19.11 robust accuracy, compared with the earlier 51.04 / 18.47. CFA obtains 55.16 / 20.54, retaining advantages of 4.43 and 1.43 points. We will report the corrected baseline.

The WRN LBGAT experiments were conducted on a separate server using the authors' learning-rate schedule and were unaffected by the local scheduler error. The reported WRN results—81.56 / 51.79 on CIFAR-10 and 59.63 / 28.55 on CIFAR-100—therefore remain applicable.

We also correct the earlier LBGAT stack label: AWP was not active in that run. The verified 150-epoch WA/AWP control gives 55.79 / 26.27 over three seeds.

W1/Q3 — Contribution beyond BGAT/LBGAT

We acknowledge BGAT as the fixed clean-target regression precedent and distinguish it from LBGAT's jointly trained natural branch. CFA uses the clean-anchor discrepancy for adversarial example generation and backbone training, and uses its input sensitivity to allocate radii under an inherited frozen classifier.

The evidence supports this framework rather than a claim that feature regression alone explains the gain. At 150 epochs with the same natural teacher and WA/AWP, BGAT obtains 61.94 / 27.81; adding sensitivity allocation gives 63.73 / 27.60; CFA gives 64.82 / 28.04. CFA improves both metrics over both controls in all three paired seeds. Separately, changing only the attack objective in the logit/sensitivity setting from label CE to anchor discrepancy improves robust accuracy by 0.58 points across three seeds, with a 0.47-point clean decrease. This identifies an attack-objective effect beyond the choice of regression target.

Reviewer_JtZLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to mitigate the clean-robust trade-off by (1) training a teacher on clean image data and (2) adversarially training a student model to align its representations with those of the clean teacher.

To train the student model, the authors propose feature anchoring, which encourages the feature vector of an adversarial example in the student model to be similar to that of the corresponding clean example in the teacher model.

The student is initialized from the teacher, and the teacher's classification head is retained and frozen during adversarial training. This design aims to preserve clean accuracy while improving adversarial robustness.

Unlike previous adversarial distillation methods that primarily focus on transferring robustness from a robust teacher, this paper aims to transfer clean knowledge from a naturally trained teacher.

Critical strengths
  • Without relying on large robust teachers or extra data, the proposed method achieves high clean accuracy and AutoAttack accuracy.
  • The method is well-motivated and effectively mitigates the clean-robust trade-off.
Critical weaknesses

[W1] Proposition 1 provides a rather straightforward bound. While this proposition aims to show that feature anchoring with a clean teacher can constrain both clean fidelity and local feature variation, the bound follows directly from the triangle inequality. Therefore, its theoretical contribution appears limited.

[W2] Although Proposition 2 provides an interesting observation, it is established under a highly simplified two-feature model, where one feature is assumed to be completely robust and the others are fully perturbable. It remains unclear whether the proposed feature anchoring mechanism can similarly eliminate non-robust features in practical neural networks.

[W3] Novelty: As far as I understand, the main contribution of this paper is the use of feature anchoring with a clean teacher. However, previous studies have already explored using a clean teacher for adversarial training. Moreover, the appendix results suggest that fixed-head logit regression also achieves competitive performance. Therefore, it remains unclear what the essential advantage of feature anchoring is over existing clean-teacher-based methods. It would be helpful to clarify whether the main performance improvement comes from feature matching or sensitivity-matched perturbation budgets.

Questions for authors

[Q1] Eq 1 shows that KL distillation converges to the logit regression as the temperture increases. However, does this result provide a theoretical justification for feature regression? Could the authors clarify how this result motivates feature-level matching rather than logit-level matching?

[Q2] It remains unclear why the mixup-trained teacher transfers better than the label-smoothed teacher. Could the authors provide further insight into the underlying mechanism? If the authors could identify the underlying factors, it would be beneficial for selecting a suitable teacher before training the student.

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer JtZL

Thank you for the thoughtful review. We explain how the theoretical observations inform CFA's design and report new experiments on the attack objective, regression target, and radius allocation.

W1 — Role of Proposition 1

We agree that the bound follows directly from the triangle inequality. The observation identifies a quantity that bounds both clean feature error and local feature variation: the worst-case discrepancy from a fixed clean reference. This motivates CFA's use of the anchor discrepancy in both the inner maximization and the outer minimization. The original BGAT formulation generates perturbations by maximizing label cross-entropy. CFA uses finite-step PGD to approximate the maximum anchor discrepancy.

On CIFAR-100, we changed the PGD objective while keeping the logit regression loss, frozen head, initialization, allocation rule, and training schedule fixed. Maximizing the anchor discrepancy improved AutoAttack accuracy in all three paired seeds, with a mean gain of 0.58 percentage points and a mean decrease of 0.47 points in clean accuracy. We will present Proposition 1 as an observation that motivates this design choice.

W2 — Scope of Proposition 2

The two-feature model assumes an unperturbable robust feature, fully perturbable weak features, and exact inner maximization. Under these assumptions, the student can reproduce the teacher's clean representation while removing its reliance on the perturbable features the teacher used to compute it. This illustrates how clean fidelity and robustness can coexist. Our network experiments measure clean and robust accuracy. Establishing whether the same mechanism operates in those networks requires measuring changes in reliance on fragile features directly.

W3/Q1 — Contribution and choice of feature target

CFA uses the anchor loss to generate perturbations and train the student, and its input sensitivity determines how the training radii are allocated. On CIFAR-100, CFA reaches 64.82% clean / 28.04% AutoAttack, compared with 63.73% / 27.60% for BGAT with the same natural teacher, WA/AWP, and sensitivity allocation rule. The complete CFA recipe improves both metrics in every paired seed.

Equation 1 motivates direct regression through the centered-logit limit of high-temperature distillation. Feature matching has a separate rationale. With a shared fixed affine head, feature matching also penalizes discrepancies in directions that the head maps to zero. We tested the value of this extra constraint using three seeds per cell, 150 epochs, and a mean training radius of 8/255 on CIFAR-100. AutoAttack also uses 8/255.

Target / radiusClean (%)AutoAttack (%)
Feature / uniform62.63 ± 0.2228.01 ± 0.20
Feature / sensitivity64.82 ± 0.0728.04 ± 0.09
Logit / uniform62.29 ± 0.1327.67 ± 0.20
Logit / sensitivity64.44 ± 0.0827.81 ± 0.06

Entries are mean ± sample SD. Allocation accounts for the larger clean gain, adding 2.19 points with feature targets and 2.15 with logit targets, with mean AutoAttack changes of +0.03 and +0.14. The feature target gives a small gain on CIFAR-100. On CIFAR-10, the full recipe with logit targets leads by 0.14 points in the harmonic mean of clean and AutoAttack accuracy across three paired seeds. We will clarify that the framework works with either target and report the dataset-dependent differences.

Q2 — Teacher representation geometry and transfer

The mixup and label-smoothing comparison motivated us to look for teacher properties that can be measured before student training. Across 16 teacher checkpoints and configurations, the teacher's within-class to between-class feature scatter ratio was positively associated with student clean accuracy (Spearman ρ = 0.91). The ratio can be computed for teachers trained with different schemes.

The two teachers have similar superclass-structure AUC (0.733 and 0.735, respectively). The student trained with the label-smoothed teacher matches its own teacher more closely (0.842 versus 0.826 cosine similarity) despite lower clean accuracy. These observations motivate testing feature scatter as a candidate selection criterion. Several checkpoints share training trajectories, so validating the criterion requires independent runs.

Thank you for your response and additional experiments.

I understand that the choice of the clean teacher can affect the student's performance. Since the student is initialized from the clean teacher, I wonder whether the teacher's robustness also affects the student's performance. In this regard, could you provide the robustness of the different naturally trained teachers?

Based on the additional results, it seems that sensitivity matching mainly improves clean accuracy, while feature matching provides a relatively small improvement in AA accuracy. Since sensitivity matching assigns smaller perturbation budgets to samples with higher gradient sensitivity, is the improvement in clean accuracy mainly due to applying weaker adversarial training to these samples?

In Figure 1, the reported results marked with stars are obtained after 150 training epochs, while the other CFA results are obtained after 100 epochs, and it seems that the stars achieved a better trade-off. Could the authors clarify the number of training epochs used for each baseline? Is it possible that the baseline methods could also achieve better performance with longer training?

Could you also provide the code for reproducibility?

AuthorsOpenReview ↗

Author Response to Reviewer JtZL

Thank you for following up on our response and additional experiments. We address your four questions about teacher robustness, radius allocation, training duration, and code availability below.

1. Teacher robustness and initialization

The standard 200-epoch ResNet-18 teachers used in our audited main CIFAR experiments have clean / robust accuracies of 95.33 / 0.00 on CIFAR-10 and 77.65 / 0.00 on CIFAR-100. These evaluations use APGD-CE and APGD-T at 8/255; APGD-CE already reduces both teachers to zero accuracy. Nevertheless, their CFA students attain substantial robustness. Thus, CFA does not require the natural teacher to have nonzero adversarial accuracy under this evaluation.

Importantly, the target-by-radius comparison uses the identical teacher checkpoint and student initialization in all four cells. Its feature/allocation differences therefore cannot be explained by differences in teacher robustness or initialization across cells. This controlled evidence does not depend on the teacher-selection correlation.

The separate 16-teacher analysis asks which natural reference transfers better. We do not have robustness measurements for every checkpoint, and that comparison does not separate initialization from supervision. Teacher robustness may contribute to its ranking; we do not claim otherwise. Feature scatter is consequently a descriptive candidate diagnostic, not a proven causal selection rule. The zero-accuracy measurements above establish that a robust teacher is not required, not that initialization is irrelevant.

2. Is the clean gain due to smaller radii on sensitive samples?

Smaller radii on highly sensitive samples are intentional: a given input perturbation can cause a larger change in the anchoring objective for such samples. The allocation rule attempts to account for that difference, while compensating with larger radii on less sensitive samples to preserve the batch mean radius. This is a redistribution of perturbation radii. Mean-radius preservation does not imply equal adversarial loss or optimization difficulty, and protecting sensitive samples may indeed contribute to the clean gain.

The empirical question is whether this allocation improves clean accuracy while retaining robustness at a common evaluation radius. In the three-seed, 150-epoch CIFAR-100 comparison, feature matching changes from 62.63 clean / 28.01 reported robust accuracy with uniform radii to 64.82 / 28.04 with sensitivity matching. Logit matching changes from 62.29 / 27.67 to 64.44 / 27.81. All are evaluated at 8/255. The principal measured effect is a clean improvement with approximately unchanged robust accuracy, consistent with the paper's trade-off objective. The feature-target advantage itself is modest and dataset-dependent.

The existing allocation control tests whether heterogeneous radii alone suffice. At 100 epochs and mean training radius 8.8/255, sensitivity matching gives 62.17 / 28.86, whereas assigning the same sensitivity-derived radius values by clean-CE difficulty rank gives 60.90 / 28.06. This reassignment preserves the radius multiset within each batch for the current model, while changing its association with samples. It supports the assignment rule beyond radius dispersion alone. This single-seed control does not isolate the causal mechanism in the final recipe or rule out protection of sensitive samples as an explanation.

3. Training duration and Figure 1

The reviewer is correct that the CFA radius-sweep points use 100 student epochs and the stars use 150. Longer training can contribute to the difference. The grey points are published baseline results under their respective protocols, not uniformly 100-epoch runs. Figure 1 is an overview of reported operating points, not a controlled experiment isolating method from training duration.

Our additional comparisons provide more direct evidence for selected close alternatives:

ComparisonStudent epochs
CFA feature/logit x uniform/sensitivity, three seeds per cell150
Strengthened BGAT and LBGAT controls with WA/AWP150
Strengthened ADR with WA/AWP200

The target/allocation effects above are measured at the same 150 epochs; they cannot be explained by giving those CFA variants more epochs than their controls. The ADR comparison also does not give CFA more student epochs. This does not exclude improvements from longer or differently tuned training for other baselines. We will distinguish these controlled comparisons from Figure 1's cross-protocol overview. CFA's 200-epoch natural pretraining is additional work and must be counted separately from student training.

4. Code

The code is already included in the supplementary materials accompanying our submission. Please refer to the submitted code supplement. We will make its location more explicit in the manuscript's reproducibility statement.

Thank you for your response. First of all, I cannot find the code supplement; it seems that the ZIP file is missing. Could you check whether it was properly uploaded? Alternatively, you could provide the code through an anonymous GitHub repository.

Regarding Q1, my original question was about comparing the robustness of different naturally trained teachers, such as those trained with mixup, label smoothing, and standard training. Since their robust accuracy may be close to zero under strong attacks, it might be helpful to consider other robustness measures or smaller perturbation budgets. Could you provide a robustness comparison among these natural teachers? ( As the discussion period is almost over, I understand if it is difficult to conduct additional experiments in time. In that case, simply answering the remaining questions would be sufficient.)

Additionally, does the standard teacher refer to the mixup-trained teacher used in the main experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.