- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies catastrophic overfitting (CO) in fast adversarial training and proposes DC-FGSM, which uses class-adaptive perturbation priors and gradient correction. Experiments show robustness gains, but novelty, causal support, and numerical consistency remain insufficient.
Critical strengths
- Addresses a practical limitation of efficient adversarial training.
- Favorable cross-architecture results and improvements when added to existing FAT methods.
- Evaluation covers multiple datasets, backbones, attacks, and checkpoints.
Critical weaknesses
- Insufficient novelty of the mechanistic interpretation. The trigger-based explanation substantially overlaps with LIET (ICCV 2025), which already studies transferable class-label information after CO to explain the effectiveness of initialization. Gradient drift also needs clearer differentiation from GradAlign and prior local-nonlinearity analyses.
- Figures 1, 2, and 6, along with the entropy and drift analyses, largely demonstrate phenomena that accompany CO rather than establish its causes. Hence, the causal claims should be reduced to a mechanistic hypothesis.
- Incomplete ablations. For instance, the PGD-2 experiment changes both initialization and the attack update, so it cannot isolate the effect of drift correction. DC-FGSM also uses JSD regularization, masking/random replacement, and historical momentum, but Figures 7–8 do not fully separate these factors.
- Verifiable numerical inconsistencies and missing reproduction details. For CIFAR-100 at = 8/255 (best checkpoint), DC-FGSM reports 27.27% AA accuracy in Table 1 but 27.09% in Table 5. In Table 8, the eight listed accuracies for FGSM-MEP average to about 32.07%, not the reported 29.5%; the reported Avg. standard deviation of 8.89 for FGSM-RS also conflicts with the mean of its eight component standard deviations (about 1.15). Reproduction also requires missing details such as the JSD coefficient, policy temperature/update rate, buffer capacity, policy update frequency, and attack step sizes/restarts.
Questions for authors
The paper addresses an important problem and shows promising robustness gains, but its novelty overlaps with prior work, its causal evidence is insufficient, and its ablations, baselines, and numerical consistency are inadequate
13 comments
(2) Temporal and intervention evidence.
Tables I and II use FGSM-AT branches from the same epoch-5 checkpoint, with interventions beginning at epoch 6(TNT). On 512 fixed examples, we measure
PGD denotes PGD-10, and inner denotes the training-time attack. A larger Gap indicates a weaker inner attack; a value near zero indicates comparable strength. We also report gradient drift, PGD-10 accuracy, and relative to a fixed reference. Epochs are zero-indexed, and “Final” is the last checkpoint.
Table I. Controlled Interventions on CIFAR-10 at
| Method | Metric | Epoch 5 | Epoch 6(TNT) | Epoch 7 | Epoch 10 | Epoch 20 | Epoch 22 | Epoch 40 | Epoch 80 | Final |
|---|---|---|---|---|---|---|---|---|---|---|
| Baseline (CO at Epoch 7) | HTAR (×) | 1.75 | 1.93 | 0.94 | 8.55 | 8.19 | 5.23 | 4.49 | 2.29 | 2.24 |
| Gradient Drift (%) | 23.97 | 40.59 | 76.30 | 92.71 | 116.21 | 112.15 | 110.15 | 107.04 | 105.53 | |
| Inner-Max Gap | 0.111 | 0.183 | 2.123 | 17.842 | 61.357 | 56.619 | 69.836 | 85.547 | 82.574 | |
| Robust Acc. (%) | 23.96 | 21.79 | 4.01 | 0.04 | 0.00 | 0.00 | 0.02 | 0.11 | 0.09 | |
| Random-Noise Intervention (CO at Epoch 22) | HTAR (×) | 1.75 | 0.85 | 0.84 | 1.11 | 1.18 | 5.97 | 14.54 | 7.73 | 5.79 |
| Gradient Drift (%) | 23.97 | 9.68 | 8.93 | 11.02 | 21.10 | 98.79 | 102.94 | 104.78 | 105.50 | |
| Inner-Max Gap | 0.111 | 0.098 | 0.112 | 0.165 | 0.320 | 13.127 | 49.754 | 76.679 | 73.313 | |
| Robust Acc. (%) | 23.96 | 24.05 | 24.26 | 26.18 | 28.24 | 0.00 | 0.00 | 0.01 | 0.01 | |
| Debiasing Intervention | HTAR (×) | 1.75 | 0.99 | 0.92 | 1.15 | 1.27 | 1.20 | 1.38 | 1.72 | 1.74 |
| Gradient Drift (%) | 23.97 | 10.22 | 9.28 | 11.63 | 15.18 | 16.37 | 20.59 | 29.17 | 31.74 | |
| Inner-Max Gap | 0.111 | 0.156 | 0.182 | 0.288 | 0.489 | 0.666 | 1.116 | 2.936 | 3.610 | |
| Robust Acc. (%) | 23.96 | 21.09 | 21.51 | 23.66 | 25.65 | 25.71 | 26.62 | 21.18 | 19.89 | |
| Debiasing + Correction | HTAR (×) | 1.75 | 1.00 | 0.95 | 1.11 | 1.24 | 1.24 | 1.28 | 1.60 | 1.62 |
| Gradient Drift (%) | 23.97 | 9.60 | 8.98 | 10.77 | 13.95 | 15.23 | 18.10 | 24.97 | 27.22 | |
| Inner-Max Gap | 0.111 | 0.135 | 0.164 | 0.248 | 0.402 | 0.537 | 0.842 | 2.128 | 2.599 | |
| Robust Acc. (%) | 23.96 | 21.63 | 22.45 | 25.16 | 27.42 | 27.26 | 28.53 | 24.13 | 22.90 |
Table I shows that the baseline and random-noise branches collapse at epochs 7 and 22, respectively; random noise therefore only delays the same failure process. Debiasing retains 19.89% final PGD-10 accuracy, while gradient correction raises it to 22.90% and reduces both drift and the Inner-Max Gap. These results support both interventions from a shared pre-CO state.
Table II. Batch-Level Dynamics Around Catastrophic Overfitting on CIFAR-10 at
| Epoch / Stage | Metric | B0–47 | B48–96 | B97–145 | B146–194 | B195–243 | B244–292 | B293–341 | B342–390 |
|---|---|---|---|---|---|---|---|---|---|
| Epoch 6 (Pre-CO) | HTAR (×) | 1.19 | 0.99 | 1.34 | 1.11 | 0.97 | 0.95 | 1.31 | 1.20 |
| Gradient Drift (%) | 11.65 | 12.31 | 11.28 | 14.45 | 15.60 | 12.71 | 11.27 | 17.16 | |
| Inner-Max Gap | 0.089 | 0.103 | 0.126 | 0.069 | 0.068 | 0.057 | 0.108 | 0.132 | |
| Robust Acc. (%) | 30.42 | 26.21 | 28.84 | 30.93 | 28.52 | 27.90 | 28.11 | 28.78 | |
| Epoch 7 (CO Transition) | HTAR (×) | 1.29 | 0.96 | 1.89 | 1.71 | 1.69 | 7.25 | 4.98 | 7.82 |
| Gradient Drift (%) | 16.38 | 20.65 | 27.22 | 28.73 | 38.85 | 58.76 | 79.48 | 97.28 | |
| Inner-Max Gap | 0.120 | 0.097 | 0.182 | 0.166 | 0.284 | 1.294 | 2.081 | 3.565 | |
| Robust Acc. (%) | 27.36 | 28.76 | 26.18 | 25.65 | 23.49 | 21.01 | 5.69 | 2.39 | |
| Epoch 8 (Post-CO) | HTAR (×) | 5.75 | 2.87 | 4.15 | 6.08 | 6.49 | 10.10 | 14.35 | 17.75 |
| Gradient Drift (%) | 99.93 | 103.67 | 103.42 | 105.11 | 105.88 | 109.82 | 110.67 | 111.79 | |
| Inner-Max Gap | 6.441 | 2.683 | 6.006 | 11.471 | 12.661 | 14.772 | 23.416 | 20.279 | |
| Robust Acc. (%) | 0.67 | 0.18 | 0.24 | 0.32 | 0.29 | 0.18 | 0.16 | 0.30 |
Table II resolves the transition at the batch level: drift accumulates throughout epoch 7, the Inner-Max Gap and HTAR rise sharply in B244–292, and robust accuracy collapses in the next window. This ordering places drift accumulation before robustness collapse and trigger amplification.
(3) Constrained comparison with LIET.
We compare LIET, D-FGSM, and DC-FGSM on CIFAR-10 at , with all initialization noise restricted to and all final perturbations projected onto the standard ball. D-FGSM uses debiasing initialization , and DC-FGSM additionally applies gradient correction. This experiment evaluates the methods under the specified training constraint.
In Table III, the mean CO onset shifts from epoch 30 for LIET to epoch 79 for D-FGSM across the three reported seeds, a delay of 49 epochs. D-FGSM still undergoes CO in these runs. Its role is therefore to extend the stable training period, while the final-checkpoint results in Table IV evaluate the additional benefit of gradient correction.
Table III. CO Dynamics of LIET and D-FGSM under Standard Projection and -Bounded Initialization Noise on CIFAR-10 at
| Method | Epoch | FGSM Acc. (%) | PGD Acc. (%) | Gradient Drift |
|---|---|---|---|---|
| LIET | ||||
| CO = 29.67 epochs | Seed 0/1/2: 31/28/30 | — | — | — |
| 10 | 24.93 | 25.83 | 7.96 | |
| 20 | 30.37 | 30.20 | 11.28 | |
| 40 | 73.67 | 0.97 | 102.88 | |
| 80 | 83.80 | 2.23 | 101.97 | |
| Final | 89.27 | 17.03 | 101.78 | |
| D-FGSM | ||||
| CO = 78.67 epochs | Seed 0/1/2: 78/85/73 | — | — | — |
| 10 | 22.80 | 23.60 | 7.72 | |
| 20 | 27.60 | 28.30 | 10.62 | |
| 40 | 32.70 | 32.30 | 13.87 | |
| 80 | 58.30 | 2.10 | 97.80 | |
| Final | 86.20 | 13.20 | 100.48 |
Table IV reports AA accuracy of 16.66%, 21.75%, and 23.46% at the best checkpoint for LIET, D-FGSM, and DC-FGSM, respectively. At the final checkpoint, AA accuracy is 0.00% for LIET, 0.05% for D-FGSM, and 19.41% for DC-FGSM. Final PGD-50 accuracy is 5.24%, 4.60%, and 29.09%, respectively. These results show that debiasing improves the best-checkpoint robustness in this setting, while gradient correction substantially improves the robustness retained at the end of training.
Because these comparisons use the same perturbation bound and projection, they establish that the reported gains can be obtained within the prescribed constraint. Figures 3 and 7 provide the more specific evidence concerning class-dependent spatial structure and transformation choice.
Table IV. Best- and Final-Checkpoint Performance under Standard Projection and -Bounded Initialization Noise on CIFAR-10 at
| Checkpoint | Method | Clean | PGD-50 | CW | AA | CR | PMA | Avg. |
|---|---|---|---|---|---|---|---|---|
| Best | LIET | 62.31 | 24.41 | 20.14 | 16.66 | 16.66 | 17.27 | 26.24 |
| D-FGSM (Ours) | 71.91 | 32.62 | 26.77 | 21.75 | 21.97 | 23.10 | 33.02 | |
| DC-FGSM (Ours) | 72.54 | 34.00 | 27.46 | 23.46 | 23.49 | 24.60 | 34.26 | |
| Final | LIET | 44.68 | 5.24 | 3.15 | 0.00 | 12.97 | 7.27 | 12.22 |
| D-FGSM (Ours) | 82.17 | 4.60 | 3.43 | 0.05 | 20.31 | 11.11 | 20.28 | |
| DC-FGSM (Ours) | 77.51 | 29.09 | 24.61 | 19.41 | 19.74 | 21.33 | 31.95 |
Avg. is the arithmetic mean of Clean and the five attack-specific accuracy columns. Accuracy values are percentages.
We thank the reviewer for the constructive feedback. We clarify the relationship to prior work, present the additional controls, and detail the numerical corrections and implementation settings.
Weakness 1 — Novelty of the Mechanistic Interpretation and Method
(1) Relationship to LIET.
LIET demonstrates the transferability of label information in perturbations after CO and uses class-specific initialization derived from gray images, together with nonuniform label smoothing, to mitigate this effect. Our analysis builds on this observation by investigating the emergence and reinforcement of the class-dependent signal during training. LIET
Our contribution has three connected aspects. First, our trigger-based interpretation identifies persistent, spatially distributed class-dependent patterns (Figures 1 and 3) and investigates how they become reinforced during training. Second, we examine how changes in the gradient along the attack trajectory weaken inner maximization and how repeated outer updates can reinforce class-biased perturbations. Third, we use class-adaptive spatial transformations and trajectory gradient correction to address these two parts of the proposed feedback process. Figure 3 and the ablations in Figure 7 compare the adaptive transformations with fixed initialization and fixed spatial transformations.
Robustness under standard perturbation constraints. Our method restricts initialization noise to [−ϵ,ϵ] and projects the final perturbation onto the standard L∞ ball. We train without early stopping and evaluate stability through the final checkpoint. As shown in Table III, debiased initialization substantially delays CO under these constraints, while Table IV shows that gradient correction further improves final-checkpoint robust accuracy. The full training objective also includes JSD consistency regularization, whose contribution is quantified separately in Tables VII and IX.
N-FGSM uses stronger noise together with an unclipped training perturbation construction. LIET's supplementary implementation description also specifies omission of perturbation clipping at larger budgets. These training choices motivate the constrained comparison below. N-FGSM, LIET supplementary material, Section B