- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MAGE, a target-supervised adaptation method for improving the adversarial robustness of frozen vision-language models. Instead of using a fixed confidence-based energy, MAGE learns a small scalar energy head on normalized CLIP features. The head is trained using energy ranking, classification recovery, margin improvement, and clean-stability losses. At test time, the method performs five bounded image updates using the learned energy gradient, without access to test labels. Experiments cover several robust CLIP backbones and target datasets. The reported results show consistent improvements over the original ET3 setup and reach 37.0% average PGD accuracy in the nine-dataset comparison.
Critical strengths
1. The method has a clear motivation and a small trainable component.
A fixed confidence energy can reinforce an incorrect prediction. Learning an energy through the outcome of a recovery step is a sensible way to address this problem. The head has only about 70k parameters for ViT-B/32, while the CLIP encoders remain frozen.
2. The main reported improvements are meaningful.
Table 2 reports improvements over original ET3 in all 16 backbone–dataset settings, with clean accuracy improving over the base in 15 of them. The averages in Table 3 also support the headline results: approximately 37.0% robust accuracy and a 7.6-point clean improvement over the robust base.
3. The paper includes useful controls rather than relying only on the main comparison.
Table B.2 compares the methods under the same deployment normalizer. MAGE still improves over ET3 under both L2 and Mean-Absolute normalization. This supports a benefit from the learned energy beyond the optimizer change alone.
4. The supervision and scope are disclosed clearly.
The authors explicitly state that MAGE uses target-dataset labels, trains separate heads, and has not demonstrated cross-dataset transfer. The appendix also provides three-seed results for several experiments and discusses unsuccessful recovery trajectories.
Critical weaknesses
1. The strong-attack evidence does not yet cover the default deployed method well enough.
The main experiments use adaptive PGD-20 through the complete defense, which is an important positive. However, the APGD-CE, APGD-DLR, and CW experiments in Appendix B.5 attack only the frozen base model and apply MAGE afterwards. They therefore do not establish robustness to those attacks when the attacker knows the defense. Table B.8 increases adaptive PGD to 100 steps, but uses a separate diagnostic configuration, with 36.2% accuracy at 20 steps rather than the default 55.5% in Table 1. The authors acknowledge this difference, so it is not a contradiction. Still, this check cannot establish attack convergence for the main configuration. Stronger attacks should be run on representative default checkpoints, with multiple starts, different losses, and per-example worst-case results.
2. The clean accuracies of the frozen base models need to be reconciled with the stated protocol.
Table A.3 says that CIFAR-10 and CIFAR-100 use their full test sets. The preprocessing and text prototypes are also described as fixed. Yet Table B.3 reports undefended OpenAI CLIP clean accuracies of 85.1, 85.6, and 85.0 across seeds. Table B.7 reports FARE4 CIFAR-100 clean accuracies of 47.5, 49.9, and 51.8. With the same frozen checkpoint, fixed prototypes, and full fixed test set, changing the energy-head seed should not change the undefended clean prediction. Different checkpoints, subsets, or preprocessing could explain this, but those differences need to be documented. This matters for both reproducibility and the interpretation of paired gains.
3. The comparisons do not fully separate the energy objective from supervision and update rules.
Appendix B.2 shows that changing ET3’s deployment normalization from L2 to Mean-Absolute raises its robust accuracy from 12.9% to 46.6%. Under the same Mean-Absolute rule, MAGE reaches 55.1%, leaving an 8.5-point advantage. This is still useful, but it shows that the normalizer accounts for a substantial part of the original gap. The authors acknowledge this on page 19, although Appendix C.4 describes the controlled comparison as differing only in the energy function and learned head. Same-normalizer controls on a representative robust backbone would help clarify Table 2.
There is also a supervision gap: MAGE uses the full labeled target training set, while ET3 uses no target labels and some prompt baselines use few-shot labels. This is disclosed, but a comparable small adapter or prompt trained with the same target supervision is still needed to establish the benefit of this particular learning approach.
4. The independent contribution of margin alignment and its relation to earlier learned-energy purification remain unclear. In Table 1, adding the margin term to ranking plus CE increases robust accuracy from 54.1% to 55.1%. The corresponding seed-level gains in Table B.3 are +2.3, −0.5, and +1.1 points. This suggests a possible benefit, but not a consistent improvement across all seeds. A matched ablation on a robust backbone would better connect the paper’s central margin-alignment claim to its main results. I strongly suggest that the paper should also discuss the earlier work AEFOP: Adversarial Energy Field Optimization for Adversarial Example Purification. Sections 2.3–2.4 of that work already learn a scalar energy field and train through recovery steps using classification and semantic-preservation objectives with a fixed classifier. MAGE has differences, including its small CLIP-based head and explicit margin objective, but the contribution beyond this closely related framework should be explained.
Questions for authors
1. Do the main gains remain under stronger attacks against the complete default MAGE pipeline?
Please evaluate representative default checkpoints using multiple restarts and losses such as CE and DLR, retaining the worst result per example. Please also clarify whether attack gradients include the normalization statistic and all dependencies through the five recovery steps.
2. Why does undefended clean accuracy change across seeds?
Please explain the CIFAR-10 values in Table B.3 and the FARE4 CIFAR-100 values in Table B.7. Were different checkpoints, prototypes, preprocessing, or test subsets used? Checkpoint identifiers and actual evaluation counts would help resolve this.
3. How much of the robust-backbone gain remains under matched update rules?
Table B.2 is helpful. Could the same-normalizer comparison be added for one representative setting from Table 2, with reasonable step-size choices for ET3?
4. How does MAGE compare with a small adaptation baseline using the same labeled target data?
One representative same-data comparison would help distinguish the benefit of the recovery-energy objective from the benefit of additional target supervision.
5. What is the independent benefit of the margin term?
Could the authors compare CE-based recovery with and without the margin term on a robust backbone, keeping ranking, stability, normalization, and training budget fixed, and report paired seed-level differences?
6. How does MAGE differ from earlier learned-energy recovery methods such as AEFOP?
Please clarify which parts of the learning principle are new and which benefits specifically follow from the CLIP feature head and margin objective. This is my major concern about the study.
6 comments
W2:Base Clean Accuracy Across Seeds
Thank you for identifying the discrepancy between Table A.3 and the seed-level results. With the checkpoint, text prototypes, preprocessing, and test examples fixed, changing the energy head’s training seed should leave the undefended Base predictions unchanged. The reported values are:
| Setting | Seed 1 | Seed 2 | Seed 3 |
|---|---|---|---|
| Table B.3: OpenAI CLIP, CIFAR-10 | 85.1% | 85.6% | 85.0% |
| Table B.7: FARE4, CIFAR-100 | 47.5% | 49.9% | 51.8% |
The relevant experiments used a randomly sampled subset of 5,000 test images for each seed, but this evaluation scope was not accurately specified in the original manuscript. Consequently, the variation in undefended clean accuracy across seeds partly reflects differences in the sampled test subsets.
W3:What Persists Under Matched Normalization and Supervision?
Thank you for asking how much of the gain survives when the comparison controls for the deployment normalizer and target labels. We acknowledge the size of the normalization effect: in Table B.2, ET3 rises from 12.9% to 46.6% when switched to MeanAbs. MAGE reaches 55.1% with the same normalizer, leaving an 8.5-point advantage in that setting.
Under matched MeanAbs deployment, we also evaluated transfer accuracy on the same perturbed inputs for each method:
| Dataset | ET3 transfer accuracy (%) | MAGE transfer accuracy (%) | MAGE gain (pp) |
|---|---|---|---|
| DTD | 14.88 | 21.90 | +7.02 |
| Caltech101 | 47.73 | 55.48 | +7.75 |
| CIFAR-100 | 17.43 | 20.31 | +2.88 |
MAGE’s transfer gain persists across all three datasets. The amount that persists in the Table 2 adaptive setting is a separate measurement, since it depends on the representative robust checkpoint, the full evaluation protocol, and the ET3 step size.
In a separate ViT-B/16 experiment, we gave MAGE and both linear heads the same 1,000 labeled target examples and five training epochs:
| Dataset | Supervised linear head: clean / transfer (%) | Adversarially trained linear head: clean / transfer (%) | MAGE: clean / transfer (%) |
|---|---|---|---|
| DTD | 72.43 / 0.90 | 48.69 / 6.86 | 34.75 / 21.90 |
| Caltech101 | 95.38 / 0.64 | 79.21 / 12.03 | 73.84 / 55.65 |
| CIFAR-100 | 68.48 / 0.00 | 26.82 / 0.47 | 27.50 / 17.81 |
Neither tested classification head reproduces MAGE’s transfer performance at the same supervision budget. We believe this is direct evidence that how the labels guide recovery matters: fitting a target classifier and learning a direction for input recovery lead to different behavior on perturbed inputs.
W1:Adaptive Evaluation of the Default Deployment
Thank you for recognizing the adaptive evaluation of the complete deployment pipeline and for distinguishing it from the appendix diagnostics. The main experiment evaluates the default deployment under adaptive PGD-20 at a radius of 2/255. MAGE outperforms original ET3 in all 16 backbone and dataset combinations under that protocol.
B.8 shows that MAGE accuracy stays within 36.2%–37.2% as the attack runs from 10 to 100 steps in that configuration. Your question about the default checkpoints is more specific: the current results do not provide a per-example worst-case comparison across losses and restarts for those checkpoints.
The main experiment is reported as adaptive to the five-step deployment. Confirming which dependencies contribute to the attack gradient, including the MeanAbs statistic, requires checking the actual computation graph.