acceptodds
Under review as a conference paper at ICLR 2027

MAGE: Margin-Aligned Gradient Energy for Target-Supervised Robust Adaptation of Vision-Language Models

Abstract

Vision-language models are vulnerable to adversarial perturbations, and fixed confidence-based energy objectives need not guide an attacked image toward its correct class. We introduce MAGE (Margin Aligned Gradient Energy), a target-supervised, parameter-efficient robust adaptation method that learns a scalar energy head on frozen vision-language features. For each target dataset, MAGE independently trains the head using its training split, class names, and labels, shaping the negative input-energy gradient to restore the ground-truth margin while preserving clean predictions. At deployment, the adapted head performs a few bounded test-time image updates without test-label access or persistent parameter updates. Under adaptive PGD-20 that differentiates through the complete deployment pipeline, MAGE improves over fixed ET3 in all evaluated backbone-dataset combinations and achieves 37.0% average robust accuracy in the nine-dataset comparison, while also improving average clean accuracy by 7.6 points. These results show that target-supervised energy adaptation is most useful when its test-time gradient is aligned with semantic recovery rather than confidence alone.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Models, Adversarial Robustness, Test-Time Defense

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aKF4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MAGE, a target-supervised adaptation method for improving the adversarial robustness of frozen vision-language models. Instead of using a fixed confidence-based energy, MAGE learns a small scalar energy head on normalized CLIP features. The head is trained using energy ranking, classification recovery, margin improvement, and clean-stability losses. At test time, the method performs five bounded image updates using the learned energy gradient, without access to test labels. Experiments cover several robust CLIP backbones and target datasets. The reported results show consistent improvements over the original ET3 setup and reach 37.0% average PGD accuracy in the nine-dataset comparison.

Critical strengths

1. The method has a clear motivation and a small trainable component.

A fixed confidence energy can reinforce an incorrect prediction. Learning an energy through the outcome of a recovery step is a sensible way to address this problem. The head has only about 70k parameters for ViT-B/32, while the CLIP encoders remain frozen.

2. The main reported improvements are meaningful.

Table 2 reports improvements over original ET3 in all 16 backbone–dataset settings, with clean accuracy improving over the base in 15 of them. The averages in Table 3 also support the headline results: approximately 37.0% robust accuracy and a 7.6-point clean improvement over the robust base.

3. The paper includes useful controls rather than relying only on the main comparison.

Table B.2 compares the methods under the same deployment normalizer. MAGE still improves over ET3 under both L2 and Mean-Absolute normalization. This supports a benefit from the learned energy beyond the optimizer change alone.

4. The supervision and scope are disclosed clearly.

The authors explicitly state that MAGE uses target-dataset labels, trains separate heads, and has not demonstrated cross-dataset transfer. The appendix also provides three-seed results for several experiments and discusses unsuccessful recovery trajectories.

Critical weaknesses

1. The strong-attack evidence does not yet cover the default deployed method well enough.

The main experiments use adaptive PGD-20 through the complete defense, which is an important positive. However, the APGD-CE, APGD-DLR, and CW experiments in Appendix B.5 attack only the frozen base model and apply MAGE afterwards. They therefore do not establish robustness to those attacks when the attacker knows the defense. Table B.8 increases adaptive PGD to 100 steps, but uses a separate diagnostic configuration, with 36.2% accuracy at 20 steps rather than the default 55.5% in Table 1. The authors acknowledge this difference, so it is not a contradiction. Still, this check cannot establish attack convergence for the main configuration. Stronger attacks should be run on representative default checkpoints, with multiple starts, different losses, and per-example worst-case results.

2. The clean accuracies of the frozen base models need to be reconciled with the stated protocol.

Table A.3 says that CIFAR-10 and CIFAR-100 use their full test sets. The preprocessing and text prototypes are also described as fixed. Yet Table B.3 reports undefended OpenAI CLIP clean accuracies of 85.1, 85.6, and 85.0 across seeds. Table B.7 reports FARE4 CIFAR-100 clean accuracies of 47.5, 49.9, and 51.8. With the same frozen checkpoint, fixed prototypes, and full fixed test set, changing the energy-head seed should not change the undefended clean prediction. Different checkpoints, subsets, or preprocessing could explain this, but those differences need to be documented. This matters for both reproducibility and the interpretation of paired gains.

3. The comparisons do not fully separate the energy objective from supervision and update rules.

Appendix B.2 shows that changing ET3’s deployment normalization from L2 to Mean-Absolute raises its robust accuracy from 12.9% to 46.6%. Under the same Mean-Absolute rule, MAGE reaches 55.1%, leaving an 8.5-point advantage. This is still useful, but it shows that the normalizer accounts for a substantial part of the original gap. The authors acknowledge this on page 19, although Appendix C.4 describes the controlled comparison as differing only in the energy function and learned head. Same-normalizer controls on a representative robust backbone would help clarify Table 2.

There is also a supervision gap: MAGE uses the full labeled target training set, while ET3 uses no target labels and some prompt baselines use few-shot labels. This is disclosed, but a comparable small adapter or prompt trained with the same target supervision is still needed to establish the benefit of this particular learning approach.

4. The independent contribution of margin alignment and its relation to earlier learned-energy purification remain unclear. In Table 1, adding the margin term to ranking plus CE increases robust accuracy from 54.1% to 55.1%. The corresponding seed-level gains in Table B.3 are +2.3, −0.5, and +1.1 points. This suggests a possible benefit, but not a consistent improvement across all seeds. A matched ablation on a robust backbone would better connect the paper’s central margin-alignment claim to its main results. I strongly suggest that the paper should also discuss the earlier work AEFOP: Adversarial Energy Field Optimization for Adversarial Example Purification. Sections 2.3–2.4 of that work already learn a scalar energy field and train through recovery steps using classification and semantic-preservation objectives with a fixed classifier. MAGE has differences, including its small CLIP-based head and explicit margin objective, but the contribution beyond this closely related framework should be explained.

Questions for authors

1. Do the main gains remain under stronger attacks against the complete default MAGE pipeline?

Please evaluate representative default checkpoints using multiple restarts and losses such as CE and DLR, retaining the worst result per example. Please also clarify whether attack gradients include the normalization statistic and all dependencies through the five recovery steps.

2. Why does undefended clean accuracy change across seeds?

Please explain the CIFAR-10 values in Table B.3 and the FARE4 CIFAR-100 values in Table B.7. Were different checkpoints, prototypes, preprocessing, or test subsets used? Checkpoint identifiers and actual evaluation counts would help resolve this.

3. How much of the robust-backbone gain remains under matched update rules?

Table B.2 is helpful. Could the same-normalizer comparison be added for one representative setting from Table 2, with reasonable step-size choices for ET3?

4. How does MAGE compare with a small adaptation baseline using the same labeled target data?

One representative same-data comparison would help distinguish the benefit of the recovery-energy objective from the benefit of additional target supervision.

5. What is the independent benefit of the margin term?

Could the authors compare CE-based recovery with and without the margin term on a robust backbone, keeping ranking, stability, normalization, and training budget fixed, and report paired seed-level differences?

6. How does MAGE differ from earlier learned-energy recovery methods such as AEFOP?

Please clarify which parts of the learning principle are new and which benefits specifically follow from the CLIP feature head and margin objective. This is my major concern about the study.

6 comments
AuthorsOpenReview ↗

W1:Adaptive Evaluation of the Default Deployment

Thank you for recognizing the adaptive evaluation of the complete deployment pipeline and for distinguishing it from the appendix diagnostics. The main experiment evaluates the default deployment under adaptive PGD-20 at a radius of 2/255. MAGE outperforms original ET3 in all 16 backbone and dataset combinations under that protocol.

EvaluationProtocolResult
Table 2Adaptive PGD-20 against the default deployment, radius 2/255MAGE leads original ET3 in all 16 settings
Appendix B.5Perturbations generated against frozen Base, followed by MAGEMultiple transfer attacks across five datasets
Appendix B.8Adaptive PGD with 10, 20, 50, and 100 steps in a separate diagnostic configurationMAGE: 37.2%, 36.2%, 36.8%, and 37.1%

B.8 shows that MAGE accuracy stays within 36.2%–37.2% as the attack runs from 10 to 100 steps in that configuration. Your question about the default checkpoints is more specific: the current results do not provide a per-example worst-case comparison across losses and restarts for those checkpoints.

The main experiment is reported as adaptive to the five-step deployment. Confirming which dependencies contribute to the attack gradient, including the MeanAbs statistic, requires checking the actual computation graph.

AuthorsOpenReview ↗

W2:Base Clean Accuracy Across Seeds

Thank you for identifying the discrepancy between Table A.3 and the seed-level results. With the checkpoint, text prototypes, preprocessing, and test examples fixed, changing the energy head’s training seed should leave the undefended Base predictions unchanged. The reported values are:

SettingSeed 1Seed 2Seed 3
Table B.3: OpenAI CLIP, CIFAR-1085.1%85.6%85.0%
Table B.7: FARE4, CIFAR-10047.5%49.9%51.8%

The relevant experiments used a randomly sampled subset of 5,000 test images for each seed, but this evaluation scope was not accurately specified in the original manuscript. Consequently, the variation in undefended clean accuracy across seeds partly reflects differences in the sampled test subsets.

AuthorsOpenReview ↗

W3:What Persists Under Matched Normalization and Supervision?

Thank you for asking how much of the gain survives when the comparison controls for the deployment normalizer and target labels. We acknowledge the size of the normalization effect: in Table B.2, ET3 rises from 12.9% to 46.6% when switched to MeanAbs. MAGE reaches 55.1% with the same normalizer, leaving an 8.5-point advantage in that setting.

Under matched MeanAbs deployment, we also evaluated transfer accuracy on the same perturbed inputs for each method:

DatasetET3 transfer accuracy (%)MAGE transfer accuracy (%)MAGE gain (pp)
DTD14.8821.90+7.02
Caltech10147.7355.48+7.75
CIFAR-10017.4320.31+2.88

MAGE’s transfer gain persists across all three datasets. The amount that persists in the Table 2 adaptive setting is a separate measurement, since it depends on the representative robust checkpoint, the full evaluation protocol, and the ET3 step size.

In a separate ViT-B/16 experiment, we gave MAGE and both linear heads the same 1,000 labeled target examples and five training epochs:

DatasetSupervised linear head: clean / transfer (%)Adversarially trained linear head: clean / transfer (%)MAGE: clean / transfer (%)
DTD72.43 / 0.9048.69 / 6.8634.75 / 21.90
Caltech10195.38 / 0.6479.21 / 12.0373.84 / 55.65
CIFAR-10068.48 / 0.0026.82 / 0.4727.50 / 17.81

Neither tested classification head reproduces MAGE’s transfer performance at the same supervision budget. We believe this is direct evidence that how the labels guide recovery matters: fitting a target classifier and learning a direction for input recovery lead to different behavior on perturbed inputs.

Reviewer_9JgCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates robust adaptation method for vision-language models. The method introduces a lightweight trainable energy head on top of a frozen visual encoder and optimizes it using energy ranking, classification restoration, margin improvement, and clean-stability objectives. Experiments show that MAGE improves adversarial robustness over compared methods.

Critical strengths
  1. Improving adversarial robustness in vision-language models is important for their reliable deployment. Exploring parameter-efficient robustness adaptation is practically relevant. 2.The proposed method achieves improvements over compared methods.
Critical weaknesses
  1. The abstract does not sufficiently highlight the importance of the research problem or the key challenges addressed by the proposed method.
  2. In the first paragraph of the Introduction, the authors claim that small adversarial perturbations can significantly affect VLM predictions. More relevant references should be provided to support this claim.
  3. The presentation should be improved. For example, page 2 contains excessive blank space, which affects the overall layout.
  4. The authors argue that LogSumExp energy may fail to provide effective guidance for adversarial recovery, motivating an energy function with restorative gradients. However, the connection between this limitation and the proposed solution is not sufficiently explained.
  5. The authors state that an energy function should have a “restorative” negative input gradient. However, the meaning of “restorative” is not clearly defined, and its relationship to adversarial robustness requires further explanation.
  6. Figures 2 and 3 should be explicitly referenced and discussed in the main text to improve readability.
  7. The proposed method aims to improve adversarial robustness while preserving clean-input stability. However, it remains unclear how these potentially competing objectives are balanced. The authors should investigate the sensitivity to their corresponding loss weights.
  8. The main experiments primarily consider adaptive PGD-20 with an (\ell_\infty) perturbation budget of (2/255). Evaluations under additional attack methods and perturbation budgets are necessary to establish the generality of the proposed robustness improvements.
Questions for authors

Please refer to Critical Weaknesses.

9 comments
AuthorsOpenReview ↗

W1–W6:Why Confidence Alone Does Not Guide Recovery

Thank you for asking us to connect the motivation more directly to the method. We also appreciate your comments on the supporting references, page layout, and placement of Figures 2 and 3. The abstract and introduction emphasize the central challenge, the introduction adds supporting references, and the main text explicitly discusses both figures.

The problem is that a model can become more confident in an incorrect prediction. If an adversarial input already makes the wrong class the highest scoring class, minimizing a fixed LogSumExp energy can reinforce that prediction. MAGE learns an energy whose descent direction is rewarded for improving the true class’s position relative to its strongest competitor. The target labels provide this signal during training; the learned update uses no test labels.

We measure recovery using the change in the true-class margin: the true-class score minus the highest competing-class score. The following experiment measures the average gain after one and five updates and reports final transfer accuracy separately. It is a separate experiment from the MeanAbs comparison in the response to ykLp.

DatasetMean gain after 1 step: ET3 / MAGEMean gain after 5 steps: ET3 / MAGEFinal transfer accuracy: ET3 / MAGE (%)
DTD15.546 / 16.67612.789 / 16.95716.20 / 22.83
Caltech10121.909 / 23.35420.313 / 22.90645.63 / 53.73
CIFAR-10021.542 / 22.93319.631 / 22.18622.50 / 18.50

MAGE produces larger mean margin gains after both one and five updates on all three datasets. Its final transfer accuracy is higher by 6.63 points on DTD and 8.10 points on Caltech101; ET3 leads by 4.00 points on CIFAR-100 in this experiment. We believe CIFAR-100 illustrates why the two measurements are useful together: the mean margin can improve even when fewer individual examples finish on the correct side of the decision boundary.

AuthorsOpenReview ↗

W7:Recovery and Stability Across Loss Weights

Thank you for suggesting a sensitivity analysis of the stability loss. We fixed the recovery weight at 1 and compared three stability weights on DTD and CIFAR-100.

DatasetStability weightClean accuracy (%)Transfer accuracy (%)Clean KL
DTD028.3018.701.234
DTD0.2536.5323.070.974
DTD0.534.6022.830.915
CIFAR-100013.8711.632.881
CIFAR-1000.2529.3318.172.250
CIFAR-1000.529.0018.502.161

At both nonzero weights, clean and transfer accuracy improve and Clean KL decreases relative to omitting the stability loss, on both datasets. In these settings, the stability objective helps preserve clean behavior while supporting recovery on perturbed inputs.

The best weight depends on the metric. A weight of 0.25 gives the highest clean accuracy on both datasets, while 0.5 gives the lowest Clean KL. Transfer accuracy peaks at 0.25 on DTD and 0.5 on CIFAR-100. This is why we report the three metrics together when assessing the effect of the stability weight.


AuthorsOpenReview ↗

W8:Attack Methods and Perturbation Budgets

Thank you for suggesting a broader robustness evaluation. We compared ET3 and MAGE on the same PGD inputs generated against the frozen Base model, varying the perturbation radius. The 2/255 column matches the transfer results in the mechanism analysis above.

DatasetRadius 1/255: ET3 / MAGE (%)Radius 2/255: ET3 / MAGE (%)Radius 4/255: ET3 / MAGE (%)
DTD21.70 / 26.6316.20 / 22.836.10 / 14.83
Caltech10159.33 / 59.1345.63 / 53.7316.00 / 42.03
CIFAR-10030.60 / 21.2322.50 / 18.509.00 / 14.17

At the largest tested radius, MAGE leads on all three datasets by 8.73, 26.03, and 5.17 points, respectively. At smaller radii, the ranking depends on the dataset. For example, on CIFAR-100, ET3 leads at 1/255 and 2/255, while MAGE leads at 4/255.

Appendix B.5 broadens the transfer evaluation to PGD-CE, APGD-CE, APGD-DLR, and CW. Appendix B.8 examines a different aspect of robustness: adaptive PGD iteration count. In that diagnostic configuration, MAGE achieves 37.2%, 36.2%, 36.8%, and 37.1% at 10, 20, 50, and 100 attack steps, respectively. The radius sweep establishes where the transfer gains persist, and B.8 shows that accuracy is stable as the adaptive attack runs longer in its diagnostic setting.

Reviewer_ykLpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MAGE, a defense for CLIP-based VLMs that learns a small scalar energy head on frozen image features, separately for each target dataset, using its labeled training split. The head is trained with energy-ranking, one-step restoration (cross-entropy and margin), and clean-stability losses. At test time, the input is updated by a few bounded energy-descent steps without labels. The method is evaluated on robust CLIP backbones (TeCoA, FARE) under adaptive PGD and compared mainly to ET3, as well as several test-time and prompt-based defenses.

Critical strengths

Good coverage of related work.

Critical weaknesses

Overall, the paper gives a somewhat misleading impression because the comparison to competing methods differs considerably between parts of the paper:

  1. There is no complexity comparison to competing methods. Only the appendix compares runtime, and only to the base model and ET3.
  2. Figure 1 compares only against the robust CLIP base, unlike the rest of the paper, which focuses on ET3 and other defenses.
  3. Table 1 shows a large clean accuracy degradation (21.6 points on standard CLIP), while the other tables do not discuss it.
  4. Table 2 presents clean and robust accuracy in a confusing way, which makes it hard to judge the cost of the robustness gain. For example, clean accuracy is not reported for ET3.
  5. In Table 3, older methods appear to outperform newer ones (for example, TTC is below the robust base), without sufficient discussion.
  6. Regarding supervision: MAGE trains on the full labeled training split of each target dataset. The clean accuracy gains suggest the head acts partly as supervised dataset-specific adaptation. A supervised baseline is needed.
  7. Normalization confound with ET3. In the main tables, ET3 uses L2 normalization, and MAGE uses Mean-Absolute normalization. Under Mean-Absolute normalization, ET3 itself rises from 12.9% to 46.6% (Table B.2), versus 55.1% for MAGE. Much of the reported gap over ET3 appears to stem from the normalizer rather than the learned energy.
Questions for authors

How does the computational cost compare to the other methods in Table 3? Why does Figure 1 compare only against the robust CLIP base? Can the authors report and discuss clean accuracy for all methods in Table 2? Why do older methods outperform newer ones in Table 3?

5 comments
AuthorsOpenReview ↗

W1, W5:nference Cost and the TTC Comparison

Thank you for asking for a clearer account of the cost and performance of test-time recovery. We measured inference latency and peak allocated PyTorch memory on an NVIDIA A40 using DTD, with a batch size of 8 and five repeats per method.

MethodLatency (ms/image)Peak memory (MiB)
Base6.239719.15
ET3, MeanAbs, 5 steps65.5831716.52
MAGE, MeanAbs, 5 steps65.4871716.52
TTC, 5 steps71.7781771.69

At the same five-step MeanAbs budget, MAGE and ET3 have essentially identical inference costs. MAGE takes about 10.5 times as long as a single Base forward pass. The distinction matters in practice: iterative recovery adds cost relative to Base, while learning the energy head does not increase the measured inference cost relative to five-step ET3.

Table 3 also shows that TTC falls below the robust Base in the reported setting. That result makes the robust Base a useful reference: an additional test-time update can incur cost without improving accuracy in a given setting. In Table 3, TTC achieves an average PGD accuracy of 19.7%, compared with 27.5% for the robust base. This indicates that its input correction does not provide a net benefit under the evaluated backbone and protocol. Additional input updates can potentially disrupt predictions that the robust backbone already handles correctly;Publication recency alone does not determine performance, and these comparisons should be understood in the context of their backbone, supervision, and deployment settings.

AuthorsOpenReview ↗

W2–W4:Figure 1 and the Tradeoff Between Clean and Robust Accuracy

Thank you for pointing out that the original figures and tables emphasized different comparisons. The original Figure 1 used the robust Base to show what changes when MAGE is added to a frozen backbone. A comparison with ET3 addresses a separate question: how learned recovery compares with fixed-energy recovery. Placing Base, ET3, and MAGE on the same nine-dataset comparison makes both questions visible.

Table 1 illustrates why clean and robust accuracy should be read together. On CIFAR-10 with standard CLIP:

MethodClean accuracy (%)Clean drop from Base (pp)PGD accuracy (%)
Base, no defense85.20.00.0
Original ET374.810.412.1
MAGE63.621.655.5

In this setting, MAGE gains 43.4 percentage points in PGD accuracy over ET3, with clean accuracy 11.2 points lower. Reporting both numbers makes the tradeoff explicit.

For the Table 2 question, the additional ET3 clean evaluation covers four backbones, four datasets, and three seeds, yielding 48 run records. The separate three-dataset MeanAbs comparison in our response to W6–W7 reports clean and transfer accuracy side by side. That comparison provides three concrete examples; it is distinct from the full Table 2 evaluation.

AuthorsOpenReview ↗

W6–W7:Matched Normalization and Target Supervision

Thank you for identifying two factors that should be examined separately: deployment normalization and the use of target labels. We acknowledge that normalization materially changes ET3’s performance. In Table B.2, switching ET3 from L2 to MeanAbs raises robust accuracy from 12.9% to 46.6%. Under the same MeanAbs normalization, MAGE reaches 55.1%, an 8.5-point advantage over ET3.

We also evaluated both methods with MeanAbs on the same adversarial inputs generated against the frozen Base model:

DatasetET3 clean / transfer (%)MAGE clean / transfer (%)MAGE transfer gain (pp)
DTD32.07 / 14.8834.75 / 21.90+7.02
Caltech10179.22 / 47.7373.75 / 55.48+7.75
CIFAR-10027.46 / 17.4343.42 / 20.31+2.88

MAGE retains a transfer advantage on all three datasets under the same normalizer. Clean accuracy improves on DTD and CIFAR-100 and decreases on Caltech101.

In a separate experiment on ViT-B/16, we compared MAGE with two linear heads using the same 1,000 labeled target examples and five training epochs:

DatasetSupervised linear head: clean / transfer (%)Adversarially trained linear head: clean / transfer (%)MAGE: clean / transfer (%)
DTD72.43 / 0.9048.69 / 6.8634.75 / 21.90
Caltech10195.38 / 0.6479.21 / 12.0373.84 / 55.65
CIFAR-10068.48 / 0.0026.82 / 0.4727.50 / 17.81

The supervised linear head has the highest clean accuracy on all three datasets. MAGE has the highest transfer accuracy, beating the adversarially trained linear head by 15.04, 43.62, and 17.34 points, respectively. We believe this is the key result of the supervision control: the same labels produce different outcomes when used to learn an input recovery direction rather than a classification head.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.