acceptodds
Under review as a conference paper at ICLR 2027

Robustness Beyond Pixels for Vision Language Models

Abstract

Prompt learning adapts frozen vision–language models using a small number of trainable tokens, but adversarial prompt-learning methods typically defend only against image-space perturbations. We study a complementary white-box setting where an adversary jointly perturbs the input image and selected residual-stream activations, as may arise in shared-encoder or untrusted-adapter pipelines. This compositional threat model is challenging because pixel and feature perturbations interact through a common forward pass, while activation scales vary across transformer depth. We propose JPFAT (Joint Pixel–Feature Adversarial Prompt Tuning). JPFAT uses alternating projected gradient ascent over a fixed product threat set, Relative Activation Budgets to calibrate feature radii to layerwise activation scale, and entropy-regularized effort allocation to prioritize attack paths without changing their feasible radii. A vulnerability-adaptive consistency loss emphasizes samples with larger output and internal representation drift. Using fixed-budget, non-adaptive pixel, feature, sequential, and joint attacks for evaluation, JPFAT improves accuracy under the joint pixel–feature adversary and its individual components while maintaining competitive clean base-to-new generalization.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
adversarial robustness, joint pixel–feature attack, prompt learning, vision language model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zmp5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the robustness of adversarial prompt learning for vision-language models, noting that existing methods primarily attack and defend in the pixel space despite learnable prompts being inserted inside the encoder. The authors propose a joint pixel-feature threat model where an adversary perturbs both the input image and intermediate residual-stream activations at selected visual encoder layers. To address this, they present JPFAT, which comprises JA-PGD to alternate updates between pixel and multi-layer feature perturbations, Relative Activation Budgets to calibrate feature perturbation radii against layerwise activation scales, gradient-effort allocation to adjust step sizes while preserving fixed projection radii, and VAAL to weight training examples by output and internal drift. Experiments across 11 few-shot datasets using CLIP ViT-B/32 and MaPLe prompts evaluate base-to-new generalization, cross-dataset transfer, single-path and joint robustness, ablations, computational cost, AutoAttack, adaptive attacks, deeper attacks, and the ViT-B/16 backbone. The primary finding is that joint pixel-feature adversarial training improves joint robustness, accompanied by a modest drop in clean accuracy.

Critical strengths
  1. Strong motivation and a cohesive methodological formulation: The paper highlights the asymmetry between prompts operating internally and defenses remaining restricted to pixel space. It formalizes a joint pixel-feature threat model, where JA-PGD, RAB, effort allocation, and VAAL serve well-defined complementary roles to form a coherent framework.

  2. Comprehensive experimental evaluation: The validation covers 11 datasets, base-to-new generalization, cross-dataset transfer, pixel/feature/joint evaluations, ablations, computational overhead, AutoAttack, adaptive attacks, deeper attack depths, and alternative backbones. The empirical results consistently support the claim that joint training outperforms pixel-only adversarial prompt baselines under the proposed threat model.

Critical weaknesses
  1. Practical relevance of the threat model and low absolute robustness: The feature perturbation path assumes white-box write access to intermediate activations, which is uncommon in typical image-based adversarial settings. Furthermore, even under this threat model, the harmonic mean of joint robust accuracy remains around 5%–6%, which limits its practical security utility. The paper also acknowledges that feature-space robustness complements rather than replaces pixel-space robustness.

  2. Methodological complexity, hyperparameter sensitivity, and reporting consistency: JPFAT introduces multiple hyperparameters across RAB budgets and steps, allocation temperature and momentum, VAAL attention temperature and floors, robust loss weight λ\lambda, and attack depth. Ablation results indicate that the performance gains of RAB and effort allocation are somewhat dataset-dependent. Additionally, baseline comparisons involve non-equivalent threat sets during training, making JPFAT's advantage under joint evaluation largely expected. Finally, discrepancies in table labels and numerical values across sections should be resolved to ensure clarity, reliability, and reproducibility.

Questions for authors
  1. Feature perturbations require white-box write access to internal activations. Could the authors elaborate on more realistic deployment scenarios for this threat model? Does JPFAT retain an advantage over baselines under black-box or transfer-based attack conditions?
  2. Does the joint evaluation protocol inherently disadvantage pixel-only trained baselines? Were the training compute, total perturbation budgets, and attack iteration counts matched across methods? Could the authors clarify the naming and numerical discrepancies across the tables?
Reviewer_bvu5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies robust prompt learning for vision–language models under joint pixel and internal activation perturbations. It proposes JPFAT, which trains a small set of prompt parameters while keeping the encoders frozen. The method combines alternating pixel–feature attacks, Relative Activation Budgets, gradient-effort allocation, and representation-drift-based sample weighting. Experiments on 11 datasets show improved accuracy under joint and individual attacks relative to the compared baselines, although absolute robust accuracy under the joint attack remains low.

Critical strengths
  • The paper clearly distinguishes pixel perturbations from internal activation perturbations and models their joint effects through a common forward pass, providing a well-defined setting for studying internal representation robustness.
  • Experiments cover base-to-new generalization and cross-dataset transfer, with separate results for clean inputs, pixel-only, feature-only, and joint attacks, making performance under different attacker capabilities easier to assess.
  • Component ablations, stronger attacks with multiple restarts, variations in attack depth, and evaluation on an additional backbone provide empirical support from several perspectives.
Critical weaknesses
  1. The practical justification for the internal-activation threat model is insufficient. The paper motivates this setting through shared encoders, untrusted adapters, and plugins, but does not specify why an attacker would have write access to activations at the selected layers while being restricted to small, element-wise bounded additive perturbations. The presence of prompt parameters inside the encoder does not itself imply such access. The connection between the threat model and practical deployment scenarios therefore needs clarification.

  2. The VAAL ablation does not separate sample reweighting from an overall increase in the robust-loss weight. Appendix E shows that, when the normalization floor is inactive, the average weight is 2+c(a), whereas disabling VAAL sets w_i=1. Enabling VAAL therefore also increases the overall strength of the robust loss. The observed gains alone do not establish that VAAL benefits from effectively identifying vulnerable samples. Comparisons with matched average weights or robust-loss strength are needed.

  3. The low joint robust accuracy limits the practical utility of the defense. In Table 2, the average HM under the joint attack improves from the best baseline’s 0.4% to 5.4%, while average joint accuracy in cross-dataset transfer is only 6.2%. These results demonstrate relative improvements but do not establish that the model remains usable under this threat. The paper should more clearly distinguish an exploratory mitigation of a strong joint threat from a defense with practical utility.

Questions for authors
  1. Please clarify the evaluation sample selection. Appendix C states that the test partition is subsampled to a fixed test batch size. How many examples are evaluated for each dataset, and is the full test set covered?

  2. Are the attack update schedules compared under matched computational budgets? JA-PGD requires two backward passes per iteration, whereas simultaneous pixel and feature updates can use one. Do the sequential-attack or leader-order experiments distinguish the benefit of alternating updates from that of additional gradient computation?

Reviewer_GGiFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Different from previous studies that define the adversary in the pixel space, this paper proposes a complementary setting where the adversary jointly perturbs the input image in the pixel space as well as the activations in the feature space. Specifically, this paper proposes JPFAT, which includes four components and alternates the pixel and feature updates. Experimental results on multiple datasets demonstrate that the joint robustness improves over pixel-only baselines.

Critical strengths
  1. This paper has a clear motivation that proposes an adversary that jointly perturbs the input image and features.
  2. When designing the feature adversary, this paper highlights the challenge that features might have significantly different magnitudes across layers, and proposes an adaptive strategy for setting hyperparameters. This design is insightful and interesting.
  3. Experimental results show the effectiveness of the proposed method, such as improved robustness even in the cross-dataset setting.
Critical weaknesses
  1. Discussion about layer selections.

This paper designs a feature-level attack on several consecutive layers in the bottom of the model (line 133). Is there any reason or motivation for this design? For example, how about attacking a specific layer, such as layer 2, instead of attacking consecutive layers? How about attacking features from middle or top layers?

  1. Generalization to different fixed attack budgets during inference.

For the evaluation, this paper reports results with fixed budgets, 0.002 for εf\varepsilon_f, 0.0004 for αf\alpha_f. Why was this budget chosen? How about evaluating using different budgets during inference?

  1. Minor: including autoattack evaluation in the main paper.

I understand the space limit in the main paper, but it would be clearer and more solid if autoattack evaluation is included in the main paper to show the generalization ability.

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.