acceptodds
Withdrawn from ICLR 2027

T-VSS: Test-Time Visual Subspace Steering for Adversarial Robustness of Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Contrastive vision-language models (VLMs), exemplified by CLIP, enable strong zero-shot visual recognition but remain highly vulnerable to adversarial perturbations. Recent test-time adaptation methods improve robustness without retraining, but they do not directly adapt the corrupted visual representation itself. Prompt-based methods adapt the learnable text prompts, while input-space methods optimize pixels or padding at test time. These approaches can improve predictions, but they do so through an indirect and expensive optimization path. We propose Test-time Visual Subspace Steering (T-VSS), a lightweight defense that performs test-time adaptation directly in the visual feature space. T-VSS first builds a sample-specific low-rank subspace from multi-view feature residuals anchored at the attacked image. It then learns a shared feature correction within this subspace using reliability-weighted entropy minimization. By constraining adaptation to a compact visual geometry, T-VSS steers attacked features toward more stable and discriminative predictions while avoiding noisy full-space updates. Experiments on fine-grained, ImageNet, and ImageNet-OOD benchmarks show that T-VSS improves adversarial robustness while maintaining competitive clean accuracy and better efficiency than prior test-time adaptation methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Adversarial Robustness, Vision-Language Models, Test-time Adaptaton

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_euSROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Test-Time Visual Subspace Steering (T-VSS), a test-time adversarial defense based on a low-rank visual feature subspace. It constructs a sample-specific subspace from anchor-relative residuals across augmented views and learns a shared low-rank feature correction through reliability-weighted entropy minimization, improving CLIP zero-shot classification robustness.

Critical strengths

S1. The manuscript is complete and clearly organized, with a coherent presentation of the motivation, method, and experimental evaluation.

S2. The experimental evaluation covers eight fine-grained classification benchmarks, ImageNet, and ImageNet out-of-distribution datasets. The paper also reports results across CLIP backbones, attack methods, and inference-efficiency settings, providing a relatively broad empirical assessment.

Critical weaknesses

W1. The rationale for using stochastic multi-view augmentation to construct the correction subspace could be more explicit. Although the paper includes view-budget comparisons and analyzes multi-view residual structure, it remains unclear why augmentation-induced feature variation is the most appropriate basis for the proposed steering operation, or how essential this particular construction is relative to alternatives.

W2. The relationship between the residual-SVD subspace and effective adversarial correction directions is not directly established. The low-rank residuals primarily describe relative feature variations induced by augmentation, rather than the clean–adversarial embedding displacement itself. Although the reported results support the empirical utility of the subspace, they do not show how much of a useful correction direction it actually captures.

Questions for authors

Q1. Could the authors clarify why stochastic data augmentation is needed to construct the visual subspace from an attacked input? What would be lost without augmentation, and could an alternative subspace construction provide comparable feature steering?

Q2. What is the practical interpretation of the anchor-relative residual matrix? Since it does not directly estimate the clean–adversarial feature displacement, could the authors quantify its relationship to the actual correction direction, for example by analyzing the projection of the clean–adversarial displacement onto the residual-SVD subspace?

Reviewer_vz1WOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes T-VSS, a test-time defense for zero-shot CLIP. It uses SVD on the feature residuals between an input image and its augmented views to obtain a low-rank subspace, then optimizes a shared feature correction with reliability-weighted entropy minimization. Both CLIP encoders remain frozen. The experiments cover eight fine-grained datasets, ImageNet, and ImageNet-OOD benchmarks. The method generally improves adversarial accuracy over the tested optimization-based defenses while keeping the inference cost relatively low. However, the experiments in adaptive attack evaluation, ablation, and efficiency evaluation are far away from being accepted by ICLR 2027.

Critical strengths
  1. Adapting features rather than prompts or pixels is a simple and practical idea. The method only updates a small coefficient vector at test time.

  2. The results cover several CLIP backbones and datasets. The ablations, especially the residual-SVD versus random-basis comparison, are useful.

Critical weaknesses
  1. The adaptive attack evaluation is insufficient. Appendix D uses 7-step EOT-PGD on ResNet-50, even with only one stochastic defended forward pass per attack step. Stronger settings and results on ViT would be more convincing.

  2. The low-rank residual analysis does not show whether the selected subspace contains useful directions for correcting adversarial displacement. The random-basis ablation helps, but it does not directly answer this question.

  3. The efficiency evaluation is insufficient. Table 4 only reports latency on UCF101 with ResNet-50. With 64 views, T-VSS takes 0.383s/image versus 0.408s for TTP, so the difference is fairly small.

  4. The method is closely related to DBD (ICLR 2026), since both use augmented views for test-time visual feature correction. The comparison in Appendix C.7 is limited to Caltech101, where DBD is much stronger on standard ViT-B/32 (98.8% vs. 76.6% robust accuracy). The advantage of T-VSS over DBD is not clear.

  5. The core component in ablation, reliability-weighted entropy minimization, is closely similar to R-TPT (CVPR 2025). And the whole ablation studies are closely similar to R-TPT (CVPR 2025).

Questions for authors
  1. How does T-VSS perform with more EOT samples, attack steps, and restarts, particularly on ViT?

  2. What is the main advantage of T-VSS over DBD? Could the authors provide a broader comparison across datasets and backbones, with appropriate calibration for DBD?

  3. How much of the clean-to-adversarial feature displacement lies in the selected subspace? Does the learned correction actually move the feature closer to its clean counterpart?

  4. Could the authors break down the runtime for view encoding, SVD, and optimization, and report latency on ViT backbones?

Reviewer_ywt3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes T-VSS, a test-time defense for CLIP-style VLMs against adversarial perturbations. Instead of adapting text prompts or input pixels, it adapts the visual features directly. It builds a sample-specific low-rank subspace from the residuals of augmented views relative to the attacked image, then learns a shared steering correction within this subspace using reliability-weighted entropy minimization. The method is evaluated on fine-grained datasets, ImageNet, and ImageNet-OOD benchmarks across several CLIP backbones.

Critical strengths
  1. The idea is very interesting: it brings steering and low dimensionality into test-time defense. These ideas are widely known and accepted, and they feel like a very natural addition to this field.
  2. Strong experimental work, comparing against many similar methods on many relevant datasets.
  3. The ablation study clearly shows the importance of each component of the defense.
Critical weaknesses
  1. Missing comparison to related work, such as the energy-guided test-time defense ET3.
  2. The clean accuracy degradation needs more discussion so the reader can judge whether the trade-off makes sense.
  3. Dependence on attack: since this method looks at the rank, an attacker who is aware of it might be able to manipulate it.
Questions for authors
  1. How does T-VSS compare to ET3?
  2. Can the authors discuss the clean accuracy degradation further, and in which settings they consider it acceptable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.