acceptodds
Withdrawn from ICLR 2027

Safe Vision-Language Models via Sparse Autoencoder Feature-Guided Embedding Steering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision language models such as CLIP provide transferable representations for retrieval, classification, generation, and visual question answering. However, pretraining on web data can also encode unsafe concepts that propagate to downstream systems. Existing safety alignment methods mainly follow two strategies. Fine tuning with constructed safe and unsafe pairs requires substantial training and can disturb broadly useful knowledge. Methods that directly manipulate pretrained weights without further training avoid this cost, but often provide limited safety control and may still impair general capabilities. We attribute these limitations in part to unsafe and general semantics being entangled in dense representations. Inspired by sparse autoencoders, which decompose dense neural representations into interpretable features, we propose Sparse Autoencoder Feature Guided Embedding Steering (SAFES-CLIP). Rather than updating the pretrained backbone, our method trains a sparse autoencoder with multiple Top-K budgets on frozen embeddings and obtains a shared sparse concept space. It identifies features associated with unsafe content using safe and unsafe calibration samples, then removes only their contribution to each input during inference. We further extend this mechanism to token representations for integration with generation and visual question answering systems. Experiments across retrieval between images and text, image classification, image generation, and visual question answering show consistent safety improvements while preserving broad general capabilities. Our core code has been provided in the supplementary material.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Models, Safety Alignment, Sparse Autoencoders, Feature Steering, Multimodal Safety, Generalization Preservation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_ms53OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Vision-language models (VLMs) are increasingly deployed in real-world applications and are typically trained on large-scale datasets collected from the internet. Such datasets may contain unsafe or harmful content, which can subsequently be propagated to downstream applications. Given a pretrained VLM, two main paradigms exist for mitigating these issues: (i) fine-tuning the model on safe/unsafe datasets to steer its behavior, and (ii) post-hoc weight manipulation or activation steering methods that do not require additional training data. Both approaches have advantages and disadvantages. Fine-tuning-based methods generally achieve stronger safety performance but risk degrading the model's general-domain capabilities, while parameter-free steering methods tend to preserve more knowledge but usually provide weaker safety improvements. This paper proposes a middle-ground approach that is neither fully training-based nor purely based on weight or activation manipulation. The key idea is to leverage interpretability techniques through sparse autoencoders, identify safety-related features using a calibration set, and modify unsafe activations at inference time on a per-instance basis during downstream tasks. The method is evaluated on CLIP-L and across four downstream tasks.

Critical strengths
  • The problem is relevant and well motivated. The paper addresses an interesting gap between existing fine-tuning-based and steering-based approaches.
  • The downstream evaluation is reasonably comprehensive, covering multiple tasks and usage scenarios for vision-language models.
Critical weaknesses
  • Evaluation on a single VLM is a significant limitation for a largely empirical paper. It is difficult to assess how well the method generalizes across architectures and model scales.

  • The proposed feature-scoring mechanism appears relatively simple. While intuitive, it is not clear how effectively it addresses the issue of polysemantic features. The authors acknowledge this concern, but its impact is neither thoroughly discussed nor experimentally investigated.

Questions for authors
  • Can the method be evaluated on additional models beyond CLIP-L? The current experimental validation is limited to a single architecture. A more convincing evaluation would include models of different scales and potentially different VLM families.

  • What is the training cost of the proposed approach, particularly the sparse autoencoder training stage? How does this cost compare to methods such as SafeCLIP?

  • How sensitive is the method to the sparse autoencoder training data? The current SAE is trained on CC3M, what happens if it is trained directly on ViSU or another dataset?

  • Does the approach generalize to previously unseen unsafe concepts that are not represented in the calibration set? Why is UWM not included as a baseline for the VQA and image-generation experiments?

  • Some reported improvements, particularly on COCO, appear relatively small and may fall within the expected variance of the evaluation. Could the authors report confidence intervals or statistical significance estimates?

  • The presentation of the tables could be improved. Rather than reporting only the delta for the proposed method in a separate row, it would be easier to compare approaches if the corresponding deltas were reported inline for all baselines and best-performing values should be highlighted consistently across columns. Also the color coding in Table 2 appears confusing (or potentially inverted?). For example, in MM-SafetyBench (where lower is better), lower values are highlighted in red, whereas in MMBench (where higher is better), lower values appear highlighted in green.

  • Does the method require paired safe/unsafe examples in the calibration set? If so, how restrictive is this requirement in practice? Discussing this limitation more explicitly, or outlining possible relaxations as future work, would strengthen the paper.

Reviewer_ufebOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SAFES-CLIP, a model to suppress unsafe concepts in CLIP embeddings while preserving the useful information in the semantics. The main motivation is that the safe and unsafe contents are entangled in the embedding space and suppressing by modifying model parameters inevitably reduces useful capabilities. SAFES-CLIP first trains an SAE to learn a sparse representation on the embeddings, then selects safety-sensitive features, and at inference time steers embeddings using the reconstructed features' contribution and removing them from the activation.

Critical strengths
  1. The core contribution is the idea to separate learning which features matter and modifying the models themselves. By learning an auxiliary representation and only intervening in the top most sensitive features is a more targeted method. This may be a useful middle ground between fine-tuning and static weight editing. the method seems well-motivated by a mechanistic angle and is potentially applicable to other models and types of embeddings.
  2. The empirical results show that SAFES-CLIP can preserve a significant amount of general capabilities over Safe-CLIP and UWM (especially for image classification in Figure 3.
Critical weaknesses

The empirical results on safety improvements are not as strong as capability perservation, especially the result on SafeGround in Appendix shows barely any improvements on safety over the baseline and worse performance than Safe-CLIP and UWM.

Questions for authors

None

Reviewer_6qnSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work trains a TopK sparse autoencoder to decompose the representation of a CLIP model. It then identifies a part of the SAE's representation responsible for the unsafety and uses it to steer the CLIP's representation to suppress such unsafe concepts. Benchmarks in image-text retrieval, image generation, and VQA show little signal that the method is useful.

Critical strengths

It was challenging to find any. The general idea of using SAEs to steer safety features in CLIP is interesting, but the paper is not well written, and I found neither novel ideas in the methodology nor results demonstrating that it performs significantly better than a simple baseline. Perhaps the most interesting is the application of CLIP+SAE on the ViSU, I2P, and MM-SafetyBench tasks.

Critical weaknesses

This work is a straightforward application of SAEs to safety-inclined tasks of the CLIP model. The first two contributions of this work are weak and not necessarily novel:

  1. We formulate safety alignment as learned intervention in a sparse feature space, a new paradigm that combines control guided by data with a frozen pretrained backbone

"Safety alignment as learned intervention in a sparse feature space" is the most standard formulation in the literature on safety/alignment/SAEs and not "a new paradigm".

  1. We instantiate this paradigm with a multi-TopK vision-language SAE, calibrated feature selection, and steering weighted by activation that acts independently on each input.

This is again the most standard methodology; see e.g. [a,b,c] and subsequent work citing them. It is challenging to find novelty in Section 3.

[a] "Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery" ECCV 2024

[b] "Interpreting CLIP with Hierarchical Sparse Autoencoders" ICML 2025

[c] "Sparse autoencoders learn monosemantic features in vision-language models" NeurIPS 2025

Regarding the third contribution of

3. We validate the framework across retrieval, classification, generation, and visual question answering, and show that data exposure explains a substantial part of Safe-CLIP’s advantage in retrieval with safe queries.

Validation of the proposed method is generally not a contribution in itself, but rather a necessity. What is the main finding? Empirical results show insignificant metric differences w.r.t. two unclearly chosen baselines.

  • a) Table 1 is unclear. Why is the difference w.r.t. CLIP highlighted and not the two baseline methods? Is the difference of ≤2\leq 2 significant or interesting? Why do both methods (UWM, SAFES-CLIP) perform relatively poorly (≤50\leq50 R@20) on the "unsafe counterparts"?
  • b) What is the difference between green/red arrows in Table 2? Why does the table not report results for the second UWM baseline method? Again, is the difference of ≤1\leq 1 in FID/CLIPScore/Circular significant or interesting?
  • c) How can the safety/alignment steering be applied to the 15 general image classification tasks such as MNIST, Food101, CIFAR100? Is the goal of Figure 3 to show that CLIP and SAFES-CLIP have the same exact result, which it fails to do?
  • d) Table 3 is also poorly presented: what is the difference between "variants" in rows and "variants" in columns (e.g. min-rank selection row vs. min-rank feature selection column)?
  • e) Table 4 again! reports infinitesimal differences in metric values such as 0.10−1.960.10-1.96 out of 100100.
  • f) In general, why do results, such as ablations in Table 3, show 4 digits but not report standard deviations/errors over repeated SAE runs etc.?

Finally, CLIP is quite an outdated model (2021), and so I would recommend analyzing the more recent SigLIP-2 and/or Meta CLIP-2 models (2025) to make the analysis interesting.

Questions for authors

Unfortunately, I don't think answers to any question could affect my assessment of this work.

Do you know what the "unsafe feature set" is? Are these just opaque values or actually labeled neurons in the SAE? Do you know that a particular feature relates to nudity? What are the safety/alignment-related concepts that you are concerned about in CLIP? It should be clearly discussed in the paper.

MM-SafetyBench is used but not cited in the paper.

Reviewer_5PyUOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAFES CLIP, a safety alignment framework for CLIP that leverages vision language sparse autoencoders (VL SAE) to suppress unsafe concepts via input driven embedding steering without fine tuning the CLIP backbone.

Critical strengths

This paper proposes SAFES CLIP, a safety alignment framework for CLIP that leverages vision language sparse autoencoders (VL SAE) to suppress unsafe concepts via input driven embedding steering without fine tuning the CLIP backbone.

Critical weaknesses

First, the paper is based on the existing VL SAE representation steering paradigms. Identifying unsafe features via activation difference statistics and subtracting reconstructed unsafe embedding components represents incremental engineering rather than a fundamental conceptual advance. The key insight of suppressing SAE features activated by harmful content has clear precedents in mechanistic interpretability literature, and the token level extension is a straightforward adaptation of the pooled embedding pipeline. Second, compared against fine tuned Safe CLIP, SAFES CLIP achieves substantially weaker unsafe to safe redirection on ViSU benchmarks. Although it preserves zero shot classification accuracy, its absolute safety improvements over vanilla CLIP are small. The ablation results further show strong performance degradation when pushing hyperparameters ρ and α for better safety, revealing a brittle safety utility trade off with no clear path toward real world deployment. Adaptive α is mentioned only as future work and is not fully validated. Third, the evaluation is confined to ViSU, MM SafetyBench and I2P benchmarks. The paper lacks testing against diverse adversarial prompts, out of distribution unsafe inputs or concept restoration attacks, which are known to defeat many concept erasure methods. The ethical statement acknowledges potential suppression of benign overlapping semantics, yet there is no systematic measurement of false suppression rates across diverse real world inputs.

Questions for authors

The inference requires running SAE encoding decoding for every input embedding, adding non trivial computational latency. The token SAE variants for Stable Diffusion and LLaVA introduce further overhead, but latency numbers are missing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.