- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper targets at a critical question: could a safety direction computed at one source model remains effective when transfers to another different model's representation space. This paper proposes a framework that allows we compute a safety vector at the representation of one LLM and, train a light-weighted aligner to map this vector to the representation space of the target LLM without accessing to unsafe data and finally steer the vector before it passes to the downstream generator model (Text-to-Image and Text-to-Video models). By validating under multiple underlying model types and different alignment methods, the results shows the effectiveness of this framework. What's more, this framework also supports multi-category steering.
Critical strengths
- This paper targets at a critical problem, where the governer can not access the unsafe data, with only the safety direction computed from another underlying model, would it be possible to leverage it to the deployed model's safety steering. Also, the paper links LLM-to-LLM transfer with Visual content moderation, which is under-explored.
- The evaluation is broad (3 sources, 5 targets across two modalities, 3 maps, two image benchmarks) and include lots of useful controlled studies. These controlled studies helps understanding the contribution of the selection of different components and hyper-parameters.
Critical weaknesses
I feel like the current pipeline is more of a combination of several existing technical methods. For example, Difference-of-means steering, aligning with affine/linear/MLP and additive intervention on text-encoder are all well-established components. Therefore, the combination seems to lead to a predictable result, since although the authors use Image and Video generation to make the setting as multi-modal, the upstream encoder is still an LLM so it's natural to use another LLM as the source model.
Questions for authors
- Could the authors clarify the technical contribution of the framework they propose? I want to see more technical design insight under this specific cross-model setting.
- Why the authors choose image generation and video generation as the downstream generator without any design that is specific to the visual modality? The target LLM is directly used as the encoder of the generator so I think the claim of first work in this "source-only interventions remain actionable through downstream image and video generators without target-side unsafe data" should be clarified.
- Would it be possible to decompose the SVD direction into a component parallel to the native direction and an orthogonal residual, and steer with each one separately under matched norm? That would help tell whether SVD’s lower ASR is really coming from the safety direction or from the rest.
- The baselines authors choose do not contain methods specifically designed for this multi-modal generation setting. For example, a very direct baseline could be leverage an LLM as rewriter to rewrite the input unsafe prompt, which only has access to the text input without changing the underlying generator.
3 comments
Response to Reviewer Vfgh (Part 2)
[Q4] LLM prompt-rewriting baseline
We agree that prompt rewriting is a relevant baseline. We use Llama-3-8B-Instruct with greedy decoding and no generated-image feedback. Ridge uses strength 5 on Schnell, Dev, and Qwen-Image and 3 on Z-Image. CLIP-Sim is measured against the original prompt; ASR is conditional on valid images, with image availability reported alongside it.
LLM prompt rewriting: safety and generation quality.
Schnell.
| Method | ASR (%) ↓ | Benign CLIP-Sim ↑ | Valid images, I2P / benign (%) |
|---|---|---|---|
| Original | 30.7 | 0.319 | 100.0 / 100.0 |
| Transfer Ridge | 11.0 | 0.306 | 100.0 / 100.0 |
| LLM prompt rewriting | 21.0 | 0.316 | 78.7 / 97.3 |
Dev.
| Method | ASR (%) ↓ | Benign CLIP-Sim ↑ | Valid images, I2P / benign (%) |
|---|---|---|---|
| Original | 28.6 | 0.309 | 100.0 / 100.0 |
| Transfer Ridge | 12.4 | 0.297 | 100.0 / 100.0 |
| LLM prompt rewriting | 20.1 | 0.307 | 77.1 / 97.3 |
Qwen-Image.
| Method | ASR (%) ↓ | Benign CLIP-Sim ↑ | Valid images, I2P / benign (%) |
|---|---|---|---|
| Original | 38.4 | 0.332 | 100.0 / 100.0 |
| Transfer Ridge | 17.1 | 0.319 | 100.0 / 100.0 |
| LLM prompt rewriting | 24.4 | 0.323 | 78.7 / 97.3 |
Z-Image.
| Method | ASR (%) ↓ | Benign CLIP-Sim ↑ | Valid images, I2P / benign (%) |
|---|---|---|---|
| Original | 30.4 | 0.319 | 100.0 / 100.0 |
| Transfer Ridge | 10.5 | 0.287 | 100.0 / 100.0 |
| LLM prompt rewriting | 18.0 | 0.312 | 78.7 / 97.3 |
Rewriting largely preserves mean benign CLIP-Sim among requests that produce valid images, while transfer achieves lower conditional ASR in these comparisons. Refusals and invalid responses leave 21.3–22.9% of I2P requests and 2.7% of benign requests without an image; these requests receive no safety credit. Consequently, the ASR denominators differ across methods. On the Dev prompts that yield images after rewriting, the paired original ASR is 28.5%, versus 20.1% after rewriting (difference: −8.4 percentage points; 95.0% paired interval [−11.2, −5.8]).
References
[1] Yuepeng Hu, Zhengyuan Jiang, and Neil Zhenqiang Gong. SafeText: Safe Text-to-image Models via Aligning the Text Encoder. arXiv preprint arXiv:2502.20623, 2025.
[2] Jaehong Yoon, Shoubin Yu, Vaidehi Patil, Huaxiu Yao, and Mohit Bansal. SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation. International Conference on Learning Representations (ICLR), 2025.
[3] Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu. Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models. Advances in Neural Information Processing Systems (NeurIPS), 2024.
[4] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023.
[5] Qwen Team. Qwen3.5: Towards Native Multimodal Agents. Qwen technical blog, 2026.
Thank the authors for the detailed response and the additional experiments. I will keep my current positive score. Please update the revised version of the paper to include the additional clarification and experimental results.
Response to Reviewer Vfgh (Part 1)
We thank the Reviewer for recognizing the importance of constructing interventions without target-side unsafe training pairs and the controlled evaluation. We also appreciate the concrete suggestions on SVD decomposition and prompt rewriting.
[W1/Q1] Technical contribution and design insight
The design separates unsafe source contrasts from benign target alignment. A reusable map transports the contrast once; generation then adds the saved vector to normal target conditioning states, keeping the generator frozen and the mapper outside inference. Although the components are established, benign alignment alone does not ensure a useful downstream safety intervention. The new decomposition explains effects beyond native-direction agreement, while Schnell→Dev demonstrates unchanged reuse with a shared encoder.
[W1/Q2] Visual generation and scope of the priority claim
Why visual generation? Visual generation tests whether a transferred text-space intervention remains useful after passing through a frozen downstream generator. Prior work motivates this interface: SafeText [1] aligns the text encoder, SAFREE [2] filters prompt embeddings in its image/video safeguard, and AdvUnlearn [3] finds the text encoder more suitable than the U-Net for robust concept erasure in its evaluation. These findings motivate text conditioning without making it universally preferable to visual-space interventions.
Applicability to text generation. Our mechanism is not inherently visual. We therefore include transfer from Llama-3-8B-Instruct to Mistral-7B-Instruct-v0.3 [4] and Qwen3.5-9B [5] as text-generating targets. Ridge transfers the source safe/unsafe contrast using benign Alpaca anchors for alignment and magnitude scaling, without target-side unsafe training pairs. We evaluate JailbreakBench responses with greedy decoding. Safety is the fraction classified as safe by Llama-Guard-3-8B; perplexity is measured with the intervened target on Pile and Alpaca.
At α=0.5, safe-response rates increase by 28.0 percentage points on Mistral and 10.0 on Qwen, with PPL increases of at most 4.3% on both evaluation corpora. At α=1, safety improves further at a larger language-modeling cost. These examples support applicability beyond visual generation. We will clarify that our claimed contribution concerns source-only safety transfer evaluated through frozen image/video generators, not a modality-specific alignment mechanism.
[Q3] SVD parallel/residual decomposition at matched norm
Decomposition. For unit native direction n and transferred SVD vector v, we form
p=(v⊤n)n,r=v−p,
and rescale each component to ∥v∥2. The positive projection makes the norm-matched parallel intervention identical to the native direction at that norm.
Evaluation protocol. Using a Llama-8B source, we evaluate each component under matched applied norm. Cells report detector-ASR (%) / relative benign CLIP loss (%). Unsteered ASR is 30.7% on Schnell and 30.4% on Z-Image.
Interpretation. The residual affects detector-ASR despite native orthogonality. On Z-Image at α=1, it gives lower ASR than the parallel component at nearly identical mean CLIP loss. At higher strengths, the residual and full SVD incur much greater loss. This establishes behavioral effects beyond the native direction, without isolating a pure safety feature or attributing all suppression to incoherence.
This diagnostic experiment uses native target information, including the native-derived Z-Image scale. It controls applied norms and generation parameters within each target; it does not demonstrate construction without target-side unsafe training pairs or matched utility across targets.