acceptodds
Under review as a conference paper at ICLR 2027

Cross-modal steering vectors exist across both image and text generation

11 views

Abstract

Are concepts and mechanisms unified across modalities in current AI models, or instead implemented separately for each one? We operationalize this broad question of modality unification by studying cross-modal steering: 1) finding vectors that successfully steer generation on tokens of one modality, which then 2) transfer zero-shot to steer generation when added to tokens of another modality. Our study goes beyond simple correlational alignment (e.g. representational similarity) because steering is fundamentally causal. We study cross-modal steering in several unified multimodal models (UMMs), since they possess shared parameters for language and image generation, focusing on these two modalities. Across diverse steered concepts such as emotion and object size, we show that steering vectors transfer zero-shot in both directions (text-to-image & image-to-text). Even for concepts that are highly unique to one modality (e.g. formal vs. colloquial language), steering vectors sometimes induce a consistent, interpretable effect in the other modality, such as formal clothing and design. This transfer has a geometric signature, too: despite a clear modality gap separating text and image representations, steering vectors with strong cross-modal transferability tend to point in similar directions for the same concept across modalities. Our findings shed light on how vision and language interact inside unified multimodal models and point toward more general, modality-agnostic approaches for controlling their behavior.

Primary area:
interpretability and explainable AI
Keywords:
unified multimodal models, vision-and-language, steering, representational alignment, interpretability, image editing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Bu22OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper systematically studies the cross-modal transfer ability of Steering Vectors in unified multimodal generation models. Through bidirectional causal interventions on three models, namely UniAR, Emu3.5, and Liquid, the authors find that concept directions extracted from text or images can achieve zero-shot control in the other modality. They further analyze the connection between this transfer ability and the geometric relations of cross-modal representations.

Critical strengths
  1. The experimental details are clearly described.

  2. The experiments are fairly comprehensive, and the unique geometric analysis has certain value.

Critical weaknesses
  1. The novelty of the paper is very limited. Can a vector learned in one model be applied to other models? If the finding only holds within the unified own models, it has little meaning. The original purpose of unified models is to provide a unified representation of different modalities, such as images and text, inside the model. Therefore, it is expected that vectors extracted from different modalities have similar or the same effects. This is a natural consequence of the design.

  2. I do not see the motivation of the paper from beginning to end. The existence of such cross-modal steering vectors inside a unified model is not very surprising. Then what is the use of such vectors? Or what is the use of this finding for the development of multimodal models? The specific pattern found in unified models cannot be extended to other multimodal models.

Questions for authors
  1. The paper extracts vectors by taking the mean of activation differences. This is a very coarse and inaccurate approach. I doubt the ability of such extracted vectors, especially abstract vectors, to transfer to other modalities and concepts.

  2. As the authors state in line 92, existing work has already studied cross-modal steering in unified models and reached conclusions different from this paper. Why is that?

  3. The authors state that "In Emu3.5 and Liquid, text-derived vectors sometimes even steer images better than image-derived ones". Why is this? It is puzzling.

  4. The paper contains statements such as "steering an image generator with a vector for formal vs. colloquial language causes people or rooms to be depicted more formally, and steering a language model with a vector for low image resolution causes the model to write shorter text and express uncertainty". I do not know how to understand the boundaries of abstract concepts such as formal and informal as defined in the paper. Moreover, the logic of the presentation is not fluent.

  5. Section 2.1 spends a large amount of space introducing very basic content, which should be summarized in the related work. The method section contains no novel content and is basically a description of simple and generic approaches.

  6. The evaluation system has flaws, and the testing and assessment contain same-family circular dependencies. The minimal pairs, image edits, and evaluation prompts are all generated by Gemini-family models, and the judge is also Gemini 3.7 Flash. The "concepts" being evaluated are partly defined by the editing behavior of Gemini. A judge from the same family may inherit the concept bias of the data generation model, thus systematically overestimating the success rate of cross-modal transfer. Although human annotation is used to validate the automatic evaluation, experiments with a single same-family model are not sufficient.

  7. The evaluation content is incomplete. The paper mainly checks whether the target attribute changes, but it does not clearly evaluate other core application metrics, such as unintended changes to other attributes or degradation of instruction-following ability.

4 comments
AuthorsOpenReview ↗

Addressing main concerns

Dear Reviewer Bu22,

Thank you for engaging with our paper in detail. We will primarily discuss your main critiques around novelty and surprise, and address the other points at the end.

W1: Novelty and surprise

We think cross-modal steering vectors are quite surprising and far from obvious. In fact, we argue that such alignment might be less obvious in UMMs than in LLaVA-style VLMs.

  • Serra et al. 2025 [1] showed how UMMs are often less unified than LLMs that were adapted to be VLMs (e.g. LLaVA), across several metrics: 1) attention patterns and 2) lower cross-modal cosine similarity in UMMs. This can be quickly checked from their Fig1 and Fig2 (https://arxiv.org/abs/2412.06646). A recent paper shows similar mixed results [2]. Before seeing these, we also might have aligned more with your view.
  • One intuition why UMMs might be less unified:
    1. LLaVA-style models force representations into textual space, whereas UMMs flexibly allow both modalities to co-exist, allowing for each modality to exist in a separate subspace when useful for the pretraining objective.
    2. UMMs focus on visual generation and not just visual understanding: A model that does visual generation also needs to model perceptual details, which might not share many similarities with language-like semantics.
  • A model being called unified does not imply how shared its representations actually are, which is why we investigate this.

We also think our finding is non-trivial when considering the steering literature. Even when studying steering vectors in the context of text generation, generalization to different prompts and task settings is an active research area [3, 4, inter alia].

We are not aware of work showing that cross-modal steering is trivial, and will be happy to directly respond to any pointers the reviewer provides. This seems to us like the primary concern we’d like to understand further. It might ultimately also improve our framing and writing of the final version.

W2: Why study UMMs

So, why work on UMMs if they are far from solved? As you point out: “If the finding only holds within the unified own models, it has little meaning.”

UMMs are in another sense the most general form of current AIs, and follow the philosophy of our field in recent years: one model that can process and generate any form of data. We therefore expect them to grow stronger and more mainstream with time, and for phenomena like cross-modal steering vectors to improve. So it is worth understanding UMMs scientifically already now.

[1] Serra, Alessandro, et al. "The narrow gate: Localized image-text communication in native multimodal models." Advances in Neural Information Processing Systems 38 (2026): 74684-74712. [2] Gao, Xin, et al. "Do Text Edits Generalize to Visual Generation? Benchmarking Cross-Modal Knowledge Editing in UMMs." arXiv preprint arXiv:2606.00477 (2026). [3] Da Silva, Patrick Queiroz, et al. "Steering off course: Reliability challenges in steering language models." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. [4] Tan, D., et al. (2024). Analysing the Generalisation and Reliability of Steering Vectors. NeurIPS 2024. https://arxiv.org/abs/2407.12404.

AuthorsOpenReview ↗

Addressing detailed questions

Thank you for the reading our paper thoroughly and raising some important questions!

Other methodological Qs

Q1: Mean of activation differences methodology as “coarse and inaccurate” approach: This is the most standard steering approach, to our understanding. It is highly established in LLMs and, in recent years, also more in vision. We extensively cite and contextualize this in Section 2.1. Mean of activation differences is also a very natural solution for achieving one unified and comparable steering framework for both modalities. In fact, simplicity of steering design can be seen as a strength: Complex steering approaches can often be heavily parametrized for little-to-no performance gain, and, critically, are less faithful to the underlying representations, similar to complex probes.

Re “I doubt the ability of such extracted vectors”: Our experiments establish at least a lower bound on this ability, i.e., we measure concept presence in generated outputs.

Q2: Why Wang & Li (2026) [5] reached different conclusions is a great question. We briefly discuss the main differences in our related work section: We suspect it is because they primarily rely on BAGEL, which is not very unified; e.g., BAGEL has separate weights for text and and image generation, which interact only through attention. Their code is not public.

Q3: Why do text-derived steering vectors sometimes work better than image-derived for images? Intuitively, image generation has been “steered” by text via prompting for many years, and text may provide a less noisy semantic space [6]. We will revise the paper with this explanation.

Q4: Regarding the boundaries and logic of abstract concepts like “formal vs. informal”: Our motivation is that cross-modal steering would be most interesting to study when the concept is not equally represented in vision and language data. This might constitute even stronger evidence of cross-modal transfer. We agree there is some subjectivity in how formal vs. colloquial language vectors would manifest visually, but we do think some abstract notion of “formality” can be visual, such as clothing or interior design.

Q5: Is Sec. 2.1 needed: We did not want to assume deep familiarity with either UMMs or steering vectors, especially since some readers might only know steering vectors for LLMs and other readers for image generation.

Q6: “Circular dependencies” using Gemini: We re-judged half of the prompts in the main evaluation (9,600 pairs) with a different judge (Claude Opus 5.5, no few-shot examples). It is stricter in absolute terms but picks the opposite candidate on only 1.1% of pairs. Recovery with Claude vs. Gemini (UniAR / Emu3.5 / Liquid):

  • text→image: 0.82 / 1.38 / 1.32 vs. 0.89 / 1.27 / 1.25
  • image→text: 0.74 / 0.45 / 0.41 vs. 0.76 / 0.61 / 0.45

We would like to emphasize that the typical circular dependency is not an issue here: The models producing the steered outputs and the judge are fully separate. When you refer to “image edits”, these are not steered outputs but the minimal pairs that are ultimately collapsed into a single vector. We also note that our randomly shuffled evaluation protocol controls for judge bias towards the steered or unsteered sample.

Q7: Other attributes and instruction following beyond concept presence: App. A.7 shows output quality against steering strength (text fluency, image coherence). Instruction following for text stays at 90% or more of its unsteered level up to the alpha strength where concept effect peaks. We are not claiming to release a proper editing tool and mainly emphasize the interesting phenomenon of cross-modal steering.

We are happy to answer any further questions. If your primary concerns are addressed, we hope you consider reflecting this in your final rating

[5] Wang, Yu, and Sharon Li. "Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steering." arXiv preprint arXiv:2607.26411 (2026). [6] Ekin, Yigit, and Yossi Gandelsman. "The unreasonable effectiveness of text embedding interpolation for continuous image steering." arXiv preprint arXiv:2603.17998 (2026).

Dear Reviewer,

Thank you again for your time and effort on this. Since we have uploaded a revised version of the paper yesterday, let us know if this adequately addresses your primary concerns. See the global response for some details what we changed and the PDF itself for the actual changes.

Dear Reviewer Bu22,

We hope you engage with our response, especially the first of the two messages: We genuinely believe these findings are far from expected and we encourage you to consider the work we cite to support this. Also for people who don't work on UMMs, and for example just on steering vectors and their robustness, these findings are interesting.

Reviewer_gSkGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates whether concept steering directions estimated in one modality can control generation in another modality in unified autoregressive multimodal models. Across three model families and eight concepts, it studies bidirectional transfer, layer dependence, geometric alignment, and examples involving modality-specific concepts.

Critical strengths
  1. The interventions provide more direct evidence of cross-modal transfer than representation similarity alone. Bidirectional evaluation across different visual-tokenization designs is a useful contribution.
  2. The paper investigates both successful and weak-transfer concepts, layer effects, and absolute steering performance in addition to relative recovery. These analyses reveal meaningful heterogeneity rather than claiming universal transfer.
  3. The appendices include output-quality measurements and human validation on 320 held-out pairs with three judgments per pair. These are important checks on the automated evaluation and should be credited.
Critical weaknesses
  1. Recovery depends on the evaluation and strength-selection procedure. Normalized AUC is computed over setting-specific strength ranges, selected using target-modality behavior and extending toward degeneration (Appendix A.4). Recovery ratios can also be large when within-modality steering is weak. Thus a high ratio is not by itself evidence of strong, usable transfer. Absolute effects at matched output-quality constraints, with strength calibration separated from evaluation prompts, would provide a more interpretable main comparison. Source-only vector estimation should also be distinguished from a target-tuning-free evaluation protocol.
  2. Output quality needs to be integrated into the principal performance claim. Appendix A.7 already measures quality, and Figure 8 shows substantial text-fluency degradation as strength increases. This is not an absence of quality evaluation; the concern is that concept AUC and relative recovery can reward effects outside a useful quality regime. Showing the transfer curves subject to a fixed quality threshold would help determine how much transfer remains practically meaningful.
Questions for authors
  1. How do the model rankings and transfer conclusions change under a matched-quality evaluation, with strength ranges calibrated on separate prompts?
  2. Which concepts transfer reliably across prompts and generation seeds? Can you provide uncertainty on both absolute AUC and recovery, especially when the denominator is small?
3 comments
AuthorsOpenReview ↗

Main response

Dear Reviewer gSkG,

Thank you for the careful review, and for appreciating the quality measurements and human validation!

W1/W2/Q1: Recovery depends on strength selection, and output quality should be part of the main claim

We agree on both points. For this rebuttal, we computed recovery by comparing the two vectors at equal output quality instead of over strength ranges we selected, with the strength calibrated on separate prompts.

For each vector, we take the steering strength at which output quality drops to 90% of the unsteered outputs (image coherence for images, fluency for text; the same quality judges as in App. A.7). We find this strength on one half of the prompts and measure concept rate at that strength on the other half, averaged over random splits. Recovery is the concept rate of the transferred vector divided by that of the intra-modality vector.

UniAREmu3.5Liquid
Text→image, recovery in the paper (Fig. 3)0.841.231.29
Text→image, recovery at equal output quality0.971.281.28
Image→text, recovery in the paper (Fig. 3)0.740.590.39
Image→text, recovery at equal output quality0.840.490.49

Since a ratio alone does not show usable transfer, here are the absolute concept rates behind it (transferred vs. intra-modality vector):

UniAREmu3.5Liquid
Text→image0.77 vs. 0.800.76 vs. 0.600.67 vs. 0.52
Image→text0.62 vs. 0.740.27 vs. 0.550.25 vs. 0.51

Our conclusions are unchanged. Text-derived vectors steer image generation as well as image-derived ones in UniAR, and better in Emu3.5 and Liquid, also in absolute terms. Image-derived vectors steer text generation less well than text-derived ones, with UniAR transferring best; in Emu3.5 and Liquid this transfer is weak in absolute terms. A threshold of 80% gives the same conclusions.

One limitation remains: the candidate strengths still come from our hand-chosen ranges; only the choice of strength within them is now made automatically and on separate prompts.

In the revised paper we will report recovery at equal output quality and the absolute concept rates alongside Table 2 in the main text. We will also state explicitly that only the direction is estimated from the source modality, while the strength is calibrated on target-modality outputs.

Q2: Which concepts transfer reliably, and uncertainty on AUC and recovery

We computed 95% bootstrap intervals over the 20 prompts for every concept, for the AUC of the transferred vector (Table A) and for recovery (Table B), both as defined in the paper. The two tables are in our second comment below, and we will add them to the appendix.

Absolute AUC. The differences between concepts that transfer well and those that do not are much larger than the uncertainty: intervals are typically within ±0.10, while, for example, image→text transfer in UniAR is 0.82 [0.74, 0.89] for emotion and 0.11 [0.06, 0.16] for size. Concepts with similar AUC do overlap, so we do not claim a strict ranking among them.

Recovery and small denominators. We agree that recovery is less certain when intra-modality steering is weak. The clearest case is chaos for text→image: in Liquid, intra-modality AUC is 0.23 and recovery is 1.73 [1.26, 2.44]; in Emu3.5, intra-modality AUC is 0.20, which sits at our exclusion threshold, and the ratio would be 3.51 [2.82, 4.44]. Still, recovery above one is not only a small-denominator effect: in Emu3.5, the intervals for age, emotion, depth and size lie entirely above 1.

Seeds. We have one generation per prompt and strength, so we cannot report variation across seeds. This was a budget decision: our evaluation already covers 134,400 judged pairs (8 concepts × 4 settings × 7 layer configurations × 10 strengths × 20 prompts, for 3 models), and we prioritized breadth over repeated seeds.

We are happy to answer any further questions.

AuthorsOpenReview ↗

Detailed Tables

Tables for Q2 (continued from our reply above)

Values are per concept with 95% bootstrap intervals over the 20 prompts, at the middle layer range. T→I = text→image, I→T = image→text.

Table A: Transferred-vector AUC

T→I UniART→I Emu3.5T→I LiquidI→T UniARI→T Emu3.5I→T Liquid
Color1.00 [1.00, 1.00]0.81 [0.76, 0.86]0.87 [0.81, 0.92]0.21 [0.13, 0.30]0.22 [0.15, 0.30]0.07 [0.03, 0.11]
Age0.94 [0.88, 0.98]0.74 [0.63, 0.86]0.70 [0.53, 0.86]0.52 [0.43, 0.61]0.24 [0.17, 0.31]0.10 [0.05, 0.16]
Emotion0.92 [0.89, 0.96]0.60 [0.50, 0.68]0.51 [0.41, 0.62]0.82 [0.74, 0.89]0.74 [0.70, 0.79]0.49 [0.43, 0.54]
Cleanness0.82 [0.74, 0.90]0.32 [0.26, 0.38]0.57 [0.47, 0.68]0.57 [0.51, 0.64]0.22 [0.15, 0.28]0.20 [0.12, 0.28]
Depth0.80 [0.75, 0.86]0.72 [0.66, 0.79]0.37 [0.30, 0.44]0.66 [0.61, 0.72]0.26 [0.20, 0.32]0.36 [0.29, 0.44]
Chaos0.76 [0.68, 0.82]0.70 [0.61, 0.80]0.39 [0.31, 0.48]0.71 [0.65, 0.76]0.46 [0.40, 0.53]0.39 [0.33, 0.45]
Size0.61 [0.50, 0.72]0.64 [0.58, 0.70]0.56 [0.49, 0.63]0.11 [0.06, 0.16]0.32 [0.25, 0.39]0.12 [0.08, 0.16]
Spatial0.03 [0.00, 0.08]0.14 [0.05, 0.24]0.10 [0.04, 0.17]0.00 [0.00, 0.02]0.00 [0.00, 0.00]0.00 [0.00, 0.02]

**Table B: Recovery (blank where intra-modality AUC is below 0.2, as in the paper, because the ratio is not meaningful there)

T→I UniART→I Emu3.5T→I LiquidI→T UniARI→T Emu3.5I→T Liquid
Color1.00 [1.00, 1.00]0.87 [0.81, 0.93]0.96 [0.91, 1.02]0.47 [0.29, 0.68]0.76 [0.53, 1.03]–
Age1.29 [1.11, 1.59]1.64 [1.36, 2.09]1.79 [1.43, 2.31]0.74 [0.62, 0.84]0.47 [0.35, 0.61]0.21 [0.10, 0.35]
Emotion1.05 [1.00, 1.11]1.18 [1.05, 1.33]1.03 [0.84, 1.22]0.99 [0.93, 1.05]1.02 [0.93, 1.11]0.54 [0.47, 0.61]
Cleanness0.94 [0.86, 1.01]0.89 [0.64, 1.19]1.40 [1.16, 1.74]0.92 [0.81, 1.04]0.33 [0.24, 0.43]0.44 [0.29, 0.60]
Depth0.85 [0.79, 0.91]1.35 [1.16, 1.57]0.79 [0.61, 0.99]0.96 [0.86, 1.10]0.56 [0.45, 0.68]0.59 [0.48, 0.70]
Chaos0.77 [0.69, 0.85]–1.73 [1.26, 2.44]0.90 [0.83, 0.99]0.50 [0.43, 0.57]0.57 [0.46, 0.69]
Size0.73 [0.59, 0.90]1.44 [1.19, 1.78]1.34 [1.18, 1.55]0.18 [0.11, 0.26]0.50 [0.39, 0.61]0.36 [0.25, 0.48]
Spatial0.07 [0.00, 0.15]––––0.02 [0.00, 0.06]

Dear Reviewer,

Thank you again for your time and effort on this. Since we have uploaded a revised version of the paper yesterday, let us know if this adequately addresses your primary concerns. See the global response for some details what we changed and the PDF itself for the actual changes.

Reviewer_RufLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether a steering direction extracted from one modality can also control generation in another. It constructs contrastive mean-difference vectors from text or image pairs and applies them to text and image generation in three unified multimodal models: UniAR, Emu3.5, and Liquid. The experiments show bidirectional transfer for several concepts, with substantial variation across models, concepts, layers, and transfer directions. Additional qualitative examples examine formality, language identity, image resolution, and motion blur. A geometric analysis finds that some text and image concept directions align despite a modality gap, while their relationship to transfer performance depends on the evaluation metric and setting.

Critical strengths
  1. The central question is clear and experimentally testable. Shared parameters alone do not show that a direction can control generation across modalities. Injecting a source-derived direction into target-modality tokens provides direct intervention evidence. The distinction between separated modality representations and partly aligned concept directions is also useful.

  2. The main empirical contribution extends beyond selected examples. The study tests both transfer directions in three models, including free-form text generation, reports unsuccessful concepts, and validates its concept judge against human annotations. Relative to the related work discussed in Section 4, the useful contribution is the bidirectional generative evidence and its dependence on model and concept, rather than a new steering estimator.

Critical weaknesses
  1. The strength-calibrated recovery metric needs a more qualified interpretation. Section 2.2 describes transfer without target-modality data, but Appendix A.4 uses target outputs from a few evaluation prompts to choose the strength range. The direction is transferred unchanged, yet the full intervention is target-calibrated. Recovery also compares normalized AUCs over different strength ranges, and the judge is explicitly instructed to score the concept even in degraded outputs. The separate quality analyses in Section 3.2 and Appendix A.7 are helpful, but do not establish how much of the reported recovery remains at comparable output quality. The distinction matters particularly when recovery exceeds one or image-derived vectors degrade text fluency more strongly.

  2. The introductory claim about geometry is stronger than the per-setting evidence. The statement that closer directions lead to better transfer is not consistently supported by Table 7. For example, Emu3.5 image-to-text has a Spearman correlation of 0.47 with recovery but only 0.04 with transferred-vector AUC; Liquid image-to-text similarly changes from 0.69 to 0.35. Section 3.4 acknowledges heterogeneity, but alignment predicting absolute transfer strength is a different conclusion from alignment correlating with a ratio that also depends on intra-modality steering. The evidence for the existence of aligned, transferable directions is stronger than the evidence for a general predictive relationship.

Questions for authors
  1. Could you explicitly scope “zero-shot” to transferring the direction, distinguish strength calibration from final evaluation, and show whether the main recovery conclusions hold under a common output-quality criterion? This would help interpret the cross-setting AUC ratios without requiring equal raw steering strengths.

  2. What geometric conclusion remains when using transferred-vector AUC instead of recovery in Table 7? Please reconcile the introductory statement with the model- and direction-dependent results, and distinguish the evidence for aligned directions from evidence that their alignment generally predicts transfer strength.

2 comments
AuthorsOpenReview ↗

Addressing main concerns

Dear Reviewer RufL,

Thank you for engaging with our work, and finding our RQ well-defined and useful!

W1/Q1: The strength-calibrated recovery metric needs a more qualified interpretation

Thank you for raising these points. We will try to address each of them and will focus on what we consider the core critiques.

Can we really call it zero-shot?

The direction transfers zero-shot, which was the primary scientific result our paper is about. This in itself is very interesting for the multimodality and interpretability literature. However, we agree that the steering strength is calibrated on target-modality outputs. In the revised paper we will explicitly scope "zero-shot" to the direction and state this calibration in Sec. 2.2. For a practitioner this does indeed matter.

Is the way we calculate recovery meaningful?

In the paper, we chose one strength range per model and setting by hand on a few evaluation prompts, shared across all concepts, so the range could not be tuned to favor any concept. Still, we agree that a comparison which does not depend on hand-chosen ranges is more convincing, and the metrics already in our paper allow one.

So for this rebuttal, we computed recovery by comparing the two vectors at equal output quality instead of over strength ranges we selected. For each vector, we take the steering strength at which output quality drops to 90% of the unsteered outputs (image coherence for images, fluency for text; the same quality judges as in App. A.7). Recovery is then the concept rate of the transferred vector at that strength, divided by the concept rate of the intra-modality vector at its own such strength.

UniAREmu3.5Liquid
Text→image, recovery in the paper (Fig. 3)0.841.231.29
Text→image, recovery at equal output quality0.971.281.28
Image→text, recovery in the paper (Fig. 3)0.740.590.39
Image→text, recovery at equal output quality0.850.520.49

Our conclusions are unchanged. Text-derived vectors steer image generation as well as image-derived ones in UniAR, and better in Emu3.5 and Liquid. Recovery above one is therefore not a product of degraded images. Image-derived vectors steer text generation less well than text-derived ones.

Thus, output quality gives an automatic criterion for the steering strength, which could in principle replace our manual choice of range; in the revised paper we will report recovery under this criterion alongside Table 2.

W2/Q2: The introductory claim about geometry is stronger than the per-setting evidence

We agree, and thank you for suggesting transferred-vector AUC as the metric here, in contrast to the recovery metric. We will revise the introductory sentence and the corresponding finding in Sec. 3.4, and separate two claims:

1. Aligned directions exist despite the modality gap. This is well supported for UniAR and Liquid: cosine similarity between text- and image-derived vectors reaches 0.60 and 0.51. 30 of 42 (concept, layer range) points lie more than two standard deviations above the random baseline.

2. Alignment predicts transfer strength. This is only a trend. Using transferred-vector AUC as you suggest, the Spearman correlation with cosine similarity is (UniAR / Emu3.5 / Liquid):

  • text→image: 0.80 / 0.42 / 0.32
  • image→text: 0.17 / 0.04 / 0.35

This metric gives a more consistent picture than recovery: all six correlations are positive, whereas with recovery two were negative. However, only UniAR text→image is significant on its own, and with 21 points per model we cannot establish a general predictive relationship.

We will therefore frame geometry as showing trends in some models and transfer directions, with mixed evidence overall. This raises an intriguing general question about steering vectors, which we will highlight in revisions: in Emu3.5, text- and image-derived vectors are nearly orthogonal (cosine at most 0.24), yet text-derived vectors steer images as well as image-derived ones. Related observations exist for text-only LLMs, where multiple near-orthogonal directions can steer the same behavior (Wollschläger et al., ICML 2025).

Overall, we would argue that our main contribution of the paper remains and has merit to the research community. The above revisions will additionally help make certain findings more robust.

We are happy to answer any further questions. If your primary concerns are addressed, we hope you consider reflecting this in your final rating.

Wollschläger, T., Elstner, J., Geisler, S., Cohen-Addad, V., Günnemann, S., & Gasteiger, J. (2025). The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence. Proceedings of the 42nd International Conference on Machine Learning (ICML). https://proceedings.mlr.press/v267/wollschlager25a.html

Dear Reviewer,

Thank you again for your time and effort on this. Since we have uploaded a revised version of the paper yesterday, let us know if this adequately addresses your primary concerns. See the global response for some details what we changed and the PDF itself for the actual changes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Update to Paper

Dear reviewers,

Thank you for the feedback so far. We have uploaded a revised version of the paper, where we focus on the most pressing aspects of each review. Changes are marked in blue:

  • Recovery at equal output quality (Sec. 3.2, new Table 2, App. B.1). We still use Fig. 3 as our main result and a more intuitive way to visually present our findings. However, in order to have a more systematic method for quantifying transfer, we now also compare transferred and intra-modality vectors at a steering strength where output quality drops to 90% of the baseline (unsteered). We report absolute concept rates next to recovery. Conclusions are unchanged. (Reviewers RufL and gSkG)
  • Scope of "zero-shot" (Sec. 2.2). We clarify that the direction of a transferred vector is estimated from the source modality alone, while strength is calibrated on target-modality outputs. (Reviewers RufL and gSkG)
  • Geometry (Introduction, Sec. 3.4). We now more distinguish two claims more clearly: aligned directions exist despite the modality gap, and alignment is only a partial predictor of transfer strength. We add the correlation with transferred-vector AUC and discuss Emu3.5, where nearly orthogonal vectors still transfer. (Reviewer RufL)
  • Uncertainty (App. B.2, new Tables 8 and 9). 95% bootstrap intervals over prompts for the AUC and recovery of every concept. (Reviewer gSkG)
  • Brief intuition: Why text-derived vectors steer images well (Sec. 3.2). (Reviewer Bu22)

Note: Tables 6 and 7 of the submission are now Tables 7 and 10.

We are happy to answer any other questions, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.