acceptodds
Under review as a conference paper at ICLR 2027

Probing Cross-Branch Compatibility in Unified Multimodal Models through Semantic Steering

Abstract

Unified multimodal models (UMMs) aim to integrate understanding and generation within a single architecture, yet these two capabilities are still largely evaluated in isolation. Strong performance on both tasks therefore tells us little about how their internal semantic representations relate to each other. A fundamental question is whether the semantic structures underlying understanding and generation are compatible across branches. To study this question, we introduce cross-branch semantic steering, an intervention-based framework that extracts semantic directions from one branch and applies them to the other. % We show that steering vectors learned from the understanding branch can transfer to generation, enabling controllable image synthesis and improved semantic faithfulness. In contrast, the reverse direction consistently shows limited effectiveness. Our analysis suggests that this asymmetry may be related to a practical representational mismatch: understanding-derived vectors capture transferable, object-centric semantics, while generation-derived vectors primarily encode low-level appearance features. Our results reveal that architectural unification does not guarantee semantic alignment, and establish cross-branch steering as a practical tool for probing multimodal representations.

Primary area:
interpretability and explainable AI
Keywords:
Unified multimodal models, Steering Method, Multimodal Modal Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_W4LYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the semantic transferability between understanding and generation branches in unified multimodal models (UMMs). The authors propose cross-branch semantic steering, which extracts layer-wise steering vectors from the hidden states of one branch using contrastive samples and injects them into the other branch during inference. Moreover, they construct UMMSteer, a contrastive dataset covering 51 concept pairs across seven semantic categories, to extract steering vectors and evaluate their cross-branch transferability. Experiments show that steering vectors from the understanding branch can effectively control image generation, while those from the generation branch fail to meaningfully influence understanding. The authors analyze this asymmetry through subspace alignment and self-steering experiments, suggesting differences in the semantic representations learned by the two branches. Additional experiments across different UMM architectures examine the generalizability of these findings and explore the potential of steering to improve semantic faithfulness in image generation.

Critical strengths
  1. Asymmetric cross-branch transferability. Steering vectors extracted from the understanding branch can effectively control image generation, while those extracted from the generation branch fail to influence understanding. This finding provides insights into the relationship between understanding and generation in UMMs.

  2. Comprehensive experiments. The authors evaluate 51 concept pairs across seven semantic categories, multiple steering methods, and different UMM architectures. Additional ablations and control experiments further support the effectiveness of understanding-to-generation steering.

Critical weaknesses
  1. Lack of Novelty. The paper's main findings regarding the representational differences and limited alignment between understanding and generation largely overlap with prior work. UniFork [1] has analyzed the different layer-wise representation alignment patterns of understanding and generation, while GapEval [2] has revealed inconsistencies between the two capabilities and limited cross-task knowledge transfer. Although both works are cited, the paper does not adequately discuss how its findings differ from or extend these studies. While cross-branch steering provides a different experimental approach, its additional insights beyond prior work remain unclear.

  2. Insufficient evidence for the claimed representation mismatch. As shown in Figure 5(a), generation-derived steering vectors fail to reliably modify the target semantic attribute even when applied back to the generation branch. This raises the possibility that the observed Gen-to-Und failure is caused by ineffective semantic direction extraction rather than an inherent representational mismatch between the two branches.

[1] Li et al., UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation, 2025. https://arxiv.org/abs/2506.17202

[2] Wang et al., Quantifying the Gap between Understanding and Generation within Unified Multimodal Models, 2026. https://arxiv.org/abs/2602.02140

Questions for authors
  1. Please clarify the main differences and additional insights of this work compared with prior studies such as UniFork and GapEval.

  2. Could the failure of Gen-to-Und steering be due to ineffective steering vector extraction rather than representation mismatch? Please provide further clarification or additional experiments to distinguish these possibilities.

Reviewer_GVzkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces "semantic steering" as a way to poll mutual semantics from branches of a unified model with the objective of better understanding unified model generation. The steering vectors are extracted with mean differences (CAA) (Panickssery et al., 2024), principal components of difference vectors (Zou et al., 2025), and linear classifiers (Li et al., 2024). The paper heavily relies on BAGEL for practical implementation and experiments. The authors show that they are succesful in steering visual attributes, while generation vectors cant influence text QA answers.

Critical strengths

The research question is interesting and worthwhile studying and I do think the initial results are motivating of further study. With BAGEL, the authors demostrated success for color steering, and some success for counts, positions and styling. The main validation of the automatic evaluator is done with human annotations which is great. The size of the image study is significant (>300), but aggregation is probably hiding the weaker results for some categories.

Critical weaknesses
  1. There is some weakness to the main thesis of the paper, I am not sure from the paper experiments that the found assymetry is truly related to cross branch incompatibility. Actually, it might simply be related to inefective extraction of steer vectors. All the experiments, including appendix ablations boild down to linear strategies to steer vector extraction.

  2. There is also a missing control which is basically a generation steer vector that suceeds within generation but fails for understanding.

  3. The generation process evolves through denoising, so the effect of perturbations might simply not survive the denoising process.

  4. Multiple runs with different generation seeds should eb used to establish robustness across most results.

  5. it seems to me like the evaluation does not necesarily establishes the preservation properties that are claimed in the paper. SSR is supposed to check target-object presence and target-attributon satisfaction, but the sole object presence does not really guarantee that the exact same instance is preserved, or says anything about unrelated attributes or surrodungin scene.

Questions for authors

As per the critical weaknesses. In particular 1) 2), and 4), and comments on 5)

Reviewer_htDFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the understanding and generation branches of unified multimodal models have compatible semantic representations. It constructs UMMSteer, a dataset of contrastive concept pairs, and uses existing activation-steering methods to extract directions from one branch and apply them to the other. The main experiments on BAGEL show that directions extracted from text-only QA in the understanding branch can influence image generation, while generation-derived directions do not reliably change understanding-side predictions. The paper attributes this asymmetry to differences in the semantic information captured by the two branches and provides additional understanding-to-generation results on other models.

Critical strengths
  1. The paper addresses an interesting question. Evaluating understanding and generation separately does not establish whether their internal representations are compatible. Investigating this relationship through interventions is a useful direction.
  2. The successful understanding-to-generation transfer is an interesting empirical result. Directions extracted from text-only QA can influence several visual attributes during generation without retraining the model. The approach is simple, and the examples make the intervention easy to understand.
  3. The paper includes useful controls and additional analyses. These include random and unrelated directions, alternative generation-side extraction strategies, different intervention positions, and out-of-template compositional prompts. These efforts help characterize the effectiveness and limitations of the approach.
Critical weaknesses
  1. The understanding-side experiment has a narrower scope than the framing suggests. The source vectors are extracted from answer-token activations in text-only QA, without processing an image. Calling this U→G is reasonable at the level of functional branches, but it does not directly establish compatibility between visual understanding and image generation. The result could reflect the ability of textual semantic directions to influence a text-conditioned generator. The paper does not sufficiently distinguish this interpretation from a broader claim about understanding–generation semantic alignment.
  2. The evidence does not establish how broadly the directional asymmetry applies. The bidirectional analysis is primarily conducted on BAGEL, which explicitly separates understanding and generation parameters through its Mixture-of-Transformer-Experts architecture. The additional models are evaluated in the U→G direction, without corresponding G→U results, so these comparisons do not establish that the asymmetry generalizes across architectures. Moreover, the distinction between autoregressive and diffusion-based generation may not capture the most relevant differences. For example, MetaMorph (ICCV 2025) (https://www.openaccess.thecvf.com/content/ICCV2025/papers/Tong_MetaMorph_Multimodal_Understanding_and_Generation_via_Instruction_Tuning_ICCV_2025_paper.pdf) predicts semantic visual features used for understanding and delegates pixel synthesis to a separate diffusion decoder. Such designs raise the possibility that transferability depends on the visual representation and intervention site, rather than on the understanding–generation distinction itself. The paper’s broader architectural conclusions are therefore stronger than the evidence supports.
  3. The reverse-transfer failure is difficult to separate from the quality of the extracted vectors. The generation-derived vectors do not reliably control the intended attribute even when applied back to generation. Their failure on understanding could therefore reflect ineffective semantic extraction, rather than an incompatibility between the branches. The alternative pooling strategies and intervention positions are useful controls, but do not fully resolve this ambiguity. Prior work such as CASteer (ICLR 2026) (https://proceedings.iclr.cc/paper_files/paper/2026/hash/435422305988b73c6cc00bcb29ba2531-Abstract-Conference.html) shows that contrastive generation activations can yield directions supporting effective concept erasure in diffusion models. Although the models and intervention setups differ, this makes the explanation that generation-derived directions mainly capture low-level appearance less conclusive.
Questions for authors
  1. What conclusions about understanding–generation compatibility are supported by the text-only extraction setup? My concern is that the experiment demonstrates transfer from textual semantics to image generation, while leaving the compatibility of visually grounded understanding and generation unresolved. How do the authors interpret this distinction?
  2. How broadly should the reported asymmetry and its architectural explanation be interpreted? The bidirectional evidence is largely specific to BAGEL. Models such as MetaMorph also organize generation around semantic visual features, with pixel synthesis handled separately. Could the observed asymmetry depend on expert separation, the representation being generated, or the intervention site, rather than reflect a general property of understanding and generation?
  3. What distinguishes ineffective semantic extraction from cross-branch incompatibility in the reverse-transfer results? Since the generation-derived vectors also provide weak G→G control, I am not yet convinced that their G→U failure identifies a mismatch between the branches. How do the authors reconcile their interpretation with prior work such as CASteer, where generation-derived activation directions support effective semantic control?
1 comment
Reviewer_htDFOpenReview ↗

final justification

Thank you for the responses and additional experiments. They address some of my concerns, but also show that the original conclusions do not hold across all the tested settings. UniPic-1 shows the opposite asymmetry, K-Steering enables reverse transfer for counting, and Nexus-Gen supports transfer in both directions at the shared visual interface.

I appreciate that the authors have revised their claims. However, it is still unclear why transfer succeeds in some settings and fails in others. It is difficult to obtain reasonable, concrete conclusions according to the current experiment results. I therefore maintain my weak reject rating. I encourage the authors to study these differences further and clarify what determines whether transfer works.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.