acceptodds
Under review as a conference paper at ICLR 2027

Cues as Compasses: Cross-Modal Condition Steering for Unified Multimodal Models

Abstract

Unified multimodal models perform image understanding and generation within one architecture, yet architectural unification does not guarantee that each task uses its cross-modal condition effectively. Analyzing attention at the decisive stage of each task, we find a common conditioning gap in two task-specific forms: before answering, the model revisits the image but spreads its attention diffusely, whereas during generation, attention to the instruction decays early, leaving later updates weakly constrained. We propose Cross-Modal Condition Steering (CMCS), a training-free framework that contrasts each condition with a targeted reference to adaptively decide where, how strongly, and when to steer according to task demands. For understanding, CMCS redirects the model's existing visual attention toward instruction-relevant regions, and accepts the resulting answer change only to the extent that the reference evidence supports it. For generation, it reinjects, for every semantic unit, the residual between predictions under the original prompt and a counterfactual that alters that unit, shaped spatially at a fixed magnitude and scheduled by the unit's stage-dependent prior. Without updating any parameters, CMCS dynamically adapts to the current query and answer without degrading the original model performance, improving accuracy on visual understanding benchmarks and strengthening prompt adherence and compositional alignment on image generation benchmarks for both backbones, with further improvements in instruction adherence for image editing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Unified Multimodal Models, Inference-Time Steering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_K6GKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper identifies a cross-modal conditioning gap in UMMs: visual attention is diffuse in understanding, while text attention decays rapidly in generation. To address this, the authors propose CMCS, a training-free framework that contrasts the original condition with task-specific references to decide where, how strongly, and when to steer. For understanding, it redirects visual attention toward instruction-relevant regions; for generation, it reinjects the instruction’s semantic residual along the sampling trajectory with stage-dependent strengths. The authors evaluated CMCS on two UMMs: BAGEL and Qwen-Image across understanding, T2I generation, and image editing, with ablation studies showing the role of condition-relevant localization.

Critical strengths

The paper proposes a training-free, input-adaptive steering framework that uses task-specific references to decide where and how strongly to intervene, without updating model parameters. This design is conceptually clean and could be practically useful for frozen unified multimodal models, as it avoids retraining and can be applied at inference time.

Critical weaknesses

However, I have several concerns:

  1. The motivation is not causally supported. The paper claims that diffuse visual attention in understanding and decaying text attention in generation are problematic, and that steering attention toward instruction-relevant regions or sustaining it over time should improve performance. However, there is no evidence that this direction is actually beneficial. The authors provide no related work supporting this hypothesis, no before/after attention distribution visualizations showing that the intervention indeed makes attention more concentrated or persistent, and no oracle analysis showing what would happen if the attention distribution or generation trajectory followed the proposed direction. For example, if we could force the attention to be concentrated as the paper suggests, would the model consistently improve across these benchmarks and other tasks? Without such validation, the motivation remains speculative rather than grounded.

  2. The improvements are incremental and scattered. In the main results (Table 1), CMCS does not show significant or consistent gains over the native backbone across all tasks. Some improvements are very small—e.g., MathVista +0.3, MME and SEEDBench-Img nearly unchanged, and GenEval even slightly negative for BAGEL. These marginal differences make it difficult to tell whether the gains are due to the proposed mechanism or simply random variation. The lack of statistical significance tests further weakens the claim that CMCS meaningfully improves condition utilization.

  3. The experimental comparison is incomplete. Table 1 contains many blank entries for baseline models, with no explanation of whether the missing results are due to unreported numbers, incompatible protocols, or unsupported tasks. More importantly, the paper only compares CMCS against the native backbone and simple spatial controls (uniform/random), but does not compare against other training-free methods mentioned in related work, such as VCD, SEGA, PASTA, or CFG variants. Without these comparisons, it is unclear whether the proposed reference-based steering is actually more effective than existing training-free intervention strategies.

  4. Generalizability is not established. The method is only evaluated on two unified models, BAGEL and Qwen-Image, which share similar flow-matching and hybrid architectures. The paper claims consistent improvements on both, but this limited evidence does not demonstrate broader applicability. Additional experiments on other unified multimodal models, such as BLIP3o or Ovis-U1, would be necessary to show that CMCS generalizes across different architectures and training paradigms. Without such results, the method’s robustness and practical value remain uncertain.

Questions for authors

See the weaknesses above.

Reviewer_pkeaOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper first analyzes BAGEL model’s attention and found that the model allocate attention over image diffusely before answering queries and the attention to key instruction constraint decays during generation, indicating the model is not paying attention to more useful part of the input for answering questions. Then, to mitigate the issue, the authors propose Cross-Modal Condition Steering that intervenes on key input constraint to examine attention to which visual position is causally affected and how answer logit changes and steer model attention or sampling trajectory accordingly. The method results in moderate improvement in performance over baselines.

Critical strengths
  • The paper provides solid attention analysis of BAGEL model and shows that the model has diffuse attention over visual information during understanding and less attention to crucial textual constraints during generation
  • The paper proposes a training-free, cheap method to dynamically decide when and where to steer model’s attention based on how a key text constraint causally affects model’s attention and output logit.
Critical weaknesses
  • The estimator / optimization target is a proxy. The link to the downstream performance is assumed rather than tested. CMCS steers attention allocation (understanding) and velocity residuals (generation). The only connection to accuracy is the hypothesis that the model’s evidence and its counterfactual direction are already correct, otherwise the steering shouldn’t necessarily be helpful to model’s downstream performance / final answer correctness. This might explain why the steering does not consistently improve or only result in very moderate improvement in performance comparing to the baseline.
  • The work is somewhat incremental. There are already methods for steering attention during inference time to improve performance but are not included in the experiment result table. The distinguishing claim made is that the counterfactual contrast gives a principled where/how-much signal, but given the final evaluation metric is still performance, other related methods should still be run (or could have included stronger baselines such as simply allocating more attention to the visual component that are related to the input question).
Questions for authors
  • Is the attention analysis done on cases where the model generate correct answers? If not, how does the attention distribution differ for correct and incorrect cases? The method is built based on the attention analysis, so it would be nice to have the analysis to be more tied to downstream performance to have a better estimator. Maybe a more finegrained analysis might reveal that attention might not always be a good heuristics.
  • Is there a reason for why the estimator is not designed to directly optimize for downstream performance (if I understood correctly), given your final evaluation metric is still around performance? If it’s not, then I would expect having an intermediate proxy to be less effective at steering model performance, even though you did steer the model to pay more attention to regions that are causally affected by unit intervention in the input.
Reviewer_7u4XOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Cross-Modal Condition Steering (CMCS), a training-free framework for improving condition utilization in multimodal understanding and generation. For understanding, the method constructs question-dependent spatial maps by contrasting the original question with neutral-anchor variants, redistributes the final answer query’s visual attention while preserving its total visual attention mass, and uses a logit-residual-based rule to determine how much of the resulting prediction change to accept. For generation, CMCS uses semantic-unit attention maps and stage-dependent priors to spatially and temporally modulate the residual obtained from classifier-free guidance. The method is evaluated on BAGEL and Qwen-based components across visual understanding, text-to-image generation, and image editing.

The problem is relevant, and the paper presents promising results, particularly on MMBench. However, the current submission contains inconsistencies in the method description and result interpretation, and the experiments do not sufficiently establish the contribution of the proposed components over simpler or existing inference-time interventions.

Critical strengths
  1. Relevant problem and practical setting. Improving the use of existing cross-modal conditions without updating model parameters is a worthwhile direction. The paper provides a concrete implementation for both autoregressive understanding and flow-based generation.

  2. Useful separation of spatial allocation and intervention magnitude. Preserving the visual attention mass in understanding and matching the per-unit injection RMS in generation make the intervention more interpretable. The uniform and random spatial controls are more informative than a simple comparison between native inference and the complete method.

  3. Promising improvements on some understanding tasks. Table 4 reports MMBench improvements from 87.83 to 88.99 for BAGEL and from 82.00 to 83.93 for the Qwen understanding component. CMCS also outperforms the spatial controls in these settings, providing evidence that question-dependent routing can be useful.

  4. Substantial implementation documentation. The appendices include pseudocode, derivations, backbone-specific insertion points, and implementation settings. The paper also reports regressions and unsuccessful qualitative examples, which helps delineate the method’s limitations.

Critical weaknesses

1. The claimed generation mechanism is inconsistent with the actual algorithm.

The abstract states that, for each semantic unit, the method uses the prediction residual between the original prompt and a counterfactual that modifies that unit. However, Section 3.3, Algorithm 2, and Equation (22) use a single residual,

rt=vtF−vt∅,r_t = v_t^{F} - v_t^{\varnothing},

shared by all semantic units. The units differ in their spatial maps and stage-dependent weights, rather than in independently computed counterfactual residuals.

This distinction affects both the novelty claim and the interpretation of semantic specificity. At a fixed state, before native renormalization, Equation (22) can be collected into

vt′=vt+rt⊙Wt,v'_t = v_t + r_t \odot W_t,

where WtW_t is a nonnegative spatial multiplier obtained by combining the unit-specific maps and coefficients. Thus, the implemented method spatially and temporally reweights a shared CFG residual; it does not separately identify each unit’s counterfactual prediction direction.

Figure 2 also describes visual-token enhancement and text-attention reweighting, whereas the equations describe answer-query attention redistribution and velocity-residual modification. The abstract, diagram, and algorithm should be reconciled. The implemented approach may still be useful, but its contribution needs to be stated accurately.

2. The comparative evaluation does not establish the incremental benefit over relevant inference-time methods.

The scores of other foundation models in Tables 1–3 provide useful context, but they do not establish whether CMCS improves over alternative interventions applied to the same backbone. Tables 4–5 primarily compare native inference, uniform routing, random routing, and CMCS.

For generation, a particularly important comparison is against appropriately selected CFG scales and simple time-dependent or interval-based guidance. Guidance scheduling is already known to affect generation quality [1], and the shared-residual formulation makes these comparisons directly relevant. The existing uniform control is useful, but it retains much of CMCS’s strength computation and eligibility logic and therefore does not replace an independently configured simple guidance baseline.

For understanding, a matched-backbone comparison with a relevant attention intervention, such as Visual Attention Redistribution [2], or an appropriate contrastive method would help establish the value of the anchor-derived maps and acceptance rule.

These comparisons should also report practical inference cost. The appendix discloses that understanding requires the original evaluation, one anchor evaluation per unit, and an additional review evaluation. Actual latency and peak memory are needed to assess the performance–cost tradeoff.

3. The ablations primarily validate spatial routing, while other central components and mechanistic claims remain insufficiently tested.

The paper emphasizes where, how strongly, and when to intervene. However, the reported ablations mainly vary the spatial maps. They do not separately establish the contributions of the semantic-type phase priors, the adaptive strength factors, or the residual-supported decision rule.

A small set of targeted comparisons would be informative: a shared schedule for all semantic types, shuffled type-specific schedules, fixed versus adaptive intervention strength, and removal of the understanding acceptance rule. Where applicable, the intervention budget should be matched so that schedule effects can be distinguished from changes in total guidance magnitude.

The attention diagnostics also admit alternative explanations. High visual-attention entropy may be appropriate for questions requiring distributed evidence, and declining direct attention to text does not necessarily imply that textual constraints have ceased to influence the latent state. The attention curves motivate the method, but do not by themselves establish the proposed causal explanation. The mechanistic claims should be qualified or supported by more direct interventions.

4. The statistical support for several gains is limited, and some performance claims contradict the reported results.

Several improvements are small. In Table 5, BAGEL’s Soft-TIFA AM increases from 66.20 to 66.38, while GM increases from 19.10 to 19.60. The additional GM gain over the uniform control is only 0.15. MathVista improves by 0.3 percentage points on each backbone. Without paired uncertainty estimates or repeated generation seeds, it is difficult to determine how stable these differences are. This does not establish that the gains are insignificant; the current reporting simply does not resolve their reliability.

Two statements also require correction:

  • The abstract’s claim of improving performance “without degrading the original model performance” is too strong. For example, BAGEL’s GenEval score decreases from 0.82 to 0.81, and Tables 1 and 3 contain additional regressions. The projection and norm constraints do not provide an accuracy-preservation guarantee.
  • Section 4.2 states that Qwen-Image gains approximately 0.11 points across all editing metrics. Table 3 instead reports mixed changes, including English perceptual quality decreasing from 7.62 to 7.51, English overall score decreasing from 7.63 to 7.61, and Chinese semantic consistency decreasing from 7.68 to 7.62.

The conclusions should reflect these tradeoffs. Reporting paired confidence intervals and the numbers of understanding examples corrected versus newly corrupted would make the evidence substantially more informative.

5. The generation benchmark versions and evaluation configuration need clarification.

Section 4.1 and Appendix D.2 describe 800 compositional prompts evaluated using Soft-TIFA AM/GM, while referring to GenEval and citing Ghosh et al. (2023). This description appears to correspond to GenEval 2 [3], rather than the original detector-based GenEval protocol.

Please clarify whether Table 2 uses the original GenEval while Table 5 and Figures 1 and 3 use GenEval 2. The exact prompt sets, number of generated images per prompt, seeds, and evaluation scripts should be specified for each experiment, with the appropriate references.

The Soft-TIFA judge also needs an exact checkpoint identifier and scoring configuration. The text names Qwen3.8-27B, while the associated reference names Qwen3.8-Max. This is a reproducibility ambiguity that should be resolved. Providing the semantic-unit extraction prompt and token-alignment procedure would further improve reproducibility.

References

[1] T. Kynkäänniemi, M. Aittala, T. Karras, S. Laine, T. Aila, and J. Lehtinen, “Applying guidance in a limited interval improves sample and distribution quality in diffusion models,” in Advances in Neural Information Processing Systems, vol. 37, 2024.

[2] S. Kang, J. Kim, J. Kim, and S. J. Hwang, “See what you are told: Visual attention sink in large multimodal models,” in International Conference on Learning Representations, 2025.

[3] A. Kamath, K.-W. Chang, R. Krishna, L. Zettlemoyer, Y. Hu, and M. Ghazvininejad, “GenEval 2: Addressing benchmark drift in text-to-image evaluation,” arXiv:2512.16853, 2025.

Questions for authors
  1. Does the implementation use any unit-specific counterfactual generation predictions, or do all units share the native CFG residual as specified in Algorithm 2? Which description should be treated as authoritative?

  2. How does CMCS compare with appropriately configured CFG scale/schedule baselines and a relevant understanding intervention on the same backbones, including actual inference cost?

  3. How much of the improvement remains after simplifying the semantic phase priors, adaptive strengths, and understanding acceptance rule? Are the key differences robust under paired uncertainty analysis and multiple generation seeds?

  4. Which benchmark versions and judge checkpoints were used for each table and figure, and can the authors reconcile the editing-results discussion with Table 3?

Reviewer_m4jpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses attention cues and spatial-temporal steering to improve the use of conditioning information in frozen unified multimodal models across understanding, generation, and editing. It studies whether existing multimodal capabilities can be used more effectively without retraining.

Critical strengths
  1. The method has practical appeal across different backbones, with meaningful gains in localized understanding and some editing tasks. The positive results should not all be characterized as negligible.

  2. Matched checkpoints, initial noise, and evaluation, together with uniform and random spatial controls, support the usefulness of condition-relevant localization. The generation branch reuses existing CFG predictions without additional model forward passes.

Critical weaknesses
  1. Attribution of the more complex components remains incomplete. A native CFG baseline is already included, but spatial ablations do not separately isolate phase priors, state-adaptive steering strength, and gating. Simple dose-matched CFG or temporal-schedule controls would help distinguish changes in overall guidance from the contributions of individual components. The understanding branch also needs measured latency.

  2. Improvements are not universal. Some generation and editing metrics decline slightly, and uncertainty analysis is missing for the main results, so absolute claims such as no performance degradation should be narrowed. The abstract's per-unit counterfactual residuals also need to be reconciled with the algorithm's shared full-prompt CFG residual to make the intervention mechanism clear.

Matched spatial ablations and concrete task-level gains provide positive evidence. The main limitations are mechanistic attribution and robustness of the gains, not an established algorithmic error. A few small negative changes or the absence of exhaustive component ablations alone do not negate the overall contribution.

Questions for authors

Please see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.