acceptodds
Withdrawn from ICLR 2027

Vision-Language Binding in In-Context Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In-context image generation models such as FLUX.2 take a text prompt and an optional reference image as visual conditioning for the output. Internally, the text, reference, and noise tokens are concatenated and processed through a single attention stream where all tokens can attend to one another, leaving open how reference information flows through the model to produce the output image. We show that an implicit cross-modal binding emerges between the text tokens and the reference image: the text tokens absorb visual reference content during the forward pass, and that absorbed content causally influences the generated output. We surface this binding with three causal interventions on FLUX.2: T2I Lens, which decodes intermediate text-token activations through a text-to-image path; Attention Knockout, which severs specific attention edges; and I2I-to-I2I Patching, which copies text token activations between editing runs. Across 2,875 editing tasks spanning SUN397, DreamBench++, and web-collected images, we observe a consistent division of labor: properties of the reference image, like color, style, and scene setting, are first written into the text tokens, which carry them to the generated image; pixel-exact properties like a specific face or instance identity bypass the text tokens and flow directly from reference to image through image-to-image attention. We further localize the binding to the semantically inert tokens of the text sequence and to a small set of binding layers, where amplifying reference-to-text attention repairs failing edits. These results show that text tokens in a multimodal DiT are not just prompt holders, but a structured channel for reference image content. More broadly, they suggest that even in unified-attention multimodal generative models, token modality structures how conditioning information is represented and routed across the network.

Primary area:
interpretability and explainable AI
Keywords:
Interpretability, attention knockout, diffusion transformer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MEG4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how tokens from different modalities represent and pass information in reference-conditioned image generation. Through activation patching and attention knockout, it shows that text tokens carry not only editing instructions but also reference-image properties such as color and style, while the direct reference-to-image pathway plays a larger role in preserving identity. The authors also use these findings to correct failures in preserving the reference style.

Critical strengths
  1. The research question and findings are insightful. Shared attention does not necessarily give different modalities the same role. Understanding these roles can help explain and control image editing.

  2. The experiments provide complementary evidence: they test what reference information can be decoded from text tokens, which properties are lost when connections are blocked, and which properties transfer through activation patching.

  3. The style-repair experiments demonstrate a practical use of the findings.

  4. The paper is well written, and the figures make the experiments and results easy to follow.

Critical weaknesses
  1. The factors determining pathway preferences remain unclear. The differences between color, style, and human identity may reflect a distinction between global and fine-grained information. The experiments show different pathway preferences, but do not establish whether these depend on whether the information is language-describable or whether it is global or fine-grained.

  2. The repair experiments only partially test the proposed mechanism. The paper analyzes pathways for color, style, and identity, but only demonstrates style repair by strengthening reference-to-text attention. It remains unclear whether similar interventions can correct color errors or whether strengthening the direct reference-to-image pathway can improve identity preservation.

Questions for authors
  1. Can the authors distinguish the language-describable explanation from the global/fine-grained explanation? One possible test is to use photos of celebrities whom the model can recognizably generate from their names. During editing, omit the names in text prompts and block reference-to-image attention. If celebrity identity survives, especially more reliably than in the existing experiments with ordinary people, this would support a role for language describability in pathway preferences.

  2. Can the authors extend the repair experiments to color and identity? Specifically, does stronger reference-to-text attention correct color errors, and does stronger reference-to-image attention improve identity preservation? Do these interventions affect instruction following?

  3. What happens if both padding and filler text are removed, leaving only the instruction and necessary special tokens? Does reference information shift to the instruction tokens, and does this affect generation quality or color/style preservation?

Reviewer_ScVzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates what role text tokens play when a reference image is included in the input. It applies three causal interventions to FLUX.2: T2I Lens, Attention Knockout, and I2I-to-I2I Patching. Through a series of experiments, the authors find that text states carry reference-image information. More specifically, color and style are primarily transmitted through text states, particularly padding positions, while identity preservation depends more strongly on direct attention from generated-image tokens to reference-image tokens.

Critical strengths
  1. A relevant question about reference conditioning. The paper investigates how reference-image information is routed through text and image states in multimodal diffusion transformers, addressing a useful question for understanding in-context image generation.
  2. Clear qualitative demonstrations. The attention-knockout and activation-patching examples clearly illustrate changes in reference color, style, and scene preservation, making the intervention effects easy to inspect.
Critical weaknesses

Limited conceptual novelty beyond existing work. The three interventions largely adapt established techniques, including Diffusion Lens (Toker et al., 2024), attention knockout (Geva et al., 2023), and activation patching (Heimersheim & Nanda, 2024). This is not inherently a weakness: existing tools can support a substantial contribution if they reveal important new mechanisms. However, the findings here appear to offer a limited conceptual advance. Reference-dependent information entering text states is a natural expectation under bidirectional multimodal attention, and prior work has already demonstrated visually enriched DiT text states (Dahary et al., 2026) and register-like information storage in padding positions during diffusion (Toker et al., 2025). The potentially distinctive contribution is therefore the proposed attribute-dependent routing: color and style through text states, versus identity through direct reference-to-image attention. Yet this division is less clear than the paper’s framing suggests. T2I Lens primarily reveals coarse reference associations, while Table 2 reports loss of identity preservation in 45.6% of cases under reference-to-text knockout, compared with 76.7% under reference-to-image knockout. These results support differing degrees of pathway dependence rather than a clean separation.

Questions for authors
  1. What mechanistic insight does the proposed attribute-dependent routing establish beyond the findings of Dahary et al. (2026) and Toker et al. (2025)?
  2. What motivated the focus on color, style, and human identity? Were other reference attributes, such as shape, material, or spatial layout, evaluated, and if so, did they show the same routing pattern?
Reviewer_FTxXOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how information from a reference image flows through the unified attention stream of an image-editing model. The authors use three complementary interventions: T2I Lens, which decodes intermediate text-token activations into images; Attention Knockout, which blocks selected attention paths; and I2I-to-I2I Patching, which transfers text-token activations between editing runs.

Critical strengths
  1. Complementary interventions provide a persuasive chain of evidence.

T2I Lens establishes that reference information can be recovered from text activations, Attention Knockout tests dependence on particular pathways, and I2I-to-I2I Patching demonstrates a causal effect of the transferred activations. In Table 3, patching all text tokens transfers color and style in 92.0% and 87.6% of cases, respectively. Appendix B further constrains the available reference-image pathways, strengthening the case for mediation through text tokens. This combination is the paper's most consequential contribution.

  1. Localizing the carrier of reference information adds explanatory value.

Patching padding tokens achieves results close to patching the entire text sequence, whereas patching instruction tokens has little effect. Appendix E further shows that the carrier changes when filler text replaces padding. These findings go beyond observing that text tokens contain visual information and begin to explain where that information resides. Repairing 31 style-transfer failures by strengthening the corresponding attention pathway also illustrates the practical value of the analysis, although broader validation is needed.

Critical weaknesses
  1. The central interpretation turns a preference for particular pathways into an overly absolute functional separation.

In Table 2, blocking attention from text queries to reference-image keys still destroys human identity in 45.6% of cases. The corresponding result from the second judge is 34.4%. These results are difficult to reconcile with the claim that blocking the text-side pathway barely affects identity. The effect could reflect general generation degradation or an indirect role of the text pathway, but distinguishing these explanations requires additional controls.

Likewise, failure to recover identity using a particular Lens or patching configuration does not establish that identity information is absent from text representations. It establishes a negative result for the tested interventions. Moreover, Appendix F reports only style-patching experiments for Qwen-Image-Edit. This supports causal style transfer through text activations, but does not establish that the same color/style-versus-identity division holds in both models. These issues directly affect the strength and scope of the paper's main mechanistic interpretation.

  1. Evaluation definitions and aggregation are insufficiently transparent, particularly for the key negative findings about identity.

Appendix G.2 states that the two knockout directions use questions with opposite polarities: one asks whether the property is lost, while the other asks whether it is preserved. Table 2 nevertheless reports destruction rates for both. Similarly, the human-identity question in Appendix G.1 asks whether the generated person is different, whereas Table 1 reports identity recovery. The paper should explicitly specify how raw judgments are converted into each reported metric.

The overall 91.1% inter-judge agreement also obscures weaker agreement in some important conditions. For example, agreement on human identity under direct reference-to-image knockout is 76.7%, while human-model agreement on the identity knockout conditions is approximately 74%-79%. Absolute negative claims about identity therefore require particular care. In addition, experiments reuse a relatively small set of reference people, style subjects, and colors; the paper should explain whether its uncertainty estimates account for the dependence among these repeated observations.

Questions for authors
  1. How do you explain the 45.6% identity-destruction rate when blocking the reference-to-text pathway? Please report results restricted to examples whose unmodified I2I baselines successfully preserve identity, and clarify whether matched generation-degradation controls distinguish identity-information loss from a general reduction in generation quality.

  2. Please specify how raw pass judgments are converted into recovery or destruction rates in Tables 1, 2, and D1, especially when the underlying questions have opposite polarities. Do the main results remain unchanged when all conditions are evaluated with a consistent question polarity?

  3. Are corresponding color and identity experiments available for Qwen-Image-Edit? If not, would you narrow the cross-model conclusion to the finding that text activations can causally transfer style?

  4. What are the per-condition results after human regrading? Do the main differences and their uncertainty estimates continue to support the interpretation when resampling is grouped by reference person or style subject?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.