- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how tokens from different modalities represent and pass information in reference-conditioned image generation. Through activation patching and attention knockout, it shows that text tokens carry not only editing instructions but also reference-image properties such as color and style, while the direct reference-to-image pathway plays a larger role in preserving identity. The authors also use these findings to correct failures in preserving the reference style.
Critical strengths
-
The research question and findings are insightful. Shared attention does not necessarily give different modalities the same role. Understanding these roles can help explain and control image editing.
-
The experiments provide complementary evidence: they test what reference information can be decoded from text tokens, which properties are lost when connections are blocked, and which properties transfer through activation patching.
-
The style-repair experiments demonstrate a practical use of the findings.
-
The paper is well written, and the figures make the experiments and results easy to follow.
Critical weaknesses
-
The factors determining pathway preferences remain unclear. The differences between color, style, and human identity may reflect a distinction between global and fine-grained information. The experiments show different pathway preferences, but do not establish whether these depend on whether the information is language-describable or whether it is global or fine-grained.
-
The repair experiments only partially test the proposed mechanism. The paper analyzes pathways for color, style, and identity, but only demonstrates style repair by strengthening reference-to-text attention. It remains unclear whether similar interventions can correct color errors or whether strengthening the direct reference-to-image pathway can improve identity preservation.
Questions for authors
-
Can the authors distinguish the language-describable explanation from the global/fine-grained explanation? One possible test is to use photos of celebrities whom the model can recognizably generate from their names. During editing, omit the names in text prompts and block reference-to-image attention. If celebrity identity survives, especially more reliably than in the existing experiments with ordinary people, this would support a role for language describability in pathway preferences.
-
Can the authors extend the repair experiments to color and identity? Specifically, does stronger reference-to-text attention correct color errors, and does stronger reference-to-image attention improve identity preservation? Do these interventions affect instruction following?
-
What happens if both padding and filler text are removed, leaving only the instruction and necessary special tokens? Does reference information shift to the instruction tokens, and does this affect generation quality or color/style preservation?