- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors use mechanistic interpretability techniques to study autoregressive image generation models. Their goal is to localize visual functions, internal processes like transmitting information from the prompt to the image, or binding color to within the boundaries of an object. They do this in a 2-step process for each target visual function. (1) they localize to a set of generated tokens, by manually coming up with a shortlist of candidate token sets (like the boundary of the object), and patching over the attention head outputs for every head of every layer in that set of tokens and seeing if the generated output image’s color changes. (2) they then localize to a sparse-ish set of attention heads by learning a soft mask over the attention heads when they do patching, supervised by a next token loss rather than by the final entire output. They use only single-object, plain background images, and use two autoregressive image generation models, Janus Pro and Emu3. The localized interventions seem reasonably successful at transferring color and shadow properties of the objects, though it’s not clear if they exactly represent the stated visual functions.
Critical strengths
- The paper is, to the best of my knowledge, the first paper to apply mechanistic interpretability for localizing internal computations to autoregressive image generation models. It is debatable whether or not what they found is a circuit, but it is certainly a step in the right direction of interpreting AR image generation models. Work on doing interpretability for models other than language models is generally healthy for the interpretability ecosystem.
- The paper provides many qualitative examples of the intervention outputs, making it easy to judge whether or not their interventions worked.
Critical weaknesses
-
The intervention experiments do not offer much evidence that the exact visual function has been captured. Fun. 1 to Fun. 3 all have the same final causal effect on the image of changing the object’s color, making it unclear that the localized heads are responsible for the claimed internal computation.
-
The authors do not discuss the prompts/data that they use anywhere in the main paper. We don’t know what the prompts looked like or how many of them there were. This is pretty critical for interpreting the quantitative results, like in Figure 5, especially given the lack of error bars, to know whether or not their methods worked. The main qualitative results seem to only cover color and shadow of single objects; preferably the results include a more diverse set of prompts.
More minor concerns:
- nearly all the figures are way too small, not readable without zooming in
- nit: lines 222-224 seem duplicated
- nit: line 266 incomplete sentence
Questions for authors
[Clarification] What components exactly of the method did the authors try to ablate in Figure 9b? The ablations seem to show that not every part of the method is necessary?
[Clarification] What are the differences between Fun.1, Fun.2, and Fun.3, and is there evidence showing that the localized heads capture these differences?
[Suggestion] The attention head masks still seem fairly dense. Do the interventions still work with a stronger sparsity regularization penalty on the masks?
9 comments
Weakness 2: Please clarify the prompts/data and sample sizes underlying Fig. 5, quantify uncertainty, and show greater prompt diversity.
Response:
Thank you for identifying where the experimental protocol needs to be made explicit. The evaluation uses manually constructed, controlled donor–recipient prompt pairs so that each intervention targets a specified visual change while shared conditions are held fixed.
- Prompt Construction, Sample Sizes and Diversity. For Fun. 1/3, donor and recipient describe the same object, view, size, background, and lighting, varying only the specified object color. For Fun. 2, object identity and composition remain fixed while object/background colors are swapped (e.g., red mug on green background green mug on red background), enabling evaluation of attribute ownership and background leakage. For Fun. 4, object identity, pose, and view remain fixed while lighting and shadow conditions vary. Fig. 5 evaluates 10 distinct donor–recipient prompt pairs for each function, covering 19 object categories and different color, object–background, and shadow variations. Representative prompt templates, with repeated constraints abbreviated, are:
Fun. 1/3: Studio photo of one {object_color} {object},
flat {background_color} background, fixed view, ...
Fun. 2: A single {object_color} {object}, centered,
plain {background_color} background, studio lighting, ...
Fun. 4: A single {object}, fixed view,
{lighting_and_shadow_condition}, ...
-
How the cases are evaluated. The generator remains frozen (Sec. 5.1). FCP first selects function-relevant token positions (Sec. 4.3; Fig. 3); DML learns head weights with those positions fixed (Sec. 4.4; Eq. 4); the retained Top- configuration is then evaluated on full generations against the corresponding regional and head controls (Sec. 4.5; Fig. 5). Fig. 5(a) measures target/reference regions, and Fig. 5(b) compares learned configurations with head controls. The metrics are defined per function rather than treating any color change as success.
-
Reliability of intervention effects. Fig. 5 evaluates all four functions using function-specific measurements, reference regions, and head controls, examining both the intended visual changes and their spatial selectivity. Fig. 8(c,d) further provides individual evaluation results and 95% confidence intervals under additional evaluation settings. Together, these complementary analyses support the reliability of the observed functional intervention effects.
Added experimental details to Sec. 5.1 of the revised manuscript.
Question 1: What is ablated in Fig. 9(b)? Are all components necessary?
Response:
Fig. 9(b) examines the contribution of individual DML objective terms (Eq. 4). No-control removes control-position supervision; No-margin removes the margin penalty; Source-only uses donor-token targets alone. All variants retain the FCP-selected positions, activation interventions, and DML mask learning (Secs. 4.2–4.4). All variants preserve positive functional effects, with Source-only achieving the highest displayed score. This demonstrates the robustness of the localization framework to simplified training objectives.
Question 3: The attention-head masks appear dense. Do functional interventions persist under stronger sparsity?
Response:
-
Sparsity is already incorporated into VFL. The regularizer in Eq. (4) penalizes total mask weight and encourages near-binary head selection (Sec. 4.4). Thus, the learned masks in Fig. 8(a) are already obtained under sparsity constraints. Our objective is to identify functionally meaningful head configurations, balancing sparsity with causal intervention effectiveness rather than maximizing sparsity alone.
-
Distributed computation motivates joint head localization. Transformer heads can interact through compositional circuits[1], while [2] identified a behavioral circuit comprising multiple interacting head groups. This motivates VFL's focus on joint head configurations rather than isolated heads (Sec. 1). Consistently, Fig. 8(e) shows non-uniform contributions from different head groups, supporting the importance of their collective functional effects. The resulting mask density therefore reflects a regularized, function-oriented localization objective.
References
[1] Elhage, N., et al. A Mathematical Framework for Transformer Circuits. Transformer Circuits, 2021, §“Two-Layer Attention-Only Transformers” → §“Three Kinds of Composition”.
[2] Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small. ICLR 2023, §3.1–§3.4, particularly §3.2 and §3.4.
Minor concerns: Small figures; duplicated and incomplete line.
Response:
Thank you. We have enlarged the relevant figure labels, removed the duplicated text, and completed the sentence specifying Fun. 2's operational token set. These revisions are reflected in Figs. 1, 2, 3, 5, 6, 8 and Lines 222–224, 266.
Response:
Thank you for asking us to distinguish functional localization from the appearance of an edited image. Color is a common experimental probe, while the three hypotheses concern different computational roles.
Different functions and observations (Sec. 3; Fig. 1). Fun. 1, Attribute transmission, concerns prompt-specified information entering the image-token sequence; Fig. 1(a) observes stronger prompt-dependent next-token differences early in generation. Fun. 2, Object-attribute ownership, concerns keeping an attribute associated with its object; Fig. 1(b) measures different hidden-state responses in object regions and far background. Fun. 3, Object-internal propagation, concerns the transfer of established information to later tokens of the same object; Figs. 1(c,d) examine within-object historical attention and local enrichment. These are three distinct internal observables rather than three names for color editing.
Different causal tests (Sec. 4.3; Fig. 3). FCP employs distinct intervention strategies: early-prefix positions for Fun. 1, object-neighborhood positions with equal-size spatial controls for Fun. 2, and future-object positions compared against onset/reference positions for Fun. 3. These tests distinguish early attribute transmission, spatial attribute ownership, and subsequent object-internal propagation, respectively. Their function-specific evaluation criteria (Sec. 4.3; Fig. 5(a)) further connect each intervention pattern to its hypothesized computational role.
From positions to heads (Secs. 4.4–4.5). Building on function-specific token positions identified by FCP, DML learns layer–head intervention configurations for each visual function. These configurations are evaluated through full autoregressive generation using function-specific measurements and causal controls (Figs. 5, 9). Together, the positional and head-level interventions provide complementary evidence that VFL localizes internal configurations associated with distinct computational roles.
In summary, the functions are distinguished by their generation-time signatures, intervention positions, and target/reference measurements. Their common color outcome is the controlled readout of different causal dependencies.