- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces IsoProbe, a data-free method designed to analyze and interpret how cross-attention heads write information from text tokens into the visual feature stream. It trains a sparse transcoder using isotropic Gaussian probes on the fixed OV operators of each cross-attention head in text-to-image diffusion models, thereby obtaining an overcomplete dictionary of "writing directions."
Critical strengths
- Treat the OV path of cross-attention as a fixed linear operator and directly learn a sparse dictionary on it, distinguishing this approach from the common practice of training SAEs on generated activations.
- Cover object ablation, style injection, object patching, gender bias, and safety benchmarks, while extending the analysis to multiple architectures.
Critical weaknesses
- Appendix B.3 shows that even without editing the sparse codes simply by performing a full replacement of all 195 heads, the output changes significantly: the LPIPS score is approximately 0.253, and the CLIP score drops from 31.40 to 31.07. This indicates that "no-edit substitution" inherently alters the image. Consequently, the changes in objects or styles observed in Figures 4, 5, and 6 may stem partly from transcoder approximation errors rather than the functional contribution of the selected sparse codes. Although Appendix E.3.2 employs an intervention that preserves the original attention output while merely adding or subtracting the decoded code change, many of the main text's conclusions still rely on substitution-based interventions.
- Object ablation is primarily based on a small number of prompts (e.g., "apple"/"pear") and the selection of top-exclusive sparse features, which are then ablated across all token positions. While Appendix E.3 extends the analysis to 23 object pairs, the magnitude of the drop in category margin remains small when using a single selection prompt; the LPIPS score for non-target objects under the selected-code intervention is 0.254, which is far higher than the 0.118 observed in the random control.
- While IsoProbe has the advantage of being data-agnostic, actual CLIP text embeddings are highly anisotropic. Therefore a discussion involving real CLIP models and actual prompts is necessary.
- The term "concept-associated" is insufficient to demonstrate a clear causal mapping between a single dictionary element and a specific concept.
Questions for authors
see weakness
3 comments
W2-1 Rationale for intervening at all text-token positions
Section 4.3 tests whether the object affected by an intervention depends on the selected code set, without separately specifying target-token positions. When intervention is restricted to the target token, the observed change could be interpreted as attenuation of the explicitly designated word's representation. We select codes from the activations of the apple and pear tokens, then suppress the selected sets at all text-token positions in the 10 shared heads. Non-target tokens are therefore not protected by exclusion from the intervention, and the same code-suppression rule applies at every position.
W2-2 Effect sizes for codes selected from a single prompt
In Appendix E.3.1, the code sets and 15 shared heads selected from a single prompt for each object pair are held fixed during evaluation on held-out prompts. Table 5 reports the mean category margins in these evaluation contexts.
For a given object crop, the category margin is the difference between the cosine similarities of its CLIP image embedding to the text embeddings of the two category descriptions. For example, for a cat crop, its similarity to "a photo of a dog" is subtracted from its similarity to "a photo of a cat." This measures relative similarity to the two categories, rather than a probability or removal success rate.
The mean margin decreases from 0.0552 before intervention to 0.0263 after suppressing the corresponding concept's codes. Suppressing the other concept's codes and applying the active-random control yield margins of 0.0503 and 0.0532, respectively. The reduction of 0.0289 should therefore be interpreted relative to the observed pre-intervention category margin and the control responses. The mean margin is reduced to approximately half its original value, but this does not quantify the fraction of concept information removed or imply a category change in every scene.
W2-3 Concept-specific responses and non-target appearance changes
As noted in the review, Appendix E.3.1 reports higher non-target LPIPS for selected-code interventions, at 0.254, than for active-random controls, at 0.118. Selected-code interventions produce larger target-category responses and also greater changes in non-target appearance. The controls match decoded write magnitude for each head, but do not match the target effect at the image level. This comparison therefore does not assess appearance preservation at an equal target change. It supports concept-specific functional responses of the selected codes, without establishing precise preservation of non-target appearance.
W3 Validation on real CLIP embeddings and prompts
The anisotropy of real CLIP text embeddings and the extent to which a finite caption dataset covers inputs encountered in practice are distinct issues. We examine whether OV approximation extends from inputs frequent in the caption training data to real inputs that are rare or absent from it. IsoProbe approximates the fixed OV operator across diverse input directions without restricting training to the distribution of a particular caption collection.
Figure 2c,d illustrates these differences using contextual token states from the actual CLIP text encoder. The COCO transcoder achieves higher fidelity on the frequent tokens shown, which are drawn from held-out COCO captions. IsoProbe achieves higher fidelity on the selected COCO-unseen tokens from P4D and Ring-A-Bell, with a larger gap between the two approaches in this group.
We also evaluated image generation from real prompts. Using COCO-1K captions excluded from COCO transcoder training, we measured distributional differences relative to unmodified SDv2.1 outputs with FID and semantic alignment between prompts and images with CLIP scores. Tables 2 and 14 report these results separately for no-edit reconstruction and safety-code suppression.
W4 Associations of individual elements and functional contributions of code sets
An activation-based association with a concept does not, by itself, establish the causal function of an individual dictionary element. We use these associations to select codes and assess the functional contributions of the resulting sets through interventions. Because contributions associated with one concept may be distributed across multiple entries, our primary unit of intervention is a code set. The abstract of the original submission explicitly refers to concept-associated code subsets, and Section 3.2 clarifies that dictionary elements are learned components introduced by the transcoder rather than literal units of the original model.
The causal evidence is provided by controlled interventions on selected codes. In Figure 4, varying the suppressed code set changes the corresponding apple or pear response while holding the transcoder, shared heads, and intervention scope across all text-token positions fixed. Style injection and object-code patching in Figure 5, together with progressive style suppression in Figure 6, produce content changes corresponding to the code manipulations. Appendix E.3.1 evaluates fixed code sets in new contexts against other-concept codes and active-random controls matched in write magnitude. Concept-associated object changes also occur when the original attention output is retained (Appendix E.3.2).
We would be happy to address any further questions or requests for clarification.
Thank you for your careful review and constructive feedback.
We have uploaded a revised manuscript in which additions and revisions are marked in blue.
W1-1 Approximation error and the functional contributions of selected codes
As noted in the review, Appendix B.3 shows that substituting transcoder outputs for OV writes can alter generated images even without code intervention. This evaluation replaces the OV writes of all 195 cross-attention heads in SDv2.1, so approximation errors across these heads jointly influence generation. Such errors may also affect the magnitude and appearance of intervention responses. Nevertheless, in the no-edit examples in Figure 10, the IsoProbe reconstructions retain the main objects and overall scene content of the original SD outputs, despite differences in visual details.
Replacing OV writes throughout the model with dictionary outputs allows us to investigate the roles of concept-associated components within shared heads. Within this approximation, we assess the contributions of selected codes by varying the code sets and intervention strengths while holding the transcoder fixed.
Figures 4-6, cited in the review, provide these comparisons. In Figure 4, suppressing apple or pear codes shifts the corresponding object toward the appearance of the other category, with the prompt, 10 shared heads, and intervention scope across all text-token positions held fixed. Figure 5 shows changes corresponding to style-code injection and dog-to-cat code patching. Figure 6 shows progressively weaker target-style characteristics as the number of suppressed entries increases.
Appendix E.3.1 compares interventions on the corresponding concept's codes, the other concept's codes, and active-random controls in the same 15 shared heads, using a common no-edit TC reconstruction as the reference. Active-random controls are drawn from active codes outside the union of the two selected sets and matched in decoded write magnitude for each head. Suppressing the corresponding concept's codes produces a larger decrease in its category margin than either control. The observation that reconstruction alters images is therefore insufficient, on its own, to explain these responses to code selection and intervention strength.
In the object experiments of Appendix E.3.2 in the original submission, we retain the original attention output and suppress the codes associated with one object or the other. The resulting object responses depend on which code set is suppressed (Table 6), showing that the functional correspondence between selected codes and generated content also occurs without full substitution.
W1-2 Extending original-output validation to gender and safety
Following a suggestion from another reviewer, we extended the original-output evaluation to gender and safety interventions on SDv2.1. We compared this path with TC substitution using the same code manipulations and initial noise.
The gender intervention examples newly added to the revised Appendix E.3.2 (Figure 19) show changes in the apparent gender presentation of the mechanic and firefighter according to the sign of the intervention along the gender-reference pattern. The safety examples added to the same section (Figure 20) compare suppression at N=50 and N=100 for four I2P cases. Reduced exposure or additional clothing is also observed when the original attention output is retained. These responses therefore occur without full substitution, although they are accompanied by other appearance changes.
Given your concern about reliance on full substitution, we would appreciate your assessment of the additional comparisons in Figures 19 and 20 of the revised Appendix E.3.2.