acceptodds
Under review as a conference paper at ICLR 2027

Sparse Decomposition of Cross-Attention OV Circuits via Isotropic Probing

Abstract

Understanding the semantic influence of text on image generation is essential for interpreting text-to-image diffusion models. Cross-attention provides a direct interface between text and the denoising network, but attention maps reveal where token information is routed rather than what each head writes into the visual feature stream. We focus on the output-value (OV) pathway, where the value and output projections compose into a fixed linear map from token embeddings to feature updates. Motivated by isotropic random measurements in matrix sensing, we introduce IsoProbe, which probes each head's OV operator with isotropic Gaussian inputs and fits a sparse transcoder to the resulting outputs. This data-free procedure yields an overcomplete dictionary of writing directions without requiring text or image samples. The resulting dictionaries approximate OV writes through input-dependent sparse combinations of learned directions. Prompt-based analyses identify concept-associated code subsets within shared heads. Injections, patches, and ablations link selected entries to changes in object, style, and attribute-associated content, providing a means to investigate their functional contributions.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, text-to-image diffusion, cross-attention, OV circuit, sparse transcoder, sparse dictionary learning, isotropic probing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mn1xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces IsoProbe, a data-free method designed to analyze and interpret how cross-attention heads write information from text tokens into the visual feature stream. It trains a sparse transcoder using isotropic Gaussian probes on the fixed OV operators of each cross-attention head in text-to-image diffusion models, thereby obtaining an overcomplete dictionary of "writing directions."

Critical strengths
  1. Treat the OV path of cross-attention as a fixed linear operator and directly learn a sparse dictionary on it, distinguishing this approach from the common practice of training SAEs on generated activations.
  2. Cover object ablation, style injection, object patching, gender bias, and safety benchmarks, while extending the analysis to multiple architectures.
Critical weaknesses
  1. Appendix B.3 shows that even without editing the sparse codes simply by performing a full replacement of all 195 heads, the output changes significantly: the LPIPS score is approximately 0.253, and the CLIP score drops from 31.40 to 31.07. This indicates that "no-edit substitution" inherently alters the image. Consequently, the changes in objects or styles observed in Figures 4, 5, and 6 may stem partly from transcoder approximation errors rather than the functional contribution of the selected sparse codes. Although Appendix E.3.2 employs an intervention that preserves the original attention output while merely adding or subtracting the decoded code change, many of the main text's conclusions still rely on substitution-based interventions.
  2. Object ablation is primarily based on a small number of prompts (e.g., "apple"/"pear") and the selection of top-exclusive sparse features, which are then ablated across all token positions. While Appendix E.3 extends the analysis to 23 object pairs, the magnitude of the drop in category margin remains small when using a single selection prompt; the LPIPS score for non-target objects under the selected-code intervention is 0.254, which is far higher than the 0.118 observed in the random control.
  3. While IsoProbe has the advantage of being data-agnostic, actual CLIP text embeddings are highly anisotropic. Therefore a discussion involving real CLIP models and actual prompts is necessary.
  4. The term "concept-associated" is insufficient to demonstrate a clear causal mapping between a single dictionary element and a specific concept.
Questions for authors

see weakness

3 comments

Thank you for your careful review and constructive feedback.

We have uploaded a revised manuscript in which additions and revisions are marked in blue.

W1-1 Approximation error and the functional contributions of selected codes

As noted in the review, Appendix B.3 shows that substituting transcoder outputs for OV writes can alter generated images even without code intervention. This evaluation replaces the OV writes of all 195 cross-attention heads in SDv2.1, so approximation errors across these heads jointly influence generation. Such errors may also affect the magnitude and appearance of intervention responses. Nevertheless, in the no-edit examples in Figure 10, the IsoProbe reconstructions retain the main objects and overall scene content of the original SD outputs, despite differences in visual details.

Replacing OV writes throughout the model with dictionary outputs allows us to investigate the roles of concept-associated components within shared heads. Within this approximation, we assess the contributions of selected codes by varying the code sets and intervention strengths while holding the transcoder fixed.

Figures 4-6, cited in the review, provide these comparisons. In Figure 4, suppressing apple or pear codes shifts the corresponding object toward the appearance of the other category, with the prompt, 10 shared heads, and intervention scope across all text-token positions held fixed. Figure 5 shows changes corresponding to style-code injection and dog-to-cat code patching. Figure 6 shows progressively weaker target-style characteristics as the number of suppressed entries increases.

Appendix E.3.1 compares interventions on the corresponding concept's codes, the other concept's codes, and active-random controls in the same 15 shared heads, using a common no-edit TC reconstruction as the reference. Active-random controls are drawn from active codes outside the union of the two selected sets and matched in decoded write magnitude for each head. Suppressing the corresponding concept's codes produces a larger decrease in its category margin than either control. The observation that reconstruction alters images is therefore insufficient, on its own, to explain these responses to code selection and intervention strength.

In the object experiments of Appendix E.3.2 in the original submission, we retain the original attention output and suppress the codes associated with one object or the other. The resulting object responses depend on which code set is suppressed (Table 6), showing that the functional correspondence between selected codes and generated content also occurs without full substitution.

W1-2 Extending original-output validation to gender and safety

Following a suggestion from another reviewer, we extended the original-output evaluation to gender and safety interventions on SDv2.1. We compared this path with TC substitution using the same code manipulations and initial noise.

The gender intervention examples newly added to the revised Appendix E.3.2 (Figure 19) show changes in the apparent gender presentation of the mechanic and firefighter according to the sign of the intervention along the gender-reference pattern. The safety examples added to the same section (Figure 20) compare suppression at N=50 and N=100 for four I2P cases. Reduced exposure or additional clothing is also observed when the original attention output is retained. These responses therefore occur without full substitution, although they are accompanied by other appearance changes.

Given your concern about reliance on full substitution, we would appreciate your assessment of the additional comparisons in Figures 19 and 20 of the revised Appendix E.3.2.

W2-1 Rationale for intervening at all text-token positions

Section 4.3 tests whether the object affected by an intervention depends on the selected code set, without separately specifying target-token positions. When intervention is restricted to the target token, the observed change could be interpreted as attenuation of the explicitly designated word's representation. We select codes from the activations of the apple and pear tokens, then suppress the selected sets at all text-token positions in the 10 shared heads. Non-target tokens are therefore not protected by exclusion from the intervention, and the same code-suppression rule applies at every position.

W2-2 Effect sizes for codes selected from a single prompt

In Appendix E.3.1, the code sets and 15 shared heads selected from a single prompt for each object pair are held fixed during evaluation on held-out prompts. Table 5 reports the mean category margins in these evaluation contexts.

For a given object crop, the category margin is the difference between the cosine similarities of its CLIP image embedding to the text embeddings of the two category descriptions. For example, for a cat crop, its similarity to "a photo of a dog" is subtracted from its similarity to "a photo of a cat." This measures relative similarity to the two categories, rather than a probability or removal success rate.

The mean margin decreases from 0.0552 before intervention to 0.0263 after suppressing the corresponding concept's codes. Suppressing the other concept's codes and applying the active-random control yield margins of 0.0503 and 0.0532, respectively. The reduction of 0.0289 should therefore be interpreted relative to the observed pre-intervention category margin and the control responses. The mean margin is reduced to approximately half its original value, but this does not quantify the fraction of concept information removed or imply a category change in every scene.

W2-3 Concept-specific responses and non-target appearance changes

As noted in the review, Appendix E.3.1 reports higher non-target LPIPS for selected-code interventions, at 0.254, than for active-random controls, at 0.118. Selected-code interventions produce larger target-category responses and also greater changes in non-target appearance. The controls match decoded write magnitude for each head, but do not match the target effect at the image level. This comparison therefore does not assess appearance preservation at an equal target change. It supports concept-specific functional responses of the selected codes, without establishing precise preservation of non-target appearance.

W3 Validation on real CLIP embeddings and prompts

The anisotropy of real CLIP text embeddings and the extent to which a finite caption dataset covers inputs encountered in practice are distinct issues. We examine whether OV approximation extends from inputs frequent in the caption training data to real inputs that are rare or absent from it. IsoProbe approximates the fixed OV operator across diverse input directions without restricting training to the distribution of a particular caption collection.

Figure 2c,d illustrates these differences using contextual token states from the actual CLIP text encoder. The COCO transcoder achieves higher fidelity on the frequent tokens shown, which are drawn from held-out COCO captions. IsoProbe achieves higher fidelity on the selected COCO-unseen tokens from P4D and Ring-A-Bell, with a larger gap between the two approaches in this group.

We also evaluated image generation from real prompts. Using COCO-1K captions excluded from COCO transcoder training, we measured distributional differences relative to unmodified SDv2.1 outputs with FID and semantic alignment between prompts and images with CLIP scores. Tables 2 and 14 report these results separately for no-edit reconstruction and safety-code suppression.

W4 Associations of individual elements and functional contributions of code sets

An activation-based association with a concept does not, by itself, establish the causal function of an individual dictionary element. We use these associations to select codes and assess the functional contributions of the resulting sets through interventions. Because contributions associated with one concept may be distributed across multiple entries, our primary unit of intervention is a code set. The abstract of the original submission explicitly refers to concept-associated code subsets, and Section 3.2 clarifies that dictionary elements are learned components introduced by the transcoder rather than literal units of the original model.

The causal evidence is provided by controlled interventions on selected codes. In Figure 4, varying the suppressed code set changes the corresponding apple or pear response while holding the transcoder, shared heads, and intervention scope across all text-token positions fixed. Style injection and object-code patching in Figure 5, together with progressive style suppression in Figure 6, produce content changes corresponding to the code manipulations. Appendix E.3.1 evaluates fixed code sets in new contexts against other-concept codes and active-random controls matched in write magnitude. Concept-associated object changes also occur when the original attention output is retained (Appendix E.3.2).

We would be happy to address any further questions or requests for clarification.

Reviewer_2An2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents IsoProbe, a sparse representation method for analyzing output-value (OV) operators in text-to-image diffusion models. The method trains sparse transcoders using isotropic Gaussian probes, without requiring text or image training samples. Experiments show that this method has certain effectiveness.

Critical strengths
  1. The paper offers an interesting perspective on cross-attention interpretability by examining OV writing operators, complementing existing analyses that primarily focus on QK routing.
  2. The proposed method exploits the linear structure of fixed OV operators to construct training pairs from isotropic Gaussian probes without requiring text or image samples.
Critical weaknesses
  1. Section 4.2 mainly evaluates OV approximation fidelity using cosine similarity and decoder coverage. However, these metrics do not directly measure errors in OV write magnitudes.

  2. Section 4.3 selects concept-exclusive codes based on apple and pear token activations. However, the subsequent ablation suppresses these codes at all text-token positions in the ten shared heads. Token-specific ablations would help clarify the contributions of different token positions.

  3. Section 4.5 reports substantial reductions in attack success rates on I2P and MMA after suppressing 50 selected codes. However, the effects on benign prompts remain unclear. Evaluating semantic alignment on benign generations would help assess the impact on normal content generation.

Questions for authors

Please see Weaknesses.

3 comments

Thank you for your careful review and constructive feedback.

We have uploaded a revised manuscript in which additions and revisions are marked in blue.

W1 Cosine fidelity and output magnitude error

In Section 4.2, cosine similarity measures directional agreement between predicted and original OV writes, while decoder coverage characterizes the decoder's relation to the writing subspace. We agree that neither metric directly measures output magnitude error.

We therefore measured the absolute difference between the norms of the transcoder output and the original OV write for each input, normalized by the latter. We averaged this relative error over inputs within each head and then uniformly across heads. Taking the absolute value before aggregation prevents overestimation and underestimation from canceling.

The evaluation covers all 195 SDv2.1 heads. For real text, we used contextual token states excluding BOS, EOS, and padding.

Evaluation inputsIsoProbe transcoder relative output norm errorCOCO transcoder relative output norm error
COCO held-out9.89%6.84%
I2P10.35%14.03%
P4D10.49%14.76%
MMA adversarial10.86%12.92%

For example, a value of 10% indicates that the predicted output norm differs from the ground-truth norm by 10% on average. IsoProbe has lower errors on the evaluated I2P, P4D, and MMA adversarial inputs, whereas the COCO transcoder is more accurate on held-out COCO inputs. This metric assesses output magnitude. It does not establish superiority in overall reconstruction error, which also depends on direction, or in image preservation.

W2 Rationale for intervening at all text-token positions

Section 4.3 tests whether the object affected by an intervention depends on the selected code set, without separately specifying target-token positions. When intervention is restricted to the target token, the observed change could be interpreted as attenuation of the explicitly designated word's representation. We select codes from the activations of the apple and pear tokens, then suppress the selected sets at all text-token positions in the 10 shared heads. Non-target tokens are therefore not protected by exclusion from the intervention, and the same code-suppression rule applies at every position.

W3 Effects of safety interventions on semantic alignment for benign prompts

Evaluations on benign prompts were included in the original submission and appear in Tables 2 and 13 of the revised manuscript. We evaluated images generated from COCO-1K prompts under safety-code interventions using FID and CLIP scores. FID measures the distributional difference from unmodified SDv2.1 outputs, while CLIP measures semantic alignment between prompts and generated images.

Table 2 shows how stronger suppression of selected codes affects both safety and benign generation. Increasing IsoProbe suppression from N=0 to N=50 substantially reduces attack success rates on I2P and MMA, while the COCO-1K CLIP score decreases from 31.07 to 30.35. For reference, unmodified SDv2.1 scores 31.40. Thus, both transcoder substitution and subsequent code suppression reduce average prompt–image alignment on benign prompts. These results characterize the trade-off between suppressing unsafe content and preserving semantic alignment. At the same time, FID increases from 40.02 to 43.52, indicating a change in the generated image distribution. Table 14 in Appendix H reports CLIP, FID, and LPIPS across a wider range of suppression strengths to characterize these effects.

Figure 27 of the revised paper (Figure 24 in the original submission) visually compares target and non-target responses in a separate style intervention. Suppressing cyberpunk-associated codes weakens cyberpunk characteristics in images generated from the target prompt. In the displayed non-target examples, Van Gogh's painterly characteristics and the identity of the pizza remain largely recognizable. These examples show that suppressing selected codes can induce different responses depending on the associated content, rather than uniformly degrading generation across prompts.

We would be happy to address any further questions or requests for clarification.

Thank you for the authors' detailed responses. I have reviewed their responses to both my concerns and those raised by the other reviewers. Most of my concerns have been addressed. Therefore, I will maintain my positive score.

Thank you for taking the time to read our responses, including those to the other reviewers. We sincerely appreciate your constructive review and your positive assessment.

Reviewer_fhapOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at the OV part of cross-attention heads in text-to-image diffusion models. Since each head's OV operator is a fixed linear map, the authors feed it random Gaussian vectors and train a sparse transcoder on the exact outputs, with no text or image data. The decoder rows then form a dictionary of "writing directions." Prompts are used to find which codes go with which concept, and those codes are zeroed, added, or swapped to edit objects, styles, gender cues, and unsafe content. Most experiments are on SD v2.1, with extra results on SD1.4, SDXL, PixArt-alpha, and an exploratory SD3 study.

Critical strengths

I liked the intervention experiments. Zeroing the apple codes in the shared heads pushes the apple toward a pear and the other way around (Fig. 4), and this is repeated on 23 object pairs with held-out contexts and random-code controls (App. E.3.1). This is what makes the dictionary more than a nice visualization. The matched comparison against the SVD ablation of Bae et al. (Fig. 3, App. G) is also fair.

Critical weaknesses

My main concern is that the transcoder substitution by itself already changes the images quite a bit. With no edit at all, swapping in the transcoder output gives FID 40.0 against SD v2.1 (Table 2, N = 0), about what the baselines change by actually editing, and reconstruction alone moves UnLearnDiffAtk from 32.39% to 26.06% (Table 2). The safety and gender tables never separate the two effects, and the substitution-free variant in App. E.3.2 is only run for objects. So I am not sure how much of the safety effect comes from the edited codes.

I am also not fully convinced that isotropic probing is better for interpretation, rather than just better off-distribution. Fidelity is higher on tokens unseen in COCO captions (Fig. 2d), but the COCO-trained transcoder is more faithful on frequent COCO tokens (Fig. 2c). It would also help to know whether the picked concept codes come back when the transcoder is retrained with a different seed; App. J only looks at reconstruction error.

Questions for authors
  1. Do the safety and gender results still hold if you only add the decoded code change to the original OV output, without the full substitution (as in App. E.3.2)? I would appreciate even a small-scale version of this.
  2. If you retrain the transcoders with different seeds, how much of the dictionary and the picked concept codes come back?
  3. In App. E.3.1 you keep 112 out of 414 scenes. Could the surviving scenes be the easy ones?
4 comments

Thank you for your careful review and constructive feedback.

We have uploaded a revised manuscript in which additions and revisions are marked in blue.

W1-1 / Q1-1 Distinguishing reconstruction error from code intervention effects

Sparse dictionary methods, including sparse autoencoders (SAEs) and cross-layer transcoders (CLTs), approximate model computations and can introduce reconstruction error that affects both model outputs and the interpretation of interventions. In our SDv2.1 substitution setting, the OV writes of all 195 cross-attention heads are replaced by transcoder outputs, so reconstruction errors across heads can jointly affect generation. The no-edit changes noted in the review were measured under this approximation of the full OV pathway (Table 2, Appendix B.3).

We replace all heads despite this reconstruction error to examine the contributions of selected codes while representing the model's OV computations through the learned sparse dictionaries. Each head's OV dictionary is learned through isotropic probing without text or image samples. In the main substitution experiments, we hold the transcoder fixed and test whether manipulating selected codes changes the corresponding generated content.

To assess whether these responses also occur without substituting reconstructed writes, the experiment in Appendix E.3.2 of the original submission, as noted in the review, retains the original attention output and applies only the decoded changes induced by the selected-code interventions. The no-edit condition therefore reproduces the original SD output. Suppressing an object's codes reduces its category margin more, on average, than suppressing the other object's codes, providing evidence of distinct functional responses to the selected sets without full substitution (Table 6).

W1-2 / Q1-2 Gender and safety interventions retaining the original attention output

Following your suggestion, we extended this evaluation to gender and safety interventions on SDv2.1. In each experiment, the original-output and TC substitution paths use the same prompt and initial noise, and intervention effects are evaluated relative to each path's no-edit output. Qualitative comparisons have been added as Figures 19 and 20 in Appendix E.3.2 of the revised manuscript.

For the gender intervention, we used the gender-reference pattern defined in Appendix F. In the mechanic and firefighter examples in Figure 19, applying the same decoded code change to both paths produces changes in apparent gender presentation in the intended direction, including when the original attention output is retained.

For the safety intervention, both paths use the same fixed safety-code sets. Across the four I2P examples in Figure 20, reconstruction error leads to differences in pose and visual details in the no-edit images (N=0), while the main subjects and overall scene content remain similar.

The examples at N=50 and N=100 show that suppressing the selected codes can reduce exposure or introduce clothing in both paths. These observations indicate that the intended effects of code intervention also occur when the original attention output is retained.

W2 Isotropic probing and its role as an interpretability tool

Figure 2c,d illustrates the differences between the two training approaches. The COCO transcoder is more accurate on the frequent COCO tokens shown, whereas IsoProbe achieves higher fidelity on the selected COCO-unseen tokens. The fidelity gap between the two transcoders is also larger for these COCO-unseen tokens than for the frequent COCO tokens. These approximation results alone do not establish superiority in semantic interpretation.

We separately analyzed concept-code associations in the learned dictionaries and tested the functional contributions of selected codes through interventions. Our analyses examine token-associated activation patterns and code responses to concept replacement (Appendices C and D). Interventions on code sets associated with different concepts within the same heads produce corresponding changes in generated content. We also reuse fixed code selections in new contexts to assess whether these functional correspondences extend beyond the selection prompts (Section 4.3 and Appendix E).

These results support the use of data-free OV dictionaries to investigate concept-associated contributions.

Q2-1 OV reconstruction across training seeds

We independently retrained the dictionaries with seeds 101 and 202 under the same SDv2.1 OV operators and training settings. Table 7 in the revised Appendix E.3.3 compares the original OV writes with the outputs of the original dictionary and the two retrained dictionaries on the same held-out object-token states. Their mean cosine similarities to the original OV writes, averaged over all 195 heads, are 0.878, 0.878, and 0.877, respectively. Pairwise cosine similarities between the reconstructed outputs range from 0.800 to 0.801.

The three dictionaries achieve comparable reconstruction fidelity to the original OV writes, while differences in their reconstruction errors lead to differences between the reconstructed outputs. This comparison concerns the summed decoded contributions of all active codes.

Q2-2 Functional responses of selected codes across retrained dictionaries

Figure 21 compares apple/pear and chair/table interventions using the same 15 heads and initial noise across dictionaries. Only the decoded code changes are applied to the original attention output.

We reselected concept-associated codes independently in each dictionary using the same procedure and evaluated interventions in contexts not used for code selection. Suppressing apple codes shifted the apple toward a pear-like appearance, while suppressing chair codes removed the chair's backrest. Both responses were observed with the original dictionary and the two retrained dictionaries. Although the resulting appearance and non-target changes varied, the presented examples show a recurring functional correspondence between the selected codes and concept changes across training seeds.

Q3 Criteria for retaining 112 of the 414 scenes

Appendix E.3.1 measures intervention responses for each of the two objects within the same scene. This requires both objects to be identifiable in the pre-intervention reference image and the evaluation bounding boxes to localize each object correctly. Applying these criteria to 414 scenes generated with SDv2.1 yielded 112 evaluation scenes spanning 19 object pairs. Selection was based solely on the reference images and did not use the extent to which the target changed or the non-target was preserved after intervention. The experiment therefore compares the responses induced by selected codes in reference scenes where both objects can be evaluated separately.

We would be happy to address any further questions or requests for clarification.

I appreciate the responses. The figures are also appreciated. While most concerns are addressed, the gender/safely results involving the substitution is qualitative. Some form of quantitative evaluation would have been nice to see, even on a small subset.

Thank you for following up and for your feedback on the added figures. We also appreciate your suggestion to complement the qualitative examples with a quantitative comparison.

We compared the two safety intervention paths on SDv2.1 using the same 240 prompt–seed pairs from I2P and P4D and the same selected safety codes. The table reports pooled detection rates, defined as the fraction of images in which NudeNet detects at least one exposed-anatomy class with confidence of at least 0.6. At N=0, no codes are suppressed, so the original-output path corresponds to unmodified SDv2.1.

Intervention pathN=0N=50N=100
Original attention output retained15.8% (SD baseline)3.8%2.5%
TC substitution13.3%2.5%0.8%

Substitution alone does not leave the detection rate unchanged, but the rate remains broadly similar (15.8% to 13.3%). The N=0 rows of Tables 2 and 13 show a similar pattern on I2P, MMA, and P4D (for example, 14.95% to 14.63% on I2P). In contrast, suppressing the selected codes lowers the rate in both settings as N increases. This trend suggests that the reduction is associated mainly with the selected codes rather than with substitution, and that it also appears without substitution.

We also evaluated text–image alignment on 200 COCO captions. The following table reports changes in mean CLIP score relative to unmodified SDv2.1.

Intervention pathN=0N=50N=100
Original attention output retained0.00 (SD baseline)-0.20-0.33
TC substitution-0.20-0.61-1.03

Applying only the decoded code change to the original attention output avoids the reconstruction error introduced by full substitution.

We would be happy to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviews, Responses, and Revision

We thank all three reviewers for their careful and constructive reviews. As the author-reviewer discussion period is drawing to a close, we briefly summarize the revision and our responses. Details are in the individual responses.

Changes in the revised manuscript

Additions and revisions are marked in blue. We added Figures 19 and 20 to Appendix E.3.2, and Table 7 and Figure 21 to the new Appendix E.3.3. As a result, figure numbers from Figure 19 of the original submission onward increased by three, and table numbers from Table 7 onward increased by one.

Shared concern on transcoder substitution and code intervention effects (fhap, mn1x)

As the reviewers noted, substitution alone can change the generated images. We report this change in the N=0 row of Table 2 and in Appendix B.3, and state it as a limitation in the conclusion. It is a limitation shared by methods that approximate model computations with sparse dictionaries.

However, the concept-specific changes observed under intervention are not explained by this approximation error alone. First, every condition uses the same transcoder, yet the change follows the manipulated codes as intended. In the main text, suppressing apple codes shifts the apple toward a pear and suppressing pear codes shifts the pear toward an apple within the same heads (Figure 4), and the target style weakens progressively as more entries are suppressed (Figure 6). Quantitatively, across object pairs, the mean category margin of an object falls to about half of its pre-intervention value when its own codes are suppressed, whereas it stays close to that value under the other object's codes or magnitude-matched random codes (Appendix E.3.1). Second, concept-specific responses also appear when we retain the original attention output and add only the decoded change from the selected codes (Appendix E.3.2). Following Reviewer fhap's suggestion, we extended this setting to gender and safety interventions and added qualitative examples as Figures 19 and 20.

Other points

  • Fidelity (2An2, mn1x, fhap). We reported the relative output-norm error on real CLIP token states in our response to Reviewer 2An2. These results describe how well each dictionary approximates the OV writes, and the evidence for interpretation comes from the concept-code analyses and interventions (Section 4.3 and Appendices C to E).
  • Effect size (mn1x). The reviewer noted that the decrease in category margin is small in absolute terms. The margin measures how much closer an object crop is to its own category than to the other, as a difference of cosine similarities, and its pre-intervention value is itself small (Table 5). The decrease should therefore be read relative to that value and the controls. As stated above, the margin falls to about half only when an object's own codes are suppressed.
  • Non-target changes (mn1x). The reviewer also noted that, in the experiment of Table 5 (Appendix E.3.1), selected-code interventions yield higher non-target LPIPS than the random control. This control is matched only in write magnitude and reduces the target margin far less than the selected codes do. The two interventions thus differ in how much they change the target, so the LPIPS gap alone does not show how much non-target appearance changes for a given target change. What this experiment does show is that the non-target object changes in appearance while its category margin stays close to the pre-intervention value on average (Table 5). A comparison at equal target-detection rates is given in Appendix E.5 under a separate two-object protocol on SDv1.4, where IsoProbe retains the non-target object more often and has lower non-target LPIPS than SAeUron (Table 8).
  • Causal role of individual elements (mn1x). We agree that an activation-based association alone cannot establish the causal role of an individual dictionary element. We therefore use these associations to select codes, and we test functional contributions by intervening on code sets.
  • Training seeds (fhap). We retrained the dictionaries with two other seeds. Fidelity to the original OV writes was similar across the three dictionaries (Table 7), and codes reselected independently in each dictionary produced object responses in the same direction in the presented examples (Figure 21). Reproducibility is thus assessed at the level of reconstruction and of the function of selected code sets.
  • Real benign prompts (2An2, mn1x). This evaluation was already in the original submission. Images generated from COCO-1K captions are evaluated with FID and CLIP in Table 2 and Table 14. With no code edit, the CLIP score on these real prompts is 31.07, compared with 31.40 for the unmodified model, and it decreases gradually with stronger suppression (30.35 at N=50). This quantifies the trade-off between suppressing unsafe content and preserving normal generation.

We would be glad to address any further questions before the discussion period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.