acceptodds
Under review as a conference paper at ICLR 2027

Uncovering Conceptual Geometry in Diffusion Models with Spatially Invariant Similarity

Abstract

How do diffusion models organize the visual world they learn to generate? We introduce Spatially Invariant Similarity (SIS), a training-free and model-agnostic method for comparing intermediate diffusion representations despite variations in spatial configuration. SIS establishes patch-wise feature correspondences and preserves the spatial evidence underlying their similarity. We further introduce Testing with SIS (TSIS), which aggregates these correspondences across generations to reveal concept-level relational structure. Across diverse pretrained diffusion models, TSIS uncovers coherent conceptual geometries: semantically related object categories exhibit shared coarse organization while retaining model-specific relationships. Extending beyond object categories, we show that color, material, and viewpoint exhibit distinct block–timestep trajectories, revealing where and when different visual factors become distinguishable during generation. Finally, generations that deviate from the reference geometry in intermediate representations consistently produce rarer and less similar final images. These results establish SIS and TSIS as tools for studying both the organization and evolution of conceptual structure within diffusion models.

Primary area:
interpretability and explainable AI
Keywords:
diffusion models, spatial correspondence, concept similarity, interpretability, semantic representations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iXaxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces spatially invariant similarity for diffusion representations by matching spatial features and weighting their correspondence with concept attention. Aggregation across generations produces concept-level affinity matrices. Class-level experiments cover thirteen diffusion models; further analyses examine attribute trajectories and the association between intermediate representational atypicality and unusual final images.

Critical strengths
  1. SIS makes similarity inspectable by retaining the spatial contributions to the score. Similar overall scores can reflect different image regions, so these maps provide information that a scalar comparison loses. The paper also recognizes that a correspondence map does not uniquely identify the semantic factor responsible for it.
  2. The analysis links intermediate representations to completed images: the low-TSIS groups have higher rarity and lower CLIP similarity in all twelve tested category–attribute comparisons. The broader class-level analysis across diffusion models adds descriptive evidence of both shared and model-specific organization.
Critical weaknesses
  1. The comparison with global mean pooling changes both spatial correspondence and attention weighting, so it establishes an advantage for the complete TSIS construction without isolating the contribution of correspondence (Figure 4). The text-encoder comparison also leaves pooling and affinity computation unspecified, making the suggestion that diffusion geometry is not simply inherited from text conditioning difficult to assess. These gaps limit attribution and cross-representation interpretation; the descriptive TSIS structures and image-space associations remain supported.
  2. The representation-selection rule is inconsistent between the main text and Appendix C. The former maximizes accuracy over block–timestep pairs, while the latter first selects a block by accuracy averaged over timesteps. The timestep near 0.8T is then described as both fixed and a tie-breaker, although several reported models use step 12 of 20. The actual procedure needs to be clear because it determines which representation underlies the reported geometry.
Questions for authors
  1. How are conditioning-text embeddings pooled and converted into affinities? If results using comparable affinity calculations or attention-controlled pooling are available, do they preserve the reported differences? The implementation clarification is needed to assess the comparison; the additional controls would strengthen attribution.
  2. Which block–timestep selection rule was actually used for the class-level results? Please reconcile the main-text and appendix descriptions. If results under alternative rules are available, do they materially change the affinities or the comparison with external vision representations?
2 comments

Thank you for the careful review and for identifying the missing implementation details and inconsistent descriptions of representation selection. Please find our responses below. Changes addressing this review are shown in red in the revised paper (orange for changes shared across reviews).

W1: Contribution of spatial correspondence

We added a three-way ablation comparing global average pooling (GAP), spatial matching without attention weighting (Vanilla), and full TSIS across seven diffusion models (Sec. 5.1, Table 1). Using 20 selected CIFAR-100 classes with 10 images per class, we report 5-fold self-classification accuracy averaged over all evaluated block–timestep pairs:

ModelGAPVanillaTSIS
SD3.5 L0.690.760.78
Lumina 2.00.810.800.70
Z-Image0.610.720.72
SANA 1.50.910.950.95
FLUX.1 Dev0.800.850.88
PixArt-α\alpha0.710.800.86
SDXL0.680.740.76

Comparing Vanilla with GAP isolates the contribution of spatial matching: accuracy improves in six of seven models. Adding attention weighting improves accuracy in four models, matches Vanilla in two, and reduces accuracy in Lumina 2.0.

W1 & Q1: Text feature extraction and affinity comparison

App. B.2 now specifies how we extract and pool the text-conditioning features:

  1. Process “a photo of {concept}” through the model’s original text encoder and conditioning pipeline.
  2. Extract text token features immediately before cross-attention or joint attention in the first diffusion block at the first denoising step. These features include model-specific transformations applied after text encoding.
  3. Select only the concept tokens and average their feature vectors into one concept representation. For multi-token concepts such as “pickup truck,” all corresponding tokens are included; the remaining prompt tokens are excluded.

Pairwise cosine similarities between these concept representations form the text-encoder affinity matrix. We use the original pretrained models without additional training or modification.

Diffusion affinities are computed with TSIS; vision-encoder affinities average image-pair similarities for each concept pair. Each matrix is symmetrized and Z-normalized using its off-diagonal mean and standard deviation. For each vision encoder, the matrices are then averaged across diffusion models and normalized again to obtain a common reference geometry.

Table 2 compares these affinity patterns using cosine similarity and L2 distance between their upper-triangular entries (i<ji<j). Negative values are clipped only for visualization. Diffusion–CLIP cosine agreement exceeds diffusion–text agreement in 12 of 13 models, with Kandinsky 3 as the exception.

Table 5 repeats the comparison on raw matrices, without normalization, symmetrization, or clipping. CLIP agreement remains higher than text agreement in 11 of 13 models. SD1.5 changes ordering (text: 0.861; CLIP: 0.856), while Kandinsky 3 favors text in both settings. We use normalized comparisons as the primary analysis because raw cosine similarity is sensitive to mean affinity levels, and raw L2 depends on both mean and scale.

W2 & Q2: Representation selection and stability across locations

The class-level selection rule is now specified consistently in Sec. 5 and App. B.1:

  1. Compute 5-fold TSIS self-classification accuracy at each candidate block–timestep pair (l,t)(l,t).
  2. Average accuracy over timesteps for each block and select the block with the highest mean accuracy (Eq. 21).
  3. At that block, select the inference step whose normalized scheduler timestep is closest to 0.8, where 1 corresponds to pure noise.

Thus, block selection uses timestep-averaged accuracy, followed by a fixed scheduler-timestep criterion. We do not jointly maximize accuracy over (l,t)(l,t) or use the timestep as a tie-breaker. The value 0.8 is a normalized scheduler timestep, not 80% of the inference steps; non-uniform scheduler spacing can produce different step indices across models. Table 4 lists the locations used for the class-level results, including Figure 11.

To assess sensitivity to the exact location, we added the analysis in App. A.2, Figure 9. For each of seven models, we select the individual (l,t)(l,t) pairs in the top 10% and bottom 10% of self-classification accuracy on 20 selected CIFAR-100 classes. We construct an affinity matrix at each location, Z-normalize it, flatten its upper-triangular entries (i<ji<j), and compute the average pairwise cosine similarity within each group.

The top-accuracy group has higher agreement in every evaluated model, showing that high-accuracy locations yield more consistent conceptual geometries. SDXL has high agreement in both groups; its candidate locations are restricted to the mid-block.

Thank you for the detailed response and the effort invested in addressing my questions. Your clarifications resolve the concerns I raised. I will keep my rating at Weak acceptance.

Reviewer_8fikOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SIS, an attention-weighted similarity measure that matches spatial diffusion features across generations, and TSIS, which aggregates these similarities into concept affinity matrices. Experiments across 13 diffusion models examine object-category relationships, block–timestep patterns for color, material, viewpoint, and the association between representational atypicality and unusual generated images.

Critical strengths
  1. SIS retains spatial contributions to similarity, allowing comparable scalar scores to be traced to different image regions. This makes individual concept relationships more inspectable than a global similarity score alone.
  2. Evaluation across 13 models provides substantial evidence of applicability across architectures.
  3. The full factorial attribute experiment enables analysis beyond object categories. The rarity experiment additionally connects intermediate similarities to final-image measurements.
Critical weaknesses
  1. Figure 4 compares complete TSIS against global mean pooling, simultaneously changing spatial matching and attention weighting, so it does not isolate the benefit attributed to correspondence. Attention-weighted pooling and matching without attention are necessary ablations, alongside controlled spatial perturbations to establish robustness to changes in layout.
  2. Section 4.2 interprets higher attribute-classification accuracy as a stronger generative role, although decoding a prompted attribute does not establish that the corresponding block controls its appearance. Interventions on high-versus low-accuracy blocks, coupled with verification of rendered attributes would substantiate this interpretation. (Experiment beyond teddy bears may also help improve convincibility)
Questions for authors
  1. How exactly are concept-specific attention maps constructed across architectures, including which prompt tokens are selected and how their attention values are aggregated? For the attribute experiments, does the map use the target attribute’s tokens, the object tokens, or the entire prompt?
  2. Which model and attribute-specific (l,t) values are used for Table 2, and are these transferred unchanged from the teddy-bear experiment? Since Figure 5 illustrates all three SANA attribute representations at t=20, please clarify whether the rarity measurements use the final denoising step or an earlier stage.
3 comments

Thank you for the careful review and constructive suggestions. Please find our responses below. Changes addressing this review are shown in green in the revised paper (orange for changes shared across reviews).

W1: Spatial correspondence, attention weighting, and layout robustness

We added a three-way ablation comparing global average pooling (GAP), spatial matching without attention weighting (Vanilla), and full TSIS across seven diffusion models (Sec. 5.1, Table 1). We report 5-fold self-classification accuracy averaged over all evaluated block–timestep pairs:

ModelGAPVanillaTSIS
SD3.5 L0.690.760.78
Lumina 2.00.810.800.70
Z-Image0.610.720.72
SANA 1.50.910.950.95
FLUX.1 Dev0.800.850.88
PixArt-α\alpha0.710.800.86
SDXL0.680.740.76

Comparing Vanilla with GAP isolates the contribution of spatial matching: accuracy improves in six of seven models. Adding attention weighting improves accuracy in four models, matches Vanilla in two, and reduces accuracy in Lumina 2.0.

We also added two robustness analyses:

  • Layout variation (App. A.1, Figure 8): For each concept, we generate five object-size variations and five position variations, keeping the same class label. We evaluate all three methods at the model-specific (l,t)(l,t). Accuracy decreases under layout variation for all three methods, but TSIS retains the highest mean accuracy across models.
  • Object area (App. A.1, Figure 7): Across 14,000 images from seven models, we estimate object area using SAM 3 and compare it with each image’s average similarity to other generations of the same model. After within-model Z-normalization, TSIS has a much weaker correlation with area (r=−0.064r=-0.064) than GAP similarity (r=−0.466r=-0.466; both p<0.001p<0.001).

W2: Attribute discriminability and intervention

We revised Sec. 5.3 to interpret self-classification accuracy as attribute discriminability and added a preliminary intervention experiment (App. C.4, Figure 14).

We generate two images with the same seed from prompts differing in color, material, and viewpoint. At each (l,t)(l,t), we average features over the top 10% of patches ranked by concept-specific attention and compute a feature-difference direction.

For each attribute, we intervene at the top 20% of block–timestep pairs ranked by self-classification accuracy, extending these regions to adjacent blocks and the subsequent denoising step. We add the direction to the feed-forward residual updates and vary its strength α\alpha. The feature-difference construction is shared across attributes; the intervention locations are selected separately.

Relative to the original generation at α=0\alpha=0, Figure 14 shows:

  • Color-selected regions shift the appearance toward blue.
  • Material-selected regions change the smooth surface toward a fur-like appearance.
  • Viewpoint-selected regions shift the orientation toward a rear view.

The changes are not fully disentangled. This experiment compares intervention strengths at the selected regions, rather than high- versus low-accuracy regions.

We also extended the attribute analysis beyond teddy bears to house, chair, sneaker, and toy cat using SANA-1.5, reporting block–timestep accuracy maps and UMAP projections (App. C.3, Figure 13).

Q1: Construction of concept-specific attention maps

We use all tokens covering the concept description. In the attribute experiments, this includes color, material, object, and viewpoint. For example, the selected span in “a photo of red fabric teddy bear, front view” is “red fabric teddy bear, front view.” The prefix is excluded, and the same span is used for all three attribute evaluations.

App. B.2 (Eqs. 22–23) specifies the architecture-dependent computation. In joint attention, image queries attend to both image and text keys, so both enter the softmax denominator. In cross-attention, the denominator includes text keys only. We average attention over the selected concept tokens and attention heads, followed by the selected blocks and timesteps.

Q2: Model and representation locations for Table 3

Table 3 (Table 2 in the original submission) uses SD3.5 Medium. The following locations from the teddy-bear analysis are transferred unchanged to vase, handbag, hat, and teapot (Sec. 5.3, Figure 12):

AttributeBlock llStep tt
Color216
Material124
Viewpoint1020

These are attribute-specific locations for the rarity experiment; Table 4 lists the separate class-level selections.

Inference steps are indexed in reverse chronological order: for 20 steps, t=20t=20 is the first and t=1t=1 the last. The three locations therefore correspond to the 5th, 17th, and 1st steps in generation order; none uses the final step. The SANA-1.5 representations at t=20t=20 in Figure 4 likewise correspond to the first step.

Thank you for the thoughtful response. I appreciate the authors’ candid and measured discussion on the remaining issues of their work. The clarifications are helpful, and as I will keep my positive rating.

We sincerely appreciate your constructive comments that have helped improve the paper. We will make sure all promised revisions are incorporated into the camera-ready version.

Reviewer_7Sc9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, authors introduce Spatially Invariant Similarity (SIS) to compare intermediate diffusion representations across generations with different spatial layouts. SIS matches each spatial feature in one generation to its most similar feature in another and weights these correspondences using concept attention maps. TSIS aggregates the scores across generation pairs and both matching directions to construct concept affinity matrices. Authors study 100 object classes across 13 diffusion models, compare their geometries with vision encoders, and examine color, material, and viewpoint across blocks and timesteps. They also relate low similarity to reference generations to higher rarity and lower image-to-image CLIP similarity.

Critical strengths
  • I appreciate the broad evaluation across 13 diffusion models, including both U-Net and transformer architectures. The method requires no additional training or backpropagation, which makes it appealing for studying models.
  • Figure 3 is particularly useful. The matched locations make it much easier to understand which visual regions contribute to the similarity score.
Critical weaknesses
  • The presentation makes the paper unnecessarily difficult to follow, particularly when introducing the method and interpreting the main results. Figure 1 presents many affinity matrices, but the caption gives little guidance on which relationships the reader should focus on or what the comparison demonstrates. In Figure 2, the roles of the base/source concepts and the scalar S are not sufficiently explained when the reader first encounters them. The text then describes this figure as the TSIS procedure (line 246), although the aggregation across generations and matching directions is not shown. Figure 4 and Table 1 have similar issues: the reader needs a clearer explanation of how the results are constructed, how to read the plots and color-coded rows, and what conclusions they support. These are central figures for understanding both the proposed similarity measure and the evidence for the paper's claims. As a result, it takes substantial effort to reconstruct the argument, and it is difficult to judge what has actually been established. I would need these explanations to be substantially clearer before recommending acceptance.
  • The new insight provided by the conceptual geometry analysis is not sufficiently established. Shared representational structure across models has been studied extensively, including in [1], while nearest-neighbor matching of diffusion features is already used in [2], which the paper cites. The specific contribution here appears to be the attention-weighted aggregation into concept affinities and its analysis across models. However, stronger alignment with CLIP than purely visual encoders is plausible given the models' text conditioning. I would like to understand what these results reveal beyond this expectation, and what questions TSIS can answer that simpler feature comparisons cannot. The brief related work section leaves much of the necessary context to the appendix.
  • Please see the questions below as well.
Questions for authors

Questions:

  • Equation 8 allows several base locations to match the same source feature. How does this affect the score when an object occupies very different fractions of the two images? The examples mostly show objects of comparable size, so it is unclear how robust the measure is to this variation.
  • What is the exact rule used to select the block and timestep? The main text describes selecting the pair with the highest cross-validated accuracy, whereas Appendix C selects the block using accuracy averaged over timesteps. It then describes both fixing the timestep near 0.8T and using this as a tie-breaker. Please reconcile these descriptions with Table 3. How stable are the reported geometries under other high-performing choices?
  • Section 4.2 only studies attribute variations of a teddy bear. While interesting, this single object category is not convincing enough to support general conclusions about how diffusion models represent color, material, and viewpoint. Do the observed patterns hold for other object categories?
  • Figure 5 does show where attributes become distinguishable across blocks and timesteps. Why does higher classification accuracy imply a stronger attribute-specific "generative role"? Can the authors also show how relationships between concepts change during denoising, rather than only how distinguishable the attribute labels are?
  • What is the practical utility of the resulting geometry? For example, does TSIS identify atypical generations more reliably than a simpler distance based on pooled features? The rarity experiment is interesting, but comparing only the highest- and lowest-scoring groups does not establish this advantage.
  • Does Table 1 use the affinity matrices after both Z-normalization and clipping negative values to zero? If so, how much does this preprocessing affect the cross-model comparisons?

Suggestions:

  • For Figure 1, I would recommend focusing on two affinity matrices and explaining the specific relationships they demonstrate. The comprehensive comparison can remain in the appendix.
  • Please make Figure 2 understandable on its own: explain the base/source roles and S in the caption, and show the aggregation across generations and matching directions if it is intended to illustrate TSIS. N_b and N_s should also be explicitly introduced as the numbers of generations when they first appear around line 237.
  • Figure 4 needs a clearer explanation of the two heatmaps and their interpretation before the reader encounters it. Table 1 would be easier to read with explicit labels for the reference encoders instead of relying on row colors.
  • The discussion of prior correspondence methods and shared representations should appear earlier, so that the reader can assess the contribution before working through the experiments.

Typos:

  • Line 322: "For almost of evaluated models" -> "For almost all evaluated models."?
  • Line 396: missing closing parenthesis in "(See Figure 11 in Appendix H."

References:

  • [1] Huh, Minyoung, et al. "The platonic representation hypothesis." arXiv preprint arXiv:2405.07987 (2024).
  • [2] Tang, Luming, et al. "Emergent correspondence from image diffusion." Advances in neural information processing systems 36 (2023): 1363-1389.
5 comments

Thank you for the detailed review and constructive feedback. We agree that the presentation should more clearly explain our method and guide readers through the main findings. Based on your suggestions, we have substantially revised the manuscript to improve its clarity and organization. Changes made in response to this review are shown in blue in the revised paper (orange for changes addressing comments from multiple reviewers).

W1 & S1–S4: Clarifying the presentation

We appreciate these suggestions and have made the following revisions:

Figure 1 (Conceptual geometry): We provide a more focused interpretation of the main results in Sec. 5.2, emphasizing the shared coarse organization of biological categories and model-specific differences in plant-related relationships. The full affinity matrices are provided in Appendix C.2 (Fig. 11).

Figure 2 (SIS procedure): We revised the figure labels and caption to explicitly define the base and source concepts, directional matching, and scalar SIS score. We also clarified the distinction between SIS and TSIS in Sec. 4: SIS computes a spatial map and scalar score for a single generation pair (Eq. 13), whereas TSIS averages these scores across generation pairs and both matching directions (Eqs. 14–15). We explicitly define NbN_b and NsN_s as the numbers of base and source generations.

Figure 4 and Table 1 (Experimental clarity): We replaced the previous Figure 4 with an ablation study (now Table 1) that separately evaluates spatial matching and attention weighting across seven diffusion models. We also clarified the interpretation of the visual-attribute heatmaps (now Fig. 4), including their axes, color scales, and conclusions. The reference encoders in the geometry comparison (now Table 2) are explicitly labeled, and we clarified how their similarity metrics are computed and interpreted.

Related Work (Earlier placement): We moved the Related Work section immediately after the Introduction (Sec. 2) and expanded the discussion of semantic correspondence, representation similarity, and shared representational structures. We also clarify how SIS and TSIS differ from prior correspondence methods by examining concept-level relational geometry across generations.

We hope these revisions make the method, experimental comparisons, and main contributions easier to understand and evaluate.

Q1: Robustness to object size and one-to-many matching

Thank you for raising this important concern. We agree that one-to-many matching could make SIS sensitive to differences in object size. To examine this, we added two analyses in Appendix A.1 (Figs. 7–8).

First, we analyzed 14,000 generated images across seven diffusion models, measuring the correlation between object area (estimated using SAM 3) and average similarity scores. TSIS shows a very weak correlation (r = −0.064, p<0.001), compared with mean pooling (r = −0.466, p<0.001), suggesting that its scores are largely insensitive to object size.

Second, we constructed a dataset with controlled variations in object size and position. TSIS maintains the highest self-classification accuracy compared with GAP and Vanilla (TSIS without attention weighting), even under these variations (Fig. 8).

These results suggest that one-to-many matching does not substantially compromise the robustness of TSIS to object size and layout variations in our experiments.

Q2: Block–timestep selection and stability

Thank you for pointing out this inconsistency. We have clarified the selection procedure in Sec. 5 (“Representation Selection”) and Appendix B.1.

Specifically, we first perform 5-fold self-class prediction at every block–timestep pair (l,t)(l,t). We then select the block with the highest accuracy averaged over all timesteps and fix the timestep to the inference step whose normalized scheduler timestep is closest to 0.8 (where 1 denotes pure noise). Thus, we do not select the (l,t)(l,t) pair with the highest individual accuracy. We have also removed the inconsistent tie-breaker description and clarified the timestep indexing in Appendix B.1 and Table 4.

To examine stability, we added an analysis in Appendix A.2 (Fig. 9) comparing affinity matrices from the top and bottom 10% of block–timestep pairs ranked by self-classification accuracy. The top-performing pairs consistently show higher agreement across all seven models, with pairwise cosine similarities of 0.68–0.89 for most models, compared with 0.25–0.49 for the bottom-performing pairs.

These results suggest that the reported conceptual geometries are not specific to a single selected block–timestep pair but remain consistent across other high-performing representations.

Q3: Generalization across object categories

Thank you for this important suggestion. We agree that evaluating a single object category is insufficient to establish generality. To address this, we added two sets of experiments in Appendix C.3.

First, Figure 12 extends the teddy bear analysis to four diffusion models (SANA-1.5, SD3.5 Medium, Lumina 2.0, and FLUX.1-dev), showing similar attribute-specific patterns across models.

Second, Figure 13 extends the analysis to four additional object categories (house, chair, sneaker, and toy cat) using SANA-1.5. We observe similar patterns in the block–timestep accuracy maps and attribute-specific UMAP structures, although their exact locations and separability vary across objects.

These results suggest that the observed attribute-specific organization is not limited to teddy bears, but extends to other object categories, while retaining model- and object-dependent differences.

Q4: Attribute discriminability and its connection to generation

Thank you for this important point. We agree that higher self-classification accuracy indicates stronger attribute discriminability, but does not necessarily imply a stronger causal role in generation. We have revised Sec. 5.3 to clarify this distinction and avoid overstating our findings.

First, to examine how concept relationships evolve during denoising, we added a temporal affinity analysis in Appendix D (Fig. 20). Using SD3.5 Medium at a fixed block (l=10l=10), we track TSIS affinities between hamster and 20 other concepts across denoising steps. The results show that broad semantic organization is largely maintained, while individual concept affinities follow different trajectories. This directly demonstrates how pairwise concept relationships change during denoising, beyond attribute classification accuracy.

Second, to investigate the connection to generation, we added preliminary feature interventions in Appendix C.4 (Fig. 14). We inject simple feature-difference directions into the residual updates at attribute-specific block–timestep regions identified by TSIS. These interventions induce visible changes in color, material, and viewpoint.

While the intervention results suggest a connection between attribute-discriminative regions and generation, the effects are not fully disentangled. We therefore present them as preliminary evidence rather than definitive causal validation. Further investigation using more targeted feature editing methods is needed to establish this connection.

Q5: Practical utility of conceptual geometry

Thank you for this valuable question. We agree that comparing only the highest- and lowest-scoring groups does not establish an advantage over simpler pooled features. To address this concern, we added two experiments.

First, we directly compare TSIS with globally average-pooled (GAP) features using all 1,000 handbag generations (Appendix C.5, Figs. 15–16). We compute Spearman correlations between intermediate similarity scores and two image-level measures: Rarity Score (RS) and CLIP similarity (CS).

TSIS achieves stronger absolute correlations in five of six comparisons. For example, TSIS shows stronger correlations with RS for color (−0.412 vs. −0.296) and viewpoint (−0.553 vs. −0.371), and with CS for material (0.466 vs. 0.282). GAP performs better only for material RS (−0.416 vs. −0.325). These results suggest that TSIS generally captures image-level atypicality more effectively than GAP, beyond the extreme-group comparison.

Second, we explore another practical use of conceptual geometry for compositional generation (Appendix C.7, Fig. 19). We compare FLUX.1-dev and SD3.5 Large using six concept pairs with different relative TSIS affinities. Across all six illustrated pairs, the model with higher concept affinity shows lower CLIP alignment when generating the two concepts together. This suggests that closely related concepts may be more difficult to generate jointly, and that TSIS can help identify model-specific differences in compositional generation behavior.

As a result, these results demonstrate the potential utility of TSIS for early atypicality screening and diagnosing compositional generation behavior. While the results are encouraging, broader evaluations are needed to establish their generality.

Q6: Effect of affinity matrix preprocessing

Thank you for raising this question. We have clarified the preprocessing procedure and added a comparison without normalization in Appendix B.2 (Table 5).

First, the main Table 2 uses symmetrized and Z-normalized affinity matrices, without clipping negative values. Clipping is applied only for visualization in Figures 1 and 11.

Second, we recomputed CS and L2 using raw affinity matrices without normalization, symmetrization, or clipping (Appendix Table 5). CLIP still achieves the highest CS for most diffusion models, although the absolute values differ substantially. Raw CS can be influenced by nonzero mean affinities, while raw L2 is sensitive to differences in mean and scale. Since L2 accumulates these differences over 4,950 concept pairs, even moderate mean offsets can produce large distances.

Z-normalization removes these differences and focuses the comparison on relative concept relationships. Under this normalization, CS and L2 yield equivalent rankings. We therefore use normalized comparisons as our primary analysis and report the raw results to clarify the effect of preprocessing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.