- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces spatially invariant similarity for diffusion representations by matching spatial features and weighting their correspondence with concept attention. Aggregation across generations produces concept-level affinity matrices. Class-level experiments cover thirteen diffusion models; further analyses examine attribute trajectories and the association between intermediate representational atypicality and unusual final images.
Critical strengths
- SIS makes similarity inspectable by retaining the spatial contributions to the score. Similar overall scores can reflect different image regions, so these maps provide information that a scalar comparison loses. The paper also recognizes that a correspondence map does not uniquely identify the semantic factor responsible for it.
- The analysis links intermediate representations to completed images: the low-TSIS groups have higher rarity and lower CLIP similarity in all twelve tested category–attribute comparisons. The broader class-level analysis across diffusion models adds descriptive evidence of both shared and model-specific organization.
Critical weaknesses
- The comparison with global mean pooling changes both spatial correspondence and attention weighting, so it establishes an advantage for the complete TSIS construction without isolating the contribution of correspondence (Figure 4). The text-encoder comparison also leaves pooling and affinity computation unspecified, making the suggestion that diffusion geometry is not simply inherited from text conditioning difficult to assess. These gaps limit attribution and cross-representation interpretation; the descriptive TSIS structures and image-space associations remain supported.
- The representation-selection rule is inconsistent between the main text and Appendix C. The former maximizes accuracy over block–timestep pairs, while the latter first selects a block by accuracy averaged over timesteps. The timestep near 0.8T is then described as both fixed and a tie-breaker, although several reported models use step 12 of 20. The actual procedure needs to be clear because it determines which representation underlies the reported geometry.
Questions for authors
- How are conditioning-text embeddings pooled and converted into affinities? If results using comparable affinity calculations or attention-controlled pooling are available, do they preserve the reported differences? The implementation clarification is needed to assess the comparison; the additional controls would strengthen attribution.
- Which block–timestep selection rule was actually used for the class-level results? Please reconcile the main-text and appendix descriptions. If results under alternative rules are available, do they materially change the affinities or the comparison with external vision representations?
2 comments
Thank you for the detailed response and the effort invested in addressing my questions. Your clarifications resolve the concerns I raised. I will keep my rating at Weak acceptance.
Thank you for the careful review and for identifying the missing implementation details and inconsistent descriptions of representation selection. Please find our responses below. Changes addressing this review are shown in red in the revised paper (orange for changes shared across reviews).
W1: Contribution of spatial correspondence
We added a three-way ablation comparing global average pooling (GAP), spatial matching without attention weighting (Vanilla), and full TSIS across seven diffusion models (Sec. 5.1, Table 1). Using 20 selected CIFAR-100 classes with 10 images per class, we report 5-fold self-classification accuracy averaged over all evaluated block–timestep pairs:
Comparing Vanilla with GAP isolates the contribution of spatial matching: accuracy improves in six of seven models. Adding attention weighting improves accuracy in four models, matches Vanilla in two, and reduces accuracy in Lumina 2.0.
W1 & Q1: Text feature extraction and affinity comparison
App. B.2 now specifies how we extract and pool the text-conditioning features:
Pairwise cosine similarities between these concept representations form the text-encoder affinity matrix. We use the original pretrained models without additional training or modification.
Diffusion affinities are computed with TSIS; vision-encoder affinities average image-pair similarities for each concept pair. Each matrix is symmetrized and Z-normalized using its off-diagonal mean and standard deviation. For each vision encoder, the matrices are then averaged across diffusion models and normalized again to obtain a common reference geometry.
Table 2 compares these affinity patterns using cosine similarity and L2 distance between their upper-triangular entries (i<j). Negative values are clipped only for visualization. Diffusion–CLIP cosine agreement exceeds diffusion–text agreement in 12 of 13 models, with Kandinsky 3 as the exception.
Table 5 repeats the comparison on raw matrices, without normalization, symmetrization, or clipping. CLIP agreement remains higher than text agreement in 11 of 13 models. SD1.5 changes ordering (text: 0.861; CLIP: 0.856), while Kandinsky 3 favors text in both settings. We use normalized comparisons as the primary analysis because raw cosine similarity is sensitive to mean affinity levels, and raw L2 depends on both mean and scale.
W2 & Q2: Representation selection and stability across locations
The class-level selection rule is now specified consistently in Sec. 5 and App. B.1:
Thus, block selection uses timestep-averaged accuracy, followed by a fixed scheduler-timestep criterion. We do not jointly maximize accuracy over (l,t) or use the timestep as a tie-breaker. The value 0.8 is a normalized scheduler timestep, not 80% of the inference steps; non-uniform scheduler spacing can produce different step indices across models. Table 4 lists the locations used for the class-level results, including Figure 11.
To assess sensitivity to the exact location, we added the analysis in App. A.2, Figure 9. For each of seven models, we select the individual (l,t) pairs in the top 10% and bottom 10% of self-classification accuracy on 20 selected CIFAR-100 classes. We construct an affinity matrix at each location, Z-normalize it, flatten its upper-triangular entries (i<j), and compute the average pairwise cosine similarity within each group.
The top-accuracy group has higher agreement in every evaluated model, showing that high-accuracy locations yield more consistent conceptual geometries. SDXL has high agreement in both groups; its candidate locations are restricted to the mid-block.