- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims to investigate the intrinsic generative capabilities of diffusion models through semantic directions in the conditioning space. The authors analyze the generative manifold induced by a fixed prompt by generating multiple samples and computing gradients with respect to the conditioning based on pairwise differences between generated images. The collected gradients are decomposed using PCA or SAE to identify semantic directions. The authors argue that these directions can be used to control image generation. To validate their approach, they evaluate tangent space recovery on a synthetic dataset and demonstrate its applicability to SDXL and SDXL-DMD. The experimental results show that controlling the identified semantic directions allows the adjustment of specific attributes in generated images.
Critical strengths
The paper presents the proposed method and its validation in a clear manner. It also shows that diffusion models have semantic directions on the generative manifold, which can be used to control specific attributes in the generation process.
Critical weaknesses
-
Limited novelty and scope: While the paper provides a clear and systematic analysis, the proposed approach has a limited scope. It focuses on extracting information already captured by samples generated under a fixed condition. Although this analysis is meaningful, it is unclear whether the findings provide sufficient new insights beyond characterizing existing variations in the generated samples.
-
Limited generalizability and practical applicability: The analysis is conducted on a limited set of predefined concepts, and extending the proposed approach to different conditions requires separate analysis for each concept. This raises concerns about its scalability, and it is difficult to identify practical applications where such an analysis would be necessary.
Questions for authors
-
Generalizability and practical applicability: The proposed approach requires separate analysis for each concept, which limits its scalability. Could the authors further discuss its applicability to a broader range of conditions and provide practical scenarios where this approach would be beneficial? Can semantic directions discovered from one concept be transferred to another concept or conditioning setting? This could provide further evidence of the scalability and broader applicability of the proposed approach.
-
Generalization across models and modalities: Could the authors extend the analysis to other diffusion models, such as DiT or SD3.x, or to diffusion models in other modalities? This could clarify whether the discovered semantic directions reflect general properties of diffusion models or are specific to the models considered.
-
Analysis of limitations and future directions: The authors acknowledge that the proposed approach has limited applicability. However, the discussion mainly describes these limitations without explaining their underlying causes. Could the authors provide further analysis of why these limitations arise within the current framework and discuss potential directions for addressing them?
6 comments
W1 & W2: Scope and practical applications
We thank the reviewer for finding our analysis clear, systematic and meaningful. We believe, however, that its scope is broader than characterizing the variation already present in the samples: ELROND turns this variation into directions that control it and reach beyond it. Directions estimated from SDXL-DMD's own collapsed samples raise its recall with respect to the teacher from 0.15 to 0.52 (mean over concepts, App. P, Tab. 20), i.e. the steered model generates parts of the teacher's distribution that its samples never contained.
Such an analysis is useful whenever one wants the variation a model can produce without knowing it in advance:
- Restoring diversity in distilled models, without training or the teacher at inference (Sec. 4.3).
- Creative exploration without naming attributes (Sec. 4.2).
- Subject-specific control in a multi-subject prompt.
- Model auditing: revealing which attributes a model varies for a concept to determine model biases.
Transfer and practical applicability beyond images. We additionally apply ELROND to RFdiffusion3, a diffusion model that generates 3D protein structures (see App. O). We select the task of motif-based generation, which keeps a small protein fragment, called a motif, fixed while generating different structures around it. We discover a shared set of noise-specific PCA subspaces from protein generations without a fixed motif and reuse them across 10 different motifs, without repeating gradient collection or PCA fitting for each condition. For each motif and method, we generate 32 candidate structures, each 100 amino acids long. Steering modifies the conditioning of the surrounding structure during high-noise denoising while preserving the specified motif constraints.
We measure diversity as mean pairwise , calculated outside the motif among samples passing a backbone-geometry screen that checks bond lengths and angles, backbone conformations, and atomic clashes. Results are averaged equally across motifs and reported as mean ± standard error, comparing native generation with ELROND and randomly scrambled steering directions at the same strength.
| Condition | Geometry pass (%) ↑ | Diversity among passers ↑ |
|---|---|---|
| RFD3 | 91.25 ± 2.12 | 0.5786 ± 0.0272 |
| Random 0.05 | 95.00 ± 2.14 | 0.5822 ± 0.0263 |
| ELROND 0.05 | 91.25 ± 2.27 | 0.6377 ± 0.0117 |
ELROND increases diversity by 10.2% over native generation and 9.5% over matched random steering, while geometry pass matches native generation and is 3.75 percentage points below random (paired -tests versus random: significant diversity gain, Holm-adjusted ; geometry ). This illustrates a practical application: generating structurally varied scaffold candidates around a prescribed motif for downstream screening, without specifying the desired structural changes in advance. The same learned subspaces support multiple design specifications, sharing the analysis cost across conditions.
Thank you
Thank you for the detailed response and additional experiments.
The transfer experiments help address my concerns. However, the evaluation is still limited to a small set of concepts and prompts. Could the authors further evaluate transferability across a broader range of unseen concepts, prompts, and categories, including generation quality and semantic consistency?
I still have concerns about the scalability and generalizability of the proposed method.
We thank the reviewer for the careful review. We address all points below.
Q1 & W2: Generalizability of the discovered directions and subspaces
As suggested by the reviewer, we study the transferability of individual directions and of whole subspaces. Both transfer well to different conditionings.
Individual directions. A direction discovered for a concept induces similar changes when applied to the same concept in a different prompt and to different concepts (App. X, Fig. 26). For example, a direction found for "A picture of a dog" that turns the dog dark does the same for dogs in longer prompts (on a beach, in a convertible) and for cats, cars, monsters, and people.
Subspaces. Steering with external directions computed for the same concept in a different prompt, or even for an entirely different concept, increases diversity on average almost as much as native directions computed for the prompt itself, with similar prompt adherence (Sec. 4.4, App. V, X).
The results of our method are therefore highly reusable across prompts and concepts, which we find an interesting result in itself and which strengthens the practical applicability of our method. Practical scenarios are discussed in W1 & W2 below.
Q2: Other models and modalities
The submission already includes results showing generalization to the DiT-based Flux Schnell (App. G). However, as requested, we now provide additional results for SD3.5 Large Turbo (MMDiT, App. W). The model is 1.8–3.2× more sensitive to the discovered directions than to random directions of the same norm (sensitivity: DreamSim distance between the image before and after the perturbation), and applying the directions increases diversity (mean pairwise DreamSim distance) substantially on every concept.
As for other modalities, in App. O we extend our method beyond images and apply it to 3D protein structures generation (see also W1 & W2, Transfer and practical applicability beyond images).
Q3: Causes of the limitations and possible remedies
We have updated the paper with a further analysis of the limitations and of future directions (Sec. 5, App. H). We discuss them one by one.