- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method for continuous attribute control in the modulation space (M-space) of a pretrained DiT. It builds attribute direction vectors from the differences between paired text representations and adds them during generation. This allows continuous attribute control without training. The paper also introduces SAMB and AGMR to preserve the image’s spatial structure and keep non-target regions unchanged during attribute control. The experiments show a good balance between control strength and content preservation.
Critical strengths
-
The method is simple and requires no additional training. It uses the existing M-space of a pretrained DiT and averages the differences between paired text representations to build reusable attribute directions. There is no need to train an adapter for each attribute or optimize model parameters. This provides a low-cost approach to continuous attribute control.
-
The proposed modules achieve a good balance between control strength and content preservation. In the reported experiments, the method outperforms the compared baselines on three attribute response metrics and the LPIPS and DINO metrics for content preservation. The ablation studies on SAMB and AGMR also support their ability to preserve non-target regions and spatial structure.
Critical weaknesses
-
The process for building attribute direction vectors lacks key details and ablation studies. For example, how many prompt pairs are used? How are these prompts chosen? How do the authors ensure that averaging removes unrelated factors and keeps only the target semantic direction, without introducing other biases?
-
The method is designed around the properties of M-space. It therefore depends on architectures such as FLUX and SD3, which take pooled conditional embeddings as input. Newer generative models, such as Qwen-Image, no longer use these embeddings, and their AdaLN is conditioned only on timestep embeddings. The proposed method therefore cannot be applied to these models.
Questions for authors
-
How is the range of the attribute control coefficient λ determined? How is the “maximum effective control strength for each method” in Section 4.1 measured? For the proposed method, how does the amount of visual change relate to λ? Is the relationship roughly linear?
-
In SAMB, how is the strength coefficient η set for the source and edit branches?
3 comments
Response to Reviewer eJCj (Part 2/2): Question 2
Response to Question 2.
We set η = 0.6: the edited branch blends its own self-attention map with the source map using weights of 0.6 and 0.4, respectively, while the source branch remains unchanged.
The ablation from Appendix N is reproduced below. Smaller η strengthens source-map blending, improving content preservation at the expense of attribute response. We select η = 0.6 to balance attribute response, continuity, and content preservation.
E: extrapolation; C: continuity.
| η | E-CLIP ↑ | E-SigLIP ↑ | E-BLIP ↑ | C-CLIP ↑ | C-SigLIP ↑ | C-BLIP ↑ | LPIPS-Alex ↓ | LPIPS-VGG ↓ | DINO ↓ | ID ↓ |
|---|---|---|---|---|---|---|---|---|---|---|
| 1.0 (no SAMB) | 0.2622 | 0.0788 | 0.5169 | 0.2735 | 0.2207 | 0.2524 | 0.2706 | 0.3066 | 0.1600 | 0.5035 |
| 0.8 | 0.2576 | 0.0773 | 0.5062 | 0.2861 | 0.2302 | 0.3144 | 0.1504 | 0.1928 | 0.0889 | 0.4065 |
| 0.7 | 0.2538 | 0.0768 | 0.4616 | 0.2792 | 0.2226 | 0.2739 | 0.1164 | 0.1599 | 0.0681 | 0.3609 |
| 0.6 (default) | 0.2506 | 0.0758 | 0.4577 | 0.2736 | 0.2117 | 0.2371 | 0.0971 | 0.1407 | 0.0502 | 0.3322 |
| 0.5 | 0.2466 | 0.0741 | 0.3749 | 0.2613 | 0.2092 | 0.2249 | 0.0843 | 0.1275 | 0.0454 | 0.2744 |
| 0.4 | 0.2419 | 0.0706 | 0.3385 | 0.2461 | 0.2012 | 0.2085 | 0.0709 | 0.1198 | 0.0388 | 0.2428 |
Thank you for the response. My concerns regarding the construction of attribute direction vectors and the strategies for selecting certain hyperparameters have been adequately addressed. I decide to maintain the original score.
Response to Reviewer eJCj (Part 1/2): Weaknesses 1-2 & Question 1
Response to Critical Weakness 1.
We thank the reviewer for requesting these details. We use N = 100 prompt pairs per attribute. Within each pair, only the target-attribute phrase changes, while the subject, non-target attributes, and context remain identical. For example, an age-control pair is “a young man in a park” and “an elderly man in a park.” Across pairs, we vary subject descriptions, non-target attributes, and contexts to reduce dependence on any particular prompt.
This construction is motivated by prior work on linear semantic representations and prompt ensembling. The Linear Representation Hypothesis and the Geometry of Large Language Models (ICML 2024) formalizes concept directions using counterfactual word pairs and estimates them by averaging representation differences across pairs. In vision-language applications, StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (ICCV 2021) constructs text directions from differences between template-averaged embeddings to reduce sensitivity to wording. These studies provide motivation and empirical precedents for our direction construction strategy.
Specifically, we compute each pair’s M-space representation difference and average these differences. Matching the descriptions within each pair helps reduce contributions from shared content, while averaging across diverse pairs attenuates wording- and context-dependent variations that are inconsistent across pairs. This is intended to emphasize the shared target-attribute direction; it does not guarantee the removal of all unrelated factors or systematic biases.
To assess sensitivity to the number and selection of prompt pairs, we conducted an ablation over 25 attributes. For each attribute, we construct a reference direction from 10,000 prompt pairs, repeatedly sample subsets of different sizes, and compute the cosine similarity between each subset-derived direction and the reference. We average over repeated subsets within each attribute and then give all attributes equal weight. At N = 100, the mean cosine similarity is approximately 0.992, indicating stable direction estimation across sampled subsets of the evaluated prompt pool. Details are provided in Appendix L.
Response to Critical Weakness 2.
We thank the reviewer for highlighting this architectural dependency. Our current method requires a pooled text condition that contributes to a globally shared modulation representation in the DiT. Both FLUX and SD3.5 provide this interface, allowing us to construct attribute directions in their respective M-spaces without additional training.
We agree that our method cannot be directly applied to architectures without this text-conditioned modulation pathway. Z-Image and Qwen-Image do not provide the corresponding interface required by our method. Consequently, our current training-free formulation cannot be directly applied to these models without modifying the method or introducing additional model-specific adaptations.
More broadly, our findings highlight shared text-conditioned modulation as a useful interface for continuous semantic control. We hope this perspective can inform future DiT architecture design by motivating explicit, accessible conditioning interfaces for controllable generation.
Response to Question 1.
We determine the effective control range for each method by progressively varying its control coefficient and jointly assessing target-attribute response and image quality. For our method, CLIP, SigLIP, and BLIP attribute responses all peak at λ = 18. However, more than 80% of the generated images at this scale exhibit hand abnormalities. We therefore set the maximum effective control strength to λ = 16 to balance attribute response and image quality. Each comparison method is calibrated using the same procedure.
By construction, the displacement in M-space is directly proportional to λ. The resulting visual changes, however, pass through a nonlinear generation process. As shown in Appendix L, the mean CLIP, SigLIP, and BLIP attribute responses exhibit a roughly linear increasing trend over much of the effective control range. At excessively large λ values, image degradation, including noise and other artifacts, contributes to the subsequent decline in these scores.