- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Asym-DiT, which introduces a pretrained representation into a diffusion transformer as a training-time query. Projected representation tokens are concatenated with the latent tokens and processed by the same transformer with an asymmetric attention mask - latent tokens cannot attend to representation tokens, and representation tokens attend only to latent tokens and themselves. The representation loss therefore reaches the latent only through backprop, and the representation tokens are dropped at inference. Experiments on text-to-music generation with an internal dataset (MusicFM features as the representation) report improved FAD over vanilla DiT, REPA, ReDi, and REG when training from scratch, larger gains when fine-tuning.
Critical strengths
-
The proposed method yields meaningful improvements both when training from scratch and when fine-tuning, and the gains grow with model size up to 1.7B parameters. It is particularly effective as a post-hoc training signal for already trained models, where REPA offers little gain.
-
The design is simple and practical. It adds no parameters and no inference cost, and since the latent never attends to the representation tokens, it can be attached to a pretrained model without altering its initial behavior. It also does not require token-level correspondence between the latent and the representation, which makes it easier to apply to modalities such as audio, where frame rates rarely match.
Critical weaknesses
-
Several aspects of the experiments would benefit from further clarification. The baselines were ported to audio by the authors, and ReDi performs worse than the vanilla model; the paper attributes this to the direction of information flow, but no further analysis is provided. Since Fig. 5b shows that Asym-DiT itself falls below the baseline at α = 1, it is hard to tell whether the comparisons reflect well-tuned baselines.
-
The fine-tuning results, which are the strongest evidence, would be more convincing with a control in which the vanilla checkpoint is fine-tuned with the identical recipe but without the auxiliary loss. The scaling comparison in Fig. 1 also contrasts fine-tuned Asym-DiT with REPA trained from scratch. In addition, since the representation tokens share all weights with the latent tokens, it is unclear whether the gains come through the latent pathway, as claimed, or through multi-task learning on the shared weights.
-
The method is presented as a general approach for diffusion transformers, but it is evaluated only on text-to-music generation, so its effectiveness in the standard image setting, where the baselines were originally developed, remains unclear.
Questions for authors
-
Could the authors report a control in which the vanilla checkpoint is fine-tuned with the same recipe but without the auxiliary loss, along with the fine-tuning details (steps, learning rate, EMA)?
-
Why was music chosen as the sole domain? Results on image domain would help confirm the advantage over REPA.
9 comments
Rebuttal by Authors
Weakness 1 (cont.)
Since Fig. 5b shows that Asym-DiT itself falls below the baseline at α = 1, it is hard to tell whether the comparisons reflect well-tuned baselines.
Ans.
[Controlled Setup] We took particular care to make the comparison controlled, by separating two layers of settings. The shared training recipe, including the codebase, data, model architecture, optimizer, learning rate, EMA, batch size, training steps, and the evaluation protocol, is identical across all methods and taken from the vanilla baseline (App. B and C), so it is not tuned in favor of Asym-DiT. All methods also use the same representation, MusicFM layer 7. The method-specific components, in contrast, follow each method’s official implementation, such as REPA’s alignment depth, loss weight, and projector, and ReDi’s token construction and loss weight. The methods therefore differ only in these components, which we then tuned as follows.
[Tuning of Method-Specific Hyperparameters] For REPA, we performed a training-time search over four configurations of loss weight and alignment depth, matching the size of our search over (Table 13); even its best configuration remains 6.65 FAD behind Asym-DiT. The drop of Asym-DiT at is one point of this search, and corresponds to the regime analyzed in our previous answer, where the representation loss exerts little pressure on the latent. For ReDi, we followed its official training configuration exactly, including the loss weight of its official implementation, and tuned its representation guidance at inference over a range that includes the value used in the ReDi paper (see our first answer to this weakness). We acknowledge that this search is smaller than for the other two methods, as it involves no retraining. Our analysis, however, suggests that further training-time tuning is unlikely to change the conclusion. ReDi’s representation queries direct only 8.4% of their attention to the latents, so adjusting its loss weight would only scale a gradient that barely reaches the latent. Consistently, even Asym-DiT falls below vanilla when only its cross-representation mask is removed at its best (52.51 vs. 46.50).
Rebuttal by Authors
Weakness 2.
The fine-tuning results, which are the strongest evidence, would be more convincing with a control in which the vanilla checkpoint is fine-tuned with the identical recipe but without the auxiliary loss.
Ans.
[Vanilla Control for Fine-tuning] We thank the reviewer for this suggestion and have added the control to Table 5. Starting from the same vanilla checkpoint (DiT-B at 300k steps and DiT-XXL at 250k steps), we fine-tuned it for the same number of steps (10k and 2.5k, respectively) with the identical recipe, without any auxiliary loss. This recipe is exactly the one used for pre-training (App. A.2): Muon with learning rate 1e-3 for the linear layers, AdamW with learning rate 5e-5 for the remaining parameters, and an EMA decay of 0.9999. All of these were tuned for the vanilla baseline and inherited unchanged; none of these shared hyperparameters were tuned for Asym-DiT. The vanilla, REPA, and Asym-DiT fine-tuning runs therefore differ only in the auxiliary loss.
| Model | FAD$_{\mathrm{MuQ}}$ ↓ | FAD$_{\mathrm{MusicFM}}$ ↓ | CLAP$_{\mathrm{MuQ-MuLan}}$ ↑ |
|---|---|---|---|
| Baseline (300k) | 43.63 | 42.39 | 0.266 |
| vanilla (+10k) | 43.35 | 42.18 | 0.269 |
| REPA finetuning (+10k) | 43.24 | 40.59 | 0.275 |
| Asym-DiT finetuning (+10k) | 33.91 | 33.38 | 0.286 |
| Model | FAD$_{\mathrm{MuQ}}$ ↓ | FAD$_{\mathrm{MusicFM}}$ ↓ | CLAP$_{\mathrm{MuQ-MuLan}}$ ↑ |
|---|---|---|---|
| Baseline-XXL (250k) | 28.53 | 30.11 | 0.310 |
| vanilla (+2.5k) | 28.50 | 30.24 | 0.310 |
| REPA finetuning (+2.5k) | 28.17 | 28.93 | 0.315 |
| Asym-DiT finetuning (+2.5k) | 23.34 | 27.11 | 0.328 |
The control shows that continued training alone barely moves the model (MuQ FAD 43.63 → 43.35 for DiT-B and 28.53 → 28.50 for DiT-XXL), and REPA adds little on top of it (0.11 and 0.33 over the control). Asym-DiT, in contrast, improves MuQ FAD by 9.44 and 5.16 over the control under the very same recipe, with consistent gains on MusicFM FAD and CLAP. The fine-tuning gain is therefore attributable to the auxiliary supervision itself, rather than to additional training steps or a favorable recipe.
The scaling comparison in Fig. 1 also contrasts fine-tuned Asym-DiT with REPA trained from scratch.
Ans.
[Revised Fig. 1] We agree, and have revised Fig. 1 (right) so that every comparison uses the same training protocol: filled markers denote pre-training from scratch, and hollow markers denote fine-tuning from the same vanilla checkpoint, for all methods. Under fine-tuning, Asym-DiT outperforms REPA on both FAD and CLAP at every scale, and its advantage persists up to the 1.7B DiT-XXL (Table 5). Under pre-training, Asym-DiT achieves the best FAD at every scale we trained from scratch, while its CLAP is comparable to or slightly below REPA’s.
Rebuttal by Authors
We thank the reviewer for the careful review, and for recognizing the simplicity and practicality of Asym-DiT and its effectiveness as a post-hoc training signal. We address each concern below.
Weakness 1.
Ans.
[Faithful Porting] We re-audited both baselines against their official repositories and confirmed that their core mechanisms are preserved, with changes limited to those required by the audio domain (App. C.1). Beyond the audit, we tuned both baselines, as described below, to ensure that each is compared under its best configuration.
[REPA Tuning] A sweep over REPA’s loss weight and alignment depth on DiT-B improves it by at most 0.55 FAD over the default configuration, while Asym-DiT remains 6.65 FAD ahead of even the best one at the same training step. The gap is therefore not explained by suboptimal REPA hyperparameters.
[ReDi Tuning] Representation guidance (RG) [1], proposed in the ReDi paper but disabled by default in its official repository, was not enabled in our submission. With RG, ReDi recovers to the vanilla level at the cost of extra inference compute, but does not surpass it, whereas Asym-DiT improves over vanilla without any inference-time change. We have updated Table 4 accordingly. Note that ReDi, following its original design, differs from Asym-DiT not only in the attention mask but also in the noise coupling, as it diffuses both streams at a shared noise level. We analyze why ReDi does not improve over vanilla in our next answer.
[1] Kouzelis, Theodoros, et al., "Boosting generative image modeling via joint image-feature synthesis." NeurIPS25.
--
Ans.
[Direction of Information Flow] We agree that the original paper asserted this without making it precise. The representation’s information can reach the latent by two routes: forward, when latent tokens attend to representation tokens (as in ReDi), or backward, when representation queries read the latent and their loss gradient flows into the latent hidden states. The lat-to-rep mask closes the forward route, so in Asym-DiT the supervision can reach the latent only through the backward pass. Intervening on this backward route, with everything else fixed, shows that the gain depends on it:
Closing the forward route is what makes this supervision usable in practice. First, the representation never conditions the latent, so it is dropped at inference at no cost. Second, as the reviewer noted, Asym-DiT can be attached to a pretrained model without changing its initial behavior, which is where it gives its largest gains (Table 5). Third, it removes the inference-time dependency that the forward route creates: ReDi’s latents direct 20.5% of their attention to representation tokens, and its FAD moves from 58.94 to 42.15 depending only on how the representation is guided at sampling time. Asym-DiT has no such dependency. We have not isolated whether closing the forward route also changes generation quality in an otherwise identical model, which would require a jointly sampled variant outside the design space we study. Our claim is that the gain is carried by the backward route, and that the asymmetry is what allows it to be obtained without making the representation a generative variable. We have revised Sec. 6.2 accordingly.