acceptodds
Under review as a conference paper at ICLR 2027

Asymmetric Diffusion Transformer: Representation Supervision via Training-Time Queries

13 views

Abstract

Pretrained representations are now widely used to train diffusion transformers, and the open question is not whether to use them but what role to give them. Existing methods answer in two ways: representation alignment (REPA) keeps the representation outside the transformer and pulls an internal activation toward it with an explicit loss, while joint diffusion brings it inside but makes it a variable the model must generate alongside the latent. We introduce **Asym-DiT**, a third option: the representation enters the transformer as a training-time query, yet never conditions the generative path, so it is dropped at inference with no added cost. An asymmetric mask lets representation tokens read the latent while the latent never reads them; the representation solves its own denoising task by reading the latent, and its loss reaches the latent only through backpropagation, reshaping the latent hidden states to expose the information the representation needs. The representation's information enters the latent through this query-induced learning rather than an explicit loss: the representation does not tell the latent what to be, only what information to expose, leaving its form free. Because the objective never fixes that form, it does not compete with generation the way an alignment loss does, and it keeps helping where REPA stalls, scaling better with model size and, unlike REPA, giving large gains when fine-tuning an already trained transformer.

Primary area:
generative models
Keywords:
Diffusion models, Representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_f89nOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Asym-DiT, which introduces a pretrained representation into a diffusion transformer as a training-time query. Projected representation tokens are concatenated with the latent tokens and processed by the same transformer with an asymmetric attention mask - latent tokens cannot attend to representation tokens, and representation tokens attend only to latent tokens and themselves. The representation loss therefore reaches the latent only through backprop, and the representation tokens are dropped at inference. Experiments on text-to-music generation with an internal dataset (MusicFM features as the representation) report improved FAD over vanilla DiT, REPA, ReDi, and REG when training from scratch, larger gains when fine-tuning.

Critical strengths
  • The proposed method yields meaningful improvements both when training from scratch and when fine-tuning, and the gains grow with model size up to 1.7B parameters. It is particularly effective as a post-hoc training signal for already trained models, where REPA offers little gain.

  • The design is simple and practical. It adds no parameters and no inference cost, and since the latent never attends to the representation tokens, it can be attached to a pretrained model without altering its initial behavior. It also does not require token-level correspondence between the latent and the representation, which makes it easier to apply to modalities such as audio, where frame rates rarely match.

Critical weaknesses
  • Several aspects of the experiments would benefit from further clarification. The baselines were ported to audio by the authors, and ReDi performs worse than the vanilla model; the paper attributes this to the direction of information flow, but no further analysis is provided. Since Fig. 5b shows that Asym-DiT itself falls below the baseline at α = 1, it is hard to tell whether the comparisons reflect well-tuned baselines.

  • The fine-tuning results, which are the strongest evidence, would be more convincing with a control in which the vanilla checkpoint is fine-tuned with the identical recipe but without the auxiliary loss. The scaling comparison in Fig. 1 also contrasts fine-tuned Asym-DiT with REPA trained from scratch. In addition, since the representation tokens share all weights with the latent tokens, it is unclear whether the gains come through the latent pathway, as claimed, or through multi-task learning on the shared weights.

  • The method is presented as a general approach for diffusion transformers, but it is evaluated only on text-to-music generation, so its effectiveness in the standard image setting, where the baselines were originally developed, remains unclear.

Questions for authors
  • Could the authors report a control in which the vanilla checkpoint is fine-tuned with the same recipe but without the auxiliary loss, along with the fine-tuning details (steps, learning rate, EMA)?

  • Why was music chosen as the sole domain? Results on image domain would help confirm the advantage over REPA.

9 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the careful review, and for recognizing the simplicity and practicality of Asym-DiT and its effectiveness as a post-hoc training signal. We address each concern below.

Weakness 1.

Several aspects of the experiments would benefit from further clarification. The baselines were ported to audio by the authors, and ReDi performs worse than the vanilla model;

Ans.

[Faithful Porting] We re-audited both baselines against their official repositories and confirmed that their core mechanisms are preserved, with changes limited to those required by the audio domain (App. C.1). Beyond the audit, we tuned both baselines, as described below, to ensure that each is compared under its best configuration.

[REPA Tuning] A sweep over REPA’s loss weight and alignment depth on DiT-B improves it by at most 0.55 FAD over the default configuration, while Asym-DiT remains 6.65 FAD ahead of even the best one at the same training step. The gap is therefore not explained by suboptimal REPA hyperparameters.

Model (75k steps)FAD$\mathrm{_MuQ}$ ↓
Baseline53.38
REPA, L4, λ=0.251.83
REPA, L4, λ=0.5 (REPA default)50.09
REPA, L4, λ=1.549.54
REPA, L8, λ=0.549.68
Asym-DiT42.89

[ReDi Tuning] Representation guidance (RG) [1], proposed in the ReDi paper but disabled by default in its official repository, was not enabled in our submission. With RG, ReDi recovers to the vanilla level at the cost of extra inference compute, but does not surpass it, whereas Asym-DiT improves over vanilla without any inference-time change. We have updated Table 4 accordingly. Note that ReDi, following its original design, differs from Asym-DiT not only in the attention mask but also in the noise coupling, as it diffuses both streams at a shared noise level. We analyze why ReDi does not improve over vanilla in our next answer.

ModelCFGRGFAD$\mathrm{_MuQ}$ ↓
Baseline5N/A42.40
ReDi5Off (ReDi default)58.94
ReDi51.542.41
ReDi5242.15
ReDi52.542.32
Asym-DiT5N/A38.41

[1] Kouzelis, Theodoros, et al., "Boosting generative image modeling via joint image-feature synthesis." NeurIPS25.

--

the paper attributes this to the direction of information flow, but no further analysis is provided.

Ans.

[Direction of Information Flow] We agree that the original paper asserted this without making it precise. The representation’s information can reach the latent by two routes: forward, when latent tokens attend to representation tokens (as in ReDi), or backward, when representation queries read the latent and their loss gradient flows into the latent hidden states. The lat-to-rep mask closes the forward route, so in Asym-DiT the supervision can reach the latent only through the backward pass. Intervening on this backward route, with everything else fixed, shows that the gain depends on it:

Backward routeSettingFAD$\mathrm{_MuQ}$ ↓Vanilla FAD$\mathrm{_MuQ}$
Asym-DiT (intact)PT, 125k40.7646.50
w/o cross-rep maskPT, 125k52.5146.50
Weakened (α=1\alpha=1)PT, 125k51.8946.50
Noise target (emptied)PT, 125k46.5046.50
Asym-DiT (intact)FT, 10k from 300k33.9143.35
Stop-gradient on latent K/VFT, 10k from 300k42.1343.35

Closing the forward route is what makes this supervision usable in practice. First, the representation never conditions the latent, so it is dropped at inference at no cost. Second, as the reviewer noted, Asym-DiT can be attached to a pretrained model without changing its initial behavior, which is where it gives its largest gains (Table 5). Third, it removes the inference-time dependency that the forward route creates: ReDi’s latents direct 20.5% of their attention to representation tokens, and its FAD moves from 58.94 to 42.15 depending only on how the representation is guided at sampling time. Asym-DiT has no such dependency. We have not isolated whether closing the forward route also changes generation quality in an otherwise identical model, which would require a jointly sampled variant outside the design space we study. Our claim is that the gain is carried by the backward route, and that the asymmetry is what allows it to be obtained without making the representation a generative variable. We have revised Sec. 6.2 accordingly.

AuthorsOpenReview ↗

Rebuttal by Authors

Weakness 1 (cont.)

Since Fig. 5b shows that Asym-DiT itself falls below the baseline at α = 1, it is hard to tell whether the comparisons reflect well-tuned baselines.

Ans.

[Controlled Setup] We took particular care to make the comparison controlled, by separating two layers of settings. The shared training recipe, including the codebase, data, model architecture, optimizer, learning rate, EMA, batch size, training steps, and the evaluation protocol, is identical across all methods and taken from the vanilla baseline (App. B and C), so it is not tuned in favor of Asym-DiT. All methods also use the same representation, MusicFM layer 7. The method-specific components, in contrast, follow each method’s official implementation, such as REPA’s alignment depth, loss weight, and projector, and ReDi’s token construction and loss weight. The methods therefore differ only in these components, which we then tuned as follows.

[Tuning of Method-Specific Hyperparameters] For REPA, we performed a training-time search over four configurations of loss weight and alignment depth, matching the size of our search over α\alpha (Table 13); even its best configuration remains 6.65 FAD behind Asym-DiT. The drop of Asym-DiT at α=1\alpha=1 is one point of this search, and corresponds to the regime analyzed in our previous answer, where the representation loss exerts little pressure on the latent. For ReDi, we followed its official training configuration exactly, including the loss weight of its official implementation, and tuned its representation guidance at inference over a range that includes the value used in the ReDi paper (see our first answer to this weakness). We acknowledge that this search is smaller than for the other two methods, as it involves no retraining. Our analysis, however, suggests that further training-time tuning is unlikely to change the conclusion. ReDi’s representation queries direct only 8.4% of their attention to the latents, so adjusting its loss weight would only scale a gradient that barely reaches the latent. Consistently, even Asym-DiT falls below vanilla when only its cross-representation mask is removed at its best α\alpha (52.51 vs. 46.50).

AuthorsOpenReview ↗

Rebuttal by Authors

Weakness 2.

The fine-tuning results, which are the strongest evidence, would be more convincing with a control in which the vanilla checkpoint is fine-tuned with the identical recipe but without the auxiliary loss.

Ans.

[Vanilla Control for Fine-tuning] We thank the reviewer for this suggestion and have added the control to Table 5. Starting from the same vanilla checkpoint (DiT-B at 300k steps and DiT-XXL at 250k steps), we fine-tuned it for the same number of steps (10k and 2.5k, respectively) with the identical recipe, without any auxiliary loss. This recipe is exactly the one used for pre-training (App. A.2): Muon with learning rate 1e-3 for the linear layers, AdamW with learning rate 5e-5 for the remaining parameters, and an EMA decay of 0.9999. All of these were tuned for the vanilla baseline and inherited unchanged; none of these shared hyperparameters were tuned for Asym-DiT. The vanilla, REPA, and Asym-DiT fine-tuning runs therefore differ only in the auxiliary loss.

ModelFAD$_{\mathrm{MuQ}}$ ↓FAD$_{\mathrm{MusicFM}}$ ↓CLAP$_{\mathrm{MuQ-MuLan}}$ ↑
Baseline (300k)43.6342.390.266
vanilla (+10k)43.3542.180.269
REPA finetuning (+10k)43.2440.590.275
Asym-DiT finetuning (+10k)33.9133.380.286
ModelFAD$_{\mathrm{MuQ}}$ ↓FAD$_{\mathrm{MusicFM}}$ ↓CLAP$_{\mathrm{MuQ-MuLan}}$ ↑
Baseline-XXL (250k)28.5330.110.310
vanilla (+2.5k)28.5030.240.310
REPA finetuning (+2.5k)28.1728.930.315
Asym-DiT finetuning (+2.5k)23.3427.110.328

The control shows that continued training alone barely moves the model (MuQ FAD 43.63 → 43.35 for DiT-B and 28.53 → 28.50 for DiT-XXL), and REPA adds little on top of it (0.11 and 0.33 over the control). Asym-DiT, in contrast, improves MuQ FAD by 9.44 and 5.16 over the control under the very same recipe, with consistent gains on MusicFM FAD and CLAP. The fine-tuning gain is therefore attributable to the auxiliary supervision itself, rather than to additional training steps or a favorable recipe.

The scaling comparison in Fig. 1 also contrasts fine-tuned Asym-DiT with REPA trained from scratch.

Ans.

[Revised Fig. 1] We agree, and have revised Fig. 1 (right) so that every comparison uses the same training protocol: filled markers denote pre-training from scratch, and hollow markers denote fine-tuning from the same vanilla checkpoint, for all methods. Under fine-tuning, Asym-DiT outperforms REPA on both FAD and CLAP at every scale, and its advantage persists up to the 1.7B DiT-XXL (Table 5). Under pre-training, Asym-DiT achieves the best FAD at every scale we trained from scratch, while its CLAP is comparable to or slightly below REPA’s.

Reviewer_RpLXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a new way to use pretrained encoders when training diffusion models. It proposes feeding the encoder’s features into the transformer as extra noisy tokens that must be denoised. These tokens can read the audio latents, but the audio latents cannot read them.

Critical strengths
  • The method adds a modest training cost but no cost at inference.
  • The work is well motivated and well executed.
Critical weaknesses

None

Questions for authors
  • REPA also uses a learned MLP projector, so it constrains expressiveness. Asym-DiT's rep branch is effectively a much larger decoder: cross-attention into latent states at every layer. Part of the gain may be decoder capacity.
  • ReDi falls far below vanilla (FAD 58.94), and REPA is slightly worse than vanilla at DiT-B in Fig. 1, unlike their image results. Not sure if they are proper baselines here
  • The method is framed as general ("any pretrained encoder"), but only audio been tested.
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for their insightful comments. We are glad the reviewer finds our work well-motivated and well-executed. We would like to address the questions raised.

Question 1.

REPA also uses a learned MLP projector, so it constrains expressiveness. Asym-DiT's rep branch is effectively a much larger decoder: cross-attention into latent states at every layer. Part of the gain may be decoder capacity.

Ans.

[Decoder Capacity vs. Supervision] We thank the reviewer for this question. We first note that Asym-DiT adds no parameters: the representation tokens are processed by the generator’s own weights, so its “decoder” is the DiT itself rather than an added module. The question is therefore whether the gain comes from the richer access this provides, namely attention into the latents at every layer, or from the supervision itself. To separate the two, we gradually extend REPA fine-tuning toward Asym-DiT, changing one factor at a time (Table 7; DiT-B, λ\lambda = 0.5, 10k fine-tuning steps).

MethodChange from the previous rowFAD$_{\mathrm{MuQ}}$ ↓CLAP$_{\mathrm{MuQ-MuLan}}$ ↑
REPAsingle layer, MLP projector43.240.275
M-REPAall layers, larger MLP41.270.275
Q-REPA-randseparate DiT-sized decoder (random init.) with layer-wise asymmetric attention42.670.272
Q-REPA-copydecoder initialized from the pretrained DiT41.580.275
Q-REPA-sharedecoder shares the generator's weights, i.e., the Asym-DiT architecture39.860.277
Asym-DiTdenoising objective instead of alignment33.910.286

The results lead to three observations.

  • Multi-layer access helps modestly. Reading all layers instead of one improves FAD from 43.24 to 41.27 (M-REPA).
  • Extra capacity alone does not explain the gain. Adding an entire DiT-sized decoder does not improve on M-REPA (42.67 and 41.58 vs. 41.27). Likewise, enlarging REPA’s projector into a Q-former (below table) brings no further gain as it grows from 1 to 12 layers (41.90 → 42.71; App. E, Table 16). In fact, Q-REPA-share, which adds no parameters, outperforms Q-REPA-copy, which carries an additional DiT (39.86 vs. 41.58).
  • The supervision makes the largest difference. Q-REPA-share and Asym-DiT have the same architecture, attention pattern, and capacity, and differ only in the objective. Replacing alignment with Asym-DiT’s denoising objective improves FAD from 39.86 to 33.91. This is the largest single step, accounting for about 64% of the total gain over REPA. The difference lies in what each objective asks of the latent. Alignment requires the latent not just to carry the representation but to look like it, whereas Asym-DiT fixes only what the latent must carry, not what it must look like (Sec. 4.2). This slack lets the latent store the information in whatever form best serves generation.

Decoder capacity therefore does not explain Asym-DiT’s gain; it comes primarily from how the representation supervises the model. This also explains why the gap is especially large in fine-tuning: since Asym-DiT does not dictate the form of the latent, it does not fight the form that the pretrained generator has already learned. We have added these analyses to Sec. 6.3 and App. E of the revised manuscript.

MethodFAD$_{\mathrm{MuQ}}$ ↓CLAP$_{\mathrm{MuQ-MuLan}}$ ↑
REPA w/ MLP43.240.275
REPA w/ 1 layer Q-former41.900.273
REPA w/ 3-layer Q-former42.920.272
REPA w/ 12-layer Q-former42.710.274
AuthorsOpenReview ↗

Rebuttal by Authors

Question 2.

ReDi falls far below vanilla (FAD 58.94).

Ans.

[Implementation Audit and RG Tuning] We shared this concern and re-audited our implementation against the official ReDi repository, confirming that its core mechanism is preserved apart from the necessary domain adaptations. We then optimized its inference-time configuration. Representation guidance (RG), proposed in the ReDi paper but disabled by default in the official repository, was not enabled in our submission. With RG, ReDi recovers to the vanilla level, at the cost of extra inference compute. We have updated Table 4 accordingly.

ModelCFGRGMuQ_FAD ↓
Baseline5N/A42.40
ReDi5Off (ReDi default)58.94
ReDi51.542.41
ReDi5242.15
ReDi52.542.32
Asym-DiT5N/A38.41

[Why ReDi Does Not Improve over Vanilla] Even with tuned RG, ReDi only matches vanilla, and our analysis in Sec. 4.3 explains why. Representation supervision in Asym-DiT is governed by two knobs: the cross-representation mask decides whether the representation query must read the latent, and the noise coupling α\alpha decides how much it needs to. ReDi sits at the unfavorable end of both. It lets representation tokens attend to one another, and it diffuses both streams at a shared noise level, which corresponds to α=1\alpha=1. Our ablations show that either setting alone is enough to push Asym-DiT itself below vanilla: removing the cross-representation mask yields FAD 52.51 (Fig. 5a), and setting α=1\alpha=1 yields 51.89 (Fig. 5b), against 46.50 for vanilla at the same step. In both cases, reading the latent adds little to the representation query’s prediction: without the mask, the query already obtains what it needs from neighboring representation tokens (Fig. 5a), and at α=1\alpha=1, the noisy token itself remains informative enough that the latent offers little beyond it (App. D). The access gap Δ\Delta (Eq. 2) is thus small, and the representation loss exerts little pressure on the latent. ReDi combines both settings, so its lack of improvement is what our analysis would lead us to expect. Asym-DiT, in other words, is not merely ReDi with a different mask; it is the configuration that turns both knobs toward latent-mediated supervision.

REPA is slightly worse than vanilla at DiT-B in Fig. 1, unlike their image results.

Ans.

[REPA in Music] Across all filled points in the revised Fig. 1 (right), REPA’s gains over vanilla at 200k iterations vary with model size. More broadly, both its acceleration and quality gains in music are more modest than in images, and we believe this gap is structural. REPA relies on a one-to-one correspondence between encoder and latent tokens, which images provide for free by resizing the input. This property, however, rarely carries over from images to music. Any way of changing the number of encoder tokens for a given clip, whether by time-stretching the audio or by feeding it to the encoder at a different sample rate, changes its tempo (and, in the latter case, its pitch), effectively turning it into a different piece of music. REPA in music must therefore resample one of the two sequences after encoding, a condition absent from its original image setting. Asym-DiT does not require such a correspondence: representation tokens keep their native frame rate and retrieve their targets from the latents through attention, rather than through a token-wise alignment loss.

Not sure if they are proper baselines here.

Ans.

[Baseline Validity] We understand the concern that baselines performing weakly in music may make the comparison less informative. First, REPA and ReDi serve as controlled comparisons rather than as targets to beat: they represent the two existing roles a pretrained representation can take (Table 1), and we run them with the same encoder, features, and codebase, so that the comparison isolates how the representation is used. The primary reference is vanilla DiT trained under identical settings, over which Asym-DiT improves consistently. Second, both baselines are faithfully implemented and carefully tuned (see our response to Reviewer WUvX and f89n for details). REPA improves over vanilla in our main comparisons, yet even its best configuration from a sweep over loss weight and alignment depth remains 6.65 FAD behind Asym-DiT (49.54 vs. 42.89). ReDi’s original underperformance stemmed from a default inference setting, which we have corrected in Table 4.

AuthorsOpenReview ↗

Rebuttal by Authors

Question 3.

The method is framed as general ("any pretrained encoder"), but only audio been tested.

Ans.

[ImageNet Result] We conducted additional experiments on ImageNet 256×256 by fine-tuning the publicly available SiT [1] and REPA [2] checkpoints with DINOv2 features as the auxiliary representation. As shown below, Asym-DiT further improves generation quality not only on the vanilla SiT-XL but also on the REPA-pretrained SiT-XL. These results provide further evidence that fine-tuning with Asym-DiT is effective beyond music generation. We have revised the Introduction to clarify that our fine-tuning experiments span both music and image generation.

ModelFID ↓
SiT-XL/2 @ 7M [1,2]8.7
+ Vanilla (+2k)9.3
+ REPA (+2k)9.3
+ Asym-DiT (+2k)7.8
SiT-XL/2 + REPA @ 4M [2]5.9
+ REPA (+2k)6.8
+ Asym-DiT (+2k)4.0

[Encoder Generality] Asym-DiT has now been validated with two different encoders across two modalities: MusicFM for music and DINOv2 for images. Within music, it is also robust to how the target is formed from the encoder, as every layer configuration in Table 11 of the revised manuscript improves over both the baseline and REPA. By “any pretrained encoder,” we mean that Asym-DiT imposes no structural requirement on the encoder, such as a token-level correspondence with the latents, since representation tokens read the latents through attention. We have revised this sentence to avoid overstating.

[1] Ma, Nanye, et al., "SiT: Exploring flow and diffusion-based generative models with scalable interpolant transformers." ECCV24.

[2] Yu, Sihyun, et al., "Representation alignment for generation: Training diffusion transformers is easier than you think." ICLR25.

AuthorsOpenReview ↗

A Different Direction for Representation Supervision

Dear Reviewer RpLX,

Thank you again for your encouraging review. Before the discussion closes, we wanted to share what we find most exciting about this work.

Can a pretrained representation improve a generator without ever entering its generative forward pass? This was the rather counterintuitive idea behind Asym-DiT. Instead of aligning hidden states to a representation (REPA), or generating it alongside the latent (ReDi), we let the representation query the latent during training, so its supervision reaches the generator through backpropagation alone.

Your decoder-capacity question helped us investigate this idea further. Even an entire second DiT could not close the gap, whereas changing only the supervision objective, with the architecture fixed, reduced FAD from 39.86 to 33.91. And on REPA's own ImageNet checkpoint, just 2k steps of fine-tuning reduced FID from 5.9 to 4.0.

We don't claim this is the final answer to representation supervision. But we believe it opens a genuinely different direction, one that already works remarkably well and offers a practical way to improve pretrained generators without changing inference.

We hope these results make the broader significance of the idea clearer, and would be grateful if you would consider raising your rating.

Reviewer_WUvXOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Asym-DiT, a way to give diffusion transformers representation supervision without forcing their hidden states to align with a pretrained encoder, as REPA does. During training, noised representation tokens (MusicFM layer-7 features reduced to 128 channels by PCA and globally rescaled) are appended to the latent sequence, and an asymmetric attention mask stops latent tokens from attending to representation tokens and stops representation tokens from attending to each other, so that each representation token acts as a query that must recover its target by reading the latents. The model is trained with Lz+λLrL_z + \lambda L_r (Eq. 1), and the representation tokens are dropped at inference, leaving inference cost unchanged (Sec. 3, Algorithm 1). The mechanism is analyzed through the access gap Δ\Delta (Eq. 2, Fig. 4), a noise-target control (Table 2), attention-mass and linear-probe analyses of the cross-representation mask (Fig. 3, Fig. 5a), and a Bayes-floor and data-processing argument (App. D). On text-to-music generation with an internal licensed corpus, Asym-DiT improves FAD-MuQ from 42.40 to 38.41 when trained from scratch at DiT-B scale, against 40.95 for REPA (Table 4); it gives larger gains as a short fine-tuning stage, from 43.63 to 35.50 for DiT-B and from 28.53 to 23.34 for the 1.7B DiT-XXL (Table 5); and it reaches the baseline's FAD in 4.0× fewer steps (Fig. 1).

Critical strengths
  1. A simple mechanism with no inference cost, backed by careful mechanistic analysis. The method changes only an attention mask and adds tokens that exist only during training (Algorithm 1), and the paper explains why it works. The access gap Δ\Delta is positive and grows during training (Fig. 4), showing that the queries actually use the latents. A noise target gives exactly the baseline FAD (Table 2: 46.50 in both cases), whereas the MusicFM target reaches 40.76 at 125k steps, indicating that the gain comes from the target's information content rather than from the extra tokens or loss term. The cross-representation mask is also well motivated: a linear probe recovers an auxiliary token far better from its nearest auxiliary neighbor than from its nearest latent token (R2≈0.65R^2 \approx 0.65 vs. 0.120.12), and masking this shortcut raises the attention mass on latents from 9.3% to 95.5% (Fig. 3), while removing the mask worsens FAD from 40.76 to 52.51 (Fig. 5a). Together, these analyses support the paper's central idea of supervising through queries that read the latents rather than aligning the latents to the target.
Critical weaknesses
  1. The evidence for generality is narrow and statistically thin compared with the paper's claims. The Abstract and Sec. 1 present Asym-DiT as a general method for diffusion transformers, and the baselines (REPA, ReDi, REG) come from image generation, yet all experiments use a single internal, unreleased text-to-music corpus (Sec. 6.1, App. A). There are no results on a standard public benchmark, such as ImageNet 256×256 under REPA's official setup or a public music benchmark used as the main evaluation. The baselines are re-implemented and ported to audio (App. C.1, e.g., REPA through sinc resampling). REPA's alignment depth and loss weight do not appear to be reported or tuned, and the poor result of the ReDi "separate tokens" variant (58.94) raises the possibility of a porting problem. Moreover, every FAD number comes from a single run with no seeds or variance reported, so differences such as REG 41.68 vs. REPA 40.95 vs. Asym-DiT 38.41 in Table 4 cannot be separated from run-to-run noise, and no sensitivity analysis for λ\lambda was found.

  2. The scaling comparison and the fine-tuning comparison do not use the same protocol. In Fig. 1 (right), Asym-DiT is applied as a fine-tuning stage on top of the vanilla checkpoint, whereas REPA is trained from scratch (Sec. 6.2, "Scaling"), so the claim that Asym-DiT scales better than REPA compares different training protocols. For Table 5, it is unclear whether the vanilla and REPA rows were trained for the same additional steps as the Asym-DiT row, and the step count for the DiT-XXL fine-tune is also unclear, since Table 9 lists "@250k + Asym 2.5k" against "@300k". In addition, what appears to be the same configuration (DiT-B with MusicFM L7) gives 35.50 in Table 5 but 34.62 in Table 8.

Questions for authors
  1. Can you provide results in REPA's official ImageNet 256×256 setup (SiT-B/L or similar, the same number of steps, the official REPA hyperparameters), or else limit the paper's claims to audio generation?
  2. In Table 5 and Fig. 1, were the vanilla and REPA models continued for the same additional steps as Asym-DiT? What are the exact step counts for the DiT-XXL fine-tune? Why does Table 5 report 35.50 while Table 8 reports 34.62 for what appears to be the same configuration?
  3. App. D (around Eq. 2) says latent-access dropout is used during training so that Δ\Delta can be estimated. This is not mentioned in Sec. 3 or Algorithm 1. Is it used in all reported runs, at what rate, and does it affect the FAD results?
  4. Sec. 6.2 reports that, with MusicFM features cached offline, per-step time is nearly unchanged at DiT-B and about 20% higher up to DiT-L. What is the per-step overhead at DiT-XXL, and what is the one-off cost of extracting and caching the MusicFM features (and fitting the PCA) for the full corpus? How large is the speed-up in Fig. 1 when measured in wall-clock time or total compute rather than in steps?
  5. Table 8 shows that Sum (L9–L12) performs on par with or slightly better than the default L7 (FAD-MuQ 33.96 vs. 34.62). How was L7 chosen as the default, and were this and other hyperparameters selected on the same prompt set used to report the final results? Relatedly, what value of λ\lambda in Eq. 1 is used at each model scale, and how sensitive are the results to it?
9 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We sincerely appreciate the reviewer for the constructive and helpful review. We address the concerns raised in the following answers.

Weakness 1.

The Abstract and Sec. 1 present Asym-DiT as a general method for diffusion transformers, and the baselines (REPA, ReDi, REG) come from image generation, yet all experiments use a single internal, unreleased text-to-music corpus (Sec. 6.1, App. A). There are no results on a standard public benchmark, such as ImageNet 256×256 under REPA's official setup.

Ans.

[ImageNet Result] Following the reviewer's suggestion, we conducted additional experiments on ImageNet 256×256 by fine-tuning the publicly available SiT [1] and REPA [2] checkpoints using Asym-DiT (Algorithm 1). We evaluated class-conditional generation following the official REPA evaluation protocol. To isolate the effect of Asym-DiT from that of additional training, we continued training each checkpoint for the same 2k steps without Asym-DiT (vanilla) and with REPA’s own objective. As shown below, Asym-DiT further improves generation quality not only on the vanilla SiT-XL but also on the REPA-pretrained SiT-XL. This suggests that Asym-DiT provides complementary benefits to REPA, rather than merely replicating the effects of representation alignment. We have updated the revised manuscript accordingly.

ModelFID ↓
SiT-XL/2 @ 7M [1,2]8.7
+ Vanilla (+2k)9.3
+ REPA (+2k)9.3
+ Asym-DiT (+2k)7.8
SiT-XL/2 + REPA @ 4M [2]5.9
+ REPA (+2k)6.8
+ Asym-DiT (+2k)4.0

[1] Ma, Nanye, et al., "SiT: Exploring flow and diffusion-based generative models with scalable interpolant transformers." ECCV24.

[2] Yu, Sihyun, et al., "Representation alignment for generation: Training diffusion transformers is easier than you think." ICLR25.

or a public music benchmark used as the main evaluation.

Ans.

[Public benchmark evaluation] Following the reviewer's comments, we expanded and moved Table 9 of the original version to Tables 8, 20, and 21 in the revised paper to compare our Asym-DiT with publicly available music models on a benchmark evaluation set. We selected the instrumental subset of the Song Describer (SDD) [3], since our models are trained on instrumental music; the full file list is disclosed in App. F. The table below is the summary of the benchmark result.

ModelFAD$_{\mathrm{MuQ}}$ (SDD) ↓
MusicGen36.55
ACE-Step 1.533.06
Stable Audio 321.51
Stable Audio Open51.41
DiffRhythm229.39
DiT-XXL34.33
DiT-XXL + Asym-DiT FT29.29

[Public vs Internal evaluation] From the comparison between REPA, we observed the similar trends in both public benchmark and internal set: in pre-training (PT), Asym-DiT outperformed REPA and baseline in FAD_MuQ; and in fine-tuning (FT), Asym-DiT achieves substantially larger gains over the baseline than REPA. Since the public set contains only 446 files (less than 10% of our internal set), FAD estimates on it are noisier; we therefore keep the internal set as the primary evaluation and report SDD as a public replication.

SettingModelFAD$_{\mathrm{MuQ}}$ (internal) ↓FAD$_{\mathrm{MuQ}}$ (SDD) ↓
Pre-training (400k)DiT-B42.4050.44
+ REPA40.9546.96
+ Asym-DiT38.4145.87
Fine-tuning (DiT-B@300k: internal 43.63, SDD 50.82)+ Vanilla (+10k)43.3550.12
+ REPA (+10k)43.2449.52
+ Asym-DiT (+10k)33.9138.71
Fine-tuning (DiT-XXL@250k: internal 28.53, SDD 36.37)+ Vanilla (+2.5k)28.5035.68
+ REPA (+2.5k)28.1735.65
+ Asym-DiT (+2.5k)23.3429.29

[Public benchmark training] Unlike in the image domain, it is extremely difficult to train a generative model in the music domain with public datasets. Among the publicly available music datasets we identified with verifiable license are the Free Music Archive (FMA) [4] and MTG-Jamendo [5]. They, however, provide only approximately 14k and 56k usable tracks, respectively, making them insufficient for large-scale training. While Freesound [6] offers 473k samples, this primarily contains general audio, making it less suitable for training music generation models.

[3] Manco, Ilaria, et al., "The song describer dataset: a corpus of audio captions for music-and-language evaluation." NeurIPS23 Workshop.

[4] Defferrard, Michaël, et al., "FMA: A dataset for music analysis." ISMIR16.

[5] Bogdanov, Dmitry, et al., "The mtg-jamendo dataset for automatic music tagging." ICML19.

[6] Fonseca, Eduardo, et al., "Freesound datasets: a platform for the creation of open audio datasets." ISMIR17.

AuthorsOpenReview ↗

Rebuttal by Authors

Weakness 1 (cont.)

REPA's alignment depth and loss weight do not appear to be reported or tuned.

Ans.

[REPA Report] We have expanded App. C.1 to document REPA's alignment depth, loss weight, and projector architecture.

[REPA Tuning] We additionally conducted a hyperparameter sweep on DiT-B to tune REPA in music domain, focusing on the loss weight and the alignment depth. The result shows that (1) REPA achieves comparable improvements across a range of loss weights, provided the weight is not too small; and (2) applying alignment at 2/3 of transformer (L8) also yields comparable performance to the official REPA setup (1/3 of transformer, i.e., L4). Importantly, the performance gap between REPA and Asym-DiT remains substantially larger than the improvements achievable through REPA hyperparameter tuning. This suggests that the gains of Asym-DiT cannot be explained simply by suboptimal hyperparameter choices for REPA. All experiments use MusicFM L7 features as the representation target.

Model (75k steps)FAD$_{\mathrm{MuQ}}$ ↓
Baseline53.38
REPA, L4, λ=0.251.83
REPA, L4, λ=0.5 (REPA default)50.09
REPA, L4, λ=1.549.54
REPA, L8, λ=0.549.68
Asym-DiT42.89

--

and the poor result of the ReDi "separate tokens" variant (58.94) raises the possibility of a porting problem.

Ans.

[ReDi Implementation and the 58.94 Result] The reviewer’s concern was well founded: the 58.94 in our original Table 4 did not represent ReDi fairly. Following the review, we re-audited our implementation against the official ReDi repository [7]. Training follows the official setup, including the separate-tokens variant (reported as the stronger variant in the original paper) and the official loss weight, with changes only where the audio domain requires them (App. C.2). The components shared with Asym-DiT (data pipeline, PCA projection, total-variance matching, MusicFM L7 features, evaluation) are identical across methods, so the audit focused on the ReDi-specific parts. It found one difference, at inference time. Representation guidance (RG) is proposed in the ReDi paper but disabled by default in the official repository, and it was not enabled in our submission. We enabled RG and swept its scale, reporting the best value, which if anything favors ReDi:

ModelCFGRGFAD$_{\mathrm{MuQ}}$ ↓
Baseline5N/A42.40
ReDi5Off (ReDi default)58.94
ReDi51.542.41
ReDi5242.15
ReDi52.542.32
Asym-DiT5N/A38.41

With RG, ReDi improves from 58.94 to 42.15 and matches the vanilla baseline (42.40), at the cost of extra inference compute and one more sampling hyperparameter. Asym-DiT reaches 38.41 with no change at inference. Table 4 now reports the RG-tuned ReDi (full sweep in App. C.3, Table 15).

[Why ReDi Does Not Improve over Vanilla] Our own ablations are consistent with this outcome. ReDi differs from Asym-DiT in two training-time settings: its representation tokens attend to one another, and both streams are diffused at a shared noise level, which corresponds to α=1\alpha=1. Applying either setting alone to Asym-DiT, within our own codebase, removes its gain and places it below vanilla at the same step (FAD$_{\mathrm{MuQ}}$ 52.51 and 51.89 vs. 46.50 at 125k; Fig. 5). These runs involve no porting, so they show that both settings are unfavorable in our setting, and ReDi combines them.

[7] Kouzelis, Theodoros, et al., "Boosting generative image modeling via joint image-feature synthesis." NeurIPS25.

AuthorsOpenReview ↗

Rebuttal by Authors

Weakness 1 (cont.)

Moreover, every FAD number comes from a single run with no seeds or variance reported, so differences such as REG 41.68 vs. REPA 40.95 vs. Asym-DiT 38.41 in Table 4 cannot be separated from run-to-run noise,

Ans.

[Run-to-run Variability] We ran three independent fine-tuning runs of Asym-DiT from the same checkpoint (DiT-B, λ=0.5\lambda = 0.5, +10k). FADMuQ\mathrm{FAD}_{\mathrm{MuQ}} is 33.75 ± 0.27, and CLAP is 0.285 ± 0.001. The run reported in revised Table 5 (33.91) is the worst of the three, so the reported number is not a favorable selection. The fine-tuning gap over the vanilla control in Table 5 (9.44 FADMuQ\mathrm{FAD}_{\mathrm{MuQ}} ) is more than 30 times this standard deviation. For the pre-training comparison in Table 4, multi-seed runs are in progress and will be reported in the final version. In the meantime, Fig. 1 (left) shows that this gap is not a fluctuation at a single checkpoint: Asym-DiT has a lower FADMuQ\mathrm{FAD}_{\mathrm{MuQ}} than REPA at every evaluated checkpoint from 75k to 400k steps.

RunFAD$_{\mathrm{MuQ}}$ ↓CLAP$_{\mathrm{MuQ-MuLan}}$ ↑
Asym-DiT (+10k) (reported)33.910.286
Asym-DiT (+10k), seed 233.430.285
Asym-DiT (+10k), seed 333.900.284
Mean ± std (n=3)33.75 ± 0.270.285 ± 0.001

and no sensitivity analysis for λ was found.

Ans.

[λ\lambda Sensitivity] We have added a λ\lambda sweep for DiT-B fine-tuning (Table 18). Every λ\lambda from 0.2 to 2.0 yields a large improvement over the vanilla control, and even the worst value in this range (36.14 FADMuQ\mathrm{FAD}_{\mathrm{MuQ}}) outperforms REPA fine-tuning (43.24) by more than 7; only a very small weight (λ=0.02\lambda = 0.02) yields a smaller gain. The conclusion of Table 5 therefore does not depend on the choice of λ\lambda. We use λ=0.5\lambda = 0.5 for DiT-B fine-tuning to match REPA’s default weight, which we also use for REPA fine-tuning and throughout the ablations in Table 7, so that the comparison is controlled. We note that λ\lambda trades off FAD and CLAP to some extent (e.g., λ=2.0\lambda = 2.0 gives the best CLAP), and report both metrics in full.

ModelλFAD$_{\mathrm{MuQ}}$↓CLAP_{\mathrm{MuQ-MuLan}}\ ↑
Baseline @300k043.630.266
Baseline (+10k)043.350.269
REPA (+10k)0.543.240.275
Asym-DiT (+10k)0.0241.510.288
Asym-DiT (+10k, original report)0.235.500.293
Asym-DiT (+10k, new report)0.533.910.286
Asym-DiT (+10k)2.036.140.296

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.