acceptodds
Under review as a conference paper at ICLR 2027

Representation Conditioning for Generation

Abstract

Representation-alignment methods effectively accelerate the diffusion transformer training. REPA, for example, uses a pretrained vision encoder to supervise an intermediate layer through an *MLP projection head* which is discarded after training. It aims to import semantics from the vision encoder to the diffusion backbone. While this objective is achieved at *early layers* by back-propagating the alignment loss, there is no guarantee that *later layers*, supervised by the velocity prediction loss alone, can well access such semantics. We propose a simple solution, representation conditioning (RECON), to improve semantics in later layers. During training, we feed the pretrained vision encoder tokens into later layers; during inference, because we no longer have the clean image, we reuse the MLP head (as a proxy to the vision encoder) and project its output tokens to the later layers. Our method adds no parameters and leads to much faster convergence: RECON is faster than SiT and faster than REPA.

Primary area:
generative models
Keywords:
Diffusion Model Acceleration, Representation Conditioning, Representation Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_d1xhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A simple representation conditioning (RECON) method is proposed, to improve semantics in later layers. Instead of discarding the MLP projection head after training, RECON keeps it at inference, pools its output into semantic tokens and injects them into later transformer layers. During training, the pretrained vision encoder tokens are transferred into later layers; during inference, the MLP head is used (as a proxy to the vision encoder) and its output tokens are feed to the later layers.

Critical strengths

REPA discards the projection head which holds strongest aligned visual semantics. The idea to reuse it at inference is intuitive. Empirically RECON obtains much faster training convergence, improves layer‑wise semantics (CKNNA, linear probing), achieves strong gFID without CFG on ImageNet‑256, and also shows gains on text‑image generation tasks.

Critical weaknesses
  1. RECON is worse than REG and multiple methods with CFG. There lacks quantitative representation‑level evidence verifying why conditional‑unconditional gap shrinks. Guidance scale sweep experiments are missing. This largely limits practical usage for real‑world generation pipelines.
  2. Noticeable fidelity‑diversity trade‑off: precision improves significantly while recall drops. It remains unclear whether this trade‑off is intrinsic or tunable via hyper‑parameters. Additional diversity analysis is required.
  3. No inference‑time overhead analysis. Training convergence speedup is evaluated, but memory and latency cost induced and extra semantic tokens are not reported.
Questions for authors

Why RECON is worse than REG with CFG? How to improve its performance? Is the computation cost and time latency less than REG? Why precision improves significantly while recall drops?

5 comments

We thank the reviewer for their time and for the valuable feedback to improve our paper. We acknowledge your concerns and address them below. We would be happy to engage in further discussion and clarify any remaining questions.

Q1. Why is RECON worse than REG with CFG?

We compared RECON and REG [1] at the same training length (2.4M steps, batch 256) on 4,000 ImageNet validation images, with xtx_t built from real latents. We measured two quantities (Table R1, Fig. R1, Fig. R2):

  • the relative CFG gap ∥vc−vu∥/∥vc∥\lVert v_c - v_u\rVert / \lVert v_c\rVert between the conditional and unconditional predictions;
  • the class accuracy of the block-8 features. We map the features to the DINOv2 space with the projection head, pool them, and take the nearest DINOv2 class mean. This measures how much class information the features carry (DINOv2 on the clean images: 59.1%).

Table R1. RECON vs. REG at 2.4M steps.

tt0.250.450.650.85
CFG gap, REG0.0910.1040.0990.090
CFG gap, RECON0.1280.1820.2000.133
Class accuracy, true label, REG59.0%59.4%60.8%69.2%
Class accuracy, true label, RECON62.7%65.3%70.7%83.4%
Class accuracy, null label, REG57.4%57.8%57.6%51.1%
Class accuracy, null label, RECON48.6%39.5%16.4%1.6%

With the true label, the features of RECON carry more class information than those of REG. With the null label, the order reverses. REG keeps 57–58% up to t=0.75t = 0.75, while RECON falls to 16% at t=0.65t = 0.65. The reason is the source of the semantic token. In REG, the class token is part of the noisy input, and both CFG branches receive the same token. So the unconditional branch keeps the class. In RECON, each branch predicts its own semantic token from its own label. So the unconditional branch loses the class at high noise. The two branches of RECON therefore differ much more, and the CFG gap is up to 2× that of REG.

Note: at high noise, the features mostly reflect the label rather than the image, so the class accuracy can go above the clean-image score for both RECON and REG.

Effect on CFG. The CFG update is (w−1)(vc−vu)(w-1)(v_c - v_u), so at one scale RECON receives about twice the update of REG. CFG therefore over-guides RECON at the scales that suit REG: larger scales raise IS but lower recall and worsen FID. Fig. R3 shows the CFG sweep of RECON (480 epochs, SDE-250, FID-10K). The best scale is w=1.5w = 1.5–1.71.7 (FID-10K 5.19 → 4.92). REG uses w=2.3w = 2.3 in its official evaluation [1]. Over t=0.45t = 0.45–0.650.65, RECON at w≈1.65w \approx 1.65–1.751.75 already receives the update of REG at w=2.3w = 2.3. This matches the optimum of our sweep.

Correction to the submission. In Section 7, we guessed that the backbone is rich in semantics without the class label, so the unconditional prediction is close to the conditional one. These measurements show the opposite. The unconditional branch of RECON loses the class, and its gap to the conditional branch is larger than in REG. CFG is less effective for RECON for this reason. We will replace the previous hunch with this analysis. Q2 gives our solution.

Figures:

Fig. R1 (relative CFG gap): https://freeimage.host/i/o9WYoEF

Fig. R2 (class accuracy of block-8 features): https://freeimage.host/i/o9WYApR

Fig. R3 (CFG sweep, FID-10K, SDE-250): https://freeimage.host/i/o9WYukv

References

[1] Wu, Ge, et al. "Representation entanglement for generation: Training diffusion transformers is much easier than you think." Advances in Neural Information Processing Systems 38 (2025).

Q2. How can the guided performance of RECON be improved?

Autoguidance. Autoguidance [2] guides with a weaker version of the same model and gives the same label to both models. Its strength therefore does not depend on how much class information an unconditional branch keeps (Q1). As the weak model, we use an early checkpoint of RECON: RECON checkpoint at 125K iterations from the 480-epoch run.

RAE [3] reports the same problem with CFG: without an interval it does not improve FID, and after a grid search over scale and interval it is still worse than autoguidance, so RAE uses autoguidance by default. For RECON at 80 epochs, autoguidance gives FID-50K 2.26 against 2.48 for the best CFG, with higher recall (0.570 vs. 0.558). Fig. R4 shows the autoguidance sweep at 480 epochs (SDE-100, FID-10K). Autoguidance lowers FID-10K from 5.32 to 4.73 and raises recall from 0.684 to 0.696, while CFG lowers recall (Fig. R3 in Q1).

Two training-free improvements of autoguidance. With zt=(1−t)z0+tz1z_t = (1-t) z_0 + t z_1 (t=1t = 1 is pure noise), a velocity prediction vv gives the clean estimate z^0=zt−tv\hat z_0 = z_t - t v. With the strong and weak clean estimates z^0s\hat z_0^s and z^0w\hat z_0^w, autoguidance reads z^0=z^0s+(w−1)Δ\hat z_0 = \hat z_0^s + (w-1)\Delta with Δ=z^0s−z^0w\Delta = \hat z_0^s - \hat z_0^w.

  1. Projected autoguidance. At each spatial position, the latent is a vector of 4 channels. There, the part of Δ\Delta parallel to z^0s\hat z_0^s does not change the direction of z^0s\hat z_0^s. It only makes it longer or shorter, and at large ww this over-saturates the image. We remove it at every position: Δ⊥=Δ−⟨Δ,z^0s⟩∥z^0s∥2z^0s,z^0=z^0s+(w−1)Δ⊥.\Delta_\perp = \Delta - \frac{\langle \Delta, \hat z_0^s\rangle}{\lVert \hat z_0^s\rVert^2}\hat z_0^s, \qquad \hat z_0 = \hat z_0^s + (w-1)\Delta_\perp . Samples then tolerate w=2.5w = 2.5–33 instead of about 2, at no extra cost. This is APG [4] without the parallel part, applied per position to the autoguidance update.
  2. Semantic token extrapolation. At sampling, gϕg_\phi predicts the semantic token u^t=Pool(gϕ(ht))\hat u_t = \mathrm{Pool}(g_\phi(h_t)) from the noisy input. At high noise, the model cannot yet tell what the final image will be, so the token is close to the class average (Table R3 in Q3). The token u^1\hat u_1, predicted from the initial noise, is the extreme case: its cosine to the class mean is 0.97. It plays the role of the unconditional prediction in CFG. The difference u^t−u^1\hat u_t - \hat u_1 is what the token knows about this sample beyond the class average. We amplify it in the strong model: ut′=u^1+(1+λ)(u^t−u^1)=u^t+λ(u^t−u^1),λ=0.05.u_t' = \hat u_1 + (1+\lambda)(\hat u_t - \hat u_1) = \hat u_t + \lambda(\hat u_t - \hat u_1), \qquad \lambda = 0.05 . The weak model is unchanged. u^1\hat u_1 needs only blocks 1–8 and gϕg_\phi, once per sample. Samples become less typical of their class, which raises recall (Q3).

Table R2.

ModelGuidancewwFIDPrec.Rec.
80 epnone–3.240.8090.561
80 epCFG, t≤0.5t \le 0.51.52.480.8330.558
80 epAG2.02.260.8250.570
80 epAG + projection3.02.170.8170.581
80 epAG + extrapolation2.02.190.8020.595
80 epAG + both2.52.100.7990.602
480 epAG1.41.800.8180.597
480 epAG + both2.01.500.7980.628

Figures:

Fig. R4 (autoguidance sweep, FID-10K, SDE-100): https://freeimage.host/i/o9WY5Ip

References

[2] Karras, Tero, et al. "Guiding a diffusion model with a bad version of itself." Advances in Neural Information Processing Systems 37 (2024).

[3] Zheng, Boyang, et al. "Diffusion transformers with representation autoencoders." International Conference on Learning Representations (2026).

[4] Sadat, Seyedmorteza, et al. "Eliminating oversaturation and artifacts of high guidance scales in diffusion models." International Conference on Learning Representations (2025).

Reviewer_cU7EOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

RECON extends REPA by conditioning later diffusion-transformer blocks on semantic tokens. During training, these tokens are pooled from clean-image encoder features and injected below a noise threshold. At inference, the existing REPA projection head is used instead to predict the missing clean-image encoder features directly from intermediate diffusion features. The method substantially improves generation over REPA across several settings.

Critical strengths
  • RECON simple and practical design. RECON reuses an existing projection head and introduces a lightweight conditioning mechanism.

  • RECON leads to substantial improvements over REPA. The gains extend across model sizes, target encoders, tokenizers, and seem visible in text-to-image settings as well. The paper also studies the robustness to fewer sampling step, and it seems promising.

  • The paper investigates conditioning sources, injection schedules, token count, and alignment depth through useful ablations, although important mechanistic questions remain, it provides interesting insights.

Critical weaknesses

1/ My main concern is the novelty relative to closely related work. The benefits of incorporating pretrained visual representations into generation are already well established. REG further demonstrates substantial gains from jointly generating semantic representations and image latents, and explicitly evaluates both class tokens and averaged DINO features. Consequently, RECON’s use of pooling does not establish a fundamental advantage over REG in compatibility with encoders that lack a class token.

The meaningful distinction is that RECON predicts the semantic token from intermediate diffusion features rather than jointly diffusing it with the image. While this provides a simple and practical extension of REPA, the paper does not sufficiently establish the advantages of this design over existing alternatives or offer a clear insight into why it should be preferable. Stronger controlled comparisons with REG, together with the currently missing comparisons with REGLUE, are needed to substantiate the contribution.

2/ One explanation is incomplete. Injecting predicted tokens at high-noise inference steps helps substantially despite their absence during training in that regime. This important result is insufficiently explained given that it seems critical in the overall performance. Additional tokens could also provide a useful computational workspace, independently of their semantic content acting like register.

3/ It seems like improved fidelity comes also with reduced coverage. Precision increases while recall decreases. The smaller gains from CFG, including in Tables 3–4, suggest possible overlap with the effects of guidance, although this is not clearly established. REG remains stronger overall when looking at best performance.

Questions for authors

1/ What advantage does predicting semantic tokens offer over jointly diffusing them? Please provide controlled comparisons with REG, including its pooled-feature variant, and REGLUE under matched training and sampling budgets. Report inference FLOPs, latency, deployed parameter counts, and uncertainty (at least for small performance differences).

2/ Why does high-noise token injection improve inference despite the training mismatch? Please isolate injection over different timestep intervals and measure prediction accuracy and token sensitivity along sampling trajectories. Also, if the authors could include controls using learned register tokens, random tokens, and pooled backbone features with comparable compute to clearly distinguish semantic conditioning from additional computational capacity.

3/ Why do the authors need to pool the semantic features so aggressively? The default single global token then makes RECON resembles the global conditioning used by REG, that is criticised in the paper. Beyond the existing one versus four token ablation, would retaining all spatial tokens improve performance? The degradation with additional tokens should be investigate, whether it arises from prediction error, compute, or the conditioning mechanism itself.

4/ How much of the improvement survives a matched guidance and diversity comparison? Please sweep CFG strength for both REPA and RECON and report FID, precision, and recall, including comparisons at matched recall. In particular, Table 3 and 4 reports only one w value, that makes it harder to validate.

Additionally , if the other could harmonise Table 1 and Table 2 where one uses Iterations, and the other Epochs, so the results can be compared more seamlessly.

The authors cite SRA in the related work but omit its use of an EMA teacher. I suggest clarifying this point and also citing LayerSync (https://arxiv.org/abs/2510.12581), which actually aligns intermediate representations within the current network using the same forward pass.

8 comments

We thank the reviewer for their time and for the valuable feedback to improve our paper. Below we have provided answers for the raised questions.

W1. Our message and novelty.

Our message. A diffusion transformer trained with representation alignment already learns to predict the semantics of the clean image. The message of RECON is that this prediction should be used at sampling, not discarded.

RECON is not tied to REPA. This principle applies to any method that trains an alignment head, whatever its target: an external encoder (REPA), an EMA copy of the model (SRA [1]), or a later layer of the same forward pass (LayerSync [2]). To test this, we added RECON to SRA and LayerSync, each with its official loss (Table A). RECON improves both, by 3.9 and 3.1 FID, as it improves REPA in the paper. So RECON is a plug-in that complements the alignment method, not an extension of one specific method.

Table A. SiT-B/2 at 100K steps, FID-50K, SDE with 250 steps, no CFG.

MethodWithout RECONWith RECON
SiT-B/2, no alignment63.73–
SRA60.2356.30
LayerSync56.9653.83

For LayerSync on SiT-B/2, we align layer 4 with layer 8.

Difference from joint diffusion. Joint diffusion methods, such as REG, ReDi [3] and REGLUE [4], also keep semantic features at sampling. But they are built around the features of a fixed encoder, which they add as a second variable that the model generates from noise, with extra input and output heads. RECON keeps the model and its training target. It predicts the features with the existing alignment head gϕg_\phi and conditions the later layers on this prediction in the same forward pass. This is why RECON also applies to self-contained methods.

References

[1] Jiang, Dengyang, et al. "Representation alignment for diffusion transformers without external components." International Conference on Learning Representations (2026).

[2] Haghighi, Yasaman, et al. "LayerSync: Self-aligning intermediate layers." International Conference on Learning Representations (2026).

[3] Kouzelis, Theodoros, et al. "Boosting generative image modeling via joint image-feature synthesis." Advances in Neural Information Processing Systems 38 (2025).

[4] Petsangourakis, Giorgos, et al. "REGLUE Your Latents with Global and Local Semantics for Entangled Diffusion." arXiv preprint arXiv:2512.16636 (2025).

Q1. What does predicting the semantic token give over diffusing it jointly? Controlled comparison with REG and REGLUE, and cost.

Why prediction rather than joint diffusion. The encoder features f(x0)f(x_0) are a deterministic function of the image, so they add no randomness to the distribution that the model must learn. Joint diffusion still gives the token its own noise. At sampling, it generates the token from noise that is independent of the image noise, and the model must keep the two consistent. A predicted token depends only on the current noisy input. The alignment loss trains gϕg_\phi to predict f(x0)f(x_0) from xtx_t. So the token is an estimate of the clean semantics from the current input, and it agrees with xtx_t at every step. One forward pass gives it, and no second variable is diffused.

Controlled comparison. Table B compares the methods at one budget: SiT-XL/2 at 200K steps with batch size 256, no CFG, and FID-50K with the SDE sampler (250 steps).

Table B. SiT-XL/2 at 200K steps, no CFG.

MethodSemantic inputSource of the token at samplingFID-50K
REPAnone–11.1
REGDINOv2 class tokendiffused with the image5.0
REGmean of DINOv2 patch tokensdiffused with the image5.5
REGLUEclass token and compressed patch featuresdiffused with the image4.6
RECONmean of DINOv2 patch tokenspredicted by gϕg_\phi from layer 84.6

Pooling in REG and in RECON. We agree that REG can also use pooled features. With the mean of the patch tokens in place of the class token, REG goes from 5.0 to 5.5. RECON uses the same mean token and reaches 4.6. So RECON is better than both versions of REG, with fewer trainable parameters and fewer training GFLOPs (Table C).

Comparison with REGLUE. RECON and REGLUE give the same FID (4.6), and we do not claim a difference between them. REGLUE diffuses the class token and 16 channels of compressed patch features, and it needs a separate compressor network. RECON uses one pooled token and adds no diffused variable.

Table C. Cost on one H100, bf16, batch size 64, SiT-XL/2 at 256 px.

REPAREGREGLUERECON
Parameters at inference675.1M676.9M677.0M683.2M
Inference GFLOPs per forward pass118.6119.1119.1121.0
Inference latency per forward pass, torch.compile (ms)37.639.539.939.4
Inference peak memory (GiB)4.684.694.734.75
Trainable parameters683.2M685.0M685.2M683.2M
Training GFLOPs per image387.4388.8391.1388.4
Training time per step (ms)242251253248

GFLOPs count one multiply-add as one FLOP. Training GFLOPs include the frozen DINOv2 encoder, and for REGLUE also its frozen compressor. For the trainable parameters of REGLUE, we do not include its compressor (16.6M parameters). REGLUE trains it before the diffusion backbone and keeps it frozen during the backbone training.

We will add Tables B and C to the paper. This adds the joint diffusion methods that were missing from our original comparison, REGLUE and REG with the mean token, and we will also add ReDi under the same budget.

W2, Q2. Why does injection at high noise help, although training never injects there?

This is a fair question. At t≥tmax⁡t \ge t_{\max}, the decoder never receives a token in training, yet it uses the token there at sampling. In short, training at low noise teaches some decoder layers to attend to the token, and these layers keep attending to it at high noise. So what the decoder learns at low noise also works at high noise. The predicted token is less accurate at high noise, but it still lowers the FM loss and improves the samples. Below, we first explain why we expect this, and then test it.

What the network learns at low noise also acts at high noise. One network serves all timesteps, so what it learns at some timesteps also acts at the others. Reward fine-tuning shows this. DRaFT [5] backpropagates only through the last sampling step, and its authors report: "although DRaFT-1 only computes gradients through the final generation step, it generalizes across time and affects the whole sampling chain." The computation of the network also stays similar across noise levels. Sun et al. [6] remove the noise conditioning and find that "most models exhibit graceful degradation, and in some cases, they even perform better without noise conditioning." For diffusion transformers, DiTFastAttn [7] reports "high similarity between the attention outputs of neighboring steps". So we expect the layers that read the token at t<tmax⁡t < t_{\max} to read it also at t≥tmax⁡t \ge t_{\max}.

How we test it. We test this on RECON at 80 epochs (SiT-XL/2, tmax⁡=0.5t_{\max} = 0.5, EMA weights), at 19 timesteps from t=0.05t = 0.05 to t=0.95t = 0.95. This covers both the timesteps where the decoder was trained with a token (t<tmax⁡t < t_{\max}) and those where it was not (t≥tmax⁡t \ge t_{\max}). We use ImageNet validation images (2,000 for the attention experiment, 10,000 for the FM loss experiment).

1. The layers that attend to the token at low noise also attend to it at high noise. Fig. R1 shows how much attention each layer puts on the predicted token Pool(gϕ(ht))\mathrm{Pool}(g_\phi(h_t)), the token used at sampling, at each timestep. Above tmax⁡t_{\max}, the pattern is the same as below it: layers 9, 11 and 19–27 attend to the token, and the other layers do not. Layer 9 stays at 32–35% at every timestep, and layer 11 at 5–8%. In layers 19–27, the attention falls slowly as the noise grows. Averaged over layers 9–28, it falls only from 7.4% to 5.0%.

2. The decoder can use the semantic token at noise levels that it never saw. Attention alone does not show that the token helps. So we compare the FM loss with and without the token, on the same noisy inputs (Fig. R2). We test two tokens: the teacher token Pool(y0)\mathrm{Pool}(y_0), which is accurate at every noise level, and the predicted token, which is used at sampling. Both tokens lower the FM loss at every timestep, also at t≥tmax⁡t \ge t_{\max} (Table D). The teacher token lowers it even more at t≥tmax⁡t \ge t_{\max} (2.73%) than at t<tmax⁡t < t_{\max} (1.45%). The predicted token follows the same curve, but its effect is about half as large. This gap comes from the prediction error of gϕg_\phi, which grows with the noise: the cosine of the predicted token to the teacher token falls from 0.85 at t=0.5t = 0.5 to 0.70 at t=0.9t = 0.9. Even so, injecting the predicted token at t≥tmax⁡t \ge t_{\max} improves FID.

Table D. The token below and above tmax⁡t_{\max}.

t<tmax⁡t < t_{\max}t≥tmax⁡t \ge t_{\max} (no token in training)
Attention on the predicted token, layers 9–28 †7.4%5.0%
FM loss decrease, teacher token ‡1.45%2.73%
FM loss decrease, predicted token ‡0.8%1.3%

† Share of attention that a patch query gives to the token, averaged over the 16 heads, the 256 patch queries, the images and the timesteps. Uniform attention gives 0.39%.

‡ (Lnone−Ltoken)/Lnone(\mathcal L_{\mathrm{none}} - \mathcal L_{\mathrm{token}}) / \mathcal L_{\mathrm{none}}, where L\mathcal L is the FM loss averaged over the images and the timesteps. All conditions use the same noisy inputs.

Figures:

Fig. R1 (attention on the predicted token, by layer and timestep): https://freeimage.host/i/oH0AgZN

Fig. R2 (FM loss decrease): https://freeimage.host/i/oH0Avyv

References

[5] Clark, Kevin, et al. "Directly fine-tuning diffusion models on differentiable rewards." International Conference on Learning Representations (2024).

[6] Sun, Qiao, et al. "Is noise conditioning necessary for denoising generative models?" International Conference on Machine Learning (2025).

[7] Yuan, Zhihang, et al. "DiTFastAttn: Attention compression for diffusion transformer models." Advances in Neural Information Processing Systems 37 (2024).

Reviewer_MGFmOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims to improve generation quality and training convergence of diffusion transformers by making the backbone more deeply supervised by the visual encoder's feature in REPA. Specifically, RECON conditions later layers on pooled encoder features during training and substitutes the MLP's predicted features during sampling. Experiments show faster convergence and improved generation quality in both class-conditional and text-to-image settings.

Critical strengths
  1. Reusing the otherwise discarded MLP output for semantic conditioning is a simple and clever idea.
  2. Figure 8 demonstrates consistent gains across B/L/XL model scales, different VAEs, and visual encoders.
Critical weaknesses
  1. The experimental analysis still lacks depth, particularly in Section 6.3 and Section 7. The results only show some interesting "effects", but cannot explain "why/how" the method works.
  • Figure 6 and 7 show stronger and eariler semantic emergence across timesteps and layers, but mostly describe the outcomes rather than explain the mechanism behind them. For example, one may ask why the operation at layer 8 can improve the LP top-1 uniformly at each layer, and how the time-dependent scheduling leads to improvement at all timesteps. Table 7 and 8 also receive little discussion beyond the final design choices.
  • Some design choices remain less explained. Recent studies such as iREPA and SRA2 highlight the importance of spatial structure over global semantics for representation alignment. However, increasing kk from 1 to 4 substantially worsen FID (attributed to amplified errors), and the paper keeps emphasizing on pooled, higher level semantics. Intuitively, more pooled tokens preserve more useful information, and should also be "at least not detrimental".
  1. The method combines training-time teacher conditioning and sampling-time self- conditioning, without clearly showing their respective roles and effectiveness. The train-inference gap also remains (at least partially) unsolved.
  • The results did not clarify how much of the overall gain comes from teacher-conditioned training vs. semantic feedback during sampling. The latter may also deserve discussion in relation to other studies such as "Conditioning Residuals" and "Registers Matter for Pixel-Space Diffusion Transformers", since empty tokens or self-derived semantic summary might also improve performance.
  • The train-inference gap mainly comes from the y0y_0 (or gϕ(h)g_\phi(h)). However, the features hh at layer 8 still have rich semantics (Figure 2b), and the paper does not clarify why the projector is essential compared with conditioning on pooled hh, which might not suffer from this problem.
  • Furthermore, training injects teacher tokens only at low noise, whereas Table 8 highlights the importance of high-noise injection, where the decoder was not trained with semantic tokens and the predicted features are less accurate.
Questions for authors

Please refer to the weaknesses.

12 comments

We thank the reviewer for their time and for the valuable feedback to improve our paper. Below we have provided answers for the raised questions.

W1. Why does the operation at layer 8 improve the LP top-1 accuracy at every layer?

The answer differs for the layers after layer 8 and for the layers up to layer 8.

Layers 9–28. These layers receive the semantic token, and the patch tokens attend to it. Table A gives the attention mass on the token, which is the share of attention that a patch query gives to the token. We average it over the 16 heads, the 256 patch queries and the images. We measure it on the RECON model of Fig. 6 (SiT-XL/2, 80 epochs, EMA weights) at t=0.25t = 0.25 . The model uses its own predicted token, as in sampling. We use 2,000 ImageNet validation images.

Table A. Attention mass on the semantic token, and LP top-1 gain over REPA.

Layer910–1819–2728
Attention mass †33.6%1.2%11.7%0.5%
LP top-1 gain over REPA (Fig. 6)+2.1+2.2+3.3+3.8

† Uniform attention over the 257 keys (256 patch tokens and the semantic token) would give 0.39%.

The token is the most semantic signal in the sequence. When patch tokens read it, they write its content into the residual stream, which carries it to all later layers. The LP gain also follows how much the token has been read so far, not just the attention at each layer. Layer 9 reads it strongly (33.6%, +2.1 gain). Layers 10–18 barely attend (1.2%) but keep the gain (+2.2). Layers 19–27 read it again (11.7%), and the gain grows to +3.3 (peak +4.0 at layer 24). Layer 28 attends least (0.5%) but inherits the accumulated gain (+3.8).

Layers 1–8. These layers do not see the token. The token changes them through the training gradients. We compare RECON with REPA trained with the same alignment loss weight as RECON, at epochs 10, 20 and 80. Both models use the training forward pass on the same inputs: 512 ImageNet validation images (one per class), the same noise, and five timesteps from t=0.05t = 0.05 to t=0.45t = 0.45. At these timesteps, RECON receives the teacher token, as in training.

The token does part of the work of the flow-matching (FM) loss LFM\mathcal L_{\mathrm{FM}} (Eq. 2): the FM loss of RECON is up to 2.4% lower than that of REPA. The FM loss therefore sends a smaller gradient to layers 1–8, and the alignment loss Lalign\mathcal L_{\mathrm{align}} (Eq. 4) receives a larger share of the gradient there (Table B).

Table B. Gradient norms † on layers 1–8, mean over the five timesteps.

ModelEpoch 10Epoch 20Epoch 80
∥∇LFM∥\lVert \nabla \mathcal L_{\mathrm{FM}} \rVertREPA0.01820.01900.0132
RECON0.01710.01650.0129
∥∇Lalign∥/∥∇LFM∥\lVert \nabla \mathcal L_{\mathrm{align}} \rVert / \lVert \nabla \mathcal L_{\mathrm{FM}} \rVertREPA2.281.982.60
RECON2.532.392.72

† ∇\nabla: gradient with respect to the parameters of layers 1–8, of the loss averaged over the 512 images.

The FM gradient of RECON on layers 1–8 is smaller, and the ratio of the alignment gradient to the FM gradient is larger. So the first layers of RECON receive a bigger share of the alignment gradient. The alignment loss then shapes these layers more strongly, which raises their semantic content and their LP accuracy.

W2. Training injects teacher tokens only at low noise. Why does injection at high noise help (Table 8)?

This is a fair question. At t≥tmax⁡t \ge t_{\max}, the decoder never receives a token in training, yet it uses the token there at sampling. In short, training at low noise teaches some decoder layers to attend to the token, and these layers keep attending to it at high noise. So what the decoder learns at low noise also works at high noise. The predicted token is less accurate at high noise, but it still lowers the FM loss and improves the samples. Below, we first explain why we expect this, and then test it.

What the network learns at low noise also acts at high noise. One network serves all timesteps, so what it learns at some timesteps also acts at the others. Reward fine-tuning shows this. DRaFT [1] backpropagates only through the last sampling step, and its authors report: "although DRaFT-1 only computes gradients through the final generation step, it generalizes across time and affects the whole sampling chain." The computation of the network also stays similar across noise levels. Sun et al. [2] remove the noise conditioning and find that "most models exhibit graceful degradation, and in some cases, they even perform better without noise conditioning." For diffusion transformers, DiTFastAttn [3] reports "high similarity between the attention outputs of neighboring steps". So we expect the layers that read the token at t<tmax⁡t < t_{\max} to read it also at t≥tmax⁡t \ge t_{\max}.

How we test it. We test this on RECON at 80 epochs (SiT-XL/2, tmax⁡=0.5t_{\max} = 0.5, EMA weights), at 19 timesteps from t=0.05t = 0.05 to t=0.95t = 0.95. This covers both the timesteps where the decoder was trained with a token (t<tmax⁡t < t_{\max}) and those where it was not (t≥tmax⁡t \ge t_{\max}). We use ImageNet validation images (2,000 for the attention experiment, 10,000 for the FM loss experiment).

1. The layers that attend to the token at low noise also attend to it at high noise. Fig. R1 shows how much attention each layer puts on the predicted token Pool(gϕ(ht))\mathrm{Pool}(g_\phi(h_t)), the token used at sampling, at each timestep. Above tmax⁡t_{\max}, the pattern is the same as below it: layers 9, 11 and 19–27 attend to the token, and the other layers do not. Layer 9 stays at 32–35% at every timestep, and layer 11 at 5–8%. In layers 19–27, the attention falls slowly as the noise grows. Averaged over layers 9–28, it falls only from 7.4% to 5.0%.

2. The decoder can use the semantic token at noise levels that it never saw. Attention alone does not show that the token helps. So we compare the FM loss with and without the token, on the same noisy inputs (Fig. R2). We test two tokens: the teacher token Pool(y0)\mathrm{Pool}(y_0), which is accurate at every noise level, and the predicted token, which is used at sampling. Both tokens lower the FM loss at every timestep, also at t≥tmax⁡t \ge t_{\max} (Table C). The teacher token lowers it even more at t≥tmax⁡t \ge t_{\max} (2.73%) than at t<tmax⁡t < t_{\max} (1.45%). The predicted token follows the same curve, but its effect is about half as large. This gap comes from the prediction error of gϕg_\phi, which grows with the noise: the cosine of the predicted token to the teacher token falls from 0.85 at t=0.5t = 0.5 to 0.70 at t=0.9t = 0.9. Even so, injecting the predicted token at t≥tmax⁡t \ge t_{\max} improves FID.

Table C. The token below and above tmax⁡t_{\max}.

t<tmax⁡t < t_{\max}t≥tmax⁡t \ge t_{\max} (no token in training)
Attention on the predicted token, layers 9–28 †7.4%5.0%
FM loss decrease, teacher token ‡1.45%2.73%
FM loss decrease, predicted token ‡0.8%1.3%

† Share of attention that a patch query gives to the token, averaged over the 16 heads, the 256 patch queries, the images and the timesteps. Uniform attention gives 0.39%.

‡ (Lnone−Ltoken)/Lnone(\mathcal L_{\mathrm{none}} - \mathcal L_{\mathrm{token}}) / \mathcal L_{\mathrm{none}}, where L\mathcal L is the FM loss averaged over the images and the timesteps. All conditions use the same noisy inputs.

Figures:

Fig. R1 (attention on the predicted token, by layer and timestep): https://freeimage.host/i/oH0AgZN

Fig. R2 (FM loss decrease): https://freeimage.host/i/oH0Avyv

References

[1] Clark, Kevin, et al. "Directly fine-tuning diffusion models on differentiable rewards." International Conference on Learning Representations (2024).

[2] Sun, Qiao, et al. "Is noise conditioning necessary for denoising generative models?" International Conference on Machine Learning (2025).

[3] Yuan, Zhihang, et al. "DiTFastAttn: Attention compression for diffusion transformer models." Advances in Neural Information Processing Systems 37 (2024).

Comments on the paper

1 comment

Dear AC,

We thank you and the reviewers for the constructive feedback. Following the reviews, we ran additional experiments, which we will add to the revised paper. Detailed responses are under each review; here we summarize the main improvements:

  • Generality (cU7E). RECON is not limited to REPA: it also improves SRA and LayerSync, two self-contained methods that do not use an external encoder, by 3.9 and 3.1 FID. This suggests that RECON can be viewed as a plug-in for other representation alignment methods, rather than an extension of REPA only.
  • Comparison with joint diffusion (cU7E, d1xh). In a controlled comparison at 200K training steps without guidance, RECON outperforms REG and matches REGLUE, with the same inference speed as REG and no extra trainable parameters over REPA.
  • Why RECON works (MGFm, cU7E). We explained and showed quantitatively why the semantic token helps even at noise levels never seen in training. We also showed that the performance and faster convergence of RECON do not come from the extra computational capacity of an additional token: with the same compute and training length, adding a learned register [1] or pooled backbone features [2] improves the FID of REPA only from 17.3 to about 16, while RECON reaches 9.3.
  • Number of semantic tokens (MGFm, cU7E). More tokens do help when they are accurate, in line with iREPA and SRA2. The drop in generation quality when increasing the number of semantic tokens comes from prediction errors in the predicted tokens.
  • Guidance (d1xh, cU7E). We identified why CFG is less effective for RECON than for REG, and proposed two training-free sampling methods tailored to RECON, which improve its FID with autoguidance from 1.80 to 1.50.
  • Precision and recall (d1xh, cU7E). We identified the cause of the lower recall and made the precision–recall balance adjustable at sampling, without retraining.

We thank all reviewers again for their time and valuable feedback, which helped us improve the paper.

Sincerely,

The Authors

[1] Starodubcev, Nikita, et al. "Registers Matter for Pixel-Space Diffusion Transformers." International Conference on Machine Learning (2026).

[2] Xiang, Weilai, et al. "Conditioning Residuals for Diffusion Models via Representation Feedback." arXiv preprint arXiv:2505.10999 (2025).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.