- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
A simple representation conditioning (RECON) method is proposed, to improve semantics in later layers. Instead of discarding the MLP projection head after training, RECON keeps it at inference, pools its output into semantic tokens and injects them into later transformer layers. During training, the pretrained vision encoder tokens are transferred into later layers; during inference, the MLP head is used (as a proxy to the vision encoder) and its output tokens are feed to the later layers.
Critical strengths
REPA discards the projection head which holds strongest aligned visual semantics. The idea to reuse it at inference is intuitive. Empirically RECON obtains much faster training convergence, improves layer‑wise semantics (CKNNA, linear probing), achieves strong gFID without CFG on ImageNet‑256, and also shows gains on text‑image generation tasks.
Critical weaknesses
- RECON is worse than REG and multiple methods with CFG. There lacks quantitative representation‑level evidence verifying why conditional‑unconditional gap shrinks. Guidance scale sweep experiments are missing. This largely limits practical usage for real‑world generation pipelines.
- Noticeable fidelity‑diversity trade‑off: precision improves significantly while recall drops. It remains unclear whether this trade‑off is intrinsic or tunable via hyper‑parameters. Additional diversity analysis is required.
- No inference‑time overhead analysis. Training convergence speedup is evaluated, but memory and latency cost induced and extra semantic tokens are not reported.
Questions for authors
Why RECON is worse than REG with CFG? How to improve its performance? Is the computation cost and time latency less than REG? Why precision improves significantly while recall drops?
5 comments
Q1. Why is RECON worse than REG with CFG?
We compared RECON and REG [1] at the same training length (2.4M steps, batch 256) on 4,000 ImageNet validation images, with built from real latents. We measured two quantities (Table R1, Fig. R1, Fig. R2):
- the relative CFG gap between the conditional and unconditional predictions;
- the class accuracy of the block-8 features. We map the features to the DINOv2 space with the projection head, pool them, and take the nearest DINOv2 class mean. This measures how much class information the features carry (DINOv2 on the clean images: 59.1%).
Table R1. RECON vs. REG at 2.4M steps.
| 0.25 | 0.45 | 0.65 | 0.85 | |
|---|---|---|---|---|
| CFG gap, REG | 0.091 | 0.104 | 0.099 | 0.090 |
| CFG gap, RECON | 0.128 | 0.182 | 0.200 | 0.133 |
| Class accuracy, true label, REG | 59.0% | 59.4% | 60.8% | 69.2% |
| Class accuracy, true label, RECON | 62.7% | 65.3% | 70.7% | 83.4% |
| Class accuracy, null label, REG | 57.4% | 57.8% | 57.6% | 51.1% |
| Class accuracy, null label, RECON | 48.6% | 39.5% | 16.4% | 1.6% |
With the true label, the features of RECON carry more class information than those of REG. With the null label, the order reverses. REG keeps 57–58% up to , while RECON falls to 16% at . The reason is the source of the semantic token. In REG, the class token is part of the noisy input, and both CFG branches receive the same token. So the unconditional branch keeps the class. In RECON, each branch predicts its own semantic token from its own label. So the unconditional branch loses the class at high noise. The two branches of RECON therefore differ much more, and the CFG gap is up to 2× that of REG.
Note: at high noise, the features mostly reflect the label rather than the image, so the class accuracy can go above the clean-image score for both RECON and REG.
Effect on CFG. The CFG update is , so at one scale RECON receives about twice the update of REG. CFG therefore over-guides RECON at the scales that suit REG: larger scales raise IS but lower recall and worsen FID. Fig. R3 shows the CFG sweep of RECON (480 epochs, SDE-250, FID-10K). The best scale is – (FID-10K 5.19 → 4.92). REG uses in its official evaluation [1]. Over –, RECON at – already receives the update of REG at . This matches the optimum of our sweep.
Correction to the submission. In Section 7, we guessed that the backbone is rich in semantics without the class label, so the unconditional prediction is close to the conditional one. These measurements show the opposite. The unconditional branch of RECON loses the class, and its gap to the conditional branch is larger than in REG. CFG is less effective for RECON for this reason. We will replace the previous hunch with this analysis. Q2 gives our solution.
Figures:
Fig. R1 (relative CFG gap): https://freeimage.host/i/o9WYoEF
Fig. R2 (class accuracy of block-8 features): https://freeimage.host/i/o9WYApR
Fig. R3 (CFG sweep, FID-10K, SDE-250): https://freeimage.host/i/o9WYukv
References
[1] Wu, Ge, et al. "Representation entanglement for generation: Training diffusion transformers is much easier than you think." Advances in Neural Information Processing Systems 38 (2025).
Q2. How can the guided performance of RECON be improved?
Autoguidance. Autoguidance [2] guides with a weaker version of the same model and gives the same label to both models. Its strength therefore does not depend on how much class information an unconditional branch keeps (Q1). As the weak model, we use an early checkpoint of RECON: RECON checkpoint at 125K iterations from the 480-epoch run.
RAE [3] reports the same problem with CFG: without an interval it does not improve FID, and after a grid search over scale and interval it is still worse than autoguidance, so RAE uses autoguidance by default. For RECON at 80 epochs, autoguidance gives FID-50K 2.26 against 2.48 for the best CFG, with higher recall (0.570 vs. 0.558). Fig. R4 shows the autoguidance sweep at 480 epochs (SDE-100, FID-10K). Autoguidance lowers FID-10K from 5.32 to 4.73 and raises recall from 0.684 to 0.696, while CFG lowers recall (Fig. R3 in Q1).
Two training-free improvements of autoguidance. With ( is pure noise), a velocity prediction gives the clean estimate . With the strong and weak clean estimates and , autoguidance reads with .
- Projected autoguidance. At each spatial position, the latent is a vector of 4 channels. There, the part of parallel to does not change the direction of . It only makes it longer or shorter, and at large this over-saturates the image. We remove it at every position: Samples then tolerate – instead of about 2, at no extra cost. This is APG [4] without the parallel part, applied per position to the autoguidance update.
- Semantic token extrapolation. At sampling, predicts the semantic token from the noisy input. At high noise, the model cannot yet tell what the final image will be, so the token is close to the class average (Table R3 in Q3). The token , predicted from the initial noise, is the extreme case: its cosine to the class mean is 0.97. It plays the role of the unconditional prediction in CFG. The difference is what the token knows about this sample beyond the class average. We amplify it in the strong model: The weak model is unchanged. needs only blocks 1–8 and , once per sample. Samples become less typical of their class, which raises recall (Q3).
Table R2.
| Model | Guidance | FID | Prec. | Rec. | |
|---|---|---|---|---|---|
| 80 ep | none | – | 3.24 | 0.809 | 0.561 |
| 80 ep | CFG, | 1.5 | 2.48 | 0.833 | 0.558 |
| 80 ep | AG | 2.0 | 2.26 | 0.825 | 0.570 |
| 80 ep | AG + projection | 3.0 | 2.17 | 0.817 | 0.581 |
| 80 ep | AG + extrapolation | 2.0 | 2.19 | 0.802 | 0.595 |
| 80 ep | AG + both | 2.5 | 2.10 | 0.799 | 0.602 |
| 480 ep | AG | 1.4 | 1.80 | 0.818 | 0.597 |
| 480 ep | AG + both | 2.0 | 1.50 | 0.798 | 0.628 |
Figures:
Fig. R4 (autoguidance sweep, FID-10K, SDE-100): https://freeimage.host/i/o9WY5Ip
References
[2] Karras, Tero, et al. "Guiding a diffusion model with a bad version of itself." Advances in Neural Information Processing Systems 37 (2024).
[3] Zheng, Boyang, et al. "Diffusion transformers with representation autoencoders." International Conference on Learning Representations (2026).
[4] Sadat, Seyedmorteza, et al. "Eliminating oversaturation and artifacts of high guidance scales in diffusion models." International Conference on Learning Representations (2025).
We thank the reviewer for their time and for the valuable feedback to improve our paper. We acknowledge your concerns and address them below. We would be happy to engage in further discussion and clarify any remaining questions.