acceptodds
Under review as a conference paper at ICLR 2027

Single-Transformer Latent Diffusion

Abstract

Latent generative models typically rely on three separate networks: an encoder that tokenizes pixels, a generator that models the latent distribution, and a decoder that maps latents back to pixels. This contrasts with multimodal transformers, where heterogeneous modalities are processed by a single backbone in one unified framework. Recent work narrows the gap by casting tokenization and generation as one latent-inference problem solved by a single generative encoder, but the decoder remains a separate network. We investigate whether this last component can be folded in too, and what it takes for one diffusion transformer to encode, denoise, and decode. We propose a simple formulation that casts all three modes as position-aligned sequence maps over one fully shared backbone; the modes differ only in lightweight input and output heads and in their conditioning, namely a mode token, the timestep, and the class label. On ImageNet-1K at matched parameters, our model, UNIFIED, improves gFID over both a separate tokenizer–generator pipeline and the two-network baseline, with the largest gains at the two smaller scales (2.79 vs. 3.04 at M, 2.44 vs. 2.77 at M) and a smaller gain at M (2.16 vs. 2.23). The results show that a single transformer can encode, denoise, and decode, and that full unification is not merely feasible but can be beneficial: the standalone decoder is a less parameter-efficient use of capacity than the shared backbone, and folding it in can improve generation quality at matched parameters.

Primary area:
generative models
Keywords:
generative models, representation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dizvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether encoding, latent generation, and image decoding can share all Transformer blocks. Starting from UNITE, it introduces UNIFIED, with lightweight task-specific heads, mode tokens, spatially aligned token layouts, and a clean-endpoint timestep convention for encoding and decoding. ImageNet-256 experiments compare parameter-matched models at three scales, examine capacity allocation and sampling behavior, and analyze cross-mode feature similarity.

Critical strengths
  1. The parameter-matched comparisons consistently favor complete sharing across model sizes. Table 2 reports generation FID improvements from 3.04 to 2.79 at Tiny, 2.77 to 2.44 at Small, and 2.23 to 2.16 at Base; Base reconstruction FID improves from 1.41 to 0.97. Only 1.63% of Base parameters are task-specific heads and tokens. The capacity-allocation comparisons in Table 3 make the architectural result more meaningful than comparison with a single arbitrary split.
  2. The paper examines important practical qualifications rather than reporting parameter savings alone. UNITE++ helps separate shared weights from the proposed interface, and the timestep/readout ablations probe design choices. Appendix C discloses the larger training and per-flow-step computation. The fixed-NFE comparison in Figure 4 and endpoint truncation in Table 5 demonstrate a potentially useful quality/sampling tradeoff. The unsuccessful QM9 result further shows that the authors do not assume the image-domain finding transfers automatically.
Critical weaknesses
  1. Equal parameter count is not equal compute, which limits the efficiency conclusion. Appendix C reports 3.58 versus 2.39 TFLOPs per image for a full training step, and 81.7 versus 50.1 GFLOPs per flow-matching forward pass, for UNIFIED versus UNITE. The fixed-NFE results are helpful, but a comparison in which UNITE receives the additional training compute is absent. Sampling choices also matter: Table 2 gives Base generation FID 2.23 versus 2.16, while Table 5 gives 2.22 versus 2.05 when both models stop at 0.99. UNIFIED remains better in that comparison, but a general quality/compute advantage needs consistently matched training and sampling budgets.
  2. The explanation for the improvement is not fully disentangled. Complete sharing, spatial alignment of the token interface, and the encoding/decoding timestep convention change together. Table 4 partially separates these factors, but does not provide a complete interface-by-timestep comparison with the same shared architecture. Higher CKA can also result from aligned inputs and positions, so it is not independent evidence that the tasks learn the same computation or that this similarity causes better generation. This matters because the distinction from earlier joint generative encoders and diffusion-based unified models rests substantially on the proposed compatibility of the three modes.
Questions for authors
  1. Fairness test: What happens if UNITE receives the same total training FLOPs as UNIFIED, with the same guidance and ODE-endpoint selection budget? Please also compare inference FLOPs at matched generation quality, since Appendix C reports a higher cost per denoising call for UNIFIED. This would establish whether the architectural improvement gives a favorable overall quality/compute tradeoff.
  2. Mechanism test: Can the readout-layout and clean-endpoint choices be compared factorially within the same shared backbone, including UNITE++ with the clean-endpoint convention? An aligned-interface model with a separate decoder would further clarify whether the improvement is caused by full sharing or primarily by the interface design.
  3. How does UNIFIED compare with SOTA methods at matched budgets? Can you compare with strong baselines such as DiT, SiT, REPA, REPA-E, and RAE at matched parameter count, training FLOPs, and sampling NFE? For text-to-image, please include small-scale PixArt/SD-style baselines or at least same-tokenizer and same-data controls. Report quality/compute tradeoffs, inference latency, and NFE-to-quality curves.
  4. Can you evaluate UNIFIED on a text-to-image dataset such as CC12M, LAION, or COCO, with a text encoder and appropriate conditioning? How does it compare with UNITE, UNITE++, and the separate pipeline under identical text conditioning, data, training compute, and sampling budget? Does full sharing still help at higher resolution and longer token sequences?
7 comments

We thank the reviewer for the feedback and for highlighting the consistent parameter-matched gains, the controls that separate weight sharing from the interface design, and our reporting of the method's practical qualifications. We answer each point below.

Note on the two UNIFIED variants. The submitted model encodes and decodes at the clean endpoint (tdec=1t_{\mathrm{dec}}=1). The submission found that decoding and generation compete there: ending the sampling ODE at t=0.99t=0.99 improved gFID from 2.16 to 2.05. The revision therefore adds a variant that conditions decoding on its own learned embedding (tdec=−1t_{\mathrm{dec}}=-1; Section 3.3). It adds one embedding vector and changes nothing else. We report both variants throughout, and the improvement over UNITE at every model size holds for both.

W1 / Q1 — Matched training and sampling budgets

With matched budgets, UNIFIED reaches a better final quality than UNITE; UNITE's advantage is limited to the early, lower-quality regime.

  • Training: given UNITE's full training budget, UNIFIED matches its final gFID with 10–19% less compute and improves on it with 3–20% more, at every size. UNITE leads early in training but plateaus over its last 240 EFLOPs, while UNIFIED keeps improving.
  • Sampling: at matched quality, UNIFIED needs fewer sampling FLOPs (2.6 vs. 3.2 TFLOPs to reach gFID 2.50), and at 5.2 TFLOPs it surpasses UNITE at 12.8 TFLOPs (2.23 vs. 2.28). UNITE is cheaper only for coarser targets (e.g., gFID 2.86 at 1.6 TFLOPs). Every model receives the same guidance search, and no model uses endpoint selection.

Training budget and training FLOPs at matched quality. We evaluate the intermediate checkpoints of every model and plot gFID against total training FLOPs (Figure 4 for Base, Figure 6 for Tiny and Small). The table below compares UNITE's final model with the two UNIFIED checkpoints whose training compute brackets it (Table 6; EFLOPs =1018=10^{18} FLOPs; every gFID is the mean over multiple evaluation seeds).

SizeMethodEpochsTraining EFLOPsgFID ↓
TinyUNITE (2:1)2002183.00
UNIFIED (tdec=−1t_{\mathrm{dec}}=-1)1201763.03
UNIFIED (tdec=−1t_{\mathrm{dec}}=-1)1602352.91
SmallUNITE2002962.76
UNIFIED (tdec=−1t_{\mathrm{dec}}=-1)1002542.78
UNIFIED (tdec=−1t_{\mathrm{dec}}=-1)1203052.65
BaseUNITE2006082.23
UNIFIED (tdec=−1t_{\mathrm{dec}}=-1)1205462.22
UNIFIED (tdec=−1t_{\mathrm{dec}}=-1)1607282.15

With 10–19% less training compute, UNIFIED is within 0.03 gFID of UNITE's final model at every size, and with 3–20% more it improves on UNITE at every size. The gain is smaller than at matched epochs (0.15 at Base). Although UNITE leads early in training, its curve flattens over its last 240 EFLOPs (2.26 → 2.23 → 2.23), while UNIFIED keeps improving up to and beyond UNITE's total budget (2.22 → 2.15 → 2.08).

Sampling budget and inference FLOPs at matched quality. With a fixed-step Euler solver, sampling FLOPs per image are NFE × one forward pass (50.1 GFLOPs for UNITE, 81.7 for UNIFIED at Base level). Figure 4 plots gFID against both NFE and sampling FLOPs:

NFEUNITE: TFLOPsUNITE: gFID ↓UNIFIED: TFLOPsUNIFIED (tdec=1t_{\mathrm{dec}}=1): gFID ↓UNIFIED (tdec=−1t_{\mathrm{dec}}=-1): gFID ↓
160.83.861.33.323.40
321.62.862.62.462.50
643.22.505.22.202.23
1286.42.3510.52.112.14
25612.82.2820.92.092.12

The two UNIFIED variants have the same architecture and therefore the same sampling FLOPs. Both UNIFIED variants are better than UNITE at every NFE, and the submitted model (tdec=1t_{\mathrm{dec}}=1) is slightly better than UNIFIED (tdec=−1t_{\mathrm{dec}}=-1) at every NFE under this solver. At matched quality, UNIFIED needs fewer sampling FLOPs for any target gFID of 2.50 or better: it reaches 2.46–2.50 at 2.6 TFLOPs, which UNITE reaches at 3.2 TFLOPs, and 2.20–2.23 at 5.2 TFLOPs, a quality UNITE does not reach even at 12.8 TFLOPs. UNITE is cheaper only for coarser targets.

Guidance budget and endpoint selection. Every model receives the same guidance search: the CFG scale is swept from 1.0 to 4.0 in steps of 0.2, guidance is applied for t∈[0.1,1]t \in [0.1, 1], and each model is reported at its best scale. The fixed-NFE results above reuse each model's best scale from this sweep. Every result above integrates the ODE over the full interval [0,1][0,1], so no model receives endpoint selection. Appendix E (Table 12) analyzes early stopping separately and gives every model the same choices of tendt_{\mathrm{end}}.

Changes: Section 4.4, Figures 4 and 6, Appendix B ("Classifier-free guidance"), Appendix C (Table 6), and Appendix E (Table 11).

W2 / Q2 — Disentangling layout, timestep, and weight sharing

Within the fully shared backbone, the generation gain requires both position alignment and the matched encoding timestep: neither full sharing alone (UNITE++) nor the layout alone improves generation. The aligned-interface model with a separate decoder, which isolates the effect of full sharing, is still training.

Table 3 of the revision separates the factors one at a time rather than in a full factorial design: each row differs from the previous one in a single factor, and UNIFIED (tenc=0t_{\mathrm{enc}}=0, tdec=1t_{\mathrm{dec}}=1) is the bridge between the layout comparison and the timestep comparison. A full factorial design is unfeasible, because one of its cells, readout tokens with tenc=1t_{\mathrm{enc}}=1, is not well defined (see the fourth point below).

ModelBackboneLayouttenct_{\mathrm{enc}}tdect_{\mathrm{dec}}gFID ↓rFID ↓
UNITEencoder + generator shared, separate decoderreadout tokens0–2.231.41
UNITE++fully sharedreadout tokens012.242.20
UNIFIEDfully sharedposition-aligned012.311.39
UNIFIEDfully sharedposition-aligned112.150.97
UNIFIEDfully sharedposition-aligned1−12.080.95
  • UNITE++ is the direct extension of UNITE. It folds the decoder into the backbone and changes nothing else: it keeps the readout tokens and adds a decoding mode at the clean endpoint. UNITE's encoder reads N(0,I)\mathcal{N}(0,I) readouts, so tenc=0t_{\mathrm{enc}}=0 describes its input correctly, and UNITE++ inherits it.
  • Layout (rows 2 and 3). UNITE++ and UNIFIED (tenc=0t_{\mathrm{enc}}=0, tdec=1t_{\mathrm{dec}}=1) have the same fully shared backbone and the same timesteps, and differ only in the layout. Position alignment restores reconstruction (rFID 2.20 → 1.39) but does not by itself improve generation.
  • Encoding timestep (rows 3 and 4). The two UNIFIED models have the same backbone and layout, and differ only in tenct_{\mathrm{enc}}. Both are evaluated at epoch 200; the submission reported the model with tenc=0t_{\mathrm{enc}}=0 at its lowest-gFID checkpoint (epoch 160, gFID 2.20). The clean endpoint improves both generation (2.31 → 2.15) and reconstruction (1.39 → 0.97).
  • UNITE++ with the clean-endpoint convention. This setting is not well defined because the encoder of UNITE++ reads pure-noise readouts adapted from UNITE, and setting tenc=1t_{\mathrm{enc}}=1 would label those latent readouts as clean.
  • Full sharing vs. interface. As the reviewer suggests, we are training UNITE with the aligned interface: its generative encoder maps image patches to latents directly without readout tokens, while keeping the standalone decoder. Comparing it with UNIFIED isolates the effect of sharing the decoder. We will report early results closer to the end of the rebuttal.
  • CKA. We agree that CKA is not independent evidence. The revision no longer says that CKA "confirms" the mechanism, and now reads: "We use CKA only to locate where the two passes diverge, while the effect of each design choice on generation is measured by the ablations in Table 3." The divergence it locates (the last block near the clean endpoint) motivated the learned decoding timestep, which is then tested by its effect on gFID (row 5).

Changes: Section 3.3 ("Decoding timestep"); Section 4.3 (Table 3, and the paragraphs "Readout tokens", "Position alignment", "Clean endpoint for encoding", "CKA analysis", and "Learned decoding timestep").

Q3 — Comparison with SOTA methods at matched budgets

ImageNet. Table 7 of the revision compares UNIFIED with two-stage and single-stage models, including DiT, SiT, REPA, and RAE, and lists the parameter count of all networks, the training epochs, and the pretrained components each method uses. Due to computational constraints, we use the baseline results reported in their papers. Our models train for 200 epochs, against 400–1600 for most of the two-stage models.

MethodPretrained AEPretrained VFMParamsEpochsgFID ↓
DiT-XL/2✓–759M14002.27
SiT-XL/2✓–759M14002.06
REPA-B✓✓214M–2.15
RAE-B✓✓631M–2.08
UNIFIED-B––215M2002.08

UNIFIED-B uses the learned decoding timestep (tdec=−1t_{\mathrm{dec}}=-1). The submitted variant (tdec=1t_{\mathrm{dec}}=1) reaches 2.15 (Table 2).

UNIFIED-B improves on REPA-B and matches RAE-B without a pretrained autoencoder or vision foundation model. Every model in Table 7 with a lower gFID than UNIFIED-B uses at least 545M parameters.

Text-to-image. Table 4 follows the DiffusionBench protocol, so every model has the same training data (JourneyDB + BLIP3-o), text encoder (Qwen3-0.6B), number of training steps, and sampler (50-step Euler, CFG scale 6.0). The baselines cover a latent diffusion model with a pretrained SD-VAE (NanoGen), one with a pretrained FLUX-VAE (DiT$^{\mathrm{DH}}$-XL), and a pixel-space model (JiT):

MethodParamsGenEval ↑DPG-Bench ↑GenAI-Bench ↑
Pretraining (100K steps)
DiT$^{\mathrm{DH}}$-XL (FLUX-VAE)875M + 84M0.4970.7890.637
JiT615M0.5160.7820.674
NanoGen (SD-VAE)611M + 84M0.5780.8040.691
UNIFIED-B218M0.4710.7920.672
UNIFIED-L554M0.5270.7940.685
+ Fine-tuning (2K steps)
DiT$^{\mathrm{DH}}$-XL (FLUX-VAE)875M + 84M0.8410.8350.688
UNIFIED-B218M0.8490.8330.729
UNIFIED-L554M0.8780.8480.748

After pretraining, UNIFIED-L (554M) improves on pixel-space JiT (615M) on all three benchmarks, and UNIFIED-B (218M) is on par with JiT on DPG-Bench and GenAI-Bench. Unlike DiT$^{\mathrm{DH}}$-XL and NanoGen, UNIFIED does not rely on a pretrained VAE, which may have been trained on large-scale external data. Instead, it learns its latent space from scratch during training on the T2I data.

After fine-tuning, UNIFIED-B (218M) is on par with DiT$^{\mathrm{DH}}$-XL (959M in total, including a pretrained FLUX-VAE) on GenEval and DPG-Bench, and better on GenAI-Bench. UNIFIED-L (554M) is better on all three.

Quality/compute trade-offs and NFE-to-quality curves. Figure 4 plots gFID against training FLOPs, NFE, and sampling FLOPs for UNITE, UNITE++, and both UNIFIED variants; the numbers are given in our response to W1 / Q1. For inference latency, generating one image at Base takes 155.5 ms for UNITE and 256 ms for both UNIFIED variants (1.64×; 50 NFE plus one decoding pass, on a single NVIDIA RTX PRO 6000). This ratio matches the ratio of FLOPs per forward pass (81.7 vs. 50.1 GFLOPs, 1.63×), so the comparisons at matched sampling FLOPs above also hold for wall-clock time.

Changes: Section 4.2 and Appendix D (Table 7); Section 4.5 and Appendix F (Table 4); Section 4.4 (Figure 4).

Reviewer_XubWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to unify the tokenizer and generator of the latent diffusions into a single model. The model is trained using three distinct passes: one for encoding, one for denoising, and one for decoding. The overall objective consists of a pixel-level ℓ2\ell_2 reconstruction loss, an LPIPS loss, and a flow matching loss.

Critical strengths
  1. The paper targets a practical problem regarding the inefficiency of running separate tokenizer, generator, and decoder networks in latent diffusion models.
  2. The empirical comparison is well-controlled.
Critical weaknesses
  1. My major concern is that, when merely targeting the image generation task, what’s the benefit of keeping the encode-generate-decode principles of the latent-space models compared with the pixel-space methods such as JiT?
  2. The writings could be further refined to improve clarity. Some expressions are ambiguous, such as:
    1. “The clean target of the generation loss is stop-gradient” in line 244-245.
    2. “our lightweight heads translate between values and the backbone width: P in x and P out x for pixel patches, and P in z and P out z for latent tokens. Only one of them is new: P in x , P in z , and P out z are the heads of UNITE’s generative encoder, used unchanged,” in line 261-263. What does “only one of them is new” refers to?
  3. The performance gains over UNITE quickly saturate when model size increases, e.g., 2.23 v.s. 2.16 with 215M model parameters.
Questions for authors

Apart from questions listed in the Weakness, could the authors clarify the key insights/takeaways of the work when comparing with the UNITE? Besides, what’s the challenge of merging the decoder and encoder into a single backbone?

4 comments

We thank the reviewer for the feedback and for highlighting the practical problem we target, the cost of separate encoder, generator, and decoder networks, and the well-controlled comparison. We answer each point below.

Note on the two UNIFIED variants. The submitted model encodes and decodes at the clean endpoint (tdec=1t_{\mathrm{dec}}=1). The submission found that decoding and generation compete there: ending the sampling ODE at t=0.99t=0.99 improved gFID from 2.16 to 2.05. The revision therefore adds a variant that conditions decoding on its own learned embedding (tdec=−1t_{\mathrm{dec}}=-1; Section 3.3). It adds one embedding vector and changes nothing else. We report both variants throughout, and the improvement over UNITE at every model size holds for both.

W1 — Benefit of the latent pipeline over pixel-space models

In UNIFIED, keeping the latent adds no pipeline complexity, and it gives better generation at the model sizes we test. Like a pixel-space model, UNIFIED is one network trained in one stage, with no pretrained autoencoder or vision foundation model. Unlike a pixel-space model, it denoises a compact latent that it learns itself, and maps the result to pixels in a single decoding pass.

Empirically, the learned latent leads to better generation. On ImageNet-1K 256×256, UNIFIED improves on pixel-space models that have about twice its parameters and three times its training epochs (Table 7 in the revision). UNIFIED-T (62M) reaches 2.78 against 2.96 for JiT-B/16 + RG (131M), and UNIFIED-B (215M) reaches 2.08 against 2.36 for JiT-L/16 (459M).

MethodDiffusion spacePretrained VFMParamsEpochsgFID ↓
JiT-B/16Pixel–131M6003.66
PixelREPA-B/16Pixel✓131M6003.17
JiT-B/16 + RGPixel–131M6002.96
PixelU-B/16Pixel✓164M6003.40
JiT-L/16Pixel–459M6002.36
UNIFIED-TLearned latent–62M2002.78
UNIFIED-SLearned latent–116M2002.43
UNIFIED-BLearned latent–215M2002.08

UNIFIED rows use the learned decoding timestep (tdec=−1t_{\mathrm{dec}}=-1). The submitted variant (tdec=1t_{\mathrm{dec}}=1) reaches 2.75, 2.41, and 2.15 (Table 2).

The same holds for text-to-image generation under an identical protocol (same training data, text encoder, training steps, and sampler; Table 4). After pretraining, UNIFIED-L (554M) improves on JiT (615M) on all three benchmarks:

MethodParamsGenEval ↑DPG-Bench ↑GenAI-Bench ↑
JiT615M0.5160.7820.674
UNIFIED-L554M0.5270.7940.685

The learned latent also makes the same network serve as a tokenizer (with rFID 0.95 at Base level), which a pixel-space model does not provide.

Changes: Section 4.2 and Appendix D (Table 7); Section 4.5 (Table 4).

W2 — Ambiguous wording

Thank you for pointing these out. We rewrote both sentences.

  1. Stop-gradient (Section 3.1). The sentence now reads: "We apply a stop-gradient to the encoder output zz in the generation loss, so the loss treats the latent as a constant. The same detached latent serves as the clean target and, once noised, as the generator’s input. Because no gradient from the generation loss reaches zz, this loss cannot push the latent toward a trivial solution. The reconstruction and generation losses still jointly update the shared weights." Section 3.4 also defines sg⁡(⋅)\operatorname{sg}(\cdot) as the operator that treats its argument as a constant during backpropagation.
  2. "Only one of them is new" (Section 3.2). "Them" referred to the four input and output heads. The sentence now reads: "Three of the four heads, PxinP_x^{\mathrm{in}}, PzinP_z^{\mathrm{in}}, and PzoutP_z^{\mathrm{out}}, are those of UNITE's generative encoder and are used unchanged. The fourth, PxoutP_x^{\mathrm{out}}, is the only head that UNIFIED adds: it lets the backbone itself return pixel patches and thereby replaces UNITE's standalone decoder."

Changes: Sections 3.1, 3.2, and 3.4.

W3 — Gains over UNITE at larger model sizes

We agree that the gain at Base is smaller than at Tiny and Small. The submitted model already improves on UNITE at Base by more than the evaluation variance, and the learned decoding timestep added in the revision widens this gain. gFID, mean over five evaluation seeds (Tables 2 and 11):

SizeUNITEUNIFIED, tdec=1t_{\mathrm{dec}}=1 (submitted)UNIFIED, tdec=−1t_{\mathrm{dec}}=-1 (revision)
Tiny3.00 (61M)2.75 (62M), gain 0.252.78 (62M), gain 0.22
Small2.76 (117M)2.41 (116M), gain 0.352.43 (116M), gain 0.33
Base2.23 (217M)2.15 (215M), gain 0.082.08 (215M), gain 0.15
  • Base gain of the submitted model. The 0.07 the reviewer quotes (2.23 vs. 2.16) comes from a single evaluation run. Over five evaluation seeds, the submitted model reaches 2.150 ± 0.011 against 2.226 ± 0.019 for UNITE with the adaptive dopri5 solver (gain 0.08), and 2.089 ± 0.013 against 2.277 ± 0.029 with a 256-step Euler solver (gain 0.19). Under both solvers, every run of the submitted model has a lower gFID than every run of UNITE (Table 11). The gain at Base is small, but it is not within the evaluation noise.
  • Learned decoding timestep. The submission found that decoding and generation compete at the clean endpoint: ending the sampling ODE at t=0.99t=0.99 improved gFID from 2.16 to 2.05. The revision gives decoding its own learned embedding (Section 3.3), which resolves this conflict during training. At Base, it raises the gain to 0.15 (2.075 ± 0.025) without selecting an endpoint. At Tiny and Small, it is on par with the submitted model, so it does not change the trend across sizes.
  • Why the gain shrinks. We attribute the trend to how capacity is spent. UNITE reserves part of its budget for a standalone decoder that receives only reconstruction gradients, whereas UNIFIED gives every parameter to the shared backbone, which also generates. When the total budget is small, generation is limited by capacity, so moving the decoder's parameters into the shared backbone helps most. Tiny UNIFIED (62M, 2.75) matches Small UNITE (117M, 2.76) with about half the parameters. At Tiny, even shifting UNITE's capacity from the decoder to the generative encoder improves gFID from 5.01 to 3.00 (Table 8). As the budget grows, generation is less limited by capacity, so the parameters freed from the decoder matter less.
  • Larger scale. UNIFIED itself keeps improving with size: in text-to-image generation, going from 218M to 554M raises GenEval from 0.471 to 0.527 after the same 100K pretraining steps (Table 4). Whether the gain over UNITE persists beyond 215M on ImageNet remains to be tested, and the revision states this.

Changes: Section 3.3; Tables 2, 8, and 11; Appendix D.

Q1 / Q2 — Takeaways relative to UNITE, and the challenge of merging the decoder

Key takeaways. We study whether a latent diffusion model can be trained with a single diffusion transformer in a single stage, without external supervision such as DINOv2 features or a GAN loss. UNITE merges the encoder and the generator but keeps a standalone decoder. Relative to UNITE, our takeaways are:

  1. The decoder need not be a separate network. One DiT can encode, generate, and decode with every backbone weight shared, given the right mode and timestep conditioning, and it improves generation over UNITE at ~60M, ~115M, and ~215M parameters.
  2. Decoder parameters are better spent in the shared backbone. At a fixed Tiny budget, moving UNITE's parameters from the decoder to the generative encoder improves gFID (5.01 → 3.79 → 3.00), and removing the standalone decoder improves it further (2.75 and 2.78 for the two variants; Table 8).
  3. How the three modes are specified matters as much as sharing the weights. We illustrate the details below.

Challenge of merging the decoder. A direct extension of UNITE does not work. We built a baseline UNITE++: it folds the decoder into the backbone and keeps UNITE's readout tokens for both encoding and decoding. It generates on par with UNITE but reconstructs much worse (rFID 2.20 vs. 1.41). We argue that decoding is the bottleneck for reconstruction, since each image location must be recovered by attending from readout tokens to latents that have no fixed spatial correspondence to it. Table 3 shows how we resolve this, one change at a time:

StepChangegFID ↓rFID ↓
UNITE–2.231.41
UNITE++decoder folded into the backbone, readout tokens kept2.242.20
UNIFIED (tenc=0t_{\mathrm{enc}}=0, tdec=1t_{\mathrm{dec}}=1)readout tokens replaced by position-aligned direct mapping2.311.39
UNIFIED (tenc=1t_{\mathrm{enc}}=1, tdec=1t_{\mathrm{dec}}=1)encoding at the clean endpoint2.150.97
UNIFIED (tenc=1t_{\mathrm{enc}}=1, tdec=−1t_{\mathrm{dec}}=-1)learned decoding timestep2.080.95
  • Position alignment removes the routing problem and restores reconstruction.
  • The encoding timestep must describe the encoder's input. UNITE encodes from Gaussian readouts, so tenc=0t_{\mathrm{enc}}=0 is correct for it. Once the encoder reads clean image patches, tenc=0t_{\mathrm{enc}}=0 labels them as pure noise, and switching to tenc=1t_{\mathrm{enc}}=1 improves both generation and reconstruction.
  • Decoding and generation conflict at the clean endpoint, where one outputs pixels and the other outputs the clean latent. A learned decoding timestep removes the conflict: with tdec=1t_{\mathrm{dec}}=1, the last-block CKA between the encoding and generation passes drops to 0.37 at t=0.995t=0.995, whereas with tdec=−1t_{\mathrm{dec}}=-1 it stays at 0.92–0.93, on par with UNITE (0.92; Figure 3).

Changes: Section 3.3; Section 4.3 (Table 3, Figure 3, and the paragraph "Learned decoding timestep"); Appendix E (Table 8).

I thank the authors for their detailed responses. However, my main concerns remain unaddressed.

Compared with pixel-space methods (e.g., JiT), the proposed UNIFIED model uses fewer parameters but requires three separate forward passes per iteration. Consequently, the training FLOPs of a 215M model could exceed those of a 400M JiT. Thus, the practical advantages of the encode-generate-decode pipeline over straightforward end-to-end pixel-space models remain unclear.

Moreover, the main contribution is not sufficiently substantiated. While the motivation is to merge the decoder into the generator, the benefits over prior work appear limited to smaller models and diminish as model capacity increases.

I therefore decide to maintain my current assessment.

Reviewer_93HnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UNIFIED uses one transformer to encode images, denoise latents, and decode images. It absorbs UNITE's separate decoder into the shared backbone. Lightweight heads and mode conditioning distinguish the tasks. The paper reports better ImageNet-256 FID at three matched parameter budgets and studies how the shared model handles these different tasks.

Critical strengths
  • The parameter-allocation experiments are useful. Tables 2-3 show gains across three scales and test where decoder parameters are better spent. The Tiny and Small improvements are reasonably clear: 3.04 to 2.79 and 2.77 to 2.44 FID.

  • The paper includes meaningful design controls. UNITE++ separates weight sharing from the token-layout change. The timestep and mode-token ablations clarify the design. Figure 4 also shows that UNIFIED at 64 evaluations can outperform UNITE at 128.

Critical weaknesses
  • My main concern is the compute trade-off. Matching parameter counts does not match training cost. Appendix C reports 3.58 versus 2.39 TFLOPs per image at Base, or 1.50x more compute. The lower-evaluation inference result is useful, but it does not establish a training benefit. I wonder whether UNITE would close the gap with the same total training budget.

  • The smaller gains need uncertainty estimates. Base improves by 0.07 FID, and the encoding-timestep change improves FID by 0.04. Tables 2 and 4 do not report variation across ImageNet training seeds or evaluation samples. This is less concerning for the larger Tiny and Small gains.

  • The sampling description is inconsistent. Section 3.4 starts at t=0, and Table 5 uses [0,1]. Appendix B instead specifies [0.1,1.0]. This may be a reporting error, but the actual interval matters for the endpoint analysis. Figure 4 also needs an explicit solver and time grid for its fixed-budget results.

Questions for authors
  • How do UNIFIED and UNITE compare under the same total training FLOPs?

  • Are the Base and timestep gains stable across repeated training and evaluation runs?

  • Which initial condition and interval produced each table? What solver and time grid were used for Figure 4?

3 comments

We thank the reviewer for the feedback and for highlighting the parameter-allocation experiments, the controls that separate weight sharing from the layout and the timestep, and the fixed-budget comparison. We answer each point below.

Note on the two UNIFIED variants. The submitted model encodes and decodes at the clean endpoint (tdec=1t_{\mathrm{dec}}=1). The submission found that decoding and generation compete there: ending the sampling ODE at t=0.99t=0.99 improved gFID from 2.16 to 2.05. The revision therefore adds a variant that conditions decoding on its own learned embedding (tdec=−1t_{\mathrm{dec}}=-1; Section 3.3). It adds one embedding vector and changes nothing else. We report both variants throughout, and the improvement over UNITE at every model size holds for both.

W1 / Q1 — Compute trade-off and comparison at matched training FLOPs

Training FLOPs. Given UNITE's full training budget, UNIFIED matches its final gFID with 10–19% less compute and improves on it with 3–20% more, at every size. UNITE leads early in training but plateaus near the end of its budget, while UNIFIED keeps improving.

We evaluated the intermediate checkpoints of every model and plotted gFID against total training FLOPs (Figure 4 for Base, Figure 6 for Tiny and Small). The table below compares UNITE's final model with the two UNIFIED checkpoints whose training compute brackets it (Table 6 in the revision; EFLOPs =1018=10^{18} FLOPs; every gFID is the mean over multiple evaluation seeds).

SizeMethodEpochsTraining EFLOPsgFID ↓
TinyUNITE (2:1)2002183.00
UNIFIED (tdec=−1t_{\mathrm{dec}}=-1)1201763.03
UNIFIED (tdec=−1t_{\mathrm{dec}}=-1)1602352.91
SmallUNITE2002962.76
UNIFIED (tdec=−1t_{\mathrm{dec}}=-1)1002542.78
UNIFIED (tdec=−1t_{\mathrm{dec}}=-1)1203052.65
BaseUNITE2006082.23
UNIFIED (tdec=−1t_{\mathrm{dec}}=-1)1205462.22
UNIFIED (tdec=−1t_{\mathrm{dec}}=-1)1607282.15
  • With 10–19% less training compute, UNIFIED is within 0.03 gFID of UNITE's final model at every size (3.03 vs. 3.00 at Tiny, 2.78 vs. 2.76 at Small, and 2.22 vs. 2.23 at Base). With 3–20% more, it improves on UNITE at every size (2.91, 2.65, and 2.15); at Small, 3% more compute gives a gain of 0.11.
  • The gain is smaller than at matched epochs: at Base, the 0.15 of Table 2 requires the full 200 epochs.
  • On whether UNITE would close the gap: although UNITE leads early in training (e.g., 2.26 vs. 2.35 at Base and 365 EFLOPs), its curve flattens over its last 240 EFLOPs (2.26 → 2.23 → 2.23), while UNIFIED keeps improving up to and beyond UNITE's total budget (2.22 → 2.15 → 2.08 from 546 to 910 EFLOPs).

Sampling FLOPs. At matched quality, both UNIFIED variants need fewer sampling FLOPs than UNITE (Figure 4). They reach gFID 2.46–2.50 at 2.6 TFLOPs per image, which UNITE reaches at 3.2 TFLOPs. At 64 NFE (5.2 TFLOPs), they reach 2.20 (tdec=1t_{\mathrm{dec}}=1) and 2.23 (tdec=−1t_{\mathrm{dec}}=-1), better than UNITE at 256 NFE (12.8 TFLOPs, 2.28) with 2.5× fewer FLOPs. UNITE is cheaper only for coarser targets (e.g., gFID 2.86 at 1.6 TFLOPs). At 256 NFE, averaged over five evaluation seeds, they reach 2.09 and 2.12, against 2.28 for UNITE (Table 11).

Changes: Section 4.4, Figure 4, Figure 6, and Appendix C (new paragraph "At matched training compute" and Table 6).

W2 / Q2 — Uncertainty of the Base and timestep gains

Across evaluation runs, yes: under both solvers, every run of both UNIFIED variants has a lower gFID than every run of UNITE. Given computational budgets, we are not able to repeat training with different seeds.

We repeated the evaluation of the Base models with five seeds, each drawing a different set of 50K initial noises. Every run uses the epoch-200 checkpoint, the model's best CFG scale, and both solvers of the paper: the adaptive dopri5 solver and a 256-step Euler solver. The table reports the mean ± standard deviation (Table 11 in the revision).

Methodtenct_{\mathrm{enc}}tdect_{\mathrm{dec}}dopri5: gFID ↓dopri5: IS ↑Euler-256: gFID ↓Euler-256: IS ↑
UNITE0–2.226 ± 0.019282.1 ± 1.42.277 ± 0.029280.4 ± 0.6
UNITE++012.239 ± 0.004327.1 ± 0.72.279 ± 0.013329.2 ± 1.4
UNIFIED112.150 ± 0.011312.8 ± 1.92.089 ± 0.013315.0 ± 1.1
UNIFIED1−12.075 ± 0.025314.4 ± 2.32.121 ± 0.010314.9 ± 1.9
  • Base gain. The gFID ranges over the five runs do not overlap (UNIFIED vs. UNITE; dopri5: 2.047–2.162 vs. 2.206–2.255; Euler: 2.070–2.133 vs. 2.253–2.318). With dopri5, UNIFIED (tdec=1t_{\mathrm{dec}}=1, the model of the submission) improves on UNITE by 0.08 on average, and UNIFIED (tdec=−1t_{\mathrm{dec}}=-1) improves on UNITE by 0.15. With Euler, the two gains are 0.19 and 0.16 respectively.
  • Numbers in the submission. The values the reviewer quotes (2.23 vs. 2.16) are seed 0 of the dopri5 columns. The revision now reports the mean over evaluation seeds wherever an epoch-200 gFID or IS of UNITE, UNITE++, or UNIFIED appears (Figures 1, 4, and 6; Tables 2, 3, and 6–10), so the Base comparison reads 2.23 (UNITE) vs. 2.15 and 2.08 (UNIFIED). Results without repeated evaluations are marked as single runs.
  • Tiny and Small. Table 11 also lists these sizes. At Tiny, five seeds give 2.998 ± 0.030 for UNITE, 2.754 ± 0.026 for UNIFIED (tdec=1t_{\mathrm{dec}}=1), and 2.783 ± 0.025 for UNIFIED (tdec=−1t_{\mathrm{dec}}=-1). At Small, five seeds give 2.761 ± 0.020 for UNITE, 2.414 ± 0.018 for UNIFIED (tdec=1t_{\mathrm{dec}}=1), and 2.426 ± 0.012 for UNIFIED (tdec=−1t_{\mathrm{dec}}=-1).
  • Encoding timestep. The 0.04 in the submission (2.20 vs. 2.16) compared two single runs. It also evaluated the model with tenc=0t_{\mathrm{enc}}=0 at epoch 160, its lowest-gFID checkpoint. The revision evaluates every model at epoch 200, where tenc=0t_{\mathrm{enc}}=0 gives 2.31 and tenc=1t_{\mathrm{enc}}=1 gives 2.15 (Tables 3 and 9). Even at its best checkpoint, tenc=0t_{\mathrm{enc}}=0 (2.20) trails tenc=1t_{\mathrm{enc}}=1. Reconstruction, which involves no sampling, also improves with tenc=1t_{\mathrm{enc}}=1 (rFID 1.39 → 0.97, PSNR 25.74 → 26.63; Table 9).
  • Training seeds. Given the computational constraints, we could not repeat ImageNet training with different seeds. The table measures evaluation variance only, and we state this in the revision.

Changes: Appendix E (new paragraph "Variation across evaluation seeds" and Table 11); seed means in Figures 1, 4, and 6 and Tables 2, 3, and 6–10; caption of Table 9 (checkpoint of the model with tenc=0t_{\mathrm{enc}}=0).

W3 / Q3 — Sampling description, and the solver and time grid of Figure 4

This was a reporting error in Appendix B, and we apologize for the confusion. The interval [0.1,1.0][0.1, 1.0] is the range of tt over which classifier-free guidance is applied, following UNITE; it is not the integration interval. The protocols are as follows.

  • All ImageNet tables (Tables 2, 3, and 6–10, the dopri5 columns of Table 11, and the training-budget panels of Figures 4 and 6): the sample starts from Gaussian noise at t=0t=0 and is integrated over the full interval [0,1][0,1] with the adaptive dopri5 solver (absolute tolerance 10−610^{-6}, relative tolerance 10−310^{-3}). The solver chooses its own steps, so there is no time grid. Guidance is applied only for t∈[0.1,1]t \in [0.1, 1], and each model uses its best CFG scale from a sweep over 1.0–4.0 in steps of 0.2.
  • Fixed-budget results (Figure 4 of the submission; now the sampling-budget panels of Figure 4 and the Euler columns of Table 11): a fixed-step Euler solver over [0,1][0,1]. With a budget of KK NFE, it takes KK steps on the grid ti=αsi/(1+(α−1)si)t_i = \alpha s_i / (1 + (\alpha - 1) s_i) with si=i/Ks_i = i/K and α=0.5\alpha = 0.5, the noise-schedule shift used in training, so the grid is denser near t=0t=0. Each model uses its best CFG scale, with the same guidance interval and 50K class-balanced samples.
  • Endpoint analysis (Table 5 of the submission; now Appendix E and Table 12): the same initial condition and solver, but the integration ends at tend<1t_{\mathrm{end}} < 1, at a fixed CFG scale of 2.0. This is the only experiment with a different interval. It now sweeps tend∈{0.95,0.98,0.99,0.995}t_{\mathrm{end}} \in \{0.95, 0.98, 0.99, 0.995\} rather than 0.99 alone.

Changes: Section 3.4 ("Inference"), Appendix B ("ODE solver", "Classifier-free guidance", and "Protocol of each result"), and the caption of Figure 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.