- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether encoding, latent generation, and image decoding can share all Transformer blocks. Starting from UNITE, it introduces UNIFIED, with lightweight task-specific heads, mode tokens, spatially aligned token layouts, and a clean-endpoint timestep convention for encoding and decoding. ImageNet-256 experiments compare parameter-matched models at three scales, examine capacity allocation and sampling behavior, and analyze cross-mode feature similarity.
Critical strengths
- The parameter-matched comparisons consistently favor complete sharing across model sizes. Table 2 reports generation FID improvements from 3.04 to 2.79 at Tiny, 2.77 to 2.44 at Small, and 2.23 to 2.16 at Base; Base reconstruction FID improves from 1.41 to 0.97. Only 1.63% of Base parameters are task-specific heads and tokens. The capacity-allocation comparisons in Table 3 make the architectural result more meaningful than comparison with a single arbitrary split.
- The paper examines important practical qualifications rather than reporting parameter savings alone. UNITE++ helps separate shared weights from the proposed interface, and the timestep/readout ablations probe design choices. Appendix C discloses the larger training and per-flow-step computation. The fixed-NFE comparison in Figure 4 and endpoint truncation in Table 5 demonstrate a potentially useful quality/sampling tradeoff. The unsuccessful QM9 result further shows that the authors do not assume the image-domain finding transfers automatically.
Critical weaknesses
- Equal parameter count is not equal compute, which limits the efficiency conclusion. Appendix C reports 3.58 versus 2.39 TFLOPs per image for a full training step, and 81.7 versus 50.1 GFLOPs per flow-matching forward pass, for UNIFIED versus UNITE. The fixed-NFE results are helpful, but a comparison in which UNITE receives the additional training compute is absent. Sampling choices also matter: Table 2 gives Base generation FID 2.23 versus 2.16, while Table 5 gives 2.22 versus 2.05 when both models stop at 0.99. UNIFIED remains better in that comparison, but a general quality/compute advantage needs consistently matched training and sampling budgets.
- The explanation for the improvement is not fully disentangled. Complete sharing, spatial alignment of the token interface, and the encoding/decoding timestep convention change together. Table 4 partially separates these factors, but does not provide a complete interface-by-timestep comparison with the same shared architecture. Higher CKA can also result from aligned inputs and positions, so it is not independent evidence that the tasks learn the same computation or that this similarity causes better generation. This matters because the distinction from earlier joint generative encoders and diffusion-based unified models rests substantially on the proposed compatibility of the three modes.
Questions for authors
- Fairness test: What happens if UNITE receives the same total training FLOPs as UNIFIED, with the same guidance and ODE-endpoint selection budget? Please also compare inference FLOPs at matched generation quality, since Appendix C reports a higher cost per denoising call for UNIFIED. This would establish whether the architectural improvement gives a favorable overall quality/compute tradeoff.
- Mechanism test: Can the readout-layout and clean-endpoint choices be compared factorially within the same shared backbone, including UNITE++ with the clean-endpoint convention? An aligned-interface model with a separate decoder would further clarify whether the improvement is caused by full sharing or primarily by the interface design.
- How does UNIFIED compare with SOTA methods at matched budgets? Can you compare with strong baselines such as DiT, SiT, REPA, REPA-E, and RAE at matched parameter count, training FLOPs, and sampling NFE? For text-to-image, please include small-scale PixArt/SD-style baselines or at least same-tokenizer and same-data controls. Report quality/compute tradeoffs, inference latency, and NFE-to-quality curves.
- Can you evaluate UNIFIED on a text-to-image dataset such as CC12M, LAION, or COCO, with a text encoder and appropriate conditioning? How does it compare with UNITE, UNITE++, and the separate pipeline under identical text conditioning, data, training compute, and sampling budget? Does full sharing still help at higher resolution and longer token sequences?
7 comments
W2 / Q2 — Disentangling layout, timestep, and weight sharing
Within the fully shared backbone, the generation gain requires both position alignment and the matched encoding timestep: neither full sharing alone (UNITE++) nor the layout alone improves generation. The aligned-interface model with a separate decoder, which isolates the effect of full sharing, is still training.
Table 3 of the revision separates the factors one at a time rather than in a full factorial design: each row differs from the previous one in a single factor, and UNIFIED (, ) is the bridge between the layout comparison and the timestep comparison. A full factorial design is unfeasible, because one of its cells, readout tokens with , is not well defined (see the fourth point below).
| Model | Backbone | Layout | gFID ↓ | rFID ↓ | ||
|---|---|---|---|---|---|---|
| UNITE | encoder + generator shared, separate decoder | readout tokens | 0 | – | 2.23 | 1.41 |
| UNITE++ | fully shared | readout tokens | 0 | 1 | 2.24 | 2.20 |
| UNIFIED | fully shared | position-aligned | 0 | 1 | 2.31 | 1.39 |
| UNIFIED | fully shared | position-aligned | 1 | 1 | 2.15 | 0.97 |
| UNIFIED | fully shared | position-aligned | 1 | −1 | 2.08 | 0.95 |
- UNITE++ is the direct extension of UNITE. It folds the decoder into the backbone and changes nothing else: it keeps the readout tokens and adds a decoding mode at the clean endpoint. UNITE's encoder reads readouts, so describes its input correctly, and UNITE++ inherits it.
- Layout (rows 2 and 3). UNITE++ and UNIFIED (, ) have the same fully shared backbone and the same timesteps, and differ only in the layout. Position alignment restores reconstruction (rFID 2.20 → 1.39) but does not by itself improve generation.
- Encoding timestep (rows 3 and 4). The two UNIFIED models have the same backbone and layout, and differ only in . Both are evaluated at epoch 200; the submission reported the model with at its lowest-gFID checkpoint (epoch 160, gFID 2.20). The clean endpoint improves both generation (2.31 → 2.15) and reconstruction (1.39 → 0.97).
- UNITE++ with the clean-endpoint convention. This setting is not well defined because the encoder of UNITE++ reads pure-noise readouts adapted from UNITE, and setting would label those latent readouts as clean.
- Full sharing vs. interface. As the reviewer suggests, we are training UNITE with the aligned interface: its generative encoder maps image patches to latents directly without readout tokens, while keeping the standalone decoder. Comparing it with UNIFIED isolates the effect of sharing the decoder. We will report early results closer to the end of the rebuttal.
- CKA. We agree that CKA is not independent evidence. The revision no longer says that CKA "confirms" the mechanism, and now reads: "We use CKA only to locate where the two passes diverge, while the effect of each design choice on generation is measured by the ablations in Table 3." The divergence it locates (the last block near the clean endpoint) motivated the learned decoding timestep, which is then tested by its effect on gFID (row 5).
Changes: Section 3.3 ("Decoding timestep"); Section 4.3 (Table 3, and the paragraphs "Readout tokens", "Position alignment", "Clean endpoint for encoding", "CKA analysis", and "Learned decoding timestep").
Q3 — Comparison with SOTA methods at matched budgets
ImageNet. Table 7 of the revision compares UNIFIED with two-stage and single-stage models, including DiT, SiT, REPA, and RAE, and lists the parameter count of all networks, the training epochs, and the pretrained components each method uses. Due to computational constraints, we use the baseline results reported in their papers. Our models train for 200 epochs, against 400–1600 for most of the two-stage models.
| Method | Pretrained AE | Pretrained VFM | Params | Epochs | gFID ↓ |
|---|---|---|---|---|---|
| DiT-XL/2 | ✓ | – | 759M | 1400 | 2.27 |
| SiT-XL/2 | ✓ | – | 759M | 1400 | 2.06 |
| REPA-B | ✓ | ✓ | 214M | – | 2.15 |
| RAE-B | ✓ | ✓ | 631M | – | 2.08 |
| UNIFIED-B | – | – | 215M | 200 | 2.08 |
UNIFIED-B uses the learned decoding timestep (). The submitted variant () reaches 2.15 (Table 2).
UNIFIED-B improves on REPA-B and matches RAE-B without a pretrained autoencoder or vision foundation model. Every model in Table 7 with a lower gFID than UNIFIED-B uses at least 545M parameters.
Text-to-image. Table 4 follows the DiffusionBench protocol, so every model has the same training data (JourneyDB + BLIP3-o), text encoder (Qwen3-0.6B), number of training steps, and sampler (50-step Euler, CFG scale 6.0). The baselines cover a latent diffusion model with a pretrained SD-VAE (NanoGen), one with a pretrained FLUX-VAE (DiT$^{\mathrm{DH}}$-XL), and a pixel-space model (JiT):
| Method | Params | GenEval ↑ | DPG-Bench ↑ | GenAI-Bench ↑ |
|---|---|---|---|---|
| Pretraining (100K steps) | ||||
| DiT$^{\mathrm{DH}}$-XL (FLUX-VAE) | 875M + 84M | 0.497 | 0.789 | 0.637 |
| JiT | 615M | 0.516 | 0.782 | 0.674 |
| NanoGen (SD-VAE) | 611M + 84M | 0.578 | 0.804 | 0.691 |
| UNIFIED-B | 218M | 0.471 | 0.792 | 0.672 |
| UNIFIED-L | 554M | 0.527 | 0.794 | 0.685 |
| + Fine-tuning (2K steps) | ||||
| DiT$^{\mathrm{DH}}$-XL (FLUX-VAE) | 875M + 84M | 0.841 | 0.835 | 0.688 |
| UNIFIED-B | 218M | 0.849 | 0.833 | 0.729 |
| UNIFIED-L | 554M | 0.878 | 0.848 | 0.748 |
After pretraining, UNIFIED-L (554M) improves on pixel-space JiT (615M) on all three benchmarks, and UNIFIED-B (218M) is on par with JiT on DPG-Bench and GenAI-Bench. Unlike DiT$^{\mathrm{DH}}$-XL and NanoGen, UNIFIED does not rely on a pretrained VAE, which may have been trained on large-scale external data. Instead, it learns its latent space from scratch during training on the T2I data.
After fine-tuning, UNIFIED-B (218M) is on par with DiT$^{\mathrm{DH}}$-XL (959M in total, including a pretrained FLUX-VAE) on GenEval and DPG-Bench, and better on GenAI-Bench. UNIFIED-L (554M) is better on all three.
Quality/compute trade-offs and NFE-to-quality curves. Figure 4 plots gFID against training FLOPs, NFE, and sampling FLOPs for UNITE, UNITE++, and both UNIFIED variants; the numbers are given in our response to W1 / Q1. For inference latency, generating one image at Base takes 155.5 ms for UNITE and 256 ms for both UNIFIED variants (1.64×; 50 NFE plus one decoding pass, on a single NVIDIA RTX PRO 6000). This ratio matches the ratio of FLOPs per forward pass (81.7 vs. 50.1 GFLOPs, 1.63×), so the comparisons at matched sampling FLOPs above also hold for wall-clock time.
Changes: Section 4.2 and Appendix D (Table 7); Section 4.5 and Appendix F (Table 4); Section 4.4 (Figure 4).
We thank the reviewer for the feedback and for highlighting the consistent parameter-matched gains, the controls that separate weight sharing from the interface design, and our reporting of the method's practical qualifications. We answer each point below.
Note on the two UNIFIED variants. The submitted model encodes and decodes at the clean endpoint (tdec=1). The submission found that decoding and generation compete there: ending the sampling ODE at t=0.99 improved gFID from 2.16 to 2.05. The revision therefore adds a variant that conditions decoding on its own learned embedding (tdec=−1; Section 3.3). It adds one embedding vector and changes nothing else. We report both variants throughout, and the improvement over UNITE at every model size holds for both.
W1 / Q1 — Matched training and sampling budgets
With matched budgets, UNIFIED reaches a better final quality than UNITE; UNITE's advantage is limited to the early, lower-quality regime.
Training budget and training FLOPs at matched quality. We evaluate the intermediate checkpoints of every model and plot gFID against total training FLOPs (Figure 4 for Base, Figure 6 for Tiny and Small). The table below compares UNITE's final model with the two UNIFIED checkpoints whose training compute brackets it (Table 6; EFLOPs =1018 FLOPs; every gFID is the mean over multiple evaluation seeds).
With 10–19% less training compute, UNIFIED is within 0.03 gFID of UNITE's final model at every size, and with 3–20% more it improves on UNITE at every size. The gain is smaller than at matched epochs (0.15 at Base). Although UNITE leads early in training, its curve flattens over its last 240 EFLOPs (2.26 → 2.23 → 2.23), while UNIFIED keeps improving up to and beyond UNITE's total budget (2.22 → 2.15 → 2.08).
Sampling budget and inference FLOPs at matched quality. With a fixed-step Euler solver, sampling FLOPs per image are NFE × one forward pass (50.1 GFLOPs for UNITE, 81.7 for UNIFIED at Base level). Figure 4 plots gFID against both NFE and sampling FLOPs:
The two UNIFIED variants have the same architecture and therefore the same sampling FLOPs. Both UNIFIED variants are better than UNITE at every NFE, and the submitted model (tdec=1) is slightly better than UNIFIED (tdec=−1) at every NFE under this solver. At matched quality, UNIFIED needs fewer sampling FLOPs for any target gFID of 2.50 or better: it reaches 2.46–2.50 at 2.6 TFLOPs, which UNITE reaches at 3.2 TFLOPs, and 2.20–2.23 at 5.2 TFLOPs, a quality UNITE does not reach even at 12.8 TFLOPs. UNITE is cheaper only for coarser targets.
Guidance budget and endpoint selection. Every model receives the same guidance search: the CFG scale is swept from 1.0 to 4.0 in steps of 0.2, guidance is applied for t∈[0.1,1], and each model is reported at its best scale. The fixed-NFE results above reuse each model's best scale from this sweep. Every result above integrates the ODE over the full interval [0,1], so no model receives endpoint selection. Appendix E (Table 12) analyzes early stopping separately and gives every model the same choices of tend.
Changes: Section 4.4, Figures 4 and 6, Appendix B ("Classifier-free guidance"), Appendix C (Table 6), and Appendix E (Table 11).