acceptodds
Under review as a conference paper at ICLR 2027

Unifying Latent Denoiser with Decoder Makes Robust Image Generation

Abstract

Despite the substantial progress of latent diffusion models, the gap between generated latents and those encoded from real data remains, posing a fundamental tension between reconstruction fidelity and generative modeling. In this study, we explore a simple-yet-effective approach that jointly learns denoising and image reconstruction from encoded image latents, termed the Denoise-decode Joint Transformer (DJT). During training, the denoising and reconstruction objectives shape the shared backbone together, while each task-specific head specializes in its corresponding task. At inference time, the model acts like a recursive transformer: it first operates as a denoiser to iteratively produce a final latent sample and then as a decoder to map the latent into an image. Experiments on ImageNet show that DJT-XL achieves gFID=**0.97**, with **30%** less parameters than the counterpart pipeline. Representation analysis further shows that the joint training strategy strengthens the alignment between denoising and decoding representations and improves their functional compatibility. The code and models will be publicly available.

Primary area:
generative models
Keywords:
Latent Diffusion Models, Image Generation, Image Reconstruction, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TeYiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Denoise-decode Joint Transformer (DJT), which shares a transformer backbone between latent denoising and image reconstruction while retaining shallow task-specific heads. A mode condition distinguishes the two tasks, and a Bernoulli router assigns training examples to denoising or reconstruction. At inference, the denoising branch first generates a latent through iterative sampling, after which the network switches to reconstruction mode to produce an image. The main experiments use frozen DINOv2 representations on ImageNet-256, with additional results in VA-VAE latent space. The authors report improvements in generation quality and a reduction in unique denoiser–decoder parameters, and investigate representation similarity, activation interchangeability, and robustness to latent perturbations.

Critical strengths
  1. The reported generation results make parameter sharing practically compelling. DJT-XL achieves unguided gFID 1.35 and guided gFID 0.97, compared with 1.51 and 1.13 for the RAE entry in Table 3. The main DJT denoiser–decoder network contains approximately 870M unique parameters, compared with 1,254M for the listed RAE denoiser and decoder. The unguided improvement is particularly useful evidence that the result is not solely attributable to guided sampling.

  2. The paper includes informative controls beyond comparing complete generation pipelines. Table 2 evaluates alternative decoders on latents generated by the same frozen RAE denoiser, and Figure 7 compares joint training with a reconstruction-only model using the DJT decoding architecture on the same DJT-generated latents. These experiments support a decoder-side benefit from joint training. The branch-point and task-allocation ablations also provide useful evidence about how extensively the two tasks can share computation.

Critical weaknesses
  1. The architectural novelty relative to ELF is limited and needs more precise positioning. ELF (Embedded Language Flows, Hu et al., 2026; arXiv:2605.10938) already uses a shared denoising/decoding network, a binary mode condition, stochastic allocation between training objectives, and a final switch to decoding after continuous-space generation. DJT cites this work, but the overlap extends beyond the Bernoulli routing procedure to the overall training and inference template. Image reconstruction introduces different output requirements and losses, and the proposed task-specific heads are a concrete architectural difference; nevertheless, the contribution should be evaluated as an image-domain adaptation and investigation of this shared-network paradigm. The current experiments establish that the adaptation works, but less clearly establish which image-specific design choices are essential beyond the inherited template.

  2. The proposed distribution-shift explanation is not sufficiently supported by the measurement protocol and mechanism-specific controls. Table 1 compares Latent-FD values across VA-VAE and DINOv2 spaces with different dimensionalities. Fréchet distance depends on representation scale: multiplying both distributions by a scalar (c) multiplies the distance by (c^2). Without an adequately specified normalization and estimation protocol, the numerical difference cannot establish that one latent space has a larger mismatch in a representation-independent sense. The C2ST results offer complementary evidence, but their interpretation requires the classifier architecture, sample sizes, and held-out evaluation protocol.

  3. The robustness experiments also warrant a narrower interpretation. Gaussian perturbations need not reproduce errors in generated latents, as the authors acknowledge, and Appendix A specifies Gaussian augmentation during reconstruction training. The fixed-latent controls in Table 2 and Figure 7 are valuable, but do not by themselves establish that improved perturbation robustness mediates the generation gains. In particular, the reconstruction-only control should be evaluated on the same robustness curves, with its augmentation settings and effective reconstruction-training budget specified. Distributional rFID should also be complemented by a measure of preservation of the corresponding image content under perturbation.

Questions for authors
  1. Novelty and design attribution: Beyond changing the output domain and reconstruction losses, which image-specific architectural choices are essential relative to ELF? What are the individual contributions of the explicit mode embedding and task-specific branches? The existing branch-point ablation is useful, but does not fully answer these questions.

  2. Distribution-gap measurement: How exactly is Latent-FD computed, including spatial aggregation, dimensionality handling, normalization, sample sizes, and covariance estimation? What classifier and train/test protocol are used for C2ST? Does the relative conclusion remain stable under reasonable changes to the representation scaling and evaluation protocol?

  3. Robustness attribution: For the reconstruction-only control in Figure 7, what latent augmentation and reconstruction-update budget are used? Does this same-architecture control exhibit a corresponding deficit in perturbation robustness, and is the difference accompanied by worse preservation of image content?

  4. Inference-cost accounting: Table 3 describes parameter counts for the complete inference pipeline, while Appendix A specifies an additional DJT-S checkpoint for autoguidance. Please distinguish the main denoiser–decoder parameter count from the complete guided-system count, apply the same convention to the baselines, and report total inference compute or latency under matched sampling settings.

6 comments
AuthorsOpenReview ↗

Response to Reviewer TeYi 1/3

Thank you for your thoughtful and constructive feedback. We sincerely appreciate the time and care you devoted to reviewing our work. Below, we provide point-to-point responses to the review.

Q1 (combined with W1): Novelty and design attribution: Beyond changing the output domain and reconstruction losses, which image-specific architectural choices are essential relative to ELF? What are the individual contributions of the explicit mode embedding and task-specific branches? The existing branch-point ablation is useful, but does not fully answer these questions.

A1: Thank you for raising the question of design attribution.

First, DJT is designed to tackle the long-standing train-test mismatch in the generative pipeline, rather than to adapt the ELF design to the image domain. ELF adopts a DiT-based architecture augmented with an additional linear reconstruction head and multiple inserted context tokens for performance optimization. In contrast, our method only adopts its Bernoulli router, a common mechanism for routing the samples of a minibatch during training. Beyond this routing component, the architectural design of our method differs substantially from that of ELF.

Specifically, our model uses a wide-head transformer following DDT and RAE, which has been verified to work well for high-dimensional latents. We also optimize the task-specific branches in terms of where to branch and how deep the branches should be, as well as the training budget for each branch. To validate these design choices in practice, we conduct the experiments at the XL scale, demonstrating the effectiveness of our architecture in large-scale settings.

We have also provided the implementation details and source code, which we believe help address the concern about architectural similarity. As for the individual contributions of the proposed components, we conduct the following ablation experiments.

To examine the role of our branch design, we replace our task-specific branches with ELF-style simple heads. As shown in the table below, the performance degrades clearly, demonstrating that the carefully designed task-specific branches are important to the effectiveness of our method.

model50k gFID50k rFID
DJT6.130.99
DJT w/o depth in branch11.141.75

To verify the role of the explicit mode embedding, we first replace the mode embedding with zero during inference and evaluate its performance. Then we conduct standalone experiments where the mode embedding is explicitly inserted as a context token. As shown in the table below, strengthening the condition improves both gFID and rFID, whereas weakening it degrades both substantially. This demonstrates that the mode token provides useful information for the shared backbone.

model50k gFID50k rFID
DJT6.130.99
DJT w/ zero mode embedding80.682.62
DJT w/ context mode embedding6.050.98

These results further demonstrate that our architecture is well suited to high-dimensional image latents and provides substantial improvements over the corresponding ELF-style design choices.

AuthorsOpenReview ↗

Response to Reviewer TeYi 2/3

Q2 (combined with W2): Distribution-gap measurement: How exactly is Latent-FD computed, including spatial aggregation, dimensionality handling, normalization, sample sizes, and covariance estimation? What classifier and train/test protocol are used for C2ST? Does the relative conclusion remain stable under reasonable changes to the representation scaling and evaluation protocol?

A2: Thank you for the careful review. We provide the detailed measurement protocol below:

Latent-FD. Each 16×16 grid is spatially averaged into a single D-dimensional vector (D=32 for VA-VAE; D=768 for DINOv2), without whitening or dimensionality reduction. DINOv2 uses the released mean/variance normalization; VA-VAE uses posterior samples normalized by its released mean/std. We compare 50,000 unguided generated samples with all 1,281,167 encoded ImageNet training images. For the real and generated pooled features separately, we estimate full D×DD\times D sample covariance matrices using float64 moment accumulation, an N−1N-1 denominator, and no shrinkage.

To account for dimensionality, we additionally report FDD\frac{FD}{D} below. The ordering persists after dimension normalization, but FDD\frac{FD}{D} remains scale-dependent. We therefore complement Latent-FD with C2ST[1], which provides a measure of empirical distinguishability by evaluating how accurately a classifier separates generated from real latents on held-out data.

Latent SpaceLatent−FDD\frac{Latent-FD}{D}
VA-VAE9.36×10−59.36\times10^{-5}
DINOv22.65×10−32.65\times10^{-3}

C2ST. C2ST[1] uses a ResNet-18 with minimal task-specific modifications to its input stem and output head, enabling it to directly process full latent tensors and distinguish real encoded latents from generated latents while retaining the standard residual backbone. Each source contains 50k samples; real latents come from ImageNet validation images. A fixed class-stratified split allocates 30k/10k/10k per source to training/validation/testing. Learning rate and checkpoint selection use validation data only. Test accuracy across three training seeds is 60.30±0.43% / 85.22±0.32% for VA-VAE/DINOv2 (mean±SD).

We further evaluate C2ST[1] across different training set sizes (5k–30k samples per source) using ResNet-18 and model architectures (half-width ResNet-18, ResNet-18, and ResNet-34) using 30k training samples per source. Across these settings, generated and real latents remain more distinguishable in DINOv2 space than in VA-VAE space.

Although Latent-FD is scale-dependent, it provides a quantitative description of the distribution shift using a widely adopted metric. As a scale-independent complement, we provide the C2ST results, so that the distribution gap is characterized as comprehensively as possible.

We have added these details to the revised version. Your comments have helped us strengthen the supporting evidence.


Q3 (combined with W3): Robustness attribution: For the reconstruction-only control in Figure 7, what latent augmentation and reconstruction-update budget are used? Does this same-architecture control exhibit a corresponding deficit in perturbation robustness, and is the difference accompanied by worse preservation of image content?

A3: The reconstruction-only control uses the same raw-latent augmentation as RAE, i.e., z+σ⋅ϵz+\sigma \cdot \epsilon with σ∼U(0,0.8)\sigma \sim \mathcal{U}(0, 0.8), applied before normalization, and the same reconstruction-loss components. It receives 50,040 updates over 20 epochs (batch size 512). The DJT model in Figure 7 instead receives approximately 10 reconstruction-equivalent epochs.

To assess its perturbation robustness, we repeat the experiment in Fig. 1(b) across Gaussian noise strengths σ. We additionally measure content preservation under the same perturbation using pixel L1 between each decoder's perturbed and clean outputs, averaged over RGB pixels in [0,1]. Both metrics use 50k validation images; ΔrFID is relative to σ=0.

ΔrFID ↓

σRAEDJT w/o GenDJT
00.00000.00000.0000
0.11.85480.01290.0062
0.390.57570.11130.0630
0.5245.22430.24830.1269
0.8306.62300.55680.2147
1337.23300.91420.3129

L1 loss ↓

σRAEDJT w/o GenDJT
00.00000.00000.0000
0.10.07320.01910.0186
0.30.16550.04150.0401
0.50.21630.05580.0539
0.80.26020.07230.0700
10.27980.08220.0798

As shown in the tables above, with joint training, DJT is more robust in terms of both rFID and the preservation of image content. This demonstrates that joint training helps the decoder better handle perturbations and thus achieve better robustness to distribution shift.

AuthorsOpenReview ↗

Response to Reviewer TeYi 3/3

Q4: Inference-cost accounting: Table 3 describes parameter counts for the complete inference pipeline, while Appendix A specifies an additional DJT-S checkpoint for autoguidance. Please distinguish the main denoiser–decoder parameter count from the complete guided-system count, apply the same convention to the baselines, and report total inference compute or latency under matched sampling settings.

A4: We agree that Table 3 should distinguish the main denoiser–decoder pipeline from the complete system including autoguidance. Counting shared weights once and including the full auxiliary checkpoint gives:

PipelineMain parametersWith autoguidanceUnguided TFLOPs/imageGuided TFLOPs/imageUnguided Latency/image
RAE DiT^DH-XL + Dec-XL1,254.0M1,450.5M16.05641.3911242.98ms
DJT-XL870.7M1,082.4M16.08941.4241266.44ms

DJT's auxiliary S checkpoint contributes 211.7M parameters, including its unused 15.3M decoding head; RAE's auxiliary denoiser contributes 196.5M. Thus, the parameter reduction is 30.6% for the main pipeline and 25.4% including the full auxiliary checkpoint, with improved performance and less than 0.1% computation overhead. Encoder and training-loss networks are excluded from both generation systems.

Thanks again for your valuable suggestions, and we hope the above responses can address your concerns.


Reference:

[1] Lopez-Paz et al., "Revisiting Classifier Two-Sample Tests." ICLR, 2017.

Reviewer_8hfzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DJT (Denoise-decode Joint Transformer), which unifies the latent denoiser and the pixel decoder into a single weight-shared Transformer. The motivation is twofold. First, separately trained denoisers and decoders in DINOv2 latent space show high representational similarity in early layers. Second, decoders in high-dimensional semantic spaces are fragile to the shift between encoded and generated latentss. The authors therefore share one trunk between the two tasks, with two lightweight task-specific heads, a mode token, and a Bernoulli router that sends 10% of training samples to reconstruction and 90% to flow-matching denoising. At inference, the denoising branch generates a clean latent and the decoding branch renders the image. On ImageNet 256x256, DJT-XL reports 0.97 gFID with guidance and 1.35 without, using roughly 30% fewer parameters than the RAE pipeline.

Critical strengths
  • Removing the standalone ViT decoder cuts unique parameters by about 30% and decoding FLOPs by about 50 againste RAE at small scale and by over 90% against the VA-VAE decoder, while matching or exceeding the strongest RAE-based pipelines on ImageNet 256. Because the gain does not come with extra complexity, the recipe is simple and likely to transfer to other latent spaces.
  • The DJT w/o Gen ablation at Figur 7 (6.58 to 5.02 gFID on the same generated latents) and the slower rFID degradation under latent perturbations at Figure 1b make a concrete case that the denoising objective makes the decoding path more tolerant of off-manifold latents. This is a useful and reusable insight for two-stage latent generative models beyond RAE.
Critical weaknesses
  • The evaluation is limited to a single representation encoder (DINOv2-B) apart from VA-VAE, a single dataset and a single resolution (ImageNet 256x256), although the central claim concerns decoding in high-dimensional representation spaces in general. Other encoders such as SigLIP2, MAE or larger DINOv2 variants would test this claim directly, and higher resolutions such as 512x512 are where decoding cost and detail fidelity matter most. More importantly, the gain of DJT is attributed to decoder robustness against off-manifold generated latents, and this benefit may shrink as generators improve and produce latents closer to the data manifold. Also do the improvement holds on top of stronger RAE baselines such as RAEv2 ?
Questions for authors
  • Could the authors clarify in the related work how DJT differs from ELF (Hu et al., 2026), which also shares the denoising network with the decoder?
  • How many reconstruction updates does DJT w/o Gen receive in Figure 7, and what is its rFID? Could latents generated by an independent frozen denoiser, such as the RAE DiTDH-S used in Table 2, be decoded by both DJT and DJT w/o Gen, and what is the gFID gap then?
  • Could training curves (gFID vs. epochs, and ideally vs. total training compute) be reported for DJT and the RAE pipeline?
  • Could DJT be applied on top of RAEv2 or with a different representation encoder, and evaluated at 512x512, to show that the robustness gain persists with stronger generators and other latent spaces?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 8hfz 1/2

Thank you for your support and constructive feedback. We sincerely appreciate the time and care you devoted to reviewing our work.

Q1: Could the authors clarify in the related work how DJT differs from ELF (Hu et al., 2026), which also shares the denoising network with the decoder?

A1: Thank you for raising the question of architectural differences. ELF is a language model that uses a DiT with an additional linear head for reconstruction and inserts many context tokens for performance optimization. We only adopt its Bernoulli router, which is a common technique for assigning the samples of a minibatch to tasks during training. The rest of our architectural design differs significantly from ELF's.

Specifically, our model uses a wide-head transformer following DDT and RAE, which has been verified to work well for high-dimensional latents. We also optimize the task-specific branches in terms of where to branch and how deep the branches should be, as well as the training budget for each branch. To make these choices practical for the community, we conduct the experiments at the XL scale, so that our design choices are supported by large-scale validation.

Additionally, we conducted ablation experiments during the rebuttal, in which either the transformer is replaced with an ELF-style DiT or the task-specific branches are reduced to ELF's simple shallow heads. All models are trained for 60 epochs due to limited computational resources. For DJT, all reconstruction losses are enabled.

model50k gFID50k rFID
DJT6.130.99
backbone switched to ELF-style DiT9.841.17
branches reduced to ELF-style heads11.141.75

As shown above, both modifications lead to clearly degraded performance, indicating that the wide-head transformer backbone and the carefully designed task-specific branches are important to the effectiveness of our method. These results further demonstrate that our architecture is well suited to high-dimensional image latents and provides substantial improvements over the corresponding ELF-style design choices.


Q2: How many reconstruction updates does DJT w/o Gen receive in Figure 7, and what is its rFID? Could latents generated by an independent frozen denoiser, such as the RAE DiTDH-S used in Table 2, be decoded by both DJT and DJT w/o Gen, and what is the gFID gap then?

A2: Thank you for this question. In Figure 7, we keep the number of reconstruction samples seen by the decoding branch comparable, so that the comparison is controlled.

Specifically, DJT w/o Gen is trained from scratch for 20 epochs with a global batch size of 512, matching the RAE decoder's batch size. It receives 50,040 optimizer updates and achieves an rFID of 0.71.

The DJT model in Figure 7 uses a reconstruction routing probability of p=0.1 and is trained for 100 epochs, which corresponds to 10 equivalent epochs for the decoding branch. We have clarified this in the revised version.

For the independent-denoiser comparison, as requested, we provide the results below. The latents are the same 50k unguided latents generated by the frozen RAE DiT^DH-S denoiser used in Table 2:

DecoderEquivalent epochsrFID ↓50k gFID ↓
DJT w/o Gen200.715.68
DJT-S (recon_prob=0.1)100.844.78
DJT-S (recon_prob=0.2)200.774.75

Despite having a higher clean-latent rFID, DJT-S improves gFID by 0.90 over reconstruction-only training. This supports the benefit of joint training for decoding latents affected by distribution shift.

AuthorsOpenReview ↗

Response to Reviewer 8hfz 2/2

Q3: Could training curves (gFID vs. epochs, and ideally vs. total training compute) be reported for DJT and the RAE pipeline?

A3: Thank you for the suggestion. We have updated the main paper and now report gFID vs. epochs, GPU hours, and Gflops in the revised version. As a quick preview: at all five shared evaluation epochs, DJT achieves a lower gFID than RAE, reaching 10.87 versus 15.47 at 100 epochs. Note that the gFID values on these curves are computed on 5k class-balanced samples, so they are higher than the 50k gFID reported in the main tables. Moreover, DJT-S achieves better generation quality with fewer GPU hours and training FLOPs, demonstrating a more favorable trade-off between training cost and generation quality.

We additionally report the average time (s) per forward step. All costs are measured on 8×A800 80GB GPUs with FP32 precision (the RAE default), using models of similar parameter counts.

MethodDecode Training ↓\downarrowDenoise Training ↓\downarrow
RAE3.0450.563
DJT (ours)1.0961.096

VAE training may need to traverse billions of images. As the amount of image data and the number of training steps increase, the overall training time will therefore favor the method that spends less time on decoder training. As shown above, our method achieves a better balance by integrating decoder and denoiser training into one joint architecture with a parallel forward loop.


Q4: Could DJT be applied on top of RAEv2 or with a different representation encoder, and evaluated at 512x512, to show that the robustness gain persists with stronger generators and other latent spaces?

A4: We train RAE DiT^DH-S and DJT-S in the RAEv2 representation space (which uses multi-layer DINOv3 features as the latent) at 256×256 resolution:

Model50k gFID ↓
RAE DiT^DH-S + Dec-B8.82
DJT-S8.59

We also train and evaluate both models at 512×512 resolution:

Model50k gFID ↓
RAE DiT^DH-S + Dec-B7.53
DJT-S6.70

These results demonstrate that the proposed unified design generalizes across diverse latent representations and image resolutions, consistently improving generation quality.


W1: More importantly, the gain of DJT is attributed to decoder robustness against off-manifold generated latents, and this benefit may shrink as generators improve and produce latents closer to the data manifold...

A5: Good question! To probe whether the advantage remains at a larger scale, we further conduct a cross-decoding experiment using XL generators and large decoders. The results are shown below.

GeneratorRAE Dec-XL gFID ↓DJT-XL decoder gFID ↓
DiT-XL1.421.29
DJT-XL1.441.29

As shown in the table, the robustness advantage of our decoder remains with strong generators. This further supports our claim that joint training helps address the train-test mismatch: the DJT-XL decoder yields better results regardless of the generator, showing that its robustness to distribution shift generalizes.

We would also like to note that, in a scaled-up generative pipeline, both the decoder and the denoiser are heavy. Beyond training efficiency, DJT's design simplifies inference deployment: it can perform denoising and decoding in parallel for batched tasks within each forward pass, which simplifies the switch scheduling between the denoiser and the decoder and saves both time and memory substantially. We regard this as another potential advantage.

Thanks again for your time devoted to review our work, and we hope the above responses can address your concerns.

Dear Reviewer 8hfz,

We kindly remind you that the discussion period will end in 30 hours. We would be grateful to know whether our responses have adequately addressed your concerns. If so, we would sincerely appreciate it if you could consider updating your rating accordingly. We are also happy to clarify any remaining questions.

Thank you again for your time and thoughtful review.

Best regards,

The Authors

Reviewer_twKjOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DJT shares a subset of transformer weights between latent denoising and image decoding. Mode conditioning (diff or dec) and sample routing select the task. The representation encoder stays frozen. The method reuses a common backbone for iterative generation and a final decoding pass, with task-specific heads. DJT-XL reports ImageNet-256 gFID of 0.97 with guidance and 1.35 without guidance.

Critical strengths
  1. Unifying generation and reconstruction is an important and promising direction and this paper presents a good effort.
  2. It further improves state-the-art, though only evaluated by FID. It's unclear whether the gap would hold under other representation-diverse evaluation metrics.
Critical weaknesses
  1. Training time not reported. Training a good reconstruction decoder requires substantially more compute than training a diffusion model. Scaling up the shared transformer blocks also means putting more compute on the reconstruction model (perceptual loss + discriminator etc). I wonder how much additional time this method requires compared to the raw RAE.

  2. This method claims to work on VAE but only VA-VAE's results are shown. VA-VAE is never a typical VAE. It's still heavily semantics-pilled. To support the claim in the paper, please show results on standard VAEs.

  3. The reconstruction performance doesn't improve over normal RAEs.

Questions for authors

Table 1 never mentions what two sets of latents are.

3 comments
AuthorsOpenReview ↗

Response to Reviewer twKj 1/2

Thank you for your thoughtful and constructive feedback. Your comments have helped us sharpen our claims and strengthen the supporting evidence.

W1: Training time not reported. Training a good reconstruction decoder requires substantially more compute than training a diffusion model. Scaling up the shared transformer blocks also means putting more compute on the reconstruction model (perceptual loss + discriminator etc). I wonder how much additional time this method requires compared to the raw RAE.

A1: Thank you for raising the question of training cost. Decoder training indeed requires a large amount of computation to achieve good reconstruction. Below, we report the average time (s) per forward step. All costs are measured on 8×A800 80GB GPUs with FP32 precision (the RAE default), using models of similar parameter counts. As requested, we report the time cost with the perceptual loss and the discriminator (GAN) loss fully activated.

MethodDecode Training ↓\downarrowDenoise Training ↓\downarrow
RAE3.0450.563
DJT (ours)1.0961.096

Modern VAE training may need to traverse billions of images. As the amount of image data and the number of training steps increase, the overall training time will therefore favor the method that spends less time on decoder training. As shown above, our method achieves a better balance by integrating decoder and denoiser training into one joint architecture with a parallel training loop.

For the ImageNet-1k (IN-1k) results reported in the paper, the statistics (with 100 epochs of denoiser training) are:

ModelDecoder Training ↓\downarrowDenoiser Training ↓\downarrowTotal Time (GPU hours)50k gFID ↓\downarrow
RAE190 (16 epochs on IN-1k)1563467.41
DJT (ours)-- (equivalent to 10 epochs on IN-1k)--2665.02

As shown above, our method also achieves better training efficiency in ImageNet-scale experiments.

We would also like to note that, in industry-level inference deployment, DJT can perform denoising and decoding in parallel within a single batched forward pass, which simplifies the switch scheduling between the denoiser and the decoder and saves both time and memory substantially. We regard this as another potential advantage.


W2: This method claims to work on VAE but only VA-VAE's results are shown. VA-VAE is never a typical VAE. It's still heavily semantics-pilled. To support the claim in the paper, please show results on standard VAEs.

A2: Thank you for this insightful question. We conducted experiments using SD-VAE as the image encoder and trained both models for 100 epochs. The results are shown below:

Model in SD-VAE space50k gFID (w/o CFG) ↓50k gFID (w/ CFG) ↓
DiT24.715.49
DJT (ours)20.875.01

Our method consistently achieves better performance in the SD-VAE latent space. Note that under this setting, the decoding cost differs significantly between the standalone SD-VAE decoder (310 Gflops) and our decoder integrated in DJT (22 Gflops).

Additionally, we have supplemented Table 1 with the statistics of SD-VAE in the revised version. SD-VAE shows an even smaller distribution gap (C2ST acc. 57.23%, Latent-FD/DD 3.30×10−53.30\times10^{-5}) than VA-VAE (60.30%, 9.36×10−59.36\times10^{-5}), and both are far below DINOv2 (85.22%, 2.65×10−32.65\times10^{-3}). This demonstrates that our observation is not restricted to the VA-VAE space and also holds for a standard VAE.


W3: The reconstruction performance doesn't improve over normal RAEs.

A3: We agree that the reconstruction performance on clean latents is not improved. However, DJT clearly decodes generated latents better, which matters more in a generative pipeline. When decoding the same latents generated by an independent RAE denoiser (the setting of Table 2 in the main paper), DJT-S reduces gFID to 4.75, surpassing both RAE's Dec-B (7.41) and Dec-XL (5.77), despite using a much smaller decoding path. This distinction is consistent with the reconstruction–generation mismatch discussed in [1], which is also the primary motivation of this paper, as the mismatch becomes more severe for high-dimensional latents.

DJT's PSNR also exceeds that of the original RAE decoder. We acknowledge that further improvement can be made on the encoder side, as discussed in Appendix G.

AuthorsOpenReview ↗

Response to Reviewer twKj 2/2

Q1: Table 1 never mentions what two sets of latents are.

A4: Thanks for the question. As stated in the bold caption of Table 1, the two sets are latents encoded from real images and latents sampled by a latent denoiser. To make this clearer, we provide the details here. For Latent-FD, the real reference set contains all 1,281,167 ImageNet training images; for C2ST, it contains the 50,000 validation images. For the generated set, both metrics use the same 50,000 class-balanced generated latents, sampled without guidance (CFG=1).

The measurement protocol follows the original implementation of C2ST [2]. The denoisers are SiT-XL for SD-VAE, LightningDiT-XL for VA-VAE, and RAE DiT^DH-XL for DINOv2, all of which are officially released checkpoints. We have stated this clearly in the revised version.

Thank you again for your careful review.


[1] Fang et al. "A Latent Distribution Perspective on Evaluating and Improving Latent Generative Models." arXiv:2609.24088, 2026.

[2] Lopez-Paz et al., "Revisiting Classifier Two-Sample Tests." ICLR, 2017.

Dear Reviewer twKj,

We kindly remind you that the discussion period will end in 30 hours. We would be grateful to know whether our responses have adequately addressed your concerns. If so, we would sincerely appreciate it if you could consider updating your rating accordingly. We are also happy to clarify any remaining questions.

Thank you again for your time and thoughtful review.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.