- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the Denoise-decode Joint Transformer (DJT), which shares a transformer backbone between latent denoising and image reconstruction while retaining shallow task-specific heads. A mode condition distinguishes the two tasks, and a Bernoulli router assigns training examples to denoising or reconstruction. At inference, the denoising branch first generates a latent through iterative sampling, after which the network switches to reconstruction mode to produce an image. The main experiments use frozen DINOv2 representations on ImageNet-256, with additional results in VA-VAE latent space. The authors report improvements in generation quality and a reduction in unique denoiser–decoder parameters, and investigate representation similarity, activation interchangeability, and robustness to latent perturbations.
Critical strengths
-
The reported generation results make parameter sharing practically compelling. DJT-XL achieves unguided gFID 1.35 and guided gFID 0.97, compared with 1.51 and 1.13 for the RAE entry in Table 3. The main DJT denoiser–decoder network contains approximately 870M unique parameters, compared with 1,254M for the listed RAE denoiser and decoder. The unguided improvement is particularly useful evidence that the result is not solely attributable to guided sampling.
-
The paper includes informative controls beyond comparing complete generation pipelines. Table 2 evaluates alternative decoders on latents generated by the same frozen RAE denoiser, and Figure 7 compares joint training with a reconstruction-only model using the DJT decoding architecture on the same DJT-generated latents. These experiments support a decoder-side benefit from joint training. The branch-point and task-allocation ablations also provide useful evidence about how extensively the two tasks can share computation.
Critical weaknesses
-
The architectural novelty relative to ELF is limited and needs more precise positioning. ELF (Embedded Language Flows, Hu et al., 2026; arXiv:2605.10938) already uses a shared denoising/decoding network, a binary mode condition, stochastic allocation between training objectives, and a final switch to decoding after continuous-space generation. DJT cites this work, but the overlap extends beyond the Bernoulli routing procedure to the overall training and inference template. Image reconstruction introduces different output requirements and losses, and the proposed task-specific heads are a concrete architectural difference; nevertheless, the contribution should be evaluated as an image-domain adaptation and investigation of this shared-network paradigm. The current experiments establish that the adaptation works, but less clearly establish which image-specific design choices are essential beyond the inherited template.
-
The proposed distribution-shift explanation is not sufficiently supported by the measurement protocol and mechanism-specific controls. Table 1 compares Latent-FD values across VA-VAE and DINOv2 spaces with different dimensionalities. Fréchet distance depends on representation scale: multiplying both distributions by a scalar (c) multiplies the distance by (c^2). Without an adequately specified normalization and estimation protocol, the numerical difference cannot establish that one latent space has a larger mismatch in a representation-independent sense. The C2ST results offer complementary evidence, but their interpretation requires the classifier architecture, sample sizes, and held-out evaluation protocol.
-
The robustness experiments also warrant a narrower interpretation. Gaussian perturbations need not reproduce errors in generated latents, as the authors acknowledge, and Appendix A specifies Gaussian augmentation during reconstruction training. The fixed-latent controls in Table 2 and Figure 7 are valuable, but do not by themselves establish that improved perturbation robustness mediates the generation gains. In particular, the reconstruction-only control should be evaluated on the same robustness curves, with its augmentation settings and effective reconstruction-training budget specified. Distributional rFID should also be complemented by a measure of preservation of the corresponding image content under perturbation.
Questions for authors
-
Novelty and design attribution: Beyond changing the output domain and reconstruction losses, which image-specific architectural choices are essential relative to ELF? What are the individual contributions of the explicit mode embedding and task-specific branches? The existing branch-point ablation is useful, but does not fully answer these questions.
-
Distribution-gap measurement: How exactly is Latent-FD computed, including spatial aggregation, dimensionality handling, normalization, sample sizes, and covariance estimation? What classifier and train/test protocol are used for C2ST? Does the relative conclusion remain stable under reasonable changes to the representation scaling and evaluation protocol?
-
Robustness attribution: For the reconstruction-only control in Figure 7, what latent augmentation and reconstruction-update budget are used? Does this same-architecture control exhibit a corresponding deficit in perturbation robustness, and is the difference accompanied by worse preservation of image content?
-
Inference-cost accounting: Table 3 describes parameter counts for the complete inference pipeline, while Appendix A specifies an additional DJT-S checkpoint for autoguidance. Please distinguish the main denoiser–decoder parameter count from the complete guided-system count, apply the same convention to the baselines, and report total inference compute or latency under matched sampling settings.
6 comments
Response to Reviewer TeYi 2/3
Q2 (combined with W2): Distribution-gap measurement: How exactly is Latent-FD computed, including spatial aggregation, dimensionality handling, normalization, sample sizes, and covariance estimation? What classifier and train/test protocol are used for C2ST? Does the relative conclusion remain stable under reasonable changes to the representation scaling and evaluation protocol?
A2: Thank you for the careful review. We provide the detailed measurement protocol below:
Latent-FD. Each 16×16 grid is spatially averaged into a single D-dimensional vector (D=32 for VA-VAE; D=768 for DINOv2), without whitening or dimensionality reduction. DINOv2 uses the released mean/variance normalization; VA-VAE uses posterior samples normalized by its released mean/std. We compare 50,000 unguided generated samples with all 1,281,167 encoded ImageNet training images. For the real and generated pooled features separately, we estimate full sample covariance matrices using float64 moment accumulation, an denominator, and no shrinkage.
To account for dimensionality, we additionally report below. The ordering persists after dimension normalization, but remains scale-dependent. We therefore complement Latent-FD with C2ST[1], which provides a measure of empirical distinguishability by evaluating how accurately a classifier separates generated from real latents on held-out data.
| Latent Space | |
|---|---|
| VA-VAE | |
| DINOv2 |
C2ST. C2ST[1] uses a ResNet-18 with minimal task-specific modifications to its input stem and output head, enabling it to directly process full latent tensors and distinguish real encoded latents from generated latents while retaining the standard residual backbone. Each source contains 50k samples; real latents come from ImageNet validation images. A fixed class-stratified split allocates 30k/10k/10k per source to training/validation/testing. Learning rate and checkpoint selection use validation data only. Test accuracy across three training seeds is 60.30±0.43% / 85.22±0.32% for VA-VAE/DINOv2 (mean±SD).
We further evaluate C2ST[1] across different training set sizes (5k–30k samples per source) using ResNet-18 and model architectures (half-width ResNet-18, ResNet-18, and ResNet-34) using 30k training samples per source. Across these settings, generated and real latents remain more distinguishable in DINOv2 space than in VA-VAE space.
Although Latent-FD is scale-dependent, it provides a quantitative description of the distribution shift using a widely adopted metric. As a scale-independent complement, we provide the C2ST results, so that the distribution gap is characterized as comprehensively as possible.
We have added these details to the revised version. Your comments have helped us strengthen the supporting evidence.
Q3 (combined with W3): Robustness attribution: For the reconstruction-only control in Figure 7, what latent augmentation and reconstruction-update budget are used? Does this same-architecture control exhibit a corresponding deficit in perturbation robustness, and is the difference accompanied by worse preservation of image content?
A3: The reconstruction-only control uses the same raw-latent augmentation as RAE, i.e., with , applied before normalization, and the same reconstruction-loss components. It receives 50,040 updates over 20 epochs (batch size 512). The DJT model in Figure 7 instead receives approximately 10 reconstruction-equivalent epochs.
To assess its perturbation robustness, we repeat the experiment in Fig. 1(b) across Gaussian noise strengths σ. We additionally measure content preservation under the same perturbation using pixel L1 between each decoder's perturbed and clean outputs, averaged over RGB pixels in [0,1]. Both metrics use 50k validation images; ΔrFID is relative to σ=0.
ΔrFID ↓
| σ | RAE | DJT w/o Gen | DJT |
|---|---|---|---|
| 0 | 0.0000 | 0.0000 | 0.0000 |
| 0.1 | 1.8548 | 0.0129 | 0.0062 |
| 0.3 | 90.5757 | 0.1113 | 0.0630 |
| 0.5 | 245.2243 | 0.2483 | 0.1269 |
| 0.8 | 306.6230 | 0.5568 | 0.2147 |
| 1 | 337.2330 | 0.9142 | 0.3129 |
L1 loss ↓
| σ | RAE | DJT w/o Gen | DJT |
|---|---|---|---|
| 0 | 0.0000 | 0.0000 | 0.0000 |
| 0.1 | 0.0732 | 0.0191 | 0.0186 |
| 0.3 | 0.1655 | 0.0415 | 0.0401 |
| 0.5 | 0.2163 | 0.0558 | 0.0539 |
| 0.8 | 0.2602 | 0.0723 | 0.0700 |
| 1 | 0.2798 | 0.0822 | 0.0798 |
As shown in the tables above, with joint training, DJT is more robust in terms of both rFID and the preservation of image content. This demonstrates that joint training helps the decoder better handle perturbations and thus achieve better robustness to distribution shift.
Response to Reviewer TeYi 3/3
Q4: Inference-cost accounting: Table 3 describes parameter counts for the complete inference pipeline, while Appendix A specifies an additional DJT-S checkpoint for autoguidance. Please distinguish the main denoiser–decoder parameter count from the complete guided-system count, apply the same convention to the baselines, and report total inference compute or latency under matched sampling settings.
A4: We agree that Table 3 should distinguish the main denoiser–decoder pipeline from the complete system including autoguidance. Counting shared weights once and including the full auxiliary checkpoint gives:
| Pipeline | Main parameters | With autoguidance | Unguided TFLOPs/image | Guided TFLOPs/image | Unguided Latency/image |
|---|---|---|---|---|---|
| RAE DiT^DH-XL + Dec-XL | 1,254.0M | 1,450.5M | 16.056 | 41.391 | 1242.98ms |
| DJT-XL | 870.7M | 1,082.4M | 16.089 | 41.424 | 1266.44ms |
DJT's auxiliary S checkpoint contributes 211.7M parameters, including its unused 15.3M decoding head; RAE's auxiliary denoiser contributes 196.5M. Thus, the parameter reduction is 30.6% for the main pipeline and 25.4% including the full auxiliary checkpoint, with improved performance and less than 0.1% computation overhead. Encoder and training-loss networks are excluded from both generation systems.
Thanks again for your valuable suggestions, and we hope the above responses can address your concerns.
Reference:
[1] Lopez-Paz et al., "Revisiting Classifier Two-Sample Tests." ICLR, 2017.
Response to Reviewer TeYi 1/3
Thank you for your thoughtful and constructive feedback. We sincerely appreciate the time and care you devoted to reviewing our work. Below, we provide point-to-point responses to the review.
A1: Thank you for raising the question of design attribution.
First, DJT is designed to tackle the long-standing train-test mismatch in the generative pipeline, rather than to adapt the ELF design to the image domain. ELF adopts a DiT-based architecture augmented with an additional linear reconstruction head and multiple inserted context tokens for performance optimization. In contrast, our method only adopts its Bernoulli router, a common mechanism for routing the samples of a minibatch during training. Beyond this routing component, the architectural design of our method differs substantially from that of ELF.
Specifically, our model uses a wide-head transformer following DDT and RAE, which has been verified to work well for high-dimensional latents. We also optimize the task-specific branches in terms of where to branch and how deep the branches should be, as well as the training budget for each branch. To validate these design choices in practice, we conduct the experiments at the XL scale, demonstrating the effectiveness of our architecture in large-scale settings.
We have also provided the implementation details and source code, which we believe help address the concern about architectural similarity. As for the individual contributions of the proposed components, we conduct the following ablation experiments.
To examine the role of our branch design, we replace our task-specific branches with ELF-style simple heads. As shown in the table below, the performance degrades clearly, demonstrating that the carefully designed task-specific branches are important to the effectiveness of our method.
To verify the role of the explicit mode embedding, we first replace the mode embedding with zero during inference and evaluate its performance. Then we conduct standalone experiments where the mode embedding is explicitly inserted as a context token. As shown in the table below, strengthening the condition improves both gFID and rFID, whereas weakening it degrades both substantially. This demonstrates that the mode token provides useful information for the shared backbone.
These results further demonstrate that our architecture is well suited to high-dimensional image latents and provides substantial improvements over the corresponding ELF-style design choices.