acceptodds
Under review as a conference paper at ICLR 2027

Decoding Without a Decoder: Direct Pixel Prediction from Diffusion Transformers

Abstract

Latent diffusion models rely on a decoder to map denoised latents to pixels, adding complexity and inference cost. We introduce Decoding Without a Decoder (DWD), which lets the diffusion transformer produce final pixels directly through a linear head. Unlike previous methods, where the denoising output leads to final pixels, DWD separates the two roles: the original denoising head continues to drive sampling, then the new DWD head predicts the pixels at the final step. We extend pretrained latent models with DWD efficiently by finetuning only the last few transformer blocks, training the DWD head with pixel quality losses while regularizing the denoising head toward the pretrained model to preserve the sample trajectory. Across latent image and video generators, DWD removes the decoder and reduces inference cost while maintaining or even improving pixel quality. On Wan video generation, it visibly reduces detail distortion and flickering, lowering warping error by 23%. Remarkably, the same separation also improves pixel diffusion models that do not even use a decoder: a final-step DWD head produces substantially better texture detail in text-to-image generation, and improves ImageNet FID from 1.55 to 1.43, a new state of the art among pixel flow models. Code and models will be released.

Primary area:
generative models
Keywords:
pixel space, video generation, image generation, latent diffusion, flow models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_P8dtOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper targets the inference cost of the VAE decoder in latent diffusion models and the oversmoothed textures of pixel diffusion models. It adds a linear pixel head in parallel with the denoising head of a pretrained DiT, finetunes the last few blocks with L1, LPIPS and adversarial losses plus a trajectory regularization toward the frozen model, and uses this head only at the final step to output pixels directly. The method is evaluated on ImageNet (SiT-REPA, REPA-E, JiT, AnisoFlow), text-to-image (FLUX.2 klein, AnisoFlow-T2I) and text-to-video (Wan2.1 14B), where it removes the decoder with comparable or better FID and lowers warping error on video.

Critical strengths

The motivation and intuition are sound.

The linear probe in Fig. 3 makes a convincing case that last-layer DiT tokens already carry rich pixel information. The idea of separating the final pixel output from the denoising output is simple and well motivated.

Critical weaknesses

Weakness 1. The method seems does not really decode, and its logic does not extend to recent few/one-step models.

At t=0, where the head acts as a pure decoder, the quality gain disappears. Sec. 3.2 states that quality “substantially worsens”, and Fig. 4(b) shows results as smooth as the baseline, although the Fig. 4 caption’s “matches or outperforms the baseline’s” conflicts with its own “recovers the gain” and should be clarified. The quality gain therefore depends on evaluating the head at a nonzero timestep, which for the latent models is t=0.08 on ImageNet and t=0.3 for FLUX.2 klein Base and Wan (Table 4). Wan’s pixel head is never even trained below t≈0.125 after shifting (p_pix = [0.02, 0.1] with shift 7). At this noise level fine structure such as small text or mesh is likely not yet determined, so the head generates the remaining trajectory in one shot under L1+LPIPS+GAN losses. This looks much closer to GAN-like generation than to decoding. The comparison thus mixes decoder removal, a sampling schedule that ends at t=0.3, and adversarial generation, and no control separates them on Wan, where the largest latent-model gain is reported (e.g., the same final timestep with a GAN-tuned latent final step and the original decoder). The t_final sweep (Fig. 4(a)) and the adversarial-loss ablation (Table 2) address this only partially, on ImageNet and text-to-image. Notably, without the adversarial loss the COCO FID gain on AnisoFlow-T2I disappears (28.93 vs. 28.86 for the baseline, Table 2). The design is also close to the denoising decoder of LTX-Video (HaCohen et al., 2024) and the early termination in PiD (Lu et al., 2026). LTX-Video is cited only as a generic latent model, and PiD’s early termination is not discussed. The argument also weakens as the step count drops. In the 4-step FLUX model the head already jumps from t=0.7 and only ties the baseline. In one-step generators (DMD, flow maps) there is no trajectory to preserve, and paired L1/LPIPS becomes ill-posed. One-step models and few-step video models, including the 4-step Wan 1.3B used as motivation in the introduction, are not evaluated, even though this is where decoder cost matters most. Since few- and single-step generation is now widely used and the paper targets generation efficiency, this limits its generality. If DWD does not apply to one-step models, simply using such a model with its decoder would likely be faster.

Weakness 2. Baselines are limited to the vanilla models, with no cost-matched methods that share the same goal.

The clearest gap is AsymFlow (Chen et al., 2026a), which converts the same FLUX.2 klein 9B into a decoder-free pixel model at roughly the same per-step cost and reports gains over its latent base, whereas DWD only ties it. The paper cites AsymFlow and borrows its Oklab output and captions but reports no numbers. L2P (Chen et al., 2026c) also removes the VAE from a pretrained latent model by training only shallow layers and reports much faster native 4K inference. FlashDecoder (arXiv:2607.14898) targets the same Wan decoder bottleneck without touching the generator. These are only examples; my main point is that, since DWD aims at efficiency, it should be compared against other methods with the same goal, not only against the vanilla base models. I appreciate that the main goal is cost reduction, so matching the baseline quality would be enough if the savings were meaningful. However, since the inference cost of DWD is reported only in FLOPs, without wall-clock time or memory, it is unclear whether it is more efficient than these methods. Finally, the existing comparisons are not controlled. For the latent models, DWD alone gets a different final timestep, retuned guidance for SiT-REPA (App. A.3), and curated ≥720p training data for Wan. Its frozen blocks are also quantized to FP8 during finetuning, which may explain its robustness under 8-bit inference (Table 3).

Also, the pixel-space results rely mainly on AnisoFlow, which is unpublished and cannot be verified. In text-to-image, the only pixel baseline is AnisoFlow-L/16-T2I, which is much weaker than current state-of-the-art generators (COCO FID 28.86 vs. 15.13 for FLUX.2 klein Base). The gains from DWD may therefore look significant mainly because of this weak baseline.

References

  • Chen et al., Asymmetric Flow Models, arXiv:2605.12964, 2026.
  • Lu et al., PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion, arXiv:2605.23902, 2026.
  • HaCohen et al., LTX-Video: Realtime Video Latent Diffusion, arXiv:2501.00103, 2024.
  • Chen et al., L2P: Unlocking Latent Potential for Pixel Generation, arXiv:2605.12013, 2026.
  • FlashDecoder: Real-Time Latent-to-Pixel Streaming Decoder with Transformers, arXiv:2607.14898, 2026.
Questions for authors

The main claim of the paper feels too broad. The title and introduction frame DWD as a way to remove the decoder for efficiency, but the strongest results come from pixel models that have no decoder at all (texture detail and ImageNet FID), and the video section focuses on warping error. It is unclear which of these is the main contribution. Could the authors clarify which one they consider central? Narrowing the scope, either to efficiency with wall-clock evidence in few-step settings or to the dual-head separation principle, might make the paper more convincing.

6 comments
AuthorsOpenReview ↗

Response to Reviewer P8dt (1/2)

We thank the reviewer for finding the motivation and intuition sound, the linear probe in Fig. 3 convincing, and the separate pixel output simple and well motivated.

W1: Whether DWD "decodes", given that the quality gain disappears at t=0t=0

DWD is designed not to decode the clean latents. Its main point is that the final pixels should come from a separate output on the DiT, evaluated before the trajectory is fully denoised. The t=0t=0 experiment is meant to support this, so we do not consider it a weakness. In the Fig. 4 caption, we meant that at tfinal=0t_{\mathrm{final}}=0 the FID goes back to roughly the baseline level, and a small positive timestep recovers DWD's gain. We will rephrase the caption.

W1: The nonzero final timestep means the head generates the remaining trajectory in one shot, closer to GAN-like generation than to decoding

t=0.3t=0.3 for video is a very small timestep under a shift of 7, which corresponds to t≈0.058t\approx0.058 before shifting. This needs to compensate for the resolution and frame count of video. The same principle applies to decoder training. Recent decoders such as RAE add noise with a standard deviation up to 0.8 to unit-variance latents during training. The adversarial loss is not essential to DWD (Table 2). The improved moving details on Wan and the fully preserved text rendering on FLUX.2 (LongText-Bench 0.909 vs. 0.908) also show that the head does not damage content.

We skip tiny timesteps for Wan (t<0.02t<0.02 before shifting) because Wan's own outputs there have strong color drift and detail degradation (presumably due to logit-normal pretraining and shared modulation). Regular sampling grids do not use these timesteps anyway. We will clarify this in the revised paper.

W1/W2: Controlled comparisons for the final timestep and the adversarial loss

On ImageNet and text-to-image, Fig. 4 controls the final timestep and Table 2 the adversarial loss. DWD keeps the sampler's own final timestep for the pixel models and the 4-step models (listed as "sampler" in Table 4). The pixel models still improve substantially under the same final timestep (Table 1), so the gains come from the dual-head separation. Fig. 4(a) also sweeps the final timestep of a latent baseline (REPA-E) with its native decoder, and this does not improve over its default setting. Setting the final timestep explicitly is also common in prior work, e.g., SiT (also used by REPA and REPA-E) sets the final sigma to 0.04, much earlier than a 250-step uniform grid.

Repeating these controls on Wan 14B is very expensive (VBench inference alone costs 260 GPU hours), so we could not run them within the rebuttal period. Instead, our new 4-step Wan 1.3B experiment keeps the baseline's exact timesteps, and DWD still reduces warping error (part 2).

W1: Without the adversarial loss, the COCO FID gain on AnisoFlow-T2I disappears

DWD's largest gain on pixel models is in patch FID, which reflects fine-grained detail. Without the adversarial loss, patch FID is still clearly better than the baseline (34.52 vs. 36.59, Table 2).

W1: Similarity to the denoising decoder of LTX-Video and the early termination in PiD

LTX-Video and PiD both use a separate decoder network (PiD uses a second pixel diffusion model), while DWD removes the decoder and reads the pixels out from the DiT itself. Early termination is not the core idea of DWD. We will add a discussion of these works and make the architectural distinction clearer.

W2: Retuned guidance for REPA

Tuning the guidance is common practice on ImageNet, and the baselines themselves use tuned guidance (e.g., AsymFlow states this explicitly). We verified that retuning the baseline guidance did not give a clearly better FID. For a fair comparison, we also retune it for each DWD model. DWD does not change the optimal guidance for most models, and REPA is the only exception.

W2: Frozen blocks quantized to FP8 during finetuning, possibly explaining the robustness under 8-bit inference

The FP8 quantization of the frozen blocks is only there to speed up training. The reference model is also quantized to FP8, while the learnable layers are trained in FP32 with BF16 mixed precision. The BF16 tail therefore learns from a slightly degraded FP8 reference, which works against DWD. Our main results and all qualitative results are under BF16. The 8-bit results are not central to the paper, and we can remove them if the reviewer prefers.

W2: Curated ≥720p training data for Wan

We added a baseline that finetunes the native Wan decoder on the same data with the same pixel losses (see our response to Reviewer Yxpz W2).

W2: Inference cost reported only in FLOPs, without wall-clock time or memory

See our response to Reviewer 5V5K (W1/Q1) for the full latency and peak VRAM table. DWD makes end-to-end generation 1.12× faster on 4-step FLUX.2 klein 9B and 1.69× faster on 4-step Wan2.1 1.3B, with 5% and 16% lower peak VRAM.

AuthorsOpenReview ↗

Response to Reviewer P8dt (2/2)

W1: Applicability to few-step and one-step models, including the 4-step Wan 1.3B used as motivation

We added the rCM-distilled 4-step Wan2.1 1.3B model from our introduction. We tune its last 8 of 30 blocks and keep the baseline's exact timesteps:

MethodWarping error (×10⁻³)↓HPSv3↑HPSv2.1↑Subject consist.↑Backgr. consist.↑Temporal flicker.↑Motion smooth.↑Dynamic degree↑Aesthetic quality↑Imaging quality↑
Wan2.1 1.3B (4-step)2.848.320.2650.9380.9620.9760.9790.8250.6600.669
 DWD2.078.460.2580.9460.9590.9840.9830.8110.6570.648
 DWD (self-distillation)2.458.450.2610.9420.9560.9820.9810.8220.6560.631

DWD reduces warping error by 27% and improves temporal flickering. Note that the 1.3B model is an unfavorable setting for DWD. It is much smaller, and tuning only its last 8 of 30 blocks leaves limited learnable capacity. The 14B model would not have this limitation, but we could not run it within the rebuttal period.

DWD does not target one-step models, since they have no trajectory to preserve. For two-step models, we acknowledge that the paired L1/perceptual loss may become ill-posed, but its target can be easily swapped. There are two options:

  • Self-distillation. The regular DWD loss is still applied at small timesteps, and the final step's L1/perceptual target becomes a chain of the reference model's native final step, renoising to the DWD timestep, and the DWD pixel output. This is the self-distillation row above, which also reduces warping error and improves temporal consistency.
  • Distill the native decoder. The final step's L1/perceptual target becomes the reference model's native final step followed by the native decoder. This may not improve quality, but it still cuts inference cost substantially. In a preliminary experiment on a 2-step MiniMax-H3 model, it reduces overall latency from 9.9 s to 6.2 s (1.6× faster).

W1: A one-step model with its decoder would likely be faster

On Wan 1.3B and MiniMax-H3, the decoder is slower than one denoiser step. Also, one-step T2I/T2V generation typically has much lower quality than two-step generation, and a decoder cannot recover the corrupted content.

W2: Comparison with methods that share the same goal (AsymFlow, L2P, FlashDecoder)

We evaluated AsymFlow finetuned from the same FLUX.2 klein 9B Base model on the same data:

MethodFID↓pFID↓OneIG↑LongText↑Training cost (H100)
FLUX.2 klein Base 9B15.1318.690.5560.909--
AsymFLUX.2 klein18.1022.620.5060.7771100 GPU hours (15K iter)
FLUX.2 klein DWD15.0518.380.5560.908350 GPU hours (26K iter)

AsymFlow is substantially weaker, partly due to its sensitivity to training data and partly due to the difficulty of adapting a latent trajectory to a pixel trajectory. DWD matches the base model at about a third of the training cost. L2P is also substantially weaker than its base model (Z-Image Turbo): OneIG 0.499 vs. 0.528, LongText 0.787 vs. 0.917. FlashDecoder is not open source, so we could not compare with it.

W2/Q1: Main goal of the paper

Practical cost reduction is not the main goal. The main goal is to find a minimal working recipe for latent models, and to test whether the same principle transfers to pixel models. We show that a second decoder network is redundant, and that two separate heads on the same backbone make a compact, high-quality design. To our knowledge, no decoding acceleration work makes this point. This understanding is our primary contribution, and cost reduction is a secondary benefit.

W2: Pixel-space results rely mainly on AnisoFlow, which is unpublished, and its text-to-image baseline is much weaker than FLUX.2

AnisoFlow is included in the supplementary material. On ImageNet it gets a much better FID than JiT without REPA (1.55 vs. 1.90). On text-to-image it is stronger than the larger latent i1-1.3B model, and much stronger than MiniT2I of the same size (0.9B):

ModelOneIG↑LongText↑COCO FID↓
i1-1.3B (latent)0.5110.85362.3
MiniT2I0.3560.10632.6
AnisoFlow-L/16-T2I0.5160.86420.1

It is expected to be weaker than FLUX.2 klein 9B, given its smaller size and different training data. DWD also improves JiT on ImageNet (1.90 to 1.85), so the pixel-model gains do not depend on AnisoFlow alone.

We hope our replies have addressed the reviewer's concerns. In particular, DWD keeps the sampler's own final timestep for the pixel and 4-step models, and our new 4-step Wan results use the baseline's exact timesteps. We kindly ask the reviewer to consider raising the score, and we would be glad to know if any concerns remain.

Thank you for the detailed response. The new wall-clock and memory results, the 4-step Wan 1.3B experiment and the finetuned-decoder baseline address several of my concerns, and stating that one-step models are out of scope is helpful. A few points remain.

1. AsymFlow (W2). Your retrained AsymFlow is much worse than the FLUX.2 base (FID 18.10 vs. 15.13, LongText 0.777 vs. 0.909), while the AsymFlow paper reports gains over its latent base. Could you evaluate the official AsymFlow checkpoint if available, or report the metrics used in that paper, so that the comparison does not depend on your retraining?

2. AnisoFlow numbers (W2). The new table lists COCO FID 20.1, OneIG 0.516 and LongText 0.864 for AnisoFlow-L/16-T2I, while Table 2 reports 28.86, 0.528 and 0.850. Could you clarify which evaluation protocol each uses?

3. Main goal (Question). The response says cost reduction is a secondary benefit, but the title and abstract present DWD mainly as removing the decoder to reduce inference cost. I would recommend at least revising the abstract, both in the PDF and on OpenReview, to match this clarified goal.

Reviewer_5V5KOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Decoding Without a Decoder (DWD), which adds a linear pixel prediction head to a pretrained diffusion Transformer and uses it at the final sampling step. The original head continues to drive denoising. The final Transformer blocks are finetuned with pixel-quality losses and trajectory regularization. Experiments span latent and pixel image generators, distilled image generation, and video generation, showing broadly preserved or improved quality while removing the standalone decoder.

Critical strengths

1.A simple architectural insight supported by relevant controls. The shared-head ablations in Table 1 demonstrate that the benefit is not explained solely by adding perceptual losses. Improvements on pixel models, which have no decoder to remove, make the separation of objectives particularly convincing.

2.Broad applicability across substantially different generators. Maintaining FLUX.2 quality without its decoder, improving AnisoFlow’s ImageNet FID from 1.55 to 1.43, and extending the approach to video collectively establish practical relevance beyond one architecture or benchmark.

Critical weaknesses

1.End-to-end efficiency is insufficiently demonstrated. Tables 1–3 report final-step FLOPs, including savings from omitting final-step classifier-free guidance, rather than complete generation latency and peak memory. These savings do not directly establish the practical speedup across different sampling lengths.

2.The video-quality improvement is not fully isolated. Table 3 shows reduced warping error alongside lower dynamic degree and imaging quality. This does not invalidate the improvement, but leaves uncertainty about the preservation of moving detail. The comparison also lacks a decoder-retaining model adapted with comparable data, pixel objectives, and training budget, making the specific contribution of decoder removal difficult to separate from finetuning.

Questions for authors

1.Efficiency clarification: What are the end-to-end latency and peak-memory improvements under matched settings, particularly for four-step FLUX.2 and Wan? Please also reconcile the “275-GFLOP” decoder cost in Section 4.3 with approximately 0.28 PFLOPs in Table 3.

2.Video validation: Does the warping-error advantage persist after controlling for motion magnitude, and against a decoder-retaining baseline finetuned with comparable data and objectives?

2 comments

We thank the reviewer for the positive assessment, and we are glad the paper gave the reviewer a substantially deeper understanding of the problem. We also appreciate the reviewer noting that the shared-head ablations isolate the benefit of the separate head beyond the perceptual losses, that the gains on pixel models (which have no decoder to remove) make the separation of objectives particularly convincing, and that DWD applies broadly across image and video generators.

W1/Q1: End-to-end latency and peak memory, particularly for 4-step FLUX.2 and Wan

This paper focuses on showing that a second decoder network is redundant and that a dual-head design preserves or improves quality. The submitted version does not yet claim significant practical speedups.

To add a practical efficiency study, we report end-to-end latency and peak VRAM for 4-step FLUX.2 klein 9B and 4-step Wan2.1 1.3B (newly added, see our response to Reviewer P8dt). FLUX.2 is evaluated at 1024×1024 and Wan at 81 frames of 480×832, both in eager mode:

ModelLatency (s)↓Peak VRAM (GB)↓
FLUX.2 klein 9B (4-step)0.77637.31
 DWD0.69535.54
Wan2.1 1.3B (4-step)4.66119.22
 DWD2.75716.15

DWD makes end-to-end generation 1.12× faster on FLUX.2 and 1.69× faster on Wan, with 5% and 16% lower peak VRAM, respectively. The Wan timings in our introduction (2.08 s for sampling and 1.15 s for decoding) were measured with torch.compile, so the absolute times differ. We will use the eager timings consistently in the revised paper.

W2/Q2: Lower dynamic degree and imaging quality in Table 3, and preservation of moving detail

Most of DWD's visible improvements are in fact on moving detail. The frames in Fig. 6 come from videos with a moving camera, where the baseline distorts and blurs details while DWD keeps them coherent and sharp. We strongly encourage the reviewer to view the supplementary videos, which show this much more clearly than still frames.

The dynamic degree gap is small (0.736 vs. 0.711), and we see no visible difference in motion between the two models. Our new 4-step Wan 1.3B experiment also gives a motion-matched comparison (see our response to Reviewer P8dt W1). There, the self-distillation variant of DWD has nearly the same dynamic degree as the baseline (0.822 vs. 0.825) and still reduces warping error by 14%.

Imaging quality in Table 3 is effectively the same (0.677 vs. 0.671), and so are HPSv3 (9.06 vs. 9.04) and HPSv2.1 (0.266 for both).

W2/Q2: Decoder-retaining baseline adapted with comparable data, pixel objectives, and training budget

We added a Wan 14B baseline that finetunes the native Wan decoder on the same data with the same pixel losses as DWD:

MethodWarping error (×10⁻³)↓HPSv3↑HPSv2.1↑Subject consist.↑Backgr. consist.↑Temporal flicker.↑Motion smooth.↑Dynamic degree↑Aesthetic quality↑Imaging quality↑
Native decoder1.529.060.2660.9510.9810.9900.9830.7360.6820.677
Finetuned decoder1.399.030.2660.9510.9820.9920.9830.7310.6810.676
DWD1.189.040.2660.9520.9820.9930.9840.7110.6850.671

Finetuning the decoder reduces warping error only from 1.52 to 1.39, while DWD reduces it further to 1.18. The other metrics are effectively tied. This suggests that most of DWD's gain comes from the method rather than from the training data.

We could not match DWD's training budget within the rebuttal period, because a decoder training iteration is about 2× slower than a DWD iteration for Wan2.1 14B (24 s vs. 12.5 s). The decoder's training loss is already flat, since the pretrained decoder is limited by the latent bottleneck rather than by its own training. We therefore report results after 2K iterations.

We also have other evidence that DWD is insensitive to the training data. For example, AsymFlow and DWD both finetune FLUX.2 klein on the same data (with little text): AsymFlow degrades LongText-Bench from 0.909 to 0.777, while DWD keeps it at 0.908 (see our response to Reviewer P8dt W2).

Q1: Reconciling the "275-GFLOP" decoder cost in Sec. 4.3 with 0.28 PFLOPs in Table 3

They are the same number in different units. We will write 0.28 PFLOPs in Sec. 4.3 to match Table 3.

We thank the reviewer again for the support and the thoughtful questions, and we are happy to make further clarifications if needed.

We have uploaded the revised PDF, which incorporates the new results and clarifications from our response (changes marked in blue). Thank you again for your support and the helpful questions.

Reviewer_YxpzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Decoding Without a Decoder (DWD), aiming to replace the VAE decoder in latent diffusion models to reduce inference cost. The main contribution is to decouple the denoising process from final pixel prediction. Specifically, DWD introduces a lightweight linear head that directly predicts pixels from the DiT's last-layer tokens at a nonzero final timestep. This allows the model to be optimized with pixel-quality losses. In addition, the authors introduce trajectory regularization to preserve the pretrained denoising behavior. The results indicate that DWD improves fine details in both image and video generation, with particularly noticeable improvements on the AnisoFlow-L/16-T2I model.

Critical strengths
  1. The proposed method provides a way to directly optimize pretrained diffusion models with pixel-quality losses.
  2. The qualitative results in the supplementary material show noticeable improvements.
  3. The method is clearly and easy to follow.
Critical weaknesses
  1. The improvements appear limited on stronger models such as FLUX.2, which already generate fine details. Does DWD mainly benefit smaller models?
  2. The authors fine-tune Wan2.1 using high-quality videos at least 720p. It is difficult to determine whether the improvements come from DWD or the training data.
  3. I am concerned about how the main contribution is presented. The authors emphasize that a linear head can replace the VAE decoder, but the method also fine-tunes several Transformer blocks, which account for most of the trainable parameters. The linear head seems to serve more as a pixel projection layer and the contribution of the linear head itself is not clearly demonstrated.
Questions for authors

1.See weekness 2. DWD appears to require substantial additional training. If we retained the vae decoder, could it achieve comparable results with less training?

2 comments

We thank the reviewer for noting that DWD lets pretrained diffusion models be optimized directly with pixel-quality losses, that the supplementary videos show visible improvements, and that the method is clear and easy to follow. Below we address the concerns about the scope of the gains and the training data, including a new decoder finetuning baseline.

W1: Limited improvements on stronger models such as FLUX.2. Does DWD mainly benefit smaller models?

No. DWD improves Wan2.1 14B, our largest model, and reduces its warping error by 23% (Table 3). The small gain on FLUX.2 comes mainly from its VAE, which compresses much less than other VAEs (FLUX.2 has 128 latent dimensions per token for a 768-dimensional pixel patch; for comparison, Wan2.1 has 64 latent dimensions per token for a 3072-dimensional pixel patch). Decoding from such a wide latent is easy for the native FLUX.2 decoder, so there is little room to improve. Nevertheless, DWD still removes the decoder there without quality loss. We will clarify this in the revised paper.

W2/Q2: Whether the Wan improvements come from DWD or from the ≥720p training data, and whether the retained VAE decoder could achieve comparable results with less training

We added a new Wan 14B baseline that finetunes the native Wan decoder on the same data with the same pixel losses as DWD. The decoder is already well trained and its training loss stays flat, so we report results after 2K iterations:

MethodWarping error (×10⁻³)↓HPSv3↑HPSv2.1↑Subject consist.↑Backgr. consist.↑Temporal flicker.↑Motion smooth.↑Dynamic degree↑Aesthetic quality↑Imaging quality↑
Native decoder1.529.060.2660.9510.9810.9900.9830.7360.6820.677
Finetuned decoder1.399.030.2660.9510.9820.9920.9830.7310.6810.676
DWD1.189.040.2660.9520.9820.9930.9840.7110.6850.671

Finetuning the decoder reduces warping error only from 1.52 to 1.39, while DWD reduces it further to 1.18. The other metrics are effectively tied. This suggests that most of DWD's gain comes from the method rather than from the training data.

We also have other evidence that DWD is insensitive to the training data. For example, AsymFlow and DWD both finetune FLUX.2 klein on the same data (with little text): AsymFlow degrades LongText-Bench from 0.909 to 0.777, while DWD keeps it at 0.908 (see our response to Reviewer P8dt W2).

W3: Presentation of the main contribution, since the finetuned Transformer blocks hold most of the trainable parameters

The linear head is meant only as a readout layer. Our main idea is that the DiT already encodes pixel structure (Fig. 3), so finetuning the last blocks plus a linear readout is enough for direct pixel prediction. We will revise the writing to make clear that the contribution is the separate pixel output on the DiT, and the linear head is its minimal form.

W3: The contribution of the linear head itself

The contribution of the separate output is shown by the w/o separate head rows in Table 1. Fusing the two heads under the same losses worsens AnisoFlow's ImageNet FID from 1.55 to 1.71, while DWD with a separate head improves it to 1.43. The pixel losses alone therefore do not explain the gain. As for the head's form, a linear head is already the minimal and cheapest choice. Adding convolutional refinement brings no consistent improvement (DWD-conv, Table 1), showing that our tested models need no decoder-like architecture.

Q2: DWD appears to require substantial additional training

DWD finetunes only the last few blocks without full backpropagation, so its training is cheaper than it may appear.

DWD costs much less than adapting a latent model to pixel space. On FLUX.2 klein 9B, AsymFlow takes 1100 H100 GPU hours (15K iterations) and gives worse FID, OneIG and LongText scores (see our response to Reviewer P8dt W2). DWD takes 350 H100 GPU hours with a saturated schedule of 26K iterations, and it matches the native decoder while preserving seed-level content.

Finetuning the native decoder is not cheap either. One Wan decoder training iteration takes 24 s, even after we optimized the decoder code for speed and memory. With the same data and losses, one DWD iteration takes 12.5 s for Wan 14B and 4.4 s for Wan 1.3B (newly added, see our response to Reviewer P8dt).

We hope our replies have addressed the reviewer's concerns. The new decoder baseline separates the effect of the training data, and the shared-head ablation in Table 1 shows the contribution of the separate output. We kindly ask the reviewer to consider raising the score, and we would be glad to know if any concerns remain.

We have uploaded the revised PDF, which incorporates the new results and clarifications from our response (changes marked in blue). We hope the revision addresses your concerns, and we would be glad to answer any further questions.

Comments on the paper

2 comments

We thank all reviewers for their time and constructive feedback. The reviewers found the motivation and intuition sound and the linear probe convincing (P8dt), the method clear with visible improvements in the supplementary videos (Yxpz), and the separation of objectives particularly convincing on pixel models (5V5K). We have responded to each reviewer's comments individually, and below we summarize the main goal of the paper, the new experiments, and the planned revisions.

Main goal of the paper. Our main goal is to find a minimal working recipe for latent models, and to test whether the same principle transfers to pixel models. We show that a second decoder network is redundant, and that two separate heads on the same DiT backbone make a compact, high-quality design. For latent models, DWD removes the decoder without quality loss, and on Wan it also improves quality visibly. The pixel-model results follow from the same principle, and the improvements further strengthen the dual-head design. Currently, cost reduction is a secondary benefit rather than the main goal. Nevertheless, our added 4-step benchmarks show significant speedups and memory savings.

Supplementary videos. We strongly encourage all reviewers to view the supplementary videos. DWD's improvements on Wan are mostly in moving detail and flickering, which are hard to judge from the still frames in the paper.

New experiments.

  • Finetuned decoder baseline (all reviewers). We finetuned the native Wan 14B decoder on the same data with the same pixel losses as DWD. It reduces warping error only from 1.52 to 1.39, while DWD reduces it to 1.18. This suggests that most of DWD's gain comes from the method rather than from the training data. Each decoder training iteration is also about 2× slower than a Wan 14B DWD iteration.
  • 4-step Wan2.1 1.3B (5V5K, P8dt). We applied DWD to the rCM-distilled 4-step model from our introduction, keeping the baseline's exact sampler timesteps. DWD reduces warping error by 27% and improves temporal flickering. We also show a self-distillation variant of the loss for few-step models.
  • End-to-end latency and memory (5V5K, P8dt). DWD makes end-to-end generation 1.12× faster on 4-step FLUX.2 klein 9B and 1.69× faster on 4-step Wan2.1 1.3B, with 5% and 16% lower peak VRAM, respectively (measured in eager mode).
  • Comparison with AsymFlow and L2P (P8dt). Finetuned from the same FLUX.2 klein 9B Base model on the same data, AsymFlow is worse on FID, pFID, OneIG and LongText at about 3× the training cost of DWD, while DWD matches the base model. L2P is also substantially weaker than its base model Z-Image Turbo.

Paper revision. We will add these results to the paper and upload the revised PDF later in the discussion period. We will also make the clarifications promised in our responses:

  • The main goal of the paper, with cost reduction as a secondary benefit (P8dt)
  • The linear head as a minimal readout, with the separate pixel output on the DiT as the contribution (Yxpz)
  • The low compression of the FLUX.2 VAE, which explains the small gain on FLUX.2 (Yxpz)
  • A discussion of LTX-Video and PiD, and the architectural distinction from them (P8dt)
  • The Fig. 4 caption and the skipped tiny timesteps in Wan's pixel loss (P8dt)
  • The decoder cost in Sec. 4.3, written in PFLOPs to match Table 3 (5V5K)

We have uploaded the revised PDF, with all changes marked in blue. It includes the new experiments and the clarifications promised in our responses.

New experiments

  • Finetuned-decoder baseline on Wan2.1 14B: Table 3 (page 9) and Appendix B.1 (page 18)
  • 4-step Wan2.1 1.3B, including a self-distillation variant: Appendix B.2 (Table 5, page 18)
  • End-to-end latency and peak memory: Appendix B.3 (Table 6, page 19)
  • Comparison with AsymFlow and L2P: Appendix B.4 (Table 7, page 19), with qualitative text-rendering comparisons in Fig. 7 (page 20)

Clarifications

  • Abstract (page 1): the summary sentence now focuses on quality, and the OpenReview abstract is updated accordingly
  • Introduction (page 2): the first contribution states that a latent DiT needs no decoder, and the Wan timings use eager measurements
  • Related work (page 2): LTX-Video and PiD
  • Sec. 3.2 (page 4): the linear head as the minimal form of the separate pixel output
  • Fig. 4 caption (page 7)
  • Sec. 4.2 (page 8): the low compression of the FLUX.2 VAE
  • Sec. 4.3 (page 9): decoder cost in PFLOPs
  • Appendix A.2 and A.3 (page 17): the FP8 reference model, the Wan final timestep before shifting, and the skipped tiny timesteps in Wan's pixel loss

We thank all reviewers again for their feedback, which has improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.