acceptodds
Withdrawn from ICLR 2027

Recasting DiT for Extreme Image Compression with Hierarchical Visual Priors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pretrained diffusion models provide powerful generative priors for extreme image compression, but their stochastic, iterative generation processes are not designed to decode compressed representations directly. We present PrismCodec, which recasts a pretrained Diffusion Transformer (DiT) as a bitstream-grounded one-step generative decoder. PrismCodec uses the decoded representation in two complementary roles: a projected codec state defines the starting state for generation, while hierarchical visual priors derived from the same decoded latent provide semantic and structural guidance. To bridge the gap between stochastic, iterative generation and deterministic one-step decoding, we develop a progressive adaptation strategy. We first learn codec-conditioned transport from a noisy neighborhood centered on the codec state, distill it into a one-step mapping, and finally anneal the source perturbation to zero during end-to-end optimization. The resulting decoder reconstructs deterministically from the received representation without additional conditioning bits. Across Kodak, CLIC, and DIV2K, PrismCodec consistently improves rate–perception performance over prior generative codecs, achieving BD-rate reductions of 70–75% in LPIPS and 75–81% in DISTS over StableCodec. Code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image compression, generative compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1GpFOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PrismCodec, a one-step flow-based extreme image compression method, that adapts an unknown Lightning latent space and DINOv3, by using 2-stage cascaded decoding and one-step distilation. Specifically, after some unknown encoding, they generate from the encoded bitstream some latent state Z and an auxiliary temporary decoded image. The latent state Z is the input to the flow-based distilled decoder, which is conditioned on 2 things: 1. The bistream-decoded image encoded into the unknown generative VAE latent space 2. DINOv3 features of this bitsream-decoded image. Together, the lightningDiT runs a one-step (distilled) generation pipeline which reconstructs the final image.

Critical strengths
  • The results seem very appealing with strong performance gains both in distortion and in perception.
  • The paper checks both distortion metrics (MSE, SSIM), perceptual distortion metrics (LPIPS, DISTS), and perception metrics (FID), which I appreciate, considering the perception-distortion-rate tradeoff (Blau & Michaeli, 2019). This is also done on multiple benchmakrs (Kodak, CLIC, and DIV2K), which I again appreciate.
Critical weaknesses
  • The paper suffers from unclear writing, with vague, missing reproducibility information, imprecise phrasing, and confusing presentation order.
    • First, the paper motivates its core contribution on a somewhat false premise. The first line in the abstract claims “Pretrained diffusion models provide powerful generative priors for extreme image compression, but their stochastic, iterative generation processes are not designed to decode compressed representations directly”, and the intro (L58-59) says : “but their generative processes are not naturally aligned with compression decoding”. However, the generative process of stochastic DDMs is naturally aligned with compression. In the seminal Ho et al. 2020 they demonstrated how diffusion models are natural lossy compressors. This line of works was continued in their use as zero-shot codecs by DiffC (Theis et al. 2022, Vonderfecht & Liu 2025), and DDCM (Ohayon et al. 2025, Vaisman et al. 2026). While not one-step, some do so at incredible speed (Vonderfecht & Liu 2025, Vaisman et al. 2026). Both latter approaches also used Flux.1 for compression, which is a DiT. OSCAR even explicitly demonstrates how the diffusion noisy process can be thought of as an analogy to compression, where higher timesteps correspond to a degradation of more noise. While the authors list some of these methods in App. A, and even DiffC appears in Tab1 - they are not mentioned in the related work section of the main part of the paper, not even compression part of the seminal DDPM paper or DiffC. Instead, the “Generative Image compression” paragraph stops at works from 2023. Therefore, the story presented throughout the main paper neglects these works exist, and how the current work is situated relative to them. An afterthought of an appendix mention doesn’t do the field justice, and it doesn’t allow for incorrect statements in the abstract or the intro to allow for a nicer, easier narrative. Instead, it undermines the presentation of the work.
    • The paper repeatedly mentions a “frozen generative VAE encoder/decoder”(e.g., L136, L175, L204), but nowhere in the paper or appendix I could find what this frozen, pre-trained generative VAE encoder is. Section 4 only mentions LightningDiT was pre-trained on ImageNet-1K. This is especially crucial since the authors claim in the reproducibility statement everything is present for reproducing the experiments. Importantly, this raises questions regarding the contribution: are the results better due to the method itself, or just because they used a modern high-capacity VAE (e.g. Flux.2 VAE) against baselines which operate on legacy SD1.5 VAES (such as StableCodec, AEIC, or DiffC). Flux.1’s VAE is known to have a very high PSNR relative to the earlier SD1-2 VAEs (see e.g. Turbo-DDCM). The SD1.5 VAE is capped to 512x512, when high-res images are compared to (and PrismCodec even trains on high resolution images (L315)). However, for examples, DiffC has a Flux.1 variant – which operates on 1024x1024 iamges. This again seems like an unfair advantage.
    • Nowhere in the paper the encoder is described. Specifically, it is missing from both fig 2+3, and the bitstream is ‘given’, and from all the paper and appendices. No architectural information or anything are known apart from eq.8 that have a rate – which should affect the encoding.
    • the rate is mentioned in eq. (8) hasn’t been described for how it’s controlled for. This just appear in eq 13. The entire core part of the method should be present in the main part of the paper, and not delegated to the appendix.
  • The authors don’t provide any explanation as to why their method is so effective (L412, L425). This is suspicious as the bitstream encoding nor the used frozen generative VAEs were described. With such a high reported margin of improvement, a better ablation is necessary. This is related to Table 4: Even when removing major parts of the algorithm, it seems the improvements are still very large (-54/58% w/o distillation, -37/46% without the zero-centered noise). How can that be? Can you add to tab. 4 variants where 2 components are removed (e.g. both semantic and structural priors, to isolate the performance on zcz_c) - so we can see what actual stage had the most effect? If in fact the generative latent VAE is much newer than the SD latent VAEs, then a fairer comparison to baselines would be to have a CodecPrism variant which uses the same VAE. Indeed, currently table 4 shows that removing the 3 stages retains a high improvement, which suggests the performance gain is heavily driven by the raw capacity of the backbone.
    • Note that PrismCodec has 1.9B parameters (Tab2), which is nearly x2 the other methods. Decoding is also therefore way longer than some of the compared approaches (I’m unsure why DiffC isn’t given here?). Also, the VRAM is neatly hidden in the appendix, where it says that the inference requires latent tiling to prevent OOM on an A100, which is a problem not found in previous approaches, even for high VRAM methods such as Turbo-DDCM. Considering image compression is a very practical problem, with real-world applications on consumer hardware, this is worth mentioning in the main part of the paper.
  • All results with BD-rate percentages (including Fig1 and Tab1) report the percentages improvements relative to StableCodec. Why? StableCodec is an older baselines which is outperformed by other baselines compared to in this paper. For example, OneDC achieves -56% on CLIC on LPIPS according to table 1, but is missing from Fig1 to where PrismCodec achieves -71%. Anchoring to StableCodec inflates the performance gains, and seems to be misleading on purpose. The % should be taken from the best method.
  • The paper would benefit from comparison to different approaches, which are more distortion oriented, such as BPG (for a classical comparison), ILLM, CRDR-D. DiffC should also be compared to in Tab 2 – when considering the time taken to decode, as a zero-shot approach. You can compare to other zero-shot sota perceptual approaches such as Turbo-DDCM to complement this.
  • The paper would benefit from having more visual examples in the appendix, and to have zoomed-out images as well. This will also help to see if the tiling introduces artifacts.

Mentioned works:

  • Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." Advances in neural information processing systems 33 (2020): 6840-6851.
  • Theis, Lucas, Tim Salimans, Matthew D. Hoffman, and Fabian Mentzer. "Lossy compression with gaussian diffusion." arXiv preprint arXiv:2206.08889 (2022).
  • Vonderfecht, Jeremy, Liu, Feng. "Lossy compression with pretrained diffusion models." In International Conference on Learning Representations, vol. 2025, pp. 687-702. 2025.
  • Guo, Jinpei, Yifei Ji, Zheng Chen, Kai Liu, Min Liu, Wang Rao, Wenbo Li, Yong Guo, and Yulun Zhang. "Oscar: One-step diffusion codec across multiple bit-rates." Advances in Neural Information Processing Systems 38 (2026): 85267-85286.
  • Careil, Marlene, Matthew J. Muckley, Jakob Verbeek, and Stéphane Lathuilière. "Towards image compression with perfect realism at ultra-low bitrates." In International conference on learning representations, vol. 2024, pp. 15544-15564. 2024.
  • Ohayon, Guy, Hila Manor, Tomer Michaeli, and Michael Elad. "Compressed image generation with denoising diffusion codebook models." In Proceedings of the 42nd International Conference on Machine Learning, vol. 267, pp. 47044-47089. 2025.
  • Vaisman, Amit, Guy Ohayon, Hila Manor, Miki Elad, and Tomer Michaeli. "Turbo-ddcm: Fast and flexible zero-shot diffusion-based image compression." In International Conference on Learning Representations, vol. 2026, pp. 151475-151520. 2026.
Questions for authors

Most of my questions are basically the weaknesses.

  • What’s the generative VAE backbone? What’s the encoding like?
  • Why do the BD-rate % compared to stablecodec?
  • How’s the VRAM usage on competing approaches?
  • Why is FID missing for Kodak in tab1? You can use smaller patches than 256x256 to have enough patches like previous works did (E.g. Turbo-DDCM).
Reviewer_ZpjcOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposed to decompose bitstream evidence into two roles: generative starting point and vision guidance, and transform stochastic iterative generation into deterministic single-step decoding via progressive adaptation. The key technology contains conditional flow adaptation, distillation, and noise annealing. Extensive experiments on the benchmark datasets demonstrate that PrismCodec outperforms SOTA methods in terms of rate-perception tradeoff. Moreover, the ablation experiments align with the three contributions outlined in the manuscript, and the logical flow is clear.

However, the current version still exhibits significant gaps in terms of theoretical guarantees for deterministic decoding, fair comparison with SOTA methods, robustness of the evidence image, consistency of high-resolution tiling, as well as reproducibility. If the authors can adequately supplement the aforementioned experiments and analyses in their rebuttal, the paper could reach an acceptable level.

Critical strengths

1、 The motivation of this work is clear: how to adapt the pretrained diffusion models to generative image compression. 2、 The design of the proposed method is hierarchical. 3、 The experimental scope and metrics are relatively comprehensive. 4、 The manuscript is well organized and well written.

Critical weaknesses

1、The paper combines several existing ideas: codec-centered source (similar to PMRF's posterior-mean centering), multi-depth DINOv3 features (similar to REPA-style alignment), VAE structural prior, and AYF-EMD distillation. The novelty lies primarily in the combination and the progressive adaptation schedule. The paper would benefit from a clearer articulation of what is fundamentally new beyond assembling known techniques. 2、While StableCodec is used as the BD-rate reference, the paper does not provide a detailed architectural or training comparison. StableCodec also uses a codec latent and structural residual with fixed text conditioning. An ablation replacing PrismCodec's conditioning with StableCodec's (e.g., fixed text + residual) would strengthen the claim that the proposed decomposition is necessary. 3、The "evidence image" x~ is lightly supervised but its quality is not analyzed. The paper supervises x~ with MSE + DINOv3 feature similarity (Eq. 15), but does not report how good x~ is as a reconstruction, nor how its quality affects final performance. Since x~ is the interface for both semantic and structural priors, its fidelity is critical. An ablation varying λev-sem or showing x~ visualizations would be informative. Additionally, we doubt that the method is similar to SPRDiff [1] in that it uses intermediate images to provide guidance, as it appears to do. [1]. Wei H, Zhou Y, Ge C, et al. Exploiting Semantic and Pixel Representations for Ultra-Low Bitrate Image Compression[J]. arXiv preprint arXiv:2606.01608, 2026. 4、The deployed decoder returns a single reconstruction per bitstream. For compression, this is desirable for reproducibility, but the paper does not analyze whether the deterministic mapping captures the full conditional posterior or whether diversity is sacrificed. The theoretical analysis in Appendix C.6 explicitly states it does not bound reconstruction error to the source image, which is honest but leaves a gap in understanding the method's theoretical guarantees. 5、Fig. 4 shows PrismCodec's PSNR and MS-SSIM curves are comparable to or slightly better than baselines, but the paper's main claims are about perceptual metrics. For a compression method, it would be useful to discuss the rate-distortion-perception tradeoff more explicitly—e.g., whether PrismCodec can be tuned to prioritize PSNR at the expense of LPIPS, and how the five λrateIII values affect this tradeoff. 6、The paper states both are frozen, but they still add inference cost. Table 2 reports total FLOPs, but it would be helpful to break down the contribution of DINOv3 vs. the DiT vs. the codec. Since DINOv3 is a large vision transformer, its inference cost may be non-trivial, especially for high-resolution images (resized to 512×512). 7、For high-resolution images (e.g., 2K), this means the semantic prior is computed at a lower resolution than the structural prior (which uses the VAE latent at full resolution). The paper does not discuss how this resolution mismatch affects performance or whether adaptive resolution would help. 8、The block structure is described in Appendix C.3, but the main text would benefit from a clearer diagram or pseudocode. 9、How are the BD-rates computed? Are they averaged over the five points? What is the reference curve? The appendix could provide more details. 10、The paper does not discuss failure cases. Are there image types where PrismCodec performs poorly? A brief discussion would be valuable. 11、How much does tiling affect latency and quality compared to full-resolution inference?

Questions for authors

See the weaknesses part.

Reviewer_SJXBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PrismCodec adapts a pretrained DiT into a deterministic one-step image decoder. The compressed representation supplies a codec state and an auxiliary evidence image, from which frozen DINOv3 and VAE encoders extract multi-depth semantic features and a structural condition. Training proceeds through conditional flow adaptation, one-step distillation, and end-to-end optimization with noise annealing. Experiments on Kodak, CLIC2020, and DIV2K report substantial perceptual bitrate savings.

Critical strengths
  1. The conditioning architecture is well motivated. The codec state provides a direct baseline for latent residual prediction, the VAE condition supplies a structural reference in the generator's native latent space, and DINO features add semantic context. Table 3 supports using both conditioning branches, while all inputs are derived from the received bitstream without additional conditioning bits.
  2. Adapting the teacher to compressed evidence before distillation is a sensible choice. Freezing the codec and condition-producing modules during Stage II keeps the compression interface fixed while the student learns a one-step map; Stage III then jointly optimizes the deployed reconstruction objective. The overall system achieves substantial perceptual gains, including LPIPS/DISTS BD-rate savings of 69.54%/74.64% on DIV2K relative to StableCodec.
Critical weaknesses
  1. The motivation for the specific depth assignment remains incomplete. DINO and DiT serve different purposes, so progressively deeper DINO features do not automatically correspond to the needs of later DiT blocks. Table 3 shows that multi-depth features are useful, but does not isolate the value of the proposed ordering over shared multi-depth conditioning or simple feature fusion.
  2. The benefit of gradual noise removal is not isolated by the current ablation. The proposed schedule is intended to ease the transition to deterministic decoding, but the no-annealing variant also retains noise at inference. Consequently, the reported gap combines the effect of the training transition with that of noisy deployment inputs, leaving the advantage over immediate deterministic fine-tuning unclear.
Questions for authors
  1. Under the same conditioning budget, how does the proposed depth assignment compare with reversed ordering or multi-depth feature fusion?
  2. Starting from the same Stage-II checkpoint, how does setting source noise to zero immediately in Stage III compare with gradual annealing, with other settings matched?
  3. If the omitted 20k Stage-II updates are instead allocated to direct one-step fine-tuning in the no-distillation variant, does the performance gap remain?
  4. Since SODEC also derives visual guidance from an auxiliary reconstruction (Appendix A.6), can you provide a comparison under common evaluation settings to clarify the gains from PrismCodec's new conditions and training procedure?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Clarifications

We sincerely thank the reviewers for their constructive feedback. We would like to clarify several key points regarding the technical contributions, experimental interpretation, and evaluation methodology. We will incorporate additional experiments and implementation details in a future revised version.



1. Relationship to Prior Diffusion-Based Compression Methods

We acknowledge that diffusion models have long been studied for lossy compression, including DDPM, DiffC, DDCM, and Turbo-DDCM. Our original wording was broader than intended, and we will clarify this relationship.

Importantly, deterministic decoding is a common property of compression systems, including those built upon stochastic generative models. DDCM and Turbo-DDCM achieve reproducible reconstruction through transmitted selections from shared noise codebooks, while DiffC also provides a deterministic probability-flow reconstruction option. Thus, stochastic diffusion formulations do not necessarily imply stochastic decoding from a fixed bitstream.

PrismCodec does not claim deterministic decoding itself as a contribution, as reproducible reconstruction from a fixed bitstream is already well established in lossy compression, including diffusion-based codecs. The distinction lies in how deterministic decoding is achieved. While DiffC provides deterministic probability-flow reconstruction and DDCM-based methods reproduce denoising trajectories using transmitted codebook selections, PrismCodec directly reconstructs from an entropy-decoded learned representation through one DiT forward pass. Our contribution is the bitstream-grounded decoding architecture and the progressive adaptation of a pretrained DiT to this one-step reconstruction process.

We will add those discussions to the revised paper.



2. Technical Contributions

PrismCodec's contribution lies in adapting a pretrained DiT for one-step reconstruction from compressed representations through three connected designs:

  1. Bitstream-grounded initialization: The decoded representation provides a codec state that directly initializes generation.
  2. Hierarchical visual priors: An evidence image derived from the same representation supplies multi-depth DINOv3 semantic guidance and a structural condition in the generator's native VAE latent space, without additional conditioning bits.
  3. Progressive adaptation: Codec-conditioned flow learning, one-step flow-map distillation, and source-noise annealing progressively adapt the pretrained DiT to deterministic reconstruction from the codec state.

We do not claim that DINO features, VAE conditioning, or flow distillation are individually novel. The contribution is their coordinated use to adapt a pretrained generative prior into a bitstream-grounded one-step decoder.



3. Clarification of the Ablation Studies

We appreciate the suggestions for more comprehensive ablations and will include additional controlled experiments in a future revised version to better isolate the contributions of hierarchical feature assignment, source-noise annealing, and flow-map distillation under matched training budgets.



4. Implementation Details and Computational Costs

We agree that further implementation details would improve reproducibility. We will provide more comprehensive descriptions of the codec architecture, entropy coding, pretrained VAE configuration, and encoding/decoding procedures. We will also clarify parameter accounting, computational costs and GPU memory measurements.



5. Why Kodak FID Is Not Reported

Kodak contains only 24 images, making FID estimation statistically unreliable under our 256 × 256 patch-based evaluation protocol. As noted by HiFiC, this protocol yields only 192 patches on Kodak, which is insufficient for reliable distributional evaluation. MS-ILLM reach the same conclusion. Recent methods, including StableCodec, DiT-IC, OneDC and AEIC, also omit Kodak FID/KID for this reason.

We observe that some papers use smaller patches, such as 64 × 64, to increase the sample count. However, standard FID implementations extract Inception-v3 features from inputs resized to 299 × 299. Upsampling 64 × 64 crops by approximately 4.7 times changes the spatial scale and may substantially affect the extracted features without recovering the missing spatial context. Moreover, patches extracted from the same 24 images remain correlated, so increasing the number of patches does not resolve the limited diversity of the source dataset.

We do not claim that Kodak FID cannot be computed. Rather, we consider its statistical reliability and interpretability insufficient under our evaluation protocol. We therefore follow established practice by reporting paired distortion and perceptual metrics on Kodak and evaluating FID on larger datasets such as CLIC2020 and DIV2K.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.