acceptodds
Withdrawn from ICLR 2027

Unified Image Generation and Compression with Hierarchical Progressive Context Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learned image compression methods typically transform images into the latent domain and model the probability distributions of latents for efficient entropy coding. Image generation models, particularly latent-space generative frameworks, rely on learned probability distributions for latent sampling. More accurate latent distribution modeling can therefore benefit both tasks. However, existing studies, including current generative image compression methods, have rarely explored this synergistic property. To address this research gap, we propose Uni-HPCM, a unified compression–generation framework that uses shared conditional latent distributions for both entropy coding and sampling. Built on the advanced Hierarchical Progressive Context Modeling (HPCM) model, Uni-HPCM is deliberately trained to jointly optimize the rate–distortion cost and perceptual quality. After training, the single unified model supports three versatile paradigms: reconstructive compression via full main-latent coding, pure image generation via full latent sampling, and generative compression across a spectrum of coding–sampling combinations. Experiments demonstrate competitive compression performance, favorable generation quality, and promising results in generative image compression. In addition, a larger model enhances both rate–distortion performance and generation quality, highlighting their potential synergy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Learned Image Compression, Unified Compression and Generation, Hierarchical Progressive Context Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LUcbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Uni-HPCM, which extends HPCM by sharing conditional latent distributions between entropy coding and sampling. Its hybrid training combines a rate–distortion objective for fully coded reconstructions with perceptual supervision for partially sampled reconstructions. The framework supports reconstructive and generative compression by varying the proportion of coded and sampled latents, and enables class-conditional generation using hyper-latents extracted from DiT-generated images. Experiments demonstrate substantial improvements over direct sampling from HPCM while retaining competitive compression performance.

Critical strengths

1.The framework extends a single codec to support reconstructive and generative compression through flexible coding–sampling combinations, without increasing the parameter count or computation of fully coded compression. This offers a practical way to add generative capabilities to an existing compression model.

2.HCST substantially improves sampling over both direct HPCM sampling and matched-budget RD-only fine-tuning (Table 2). These results support the effectiveness of adapting the context model and decoder to trajectories containing sampled latents.

Critical weaknesses

1.The value of the complete class-conditional generation pipeline remains insufficiently established. CCLS first uses DiT to generate a complete image and then extracts its hyper-latent for resampling. Although this dependency is disclosed and Table 9 shows improvements over the chosen teacher configuration, Table 1 reports codec-only costs. Comparisons at matched end-to-end computation are needed to establish the benefit of this additional generation stage.

2.The evidence for compression–generation synergy is limited. Table 4 shows modestly worse overall rate–distortion performance than HPCM-Base, while improvements from increasing model capacity do not isolate the benefit of joint training. Moreover, Table 2 fixes λ\lambda and kk without reporting actual bitrates. Since HCST also updates the hyperprior, the sampling gains may partly reflect richer conditioning information, leaving the contribution of improved shared distribution modeling unclear.

Questions for authors

1.What are the end-to-end latency and computation of CCLS, including DiT and hyper-latent extraction? Does its improvement over teacher-only generation persist at matched total computation and across reasonable teacher guidance settings?

2.What are the hyper-latent and coded main-latent bitrates for the variants in Table 2? Do the quality improvements remain at matched actual bitrate, particularly when all main-latent subsets are sampled?

Reviewer_LE6cOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Uni-HPCM, a method for (generative) image compression and class-conditional image generation, and allows for a single model to perform multi-bitrate compression. The idea is to reuse the latent distribution learned for entropy coding also for image generation. If you want a distortion oriented compression, you can transmit all latents in the hierarchy, whereas you can transmit some of them when you want generative compression.

The underlying idea is not novel, but its application to HPCM is a different implementation.

Critical strengths

It seems that the proposed method is quite efficient, time wise and parameter wise.

Critical weaknesses
  • My main concern is the novelty of the paper. The authors emphasize throughout the paper and abstract how they examine whether you can use the same learned space for both generation and compression, how this synergy is important and unexamined, etc. However, this idea has already been explored in the past by the likes of DiffC (Theis et al. 2022) and even the seminal DDPM paper (Ho et al. NeurIPS 2020) - both uncited. Specifically, DiffC shows how a pretrained diffusion model can progressively perform lossy compression at different BPPs by transimiting different amount of latent codes and continuing with generation from it (either DDPM or DDIM). This idea is just like the progressive mechanism proposed here. DDCM (Ohayon et al. ICML 2025), which IS cited by the authors, is merely regarded as a method with codebooks, but it also demonstrated similarly to DiffC that a single generative model can support both compression and generation. Additionally, they specifically show conditional image generation using their approach, yet the authors do not mention this at all. Furthermore, Turbo-DDCM (Vaisman et al. ICLR 2026) further show you can transmit the latent codes of DDCM halfway and continue with DDIM as well). It is widely known that DDPM is kind of hierarchical/progressive by nature, as earlier timesteps correspond to lower frequency of the images. The authors also should discuss ARPC (Zhang et al. ICLR 2026), which also learns how to submit progressively compressed latents depending on the required bitrate. As a result, it seems that the paper primarly adapts existing ideas to a new architecture of HPCM rather than proposing some new idea as the authors claim: the idea of transmitting earlier subsets of some hierarchical latent representation and sampling the unsubmitted ones is not new by itself. Also, learning a dist for entropy coding and sampling from a learned distribution in a generative manner is not by itself new (like p(x) in DDPMs).
    • The ‘hybrid coding sampling tranining’ losses aren’t novel as well, since it’s well established in the literature to combined perceptual, rate, and reconstruction objectives (e.g. CRDR, Iwai et al. WACV 2024).
  • Fig. 1 shows that when fewer latents are submitted and more are generated, then PSNR decreases and LPIPS increases - meaning worse distortion and worse perceptual distortion. While this is expected for lower BPPs, and although you can see improvement over the baseline HPCM - it's not clear that there’s any perceptual improvement/stays the same numerically. Why not include FID to show that as you generate more the distribution becomes more realistic as more latents are sampled? You also do not include the actual BPP for each reconstruction so it’s imposible to compare the rate in which the distortion degrades as we change the BPP.
  • The paper lacks in evaluation.
    • For image compression there’s only ONE qualitative figure in the entire submission (incl appendices) - and it’s the Fig 1 which shows a SINGLE compressed image against only the HPCM base. All the other qualitative figures (6 and 9) only show results for class-conditoinal generation. This is insufficient for an image compression paper, especially for one making claims about perceptual quality - and even more when no actual perceptual metric reported on the compression (see below). The paper should include multiple image reconstructions, at multiple bitrates matched against competing baselines.
    • The paper repeatedly consider LPIPS and DISTS as both perceptual distortion or perceptual quality metrics. However, both are full-reference perceptual distortion metrics that measure similarlity to the original ground truth image - just in a different space (compared to PSNR which is in pixel space). This is very different from perception as defined in the perception-distortion-rate tradeoff paper – where it’s a DISTRIBUTIONAL metric. Thus, the authors should include FID or KID, as is very common in papers in the field (e.g. PerCO, DiffC, DDCM, etc…) Otherwise, this might simply be a distortion-based method. This is true also for the quantitative figures (8/5). The authors claim to improve perceptual compression - yet without any perceptual metrics for compression, this remains to be demonstrated.
    • The compression comparisons are missing important benchmarks, such as the methods states above – and even StableCodec which is cited yet uncompared to. These are methods that directly address flexibly bitrate control, perceptual fidelity, and efficient compression. Additionally, the authors claim (L375) that their method covers a wider bitrate range than all compared methods. However, this might be simply due to the choice of what models and checkpoints the authors used for comparison. Specifically, the official MS-ILLM checkpoints go all the way from 0.002 to 0.9 BPP (code), yet the comparison includes only a small subset of this range that ends in 0.1 bpp. Why are the higher BPP checkpoints missing? CRDR also reaches up to 0.8BPP. Why are the main generative compression comparisons limited to Kodak in Fig5, while CLIC is used just for distortion-oriented compression in Fig4? Kodak is a small dataset of only 24 images, and comparing most methods only on it is not sufficient. If resolution is an issue for the other baselines, then it’s common even to downsample CLIC. Also, DIV2K is another relevant large dataset that can help compare more methods.
    • Why does Tab 4 in the appendix suddenly includes comparisons to GLIC and DCAE but they aren’t included in the quantitative compression comparisons in the main figure? Their BD rate is better than the proposed method.
    • Tab 4 also shows that the best BD-rates are actually achieved by the baseline HPCM-Large or GLIC - and not by the proposed method. As no perception metric was computed for compression, this undermines the claims of overall improved performance over the baseline. Indeed, Uni-HPCM-Base actually performs the second worst out of all compared methods, outperforming only TCM (which appears in Fig4). Here especially it is very important to compare against other perceptually-focused methods like PerCO , DiffC, and DDCM, if the authors aim to claim that their method is better perceptually (so that we can see if other perceptual methods are also worse on the BD-rate)
    • Why do HPCM base and UNI HPCM results on CLIC (fig4) seem almost identical, where on the smaller Kodak we see difference?
    • The runtime experiment in table 3 uses different GPUs for each method, according to L465 (3090 for uin-HPCM - and whatever the other methods used are plugged in as is), and maybe even different evaluation settings completely. This is not a valid controlled wall-clock runtime and speed comparison. Also, it isn’t reported which configuration of each bitrate is used in each model, compred to Uni-HPCM and the resulting BPP. How does the runtime change when the BPP change for Uni-HPCM and for competing methods? It might be that other methods are timed on stronger, longer compression (or vice versa).
  • The authors claim that (eg last line in the abstract) increasing model capacity improves both rate-distortion performance and generation quality – thus this highlight their synergy. Does it? Increasing the capacity can help both tasks separately, simply because each has more parameters available for optimization. It doesn’t necessarly mean that the two mutually benefit from joint optimization.
  • I’m unsure of the significance of class conditional generation. First, class conditional generation was a stepping stone towards text conditional models, which are prevalent today. More importantly however, it seems to me that the class conditional generation is not standalone? From my understanding, you first use a pretrained class-conditional model (here DiT-XL) to generate an image, extract its hyper latents, and then sample using the main latents of UNI HPCM? Thus, class conditional requires running a different class-conditional model beforehand, making the procedure somewhat circular? Is it not then just class-conditional refinement and not generation?
    • This is also relevant to table 1 - where the sampling steps and parameter counts are marked with a dagger, indicating they only refer to the codec. This in my opinion is not clear enough that there’s a heavy additional cost of the pre-trained DiT (only in app F there’s a clarification it requires another 675M params and additional 25 steps).If I understood correctly, this puts in question the fairness of all the class-conditional image generation results, since all compared models actually generated the images themselves rather than relying on a previous pre-trained model.
  • Some of the paper’s citations are also incorrect or missing.
    • The Kodak24 dataset is credited to the Kodak company without a date. While the original images attributions is Indeed for Eastman Kodak – the commonly used Kodak24 dataset was collected and widely attributed to Rich Franzen 1999.
    • The pro subset of CLIC2020 is cited as some generic entry (“Challenge on Learned Image Compression (CLIC). Clic professional validation set, 2020. Dataset used in the Challenge on Learned Image Compression.”) and should be cited properly “Toderici, G., Shi, W., Timofte, R., Theis, L., Balle, J., Agustsson, E., Johnston, N., and Mentzer, F. Workshop and challenge on learned image compression (CLIC2020), 2020. URL http://www.compression.cc. “
    • The Tecnick dataset is not cited at all, nor are the papers introducing LPIPS, FID, or DISTS. Proper attribution is needed.

Minor:

  • I am unsure of the necessity of the discussion section. It largely repeats how increased capacity improves performances (which I already address above). The limitations could be combined into the conclusions.
Questions for authors
  • The submission doesn’t include a reproducibility statement - and while not necessary by ICLR guidelines, I’m concerned regarding code release. Are you planning to release the code? This is especially important as many implementation details are missing from the paper (e.g. configurations of the two-stage adversarial procedure).
Reviewer_suTnOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Uni-HPCM, which reuses the conditional latent distributions of the HPCM entropy model for both entropy coding and latent sampling. Hybrid Coding–Sampling Training (HCST) optimizes rate–distortion on fully coded reconstructions and perceptual quality on mixed coded and sampled ones. At inference, a single model supports standard compression and generative compression with adjustable bitrate. It also supports class-conditional generation, which samples all subsets conditioned on hyper-latents extracted from DiT-XL/2 images. Experiments show slightly lower rate–distortion performance than HPCM-Base, competitive perceptual quality in generative compression, and a reasonable ImageNet-256 FID.

Critical strengths
  • The idea is simple and intuitive. Sampling directly from the codec’s own conditional Gaussians along HPCM’s coarse-to-fine schedule is a natural way to obtain variable-rate generative decoding. It adds no parameters or extra decoder cost.
  • The experiments show that the proposed method makes the entropy model usable as a sampler. Uni-HPCM reduces LPIPS in generative compression and FID in class-conditional generation compared with untrained HPCM sampling. At the same time, it largely preserves the rate–distortion performance of the original codec.
Critical weaknesses
  • I have concerns about the CCLS setting. CCLS essentially uses images generated by DiT-XL/2 as the conditioning information: it extracts the hyper-latent from each DiT image and resamples the main latents from it. As a result, the class semantics and global structure of the outputs largely come from DiT rather than from the codec, as can be seen in Figs. 6 and 9, where the generated images are near-copies of the conditioning images. Table 8 points the same way: FID degrades from 11.76 to 25.57 as λ decreases. This is consistent with FID tracking how much of the teacher image survives, rather than with any generative capability of the codec.
  • The claim of “synergy” is not well supported. The experimental results suggest a trade-off between generation and compression rather than a synergy: Uni-HPCM gains sampling ability at the cost of rate–distortion performance, as shown in Table 4.
Questions for authors
  • How much of the output is inherited from the teacher image? Please report the bitrate of ẑ^tea and the LPIPS/PSNR between each CCLS sample and its DiT conditioning image. If the samples remain very close to the conditioning images, in what sense is CCLS generation by the codec rather than a resampled reconstruction of DiT outputs?
  • The DiT-XL/2 performance in Table 1 (FID 2.27, 250 steps) differs substantially from the DiT images actually used as CCLS conditioning (FID 16.19 in Table 9).
    • Why was this non-standard configuration chosen as the teacher, and why is it not stated alongside Table 1?
    • What is the performance of CCLS when conditioned on DiT-XL/2 images generated under the configuration reported in Table 1? If CCLS does not improve over a teacher at its standard operating point, the gain over the current teacher may reflect added diversity to an over-guided sampler rather than the generative ability of the codec.
  • HPCM-Base coding is MSE-optimized, so outperforming it on LPIPS/DISTS is expected. A natural ablation is HPCM fine-tuned with the same L1 + LPIPS + GAN objective on fully coded reconstructions across λ.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.