acceptodds
Under review as a conference paper at ICLR 2027

Learned Image Compression with Vector Quantization-Based Prior Modeling

Abstract

Entropy modeling is essential for learned image compression (LIC), where the latents are usually scalar-quantized and subsequently encoded based on estimated priors. Compared to scalar quantization (SQ), vector quantization (VQ) is theoretically more efficient for compression performance. However, the codebook size in VQ grows exponentially with the bitrate, resulting in an extremely high-dimensional optimization landscape. Such a complex landscape poses significant challenges to convergence and often leads to severe codebook collapse, making it difficult to achieve effective rate-distortion (RD) optimization. To make VQ tractable within an end-to-end framework, we propose Vector Quantization-Based Prior Modeling (VQPM). VQPM applies VQ to a subset of the latents with the lowest spatial resolution. These low-resolution features inherently capture the image's global structure and are coded first, without any decoded context. VQ is particularly suitable here as it can model the dependencies within the vector itself, compensating for the lack of external context. The vector-quantized latents then serve as global structural guidance for the subsequent hyperprior and contextual prior modeling. We further introduce an RD-oriented codebook training strategy that optimizes the codebook using end-to-end RD gradients. Experimental results show that VQPM achieves state-of-the-art RD performance, reducing PSNR BD-rate over VTM-22.0 by 20.29%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Learned Image Compression, Context Modeling, Vector Quantization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qWqEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VQPM, which employs vector quantization (VQ) not as a quantizer for the entire latent representation, but as a prior for entropy modeling in learned image compression. Within the hierarchical context model (HPCM), VQ is applied only to a small subset of the low-resolution latents that are coded first and therefore have limited contextual information available. The resulting VQ representation then serves as global structural information for the hyperprior and the subsequent context modeling. The paper further proposes a training strategy that decomposes each codeword into an EMA-updated centroid and an offset optimized with the rate–distortion (RD) loss, so that codebook learning better reflects the compression objective. Experiments on Kodak, CLIC Pro Val, and Tecnick report BD-rate reductions of 20.29%, 19.94%, and 23.95%, respectively, over VTM-22.0, slightly outperforming prior state-of-the-art methods while maintaining relatively low computational complexity.

Critical strengths
  • The main strength of the paper is that it achieves a meaningful performance improvement over an already strong baseline with a relatively simple idea. Applying VQ only to a small subset of the first-coded latents, where contextual information is most limited, is a well-motivated way to exploit VQ as a prior while avoiding the optimization difficulties of quantizing the entire latent representation. Conditioning the entropy model of the hyper-latent on the VQ-reconstructed latents, which turns a fixed factorized prior into a content-adaptive prior, can also be regarded as a novel component.
  • On the same HPCM-Large backbone, VQPM yields approximately 2% BD-rate improvement with only about a 1.5–1.8% increase in computation and 0.11M additional parameters (0.12–0.14%). This is a non-negligible gain in the relatively mature field of MSE-oriented learned image compression. The method also transfers to a Channel-AR entropy model without additional modifications, yielding a 1.47% improvement.
  • The ablation studies are reasonably comprehensive, covering the number of groups to which VQ is applied, a comparison with an SQ-based prior, bit decomposition at low and high rates (the cost of the VQ indices versus the bits saved), codebook size and training hyperparameters, codebook utilization, and per-image codebook usage patterns.
  • The evaluation is transparent. Rates are measured from actual arithmetic-coded bitstreams, end-to-end codec latency including entropy coding is reported, and comparisons with practical codecs such as DCVC-RT-Intra and JPEG-AI are presented, including results that are unfavorable to the proposed method.
Critical weaknesses
  1. Insufficient comparison for the proposed VQ training strategy. Several recent approaches improve VQ training, including the rotation trick (Fifty et al.), DiVeQ (Vali et al.), and SimVQ (Zhu et al.). However, the proposed RD-oriented VQ training strategy is compared only against standard VQ training, so it remains unclear whether it offers an advantage over these more recent techniques. A controlled comparison against recent VQ training methods, rather than only standard VQ, is needed to substantiate the claimed effectiveness of the proposed strategy.

  2. Source of the complexity advantage. The claim in Fig. 1(a) and Table 1 that the proposed model outperforms HPCM-Large with lower computational complexity is enabled by externally adopted efficiency modules (PCBlock and PATConv), whereas VQPM itself increases computation by approximately 1.5–1.8%. This distinction is explained only in Appendix D.4. The main paper should clearly report the computational overhead attributable specifically to VQPM.

  3. Limited reporting of the cleanest comparison. The HPCM-Large + VQPM result, which most cleanly isolates the contribution of VQPM itself (−1.99% BD-rate), is reported only on Kodak and only in the appendix. Direct BD-rate comparisons against HPCM-Large should also be provided on CLIC Pro Val and Tecnick, as well as under MS-SSIM.

  4. Rate accounting. Although this is a broader issue in the field, the reported gains are relatively modest, which makes precise and consistent rate accounting particularly important. The paper should clearly state whether the reported rates are computed from the total file size including headers or only from the coded stream payload, and whether the rates of the competing methods were obtained under a consistent protocol.

Minor

  • In Fig. 3, the color assigned to each model should be consistent across the three RD plots. In particular, HPCM-Large should use a color that is clearly distinguishable from that of the proposed model.
Questions for authors
  1. Can the authors compare the proposed RD-oriented VQ training strategy against recent VQ training methods, such as the rotation trick, DiVeQ, and SimVQ, by replacing the VQ module of VQPM with each of them while keeping the rest of the architecture, the RD loss, the codebook size, and the token dimension unchanged?
  2. Can the authors report in the main paper the computational overhead (kMACs/pixel, parameters, and runtime) attributable specifically to VQPM, separately from that of the adopted efficiency modules (PCBlock and PATConv)?
  3. Can the authors provide direct BD-rate comparisons of HPCM-Large + VQPM against HPCM-Large on CLIC Pro Val and Tecnick, as well as under MS-SSIM, and include these results in the main paper?
3 comments

Thank you for identifying the same-backbone comparison as the clearest way to assess VQPM, and for specifying the controls you would like to see for codebook training.

Q1. How does our training strategy compare with Rotation Trick, DiVeQ, and SimVQ?

We compare Rotation Trick, DiVeQ, and SimVQ with the training strategies in Table 4, using the same codec for all methods. In fact, We have explored these methods while developing our RD-oriented VQ training. Rotation Trick gives little improvement, and DiVeQ, a differentiable VQ method, provides only a modest gain. SimVQ performs worse in our framework, possibly because its parameterization restricts codebook optimization. Our RD-oriented training achieves the best RD performance, codebook usage, and perplexity among the compared methods. We will include this comparison in the paper.

VQ training methodBD-rateCodebook usagePerplexity
Standard VQ-VAE (Table 4)0.00%78.24%524.00
EMA without RD offset (Table 4)−1.13%90.62%581.32
Full RD-oriented training (Table 4)−1.86%97.95%756.54
Rotation Trick-0.12%80.22%531.98
DiVeQ-0.66%88.28%576.00
SimVQ1.19%60.55%528.69

Q2. What computational overhead does VQPM add on a fixed backbone?

The lower computational complexity in Fig. 1(a) and Table 1 comes from the backbone modifications, including PCBlock and PATConv. Table 5 in the revised manuscript separately evaluates VQPM by comparing each backbone with and without the module. On both the original HPCM-Large and efficient HPCM backbones, VQPM adds approximately 19 kMACs/pixel (1.5–1.8%) and 0.11M parameters.

We further measure end-to-end encoding and decoding times with and without VQPM on each backbone using the same RTX 3090, including entropy coding. As shown in the table below, VQPM adds 4–5 ms to both encoding and decoding. We will include these comparisons in the paper to explicitly report VQPM’s computation, parameter, and runtime overhead separately from the efficiency gains of the backbone modifications.

BackboneEncoding without VQPM (ms)Encoding with VQPM (ms)Decoding without VQPM (ms)Decoding with VQPM (ms)
Original HPCM-Large119124122127
Efficient HPCM111115109113

Q3. Direct comparison of HPCM-Large with and without VQPM.

We directly compare HPCM-Large with and without VQPM while keeping the original backbone unchanged. As shown in the table below, VQPM consistently reduces BD-rate on Kodak, CLIC Pro Val, and Tecnick under both PSNR and MS-SSIM. These results extend the Kodak PSNR comparison in Section 4.3 (Table 5) of the revised manuscript and confirm that VQPM provides rate–distortion gains independently of the backbone modifications. We will include these results in the paper.

BD-rate of HPCM-Large + VQPM relative to HPCM-LargeKodakCLIC Pro ValTecnick
PSNR−1.99%-1.78%-1.72%
MS-SSIM-2.05%-2.04%-1.95%

Weakness 4. File size, headers, and comparison protocol.

We compute VQPM’s bitrate from the complete encoded file for each image. Specifically, we report 8B/(H×W)8B/(H \times W) bpp, where BB is the file size in bytes and H×WH \times W is the number of pixels in the original image. The byte count includes the file header and the arithmetic-coded streams for the VQ indices, the hyperlatent, and the remaining main latents.

The rates reported for the competing methods in Table 1 follow the same convention. For each method, the byte count includes header bytes, and the denominator is the original image’s pixel count. We will clarify in Section 4.1 that this full-file bpp definition applies to both VQPM and the competing methods in Table 1.

Minor comment. Colors in Fig. 3.

We will use the same color and line style for each method across all three panels and give VQPM and HPCM-Large clearly distinct curves.

Thank you for suggesting that we report the same-backbone comparisons in the main paper. We have expanded Table 5 in Section 4.3 to compare the original HPCM-Large and efficient HPCM backbones, each with and without VQPM. The table now reports PSNR and MS-SSIM BD-rates on Kodak, CLIC Pro Val, and Tecnick, together with end-to-end encoding and decoding times.

We reproduce Table 5 below for ease of reference. All BD-rates are relative to the original HPCM-Large without VQPM. Encoding and decoding times were measured on the same RTX 3090 and include entropy coding. As shown in the table, adding VQPM improves RD performance on both backbones across all three datasets under both metrics. VQPM adds about 4~5 ms to both encoding and decoding on both backbones.

ModelPSNR KodakPSNR CLIC Pro ValPSNR TecnickMS-SSIM KodakMS-SSIM CLIC Pro ValMS-SSIM TecnickkMACs/pixelParams (M)Enc./Dec. (ms)
HPCM-Large0.00%0.00%0.00%0.00%0.00%0.00%1261.2989.71119/122
HPCM-Large + VQPM−1.99%−1.78%−1.72%−2.05%−2.04%−1.95%1280.2789.82124/127
Current Efficient HPCM+0.99%+1.04%+0.92%+1.15%+1.19%+1.09%1064.8179.31111/109
Current Efficient HPCM + VQPM−1.18%−1.56%−1.74%−1.45%−1.59%−1.78%1083.6779.42115/113

We have also revised the manuscript in response to your other comments. Appendix D.6 now compares our codebook training strategy with Rotation Trick, DiVeQ, and SimVQ using the same codec for all methods. Section 4.1 clarifies that bitrate is computed from the complete encoded file, including headers and coded streams. In Fig. 3, HPCM-Large and VQPM are now shown in blue and red, respectively, across all three panels.

Your comments helped us present the contribution of VQPM more clearly. We hope our earlier point-by-point response and these revisions address the questions you raised. If your schedule permits, we would be grateful if you could take a look at the revised comparison. We would be happy to answer any remaining questions before the discussion closes. Thank you again for your time and careful review!

Reviewer_qWqEOpenReview ↗

Response to authors

Thank you for your detailed response. I will maintain my original positive rating.

Reviewer_T2LCOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a vector quantization-based prior modeling method (VQPM). VQPM applies VQ to a subset of latent variables with the lowest spatial resolution, such as the first-coded low-resolution latents in HPCM, the first spatial position of every non-overlapping 2×2 block within the first group of Channel Autoregressive methods. The vector-quantized latents then serve as global content guidance for the subsequent hyperprior and contextual prior modeling. Through an RD-oriented codebook training strategy, the scheme achieves 2.08% and 1.47% bitrate reduction compared with HPCM and the original Channel AR LIC, respectively.

Critical strengths
  1. Reasonable motivation: For Autogressive entropy modeling methods, the first slice often does not have a strong conditional prior to assist modeling. Therefore, using VQ instead of SQ can more fully exploit VQ’s ability to better exploit inter-dimensional dependencies and benefit from the space-filling properties of vector quantization cells.
  2. Detailed method introduction: The paper describes its method and implementation details in detail, which helps understanding and reproducibility.
  3. Extensive experiments and visualizations: The paper conducts many experiments and visualizations to demonstrate the effectiveness of the scheme.
Critical weaknesses

We understand that the contribution emphasized throughout the paper is proposing to use VQ to replace the first part in autogressive entropy modeling, and to make full use of this VQ-quantized latent part to further improve the rate-distortion performance-complexity tradeoff of existing schemes such as HPCM.

  1. Performance: We understand that the proposed method replaces the subset of the first part of Autogressive entropy modeling with VQ. Although it makes as much use as possible of this VQ-quantized global information to help generate the hyperprior and contextual prior modeling, the performance gain it brings is still relatively low. Meanwhile, regarding complexity: the main experiment improves a lightweight baseline based on HPCM, and the main experiment ultimately achieves both a small rate-distortion performance gain and reduced complexity. However, in Appendices D.4 and E, when experiments are conducted on existing schemes HPCM and Channel-AR LIC, the rate-distortion performance gain decreases and complexity increases. These results weaken the paper’s contribution and the effectiveness of the method.
  2. Analysis: Although the paper does a lot of visualization and analysis on the VQ codebook, we are curious whether directly comparing the visualization or effectiveness analysis of the first-part latent, subsequent hyperprior, and contextual prior when using VQ versus SQ, respectively, would better demonstrate the VQ benefits emphasized in this paper.
Questions for authors

We mainly focus on clearer contribution explanations, please refer to the weaknesses for details.

2 comments

Thank you for raising the distinction between the performance of the complete codec and the contribution of VQPM on an existing backbone.

Weakness 1. RD gain and complexity on the original backbones.

Thank you for this comment. The presentation in our previous manuscript may not have made the comparison across backbones more clear. We would like to clarify that VQPM achieves notable and comparable RD improvements on both the original HPCM-Large and the current efficient backbone. These results were reported in Table 2 and Appendix D.4 (Table 6) of the original manuscript. We have now moved the backbone comparison from Appendix D.4 to Section 4.3 (Table 5) of the revised manuscript to make VQPM’s contribution on each backbone explicit in the main text.

Specifically, VQPM achieves RD gains of 1.99% on the original HPCM-Large and 2.08% on the efficient backbone on Kodak. We present the two comparisons separately below, with each backbone without VQPM serving as its own BD-Rate anchor.

VariantBD-Rate ↓kMACs/pixelParameters
HPCM-Large0.00%1261.2989.71M
HPCM-Large + VQPM−1.99%1280.2789.82M
VariantBD-Rate ↓kMACs/pixelParameters
Current efficient backbone0.00%1064.8179.31M
Current efficient backbone + VQPM−2.08%1083.6779.42M

The benefit lies in making better use of the structural information carried by the first-coded latents. At this early coding stage, decoded context is unavailable, whereas VQ can exploit dependencies within each latent vector to form a compact representation of the image’s global structure. This representation then benefits the coding of the remaining latents. It enables the hyperprior to adapt to individual image content and provides structural guidance for contextual probability estimation, particularly when decoded context is still limited. A small VQ-coded subset can therefore improve entropy modeling beyond the subset itself, while keeping VQ optimization tractable.

Building on this entropy-modeling benefit, our framework uses VQPM to improve hyperprior and contextual prior modeling, alongside backbone modifications that improve computational efficiency. VQPM introduces a small computational overhead because VQCPM reuses the existing context model and VQHPM applies lightweight MLP blocks to a single globally pooled feature vector per image. The Channel-AR result in Table 6 further supports consistent RD improvements across architectures with unchanged VQ hyperparameters.

Weakness 2. How do SQ and VQ differ in the first latent group and in subsequent hyperprior and context modeling?

Thank you for this comment. SQ quantizes each latent coefficient separately, whereas VQ jointly quantizes a vector through a learned codebook. This allows VQ to exploit intra-vector dependencies in the first latent group, where no previously decoded context is available. The decoded vectors then provide global structural guidance for subsequent hyperprior and contextual prior modeling. VQ thus compensates for the lack of decoded context at the earliest coding stage and provides a more effective compact structural prior than SQ.

As shown in Table 2, compared with the baseline, the SQ-coded prior degrades RD performance on Kodak by +1.01%, whereas Full VQPM improves RD performance by −2.08%. To further examine this difference, we visualize the first-coded latent energy and subsequent bitrate distributions on kodim05, kodim08, and kodim18 (the kodim08 comparison is presented in Section 4.4 (Fig. 5) of the revised manuscript, with the kodim05 and kodim18 comparisons provided in Appendix J (Figs. 16 and 17)) at comparable PSNR. We compute the energy at each spatial location, and use normalized energy maps to show how the first-group features are distributed across the image. We also visualize the bitrate maps of the hyperlatent z^\hat{z} and the remaining main latents y^SQ\hat{y}_{\mathrm{SQ}}, together with their bitrate differences.

For the first latent group, the normalized energy maps show that VQ produces a more widely distributed energy pattern than SQ, consistent with its role as global structural guidance. For hyperprior modeling, conditioning on the VQ-coded group reduces the coded bitrate of z^\hat{z}. For subsequent context modeling, Full VQPM also reduces the bitrate of the remaining latents y^SQ\hat{y}_{\mathrm{SQ}}. The bitrate difference maps show that VQPM allocates more bits to some textured regions while reducing the bitrate in other regions, yielding a lower overall rate. These comparisons show how the first-group representation affects subsequent hyperprior and contextual prior modeling and support the RD improvement reported in Table 2.

Reviewer_T2LCOpenReview ↗

Response to Authors

Thanks for the author's response. The author's further analysis of VQ and SQ has solved my problem. In addition, regarding performance concerns, the authors have further clarified that although I consider VQPM's performance includes both lightweight and performance enhancing operations, this does not affect the contribution of this paper. Therefore, I will improve my score.

Reviewer_Q6vEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Vector Quantization-Based Prior Modeling (VQPM) for learned image compression. The main motivation is that the first-coded latent subset in progressive entropy models lacks decoded context, while directly applying vector quantization (VQ) to the full latent space is difficult due to the rapidly growing codebook optimization space. To address this, the paper applies VQ only to a compact, low-resolution, first-coded latent subset and uses the resulting discrete representation as prior information for subsequent entropy modeling. Experiments on three datasets show strong rate-distortion performance.

Critical strengths

The motivation is sensible and the design is well targeted to a real weakness of progressive entropy models. Instead of applying VQ to the entire latent representation, the paper restricts VQ to a small first-coded, low-resolution subset where decoded context is unavailable. This is a reasonable way to exploit the joint modeling ability of VQ while keeping the codebook optimization problem tractable. The proposed use of this subset to condition both the hyperprior and the subsequent contextual entropy model is technically coherent.

Critical weaknesses

The novelty positioning and comparison to closely related/recent work are currently incomplete. The high-level motivation of first transmitting a compact structural representation and then using it to improve subsequent entropy modeling appears closely related to prior work on self-generated/global priors for learned image compression, e.g. Deeply-Conditioned Image Compression via Self-Generated Priors. The specific realization here using a compact VQ-coded subset may still be novel, but the current paper does not sufficiently distinguish which part of the contribution is new: the use of an early structural prior itself, or specifically the VQ-based construction of that prior.

Similarly, the proposed “RD-oriented” VQ optimization should be positioned more carefully relative to recent rate-distortion-aware VQ methods such as RDVQ. In the current formulation, the hard VQ assignments are fixed during back-propagation, and the VQ index-rate term updates the categorical model rather than directly differentiating through the discrete assignment. I would therefore like a clearer explanation of how the proposed training strategy differs conceptually from recent entropy/rate-aware VQ optimization methods.

The SOTA comparison also appears incomplete. In particular, DHIC-Regu (ICLR 2026) is a highly relevant learned image compression baseline using the same standard datasets and VTM-based evaluation protocol, and it is not included in the comparison. This omission matters because the current tables may give the impression that VQPM is uniformly best across the evaluated datasets, while a recent strong method is competitive and reportedly slightly better on Tecnick. Given the submission timing, I believe such recent pre-July 2026 methods should be included or at least discussed carefully.

Questions for authors
  1. Can the authors clarify the novelty of VQPM relative to prior methods that first encode a compact/self-generated structural representation and use it to condition subsequent entropy modeling?In particular, how does the proposed motivation and mechanism differ from Deeply-Conditioned Image Compression via Self-Generated Priors? Is the main novelty intended to be the idea of an early structural prior, or specifically the use of a selectively VQ-coded low-resolution subset as that prior?

  2. Could the authors provide a comparison with recent strong LIC methods that are currently missing, especially DHIC-Regu? Since DHIC-Regu is an ICLR 2026 method evaluated on the same standard datasets with a closely comparable VTM-based protocol, including it would help establish whether VQPM is indeed consistently state-of-the-art. I would be particularly interested in the comparison on Tecnick.

  3. Could the authors provide a VQ-only control, without VQCPM or VQHPM? Currently, the transition from the baseline to “VQCPM only” simultaneously changes the quantization method for the early subset and introduces VQ-based conditioning. Therefore, it is difficult to determine how much of the gain comes from VQ itself versus its use as a prior. A controlled SQ/VQ × no-conditioning/conditioning comparison would materially affect my assessment of the claimed contribution.

3 comments

Thank you for recognizing the motivation for quantizing the first group and for asking us to distinguish the benefit of VQ from the benefit of using that group as a prior.

Q1. What is new relative to self-generated structural priors?

Some existing methods extract and encode separate representations as global or structural priors. For example, DCIC-sgp (Zhao et al.) extracts a structural prior from the input image to condition the transforms and entropy model, while Informer (Kim et al.) derives a global hyperprior from the main latent for entropy modeling.

Unlike these approaches, our novelty lies in vector-quantizing the first-coded, low-resolution group of the main latent to exploit inter-channel dependencies without decoded context. The decoded codewords are used for reconstruction and provide global structural guidance for content-adaptive hyperprior modeling and context modeling of the remaining scalar-quantized latents (Fig. 2). This design does not require extracting and coding a separate global or structural prior. We will clarify this distinction in the main text.

Weakness. How does the RD-oriented update differ from RDVQ?

RDVQ (Jiang et al.) uses a differentiable soft assignment to estimate the VQ index rate during training, allowing the rate gradient to update the encoder. It retains hard assignments for reconstruction and entropy coding.

Unlike RDVQ, our method introduces a learnable RD-oriented offset for each codeword, ek=μk+δke_k=\mu_k+\delta_k, where μk\mu_k tracks the assigned features by EMA. During backpropagation, we keep the nearest-neighbor assignment kk fixed: the VQ index rate updates the categorical index model, while δk\delta_k receives gradients from reconstruction distortion and from the rates of the hyperlatent and remaining main latents, which use the decoded codewords as an entropy prior (Eq. 8 and Algorithm 1). This enables the codewords to balance reconstruction quality with their usefulness as priors for downstream coding, improving the overall rate–distortion trade-off. We will revise the main text to clarify this distinction.

Q2. Comparison with DHIC-Regu, particularly on Tecnick.

The table below reports PSNR BD-Rate results from Table 1 of each paper, with VTM-22.0 as the common anchor. Compared with DHIC-Regu, VQPM achieves lower BD-Rates on Kodak and CLIC Pro Valid and a better average across the three datasets. DHIC-Regu achieves a slightly lower BD-Rate on Tecnick. We will add DHIC-Regu to Table 1 and discuss these results in the main text.

MethodKodakCLIC Pro ValTecnickAverage
DHIC-Regu−19.73%−18.13%−24.09%−20.65%
VQPM−20.29%−19.94%−23.95%−21.39%

Q3. VQ-only control.

In our framework, disabling VQCPM means that the progressive context model (PCM) cannot use the first latent group, even though the decoder has already recovered it. When PCM predicts the first batch of SQ latents, its main-latent input is normally [0,y^vq][0,\hat y_{\mathrm{vq}}] (Eq. 5). Masking the VQ group changes this input to [0,0][0,0]. The hyperprior remains available, but PCM has no decoded main-latent values to guide this first prediction. At later steps, PCM can use previously decoded SQ values, while the first group remains masked. Losing access to this group may therefore hurt rate–distortion performance.

We conduct a VQ-only control experiment with VQCPM and VQHPM disabled. We encode the first latent group with VQ and use the quantized latents for image reconstruction. VQ-only increases Kodak BD-rate by 2.32% relative to the baseline. We will report the VQ-only result and discuss this comparison in the revised manuscript.

ConfigurationPCM uses first groupHyperlatent entropy model uses first groupKodak BD-rate vs. baseline
VQ-onlyNoNo2.32%
VQCPM onlyYesNo−1.29%

Thank you for your careful review and helpful comments on structural priors and VQ training. In our earlier response, we explained how our proposed RD-oriented codebook update differs from RDVQ’s differentiable soft assignment. We now compare the two strategies experimentally using the same codec. For the RDVQ-style baseline, we use soft assignment to estimate the first-group index rate during training. This allows the index-rate gradient to update the encoder. We retain hard assignments for reconstruction and entropy coding. All variants use the same codec, codebook size, and token dimension.

The table below reports PSNR BD-rate on Kodak, with standard VQ-VAE as the anchor. Our RD-oriented training achieves a BD-rate of −1.86%, compared with -1.17% for the RDVQ-style variant. In our method, the learned offset receives gradients from both reconstruction distortion and the rates of the hyperlatent and remaining main latents. This allows the codewords to be optimized for their use in both reconstruction and subsequent entropy modeling.

VQ training methodBD-rateCodebook usagePerplexity
Standard VQ-VAE (Table 4)0.00%78.24%524.00
EMA without RD offset (Table 4)−1.13%90.62%581.32
Full RD-oriented VQ (Table 4)−1.86%97.95%756.54
RDVQ-1.17%90.82%612.09

We also appreciate your suggestion to discuss DCIC-sgp. DCIC-sgp obtains this information by encoding a separate structural prior, which also conditions its analysis and synthesis transforms. The authors report that their full model takes 2.6–3.2 times as long to encode/decode as TCM. VQPM reuses the first-coded latent group for both reconstruction and entropy modeling, adding only several ms to decoding on our model. We have also added DHIC-Regu (ICLR 2026) to Table 1 of the revised manuscript for comparison across all three datasets. VQPM achieves lower BD-Rates on Kodak and CLIC Pro Valid and a better average across the three datasets. DHIC-Regu achieves a slightly lower BD-Rate on Tecnick.

Your comments helped us clarify the contribution of the first-coded group and strengthen the experimental comparisons. We hope our earlier point-by-point response and these revisions address the questions you raised. If you have a chance to look over the additional results before the discussion closes, we would be happy to answer any remaining questions. Thank you again for your time and for reviewing our work!

Thank you for the detailed responses and additional experiments. Most of my concerns have now been addressed. I will maintain my original positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.