- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes VQPM, which employs vector quantization (VQ) not as a quantizer for the entire latent representation, but as a prior for entropy modeling in learned image compression. Within the hierarchical context model (HPCM), VQ is applied only to a small subset of the low-resolution latents that are coded first and therefore have limited contextual information available. The resulting VQ representation then serves as global structural information for the hyperprior and the subsequent context modeling. The paper further proposes a training strategy that decomposes each codeword into an EMA-updated centroid and an offset optimized with the rate–distortion (RD) loss, so that codebook learning better reflects the compression objective. Experiments on Kodak, CLIC Pro Val, and Tecnick report BD-rate reductions of 20.29%, 19.94%, and 23.95%, respectively, over VTM-22.0, slightly outperforming prior state-of-the-art methods while maintaining relatively low computational complexity.
Critical strengths
- The main strength of the paper is that it achieves a meaningful performance improvement over an already strong baseline with a relatively simple idea. Applying VQ only to a small subset of the first-coded latents, where contextual information is most limited, is a well-motivated way to exploit VQ as a prior while avoiding the optimization difficulties of quantizing the entire latent representation. Conditioning the entropy model of the hyper-latent on the VQ-reconstructed latents, which turns a fixed factorized prior into a content-adaptive prior, can also be regarded as a novel component.
- On the same HPCM-Large backbone, VQPM yields approximately 2% BD-rate improvement with only about a 1.5–1.8% increase in computation and 0.11M additional parameters (0.12–0.14%). This is a non-negligible gain in the relatively mature field of MSE-oriented learned image compression. The method also transfers to a Channel-AR entropy model without additional modifications, yielding a 1.47% improvement.
- The ablation studies are reasonably comprehensive, covering the number of groups to which VQ is applied, a comparison with an SQ-based prior, bit decomposition at low and high rates (the cost of the VQ indices versus the bits saved), codebook size and training hyperparameters, codebook utilization, and per-image codebook usage patterns.
- The evaluation is transparent. Rates are measured from actual arithmetic-coded bitstreams, end-to-end codec latency including entropy coding is reported, and comparisons with practical codecs such as DCVC-RT-Intra and JPEG-AI are presented, including results that are unfavorable to the proposed method.
Critical weaknesses
-
Insufficient comparison for the proposed VQ training strategy. Several recent approaches improve VQ training, including the rotation trick (Fifty et al.), DiVeQ (Vali et al.), and SimVQ (Zhu et al.). However, the proposed RD-oriented VQ training strategy is compared only against standard VQ training, so it remains unclear whether it offers an advantage over these more recent techniques. A controlled comparison against recent VQ training methods, rather than only standard VQ, is needed to substantiate the claimed effectiveness of the proposed strategy.
-
Source of the complexity advantage. The claim in Fig. 1(a) and Table 1 that the proposed model outperforms HPCM-Large with lower computational complexity is enabled by externally adopted efficiency modules (PCBlock and PATConv), whereas VQPM itself increases computation by approximately 1.5–1.8%. This distinction is explained only in Appendix D.4. The main paper should clearly report the computational overhead attributable specifically to VQPM.
-
Limited reporting of the cleanest comparison. The HPCM-Large + VQPM result, which most cleanly isolates the contribution of VQPM itself (−1.99% BD-rate), is reported only on Kodak and only in the appendix. Direct BD-rate comparisons against HPCM-Large should also be provided on CLIC Pro Val and Tecnick, as well as under MS-SSIM.
-
Rate accounting. Although this is a broader issue in the field, the reported gains are relatively modest, which makes precise and consistent rate accounting particularly important. The paper should clearly state whether the reported rates are computed from the total file size including headers or only from the coded stream payload, and whether the rates of the competing methods were obtained under a consistent protocol.
Minor
- In Fig. 3, the color assigned to each model should be consistent across the three RD plots. In particular, HPCM-Large should use a color that is clearly distinguishable from that of the proposed model.
Questions for authors
- Can the authors compare the proposed RD-oriented VQ training strategy against recent VQ training methods, such as the rotation trick, DiVeQ, and SimVQ, by replacing the VQ module of VQPM with each of them while keeping the rest of the architecture, the RD loss, the codebook size, and the token dimension unchanged?
- Can the authors report in the main paper the computational overhead (kMACs/pixel, parameters, and runtime) attributable specifically to VQPM, separately from that of the adopted efficiency modules (PCBlock and PATConv)?
- Can the authors provide direct BD-rate comparisons of HPCM-Large + VQPM against HPCM-Large on CLIC Pro Val and Tecnick, as well as under MS-SSIM, and include these results in the main paper?
3 comments
Thank you for suggesting that we report the same-backbone comparisons in the main paper. We have expanded Table 5 in Section 4.3 to compare the original HPCM-Large and efficient HPCM backbones, each with and without VQPM. The table now reports PSNR and MS-SSIM BD-rates on Kodak, CLIC Pro Val, and Tecnick, together with end-to-end encoding and decoding times.
We reproduce Table 5 below for ease of reference. All BD-rates are relative to the original HPCM-Large without VQPM. Encoding and decoding times were measured on the same RTX 3090 and include entropy coding. As shown in the table, adding VQPM improves RD performance on both backbones across all three datasets under both metrics. VQPM adds about 4~5 ms to both encoding and decoding on both backbones.
| Model | PSNR Kodak | PSNR CLIC Pro Val | PSNR Tecnick | MS-SSIM Kodak | MS-SSIM CLIC Pro Val | MS-SSIM Tecnick | kMACs/pixel | Params (M) | Enc./Dec. (ms) |
|---|---|---|---|---|---|---|---|---|---|
| HPCM-Large | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 1261.29 | 89.71 | 119/122 |
| HPCM-Large + VQPM | −1.99% | −1.78% | −1.72% | −2.05% | −2.04% | −1.95% | 1280.27 | 89.82 | 124/127 |
| Current Efficient HPCM | +0.99% | +1.04% | +0.92% | +1.15% | +1.19% | +1.09% | 1064.81 | 79.31 | 111/109 |
| Current Efficient HPCM + VQPM | −1.18% | −1.56% | −1.74% | −1.45% | −1.59% | −1.78% | 1083.67 | 79.42 | 115/113 |
We have also revised the manuscript in response to your other comments. Appendix D.6 now compares our codebook training strategy with Rotation Trick, DiVeQ, and SimVQ using the same codec for all methods. Section 4.1 clarifies that bitrate is computed from the complete encoded file, including headers and coded streams. In Fig. 3, HPCM-Large and VQPM are now shown in blue and red, respectively, across all three panels.
Your comments helped us present the contribution of VQPM more clearly. We hope our earlier point-by-point response and these revisions address the questions you raised. If your schedule permits, we would be grateful if you could take a look at the revised comparison. We would be happy to answer any remaining questions before the discussion closes. Thank you again for your time and careful review!
Response to authors
Thank you for your detailed response. I will maintain my original positive rating.
Thank you for identifying the same-backbone comparison as the clearest way to assess VQPM, and for specifying the controls you would like to see for codebook training.
Q1. How does our training strategy compare with Rotation Trick, DiVeQ, and SimVQ?
We compare Rotation Trick, DiVeQ, and SimVQ with the training strategies in Table 4, using the same codec for all methods. In fact, We have explored these methods while developing our RD-oriented VQ training. Rotation Trick gives little improvement, and DiVeQ, a differentiable VQ method, provides only a modest gain. SimVQ performs worse in our framework, possibly because its parameterization restricts codebook optimization. Our RD-oriented training achieves the best RD performance, codebook usage, and perplexity among the compared methods. We will include this comparison in the paper.
Q2. What computational overhead does VQPM add on a fixed backbone?
The lower computational complexity in Fig. 1(a) and Table 1 comes from the backbone modifications, including PCBlock and PATConv. Table 5 in the revised manuscript separately evaluates VQPM by comparing each backbone with and without the module. On both the original HPCM-Large and efficient HPCM backbones, VQPM adds approximately 19 kMACs/pixel (1.5–1.8%) and 0.11M parameters.
We further measure end-to-end encoding and decoding times with and without VQPM on each backbone using the same RTX 3090, including entropy coding. As shown in the table below, VQPM adds 4–5 ms to both encoding and decoding. We will include these comparisons in the paper to explicitly report VQPM’s computation, parameter, and runtime overhead separately from the efficiency gains of the backbone modifications.
Q3. Direct comparison of HPCM-Large with and without VQPM.
We directly compare HPCM-Large with and without VQPM while keeping the original backbone unchanged. As shown in the table below, VQPM consistently reduces BD-rate on Kodak, CLIC Pro Val, and Tecnick under both PSNR and MS-SSIM. These results extend the Kodak PSNR comparison in Section 4.3 (Table 5) of the revised manuscript and confirm that VQPM provides rate–distortion gains independently of the backbone modifications. We will include these results in the paper.
Weakness 4. File size, headers, and comparison protocol.
We compute VQPM’s bitrate from the complete encoded file for each image. Specifically, we report 8B/(H×W) bpp, where B is the file size in bytes and H×W is the number of pixels in the original image. The byte count includes the file header and the arithmetic-coded streams for the VQ indices, the hyperlatent, and the remaining main latents.
The rates reported for the competing methods in Table 1 follow the same convention. For each method, the byte count includes header bytes, and the denominator is the original image’s pixel count. We will clarify in Section 4.1 that this full-file bpp definition applies to both VQPM and the competing methods in Table 1.
Minor comment. Colors in Fig. 3.
We will use the same color and line style for each method across all three panels and give VQPM and HPCM-Large clearly distinct curves.