- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Rather than proposing a new method, this paper presents a systematic analysis of how the value of existing per-image latent optimization (SGA) varies with bitrate. The authors apply an identical protocol to three codecs (AEIC, StableCodec, and bmshj2018) with their decoders and entropy models frozen, and adopt the rate-equivalent margin—the additional fraction of bits the amortized encoder requires to match the quality of the optimized latent—as the main metric. The results show that the benefit of per-image optimization increases as the rate decreases and saturates at a backbone-specific ceiling; at the lowest rates, the amortized encoder requires 1.7–2.2× as many bits to reach the same quality. Based on a continuous-relaxation analysis, transfer to metrics that were not optimized, and objective-control experiments, the authors argue that at extremely low bitrates the amortized encoder, rather than the decoder, is the primary bottleneck.
Critical strengths
- To my knowledge, this is the first systematic quantification of how the value of per-image optimization, which has often been regarded as a minor refinement, depends on the operating bitrate.
- Since optimization simultaneously improves quality and reduces bitrate, the authors introduce the rate-equivalent margin and design the evaluation carefully, e.g., by separating the contributions of rate reduction and quality improvement and by cross-validating the results with BD-rate.
- The continuous-relaxation analysis separates the available per-image headroom from the fraction actually captured by optimization. This shows that discretization-aware optimization is a key component and provides concrete evidence of the remaining room for encoder-side improvements.
- The nature of the observed gains is clarified through control experiments with alternative objectives and by re-running the protocol on the MSE-trained codec with its own training objective. Unfavorable results on metrics such as MUSIQ and CLIP-IQA are reported rather than omitted, which is commendable.
- Rates are measured from actual entropy-coded bitstreams, and the paper transparently documents the exclusion of corrupted measurements, reproducibility issues across GPU architectures, and consistency checks on result files, which increases confidence in the reported measurements.
Critical weaknesses
-
Limited methodological novelty and overstated framing. SGA and per-image RDO are existing techniques, and the primary contribution lies in measurement and interpretation. The key question is therefore whether the resulting insight is sufficiently novel and general. In this regard, referring to the observed trend as a "law" based on only three backbones, and only two in the extremely low-rate regime, appears overstated.
-
Amortization gap is confounded with objective mismatch. The evaluation metric F is directly optimized and then used for evaluation, whereas the training objectives of AEIC and StableCodec are not identical to F. Control experiments that optimize each codec under its own training objective are missing for the two diffusion-based codecs. In particular, since the training objectives of AEIC and StableCodec include an adversarial loss, it is important to optimize their latents with their respective original training losses and compare the resulting realism.
-
Reliability of the FID comparison. The paper compares the FID of the base models with that obtained through RDO. However, even with patch-FID, the sample size appears too small for a reliable distributional comparison. This is particularly problematic in a paired setting such as image compression, where an insufficient number of samples can make FID behave more like a measure of image-level similarity than a robust distributional metric. Consequently, the FID improvement observed in some bitrate ranges may simply reflect the optimization of F, which consists of distortion-related terms, rather than a genuine improvement in distributional realism.
-
Mismatched control for the extreme-rate vs. conventional-rate comparison. The only codec evaluated at conventional bitrates is an MSE-trained model, which makes it difficult to fairly support the claim that the benefit is small at conventional bitrates. The same analysis should be performed on perceptual codecs operating at conventional bitrates, such as HiFiC or MS-ILLM, before a general claim about rate dependence can be established.
-
Computational cost limits practical applicability. The optimization requires approximately 0.5–3 GPU hours per 2K image, yet the paper describes the required budget as "modest." Moreover, reducing the budget to one quarter retains only about one quarter of the gain, suggesting that the method operates in a regime where the benefit remains roughly proportional to computational cost.
-
Small sample sizes. The main experiments use only 22–24 images (11 for bmshj2018), while the mechanism analysis, the objective-control experiments, and the optimization-budget experiments each use only six images. Regardless of the small p-values obtained from Page's test, the limited number of images restricts the generalizability of the conclusions.
Questions for authors
- Given that SGA and per-image RDO are existing techniques, what do the authors consider the core novel insight of this work beyond the measurement itself? Can the authors justify calling the observed trend a "law" based on three backbones, only two of which operate in the extremely low-rate regime?
- Can the authors provide control experiments for AEIC and StableCodec in which the latents are optimized under each codec's own training objective, and compare the resulting margins and realism with those obtained under F? This would help separate the amortization gap from objective mismatch.