acceptodds
Under review as a conference paper at ICLR 2027

Per-Image Latent Optimization at Extreme Bitrates

Abstract

Neural codecs compress with an amortized encoder: a single network, trained to perform well on average, encodes every image in one forward pass. Per-image latent rate–distortion optimization (RDO) instead refines the transmitted latent for each individual image at encoding time. It is commonly regarded as a marginal refinement, worth a few tenths of a dB at conventional rates, but that assessment was formed when bits were plentiful. Generative decoders now sustain reconstruction at bitrates orders of magnitude lower, where every bit the encoder misallocates is a visible fraction of the budget. We show that in this regime the value of per-image optimization is not a constant of the technique but grows systematically as the operating rate falls. We measure the gain as a rate-equivalent margin, the additional bits the amortized encoder requires on the same image's own rate ladder to match the optimized quality. Across three architecturally distinct codecs whose ladders span three orders of magnitude in rate, the median margin rises as rate decreases and saturates at a backbone-specific ceiling. At the lowest rates, the amortized encoder needs to times its bits to match per-image optimization, corresponding to median per-image BD-rate savings of –. A continuous-relaxation analysis traces this growth to two coupled trends: the per-image suboptimality of the amortized encoder widens steadily as rate falls, and the share of it that discretization-aware optimization captures becomes meaningful only at the lowest rates. The gains transfer to held-out perceptual metrics the optimizer never saw, precisely in the low-rate regime; optimizing a mismatched objective instead erases or reverses them. Practically, the gain saturates in compute: doubling the standard optimization budget adds at most a relative , so a modest budget suffices whenever content is encoded once and decoded many times. Together, these results identify the amortized encoder, rather than the decoder, as a principal bottleneck of neural compression at extreme rates.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural image compression, low bitrate image compression, rate-distortion optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NHKyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Rather than proposing a new method, this paper presents a systematic analysis of how the value of existing per-image latent optimization (SGA) varies with bitrate. The authors apply an identical protocol to three codecs (AEIC, StableCodec, and bmshj2018) with their decoders and entropy models frozen, and adopt the rate-equivalent margin—the additional fraction of bits the amortized encoder requires to match the quality of the optimized latent—as the main metric. The results show that the benefit of per-image optimization increases as the rate decreases and saturates at a backbone-specific ceiling; at the lowest rates, the amortized encoder requires 1.7–2.2× as many bits to reach the same quality. Based on a continuous-relaxation analysis, transfer to metrics that were not optimized, and objective-control experiments, the authors argue that at extremely low bitrates the amortized encoder, rather than the decoder, is the primary bottleneck.

Critical strengths
  • To my knowledge, this is the first systematic quantification of how the value of per-image optimization, which has often been regarded as a minor refinement, depends on the operating bitrate.
  • Since optimization simultaneously improves quality and reduces bitrate, the authors introduce the rate-equivalent margin and design the evaluation carefully, e.g., by separating the contributions of rate reduction and quality improvement and by cross-validating the results with BD-rate.
  • The continuous-relaxation analysis separates the available per-image headroom from the fraction actually captured by optimization. This shows that discretization-aware optimization is a key component and provides concrete evidence of the remaining room for encoder-side improvements.
  • The nature of the observed gains is clarified through control experiments with alternative objectives and by re-running the protocol on the MSE-trained codec with its own training objective. Unfavorable results on metrics such as MUSIQ and CLIP-IQA are reported rather than omitted, which is commendable.
  • Rates are measured from actual entropy-coded bitstreams, and the paper transparently documents the exclusion of corrupted measurements, reproducibility issues across GPU architectures, and consistency checks on result files, which increases confidence in the reported measurements.
Critical weaknesses
  1. Limited methodological novelty and overstated framing. SGA and per-image RDO are existing techniques, and the primary contribution lies in measurement and interpretation. The key question is therefore whether the resulting insight is sufficiently novel and general. In this regard, referring to the observed trend as a "law" based on only three backbones, and only two in the extremely low-rate regime, appears overstated.

  2. Amortization gap is confounded with objective mismatch. The evaluation metric F is directly optimized and then used for evaluation, whereas the training objectives of AEIC and StableCodec are not identical to F. Control experiments that optimize each codec under its own training objective are missing for the two diffusion-based codecs. In particular, since the training objectives of AEIC and StableCodec include an adversarial loss, it is important to optimize their latents with their respective original training losses and compare the resulting realism.

  3. Reliability of the FID comparison. The paper compares the FID of the base models with that obtained through RDO. However, even with patch-FID, the sample size appears too small for a reliable distributional comparison. This is particularly problematic in a paired setting such as image compression, where an insufficient number of samples can make FID behave more like a measure of image-level similarity than a robust distributional metric. Consequently, the FID improvement observed in some bitrate ranges may simply reflect the optimization of F, which consists of distortion-related terms, rather than a genuine improvement in distributional realism.

  4. Mismatched control for the extreme-rate vs. conventional-rate comparison. The only codec evaluated at conventional bitrates is an MSE-trained model, which makes it difficult to fairly support the claim that the benefit is small at conventional bitrates. The same analysis should be performed on perceptual codecs operating at conventional bitrates, such as HiFiC or MS-ILLM, before a general claim about rate dependence can be established.

  5. Computational cost limits practical applicability. The optimization requires approximately 0.5–3 GPU hours per 2K image, yet the paper describes the required budget as "modest." Moreover, reducing the budget to one quarter retains only about one quarter of the gain, suggesting that the method operates in a regime where the benefit remains roughly proportional to computational cost.

  6. Small sample sizes. The main experiments use only 22–24 images (11 for bmshj2018), while the mechanism analysis, the objective-control experiments, and the optimization-budget experiments each use only six images. Regardless of the small p-values obtained from Page's test, the limited number of images restricts the generalizability of the conclusions.

Questions for authors
  1. Given that SGA and per-image RDO are existing techniques, what do the authors consider the core novel insight of this work beyond the measurement itself? Can the authors justify calling the observed trend a "law" based on three backbones, only two of which operate in the extremely low-rate regime?
  2. Can the authors provide control experiments for AEIC and StableCodec in which the latents are optimized under each codec's own training objective, and compare the resulting margins and realism with those obtained under F? This would help separate the amortization gap from objective mismatch.
Reviewer_7LmyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how per-image latent optimization gains change with bitrate. Experiments on three codecs show larger gains at lower rates, with additional analyses of the optimization gap and perceptual quality.

Critical strengths
  1. The motivation is reasonable, and the low-rate gains are substantial.
  2. The evaluation is fairly careful, using true coded rates, BD-rate, and held-out metrics.
  3. The analysis provides some useful insights beyond the main results.
Critical weaknesses
  1. The gains on bmshj2018 mainly come from objective retargeting rather than the claimed amortization gap. This weakens the generality of the proposed law.
  2. The continuous-relaxation analysis is not a convincing upper bound, especially given the large gap between soft and actual rates.
  3. The experiments use relatively few images, and the mechanism analysis is even more limited.
  4. The effects of bitrate and differences between rate-specific checkpoints are not clearly separated.
Questions for authors
  1. How much of the gain on AEIC and StableCodec comes from objective mismatch?
  2. Why should the continuous solution be considered an upper bound when its rate estimate is inaccurate?
  3. Could the observed trend be partly explained by differences in checkpoint training rather than bitrate itself?
Reviewer_ucvfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates per-image latent optimization for neural image compression at extreme low bitrates. The authors observe that the gains over amortized encoding increase as bitrate decreases and eventually saturate, a phenomenon termed the "divergence law." Experiments on three codecs and several analyses are provided to characterize and explain this behavior.

Critical strengths

The paper identifies an interesting rate-dependent behavior of per-image optimization, providing useful insights into encoder-side limitations at extreme bitrates.

The experiments include several valuable analyses, such as bit-shedding decomposition, objective controls, optimization-budget studies, and held-out metric evaluations.

The systematic investigation across different codecs provides useful empirical observations beyond conventional RD comparisons.

Critical weaknesses

Limited evidence across backbones. Although the authors acknowledge that bmshj2018 mainly demonstrates objective-retargeting gains, only two objective-aligned backbones support the central amortization-gap interpretation.

Generality of the divergence law. The observed behavior is backbone-dependent, and the evaluation covers relatively few codecs. More evidence is needed to establish the generality of the proposed empirical law.

Limited mechanistic validation. The continuous-relaxation analysis is conducted on only six images and one backbone. It remains unclear whether the proposed mechanism consistently explains the observations across different codecs.

Potential influence of quantization mismatch. Previous studies on quantization train-test mismatch have highlighted discrepancies introduced by relaxation-based approximations, which can be particularly relevant at low bitrates. Given the optimistic rate estimates used in the continuous-relaxation analysis, it remains unclear to what extent the growing slack reflects an increasing amortization gap rather than relaxation-related effects.

High computational complexity. The reported optimization cost of 0.5–3 GPU-hours per image is substantial. A more complete comparison of encoding time and RD performance would help assess practical applicability.

Limited evaluation scale. The relatively small evaluation sets, despite the additional Kodak experiments, limit the evidence for generalization across diverse image content and resolutions.

Questions for authors

Can the authors provide additional evidence from another objective-aligned backbone to strengthen the amortization-gap interpretation?

Can the proposed mechanism of growing slack and late-onset capture be validated on another backbone, such as StableCodec?

Could the authors further clarify whether the growing slack is influenced by quantization-relaxation mismatch, considering the related issues discussed in previous studies on learned image compression?

Can the authors provide a more detailed comparison of encoding time and RD performance to clarify the practical tradeoff of per-image optimization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.