acceptodds
Under review as a conference paper at ICLR 2027

ResARC: Residual-Aware AutoRegressive Coding for Ultra-Low Bitrate Image Compression

18 views

Abstract

Progressive autoregressive image codecs provide an appealing paradigm for generative compression by quantizing continuous latents into discrete tokens, transmitting coarse-to-fine prefix tokens and generating the remaining suffix tokens at the decoder. However, their reconstruction quality is fundamentally limited by two residuals introduced along this pipeline: the quantization residual, arising from information loss during discrete tokenization, and the generation residual, resulting from imperfect autoregressive generation of the suffix tokens. To address these limitations, we introduce ResARC, a residual-aware autoregressive codec that explicitly compensates for both residuals at the decoder. Specifically, we generate the quantization residual with a diffusion transformer conditioned on the autoregressive decoding context, while requiring no additional side information. In parallel, we compute the generation residual at the encoder and employ a learned Generation Residual Codec to efficiently compress and transmit it for decoder-side correction. The recovered residuals are then integrated with the reconstructed latent representation and decoded through an adapted VAE decoder. Extensive experiments demonstrate that ResARC achieves competitive perceptual similarity while substantially improving distributional fidelity over leading generative codecs across the ultra-low bitrate regime.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Lossy Image Compression, Residual-aware Image Compression, Visual AutoRegressive Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_X3QLOpenReview ↗
Rating
1(−2)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
1(−1)
Read review
Paper summary

ResARC improves ultra-low bitrate autoregressive image compression by synthesizing quantization residuals without extra bits and transmitting compressed generation residuals. Comparisons with generative codecs on DIV2K and CLIC2020 show improved perceptual similarity and distributional fidelity, while ablations support the complementary benefits of both residual branches.

Critical strengths

Meaningful validation of the design. Branch ablations and compensation-strategy comparisons support the complementary benefits of synthesizing quantization residuals and transmitting generation residuals. This strengthens the case that the architecture contributes to the improvements.

Critical weaknesses
  1. Limited methodological novelty. The contribution primarily lies in decomposing reconstruction errors and integrating established techniques into a residual-aware compression architecture. While the design is coherent and the reported gains are substantial, the methodological advance appears relatively incremental. This limits the strength of the paper’s originality claim.
  2. The computational cost is not evaluated. ResARC combines a 2B autoregressive backbone, encoder-side suffix-generation replay, a residual codec, and diffusion refinement, but reports no encoding or decoding latency or peak memory. Without these measurements, it is difficult to assess whether the bitrate savings justify the additional computation, limiting the practical significance of the results.
Questions for authors
  1. Tables 1–3 examine the contributions of the residual branches and model adaptation. Do these controls also account for increased model capacity, or could additional capacity explain part of the performance improvement over ARPC?
  2. What are the encoding latency, decoding latency, and peak memory at the evaluated resolution? Are measurements available against ARPC and competitive diffusion codecs under identical hardware conditions?
  3. A key concern is the implicit orthogonal additivity assumption underlying the residual decomposition in Equation 7. The two residuals are inherently coupled — coarser quantization increases both errors by degrading token information — and their frequency spectra overlap substantially. Could the authors justify this assumption, quantify the correlation between the two residuals, and clarify whether the two branches deliver truly complementary gains or involve redundant compensation?
9 comments
AuthorsOpenReview ↗

Response to Reviewer X3QL(1/2)

Q1. Could increased model capacity explain the improvements over ARPC, rather than the residual-specific design?

A1. We compare ResARC with ARPC [5] and an enlarged AR prior without our residual-specific design on DIV2K. Specifically, the enlarged AR prior adds two Transformer blocks and rank-64 LoRA to Infinity-2B [6], introducing an additional 230.76M trainable parameters. As shown in Table R3.1, the enlarged AR prior without our residual-specific design requires 19.64% and 262.57% more bitrate than ResARC at matched FID and KID, respectively. Moreover, ResARC achieves better BD-rates across all four metrics than the enlarged AR prior, despite having fewer total parameters (3.64B vs. 3.77B). These results demonstrate that the improvements over ARPC cannot be explained simply by increasing model capacity and support the effectiveness of our residual-specific design.

Table R3.1: Comparison with ARPC and an enlarged AR prior on DIV2K. BD-rate (%) is relative to ResARC.

\hline \text{Method} & \text{Params.}\downarrow & \text{LPIPS}\downarrow & \text{DISTS}\downarrow & \text{FID}\downarrow & \text{KID}\downarrow \\\\ \hline \text{ARPC [5]} & 3.54\mathrm{B} & +40.61 & +75.24 & +104.00 & +271.91 \\\\ \text{Enlarged AR prior} & 3.77\mathrm{B} & +6.38 & +6.35 & +19.64 & +262.57 \\\\ \textbf{ResARC (Ours)} & 3.64\mathrm{B} & \mathbf{0.00} & \mathbf{0.00} & \mathbf{0.00} & \mathbf{0.00} \\\\ \hline \end{array}$$ **W2 & Q2. What are the encoding latency, decoding latency, and peak memory compared with ARPC and competitive diffusion codecs under the same hardware conditions?** **A2.** Thank you for this suggestion. We evaluate the computational efficiency of ResARC against leading generative codecs on DIV2K at 1024 × 1024 resolution, using an RTX 3090 GPU with a batch size of 1. As shown in Table R3.2, ResARC requires 2.98–3.23 s for encoding and 3.39–3.65 s for decoding. **Its decoding latency is comparable to ARPC [5] and competitive with the evaluated diffusion-based codecs.** Moreover, ResARC reduces peak GPU memory usage by 13.8% compared with ARPC (13.62 vs. 15.80 GiB). Importantly, ResARC achieves BD-rate reductions of 28.88%, 42.94%, 50.98%, and 73.11% over ARPC in terms of LPIPS, DISTS, FID, and KID, respectively. These results demonstrate that **ResARC achieves a favorable rate--perception--latency trade-off while reducing peak GPU memory usage compared with ARPC.** Table R3.2: **End-to-end coding latency, peak GPU memory usage, and BD-rate (%) comparison on DIV2K (1024 × 1024).** $$\def\rcell#1{\smash{\begin{array}{c}\hspace{4.4em}\\\\#1\\\\\hspace{4.4em}\end{array}}} \def\rmetrics#1#2#3#4{\begin{array}{cccc}\rcell{#1} & \rcell{#2} & \rcell{#3} & \rcell{#4}\end{array}} \begin{array}{lcccc} \hline \text{Method} & \text{Enc. (s)}\downarrow & \text{Dec. (s)}\downarrow & \begin{array}{c} \text{Peak GPU} \\\\ \text{memory (GiB)}\downarrow \end{array} & \begin{array}{c} \text{BD-rate vs. ARPC }(\\%) \\\\ \hline \rmetrics{\text{LPIPS}\downarrow}{\text{DISTS}\downarrow}{\text{FID}\downarrow}{\text{KID}\downarrow} \end{array} \\\\ \hline \text{PerCo (SD) [1]} & 0.15 & 9.13 & 4.86 & \rmetrics{+190.35}{+639.33}{+770.14}{+2936.40} \\\\ \text{DiffC [2]} & 1.99 & 6.24 & 4.99 & \rmetrics{+31.40}{+359.73}{+417.41}{+792.74} \\\\ \text{DiffEIC [3]} & 0.48 & 22.64 & 9.21 & \rmetrics{+106.16}{+520.97}{+423.82}{+639.93} \\\\ \text{ResULIC [4]} & 0.32 & 4.12 & 9.17 & \rmetrics{+35.52}{+297.70}{+236.68}{+801.30} \\\\ \hline \text{ARPC [5]} & 1.81\text{–}2.56 & 3.57\text{–}5.89 & 15.80 & \rmetrics{0.00}{0.00}{0.00}{0.00} \\\\ \textbf{ResARC (Ours)} & 2.98\text{–}3.23 & 3.39\text{–}3.65 & 13.62 & \rmetrics{\mathbf{-28.88}}{\mathbf{-42.94}}{\mathbf{-50.98}}{\mathbf{-73.11}} \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer X3QL(2/2)

Q3. Does the residual decomposition require orthogonality, and do the two branches provide complementary gains?

A3. Thank you for this insightful comment. The two residuals arise from different processes: the quantization residual captures details lost during tokenization, while the generation residual captures suffix generation errors affecting local structures and textures. The residual decomposition in Equation (7) is an exact algebraic identity and does not require orthogonality or disjoint frequency spectra. We further analyze their correlations on DIV2K. For each prefix depth, we compute the angle between the pooled quantization and generation residual vectors. As shown in Figure 3(a), these angles range from 84.0° to 92.5°, suggesting that the pooled residual vectors are approximately orthogonal. Furthermore, Figure 3(b) shows the mean signed Pearson correlation for each residual pair, with lower values for quantization–generation pairs than for pairs of generation residuals at different prefix depths. Table R3.3 also reports a mean absolute Pearson correlation of 0.076 between quantization and generation residuals (Q–G), compared with 0.411 between generation residuals (Within-G) at different transmitted prefix depths for the same image. These results indicate substantially weaker linear correlation between quantization and generation residuals than among generation residuals at different prefix depths.

Furthermore, Table 2 quantitatively shows that the generation-only and quantization-only configurations require 1.90% and 5.38% more bitrate, respectively, than full ResARC at matched DISTS. Together, these results support complementary gains from compensating both residuals.

Table R3.3: Pearson correlations by depth (mean ∣r∣|r|).

\hline k & \text{Q–G} & \text{Within-G} \\\\ \hline 5 & 0.104 & 0.362 \\\\ 6 & 0.097 & 0.416 \\\\ 7 & 0.077 & 0.444 \\\\ 8 & 0.060 & 0.447 \\\\ 9 & 0.052 & 0.426 \\\\ 10 & 0.068 & 0.370 \\\\ \textbf{All} & \mathbf{0.076} & \mathbf{0.411} \\\\ \hline \end{array}$$ **W1. Limited methodological novelty.** **A4.** Thank you for this comment. To our knowledge, **ResARC is the first to explicitly distinguish and compensate for quantization and generation residuals in autoregressive generative compression**, both of which are key factors affecting perceptual compression performance. The originality of our method lies in linking the explicit formulation of these two residuals to distinct compensation mechanisms based on their different roles in reconstruction. Specifically, we develop compensation strategies tailored to their distinct characteristics, namely, modeling the quantization residual using a DiT and explicitly encoding and transmitting the generation residual, **thereby achieving competitive performance against leading generative codecs.** **References:** [1] PerCo (SD): Open Perceptual Compression. [2] Lossy Compression with Pretrained Diffusion Models. [3] Towards Extreme Image Compression with Latent Feature Guidance and Diffusion Prior. [4] Ultra Lowrate Image Compression with Semantic Residual Coding and Compression-aware Diffusion. [5] AutoRegressive-based Progressive Coding for Ultra-Low Bitrate Image Compression. [6] Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis.

After carefully reviewing the rebuttals, my primary concerns regarding methodological novelty and computational cost remain unresolved. Consequently, I am adjusting my score as the trade-offs introduced by this design are, in my view, unacceptable for practical deployment.

  1. Unacceptable Computational Cost and Latency: The authors' response regarding latency (Table R2.5 / Table 5) reports an encoding time of 2.98–3.23s and a decoding time of 3.39–3.65s on an RTX 3090 GPU. While the authors argue that the additional DiT and generation residual codec (GRC) modules account for less than 2% of the total decoding time, this defense misses the forest for the trees. The baseline itself (Infinity-2B autoregressive backbone) is already prohibitively heavy and slow. Stacking heavy diffusion-based generation and explicit residual coding on top of an already sluggish 2B–3.6B parameter autoregressive model results in an end-to-end decoding latency of over 3.5 seconds per image. In ultra-low bitrate image compression scenarios, a 3.5+ second decoding delay is a critical bottleneck that renders the method practically infeasible. The authors have failed to provide any viable pathway or lightweight alternative for practical deployment.
  2. Incremental Novelty and Component Stitching: Regarding the concern over technical novelty, the authors position ResARC as the first to explicitly distinguish and compensate for quantization and generation residuals. However, a close inspection reveals that the framework is essentially an engineering composition of existing, off-the-shelf generative components—namely, applying Flow Matching via a DiT for quantization residuals and a traditional analysis/synthesis transform (with a four-pass entropy model) for generation residuals, tied together with an ARPC backbone. The post-hoc geometric analysis (such as the Q-G angle and Pearson correlation in Figure 3) merely rationalizes the architectural choices with descriptive statistics rather than providing a fundamental breakthrough in compression theory or conceptual design. The contribution remains an incremental modular combination rather than a conceptual advancement.
Reviewer_prC2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ResARC, which builds on the ARPC framework. It divides the latent error into two parts: quantization residual and generation residual. The quantization residual remains after quantization using the full token sequence. The generation residual comes from errors in the generated tokens. A conditional DiT generates the quantization residual. A learned codec compresses and transmits the generation residual. These two components improve perceptual quality at ultra-low bitrates.

Critical strengths
  • The motivation is clear. Both residuals have clear definitions and can be directly computed. The method can be integrated into a progressive AR coding framework.
  • The method achieves better DISTS and FID results than ARPC on DIV2K and CLIC2020. Ablations on the two branches and different compensation strategies support the effectiveness of the design.
Critical weaknesses
  • Another way to reduce quantization error is to increase the vocabulary size of BitVAE. The current ablations only study residual compensation with a fixed tokenizer. They do not show whether adding DiT is better than increasing the vocabulary size. Therefore, it's unclear whether the quality gain from DiT justifies its extra computational cost. The authors should directly compare these two approaches.
  • Is the Generation Residual Codec trained with fixed rate-distortion loss weights? The overall bitrate is varied by changing the number of transmitted scales, kk. For a fixed kk, how does changing the bitrate of the Generation Residual Codec affect reconstruction quality? Under the same bitrate, what is the optimal bit allocation between the image token bitstream and the generation residual bitstream?
  • In Table 5, the generation residual bitrate rises from 0.000229 bpp at k=7k=7 to 0.000279 at k=8k=8 and 0.000314 at k=9k=9. It then drops to 0.000198 at k=10k=10. Why does the residual bitrate peak at an intermediate kk? Does the autoregressive model have its largest generation error at k=9k=9?
  • ResARC adds a DiT for quantization error compensation and a Generation Residual Codec to ARPC. The authors should analyze how these additional modules affect encoding and decoding latency.
Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer prC2 (1/4)

Q1. Does the quality gain from the DiT justify its additional decoding cost compared with enlarging the BitVAE vocabulary?

A1. We politely note that although increasing the BitVAE vocabulary can reduce quantization error, it can also increase the bitrate required to transmit image tokens, as observed in ARPC [1]. To directly evaluate this trade-off, we conducted a systematic comparison between our DiT-based approach for compensating for the quantization residual and directly enlarging the BitVAE vocabulary. Specifically, we compared our framework, which combines a tokenizer with a vocabulary size of 2³² and our quantization residual generator (DiT), against a model using a larger vocabulary of 2⁶⁴ without quantization residual prediction. The experiments were conducted on the DIV2K validation set at 1024 × 1024, with generation residual coding disabled. The R-D curves in Figure 16 show that our quantization residual generation design achieves better compression performance than directly enlarging the BitVAE vocabulary size across LPIPS, DISTS, FID, and KID at comparable bitrates. Specifically, with seven transmitted scales, our DiT-based approach achieves FID 21.61 and KID 2.73 × 10⁻⁴ at 0.012 bpp, while the larger vocabulary yields FID 86.93 and KID 2.487 × 10⁻² at 0.013 bpp. Moreover, we compared our quantization residual generation design, which employs an 80.24M-parameter DiT, with an enlarged AR prior containing 230.76M additional trainable parameters. As shown in Table R2.1, enlarging the AR prior saves 1.89% bitrate at matched LPIPS but requires 6.18% more bitrate at matched FID and 98.68% more at matched KID. Taken together, the quality gains across all four metrics over vocabulary expansion and the FID and KID gains over AR expansion justify our DiT-based quantization residual generation design, which requires no additional transmitted bits and only 0.04 s per image on an RTX 3090 GPU. These results support our quantization residual generation design over using a larger 2⁶⁴ BitVAE vocabulary size or directly increasing the AR model capacity.

Table R2.1: Model size and BD-rate (%) on DIV2K. BD-rate is computed relative to our configuration with only the quantization residual branch.

\hline \text{Method} & \text{Params.}\downarrow & \text{LPIPS}\downarrow & \text{DISTS}\downarrow & \text{FID}\downarrow & \text{KID}\downarrow \\\\ \hline \text{Enlarging AR Prior} & 3.77\mathrm{B} & \mathbf{-1.89} & +0.90 & +6.18 & +98.68 \\\\ \textbf{Ours (only Quantization Residual Branch)} & \mathbf{3.62B} & 0.00 & \mathbf{0.00} & \mathbf{0.00} & \mathbf{0.00} \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer prC2 (2/4)

Q2. Is the rate weight fixed when training the generation residual codec? How does the generation residual bitrate affect quality at fixed k? Which bitrate allocation between image tokens and generation residuals performs best at a comparable total bitrate?

A2. ResARC uses a fixed rate weight of λR=1.0\lambda_R=1.0 when training the generation residual codec. Nevertheless, since the codec is conditioned on the transmitted prefix depth k, it can adapt its generation residual bitrates to different numbers of transmitted token scales. To investigate the effect of the generation residual bitrate on reconstruction quality, we train the codec with different rate weights λR\lambda_R. As shown in Table R2.2, at k∈7,8,10k\in\\{7,8,10\\}, a higher generation residual bitrate generally corresponds to better reconstruction quality. However, increasing the generation residual bitrate does not necessarily yield the best overall rate-perception trade-off. Table R2.3 shows that the default λR=1.0\lambda_R=1.0 achieves the lowest BD-rate across all four metrics among the three evaluated weights λR∈0.5,1.0,2.0\lambda_R \in \\{0.5,1.0,2.0\\}, supporting our default choice.

To further investigate the bitrate allocation between transmitted image tokens and generation residuals under comparable total bitrates during inference, we additionally vary the allocation by partially masking image tokens in the last transmitted scale, thereby reducing the image token bitrate while allowing more bits for the generation residual. As shown in Figure 11, at k=5k=5, reducing the image token-to-generation residual bitrate ratio from the default 12.45:1 to 7.69:1 lowers the total bitrate by only 0.86%, while increasing FID by 4.01%. Further reducing the ratio to 6.98:1 results in an 8.50% higher FID than the default allocation. These results demonstrate that at k=5k=5, ResARC's default bitrate allocation of 12.45:1 achieves the best reconstruction quality across all four metrics among the evaluated schemes at comparable total bitrates.

Table R2.2: Reconstruction quality at different generation residual bitrates.

\hline k & \lambda_R & \text{Gen. Res. bpp} & \text{LPIPS}\downarrow & \text{DISTS}\downarrow & \text{FID}\downarrow & \text{KID}\downarrow \\\\ \hline 7 & 1.0 & 2.29\times10^{-4} & 0.2530 & 0.0993 & 21.06 & 2.30\times10^{-4} \\\\ 7 & 0.5 & 3.67\times10^{-4} & \mathbf{0.2524} & \mathbf{0.0990} & \mathbf{21.02} & \mathbf{2.04\times10^{-4}} \\\\ \hline 8 & 1.0 & 2.79\times10^{-4} & 0.2142 & 0.0865 & 18.48 & 1.05\times10^{-4} \\\\ 8 & 0.5 & 4.28\times10^{-4} & \mathbf{0.2140} & \mathbf{0.0863} & \mathbf{18.38} & \mathbf{9.80\times10^{-5}} \\\\ \hline 10 & 1.0 & 1.98\times10^{-4} & \mathbf{0.1517} & 0.0617 & 13.97 & -7.32\times10^{-5} \\\\ 10 & 0.5 & 2.65\times10^{-4} & 0.1518 & \mathbf{0.0616} & \mathbf{13.96} & -\mathbf{8.33\times10^{-5}} \\\\ \hline \end{array}$$ Table R2.3: **BD-rate (%) for different rate weights when training the GRC.** $$\begin{array}{ccccc} \hline \lambda_R & \text{LPIPS}\downarrow & \text{DISTS}\downarrow & \text{FID}\downarrow & \text{KID}\downarrow \\\\ \hline 0.5 & +1.43 & +0.02 & +3.95 & +20.43 \\\\ 2.0 & +23.17 & +21.78 & +22.90 & +36.37 \\\\ \textbf{1.0 (Ours)} & \mathbf{0.00} & \mathbf{0.00} & \mathbf{0.00} & \mathbf{0.00} \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer prC2 (3/4)

Q3. Why does the generation residual bitrate peak at an intermediate prefix depth k? Does the autoregressive model have its largest generation error at k = 9?

A3. We would like to clarify that the peak in generation residual bitrate does not indicate the largest generation error at k=9. As more token scales are losslessly transmitted to the decoder and fewer scales need to be generated lossily, the magnitude of the generation residual decreases monotonically. As shown in Table R2.4, the mean generation residual L2 norm consistently decreases from 1.72 × 10² at k=5 to 9.05 × 10¹ at k=10, including 1.05 × 10² at k=9.

The non-monotonic behavior of the generation residual bitrate can be explained by the rate-perception trade-off across different overall bitrate regimes. Our generation residual codec is conditioned on the transmitted prefix depth k, allowing it to adapt its coding behavior to the residual characteristics and bitrate budgets at different prefix depths. Specifically, at smaller k, the overall bitrate is extremely low, requiring the codec to tightly constrain the generation residual bitrate to minimize bitrate overhead while preserving reconstruction quality. As k increases, the overall bitrate increases with additional transmitted token scales, so a modest increase in generation residual bitrate incurs a smaller relative overhead. When k increases further, fewer token scales need to be generated lossily, and the residual magnitude decreases, thereby reducing the required generation residual bitrate. Moreover, Figure 10 in Appendix D shows that the proportion of the total bitrate allocated to the generation residual generally decreases as k increases.

Therefore, the intermediate peak in generation residual bitrate reflects the adaptive rate-distortion behavior of the generation residual codec, rather than an increase in the magnitude of the generation residual.

Table R2.4: Mean generation residual ℓ2\ell_2 norm and bitrates on DIV2K.

\hline k & \text{Mean }\ell_2\text{ Norm} & \text{Gen. Res. bpp} \\\\ \hline 5 & 1.72\times10^{2} & 2.30\times10^{-4} \\\\ 6 & 1.50\times10^{2} & 2.30\times10^{-4} \\\\ 7 & 1.31\times10^{2} & 2.29\times10^{-4} \\\\ 8 & 1.18\times10^{2} & 2.79\times10^{-4} \\\\ 9 & 1.05\times10^{2} & 3.14\times10^{-4} \\\\ 10 & 9.05\times10^{1} & 1.98\times10^{-4} \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer prC2 (4/4)

Q4. How do the additional DiT and Generation Residual Codec affect encoding and decoding latency?

A4. Thank you for this suggestion. We conduct a detailed latency analysis of the additional modules and benchmark ResARC against leading generative codecs on DIV2K at 1024 × 1024 resolution, using RTX 3090 GPUs with a batch size of 1. As shown in Table R2.5, the quantization residual generator (i.e., DiT) requires no additional encoding computation and takes only approximately 0.04 s for decoding. Meanwhile, the generation residual codec (i.e., GRC) requires approximately 0.02 s for both encoding and decoding. Together, these two residual branches account for less than 2% of the total decoding time, indicating that the additional modules introduce only minor computational overhead.

Moreover, as shown in Table R2.6, ResARC requires 2.98–3.23 s for encoding and 3.39–3.65 s for decoding. Its decoding latency is comparable to ARPC [1] and competitive with the evaluated diffusion-based codecs. Importantly, ResARC achieves BD-rate reductions of 28.88%, 42.94%, 50.98%, and 73.11% over ARPC in terms of LPIPS, DISTS, FID, and KID, respectively.

These results demonstrate that ResARC achieves substantial rate-perception improvements while introducing only minor additional latency from its residual coding branches, offering a favorable rate--perception--latency trade-off.

Table R2.5: Encoding and decoding latency breakdown of ResARC on DIV2K (1024 × 1024).

\def\rpair#1#2{\begin{array}{cc}\rcell{#1} & \rcell{#2}\end{array}} \begin{array}{lccc} \hline \text{Module} & \begin{array}{c} \text{Encoding} \\\\ \hline \rpair{\text{Min. }(k=5)}{\text{Max. }(k=10)} \end{array} & \begin{array}{c} \text{Decoding} \\\\ \hline \rpair{\text{Min. }(k=5)}{\text{Max. }(k=10)} \end{array} & \text{Params.} \\\\ \hline \text{DiT (Quant. Res.)} & \rpair{\text{–}}{\text{–}} & \rpair{0.04\\,(1.18\\%)}{0.04\\,(1.10\\%)} & 0.08\mathrm{B} \\\\ \text{GRC (Gen. Res.)} & \rpair{0.02\\,(0.67\\%)}{0.02\\,(0.62\\%)} & \rpair{0.02\\,(0.59\\%)}{0.02\\,(0.55\\%)} & 0.02\mathrm{B} \\\\ \text{VAE} & \rpair{0.16\\,(5.37\\%)}{0.16\\,(4.95\\%)} & \rpair{0.53\\,(15.63\\%)}{0.54\\,(14.79\\%)} & 0.11\mathrm{B} \\\\ \text{AR + Entropy Coding} & \rpair{2.75\\,(92.28\\%)}{3.00\\,(92.88\\%)} & \rpair{2.75\\,(81.12\\%)}{3.00\\,(82.19\\%)} & 2.21\mathrm{B} \\\\ \text{T5 encoder} & \rpair{0.05\\,(1.68\\%)}{0.05\\,(1.55\\%)} & \rpair{0.05\\,(1.48\\%)}{0.05\\,(1.37\\%)} & 1.22\mathrm{B} \\\\ \hline \textbf{Total} & \rpair{\mathbf{2.98\\,(100.00\\%)}}{\mathbf{3.23\\,(100.00\\%)}} & \rpair{\mathbf{3.39\\,(100.00\\%)}}{\mathbf{3.65\\,(100.00\\%)}} & \mathbf{3.64B} \\\\ \hline \end{array}$$ Table R2.6: **End-to-end coding latency and BD-rate (%) comparison on DIV2K (1024 × 1024).** $$\def\rcell#1{\smash{\begin{array}{c}\hspace{4.4em}\\\\#1\\\\\hspace{4.4em}\end{array}}} \def\rmetrics#1#2#3#4{\begin{array}{cccc}\rcell{#1} & \rcell{#2} & \rcell{#3} & \rcell{#4}\end{array}} \begin{array}{lccc} \hline \text{Method} & \text{Enc. (s)}\downarrow & \text{Dec. (s)}\downarrow & \begin{array}{c} \text{BD-rate vs. ARPC }(\\%) \\\\ \hline \rmetrics{\text{LPIPS}\downarrow}{\text{DISTS}\downarrow}{\text{FID}\downarrow}{\text{KID}\downarrow} \end{array} \\\\ \hline \text{PerCo (SD) [2]} & 0.15 & 9.13 & \rmetrics{+190.35}{+639.33}{+770.14}{+2936.40} \\\\ \text{DiffC [3]} & 1.99 & 6.24 & \rmetrics{+31.40}{+359.73}{+417.41}{+792.74} \\\\ \text{DiffEIC [4]} & 0.48 & 22.64 & \rmetrics{+106.16}{+520.97}{+423.82}{+639.93} \\\\ \text{ResULIC [5]} & 0.32 & 4.12 & \rmetrics{+35.52}{+297.70}{+236.68}{+801.30} \\\\ \hline \text{ARPC [1]} & 1.81\text{–}2.56 & 3.57\text{–}5.89 & \rmetrics{0.00}{0.00}{0.00}{0.00} \\\\ \textbf{ResARC (Ours)} & 2.98\text{–}3.23 & 3.39\text{–}3.65 & \rmetrics{\mathbf{-28.88}}{\mathbf{-42.94}}{\mathbf{-50.98}}{\mathbf{-73.11}} \\\\ \hline \end{array}$$ **References:** [1] AutoRegressive-based Progressive Coding for Ultra-Low Bitrate Image Compression. [2] PerCo (SD): Open Perceptual Compression. [3] Lossy Compression with Pretrained Diffusion Models. [4] Towards Extreme Image Compression with Latent Feature Guidance and Diffusion Prior. [5] Ultra Lowrate Image Compression with Semantic Residual Coding and Compression-aware Diffusion.
Reviewer_nqKfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ResARC, a residual-aware autoregressive framework for ultra-low-bitrate image compression. Building upon progressive autoregressive coding, the authors identify two sources of reconstruction errors: quantization residuals introduced by discrete tokenization and generation residuals arising from imperfect autoregressive prediction. To address these errors, the proposed method employs a conditional diffusion transformer to generate quantization residuals without additional bitrate overhead, while introducing a learned residual codec to compress and transmit generation residuals. The two residuals are combined with the reconstructed latent and decoded through an adapted VAE decoder.

Experiments on DIV2K and CLIC2020 demonstrate competitive rate-perception performance compared with existing generative compression methods, particularly in terms of DISTS and distributional fidelity metrics. Ablation studies further investigate the contributions of the two residual branches, their compensation strategies, and the training objectives.

Critical strengths
  1. The paper identifies quantization and generation residuals in autoregressive image compression and proposes separate compensation mechanisms. While the technical novelty is relatively limited, the overall motivation is reasonable.

  2. The paper includes comparisons with existing methods and ablation studies to evaluate the proposed components. Although the performance improvements are not particularly impressive, the results provide some support for the effectiveness of the proposed approach.

Critical weaknesses
  1. Further justification of the residual compensation design. The paper adopts two parallel branches to compensate for quantization and generation residuals. It would be interesting to investigate an alternative sequential design, where the generation residual is first used to correct the autoregressively reconstructed latent, followed by diffusion-based enhancement. Such a comparison could provide further insights into the design choices and the interaction between the two residuals.

  2. Further analysis of the generation residual's contribution. Although the ablation results demonstrate improvements from the generation residual branch, its benefits are not particularly evident in some qualitative comparisons. A more detailed analysis of its contribution, especially the improvements brought by the transmitted residual information, would help better understand the effectiveness of this component.

Questions for authors
  1. Residual compensation design. What motivates the parallel compensation of quantization and generation residuals? Would a sequential design, where the generation residual first corrects the autoregressively reconstructed latent and the diffusion model subsequently enhances the corrected representation, provide potential benefits? A comparison or further discussion would help clarify this design choice.

  2. Contribution of the generation residual. While the ablation results show improvements from the generation residual branch, its contribution is less evident in some qualitative examples. Could the authors provide further analysis of how much improvement comes from the transmitted residual information itself, rather than the additional conditional synthesis network?

  3. Training objective ablations. Appendix Table 8 provides ablation results for different loss combinations. Could the authors additionally provide the corresponding rate-perception curves for the baseline objective, each individual perceptual loss (LPIPS or DISTS), and their combination? This would help assess the effectiveness and consistency of these objectives across different bitrates.

2 comments
AuthorsOpenReview ↗

Response to Reviewer nqKf (1/2)

W1 & Q1. What motivates the parallel compensation of quantization and generation residuals, and could correcting generation errors before diffusion refinement improve performance?

A1. Thank you for this suggestion. As explained in Sections 4.1–4.3 and Equation (7), the two residuals arise from different processes and exhibit different properties: the quantization residual captures information lost during tokenization, including fine-grained details that are vital for reconstruction, whereas the generation residual captures suffix generation errors and may affect both local details and image structures. These differences motivate separate, parallel compensation branches whose outputs are fused additively in latent space, as formulated in Equation (12).

We implemented the suggested sequential variant, in which the autoregressively reconstructed latent is first corrected using the decoded generation residual, and the corrected latent is then injected into the DiT as an additional condition. We compared both variants on DIV2K at 1024 × 1024 resolution, using the same training budget and a shared VAE decoder. As shown in Table R1.1, the sequential variant slightly reduces BD-rate at matched LPIPS but increases BD-rate at matched DISTS, FID, and KID. These results suggest that the original autoregressive decoding conditions, consisting of the Transformer’s final-scale hidden features and reconstructed token embeddings, already provide sufficient context for predicting the quantization residual. Conditioning the DiT additionally on the latent corrected by the decoded generation residual does not yield a consistent rate–perception improvement in this comparison. We therefore retain the parallel compensation design, which allows the DiT to begin quantization residual generation without waiting for the generation residual to be decoded. The two branches can thus run concurrently when hardware resources permit, potentially reducing decoding latency.

Table R1.1: Comparison of parallel and sequential residual compensation on DIV2K. BD-rate (%) is computed relative to the parallel configuration.

\hline \text{Model} & \text{LPIPS}\downarrow & \text{DISTS}\downarrow & \text{FID}\downarrow & \text{KID}\downarrow \\\\ \hline \text{Sequential} & \mathbf{-0.13} & +0.09 & +0.14 & +0.53 \\\\ \textbf{Parallel (Ours)} & 0.00 & \mathbf{0.00} & \mathbf{0.00} & \mathbf{0.00} \\\\ \hline \end{array}$$ **W2 & Q2. How much of the gain comes from the transmitted generation residual beyond the conditional synthesis network?** **A2. (1) Visual evidence.** Figure 17 qualitatively compares the autoregressive baseline with the variant that incorporates the generation residual branch. As highlighted in the red boxes, **the generation residual branch reduces spurious doubled edges along the wooden planks and recovers finer window-lattice details beneath the dome.** Moreover, the DISTS metric improves from 0.0594 to 0.0524 at similar bitrates of 0.0672 and 0.0674 bpp, respectively. Table 2 also shows that adding the generation residual branch yields lower BD-rates than the autoregressive baseline across all four metrics. **(2) Contribution of the generation residual beyond the conditional synthesis network.** To distinguish the contribution of the transmitted generation residual from that of the conditional synthesis network, we introduce a zero-latent variant that replaces the entropy-decoded generation residual latent with zeros before the conditional synthesis transform, while retaining the same network architecture and other conditioning signals. For a fair comparison, we retrain this variant under the same training settings, allowing the conditional synthesis network to adapt to the zero-latent input. As shown in Table R1.2, the zero-latent variant exhibits higher BD-rates than our original generation residual-only across all four metrics, requiring 22.18%, 38.65%, 57.59%, and 110.02% more bitrate at matched LPIPS, DISTS, FID, and KID, respectively. **These results indicate that the transmitted generation residual provides substantial rate–perception gains beyond those achievable by the conditional synthesis network alone.** **Table R1.2: Comparison of zero-latent and generation residual inputs on DIV2K.** BD-rate (%) is computed relative to the generation residual configuration. $$\begin{array}{lcccc} \hline \text{Config.} & \text{LPIPS}\downarrow & \text{DISTS}\downarrow & \text{FID}\downarrow & \text{KID}\downarrow \\\\ \hline \text{Zero latent} & +22.18 & +38.65 & +57.59 & +110.02 \\\\ \textbf{G-only (Ours)} & \mathbf{0.00} & \mathbf{0.00} & \mathbf{0.00} & \mathbf{0.00} \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer nqKf (2/2)

Q3. Could the authors provide rate-perception curves for the baseline objective, LPIPS, DISTS, and their combination across bitrates?

A3. We thank the reviewer for this suggestion.

We present rate-perception curves on DIV2K in Figures 14 and 15 for the two residual branches, comparing the baseline objective, the baseline with LPIPS or DISTS individually, and the baseline with both losses. We report LPIPS, DISTS, FID, and KID as evaluation metrics. For the quantization residual generator, Figure 14 shows that adding LPIPS as the sole perceptual loss consistently improves LPIPS and FID and achieves the lowest KID score among the evaluated settings. For the generation residual codec, Figure 15 shows that settings with DISTS supervision substantially outperform those without it, and using DISTS as the sole perceptual loss achieves lower DISTS, FID, and KID scores than the joint LPIPS and DISTS objective, while the latter achieves a slightly lower LPIPS score. These curves complement the BD-rate results in Table 14 and support our branch-specific choice of perceptual losses: LPIPS for the quantization residual generator and DISTS for the generation residual codec.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive feedback. Our paper introduces ResARC, which, to our knowledge, is the first autoregressive image codec to explicitly distinguish and compensate for quantization and generation residuals. ResARC employs a diffusion transformer to predict the quantization residual without additional transmitted bits and a learned codec to compress and transmit the generation residual. By tailoring the compensation strategies to the distinct characteristics of each residual, ResARC achieves competitive rate–perception performance against leading generative codecs at ultra-low bitrates.

We have responded to the questions and concerns raised by each reviewer and revised the manuscript based on the constructive suggestions. The revised manuscript includes additional experiments and clarifications, and all changes are highlighted in blue. We appreciate the reviewers’ time and thoughtful suggestions and welcome any further discussion.

Area_Chair_Vwy1OpenReview ↗

Please join the discussion before it closes

Thank you to the authors for the rebuttal and to the reviewers for engaging in the discussion. As the discussion period draws to a close, I encourage everyone to keep the conversation going. I’d appreciate a brief final assessment from each reviewer. After reading the rebuttal and the discussion so far, what is the main reason you lean toward acceptance or rejection, and what evidence supports that view? Please note which concerns the rebuttal resolved and which remaining concerns, if any, are decisive for your recommendation.

If your assessment differs from another reviewer’s, please point to the specific claim or evidence you interpret differently. Authors are welcome to clarify any factual points that remain open. This will help us identify where we agree and understand the reasons for any remaining disagreement.

AuthorsOpenReview ↗

Brief Follow-up on Our Rebuttal

Dear Reviewers,

Thank you again for your time and thoughtful feedback. If you have a moment before the discussion closes, we would be grateful to hear whether any key concerns remain and how our responses and additional experiments inform your assessment.

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.