acceptodds
Under review as a conference paper at ICLR 2027

ResGVC: Residual Coding for High-fidelity Generative Video Compression

Abstract

Generative video compression (GVC) methods usually employ decoder-side diffusion models to denoise reconstructed pictures based on generative priors. Besides, most of these methods are applied in latent space rather than directly in pixel domain. In this paper, we propose ResGVC, a residual coding framework for GVC that guides denoising toward original pictures by transmitting residual signals. ResGVC first encodes the source pictures within a learned video codec and then decomposes the distortion between the codec reconstruction and the source pictures into a sequence of residual segments, where each residual segment is acquired at each denoising step. Residual segments are then compressed using a novel residual codec with a diffusion transformer as entropy model. Unlike prior methods, ResGVC introduces explicit source constraints throughout denoising rather than relying on decoder-side generation alone. The residual codec also enables more efficient context modeling capability based on the powerful generative priors of diffusion transformer. Moreover, operating directly in pixel space avoids the irreversible information loss caused by latent representation bottlenecks. Experimental results show that, compared with the previous state-of-the-art method, ResGVC achieves 63.7% and 83.8% BD-rate reductions in terms of LPIPS and PSNR, respectively, while also delivering superior visual quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
generative video compression, residual coding, diffusion model, entropy model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pDJLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ResGVC, a residual coding framework for GVC that guides denoising toward original pictures by transmitting residual signals. ResGVC first encodes the source pictures within a learned video codec and then decomposes the distortion between the codec reconstruction and the source pictures into a sequence of residual segments, where each residual segment is acquired at each denoising step. Residual segments are then compressed using a novel residual codec with a diffusion transformer as entropy model. Experimental results show that, compared with the previous state-of-the-art method, ResGVC achieves 63.7% and 83.8% BD-rate reductions in terms of LPIPS and PSNR, respectively, while also delivering superior visual quality.

Critical strengths
  1. The proposed method that is inspired from the diffusion process to progressively encode the residual between the raw and compressed frames is novel;

  2. The experiments show promising results and ablation studies are informative.

Critical weaknesses
  1. The proposed method uses DCVC-UF as its base codec, however, it is not compared with in the experiments.

  2. In Table 1 and Figure 5, the FloLPIPS performance of the proposed method is always lower than LPIPS performance, especially, the FloLPIPS is worse than YODA on MCL-JCV given better LPIPS. This raises a concern that whether the temporal consistency of the proposed method is not satisfying.

  3. Using DCVC-UF as its base codec should be clearly claimed in the main text, instead of only in the Appendix.

Questions for authors

Please see Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer pDJL

Thank you very much for your careful review of our paper and insightful comments. We hope the following responses can alleviate your concerns.

W1: Additional baseline

According to your advice, we have included DCVC-UF as an additional baseline. As shown in Table R3.1, for perceptual metrics, ResGVC demonstrates substantial improvements over DCVC-UF. For PSNR and MS-SSIM, although ResGVC performs worse than DCVC-UF, the gap remains relatively small, and it still outperforms the GVC method S2VC which achieves the best fidelity- oriented performance among all GVC baselines.

Table R3.1. BD-rate (%) ↓\downarrow / BD-metric ↑\uparrow averaged on UVG, HEVC B, and MCL-JCV. “N/A” arises from insufficient overlap between rate-metric curves.

MethodLPIPSDISTSFIDFloLPIPSPSNRMS-SSIM
DCVC-UFN/A / -0.137N/A / -0.09365.8 / -9.4N/A / -0.178-94.1 / 7.265-91.9 / 0.109
S2VC382.8 / -0.064541.7 / -0.039-52.7 / 9.8600.4 / -0.068-75.8 / 3.143N/A / 0.077
YODA0.0 / 0.0000.0 / 0.0000.0 / 0.00.0 / 0.0000.0 / 0.0000.0 / 0.000
ResGVC (Ours)-63.7 / 0.030-21.2 / 0.003-67.3 / 13.3-26.2 / 0.008-83.8 / 4.379-84.5 / 0.086

W2: Temporal consistency

We acknowledge that the temporal consistency of ResGVC has room for improvement indeed. However, in most cases on test datasets, the FloLPIPS of ResGVC is still competitive with or better than that of YODA. In the supplementary material, we also upload an MP4 video for the reviewer to compare the temporal consistency between ResGVC and YODA.

Regarding the relatively weaker FloLPIPS performance of ResGVC in some cases, we believe one possible reason is that ResGVC learns to maintain temporal consistency in a GOP-by-GOP manner, which is more challenging than the frame-by-frame processing adopted by YODA. We will improve this point in our future work.

W3: Clarifying the base codec

We have clearly claimed in Section 3.2 of our revised manuscript that DCVC-UF is used as our base codec.

Thank the authors for the response and additional results. I would keep my weak acceptance recommendation.

AuthorsOpenReview ↗

Response to Reviewer pDJL

Thank you for your reply! We appreciate your suggestions and feedback all along.

Reviewer_ikzFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ResGVC combines a learned video codec with iterative pixel-space residual refinement. A VideoPiT-based entropy model supports the transmission of selected residuals, while subsequent residuals are generated without additional bits. The paper evaluates the framework on HEVC B, UVG, and MCL-JCV, using two residual-coding steps followed by three residual-generation steps.

Critical strengths
  1. A meaningful approach to source-constrained generative reconstruction. Transmitting selected residuals introduces explicit source information during refinement, rather than relying entirely on decoder-side generation. Combining this with residual generation provides a reasonable mechanism for controlling the perception–fidelity–rate trade-off.

  2. Promising results with useful component ablations. The reported results are competitive across several metrics and datasets. Ablations on pixel-space residual coding, entropy modeling, and the coding/generation split provide evidence for some design choices.

Critical weaknesses
  1. Missing baselines weaken the state-of-the-art claims. The evaluation omits relevant methods including GVC-RT, GVC1D and S²VC. Comparisons under consistent quality and runtime protocols are needed to establish the claimed advantage over recent generative codecs. Public implementations are available for at least GVC-RT, GVC1D and S²VC.

  2. The residual framework’s contribution is confounded with base-codec selection and fine-tuning. ResGVC jointly fine-tunes its underlying DCVC-UF, yet compares against DCVC-FM rather than DCVC-UF. Without comparisons against both pretrained DCVC-UF and independently fine-tuned DCVC-UF using matching data and rate–distortion objectives, the gains cannot be attributed cleanly to the proposed residual framework.

  3. The GNVC-VD quality–speed comparison lacks implementation consistency. GNVC-VD’s quality results come from its original paper, while runtime is measured using an untrained architectural reimplementation whose equivalence to the complete codec is not established. Table 2 reports 439/4,225 ms per frame on H20, versus the original paper’s 153/1,557 ms at 1080p on A800. These cross-device timings are not directly comparable. Resolution, model configuration, inference pipeline and optimization details must be established before drawing quality–speed conclusions.

  4. Complexity reporting does not establish the complete deployment cost. Table 2 omits resolution, precision, CUDA/kernel settings and baseline optimization details. It is unclear whether the 593.9M parameters include DCVC-UF and whether timings cover the base codec, actual entropy coding and all encoder-side and decoder-side reconstruction operations. Component-wise parameters/runtime, MACs and peak memory are missing. Including DCVC-UF and GLC-Video would quantify the residual framework’s additional cost. Consistent end-to-end tests on RTX 4090 and A100 would further clarify hardware sensitivity and practical efficiency.

  5. The visual evidence does not adequately establish overall viewing quality or temporal stability. Although FloLPIPS and cross-frame examples are provided, Fig. 7 emphasizes a single-frame printed-texture crop, while Fig. 12 mainly examines rigid background textures. These examples insufficiently cover moving objects, deformation, fast motion and occlusions. No user study is reported. Matched-bitrate video playback comparisons and subjective evaluation are needed, as improved LPIPS/DISTS alone do not establish a better viewing experience.

  6. Diffusion-specific gains are not isolated from Transformer modeling and iterative residual refinement. Table 3 compares VideoPiT with a CNN entropy model, but this does not distinguish diffusion-specific benefits from differences in architecture and capacity. Comparisons against a capacity-matched non-diffusion Transformer and conventional iterative residual refinement under matched training and rate–distortion settings are missing. The advantage of diffusion-based entropy modeling and denoising-path decomposition therefore remains insufficiently justified.

Questions for authors
  1. VideoPiT training. Is VideoPiT optimized solely through the rate–distortion objective in Eq. (13), or does it receive additional flow-matching supervision or explicit timestep/noise-level conditioning?

  2. Configuration changes. Can resolution, GOP size and the residual-coding/generation split be changed using the same checkpoint, or do they require retraining?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ikzF (Part 1)

Thank you for your detailed review and thoughtful feedback. Your questions regarding the depth of our analysis are very important, and we appreciate the opportunity to clarify our work.

W1: Additional baselines

According to your advice, we have included GVC-RT, GVC1D, and S2VC as additional baselines. Table R2.1 reports the BD-rate and BD-metric results, together with the coding speed measured as the average per-frame encoding and decoding time on an NVIDIA H20 GPU. As shown in Table R2.1, ResGVC demonstrates substantial improvements on most perceptual metrics and achieves the best performance in terms of PSNR and MS-SSIM.

Moreover, ResGVC achieves competitive coding speed compared with S2VC, GVC1D, and YODA, while falling behind GVC-RT, which prioritizes a lightweight architecture at the non-trivial expense of rate-perception performance. Besides, we suggest that generative video codecs represent a valuable new paradigm relative to conventional codecs and fidelity-oriented neural codecs. Given the known complexity costs associated with generative models (e.g., diffusion models and autoregressive models), we would first like to explore the rate-perception performance ceiling of ResGVC under relaxed complexity constraints, before progressively introducing practical complexity budgets.

Table R2.1. BD-rate (%) ↓\downarrow / BD-metric ↑\uparrow averaged over UVG, HEVC B, and MCL-JCV. “N/A” arises from insufficient overlap between rate-metric curves.

MethodLPIPSDISTSFIDFloLPIPSPSNRMS-SSIMEnc. / Dec. Time (ms)
GVC-RT103.2 / -0.034N/A / -0.029-20.6 / 4.2130.6 / -0.040-15.7 / 0.171-75.6 / 0.0729.8 / 23.4
GVC1D-26.8 / 0.008N/A / -0.020-61.4 / 11.6-29.7 / 0.009-62.9 / 1.859-66.5 / 0.0561025.8 / 755.5
S2VC382.8 / -0.064541.7 / -0.039-52.7 / 9.8600.4 / -0.068-75.8 / 3.143N/A / 0.077182.8 / 1799.5
YODA0.0 / 0.0000.0 / 0.0000.0 / 0.00.0 / 0.0000.0 / 0.0000.0 / 0.0001138.8 / 729.6
ResGVC (Ours)-63.7 / 0.030-21.2 / 0.003-67.3 / 13.3-26.2 / 0.008-83.8 / 4.379-84.5 / 0.086346.1 / 771.7

W2: Base-codec performance

According to your advice, we provide the evaluation results of the pretrained DCVC-UF, together with those of the fine-tuned DCVC-UF trained on data and rate-distortion objectives matched to ResGVC. As shown in Table R2.2, ResGVC outperforms the fine-tuned DCVC-UF across all metrics, thereby demonstrating the effectiveness of the proposed residual coding and residual generation framework. Moreover, ResGVC significantly surpasses the pretrained DCVC-UF on perceptual metrics at the cost of a relatively small reduction in PSNR and MS-SSIM, because the pretrained DCVC-UF is designed to optimize fidelity-oriented metrics only.

Table R2.2. BD-rate (%) ↓\downarrow / BD-metric ↑\uparrow averaged over UVG, HEVC B, and MCL-JCV. “N/A” arises from insufficient overlap between rate-metric curves.

MethodLPIPSDISTSFIDFloLPIPSPSNRMS-SSIM
DCVC-UFN/A / -0.184N/A / -0.108425.7 / -32.9N/A / -0.199-70.4 / 2.901-60.9 / 0.029
Fine-tuned DCVC-UF50.3 / -0.01670.7 / -0.01942.6 / -10.140.5 / -0.01614.9 / -0.4612.2 / -0.024
ResGVC (Ours)0.0 / 0.0000.0 / 0.0000.0 / 0.00.0 / 0.0000.0 / 0.0000.0 / 0.000
AuthorsOpenReview ↗

Response to Reviewer ikzF (Part 2)

W3 & W4: Detailed complexity analysis

Currently, we have no access to A800 GPUs to test our reimplementation of the closed-source method GNVC-VD. To address your concerns, we replace GNVC-VD with recent open-source baselines, including DCVC-FM, DCVC-UF, GLC-Video, GVC-RT, S2VC, GVC1D, and YODA, and ensure that all methods are evaluated under the same test environment. As presented in Table R2.3 and Table R2.4, ResGVC exhibits competitive coding speed relative to GVC baselines such as S2VC, GVC1D, and YODA, with the exception of GVC-RT, which employs a lightweight architecture at a non-trivial cost to rate-perception performance.

The deployment details are provided below to address your concerns point by point:

  • For resolution, all tests are conducted on sequences with a resolution of 1920$\times$1080.

  • For model configuration, inference pipeline, precision, CUDA/kernel settings, and optimization details, each baseline is evaluated using its official open-source implementation. Our model is configured with DCVC-UF as the learned video codec. In the proposed residual codec, the analysis transform gag_\mathrm{a} and synthesis transform gsg_\mathrm{s} are built upon the encoder and decoder architecture of DCVC-UF, while VideoPiT consists of six PiT blocks from [1]. The model implementation details are provided in Appendix A.1. Our inference pipeline is illustrated in Figure 2 of the manuscript. The total number of denoising steps TT is set to 5, and the skip mode described in Section 3.3 allocates two steps to residual coding and three steps to residual generation. Our model is tested using 16-bit floating-point precision, PyTorch 2.9.1, and CUDA 12.8, with optimization limited to FlashAttention [2] which is commonly adopted for attention computation.

  • The reported 593.9M parameters include DCVC-UF. The measured timings cover the entire inference pipeline, including the base codec, actual entropy coding, and all encoder-side and decoder-side reconstruction operations.

  • MACs and peak memory usage are reported in Table R2.4. Component-wise parameters, runtime, and MACs are reported in Table R2.5.

  • DCVC-UF and GLC-Video have been included in the complexity comparisons in Table R2.3 and Table R2.4.

  • In addition to H20, we also report the encoding and decoding time measured on H100, as shown in Table R2.3. At present, we have no access to RTX 4090 or A100 GPU resources within our institution. We therefore use H100 as an alternative, which is also used in [3].

Table R2.3. Encoding / decoding time (ms) comparisons on 1920$\times$1080 sequences. The time is averaged over both intra- and inter-frames in the sequences.

MethodH20 (96GB)H100 (80GB)
DCVC-FM207.5 / 167.6142.8 / 119.3
DCVC-UF7.2 / 7.91.7 / 2.0
GLC-Video470.4 / 297.7249.1 / 160.3
GVC-RT9.8 / 23.46.3 / 13.4
S2VC182.8 / 1799.589.2 / 809.5
GVC1D1025.8 / 755.5228.4 / 180.3
YODA1138.8 / 729.6513.8 / 324.2
ResGVC (Ours)346.1 / 771.7137.2 / 323.9

Table R2.4. MACs, model parameters, and peak GPU memory on 1920$\times$1080 sequences. Model parameters count both intra- and inter-models.

MethodEnc. KMACs/pixelDec. KMACs/pixelParams (M)Peak GPU Memory (GB)
DCVC-FM1130.6860.444.910.1
DCVC-UF142.3148.2162.72.7
GLC-Video3189.02084.7225.814.4
GVC-RT145.7335.4135.23.7
S2VC1647.09346.71842.322.8
GVC1D8425.16389.51771.674.1
YODA10411.96856.52050.336.3
ResGVC (Ours)10326.825360.2593.919.5

Table R2.5. Component-wise parameters, runtime, and MACs of ResGVC on 1920$\times$1080 sequences. Runtime is measured on an NVIDIA H20 GPU.

ComponentParams (M)Runtime (ms)KMACs/pixel
VideoPiT359.7141.04896.3
Analysis transform gag_\mathrm{a}15.175.030.3
Synthesis transform gsg_\mathrm{s}56.2911.1146.1
Learned video codec162.7218.0329.6

Note that our learned video codec has higher MACs than the DCVC-UF baseline because it additionally reconstructs pixel-space frames at the encoder side to provide context for residual coding.

AuthorsOpenReview ↗

Response to Reviewer ikzF (Part 3)

W5: Viewing experience

According to your advice, we conduct a user study to compare ResGVC with the baselines, including YODA and DCVC-UF. Tables R2.6 and R2.7 demonstrate the superiority of ResGVC in viewing experience in the aspects of spatial detail naturalness, temporal consistency, and overall visual quality.

User study details: Participants view the original video alongside two reconstructed videos from ResGVC and a baseline, which are played synchronously. We randomly sample bitrate-matched videos from the HEVC B, UVG, and MCL-JCV datasets to construct 30 test triplets for each baseline. We invite 25 participants aged 20-40 to indicate their preference on overall visual quality, spatial detail naturalness, and temporal consistency.

Moreover, we provide cross-frame visual examples covering moving objects, deformation, fast motion, and occlusions, as shown in Fig. 8, Fig. 17, Fig. 18, and Fig. 19 in our revised manuscript. These cases also demonstrate the effectiveness of our method in perceptual naturalness and temporal consistency.

Table R2.6. User preference for ResGVC compared with DCVC-UF.

ResGVCNeutralDCVC-UF
Spatial detail naturalness83%6%11%
Temporal consistency39%37%24%
Overall visual quality71%17%12%

Table R2.7. User preference for ResGVC compared with YODA.

ResGVCNeutralYODA
Spatial detail naturalness82%12%6%
Temporal consistency78%15%7%
Overall visual quality88%4%8%

W6: Disentangling diffusion process and entropy model architecture

We thank the reviewer for this insightful point. We first scale down our transformer-based entropy model (Params: 359.7M) to a lightweight transformer (Params: 60.7M) to match the capacity of the CNN-based entropy model (Params: 62.9M) for a fair comparison. We then train these models with one and five denoising steps, respectively, while keeping the other training configurations the same. The setting with one denoising step represents a non-diffusion process without iterative denoising, whereas the setting with five denoising steps represents the iterative residual refinement.

Table R2.8 presents the test results of these models under different denoising-step settings. Comparing Test (f) with Test (c), the lightweight transformer outperforms the CNN-based model under the multi-step denoising setting. Comparing Test (d) with Test (a), it outperforms the CNN-based model under the one-step residual generation (RG) setting while comparing Line (e) with Test (b), it performs slightly worse under the one-step residual coding (RC) setting.

These observations indicate that under one-step denoising, the transformer architecture is better suited for RG than for RC. As the number of steps increases, the transformer shows a greater advantage over the CNN, which further demonstrates the effectiveness of the proposed iterative residual refinement coupled with a transformer-based entropy model.

Model simplification details: We reduce the number of PiT blocks [1] in VideoPiT from six to two, and replace the pixel-wise adaptive layer normalization (AdaLN) with the patch-wise (16$\times$16) AdaLN.

Table R2.8. BD-rate (%)↓\downarrow comparisons of entropy models under different denoising step settings.

TestEntropy Model ArchitectureParams (M)Total denoising stepsRC / RG stepsLPIPSDISTSPSNRFloLPIPS
(a)CNN62.910 / 134.541.015.829.2
(b)CNN62.911 / 015.118.45.013.6
(c)CNN62.952 / 317.126.612.218.0
(d)Lightweight Transformer60.710 / 129.234.714.725.1
(e)Lightweight Transformer60.711 / 016.522.07.218.4
(f)Lightweight Transformer60.752 / 37.810.24.28.8
(g)Transformer359.710 / 118.626.17.322.5
(h)Transformer359.711 / 010.714.76.99.4
ResGVCTransformer359.752 / 30.00.00.00.0

Q1: VideoPiT training

VideoPiT is optimized using the rate–distortion objective in Eq.(13), with explicit timestep conditioning within each PiT block following the original design in [1]. We do not employ additional flow-matching supervision, such as the flow-matching loss, because for the compression task addressed by our method, we intend for the optimization to focus exclusively on the rate–distortion objective.

Reviewer_JwyeOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a residual coding framework for generative video compression that combines residual coding and residual generation along a denoising trajectory. The approach achieves promising rate-distortion performance and visual quality compared with existing generative video codecs. However, the experimental evaluation lacks comparisons with the initialized base codec, a sufficiently detailed complexity analysis, and a more comprehensive investigation of the interplay between residual coding and residual generation. Further experiments are needed to establish the benefits of the proposed design under comparable latency and model complexity, as well as full-sequence evaluation settings.

Critical strengths
  1. The proposed method outperforms all compared generative video codecs, with good visual quality.
  2. The paper presents a clean and unified residual coding paradigm, flexibly combining residual coding and residual generation to allocate bitrate along the denoising trajectory.
Critical weaknesses
  1. The paper uses DCVC-UF-L to initialize the learned video codec but does not compare against DCVC-UF-L in the experiments. Comparisons should also account for coding latency. For example, DCVC-FM, VTM-LDB (as evaluated in this paper), and YODA operate at single-frame latency, whereas the proposed method has an 8-frame latency. Given this difference, the paper should also compare against VTM-RA and DCVC-UF-L.
  2. The complexity analysis should be more detailed. The parameter counts, FLOPs (or MACs), and inference speeds of the learned video codec, each RC step, and each RG step should be reported separately. Additionally, FLOPs (or MACs) should be included in Table 2.
  3. The interplay between RC and RG is not sufficiently investigated. Table 4 only demonstrates the best trade-off under a fixed budget of five denoising steps. Would the conclusion hold with fewer or larger total step budgets? Does combining RC and RG consistently outperform using RC or RG alone under different step budgets?
  4. In Table 3, I suspect that the CNN-based entropy model is smaller than the pixel-space RC model, given that the CNN-based model is derived from a DCVC-UF model. Would the DiT-based model still demonstrate a similarly large performance gain if its parameter count were reduced to match that of the CNN-based model?

Considering these, my current stance is borderline, but I would be happy to become more positive if the above concerns are adequately addressed.

Questions for authors
  1. Is the pixel-space diffusion model trained from scratch or initialized from a pretrained model?
  2. I would be interested in seeing the performance of the learned video codec alone (i.e., step 0).
  3. DCVC-UF-L supports evaluation on full sequences, whereas this paper evaluates only the first 96 frames. Could the authors provide comparisons using the full-sequence evaluation setting?
6 comments
AuthorsOpenReview ↗

Response to Reviewer Jwye (Part 1)

Thank you for your insightful questions. We have provided detailed, point-by-point responses and hope they address your concerns effectively.

W1: Additional baselines

According to your advice, we have included DCVC-UF-L and VTM-RA (GOP 8) as additional baselines, each with an 8-frame latency. We notice that the "DCVC-UF-L" you mentioned refers to the same codec as "DCVC-UF" described in our manuscript to initialize the learned video codec. To avoid ambiguity, we uniformly use "DCVC-UF" in our response below.

As shown in Table R1.1, for perceptual metrics, ResGVC demonstrates substantial improvements over DCVC-UF and VTM-RA. For PSNR and MS-SSIM, although ResGVC performs worse than these two fidelity-oriented baselines, the gap remains relatively small, and it still outperforms the generative video compression (GVC) method S2VC which achieves the best fidelity-oriented performance among all GVC baselines.

Table R1.1. BD-rate (%) ↓\downarrow / BD-metric ↑\uparrow averaged over UVG, HEVC B, and MCL-JCV. “N/A” arises from insufficient overlap between rate-metric curves.

MethodLPIPSDISTSFIDFloLPIPSPSNRMS-SSIM
DCVC-UFN/A / -0.137N/A / -0.09365.8 / -9.4N/A / -0.178-94.1 / 7.265-91.9 / 0.109
VTM-RAN/A / -0.131N/A / -0.08922.3 / -3.3N/A / -0.158-95.0 / 7.658-93.4 / 0.111
S2VC382.8 / -0.064541.7 / -0.039-52.7 / 9.8600.4 / -0.068-75.8 / 3.143N/A / 0.077
YODA0.0 / 0.0000.0 / 0.0000.0 / 0.00.0 / 0.0000.0 / 0.0000.0 / 0.000
ResGVC (Ours)-63.7 / 0.030-21.2 / 0.003-67.3 / 13.3-26.2 / 0.008-83.8 / 4.379-84.5 / 0.086

W2: Detailed complexity analysis

According to your advice, we provide a detailed complexity analysis of ResGVC in Table R1.2, which includes the number of parameters, KMACs/pixel, and inference speed of the learned video codec, each RC step, and each RG step. As shown in Table R1.2, compared with the learned video codec, the RC and RG steps are more computationally intensive, because they incorporate VideoPiT which entails non-trivial complexity.

The MACs have been reported in Table 2 which is also presented here as Table R1.3. It is noted that ResGVC exhibits relatively higher MACs than recent GVC baselines, including YODA, GVC1D, and S2VC, while its coding time remains competitive. This efficiency could be attributed to our architectural design, where most of the computational burden is concentrated in VideoPiT, whose attention computations can be effectively accelerated using FlashAttention [1].

Table R1.2. Component-wise complexity on 1920$\times$1080 sequences. Runtime is measured on an NVIDIA H20 GPU.

ComponentParams (M)Runtime (ms)KMACs/pixel
Learned video codec162.718.0329.6
VideoPiT359.7141.04896.3
gag_\mathrm{a}15.25.030.3
gsg_\mathrm{s}56.311.1146.1
Residual entropy encoding (REE)---1.9---
Residual entropy decoding (RED)---0.9---
Each RC step: encoding (VideoPiT + gag_\mathrm{a} + gsg_\mathrm{s} + REE)431.2159.05072.7
Each RC step: decoding (VideoPiT + gsg_\mathrm{s} + RED)416.0153.05042.4
Each RG step (VideoPiT + gsg_\mathrm{s})416.0152.15042.4

Table R1.3. Complexity comparisons on 1080$\times$1920 sequences. “Enc. / Dec. Time” denotes the per-frame encoding/decoding time, measured on an NVIDIA H20 GPU. Model parameters include both the intra- and inter-models.

MethodEnc. / Dec. Time (ms)Enc. KMACs/pixelDec. KMACs/pixelParams (M)Peak GPU Memory (GB)
DCVC-FM207.5 / 167.61130.6860.444.910.1
DCVC-UF7.2 / 7.9142.3148.2162.72.7
GLC-Video470.4 / 297.73189.02084.7225.814.4
GVC-RT9.8 / 23.4145.7335.4135.23.7
S2VC182.8 / 1799.51647.09346.71842.322.8
GVC1D1025.8 / 755.58425.16389.51771.674.1
YODA1138.8 / 729.610411.96856.52050.336.3
ResGVC (Ours)346.1 / 771.710326.825360.2593.919.5

Note that our learned video codec has higher MACs than the DCVC-UF baseline because it additionally reconstructs pixel-space frames at the encoder side to provide context for residual coding.

AuthorsOpenReview ↗

Response to Reviewer Jwye (Part 2)

W3: Interplay between RC and RG

Thanks for your inspiring questions. We have conducted experiments to compare performance under different total denoising step budgets T=1,3,5,7,9T=1,3,5,7,9. As presented in Table R1.4, combining RC and RG consistently outperforms using RC or RG alone for T=3,5,7,9T=3,5,7,9. For T=1T=1, allocating the only denoising step to RC achieves better performance.

These observations demonstrate that, under a fixed total number of denoising steps, appropriately allocating the RC and RG steps by the proposed skip mode of the residual codec yields optimal performance. Furthermore, by exploiting the iterative denoising property of diffusion, our proposed multi-step residual refinement can surpass approaches that transmit or generate the residual signal only once.

Table R1.4. BD-rate (%) ↓\downarrow w.r.t. LPIPS / DISTS / PSNR / FloLPIPS. For each TT, the best result is highlighted in bold. “Optimal RC + RG steps” denotes the allocation of RC and RG steps that achieves the best BD-rate for each TT.

SettingT=1T=1T=3T=3T=5T=5T=7T=7T=9T=9
Pure RC10.7 / 14.7 / 6.9 / 9.48.1 / 9.6 / 3.8 / 8.56.7 / 7.3 / 3.4 / 4.25.9 / 7.5 / 0.7 / 3.64.3 / 4.2 / -0.4 / 2.9
Pure RG18.6 / 26.1 / 7.3 / 22.513.7 / 15.5 / 8.8 / 15.011.4 / 11.6 / 9.2 / 12.810.3 / 9.9 / 11.7 / 9.010.1 / 9.4 / 15.6 / 9.1
Combining RC and RG---5.1 / 6.3 / 3.1 / 4.20.0 / 0.0 / 0.0 / 0.0-1.9 / -2.6 / -2.0 / -2.7-2.3 / -3.5 / -3.3 / -3.0
Optimal RC + RG steps---1 + 22 + 34 + 35 + 4

W4: Comparisons of CNN-based and DiT-based entropy models with matched parameter counts

According to your advice, we scale down our DiT-based model (Params: 359.7M) to a lightweight transformer architecture (Params: 60.7M) to match the parameter count of the CNN-based model (Params: 62.9M). We train these models under the same training configuration for a fair comparison. As shown in Table R1.5, the lightweight DiT-based model still outperforms the CNN-based model, thereby demonstrating the superiority of our DiT-based entropy model for residual coding and residual generation over a CNN-based entropy model.

Table R1.5. BD-rate (%)↓\downarrow comparisons of different entropy models. All models are tested under our default setting, i.e., two steps for residual coding and three steps for residual generation.

Entropy Model ArchitectureParams (M)LPIPSDISTSPSNRFloLPIPS
CNN62.917.126.612.218.0
Lightweight Transformer60.77.810.24.28.8
Transformer (Ours)359.70.00.00.00.0

Model simplification details: We reduce the number of PiT blocks [1] in VideoPiT from six to two, and replace the pixel-wise adaptive layer normalization (AdaLN) with the patch-wise (16$\times$16) AdaLN.

Q1: Pixel-space diffusion model training

The pixel-space diffusion model is trained from scratch, as described in Appendix A.2.

Q2: Performance of the learned video codec alone

We report the performance of the learned video codec alone (i.e., step 0) in Table R1.6, which shows that relying solely on the learned video codec leads to clear degradation across all metrics compared with ResGVC. This demonstrates the effectiveness of the proposed residual coding and generation framework.

Table R1.6. BD-rate (%) ↓\downarrow averaged over UVG, HEVC B, and MCL-JCV.

MethodLPIPSDISTSPSNRFloLPIPS
Learned video codec alone (step 0)69.293.79.652.5
ResGVC (step 5)0.00.00.00.0
AuthorsOpenReview ↗

Response to Reviewer Jwye (Part 3)

Q3: Full-sequence evaluation

Sure. As shown in Table R1.7, we report full-sequence evaluation results averaged over UVG, HEVC B, and MCL-JCV. Compared with the fidelity-oriented NVC method DCVC-UF, ResGVC still shows significant improvements in perceptual metrics. Although ResGVC performs worse than DCVC-UF on fidelity-oriented metrics, the gap is relatively small, and it can significantly outperform the recent GVC method YODA.

Table R1.7. BD-rate (%)↓\downarrow / BD-metric$\uparrow$ averaged over UVG, HEVC B, and MCL-JCV with full-sequence testing. “N/A” arises from insufficient overlap between rate-metric curves.

MethodLPIPSDISTSFIDFloLPIPSPSNRMS-SSIM
VTM-LDBN/A / -0.152N/A / -0.09966.0 / -8.4N/A / -0.179-91.7 / 7.043-88.9 / 0.101
VTM-RAN/A / -0.133N/A / -0.08769.8 / -7.7N/A / -0.159-94.0 / 7.488-91.6 / 0.104
DCVC-UFN/A / -0.136N/A / -0.09279.0 / -8.1N/A / -0.176N/A / 7.456-93.1 / 0.107
YODA0.0 / 0.0000.0 / 0.0000.0 / 0.00.0 / 0.0000.0 / 0.0000.0 / 0.000
ResGVC (Ours)-60.6 / 0.027-10.7 / 0.001-69.4 / 12.1-27.8 / 0.008-80.9 / 4.196-82.5 / 0.083

References

[1] Yongsheng Yu, Wei Xiong, Weili Nie, Yichen Sheng, Shiqiu Liu, and Jiebo Luo. Pixeldit: Pixel diffusion transformers for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14273–14282, 2026.

Comments on the paper

2 comments
Area_Chair_AGJGOpenReview ↗

Closing discussion after the rebuttal

Thank you to the authors for the rebuttal and to the reviewers for engaging in the discussion. As we wrap up, I’d appreciate a brief final assessment from each reviewer. After reading the rebuttal and the discussion so far, what is the main reason you lean toward acceptance or rejection, and what evidence supports that view? Please note which concerns the rebuttal resolved and which remaining concerns, if any, are decisive for your recommendation.

If your assessment differs from another reviewer’s, please point to the specific claim or evidence you interpret differently. Authors are welcome to clarify any factual points that remain open. This will help us identify where we agree and understand the reasons for any remaining disagreement.

AuthorsOpenReview ↗

General Response

Dear Reviewers,

We sincerely appreciate your effort and thoughtful feedback on ResGVC. We have provided point-by-point responses to all the concerns raised and carefully revised the manuscript based on the constructive suggestions. The modifications of the manuscript are highlighted in blue.

We believe that these responses help clarify and strengthen the manuscript, and we would greatly appreciate it if you could take a look.

Thank you again for your valuable time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.