acceptodds
Under review as a conference paper at ICLR 2027

GENERATIVE VIDEO COMPRESSION FOR SPATIAL DETAIL AND TEMPORAL CONSISTENCY

Abstract

Generative video compression relies on visual priors to restore detail at low bitrate, while maintaining temporal consistency across frames remains challenging. Frame-wise synthesis with image generative priors can produce inconsistent textures even with temporal conditioning, resulting in flickering artifacts. In this paper, we present GVC-ST, a generative video compression framework for rich spatial detail and stable temporal consistency. GVC-ST divides the video sequence into a reference chain and non-reference frames with distinct responsibilities. The reference chain propagates recovered detail and maintains temporal consistency across the sequence, where each frame is reconstructed by one-step diffusion with forward attention to the preceding reconstructed reference frame. For non-reference frames, each frame is reconstructed from its neighboring references and can optionally be replaced by frame interpolation, achieving high detail without affecting the reference chain. Experimental results on three benchmarks show that GVC-ST achieves the best FID among all compared generative codecs while delivering substantially less temporal flickering than prior generative approaches, as measured by and .

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Generative Video Compression, Learned Video Compression, Diffusion Models, Temporal Consistency, Perceptual Video Quality

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AbNnOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the authors divide frames into a reference chain and non-reference frames. Reference frames are responsible for propagating detail and maintaining temporal consistency, while non-reference frames use bidirectional latent refinement from neighboring references and adaptively choose between coding and interpolation. The method is not lightweight; but overall it is a generative compression framework with a clear coding structure. Although many components are existing ones, the combination of "reference/non-reference division + forward reference attention + adaptive interpolation for non-reference frames" is meaningful, and the ablations support the role of each module.

Critical strengths
  1. Temporal consistency is indeed the core pain point in generative compression.

  2. The coding structure is a good idea, and the division of labor is reasonable.

  3. The main results show the best FID, and Ewarp/CLIP-F also show temporal consistency advantages; the ablation table makes the contribution of each component visible. The adaptive coding/interpolation examples are convincing.

Critical weaknesses
  1. Computational complexity is the most substantial weakness of this paper, and the main text does not address it head-on.

  2. The main text lacks fidelity metrics, so the rate-distortion-perception trade-off is incompletely presented.

  3. The evaluation metrics are coupled with the training or selection objectives.

  4. The method is a system integration of a large model and multiple modules, and component sensitivity is not discussed. The ablation never isolates the effect of the fixed reference interval.

Questions for authors
  1. The compute cost is the deciding factor for the paper. Both the encoder and decoder are very slow. The authors acknowledge in the limitation section that "DiT dominates," but the main text does not give runtime, memory, or multiples relative to VTM/DCVC-FM. I wonder whether this is a quality-is-great-but-unusable method.

  2. The main tables are mainly LPIPS/FID/Ewarp/CLIP-F, while PSNR/MS-SSIM are placed in the appendix or curves. This makes me feel the paper is avoiding the question of whether the generated content deviates from the original frames. PSNR does not have to be a strength, but it needs to appear in the paper. LPIPS is used in Stage II training, and Ewarp is used in Stage III and mode selection, and then they are used as evaluation metrics. The evaluation metrics are coupled with the training or selection objectives.

  3. Due to compute constraints, reporting only one rate point is understandable, but the paper cannot therefore claim "best among all compared generative codecs" without qualification. At the very least, the claim should be narrowed to best among generative codecs compared at full multi-rate settings.

  4. The proposed method is still a system built on a strong pretrained backbone plus multi-stage training. Why is this structure necessary for generative video compression, rather than just an effective combination of existing modules? The current ablation shows it works, but the theoretical analysis of why it works is not fully satisfying. If FLUX.2 were replaced with another DiT backbone, how much of the core gain would be expected to remain?

3 comments
AuthorsOpenReview ↗

Response to Reviewer AbNn (Part 1)

Thank you for your careful review and constructive suggestions. We have added runtime and GPU memory comparisons, included PSNR and MS-SSIM results in the main text, and expanded the evaluation with metrics not used for training or mode selection. We have also narrowed the FID claim and added a fully recursive ablation. Below, we explain the design rationale, discuss the limitation of testing only one DiT backbone, and report reference-spacing and selection-weight studies.

Q1, W1. Runtime, GPU Memory, and Practicality

We have added encoding and decoding times and peak GPU memory to the Computational Cost paragraph and Table 3 in the main text. All results below were measured at 1080p on a single NVIDIA A100-SXM4-80GB GPU and include actual entropy coding, decoding, and bitstream packing.

MethodEncoding time (s/frame)Peak encoding memory (GiB)Decoding time (s/frame)Peak decoding memory (GiB)
DCVC-FM0.338.770.278.01
GLC-Video0.436.940.285.25
YODA0.9827.370.6526.62
GVC-ST-BASE0.8120.690.7419.95
GVC-ST1.2231.370.6430.20

GVC-ST and YODA have similar runtimes, with nearly identical decoding times. GVC-ST requires more time and memory than DCVC-FM. Our current implementation does not yet support real-time encoding or decoding. This work focuses on improving perceptual compression efficiency and temporal consistency at low bitrates.

The encoder generates and scores both coding and interpolation candidates, while the decoder runs only the selected reconstruction mode. When interpolation is selected for a non-reference frame, the decoder uses RIFE and skips DiT inference for that frame. As a result, the full method has a shorter decoding time than GVC-ST-BASE.

We retain the discussion of computational cost in the Limitations section. Future work will explore more efficient generative backbones and mode selection without generating both candidates, to reduce time and memory costs.

Q2, W2–3. Reconstruction Fidelity and Metrics Coupled with Optimization

We have added PSNR/MS-SSIM RD curves and results across multiple rates to Figure 4 and Table 2 on page 8 of the main text. On HEVC-B, UVG, and MCL-JCV, GVC-ST has better overall PSNR and MS-SSIM results than GLC-Video and YODA.

To address the concern about using the same metrics for optimization and evaluation, we also use FID, KID, CLIP-F, and DINOv2 consistency. FID and CLIP-F are reported in Table 1 in the main text; KID and DINOv2 consistency are reported in Figure 10 and Table 5 in the appendix. These scores are not used directly as training losses or for RDO mode selection.

Across all three datasets, GVC-ST has better overall results on all four metrics than GLC-Video and YODA. This shows that the gains also appear in evaluation scores that we do not directly optimize, beyond LPIPS and EwarpE_{\mathrm{warp}}.

Q3. Limitations of the GNVC-VD Comparison at a Single Operating Point

We have narrowed the claim in the abstract, introduction, and Section 4.2 to "the best FID BD-Metric among the open-source generative baselines". This wording clearly defines the methods and metric being compared.

Since GNVC-VD has no public code, we reimplemented it following the architecture and training procedure described in its paper. Due to limited computing resources, we trained one model at a single rate setting and reported one operating point per dataset. The revised paper states that GNVC-VD is excluded from the multi-rate BD comparison in Table 1. We compare with GNVC-VD only near its reported operating points in Figure 3.

AuthorsOpenReview ↗

Response to Reviewer AbNn (Part 2)

Q4. Design Rationale and DiT Backbone Dependence

We explain the design through the link between reference dependencies and the optimization objective. To study this, we added fully recursive configuration F in Table 4. Every frame is coded and updates the codec and generative reference states used by later frames. Each frame therefore recovers details and serves as a reference. Changes in its generated details affect the reference conditions for later generation, linking local perceptual optimization with later reconstruction.

With role separation, the next reference frame depends directly on the previous reference frame. A non-reference frame is reconstructed after both neighboring references are available. Refining a non-reference frame or changing its reconstruction mode leaves later reference states unchanged. This allows local detail recovery while both references constrain transitions to neighboring frames.

This also lets us split mode selection into independent decisions. Given reference reconstructions R\mathcal R and fixed candidates, let Jt(mt∣R)J_t(m_t\mid\mathcal R) denote the cost for non-reference frame tt, including rate, perceptual quality, and temporal consistency with both neighboring references. Each frame's choice leaves the reference conditions for other choices unchanged, so:

= \sum_t\min_{m_t}J_t(m_t\mid\mathcal R).$$ **Thus, choosing each frame's mode independently minimizes the total selection cost for the fixed reference chain and candidate set.** Forward reference attention propagates details. Bidirectional refinement improves local candidates under fixed reference conditions. Adaptive selection balances rate, perceptual quality, and temporal consistency. Stage III freezes the codec and one-step reconstruction module and trains only the local refinement module, keeping the reference conditions fixed. In the experiments, **E has lower LPIPS and whole-sequence $E_{\mathrm{warp}}$ BD-Metrics than fully recursive F in Table 4. This supports the combined benefit of role separation and adaptive synthesis.** We also compare GVC-ST-BASE and GVC-ST using the same FLUX.2 pretrained backbone. Table 1 shows overall improvements on all four metrics across three datasets, showing that the design adds gains on top of the same pretrained prior. **Computing resources are limited, and adapting and training a different DiT takes considerable time. We have therefore not yet compared multiple DiT backbones and cannot say how much of the gain would remain after changing the backbone.** We state this limitation on page 10 of the revised paper, in the Limitations section. Future work will explore lighter DiT backbones to reduce computational cost and further evaluate our method across different generative backbones.
AuthorsOpenReview ↗

Response to Reviewer AbNn (Part 3)

W4. Component Sensitivity and Fixed Reference Spacing

We added sensitivity experiments on reference spacing and mode-selection weights to examine their effects separately. All tests cover all 30 MCL-JCV videos at four rate points, using 96 frames per video.

Reference spacing. We used configuration D with frozen model weights, no bidirectional refinement, no per-frame adaptive synthesis, and a GOP length of 32. We tested fixed spacings of 1, 2, 3, and 4. Spacing 2 matches the structure used in the paper. GOP-level adaptation generates all four candidates and selects the spacing with the lowest cost, R+0.68,LPIPS‾+λt,Ewarp‾R+0.68\\,\overline{\mathrm{LPIPS}}+\lambda_t\\,\overline{E_{\mathrm{warp}}}. To compare E_warp at similar LPIPS coding efficiency, we adjusted the temporal weight on this test set. We chose λt=0.01236\lambda_t=0.01236, giving an LPIPS BD-Rate close to fixed spacing 3. Here, RR is actual BPP, both metrics are GOP averages, and the spacing experiment uses RAFT-Small for E_warp. The baseline is fixed spacing 2:

Reference spacingLPIPS BD-Rate ↓PSNR BD-Metric ↑E_warp BD-Metric ↓CLIP-F BD-Metric ↑
Fixed 1+3.92%+0.38 dB+15.59+0.0002
Fixed 2 (spacing used in the paper)0.00%0.00 dB0.000.0000
Fixed 3−9.14%−0.63 dB−4.60+0.0018
Fixed 4+10.78%−1.06 dB−6.97+0.0022
GOP-level adaptive−9.16%−0.37 dB−8.33+0.0020

Fixed spacing 3 gives the best perceptual coding efficiency among the four fixed settings. Increasing the spacing to 4 improves temporal metrics but worsens LPIPS coding efficiency and PSNR. GOP-level adaptation gives a lower E_warp BD-Metric at an LPIPS BD-Rate close to fixed spacing 3, showing the potential of adaptive reference structures. Future work will explore more efficient content-based spacing selection to reduce the cost of generating multiple candidates.

Selection weights. The paper uses βperc=0.68\beta_{\mathrm{perc}}=0.68 and βtemp=0.02\beta_{\mathrm{temp}}=0.02, selected by offline search over a large set of videos. All videos, rates, and datasets use these weights. We tested each weight from 1/100 to 100 times its paper value to examine changes across two orders of magnitude in each direction. We kept models and candidates fixed, changed one weight at a time, and kept the other at its paper value. Results are relative to the paper weights. LPIPS uses BD-Rate and E_warp uses BD-Metric; lower is better for both.

We fixed the temporal weight at 0.02 and varied the perceptual weight:

Weight multiplierPerceptual weightTemporal weightLPIPS BD-Rate ↓E_warp BD-Metric ↓
1/1000.00680.02+0.588%+0.0378
1/500.01360.02+0.588%+0.0378
1/100.0680.02+0.519%+0.0300
1 (paper weights)0.680.020.000%0.0000
106.80.02−2.317%+0.5197
50340.02+1.529%+7.4635
100680.02+4.452%+10.5344

We fixed the perceptual weight at 0.68 and varied the temporal weight:

Weight multiplierPerceptual weightTemporal weightLPIPS BD-Rate ↓E_warp BD-Metric ↓
1/1000.680.0002−0.434%+6.3516
1/500.680.0004−1.858%+4.0479
1/100.680.002−2.207%+0.0528
1 (paper weights)0.680.020.000%0.0000
100.680.2+0.428%+0.0196
500.681+0.472%+0.0219
1000.682+0.472%+0.0219

Reducing the perceptual weight or increasing the temporal weight caused small changes; changes in the opposite directions could have larger effects. Sensitivity therefore depends on the direction of the change. The paper weights give the lowest E_warp BD-Metric among the tested settings. Some other settings improve LPIPS coding efficiency but increase E_warp BD-Metric. These results support the paper weights as a good balance between perceptual compression efficiency and temporal consistency.

Closing remarks

Thank you again for your careful review. We hope these clarifications and additional results address your concerns and help with your final assessment. We would be happy to provide any further clarification.

Reviewer_fHY1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GVC-ST is a generative video compression framework designed to improve spatially-aware perceptual quality and temporal consistency at low bitrates. It adopts different reconstruction strategies for reference and non-reference frames. Forward-reference attention propagates visual details through reference frames, while bidirectional-reference attention further refines non-reference frames. An adaptive frame synthesis mechanism jointly optimizes explicit encoding and frame interpolation. Experiments on three datasets show that GVC-ST outperforms the compared methods in perceptual compression performance.

Critical strengths
  1. Separating reference and non-reference frames is a reasonable design choice. By keeping non-reference frames out of the reference chain, the method avoids propagating their reconstruction errors to subsequent frames. Forward reference attention helps preserve visual details across reference frames, while bidirectional refinement improves the temporal consistency of non-reference frames.
  2. The paper evaluates GVC-ST on three standard video compression benchmarks using perceptual quality, distortion, and temporal consistency metrics. GVC-ST achieves the best FID-based BD-metrics among the compared generative codecs and substantially reduces Ewarp compared with GLC-Video and YODA. These results provide evidence for the effectiveness of the proposed approach.
Critical weaknesses
  1. The separation of reference and non-reference frames is intended to prevent error propagation, but it also prevents useful motion and texture information from non-reference frames. This may hurt coding efficiency in scenes with fast motion or occlusions. However, all configurations in Table 2 use the same frame-role separation strategy. It remains unclear whether the benefits of avoiding error propagation outweigh the loss of intermediate frame information.
  2. GNVC-VD is evaluated at only one bitrate, making it difficult to assess its rate-distortion performance across different operating points. There is also a mismatch in the reference structures: GVC-ST uses future reference frames for bidirectional refinement and interpolation, whereas VTM-17.0 is evaluated in a low-delay configuration.
  3. The ablation study does not fully separate the contributions of bidirectional refinement and adaptive synthesis. There is no configuration that retains adaptive synthesis while removing bidirectional refinement, and the paper does not report how often each reconstruction mode is selected at different bitrates or for different video content. Ewarp is also used for mode selection and can be affected by interpolation-induced smoothing and optical flow estimation errors. The method is computationally expensive, as the encoder needs to generate both coded and interpolated candidates. Encoding and decoding times, peak GPU memory usage, and runtime comparisons with baselines are needed to judge whether the quality improvements justify the additional cost.
Questions for authors
  1. Can the authors add an ablation that retains adaptive synthesis but removes bidirectional refinement, and report the proportion of frames selected for interpolation at different bitrates to clarify the contribution of each module?
  2. Can the authors evaluate temporal consistency on longer sequences and more challenging motion scenarios using perceptual metrics or subjective evaluations that are not part of the mode selection objective?
  3. Can the authors report encoding and decoding times, peak GPU memory usage, and the additional cost of generating both reconstruction candidates, with comparisons to existing methods?
3 comments
AuthorsOpenReview ↗

Response to Reviewer fHY1 (Part 1)

Thank you for your careful review and constructive suggestions. We have added an ablation that keeps adaptive synthesis without bidirectional refinement, reported mode selection ratios, and evaluated difficult motion sequences with metrics not used for mode selection. We also provide runtime and GPU memory comparisons, add a fully recursive baseline to examine role separation, and clarify the scope of the GNVC-VD comparison.

Q1, W3. Bidirectional Refinement and Adaptive Synthesis: Ablations and Mode Ratios

We have added configuration E, which keeps adaptive synthesis and removes bidirectional refinement, to Section 4.3 and the ablation table of the revised paper. E selects between the coded candidate from B and the interpolated candidate from D. The full model G instead uses the coded candidate from C, which includes bidirectional refinement. E and G use the same reference frames and selection weights.

We evaluated all 30 MCL-JCV videos at four rate points, using 96 frames per video and a GOP length of 32. All four quality columns below are BD-Metrics relative to A; lower is better.

VariantRole SeparationForward AttentionBidirectional RefinementSynthesis ModeLPIPS ↓EwarpE_{\mathrm{warp}} ↓EwarpRefE_{\mathrm{warp}}^{\mathrm{Ref}} ↓EwarpNonRefE_{\mathrm{warp}}^{\mathrm{NonRef}} ↓
A: GVC-ST-BASE✓××Coding0.00000.000.000.00
B✓✓×Coding−0.0009−0.43−2.77+5.19
C✓✓✓Coding−0.0010−3.96−2.77−11.57
D✓✓×Interpolation−0.0053−19.61−5.59−31.55
E (added)✓✓×Adaptive−0.0063−22.26−5.36−33.73
F: Full recursion×✓×Coding−0.0041−5.60−1.49−3.20
G: Full GVC-ST✓✓✓Adaptive−0.0063−22.56−5.32−34.86

D→E shows that adaptive synthesis provides gains even without bidirectional refinement. E→G further improves the temporal metrics for the whole sequence and non-reference frames, with the same LPIPS BD-Metric at the reported precision.

The following table reports the fraction of eligible non-reference frames selected for interpolation. The remaining frames use coding. Each video has 45 eligible frames at each rate point.

VideosRate pointE: No bidirectional refinementG: Full model
MCL-JCV allrate090.96%89.33%
MCL-JCV allrate2189.41%88.00%
MCL-JCV allrate4288.15%87.04%
MCL-JCV allrate6388.67%88.30%
MCL-JCV allAll four rates89.30%88.17%

Most eligible non-reference frames use interpolation; only a small fraction use coding. Figure 7 in Section 4.3 (page 10) shows examples of both choices.

In the left example, complex motion and deformation cause clear structural errors in the interpolated candidate. Coding better preserves the structure and is therefore selected. In the right example, interpolation gives better perceptual and temporal quality without sending coded data for the current frame, so it is selected. Adaptive synthesis saves bits on most frames and retains coding for frames that interpolation cannot reconstruct well, improving perceptual compression efficiency and temporal consistency.

Q2, W3. Challenging Motion and Independent Temporal Evaluation

We have added evaluations using CLIP-F and DINOv2 consistency. Neither metric is used for mode selection, and neither relies on optical flow. CLIP-F measures semantic feature consistency between adjacent frames. DINOv2 consistency compares visual features between adjacent frames and between each frame and the first frame, providing an additional measure of consistency across frames. We have added DINOv2 consistency results to Appendix A.3 and the five-sequence evaluation to Appendix A.6, Figure 12 of the revised paper.

We used the mean EwarpE_{\mathrm{warp}} of the source videos as an indicator of motion complexity. We ranked all 30 MCL-JCV sequences and selected the five with the highest values: SRC25, SRC22, SRC04, SRC10, and SRC09. This selection uses only the source videos and does not depend on any method's reconstructed frames. We evaluated 96 frames per sequence with a GOP length of 32. GVC-ST uses four rate points.

As shown in Figure 12, within comparable bitrate ranges, GVC-ST generally achieves lower EwarpE_{\mathrm{warp}} and higher CLIP-F than GLC-Video and YODA on these five sequences. DINOv2 consistency generally improves over GVC-ST-BASE on all five sequences. It is also generally higher than GLC-Video and YODA on SRC22, SRC10, and SRC09. These results show gains on metrics that are outside the mode selection objective and do not rely on optical flow.

AuthorsOpenReview ↗

Response to Reviewer fHY1 (Part 2)

Q3, W3. Runtime, GPU Memory, and Dual-Candidate Cost

We added encoding/decoding time and peak GPU memory comparisons to the Computational Cost paragraph and Table 3 on page 9. Results were measured at 1080p on a single NVIDIA A100-SXM4-80GB GPU.

MethodEncoding time (s/frame)Peak encoding memory (GiB)Decoding time (s/frame)Peak decoding memory (GiB)
DCVC-FM0.338.770.278.01
GLC-Video0.436.940.285.25
YODA0.9827.370.6526.62
GVC-ST-BASE0.8120.690.7419.95
GVC-ST1.2231.370.6430.20

GVC-ST and YODA have encoding and decoding times of the same order of magnitude, with nearly identical decoding times.

To quantify the extra cost of the two candidates, we compare C, which generates only the coded candidate, with the full model G. They use the same coded candidate and reference reconstruction chain. G also runs RIFE using the two reference frames to generate an interpolated candidate, then scores and selects between the two. Generating this second candidate does not require another codec or DiT run. Configuration labels follow the paper.

VariantRole SeparationForward AttentionBidirectional RefinementSynthesis ModeEncoding time (s/frame)Peak encoding memory (GiB)Decoding time (s/frame)Peak decoding memory (GiB)
A: GVC-ST-BASE✓××Coding0.8120.690.7419.95
B✓✓×Coding1.1231.201.1130.16
C✓✓✓Coding1.1231.321.1130.17
D✓✓×Interpolation0.7330.910.6430.19
E✓✓×Adaptive1.2331.250.6430.19
G: Full GVC-ST✓✓✓Adaptive1.2231.370.6430.20

In C→G, encoding time increases from 1.12 to 1.22 s/frame. The extra cost is about 0.10 s/frame, including interpolation, perceptual and temporal scoring, mode selection, and packing. Peak encoding memory increases from 31.32 to 31.37 GiB.

The decoder runs only the selected mode. When interpolation is selected for a non-reference frame, it uses RIFE and skips DiT. In this test, decoding time therefore decreases from 1.11 s/frame for C to 0.64 s/frame for G. Together with the quality results in Q1, adaptive synthesis improves perceptual compression efficiency and temporal metrics with about 0.10 s/frame of extra encoding time, while reducing decoding time.

W1. Role Separation: Benefits and Information Loss

In our framework, adaptive synthesis is built on role separation: non-reference frames do not update the reference states used for later coding and generation. This lets us choose coding or interpolation independently for each non-reference frame without changing later predictions.

To test the effect of using intermediate frames for later prediction, we have added fully recursive configuration F to Section 4.3 and Table 4 on page 10 of the revised paper. Every frame in F is coded and updates the reference states. B, E, and F all use forward attention without bidirectional refinement; G is the full model.

The experiment covers all 30 MCL-JCV videos at four rate points, using 96 frames per video and a GOP length of 32. All four quality metrics below are BD-Metrics relative to GVC-ST-BASE (A); lower is better.

VariantRole SeparationForward AttentionBidirectional RefinementSynthesis ModeLPIPS ↓EwarpE_{\mathrm{warp}} ↓EwarpRefE_{\mathrm{warp}}^{\mathrm{Ref}} ↓EwarpNonRefE_{\mathrm{warp}}^{\mathrm{NonRef}} ↓
B✓✓×Coding−0.0009−0.43−2.77+5.19
E✓✓×Adaptive−0.0063−22.26−5.36−33.73
F: Full recursion (added)×✓×Coding−0.0041−5.60−1.49−3.20
G: Full GVC-ST✓✓✓Adaptive−0.0063−22.56−5.32−34.86

The Ref/NonRef columns use the same frame positions in all configurations. In F, these groups are only used to compare matching positions.

B versus F shows that role separation has a performance cost when all frames are coded: B has worse LPIPS and whole-sequence EwarpE_{\mathrm{warp}} than F. We therefore do not attribute the overall gains solely to preventing error propagation.

With role separation, E enables adaptive synthesis to save bits through interpolation while retaining the coded candidate for frames that interpolation cannot reconstruct accurately. E has better LPIPS and whole-sequence EwarpE_{\mathrm{warp}} than fully recursive F. The full model G further improves the temporal metrics for the whole sequence and non-reference frames. These results show that role separation makes independent mode selection possible. This allows adaptive synthesis to make up for the prediction cost and improve overall perceptual compression efficiency and temporal performance.

AuthorsOpenReview ↗

Response to Reviewer fHY1 (Part 3)

W2. Limitations of the GNVC-VD Comparison at a Single Operating Point

We agree that a single operating point cannot show the full rate-distortion performance of GNVC-VD or establish which method is better across the entire bitrate range.

Since GNVC-VD has no public code, we reimplemented it following the architecture and training procedure described in its paper. Due to limited computing resources, we trained one model at a single rate setting and reported one operating point on each of HEVC-B, UVG, and MCL-JCV.

The experimental settings in the revised paper now state that GNVC-VD is excluded from the multi-rate BD comparison in Table 1. Its results are shown as separate points in Figure 3, and we compare with it only near those operating points. We have also narrowed the claim in the abstract, introduction, and Section 4.2 to "the best FID BD-Metric among the open-source generative baselines", making the comparison scope clear.

Closing remarks

Thank you again for your careful review. We hope these clarifications and additional results address your concerns and help with your final assessment. We would be happy to provide any further clarification.

Reviewer_jsijOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GVC-ST, a generative video compression framework that separates reference and non-reference frames. It combines forward reference attention, bidirectional latent refinement, and adaptive selection between coding and frame interpolation to balance perceptual quality and temporal consistency. Experiments on three benchmarks demonstrate improvements over several generative compression baselines.

Critical strengths

1.The method is well motivated, with a clear connection between reference propagation, non-reference refinement, and adaptive frame synthesis. 2.The framework achieves competitive perceptual quality and improved temporal consistency compared with several generative video compression methods.

Critical weaknesses

1.The ablation studies are mainly conducted on one dataset, and the sensitivity of the adaptive selection mechanism to its weighting parameters is insufficiently explored. 2.The framework is computationally expensive due to the large diffusion transformer, while generating and evaluating both coding and interpolation candidates introduces additional overhead. 3.Although temporal consistency improves over several generative baselines, the results do not demonstrate consistent improvements over traditional codecs across all temporal metrics.

Questions for authors

1.How does GVC-ST differ fundamentally from recent generative video compression methods that use hierarchical reference structures, adaptive keyframe selection, or latent grouping? 2.Would adaptive reference-frame spacing improve performance in videos with different motion complexity, compared with the proposed fixed reference structure? 3.Could the authors provide more comprehensive ablations across datasets, sensitivity analyses for the adaptive selection weights, and quantitative runtime and memory comparisons to clarify the robustness and practical efficiency of the proposed framework?

3 comments
AuthorsOpenReview ↗

Response to Reviewer jsij (Part 1)

Thank you for your review. We have expanded the related-work comparison and ablations on HEVC-B and UVG. We also report reference-spacing and weight studies, runtime and GPU memory costs, and temporal comparisons with traditional codecs.

Q1. Fundamental Differences from Recent Generative Video Codecs

Following your suggestion, we reviewed recent related work. These methods do use role separation. GVCHR groups latents by their importance as references, assigns different quality levels to the groups, and restricts reference access. T-GVC separates keyframes from intermediate frames and transmits keyframes and sparse motion trajectories to guide intermediate-frame generation. GoL separates transmitted latent anchors from the latents to be generated, then jointly refines the anchors and reconstructs the remaining latents.

Our design focuses on the trade-off between perceptual compression efficiency and temporal consistency. In generative coding, recovering rich detail can improve the perceptual quality of individual frames. However, changes in generated textures across frames can cause flicker and reduce temporal consistency. We therefore consider both how details are recovered and how they remain consistent over time.

GVC-ST assigns different roles to the two types of frames: the reference chain propagates recovered details and maintains temporal consistency. Non-reference frames recover local details and choose between coding and interpolation without changing the reference chain. Reference frames use one-step diffusion with forward reference attention to pass generated details along the chain. For each non-reference frame, the coded candidate is locally refined using both neighboring references. The encoder then chooses between coding and interpolation based on rate, perceptual quality, and temporal consistency. Non-reference frames do not update subsequent reference states, so their local reconstruction leaves the reference chain unchanged. Together, these steps combine detail recovery in individual frames, detail propagation across frames, and mode selection to jointly optimize perceptual compression efficiency and temporal consistency.

We added these works and their mechanism comparison to Section 2 (Related Work), page 3 of the revised manuscript.

Q2. Adaptive Reference-Frame Spacing

We compared fixed and GOP-level adaptive reference spacing to examine their effects on perceptual compression efficiency and temporal consistency.

  1. Fixed reference spacing. We tested fixed spacings of 1, 2, 3, and 4. Spacing 2 matches the reference structure used in the paper.

  2. GOP-level adaptive spacing. For each GOP, we generated candidates at all four spacings and chose the one with the lowest cost, R+0.68,LPIPS‾+λt,Ewarp‾R+0.68\\,\overline{\mathrm{LPIPS}}+\lambda_t\\,\overline{E_{\mathrm{warp}}}. To compare E_warp at similar LPIPS coding efficiency, we adjusted the temporal weight on this test set. We chose λt=0.01236\lambda_t=0.01236, which gives an LPIPS BD-Rate close to that of fixed spacing 3. This reduces the effect of differences in perceptual quality on the comparison.

We used configuration D with fixed model weights and disabled bidirectional refinement and per-frame adaptive synthesis. We evaluated all 30 MCL-JCV videos at four rate points, using 96 frames per video and a GOP length of 32. In the cost function, RR is the actual BPP. LPIPS and E_warp are averaged within each GOP, and E_warp is computed using RAFT-Small. The table below uses fixed spacing 2 as the baseline:

Reference spacingLPIPS BD-Rate ↓PSNR BD-Metric ↑E_warp BD-Metric ↓CLIP-F BD-Metric ↑
Fixed 1+3.92%+0.38 dB+15.59+0.0002
Fixed 2 (spacing used in the paper)0.00%0.00 dB0.000.0000
Fixed 3−9.14%−0.63 dB−4.60+0.0018
Fixed 4+10.78%−1.06 dB−6.97+0.0022
GOP-level adaptive−9.16%−0.37 dB−8.33+0.0020

Fixed spacing 3 achieves the best perceptual coding efficiency among the four fixed settings. Increasing the spacing to 4 improves the temporal metrics but worsens LPIPS coding efficiency and PSNR. This shows the trade-off between reference-frame bitrate and intermediate-frame reconstruction quality. GOP-level adaptation achieves a lower E_warp BD-Metric at an LPIPS BD-Rate close to that of fixed spacing 3.

These results show the potential of adaptive reference structures to improve the trade-off between perceptual compression efficiency and temporal consistency. Future work will explore more efficient content-based spacing selection and more flexible reference structures to reduce the cost of generating multiple candidates.

AuthorsOpenReview ↗

Response to Reviewer jsij (Part 2)

Q3, W1. Cross-Dataset Ablations

We chose MCL-JCV because its 30 sequences let us evaluate each component on a larger video set. Following your suggestion, we extended the ablations to all five HEVC-B and all seven UVG sequences to test whether the gains hold on other datasets.

We used existing model weights for each configuration, testing four rate points, 96 frames per sequence, and GOP length 32. All datasets and rate points used the same mode-selection weights. Labels follow Table 4 in the revised manuscript. All four metrics are BD-Metrics relative to GVC-ST-BASE (A) on each dataset; lower is better.

HEVC-B

VariantRole SeparationForward AttentionBidirectional RefinementSynthesis ModeLPIPS ↓E_warp ↓E_warp Ref ↓E_warp NonRef ↓
A (BASE)✓——Coding0.00000.000.000.00
B✓✓—Coding−0.0044−5.66−5.34−3.73
C✓✓✓Coding−0.0031−9.65−5.34−18.42
D✓✓—Interpolation−0.0105−29.40−8.17−41.99
E✓✓—Adaptive−0.0105−29.39−8.16−41.87
G (GVC-ST)✓✓✓Adaptive−0.0105−29.36−8.15−41.83

UVG

VariantRole SeparationForward AttentionBidirectional RefinementSynthesis ModeLPIPS ↓E_warp ↓E_warp Ref ↓E_warp NonRef ↓
A (BASE)✓——Coding0.00000.000.000.00
B✓✓—Coding+0.0003−1.29−1.53−0.68
C✓✓✓Coding+0.0007−2.83−1.53−5.44
D✓✓—Interpolation−0.0063−12.98−2.39−16.14
E✓✓—Adaptive−0.0063−12.98−2.39−16.13
G (GVC-ST)✓✓✓Adaptive−0.0063−12.97−2.39−16.13

The full model G improves all four metrics over baseline A on both datasets, showing that the gains of the overall framework also hold beyond MCL-JCV. The B-to-C comparison also shows that bidirectional refinement reduces both overall and non-reference-frame E_warp on both datasets.

The results for D, E, and G are close. On these two datasets, the adaptive method mainly chooses interpolation. This makes its reconstruction process similar to fixed-interpolation setting D. These results support the framework's gains across datasets and show that each component's added benefit depends on video content and interpolation quality.

Q3, W1. Sensitivity to Selection Weights

We added a sensitivity study of the adaptive selection weights. The perceptual and temporal weights were chosen through an offline search over a large set of videos. The paper uses βperc=0.68\beta_{\mathrm{perc}}=0.68 and βtemp=0.02\beta_{\mathrm{temp}}=0.02, with the same weights for all videos, rate points, and datasets.

We varied each weight from 1/100 to 100 times its value in the paper, spanning two orders of magnitude in each direction. This tests how large changes in the weights affect perceptual compression efficiency and temporal consistency. We evaluated all 30 MCL-JCV videos at four rate points, using 96 frames per video. The model and reconstruction candidates were fixed. We varied one weight at a time, keeping the other at its paper value. Results are relative to the paper weights. LPIPS is reported as BD-Rate and E_warp as BD-Metric; lower is better for both.

We fixed the temporal weight at 0.02 and varied the perceptual weight:

Weight multiplierPerceptual weightTemporal weightLPIPS BD-Rate ↓E_warp BD-Metric ↓
1/1000.00680.02+0.588%+0.0378
1/500.01360.02+0.588%+0.0378
1/100.0680.02+0.519%+0.0300
1 (paper weights)0.680.020.000%0.0000
106.80.02−2.317%+0.5197
50340.02+1.529%+7.4635
100680.02+4.452%+10.5344

We fixed the perceptual weight at 0.68 and varied the temporal weight:

Weight multiplierPerceptual weightTemporal weightLPIPS BD-Rate ↓E_warp BD-Metric ↓
1/1000.680.0002−0.434%+6.3516
1/500.680.0004−1.858%+4.0479
1/100.680.002−2.207%+0.0528
1 (paper weights)0.680.020.000%0.0000
100.680.2+0.428%+0.0196
500.681+0.472%+0.0219
1000.682+0.472%+0.0219

In these tests, reducing the perceptual weight or increasing the temporal weight caused only small changes in the results. Opposite changes could have larger effects, showing that sensitivity depends on direction. The weights used in the paper have the lowest E_warp BD-Metric among the tested settings. Some other settings improve LPIPS coding efficiency, but all come with an increase in E_warp BD-Metric. These results support our choice of weights for balancing perceptual compression efficiency and temporal consistency.

AuthorsOpenReview ↗

Response to Reviewer jsij (Part 3)

Q3, W2. Runtime and GPU Memory

We added encoding/decoding time and peak GPU memory comparisons to the Computational Cost paragraph and Table 3 on page 9. Results were measured at 1080p on a single NVIDIA A100-SXM4-80GB GPU.

MethodEncoding time (s/frame)Peak encoding memory (GiB)Decoding time (s/frame)Peak decoding memory (GiB)
DCVC-FM0.338.770.278.01
GLC-Video0.436.940.285.25
YODA0.9827.370.6526.62
GVC-ST-BASE0.8120.690.7419.95
GVC-ST1.2231.370.6430.20

In this test, GVC-ST and YODA have encoding and decoding times of the same order, with nearly identical decoding times. The encoder generates and scores both candidates, while the decoder runs only the selected reconstruction mode. For non-reference frames selected for interpolation, the decoder uses RIFE and skips DiT inference, reducing decoding cost.

W3. Temporal Consistency Compared with Traditional Codecs

We agree that GVC-ST does not consistently outperform traditional codecs on all temporal metrics. Our goal is to recover rich detail at low bitrates while keeping it consistent across frames and reducing flicker. The temporal improvements are mainly relative to existing generative methods.

We have added a comparison of frames 4–6 from MCL-JCV videoSRC05 to Appendix A.6, Figure 13. The table reports each method's actual bitrate and average metrics over all 96 frames.

MethodBPPLPIPS ↓E_warp ↓CLIP-F ↑
VTM-17.00.0164270.31023.820.978
GLC-Video0.0198360.10880.520.976
YODA0.0193970.094108.130.974
GVC-ST0.0205920.16037.240.977

At these operating points, GVC-ST recovers more texture and has a lower LPIPS than VTM-17.0, with similar CLIP-F scores. VTM-17.0 has a lower E_warp but smoother textures, so this score alone does not indicate better detail recovery. Compared with GLC-Video and YODA, GVC-ST achieves lower E_warp and higher CLIP-F. The consecutive frames also show more consistent texture details.

Closing remarks

Thank you again for your careful review. We hope these clarifications and additional results address your concerns and help with your final assessment. We would be happy to provide any further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.