- Rating
- 2(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
In this paper, the authors divide frames into a reference chain and non-reference frames. Reference frames are responsible for propagating detail and maintaining temporal consistency, while non-reference frames use bidirectional latent refinement from neighboring references and adaptively choose between coding and interpolation. The method is not lightweight; but overall it is a generative compression framework with a clear coding structure. Although many components are existing ones, the combination of "reference/non-reference division + forward reference attention + adaptive interpolation for non-reference frames" is meaningful, and the ablations support the role of each module.
Critical strengths
-
Temporal consistency is indeed the core pain point in generative compression.
-
The coding structure is a good idea, and the division of labor is reasonable.
-
The main results show the best FID, and Ewarp/CLIP-F also show temporal consistency advantages; the ablation table makes the contribution of each component visible. The adaptive coding/interpolation examples are convincing.
Critical weaknesses
-
Computational complexity is the most substantial weakness of this paper, and the main text does not address it head-on.
-
The main text lacks fidelity metrics, so the rate-distortion-perception trade-off is incompletely presented.
-
The evaluation metrics are coupled with the training or selection objectives.
-
The method is a system integration of a large model and multiple modules, and component sensitivity is not discussed. The ablation never isolates the effect of the fixed reference interval.
Questions for authors
-
The compute cost is the deciding factor for the paper. Both the encoder and decoder are very slow. The authors acknowledge in the limitation section that "DiT dominates," but the main text does not give runtime, memory, or multiples relative to VTM/DCVC-FM. I wonder whether this is a quality-is-great-but-unusable method.
-
The main tables are mainly LPIPS/FID/Ewarp/CLIP-F, while PSNR/MS-SSIM are placed in the appendix or curves. This makes me feel the paper is avoiding the question of whether the generated content deviates from the original frames. PSNR does not have to be a strength, but it needs to appear in the paper. LPIPS is used in Stage II training, and Ewarp is used in Stage III and mode selection, and then they are used as evaluation metrics. The evaluation metrics are coupled with the training or selection objectives.
-
Due to compute constraints, reporting only one rate point is understandable, but the paper cannot therefore claim "best among all compared generative codecs" without qualification. At the very least, the claim should be narrowed to best among generative codecs compared at full multi-rate settings.
-
The proposed method is still a system built on a strong pretrained backbone plus multi-stage training. Why is this structure necessary for generative video compression, rather than just an effective combination of existing modules? The current ablation shows it works, but the theoretical analysis of why it works is not fully satisfying. If FLUX.2 were replaced with another DiT backbone, how much of the core gain would be expected to remain?
3 comments
Response to Reviewer AbNn (Part 2)
Q4. Design Rationale and DiT Backbone Dependence
We explain the design through the link between reference dependencies and the optimization objective. To study this, we added fully recursive configuration F in Table 4. Every frame is coded and updates the codec and generative reference states used by later frames. Each frame therefore recovers details and serves as a reference. Changes in its generated details affect the reference conditions for later generation, linking local perceptual optimization with later reconstruction.
With role separation, the next reference frame depends directly on the previous reference frame. A non-reference frame is reconstructed after both neighboring references are available. Refining a non-reference frame or changing its reconstruction mode leaves later reference states unchanged. This allows local detail recovery while both references constrain transitions to neighboring frames.
This also lets us split mode selection into independent decisions. Given reference reconstructions and fixed candidates, let denote the cost for non-reference frame , including rate, perceptual quality, and temporal consistency with both neighboring references. Each frame's choice leaves the reference conditions for other choices unchanged, so:
= \sum_t\min_{m_t}J_t(m_t\mid\mathcal R).$$ **Thus, choosing each frame's mode independently minimizes the total selection cost for the fixed reference chain and candidate set.** Forward reference attention propagates details. Bidirectional refinement improves local candidates under fixed reference conditions. Adaptive selection balances rate, perceptual quality, and temporal consistency. Stage III freezes the codec and one-step reconstruction module and trains only the local refinement module, keeping the reference conditions fixed. In the experiments, **E has lower LPIPS and whole-sequence $E_{\mathrm{warp}}$ BD-Metrics than fully recursive F in Table 4. This supports the combined benefit of role separation and adaptive synthesis.** We also compare GVC-ST-BASE and GVC-ST using the same FLUX.2 pretrained backbone. Table 1 shows overall improvements on all four metrics across three datasets, showing that the design adds gains on top of the same pretrained prior. **Computing resources are limited, and adapting and training a different DiT takes considerable time. We have therefore not yet compared multiple DiT backbones and cannot say how much of the gain would remain after changing the backbone.** We state this limitation on page 10 of the revised paper, in the Limitations section. Future work will explore lighter DiT backbones to reduce computational cost and further evaluate our method across different generative backbones.Response to Reviewer AbNn (Part 3)
W4. Component Sensitivity and Fixed Reference Spacing
We added sensitivity experiments on reference spacing and mode-selection weights to examine their effects separately. All tests cover all 30 MCL-JCV videos at four rate points, using 96 frames per video.
Reference spacing. We used configuration D with frozen model weights, no bidirectional refinement, no per-frame adaptive synthesis, and a GOP length of 32. We tested fixed spacings of 1, 2, 3, and 4. Spacing 2 matches the structure used in the paper. GOP-level adaptation generates all four candidates and selects the spacing with the lowest cost, . To compare E_warp at similar LPIPS coding efficiency, we adjusted the temporal weight on this test set. We chose , giving an LPIPS BD-Rate close to fixed spacing 3. Here, is actual BPP, both metrics are GOP averages, and the spacing experiment uses RAFT-Small for E_warp. The baseline is fixed spacing 2:
| Reference spacing | LPIPS BD-Rate ↓ | PSNR BD-Metric ↑ | E_warp BD-Metric ↓ | CLIP-F BD-Metric ↑ |
|---|---|---|---|---|
| Fixed 1 | +3.92% | +0.38 dB | +15.59 | +0.0002 |
| Fixed 2 (spacing used in the paper) | 0.00% | 0.00 dB | 0.00 | 0.0000 |
| Fixed 3 | −9.14% | −0.63 dB | −4.60 | +0.0018 |
| Fixed 4 | +10.78% | −1.06 dB | −6.97 | +0.0022 |
| GOP-level adaptive | −9.16% | −0.37 dB | −8.33 | +0.0020 |
Fixed spacing 3 gives the best perceptual coding efficiency among the four fixed settings. Increasing the spacing to 4 improves temporal metrics but worsens LPIPS coding efficiency and PSNR. GOP-level adaptation gives a lower E_warp BD-Metric at an LPIPS BD-Rate close to fixed spacing 3, showing the potential of adaptive reference structures. Future work will explore more efficient content-based spacing selection to reduce the cost of generating multiple candidates.
Selection weights. The paper uses and , selected by offline search over a large set of videos. All videos, rates, and datasets use these weights. We tested each weight from 1/100 to 100 times its paper value to examine changes across two orders of magnitude in each direction. We kept models and candidates fixed, changed one weight at a time, and kept the other at its paper value. Results are relative to the paper weights. LPIPS uses BD-Rate and E_warp uses BD-Metric; lower is better for both.
We fixed the temporal weight at 0.02 and varied the perceptual weight:
| Weight multiplier | Perceptual weight | Temporal weight | LPIPS BD-Rate ↓ | E_warp BD-Metric ↓ |
|---|---|---|---|---|
| 1/100 | 0.0068 | 0.02 | +0.588% | +0.0378 |
| 1/50 | 0.0136 | 0.02 | +0.588% | +0.0378 |
| 1/10 | 0.068 | 0.02 | +0.519% | +0.0300 |
| 1 (paper weights) | 0.68 | 0.02 | 0.000% | 0.0000 |
| 10 | 6.8 | 0.02 | −2.317% | +0.5197 |
| 50 | 34 | 0.02 | +1.529% | +7.4635 |
| 100 | 68 | 0.02 | +4.452% | +10.5344 |
We fixed the perceptual weight at 0.68 and varied the temporal weight:
| Weight multiplier | Perceptual weight | Temporal weight | LPIPS BD-Rate ↓ | E_warp BD-Metric ↓ |
|---|---|---|---|---|
| 1/100 | 0.68 | 0.0002 | −0.434% | +6.3516 |
| 1/50 | 0.68 | 0.0004 | −1.858% | +4.0479 |
| 1/10 | 0.68 | 0.002 | −2.207% | +0.0528 |
| 1 (paper weights) | 0.68 | 0.02 | 0.000% | 0.0000 |
| 10 | 0.68 | 0.2 | +0.428% | +0.0196 |
| 50 | 0.68 | 1 | +0.472% | +0.0219 |
| 100 | 0.68 | 2 | +0.472% | +0.0219 |
Reducing the perceptual weight or increasing the temporal weight caused small changes; changes in the opposite directions could have larger effects. Sensitivity therefore depends on the direction of the change. The paper weights give the lowest E_warp BD-Metric among the tested settings. Some other settings improve LPIPS coding efficiency but increase E_warp BD-Metric. These results support the paper weights as a good balance between perceptual compression efficiency and temporal consistency.
Closing remarks
Thank you again for your careful review. We hope these clarifications and additional results address your concerns and help with your final assessment. We would be happy to provide any further clarification.
Response to Reviewer AbNn (Part 1)
Thank you for your careful review and constructive suggestions. We have added runtime and GPU memory comparisons, included PSNR and MS-SSIM results in the main text, and expanded the evaluation with metrics not used for training or mode selection. We have also narrowed the FID claim and added a fully recursive ablation. Below, we explain the design rationale, discuss the limitation of testing only one DiT backbone, and report reference-spacing and selection-weight studies.
Q1, W1. Runtime, GPU Memory, and Practicality
We have added encoding and decoding times and peak GPU memory to the Computational Cost paragraph and Table 3 in the main text. All results below were measured at 1080p on a single NVIDIA A100-SXM4-80GB GPU and include actual entropy coding, decoding, and bitstream packing.
GVC-ST and YODA have similar runtimes, with nearly identical decoding times. GVC-ST requires more time and memory than DCVC-FM. Our current implementation does not yet support real-time encoding or decoding. This work focuses on improving perceptual compression efficiency and temporal consistency at low bitrates.
The encoder generates and scores both coding and interpolation candidates, while the decoder runs only the selected reconstruction mode. When interpolation is selected for a non-reference frame, the decoder uses RIFE and skips DiT inference for that frame. As a result, the full method has a shorter decoding time than GVC-ST-BASE.
We retain the discussion of computational cost in the Limitations section. Future work will explore more efficient generative backbones and mode selection without generating both candidates, to reduce time and memory costs.
Q2, W2–3. Reconstruction Fidelity and Metrics Coupled with Optimization
We have added PSNR/MS-SSIM RD curves and results across multiple rates to Figure 4 and Table 2 on page 8 of the main text. On HEVC-B, UVG, and MCL-JCV, GVC-ST has better overall PSNR and MS-SSIM results than GLC-Video and YODA.
To address the concern about using the same metrics for optimization and evaluation, we also use FID, KID, CLIP-F, and DINOv2 consistency. FID and CLIP-F are reported in Table 1 in the main text; KID and DINOv2 consistency are reported in Figure 10 and Table 5 in the appendix. These scores are not used directly as training losses or for RDO mode selection.
Across all three datasets, GVC-ST has better overall results on all four metrics than GLC-Video and YODA. This shows that the gains also appear in evaluation scores that we do not directly optimize, beyond LPIPS and Ewarp.
Q3. Limitations of the GNVC-VD Comparison at a Single Operating Point
We have narrowed the claim in the abstract, introduction, and Section 4.2 to "the best FID BD-Metric among the open-source generative baselines". This wording clearly defines the methods and metric being compared.
Since GNVC-VD has no public code, we reimplemented it following the architecture and training procedure described in its paper. Due to limited computing resources, we trained one model at a single rate setting and reported one operating point per dataset. The revised paper states that GNVC-VD is excluded from the multi-rate BD comparison in Table 1. We compare with GNVC-VD only near its reported operating points in Figure 3.