- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Graph-Factored Sequential Coding (GFSC), treating a configurable frame-delay budget as a coding resource. For finite-alphabet, strictly positive -order Markov sources, it shows that the GFSC information sum-rate attains the joint-coding rate-distortion function over a nontrivial distortion region. The practical extension introduces temporal joint modeling of lookahead representations into DCVC, VCT, and DHVC.
Critical strengths
- Well-motivated setting. The paper connects limited-delay sequential coding with an information-theoretic analysis and a practical neural video compression design.
- Evaluation across architectures and delay budgets. On UVG, the BD-rate reductions relative to the respective causal counterparts at are 20.57%, 25.29%, and 28.22% for DCVC; 28.38%, 48.00%, and 55.78% for VCT; and 7.39%, 10.17%, and 11.91% for DHVC.
Critical weaknesses
-
The connection between the optimal auxiliary construction and the implemented modules remains unclear. The theoretical result relies on clique-wise auxiliary variables with specific conditional-independence properties; edge-wise linear interpolation is a specialization for Gauss-Markov sources. In the practical codec, however, the motion and context branches use convolutional temporal joint modeling and windowed spatial-temporal attention, without explicitly enforcing those properties. How much of the gain comes from the proposed graph-factored structure, as opposed to giving the encoder access to future frames and applying generic feature fusion? Please compare GFSC with simple lookahead-feature concatenation, standard temporal attention, and an explicitly structured variant under matched , parameter counts, training budgets, and test conditions. Figure 2's approximately banded inverse covariance is also an important motivation for the Markov analysis; please specify the video features, sample size, and estimation procedure behind it.
-
The causal baseline and training protocol need clearer documentation. Appendix D.1 states that all models are trained with 7-frame sequences and evaluated with GOP , but describes 50 training epochs for VCT/DHVC and later states that training lasts 30 epochs. Please clarify the initialization, training stages, and budgets for each causal and GFSC-enhanced model. In addition, please evaluate three models under the same test sequences, first 96 frames, GOP, RGB-PSNR metric, and bitrate accounting: (i) the official released DCVC checkpoint, without retraining; (ii) the causal DCVC model trained by the authors; and (iii) DCVC-GFSC at each reported . Report rate-distortion curves and BD-rates against a common anchor. This comparison would separate differences due to the authors' DCVC training from the overall gain obtained by adding lookahead and GFSC, and show performance relative to the released model. It would not, by itself, isolate the GFSC module from access to future frames.
-
Generality on stronger, more recent causal codecs is not established. The integration experiments cover DCVC, VCT, and DHVC. DCVC-B is a separate delayed-coding baseline, not a GFSC extension of a recent causal codec. Why was GFSC not evaluated on DCVC-FM or DCVC-RT? An integration into at least one stronger recent causal codec, comparing its original and GFSC-enhanced versions under matched training, test conditions, and delay budget , would better establish the claimed generality. Please report rate-distortion curves, BD-rate, parameter counts, and encoding/decoding latency.
4 comments
Response to Reviewer tGAE (2/2): Training protocol and further evaluations
W2: Causal Baseline and Training Protocol
Answer: We thank the reviewer for identifying the inconsistency in the training description and suggesting a more explicit comparison of the causal baselines. We address the questions below:
(1) Training protocol clarification. The 30 epochs reported in Appendix D.1 were a typo, which we have corrected.
-
VCT/DHVC: Both causal and GFSC-enhanced models are randomly initialized and trained from scratch in a single stage for 50 epochs, following the training settings described in Appendix D.1.
-
DCVC: As in Appendix D.1, we follow the three-stage training procedure of OpenDCVCs [1], progressively increasing the training GOP from 3 to 5 to 7 frames. The GFSC delayed modules are introduced only in the final stage. The causal and GFSC variants otherwise follow the same training protocol.
(2) Comparison with the official DCVC checkpoint. We thank the reviewer for suggesting a comparison with the official DCVC checkpoint to better identify the performance gains introduced by GFSC. These experimental results were already included in our submission. To make the comparison clearer, we have slightly revised Table 2 to explicitly distinguish the official released DCVC checkpoint, our retrained causal DCVC, and DCVC-GFSC at different delay budgets. All models are evaluated on the same test sequences using the first 96 frames, GOP = 32, and RGB-PSNR. We also highlight the corresponding model names in blue in Table 2 for clarity. For convenience, we summarize the BD-rates below, using the official DCVC as the common reference.
| Method | Delay | UVG BD-rate (%) |
|---|---|---|
| DCVC (official released) | 0 | 0.00 |
| DCVC (retrained causal) | 0 | -8.57 |
| DCVC-GFSC | 1 | -20.57 |
| DCVC-GFSC | 2 | -25.29 |
| DCVC-GFSC | 3 | -28.22 |
This comparison directly distinguishes the effect of our causal DCVC training from the additional RD performance gains achieved by GFSC. The complementary feature-fusion ablations (Appendix D. 5) in our response to Reviewer zgdi (W2/Q2) further isolate the effects of different fusion architectures under matched delay budgets.
Q3: Stronger coding baselines
Answer: We have now integrated GFSC into DCVC-FM, and added matched-limited-delay VVC-RA comparisons on UVG (new Fig. 21 and Table 5). All BD-rates below use causal DCVC-FM as the reference:
| Method | Delay | UVG BD-rate (%) |
|---|---|---|
| DCVC-FM (causal) | 0 | 0.00 |
| VVC-RA | 1 | +53.96 |
| VVC-RA | 3 | +50.08 |
| DCVC-FM-GFSC | 1 | -22.52 |
New Fig. 21 presents the rate–PSNR curves, while Table 5 includes computational cost analysis. The DCVC-FM extension currently covers because of computational resource constraints. Together with our existing results on DCVC, VCT, and DHVC, these findings provide further empirical evidence of GFSC's generality across different coding backbones. More importantly, our central contribution lies in the rigorous theoretical foundation for finite-delay optimality. Theorem 3.7 establishes that a -frame delay is sufficient to attain joint-coding RD optimality for general positive -order Markov sources within a nontrivial distortion region, independently of any particular neural codec architecture.
We hope that our response addresses your concern and supports a positive re-evaluation of the submission. Please let us know if you have any further questions regarding our work.
[1] Y. Zhang and F. Zhu, OpenDCVCs: A PyTorch Open Source Implementation and Performance Evaluation of the DCVC Series Video Codecs. 2025.
Follow up on Rebuttal Response
Dear reviewer tGAE,
As the rebuttal period is approaching its end, we wanted to check whether our responses have adequately addressed your concerns. We would be happy to provide any further clarification if needed.
Best regards,
The Authors
Additional Results for Q3: Stronger Coding Baselines
We would like to share an additional experimental update following our previous response to Q3. We have extended the DCVC-FM experiments to and further evaluated the models on MCL-JCV. The updated results are summarized below, with BD-rates measured relative to causal DCVC-FM.
| Method | Delay | UVG BD-rate (%) | MCL-JCV BD-rate (%) |
|---|---|---|---|
| DCVC-FM (causal) | 0 | 0.00 | 0.00 |
| VVC-RA | 1 | +53.96 | +6.90 |
| VVC-RA | 3 | +50.08 | +3.34 |
| DCVC-FM-GFSC | 1 | −22.52 | -20.44 |
| DCVC-FM-GFSC | 2 | -25.29 | -22.90 |
The additional experiments show that GFSC consistently improves upon causal DCVC-FM across both datasets, with further rate-distortion gains at . These results strengthen our empirical evidence that GFSC generalizes to more advanced neural video codecs and different delay budgets. The updated rate–PSNR curves and computational complexity comparisons are included in new Fig. 21 and Table 5 of the revised manuscript.
We sincerely appreciate your time and consideration. We hope these additional results further address your concerns and supports a positive re-evaluation of our submission.
Response to Reviewer tGAE (1/2): Theoretical construction, feature-fusion ablation, and temporal-dependence analysis
We thank the reviewer for the thorough and constructive feedback. We organize our response into two comments. Part 1: Clarification of the GFSC framework and its theory-guided implementation, additional feature-fusion ablations, and the detailed temporal-dependence estimation procedure for Fig. 2. Part 2: Clarification of the training protocols, comparisons between official and retrained DCVC baselines, and additional evaluations on stronger codecs (DCVC-FM and VVC-RA).
Q1: GFSC versus its neural implementation.
Answer: We thank the reviewer for highlighting the importance of clarifying the connection between the theoretical construction and its neural implementation. As clarified in our response to Reviewer zgdi (W2/Q2), GFSC is a general delayed sequential coding framework defined by the admissible information flow in Eq. (5), rather than a specific neural architecture. Importantly, any coding architecture satisfying these information-flow constraints is a valid GFSC instantiation.
The connection between the theoretical auxiliary construction and neural implementation is explained in our response to Reviewer zgdi (Q3). Specifically, Appendix B.2 constructs clique-wise auxiliaries that capture local source dependencies, together with node-wise refinements that achieve joint-coding RD optimality within the characterized distortion region. The Gaussian specialization provides an explicit example through linear interpolation. These constructions motivate jointly transforming local temporal features before quantization. TJM modules realize this principle through content-adaptive spatial associations and cross-frame interactions, providing a learnable implementation of the theory-inspired joint representation.
We then compare three valid GFSC implementations under the same delay budgets (K=1,2,3) and training/evaluation protocol: feature concatenation + 1×1 convolution, residual CNN fusion, and the adopted TJM. The following BD-rates (%) are relative to causal DCVC:
Feature fusion ablation
Full results and parameter/runtime are in new Table 3 (Appendix D.5). In particular, CNN and TJM have closely matched parameters, while the simpler concat variant is smaller. All three fusion designs improve on causal DCVC, and TJM performs best for every K. This separates the benefit of access to limited future information from the additional benefit of the chosen fusion design. Our framework contribution is not conditional on spatial-temporal attention module.
Q1: Temporal dependence estimation
Answer: Thank you for pointing out the missing details of temporal dependence analysis. We added Appendix D.4 to document the analysis. Using all seven UVG sequences (3,900 frames), we extract unquantized single-frame features with the frozen Cheng2020Anchor analysis transform [1], pool each feature map to an 8×8 grid, and apply PCA. Splitting the sequences into non-overlapping 32-frame clips yields 117 clips, or 7,488 spatial-patch trajectories (117×64). We estimate a 32×32 temporal covariance matrix by averaging feature inner products across these trajectories and invert it to obtain the information matrix shown in Fig. 2 (left). Fig. 2 (right) averages the absolute normalized precision entries (absolute partial correlations) at each temporal separation. This analysis provides empirical evidence of temporal locality.
[1] Cheng, Zhengxue, et al. Learned image compression with discretized gaussian mixture likelihoods and attention modules. CVPR 2020.