acceptodds
Under review as a conference paper at ICLR 2027

Video Compression under Limited Delay: Theoretical Optimality and System Design

Abstract

Traditional neural video compression (NVC) methods predominantly follow strictly causal predictive-coding paradigm, treating zero-frame delay as a fixed constraint rather than delay as an explicit design parameter. From an information-theoretic perspective, allowing limited frame delay enables a codec to exploit correlations with future frames within the delay window, thereby reducing uncertainty that is inaccessible to strictly causal prediction and potentially improving the rate-distortion tradeoff. In this work, we formalize this intuition through a rigorous rate–distortion analysis builds on the clique factorization of Markov networks, and propose Graph-Factored Sequential Coding (GFSC), a principled framework for sequential video coding under limited delay. To our knowledge, we provide the first theoretical result showing that, for general -order Markov sources with positive distributions, a -frame delay is sufficient for GFSC to attain joint-coding rate-distortion optimality over a nontrivial distortion region, a guarantee unavailable to zero-delay causal coding in general. In practice, we instantiate GFSC through lightweight neural modules, allowing existing causal predictive codecs to be augmented without modifying their core architectures. Experiments across multiple NVC models and datasets show that GFSC consistently improves rate-distortion performance over the corresponding causal counterparts. Under matched delay budgets, GFSC also achieves competitive performance compared with representative bidirectional codecs.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
delayed sequential coding, rate-distortion theory, neural lossy compression, probabilistic graphical model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tGAEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Graph-Factored Sequential Coding (GFSC), treating a configurable frame-delay budget KK as a coding resource. For finite-alphabet, strictly positive KK-order Markov sources, it shows that the GFSC information sum-rate attains the joint-coding rate-distortion function over a nontrivial distortion region. The practical extension introduces temporal joint modeling of lookahead representations into DCVC, VCT, and DHVC.

Critical strengths
  1. Well-motivated setting. The paper connects limited-delay sequential coding with an information-theoretic analysis and a practical neural video compression design.
  2. Evaluation across architectures and delay budgets. On UVG, the BD-rate reductions relative to the respective causal counterparts at K=1,2,3K=1,2,3 are 20.57%, 25.29%, and 28.22% for DCVC; 28.38%, 48.00%, and 55.78% for VCT; and 7.39%, 10.17%, and 11.91% for DHVC.
Critical weaknesses
  1. The connection between the optimal auxiliary construction and the implemented modules remains unclear. The theoretical result relies on clique-wise auxiliary variables with specific conditional-independence properties; edge-wise linear interpolation is a specialization for Gauss-Markov sources. In the practical codec, however, the motion and context branches use convolutional temporal joint modeling and windowed spatial-temporal attention, without explicitly enforcing those properties. How much of the gain comes from the proposed graph-factored structure, as opposed to giving the encoder access to future frames and applying generic feature fusion? Please compare GFSC with simple lookahead-feature concatenation, standard temporal attention, and an explicitly structured variant under matched KK, parameter counts, training budgets, and test conditions. Figure 2's approximately banded inverse covariance is also an important motivation for the Markov analysis; please specify the video features, sample size, and estimation procedure behind it.

  2. The causal baseline and training protocol need clearer documentation. Appendix D.1 states that all models are trained with 7-frame sequences and evaluated with GOP =32=32, but describes 50 training epochs for VCT/DHVC and later states that training lasts 30 epochs. Please clarify the initialization, training stages, and budgets for each causal and GFSC-enhanced model. In addition, please evaluate three models under the same test sequences, first 96 frames, GOP, RGB-PSNR metric, and bitrate accounting: (i) the official released DCVC checkpoint, without retraining; (ii) the causal DCVC model trained by the authors; and (iii) DCVC-GFSC at each reported KK. Report rate-distortion curves and BD-rates against a common anchor. This comparison would separate differences due to the authors' DCVC training from the overall gain obtained by adding lookahead and GFSC, and show performance relative to the released model. It would not, by itself, isolate the GFSC module from access to future frames.

  3. Generality on stronger, more recent causal codecs is not established. The integration experiments cover DCVC, VCT, and DHVC. DCVC-B is a separate delayed-coding baseline, not a GFSC extension of a recent causal codec. Why was GFSC not evaluated on DCVC-FM or DCVC-RT? An integration into at least one stronger recent causal codec, comparing its original and GFSC-enhanced versions under matched training, test conditions, and delay budget KK, would better establish the claimed generality. Please report rate-distortion curves, BD-rate, parameter counts, and encoding/decoding latency.

Questions for authors

See Critical Weaknesses

Minor Comment: References

  • Sequential Coding of Correlated Sources is cited as 2002, but the cited journal article was published in 2000 (DOI).
  • On Universal Quantization is cited as 2003, but the cited journal article was published in 1985 (DOI).
4 comments
AuthorsOpenReview ↗

Response to Reviewer tGAE (1/2): Theoretical construction, feature-fusion ablation, and temporal-dependence analysis

We thank the reviewer for the thorough and constructive feedback. We organize our response into two comments. Part 1: Clarification of the GFSC framework and its theory-guided implementation, additional feature-fusion ablations, and the detailed temporal-dependence estimation procedure for Fig. 2. Part 2: Clarification of the training protocols, comparisons between official and retrained DCVC baselines, and additional evaluations on stronger codecs (DCVC-FM and VVC-RA).

Q1: GFSC versus its neural implementation.

Answer: We thank the reviewer for highlighting the importance of clarifying the connection between the theoretical construction and its neural implementation. As clarified in our response to Reviewer zgdi (W2/Q2), GFSC is a general delayed sequential coding framework defined by the admissible information flow in Eq. (5), rather than a specific neural architecture. Importantly, any coding architecture satisfying these information-flow constraints is a valid GFSC instantiation.

The connection between the theoretical auxiliary construction and neural implementation is explained in our response to Reviewer zgdi (Q3). Specifically, Appendix B.2 constructs clique-wise auxiliaries that capture local source dependencies, together with node-wise refinements that achieve joint-coding RD optimality within the characterized distortion region. The Gaussian specialization provides an explicit example through linear interpolation. These constructions motivate jointly transforming local temporal features before quantization. TJM modules realize this principle through content-adaptive spatial associations and cross-frame interactions, providing a learnable implementation of the theory-inspired joint representation.

We then compare three valid GFSC implementations under the same delay budgets (K=1,2,3K=1,2,3) and training/evaluation protocol: feature concatenation + 1×11\times1 convolution, residual CNN fusion, and the adopted TJM. The following BD-rates (%) are relative to causal DCVC:

Feature fusion ablation

MethodDelay KKUVG BD-rate (%)MCL-JCV BD-rate (%)
DCVC-causal00.000.00
GFSC-Concat1-14.37-9.20
GFSC-Concat2-15.32-10.51
GFSC-Concat3-16.09-11.00
GFSC-CNN1-15.58-8.51
GFSC-CNN2-18.43-10.33
GFSC-CNN3-21.13-11.65
GFSC-TJM1-20.57-10.90
GFSC-TJM2-25.29-17.32
GFSC-TJM3-28.22-20.30

Full results and parameter/runtime are in new Table 3 (Appendix D.5). In particular, CNN and TJM have closely matched parameters, while the simpler concat variant is smaller. All three fusion designs improve on causal DCVC, and TJM performs best for every KK. This separates the benefit of access to limited future information from the additional benefit of the chosen fusion design. Our framework contribution is not conditional on spatial-temporal attention module.

Q1: Temporal dependence estimation

Answer: Thank you for pointing out the missing details of temporal dependence analysis. We added Appendix D.4 to document the analysis. Using all seven UVG sequences (3,900 frames), we extract unquantized single-frame features with the frozen Cheng2020Anchor analysis transform [1], pool each feature map to an 8×88\times8 grid, and apply PCA. Splitting the sequences into non-overlapping 32-frame clips yields 117 clips, or 7,488 spatial-patch trajectories (117×64117\times64). We estimate a 32×3232\times32 temporal covariance matrix by averaging feature inner products across these trajectories and invert it to obtain the information matrix shown in Fig. 2 (left). Fig. 2 (right) averages the absolute normalized precision entries (absolute partial correlations) at each temporal separation. This analysis provides empirical evidence of temporal locality.

[1] Cheng, Zhengxue, et al. Learned image compression with discretized gaussian mixture likelihoods and attention modules. CVPR 2020.

AuthorsOpenReview ↗

Response to Reviewer tGAE (2/2): Training protocol and further evaluations

W2: Causal Baseline and Training Protocol

Answer: We thank the reviewer for identifying the inconsistency in the training description and suggesting a more explicit comparison of the causal baselines. We address the questions below:

(1) Training protocol clarification. The 30 epochs reported in Appendix D.1 were a typo, which we have corrected.

  • VCT/DHVC: Both causal and GFSC-enhanced models are randomly initialized and trained from scratch in a single stage for 50 epochs, following the training settings described in Appendix D.1.

  • DCVC: As in Appendix D.1, we follow the three-stage training procedure of OpenDCVCs [1], progressively increasing the training GOP from 3 to 5 to 7 frames. The GFSC delayed modules are introduced only in the final stage. The causal and GFSC variants otherwise follow the same training protocol.

(2) Comparison with the official DCVC checkpoint. We thank the reviewer for suggesting a comparison with the official DCVC checkpoint to better identify the performance gains introduced by GFSC. These experimental results were already included in our submission. To make the comparison clearer, we have slightly revised Table 2 to explicitly distinguish the official released DCVC checkpoint, our retrained causal DCVC, and DCVC-GFSC at different delay budgets. All models are evaluated on the same test sequences using the first 96 frames, GOP = 32, and RGB-PSNR. We also highlight the corresponding model names in blue in Table 2 for clarity. For convenience, we summarize the BD-rates below, using the official DCVC as the common reference.

MethodDelay KKUVG BD-rate (%)
DCVC (official released)00.00
DCVC (retrained causal)0-8.57
DCVC-GFSC1-20.57
DCVC-GFSC2-25.29
DCVC-GFSC3-28.22

This comparison directly distinguishes the effect of our causal DCVC training from the additional RD performance gains achieved by GFSC. The complementary feature-fusion ablations (Appendix D. 5) in our response to Reviewer zgdi (W2/Q2) further isolate the effects of different fusion architectures under matched delay budgets.

Q3: Stronger coding baselines

Answer: We have now integrated GFSC into DCVC-FM, and added matched-limited-delay VVC-RA comparisons on UVG (new Fig. 21 and Table 5). All BD-rates below use causal DCVC-FM as the reference:

MethodDelay KKUVG BD-rate (%)
DCVC-FM (causal)00.00
VVC-RA1+53.96
VVC-RA3+50.08
DCVC-FM-GFSC1-22.52

New Fig. 21 presents the rate–PSNR curves, while Table 5 includes computational cost analysis. The DCVC-FM extension currently covers K=1K=1 because of computational resource constraints. Together with our existing results on DCVC, VCT, and DHVC, these findings provide further empirical evidence of GFSC's generality across different coding backbones. More importantly, our central contribution lies in the rigorous theoretical foundation for finite-delay optimality. Theorem 3.7 establishes that a KK-frame delay is sufficient to attain joint-coding RD optimality for general positive KK-order Markov sources within a nontrivial distortion region, independently of any particular neural codec architecture.

We hope that our response addresses your concern and supports a positive re-evaluation of the submission. Please let us know if you have any further questions regarding our work.

[1] Y. Zhang and F. Zhu, OpenDCVCs: A PyTorch Open Source Implementation and Performance Evaluation of the DCVC Series Video Codecs. 2025.

AuthorsOpenReview ↗

Follow up on Rebuttal Response

Dear reviewer tGAE,

As the rebuttal period is approaching its end, we wanted to check whether our responses have adequately addressed your concerns. We would be happy to provide any further clarification if needed.

Best regards,

The Authors

AuthorsOpenReview ↗

Additional Results for Q3: Stronger Coding Baselines

We would like to share an additional experimental update following our previous response to Q3. We have extended the DCVC-FM experiments to K=2K=2 and further evaluated the models on MCL-JCV. The updated results are summarized below, with BD-rates measured relative to causal DCVC-FM.

MethodDelay KKUVG BD-rate (%)MCL-JCV BD-rate (%)
DCVC-FM (causal)00.000.00
VVC-RA1+53.96+6.90
VVC-RA3+50.08+3.34
DCVC-FM-GFSC1−22.52-20.44
DCVC-FM-GFSC2-25.29-22.90

The additional experiments show that GFSC consistently improves upon causal DCVC-FM across both datasets, with further rate-distortion gains at K=2K=2. These results strengthen our empirical evidence that GFSC generalizes to more advanced neural video codecs and different delay budgets. The updated rate–PSNR curves and computational complexity comparisons are included in new Fig. 21 and Table 5 of the revised manuscript.

We sincerely appreciate your time and consideration. We hope these additional results further address your concerns and supports a positive re-evaluation of our submission.

Reviewer_r3yKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates neural video compression (NVC) under a limited-delay setting. Moving away from the conventional strictly causal zero-delay paradigm, the authors propose a principled framework named Graph-Factored Sequential Coding (GFSC). By leveraging a K frame lookahead window and formulating the rate-distortion analysis via Markov network clique factorizations, the paper demonstrates both theoretically and empirically that a limited delay can bridge the performance gap between causal and bidirectional NVC. Experiments across several mainstream causal NVC models (e.g., DCVC, VCT, and DHVC) show consistent rate-distortion improvements.

Critical strengths

Strong Theoretical Foundation: Unlike many empirical works in neural video compression that heuristically adjust temporal receptive fields, this paper provides a rigorous rate-distortion analysis using clique factorizations of Markov networks. The proof that K-frame delay is sufficient to achieve joint coding rate-distortion optimality for K-th order Markov sources is a solid theoretical contribution. Plug-and-Play Practical Design: The proposed GFSC framework is designed as a lightweight, window-based spatial-temporal attention module. It seamlessly enhances existing causal models without requiring a complete overhaul of their core architectures, making it highly practical for real-world deployment. Comprehensive Evaluation: The experiments are thorough, evaluated across diverse datasets (e.g., UVG, MCL-JCV) and multiple baseline architectures. The consistent BD-rate gains validate the effectiveness of the proposed design.

Critical weaknesses

Clarity on Computational Overhead: While GFSC is described as "lightweight," the paper lacks a detailed, empirical analysis of the actual computational overhead introduced by the K-frame spatial-temporal attention module. Specifically, metrics such as encoding/decoding latency (in milliseconds), GPU memory footprint, and FLOPs relative to K should be reported to substantiate real-time feasibility claims. Limited Discussion on Failure Cases: The paper highlights performance gains uniformly, but does not discuss scenarios where a limited lookahead window might fail or introduce artifacts (e.g., heavy camera panning, abrupt scene cuts, or fast-moving non-linear motion where future frames provide misleading temporal contexts). Comparison with Advanced Bidirectional Schemes: The comparison against bidirectional baselines is somewhat brief. It remains unclear how GFSC behaves when the delay budget K matches the actual GOP (Group of Pictures) structure or bi-directional reference distances used in standard configurations. DCVC is not the SOTA of e2e coding.

Questions for authors

Complexity vs. Performance Trade-off: Could the authors provide a runtime and memory consumption breakdown for different values of K?How does the decoding latency scale in practice under strict streaming constraints? Handling Scene Transitions: How does GFSC handle sudden scene changes or discontinuous motion within the K-frame lookahead window? Does the temporal attention mechanism occasionally over-rely on irrelevant future frames, and how is this mitigated? Choice of K in Practice: In real-world interactive video communications, what is the recommended sweet spot for K considering both BD-rate savings and buffering delay tolerance? Experiments: Can the author provide results on more recent e2e benchmarks like DCVC-DC or DCVC-FM? And VVC-RA should also be included.

4 comments
AuthorsOpenReview ↗

Response to Reviewer r3yk (1/2): Computational overhead and challenging content

We thank the reviewer for recognizing the theoretical contribution, and the plug-and-play design. We are particularly encouraged by reviewer's recognition of our rigorous rate-distortion analysis as a solid theoretical foundation for limited-delay video compression. Below, we address the questions with additional computational and scene-transition analyses.

Q1: Computational overhead

Answer: Thank you for the suggestion to provide a detailed computational analysis, which is important for pratical codec design. Table 2 already reports encoding/decoding latency, GPU memory, and FLOPs of GFSC under different delay budgets. We further provide a module-level runtime breakdown of the feature-fusion modules (FFM) in newly added Table 3 (Appendix D.5). The computational overhead results are summarized below:

Computational complexity

MethodDelay KKEnc. (s/frame)Dec. (s/frame)FFM (s/frame)Memory (GB)FLOPs (G/frame)
DCVC-causal00.520.440.05.544430.53
DCVC-GFSC10.650.440.135.525916.28
DCVC-GFSC20.800.440.255.527451.81
DCVC-GFSC30.880.440.425.528987.58

The results show that encoder computation increases approximately linearly with the delay budget KK, as reflected in the FFM runtime and FLOPs, while decoding latency and GPU memory remain approximately unchanged. Under matched delay budgets, GFSC also achieves lower computational latency than DCVC-B (Table 2). Furthermore, the newly added DCVC-FM results in Table 5 (Appendix D.7) demonstrate that GFSC achieves lower absolute latency when integrated into a faster causal coding backbone, highlighting the dependence of computational cost on the underlying codec architecture.

Overall, these results characterize the computational overhead of GFSC and demonstrate its configurable trade-off between compression efficiency and latency, with KK controlling the additional computation and lookahead buffering.

Q2: Large motion and scene transitions.

Answer: We thank the reviewer for highlighting the importance of evaluating GFSC under challenging motion conditions. Following the content-based grouping in DCVC-RT [1], we evaluate GFSC-1 against causal DCVC on three MCL-JCV subsets. The results below report the mean per-sequence BD-rates.

Challenging content performance

MethodSmall motion BD-rate (%)Large motion BD-rate (%)Scene cut BD-rate (%)
DCVC-causal0.00.00.0
DCVC-GFSC-1-16.59-8.02-9.19

GFSC-1 achieves average bitrate savings across all three subsets, although the gains are smaller for large motion and scene changes. These results identify scenarios where the benefits of limited lookahead are reduced. To further investigate GFSC's behavior around scene transitions, we provide per-frame PSNR and bitrate analyses (new Fig. 18, Appendix D.6) and future-frame attention-mass distributions (new Fig. 19, Appendix D.6). Notably, Fig. 19 shows that the attention module adaptively reduces its attention to future-frame features when the lookahead window crosses a scene transition, rather than relying on future information indiscriminately. This adaptive behavior helps explain why GFSC-1 maintains average RD improvements on scene-change content. These findings also motivate further investigation into more robust delayed temporal modeling under scene transitions and complex motion.

[1] Jia, Zhaoyang, et al. Towards practical real-time neural video compression. CVPR 2025.

AuthorsOpenReview ↗

Response to Reviewer r3yk (2/2): Practical choice of delay and stronger baselines

Q3: Practical choice of KK.

Answer: Theorem 3.7 links a sufficient lookahead to the source's Markov order. For real video, the temporal-dependence analyses on UVG (Fig. 2) and MCL-JCV (new Fig. 17) show the strongest dependencies within approximately three frame intervals. Table 2 further reveals a clear RD-latency trade-off. Computational cost grows approximately linearly with KK, while BD-rate gains exhibit diminishing returns. The additional lookahead buffering is also growing linearly with KK. Thus, K=1K=1 is preferable under tight latency budgets. K=3K=3 is a useful RD-latency balance within our tested range when the budget permits. This follows the source-locality and cost analysis discussed in our response to Reviewer zgdi (Q3).

Q4: Stronger coding baselines.

Answer: We appreciate the reviewer's suggestion to evaluate GFSC on more recent neural codecs and VVC-RA. We have extended GFSC to DCVC-FM [1], and additionally evaluated VVC-RA under limited-delay configurations. The newly added Fig. 21 and Table 5 (Appendix D.7) report the RD comparisons and computational costs.

BD-rate comparison on UVG

MethodDelay KKBD-rate (%)
DCVC-FM00.00
VVC-RA-1153.96
VVC-RA-3350.08
DCVC-FM-GFSC1-22.52

These results further demonstrate that the benefits of GFSC extend to stronger neural coding backbones. Regarding the GOP structure, conventional hierarchical random-access coding can employ larger GOPs (e.g., 32 frames) with substantial frame-reordering delay, whereas our work specifically targets small, explicitly bounded delays. We therefore evaluate VVC-RA at K=1,3K=1,3 for comparison within this regime.

Due to computational resource constraints, the DCVC-FM extension currently focuses on K=1K=1. We emphasize that our central contribution is not merely demonstrating further compression gains from future-frame information, but establishing a rigorous theoretical foundation for finite-delay optimality. The guarantee in Theorem 3.7 is independent of any specific neural codec architecture. Our extensive evaluations across DCVC, VCT, DHVC, and the newly added DCVC-FM provide strong empirical support for the practical effectiveness and generalizability of this theory-guided coding framework.

We again thank you for your support of our paper. We hope that our responses adequately address your concerns and contribute to an overall positive evaluation.

[1] Li, Jiahao, Bin Li, and Yan Lu. Neural video compression with feature modulation. CVPR 2024.

Thanks for the response. I would keep my scores.

Thank you very much for your response! We sincerely appreciate your positive assessment of our work. We are also grateful for your time and effort reviewing our paper.

Reviewer_zgdiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GFSC, a video compression framework that uses limited future frames to improve rate-distortion performance. The authors provide theoretical analysis based on Markov models and implement the method on existing neural video compression models. Experiments on several datasets show improvements over causal compression methods.

Critical strengths
  1. The paper provides a theoretical analysis of video compression with limited frame delay.
  2. The method can be applied to different existing compression models and shows consistent improvements in rate-distortion performance.
Critical weaknesses
  1. It is already well known that using future frames can improve video compression performance, as shown by existing bidirectional coding methods. However, for strictly real-time applications, future frames are not available without introducing additional delay. The paper does not clearly explain which practical scenarios require this setting.

  2. The main improvement comes from using future-frame information, which is expected. The practical implementation mainly introduces temporal attention to fuse future features. It is unclear what advantage GFSC provides over simpler future-frame fusion methods.

Questions for authors
  1. What are the target applications of the proposed limited-delay setting? How practical is this method for real-time video compression, considering the additional buffering and encoding latency?

  2. Can the authors compare GFSC with a simple future-frame attention or feature fusion baseline under the same delay budget? This would help clarify whether the improvements come from the proposed framework or simply from using future information.

  3. How does the theoretical optimality result relate to the actual neural network implementation? Is there any evidence that the proposed attention module approximates the theoretically optimal construction?


I think this paper demonstrates performance improvements in limited-delay video compression. However, I have two main concerns. First, the use of future K frames to improve compression performance limits the practical application scenarios of this method. Second, it remains unclear whether the performance improvements stem from the unique design of GFSC or simply from additional future-frame information. As I am not an expert in this area, I may have some misunderstandings. I lean towards a borderline reject and therefore maintain my rating of 2.

3 comments
AuthorsOpenReview ↗

Response to Reviewer zgdi (1/2): Theoretical contribution and targeted applications

We appreciate the reviewer's recognition of our theoretical analysis and the consistent improvements across compression models. We are particularly encouraged that the theoretical contribution has also been recognized by reviewer r3yk. Below, we further clarify our theoretical contribution and practical applications in Part 1, followed by additional fusion ablations and the theory-to-implementation connections in Part 2.

W1: Theoretical contribution—finite-delay joint-coding optimality

Answer: Our central contribution is a rigorous finite-delay optimality theorem. Theorem 3.7 establishes that, for general KK-order Markov sources with positive distributions (Definition 3.6), a KK-frame delay suffices to attain the joint-coding RD optimality over a nontrivial distortion region, despite joint coding observes the entire source sequence. Thus, the source’s Markov order—not its sequence length—determines a sufficient delay for achieving the full-sequence coding optimum.

This establishes substantially more than the familiar observation that future frames improve compression. A bounded amount of future information is sufficient to close the RD gap to joint coding within the characterized region. The central challenge is to construct coding auxiliaries that simultaneously satisfy the sequential coding structure and achieve the joint-coding RD optimality. Our proof (Appendix B.2) resolves this through clique-wise auxiliary constructions and node-wise refinements, extending finite-delay optimality beyond Gaussian sources to general Markov sources.

This source-dependent delay principle also guides the practical design. The temporal-dependence analyses on UVG (Fig. 2) and MCL-JCV (new Fig. 17 in Appendix D.4) motivating short lookahead windows. We explain how these analyses guide the practical choice of KK in Part 2.

W1/Q1: Practical applications of limited-delay coding

Answer: We thank the reviewer for highlighting the importance of clarifying the practical applications of limited-delay coding. Our target applications include latency-constrained live video streaming and live transcoding, where practical encoders already offer small, configurable frame-delay budgets. The following documented configurations demonstrate a practical regime where a few frames of delay are traded for compression efficiency. The entries directly quantify B-frame-related structural-delay increments. GFSC addresses the same small-delay regime through an explicit lookahead budget.

Application / systemConfigurationAdded structural delayLatency at 60 fps
Live video streaming / Videon LiveEdge Node [1]1 / 2 / 3 B-frames per sub-GOP; the recommended setting uses 21 / 2 / 3 frames16.7 / 33.3 / 50.0 ms
Live transcoding / AMD-Xilinx Video SDK [2-3]1 B-frame for medium motion; 2 for static/slow motion1 / 2 frames16.7 / 33.3 ms

GFSC adds two latency costs: buffering latency for future frames and computational latency. At 60 fps, K=1,2,3K=1,2,3 require 16.7, 33.3, and 50.0 ms of buffering. Table 2 reports the computational cost separately. Applications can use their remaining latency budget, after communication and processing, to accommodate these additional costs. In summary, GFSC provides a practical framework for exploiting an application's available delay budget to improve compression efficiency, with KK explicitly controlling the trade-off

[1] Videon, Node Encoder Setting Tips.

[2] Xilinx Video SDK, Number of B-Frames.

[3] Xilinx Video SDK, Encoder Latency Configuration.

AuthorsOpenReview ↗

Response to Reviewer zgdi (2/2): Feature fusion ablations and theory-guided implementation

W2/Q2: GFSC framework definition and feature fusion ablation

Answer: Thank you for the suggestion to clarify the definition of GFSC and the source of its performance gains. GFSC is a general delayed sequential coding framework, rather than a specific neural architecture. As defined in Section 3.2, GFSC specifies the admissible information flow:

  • Encoder jointly processes the current and up to KK future frames, together with previously encoded messages.
  • Decoder reconstructs each frame using only the messages received so far.

Importantly, GFSC does not impose any specific feature fusion mechanism within this information flow. Therefore, simple feature concatenation, CNN fusion, and spatial-temporal attention are all valid GFSC instantiations under the same delay budget.

To distinguish the benefits of limited lookahead from those of the particular fusion design, we conduct additional controlled ablations for K=1,2,3K=1,2,3, comparing concatenation with a 1×11\times1 convolution (channel dimension matching), residual CNN fusion, and TJM adopted in the work, under the same training/evaluation protocol. BD-rates are relative to causal DCVC.

Feature fusion ablation

MethodDelay KKUVG BD-rate (%)MCL-JCV BD-rate (%)
DCVC-causal00.000.00
GFSC-Concat1-14.37-9.20
GFSC-Concat2-15.32-10.51
GFSC-Concat3-16.09-11.00
GFSC-CNN1-15.58-8.51
GFSC-CNN2-18.43-10.33
GFSC-CNN3-21.13-11.65
GFSC-TJM1-20.57-10.90
GFSC-TJM2-25.29-17.32
GFSC-TJM3-28.22-20.30

Full results, including computational complexity, are provided in newly added Table 3 and Figure 18 (Appendix D.5). All three fusion methods consistently outperform causal DCVC, demonstrating that the benefits of limited lookahead are not specific to the TJM modules. Meanwhile, TJM achieves the largest BD-rate reductions across all tested delay budgets and datasets, demonstrating its effectiveness in exploiting future-frame information. Overall, GFSC provides the general coding framework for exploiting limited lookahead, while the fusion architecture determines how effectively the available temporal information is utilized.

Q3: From theoretical construction to neural implementation

Answer: Thank you for highlighting the importance of connecting our theoretical results to the practical implementation. The theory guides our design in three aspects:

  1. Selecting the delay budget. Theorem 3.7 establishes a sufficiency delay determined by the source's Markov order rather than sequence length. Figs. 2 and 17 show that the strongest observed dependencies on UVG and MCL-JCV are concentrated within approximately three frame intervals. Table 2 shows approximately linear computational growth with KK, while RD gains exhibit diminishing returns. Together, these observations support K=3K=3 as a practical balance between compression efficiency and latency, when the application budget permits.

  2. Controlling the admissible information flow. Equation (5) specifies the encoder's access to the current and up to KK future frames, with previously encoded messages. Our implementation follows this structure by jointly transforming delayed-window features before quantization, while the decoder uses only messages already received. All feature fusion variants above follow this same information flow.

  3. Constructing the coding representation. The general-source proof in Appendix B.2 constructs clique-wise auxiliaries from local source windows, such that source components become conditionally independent given the auxiliaries. Node-wise refinements further achieve RD optimality within the characterized distortion region. This motivates a joint transform before quantization. The Gaussian specialization makes the transform explicit: X~j,k=ωj,kXj+(1−ωj,k)Xk\tilde X_{j,k}=\omega_{j,k}X_j+(1-\omega_{j,k})X_k (Figure 5). For real video, constructing such representations requires determining both which spatial features to combine and how much each frame contributes. Spatial-temporal attention naturally supports these operations through content-adaptive spatial and cross-frame interactions. The newly added ablation in Figure 18 and Table 3 (Appendix D.5) further demonstrates the empirical effectiveness of this design.

We hope that our response addresses your concern and supports a positive re-evaluation of the submission. Please let us know if you have any further questions regarding our work.

AuthorsOpenReview ↗

Follow up on Rebuttal Response

Dear reviewer zgdi,

As the rebuttal period is approaching its end, we wanted to check whether our responses have adequately addressed your concerns. We would be happy to provide any further clarification if needed.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of submission

We sincerely thank all three reviewers for their thoughtful feedback. We are particularly encouraged by the recognition of our theoretical contribution and the consistent gains across neural video codecs. For ease of reference, we summarize the central result and the main additions in our revised manuscript.

Paper summary. Most neural video codecs use strictly causal prediction. Although bidirectional coding demonstrates the value of future frames, a more fundamental question remains: must a codec access a long future sequence to approach the optimal rate-distortion performance? We address this question by developing Graph-Factored Sequential Coding (GFSC), a principled framework for sequential coding under limited delay.

Central theoretical contributions

  • A surprising finite-delay optimality result. Theorem 3.7 proves that, for positive KK-order Markov sources, a KK-frame delay suffices to attain the full-sequence joint-coding RD optimality over a nontrivial high-resolution distortion region. This result is particularly significant because joint coding has access to the entire source sequence, whereas GFSC requires only a bounded lookahead determined by the source's Markov order, independent of sequence length.
  • A constructive explanation. Building on clique factorization of Markov networks, our proof constructs clique-wise auxiliaries and node-wise refinements that satisfy the sequential coding structure while achieving the joint-coding RD optimality. These results guide GFSC's local-window information flow and joint representations.

Reviews and Revisions. All three reviewers recognized the novelty and significance of our theoretical contribution, with no substantive concerns raised about the correctness of the central optimality result. Their remaining concerns primarily focused on clarifying the framework and its practical implementation, as well as providing additional experimental analyses and stronger baselines.

  • Theoretical contribution and framework clarification. We clarified that GFSC is a general delayed sequential coding framework, not a specific neural architecture. It defines the admissible encoder/decoder information flow without other architectural restrictions. More importantly, our central contribution is not merely demonstrating that future frames improve compression, but establishing a rigorous sufficient-delay guarantee for achieving joint-coding RD optimality. We further explained how the theoretical construction guides practical delay selection, information flow, and joint coding representations. Additional temporal-dependence analysis on MCL-JCV (new Fig. 17) and UVG (Fig. 2) supports the practical motivation for short delay windows.

  • Implementation and feature-fusion ablations. Several reviewers questioned whether the compression gains arise from GFSC's limited-lookahead framework or the particular TJM architecture. To clarify this, we conducted extensive controlled ablations using simple concatenation, CNN fusion, and TJM under identical delay budgets and training/evaluation protocols. All three fusion methods consistently improve over causal DCVC, demonstrating that substantial gains persist even with simple fusion architectures. These results demonstrate the generality of GFSC and the additional effectiveness of TJM in exploiting available future information. We also added complexity and scene-transition analyses to further characterize the practical behavior of GFSC.

  • Stronger coding baselines and experimental generality. We extended GFSC to the more recent DCVC-FM model and added limited-delay VVC-RA comparisons. Due to computational resource constraints, the DCVC-FM-GFSC extension currently covers K=1K=1. Experiments with larger delay budgets are ongoing. Together with the original results on DCVC, VCT, and DHVC, these experiments provide extensive empirical evidence that GFSC's compression benefits generalize across different neural coding architectures.

Overall Perspective. We are encouraged by the reviewers' recognition of our theoretical contributions. Their remaining concerns regarding clarification and experimental evaluation have been addressed through substantial additions to the revised manuscript, with all changes highlighted in blue.

We sincerely thank all reviewers for their engagement throughout the process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.