acceptodds
Under review as a conference paper at ICLR 2027

Unified Generative Video Compression with Autoregressive and Bidirectional Video Diffusion

Abstract

Video diffusion models can reconstruct visually compelling content from severely compressed representations at ultra-low bitrates, but they impose a rigid deployment trade-off. Bidirectional (BI) models achieve high quality through dense full-sequence attention, yet require the entire sequence before reconstruction and incur substantial latency. Autoregressive (AR) models enable low-delay streaming, but their restricted temporal context reduces compression efficiency. This tension is particularly relevant when a video stream must be decoded in real time yet is also stored for later, quality-oriented playback, making AR and BI models natural choices for the live and replay stages, respectively. To support both stages, we present UniGVC, a Unified Generative Video Codec that supports two decoding modes from a single bitstream. UniGVC-BI employs a BI-DiT with dense attention for quality-oriented random-access decoding, whereas UniGVC-AR uses windowed AR-DiT for low-delay streaming. Both paths share a hierarchical latent codec that recurrently produces the same conditions with ultra-low bitrates, allowing UniGVC-BI to serve as a conditional teacher for single-step distillation of UniGVC-AR. A tiny conditional video decoder further reduces streaming latency while preserving most of the reconstruction quality. On standard 1080p benchmarks, UniGVC-BI achieves the best overall rate-distortion-perception trade-off among the evaluated methods and produces temporally stable video at rates as low as 0.0005 bits per pixel. Meanwhile, UniGVC-AR retains competitive reconstruction quality while decoding 1080p video at 36.85 FPS, over faster than UniGVC-BI. Code and models will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Generative Video Compression, Autoregressive Video Generation, Ultra-Low Bitrate

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oVJ2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UniGVC uses one hierarchical latent codec and bitstream to support two generative decoders: a 50-step bidirectional DiT for buffered, quality-oriented reconstruction, and a single-step autoregressive DiT with windowed attention, KV caching, and a conditional tiny decoder for low-delay decoding. Both branches start from Wan2.1-1.3B; codec–LoRA joint training precedes BI-supervised AR distillation and pixel-space refinement. Experiments use the first 96 frames of 1080p HEVC-B, UVG, and MCL-JCV sequences. AR reaches 36.85 decoding FPS on a B300, over 250× faster than the paper’s own BI configuration.

Critical strengths

S1. A useful single-bitstream, multi-mode objective. Separating the transmitted representation from the receiver’s quality–latency choice could support live viewing and higher-quality replay without duplicate bitstreams. The paper develops a coherent implementation of this objective.

S2. Compression-specific conditioning and substantive decoder engineering. Bitstream-derived conditions provide source-aligned temporal context without feeding generated outputs back into subsequent predictions, enabling parallel diffusion training. Table 2 also identifies the original VAE decoder as a bottleneck after one-step distillation and demonstrates useful quality recovery: with adversarial DMD and pixel fine-tuning already enabled, codec conditioning reduces the BD-LPIPS gap from +0.058 to +0.001. These are meaningful results, even though their independent contributions require stronger validation.

Critical weaknesses

W1. The main AR evaluation does not validate causal online streaming.

Appendix D.1 generates captions from complete 96-frame videos, including future motion and events. Causal attention does not remove this non-causal side information. Although Table 9 provides a fixed-prompt ablation on HEVC-B, the main AR comparisons across all three benchmarks are not reproduced using fixed prompts or captions restricted to currently available frames. The reported quality therefore does not establish performance in the intended online setting.

W2. Decoder throughput does not establish end-to-end real-time performance, and deployment costs are incompletely reported.

The 36.85 FPS result measures decoding on a B300, while 1080p encoding requires 86 ms/frame. The reported >250× speedup is relative to the paper’s own 50-step BI configuration. End-to-end latency, including caption generation, text encoding and initial buffering, is not established. FLOPs/MACs, peak memory, KV-cache costs and specialized inference optimizations are insufficiently documented. Performance on other GPUs is also unvalidated, limiting the practical scope of the deployment claim.

W3. Perceptual superiority lacks convincing human and full-video validation against stronger generative baselines.

The user study includes only VTM-LD, DCVC-RT and GLC-Video, despite the paper acknowledging that small perceptual-metric gaps may not reflect human judgments. Fig. 10 also raises concerns about source-appearance and pose fidelity despite improved semantic plausibility. The supplied still images and temporal strips cannot adequately establish identity preservation, motion consistency or source fidelity. Full-video comparisons with and without captions, together with human comparisons against stronger generative baselines, are needed.

W4. The methodological contribution is insufficiently isolated from established designs and simpler alternatives.

GLVC already uses recurrent latent coding, GNVC-VD applies video-DiT refinement, and CausVid performs BI-to-causal distribution matching. Their integration does not by itself establish the benefit of UniGVC’s specific formulation. Missing controls include joint codec–BI/AR training versus a shared codec with independently adapted decoders, and quantitative contextual-autoregression comparisons against TF/SF. Table 2 provides useful decoder ablations, but does not isolate acceleration from UniGVC-specific changes versus inherited DCVC-family optimizations, including DC blocks and the DCVC-UF entropy-decoding strategy.

W5. Baseline coverage and cross-method evaluation consistency are insufficient.

GVC-RT and YODA are omitted. GVC1D and S2VC are included only through author-provided results rather than local reproduction under common settings. The GitHub repository was made open-source before the submission deadline. External results for unavailable methods such as GLVC and GNVC-VD are understandable, but their compatibility in preprocessing, frame selection, metric implementation and operating points is insufficiently documented. Consistent quality and runtime comparisons with reproducible generative baselines are needed to substantiate the claimed quality–latency advantage.

W6. Architectural gains are not disentangled from training scale and additional optimization.

The system uses one million 720p clips and 11K 1080p clips, followed by joint training, high-resolution adaptation, adversarial distillation and pixel-space refinement. Although Table 2 ablates selected training components, matched-data and matched-budget controls are missing for the central architectural comparisons. The results therefore establish the performance of the complete recipe, but not how much improvement comes from the proposed architecture rather than training data and additional optimization stages.

Questions for authors

Q1. Bitrate accounting.

Do the main bpp values and BD-rate calculations include the compressed sequence-caption bytes, or is this overhead reported separately in Appendix D?

Q2. Baseline availability.

What was the cutoff date for determining baseline availability, and were these checks updated before submission? For GVC1D and S2VC, did “unavailable” refer to code, checkpoints or both?

Q3. Supplementary videos.

Why were reconstructed videos not provided, given their importance for assessing perceptual quality, temporal consistency and source fidelity?

4 comments
AuthorsOpenReview ↗

Response to Reviewer oVJ2 (Part 1/4)

We sincerely thank Reviewer oVJ2 for the thorough evaluation and suggestion on UniGVC, and for recognizing its core contributions on supporting multi-mode decoding with a single bitstream, compression-specific autoregressive conditioning and substantive decoder optimization. Our point-by-point responses are provided below.

W1, W2. Clarification on live-streaming application and decoder throughput.

We thank the reviewer for raising these concerns. We would like to first emphasize that UniGVC-AR claims real-time decoding capability at the receiver side, rather than end-to-end real-time encoding and decoding on a single device. We believe it is important to distinguish these two concepts when evaluating practical video coding systems.

In fact, practical video coding and live-streaming systems commonly have two characteristics:

  1. A reserved fixed latency between the encoder and decoder. Unlike interactive applications requiring extremely low latency, many live-streaming applications, such as live broadcasting, operate with a fixed delivery delay, often on the order of minutes. Real-time operation in such scenarios does not require zero-latency end-to-end processing.
  2. Asymmetric computational resources. The encoder side typically has access to substantially more computational resources and can leverage parallel encoding across multiple GPUs or machines, whereas the decoder side is generally deployed on a single client device.

Under this common streaming architecture, UniGVC can support practical live-streaming applications:

  • Each GOP (96 frames by default) can be independently encoded. Upon receiving the first frame of a GOP, the encoder can immediately allocate available computational resources to process that GOP. By encoding multiple GOPs in parallel, the overall encoding throughput can be scaled to match the incoming video frame rate.
  • The full-GOP caption is generated after the corresponding GOP has been collected, and is transmitted as side information within the reserved delivery latency. Therefore, our streaming deployment does not require the caption to be available at the beginning of the GOP.

After the system reaches a steady state, both the encoding and decoding sides can operate in real time, given sufficient parallel computational resources at the encoder and a reserved fixed latency.

We acknowledge that our current encoder implementation requires approximately 86 ms per frame at 1080p (11.6 FPS) on a single GPU. However, this does not contradict our receiver-side real-time decoding claim, as the intended deployment architecture allows parallel computational resources at the encoder and a fixed delivery latency between the two sides.

W2. Deployment cost.

We provide additional measurements of the computational and memory requirements of UniGVC at 1080p resolution, including peak GPU memory, MACs per pixel, and the memory required for the autoregressive KV cache.

MethodEnc. Peak MemoryEnc. MACs/pixelDec. Peak MemoryDec. MACs/pixelDec. KV Cache
UniGVC-BI22.26 GiB2901K28.02 GiB2043M0
UniGVC-AR22.26 GiB2901K13.49 GiB3156K2.89 GiB

We clarify that we do not employ additional specialized inference optimizations beyond the proposed model design and decoding procedure. Furthermore, our current experimental environment does not provide other GPU models for cross-device evaluation. We will explicitly state this hardware limitation and report the above computational costs to better characterize the deployment requirements.

For sequence-level text conditioning, the additional text-related operations can be completed within the reserved fixed latency between encoding and decoding. Using HEVC-B with a GOP size of 96 as an example, we provide a detailed breakdown of the additional latency introduced by textual side information:

Textual Processing StageLatency
Encoder side: Waiting for a complete GOP (e.g., 30 FPS)3.2 s
Encoder side: Qwen3.5-397B-A17B API (video caption generation)6.5 s
Encoder side: Text encoding61.6 ms
Decoder side: Text decoding0.05 ms
Decoder side: Wan2.1 text encoder22.3 ms
Decoder side: KV cache construction4.2 ms
Total textual latency9.8 s

The additional latency introduced by textual processing is approximately 9.8 seconds, primarily due to video caption generation (6.5 seconds). This overhead can be accommodated within the minute-level fixed delivery latency commonly available in live-streaming applications.

We will add a dedicated discussion in the revised final manuscript. In particular, we will explicitly distinguish receiver-side real-time decoding from end-to-end real-time processing and clarify that extremely low-latency applications, such as video conferencing, are beyond the current scope of this work.

AuthorsOpenReview ↗

Response to Reviewer oVJ2 (Part 2/4)

W3. Human preference against additional baselines.

We have conducted additional full-video comparisons, including comparisons with and without sequence-level captions, and extended the human preference study to include the raised and stronger generative baselines: GVC-RT [1], YODA [2], GVC1D [3] and S2VC [4]. The experimental protocol follows the same settings as Figure 8. For each comparison, we collected at least 30 valid human votes. The results on the HEVC-B test set are summarized below:

UniGVC Preferred (%) ↑GVC1D, 0.0058 bppGVC-RT, 0.0043 bppYODA, 0.0061 bppS2VC, 0.0057 bpp
UniGVC-BI, 0.0038 bpp72.7%93.3%87.9%76.7%
UniGVC-AR, 0.0038 bpp67.7%96.9%84.8%63.3%
UniGVC-AR, 0.0038 bpp (w/o captions)64.5%93.3%83.3%60.0%

The additional human evaluations show that UniGVC maintains a strong perceptual preference over these generative baselines, despite operating at lower bitrates.

W4. Clarification on methodological contribution.

We appreciate the reviewer for pointing out the connections to existing works. However, we would like to clarify that UniGVC is not simply an integration of GLVC, GNVC-VD, and CausVid. The central research question of UniGVC is fundamentally different: we investigate how to develop a single compressed representation that supports two distinct generative decoding modes, namely bidirectional DiT (BI-DiT) for quality-oriented reconstruction and autoregressive DiT (AR-DiT) for low-latency reconstruction.

To the best of our knowledge, this unified single-bitstream, multi-mode decoding objective has not been explored in the aforementioned works. Specifically:

  • Compared with GLVC, which primarily investigates latent-space video compression using the Wan2.1 VAE, UniGVC explores a unified codec jointly supporting BI-DiT and AR-DiT decoding. In addition, we propose HEM to improve coding efficiency under ultra-low-bitrate conditions.

  • Compared with GNVC-VD, which employs ControlNet to guide BI-DiT for generative video compression, UniGVC investigates the joint optimization of the codec with both BI-DiT and AR-DiT. In particular, we develop codec-conditioned autoregressive training and inference strategies, together with distillation-based acceleration for AR decoding.

  • Compared with CausVid, which transforms a bidirectional video generation model into an autoregressive model through DMD, UniGVC addresses the distinct problem of generative video compression. We further develop codec-conditioned adversarial DMD distillation and a Conditional Tiny Decoder to accelerate generative reconstruction under compressed codec conditions.

W4. Additional ablation and discussion.

To further address the concerns regarding the specific design choices, we have conducted additional quantitative ablations.

1. Comparison between independently optimized and unified codecs.

We compare the unified codec against two independently optimized codecs, where the codec is jointly trained with either BI-DiT or AR-DiT alone. The independent codecs achieve better rate–perception performance, quantifying the performance cost of sharing a single representation across two decoding modes.

MethodLPIPS (%) ↓DISTS (%) ↓
Independent-BI-5.42-12.23
UniGVC-BI00
MethodLPIPS (%) ↓DISTS (%) ↓
Independent-AR-2.81-6.15
UniGVC-AR00

2. Quantitative comparison of contextual autoregression strategies.

We further provide quantitative BD-rate comparisons between Diffusion Forcing (DF), Teacher Forcing (TF), and Self Forcing (SF), using DF as the anchor:

MethodLPIPS (%) ↓DISTS (%) ↓
SF+44.57+61.24
TF+19.81+35.40
DF (Ours)00

3. Clarification on UniGVC-AR's decoder acceleration.

Regarding the comparison with DCVC-family optimizations, we would like to clarify that our decoder acceleration primarily targets the computational bottlenecks of the DiT and VAE reconstruction pipeline, rather than the entropy coding process of the codec.

Specifically, DC blocks and the DCVC-UF entropy-decoding strategy optimize different components of the coding pipeline and are therefore not directly comparable to our generative decoder acceleration. Table 2 has explicitly isolated the codec decoding cost and focuses on our acceleration and reconstruction improvement of the DiT and VAE components.

AuthorsOpenReview ↗

Response to Reviewer oVJ2 (Part 3/4)

W5, Q2. Additional baseline under consistent evaluation.

We thank the reviewer for the suggestion. we have extended the baseline evaluation to include GVC-RT [1] and YODA [2]. Furthermore, we have used the official implementations to reproduce GVC1D [3] and S2VC [4] under our evaluation settings, rather than relying solely on author-provided results.

The updated comparisons are reflected in Figures 6 and 7 of the revised manuscript. The results show that UniGVC maintains the best overall rate–distortion–perception performance among the evaluated methods, while achieving temporal characteristics closer to the ground-truth videos.

Regarding runtime comparisons, we would like to clarify that the primary objective of UniGVC is to provide unified bidirectional and autoregressive decoding from a single bitstream, rather than simply accelerating an individual decoding architecture. Since the aforementioned methods do not provide the same bidirectional decoding capability, a direct comparison of their decoding speeds would not represent equivalent functionality. We therefore focus our runtime evaluation on characterizing the quality–latency operating points supported by UniGVC, rather than claiming universal decoding-speed superiority over all baselines.

W6. Clarification on architectural gains, training scale and additional optimization.

We would like to clarify that the training data and optimization procedures used in UniGVC follow established practices in recent generative video compression and high-resolution video generation works. They are introduced to address specific technical requirements of our framework, rather than to provide an unfair advantage over prior methods.

Specifically:

  • Video codec training data: Following GLVC [5], we use 1 million 720p video clips from Koala36M for training. Importantly, the codec is trained using videos at resolutions up to 480p, following the established training scale for learning latent-space codecs under the Wan2.1 VAE.

  • 1080p DiT adaptation: UltraVideo [6] identifies the limitation of directly applying the pretrained Wan2.1 DiT to 1080p video inference. Accordingly, our high-resolution DiT adaptation follows the 1080p training data and adaptation strategy established in UltraVideo.

  • Joint training is essential to establish a shared codec that supports both BI-DiT and AR-DiT decoding.

  • Adversarial distillation and pixel-space refinement: These stages are introduced to address the reconstruction-quality bottlenecks of one-step generative decoding. Their contributions have been separately investigated in Table 2.

Therefore, these training stages are closely tied to the technical requirements of UniGVC, and their respective contributions are examined through component-level ablations.

Furthermore, to more directly validate the architectural contribution of the proposed HEM entropy model in Figure 2, we conduct an additional experiment on HEVC-C. Specifically, we independently train the codec with BI-DiT, replacing only the proposed HEM with a standard Factorized Hyperprior.

Entropy ModelLPIPS (%) ↓DISTS (%) ↓
Factorized Hyperprior+36.35+52.68
HEM (Ours)00

Q1. Bitrate accounting.

We thank the reviewer for requesting this clarification. We confirm that the bpp values and BD-rate calculations reported in the main paper include the compressed sequence-caption bytes.

Q2. Baseline availability.

In the revised evaluation, we have further checked baseline availability (the cutoff date is 2026/11/10) and used the official implementations to reproduce GVC-RT, GVC1D, YODA, and S2VC under our evaluation settings. Please refer to our response to W5 and the updated results in Figures 6 and 7.

Q3. Supplementary videos.

We apologize that supplementary videos were not included in the original submission. Our code and data release are currently undergoing internal review. Meanwhile, as described in our response to W3, we have extended the full-video human preference study to include additional generative baselines and comparisons, providing further subjective evidence of the proposed method's performance.

We commit to releasing the demonstration videos, source code, and model checkpoints upon paper acceptance, subject to the completion of the internal release process.

AuthorsOpenReview ↗

Response to Reviewer oVJ2 (Part 4/4)

References
[1] Dang et al., 2026, "GVC-RT: Towards Real-Time Generative Video Compression at Ultra-Low Bitrates".
[2] Li et al., 2026, "YODA: Yet Another One-step Diffusion-based Video Compressor".
[3] Zheng at al., 2026, "Generative Video Compression with One-Dimensional Latent Representation".
[4] Xue et al., 2026, "Single-step Diffusion-based Video Coding with Semantic-Temporal Guidance".
[5] Guo et al., 2025, "Generative Latent Video Compression".
[6] Xue et al., 2025, "UltraVideo: High-Quality UHD Video Dataset with Comprehensive Captions".

Reviewer_tV4KOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UniGVC uses a shared compressed representation to support a high-quality bidirectional diffusion decoder and a low-latency autoregressive decoder. The framework combines hierarchical entropy modeling, joint codec–diffusion training, single-step distillation, and a lightweight conditional pixel decoder.

Critical strengths
  1. The proposed method supports multiple decoding quality-latency operating points.

  2. The method integrates codec desgin and generative decoding coherently.

Critical weaknesses
  1. The evaluation does not fully establish the claimed live-streaming setting. Appendix D.1 generates a sequence-specific caption from the complete 96-frame video, and both decoding modes use this caption. Such information is unavailable at the beginning of a genuinely live stream. Consequently, the experiments establish fast decoding with precomputed sequence-level information, but do not fully establish causal operation from online acquisition.
  2. Decoder throughput should be distinguished from complete system throughput. Appendix Table 6 reports approximately 86 ms per frame for 1080p encoding, corresponding to approximately 11.6 FPS. The listed encoding pipeline also does not include caption generation. The real-time decoding claim should therefore remain explicitly scoped to the receiver-side computation.
  3. Random-access support needs a more precise operational definition. The codec maintains recurrent state, while the principal evaluation uses an intra-period of −1. Processing a buffered segment with bidirectional attention does not by itself establish independent access to arbitrary temporal positions. The required state reconstruction, refresh points, and associated bitrate overhead should be specified.
  4. The cost of sharing the representation is insufficiently quantified. A comparison against separately optimized BI-only and AR-only codecs would reveal the rate–distortion penalty, if any, incurred by the unified bitstream.
  5. Textual side-information costs are reported, which is valuable. However, their inclusion in the main rate curves, caption-generation latency, and caption-update behavior for longer streams should be made explicit.
Questions for authors
  1. Responses to the weaknesses.

  2. And also:

  • What quality and latency are obtained when captions are restricted to the first frame or previously observed frames?
  • How are random-access entry points and recurrent codec states handled?
  • How does the shared codec compare with independently optimized codecs for the two decoding modes?
4 comments
AuthorsOpenReview ↗

Response to Reviewer tV4K (Part 1/2)

We appreciate the positive overall assessment from Reviewer tV4K, particularly on the multiple decoding quality-latency operating supports of UniGVC. To address the raised concerns, we are happy provide additional discussion and ablation.

W1, W2. Discussion on streaming and throughput.

We thank the reviewer for raising these concerns. We would first like to clarify two common characteristics of practical video coding and streaming systems:

  1. A reserved fixed latency between the encoder and decoder: Practical video streaming systems typically operate with a certain delivery latency between encoding and decoding, rather than requiring zero-latency end-to-end processing. This is particularly common in live broadcasting, where the receiver operates with a fixed delay relative to the video source, often on the order of minutes.

  2. Asymmetric computational resources: The encoder side typically has access to substantially more computational resources and can leverage parallel encoding across multiple GPUs or machines, whereas the decoder side is generally deployed on a single client device.

Accordingly, the real-time claim of UniGVC-AR specifically refers to receiver-side real-time decoding, rather than end-to-end real-time encoding and decoding on a single device. We believe this distinction is important for understanding the intended application scenario of our method.

We further explain how UniGVC can be deployed in practical live-streaming scenarios.

  • Each GOP (96 frames by default) can be independently encoded. Upon receiving the first frame of a GOP, the encoder can immediately allocate available computational resources to process that GOP. By encoding multiple GOPs in parallel, the overall encoding throughput can be scaled to match the incoming video frame rate.
  • The full-GOP caption is generated after the corresponding GOP has been collected, and is transmitted as side information within the reserved delivery latency. Therefore, our streaming deployment does not require the caption to be available at the beginning of the GOP.

After the system reaches a steady state, both the encoding and decoding sides can operate in real time, given sufficient parallel computational resources at the encoder and a reserved fixed latency.

We acknowledge that the current encoder implementation does not achieve real-time encoding on a single GPU (approximately 11.6 FPS at 1080p, as reported in Appendix Table 6). However, this does not contradict our receiver-side real-time decoding claim, given the aforementioned asymmetric streaming architecture.

W3, Q2. Discussion on random-access decoding.

We would like to clarify that the random-access decoding in UniGVC-BI is achieved through inserting I-frames, which differs from the classical video coding. In our intended random-access configuration, the GOP size is set to 96 frames by default, with an I-frame inserted at the beginning of each GOP. Each GOP can therefore be independently encoded and decoded. Specifically, the recurrent codec states are reset when encoding an I-frame, allowing subsequent frames within the GOP to be processed independently of the preceding GOP.

Consequently, random access is supported at GOP boundaries, rather than at arbitrary individual frame positions. To access a particular GOP, the decoder only needs to receive its corresponding I-frame and the subsequent coded representations within that GOP, without reconstructing the codec states from previous GOPs. We will explicitly clarify these configuration in the revised final manuscript.

W4, Q3. Comparison between shared codec and independently optimized codec.

We thank the reviewer for suggesting this comparison. Specifically, we train two independently optimized codecs: one jointly optimized with the BI-DiT only (Independent-BI), and the other jointly optimized with the AR-DiT only (Independent-AR). We then compare each independent codec against the corresponding decoding mode of UniGVC on HEVC-C using BD-rate. The independent codecs achieve better rate–perception performance, quantifying the performance cost of sharing a single representation across two decoding modes.

MethodLPIPS (%) ↓DISTS (%) ↓
Independent-BI-5.42-12.23
UniGVC-BI00
MethodLPIPS (%) ↓DISTS (%) ↓
Independent-AR-2.81-6.15
UniGVC-AR00
AuthorsOpenReview ↗

Response to Reviewer tV4K (Part 2/2)

W5. Discussion on textual prompt.

We would like to clarify the cost and behavior of textual side information.

Main rate–quality curves. We confirm that the main rate–quality results reported in the main text use sequence-level captions as additional conditioning information. We will explicitly clarify this term in the revised manuscript.

Caption-generation latency. When sequence-level prompts are enabled, the additional text-related operations can be completed within the reserved fixed latency between encoding and decoding. To quantify this overhead, we provide a breakdown of the additional latency on HEVC-B.

Textual Processing stageLatency
Encoder side: Waiting for a complete GOP (e.g., 30 FPS)3.2 s
Encoder side: Qwen3.5-397B-A17B API (video caption generation)6.5 s
Encoder side: Text encoding61.6 ms
Decoder side: Text decoding0.05 ms
Decoder side: Wan2.1 text encoder22.3 ms
Decoder side: KV cache construction4.2 ms
Total textual latency9.8 s

The additional latency introduced by the text-conditioning pipeline is approximately 9.8 seconds, primarily due to the video caption API call (6.5 seconds). This overhead can be accommodated within the minute-level fixed delivery latency commonly available in live-streaming applications.

Caption-update behavior. In our default configuration, one sequence-level caption is extracted and encoded for every GOP (96 frames). The encoder extracts the caption based on the GOP and transmits it as side information. Upon receiving and decoding the caption, the decoder caches its keys and values in the DiT for decoding the current GOP.

Q1. How quality and latency are obtained when captions are restricted to the first frame or previously observed frames?

To address this concern, we have conducted additional experiments on the HEVC-B test set using UniGVC-AR. Specifically, we investigate three caption-generation strategies for each GOP (96 frames by default):

CaptionLPIPSDISTSTotal Textual Latency
None (General Only)000
From the first frame-1.37-3.595.8 s
From the first 48 frames-2.83-5.378.1 s
From the entire 96 frames-2.64-6.049.3 s

The results demonstrate that even when captions are restricted to causally available visual information, they can still provide meaningful improvements over general-only conditioning.

Regarding latency, restricting caption generation to the first frame or the first 48 frames reduces the time required to collect the frames before caption generation. However, the overall latency remains substantially influenced by the external video caption API, which incurs more than 5 seconds of response time per caption in our experiments due to network and API response overhead.

Reviewer_tV4KOpenReview ↗

Responses to the rebuttal

The rebuttal clarifies the system positioning and acknowledges the performance cost of shared representations, but it has not fully addressed the gap between experimental setup and actual deployment. The reviewer tend to keep the score.

AuthorsOpenReview ↗

Response to Reviewer tV4K

Thank you for your reply, and we are glad that most of the concerns have been solved.

We agree that there remains a gap between our experimental setup and practical deployment. Our work demonstrates the feasibility of UniGVC-AR's low-delay streaming under a favorable computational setting, rather than providing a fully deployment-ready solution for diverse receiving devices. We have explicitly acknowledged this limitation in the revised manuscript's Limitations section.

Nevertheless, we believe the demonstrated capability is meaningful given the substantial computational demands of video diffusion models. To the best of our knowledge, achieving low-delay decoding while maintaining competitive reconstruction quality remains a challenging problem, even with powerful hardware. Existing approaches often face a trade-off between decoding speed and generation quality. UniGVC takes a step toward addressing this challenge by enabling low-delay autoregressive decoding while retaining a high-quality bidirectional decoding option from the same bitstream. We hope this progress could be considered alongside the acknowledged deployment limitations when assessing the overall contribution of this work.

Once again, thank you for your time and overall feedback for our UniGVC!

Reviewer_dmNoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UniGVC propose a method to support bidirectional and autoregressive video reconstruction from a single compressed representation. A shared recurrent latent codec supplies conditions to both diffusion decoders, enabling quality-oriented playback and lower-latency decoding without transcoding the bitstream. The autoregressive branch is accelerated through one-step distillation and a conditional tiny decoder. Experiments on three standard video benchmarks demonstrate strong perceptual reconstruction, with autoregressive decoding reaching 36.85 FPS at 1080p on a B300 GPU.

Critical strengths
  • Useful shared-bitstream capability. Supporting both quality-oriented and fast decoding from the same representation provides a clear practical benefit.

  • Strong reconstruction evidence. Rate–quality curves, temporal measures, qualitative examples, and human preferences provide complementary support for reconstruction quality.

Critical weaknesses
  • Incomplete live-streaming validation. Evaluation uses captions generated from complete 96-frame videos, including information unavailable at the beginning of a live stream I think. The reported 1080p encoder also takes 86 ms per frame, approximately 11.6 FPS, before caption-generation time is considered. The 36.85-FPS result establishes real-time decoding, but does not establish the motivating end-to-end live-streaming workflow claimed.

  • Limited validation of contextual autoregression. Using codec conditions rather than previously generated latents as history is an important methodological choice. However, its direct comparison with conditional teacher forcing and self forcing is limited to qualitative examples in Figure 4. The quantitative ablations do not establish the rate–quality or temporal-stability differences among these alternatives.

  • Unclear contributions in distillation. CausVid already establishes bidirectional-teacher-to-causal-student distillation using DMD. UniGVC extends this approach to codec-conditioned reconstruction and adds adversarial supervision, but Table 2 compares the complete adversarial DMD procedure against its absence rather than against DMD alone. The results therefore demonstrate the combined procedure’s effectiveness without isolating the contribution of the added adversarial design, leaving this component’s methodological advance insufficiently established.

Questions for authors
  • Can the existing DF, TF, and SF outputs support quantitative comparisons at comparable bitrates?
  • Is a DMD-only control available under the same codec condition and student architecture? This would clarify how much improvement comes from the added adversarial supervision.
4 comments
AuthorsOpenReview ↗

Response to Reviewer dmNo (Part 1/2)

We sincerely thank Reviewer dmNo for the positive assessment of UniGVC, and for recognizing the useful shared-bitstream capability and strong reconstruction evidence presented in UniGVC. We are willing to provide additional discussion accordingly.

W1. Clarification on live-streaming application.

We thank the reviewer for raising this concern. We would first like to clarify two common characteristics of practical video coding and streaming systems:

  1. A reserved fixed latency between the encoder and decoder: Practical streaming systems typically allow a certain delivery latency between encoding and decoding, rather than requiring zero-latency end-to-end processing. This is particularly common in live broadcasting, where the decoder operates with a fixed delay relative to the video source, which can be on the order of minutes.

  2. Asymmetric computational resources: The encoder side typically has access to more computational resources and can leverage parallel encoding across multiple GPUs or machines, whereas the decoder side is usually deployed on a single client device.

Accordingly, the real-time claim of UniGVC-AR refers to real-time decoding, rather than zero-latency end-to-end processing. We further detail below how UniGVC can be deployed in practical real-time streaming scenarios.

Without sequence-level prompts, each GOP (96 frames by default) can be independently encoded and decoded. Upon receiving the first frame of a GOP, the encoder can immediately allocate available computational resources to process that GOP. By encoding multiple GOPs in parallel using multiple computational resources, the overall encoding throughput can be scaled to match the frame rate of the incoming video stream.

When sequence-level prompts are enabled, the additional text-related operations can be completed within the reserved fixed latency between encoding and decoding. To quantify this overhead, we provide a detailed breakdown of the additional latency introduced by text conditioning on the HEVC-B test set.

Textual Processing stageLatency
Encoder side: Waiting for a complete GOP (e.g., 30 FPS)3.2 s
Encoder side: Qwen3.5-397B-A17B API (video caption generation)6.5 s
Encoder side: Text encoding61.6 ms
Decoder side: Text decoding0.05 ms
Decoder side: Wan2.1 text encoder22.3 ms
Decoder side: KV cache construction4.2 ms
Total textual latency9.8 s

The additional latency introduced by the text-conditioning pipeline is approximately 9.8 seconds, primarily due to video caption generation (6.5 seconds). This overhead can be readily accommodated within the minute-level fixed delivery latency commonly available in live-streaming applications.

In summary, UniGVC-AR claims real-time decoding capability and supports practical live-streaming scenarios. After the system reaches a steady state, both the encoding and decoding sides can operate in real time, given sufficient parallel computational resources at the encoder and a reserved fixed latency.

Nevertheless, we acknowledge that the current encoder complexity is not optimized for extremely low-latency applications, such as video conferencing, where the available encoding–decoding delay is highly constrained. Addressing this limitation is an important direction for our future work. We will include a dedicated discussion of these practical deployment considerations and the associated limitations in the revised manuscript.

W2, Q1. Quantitative comparisons of DF, TF, and SF.

We appreciate the reviewer’s suggestion. We agree that the qualitative comparisons in Figure 4 alone are insufficient to fully characterize the differences among Diffusion Forcing (DF), Teacher Forcing (TF), and Self Forcing (SF). To address this concern, we have conducted additional quantitative comparisons and report the corresponding BD-rate results for LPIPS and DISTS. The experiments are performed on the HEVC-C test set using four bitrate points uniformly selected within the range of 0.001–0.006 bpp. DF, TF, and SF are evaluated under the same experimental settings, with DF serving as the anchor.

MethodLPIPS (%) ↓DISTS (%) ↓
SF+44.57+61.24
TF+19.81+35.40
DF (Ours)00

These findings provide quantitative support for the qualitative observations in Figure 4 and further justify our choice of codec-conditioned autoregressive history.

AuthorsOpenReview ↗

Response to Reviewer dmNo (Part 2/2)

W3, Q2. Ablation on the DMD-only distillation.

To address this concern, we have conducted an additional ablation study by introducing a DMD-only distillation baseline between Exp. 2 and Exp. 3 in Table 2. Specifically, we maintain exactly the same student architecture and experimental settings as those in Table 2, while removing the adversarial supervision from the distillation objective. The metrics are evaluated using UniGVC-BI as the anchor.

MethodBD-LPIPS ↓BD-DISTS ↓
No distillation (from Table 2 Exp. 2)+0.069+0.142
DMD only+0.057+0.104
Adv. DMD (from Table 2 Exp. 3)+0.041+0.045

The results provide a clearer picture of the respective contributions of the two components.

Thank you for the reply. My concerns have been addressed and I will maintain the positive score.

AuthorsOpenReview ↗

General response to Reviewer dmNo

Thank you for your reply! We are pleased to be able to address all of your concerns. We appreciate your suggestions and positive feedback all along.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on the response

Dear Reviewers,

We appreciate your participation in providing feedback and thorough review for UniGVC, and we have posted our point-by-point responses for every raised concern.

We believe these responses significantly strengthen the manuscript, and we'd really appreciate it if you could take a look. With the discussion period ending on November 18, we would like to make sure there's enough time for any follow-up question.

Thank you again for your time and effort in reviewing our work.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.