acceptodds
Under review as a conference paper at ICLR 2027

Learning Selective Reference Dynamics for Perceptual Video Compression

Abstract

Recurrent video codecs must exploit decoded history while limiting the reuse of reference errors. We introduce **SRDVC**, which separates predictive reference evolution from reference-conditioned coding. A **Latent Dynamical System (LDS)** factors reference prediction into dynamics estimation, state evolution, and observation prediction. **Dynamical Manifold Alignment (DMA)** uses current-step features to query the predicted reference, while residual paths convey content not captured by the prediction. A local perturbation model motivates direction-dependent propagation: under bounded total input, uniform contraction of an orthogonal component bounds its accumulation. This conditional analysis provides a design rationale, while temporal-subspace probes characterize the behavior learned by the codec. At matched compute on **UVG**, replacing LDS, DMA, or both degrades perceptual rate–distortion performance; replacing both increases **LPIPS/DISTS BD-Rate by 52.41%/65.76%**. Across these variants, stronger relative directional selectivity accompanies better compression. On **UVG, MCL-JCV, and HEVC-B**, SRDVC achieves mean BD-Rate reductions of **72.67% under LPIPS** and **63.97% under DISTS** relative to **GLC-video**, with efficient steady-state 1080p P-frame coding and favorable viewer preferences over the tested subjective-study baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural Video Compression, World Model, Perceptual Compression

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_j3MHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a perceptual video codec, SRDVC, which decouples predictive reference evolution from reference-conditioned coding. Specifically, an LDS module decomposes dynamics estimation, state evolution, and observation prediction, while a DMA module imposes asymmetric source allocation for reference utilization. The resulting perceptual-quality-optimized video codec achieves improvements over GLC-Video of −72.67 in LPIPS and −63.97 in DISTS.

Critical strengths
  1. The motivation for decoupling predictive reference evolution from reference-conditioned coding is insightful.
  2. The paper provides relatively comprehensive ablation studies and analytical experiments.
Critical weaknesses
  1. I find the overall organization and presentation of the paper rather unusual, specifically in the following aspects:
    1. The paper lacks a clear logical progression from motivation, analysis of existing methods, and problem formulation to the proposed method and its experimental validation. Instead, the entire paper feels like a collection of loosely connected components, making it difficult to assess whether the motivation, methodology, and experimental results reasonably support one another.
    2. The presentation is highly fragmented, even across individual sections and paragraphs. Moreover, many essential theoretical derivations, methodological details, and even experimental settings are only briefly mentioned in the main text, with readers repeatedly referred to the supplementary material for these critical details. This substantially increases the difficulty of reading and understanding the paper.
    3. The paper exhibits a peculiar defensive writing style. In many places, explanations seem to appear abruptly, seemingly intended to preempt potential questions or criticisms, without fitting naturally into the surrounding discussion. One example is Lines 179–182.
    4. The paper contains numerous paper-specific terms and abbreviations that are not adequately explained. For example, it is unclear what "MASP" in Line 298 refers to.
  2. The paper lacks comparisons with recent methods at a comparable level, such as GVC-RT: Towards Real-Time Generative Video Compression at Ultra-Low Bitrates. In addition, the rate-distortion curves suggest that the performance advantages of the proposed method are mainly concentrated in the high-bitrate region, while its performance at low bitrates is slightly inferior.
Questions for authors

Please refer to the above weaknesses to clarify the main logical chain and clear expression of this paper.

Reviewer_4t51OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SRDVC, which separates reference prediction from reference use. LDS predicts the current reference from decoded history through dynamics estimation, state update, and observation prediction. DMA uses current-step features as queries and the predicted reference as keys and values for reference-conditioned coding.

Critical strengths
  • The model separately handles reference evolution and reference-conditioned coding. The overall structure is clear and has practical value.
  • The paper is well written and well presented. The explanations are clear and follow a logical flow.
Critical weaknesses
  • Limited novelty: LDS adopts the decomposition used in latent world models: dynamics estimation, state update, and observation prediction. DMA mainly uses cross-attention between current features and the predicted reference. The contribution largely lies in organizing and combining these existing components within a video codec.
  • Experiments
    • Comparisons with diffusion-based video compression methods are insufficient. The authors should also consider DiffVC, DiffVC-OSD, Free-GVC, and YODA.
    • The authors should consider more perceptual quality metrics, such as FID and FVD.
    • The paper uses synthetic perturbations to estimate directional selectivity. However, it does not examine how actual compression errors relate to these directions. As shown in paper, changes are close to zero, with no consistent ordering between the two directions. Thus, the link between selective reference dynamics and practical compression gains remains unclear. The authors should provide more detailed explanation.

Reference:

[1] Ma W, Chen Z. Diffusion-based perceptual neural video compression with temporal diffusion information reuse[J]. ACM Transactions on Multimedia Computing, Communications and Applications, 2025, 21(12): 1-22.

[2] Ma W, Chen Z. Diffvc-osd: One-step diffusion-based perceptual neural video compression framework[C]//2025 International Conference on Visual Communications and Image Processing (VCIP). IEEE, 2025: 1-5.

[3] Ling X, Zhou C, Li C, et al. Free-gvc: Towards training-free extreme generative video compression with temporal coherence[J]. arXiv preprint arXiv:2602.09868, 2026.

[4] Li X, Zhang J, Shi J, et al. Yet another one-step diffusion-based video compressor[J]. arXiv preprint arXiv:2601.01141, 2026, 1(2).

Questions for authors

See weakness.

Reviewer_Lf4sOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces SRDVC, a perceptual neural video codec that combines factorized latent reference evolution with an asymmetric reference-conditioning interface through two proposed modules:

  • LDS (Latent Dynamical System) estimates dynamics, advances a compact State, and predicts an Observation using only decoded history
  • DMA (Dynamical Manifold Alignment) assigns current features to Queries and the predicted reference to Keys and Values.
Critical strengths

Strong perceptual compression performance. Compared with GLC-video, SRDVC achieves average BD-Rate reductions of 72.67% under LPIPS and 63.97% under DISTS across three datasets.

Critical weaknesses

DMA is essentially a cross-attention mechanism, with current encoded features as the queries and the predicted reference as the keys and values. The paper neither adequately justifies the term “dynamic manifold alignment” nor establishes substantive novelty over GVC1D’s attention-based fusion. Renaming concepts or changing feature sources is insufficient to establish novelty; the authors must clarify the differences in mechanism and validate the resulting compression gains through targeted experiments.

I also see no clear connection between the theoretical analyses, including the local perturbation model, and practical video coding challenges. These analyses appear to be supplementary formalism rather than theoretical foundations for the method. The authors must clarify their practical relevance, justify their assumptions in realistic coding settings, and explain how they guide the method’s design or help interpret the experimental results.

Questions for authors
  1. What specifically distinguishes DMA from standard cross-attention and GVC1D’s attention-based fusion, beyond differences in feature sources? What justifies the term “dynamic manifold alignment”?
  2. Which practical video coding challenges do the theoretical analyses, including the local perturbation model, address? How well do their assumptions reflect realistic coding conditions?
Reviewer_dV4QOpenReview ↗
Rating
2(−1)
Confidence
2(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the updating and use of recurrent references in perceptual video compression. It identifies an important issue in prior methods: useful historical information and historical errors may propagate together. To address this issue, the authors propose SRDVC: LDS predicts future references from historical information through dynamics estimation, state evolution, and observation prediction. DMA uses current-step features as queries and the predicted reference as keys and values, with a residual path separating out new content that was not predicted for use at the next step. The authors also formulate and justify a local model: if a class of perturbations undergoes sustained contraction and the total incoming perturbation at each step is bounded, its accumulation can remain bounded. This motivates their design. The model achieves average LPIPS/DISTS BD-Rate reductions of 72.67%/63.97% relative to GLC-video, together with efficient steady-state 1080p P-frame coding on an A100 GPU.

Critical strengths
  1. The paper identifies the tension between retaining historical information and accumulating errors, and separates different components of historical information through its LDS + DMA design. The problem formulation and the rationale for the proposed improvements are clear, while perceptual rate–distortion results and a viewer preference study support their practical value. Although the method does not outperform GVC1D on DISTS on every dataset, its combination of quality and efficiency remains attractive.
  2. The replacement experiments in Table 2 preserve the specified interfaces and other components, and Table 4 confirms that MAC counts across A–D differ by less than 1%. Replacing LDS, DMA, or both substantially increases perceptual BD-Rate, so the full model's improvement cannot simply be attributed to increased computation. This supports the value of the proposed structures within the tested codec.
Critical weaknesses
  1. The authors do not explain a mechanism that explicitly identifies and suppresses historical errors. SRDVC provides an architecture that may learn to suppress error propagation, along with indirect diagnostics, but does not establish which historical errors are attenuated, by which operation, or to what extent. Although the architecture assigns clear roles to its components, it does not include explicit denoising or error rejection.
  2. The error propagation experiments do not sufficiently validate accumulation under persistent error inputs. Specifically, the experiments characterize how the model responds to disturbances, but a single injected perturbation cannot directly establish its behavior under long-term error accumulation. For example, they do not establish whether accumulated errors are identified and removed once they reach a certain level, or whether the model can keep errors below a threshold over the long term.
Questions for authors
  1. Regarding Weakness 1: Could the authors explain which specific operations are responsible for suppressing the propagation of historical reconstruction errors?
  2. Regarding Weakness 2: Can the existing evidence distinguish suppression of actual reconstruction errors from a smaller relative response to perturbations?
  3. Is there evidence that the accumulation of the relevant perturbations remains bounded, beyond the finite-horizon response to a single perturbation?

Comments on the paper

1 comment
Area_Chair_CuDbOpenReview ↗

Closing discussion after the rebuttal

Thank you to the authors for the rebuttal and to the reviewers for engaging in the discussion. As we wrap up, I’d appreciate a brief final assessment from each reviewer. After reading the rebuttal and the discussion so far, what is the main reason you lean toward acceptance or rejection, and what evidence supports that view? Please note which concerns the rebuttal resolved and which remaining concerns, if any, are decisive for your recommendation.

If your assessment differs from another reviewer’s, please point to the specific claim or evidence you interpret differently. Authors are welcome to clarify any factual points that remain open. This will help us identify where we agree and understand the reasons for any remaining disagreement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.