acceptodds
Under review as a conference paper at ICLR 2027

RIFT: Residual-Informed Flow Transfer for One-Step Generative Video Compression

Abstract

Recent advances in video generation have enabled generative video compression to reconstruct details lost during coding using pretrained spatiotemporal priors. However, a substantial gap remains between codec-induced degradation and the perturbation trajectories learned by flow-matching models. In this paper, we identify a compression-dependent directional mismatch that degradation-strength-based timestep calibration alone cannot resolve. Compressed latents remain strongly aligned across codec operating points, while their reconstruction residuals vary substantially in magnitude and direction. To bridge this gap, we propose RIFT, a Residual-Informed Flow Transfer framework for one-step generative video compression. At its core, RIFT introduces a Residual Geometry Inducer (RGI) that converts the pretrained flow prediction into a content-adaptive restoration update using decoder-visible codec cues. RGI integrates flow-timestep conditioning with a learned full-band residual-strength estimate and content-conditioned direction and gain modulation, aligning the orientation and magnitude of the update with the compression-dependent residual geometry. Extensive experiments on three public video compression benchmarks show that RIFT outperforms competing generative video compression methods.

Primary area:
generative models
Keywords:
Generative video compression, flow matching, one-step restoration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bXRBOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the mismatch between the perturbation trajectories learned by pretrained flow-matching models in generative video codecs and the degradation patterns introduced by compression. To tackle this problem, the authors propose a Residual-Informed Flow Transfer (RIFT) framework. Based on a Residual Geometry Inducer (RGI), the framework estimates residual strength and modulates the restoration direction and local gain, aligning the direction and magnitude of flow-model updates with the geometry of compression-induced residuals. This ultimately improves perceptually optimized rate–distortion performance.

Critical strengths
  1. Reasonable motivation supported by theoretical and experimental analysis. The paper demonstrates that compressed latents remain highly aligned across different QPs, while their reconstruction residuals differ substantially in both magnitude and direction.
  2. Clearly motivated method design. The proposed modules correspond well to the phenomena identified in the motivation and analysis.
Critical weaknesses
  1. The performance results do not yet convincingly validate the motivation and the proposed method.
  • The evaluation omits several recent and competitive baselines, including Generative Video Compression with One-Dimensional Latent Representation, YODA: Yet Another One-step Diffusion-based Video Compressor, and GVC-RT: Towards Real-Time Generative Video Compression at Ultra-Low Bitrates.
  • The results in the appendix suggest that the proposed method performs slightly worse at 720p.
  • The motivation and analysis emphasize the mismatch between the direction of the model update and that of the latent reconstruction residual. However, in Table 1, ablating this component results in only small changes in BD-Rate (LPIPS) and BD-Rate (DISTS), at 0.96% and 3.76%, respectively. In contrast, the main performance gains appear to come from the learned gain module. This somewhat weakens the paper's claimed contribution.
  1. Key analyses of computational complexity and encoding/decoding latency are missing.
Questions for authors

My question mainly concerns clearer explanations of performance and complexity, as well as verifying the motivation and methods of this paper. If the author completes these reasonable explanations, I will consider improving my score.

5 comments
AuthorsOpenReview ↗

Answer to Reviewer bXRB

Thank you for identifying baseline coverage, the directional mechanism, and runtime as key questions. We have expanded the spatial-performance comparisons to include YoDA, GVC-RT, GVC1D, VVC, and DCVC-UF, and added detailed temporal and runtime results.

[W1] 1. Thank you for suggesting these additional baselines. When we began this work, the methods you mentioned had not yet released their code, so they were not included in the initial version. We have added their reported performance to the revised manuscript. During our reproduction, we found that YoDA's measured bitrate differs from the entropy-estimated bitrate reported for its Fast setting. We use the actual bitrate in our evaluation; the other results are consistent with those reported in the original paper. The official GVC1D evaluation resizes videos to 1792 × 1024, which does not allow for a fair comparison at the original resolution. When we used the original resolution as input, GVC1D's performance declined substantially. We therefore reimplemented its evaluation pipeline by padding the bottom and right edges to multiples of 256, then cropping the output back to the original dimensions. This padding adds substantial overhead, so GVC1D's actual bitrate is higher than the value shown in the figure. We include these results for reference.

  1. Thank you for your suggestion regarding the 720p results. On reviewing our evaluation, we found an issue in the preprocessing script: instead of resizing the video before cropping, as described in the paper, the script applied a center crop directly. This inadvertently removed background content, which is important for generative methods, and made the methods' performance appear more similar. We corrected the preprocessing and reran the evaluation; the results changed modestly. We also added results for additional methods. Because of their long inference times, we were unable to run YoDA and GVC1D at 720p. The GVC-RT results suggest that at lower resolutions, less texture detail needs to be synthesized, which may narrow the performance gap between generative models. This should not be interpreted as evidence that our method performs worse.

  2. We understand your concerns about the directional module. The initial ablation results also gave us pause, but the module' s role is not fully captured by objective metrics alone. In our framework, RGI is a lightweight global fine-tuning head. Its direction head has only about 200K parameters. After the DiT establishes the overall latent restoration direction, this component makes position-specific adjustments. It consists entirely of 2D convolution layers so that it can learn small, local changes in direction. In our results, the head helps calibrate generated texture details and improve reconstruction fidelity. It yields a modest PSNR gain while maintaining similar LPIPS and DISTS scores. Given the module's small size, its measurable objective benefit, and its potential to improve visual fidelity, we decided to retain it and report its results transparently. We hope this explanation addresses your concern.

[W2] Thank you for raising the runtime question. We have added latency comparisons between our method and other generative methods, measured on an H800, in the revised appendix. We hope these results address your concern.

Reviewer_bXRBOpenReview ↗

Response to Authors

Thanks for your reply. It partially addresses my concern about the performance, but I still consider the experimental results are not strong enough to support the motivation emphasized in the earlier sections of the paper—the mismatch between the direction of the model update and that of the latent reconstruction residual. Therefore, I will keep my score unchanged.

AuthorsOpenReview ↗

Answer to Reviewer bXRB

Thank you for your response. We realize that our presentation may not have conveyed the central motivation of our work clearly enough. We would therefore like to explain the reasoning behind our approach more explicitly.

Beginning with GNVC-VD, a common formulation treats compressed latents obtained at different QP settings as samples with different information content within a shared distribution. This combined distribution is then associated with a single timestep along the DiT flow trajectory. For example, GNVC-VD assumes that all compressed latents correspond to t=0.7t=0.7 and uses noise injection to simulate the corresponding intermediate distribution. This formulation also appears to be implicitly adopted in many subsequent works.

In our work, we examine the residual distributions and actual flow directions across different QP settings in greater detail. We find that the residuals behave as directional Gaussian perturbations, with clear angular differences across QPs. Adding isotropic Gaussian noise does not resolve this issue, because standard Gaussian noise does not remove the underlying directional bias of the residuals. We believe that reconciling these differences may impose an additional burden on the DiT during training and lead to a suboptimal solution.

To address this issue, we interpret residual restoration as different stages along the DiT’s trajectory toward the clean latent. We assign residuals to different timesteps through a mapping based on their magnitudes, and introduce explicit constraints and architectural designs to distinguish different QP conditions more clearly within the DiT. This provides more structured guidance than injecting a single QP embedding token and relying on the model to learn these distinctions implicitly.

Building on this design, we apply lightweight external modulation to the direction and magnitude of the residual features predicted by the DiT. This small refinement model uses regional estimates of residual magnitude and finer directional adjustments to adapt to different spatial regions within the residual at each QP. More precisely, it performs a second stage of detail refinement under the overall direction and magnitude established by the DiT.

Our contribution therefore extends beyond the external RGI module. RGI provides additional refinement tailored to different residual conditions, while the main capacity to calibrate direction and residual magnitude is already incorporated into the DiT through cross-attention and related conditioning mechanisms.

Experimentally, our method achieves state-of-the-art or near-state-of-the-art performance in subjective quality, objective metrics, and temporal consistency. We provide additional results covering nearly one-third of the test set to demonstrate that analyzing and handling different residual distributions separately improves generative video reconstruction. We believe that these results provide strong evidence that our design helps mitigate the risk of the DiT reaching suboptimal solutions because of conflicting residual magnitudes and directions across QP settings.

And please also consider the qualitative video comparisons in the appendix. One potential advantage of our method is that it avoids some of the regular texture patterns that can cover entire reconstructed frames. In our interpretation, such patterns may arise when differences in residual direction are not fully learned, causing the DiT to become trapped in local optima during reconstruction. These patterns can readily improve LPIPS and DISTS without contributing meaningful reconstruction detail. Their reduction therefore provides further evidence that our design improves local optimization and generalization during DiT training.

Reviewer_vKVLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

introduces a Residual Geometry Inducer (RGI) that converts the pretrained flow prediction into a content-adaptive restoration update using decoder-visible codec cues.

RGI integrates flow-timestep conditioning with a learned full-band residual-strength estimate and content-conditioned direction and gain modulation, aligning the orientation and magnitude of the update with the compression-dependent residual geometry.

Experiments show promising results of RIFT in comparison with competing generative video compression methods.

Critical strengths
  1. The proposed method is promising and has some novelty in the area of generative video compression;
  2. The experiments show good perceptual quality in terms of LPIPS, FID and DISTS in comparison with previous video compression methods.
Critical weaknesses
  1. Figure 2, which is essential for understanding the whole proposed framework, is incremental and has some misleading information: In "Restoration Modules", the relationship between the modules are (Multi-scale Error Estimator -> Strength Fusion) & (Direction Adaptation) -> Base Gain Modulation, however, "Multi-scale Error Estimator -> Strength Fusion" does not appear in the above " Generative Restoration" figure, which should actually be the inputs to the Base Gain Modulation; Similarly, I think the Full-band RMS Estimator should be one of the inputs to Output Modulation as well. The current figure is not clear and misleading, making readers difficult to understand the overall framework in an easy way.

  2. The most essential concern is about the metrics used in experiments. LPIPS, DISTS and FID are all perceptual metrics for images, which do not take temporal consistency into consideration. Therefore, they are not enough for the evaluation of video perceptual quality. The temporal perceptual metrics are necessary for generative video compression methods.

  3. Besides, the traditional standard video codecs are missed from comparison, such as HEVC, VVC, etc.

  4. Moreover, the latest learned video compression methods, e.g., DCVC-UF series (https://github.com/microsoft/DCVC), are not compared with.

Questions for authors

Please see above Weaknesses.

6 comments
AuthorsOpenReview ↗

Answer to Reviewer vKVL

Thank you for the specific suggestions on the framework, temporal assessment, and baseline coverage. For detailed supplementary experimental results, please refer to the newly added appendix section of the paper. For the additional subjective comparison videos, please check the supplementary materials.

[W1] Thank you for pointing this out. This was an error in the figure, and I have corrected it in the revised manuscript.

[W2] Thank you for raising this concern. I have added further evaluations in the appendix of the revised manuscript. Following another reviewer's suggestion, I included recent generative video coding methods and evaluated them under a consistent temporal testing protocol. The results show that our method achieves the best or near-best temporal performance, maintaining strong consistency across frames. Comparison videos are included in the supplementary materials. I sincerely hope these results address your concern.

[W3 & W4] Thank you for the suggestion. I have added 1080p performance comparisons for the relevant methods. At the low bitrates considered in this work, current generative methods offer a clear advantage in perceptual quality over conventional codecs. We have therefore focused our qualitative comparisons on recent generative methods such as GVC1D, GVC-RT, and YoDA. If you would also find direct visual comparisons with conventional codecs and DCVC-UF helpful, we would be happy to add these examples to the supplementary material.I hope the additional experiments address your concerns.

We have added qualitative comparison figures with VVC and DCVC-UF to the supplementary attachment. We hope these additional results address your remaining concerns.

Thanks to the authors for the rebuttal and revisions.

However, the same as the concern raised by the reviewer Q4ZL, the authors have changed the training loss in the revised manuscript. It is not convincing to me that it was an "error" in the original manuscript.

Besides, in Figure 13 of the revised manuscript, the proposed method performs not well in terms of FloLPIPS.

Therefore, I'd like to keep my original rating.

Reviewer_Q4ZLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses a relevant problem in generative video compression and proposes an interesting approach. The method is technically sound and achieves some improvements in perceptual reconstruction quality. However, I have concerns about the experimental evaluation, particularly the lack of a thorough analysis of temporal consistency between frames, which is fundamental for video compression. The method leads to improvements at frame-level, using perceptual metrics. But this does not necessarily guarantee temporally coherent reconstructions, and generative methods may introduce flickering or inconsistent details across frames. Furthermore, the ablation studies provide limited evidence for the actual contribution of the proposed directional refinement.

Critical strengths
  1. Interesting motivation and technical contribution. The paper identifies a meaningful mismatch between codec-induced distortions and the trajectories learned by pretrained flow models. The analysis of compression-dependent residual geometry provides an interesting motivation for adapting generative priors beyond simple timestep conditioning.

  2. Promising perceptual reconstruction results. RIFT achieves improvements over existing generative video compression methods across multiple benchmarks, particularly in terms of LPIPS and DISTS. The use of actual entropy-coded bitstreams strengthens the practical relevance of the evaluation, while the ablation studies demonstrate the effectiveness of several proposed components.

Critical weaknesses
  1. Lack of temporal consistency evaluation. The main weakness is the absence of a dedicated evaluation of temporal consistency, which is essential for video compression. While the method achieves improvements in frame-level perceptual metrics, these do not guarantee temporally coherent reconstructions. Generative restoration may introduce flickering or inconsistent details across consecutive frames. Without quantitative temporal consistency metrics or an analysis of temporal artifacts, it is difficult to assess the actual quality of the reconstructed videos. This significantly weakens the experimental validation.

  2. Limited evidence supporting the contribution of directional refinement. A central motivation of the paper is the need to explicitly model the direction of codec-induced residuals. However, the ablation study shows that removing directional refinement leads to relatively small performance degradation (+0.96% BD-rate in LPIPS and +3.76% in DISTS), compared to the much larger impact of removing learned gain. This raises questions about whether the improvements primarily come from magnitude adaptation rather than the proposed directional modeling.

Questions for authors
  1. Can the authors provide quantitative evidence of temporal consistency, such as temporal warping errors or flickering metrics, ideally complemented by video comparisons at matched bitrates? In particular, how does RIFT preserve temporal coherence across consecutive frames and GOP boundaries? This is my main concern, and such evidence would substantially affect my assessment of the paper.

  2. The ablation study shows relatively limited degradation when directional refinement is removed (+0.96% in LPIPS and +3.76% in DISTS). Can the authors further clarify the actual contribution of this component? In particular, how does RIFT compare against a simpler baseline relying only on timestep conditioning and learned gain, under comparable model capacity and training conditions? This would help establish whether explicit directional modeling is essential to the reported improvements.

6 comments
AuthorsOpenReview ↗

Answer to Reviewer Q4ZL

Thank you for focusing on temporal consistency and the role of directional refinement. We have added controlled directional experiments and a complete temporal evaluation. For detailed supplementary experimental results, please refer to the newly added appendix section of the paper. For the additional subjective comparison videos, please check the supplementary materials.

[W1] We have added temporal consistency evaluations, including Ewarp and FloLPIPS. In response to another reviewer's request, we also evaluated additional recent codecs that claim state-of-the-art performance. The results are provided in the appendix and supplementary video. These evaluations show that our method achieves state-of-the-art or near-state-of-the-art objective temporal performance.

I also realized that the initial manuscript omitted the temporal losses used in training, which may have caused confusion. In practice, we use temporal losses as constraints in both the codec and generation stages. Our method models the temporal behavior of residuals separately for each QP. This helps the DiT learn temporally consistent textures at each QP, while avoiding interference between texture signals of different strengths across QPs. Across GOPs, we use the standard approach of injecting the previous latent; since this is not novel, we did not present it as a contribution. We hope these results clarify the method's temporal performance.

[W2] We understand your concerns about this module. The initial ablation results also gave us pause, but the module's role is not fully captured by the objective metrics alone. In our framework, RGI is a lightweight global fine-tuning head. Its direction head is a small component of about 200K parameters. Once the DiT has established the overall latent restoration direction, this head makes position-specific adjustments. This is why it consists entirely of 2D convolution layers: they are designed to learn small, local changes in direction. In our results, the head helps calibrate generated texture details and improve reconstruction fidelity. It yields a modest PSNR gain while maintaining similar LPIPS and DISTS scores. Given its small size, its measurable objective benefit, and its potential to improve visual fidelity, we decided to retain the module and report its results transparently. We hope this addresses your concern.

Reviewer bXRB raised further questions about our direction head in a follow-up response. Since your question concerns the same issue, we provide this clarification to address any potential concerns. Our contribution lies in how we condition and optimize the DiT to handle residuals with different magnitudes and directions across QPs. We assign residuals to timesteps according to their magnitudes and use token conditioning and cross-attention to guide direction and magnitude calibration within the DiT. RGI then provides lightweight spatial refinement, while fusion with the original latent supports temporal consistency. The direction head therefore forms part of this broader solution, whose effectiveness is supported by our high-quality, temporally stable single-step reconstruction and state-of-the-art or near-state-of-the-art results. We hope this explanation addresses any concerns you may have. For further details, please refer to our response to Reviewer bXRB. We look forward to discussing these points with you.

The new temporal consistency evaluation substantially addresses my main concern, and I appreciate the effort to provide comparisons using Ewarp and FloLPIPS. However, I noticed an important discrepancy between the original and revised training objectives (Section 4.2). In addition to introducing the temporal loss, the revised manuscript changes the coefficient of the DISTS loss from 0.1 to 1 and that of the latent loss from 1 to 0.1. Could the authors clarify whether these were reporting errors in the original manuscript or whether the training objectives were modified during the rebuttal? In particular, were the original and newly reported results obtained using the same trained models and loss configuration?

Additionally, could the authors clarify how temporal information is propagated across GOP boundaries? The original manuscript explicitly stated that GOPs were processed independently, while the rebuttal mentions the injection of the previous latent.

The new temporal results positively affect my assessment, but clarifying these methodological details would be important before reconsidering my final recommendation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Revision Comment

We thank all reviewers again for their constructive suggestions. In response, we have revised the manuscript, corrected the figure issues, clarified the loss functions and experimental details omitted from the initial submission, and added subjective comparison videos to the supplementary materials. The main revisions are as follows:

  1. Clarified the temporal losses used during training, which were omitted from the initial manuscript.
  2. Corrected the inconsistency between the 720p evaluation procedure and its description in the manuscript.
  3. Added comparisons with relevant methods at 1080p.
  4. Added temporal performance evaluations and comparative analyses.
  5. Expanded the explanation of the role of the directional head.
  6. Added an inference-time evaluation.
  7. Included subjective comparison videos in the supplementary materials.

We hope these additions address the reviewers' and AC's questions about our method. Finally, I am sincerely grateful to all three reviewers, whose expertise is directly relevant to this field, for their thoughtful guidance. Your feedback has been invaluable in improving this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.