- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the mismatch between the perturbation trajectories learned by pretrained flow-matching models in generative video codecs and the degradation patterns introduced by compression. To tackle this problem, the authors propose a Residual-Informed Flow Transfer (RIFT) framework. Based on a Residual Geometry Inducer (RGI), the framework estimates residual strength and modulates the restoration direction and local gain, aligning the direction and magnitude of flow-model updates with the geometry of compression-induced residuals. This ultimately improves perceptually optimized rate–distortion performance.
Critical strengths
- Reasonable motivation supported by theoretical and experimental analysis. The paper demonstrates that compressed latents remain highly aligned across different QPs, while their reconstruction residuals differ substantially in both magnitude and direction.
- Clearly motivated method design. The proposed modules correspond well to the phenomena identified in the motivation and analysis.
Critical weaknesses
- The performance results do not yet convincingly validate the motivation and the proposed method.
- The evaluation omits several recent and competitive baselines, including Generative Video Compression with One-Dimensional Latent Representation, YODA: Yet Another One-step Diffusion-based Video Compressor, and GVC-RT: Towards Real-Time Generative Video Compression at Ultra-Low Bitrates.
- The results in the appendix suggest that the proposed method performs slightly worse at 720p.
- The motivation and analysis emphasize the mismatch between the direction of the model update and that of the latent reconstruction residual. However, in Table 1, ablating this component results in only small changes in BD-Rate (LPIPS) and BD-Rate (DISTS), at 0.96% and 3.76%, respectively. In contrast, the main performance gains appear to come from the learned gain module. This somewhat weakens the paper's claimed contribution.
- Key analyses of computational complexity and encoding/decoding latency are missing.
Questions for authors
My question mainly concerns clearer explanations of performance and complexity, as well as verifying the motivation and methods of this paper. If the author completes these reasonable explanations, I will consider improving my score.
5 comments
Response to Authors
Thanks for your reply. It partially addresses my concern about the performance, but I still consider the experimental results are not strong enough to support the motivation emphasized in the earlier sections of the paper—the mismatch between the direction of the model update and that of the latent reconstruction residual. Therefore, I will keep my score unchanged.
Answer to Reviewer bXRB
Thank you for your response. We realize that our presentation may not have conveyed the central motivation of our work clearly enough. We would therefore like to explain the reasoning behind our approach more explicitly.
Beginning with GNVC-VD, a common formulation treats compressed latents obtained at different QP settings as samples with different information content within a shared distribution. This combined distribution is then associated with a single timestep along the DiT flow trajectory. For example, GNVC-VD assumes that all compressed latents correspond to and uses noise injection to simulate the corresponding intermediate distribution. This formulation also appears to be implicitly adopted in many subsequent works.
In our work, we examine the residual distributions and actual flow directions across different QP settings in greater detail. We find that the residuals behave as directional Gaussian perturbations, with clear angular differences across QPs. Adding isotropic Gaussian noise does not resolve this issue, because standard Gaussian noise does not remove the underlying directional bias of the residuals. We believe that reconciling these differences may impose an additional burden on the DiT during training and lead to a suboptimal solution.
To address this issue, we interpret residual restoration as different stages along the DiT’s trajectory toward the clean latent. We assign residuals to different timesteps through a mapping based on their magnitudes, and introduce explicit constraints and architectural designs to distinguish different QP conditions more clearly within the DiT. This provides more structured guidance than injecting a single QP embedding token and relying on the model to learn these distinctions implicitly.
Building on this design, we apply lightweight external modulation to the direction and magnitude of the residual features predicted by the DiT. This small refinement model uses regional estimates of residual magnitude and finer directional adjustments to adapt to different spatial regions within the residual at each QP. More precisely, it performs a second stage of detail refinement under the overall direction and magnitude established by the DiT.
Our contribution therefore extends beyond the external RGI module. RGI provides additional refinement tailored to different residual conditions, while the main capacity to calibrate direction and residual magnitude is already incorporated into the DiT through cross-attention and related conditioning mechanisms.
Experimentally, our method achieves state-of-the-art or near-state-of-the-art performance in subjective quality, objective metrics, and temporal consistency. We provide additional results covering nearly one-third of the test set to demonstrate that analyzing and handling different residual distributions separately improves generative video reconstruction. We believe that these results provide strong evidence that our design helps mitigate the risk of the DiT reaching suboptimal solutions because of conflicting residual magnitudes and directions across QP settings.
And please also consider the qualitative video comparisons in the appendix. One potential advantage of our method is that it avoids some of the regular texture patterns that can cover entire reconstructed frames. In our interpretation, such patterns may arise when differences in residual direction are not fully learned, causing the DiT to become trapped in local optima during reconstruction. These patterns can readily improve LPIPS and DISTS without contributing meaningful reconstruction detail. Their reduction therefore provides further evidence that our design improves local optimization and generalization during DiT training.
Answer to Reviewer bXRB
Thank you for identifying baseline coverage, the directional mechanism, and runtime as key questions. We have expanded the spatial-performance comparisons to include YoDA, GVC-RT, GVC1D, VVC, and DCVC-UF, and added detailed temporal and runtime results.
[W1] 1. Thank you for suggesting these additional baselines. When we began this work, the methods you mentioned had not yet released their code, so they were not included in the initial version. We have added their reported performance to the revised manuscript. During our reproduction, we found that YoDA's measured bitrate differs from the entropy-estimated bitrate reported for its Fast setting. We use the actual bitrate in our evaluation; the other results are consistent with those reported in the original paper. The official GVC1D evaluation resizes videos to 1792 × 1024, which does not allow for a fair comparison at the original resolution. When we used the original resolution as input, GVC1D's performance declined substantially. We therefore reimplemented its evaluation pipeline by padding the bottom and right edges to multiples of 256, then cropping the output back to the original dimensions. This padding adds substantial overhead, so GVC1D's actual bitrate is higher than the value shown in the figure. We include these results for reference.
Thank you for your suggestion regarding the 720p results. On reviewing our evaluation, we found an issue in the preprocessing script: instead of resizing the video before cropping, as described in the paper, the script applied a center crop directly. This inadvertently removed background content, which is important for generative methods, and made the methods' performance appear more similar. We corrected the preprocessing and reran the evaluation; the results changed modestly. We also added results for additional methods. Because of their long inference times, we were unable to run YoDA and GVC1D at 720p. The GVC-RT results suggest that at lower resolutions, less texture detail needs to be synthesized, which may narrow the performance gap between generative models. This should not be interpreted as evidence that our method performs worse.
We understand your concerns about the directional module. The initial ablation results also gave us pause, but the module' s role is not fully captured by objective metrics alone. In our framework, RGI is a lightweight global fine-tuning head. Its direction head has only about 200K parameters. After the DiT establishes the overall latent restoration direction, this component makes position-specific adjustments. It consists entirely of 2D convolution layers so that it can learn small, local changes in direction. In our results, the head helps calibrate generated texture details and improve reconstruction fidelity. It yields a modest PSNR gain while maintaining similar LPIPS and DISTS scores. Given the module's small size, its measurable objective benefit, and its potential to improve visual fidelity, we decided to retain it and report its results transparently. We hope this explanation addresses your concern.
[W2] Thank you for raising the runtime question. We have added latency comparisons between our method and other generative methods, measured on an H800, in the revised appendix. We hope these results address your concern.