- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a promising look-ahead RDO framework for joint fidelity and perceptual optimization for DCVC-UF, achieving notable gains on HEVC Class C/D sequences. However, limited evaluation, insufficient perceptual validation, unclear computational overhead, and a lack of comparisons with existing in-domain RDO methods weaken the evidence for its effectiveness and practicality.
Critical strengths
- Significant performance improvements on HEVC Class C and D test sequences, particularly at high bitrates.
- The design approach incorporating look-ahead RDO (Rate-Distortion Optimization) is sound.
Critical weaknesses
- These methods were evaluated only on low-resolution HEVC Class C and D datasets (96 frames). There is a lack of further evaluation regarding: (1) full-sequence evaluation (similar to DCVC-RT and DCVC-UF); and (2) high-resolution benchmarks such as Class A, Class B, UVG, and MCL-JCV. Consequently, the results lack sufficient persuasiveness.
- Given that LPIPS and DISTS have been incorporated into the RDO process, using them solely as perceptual metrics is not sufficiently objective. Please (1) include additional perceptual metrics such as LPIPS-VGG, FID, KID, and VMAF; and (2) conduct a user study (if feasible) or at least provide a visual comparison between "PSNR-only optimization" and "joint PSNR-perceptual optimization" to demonstrate the effectiveness of perceptual training.
- In Table 2, the "search" method introduces additional overhead compared to "Adam" due to the requirement for multiple forward passes. It remains unclear whether "search" still outperforms "Adam" under an equivalent computational budget.
- The complexity of the optimization process is not presented. Information regarding computational complexity and optimization time should be included.
- This paper compares the proposed method only with DCVC-DT[1], an adaptive method designed for out-of-domain adaptation rather than in-domain RDO. Therefore, it is necessary to conduct comparisons with additional in-domain RDO methods.
[1] Neural Video Compression with Domain Transfer, Zhang et al., ISCAS 2026
Questions for authors
- In Figure 1, the title of the RD curve should not include the phrase "at QP63".
- In Figure 2, the "Online Modulation" indicator points to the latent feature after the "AE" (Autoencoder). It is suggested to point it to the latent feature before the "AE," as the data transforms into a bitstream after entropy encoding and is no longer a latent feature.
- The "Lookahead ROI" mechanism introduces additional latency in scenarios such as real-time streaming—a factor that cannot be overlooked. As noted in UI2C [2], lookahead processing of subsequent frames typically yields additional benefits.
[2] Real-Time Neural Video Compression with Unified Intra and Inter Coding, Xiang et al., CVPR2026
3 comments
Response to Weakness 3: Comparison Between Residual Direction Search and Adam Under Matched Budgets
Thank you for this careful observation. Evidence from our ablation study in Table 2 and Figure 5 directly supports the superior compute-efficiency of our residual direction search over standard Adam:
- Matched Iteration Count: In Table 2,
search (50)achieves PSNR BD-Rate, more than doubling the PSNR BD-Rate ofAdam (50)at the same iteration count. - Matched Wall-Clock Runtime Budget: Measured on a single NVIDIA A800 GPU on HEVC Class D,
search (500)takes 4.74 hours and achieves PSNR, LPIPS, and DISTS. Meanwhile,Adam (1000)takes 1.90 hours (so doubling Adam again to ~2,500 steps to match the 4.74-hour budget ofsearch (500)yields diminishing returns as shown by the convergence curve in Figure 5(a), where Adam plateaus well above the residual search trajectory). Notably, even lightweightsearch (50)(25.6 minutes) outperformsAdam (50)(6.6 minutes) by over in PSNR BD-Rate, andsearch (500)outperformsAdam (1000)in LPIPS ( vs. ) while approaching its DISTS ( vs. ) at half the gradient backpropagation steps. - Mechanism Behind Efficiency: Candidate evaluations in the residual direction search require forward passes only (no backward gradient computation) and directly evaluate the exact hard-rounded objective, preventing Adam from stalling on flat integer quantization intervals (as confirmed by the bimodal step-size selection in Figure 5(b)). We will explicitly include these runtime figures and matched-budget analyses in Table 2.
Response to Weakness 4: Computational Complexity and Necessity of Test-Time Optimization
We appreciate your suggestion to clearly report computational complexity and runtime:
- Necessity of Test-Time Optimization: Implicit-motion codecs such as DCVC-UF sacrifice explicit motion estimation and compensation to achieve ultra-fast decoding. While this design greatly improves throughput, it creates an inherent high-bitrate bottleneck where coupled temporal errors cannot be resolved by a single amortized forward pass. Encoder-side test-time latent refinement eliminates this amortization gap without modifying network weights or bitstream syntax, preserving full compatibility with the frozen decoder. This paradigm aligns directly with video-on-demand, streaming distribution, and media archiving workflows, where a video is encoded once offline and decoded many times.
- Exact Latency Overhead:
- Decoder Latency Overhead: Zero (). Because only the transmitted latent is refined and the decoder parameters remain unchanged, our method preserves the exact decoding throughput of the DCVC-UF baseline.
- Encoder Latency Overhead: Measured on a single NVIDIA A800 GPU under the ablation setting of Table 2 on HEVC Class D, the encoding runtime of our residual direction search is 4.74 hours for 500 steps and 25.6 minutes for 50 steps. Importantly, as shown in Table 2, even a lightweight 50-step configuration already reduces DISTS BD-Rate by and PSNR BD-Rate by while reducing encoding time by over . We will add these exact runtime figures and speed-quality trade-offs to Table 2 and Section 4.3 in the revised manuscript.
Response to Weakness 5: Comparison with In-Domain Enhancement and RDO Methods
Thank you for pointing out the distinction between out-of-domain adaptation and in-domain enhancement. Existing in-domain enhancement methods for neural video codecs primarily rely on learned filtering modules (such as in-loop contextual filtering and out-of-loop reconstruction enhancement), rather than encoder-side test-time latent optimization.
We actually explored adapting such in-loop and out-of-loop convolutional filtering modules onto the frozen DCVC-UF backbone, and observed two fundamental limitations:
- Limited Coding Gain: Lightweight pre-/post-processing and convolutional filtering modules yield only ~5% BD-Rate improvement on DCVC-UF, as post-filtering cannot recover high-frequency details already discarded by coarse latent quantization at the encoder.
- Architectural and Bitstream Incompatibility: In-loop filtering modifies the decoder architecture and requires retraining network weights, breaking compatibility with the frozen DCVC-UF decoder—whereas our method achieves (PSNR) / (DISTS) BD-Rate savings with zero decoder modification.
By contrast, DCVC-DT* represents the closest encoder-side, decoder-preserving latent RDO baseline (which we adapted directly to the in-domain DCVC-UF anchor under identical step budgets). We will clarify this distinction and discuss in-domain filtering approaches in Section 2.3 and Section 4.2 of the final manuscript.
Responses to Questions for Authors
- Question 1 (Figure 1 Title): Thank you for catching this! We will remove
"at QP63"from the title of Figure 1 in the final manuscript so that it accurately reflects the full RD curve across all QPs. - Question 2 (Figure 2 Arrow Placement): Thank you for the helpful suggestion. We will update Figure 2 in the final version so that the
"Online Modulation"indicator points directly to the latent feature before quantization and Arithmetic Encoding (AE), accurately matching our formulation. - Question 3 (Lookahead Latency vs. Offline Application Scope): We fully agree with your observation and will cite UI2C (Xiang et al., CVPR 2026) in our revised discussion. Because our framework is designed specifically for offline encoding workflows (e.g., Video-on-Demand, streaming distribution, and media archiving, where content is encoded once offline and decoded many times, as discussed in Weakness 4), the 1-chunk (8-frame) lookahead buffering delay does not impact real-time user playback, while preserving zero decoder overhead is paramount.
Thank you once again for your constructive review, which has been genuinely helpful in strengthening the rigor and clarity of our work. We hope our responses and the newly added 1080p results, runtime analyses, and technical clarifications have addressed your concerns regarding evaluation scope, perceptual optimization, and computational efficiency. We would be very grateful if you could revisit our rebuttal and updated paper, and reconsider your assessment in light of these revisions. Should any questions remain, we would be happy to discuss them further.
Best regards,
Authors of Submission 1637
We sincerely thank Reviewer xn4z for the constructive, thorough, and insightful review, which has been genuinely helpful in strengthening the rigor and clarity of our work. Below, we address Weakness 1 and Weakness 2 first, followed by our responses to Weakness 3–5 and your questions in separate comments.
Response to Weakness 1: Evaluation on High-Resolution Benchmarks and Full Sequences
Thank you for highlighting the importance of broader benchmark validation. Because our method performs instance-specific latent optimization at the encoder side, running full-sequence evaluations across all high-resolution benchmarks requires substantial computation time within the short rebuttal period. Nevertheless, to verify that our framework scales effectively to high-resolution content, we conducted an evaluation on the representative 1080p sequence BQTerrace (1920×1080, 96 frames) from HEVC Class B under the exact same protocol as our main experiments:
These results confirm that our lookahead ROI and joint modulation generalize effectively from lower resolutions to 1080p high-definition sequences with complex water textures and camera motion. We will include the complete 1080p benchmark results (full HEVC Class B, UVG, and MCL-JCV) as well as full-sequence evaluations in the final version of the paper.
Response to Weakness 2: Objective Perceptual Validation and Purpose of Instance-Specific Overfitting
We respectfully clarify a fundamental distinction between model-level generalization and encoder-side latent optimization:
DCVC-DT*in Table 1) improves PSNR by −8.73% on Class C but degrades DISTS by +12.27% due to over-smoothing.