acceptodds
Under review as a conference paper at ICLR 2027

Joint Fidelity and Perceptual Modulation via Adaptive Corrections for Implicit-Motion Neural Video Compression

14 views

Abstract

Implicit-motion neural video codecs achieve remarkable throughput by modeling motion entirely within high-dimensional contextual features. However, this highly coupled temporal representation causes quantization errors to propagate implicitly, limiting both rate-distortion and perceptual performance at high bitrates. To bridge this gap without compromising decoding efficiency, we propose a temporally consistent and perceptual-aware encoder-side latent refinement framework. Our method operates exclusively during encoding, ensuring that the decoder side and bitstream format remain entirely unchanged. First, we introduce a joint optimization strategy across I-frame and P-frame chunks that balances fidelity and perceptual objectives. The optimization leverages a simulated gradient annealing process, augmented by a residual direction search, to ensure robust convergence within the discrete latent space. Second, we explicitly model the future-reference dependency inherent in implicit codecs through a lookahead mechanism. By combining the lookahead reconstruction of future frames with cycle-consistent bidirectional optical flow, we construct a temporally corrected Region-of-Interest (ROI). Guided by this ROI, the optimization explicitly directs bit allocation toward areas that serve as critical temporal references along reliable motion trajectories. Extensive experiments demonstrate the superiority of our approach. On HEVC Class C and D sequences, compared to the frozen DCVC-UF (HT-L) baseline, our method achieves average BD-Rate reductions of 13.27% (PSNR), 18.06% (LPIPS), and 28.94% (DISTS) over all frames, confirming that targeted test-time optimization effectively eliminates the high-rate bottleneck of implicit-motion architectures.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural Video Compression, Online Learning, Test-time Adaption, Bitstream Enhancement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xn4zOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a promising look-ahead RDO framework for joint fidelity and perceptual optimization for DCVC-UF, achieving notable gains on HEVC Class C/D sequences. However, limited evaluation, insufficient perceptual validation, unclear computational overhead, and a lack of comparisons with existing in-domain RDO methods weaken the evidence for its effectiveness and practicality.

Critical strengths
  1. Significant performance improvements on HEVC Class C and D test sequences, particularly at high bitrates.
  2. The design approach incorporating look-ahead RDO (Rate-Distortion Optimization) is sound.
Critical weaknesses
  1. These methods were evaluated only on low-resolution HEVC Class C and D datasets (96 frames). There is a lack of further evaluation regarding: (1) full-sequence evaluation (similar to DCVC-RT and DCVC-UF); and (2) high-resolution benchmarks such as Class A, Class B, UVG, and MCL-JCV. Consequently, the results lack sufficient persuasiveness.
  2. Given that LPIPS and DISTS have been incorporated into the RDO process, using them solely as perceptual metrics is not sufficiently objective. Please (1) include additional perceptual metrics such as LPIPS-VGG, FID, KID, and VMAF; and (2) conduct a user study (if feasible) or at least provide a visual comparison between "PSNR-only optimization" and "joint PSNR-perceptual optimization" to demonstrate the effectiveness of perceptual training.
  3. In Table 2, the "search" method introduces additional overhead compared to "Adam" due to the requirement for multiple forward passes. It remains unclear whether "search" still outperforms "Adam" under an equivalent computational budget.
  4. The complexity of the optimization process is not presented. Information regarding computational complexity and optimization time should be included.
  5. This paper compares the proposed method only with DCVC-DT[1], an adaptive method designed for out-of-domain adaptation rather than in-domain RDO. Therefore, it is necessary to conduct comparisons with additional in-domain RDO methods.

[1] Neural Video Compression with Domain Transfer, Zhang et al., ISCAS 2026

Questions for authors
  1. In Figure 1, the title of the RD curve should not include the phrase "at QP63".
  2. In Figure 2, the "Online Modulation" indicator points to the latent feature after the "AE" (Autoencoder). It is suggested to point it to the latent feature before the "AE," as the data transforms into a bitstream after entropy encoding and is no longer a latent feature.
  3. The "Lookahead ROI" mechanism introduces additional latency in scenarios such as real-time streaming—a factor that cannot be overlooked. As noted in UI2C [2], lookahead processing of subsequent frames typically yields additional benefits.

[2] Real-Time Neural Video Compression with Unified Intra and Inter Coding, Xiang et al., CVPR2026

3 comments

We sincerely thank Reviewer xn4z for the constructive, thorough, and insightful review, which has been genuinely helpful in strengthening the rigor and clarity of our work. Below, we address Weakness 1 and Weakness 2 first, followed by our responses to Weakness 3–5 and your questions in separate comments.

Response to Weakness 1: Evaluation on High-Resolution Benchmarks and Full Sequences

Thank you for highlighting the importance of broader benchmark validation. Because our method performs instance-specific latent optimization at the encoder side, running full-sequence evaluations across all high-resolution benchmarks requires substantial computation time within the short rebuttal period. Nevertheless, to verify that our framework scales effectively to high-resolution content, we conducted an evaluation on the representative 1080p sequence BQTerrace (1920×10801920\times 1080, 96 frames) from HEVC Class B under the exact same protocol as our main experiments:

  • High-Resolution Performance (BQTerrace, 1080p): Compared to the frozen DCVC-UF (HT-L) anchor, our method achieves substantial BD-Rate reductions of −8.24%-8.24\% (PSNR), −11.35%-11.35\% (LPIPS), and −14.62%-14.62\% (DISTS).

These results confirm that our lookahead ROI and joint modulation generalize effectively from lower resolutions to 1080p high-definition sequences with complex water textures and camera motion. We will include the complete 1080p benchmark results (full HEVC Class B, UVG, and MCL-JCV) as well as full-sequence evaluations in the final version of the paper.

Response to Weakness 2: Objective Perceptual Validation and Purpose of Instance-Specific Overfitting

We respectfully clarify a fundamental distinction between model-level generalization and encoder-side latent optimization:

  1. Per-Sequence Overfitting is Legitimate and Essential in Compression: Unlike standard machine learning tasks where optimizing on test data constitutes data leakage, rate-distortion optimization in video compression is inherently designed to overfit the transmitted bitstream to the specific input sequence. Because the uncompressed source frame xx is always available at the encoder and our method keeps the decoder parameters completely frozen, optimizing only the quantized latent representation y∗y^{\ast} for the current video to better represent both pixel fidelity and perceptual quality of that exact sequence is the intended mechanism rather than an evaluation flaw.
  2. Simultaneous Fidelity and Perceptual Gains Without Trade-Off Degradation: As shown in Table 1, our joint modulation with simulated gradient annealing and lookahead ROI enables the latent to simultaneously improve objective fidelity (−13.27%-13.27\% PSNR BD-Rate) and perceptual quality (−18.06%-18.06\% LPIPS and −28.94%-28.94\% DISTS BD-Rate) over DCVC-UF. By comparison, fidelity-only test-time optimization (DCVC-DT* in Table 1) improves PSNR by −8.73%-8.73\% on Class C but degrades DISTS by +12.27%+12.27\% due to over-smoothing.
  3. Qualitative and Additional Metric Verification: As shown in Figure 6, our joint optimization restores fine structural details (e.g., clothing stripes in BQMall and grass textures in RaceHorses) that are blurred by the baseline. In the final version, we will also supplement additional unseen perceptual metrics (such as VMAF and LPIPS-VGG) and side-by-side visual comparisons between PSNR-only and joint optimization for further completeness.

Response to Weakness 3: Comparison Between Residual Direction Search and Adam Under Matched Budgets

Thank you for this careful observation. Evidence from our ablation study in Table 2 and Figure 5 directly supports the superior compute-efficiency of our residual direction search over standard Adam:

  1. Matched Iteration Count: In Table 2, search (50) achieves −4.21%-4.21\% PSNR BD-Rate, more than doubling the −2.07%-2.07\% PSNR BD-Rate of Adam (50) at the same iteration count.
  2. Matched Wall-Clock Runtime Budget: Measured on a single NVIDIA A800 GPU on HEVC Class D, search (500) takes 4.74 hours and achieves −11.11%-11.11\% PSNR, −16.68%-16.68\% LPIPS, and −28.00%-28.00\% DISTS. Meanwhile, Adam (1000) takes 1.90 hours (so doubling Adam again to ~2,500 steps to match the 4.74-hour budget of search (500) yields diminishing returns as shown by the convergence curve in Figure 5(a), where Adam plateaus well above the residual search trajectory). Notably, even lightweight search (50) (25.6 minutes) outperforms Adam (50) (6.6 minutes) by over 2×2\times in PSNR BD-Rate, and search (500) outperforms Adam (1000) in LPIPS (−16.68%-16.68\% vs. −16.12%-16.12\%) while approaching its DISTS (−28.00%-28.00\% vs. −29.10%-29.10\%) at half the gradient backpropagation steps.
  3. Mechanism Behind Efficiency: Candidate evaluations in the residual direction search require forward passes only (no backward gradient computation) and directly evaluate the exact hard-rounded objective, preventing Adam from stalling on flat integer quantization intervals (as confirmed by the bimodal step-size selection in Figure 5(b)). We will explicitly include these runtime figures and matched-budget analyses in Table 2.

Response to Weakness 4: Computational Complexity and Necessity of Test-Time Optimization

We appreciate your suggestion to clearly report computational complexity and runtime:

  1. Necessity of Test-Time Optimization: Implicit-motion codecs such as DCVC-UF sacrifice explicit motion estimation and compensation to achieve ultra-fast decoding. While this design greatly improves throughput, it creates an inherent high-bitrate bottleneck where coupled temporal errors cannot be resolved by a single amortized forward pass. Encoder-side test-time latent refinement eliminates this amortization gap without modifying network weights or bitstream syntax, preserving full compatibility with the frozen decoder. This paradigm aligns directly with video-on-demand, streaming distribution, and media archiving workflows, where a video is encoded once offline and decoded many times.
  2. Exact Latency Overhead:
    • Decoder Latency Overhead: Zero (0 ms0\text{ ms}). Because only the transmitted latent y∗y^{\ast} is refined and the decoder parameters remain unchanged, our method preserves the exact decoding throughput of the DCVC-UF baseline.
    • Encoder Latency Overhead: Measured on a single NVIDIA A800 GPU under the ablation setting of Table 2 on HEVC Class D, the encoding runtime of our residual direction search is 4.74 hours for 500 steps and 25.6 minutes for 50 steps. Importantly, as shown in Table 2, even a lightweight 50-step configuration already reduces DISTS BD-Rate by −9.76%-9.76\% and PSNR BD-Rate by −4.21%-4.21\% while reducing encoding time by over 11×11\times. We will add these exact runtime figures and speed-quality trade-offs to Table 2 and Section 4.3 in the revised manuscript.

Response to Weakness 5: Comparison with In-Domain Enhancement and RDO Methods

Thank you for pointing out the distinction between out-of-domain adaptation and in-domain enhancement. Existing in-domain enhancement methods for neural video codecs primarily rely on learned filtering modules (such as in-loop contextual filtering and out-of-loop reconstruction enhancement), rather than encoder-side test-time latent optimization.

We actually explored adapting such in-loop and out-of-loop convolutional filtering modules onto the frozen DCVC-UF backbone, and observed two fundamental limitations:

  1. Limited Coding Gain: Lightweight pre-/post-processing and convolutional filtering modules yield only ~5% BD-Rate improvement on DCVC-UF, as post-filtering cannot recover high-frequency details already discarded by coarse latent quantization at the encoder.
  2. Architectural and Bitstream Incompatibility: In-loop filtering modifies the decoder architecture and requires retraining network weights, breaking compatibility with the frozen DCVC-UF decoder—whereas our method achieves −13.27%-13.27\% (PSNR) / −28.94%-28.94\% (DISTS) BD-Rate savings with zero decoder modification.

By contrast, DCVC-DT* represents the closest encoder-side, decoder-preserving latent RDO baseline (which we adapted directly to the in-domain DCVC-UF anchor under identical step budgets). We will clarify this distinction and discuss in-domain filtering approaches in Section 2.3 and Section 4.2 of the final manuscript.

Responses to Questions for Authors

  1. Question 1 (Figure 1 Title): Thank you for catching this! We will remove "at QP63" from the title of Figure 1 in the final manuscript so that it accurately reflects the full RD curve across all QPs.
  2. Question 2 (Figure 2 Arrow Placement): Thank you for the helpful suggestion. We will update Figure 2 in the final version so that the "Online Modulation" indicator points directly to the latent feature yy before quantization and Arithmetic Encoding (AE), accurately matching our formulation.
  3. Question 3 (Lookahead Latency vs. Offline Application Scope): We fully agree with your observation and will cite UI2C (Xiang et al., CVPR 2026) in our revised discussion. Because our framework is designed specifically for offline encoding workflows (e.g., Video-on-Demand, streaming distribution, and media archiving, where content is encoded once offline and decoded many times, as discussed in Weakness 4), the 1-chunk (8-frame) lookahead buffering delay does not impact real-time user playback, while preserving zero decoder overhead is paramount.

Thank you once again for your constructive review, which has been genuinely helpful in strengthening the rigor and clarity of our work. We hope our responses and the newly added 1080p results, runtime analyses, and technical clarifications have addressed your concerns regarding evaluation scope, perceptual optimization, and computational efficiency. We would be very grateful if you could revisit our rebuttal and updated paper, and reconsider your assessment in light of these revisions. Should any questions remain, we would be happy to discuss them further.

Best regards,
Authors of Submission 1637

Reviewer_vg9vOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces test time optimization for I and P frame chunks. It proposes a temporally correlated ROI. And it achieves SOTA result on HEVC class C and D.

Critical strengths

The results are SOTA and the method is clear.

Critical weaknesses
  1. The combination of loss terms is not an innovation.
  2. I have concerns about the experiments. LPIPS and DISTS are already explicitly optimized through the loss term. But experiments are still based on them.
  3. The test-time optimization introduced here might result in a big latency overhead.
Questions for authors
  1. Is test-time optimization necessary here? This looks computational intensive.
  2. What is the latency overhead introduced by your method.
3 comments

We thank Reviewer vg9v for acknowledging our state-of-the-art results and the clarity of our method. Below, we address your comments regarding our technical contributions, experimental evaluation, and latency profile.


Response to Weakness 1: Technical Contributions Beyond Loss Combination

While incorporating perceptual terms into the rate-distortion objective is a starting point, our primary technical contributions resolve two structural bottlenecks specific to implicit-motion neural video codecs:

  1. Cross-Chunk Reference Modeling via Lookahead and Cycle-Consistent ROI (Section 3.3): Implicit-motion codecs such as DCVC-UF pack eight P-frames into a single latent without explicit optical flow, causing temporal features to become tightly coupled across chunks. Optimizing each chunk independently degrades inter-chunk propagation. To address this, we introduce a future-chunk lookahead mechanism paired with a bidirectional optical flow cycle-consistency mask (Equation 11). This formulation identifies reliable motion trajectories where anchor errors propagate into future frames and directs bitrate allocation specifically to those critical reference regions. As shown in Table 4, adding our lookahead and temporally corrected ROI reduces PSNR BD-Rate from -11.53% to -15.00% and LPIPS BD-Rate from -15.98% to -20.71% compared to optimizing the local objective alone.

  2. Discrete Latent Optimization via Simulated Annealing and Residual Search (Section 3.2): Because hard rounding y^=round(y)\hat{y} = \mathrm{round}(y) yields zero gradients, standard Adam updates struggle on the piecewise-flat quantization landscape. Beyond Stochastic Gumbel Annealing, we propose a residual direction search (Equations 6 and 7) that evaluates multiple candidate step sizes directly under the hard-rounded objective at each iteration. As demonstrated in Table 2 and Figure 6(a), this search accelerates early convergence and achieves an additional 7.9% DISTS BD-Rate reduction over standard Adam at matched step counts.


Response to Weakness 2: Instance-Specific Overfitting and Joint Objective-Subjective Gains

We respectfully clarify a fundamental distinction between model-level generalization and encoder-side latent optimization:

  1. Per-Sequence Overfitting is Legitimate and Essential in Compression: Unlike standard machine learning tasks where optimizing on test data constitutes data leakage, rate-distortion optimization in video compression is inherently designed to overfit the transmitted bitstream to the specific input sequence. Because our method keeps the decoder parameters completely frozen and optimizes only the quantized latent representation y∗y^* for the current video, fitting y∗y^* to better represent both pixel fidelity and perceptual quality of that exact sequence is the intended mechanism rather than an evaluation flaw.

  2. Jointly Achieving Objective Fidelity and Subjective Quality Without Trade-Off Degradation: Simply adding LPIPS and DISTS to an optimizer does not guarantee improvements across both objective and subjective dimensions. In neural video compression, naive perceptual optimization typically causes severe pixel-level distortion due to the perception-distortion trade-off. In contrast, our joint modulation with simulated gradient annealing and lookahead ROI enables the latent to simultaneously improve objective fidelity (-13.27% PSNR BD-Rate) and perceptual quality (-18.06% LPIPS and -28.94% DISTS BD-Rate) over DCVC-UF. By comparison, fidelity-only test-time optimization (DCVC-DT* in Table 1) improves PSNR by -8.73% on Class C but degrades DISTS by +12.27%.


Response to Weakness 3: Latency Overhead

Our framework operates exclusively on the encoder side while leaving the decoder completely frozen, resulting in zero decoder latency overhead (0 ms). Detailed encoder runtime measurements and speed-quality trade-offs are provided in our response to Question 2 below.

Response to Question 1: Necessity of Test-Time Optimization

Implicit-motion codecs such as DCVC-UF sacrifice explicit motion estimation and compensation to achieve ultra-fast decoding. While this design greatly improves throughput, it creates an inherent high-bitrate bottleneck where coupled temporal errors cannot be resolved by a single amortized forward pass. Encoder-side test-time latent refinement eliminates this amortization gap without modifying network weights or bitstream syntax, preserving full compatibility with the frozen decoder. This paradigm aligns directly with video-on-demand, streaming distribution, and media archiving workflows, where a video is encoded once offline and decoded many times.


Response to Question 2: Exact Latency Overhead

  • Decoder Latency Overhead: Zero (0 ms). Because only the transmitted latent yy is refined and the decoder parameters remain unchanged, our method preserves the exact decoding throughput of the DCVC-UF baseline.
  • Encoder Latency Overhead: Measured on a single NVIDIA A800 GPU under the ablation setting of Table 2 on HEVC Class D, the encoding runtime of our residual direction search is 4.74 hours for 500 steps and 25.6 minutes for 50 steps. Importantly, as shown in Table 2, even a lightweight 50-step configuration already reduces DISTS BD-Rate by -9.76% and PSNR BD-Rate by -4.21% while reducing encoding time by over 11x. We will add these exact runtime figures and speed-quality trade-offs to Table 2 and Section 4.3 in the revised manuscript.

We hope these clarifications address your concerns regarding our methodological novelty, experimental soundness, and practical efficiency. We would greatly appreciate it if you could reconsider your assessment in light of these revisions.

Thanks for the reply, I would raise my score to 2(weak reject). I think the method makes sense and the response addresses my concern about overfitting.

Reviewer_xtjcOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an adaptive test-time optimization technique to improve video compression in terms of rate-distortion. The focus is on only modifying the latent at the encoder side, such that the same decoder can be used. The proposed modification/optimization objective consists of including perceptual quality instead of only fidelity to the optimization function. Further, temporal knowledge due to processing multiple frames is used to allocate bits in meaningful regions by the proposed lookahead mechanism using bi-directional optical flow. The method has been evaluated on two classes of the HEVC dataset, showing significant improvements compared to the non-optimized baseline. The design choices and some variations of them have been ablated, showing their empirical benefit.

Critical strengths

S1: Significant improvement over the baseline by using test-time optimization:

  • This shows the benefit of the proposed method compared to not using any test-time optimization.
  • Compared to one other adaptive method, it shows improvement on the perceptual metrics, not on PSNR, due to using perceptual loss explicitly in the loss function.

S2: Proposed modifications are ablated, and their benefits are shown.

Critical weaknesses

W1: Limited experiments, and therefore it is difficult to draw broader conclusions about the usability/benefit of the method wrt. to related work:

  • Only compared to a single other optimization-based method, while the related work section lists multiple. Compared to that one, the improvements are mainly visible in the perceptual metric, which is kind of expected as the perceptual is part of the optimized objective.
  • Test-time optimization adds computational costs to the method; however, this has not been reported nor compared between the different adaptive methods. While some added costs are fine, it is still relevant information about the method and should be part of the comparisons to give the full picture of the advantages and disadvantages of the method, such that the reader knows about them when considering applying the method. Currently, it only provides a partial picture of the performance, focusing on rate-distortion while ignoring the computational costs.
  • The method has only been evaluated on two subgroups of a single dataset, which is of rather small resolution, while the used baseline method has also been evaluated on higher resolutions and multiple datasets (1920x1080, UVG, and MCL-JCV). This also limits insight into the method's current practical usability.
  • Adding test-time optimization, and having improvements on the metrics when using them as an objective, are both not very surprising, and therefore, really broader comparisons are needed to learn more about the benefits of the method compared to related work.

W2: The method is hard to follow, a lot of guessing is needed to follow the method description due to imprecise or incomplete terminology:

  • Overview figure (Fig. 2) not really helpful as terminology is not consistent with method description/variable names
  • y* mentioned in text but not in formula or method figure
  • using imprecise terminology, e.g., calling a loss the "official UF 8-frame MSE" in the main paper, without giving the references (only when looking at the appendix, it becomes clear that it is the official DCVC-UF (HT-L) training loss), many used metrics are missing the paper references, e.g., also DISTS (from Ding et al., "Image Quality Assessment: Unifying Structure and Texture Similarity", TPAMI 2020)
  • Not really clear what A and B are in Equation 3. While L_A is defined (even twice, eq. 2 and 5, L_B isn't). The terms are only later introduced.
  • Abbreviations (e.g., DPB) are used without spelling out the full name.
  • Inconsistency about the used Optical Flow network: Figure 4 mentions FlowNet, text mentions SpyNet, both of which are rather old (2015 and 2017), and some of the very first network-based optical flow methods. This choice requires further discussion to explain why these have been used instead of newer, more accurate optical flow methods.
Questions for authors

Answering the following questions would provide a better picture of the method, however, the difficulty in following the method and the limited insights will remain:

  • Why are only smaller resolutions evaluated? Would it also work on 1920x1080? Why or why not?
  • How much does the test-time adaptation increase the encoding time compared to DCVC-UF (HT-L) and other adaptive-optimization techniques? While the focus is on offline encoding, these numbers should still be reported for completeness, and to give an estimate of how expensive the obtained improvements are.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.