acceptodds
Under review as a conference paper at ICLR 2027

Reliable and Propagation-Aware Test-Time Latent Adaptation for Video Compression

Abstract

Neural video codecs rely on pretrained amortized encoders to generate coding latents, which may not be optimal for individual test videos. Encoder-side test-time latent adaptation provides an attractive way to improve instance-specific rate-distortion performance while keeping the pretrained codec and decoder unchanged. Extending latent refinement to chunk-based video coding, however, introduces two challenges: optimization is driven by differentiable surrogates that may not reflect native coding outcomes, and each adapted reconstruction changes the coding state used for subsequent chunks. We present, to our knowledge, the first encoder-side latent adaptation framework specifically designed for chunk-based neural video compression. At the chunk level, a gradient-conditioned learned optimizer generates coupled updates for the primary and hyper latents, while a distortion-constrained safety gate controls their local application with a fallback to the original encoder output. At the sequence level, candidate update policies are evaluated over a short observed prefix through separate causal closed-loop coding trajectories, using native bitstreams and decoded reconstructions to capture both actual coding behavior and temporal propagation effects. The selected policy is then causally applied from the state produced by the transmitted prefix, while all candidate trajectories remain private to the encoder. Our method requires no modification to the pretrained codec, decoder, or bitstream syntax, and introduces no additional signaling overhead. Experiments on six standard benchmark groups demonstrate consistent improvements over DCVC-UF, achieving an average BD-Rate reduction of 3.2% and up to 4.9% on individual datasets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural video compression, test-time adaptation, latent optimization, learned optimizer, rate–distortion optimization, temporal propagation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eLUtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes encoder-side test-time latent adaptation for a frozen DCVC-UF codec. Gradient-conditioned update modules refine the primary and hyper latents, with a distortion-constrained safety gate and propagation-aware sequence-level policy selection based on native prefix rate–distortion evaluation. Only the original-policy prefix is transmitted; the committed policy is applied to suffix chunks. The method preserves the decoder and bitstream syntax and reports an average BD-Rate reduction of approximately 3.2% across six benchmark groups.

Critical strengths
  1. Improving compression through encoder-side latent refinement while keeping all codec parameters and decoding-side operations unchanged is practically appealing.
  2. The experiments report both aggregate gains and per-sequence failure cases, helping readers assess the method's applicability.
Critical weaknesses
  1. Missing computational cost. Please report MACs/FLOPs, encoding time, and peak GPU memory for the original DCVC-UF and the full method at matched resolutions and configurations. Separate the costs of learned latent refinement, the safety gate, and closed-loop candidate trajectories. Ten update steps require repeated differentiable forward evaluations and gradient computations; their cost cannot be represented simply as one forward pass plus ten backward passes.

  2. Insufficient ablation of the refinement budget. Why is M=10M=10 selected? Please compare different numbers of update steps, such as 1, 5, 10, and 20, reporting native BD-Rate and encoding time on each dataset. Clarify whether changing MM requires retraining the learned optimizer. Comparing ten learned updates with twenty plain Adam steps does not isolate the effect of the refinement budget; an Adam baseline with the same safety control and policy selection under a matched budget would be more informative.

  3. Missing comparison with DCVC-DT. DCVC-DT also performs encoder-side latent refinement with frozen codec parameters and introduces dynamic RD adjustment, but it is neither cited nor evaluated. Please provide a comparison on the same backbone with matched evaluation protocols and encoding-time budgets. If its frame-level mechanism cannot be directly adapted to chunk-based coding, explain the limitations and provide an appropriate controlled baseline. Comparing with DCVC-DC alone does not evaluate DCVC-DT's adaptation mechanism.

  4. Unclear quantization implementation. Please specify the actual optimization variables and the quantization forward pass and gradient estimator used at each refinement step. If a continuous relaxation is used, explain how the final discrete symbols are obtained for native entropy coding. Does safety-gate verification use the same quantization forward pass as native coding? This detail is important for interpreting the relationship between the differentiable reconstruction and native coding outcomes.

Questions for authors

See Weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer eLUt (1/4)

Thank you for the detailed and technically precise review. We appreciate the recognition of frozen-decoder latent refinement and of our aggregate and per-sequence evaluation. We address each requested clarification and controlled comparison below.

[Weakness 1] Missing computational cost

  1. Missing computational cost. Please report MACs/FLOPs, encoding time, and peak GPU memory for the original DCVC-UF and the full method at matched resolutions and configurations. Separate the costs of learned latent refinement, the safety gate, and closed-loop candidate trajectories. Ten update steps require repeated differentiable forward evaluations and gradient computations; their cost cannot be represented simply as one forward pass plus ten backward passes.

We agree that the original submission did not report enough information to interpret the computational cost. Encoding time and peak memory are now measured under matched codec, resolution, frame count, and native bitstream configurations. The verified peak-memory results for HT-L are:

DatasetDCVC-UF HT-LFull method
HEVC D1.346 GiB7.762 GiB
HEVC C1.641 GiB14.584 GiB
HEVC E2.172 GiB26.135 GiB
HEVC B3.240 GiB50.281 GiB

To address the reviewer's important warning about the ten update steps, we count the operations on the actually executed path rather than approximating the method as “one forward plus ten backward passes.” Native coding trajectories are measured using Nsight Compute tensor-path counters, while differentiable refinement and safety verification are measured using PyTorch's operator-level FLOP counter; ADD/MUL count as one FLOP and FMA as two FLOPs. For 128 P frames (sixteen eight-frame chunks), we reconstruct the total as 65Crefine+Cdist+8Cgate,cached+64Cgate+8Cbase-traj+72Ccand-traj65C_{\mathrm{refine}}+C_{\mathrm{dist}}+8C_{\mathrm{gate,cached}}+64C_{\mathrm{gate}}+8C_{\mathrm{base\text{-}traj}}+72C_{\mathrm{cand\text{-}traj}}, thereby including every repeated differentiable evaluation, gradient computation, native verification, and closed-loop trajectory that is actually executed.

The final matched-resolution results are:

DatasetResolutionDCVC-UF FLOPs/128PFull FLOPs/128P
HEVC D416×240416\times2403.695 T261.708 T
HEVC C832×480832\times48013.967 T1039.776 T
HEVC E1280×7201280\times72032.076 T2717.087 T
HEVC B1920×10801920\times108072.099 T6151.932 T

The full-method FLOPs separate as follows:

DatasetLearned refinementSafety gateClosed-loop trajectoriesTotal
HEVC D208.371 T21.950 T31.386 T261.708 T
HEVC C833.114 T87.763 T118.899 T1039.776 T
HEVC E1922.572 T521.846 T272.669 T2717.087 T
HEVC B4356.255 T1182.424 T613.253 T6151.932 T

The totals scale monotonically with resolution and are 70.83–85.33×70.83\text{--}85.33\times those of the frozen codec. Learned refinement is the dominant component (70.8–80.1%70.8\text{--}80.1\% of full FLOPs), followed by closed-loop trajectories (10.0–12.0%10.0\text{--}12.0\%) and safety verification (8.4–19.2%8.4\text{--}19.2\%). The component sums exactly recover the reported full-method totals.

We also decompose encoder time on one representative sequence per dataset, averaged over the four QPs. Each entry reports seconds per 128 P frames, followed by its fraction of the complete encoder time:

ComponentHEVC DHEVC C
Learned latent refinement106.316 s (92.83%)129.244 s (93.53%)
Safety verification6.736 s (5.84%)7.007 s (5.07%)
Closed-loop candidate trajectories1.179 s (1.03%)1.540 s (1.11%)
Native suffix encoding0.065 s (0.06%)0.080 s (0.06%)
Complete encoder114.562 s138.191 s

The transmitted native prefix path takes a further 0.151 s on HEVC D and 0.202 s on HEVC C; all remaining orchestration accounts for only 0.115 s and 0.118 s, respectively. Thus, more than 92% of the full encoding time is spent in the actually executed learned-refinement pipeline, including latent preparation and all ten differentiable update steps. Safety verification contributes approximately 5 –\text{--} 6%, whereas the native closed-loop candidate transitions and suffix bitstream generation are comparatively small. This explicit measurement confirms the reviewer's observation that the cost cannot be represented by a schematic “one forward plus ten backward passes.” We keep the learned-refinement, safety-verification, closed-loop-search, and native-encoding costs separate rather than folding them into a single opaque total.

AuthorsOpenReview ↗

Response to Reviewer eLUt (2/4)

[Weakness 2] Insufficient ablation of the refinement budget

  1. Insufficient ablation of the refinement budget. Why is (M=10) selected? Please compare different numbers of update steps, such as 1, 5, 10, and 20, reporting native BD-Rate and encoding time on each dataset. Clarify whether changing (M) requires retraining the learned optimizer. Comparing ten learned updates with twenty plain Adam steps does not isolate the effect of the refinement budget; an Adam baseline with the same safety control and policy selection under a matched budget would be more informative.

We agree that comparing ten learned updates only with twenty plain Adam steps does not isolate the refinement budget. We therefore make two controlled comparisons. First, we vary the learned budget over M∈{1,5,10,20}M\in\{1,5,10,20\} while keeping safety control and policy selection fixed. A separate learned optimizer is trained for each MM, so changing MM in this controlled ablation does require retraining because the module learns to imitate an optimizer trajectory of the corresponding length. Second, we evaluate Adam with the same safety control and policy selection at 10 and 20 steps. Adam-10 provides the near-matched-budget control for Learned-10, thereby isolating the update rule from both the refinement budget and the deployment mechanisms.

RuleStepsHEVC BHEVC CHEVC DHEVC EAverageEnc. s/128P
Learned1-1.276%-1.445%-1.233%-0.453%-1.101%23.214
Learned5-4.232%-4.690%-4.455%-1.312%-3.672%65.501
Learned10-5.028%-5.120%-5.214%-1.574%-4.234%121.662
Learned20-5.586%-5.668%-5.532%-1.602%-4.597%231.164
Adam10 (near-matched)-4.339%-4.481%-4.203%-1.055%-3.520%110.776
Adam20-5.288%-5.543%-5.508%-1.603%-4.486%224.695

The learned-budget rows report native BD-Rate on every requested dataset under the final safety tolerance and an identical policy-selection protocol. Runtime increases from 23.214, 65.501, and 121.662 to 231.164 s/128 P frames for M=1,5,10,20M=1,5,10,20, respectively. Although M=20M=20 achieves the strongest average RD result, its 0.363-percentage-point improvement over M=10M=10 nearly doubles the encoding time; this diminishing return motivates the default M=10M=10.

For the update-rule control, Adam uses exactly the same safety control and policy selection as the learned method. Adam-10 takes 110.776 s/128 P frames, close to the 121.662 s of Learned-10, while its average BD-Rate is -3.520%, compared with -4.234% for Learned-10. Under these comparable encoding-time budgets, the learned optimizer improves the average result by 0.714 percentage points. The same trend remains at 20 steps: Learned-20 achieves -4.597% at 231.164 s, whereas Adam-20 achieves -4.486% at 224.695 s.

We expanded the ablation table and explicitly state the checkpoint-training requirement for each learned budget.

AuthorsOpenReview ↗

Response to Reviewer eLUt (3/4)

[Weakness 3] Missing comparison with DCVC-DT

  1. Missing comparison with DCVC-DT. DCVC-DT also performs encoder-side latent refinement with frozen codec parameters and introduces dynamic RD adjustment, but it is neither cited nor evaluated. Please provide a comparison on the same backbone with matched evaluation protocols and encoding-time budgets. If its frame-level mechanism cannot be directly adapted to chunk-based coding, explain the limitations and provide an appropriate controlled baseline. Comparing with DCVC-DC alone does not evaluate DCVC-DT's adaptation mechanism.

Thank you for pointing out this closely related work. We agree that comparison with DCVC-DC alone does not evaluate DCVC-DT's adaptation mechanism. We now cite DCVC-DT and compare it on the same DCVC-DC backbone. All methods use native YUV420 bitstreams, 96 frames, intra period 32, four rate points, and the same weighted YUV PSNR. We evaluate both the original frame-wise mechanism and a controlled joint-miniGOP variant under matched encoding-time budgets.

On HEVC D:

MethodTemporal unitIterationsNative BD-Rate vs. DCVC-DCEnc. s/96 framesRelative time
DCVC-DCFrame--0.000%3.2521.00×1.00\times
DCVC-DT, frame-wiseFrame10-0.227%118.00836.29×36.29\times
DCVC-DT, frame-wiseFrame40-0.781%452.082139.03×139.03\times
DCVC-DT, joint mini-GOPmini-GOP-443-0.672%455.093139.96×139.96\times
Oursmini-GOP-410 learned updates-1.421%450.312138.49×138.49\times

On HEVC C:

MethodTemporal unitIterationsNative BD-Rate vs. DCVC-DCEnc. s/96 framesRelative time
DCVC-DCFrame--0.000%7.8611.00×1.00\times
DCVC-DT, frame-wiseFrame10-0.207%203.95925.95×25.95\times
DCVC-DT, frame-wiseFrame40-0.700%867.436110.35×110.35\times
DCVC-DT, joint mini-GOPmini-GOP-447-0.779%887.257112.87×112.87\times
Oursmini-GOP-410 learned updates-1.739%869.910110.67×110.67\times

Across the two datasets, we compare the original frame-wise DCVC-DT mechanism, its joint mini-GOP counterpart, and our learned adaptation on the same DCVC-DC backbone and evaluation protocol. On HEVC D, frame-wise DCVC-DT at M=40M=40 is time-matched to ours within 0.4%, while joint mini-GOP DCVC-DT at M=43M=43 is within 1.1%. Our method achieves the largest RD improvement, reducing BD-Rate by 1.421%, compared with 0.781% for the time-matched frame-wise baseline and 0.672% for the joint mini-GOP baseline. On HEVC C, increasing frame-wise DCVC-DT from 10 to 40 iterations improves its BD-Rate from -0.207% to -0.700%. The 867.436-second frame-wise M=40M=40 run differs from our 869.910 seconds by only 0.28%. Single-point calibration on RaceHorses at rate index 1 selects M=47M=47 for joint mini-GOP DCVC-DT: its verified 887.257-second encoding time differs from ours by only 1.99%. Under these matched budgets, joint mini-GOP DCVC-DT achieves -0.779%, whereas our method reaches -1.739%, an additional 0.960 percentage-point reduction.

DCVC-DT's original frame-wise mechanism cannot be transferred verbatim to DCVC-UF because DCVC-UF jointly encodes an eight-frame chunk and shares a structured latent update across that unit. Treating each frame independently would break the backbone's native chunk structure. The joint-miniGOP experiment therefore serves as a controlled baseline for structured temporal adaptation rather than claiming an exact architectural equivalence.

We added DCVC-DT to Related Work and include same-backbone, same-protocol, and matched-runtime comparisons on both HEVC C and D.

AuthorsOpenReview ↗

Response to Reviewer eLUt (4/4)

[Weakness 4] Unclear quantization implementation

  1. Unclear quantization implementation. Please specify the actual optimization variables and the quantization forward pass and gradient estimator used at each refinement step. If a continuous relaxation is used, explain how the final discrete symbols are obtained for native entropy coding. Does safety-gate verification use the same quantization forward pass as native coding? This detail is important for interpreting the relationship between the differentiable reconstruction and native coding outcomes.

We apologize that these implementation details were unclear. The optimization variables are the continuous primary and hyper latents (yt,zt)(\mathbf{y}_t,\mathbf{z}_t). During differentiable refinement, quantization is represented by hard rounding in the forward pass with a straight-through gradient estimator in the backward pass. The learned optimizer consumes the current latents and their gradients and predicts coupled updates for both components.

After refinement, the adapted continuous latents are passed to the codec's unmodified native compress path. This path performs the codec's original discrete quantization and entropy coding, producing the actual transmitted symbols and bitstream. The decoder reconstructs exclusively from that native bitstream.

The safety gate does not reuse the differentiable STE forward pass for its final decision. Each candidate is recompressed and decoded through the codec's native quantization and entropy-coding path, and acceptance uses the resulting native reconstruction and bitstream behavior. Therefore, the differentiable quantization path generates the update direction, while the native discrete path determines both safety acceptance and deployment. This explicitly separates differentiable reconstruction quality from the native coding outcome highlighted by the reviewer.

We added these statements immediately after the refinement objective in Section 3.2 and clarified the distinction between differentiable generation and native verification in Section 3.3.

[Questions for Authors]

See Weaknesses

All questions raised through the four weaknesses are addressed individually above.

We sincerely appreciate the reviewer’s detailed suggestions. The added cost accounting, refinement-budget controls, DCVC-DT comparison, and quantization description have materially improved the completeness and reproducibility of the paper.

Reviewer_TST8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes encoder-side test-time latent adaptation for chunk-based neural video compression. It combines learned dual-latent updates, distortion-constrained safety control, and causal policy selection using native rate–distortion evaluation over independent prefix trajectories. Experiments on six benchmark groups report an average 3.2% BD-Rate reduction over DCVC-UF, without modifying the decoder or adding signaling overhead.

Critical strengths
  1. Well-motivated encoder-side adaptation. The paper addresses a relevant problem through encoder-side test-time optimization. Allocating additional computation to the encoder while keeping the decoder unchanged is consistent with the computational asymmetry of traditional video codecs.

  2. Evaluation with actual bitstreams. The method is evaluated using actual bitstreams and decoded reconstructions across six benchmark groups, providing empirical evidence of rate–distortion improvements beyond the differentiable optimization objective.

  3. Useful component ablations. The ablations demonstrate the practical value of safety control, showing that it mitigates dataset-level degradations caused by unconstrained updates and improves overall rate–distortion performance.

Critical weaknesses
  1. About background setting. The authors should establish and justify the causal, chunk-wise adaptation setting before introducing the challenges, rather than leaving its constraints and motivation to later descriptions. They should also clarify which challenges are specific to or exacerbated by this setting, and discuss how it differs from jointly optimizing the latents of multiple chunks at the sequence level.

  2. About update reliability. While Table 2 and Fig. 5(a) support the effectiveness of the gate, additional analysis of why unconstrained updates sometimes degrade performance would be helpful, e.g., separating the effects of surrogate mismatch, update magnitude, and temporal propagation.

  3. About notation consistency. The notation switches from v∈{y,z}v \in \{y,z\} to uu in Eqs. (4)–(5) without explanation, while Eq. (7) uses uu for the latent pair. Please define these symbols explicitly and use consistent notation.

  4. About adaptation to changing content. The scaling policies are predefined, and the selected policy remains fixed throughout the suffix. Although latent updates and safety gating remain chunk-dependent, it would be helpful to examine whether this design sufficiently adapts to changes in motion or scene content, and whether periodic policy reselection offers further benefits.

  5. About cross-chunk gradient propagation. DCVC-UF emphasizes the importance of cascaded training. However, the proposed latent updates are driven by the current chunk’s objective, while cross-chunk effects are addressed through policy selection rather than gradient propagation. This design choice and its implications for sequence-level rate–distortion optimization deserve further discussion.

  6. About the rate–distortion–complexity trade-off. Please report results for both DCVC-UF HT-S and HT-L, with and without the proposed adaptation, together with encoding time, memory usage, and prefix-search overhead. These comparisons would clarify whether the smaller HT-S codec combined with the proposed method offers a better trade-off than directly using HT-L, while accounting for the benefit of retaining a lightweight decoder. It would also be useful to assess the gains from applying the proposed method to HT-L. The reported RD improvements alone do not establish whether the method provides a better rate–distortion–complexity trade-off than simply using a larger codec.

Questions for authors

see weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer TST8 (1/4)

Thank you for the careful review and for recognizing the motivation, native-bitstream evaluation, and practical role of safety control. Your comments helped us clarify the causal setting and strengthen the mechanism and complexity analyses. We respond point by point below.

[Weakness 1] Background setting

  1. About background setting. The authors should establish and justify the causal, chunk-wise adaptation setting before introducing the challenges, rather than leaving its constraints and motivation to later descriptions. They should also clarify which challenges are specific to or exacerbated by this setting, and discuss how it differs from jointly optimizing the latents of multiple chunks at the sequence level.

We agree that this setting should have been established before introducing the technical challenges. In our setting, chunk tt is observed once, encoded causally from state sts_t, and immediately committed to the bitstream. The resulting decoded reconstruction updates st+1s_{t+1}. Future chunks are unavailable when the current chunk is committed. This differs from offline sequence-level latent optimization, which assumes access to multiple future chunks and repeatedly backpropagates through their joint trajectory.

This causal constraint clarifies which challenges are specific to, or amplified by, our setting. Surrogate-to-native mismatch is not unique to chunk-based coding, but it becomes more consequential because a misleading update is committed to a native bitstream and cannot subsequently be revised. Temporal propagation is intrinsic to causal predictive coding: an adapted reconstruction changes the state used by all later chunks, so a locally beneficial update can harm the suffix. Finally, the encoder cannot use future chunks to choose or correct the current update. These constraints motivate native local safety verification and encoder-private closed-loop prefix trajectories.

This setting is fundamentally different from jointly optimizing the latents of multiple chunks at sequence level. Joint optimization assumes that several chunks are simultaneously available, retains their differentiable trajectory, and revisits earlier latent decisions using future losses. Our setting observes each chunk once and commits it causally; the prefix is used only to select a policy from native closed-loop evidence, not to backpropagate future gradients into already transmitted chunks.

We added this definition and the contrast with offline sequence optimization to the Introduction and the beginning of Method, before discussing surrogate mismatch and temporal propagation.

[Weakness 2] Why unconstrained updates degrade

  1. About update reliability. While Table 2 and Fig. 5(a) support the effectiveness of the gate, additional analysis of why unconstrained updates sometimes degrade performance would be helpful, e.g., separating the effects of surrogate mismatch, update magnitude, and temporal propagation.

We agree and appreciate this concrete suggestion. Table 2 and Fig. 5(a) establish that the gate is useful, but they do not by themselves identify why an unconstrained update fails. We therefore organize the additional analysis around three factors:

  1. Surrogate mismatch: whether a favorable differentiable-objective change agrees with the native bitstream RD change.
  2. Update magnitude: how native endpoint degradation varies with the norm of the latent modification and how safety scaling suppresses excessive steps.
  3. Temporal propagation: whether an update that is locally acceptable at chunk tt produces cumulative gains or losses in subsequent coding states.
FactorDiagnosticResult
Surrogate mismatchSurrogate/native distortion changes over 128 first-chunk policy endpointsDirection agreement is 86.72% (Pearson r=0.994r=0.994, Spearman ρ=0.977\rho=0.977), but 13/128 endpoints (10.16%) are false improvements: the surrogate improves while native distortion degrades
Update magnitudeLatent-pair norm, unsafe native endpoints, and gate outcomesNorm spans 1.592--87.395; 55/128 unattenuated endpoints exceed the 1% native-distortion budget, and safety control restores 45/55 (81.82%) to the budget; γ<1\gamma<1 for 37.50% and analytic fallback occurs for 29.69%
Temporal propagationLocal vs. 8-chunk native closed-loop RD over 128 policy trajectoriesThe locally best policy matches the 8-chunk best policy in only 4/16 tasks (25%); one trajectory changes from local improvement to cumulative degradation, and mean gain decays from −2.026%-2.026\% at K=1K=1 to −1.819%-1.819\% at K=8K=8
AuthorsOpenReview ↗

Response to Reviewer TST8 (2/4)

The analysis covers four representative sequences with different motion and resolution characteristics (RaceHorses, BasketballDrill, FourPeople, and BQTerrace), four QPs, and eight nonzero policies per task. The high global surrogate/native correlations show that the differentiable signal is informative on average, while the 10.16% false-improvement rate explains why unconstrained reliance on that signal is unreliable. Update magnitude alone is also insufficient: its correlation with native change is only r=0.287r=0.287, and 42.97% of the unattenuated endpoints violate the native 1% budget. Finally, temporal evidence changes the policy ranking in 75% of the sequence--QP tasks, directly motivating selection from accumulated native closed-loop evidence rather than from one locally favorable chunk.

These diagnostics distinguish three failure sources rather than attributing all degradations to one mechanism. They also clarify that safety control and policy selection solve different failure modes: the former regulates the admissible chunk-level update, whereas the latter evaluates sequence-level consequences using native closed-loop trajectories.

Accordingly, Section 4.4 is organized around these three mechanisms and reports the aggregate endpoint and trajectory statistics above.

[Weakness 3] Notation consistency

  1. About notation consistency. The notation switches from v∈{y,z}v\in\{y,z\} to uu in Eqs. (4)--(5) without explanation, while Eq. (7) uses uu for the latent pair. Please define these symbols explicitly and use consistent notation.

We apologize for the inconsistent notation. We unified it throughout the main paper and appendix and now use ut(m)=(yt(m),zt(m))\mathbf{u}_t^{(m)}=(\mathbf{y}_t^{(m)},\mathbf{z}_t^{(m)}) for the primary--hyper latent pair at refinement step mm. When an equation must refer to one component, we use ut,c(m)\mathbf{u}_{t,c}^{(m)}, with c∈y,zc\in\\{y,z\\}, rather than introducing a second overloaded latent symbol. Polices are denoted by aa, with component-wise scaling coefficients (αya,αza)(\alpha_y^a,\alpha_z^a).

Equations (4)--(7), their surrounding definitions, and the appendix derivation now use this same notation.

[Weakness 4] Adaptation to changing content

  1. About adaptation to changing content. The scaling policies are predefined, and the selected policy remains fixed throughout the suffix. Although latent updates and safety gating remain chunk-dependent, it would be helpful to examine whether this design sufficiently adapts to changes in motion or scene content, and whether periodic policy reselection offers further benefits.

We would like to clarify that the policy fixes only the global scaling pair. The actual update remains chunk-dependent because gradients, learned update directions, safety scaling, and fallback decisions are recomputed for every chunk. Therefore, a fixed sequence-level policy does not imply a fixed latent modification.

To test whether explicit reselection is beneficial, we compare the default one-shot policy with reselection every eight chunks on three representative sequences under the same native evaluation protocol and the final ϵ=1%\epsilon=1\% configuration:

SequenceOne-shot BD-RatePeriodic BD-RatePeriodic −- one-shotAdditional search time
HEVC C / BQMall-5.411%-5.475%-0.063%622.25 s
HEVC C / BasketballDrill-4.857%-4.821%+0.036%492.49 s
HEVC E / FourPeople-1.333%-1.133%+0.199%1352.69 s
Mean-3.867%-3.810%+0.057%822.48 s

Periodic reselection changes the selected policy frequently (64 changes over 100 reselection decisions), confirming that the content-dependent evidence is not static. However, these changes do not translate into a consistent native RD benefit: periodic reselection helps BQMall slightly, but degrades BasketballDrill and FourPeople, yielding a 0.057% mean BD-Rate loss while adding 822.48 seconds of search per sequence. This indicates that the chunk-dependent gradients and safety decisions already provide the useful local adaptation, whereas repeatedly committing a policy from another short observation window adds substantial cost and can react to noisy local evidence.

Because policy selection is encoder-private and the selected scaling is applied before native coding, neither schedule requires decoder signaling. We therefore retain one-shot selection after the observed prefix and now state periodic reselection as an optional but empirically unattractive extension for strongly non-stationary content.

AuthorsOpenReview ↗

Response to Reviewer TST8 (3/4)

[Weakness 5] Cross-chunk gradient propagation

  1. About cross-chunk gradient propagation. DCVC-UF emphasizes the importance of cascaded training. However, the proposed latent updates are driven by the current chunk’s objective, while cross-chunk effects are addressed through policy selection rather than gradient propagation. This design choice and its implications for sequence-level rate–distortion optimization deserve further discussion.

Thank you for highlighting this important design choice. DCVC-UF's cascaded training and our test-time adaptation solve different optimization problems. Cascaded training is an offline procedure that optimizes codec parameters using multi-chunk training trajectories. Our method freezes those parameters and adapts per-video latents after deployment under causal bitstream commitment. Backpropagating test-time gradients through multiple chunks would require retaining the entire differentiable codec state, repeatedly revisiting already observed chunks, and delaying bitstream commitment. This conflicts with the causal test-time setting and substantially increases memory and latency. Moreover, the ultimate deployment objective is native entropy coding with discrete symbols, for which a long differentiable surrogate trajectory can accumulate mismatch.

Our design therefore separates responsibilities:

  • the learned optimizer produces an efficient direction from the current chunk's differentiable objective;
  • the safety gate verifies whether the update is locally admissible under native behavior;
  • causal native candidate trajectories measure how the update policy propagates through subsequent coding states.

This is not an approximation to full offline sequence-level gradient optimization under the same information set; it is a deliberate causal alternative that uses native evaluation to account for cross-chunk effects without future-gradient access.

We added this distinction and its implications to Section 3.3 and the Discussion.

[Weakness 6] Rate-distortion-complexity trade-off for HT-S and HT-L

  1. About the rate–distortion–complexity trade-off. Please report results for both DCVC-UF HT-S and HT-L, with and without the proposed adaptation, together with encoding time, memory usage, and prefix-search overhead. These comparisons would clarify whether the smaller HT-S codec combined with the proposed method offers a better trade-off than directly using HT-L, while accounting for the benefit of retaining a lightweight decoder. It would also be useful to assess the gains from applying the proposed method to HT-L. The reported RD improvements alone do not establish whether the method provides a better rate–distortion–complexity trade-off than simply using a larger codec.

We agree that RD improvements alone are insufficient to establish a favorable complexity trade-off. We therefore compare all four configurations on HEVC Classes B –\text{--} E under matched protocols, using HT-S within each dataset as the RD anchor. The adapted HT-L results use the final ϵ=1%\epsilon=1\% configuration.

DatasetConfigurationBD-Rate vs. HT-SEnc. s/128 PPrefix/Suffix FPSPeak memory GiBDec. FPS
HEVC DHT-S0.000%0.04172985.17 / 3158.351.2151947.67
HT-S + Ours-5.086%99.27200.728 / 5.6646.5711953.16
HT-L-20.074%0.05722223.32 / 2253.631.3461358.59
HT-L + Ours-23.773%116.63570.613 / 5.2097.7621378.49
HEVC CHT-S0.000%0.07281750.68 / 1764.961.5181191.99
HT-S + Ours-4.986%103.91870.699 / 5.17611.7271332.04
HT-L-20.158%0.11401087.11 / 1160.831.641857.11
HT-L + Ours-23.869%131.93870.551 / 4.03214.584878.12
HEVC EHT-S0.000%0.1272943.20 / 1078.842.016790.99
HT-S + Ours-0.827%160.01560.440 / 4.44220.477795.61
HT-L-11.365%0.1929644.59 / 683.392.172510.41
HT-L + Ours-12.744%220.16200.329 / 2.46926.135509.15
HEVC BHT-S0.000%0.2208574.47 / 584.823.035383.16
HT-S + Ours-4.561%302.68860.234 / 2.17438.645384.83
HT-L-21.360%0.3811332.34 / 339.533.240251.23
HT-L + Ours-25.041%433.93430.162 / 1.62350.281250.93
AuthorsOpenReview ↗

Response to Reviewer TST8 (4/4)

The result answers the two comparisons separately. First, adaptation improves both HT-S and HT-L, demonstrating that the gain is not restricted to one model scale. Relative to the unadapted HT-L backbone, the final method reduces BD-Rate by 5.214%, 5.120%, 1.574%, and 5.028% on HEVC D, C, E, and B, respectively. Second, HT-S + Ours retains the lightweight HT-S decoder and consistently uses less peak memory than HT-L + Ours, but it does not surpass the RD performance of the substantially larger unadapted HT-L backbone. Thus, adaptation is valuable when the lightweight decoder must be retained, whereas directly selecting HT-L remains the better choice when RD performance is prioritized and a larger decoder is acceptable. We have revised the paper to state this limitation explicitly rather than implying that adaptation universally dominates scaling the codec.

Peak-memory measurements over HEVC B –\text{--} E further show the expected resolution dependence. For example, HT-S/HT-S+Ours/HT-L/HT-L+Ours use 1.215/6.571/1.346/7.762 GiB on HEVC D and 3.035/38.645/3.240/50.281 GiB on HEVC B.

The revised paper includes this four-configuration table and discusses the benefit of retaining a lightweight decoder separately from the encoder-side adaptation cost.

[Questions for Authors]

see weakness.

All questions raised through the six weaknesses are addressed individually above.

We sincerely appreciate these comments, which helped us state the causal setting more clearly and substantially strengthen the mechanism and rate–distortion–complexity analyses.

Reviewer_gTxuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates test-time latent adaptation for neural video compression. Specifically, it tackles a challenging and practical problem: how to refine latents for chunk-based video coding at the encoder side without altering the pretrained decoder or the bitstream syntax. To prevent error propagation and surrogate-to-native discrepancies, the authors introduce a sequence-level evaluation scheme with a candidate update strategy and a distortion-constrained safety gate.

Overall, the manuscript is well-written, and the motivation is clear and timely. The proposed method successfully improves the rate-distortion performance across multiple standard benchmarks (achieving consistent BD-rate reductions over DCVC-UF). The idea of maintaining the original decoding procedure while optimizing encoder-side trajectories is solid and practically appealing.

Critical strengths

High Practical Value: A major strength of this work is that it requires no modifications to the pre-trained decoder or the bitstream syntax. This makes the method fully compatible with existing standards/frameworks, avoiding deployment nightmares in real-world streaming applications.

Thoughtful Handling of Temporal Propagation: Unlike traditional frame-level or chunk-level fine-tuning that often ignores inter-frame dependencies, the proposed sequence-level evaluation explicitly models temporal error accumulation and state propagation.

Robust Safety Mechanism: The introduction of the distortion-constrained safety gate provides a reliable fallback mechanism, effectively preventing catastrophic degradation when the optimization surrogate deviates from the native codec behavior.

Critical weaknesses

High Computational Overhead at the Encoder: Test-time adaptation (TTA) inherently introduces additional computational cost during encoding. While the paper highlights performance gains, a thorough discussion and breakdown of the runtime/latency overhead (e.g., training/optimization time per chunk or sequence compared to baseline encoders) are missing or sparse. This is crucial for video compression, where encoding speed often dictates practical adoption.

Hyper-parameter Sensitivity: The framework relies on several components, such as the gradient-conditioned learned optimizer and the threshold for the safety gate. It remains somewhat unclear how sensitive the performance is to these hyper-parameters across sequences with varying motion complexities (e.g., high-motion sports vs. low-motion conferencing).

Clarity on Sequence-Level Evaluation: In Section 3, the description of how candidate update strategies are evaluated on the "observed prefix" needs tighter mathematical or algorithmic formalization. The trade-off between the length of the prefix and the reliability of the estimated propagation effect requires more empirical justification.

Questions for authors

Computational Cost: Could the authors provide a concrete comparison of the encoding time (e.g., in frames per second or total encoding time per sequence) between the baseline (DCVC-UF) and your method? How does the optimization time scale with the chunk size or resolution?

Safety Gate Threshold: How is the threshold for the distortion-constrained safety gate determined? Is it fixed across all datasets and bitrates, or does it require manual tuning per sequence/bitrate point?

Failure Cases: Are there any specific scenarios (e.g., abrupt scene cuts or extremely complex non-linear motion) where the TTA mechanism fails or triggers the fallback mechanism too frequently, leading to performance drops?

Generalization to Other Backbones: While the experiments focus heavily on DCVC-UF, can this adaptation framework be seamlessly plugged into other paradigms, such as autoregressive or non-autoregressive codecs with different temporal structures?

7 comments
AuthorsOpenReview ↗

Response to Reviewer gTxu (1/5)

Thank you for taking the time to review our paper and for recognizing the practical value of preserving the pretrained decoder and bitstream syntax, explicitly treating temporal propagation, and introducing safety control. We address every weakness and question below.

[Weakness 1] High computational overhead at the encoder

High Computational Overhead at the Encoder: Test-time adaptation (TTA) inherently introduces additional computational cost during encoding. While the paper highlights performance gains, a thorough discussion and breakdown of the runtime/latency overhead (e.g., training/optimization time per chunk or sequence compared to baseline encoders) are missing or sparse. This is crucial for video compression, where encoding speed often dictates practical adoption.

We agree that the original paper did not provide a sufficiently concrete runtime breakdown. The revised paper now reports controlled measurements using the same A800 GPU, codec configuration, resolution, frame count, and native bitstream implementation. Each sequence contains one untimed I frame followed by 128 timed P frames (16 chunks). Model loading, file I/O, metric computation, and result serialization are excluded. Four synthetic native-codec calls are executed before formal measurement to remove initialization effects. Prefix and suffix chunks are measured separately because the former evaluate multiple causal candidate trajectories, whereas the latter deploy only the selected policy.

For DCVC-UF HT-L, the results are:

DatasetResolutionMethodEnc. s / 128 PPrefix FPSSuffix FPSDec. FPSBD-Rate %
HEVC D416×240416\times240DCVC-UF0.0572223.322253.631358.590.00
Ours116.6360.6135.2091378.49-5.214
HEVC C832×480832\times480DCVC-UF0.1141087.111160.83857.110.00
Ours131.9390.5514.032878.12-5.120
HEVC E1280×7201280\times720DCVC-UF0.193644.59683.39510.410.00
Ours220.1620.3292.469509.15-1.574
HEVC B1920×10801920\times1080DCVC-UF0.381332.34339.53251.230.00
Ours433.9340.1621.623250.93-5.028

The table provides both forms of comparison requested by the reviewer: total encoding seconds for a 128-P-frame sequence and throughput for the two chunk types. Adaptation substantially increases encoder computation, and the prefix is consistently the dominant part: prefix throughput is 7.3 –\text{--} 10.0 times lower than suffix throughput because each prefix chunk evaluates multiple causal closed-loop candidate trajectories, whereas each suffix chunk deploys only the selected policy. The full encoding time also increases with resolution, from 116.636 s at 416 $\times $ 240 to 433.934 s at 1920 ×\times 1080. These costs yield native BD-Rate reductions of 5.214%, 5.120%, 1.574%, and 5.028% on HEVC D, C, E, and B, respectively. In contrast, decoding throughput differs from DCVC-UF by at most 2.5% across the four datasets and shows no systematic slowdown. This is expected because our method transmits a standard native bitstream and introduces no decoder-side operation.

Accordingly, we added a dedicated complexity subsection reporting per-prefix-chunk, per-suffix-chunk, and per-sequence encoding time, together with decoding time and a precise definition of the timing boundary.

[Weakness 2] Hyper-parameter sensitivity

Hyper-parameter Sensitivity: The framework relies on several components, such as the gradient-conditioned learned optimizer and the threshold for the safety gate. It remains somewhat unclear how sensitive the performance is to these hyper-parameters across sequences with varying motion complexities (e.g., high-motion sports vs. low-motion conferencing).

Thank you for raising this point. We added both an overall hyper-parameter ablation in the main paper and a motion-stratified sensitivity analysis in the appendix. Motion complexity is measured by temporal information, i.e., the mean temporal standard deviation of successive luma differences over 129 frames. The 16 HEVC B –\text{--} E sequences are divided by the global median into eight low-motion and eight high-motion sequences.

For the safety tolerance, all four QPs and all 16 sequences are evaluated:

ϵ\epsilonLow motion BD-RateHigh motion BD-RateAll sequencesGate-active ratio
0%-1.622%-6.282%-3.952%62.3%
0.5%-2.040%-6.454%-4.247%77.4%
1%-2.422%-6.545%-4.484%86.6%
3%-1.984%-6.470%-4.227%97.0%
AuthorsOpenReview ↗

Response to Reviewer gTxu (2/5)

The motion-stratified results reveal different sensitivity patterns. Low-motion content is more sensitive to the safety tolerance: its mean BD-Rate changes from -1.622% at ϵ=0\epsilon=0 to -2.422% at 1%1\%, but weakens to -1.984% at 3%3\%, giving a range of 0.800 percentage points. In contrast, high-motion content remains consistently strong across the same sweep (-6.282% to -6.545%, with a range of only 0.263 percentage points). This indicates that conservative gating can unnecessarily reject small but useful modifications on relatively stable low-motion sequences, whereas the larger adaptation gains on high-motion sequences are less sensitive to moderate changes in the acceptance threshold. Nevertheless, an overly permissive gate degrades both groups. We therefore fix ϵ=1%\epsilon=1\% across all datasets, sequences, and QPs, without sequence- or bitrate-specific tuning.

For the learned-refinement budget, we evaluate the complete method with the same safety control and policy-selection protocol. The runtime column is measured on RaceHorses over four QPs using the same 128-P-frame timing boundary:

MMLow motion BD-RateHigh motion BD-RateAll SequencesEnc. s / 128 P
1-0.767%-1.541%-1.154%23.214
5-2.039%-5.707%-3.873%65.501
10-2.422%-6.545%-4.484%121.662
20-2.414%-7.355%-4.884%231.164

The dependence on the refinement budget is also motion-dependent. For low-motion sequences, most of the benefit is obtained by M=5–10M=5 \text{--} 10: the mean improves from -0.767% at M=1M=1 to -2.039% at M=5M=5 and -2.422% at M=10M=10, while increasing to M=20M=20 produces no further gain (-2.414%). High-motion sequences benefit much more strongly from additional refinement, improving from -1.541% at M=1M=1 to -5.707%, -6.545%, and -7.355% at M=5,10,20M=5,10,20, respectively. Thus, the learned optimizer is not disproportionately fragile on complex motion; rather, complex-motion content offers greater optimization headroom and continues to benefit from a larger budget, whereas low-motion content saturates earlier. Encoding time grows approximately linearly with MM, as expected. The separate refinement-budget response discusses the performance –\text{--} complexity choice of the default M=10M=10 in detail. A different MM requires a separately trained learned-optimizer checkpoint because the module imitates an optimization trajectory of the corresponding length.

Overall, the two hyper-parameters affect the motion groups differently: safety tolerance matters more for low-motion sequences, while refinement depth matters more for high-motion sequences. Importantly, every tested setting improves both motion groups, so the framework remains effective across the evaluated motion-complexity range rather than relying on one content regime. The revised main paper reports the aggregate ablations, while the appendix reports this low-/high-motion breakdown, the temporal-information grouping protocol, and the per-sequence results.

[Weakness 3] Clarity on sequence-level evaluation

Clarity on Sequence-Level Evaluation: In Section 3, the description of how candidate update strategies are evaluated on the "observed prefix" needs tighter mathematical or algorithmic formalization. The trade-off between the length of the prefix and the reliability of the estimated propagation effect requires more empirical justification.

We agree that the original description was not sufficiently formal. We now define the observed prefix explicitly as the first KK chunks PK={1,…,K}\mathcal{P}_K=\{1,\ldots,K\}. For each candidate policy pp, the encoder starts from the same initial coding state and executes a separate causal closed-loop native trajectory over PK\mathcal{P}_K. Its prefix score is computed from the resulting native bit count and decoded reconstruction. Only the original-policy prefix is transmitted; all alternative trajectories remain private to the encoder. After selecting p⋆p^\star, the suffix is encoded causally from the coding state produced by the transmitted prefix, preventing future-information leakage and avoiding any re-encoding of already transmitted prefix chunks.

We empirically examine reliability at two levels. First, under the final ϵ=1%\epsilon=1\% configuration, we analyze all 212 sequence –\text{--} QP records (53 sequences and four QPs). A chunk-local winner disagrees with the final K=8K=8 accumulated-RD policy in 54.0% of cases. We then compare four ways of converting the observed chunks into one sequence-level policy; regret is measured relative to the policy minimizing accumulated native RD over the same prefix:

AuthorsOpenReview ↗

Response to Reviewer gTxu (3/5)

Prefix policy estimatorEvidence usedMean regret90th-percentile regret
First chunkEarliest local observation only0.391%1.308%
Last chunkLatest local observation only0.494%0.873%
Majority voteVotes of the eight chunk-local winners0.092%0.326%
Accumulated native RD (ours)Joint native-RD evidence from all eight chunks0.000%0.000%

The high local-policy disagreement shows that the preferred update is not temporally uniform, while the regret table shows that aggregating native closed-loop costs is more reliable than selecting from one local observation or voting over local winners.

Second, we measure the downstream consequence of the prefix estimate by varying K∈{2,4,8,16}K\in\{2,4,8,16\} with M=10M=10 and ϵ=1%\epsilon=1\% on HEVC B –\text{--} E. Because every candidate is evaluated once per observed chunk, prefix-search work grows linearly with KK; the last column reports this cost relative to the adopted K=8K=8 setting:

KKHEVC BHEVC CHEVC DHEVC EAverageRelative prefix-search work
2-3.443%-4.741%-4.871%-1.085%-3.535%0.25×0.25\times
4-4.712%-4.910%-4.846%-1.443%-3.978%0.50×0.50\times
8-5.028%-5.120%-5.214%-1.574%-4.234%1.00×1.00\times
16-4.918%-5.036%-4.994%-1.775%-4.181%2.00×2.00\times

The performance gain exhibits the expected reliability –\text{--} cost saturation. Extending the prefix from K=2K=2 to 4 improves the four-dataset average by 0.443 percentage points, and extending it from 4 to 8 adds another 0.256 points, indicating that additional closed-loop observations improve the policy estimate. Extending it again from 8 to 16 doubles the search work but does not improve the aggregate result: HEVC E improves, while HEVC B, C, and D weaken, and the average changes from -4.234% to -4.181%. Thus, additional observations initially reduce the error of the propagation estimate, but beyond eight chunks the estimate has largely stabilized and content non-stationarity can offset the benefit of a longer prefix. We therefore adopt K=8K=8 as the point at which the measured reliability gain saturates before the search cost doubles.

Accordingly, Section 3.3 now contains a mathematical definition of the observed prefix, independent causal candidate trajectories, the native prefix objective, the committed policy, and the transmitted state. The revised ablation table provides the empirical justification for K=8K=8.

[Question 1] Computational cost and scaling

Computational Cost: Could the authors provide a concrete comparison of the encoding time (e.g., in frames per second or total encoding time per sequence) between the baseline (DCVC-UF) and your method? How does the optimization time scale with the chunk size or resolution?

The direct comparison is provided under Weakness 1 in both FPS and total sequence time. Under the same 128-P-frame and eight-frame-chunk configuration, the full adapted time increases from 116.636 s on HEVC D (416×240416\times240), to 131.939 s on HEVC C (832×480832\times480), 220.162 s on HEVC E (1280×7201280\times720), and 433.934 s on HEVC B (1920×10801920\times1080). Thus, the measured optimization cost increases monotonically with spatial resolution, although it is not expected to follow pixel count exactly because kernel efficiency, padding, and memory behavior also vary by resolution.

DCVC-UF fixes its coding unit to eight frames; changing chunk size would require redesigning and retraining the backbone rather than varying a test-time hyper-parameter. We therefore do not claim an empirical chunk-size scaling law. Instead, we report prefix and suffix cost per native eight-frame chunk and explicitly state this architectural constraint.

[Question 2] Safety-gate threshold

Safety Gate Threshold: How is the threshold for the distortion-constrained safety gate determined? Is it fixed across all datasets and bitrates, or does it require manual tuning per sequence/bitrate point?

The threshold is selected once through the HEVC B –\text{--} E ablation above. The final ϵ=1%\epsilon=1\% is then fixed across all datasets, sequences, and QPs. It does not require manual tuning per sequence or bitrate. The same configuration is used when evaluating UVG and MCL-JCV.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.