- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes encoder-side test-time latent adaptation for a frozen DCVC-UF codec. Gradient-conditioned update modules refine the primary and hyper latents, with a distortion-constrained safety gate and propagation-aware sequence-level policy selection based on native prefix rate–distortion evaluation. Only the original-policy prefix is transmitted; the committed policy is applied to suffix chunks. The method preserves the decoder and bitstream syntax and reports an average BD-Rate reduction of approximately 3.2% across six benchmark groups.
Critical strengths
- Improving compression through encoder-side latent refinement while keeping all codec parameters and decoding-side operations unchanged is practically appealing.
- The experiments report both aggregate gains and per-sequence failure cases, helping readers assess the method's applicability.
Critical weaknesses
-
Missing computational cost. Please report MACs/FLOPs, encoding time, and peak GPU memory for the original DCVC-UF and the full method at matched resolutions and configurations. Separate the costs of learned latent refinement, the safety gate, and closed-loop candidate trajectories. Ten update steps require repeated differentiable forward evaluations and gradient computations; their cost cannot be represented simply as one forward pass plus ten backward passes.
-
Insufficient ablation of the refinement budget. Why is selected? Please compare different numbers of update steps, such as 1, 5, 10, and 20, reporting native BD-Rate and encoding time on each dataset. Clarify whether changing requires retraining the learned optimizer. Comparing ten learned updates with twenty plain Adam steps does not isolate the effect of the refinement budget; an Adam baseline with the same safety control and policy selection under a matched budget would be more informative.
-
Missing comparison with DCVC-DT. DCVC-DT also performs encoder-side latent refinement with frozen codec parameters and introduces dynamic RD adjustment, but it is neither cited nor evaluated. Please provide a comparison on the same backbone with matched evaluation protocols and encoding-time budgets. If its frame-level mechanism cannot be directly adapted to chunk-based coding, explain the limitations and provide an appropriate controlled baseline. Comparing with DCVC-DC alone does not evaluate DCVC-DT's adaptation mechanism.
-
Unclear quantization implementation. Please specify the actual optimization variables and the quantization forward pass and gradient estimator used at each refinement step. If a continuous relaxation is used, explain how the final discrete symbols are obtained for native entropy coding. Does safety-gate verification use the same quantization forward pass as native coding? This detail is important for interpreting the relationship between the differentiable reconstruction and native coding outcomes.
Questions for authors
See Weaknesses
4 comments
Response to Reviewer eLUt (2/4)
[Weakness 2] Insufficient ablation of the refinement budget
- Insufficient ablation of the refinement budget. Why is (M=10) selected? Please compare different numbers of update steps, such as 1, 5, 10, and 20, reporting native BD-Rate and encoding time on each dataset. Clarify whether changing (M) requires retraining the learned optimizer. Comparing ten learned updates with twenty plain Adam steps does not isolate the effect of the refinement budget; an Adam baseline with the same safety control and policy selection under a matched budget would be more informative.
We agree that comparing ten learned updates only with twenty plain Adam steps does not isolate the refinement budget. We therefore make two controlled comparisons. First, we vary the learned budget over while keeping safety control and policy selection fixed. A separate learned optimizer is trained for each , so changing in this controlled ablation does require retraining because the module learns to imitate an optimizer trajectory of the corresponding length. Second, we evaluate Adam with the same safety control and policy selection at 10 and 20 steps. Adam-10 provides the near-matched-budget control for Learned-10, thereby isolating the update rule from both the refinement budget and the deployment mechanisms.
| Rule | Steps | HEVC B | HEVC C | HEVC D | HEVC E | Average | Enc. s/128P |
|---|---|---|---|---|---|---|---|
| Learned | 1 | -1.276% | -1.445% | -1.233% | -0.453% | -1.101% | 23.214 |
| Learned | 5 | -4.232% | -4.690% | -4.455% | -1.312% | -3.672% | 65.501 |
| Learned | 10 | -5.028% | -5.120% | -5.214% | -1.574% | -4.234% | 121.662 |
| Learned | 20 | -5.586% | -5.668% | -5.532% | -1.602% | -4.597% | 231.164 |
| Adam | 10 (near-matched) | -4.339% | -4.481% | -4.203% | -1.055% | -3.520% | 110.776 |
| Adam | 20 | -5.288% | -5.543% | -5.508% | -1.603% | -4.486% | 224.695 |
The learned-budget rows report native BD-Rate on every requested dataset under the final safety tolerance and an identical policy-selection protocol. Runtime increases from 23.214, 65.501, and 121.662 to 231.164 s/128 P frames for , respectively. Although achieves the strongest average RD result, its 0.363-percentage-point improvement over nearly doubles the encoding time; this diminishing return motivates the default .
For the update-rule control, Adam uses exactly the same safety control and policy selection as the learned method. Adam-10 takes 110.776 s/128 P frames, close to the 121.662 s of Learned-10, while its average BD-Rate is -3.520%, compared with -4.234% for Learned-10. Under these comparable encoding-time budgets, the learned optimizer improves the average result by 0.714 percentage points. The same trend remains at 20 steps: Learned-20 achieves -4.597% at 231.164 s, whereas Adam-20 achieves -4.486% at 224.695 s.
We expanded the ablation table and explicitly state the checkpoint-training requirement for each learned budget.
Response to Reviewer eLUt (3/4)
[Weakness 3] Missing comparison with DCVC-DT
- Missing comparison with DCVC-DT. DCVC-DT also performs encoder-side latent refinement with frozen codec parameters and introduces dynamic RD adjustment, but it is neither cited nor evaluated. Please provide a comparison on the same backbone with matched evaluation protocols and encoding-time budgets. If its frame-level mechanism cannot be directly adapted to chunk-based coding, explain the limitations and provide an appropriate controlled baseline. Comparing with DCVC-DC alone does not evaluate DCVC-DT's adaptation mechanism.
Thank you for pointing out this closely related work. We agree that comparison with DCVC-DC alone does not evaluate DCVC-DT's adaptation mechanism. We now cite DCVC-DT and compare it on the same DCVC-DC backbone. All methods use native YUV420 bitstreams, 96 frames, intra period 32, four rate points, and the same weighted YUV PSNR. We evaluate both the original frame-wise mechanism and a controlled joint-miniGOP variant under matched encoding-time budgets.
On HEVC D:
| Method | Temporal unit | Iterations | Native BD-Rate vs. DCVC-DC | Enc. s/96 frames | Relative time |
|---|---|---|---|---|---|
| DCVC-DC | Frame | -- | 0.000% | 3.252 | |
| DCVC-DT, frame-wise | Frame | 10 | -0.227% | 118.008 | |
| DCVC-DT, frame-wise | Frame | 40 | -0.781% | 452.082 | |
| DCVC-DT, joint mini-GOP | mini-GOP-4 | 43 | -0.672% | 455.093 | |
| Ours | mini-GOP-4 | 10 learned updates | -1.421% | 450.312 |
On HEVC C:
| Method | Temporal unit | Iterations | Native BD-Rate vs. DCVC-DC | Enc. s/96 frames | Relative time |
|---|---|---|---|---|---|
| DCVC-DC | Frame | -- | 0.000% | 7.861 | |
| DCVC-DT, frame-wise | Frame | 10 | -0.207% | 203.959 | |
| DCVC-DT, frame-wise | Frame | 40 | -0.700% | 867.436 | |
| DCVC-DT, joint mini-GOP | mini-GOP-4 | 47 | -0.779% | 887.257 | |
| Ours | mini-GOP-4 | 10 learned updates | -1.739% | 869.910 |
Across the two datasets, we compare the original frame-wise DCVC-DT mechanism, its joint mini-GOP counterpart, and our learned adaptation on the same DCVC-DC backbone and evaluation protocol. On HEVC D, frame-wise DCVC-DT at is time-matched to ours within 0.4%, while joint mini-GOP DCVC-DT at is within 1.1%. Our method achieves the largest RD improvement, reducing BD-Rate by 1.421%, compared with 0.781% for the time-matched frame-wise baseline and 0.672% for the joint mini-GOP baseline. On HEVC C, increasing frame-wise DCVC-DT from 10 to 40 iterations improves its BD-Rate from -0.207% to -0.700%. The 867.436-second frame-wise run differs from our 869.910 seconds by only 0.28%. Single-point calibration on RaceHorses at rate index 1 selects for joint mini-GOP DCVC-DT: its verified 887.257-second encoding time differs from ours by only 1.99%. Under these matched budgets, joint mini-GOP DCVC-DT achieves -0.779%, whereas our method reaches -1.739%, an additional 0.960 percentage-point reduction.
DCVC-DT's original frame-wise mechanism cannot be transferred verbatim to DCVC-UF because DCVC-UF jointly encodes an eight-frame chunk and shares a structured latent update across that unit. Treating each frame independently would break the backbone's native chunk structure. The joint-miniGOP experiment therefore serves as a controlled baseline for structured temporal adaptation rather than claiming an exact architectural equivalence.
We added DCVC-DT to Related Work and include same-backbone, same-protocol, and matched-runtime comparisons on both HEVC C and D.
Response to Reviewer eLUt (4/4)
[Weakness 4] Unclear quantization implementation
- Unclear quantization implementation. Please specify the actual optimization variables and the quantization forward pass and gradient estimator used at each refinement step. If a continuous relaxation is used, explain how the final discrete symbols are obtained for native entropy coding. Does safety-gate verification use the same quantization forward pass as native coding? This detail is important for interpreting the relationship between the differentiable reconstruction and native coding outcomes.
We apologize that these implementation details were unclear. The optimization variables are the continuous primary and hyper latents . During differentiable refinement, quantization is represented by hard rounding in the forward pass with a straight-through gradient estimator in the backward pass. The learned optimizer consumes the current latents and their gradients and predicts coupled updates for both components.
After refinement, the adapted continuous latents are passed to the codec's unmodified native compress path. This path performs the codec's original discrete quantization and entropy coding, producing the actual transmitted symbols and bitstream. The decoder reconstructs exclusively from that native bitstream.
The safety gate does not reuse the differentiable STE forward pass for its final decision. Each candidate is recompressed and decoded through the codec's native quantization and entropy-coding path, and acceptance uses the resulting native reconstruction and bitstream behavior. Therefore, the differentiable quantization path generates the update direction, while the native discrete path determines both safety acceptance and deployment. This explicitly separates differentiable reconstruction quality from the native coding outcome highlighted by the reviewer.
We added these statements immediately after the refinement objective in Section 3.2 and clarified the distinction between differentiable generation and native verification in Section 3.3.
[Questions for Authors]
See Weaknesses
All questions raised through the four weaknesses are addressed individually above.
We sincerely appreciate the reviewer’s detailed suggestions. The added cost accounting, refinement-budget controls, DCVC-DT comparison, and quantization description have materially improved the completeness and reproducibility of the paper.
Response to Reviewer eLUt (1/4)
Thank you for the detailed and technically precise review. We appreciate the recognition of frozen-decoder latent refinement and of our aggregate and per-sequence evaluation. We address each requested clarification and controlled comparison below.
[Weakness 1] Missing computational cost
We agree that the original submission did not report enough information to interpret the computational cost. Encoding time and peak memory are now measured under matched codec, resolution, frame count, and native bitstream configurations. The verified peak-memory results for HT-L are:
To address the reviewer's important warning about the ten update steps, we count the operations on the actually executed path rather than approximating the method as “one forward plus ten backward passes.” Native coding trajectories are measured using Nsight Compute tensor-path counters, while differentiable refinement and safety verification are measured using PyTorch's operator-level FLOP counter; ADD/MUL count as one FLOP and FMA as two FLOPs. For 128 P frames (sixteen eight-frame chunks), we reconstruct the total as 65Crefine+Cdist+8Cgate,cached+64Cgate+8Cbase-traj+72Ccand-traj, thereby including every repeated differentiable evaluation, gradient computation, native verification, and closed-loop trajectory that is actually executed.
The final matched-resolution results are:
The full-method FLOPs separate as follows:
The totals scale monotonically with resolution and are 70.83–85.33× those of the frozen codec. Learned refinement is the dominant component (70.8–80.1% of full FLOPs), followed by closed-loop trajectories (10.0–12.0%) and safety verification (8.4–19.2%). The component sums exactly recover the reported full-method totals.
We also decompose encoder time on one representative sequence per dataset, averaged over the four QPs. Each entry reports seconds per 128 P frames, followed by its fraction of the complete encoder time:
The transmitted native prefix path takes a further 0.151 s on HEVC D and 0.202 s on HEVC C; all remaining orchestration accounts for only 0.115 s and 0.118 s, respectively. Thus, more than 92% of the full encoding time is spent in the actually executed learned-refinement pipeline, including latent preparation and all ten differentiable update steps. Safety verification contributes approximately 5 – 6%, whereas the native closed-loop candidate transitions and suffix bitstream generation are comparatively small. This explicit measurement confirms the reviewer's observation that the cost cannot be represented by a schematic “one forward plus ten backward passes.” We keep the learned-refinement, safety-verification, closed-loop-search, and native-encoding costs separate rather than folding them into a single opaque total.