acceptodds
Withdrawn from ICLR 2027

JRCC: Joint Residue and Conditional Coding for Overfitted Neural Video Compression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Residue coding has long been one of the most widely used paradigms for exploiting inter-frame correlation in both conventional and learned video coding. More recently, conditional coding has emerged as a more efficient paradigm for exploiting inter-frame correlation in offline-learned video coding. This raises a natural question: can conditional coding also be effectively used in overfitted learned video coding? The key difference between overfitted and offline-learned video coding is that, in the former, the neural network model parameters must be transmitted. Therefore, using a large model to exploit conditional dependencies between neighboring frames is less suitable for the overfitted setting. In this paper, we find that directly applying conditional coding degrades rate–distortion performance under this overfitted setting, while a moderate increase in conditioning capacity can partially alleviate the degradation, but further scaling alone does not consistently resolve it. We therefore propose a joint residue-conditional coding framework that retains explicit motion-compensated prediction while using motion-aligned temporal features to condition residue synthesis. Experiments on multiple benchmark datasets under random-access (RA) and low-delay (LD) configurations show that the proposed framework consistently improves Cool-Chic 4.1, achieving average BD-rate reductions of 9.7% and 27.8%, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video compression, Compact neural codec, Cool-Chic

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_93hJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes JRCC (Joint Residue and Conditional Coding) for overfitted neural video compression. The central idea is to retain the explicit prediction-residue structure of residue coding while injecting motion-aligned temporal features into residue synthesis. The authors argue that directly applying conditional coding to overfitted codecs is inefficient because the decoder parameters themselves must be transmitted, making increased conditional-model capacity costly. JRCC therefore reuses already decoded motion information to align reference-frame features and conditions the residue synthesis network on these temporal features. Experiments under both random-access and low-delay configurations report average BD-rate reductions of 9.7% and 27.8%, respectively, over Cool-Chic 4.1.

Critical strengths
  1. In the overfitted setting, frame-specific decoder parameters contribute directly to bitrate, so simply increasing conditional decoder capacity is indeed undesirable.

  2. JRCC reuses decoded motion for feature alignment instead of estimating a second motion field, and the temporal-conditioning path is intentionally lightweight.

Critical weaknesses
  1. JRCC mainly augments the existing Cool-Chic 4.1 residue-coding framework with a motion-aligned temporal feature-conditioning branch and feeds the resulting temporal features into residue synthesis. While this design is intuitive and effective, the individual operations, including feature extraction, motion warping, feature fusion, gated refinement, and concatenation-based conditioning, are relatively standard. The paper would benefit from stronger comparisons against simpler alternatives such as warped-reference concatenation, feature modulation, additive conditioning, or lightweight attention-based conditioning to better establish that the proposed “joint residue and conditional coding” formulation provides a distinct methodological contribution.

  2. JRCC includes its frame-specific model parameters in the transmitted bitrate, whereas offline-trained neural codecs such as DCVC-B and DCVC-FM exclude pretrained model parameters from bitrate. Although this difference reflects the underlying coding paradigms, it makes direct comparison less straightforward. In addition, Table 1 shows that JRCC still underperforms VTM-24.0 and the learned conditional codecs in several settings. The most convincing evidence is therefore that JRCC improves the directly comparable Cool-Chic 4.1 overfitted baseline, rather than that it is broadly superior to modern video codecs. More comparisons with recent overfitted neural codecs would strengthen the empirical positioning.

  3. JRCC increases decoding complexity from 831.3 to 1231.1 MACs/pixel, corresponding to roughly a 48% increase, despite being repeatedly described as lightweight and compact. The paper does not report actual decoding latency, encoding/optimization time, memory consumption, or the proportion of bitrate contributed by the additional conditioning parameters. Since compactness and transmitted model cost are central motivations of the work, a more complete rate–distortion–complexity analysis is necessary.

  4. Evaluation is restricted to the first 33 frames of HEVC Classes A, B, E and UVG, following prior Cool-Chic settings. This leaves several important questions unanswered, including long-sequence error propagation, robustness to scene changes and high-motion content, and behavior under longer temporal dependencies. The ablation study also does not isolate several important design choices, such as motion alignment itself, reuse of the pixel-domain blending map for feature fusion, conditioning of the residue versus the prediction-weight map, or alternative feature-fusion strategies. In addition, the much larger gain under LD than RA deserves deeper analysis rather than only being reported numerically.

  5. The paper argues that conditional coding is less suitable for overfitted compression, but the evidence only demonstrates this for the specific conditional architecture and capacity-scaling strategy evaluated here. More parameter-efficient conditional designs may behave differently, so the broader claim should be weakened. In addition, the rate–distortion objective in Eq. (8) explicitly includes motion and residue rates but not a model-rate term, even though transmitted frame-specific network parameters are central to the paper’s motivation and are counted in the final bitrate.

Questions for authors

How do the authors distinguish the proposed JRCC from a conventional residue-coding framework augmented with an additional temporal feature-conditioning branch?

Since JRCC still underperforms some offline-trained codecs such as VTM-24.0, DCVC-B, and DCVC-FM in several settings, how should readers interpret the intended scope of the method?

JRCC increases decoding complexity from 831.3 to 1231.1 MACs/pixel. Could the authors report practical encoding time, decoding latency, memory consumption, and the additional model-bit overhead introduced by the temporal-conditioning modules?

Reviewer_9v3dOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose JRCC for overfitted NVC. The authors investigate whether conditional coding, which has been successful in offline-trained neural video codecs, can be effectively applied to overfitted codecs where decoder parameters must also be transmitted. Their experiments suggest that directly adopting conditional coding leads to inferior rate-distortion performance under limited model capacity.

To address this issue, JRCC retains explicit motion-compensated prediction while incorporating motion-aligned temporal features into residue synthesis. The proposed framework introduces lightweight feature extraction and gated context refinement modules, reusing decoded motion information for temporal alignment.

Critical strengths
  1. The obsevation is meaningful. The observation that conditional coding performs worse in the overfitted setting is interesting, and the capacity experiments provide useful evidence.
  2. The RD performance is great across various coding configuration. JRCC achieves consistent RD improvements over Cool-Chic 4.1 under both RA and LD configurations.
Critical weaknesses
  1. No decoding latency or runtime results are reported, which is a significant omission for a overfitted video coding paper. As a practical coding problem, I find it difficult to assess the overall value of JRCC without runtime measurements, especially considering the reported 48% increase in MACs.

  2. The proposed method mainly combines existing residue coding and conditional coding techniques, while the additional feature extraction and refinement modules are relatively conventional. I find the technical novelty somewhat limited, particularly given the substantial increase in decoding complexity.

Questions for authors

See weaknesses 1. Please provide detailed experimental results about the runtime and platform.

Reviewer_1d5iOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies inter-frame reconstruction in overfitted neural video coding, where the frame-specific decoder parameters are transmitted in the bitstream. Building on Cool-Chic 4.1, it reports that replacing residue coding with conditional coding degrades RD performance, and that enlarging the conditioning branch helps only partially because the extra model bits offset the latent-rate savings. The proposed JRCC keeps the prediction–residue reconstruction, but feeds the residue synthesis network with lightweight temporal features extracted from the decoded references, aligned with the already-decoded motion, β-fused, and refined by a gated module. On HEVC Classes A/B/E and UVG (first 33 frames), JRCC reduces BD-rate over Cool-Chic 4.1 by 9.7% (RA) and 27.8% (LD), while decoding complexity rises from 831 to 1231 MACs/pixel.

Critical strengths
  • A relevant question with an informative capacity analysis. The paper examines how conditional coding behaves when decoder parameters count toward the bitrate, a question specific to overfitted codecs that has received little attention. Table 3, which separates latent-only from total BD-rate as the conditioning capacity grows, makes this trade-off concrete and is useful beyond this specific method.
  • Simple, lightweight design with consistent gains. The conditioning path reuses the decoded motion field and blending map, so no extra motion information is transmitted and only a small number of parameters is added. JRCC improves over Cool-Chic 4.1 on all four datasets under both RA and LD and on every HEVC-B sequence.
Critical weaknesses
  • Limited novelty. Keeping the prediction P and conditioning the residual on temporal features is conditional residual coding, which already exists (e.g., Brand et al., 2024, which the paper cites, and MaskCRT). The paper neither discusses nor compares with these works. So the contribution reads as applying a known idea to Cool-Chic, not as a new framework.
  • The main claim is not well supported. The paper says conditional coding fails here because model parameters are costly to transmit. But Table 3 shows that even when model bits are ignored, the best conditional variant reaches only −1.05% (vs. −10.20% for JRCC), so the real problem looks like a weak baseline that drops P entirely. Also, the variant without FE, which just uses the aligned reference pixels (essentially P), already gets −8.01% (Table 4), so most of the gain may simply come from letting the synthesis network see P.
  • Weak absolute performance and narrow evaluation. JRCC is still clearly behind HM (in LD), VTM, and DCVC-B/FM, and it is compared with only one overfitted codec. All tests use only 33 frames, with no variance across runs, no encoding time, and ablations on a single setting (HEVC-B, RA).
Questions for authors

Questions that could change my assessment

  • How does JRCC differ, conceptually and empirically, from conditional residual coding (Brand et al., 2024; MaskCRT) and from the temporal context modeling in CNVC? Comparisons with CNVC and C3 would clarify where JRCC stands among overfitted video codecs.
  • Please report (a) Cool-Chic 4.1 with P concatenated to the synthesis input (no FE/CR), (b) a conditional-coding baseline conditioned on both C_f and P, and (c) Cool-Chic 4.1 with a wider synthesis network matched to ~1231 MACs/pixel. Please also give JRCC’s parameter count, model bpp, and latent-only BD-rate in the format of Table 3.
  • The LD gain (27.8%) is much larger than the RA gain, and in Fig. 9 JRCC halves the rate while gaining 1.75 dB (0.048 bpp / 35.14 dB vs. 0.099 bpp / 33.39 dB). Was Cool-Chic 4.1 run with its official LD settings (P-frame decoder configuration, iterations, rate allocation)? Per-frame PSNR and rate curves would help rule out error accumulation in the baseline.
  • What are the per-frame encoding times of JRCC and Cool-Chic 4.1 (GPU, number of iterations, parallelism), and how do they compare under the same iteration or wall-clock budget? Since FE and feature warping are recomputed at every iteration, the gains should also be assessed at equal encoding cost.
  • Can you provide results on longer sequences (e.g., 96 frames or full length), at least one LD ablation, and the variance over several random seeds? Several ablation differences (0.7–2 percentage points in Tables 6–7) may be within optimization noise.

Clarification requests

  • Table 5: please give the value of X and the FE hidden width, and clarify whether the MACs correspond to the worst-case B-frame configuration or the 33-frame average.
  • Please fully specify the conditional-coding variant (whether it outputs α, its channel widths, and whether it uses the same FE/CR).
  • Total-rate BD-rate includes intra frames that are identical across methods, so an inter-only BD-rate would be informative.
  • Figs. 8–9 compare reconstructions at mismatched rates; rate-matched examples would be more convincing. Minor: “residual” is the standard term, both panels of Fig. 4 are titled “HEVC Class B” (please label RA/LD), and the exact Cool-Chic 4.1 version and code availability should be stated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.