acceptodds
Under review as a conference paper at ICLR 2027

Optimization Is Part of the Codec: Training Compressible Implicit Neural Video Representations

Abstract

Implicit neural representations (INRs) compress a video by fitting a network to it and then pruning and quantizing the network into a bitstream. This pipeline implicitly assumes that improvements in the dense representation translate into improvements after compression. We show that this assumption can fail: as dense fitting improves reconstruction quality, compressed quality eventually declines. The decline stems from a spectral drift of weight energy toward weakly excited input directions, which lowers a sensitivity factor shared by pruning and quantization. A local spectral analysis of the optimizer indicates that weight decay sets where the weights are driven, while the learning rate sets how far. Without weight decay, a larger learning rate deepens the drift, whereas with weight decay it reverses the drift. Building on this, we propose SpecFit (Spectrally-Guided Dense Fitting), a training strategy that pairs moderate weight decay with a large learning rate to reverse the drift. SpecFit leaves the architecture, pruning criterion, and quantizer unchanged, making it a drop-in strategy for NeRV-based video compression. Relative to the standard training of each backbone, it reduces the PSNR BD-rate by 23–28% on average across three NeRV backbones, and it remains effective under a post-training quantization pipeline. Within the standard three-stage pipeline, SpecFit achieves rate–distortion performance competitive with NVRC without NVRC's learned entropy-modeling framework. This simpler decoder achieves a 65.8 speedup in model-decoding time and increases full decoding throughput from 3.08 to 19.75 FPS. Our findings reveal that optimization in video INRs is not merely a procedure for improving dense reconstruction, but a key mechanism that determines how compressible the learned representation ultimately becomes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Compression、implicit neural representation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TkX3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how dense-stage optimization affects the compressibility of implicit neural representations for video compression. The authors identify a phenomenon termed "spectral drift," where prolonged dense fitting shifts weight energy toward weakly excited input directions, making the model more sensitive to pruning and quantization. Based on a spectral analysis of optimization dynamics, they propose SpecFit, which combines decoupled weight decay with a larger learning rate during dense fitting.

Critical strengths
  1. The paper provides a useful perspective on why better dense reconstruction quality does not necessarily translate into better compressed performance. The proposed spectral analysis offers an intuitive explanation connecting weight energy distribution with sensitivity to pruning and quantization.

  2. SpecFit requires only modifications to dense-stage optimization without introducing additional architectural complexity. The reported BD-rate improvements across multiple INR backbones are encouraging and suggest that dense-stage optimization is an important but relatively underexplored factor in INR-based video compression.

Critical weaknesses

The theoretical analysis relies on simplifying assumptions that are not sufficiently validated. The analysis in Section 3.2 approximates AdamW using a scalar preconditioner and assumes stationary input statistics. However, actual AdamW uses coordinate-wise adaptive preconditioning, and intermediate feature distributions evolve throughout training. These assumptions limit how directly the proposed spectral dynamics explain the observed behavior. Moreover, the conclusion that weakly excited modes accumulate disproportionately large weight energy depends on the relationship between the target cross-covariance and the input eigenspectrum, which is not explicitly characterized.

Questions for authors
  1. How well does the proposed spectral analysis explain the behavior of actual AdamW with coordinate-wise preconditioning and non-stationary intermediate features? In particular, can the authors clarify under what conditions the target cross-covariance causes weakly excited input modes to accumulate disproportionately large weight energy?

  2. Could the authors provide an unregularized NVRC-net baseline under the same compression pipeline and fine-tuning schedule as NVRC-net+SpecFit?

Reviewer_oiU3OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the inconsistency between dense fitting and final compression quality in implicit neural video representation compression. The authors point out that as the reconstruction quality of dense fitting continues to improve, the quality after pruning and quantization may instead decline. The reason is that the weight energy gradually shifts toward directions with weaker input excitation, which the authors call spectral drift. The authors further analyze the influence of the optimizer on the spectral position of weights and propose a SpecFit training strategy that combines moderate weight decay with a larger learning rate. This method does not change the network structure, the pruning criterion, or the quantizer. Experiments cover multiple datasets and multiple NeRV backbones and report clear bitrate savings.

Critical strengths

The problem perspective of this paper is novel. It points out that dense fitting not only affects reconstruction quality but also changes the compressibility of the model. The method is simple and easy to integrate. It only adjusts the optimization hyperparameters in the dense fitting stage, does not increase inference overhead, and does not change the existing compression pipeline. The experimental scope is broad, covering multiple datasets, multiple NeRV backbones, different model scales, and different pruning rates and quantization bit widths. The main results are competitive, achieving a clear BD rate gain relative to VTM on HEVC B class data.

Critical weaknesses
  1. The theoretical analysis is built on a simplified model with linear layers, a fixed input distribution, and a fixed target, and there is a gap between it and the dynamics of real deep networks and AdamW.

  2. The paper claims that the sensitivity factors are basically fixed after dense fitting ends, but pruning fine-tuning and quantization-aware training both continue to change the weights.

Questions for authors

Answer the questions raised by the critical weakness.

Reviewer_teQEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SpecFit, a training method targeting INR-based video compression. It suggests that the compression pipeline (fitting, pruning and quantization) could cause bad compression performance as the model reconstruction quality may not be recovered if the fitting stage simply focuses on minimizing the distortion. The proposed method applies optimizer refinement (AdamW with moderate weight decay and a large learning rate) for regularizing the training in the fitting stage. The trained INR after compression shows strong improvement, where the BD-rate is reduced by more than 20% for three NeRV-based backbones, measured in PSNR. It reports performance competitive with NVRC, without using learned entropy models. The paper also provides different ablation studies for validating the effectiveness of the method.

Critical strengths
  • The paper focuses on the training process improvement for INR-based video compression, which is effectively the encoding process, where most existing works simply use a similar recipe. It shows that by introducing optimization improvement, existing models have obtained significant compression performance gain, measured in BD-rate (~20%).
  • The performance is very promising considering that no learned entropy model is used. The decoding is very fast too.
Critical weaknesses
  • Novelty is limited. The proposed method utilizes AdamW with moderate weight decay (λ = 0.05) and a large learning rate for improving the robustness for pruning and quantization.

    • A similar method (adam + l2 + scheduled weight decay) has already been used by NVRC, although with a different purpose.
    • While the authors didn't claim that the proposed method is better than learned compression (entropy modeling), learned compression optimizes rate-distortion directly - I think the authors should provide evidence that the proposed method is better, at least in some cases.
    • Table 12 should include the wider range of Adam + l2. In fact, AdamW with 0 weight decay is exactly Adam, and it has much higher Full PSNR than l2 = 2e-6 (the lowest for Adam + l2), so the Adam + l2 range is clearly not enough. The results are only reported on a single video.
  • The experimental settings. Different experiments utilize different baseline methods, where many of the details have not been provided.

    • BD-rate comparisons with other codecs are only reported on HEVC Class B and MCL-JCV (Table 1/13). Commonly used datasets like UVG are only used for comparing against each backbone's own baseline (Table 2) and for ablations, with no comparison against VTM, NVRC or DCVC. Some ablation studies use a single video.
    • NVRC-net is never specified. Figure 3c shows that NVRC-net has the same complexity as HiNeRV, but the text claims it uses the same representation architecture as NVRC (an adjusted version of HiNeRV), without learned entropy models. Table 3 also shows that NVRC-net's frame reconstruction is faster than NVRC, which is not likely if they hold the same representation. Is this actually the original HiNeRV?
    • Following the last point: in Table 2, HiNeRV + SpecFit has a 34% BD-rate gain on HEVC Class B, but if NVRC-net+SpecFit is based on HiNeRV, Figure 3c shows a ~50% BD-rate gain over HiNeRV (assuming Figure 3c is on HEVC Class B, which is not stated).
    • The detailed compression settings, like the quantization level and the pruning ratio used for the main experiments, are not provided.
    • Figure 3a only uses MSNeRV+SpecFit (without the MSNeRV baseline curve), while 3b/c use NVRC-net+SpecFit. The abstract combines the RD claim (partly based on MSNeRV+SpecFit) with the speed claims (based on NVRC-net+SpecFit). Figure 10 uses HiNeRV but not NVRC.
    • The baseline INRs have different scales, which have never been provided, e.g., what size of NVRC is used for the encoding/decoding time benchmark.
    • NeuroQuant is a compression method which can also be applied to different models, and should not be considered as a backbone. Although the text states that HiNeRV is used as its backbone, Table 2/14 and Figure 9 list NeuroQuant as a backbone, and its pipeline details (e.g., whether pruning is kept, the bit allocations, and how the rate points are produced) are not provided.
    • VTM is configured with a GOP size of 16 and an intra period of 32. GOP 16 is generally short for the random access configuration, which may weaken the VTM anchor used for the BD-rate results.
Questions for authors
  • Why does NVRC have no speed-up with the x3 GPU setting? How exactly is the evaluation being done here? Does each GPU do partial entropy decoding and frame decoding?
  • Could you please explain where the speedup over NVRC comes from, for both encoding (Figure 3b) and decoding (Table 3, model decoding and frame reconstruction)?
  • What are the criteria for choosing baselines in different experiments?
  • What are the test conditions for the DCVC models? In Table 13, DCVC-DC outperforms DCVC-FM and DCVC-RT on HEVC Class B significantly in PSNR.
  • Adding a related-work section will be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.