Optimization Is Part of the Codec: Training Compressible Implicit Neural Video Representations

Implicit neural representations (INRs) compress a video by fitting a network to it and then pruning and quantizing the network into a bitstream. This pipeline implicitly assumes that improvements in the dense representation translate into improvements after compression. We show that this assumption can fail: as dense fitting improves reconstruction quality, compressed quality eventually declines. The decline stems from a spectral drift of weight energy toward weakly excited input directions, which lowers a sensitivity factor shared by pruning and quantization. A local spectral analysis of the optimizer indicates that weight decay sets where the weights are driven, while the learning rate sets how far. Without weight decay, a larger learning rate deepens the drift, whereas with weight decay it reverses the drift. Building on this, we propose SpecFit (Spectrally-Guided Dense Fitting), a training strategy that pairs moderate weight decay with a large learning rate to reverse the drift. SpecFit leaves the architecture, pruning criterion, and quantizer unchanged, making it a drop-in strategy for NeRV-based video compression. Relative to the standard training of each backbone, it reduces the PSNR BD-rate by 23–28% on average across three NeRV backbones, and it remains effective under a post-training quantization pipeline. Within the standard three-stage pipeline, SpecFit achieves rate–distortion performance competitive with NVRC without NVRC's learned entropy-modeling framework. This simpler decoder achieves a 65.8$\times$ speedup in model-decoding time and increases full decoding throughput from 3.08 to 19.75 FPS. Our findings reveal that optimization in video INRs is not merely a procedure for improving dense reconstruction, but a key mechanism that determines how compressible the learned representation ultimately becomes.