Parameter-Efficient Visual Learning with Information-Preserving Transformations for Deep Convolutional Neural Networks

Deep neural networks typically transform representations through successive layers while discarding information that is considered unnecessary for the next transformation. Inspired by predictive coding, we propose the Representation Error Residual (RER), a feedforward representation in which each layer explicitly separates its predictable component from the information that cannot be reconstructed by the layer's own transformation. Given an input representation, a convolutional pathway produces the primary feature representation while a tied reconstruction pathway estimates the input. Their difference forms a local novelty signal which is projected into the feature space and combined with the primary representation, conserving information that is poorly represented by the current prediction for subsequent layers. We develop RERNet, a convolutional architecture built from novelty-preserving blocks, and investigate how the resulting representation evolves with depth. On CIFAR-100, an 8-block RERNet-L achieves 80.95% accuracy, comparable to a ResNet-18 (81.1%) with only 34% as many parameters, and approaches the performance of a ResNet-34 (83.0%) with only 18% as many parameters. We therefore view RER as both a representation mechanism and a means of trading parameter capacity for computation, providing a compact alternative to conventional residual representations.