acceptodds
Under review as a conference paper at ICLR 2027

Parameter-Efficient Visual Learning with Information-Preserving Transformations for Deep Convolutional Neural Networks

Abstract

Deep neural networks typically transform representations through successive layers while discarding information that is considered unnecessary for the next transformation. Inspired by predictive coding, we propose the Representation Error Residual (RER), a feedforward representation in which each layer explicitly separates its predictable component from the information that cannot be reconstructed by the layer's own transformation. Given an input representation, a convolutional pathway produces the primary feature representation while a tied reconstruction pathway estimates the input. Their difference forms a local novelty signal which is projected into the feature space and combined with the primary representation, conserving information that is poorly represented by the current prediction for subsequent layers. We develop RERNet, a convolutional architecture built from novelty-preserving blocks, and investigate how the resulting representation evolves with depth. On CIFAR-100, an 8-block RERNet-L achieves 80.95% accuracy, comparable to a ResNet-18 (81.1%) with only 34% as many parameters, and approaches the performance of a ResNet-34 (83.0%) with only 18% as many parameters. We therefore view RER as both a representation mechanism and a means of trading parameter capacity for computation, providing a compact alternative to conventional residual representations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Deep Learning, Computer Vision

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_DntGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Representation Error Residual (RER) blocks for parameter-efficient convolutional networks. A convolutional representation is mapped back to the input space through a weight-tied transposed convolution; the resulting reconstruction residual is projected through a separate 1 × 1 convolution and batch normalization and added to the forward representation. The evaluated architecture also includes an outer residual shortcut and a depth-dependent novelty-strength schedule. On CIFAR-100, the paper reports 80.95% accuracy with 3.78M parameters for RERNet-L8, compared with 81.10% and 11.23M parameters for ResNet-18. Experiments also cover CIFAR-10, Oxford-IIIT Pet segmentation, qualitative novelty maps, and inference costs on one GPU. The intended contribution is a compact architecture that trades additional computation for a smaller parameter footprint.

Critical strengths
  1. A simple and testable architectural idea. Tying the reconstruction operator to the forward convolution avoids the parameter cost of a separate decoder. The novelty projection can be incorporated into a conventional residual network, making the proposal concrete and easy to investigate. The paper also usefully distinguishes its feedforward construction from iterative predictive-coding inference (§2.2).

  2. A promising measured parameter–accuracy trade-off. Table 1 reports nearly the same CIFAR-10/100 accuracy as ResNet-18 using approximately one third of its parameters. Reporting MACs and measured latency, alongside parameter count, makes the computational trade-off visible rather than implying that fewer parameters automatically mean fewer operations. The segmentation experiment provides an additional, albeit less competitive, test of the learned backbone.

Critical weaknesses
  1. The experiments do not isolate the contribution of the reconstruction residual. Table 1 varies network width and depth, but the provided materials do not report controlled comparisons that disable the novelty contribution within otherwise unchanged RER blocks, replace reconstruction residuals with projected inputs, or use matched-capacity residual blocks. Consequently, the benefit could arise from architectural differences or the additional transformation rather than the proposed residual mechanism. This matters because the information-preservation interpretation is the central explanation for the results. Weight tying alone does not establish that the transpose reconstructs precisely the information captured by the forward mapping; normalization and nonlinearity further complicate this interpretation. Appendix A.2 explicitly discusses leaving normalization uncorrected in the reconstruction, but supplies no quantitative validation. Figure 3 shows structured maps, which is suggestive, but cannot establish complementary information or an accuracy contribution. Controlled ablations would directly address this concern; a formal preservation theorem is not necessary.

  2. The experiments do not establish an advantage over residual baselines at matched parameter budgets. Table 1 reports substantially fewer parameters and similar accuracy for RERNet-L8 relative to standard ResNet-18, but does not show whether a narrower residual model at approximately 3.8M parameters would perform similarly. The evaluation protocols also need to be aligned with the reported results: the supplied train_cifar.py evaluates the official test split after every epoch and saves the checkpoint with the highest test accuracy, whereas train_mobilenetv2.py selects its checkpoint on a held-out portion of the training set. It is unclear which of these procedures produced each Table 1 entry. Clarifying this mapping and providing comparisons under a common validation-based selection protocol would strengthen the evidence. The tables also omit run-to-run variation, limiting interpretation of the 0.15 percentage-point CIFAR-100 gap.

Questions for authors
  1. Can you compare the full RERNet-L8 with (a) α_i = 0 for all RER blocks while retaining their original forward structure, nonlinearities, and shortcuts, and (b) a projected-input branch without the reconstruction term? Setting the constructor's alpha_max=0 currently substitutes ResidualBlock and changes the pre-addition activation, so it would not isolate the novelty contribution. Please report parameter/compute differences and whether the benefit is consistent across runs.

  2. Which checkpoint-selection procedure produced each Table 1 result? With a common held-out validation protocol and multiple independent runs, how does RERNet-L8 compare with a residual network narrowed to approximately the same parameter budget?

  3. Could you map the supplied training configurations to Tables 1–2, including the MobileNet-V2 variant and classifier head, and reconcile them with Appendix A.1? For example, the classification wrapper sets αmax=1, while the segmentation entry point uses the RERNet default of 0.5 and defaults to AdamW rather than the stated SGD pipeline.

1 comment

We thank the reviewer for their careful evaluation and constructive feedback.

Weakness 1: The contribution of the reconstruction residual is not isolated

We evaluate the following variants:

  1. RER-free: Retain the forward transformation and post-activation residual pathway, but set the novelty contribution to zero
  2. Residual-free: Retain the forward transformation and reconstruction pathway, but set the residual contribution to zero
  3. Constraint-free: Use independent trainable weights for the reconstruction pathway

These experiments highlight that the reconstruction-error term provides a benefit beyond the additional projection and the underlying forward transformation.

Weakness 2: Normalization complicates the information-preservation interpretation

We agree that BatchNorm is not merely a neutral operation in this context. Its affine transformation and normalization statistics can affect the relationship between the forward feature pathway and the reconstruction-error pathway. We avoid interpreting the reconstruction pathway as an exact inverse or claiming that it guarantees information preservation. Instead, we describe it as a learned reconstruction-based correction that provides an additional representation of the discrepancy between the block input and its reconstruction.

The reconstruction pathway can attenuate, counteract, or reinforce particular components of the direct feature contribution, depending on the learned reconstruction weights, novelty projection, BatchNorm parameters, and activation pattern. Changes to the features therefore independently affect the direct feature representation and the reconstruction-derived correction. This is to fulfill the primary motivation of performing more work per parameter.

Weakness 3: Comparisons with matched-parameter residual networks are missing

Our intention was to only test over popular standard architectures; modifying channel counts without a representative ablation for the architectures risks selection bias and introducing failure modes which are not attributable to the prior works hence it was considered beyond the scope of this study.

Weakness 4: Training protocols are inconsistent

We chose a training pipeline which achieves state-of-the-art reported results for the tested ResNet variants. We acknowledge in the appendix that the stated MobileNet-V2 comparison uses a different policy as it failed to scale with the same training pipeline that was used for all RERNet and ResNet variants; we provided code to train it both ways. We did not wish for the table to underrepresent known reported accuracies for the tested architectures hence we describe the best results reproducible from our code.

Weakness 5: Run-to-run variation is not reported

We have added mean and standard deviation for RERNet variants over 5 runs. We limited multi-run experiments to CIFAR-100 which is the harder baseline. Variations between runs is low so we believe single CIFAR-10 results are reasonably representative of model results.

Questions for Authors

Q1. Can you compare the full RERNet-L8 with (a) αi=0\alpha_i=0 and (b) a projected-input branch?

An RER-free (effectively αi=0\alpha_i=0) RERNet-L8 model achieves 79.03$\pm$0.29 on CIFAR100. A residual-free RERNet-L8 model achieves 80.25$\pm$0.40 on CIFAR100.

Q2. Which checkpoint-selection procedure produced each Table 1 result?

We report the accuracy of the best performing checkpoint on the test set across training epochs. We provide an alternative MobileNet-V2 training policy as it failed to scale with the same training pipeline that was used for all RERNet and ResNet variants.

Q3. Can you map the supplied training configurations to Tables 1–2 and reconcile the differences in αmax⁡\alpha_{\max} and optimizer settings?

For classification, we use the SGD-based training pipeline described in Appendix A.1. For semantic segmentation, we use AdamW. This choice is motivated by the different optimization characteristics of dense prediction tasks, where the model produces spatially distributed predictions and gradients are influenced by pixel-level class imbalance and the interaction between encoder and decoder components. AdamW provides adaptive parameter-wise updates and decoupled weight decay, which we found suitable for optimizing the segmentation architecture.

The segmentation implementation also uses the default RERNet value of (\alpha_{\max}=0.5), whereas the classification wrapper explicitly sets (\alpha_{\max}=1.0). These are task-specific configurations rather than a single shared training setup.

We revised the experimental description and Appendix A.1 to explicitly distinguish the classification and segmentation configurations, including the optimizer, learning-rate schedule and novelty scaling parameters.

Reviewer_EcPDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RERNet, a convolutional architecture that augments feature transformations with a local reconstruction-error pathway. Each RER block uses a transposed convolution, with weights tied to the forward convolution, to reconstruct its input from its output features. It subtracts this reconstruction from the actual input, projects the resulting error into the output feature space using a learned pointwise convolution, and adds the projected error to the output features. The intention is to retain useful information that the forward transformation represents poorly, enabling competitive accuracy with fewer learned parameters. Weight tying limits the additional parameter cost of reconstruction, although the reconstruction and projection operations introduce additional computation. The approach draws inspiration from predictive coding, under which higher-level representations generate predictions of lower-level activity, and discrepancies between predictions and observations drive updates to internal representations. In hierarchical formulations, feedback pathways carry predictions downward, while prediction errors propagate upward. These interactions typically form a recurrent, iterative inference process. RERNet adopts a single-pass version of this computational motif: each RER block computes and incorporates a reconstruction error once, without iterative inference or an explicit reconstruction-loss objective. Experimentally, RERNet-L8 achieves classification accuracy close to ResNet-18 on CIFAR-10 and CIFAR-100 using approximately one-third of its parameters. This saving comes with approximately 2.3 times the reported FLOPs and 28% higher inference latency on the tested GPU. Compared with ResNet-34, it uses approximately 18% of the parameters and runs faster, but achieves lower accuracy.

Critical strengths
  1. Proposed a parameter-efficient single-pass interpretation and implementation of predictive coding compatible with ResNets that achieves competitive accuracy with substantially fewer parameters. This strength enhances the paper's impact because AI models move to edge devices with constrained resources, parameter/memory efficiency has become a pressing concern and this paper is a step in the direction of mitigating this concern.
Critical weaknesses
  1. The paper does not adequately position or evaluate its contribution against directly relevant prior architectures. Han et al. [1] already investigate local prediction-error corrections for image classification, additional computation at fixed parameter count, and prediction-error saliency maps. Wen et al. [2] study predictive-coding classifiers with both independent and tied forward/feedback weights. Deep Back-Projection Networks [3] likewise implement the closely related sequence of feature transformation, reconstruction, error projection, and additive correction. These works are neither cited nor substantively compared with RERNet. Their omission leaves unclear which aspects of the proposed mechanism are new and whether its particular single-pass formulation offers a favorable accuracy–parameter–computation trade-off. An explicit architectural comparison and controlled experiments against closely related classifiers are needed to establish the significance of the contribution.
  2. The experimental section is the next biggest weakness. The experiments do not establish that the proposed reconstruction-error mechanism causes the reported parameter efficiency.
    1. Comparisons with ResNet-18/34 simultaneously change network width, depth, parameter count, and computation.
    2. The essential controls are missing that would distinguish the benefit of reconstruction-error correction from architectural choices and additional computation per parameter:
      1. the same backbone trained without the RER branch;
      2. a direct input-projection branch P(x)P(x) replacing P(x−x^)P(x-\hat{x});
      3. and a baseline that allocates comparable computation to repeated application of shared convolutional weights. That is, use the same backbone simply pass the output of the convolution layer back into itself. The total compute should match the RERNet performance. It has been shown by prior work [4,5,6] that repeating conv layer like so can lead to improvement in accuracy with the same parameter count.
    3. The effects of weight tying and the novelty-strength schedule are also not isolated.
    4. Furthermore, the interpretation of the residual as information discarded by the forward transformation is insufficiently justified: a tied transposed convolution is generally not an inverse, and projecting and adding the resulting error does not guarantee information preservation. The qualitative maps in Figure 3 cannot establish that novelty features are complementary to the main features or causally improve predictions. Consequently, the central explanation for RERNet’s effectiveness remains unsubstantiated, even if the reported classification results are accepted.
    5. Finally, CIFAR datasets are not really representative of real-world data owing to their small size and lack of diversity. It is well known that many approaches that succeed on CIFAR fail on more representative datasets like imagenet or ecoset. Most of the relevant prior from from even 2018 has also evaluated on imagenet.

[1] Kuan Han, Haiguang Wen, Yizhen Zhang, Di Fu, Eugenio Culurciello, and Zhongming Liu. “Deep Predictive Coding Network with Local Recurrent Processing for Object Recognition.” Advances in Neural Information Processing Systems, 31, 2018. [2] Haiguang Wen, Kuan Han, Junxing Shi, Yizhen Zhang, Eugenio Culurciello, and Zhongming Liu. “Deep Predictive Coding Network for Object Recognition.” Proceedings of the 35th International Conference on Machine Learning, PMLR 80:5266–5275, 2018. [3] Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. “Deep Back-Projection Networks for Super-Resolution.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. [4] David Eigen, Jason Rolfe, Rob Fergus, and Yann LeCun. “Understanding Deep Architectures using a Recursive Convolutional Network.” arXiv preprint arXiv:1312.1847, 2013, revised 2014. [5] Ming Liang and Xiaolin Hu. “Recurrent Convolutional Neural Network for Object Recognition.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3367–3375, 2015. [6] Qianli Liao and Tomaso Poggio. “Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex.” arXiv preprint arXiv:1604.03640, 2016, revised 2020.

Questions for authors
  1. Why did you choose to add a new trainable 1x1 projection, instead of passing e through the conv layer itself?
  2. Can you provide results for the same backbone trained with the RER branch disabled, and for a baseline allocating comparable computation to repeated shared-weight convolutions or residual blocks? These comparisons would help determine whether reconstruction-error correction provides benefits beyond architectural choices and additional computation per parameter.
  3. How does an RER block differ mathematically from a single local prediction-error correction in Han et al. [1], the tied-weight predictive-coding formulation of Wen et al. [2], and the reconstruction-error projection units of Haris et al. [3]? Which differences constitute the principal contribution, and what evidence supports their advantages?
  4. Which reported configurations use the outer shortcut in Eq. 9 rather than the formulation in Eq. 7, and what value of αmax⁡\alpha_{\max} is used? How many independent training runs underlie the reported accuracies, and what is their variability, particularly for the near-equality between RERNet-L8 and ResNet-18?

[1] Kuan Han, Haiguang Wen, Yizhen Zhang, Di Fu, Eugenio Culurciello, and Zhongming Liu. “Deep Predictive Coding Network with Local Recurrent Processing for Object Recognition.” Advances in Neural Information Processing Systems, 31, 2018. Paper [2] Haiguang Wen, Kuan Han, Junxing Shi, Yizhen Zhang, Eugenio Culurciello, and Zhongming Liu. “Deep Predictive Coding Network for Object Recognition.” Proceedings of the 35th International Conference on Machine Learning, PMLR 80:5266–5275, 2018. Paper [3] Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. “Deep Back-Projection Networks for Super-Resolution.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. Paper

1 comment

We thank the reviewer for their careful evaluation and constructive feedback.

Weakness 1: Missing directly relevant prior architectures

We thank the reviewer for identifying several important works that are closely related to the motivation and design of RERNet. We have cited the works and distinguish RERNets in computational structure and objective. We do not claim that reconstruction-error correction, weight tying, or prediction-error-based feature refinement are individually new. The contribution we seek to investigate is the feedforward block formulation integrated into a residual architecture.

Weakness 2: Missing controlled experiments

We agree that this is an essential control experiment. We evaluate the following variants:

  1. RER-free variant: Retain the forward transformation and post-activation residual pathway, but set the novelty contribution to zero
  2. Residual-free variant: Retain the forward transformation and reconstruction pathway, but set the residual contribution to zero
  3. Constraint-free variant: Use independent trainable weights for the reconstruction pathway

Weakness 3: Weight tying and novelty-strength schedule

  • We add ablations that show tied versus independently parameterized reconstruction weights perform similarly.
  • During the early stages of this study, we compared using linear annealing, constant and learnable α\alpha. Using constant novelty strength or making α\alpha learnable both diminish performance, and linear annealing is largely similar in performance to the cosine annealing used.

Weakness 4: The reconstruction doesn't preserve information

The reconstruction: x^=WT∗z\hat{x}=W^T*z is not interpreted as an exact recovery of the original input. The motivation is to create a pathway where model parameters attenuate, counteract, or reinforce particular components of their direct feature contribution, modulated by minimal learned novelty projection and BatchNorm parameters. This is to fulfill the primary motivation of performing more work per parameter. We revise the manuscript to make these distinctions explicit. We have removed strong claims about information preservation with a more precise description of the mechanism as a reconstruction-error-based residual representation. We also revised the interpretation of Figure 3 without strong claims of class-specific semantic saliency.

Weakness 5: Evaluation beyond CIFAR

We regret that with available computational resources we were unable to perform experiments with ImageNet1k within the timeframe. We consider it important future work.

Questions for Authors

Q1. Why use a trainable 1×1 projection?

The reconstruction error is defined in the input space of the block, whereas the primary feature representation is defined in the output space of the forward transformation. A 1×1 projection provides a channel-wise transformation that aligns the reconstruction-error representation with the output feature dimensions while retaining its spatial structure.

Q2. Can you provide the controlled ablations?

We have added architecture ablations in the results section.

Q3. How does RERNet differ mathematically from Han et al., Wen et al., and Deep Back-Projection Networks?

  • Han et al. (2018): Use local recurrent prediction-error processing to refine representations. RERNet instead computes a reconstruction error and incorporates its projected form through a feedforward block.
  • Wen et al. (2018): Investigate predictive-coding networks for object recognition, including forward and feedback interactions. RERNet uses a tied reconstruction operation within a single-pass feedforward architecture.
  • Haris et al. (2018): Use back-projection operations to iteratively correct reconstruction errors, primarily for super-resolution. RERNet incorporates a projected reconstruction error directly into the feature representation without the same iterative back-projection procedure. RERNets are inspired by the predictive coding mechanism to formulate a weight-tied compute-bound block where the contribution of model parameters like weights and BatchNorm parameters is complementary to their contribution to the main convolutional pathway.

Q4. Which configurations use the outer shortcut? What are the values of αmax\alpha_{max}? How many independent runs were performed?

The configurations all use the outer shortcut described in Eq. 9. The ablations over the RER-free variant and Residual-free variant motivate why the combined pathway was used for final results. All classification tests use αmax=1\alpha_{max}=1 and segmentation tests use αmax=0.5\alpha_{max}=0.5 We add mean and standard deviation for RERNet variants over 5 runs. Due to time and computational constraints, we limited multi-run experiments to CIFAR-100 which is the harder baseline. Variations between runs is low and we believe single run CIFAR-10 results are reasonably representative of model results.

Reviewer_nhzfOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors of the paper "Parameter-Efficient Visual Learning with Information-Preserving Transformations for Deep Convolutional Neural Networks" introduce a very interesting concept of Representation Error Residual (RER) in place of standard residual architectures. RER models almost match the performance of vanilla Resnet models. The core idea is to take the difference between input and what is left as output after the forward pass (the residual is called novelty signal) and via downprojection with 1x1 conv, add it to the features. The core idea is backed up by predictive-coding paradigm. On two standard computer vision benchmarks (top-1 acc and segmentation), the authors show that RER models achieve similar performance while keeping the number of parameters lower.

Critical strengths

The paper is easy to follow. It is well written, you can feel that the human worked on it, it is not AI slop. The research problem is stated, the motivation is clear, the existing base architectures are known. The benchmark dataset are standard in computer vision literature. The idea is very elegant, it is also explained well -- but the simplicity of solution is striking.

Critical weaknesses

The experimental section is poor. Although the pipeline how to train the network is correct, the results are either insufficient or not informative or biased -- see the questions for authors for the critique, improvements and suggestions. Logic flaw: by introducing BN we are actually cheating: batch norm of the output from 1x1 convolution blocked can be decomposed into part related to the original xx and the reconstruction x^\hat{x}.And the first part, related to the original input of the block is somehow what the vanilla residual connection is responsible for.

Questions for authors

1 Regarding the experiment enhancements: (a) in order to be fair w.r.t. to performance-capacity trade-off, I strongly suggest to compare pairs of models that either match the performance or the capacity. So for instance use some variants of Resnet9 (Moddy et al. on Github) or lighter Resnets: 20,32,56, WRN40-4 to compare with similar number of parameters. (b) Most of computational savings is caused by decreasing the number of massive blocks. Let me explain: in the largest variant: L8 we have 3.78M caused by the fact that the last five conv are 256->256. In standard Resnet18 model we have most of the paraameters towards the end of the model. We would have to scale down the width of Resnet models to have fair comparison. (c) Ablations: it is really not sure what is the role of alpha, or P (it could be pre-fixed, not learnable random projection) and most importantly BatchNorms (see weakness 2) (d) Report and communicate the number of seeds (e) Report and inform what is the relation between the reconstruction and the input to the block (Euclidean distance, cos similiarities etc.). Additionally, use oracle reconstruction to measure how much this component is actually responsible for the loss -- i.e. is the residual indeed better-conditioned input than raw input. Also tracking the norm (recons vs original) would be useful. (f) Introduce XL variants with massive number of parameters that much typical Resnet models and then we would be able to compare the performance with similar number of parameters. 2 What would happen in the framework if LayerNorm replace BatchNorm layers? 3 How would you relate to Deep Predictive Coding Network with Local Recurrent Processing for Object Recognition (Kuan Han, Haiguang Wen, Yizhen Zhang, Di Fu, Eugenio Culurciello, and Zhongming Liu)? 4 Results that you show are with or without optional shortcut signal?

1 comment

We thank the reviewer for their careful evaluation and constructive feedback.

General criticism: The experimental section is insufficient

We agree that our block contains a conventional residual connection, and we will clarify how this connection differs from the additional novelty pathway. The novelty stream contains both an input-dependent component and an offset-dependent component. The former depends on the interaction between the forward and reconstruction weights and the BatchNorm scaling, while the latter depends on the learned BatchNorm offset and the reconstruction weights. The reconstruction pathway can attenuate, counteract, or reinforce features depending on the learned novelty projection and activation pattern. Changes to the features therefore independently affect the direct feature representation and the reconstruction-derived correction. This is to fulfill the primary motivation of performing more work per parameter. We do not claim that the novelty pathway guarantees information preservation, but that it provides an additional reconstruction-error-derived signal that can modulate the feature representation alongside the conventional residual connection.

Q1a: Matched modelperformance or capacity

We have also report performance of WideResNet40-4. We intended to only test over popular standard architectures; modifying channel counts without a representative ablation for the architectures risks selection bias and introducing failure modes which are not attributable to the prior works hence it was considered beyond the scope of this study.

Q1c: α\alpha, projection PP, and BatchNorm

We evaluate the following variants:

  1. RER-free variant: Retain the forward transformation and post-activation residual pathway, but set the novelty contribution to zero
  2. Residual-free variant: Retain the forward transformation and reconstruction pathway, but set the residual contribution to zero
  3. Constraint-free variant: Use independent trainable weights for the reconstruction pathway.

Q1d: Number of seeds

We agree that reporting only a single run does not provide sufficient information about the variability of the results. We have added mean and standard deviation for RERNet variants over 5 runs. We limited multi-run experiments to CIFAR-100 which is the harder baseline. Variations between runs is low so we believe single run CIFAR-10 results are also reasonably representative of model results.

Question 1e: Relationship between reconstruction and input

We acknowledge that a tied transposed convolution is not an inverse of the forward convolution. The effects of stride, padding, nonlinear activation, and normalization mean that reconstruction is coarse. The error term is not explicitly minimised; the goal is to create a representation where model parameters have complementary contributions to the output compared to their normal contribution to the convolutional pathway.

Q1(f): XL variants

We tested three architectural variants (L,M,S) to provide a representative sample of models feasible for ablation with our computational budget. Increasing parameter dimensions further was limited by memory constraints. In our tests adding more blocks beyond the stated architectures have diminishing returns. An RERNet-10L performs very similarly to RERNet-8L; adding more blocks diminishes accuracy. We plan to explore layer scalability as future work.

Q2: LayerNorm vs BatchNorm

We tested with BatchNorm as it is a common standard in vision models. We consider exploring different functional choices for activation normalisation important future work.

Question 3: Deep Predictive Coding Network with Local Recurrent Processing

We thank the reviewer for pointing out the closely related work by Han et al. (2018). The proposed RER mechanism shares the broader motivation of using prediction errors to refine internal representations. However, the architectural formulation and computational structure are different. The Deep Predictive Coding Network uses local recurrent processing to iteratively update representations based on prediction errors. In contrast, RERNet uses a feedforward, single-pass formulation in which the input is reconstructed from the current feature representation, the reconstruction error is projected into the feature space, and the projected signal is incorporated into the block output. While our work is predictive-coding-inspired it is not a direct implementation of the full iterative predictive-coding inference procedure.

Question 4: Are the results obtained with or without the optional shortcut?

We agree that the original manuscript does not describe the shortcut configuration sufficiently clearly. The configurations for the reported results all use the outer shortcut described in Eq. 9. As described, the ablations over the RER-free variant and Residual-free variant will motivate the chouce of the combined pathway as the best.

Reviewer_jWD6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes the Representation Error Residual (RER) block, inspired by predictive coding. Each block reconstructs its input with a weight-tied transposed convolution and adds a 1×1 projection of the reconstruction error to the output, scaled by a depth-dependent α, optionally alongside a residual shortcut. On CIFAR-100, the 8-block RERNet-L reaches 80.95% top-1 with 3.78M parameters, versus 81.10% for ResNet-18 (11.2M) and 83.05% for ResNet-34 (21.3M); segmentation on Oxford-IIIT Pet and novelty-map visualizations are also reported.

Critical strengths

Feeding a layer's own reconstruction error forward as an extra feature stream is a simple idea with a clear link to predictive coding. Weight tying keeps the added parameter cost small, and the paper is open about trading computation for parameters.

Critical weaknesses
  1. The RER mechanism is never isolated: Sec. 3.6 indicates that RERNet uses RER blocks with residual shortcuts (Eq. 9), but it is compared only with deeper and wider ResNet-18/34. An ablation with the same 8-block architecture and α = 0, or with a matched-cost replacement branch, is needed to attribute the parameter savings to RER rather than to a smaller backbone.
  2. The efficiency claim is not supported: RERNet-L8 uses 2.3× the MACs of ResNet-18 and has higher latency (0.77 vs 0.60 ms) at slightly lower accuracy, and it is worse on segmentation. Comparisons with ResNets of matched parameters or matched compute are needed.
  3. The evaluation is too limited for ICLR: only CIFAR-10/100 and a small pet segmentation set, while ImageNet results should be reported at least. MobileNet-V2 is also trained with a different augmentation policy, so that comparison is not like-for-like.
  4. The claim that novelty maps track class-specific saliency (Fig. 3) rests on five images with no quantitative measure. A quantitative check against saliency or segmentation masks is needed.
  5. Closely related work is missing, including predictive-coding CNNs for recognition (PredNet, Lotter et al., 2017; PCN, Wen et al., 2018) and classifiers with reconstruction paths (Ladder Networks, Rasmus et al., 2015).
Questions for authors
  1. What is the accuracy of the same RERNet-L8 architecture with α = 0, and with the novelty branch replaced by a matched-cost 1×1 convolution?
  2. How does RERNet compare with ResNets of matched parameter count or matched MACs on CIFAR-100?
  3. Can you report results on a larger dataset such as ImageNet?
1 comment

We thank the reviewer for their careful evaluation and constructive feedback.

Weakness 1: Isolating the RER mechanism

We performed a controlled ablation that preserves the underlying architecture while modifying the reconstruction-error pathway. Specifically, we evaluate:

  1. RER-free variant: Retain the forward transformation and post-activation residual pathway, but set the novelty contribution to zero.
  2. Residual-free variant: Retain the forward transformation and reconstruction pathway, but set the residual contribution to zero.
  3. Constraint-free variant: Use independent trainable weights for the reconstruction pathway. An RER-free RERNet-L8 model achieves 79.03$\pm$0.29 on CIFAR100. A residual-free RERNet-L8 model achieves 80.25$\pm$0.40 on CIFAR100. The constraint-free variant performs similarly to the weight-tied model with 80.900.32 with 1.74x the parameters of the weight-tied RERNet.

Weakness 2: The efficiency claim is not supported

Reviewer comment: RERNet-L8 higher MACs and latency (0.77 vs. 0.60 ms) at slightly lower accuracy than ResNet18

Parameter efficiency does not necessarily imply lower MACs, memory traffic, or inference latency. Our initial motivation was to investigate whether reconstruction-error representations could provide useful additional information without requiring the same number of independently parameterized convolutional transformations as conventional residual architectures, to do more work per parameter.
We therefore revise the efficiency claims to distinguish parameter efficiency from computational efficiency. For segmentation, we acknowledge that the initial results do not demonstrate an advantage over the residual baseline. We will report these results more cautiously and distinguish the feasibility of using RERNet as a backbone from evidence of superior segmentation performance.

Weakness 3: Limited evaluation and inconsistent MobileNet-V2 augmentation

We chose a training pipeline which achieves state-of-the-art reported results for the tested ResNet variants. We acknowledge in the appendix that the stated MobileNet-V2 comparison uses a different augmentation policy as it failed to scale with the training pipeline that was used for all RERNet and ResNet variants; we provided code to train it both ways. We did not wish for the table to underrepresent known reported accuracies for the tested architectures hence we describe the best results reproducible from our code.

Weakness 4: Insufficient quantitative evidence for class-specific saliency

The purpose of the original visualization was to illustrate the spatial structure of the projected reconstruction-error representation. We acknowledge that the visualizations alone cannot establish that the novelty signal corresponds to semantic saliency and we revise the description of Figure 3.

Weakness 5: Missing related work

We thank the reviewer for identifying these relevant works. We clarify that RERNets are inspired by the prediction coding mechanism to formulate a weight-tied feedforward block where the contribution of model parameters like weights and BatchNorm parameters is complementary to their contribution to the main convolutional pathway.

The mentioned works use either recurrent computation local to the layer or, for given prediction error, they are trained to explicitly minimise this error. PredNets are recurrent neural networks, and are similar in spirit where error is propagated as features, however this error is calculated exactly with respect to the next sequential input which is also the optimisation target of the model. PCNs also use the transpose convolution but as feedback weights from the next layer to the previous in recurrent computation. Ladder networks are trained for balancing auxiliary unsupervised task loss with the original classification loss.

Questions for Authors

Q1. What is the accuracy of RERNet-L8 with =0, and novelty branch replaced by a matched-cost 1×1 convolution?

An RER-free RERNet-L8 model achieves 79.03$\pm$0.29 on CIFAR100. A residual-free RERNet-L8 model achieves 80.25$\pm$0.40 on CIFAR100.

Q2. How does RERNet compare with ResNets of matched parameter count or matched MACs on CIFAR-100?

We only tested over popular standard architectures; modifying channel counts without a representative ablation for the architectures risks selection bias and introducing failure modes which are not attributable to the prior works hence it was considered beyond the scope of this study.

Q3. Can you provide results on ImageNet?

We regret that with our available computational resources we were unable to perform experiments with ImageNet1k within the timeframe and we consider it important future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.