acceptodds
Under review as a conference paper at ICLR 2027

Inheritance Between Feedforward and Convolutional Networks via Model Projection

Abstract

Neural-network techniques are often transferred across architecture families by analogy, but such transfer is valid only when the assumptions required by a technique are preserved. We introduce this idea as inheritance between model classes. Using a unified node-level framework with tensor-valued activations, we prove that generalized feedforward networks (GFFNs) form a strict subset of generalized convolutional networks (GCNNs), so GCNN properties transfer directly to GFFNs. The reverse direction is not automatic: standard CNN nodes use spatial kernels, while FFN nodes use one scalar weight per input contribution. We introduce model projection to recover a restricted reverse inheritance path. Projection freezes each convolutional input-channel sub-function and learns one scalar coefficient for each input-output channel contribution, giving projected CNN nodes the GFFN-style trainable structure of scalar-weighted input recombination. This inherited structure leads naturally to transfer learning. Across multiple ImageNet-pretrained CNN backbones and downstream image-classification datasets, model projection is competitive with standard transfer learning baselines and provides an effective initialization for subsequent full fine-tuning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Transfer Learning, Convolutional Neural Networks, Model Projection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hZqkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission studies the concept of inheritance, i.e. the idea that techniques can be transferred between different families of architecture within a proper formal framework. Under the formalization of neural network provided in the manuscript, the authors show that generalized convolutional networks (GCNN) can be regarded as a more general model class than generalized feedforward networks (GFFN). The authors use this formalization also to motivate a fine-tuning technique, that they call projection. This technique freezes convolutional filters but combines them with learnable weights.

While the submission’s overall premise of aiding the transfer of insights between architectural families is interesting, I found major parts of the paper rather disconnected from this motivation. I did not find any mistakes in the paper’s proofs. The experimental evaluation uses a large variety of datasets and is well documented.

Overall, this paper is a potentially interesting contribution to CNN finetuning but in the current form overstates its contributions and could benefit from more detailed experimental evaluation.

I comment on critical strengths, weaknesses, and questions below. I am open to increasing my score if weaknesses are addressed during the rebuttal phase.

Critical strengths

The experimental results suggest that the proposed projection method has promising performance in CNN finetuning on many image classification datasets.

Critical weaknesses

I believe that motivation and experimental evaluation are somewhat disconnected. The authors motivate their work with the transfer of techniques or practices between different model families, but their evaluation focuses only on fine tuning of pretrained CNNs. It is not clear to me how this connects to the idea of inheritance. The paper appears to me more convincing as the proposal of a CNN finetuning technique than as a general study of inheritance.

It is known that an FNN with an imposed weight sharing pattern is equivalent to a CNN. I believe it would be good to discuss how the authors’ formalization relates to this perspective and potentially soften the strong claim that “There is currently no inheritance framework between common deep learning models.” [l.71]

Questions for authors

The authors report standard deviations for their accuracy results. Across how many samples is this standard deviation calculated? What constitutes a sample (e.g., is it an independently chosen seed)?

How does the authors’ approach relate to the one developed in Mo, Cheng, & Fan (2019) arXiv:1906.04950? (I acknowledge that this work is not peer reviewed, but think discussing it might still be interesting as it seems closely related.)

Reviewer_1mkuOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a generalization of FFN and CNN layers, representing both in the form of a tensor dot product. Under this view, GFFNs are a strict subset of GCNNs.

They use this to motivate a finetuning method that consists in learning a scaling vector applied to frozen convolutional weights.

Experiments are done on finetuning on Cifar-10,Cifar100 and Flowers and compared to the LoRA and Para methods

Critical strengths

The attempt to propose generalized view of feedforward and convolution as tensor dot product is nice.

The proposed finetuning method seems competitive.

Critical weaknesses

First, I do not understand the general direction of this paper and why the GFFN and GCNN framework are needed to define the finuting method.

The definition of GFFN relies on the ambiguity surrounding feedforward layers applied to tensors such as images. As stated in line 198, the behavior of a GFFN differs from that of a standard feedforward layer (and is closer to what is seen in architectures such as MLP-Mixer). Similarly, it is unclear whether the definition of GCNN differs from that of a standard convolution.

Consequently, my interpretation of Theorem 3.5 is that "something that is not really a feedforward layer is a strict subset of something that is not really a convolution" (I assume GFFN/GCNN refers here to the sets of functions that can be represented by these "nodes").

I do not know what practical insight to draw from this, especially since the opposite inclusion can be shown for standard convolutions and feedforward layers. Indeed, assuming zero padding, a convolution can be expressed as a 2D Toeplitz matrix of shape ((d_{\mathrm{out}}n_1n_2,,dn_1n_2)) applied to flattened inputs, which is equivalent to applying a feedforward layer to these same flattened inputs. The Toeplitz constraint leads to the opposite inclusion: every convolutional layer can be expressed as a feedforward layer, but not vice versa. The main difference between this result and Theorem 3.5 lies in how linear layers are applied to image tensors.

Given this, it is very difficult to connect these theoretical results to the proposed fine-tuning method. How does establishing a relationship between a feedforward variant and a convolutional variant lead to a practical fine-tuning method for standard CNNs?

Furthermore, the vague terminology makes this work difficult to read. What exactly is meant by "techniques", "intuitions", or "results"? Is there confusion between the terms "layers", "models", and "nodes"?

Concerning the FineTuning method mainlydescribed in Fig 1, it would had been great to explain the difference with LoRA or other variant. And the method IA3 described in "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning" Liu et al Neurips 22 seems close to your solution (element wise multiplication of activations but applied to transformers)

Finally, several notations are incorrect or do not make sense: Eq 1 and l146: “et Wj containing a vector of weights Wjk \in R^d”: it seems that wj \in R^d not W_jk, otherwise Eq would multiply a real value Z_k by a vector Wjk \in R^d that cannot be added with a bias b_j\in R L159: B_j\R^m_1 \times m_2: m_1 and m_2 are not defined and usually for a convolution Bj is \in R Eq 2 and 4: is Z_k \ast Fjk and Fjk \ast Z_k equivalent in your setting, because for deeplearning convolution (that are cross-correlation) this is not true in general Definition 3.1: Z\in R^{d \times D}, and Wjk in R^d, could you explain the multiplication done between Z_k\in R^D and Wjk \in R^d ? Many others inconsistencies in the notations

There are many other inconsistencies in the notation.

Questions for authors

To me, the main issue lies in the assumption that what applies to GCNNs and GFFNs also applies to standard CNNs and FFNs, whereas these are, in my view, different objects. Could the authors explain why the results would extend to standard CNNs and FFNs?

Could the authors better explain what motivates the proposed fine-tuning method and compare it with existing approaches to clarify its novelty?

Could the authors also carefully review and correct the inconsistencies in the mathematical notation throughout the paper.

Reviewer_Bje1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents the concept of generalized feedforward neural networks, generalized convolutional neural networks and the concept of inheritance between these two families of functions. The authors present a projection framework to transport the study of one functional family to the other. They also provide an extensive experimental evaluation to validate their theoretical statements.

Critical strengths
  • They provide an extensive experimental validation of the framework.
Critical weaknesses
  1. Many of the presented findings are already known results from the theory of CNNs and FNNs, e.g., the equivalence between FNNs and 1×11 \times 1 convolutions which extends as it is to the generalized case.
  2. Many statements are too broad and too strong, for example Corollary 3.6 as stated is too strong. Indeed, properties of machine learning models do not always depend only on the structure of their associated functional space but also they may depend on their parameterization.
  3. The theory of functions "separable by input" is largely studied in the machine learning literature and these connection are not properly discussed and cited in the manuscript. For example, this notion can be seen as a simplifed version of Kolmogorov-Arnold representation theorem and as a superposition of simple ridge functions.
  4. Definition 3.10 of inhomogeneus input is not well-defined since it is dependant on the weight choice. This hinders the validity of both Theorem 3.11 and Corollary 3.12 since their statements hinge on this definition.

For these reasons I propose a negative score, however I am open to discussion and eventually to reasses my evaluation.

Questions for authors

Can the authors address the issues raised in "Critical Weaknesses"? In particular, point (3) and (4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.