acceptodds
Under review as a conference paper at ICLR 2027

Distributed Convolutional Neural Networks: Learning Disentangled Feature Embeddings

Abstract

This paper proposes a loss function for training the distributed convolutional neural network (DisCNN), which embeds positive-class features into a compact ball and drives the embeddings of dissimilar negative-class features toward the origin. We theoretically validate that the convolutional layers of DisCNN exclusively extract positive features while completely ignoring any dissimilar features, thereby fully disentangling positive features from dissimilar features. Furthermore,We verify a positive monotonic association between the presence probability of positive features and the L2-norms of feature embeddings. The model achieves favorable generalization performance on test data, even on training-unseen samples bearing partial positive features. Moreover, a preliminary experiment demonstrates DisCNN’s potential for object detection.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
distributed neural networks, convolutional neural networks, representation learning, feature disentanglement, loss design, learning theory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HDPeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DisCNNs, a biologically inspired distributed CNN architecture for single-category feature learning. The authors introduce the N2O loss, which encourages positive-class embeddings to form a compact region while pushing dissimilar negative features toward the origin, thereby disentangling target-class features from irrelevant ones. The paper shows that the learned representations selectively capture positive features.

Critical strengths

This paper makes an interesting attempt to CNN architecture, and the experimental results appear to be genuine and carefully conducted. I believe this work represents a promising starting point. However, the current study is still at a relatively early stage, and many aspects require further reconsideration and refinement. Please see the Weaknesses and Suggestions sections for detailed comments.

Critical weaknesses
  1. The paper aims to learn category-specific feature maps for each class, but the benefits of this design seem limited. First, many object categories share common components, such as wheels in cars and bicycles, so learning a shared representation may be more efficient. Second, the biological motivation is not consistent with the proposed architecture: the human visual system shares low-level features across categories rather than building a totally separate recognition system for each category, for example, assigning one eye to each object category. In contrast, DisCNNs are category-specific throughout the entire network. Finally, building a separate model for each category raises an important scalability issue. More specifically, the performance ceiling is constrained by the capacity of an individual model, making it difficult to achieve further performance improvements.

  2. The proposed model indeed has fewer parameters, but this comes at a significant cost. As the number of classes and samples in the dataset increases, the number of negative samples also grows. First, this leads to a severe class imbalance problem, making training difficult when negative samples vastly outnumber positive samples. Second, each model needs to be trained on the entire dataset, resulting in a very high overall training cost.

  3. The proposed method is essentially a one-vs-all classifier. This type of classifier has been extensively studied since the last century and remains widely used in modern anomaly detection and out-of-distribution detection. Therefore, the core idea of the proposed approach is not particularly novel.

  4. The authors incorrectly use the term “theoretical proof.” A theoretical proof refers to a mathematical derivation or proof, rather than empirical validation through experiments.

  5. Evaluating the method on only a single dataset is insufficient for a pattern recognition study. The authors should validate the effectiveness of the proposed method on at least three benchmarks and provide comparisons with existing methods.

Questions for authors
  1. Based on the proposed method, I suggest that the authors consider exploring its application to anomaly detection or out-of-distribution detection.

  2. The authors should pay more attention to the writing and presentation of the paper. I suggest studying and emulating the overall structure and presentation style of well-written papers. In practice, reviewers often have limited time, saying, 20 minutes, and if readers cannot quickly find the information they are looking for in the expected places, they may lose interest in reading further. The abstract and introduction are relatively well written, but the writing and organization of the later sections are considerably less standard and could be substantially improved.

  3. I encourage the authors to engage more actively with researchers in pattern recognition, rather than developing the work in isolation. Such interactions could help broaden their perspective and identify or correct potential biases in their current understanding, which are particularly important for developing into a top researcher.

Reviewer_vzogOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This manuscript addresses binary classification problems and proposes DisCNN, which possesses relatively low parameter complexity. The so-called "theorems" presented in the manuscript merely describe specific experimental results of DisCNN on a particular dataset, rather than constituting general conclusions. Presenting these results in the form of theorems is therefore inappropriate.

Critical strengths

The DisCNN proposed in this paper has a relatively small number of parameters.

Critical weaknesses

This manuscript reads like a middle school laboratory report rather than a scientific paper. The so-called theorems presented therein are merely statements of specific results obtained from a particular model on a specific dataset, rather than generalizable conclusions.

Questions for authors

The final features output by DisCNN are activated by ReLU, such that the maximum angle between feature vectors of negative sample pairs is 90 degrees. It is recommended to refer to the following references.

  • H. Dang, T. Tran, et al. Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Features Model, ICML2024

  • G. R. Kini, V. Vakilian, et al, SYMMETRIC NEURAL-COLLAPSE REPRESENTATIONS WITH SUPERVISED CONTRASTIVE LOSS: THE IMPACT OF RELU AND BATCHING, ICLR2024

Otherwise, without ReLU, in the case of only two classes, the angle between feature vectors of negative sample pairs can reach 180 degrees, and in the case of KK classes, the angle between feature vectors of different classes can uniformly reach arccos(-1/(K-1)).

  • Z. Zhu, et al, A Geometric Analysis of Neural Collapse with Unconstrained Features, NeurIPS2021
Reviewer_uUMGOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces something called the distributed convolutional neural network (DisCNN). From my understanding, it's a VGG-style CNN combined with a loss function for binary classification. Specifically, they remove the multi-class softmax classification head and instead train a separate CNN as a binary classifier for each class: positive for samples belonging to that class, and negative for everything else. The paper also discusses (i) some geometric properties of the embeddings before the classification head; and (ii) the application in object detection, such as detecting cars in images.

Critical strengths

None.

Critical weaknesses

The paper has no related-work section. The problem posed in the introduction, the proposed loss, and the claimed object detection capabilities each have directly related peer-reviewed work published between 2018 and 2024 (below are a few examples). The paper neither cites nor compares against these works.

Very similar from the problem description through to the proposed solution: Liang et al., "Training Interpretable CNNs by Differentiating Class-specific Filters", ECCV 2020

Similar to the N2O loss in the paper: Dhamija, Günther & Boult, "Reducing Network Agnostophobia", NeurIPS 2018

Similar to geometric properties observed in Lemma 2: Papyan, Han & Donoho, "Prevalence of Neural Collapse during the Terminal Phase of Deep Learning Training", PNAS 2020 Dang et al., "Neural Collapse for CE Class-Imbalanced Learning with Unconstrained ReLU Feature Model", ICML 2024

The paper has a few other obvious flaws:

  • no baselines.
  • all theorem proofs are actually experimental results, there are no proofs.
Questions for authors

Given the critical weaknesses listed above, my position is a clear reject. I have no further questions for the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.