acceptodds
Under review as a conference paper at ICLR 2027

To Gate or Not to Gate? When Does an Input-Dependent Gating Pay Off in Multi-Class Dendritic Gated Networks

Abstract

Dendritic Gated Networks (DGNs) consist of neurons that each have several weight vectors (their branches), each paired with a fixed random hyperplane. The input, together with the hyperplanes, determines which branches are active, contribute to the output and get updated. This mechanism makes DGNs resistant to catastrophic forgetting, able to approximate arbitrary nonlinear functions while being convex in their weights, and trainable without backpropagation. DGNs are binary classifiers, and existing multi-class extensions support classes one of three ways: training one full network per class (one-versus-all), training networks over an efficient binary code, or applying a shared gated expansion, in which one fixed set of random half-space gates is applied once per input, the per-branch blocks are concatenated into a single feature vector, and independent Bernoulli readouts read that vector. We propose training those readouts jointly by softmax regression: the learning objective stays jointly convex, the update stays local up to one shared normalizer per sample, and the design keeps the expansion's one to two orders of magnitude fewer parameters than the one-versus-all formulation, and its single expansion in place of the forward passes, which is what makes tractable. We then investigate when using input-dependent gates pays off, against controls matched in dimension, parameter count, update rule and training protocol. Under domain-incremental training with smooth image rotation ( and ) and pixel noise (), our proposed method outperforms the dense, static-gate and linear controls on all-task and first-task accuracy. Given the same frozen features, it also outperforms the training-free nearest-class-mean classifier, and against and over all tasks under rotation and noise. With a replay buffer on split-CIFAR-100 it also outperforms a backpropagation-trained network with Dark Experience Replay++ (DER++) at every buffer size we ran, against at rows. We attribute the gating margin to regime structure, where the input-label mapping differs between regimes that can be identified directly from the input itself. In a class-incremental problem, which has no such regime structure, our method's margin over its static-gate control shrinks to the margin it already has under i.i.d. training, as predicted. It still outperforms one-versus-all and, storing no past data, reaches against for a backpropagation-trained network with Learning without Forgetting, but an untrained dense random projection of the same width matches it and the nearest-class-mean classifier leads it ().

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
continual learning, catastrophic forgetting, dendritic gated networks, local learning rules

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mwKNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed the SGE-softmax method, which replaces the independent Bernoulli readout mechanism of SGE with softmax, and establishes the joint convexity of the method. Through domain-incremental and class-incremental benchmarks on the MNIST and CIFAR-100 datasets, the authors investigate when input-dependent gating mechanisms become effective, suggesting that empirical gating margins primarily emerge under conditions characterized by a regime structure.

Critical strengths
  1. The paper improves the SGE method using softmax, it is simple and effective.
  2. The paper implements experiments on class-incremental and domain-incremental scenarios, validating the method's effectiveness from different perspectives.
Critical weaknesses
  1. The novelty of the paper is limited. The authors' contribution only is modifying SGE using softmax to handle multi-class classification.
  2. SGE employs a set of hyperplanes to reduce the number of parameters. How are it initialized? What is the impact of different initialization methods? Why is using a set of hyperplanes still viable?
  3. The proposed method lacks comparisons with state-of-the-art methods (such as PTM, DER, and Tagfex), undermining the persuasiveness of its effectiveness for continual learning.
  4. The performance of the proposed method remains limited in class-incremental tasks.
  5. The paper reports only accuracy as an evaluation metric for continual learning, which is insufficient to demonstrate effectiveness. Since the authors claim GDN mitigates catastrophic forgetting, reporting metrics such as Forgetting, Forward Transfer (FWT), and Backward Transfer (BWT) would better illustrate the model's ability to handle the issue.
  6. There is no comparison regarding parameter counts in this work. Compared to other methods, is the performance gain simply a result of an increase in model parameters?
Questions for authors

Please refer to the weakness.

Reviewer_3QexOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when input-dependent gating benefits multi-class Dendritic Gated Networks (DGNs) in continual learning. The authors introduce SGE-softmax, which extends existing Shared Gated Expansions (SGEs) with joint softmax regression while preserving convex optimization and avoiding backpropagation. Experiments on class-incremental and domain-incremental learning suggest that input-dependent gating is particularly beneficial under identifiable distribution shifts, while offering limited advantages in class-incremental settings.

Critical strengths
  • The paper investigates when input-dependent gating provides advantages over alternative feature representations, addressing an important question in continual learning.
  • SGE-softmax provides a straightforward multi-class extension with a convex objective and substantially fewer parameters than the one-versus-all DGN formulation.
  • The evaluation includes static gating, random projections, linear classifiers, and continual learning baselines across different learning protocols, offering useful empirical observations.
Critical weaknesses
  • The core gated expansion and regime-structure hypothesis are inherited from existing work. The primary algorithmic contribution is replacing independent Bernoulli readouts with standard softmax regression, whose convexity is already well established.
  • The CIFAR-100 experiments rely on frozen pretrained ResNet-18 features, which substantially simplify the representation-learning problem. Its advantage over conventional backpropagation-based methods is therefore difficult to interpret.
  • The paper does not provide a formal relationship between regime identifiability, gate activation patterns, and the expected performance advantage, making the central conclusion primarily empirical rather than theoretically established.
  • The paper emphasizes parameter efficiency and advantages over backpropagation-based continual learning methods, but the approaches use different architectures, optimization schemes, and update budgets. More carefully matched computational budgets and standard baseline implementations are needed.

Typo:

Page 8, Line 414 "Definition 1 ," -> "Definition 1,"

Page 8, Line 424 "samd width" -> "same width"

There are more typos in the manuscript that are not listed here. The authors should check the writing.

Questions for authors
  • Beyond the use of standard softmax regression, what is the main methodological novelty over prior work? Can the authors clarify which theoretical or practical capabilities are genuinely new?
  • How does SGE-softmax compare with stronger pretrained-feature continual learning methods, particularly RanPAC, under identical backbone and training protocols?
  • Can the authors provide a more controlled comparison of computational efficiency and performance against standard continual learning baselines, including the original DER++ implementation?
Reviewer_Rap2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when input-dependent random gating benefits multi-class continual learning. It extends an existing Shared Gated Expansion (SGE) by replacing independent Bernoulli readouts with a jointly trained softmax classifier. The feature expansion remains fixed, so the readout objective is jointly convex and its update requires only the current prediction error, the expanded features, and a shared softmax normalizer. The paper evaluates class-incremental and synthetic domain-incremental protocols on MNIST and CIFAR-100, using frozen ImageNet ResNet-18 features for CIFAR-100. It reports positive margins over matched static-gate controls under drift, sensitivity to feature centering, and smaller gating-specific gains under class-incremental learning. The authors interpret these findings through regime structure: input-identifiable modes requiring different readout mappings. The paper also reports cases in which a prototype classifier or a longer-trained dense random expansion outperforms SGE-softmax, and includes replay comparisons with its implemented continual-learning baselines.

Critical strengths
  1. A simple and practical multi-class formulation. SGE-softmax provides a straightforward way to train the existing shared expansion with a coupled multi-class objective while preserving a fixed representation and a jointly convex readout problem. The update is explicitly stated and easy to implement. The parameter and computation advantages over one versus all DGNs are inherited from the shared expansion, and the softmax extension preserves them with little additional machinery.
  2. Useful controls and candid reporting of negative results. The paper goes beyond a performance only comparison: it includes linear, static gate, random ReLU, and nearest class mean controls, and examines training budgets, branch counts, data splits, and centering. The matched static-softmax comparison on rotated CIFAR-100 shows a positive margin (.500 versus .473 over all tasks). The authors also report that prototypes outperform SGE-softmax on split-CIFAR-100 (.526 versus .468), and that a longer-trained random expansion overtakes it. These results help separate benefits of frozen representations from those specifically attributable to gating.
Critical weaknesses
  1. The main nonlinear comparison does not match the proposed readout rule. Section 2.4 states that fixed-expansion controls use matched readout rules, but Table 6 explicitly uses Bernoulli readouts for dense random ReLU while SGE-softmax uses softmax. Section 5.2 identifies the MNIST dense control as Bernoulli as well. The missing dense-ReLU-plus-softmax comparison matters because changing the readout can change the ranking: at 40 epochs on split-CIFAR-100, dense ReLU scores 0.44, SGE with Bernoulli readouts 0.425, and SGE-softmax 0.468. The Bernoulli-to-Bernoulli drift comparisons do provide partial support for the gated feature map, and the matched static-softmax results are useful. However, the evidence does not yet isolate the proposed method's advantage over an equally wide nonlinear expansion trained with the same objective and class-update policy. This is central to the paper's attribution of the gains.
  2. Regime structure is a plausible interpretation but is not sufficiently validated as the boundary of gating benefits. Definition 1 and Table 11 do not fully distinguish a change in the correct input-label mapping from covariate shift or limitations of a linear readout. A within-domain versus transfer accuracy gap shows that domain-specific linear fitting helps; it does not establish that the random gates identify those regimes or reduce interference between them. Moreover, Section 6 attributes the contrast with dense ReLU to the gated map writing to different weights for different inputs. Fixed ReLU features also have input-dependent zero activations, so their linear readout updates have input-dependent support. The experiments therefore need a more precise explanation of what distinguishes the two feature maps. Since the shared expansion and the qualitative regime hypothesis are attributed to prior work, the significance of this submission rests substantially on the strength of this empirical explanation.
Questions for authors
  1. Can you report dense random ReLU with softmax, using the same centering, class-update mask where applicable, validation selection, and training-budget sweep as SGE-softmax? Does the advantage persist on the drift protocols? This limited comparison could change my assessment of the gating-specific contribution.
  2. How is regime structure operationally distinguished from a transfer gap of a restricted classifier? On the existing drift protocols, can you provide a direct check of cross-domain gate overlap or gradient interference, or a sequential-versus-joint comparison on the same mixture of drifted domains? Please also revise the distinction from ReLU updates. Evidence supporting the mechanism, or an appropriate narrowing of the claim, would affect my assessment.
  3. Section 4 computes the class-incremental centering vector from the whole training set. Does this assume access to inputs from future tasks before sequential training? Given the sensitivity in Table 10, do the main class-incremental results persist when the centering vector is fixed from the first task or an independent calibration set? This is a question about future-training-input access, not an allegation of test-label leakage. The domain-incremental protocol uses the initial domain mean and is not subject to the same concern.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.