This paper has been rushed for submission without careful proofreading. I am highly unlikely to change my mind.
I have 3 major concerns that I don't think the rebuttal can address.
1. Framing and positioning overclaim.
You are overclaiming in so many places; here are two examples:
a. The introduction says existing CL work is “typically evaluated only through the ability to remember previously learned tasks.” This mischaracterizes the field. Many methods explicitly target forward transfer, feature/representation learning, and plasticity. Basically, the GEm paper that you cited introduced FWT (forward transfer).
b. The claim to “formulate continual learning as a joint problem of retention and composition” mostly relabels the protocol. Recombining classes from different tasks into new binary pairs is a re-evaluation of standard split benchmarks, not a new problem formulation. It basically redefines class-incremental learning, not a problem you identified.
2. Experiments cannot be reproduced as you haven't included the code.
Datasets are too small to support the generality claimed. The main results use only MNIST and CIFAR-10, and CIFAR-100 appears only in the supplement. Standard class-incremental benchmarks such as CIFAR-100 with 10/20 splits as a main result, TinyImageNet, ImageNet-100/ImageNet-R and CORe50 are missing.
Non-standard backbones. The MLP has two hidden layers (64, 128). The CNN is a custom “Simple ResNet” with kernel size 5, not the backbones used in the class-incremental literature, such as reduced ResNet-18 (GEM/A-GEM, DER), ResNet-32 (iCaRL) or ResNet-18 on ImageNet subsets. This makes direct comparison to published numbers impossible. Given recent CL work on pretrained models (e.g., ViT backbones), a pretrained transformer comparison would also show whether gain inference scales.
Single seed (42), no error bars or variance.
No compute or wall-clock comparison, even though the Kalman-gain computation is acknowledged as expensive.
No ablation of target propagation versus plain backprop with the same gain inference, so it is unclear which component drives the results.
3. Bibliography errors.
Paper cites:
JM Bernardo, MJ Bayarri, JO Berger, AP Dawid, D Heckerman, AFM Smith, M West, et al. The variational bayesian em algorithm for incomplete data: with application to scoring graphical model structures. Bayesian Statistics, 7(453-464):210, 2003.
While all authors are fabricated, please see: https://mlg.eng.cam.ac.uk/zoubin/papers/valencia02.pdf
Also, the paper cites:
Eric Elmoznino, Thomas Jiralerspong, Yoshua Bengio, and Guillaume Lajoie. Towards a Formal Theory of Representational Compositionality. June 2025. URL https://openreview. net/forum?id=fXCfT7ErvL.
Sarthak Mittal, Yoshua Bengio, and Guillaume Lajoie. Is a Modular Architecture Enough? October 2022. URL https://openreview.net/forum?id=3-3XMModtrx.
They don't have a venue.
- The protocol is not class-incremental, and the comparison is unfair.
Class-incremental learning requires predicting among all classes seen so far without task identity at test time.
GaTaP instead receives a labelled support set (64 examples per class) from the evaluation task and re-infers Γ before prediction. That is task-identity information plus test-time few-shot adaptation. The setting is closer to task-incremental or meta-learning.
Minimum fair baselines would include:
Linear probe or few-shot fine-tuning of the head on the same support set with a frozen backbone.
Task-incremental baselines with comparable information.
The gating/mask methods discussed in Related Work: HAT, PackNet, SupSup-style supermasks, MNTDP, RPS-Net, none of which are compared.
The statement that GaTaP “does not require explicit knowledge of task transitions” conflicts with requiring single-task epochs and a task-specific support set. Am I wrong?
Also, try to add more buffer-free SOTA methods and compare with them, such as:
WSN: Forget-free Continual Learning with Winning Subnetworks Haeyong Kang, ICML 2022
SNV: Shapley Neuron Values for Continual Learning: Which Neurons Matter Most? Vahedifar, ICML 2026.
-
The “modular subnetwork” story is not what is evaluated.
Tables 2 and 4 set the number of subnetworks to 1 for MNIST and CIFAR-10. The main results therefore come from a single network with per-unit (or per-spatial-location) gains, not from recombining parallel modules as Fig. 1 and §3.1 suggest. With spatial gating, Γ has C×H×W entries per layer. That makes it a high-capacity task-specific parameter fitted at test time, which further weakens the compositional interpretation.
-
Sparsity is claimed but not shown.
α is 1e-5 (MNIST, CIFAR-10) and 1e-7 (CIFAR-100). At these values the soft threshold is likely nearly inactive. The paper reports no sparsity statistics for Γ, so “sparse routing” is unsupported.
-
Internal inconsistencies about the inner loop.
The Results section and A.3.1 say a single closed-form step / single iteration is used.
Tables 2, 4 and 6 list “Gamma inference steps (train/test) 10/10.”
Algorithm 1, line 9, computes h_L “followed by Γ” but never says which Γ is used in the forward pass that produces h_L before Γ has been inferred.
K_L in Eq. 9/81 is undefined, since there is no layer above L. How y is represented (one-hot? MSE?) is also not stated.
- This weakness took so much time to identify each error, and it showed me that you didn't carefully proofread your paper. Mathematical errors and notation problems.
Laplace prior: p(Γ) ∝ exp(+α‖Γ‖₁) (Sec. 3.1) is not normalizable. It should be exp(−α‖Γ‖₁). Eq. 42 then uses the correct sign.
Posterior structure: Q(r, Γ | s, o) is per-sample, yet Γ is said to be constant across all samples of a task. The posterior over Γ should condition on the whole support set.
Noise inside losses: the random noise ε_i appears inside deterministic residuals and objectives (Eqs. 12, 13, 16, 69–72, 87, 93–94, 99–100). It should not be there. Eq. 45 also writes Σ₁ = ε₁I, conflating the noise variable with its variance.
Inconsistent gating operator: Eq. 2 uses Diag(Γ)W, while Eq. 66 uses Γ ⊙ W (vector ⊙ matrix, a dimension mismatch).
Dimensions: A.1 states W₁ ∈ R^{d₂} × R^{d₁}; it should be R^{d₁×d₀}. Sec. 3.1 says Γ_i^(k) ∈ R^{d_i}, while A.4.1 gives Γ ∈ R^{C_i} in the bullet list and R^{C×H×W} a few lines later.
Not truly bilevel: Eq. 18 writes Γ*(W), but Eq. 86 states that gradients are not propagated through Γ*. The procedure is alternating optimization, not bilevel.
Broken reference: “The full algorithm … is given in [1].”
Minor error:
Figure 1 takes roughly half a page to show a sum of row-gated matrices, which Eq. 1–2 already conveys. It could be a single-column schematic.
I believe weaknesses 1- 3 cannot be addressed during rebuttal. I can not verify the paper's results and numbers if you provide code in such a short time. The other weaknesses can be addressed. I am waiting for your response, and if I have proven I am wrong, I will likely increase my score.