- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Continually-Recyclable Unit-Gating, a procedure for training ReLU-like RNNs in a continual learning domain. The method involves explicitly allocating units to specific tasks encountered using learnable gating parameters as an approximate L0 penalty. To avoid catastrophic forgetting, they enforce that units assigned to earlier tasks can provide input to units assigned later, but not vice versa. Using tasks drawn as sequences of synthetic dynamical systems, commonly studied cognitive RNN tasks, and some real data, they show that CRUG achieves competitive performance in DSR continual learning across many domains.
Critical strengths
The paper is well-organized, technically sound, and reasonably easy to follow. The authors demonstrates the efficacy of the proposed method across a fairly comprehensive set of benchmarks and baseline comparisons.
Critical weaknesses
The proposed method seems quite bespoke and overengineered, and by design only works for networks with threshold-linear units. Given the number of moving parts in the setup, I think some ablations would be useful to identifying what about the proposed method is doing the heavy lifting in continual learning performance. For instance, how important was hard gradient masking to prevent updates to previous units? Is it possible that some intermediate level of soft masking might perform even better?
Questions for authors
- If I am understanding correctly, the gating introduced here is applied symmetrically, in that suppressing a unit's input to other units also suppresses its own updateability. Is there any reason why separate gatings are not used for input and output?
- Why are encoder and decoder weights left untrained?
- The networks are trained with continuous gating parameters, but at the end of each task, these are thresholded. How come this thresholding process doesnt cause a mismatch in model predictions during training and after the thresholding that needs to be calibrated?
- Eq 3: is outer supposed to be max? otherwise .
4 comments
Comment 2
Q1. Symmetric gating
The reviewer understands correctly: in Eq. 4, the same scales both the state update of unit (through , , , and ) and its contribution to other units (through ). This is deliberate, since the gate encodes a single decision, namely, whether unit participates in the current task and is committed to it. With separate input and output gates, a unit with an open input gate and a closed output gate would update its state, but this state would reach neither other units nor the readout, so it could not affect the reconstruction. Conversely, a unit with a closed input gate and an open output gate would receive no input from the network. Its state would then evolve independently of the task’s dynamics and, at most, converge to a constant, whose effect on other units their bias terms already provide. A unit therefore contributes only when both gates are open, and the decision to keep it would depend on their product, which is again one gate per unit. Separate gates would also add no expressivity, because the strengths of a unit’s incoming and outgoing connections are already learned freely by the corresponding rows and columns of (and by , , and ). Whether a unit’s parameters are frozen is governed separately by the gradient masks (Eq. 6), which set the gradients of all parameters updating committed units to zero and leave those of free units unchanged.
Q2. Fixed encoder and decoder
Keeping the encoder and decoders fixed is a deliberate design choice. Our goal is to show that CRUG’s performance arises from how it allocates, protects and reuses capacity in the recurrent dynamics. Fixing the encoder and decoders places all learned structure in the latent dynamics, the component CRUG acts on. Each system’s dynamics must be learned and represented by the recurrent latent model. Trainable encoders or decoders could partly compensate for limitations or interference in the latent dynamics, which would make it harder to attribute the observed retention and reconstruction quality to the proposed method.
CRUG learns and retains all tasks that way, also keeping the number of trainable parameters smaller. Overall though, our goal here was to isolate continual learning in the latent dynamics. We now make this design choice more explicit in Appendix A.1.
Q3. Thresholding of continuous gates
Thresholding leaves the trained model unchanged, for two reasons. First, retained units (> ) are not rounded to one: their learned gate values are absorbed into the stored weights (Eqs. 10–11), so their effective parameters after commitment are identical to those used during training. Retained units therefore behave after commitment exactly as they did during training. Second, released units ( <= ) contribute nothing during training: the stretched-and-clipped gates (Eq. 3; Louizos et al., 2018) allow exact zeros, and the capacity penalty drives unneeded gates toward this limit. Since almost all of these gates are already 0 during training, the corresponding units did not contribute to the dynamics before commitment either, and releasing them does not change the model.
We now also verified this empirically. Across all three DSR task sequences, 65–92% of the gates per task equal 0 exactly. Only 4 of 12,295 released gates (over all runs on the three DSR task groups) are nonzero (new Fig. 13B). Accordingly, and remained largely unchanged between the model before and after commitment, which confirms that commitment does not alter the reconstruction (new Fig. 13A).
Thresholds up to 0.7 yield the same or very similar reconstruction quality, so the result does not depend on the exact choice of threshold. Performance degrades only at , which also releases units with gate values between 0.5 and 0.9, and when the absorbed gate values are replaced by binary gates (both p = 0.002; new Fig. 13D). This shows that the absorption of gate values is what keeps the trained and the committed model consistent, so no calibration step is needed.
Q4. Eq 3
We thank the reviewer for checking this carefully. Eq. 3 is correct as written:
The outer operation is a minimum and the inner one a maximum. With , the stretched sigmoid takes values in [-0.1, 1.1]. The inner maximum clips values below 0, so that lies in [0, 1.1), and the outer minimum clips values above 1, so that lies in [0, 1]. The stretch beyond [0, 1] followed by clipping allows gates to take the values 0 and 1 exactly (Louizos et al., 2018). To avoid ambiguity, we now write Eq. 3 as , consistent with Appx. A.4.
Thanks to the authors for their clarifications and additional experiments; the ablation experiments are particularly helpful. I still have (admittedly subjective) reservations about the elegance of the approach, but nonetheless am happy to maintain my positive score.
Comment 3
We thank the reviewer for their response and positive feedback and appreciate their continued support of our work.
Comment 1:
We thank the reviewer for the positive assessment and the precise technical questions, and we appreciate that the reviewer finds the paper clear, technically sound and comprehensively evaluated. We address each point below.
W1a (threshold-linear units): CRUG does not rely on threshold-linear units. The gates (Eq. 4) and gradient masks (Eq. 6) act on rows and columns of the recurrent parameters, independently of the activation function, so CRUG applies to any RNN whose units interact through weight matrices. To demonstrate this, we applied CRUG to a 160-unit LSTM on the same four-system sequence with the same protocol (new Appx. A.11). Trained sequentially without a continual learning method, the LSTM suffers catastrophic forgetting. With CRUG, it retains all four systems and reaches an overall Dstsp of 2.14, even better than interleaved training (3.07), while committing 103 of 160 units.
W1b. Design of CRUG and ablations
The components of CRUG follow from one requirement of DSR: the recurrent dynamics of previously learned systems must remain exactly intact, because small changes to recurrent parameters can qualitatively alter the attractor generated during autonomous simulation, even when one-step predictions remain nearly unchanged (Eisenmann et al., 2023; NeurIPS). To test which components are needed for this, we followed the reviewer’s suggestion and added ablations of each of CRUG’s four components: allocation, commitment, protection and forward transfer (new Experiments section 4.3, Appendix A.13 and A.16, Figs. 12–15).
Hard versus soft masking (gradient protection). As asked by the reviewer, we replaced the hard gradient mask by a soft mask that scales the gradients of committed units by ϵ∈{0.001,0.01,0.1,1}, with everything else unchanged (Fig. 14). No intermediate level of masking worked: only hard masking (ϵ=0) preserved earlier systems. Already at ϵ=0.001, earlier systems deteriorated gradually, and for ϵ >= 0.01 they were lost almost immediately once training on the next task started. The following table summarizes all further ablations.
In brief, the L0 gates were required for compact allocation, since with an L1 penalty the first task alone consumed almost the entire network. Commitment left the trained model unchanged as long as the gate values were absorbed exactly, and forward transfer was used more strongly the more dynamical structure the systems shared.
What does the heavy lifting? Three components are each necessary: hard protection prevents forgetting, exact absorption of the gate values keeps the committed dynamics identical to the trained ones, and the L0 gates keep the allocation compact. Forward transfer is optional. It improves capacity efficiency when systems share dynamical structure, but CRUG also learns sequentially without it (Fig. 3).