acceptodds
Under review as a conference paper at ICLR 2027

The RNN Homunculus: Learning to Gate Long-Term Memory

Abstract

Learning long-range dependencies with recurrent neural networks (RNNs) is difficult because of the vanishing gradient problem. Gated recurrent neural networks like GRUs and LSTMs address this problem and are commonly explained by their ability to create additive copy paths through time. This explanation is correct, but incomplete: useful RNN gate values must themselves be learned, which is typically done with another RNN. We study this gate-learning problem through carefully controlled sequence classification tasks. Our analysis shows that gated architectures struggle when all training samples require long-range memory and thus equally suffer from the vanishing gradient problem. We intervene by adding short-dependency samples which let gated models learn solutions that capture long-dependency samples, but also generalize across different class labels. When properly learned, gated RNNs match or even exceed the test accuracy of modern architectures like Mamba and xLSTM. Overall, our results highlight gate learning as a central part of learning long-range memory and show how training data can support this process by exposing the same task over shorter dependencies.

Primary area:
learning on time series and dynamical systems
Keywords:
Deep Learning, Recurrent Neural Network, Gated Neural Network

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CeXVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper looks at why LSTMs and GRUs still have problems learning long-range dependencies, even though their gates are supposed to help with this. The authors argue that gates themselves need to be learned, which is also difficult for long sequences.Adding some short-dependency examples during training greatly improves performance on longer sequences. They also show some transfer across classes, and results comparable to Mamba and xLSTM. Interesting observation, but their experiments are limited to fairly simple synthetic tasks. Was it gate learning that helped improve results?

Critical strengths

Interesting observation that gates themselves need to be learned and can suffer from vanishing gradients. Clear, well-controlled experiments showing large improvements from short-dependency training examples. Good analysis of gradient propagation and transfer across classes, with comparisons to modern architectures.

Critical weaknesses

Only simple synthetic XOR tasks. How well would that generalize to actual problems. No causal demonstration that improved gate learning, rather than other optimization effects, explains the results.

Questions for authors

isolate the causal role of gate learning

Evaluate with more complex tasks, for example on sequential MNIST? Something with more complex memory management

1 comment
AuthorsOpenReview ↗

Freezing to show gate causality and sequential MNIST

Thank you for your time and constructive feedback!

1. sequential MNIST task

Only simple synthetic XOR tasks. How well would that generalize to actual problems.

Evaluate with more complex tasks, for example on sequential MNIST? Something with more complex memory management

We extend to sequential MNIST [4], with the LSTM setup from [5] and RNN, GRU from [6].

We call a training image easy if a vanilla RNN can recognizie it. We removed these easy images from the training set through cross-validation. Additionally, we made a control training set where we removed the same number of images of each digit, at random.

Without the easy images (easy removed set), the LSTM and GRU have lower accuracy than with the full training set and with random removal.

Test accuracy (%)FullRandom removedEasy removed
LSTM98.5±0.296.3±0.493.3±2.0
GRU99.0±0.197.7±0.395.7±1.3

Gated RNNs thus depend on their easy samples for a more complex task too, as they do on temporal XOR. We added this experiment to the paper, in Section 4.6.

2. Causal role of gate learning

No causal demonstration that improved gate learning, rather than other optimization effects, explains the results.

isolate the causal role of gate learning

To further isolate the causal role of gate learning we added component-freezing experiments. Under mixed hard+short training, we freeze either the gate parameters or the candidate-state parameters together with the classification head.

The following table shows avg test accuracy (%) at sequence length 120 over 9 runs.

ModelFully trainableOnly gates trainableFrozen gates
GRU85.8682.0360.36
LSTM87.6171.8855.74

Training only gates achieves substantial long-range accuracy despite other components being fixed. Freezing gates leads to lower accuracy at long lengths. These results provide causal evidence that gate adaptation contributes to long-range learning.

We added the full freezing experiment in Section 4.4 and Fig. 7, where results across more sequence lengths are provided. We also revised Section 4.3 to not overclaim about gate learning.

References:

[4] Le et al., A simple way to initialize recurrent networks of rectified linear units, arXiv 2015.

[5] Wisdom et al., Full-capacity unitary recurrent neural networks, NeurIPS 2016.

[6] Turkoglu et al., Gating revisited: Deep multi-layer RNNs that can be trained, TPAMI 2021

Reviewer_meazOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how gated recurrent neural networks learn to preserve information over long temporal dependencies, focusing on the learning of gate values that enable effective memory retention. The paper introduces the RNN Homunculus perspective, arguing that although gating mechanisms provide additive paths that can mitigate vanishing gradients, learning appropriate gate configurations remains challenging. The hypothesis is investigated through controlled temporal XOR classification tasks involving vanilla RNNs, GRUs, LSTMs, and more recent recurrent architectures. The experiments examine the effects of mixing short- and long-dependency samples, the transfer of learned behaviors across different class groups, and the evolution of gradients and gating dynamics during training.

Critical strengths

The main strength of the paper is its carefully controlled experimental analysis of long-range learning in gated recurrent networks. The experiments provide interesting evidence that short-dependency training can improve long-range generalization, including partial transfer across different class groups.

Critical weaknesses

The two main weaknesses of the paper concern novelty and experimental evidence.

Limited theoretical novelty. The so-called RNN Homunculus problem appears to be largely a reinterpretation of established challenges in vanishing gradients and temporal credit assignment, rather than a fundamentally new theoretical problem. The mathematical analysis primarily relies on well-known properties of gradient propagation, while the benefits of training with shorter dependencies are closely related to existing ideas in curriculum learning. More importantly, the experiments do not establish a causal connection between the proposed gate-learning mechanism and the observed improvements, leaving the central hypothesis insufficiently validated.

Limited experimental scope. The entire evaluation relies on simple synthetic temporal XOR tasks with small recurrent models and relatively short sequences. Although these experiments are well controlled, they provide limited evidence that the proposed interpretation extends to more complex sequence modeling problems. Given that the paper aims to provide a broader understanding of long-term memory learning in recurrent architectures, this restricted evaluation significantly limits the relevance and impact of its conclusions.

Additional concerns. The positioning with respect to curriculum learning and related work on recurrent optimization could be strengthened. The homunculus analogy also introduces some conceptual ambiguity without clearly identifying a distinct mathematical problem. The comparison with Mamba and xLSTM is difficult to interpret beyond the specific synthetic setting, and some claims about their relative performance appear broader than the experimental evidence supports.

Questions for authors

The following questions concern the novelty of the proposed perspective and the extent to which the experimental evidence supports its main conclusions.

Q1. Theoretical novelty. What fundamentally distinguishes the proposed RNN Homunculus problem from the established vanishing gradient and temporal credit assignment problems? Given that the mathematical analysis largely follows standard gradient propagation arguments, what new theoretical understanding does the proposed formulation provide?

Q2. Evidence for the proposed mechanism. The experiments demonstrate the benefits of short-dependency training, but how do they establish that these improvements are specifically caused by learning effective gating policies? What evidence distinguishes this interpretation from more general optimization or curriculum learning effects?

Q3. Scope and generalizability. Given that all experiments are conducted on closely related synthetic temporal XOR tasks with very small models, what evidence supports the broader conclusions about long-term memory learning in recurrent architectures? How can the proposed mechanism be distinguished from behaviors specific to this simplified experimental setting?

1 comment
AuthorsOpenReview ↗

Novelty, freezing to show proposed mechanism, and sequential MNIST

Thank you for your time, effort, and detailed feedback!

1. Theoretical novelty and relation to prior work

What fundamentally distinguishes the proposed RNN Homunculus problem from the established vanishing gradient and temporal credit assignment problems?

The homunculus analogy also introduces some conceptual ambiguity…

Exactly! The gradient propagation analyses are standard, which makes it interesting to observe that nobody did this before. The homunculus perspective is to point out gates are learnt with structure identical to an RNN, which with standard gradient analysis is proved to have vanishing gradient and temporal credit assignment problems. Yet no work seem to apply the same analysis to the gates, which is exactly why we used similar analysis: the problem already exists with the mathematical foundations we already have, only we apply it to the gate-RNN as well, which has not been done before. We clarified the introduction of the paper.

The positioning with respect to curriculum learning and related work on recurrent optimization could be strengthened.

Agreed. There are existing researches on gate saturation and initialization (Gu et al., 2020; Tallec and Ollivier, 2018), and how curricula shape long-memory mechanisms (Khajehabdollahi et al., 2024). Our analysis shows gate learnability is a problem that can happen before sigmoid saturation. Our work connects optimization and curriculum perspectives through the gradients available to gates. Short-dependency samples shorten the long recurrent paths that attenuate gate-learning signals, and provide these signals for shared gate functions. This is a gradient-based explanation for recurrent optimization and curriculum learning. We carefully managed confounders in our controlled experiments to test this connection. We added more references and detailed discussions in the revised paper, Section 2. We also added more experiments related to recurrent optimization in Section 4.4.

2. Evidence for the proposed mechanism

What evidence distinguishes this interpretation from more general optimization or curriculum learning effects?

We added more interventions based on freezing weights, restricting which components can learn, this makes the causal relationship more strict.

During training with short and long samples mixed, we freeze either the gate parameters or the candidate-state parameters together with the classification head. At sequence length 120, mean accuracy over 9 runs is:

ModelFully trainableOnly gates trainableFrozen gates
GRU85.8682.0360.36
LSTM87.6171.8855.74

Freezing gates leads to a larger loss of accuracy at long lengths. These interventions strengthen the evidence for a contribution from gate learning. We added the full freezing experiment in Section 4.4 and Fig. 7, where results across more sequence lengths are provided. We also revised Section 4.3 to not overclaim about gate learning.

3. Scope and generalizability

...evidence that the proposed interpretation extends to more complex sequence modeling problems.

...what evidence supports the broader conclusions about long-term memory learning in recurrent architectures?

We agree that experiments with unknown confounders would strengthen the paper, so we extend to sequential MNIST [4], with the LSTM setup from [5] and RNN, GRU from [6].

We call a training image easy if a vanilla RNN can recognize it. We removed these easy images from the training set through cross-validation. Additionally, we made a control training set where we removed the same number of images of each digit, at random.

Without the easy images (easy removed set), the LSTM and GRU have lower accuracy than with the full training set and with random removal.

Test accuracy (%)FullRandom removedEasy removed
LSTM98.5±0.296.3±0.493.3±2.0
GRU99.0±0.197.7±0.395.7±1.3

Gated RNNs thus depend on their easy samples for a more complex task too, as they do on temporal XOR. We added this experiment to the paper, in Section 4.6.

...some claims about their relative performance appear broader than the experimental evidence supports.

We made clearer the scope of modern architecture experiments in Section 4.5 and removed possibly overly broad claims.

References:

[4] Le et al., A simple way to initialize recurrent networks of rectified linear units, arXiv 2015.

[5] Wisdom et al., Full-capacity unitary recurrent neural networks, NeurIPS 2016.

[6] Turkoglu et al., Gating revisited: Deep multi-layer RNNs that can be trained, TPAMI 2021

Reviewer_u8pQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how gated recurrent networks learn the gate configurations needed for long-range memory. A conditional gradient analysis shows that early gate-learning signals can also be attenuated by recurrent Jacobian products, including for input-only gates. Temporal-XOR experiments replace a fraction of hard training examples with equally long examples whose informative tokens lie near the final prediction. This improves hard-only test accuracy and partially transfers across token groups. The paper interprets the result as learning reusable selective-memory behavior from short dependencies.

Critical strengths
  1. The setup is simple and well controlled. Sequence length and training-set size stay fixed.
  2. The partial transfer across token groups is interesting, and each setting is tested over 25 runs.
Critical weaknesses

However, I think the proposed method has following drawbacks:

  1. The novelty is not clear enough. Prior work already studies gate learnability [1] and curriculum benefits for long-memory learning [2]. The fixed-length setup here is different, but I am not sure what new insight it adds.
  2. Easy examples can help train the whole model. The current probes do not show that learning better gates is the main reason for the gains.
  3. The experiments are limited to synthetic XOR tasks. It is also unclear whether the gains remain with chrono initialization [3]. References

[1] Gu et al. Improving the Gating Mechanism of Recurrent Neural Networks. ICML, 2020.

[2] Khajehabdollahi et al. Emergent Mechanisms for Long Timescales Depend on Training Curriculum and Affect Performance in Memory Tasks. ICLR, 2024.

[3] Tallec and Ollivier. Can Recurrent Neural Networks Warp Time?. ICLR, 2018.

Questions for authors
  1. Do the gains remain with chrono initialization or retention-oriented gate biases?
  2. What happens if you freeze or reset the gates, compared with doing the same to other components?
1 comment
AuthorsOpenReview ↗

Novelty, freezing to show gate causality, initialization experiment, and sequential MNIST

Thank you for your time, effort, and engagement!

1. Related work & novelty

Prior work already studies gate learnability [1] and curriculum benefits for long-memory learning [2].

...chrono initialization [3].

Thanks for the refs!

  • [1] addresses gate saturation; we differ because the gradient reaching early gate can already be attenuated before the saturation.
  • [2] show curricula shaping the mechanisms for long memory. Through more controlled settings (your mentioned fixed length for example), we try to better understand why curriculum works and connect it to our gradient analysis.
  • [3] motivate gate bias initialization, we now add new experiments to show short support remains useful with initialization, see experiments in Section 4.4 in the revised paper.

We added all 3 references and clarified the connections in our revised paper's related work.

2. Gate learning & component freezing

Easy examples can help train the whole model.

What happens if you freeze or reset the gates...

Yes, even stronger causality can be shown by your suggestion. We freeze gate parameters or other components (candidate-state parameters and classification head) and retrain the models and compare with no freezing. This shows the causal contribution of gate adaptation to long-range learning, table below shows mean test accuracy (%) over 9 runs.

Model / frozen componentsT=20T=60T=120
GRU: none frozen97.1191.7485.86
GRU: gate frozen99.9780.1860.36
GRU: candidate + head frozen81.5484.1982.03
LSTM: none frozen100.0100.087.61
LSTM: gate frozen99.9964.4955.74
LSTM: candidate + head frozen88.4985.4071.88
Vanilla RNN: none frozen97.1749.6449.85

At long lengths, freezing gates is more detrimental than freezing other components. These results show direct causal relation how adapting gate parameters supports learning long-range solutions. We added the full freezing experiment in Section 4.4 and Fig.7, and revised Section 4.3 to not overclaim about gate learning.

3. Retention-oriented initialization

Do the gains remain with chrono initialization or retention-oriented gate biases?

Yes, the gains remain. We initialize both bias terms of GRU update gate and LSTM forget gate to 2. The following table shows test accuracy of different model init and data setting at different sequence length, averaged over 25 runs.

Model / settingT=10T=70T=140T=200
GRU: retention biases, hard-only100.0093.1358.1055.54
GRU: default initialization, hard+short100.0095.0390.8389.31
GRU: retention biases, hard+short100.00100.00100.0098.95
LSTM: retention biases, hard-only100.00100.0086.3674.10
LSTM: default initialization, hard+short100.0098.1693.8290.72
LSTM: retention biases, hard+short100.0099.9992.8896.92

Retention biases are not sufficient at long lengths, there the benefits of short-dependency samples are shown. The full initialization experiment we added in Section 4.4 and Fig. 6.

4. sequential MNIST task

The experiments are limited to synthetic XOR tasks.

Agreed. We added sequential MNIST [4] as a setting with unknown confounders. We sue the LSTM setup from [5] and RNN, GRU from [6].

We call a training image easy if a vanilla RNN cannot recognize it. We removed these easy images from the training set through cross-valdiation. Additionally, we made a control training set where we removed the same number of images of each digit, at random.

Without the easy images (easy removed set), the LSTM and GRU have lower accuracy than with the full training set and with random removal.

Test accuracy (%)FullRandom removedEasy removed
LSTM98.5±0.296.3±0.493.3±2.0
GRU99.0±0.197.7±0.395.7±1.3

Gated RNNs thus depend on their easy samples for a more complex task too, as they do on temporal XOR. We added this experiment to the paper, in Section 4.6.

References:

[4] Le et al., A simple way to initialize recurrent networks of rectified linear units, arXiv 2015.

[5] Wisdom et al., Full-capacity unitary recurrent neural networks, NeurIPS 2016.

[6] Turkoglu et al., Gating revisited: Deep multi-layer RNNs that can be trained, TPAMI 2021

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.