- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper looks at why LSTMs and GRUs still have problems learning long-range dependencies, even though their gates are supposed to help with this. The authors argue that gates themselves need to be learned, which is also difficult for long sequences.Adding some short-dependency examples during training greatly improves performance on longer sequences. They also show some transfer across classes, and results comparable to Mamba and xLSTM. Interesting observation, but their experiments are limited to fairly simple synthetic tasks. Was it gate learning that helped improve results?
Critical strengths
Interesting observation that gates themselves need to be learned and can suffer from vanishing gradients. Clear, well-controlled experiments showing large improvements from short-dependency training examples. Good analysis of gradient propagation and transfer across classes, with comparisons to modern architectures.
Critical weaknesses
Only simple synthetic XOR tasks. How well would that generalize to actual problems. No causal demonstration that improved gate learning, rather than other optimization effects, explains the results.
Questions for authors
isolate the causal role of gate learning
Evaluate with more complex tasks, for example on sequential MNIST? Something with more complex memory management
Freezing to show gate causality and sequential MNIST
Thank you for your time and constructive feedback!
1. sequential MNIST task
We extend to sequential MNIST [4], with the LSTM setup from [5] and RNN, GRU from [6].
We call a training image easy if a vanilla RNN can recognizie it. We removed these easy images from the training set through cross-validation. Additionally, we made a control training set where we removed the same number of images of each digit, at random.
Without the easy images (easy removed set), the LSTM and GRU have lower accuracy than with the full training set and with random removal.
Gated RNNs thus depend on their easy samples for a more complex task too, as they do on temporal XOR. We added this experiment to the paper, in Section 4.6.
2. Causal role of gate learning
To further isolate the causal role of gate learning we added component-freezing experiments. Under mixed hard+short training, we freeze either the gate parameters or the candidate-state parameters together with the classification head.
The following table shows avg test accuracy (%) at sequence length 120 over 9 runs.
Training only gates achieves substantial long-range accuracy despite other components being fixed. Freezing gates leads to lower accuracy at long lengths. These results provide causal evidence that gate adaptation contributes to long-range learning.
We added the full freezing experiment in Section 4.4 and Fig. 7, where results across more sequence lengths are provided. We also revised Section 4.3 to not overclaim about gate learning.
References:
[4] Le et al., A simple way to initialize recurrent networks of rectified linear units, arXiv 2015.
[5] Wisdom et al., Full-capacity unitary recurrent neural networks, NeurIPS 2016.
[6] Turkoglu et al., Gating revisited: Deep multi-layer RNNs that can be trained, TPAMI 2021