- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper studies RNNs with the hidden state evolving according to and tasked with remembering continuous quantities, such as a robot's heading angle. The authors assume that the ideal recurrent updates are equivariant under the action of a Lie group , i.e.,
The paper assumes that the possible values of the remembered quantity of interest are encoded as a manifold in the state space. For example, an angle can be represented as , so the possible memory states form a circle. The "memory directions" in paper's language are the tangent directions to this manifold, corresponding to infinitesimal changes in the remembered quantity. They are identified with the tangent space of the group orbit, given by the image of the infinitesimal group action
The paper then studies three properties of these memory directions. First, it shows that under exact equivariance, they have zero exponential growth rates. Second, it derives perturbation bounds for "imperfectly equivariant" dynamics, distinguishing direct perturbations of memory directions from indirect ones through transverse directions. Third, it characterizes how the geometry of the memory representation affects noise amplification in noise recurrent dynamics.
Critical strengths
The paper includes some potentially interesting results, e.g. on how perturbations that break equivariance affect memory directions. However, their significance and implications were difficult for me to assess due to the lack of clarity.
Critical weaknesses
Poor writing, lack of clear central contribution: The introduction is very poorly written. It is not explained what problem is considered and what the used terms mean (what is memory, heading, state, etc?). The posed questions and the contributions are not clearly formulated either (which stored values are meant here? perturbations to what? what are protected directions/protection?).
The remaining paper structure is not explained, making the overall message very hard to extract. The presented results appear to be rather disconnected facts about equivariant recurrent dynamics and the tangent map , defined above.
The setting considered in the experiments is also hard or impossible to understand due to insufficient details. E.g., what are the recurrent dynamics considered in Figure 2? This is never specified.
Unclear technical/mathematical novelty: To the best of my understanding, all the results rely on classical mathematical approaches. The technical challenges and/or novelty are not sufficiently explained.
Limited connection to realistic RNNs: The analysis assumes recurrent dynamics that are equivariant under a known continuous group action, and the main experiments seem to use specifically constructed architectures for given symmetries. It is unclear to what extent these models relate to any RNNs used in practice.
Questions for authors
-
What is the main technical novelty of the results?
-
What are the concrete recurrent dynamics used in Figure 2? What is the purpose of these experiments?
-
What is the intended applicability of the framework? Are there examples of commonly used RNNs for which the assumed symmetry structure holds?