acceptodds
Under review as a conference paper at ICLR 2027

Symmetry, Defects, and Diffusion in Continuous-Memory Recurrent Networks

Abstract

Recurrent networks that store continuous variables must preserve differences between remembered values despite imperfect updates and state noise. We separate three problems: identifying memory directions, bounding their deterministic distortion, and controlling noise in the decoded coordinates. For ideal dynamics, exact equivariance for each input and uniform nondegeneracy along the trajectory prevent exponential growth or decay in the directions generated by continuous symmetry. For perturbed dynamics, local and finite-horizon bounds distinguish direct memory damage from leakage out of and back into its tangent space. With a fixed transverse contraction gap and no direct damage, the leakage contribution is second order. For noisy dynamics, a decoder-aware local covariance bound yields geometry design rules under a fixed coordinate metric and representation budget: equally scaled orthogonal directions are optimal for isotropic noise with least-squares decoding, whereas unequal noise can favor unequal allocation. In twenty fresh paired training units, covariance regularization improves noisy two-angle memory while meeting a specified clean-error equivalence margin; direct noise training incurs a clean-error tradeoff. In coupled four- and eight-angle integrators, the preference between anisotropic and isotropic regularization reverses when evaluation noise changes. These results connect symmetry-protected memory to measurable perturbation bounds and noise-specific representation design.

Primary area:
learning on time series and dynamical systems
Keywords:
recurrent neural networks, continuous memory, equivariance, dynamical systems, perturbation bounds, representation geometry, noise robustness

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_2pr9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies RNNs with the hidden state xtx_t evolving according to xt+1=Fut(xt).x_{t+1}=F_{u_t}(x_t). and tasked with remembering continuous quantities, such as a robot's heading angle. The authors assume that the ideal recurrent updates FuF_u are equivariant under the action of a Lie group GG, i.e., Fu(g⋅x)=g⋅Fu(x).F_u(g\cdot x)=g\cdot F_u(x).

The paper assumes that the possible values of the remembered quantity of interest are encoded as a manifold in the state space. For example, an angle θ\theta can be represented as x(θ)=r(cos⁡θ,sin⁡θ)x(\theta)=r(\cos\theta,\sin\theta), so the possible memory states form a circle. The "memory directions" in paper's language are the tangent directions to this manifold, corresponding to infinitesimal changes in the remembered quantity. They are identified with the tangent space of the group orbit, given by the image of the infinitesimal group action Axξ=dds∣s=0exp⁡(sξ)⋅x.A_x\xi=\left.\frac{d}{ds}\right|_{s=0}\exp(s\xi)\cdot x.

The paper then studies three properties of these memory directions. First, it shows that under exact equivariance, they have zero exponential growth rates. Second, it derives perturbation bounds for "imperfectly equivariant" dynamics, distinguishing direct perturbations of memory directions from indirect ones through transverse directions. Third, it characterizes how the geometry of the memory representation affects noise amplification in noise recurrent dynamics.

Critical strengths

The paper includes some potentially interesting results, e.g. on how perturbations that break equivariance affect memory directions. However, their significance and implications were difficult for me to assess due to the lack of clarity.

Critical weaknesses

Poor writing, lack of clear central contribution: The introduction is very poorly written. It is not explained what problem is considered and what the used terms mean (what is memory, heading, state, etc?). The posed questions and the contributions are not clearly formulated either (which stored values are meant here? perturbations to what? what are protected directions/protection?).

The remaining paper structure is not explained, making the overall message very hard to extract. The presented results appear to be rather disconnected facts about equivariant recurrent dynamics and the tangent map AxA_x, defined above.

The setting considered in the experiments is also hard or impossible to understand due to insufficient details. E.g., what are the recurrent dynamics considered in Figure 2? This is never specified.

Unclear technical/mathematical novelty: To the best of my understanding, all the results rely on classical mathematical approaches. The technical challenges and/or novelty are not sufficiently explained.

Limited connection to realistic RNNs: The analysis assumes recurrent dynamics that are equivariant under a known continuous group action, and the main experiments seem to use specifically constructed architectures for given symmetries. It is unclear to what extent these models relate to any RNNs used in practice.

Questions for authors
  1. What is the main technical novelty of the results?

  2. What are the concrete recurrent dynamics used in Figure 2? What is the purpose of these experiments?

  3. What is the intended applicability of the framework? Are there examples of commonly used RNNs for which the assumed symmetry structure holds?

Reviewer_rbqPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on RNNs that store a continuous variable, e.g., a ring attractor network. The authors study how symmetry, perturbations, and noise affect the memory in these networks.

Critical strengths
  • Understanding how RNNs maintain continuous memories is an interesting and relevant problem.
Critical weaknesses
  • I found the paper very difficult to follow and struggled to identify its main contribution. The presentation is very poor, and most of the text sounds like the output of an LLM. I was unable to extract anything meaningful from this paper despite spending some time reading it carefully.
Questions for authors
  • Could the authors explain in simple words the main contributions of their paper? Why is it relevant, and why should the ML community care?
Reviewer_VeDPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

From what I was able to understand, this paper attempts to answer some questions related to continuous memory in recurrent neural networks, i.e., how and under what conditions a recurrent neural network can maintain a continuous value in its memory faithfully and how to make the representation robust to noise. The paper provides several theorems and bounds related to these questions.

Critical strengths

Typically, I would say that understanding memory mechanisms in recurrent networks can provide a window into how neural circuits -- both artificial and biological -- represent and maintain memories of certain concepts while remaining robust to the effects of noise. This is useful for both neuroscience and interpretability. However, this is a strength of the topic, and not of this paper. In its current form, I am unable to evaluate properly whether this paper has any strengths other than the chosen problem domain.

Critical weaknesses
  • Unfortunately, the paper is just not understandable in its current form. While I have a fairly reasonable background in dynamical systems, RNNs, continuous attractors, etc., none of the details in this paper were clear to me. This is absolutely critical and almost entirely influences my decision to reject this paper. The claims are not clearly presented and it's unclear what I should take away from this paper. For example, Ságodi et al. (2024) provide a great account of the robustness of continuous attractor "memory" networks -- but it's unclear how this work relates to that, what it adds on top, what the exact evidence and experiments are, etc. I think this should be reason enough to reject this paper.
  • There is unfortunately also a potential anonymity concern. The related work section refers to a preprint, Mo (2026), and specifies what this work adds on top of it. Curious to find any way to understand what was going on here, I went to read Mo (2026) at the linked URL: https://arxiv.org/abs/2605.03338... only to end up on a preprint page with the exact same title as this submission and what seems to be like the exact same content. The cited preprint happens to be v1 of the same paper with a slightly different title (although the URL doesn't specifically pin that version). This seems like a clear breach of the double-blind review policy, and in conjunction with my previous comment, I believe this to be a result of improper and poor verification on the authors' part.

I think the reasons above should be more than enough evidence to reject this paper, and while I was unable to assess the scientific claims presented in this manuscript, I lean strongly towards rejecting it in its current form for the reasons above.

Questions for authors

I don't think these questions can change my assessment per se but I am genuinely curious if the authors can answer me on:

  • What is the claim here, stated very clearly?
  • What is the exact supporting evidence that isn't shrouded in hard-to-parse maths?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.