acceptodds
Under review as a conference paper at ICLR 2027

Attractor Dynamics of Energy Transformer

Abstract

Energy Transformer (ET) performs image completion by energy descent over image tokens, but its energy does not explicitly respect the translational structure of the image grid, limiting its ability to form continuous attractors. We introduce the Attractor Energy Transformer (AET), which incorporates a translation-invariant spatial kernel on the toroidal token grid . We prove that the resulting spatial energy is invariant to cyclic translations, giving rise to a two-dimensional critical manifold of equivalent states. We further show that a rectangular boundary acts as a pinning perturbation that induces a restoring force along this manifold. On COCO2014 image completion, AET consistently outperforms ET, with the improvement persisting when the kernel topology is fixed. A sliding-window experiment provides further evidence for the theoretical analysis. The gap between the toroidal and rectangular topologies is smallest for an interior mask and larger at every offset that puts the masked region on the identified edge, consistent with the pinning predicted for the rectangle.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
energy-based models, energy landscape, torus, continuous attractor, energy transformer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pxLAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces Attractor Energy Transformer (AET), which incorporates a translation-invariant spatial kernel K\mathcal{K} on the toroidal token grid T2\mathbb{T}^2, and outperforms Energy Transformer on COCO2014 image completion.

Critical strengths

The argument for incorporating translation symmetry into the energy function of Transformers is potentially meaningful.

Critical weaknesses
  1. The problem targeted by this work is limited in significance and scope. It presents only marginal improvements over Energy Transformer (ET) on image completion, which is a fairly peripheral application. Furthermore, it offers little valuable insight into why these improvements are essential or how they will have a larger influence.
  2. The technical soundness and motivation are questionable. The paper relies on a relatively unconventional tool, geometric analysis on torus, without sufficient and effective contextualization within related literature, making it hard to understand. For instance, Figure 1 lacks sufficient annotations and a clear, detailed caption.
  3. The overall organization and writing quality need substantial improvement. Issues include, but are not limited to: insufficient explanation of how the spatial energy term in Equation 11 interacts with the other components, a lack of clear contrast between the token dynamics in Section 3.2 and standard ET, and unclear purposes behind the experiments presented in Sections 5.2 and 5.3.
Questions for authors
  1. Could the authors elaborate on the motivation behind the statement in lines 30–32 more clearly and specifically: “As a result, the energy does not possess an explicit translational symmetry, and its attractors are not naturally organized into continuous families of translated state”? Why does this limitation arise, and why is addressing it important?
  2. Why introduce the kernel of the torus T2\mathbb{T}^2 in lines 35-36? The transition to its introduction is rather unnatural and unjustified.
  3. How does the theoretical analysis in Section 4 bring any new understanding or inform any new design, particularly in the context of image completion?
  4. Why is CDAT not compared as a baseline, as the authors point out in Section 6.2 that it is the closest work?
5 comments

We thank the reviewer for the candid assessment. We address the specific questions below and have revised the presentation to make the scope and contribution of the paper clearer. The contribution of the paper is a controlled analysis of how a translation invariant spatial kernel changes the energy landscape and the boundary behavior of ET, together with the completion results that follow from it.

Q1. Why the lack of translational symmetry in ET is significant

Image completion requires restoring missing patches from their spatial context, so the model must represent two dimensional spatial relationships. In standard ET, attention is content based and permutation equivariant. Positional encodings inject order, but the energy has no explicit relative displacement interaction and therefore no flat directions along translations. As a result, the completion can be sensitive to small spatial shifts of the content.

AET adds a kernel K(r−r′)\mathcal K(\mathbf r-\mathbf r^{\prime}) that depends only on relative displacement. This provides a uniform spatial prior and, in the idealized case, a flat valley along translations. The 0.9 dB gain over ET in Table 1 is modest. We agree it is not the sole contribution of the paper. The main point is the mechanism. The kernel changes the geometry of the spatial energy, and this change is measurable at the boundary. The paper is an analysis of that mechanism, not a claim of a large performance jump.

Q2. Why introduce the torus T2\mathbb T^2

On a rectangle, a discrete convolution is not translation invariant at the boundary. Edge tokens have fewer neighbors than interior tokens. On the torus, the kernel is exactly circulant, so every token has the same neighborhood. The torus is therefore a computational topology on the token grid, not an assumption that natural image content is periodic.

The rectangle is then modeled as a symmetry breaking perturbation of the torus. This separation is what makes the boundary effect measurable. The unperturbed problem has the translation symmetry, and the perturbation is the boundary term PP. This is the logical role of the torus in Section 4.3, and we will make the transition more explicit in the revision.

Q3. What the theoretical analysis contributes

The theory is an idealized account of the mechanism. The trained model is an approximate realization, and the experiments test its predictions. Its role is to motivate the translation-invariant kernel and the boundary experiment.

  1. Theorem 1 identifies the relative displacement condition for translation invariance. It motivates the toroidal kernel.
  2. Theorem 2 gives a sufficient condition for a 2D critical manifold. The condition is a critical field with independent spatial derivatives and a positive normal gap. The trained model realizes the approximate critical field, and Appendix A.8 reports the actual near neutral normal spectrum.
  3. Theorem 3 contains an exact statement and a conditional one. The projected identity Ma˙=−∇p\mathbf M\dot{\mathbf a}=-\nabla p is exact on the orbit and does not require the normal gap. It predicts the boundary pinning that the cyclic roll and sliding window experiments measure. The persistence of a nearby slow manifold and the O(ϵ2)O(\epsilon^2) reduction are conditional on ∥P∥/∣μ∣≪1\|P\|/|\mu|\ll1.

The theory is therefore explanatory and design guiding. The performance gain comes from the translation-invariant spatial kernel that the theory motivates, while the theory explains why the boundary effect appears and how to measure it.

Q4. Why CDAT is not used as a baseline

CDAT is designed for graph tasks with excitation inhibition modulation. Its interaction geometry is a graph, not a regular 2D grid, and its readout and training objective are task specific. A direct image completion baseline would require a nontrivial adaptation, namely grid to graph construction, a different readout and a different objective. Such an adaptation would not isolate the effect of the spatial kernel. Section 6.2 clarifies this. We agree that the main experiments are limited to COCO2014. The STL10 results are in Appendix A.11.

Reviewer_pxLAOpenReview ↗

Comments on the response

Thank you for the response.

I have a few further questions: How does the update differ from ET exactly? What is the rationale behind Equation 5 in Section 3.2? I don’t see any justification.

We thank the reviewer for the follow-up.

Equation 5 is a definition of the AET dynamics, so it is a construction rather than a theorem. It is CANN-inspired, as motivated in Section 1. The first two terms are a linear interaction on the torus, a leak term plus the translation-invariant kernel, and the last term is the ET update on the normalized field. Appendix A.1 derives the first two terms as the negative functional gradient of FspF_{\mathrm{sp}}, which is the rationale for this form.

The difference from ET is exact. In our notation, ET updates the token field by

=-\frac{\partial E}{\partial \mathbf{g}(\mathbf{r},t)},$$ with $E=E^{\mathrm{ATT}}+E^{\mathrm{HN}}$. AET updates it by $$\tau\frac{\partial \mathbf{x}(\mathbf{r},t)}{\partial t} =-\mathbf{x}(\mathbf{r},t) +\int_{\mathbb{T}^{2}}\mathcal{K}(\mathbf{r}-\mathbf{r}')\,\mathbf{x}(\mathbf{r}',t)\,d\mathbf{r}' -\frac{\partial E}{\partial \mathbf{g}(\mathbf{r},t)}.$$ The attention and Hopfield terms are unchanged. The difference is exactly the two additional terms $-\mathbf{x}(\mathbf{r},t)$ and $\int_{\mathbb{T}^{2}}\mathcal{K}(\mathbf{r}-\mathbf{r}')\,\mathbf{x}(\mathbf{r}',t)\,d\mathbf{r}'$. Removing these two terms recovers the ET update.
Reviewer_5vuyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focused on Energy Transformer (proposed by Hoover, 2023), and proposed Attractor Energy Transformer (AET) where added a translation-invariant spatial kernel, and proved three critical theorems:

  1. F_sp is invariant under cyclic translation
  2. The translation orbit of a critical field is a two-dimensional critical manifold whose tangent space lies in the Hessian kernel
  3. A rectangular boundary acts as a symmetry-breaking perturbation whose projection along the orbit is gradient flow on a pinning potential p(a) And 3 experiments on COCO2014 were conducted to support the theoritical results, translation-invariant kernels create flat directions (continuous attractors), rectangular boundaries break this symmetry and induce a measurable pinning force, showing that topological choices in the spatial part of an energy-based model have measurable consequences for the energy landscape.
Critical strengths

The premise is sound and the fix is cheap: ET's token interactions genuinely carry no notion of relative displacement, and importing the CANN idea of a displacement-only kernel is a natural thing to try. The theory is presented carefully and I found no technical error in it — the proof of Thm 2 (A.3, p.11–12, line 535 ff.) is clean, and the chain from Thm 1 to the tangent-space inclusion is correct. The experimental control of evaluating both topologies with the same trained weights is exactly the right design, the hyperparameters and sliding-window geometry are documented well enough to reproduce, and code is included. I also want to credit the authors for reporting results that cut against their own framing: many submissions would have buried §5.1 and §6.1.

Critical weaknesses
  1. The theorems are close to definitional, and the only hard condition is never checked: Thm 1 states that if K depends only on r−r′ then F_sp(T_a x) = F_sp(x) (section 4.1); the proof is a change of variables. Thm 2 is the standard statement that critical points of an equivariant functional come in group orbits and that the orbit directions lie in the Hessian kernel; the torus plays no special role. What is not standard is the hypothesis attached to Thm 2 — "If H_sp(x*) is positive definite on a complement of this tangent space with spectral gap μ>0\mu > 0" (section 4.2) — and μ\mu is never computed or bounded anywhere in the paper.
Questions for authors

Unverified premise of the attractor manifold: Theorem 2 is conditional on a nonconstant critical field x∗=Kx∗x^* = Kx^* with linearly independent spatial derivatives and a spectral gap μ\mu, yet such a field requires a zero-measure eigenvalue condition (λ(k)=1\lambda(k) = 1) whose existence in the trained model is never verified—and the cyclic-roll experiment only validates Theorem 1, which holds trivially for all fields. Could the authors provide spectral evidence for critical fields (and μ\mu) in the trained model, and justify why ϵ=1\epsilon = 1 for the rectangle still lies within the small-perturbation regime of the O(ϵ2)O(\epsilon^2) reduced dynamics?

1 comment

We thank the reviewer for identifying the hard premise of the paper. The paper describes Theorem 2 as the idealized condition for strict asymptotic normal attraction, and the trained model as an approximate saddle type slow manifold in Section 4.2. We now provide the spectral evidence.

Q1. Critical fields and the normal spectral gap

We compute the spectrum of the symmetric part of the channel mixing operator K\mathcal K on the trained AET weights. Table R1 reproduces the six eigenvalues nearest unity from Appendix A.8, Table 1. Table R2 reproduces the threshold counts from Appendix A.8, Table 2.

Table R1. Eigenvalues nearest unity, reproduced from Appendix A.8, Table 1.

λ\lambda∣λ−1∣\lvert\lambda-1\rvert
0.99915918.41×10−48.41\times10^{-4}
0.99837451.63×10−31.63\times10^{-3}
0.99816571.83×10−31.83\times10^{-3}
1.00196731.97×10−31.97\times10^{-3}
1.00207892.08×10−32.08\times10^{-3}
0.99774932.25×10−32.25\times10^{-3}
…\dots…\dots

Table R2. Threshold counts, reproduced from Appendix A.8, Table 2. Self conjugate modes are counted once.

∣λ−1∣\lvert\lambda-1\rvertcount
<0.01<0.0173
<0.005<0.00533
<0.002<0.0028

The closest eigenvalue is 0.9991591, with relative residual ∣λ−1∣=8.4×10−4|\lambda-1|=8.4\times10^{-4}. By Theorem 2, the translation orbit is two dimensional. The normal spectrum is mixed, and its minimum eigenvalue is negative. The entries above unity give μmin⁡≤−2.08×10−3\mu_{\min}\le -2.08\times10^{-3}. The trained model is therefore an approximate saddle type slow manifold with a mixed normal spectrum. We do not claim asymptotic normal attraction.

Q2. Cyclic roll

The T2\mathbb T^2 cyclic roll only validates Theorem 1, which holds for every field. It is a sanity check that the implementation realizes the circulant kernel to float32 precision, with mean relative change 4.1×10−84.1\times10^{-8} in the paper's Table 2. It is not evidence for a critical field. The rectangle roll is the actual measurement of the boundary term PP and the projected force.

Q3. Justification of ϵ=1\epsilon=1 in the O(ϵ2)O(\epsilon^2) regime

We agree that ϵ=1\epsilon=1 is not infinitesimal. In Theorem 3, the projected identity Ma˙=−∇p\mathbf M\dot{\mathbf a}=-\nabla p is exact for any exact critical field, and for the trained model's approximate fields it holds up to the 10−310^{-3} residual. The rectangle experiment measures the projected force, so it tests the leading O(ϵ)O(\epsilon) term. The O(ϵ2)O(\epsilon^2) reduction in Theorem 3 is conditional on a normal gap. In Theorem 3 and Appendix A.4, the state is O(ϵ∥P∥/∣μ∣)O(\epsilon\|P\|/|\mu|), the reduced field is O(ϵ2∥P∥2/∣μ∣)O(\epsilon^2\|P\|^2/|\mu|), and the relevant parameter is ∥P∥/∣μ∣\|P\|/|\mu|. The trained model's μ\mu is negative, as shown in Q1, so the O(ϵ2)O(\epsilon^2) reduction serves as a formal motivation. We therefore do not claim a rigorous O(ϵ2)O(\epsilon^2) bound at ϵ=1\epsilon=1.

The paper's Table 2 reports a mean energy change of 5.5×10−35.5\times10^{-3}. Table R3 reproduces the force RMS from Appendix A.5.

Table R3. Force RMS under cyclic rolls, reproduced from Appendix A.5.

TrainingEvaluationHorizontal RMSVertical RMS
T2\mathbb T^2T2\mathbb T^25.5×10−45.5\times10^{-4}5.3×10−45.3\times10^{-4}
T2\mathbb T^2rectangle3640
rectangleT2\mathbb T^22.6×10−32.6\times10^{-3}2.6×10−32.6\times10^{-3}
rectanglerectangle172184

The force RMS on the rectangle is 36 to 40 after T2\mathbb T^2 training. These are O(ϵ)O(\epsilon) effects. We have made this separation explicit in Theorem 3 and Appendix A.4.

Summary of changes

  1. Appendix A.8 now reports the spectrum summary and the normal gap.
  2. Section 4.2 now states the negative minimum and the approximate saddle type slow manifold.
  3. Theorem 3 and Appendix A.4 separate the exact projected identity from the conditional O(ϵ2)O(\epsilon^2) reduction and state the ∥P∥/∣μ∣\|P\|/|\mu| condition.
  4. Future Work now describes the spectral analysis as a first step and the positive gap or nonlinear finite time analysis as a remaining step.
Reviewer_r4AWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Attractor Energy Transformer (AET), which augments ET with a translation-invariant spatial kernel on a 2D torus. The authors theoretically show that cyclic translations preserve spatial energy, non-degenerate critical fields form a 2D critical manifold, and rectangular boundaries break this symmetry by inducing a pinning potential.

Critical strengths
  1. Linking Continuous Attractor Neural Networks (CANNs) and slow-manifold perturbation theory to Energy Transformers is conceptually appealing.
Critical weaknesses
  1. Experiments are limited to COCO2014 patch completion at 13M parameters. Ablations separating J0J_0 and WtrainW_{\text{train}}, and comparisons with standard relative positional encodings (e.g., 2D RoPE), are missing.

  2. Natural images are not periodic, so wrapping opposite crop borders may introduce artifacts. It remains unclear why the T2\mathbb{T}^2 kernel improves PSNR by up to 2.06 dB over the rectangular kernel without causing such artifacts.

Questions for authors

Please refer to weakness part.

1 comment

We thank the reviewer for the constructive suggestions. In response to the missing ablations, we have added the 2D RoPE comparison and the kernel component ablation in this revision. They now appear in Appendix A.9 and A.10. We reproduce them below. The appendix versions remain. We also clarify the torus and periodicity question.

1. Replacing learnable positional encoding with 2D RoPE

We have added the comparison to Appendix A.9. Table R1 reproduces it.

Table R1. 2D RoPE comparison, reproduced from Appendix A.9.

ModelPSNRSSIM
AET with learned PE22.180.7472
AET with 2D RoPE22.420.7559

This supports the complementary roles of the two mechanisms. The spatial kernel acts on the energy landscape through relative displacement and translation invariance. RoPE acts inside attention through relative position hints. The comparison is consistent with the two being largely orthogonal. It is not a claim that the kernel alone accounts for the improvement.

2. Ablation on the spatial kernel components

We have added the ablation to Appendix A.10. Table R2 reproduces it.

Table R2. Kernel component ablation, reproduced from Appendix A.10.

ModelPSNRSSIM
Standard ET21.290.7182
Fixed Gaussian only17.810.6278
Learned term only22.110.7413
Full AET22.180.7472

The Gaussian only model is over smoothed. As a low pass filter it erases high frequency texture. The learned term alone is competitive but its spectrum is less constrained. The full kernel combines the Gaussian anchor, which provides locality and stability, with the learned term, which provides detail and adaptivity.

3. Generalization beyond COCO2014

We agree that the main experiments are limited to COCO2014 at a matched 13M parameter budget in the paper's Table 1. This budget is deliberate. It keeps ET, AET and ViT comparable. We have added the STL10 results to Appendix A.11. Table R3 reproduces the STL10 table.

Table R3. STL10 comparison, reproduced from Appendix A.11.

AET on T2\mathbb T^2ETViT
Params13M13M13M
PSNR23.7022.2723.50
SSIM0.79000.74260.7779

AET on T2\mathbb T^2 is the best on both metrics, and it improves over ET.

4. Periodicity of natural images and boundary artifacts

We agree that natural images are not periodic. The torus is a computational topology for the token grid, not an assumption that image content is periodic.

On a rectangle, edge tokens have truncated neighborhoods. On the torus, every token has the same neighborhood, so the discrete kernel is exactly translation invariant and circulant. The rectangle is then modeled as a symmetry breaking perturbation, and the boundary term is measured directly by the cyclic roll and sliding window experiments. The 2.06 dB in the paper's Table 3 is the sliding window gap between AET on T2\mathbb T^2 and AET on the rectangle, which are trained separately on the two topologies, at the seam offset (0,64)(0,64). The cyclic roll experiment in the paper's Table 2 uses the same weights and changes only the padding topology at evaluation. The paper's Table 2 reports 4.1×10−84.1\times10^{-8} on T2\mathbb T^2 and 5.5×10−35.5\times10^{-3} on the rectangle. The paper's Table 1 reports a 0.9 dB gain for AET on T2\mathbb T^2 versus ET and a 0.085 dB gap between AET on T2\mathbb T^2 and AET on the rectangle.

On artifacts, the Gaussian anchor is local, the learned term is regularized, and the attention and Hopfield terms are content based, so unrelated opposite borders are not strongly coupled by attention. This limits, but does not eliminate, the risk of wraparound artifacts. The 2.06 dB seam gap in the paper's Table 3 and the pinning force RMS 36 to 40 in Table R4 are the corresponding boundary measurements. Table R4 reproduces the force RMS from Appendix A.5.

Table R4. Force RMS under cyclic rolls, reproduced from Appendix A.5.

TrainingEvaluationHorizontal RMSVertical RMS
T2\mathbb T^2T2\mathbb T^25.5×10−45.5\times10^{-4}5.3×10−45.3\times10^{-4}
T2\mathbb T^2rectangle3640
rectangleT2\mathbb T^22.6×10−32.6\times10^{-3}2.6×10−32.6\times10^{-3}
rectanglerectangle172184

Summary of changes

  1. The RoPE comparison is in Appendix A.9.
  2. The kernel component ablation is in Appendix A.10.
  3. The STL10 comparison is in Appendix A.11.

Comments on the paper

1 comment

Dear reviewers, the discussion period is coming to an end. Can you engage with the authors, please? Thanks

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.