- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces Attractor Energy Transformer (AET), which incorporates a translation-invariant spatial kernel on the toroidal token grid , and outperforms Energy Transformer on COCO2014 image completion.
Critical strengths
The argument for incorporating translation symmetry into the energy function of Transformers is potentially meaningful.
Critical weaknesses
- The problem targeted by this work is limited in significance and scope. It presents only marginal improvements over Energy Transformer (ET) on image completion, which is a fairly peripheral application. Furthermore, it offers little valuable insight into why these improvements are essential or how they will have a larger influence.
- The technical soundness and motivation are questionable. The paper relies on a relatively unconventional tool, geometric analysis on torus, without sufficient and effective contextualization within related literature, making it hard to understand. For instance, Figure 1 lacks sufficient annotations and a clear, detailed caption.
- The overall organization and writing quality need substantial improvement. Issues include, but are not limited to: insufficient explanation of how the spatial energy term in Equation 11 interacts with the other components, a lack of clear contrast between the token dynamics in Section 3.2 and standard ET, and unclear purposes behind the experiments presented in Sections 5.2 and 5.3.
Questions for authors
- Could the authors elaborate on the motivation behind the statement in lines 30–32 more clearly and specifically: “As a result, the energy does not possess an explicit translational symmetry, and its attractors are not naturally organized into continuous families of translated state”? Why does this limitation arise, and why is addressing it important?
- Why introduce the kernel of the torus in lines 35-36? The transition to its introduction is rather unnatural and unjustified.
- How does the theoretical analysis in Section 4 bring any new understanding or inform any new design, particularly in the context of image completion?
- Why is CDAT not compared as a baseline, as the authors point out in Section 6.2 that it is the closest work?
5 comments
Comments on the response
Thank you for the response.
I have a few further questions: How does the update differ from ET exactly? What is the rationale behind Equation 5 in Section 3.2? I don’t see any justification.
We thank the reviewer for the follow-up.
Equation 5 is a definition of the AET dynamics, so it is a construction rather than a theorem. It is CANN-inspired, as motivated in Section 1. The first two terms are a linear interaction on the torus, a leak term plus the translation-invariant kernel, and the last term is the ET update on the normalized field. Appendix A.1 derives the first two terms as the negative functional gradient of , which is the rationale for this form.
The difference from ET is exact. In our notation, ET updates the token field by
=-\frac{\partial E}{\partial \mathbf{g}(\mathbf{r},t)},$$ with $E=E^{\mathrm{ATT}}+E^{\mathrm{HN}}$. AET updates it by $$\tau\frac{\partial \mathbf{x}(\mathbf{r},t)}{\partial t} =-\mathbf{x}(\mathbf{r},t) +\int_{\mathbb{T}^{2}}\mathcal{K}(\mathbf{r}-\mathbf{r}')\,\mathbf{x}(\mathbf{r}',t)\,d\mathbf{r}' -\frac{\partial E}{\partial \mathbf{g}(\mathbf{r},t)}.$$ The attention and Hopfield terms are unchanged. The difference is exactly the two additional terms $-\mathbf{x}(\mathbf{r},t)$ and $\int_{\mathbb{T}^{2}}\mathcal{K}(\mathbf{r}-\mathbf{r}')\,\mathbf{x}(\mathbf{r}',t)\,d\mathbf{r}'$. Removing these two terms recovers the ET update.
We thank the reviewer for the candid assessment. We address the specific questions below and have revised the presentation to make the scope and contribution of the paper clearer. The contribution of the paper is a controlled analysis of how a translation invariant spatial kernel changes the energy landscape and the boundary behavior of ET, together with the completion results that follow from it.
Q1. Why the lack of translational symmetry in ET is significant
Image completion requires restoring missing patches from their spatial context, so the model must represent two dimensional spatial relationships. In standard ET, attention is content based and permutation equivariant. Positional encodings inject order, but the energy has no explicit relative displacement interaction and therefore no flat directions along translations. As a result, the completion can be sensitive to small spatial shifts of the content.
AET adds a kernel K(r−r′) that depends only on relative displacement. This provides a uniform spatial prior and, in the idealized case, a flat valley along translations. The 0.9 dB gain over ET in Table 1 is modest. We agree it is not the sole contribution of the paper. The main point is the mechanism. The kernel changes the geometry of the spatial energy, and this change is measurable at the boundary. The paper is an analysis of that mechanism, not a claim of a large performance jump.
Q2. Why introduce the torus T2
On a rectangle, a discrete convolution is not translation invariant at the boundary. Edge tokens have fewer neighbors than interior tokens. On the torus, the kernel is exactly circulant, so every token has the same neighborhood. The torus is therefore a computational topology on the token grid, not an assumption that natural image content is periodic.
The rectangle is then modeled as a symmetry breaking perturbation of the torus. This separation is what makes the boundary effect measurable. The unperturbed problem has the translation symmetry, and the perturbation is the boundary term P. This is the logical role of the torus in Section 4.3, and we will make the transition more explicit in the revision.
Q3. What the theoretical analysis contributes
The theory is an idealized account of the mechanism. The trained model is an approximate realization, and the experiments test its predictions. Its role is to motivate the translation-invariant kernel and the boundary experiment.
The theory is therefore explanatory and design guiding. The performance gain comes from the translation-invariant spatial kernel that the theory motivates, while the theory explains why the boundary effect appears and how to measure it.
Q4. Why CDAT is not used as a baseline
CDAT is designed for graph tasks with excitation inhibition modulation. Its interaction geometry is a graph, not a regular 2D grid, and its readout and training objective are task specific. A direct image completion baseline would require a nontrivial adaptation, namely grid to graph construction, a different readout and a different objective. Such an adaptation would not isolate the effect of the spatial kernel. Section 6.2 clarifies this. We agree that the main experiments are limited to COCO2014. The STL10 results are in Appendix A.11.