- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This manuscript compares four Transformer-based associative memory retrieval mechanisms, namely feed-forward (FFT), diffusion (DiT), recurrent (RT), and energy-based (EBT). The central claim is that explicit energy minimization scales best among all mechanisms, and that this advantage grows with model size. Experiments are conducted on an autoassociative image reconstruction task using COCO image latents.
Critical strengths
- The research question is well-defined and the experimental design is clean. Isolating the retrieval mechanism while keeping the Transformer backbone consistent is the right approach to answering this question.
- The scaling law framework makes a substantive methodological contribution. Fitting power-law exponents rather than reporting single-point comparisons is a meaningful choice that yields more informative conclusions.
- Experimental details are thorough. Appendix B provides sufficient information for reproducibility, including model configurations, training objectives, computational cost accounting, and the fitting procedure.
Critical weaknesses
- Generalizability of the findings is unverified. All experiments are based on autoassociative image reconstruction with a fixed masking corruption scheme. Whether the scaling advantage of EBT extends to heteroassociative settings remains untested. The manuscript acknowledges this in the limitations section, but the claims made in the abstract and introduction go beyond what the experiments actually support.
- EBT hyperparameters are not tuned, while the baselines are. The manuscript notes this asymmetry, but it prevents the comparison from establishing a fair upper bound for the competing methods. At minimum, a sensitivity analysis over EBT step size and number of descent steps should be provided.
- The anomalous generalization performance of DiT lacks explanation. DiT consistently stores fewer memories than FFT and RT (Table B.4), yet generalizes better to corrupted queries (Figure 4, Table B.6). The manuscript acknowledges this dissociation but offers no mechanistic explanation for it.
- Single training runs limit the reliability of the scaling law fits. Appendix B.2 states that each experimental configuration is trained only once. Scaling law fits are sensitive to noise at individual data points, particularly at the smallest and largest model scales.
Questions for authors
- EBT hyperparameters were not tuned while the baselines were. The authors are encouraged to provide a sensitivity analysis over step size and number of descent steps.
- DiT stores fewer memories than other mechanisms yet generalizes better to corrupted queries. The authors are encouraged to provide a mechanistic explanation for this dissociation.
- Each configuration is trained only once. The authors are encouraged to demonstrate the reliability of the fitted scaling exponents through repeated runs or confidence intervals.
We thank the reviewer for the positive assessment, and for noting the well-defined question, clean experimental design, scaling-law framework, and thorough experimental details.
W1: “Generalizability of the findings is unverified. All experiments are based on autoassociative image reconstruction with a fixed masking corruption scheme. Whether the scaling advantage of EBT extends to heteroassociative settings remains untested. [...] the claims made in the abstract and introduction go beyond what the experiments actually support.”
Response: We thank the reviewer and agree that our claims should match the scope of our experiments. We have scoped the paper to autoassociative retrieval, and now present implications for language and world models as motivation rather than conclusions. We also checked that the results are not tied to the input format, and found that with 16 tokens per image instead of 4, EBT's relative advantage over FFT is preserved (e.g., 8k vs. 3k memories at XS), suggesting that our findings are robust to the input format and size.
W2 + Q1: “EBT hyperparameters are not tuned, while the baselines are. The manuscript notes this asymmetry, but it prevents the comparison from establishing a fair upper bound for the competing methods. At minimum, a sensitivity analysis over EBT step size and number of descent steps should be provided.”
Response: To clarify, all models, including EBT, had their learning rates tuned, and the baselines additionally had their architecture-specific settings tuned (RT passes and DiT integration steps, Appendix B.2). Only EBT's energy-specific hyperparameters were left at default settings. We add a sensitivity analysis over the number of descent steps: more steps improve EBT, with 8 steps memorizing 2,000 memories at 3XS in roughly half the training steps of our default 4, while 2 steps is too few to train within our budget. Our reported results therefore use a non-optimal setting and are likely conservative. We now note this in the Limitations section. We also find that removing spectral normalization does not change performance.
W3 + Q2: “The anomalous generalization performance of DiT lacks explanation. DiT consistently stores fewer memories than FFT and RT (Table B.4), yet generalizes better to corrupted queries (Figure 4, Table B.6). The manuscript acknowledges this dissociation but offers no mechanistic explanation for it.”
Response: We thank the reviewer for this insightful question. Our hypothesis is that DiT is trained on all interpolated states between the query and the memory, so its velocity field is fit over a neighborhood around each memory rather than a single input. This widens the region from which retrieval succeeds (robustness), but requires fitting many more inputs per memory, lowering capacity. EBT avoids this tradeoff, as a single learned minimum provides both precise storage and a basin of attraction. We have added this discussion to Sec. 4.2.
W4 + Q3: “Single training runs limit the reliability of the scaling law fits. Appendix B.2 states that each experimental configuration is trained only once. Scaling law fits are sensitive to noise at individual data points, particularly at the smallest and largest model scales.”
Response: We trained 3 seeds at two model sizes for all four architectures. Measured capacities were identical across seeds (capacity is measured at 1,000-memory granularity), so the fitted exponents and the ranking of architectures are unchanged. Doubling the training budget to 400k steps also leaves the relative capacities unchanged.
We hope these additions address your concerns, and would be grateful if you would consider raising your score.