- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies different fast-weight update rules. Specifically, it studies the effect of adding momentum to the delta-rule update, as proposed by several recent works. The authors provide a theoretical derivation that models the updates as continuous-time flows. The theory shows that in the 2-key case, sub-unit-gain delta rules cannot reduce the risk below some bound. However, other update rules with larger effective step sizes, such as momentum, can go below this bound. For k>2 keys the best update rule depends on the data itself. Finally, the authors test their theory on a synthetic setup that includes frozen image features.
Critical strengths
The studied problem is valuable, as architecture research provides core contributions to LLM design.
Critical weaknesses
-
The experiments do not represent a real scenario - the values are hand-crafted as one-hot vectors, the keys are features from a frozen image classifier, and they arrive according to a synthetic schedule. In real-world scenarios keys and values are both projections of the same token v = Wv @ x, k = Wk @ x and their order obeys the natural distribution of the data. To connect the theory to a real setting the authors should perform their experiments on real data taken from trained fast-weight models.
-
The theory does not cover per-token update rules, which are the rules used in practice, and specifically, the ones used by the models the authors use to motivate their work (Titans, Deltanet). This makes it even harder to connect the theory to actual settings.
Questions for authors
See weaknesses
3 comments
Q2
We believe this is potentially an overlooked point: the per-token setting is already covered in our experiments. The results are reported in Table 1 (page 8) and Appendix A.4.2 (Tables 15–16) of both the initial submission and the revision. The comparison covers three per-token update rules: the delta rule (P, projection), the reflect–project rule (RP), and momentum (H). We also agree with the reviewer’s suggestion that extending the experiments to trained fast-weight models would help validate our theory in practice. Thus, our additional experiments aim to answer two questions: When tokens in real text arrive one by one, which update rule has lower retrieval error? Does the result depend on the gain used for each update?
In our new experiments, we used keys and values extracted from the same five trained checkpoints as in our response to Question 1. For each checkpoint, we recorded the keys and values generated on 100 WikiText documents. We replayed these sequences using the three per-token update rules and measured the memory retrieval error. We compared the fixed learning rate (θ = 1) with a learning rate (θ < 1) based on the mean gate value of the trained model. The results are shown in the table below.
| Checkpoint | θ = 1 | Ranking | θ < 1 | Ranking |
|---|---|---|---|---|
| OpenRaiser AdamW | 0.2653 / 0.5433 / 0.3570 | P < H < RP | 0.3735 / 0.3269 / 0.3347 | RP < H < P |
| OpenRaiser AdamP | 0.3357 / 0.6555 / 0.4544 | P < H < RP | 0.4758 / 0.4258 / 0.4362 | RP < H < P |
| Gated DeltaNet 340M | 0.4179 / 0.7118 / 0.5411 | P < H < RP | 0.4215 / 0.3976 / 0.4034 | RP < H < P |
| Gated DeltaNet 1.3B | 0.4802 / 0.8436 / 0.5812 | P < H < RP | 0.4460 / 0.4370 / 0.4369 | H ≈ RP < P† |
| Qwen3.5-0.8B | 0.0987 / 0.1808 / 0.1179 | P < H < RP | 0.1050 / 0.0974 / 0.0993 | RP < H < P |
| Each cell reports P / RP / H; lower is better. θ is the learning rate defined in Section 2.1. | ||||
| † For Gated DeltaNet 1.3B, H and RP perform similarly. |
With θ = 1, the ranking is P < H < RP for all five checkpoints. With θ < 1, the ranking changes to RP < H < P for four of the five checkpoints: RP attains the lowest risk, followed by H and then P. This is consistent with Theorem 3.6, which shows that RP can attain lower retrieval risk than H. We have added these results to Section 5 (paragraph “Trained checkpoints”, page 9), with further details in Appendix A.7 (Table 21).
Response to rebuttal
I thank the authors for their rebuttal.
Adding keys and values sampled from trained fast-weight models' geometry narrows the theory-experiment gap. However, manually setting the energy ratio does not reflect real-world statistics. Why not report results weighted by a histogram of the energy ratio measured from the actual memory states of the trained models? Also, token order should not follow a synthetic access process but should be taken directly from the natural text.
Regarding Q2, my concern was that the theory, not the experiments, does not cover per-token updates. This remains unresolved.
Overall, the new version improves over the previous one, yet my main concerns remain, which I believe are too substantial to be addressed within this revision. I will keep my score.
Q1
We thank the reviewer for suggesting experiments on real data from trained fast-weight models. We have added experiments using keys and values extracted from five trained model checkpoints run on real text (WikiText-103). Following our analysis of the strong and weak directions in Section 3, we constructed 359 key–value sets, each containing two key–value pairs generated by one of the five selected checkpoints: OpenRaiser (AdamW and AdamP), Gated DeltaNet (340M and 1.3B), and Qwen3.5-0.8B.
We set different initial memory errors and examined how well the three update rules (P, RP, and H) corrected them. Besides an empty initial memory (W0=0), we first placed all error energy in the weak direction and then tested initial errors with 1%, 5%, or 10% of their energy in the strong direction and the remainder in the weak direction. We applied P, RP, and H to each initial memory and compared their retrieval risks. The results are shown in the table below (each cell reports P / RP / H; lower is better).
The results on trained models are consistent with Propositions 3.2 and 3.4 in Section 3: when the initial error lies in the weak direction, the ranking is H < RP < P. In our experiments, as the proportion of error in the strong direction increases, the ranking changes to RP < H < P and then to RP < P < H. We have added these results to Section 5 (page 9), with details in Appendix A.6 (Table 20).