- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose open-world AI should be able to learn new tasks from a few examples. The 3 things needed to achieve this are: 1. It continually learns; 2. It learns features that are useful outside the current task; 3. It splits its learnings into groups predicted separately. The authors claim the part of the model that learns from examples improves further upon seeing more. Furthermore, the authors claim the model should handle similar tasks separately from the current task and then combine them. They test this on language models (1.5B and 8B parameters) trained on groups of similar documents. Memory Mosaics, a model that the authors claim uses this way of learning, gets better when trained on longer texts. They also show that standard Transformers and other popular models do not get better when trained on longer texts. The authors also build "two-stage" models with memories for current documents separate from memories for other similar documents and keep improving as similar documents are added. Models with a single memory stop improving around their training length.
Critical strengths
- As the authors postulate, Memory Mosaics improves with longer training contexts, while the other models mostly get worse. This is good evidence to support the paper's claims.
- Two-stage Memory Mosaics+ keeps improving with related context up to 8 times its training length, at both 1.5B and 8B. The one-stage models peak around their training length. This can be useful for long-context models.
Critical weaknesses
-
Fig. 5 doesn't show that separate memories for the current and related documents are the contributing reasons for gains. The two-stage models also use the "+" design. No one-stage "+" model is tested in the Fig. 5 setup.
-
The paper claims Memory Mosaics improves because it's consistent, but this is not tested. Memory Mosaics also differs from the other models in position encoding, key and value construction, and bandwidth. There aren't any experiments which change consistency and keep all else equal. The paper also does not check which models are actually consistent.
Questions for authors
- In the Figure 5 setting, do one-stage Transformer+ and Memory Mosaics+ also avoid the decline?
- Do the models trained at l = 64k peak near 64k when evaluated out to 128k?
- Do two-stage models degrade less than one-stage models when the related context comes from a mismatched cluster?
- Can you vary consistency within a single model family, for example Memory Mosaics with a constant bandwidth?
- Do the learned bandwidth parameters make Memory Mosaics consistent?
- Given your definition of consistent, is NoPE full attention consistent?
- How is top-1 accuracy at evaluation length l′ computed?
- At what model scale were Tables 1-3 run?
- How much do the results vary across random seeds?
- What causes Hybrid DeltaNet's drop between l = 16k and 32k?
- What is already covered in Zhang et al. (2025) and Zhang & Bottou (2026) vs what is novel in this paper?
Thanks to the reviewers and next steps
We thank the reviewers for their time and thoughtful feedback.
The introduction of this paper sets out an ambitious goal toward open-world AI, one that a single conference paper can hardly fully support. Colleagues who read the paper told us that it is too dense; the reviewers found it incomplete, noting the lack of a separate related-work section, the unclear distinction between inference-time learning and in-context learning, the question of consistency versus a sufficiently small learner error, and the missing one-stage + control, among others.
We see these as two views of the same problem: we tried to fit too much into one paper. Seen from one side, the result is dense; seen from the other, certain helpful explanations, context, and controls are missing.
Either way, it makes it hard for readers to engage with the paper's new theory and experiments. We will rewrite the work to be more accessible and complete, and return with either a longer version or two separate ones at a future venue. We thank the reviewers again; their comments will make the next version substantially better.