- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes SMem. Each block of tokens is encoded independently into memory rows, and a reader combines block-local self attention with cross attention to the rows of earlier blocks. Exact composition, deletion, and path independence follow from the masking pattern. Models from 160M to 1.5B are trained on FineWeb-Edu with and , under two recipes. I think the experimental design of this paper needs improvement, and the use of AI throughout the paper is too obvious and heavy, making it difficult for me to establish a complete logical chain for the core idea.
Critical strengths
The paper directly tests whether block independent context encoding can be trained from scratch at moderate scale. It includes several relevant controls: a Block-Attention-style arm, a CEPE-style reader, and windowed transformer readings.
Critical weaknesses
-
Many propositions and formulas bear little relation to the main idea and confuse the logic. Proposition 1, Theorem 1, and Lemma 1 restate what block-local masking already implies. The containment argument, the round elimination sketch, and the permutation chain testbed in Section 5 are not connected to SMem's perplexity, retrieval, or serving behavior.
-
I think this manuscript shows heavy AI traces, and its logic is hard to follow. Sentences stack parenthetical qualifications and numerical ranges, terms such as roll gap and retrievable-rare stratum lack clear definitions, and paragraphs switch between recipes and scales without signaling the comparison.
-
The evidence does not establish SMem as a practical long-context architecture. Training uses 1024 tokens, and there is no RULER or LongBench evaluation. Under the tuned recipe, SMem trails the matched transformer by 1.7% to 2.8% and the RoPE transformer by 7.4% to 8.5%, and its retrieval beyond the window falls to near zero.
Questions for authors
- Under the tuned recipe, what is the perplexity gap at 1.5B, and does far-range retrieval survive at that scale?
- With a transformer given the same and depth split as SMem, how much of the decode gain remains?
- How does SMem perform on RULER or LongBench at contexts beyond 4096 tokens?
- How do the transformer baselines perform at length after brief fine-tuning on longer contexts?