- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper views sequence layers, including linear attention, DeltaNet, and Mesa, as online learners and compares them through regret. It establishes a lower bound showing that fixed memory capacity forces recall errors that attention can avoid. Building on this perspective, the authors propose two layers that adapt their step sizes and timescales without learned gates, and evaluate them on synthetic tasks and enwik8 language models with up to 340M parameters.
Critical strengths
-
The paper is clearly written and easy to follow.
-
It tackles a real gap, and framing sequence layers through regret and online learning is a clever idea.
-
The length-extrapolation study is really nice. Re-evaluating the same checkpoints with RoPE interpolation separates positional effects from memory effects, and the bounded-state rules stay below their training-length bpc at 16× context across all four scales.
Critical weaknesses
-
At 340M, Gated DeltaNet has the best validation bpc (1.2048 vs. 1.2102 for KT-Delta), but the abstract says KT-Delta has the best test bpc. Why lead with test? Please report test bpc for all rules at all scales and add DeltaNet seeds at 340M, since the claimed sign flip (−0.0098) rests on a single run. Also, validation bpc gets worse for every rule from 170M to 340M. Are these runs overfitting at 3×10^8 tokens?
-
The synthetic regret rankings in Table 6 differ from the bpc rankings in Table 2. This limits what the synthetic results establish about predictive utility. Which regret statistic and comparator class, or , should inform predictions about LM performance, and under what assumptions? Measuring the relationship between regret probes and bpc on the trained enwik8 models would strengthen this claim. Including Mesa in the enwik8 comparison would also be valuable given its central role in the theoretical framework.
-
If I expand (3), I get , where depends on the data. That looks like an implicit forget/amplify gate. Is that right? If so, an ablation with would show whether the gain over DeltaNet comes from the wealth-based step or from this hidden decay.
-
App. C.1 says KT-Delta's recurrence is inherently sequential, while linear attention and DeltaNet can be computed chunkwise in parallel, yet KT-Delta is reported faster than both. Softmax also runs out of memory past 8K tokens at eval. Which kernels did you use for the baselines, and does eval build the full attention matrix? A comparison against standard optimized kernels on the same hardware would make the throughput claim much more convincing.
I would frame the paper as a unifying view through regret, limit the claimed guarantees to the configurations actually trained, and strengthen the empirical evidence with more seeds, optimized baselines, Mesa at scale, and both validation and test results. The core idea is clever, but the paper in its current form is not strong enough for acceptance.
Questions for authors
Please check weaknesses section.