- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper asks which apparent redundancies can be removed from Arm-A, a 17.4M-parameter language model derived from the BDH-GPU architecture.
Critical strengths
- Transparent reporting: strengthens its own baseline and reports unfavorable results.
- Thorough correctness checks before any timing.
Critical weaknesses
- Limited novelty: Chunked linear attention with resets at document boundaries is similar to cited works like RetNet, and Mamba-2. The neuron finding, that input-dependent selection beats a fixed one, similar to Deja Vu, CATS and TEAL. What remains is engineering on one non-standard model.
- Presentation not self-contained:
- The numbers in the abstract (also other places) offers little reference to gauge the significance
- Internal names are never defined: Arm-A, MB16×4, RB512, "mature executor," "historical dense reference" ...
- It refers to results that aren't in the paper: "the earlier canonical-BDH negative result," "the earlier logit-perturbation protocol" ... which makes it confusing to read.
- Supplements promised ... but no supplements provided ...: the selector results, data hashes and other-GPU results exist only "in the supplement,"
- Speedup claims are not convincing, for instance:
- The dense baselines were capped at 512 token blocks; So possibly some of the 1.52x may come from block sizes ...
- The matched comparison runs in uncompiled PyTorch. Their own compiled run is about two times faster, and there is no compiled dense baseline.
- There is no comparison to existing chunked linear-attention kernels.
- Generally just hard to isolate the improvements.
Questions for authors
Plz address my concerns in the weakness.