- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Nexus, an attention mechanism that refines queries and keys through separate causal self-attention operations before computing their final matching scores. Each refinement uses the prefix visible at the corresponding position, while the final attention aggregates the original values. The construction can be nested to obtain higher-order attention and reuses the existing projection parameters. The paper evaluates Nexus through language-model pretraining at multiple scales and supervised fine-tuning of pretrained models.
Critical strengths
- Clear architectural design; Nexus introduces contextual information into query–key matching while preserving causality and reusing existing projections.
- Useful theoretical analysis; the paper demonstrates how additional visible tokens can change relative attention weights within a Nexus operation, distinguishing it from the specified class of token-local scoring functions.
- Evaluation across multiple model scales; experiments include both training from scratch and adapting pretrained models, with results averaged over three independent training runs.
- Relevant architectural controls; Nexus achieves slightly higher average accuracy than the 11-layer Standard model at approximately matched analytical computation and with fewer parameters. It also outperforms the tested shared hidden-state-loop baseline at lower analytical cost.
Critical weaknesses
- Comparisons with alternative attention mechanisms are limited; MTA and Differential Transformer are evaluated only at 70M. Other relevant methods, including context-aware self-attention, DCMHA, and Tensor Product Attention, are discussed but not evaluated. Comparisons with suitable causal implementations would strengthen the empirical assessment.
- The advantage over MTA is mixed; both methods achieve a reported average accuracy of 0.406, while MTA requires 1.07× Standard’s analytical forward computation compared with 1.44× for Nexus. Nexus achieves lower NLL, but its additional computation does not produce a higher displayed task average.
- The parameter-efficiency evidence could be strengthened; the shared hidden-state-loop baseline repeats only the attention sublayer. Reusing complete Standard Transformer blocks would provide another way to increase computational depth without adding parameters. Comparing these alternatives under similar parameter and computational budgets would better assess the benefit of query/key refinement. The existing architectural controls are also limited to 70M.
- Uncertainty around the improvements is insufficiently characterized; although results are averaged over three training runs, standard deviations or confidence intervals are not reported. Cross-run win counts do not establish whether small differences, such as the 0.3-percentage-point gain over the 11-layer Standard model, are statistically reliable.
Questions for authors
- Can the authors evaluate additional relevant attention mechanisms, particularly contextual query/key methods, and extend the strongest baseline comparisons beyond 70M?
- How do Nexus and MTA compare under matched total training computation? Does Nexus’s lower NLL translate into downstream improvements at larger scales or with further training?
- How does Nexus compare with a Standard Transformer that reuses complete blocks, with both parameter count and computation approximately matched?
- Can the authors provide the individual training-run results and uncertainty estimates for the differences against the 11-layer Standard, shared hidden-state-loop, and MTA baselines?