- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
SET added a multi-resolution causal convolution path parallel to self-attention and an additional causal convolution after the MLP. On WikiText-103, it provides a noticeable improvement under learned positional embeddings but a small improvement with RoPE. SET is also about 1.85× slower per step due to its higher computation complexity.
Critical strengths
-
The paper provides relatively careful controlled ablations, including parameter matching, and positional-encoding controls.
-
It is also transparent about negative results and computational cost, which makes the actual scope of the contribution clear.
Critical weaknesses
- Limited practical benefit in the tested RoPE setting
The positional argument in Section 4.1 is restricted to learned absolute positional embeddings and does not establish a corresponding limitation of RoPE-based attention. Empirically, the benefit is substantially smaller with RoPE. Separately, the reported implementation requires 1.85× the original baseline’s step time, and longer baseline training achieves lower loss in the tested wall-clock comparisons. These results leave the practical justification for the additional computational cost insufficiently established.
- Limited evaluation breadth and evidence of broader applicability
The primary controlled evaluation is concentrated on WikiText-103. The C4 results in Table 10 provide some evidence on a second corpus, but report only language-modeling loss and are subject to acknowledged limitations in training adequacy and experimental metadata. Consequently, the current evidence does not establish whether the observed NLL improvements translate into broader downstream or cross-domain benefits.
Questions for authors
Please see the clarification questions raised in the Weaknesses section.
2 comments
SET yields only a small reduction in loss with RoPE, and the paper reports a substantial increase in training time per step. The authors’ response does not explain why this trade-off is worthwhile in practice. The response also provides no new evidence that the lower loss translates into better performance on downstream tasks. Neither of my two concerns has been adequately addressed, so my rating remains unchanged.
Under RoPE, full SET reduces mean validation NLL from 3.87901 to 3.84695 relative to the parameter-matched wider-FFN Transformer, with all three paired seeds favoring SET. Public WikiText-103 test NLL also improves from 3.8835 to 3.8510.
The same direction appears in the submitted C4 pairs, where SET lowers loss by 3.9%, 5.1%, 4.0%, and 2.9% at the reported 125M, 350M, 760M, and 1B configurations. For incremental generation, caching reduces SET latency from 20.292 to 14.019 ms/token and peak allocated memory from 464041984 to 440651264 bytes at prompt 128 and decode 64. The maximum cached-versus-full logit difference is 0.00003481, and an independent rerun reproduces the 1.447-fold speedup.