- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that softmax attention performs Nadaraya-Watson kernel regression and proposes Cubit, which replaces it with kernel ridge regression. Cubit keeps the standard query-key softmax weights but first transforms the values with the inverse of a regularized similarity matrix built from a separate key-like projection. Under causal masking, this reduces to a traingular solve. A learned, bounded per-token rescaling of the values (called as Limited-Range Rescale) stabilizes training. The authors train decoder-only models at 125M, 350M and 1.3B parameters on Arxiv, Books3 and FineWeb-Edu datasets with training sequence lengths from 512 to 8192 tokens. They compare against a standard Transformer and DeltaFormer and report consistently lower validation loss, typically by 0.02-0.04 below xthe Transformer, along with slightly higher average zero-shot accuracy on seven benchmarks including ARC, HellaSwag and PIQA. They claim the advantage grows with training sequence length, and they include ablations on the separate key projection and the rescaling as well as a local linear regression variant.
Critical strengths
-
The paper treats token mixing as a choice of regression estimator, which gives a principled reason to try a different aggregation rule and a way for comparing attention variants. The local linear regression variant in the appendix shows that this framing can produce more than one design. Cubit adds one triangular solve and a value rescaling to standard attention, so existing architectures can adopt it with little effort.
-
Under causal masking, the triangular structure keeps the method autoregressive without computing a separate matrix inverse for each position. Cubit improves on the Transformer in every reported experiment.
Critical weaknesses
-
The paper motivates Cubit with the guarantees of kernel ridge regression like explicit regularization, RKHS convergence rates and robustness to noise. Yet the implemented "kernel matrix" is a causally masked, row-normalized softmax matrix. That matrix is lower-triangular and therefore not symmetric, so it is not a Gram matrix of any kernel, and its quadratic form can be indefinite. It is invertible, which makes the triangular solve well defined, but no RKHS underlies it and the Representer theorem does not apply. The weighting uses a different query-key similarity from the matrix being inverted, and the ridge term starts at 1e-10, which is practically 0. The resulting operator, softmax attention over values transformed by an inverse of a triangular matrix with a per-token scale, closely resembles DeltaFormer, one of the baselines. One of the contributions claimed in the paper is a better replacement for attention. Without valid theory, the model used in experiments is an empirical variant of DeltaFormer, and the authors neither acknowledge nor analyze the differences.
-
The gains over the Transformer are small, typically 0.02-0.04 in validation loss. The major claim that the advantage of Cubit grows with sequence length, is not an overstatement. On both Arxiv and Books3, the gap between narrows from 0.021 to 0.0187 and 0.0346 to 0.0325 respectively. Figure 3 plots training loss rather than validation loss, which would have been more informative, The paper also reports no throughput, memory or wall-clock cost for the added triangular solve, so it is difficult to assess whether a small gain is a valid tradeoff.
Questions for authors
- What are the training throughput, peak memory usage and wall-clock time of Cubit when compared to Transformer and DeltaFormer at 1024 and 8192 tokens? Does Cubit still win when the baselines receive the same FLOPs?
- Both DeltaFormer and Cubit apply softmax attention to values transformed by the inverse of a lower-triangular matrix. What exactly does Cubit compute that DeltaFormer does not? Can you show an ablation that gradually changes Cubit to DeltaFormer by small changes to the matrix and design?
- Can you report Figure 3 in validation loss rather than training loss?
- How does change in the course of training? Does it stay near 1e-10?
2 comments
Response to Reviewer dbzF(Part 2/2)
Q4: From DeltaFormer gradually to Cubit
A4: The following is from DeltaFormer to Cubit.
From DeltaFormer to Cubit
| Method | Formulation | 5k | 10k | 15k | 20k | 25k | 30k | 35k | 40k | 45k | 50k |
|---|---|---|---|---|---|---|---|---|---|---|---|
| DeltaFormer | and set diagnal value to be 1 | 4.1395 | 3.9361 | 3.8316 | 3.7734 | 3.7121 | 3.6482 | 3.6082 | 3.5724 | 3.5270 | 3.4705 |
| Cubit with shared R | 4.1432 | 3.9328 | 3.8313 | 3.7733 | 3.7134 | 3.6465 | 3.6058 | 3.5709 | 3.5227 | 3.4661 | |
| Cubit with Seperate R | 4.1357 | 3.9293 | 3.8258 | 3.7659 | 3.7052 | 3.6394 | 3.5995 | 3.5637 | 3.5176 | 3.4597 |
Q5: Can you report Figure 3 in validation loss rather than training loss?
A5: The following is the validation loss for Figure 3.
Diff Length Performance: Transformer VS Cubit
| Method | Length | 5k | 10k | 15k | 20k | 25k | 30k | 35k | 40k | 45k | 50k |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Transformer | 512 | 4.4865 | 4.2676 | 4.1182 | 4.0698 | 3.9963 | 3.9339 | 3.8610 | 3.8313 | 3.7778 | 3.7443 |
| Cubit | 512 | 4.4295 | 4.2254 | 4.0799 | 4.0319 | 3.9554 | 3.9003 | 3.8251 | 3.7961 | 3.7432 | 3.7110 |
| Transformer | 1024 | 4.1829 | 3.9666 | 3.8620 | 3.8012 | 3.7427 | 3.6778 | 3.6356 | 3.5973 | 3.5527 | 3.4960 |
| Cubit | 1024 | 4.1357 | 3.9293 | 3.8258 | 3.7659 | 3.7052 | 3.6394 | 3.5995 | 3.5637 | 3.5176 | 3.4597 |
| Transformer | 2048 | 3.9183 | 3.7805 | 3.6773 | 3.6389 | 3.5205 | 3.4631 | 3.4387 | 3.4006 | 3.3846 | 3.3673 |
| Cubit | 2048 | 3.8698 | 3.7385 | 3.6343 | 3.5967 | 3.4779 | 3.4199 | 3.3947 | 3.3584 | 3.3433 | 3.3265 |
| Transformer | 4096 | 3.7455 | 3.6394 | 3.5440 | 3.4668 | 3.4235 | 3.3919 | 3.3094 | 3.3154 | 3.3114 | 3.2985 |
| Cubit | 4096 | 3.6788 | 3.5776 | 3.4832 | 3.4050 | 3.3669 | 3.3352 | 3.2524 | 3.2578 | 3.2555 | 3.2430 |
| Transformer | 8192 | 3.6338 | 3.4524 | 3.4260 | 3.3345 | 3.3409 | 3.2739 | 3.2439 | 3.1980 | 3.2179 | 3.1942 |
| Cubit | 8192 | 3.5675 | 3.3904 | 3.3640 | 3.2743 | 3.2818 | 3.2160 | 3.1850 | 3.1412 | 3.1598 | 3.1366 |
Diff Length Performance: Deltaformer VS Cubit
| Method | Length | 5k | 10k | 15k | 20k | 25k | 30k | 35k | 40k | 45k | 50k |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Deltaformer | 512 | 4.4390 | 4.2357 | 4.0901 | 4.0457 | 3.9694 | 3.9128 | 3.8391 | 3.8092 | 3.7563 | 3.7242 |
| Cubit | 512 | 4.4295 | 4.2254 | 4.0799 | 4.0319 | 3.9554 | 3.9003 | 3.8251 | 3.7961 | 3.7432 | 3.7110 |
| Deltaformer | 1024 | 4.1395 | 3.9361 | 3.8316 | 3.7734 | 3.7121 | 3.6482 | 3.6082 | 3.5724 | 3.5270 | 3.4705 |
| Cubit | 1024 | 4.1357 | 3.9293 | 3.8258 | 3.7659 | 3.7052 | 3.6394 | 3.5995 | 3.5637 | 3.5176 | 3.4597 |
| Deltaformer | 2048 | 3.8834 | 3.7550 | 3.6485 | 3.6120 | 3.4937 | 3.4354 | 3.4131 | 3.3751 | 3.3601 | 3.3405 |
| Cubit | 2048 | 3.8698 | 3.7385 | 3.6343 | 3.5967 | 3.4779 | 3.4199 | 3.3947 | 3.3584 | 3.3433 | 3.3265 |
| Deltaformer | 4096 | 3.6973 | 3.5991 | 3.5061 | 3.4282 | 3.3847 | 3.3556 | 3.2732 | 3.2798 | 3.2796 | 3.2657 |
| Cubit | 4096 | 3.6788 | 3.5776 | 3.4832 | 3.4050 | 3.3669 | 3.3352 | 3.2524 | 3.2578 | 3.2555 | 3.2430 |
| Deltaformer | 8192 | 3.5919 | 3.4161 | 3.3918 | 3.2997 | 3.3082 | 3.2412 | 3.2113 | 3.1655 | 3.1854 | 3.1615 |
| Cubit | 8192 | 3.5675 | 3.3904 | 3.3640 | 3.2743 | 3.2818 | 3.2160 | 3.1850 | 3.1412 | 3.1598 | 3.1366 |
Q6: How does change in the course of training? Does it stay near 1e-10?
A6: Because we use key normalziation, softmax and RoPE, the inverse matrix condition number is relatively very small and so that solution is stable and the could be very small. If the condition number is very large, we have to increase it.
Response to Reviewer dbzF(Part 1/2)
Dear Reviewer dbzF,
Thank you for the support. We will address the concerns below.
Q1: Without valid theory, the model used in experiments is an empirical variant of DeltaFormer, and the authors neither acknowledge nor analyze the differences.
A1: The following is the difference between Deltaformer and Cubit.
Difference motivation The DeltaFormer also discusses how to use the inverse matrix to improve the performance, so that we can discuss the difference here. First, the motivation is different. The DeltaFormer is motivated by combining DeltaNet and Transformer, while the Cubit is motivated by the Kernel Ridge Regression.
Different Implementaion The implementation is different, as the DeltaFormer uses two different embeddings w and k to construct the inverse matrix, while Cubit suggests that we have to use one specific embedding (e.g., only k or only r) so that the matrix is invertible.
The diagonal value process Motivated by DeltaNet, the DeltaFormer sets the diagonal value to 1 to calculate the inverse matrix, while the Cubit can be sure that there exists an inverse matrix so that the diagonal value does not necessarily have to be 1. Such an error may gradually increase with the sequence length increase (empirically validated in Figure 3)
When q=ki, the KRR/Cubit will return vi, while Transformer(Nadaraya-Watson Regression) not.
The text model erros are related to associate recall. When q=ki, we may need the perfect vi
Transformer=NW(q=kt,k,v)=kernel(kt,kt)+∑j=tkernel(kt,kj)kernel(kt,kt)vt+∑i=tkernel(kt,kt)+∑j=tkernel(kt,kj)kernel(kt,ki)vi
With the context length increase, the kernel(kt,kt)+∑j=tkernel(kt,kj)kernel(kt,kt) will decreases, suggests that the the needed vt will be difficult to be retrive. If the context is sufficient long, the retrived value is only noise
For KRR, we have th following
KRR(q=kt,k,v)=(qk)(kkT)−1v=(ktk)(kkT)v=vt.
Q2: The gains over the Transformer are small, typically 0.02-0.04 in validation loss.
A2: For language modeling, the 0.04 loss is actually a large loss gain. For example, DeltaNet Gated Attention performance gain is only 0.018 loss gain compared to Transformer++ for 1.3B in Table 1 [1]
Reference:
[1] Yang, S., Wang, B., Zhang, Y., Shen, Y., & Kim, Y. (2024). Parallelizing linear transformers with the delta rule over sequence length. Advances in neural information processing systems, 37, 115491-115522.
Q3: The time cost
A3: The following is the time cost.