acceptodds
Under review as a conference paper at ICLR 2027

Cubit: Token Mixer with Kernel Ridge Regression

Abstract

Since its introduction in 2017, the Transformer has become one of the most widely adopted architectures in modern deep learning. Despite extensive efforts to improve positional encoding, attention mechanisms, and feed-forward networks, the core token-mixing mechanism in Transformers remains attention. In this work, we show that the attention module in Transformers can be interpreted as performing Nadaraya–Watson regression, where it computes similarities between tokens and aggregates the corresponding values accordingly. Motivated by this perspective, we propose Cubit, a potential next-generation architecture that leverages Kernel Ridge Regression (KRR), while the vanilla Transformer relies on Nadaraya-Watson regression. Specifically, Cubit modifies the classical attention computation by incorporating the closed-form solution of KRR, combining value aggregation through kernel similarities with normalization via the inverse of the kernel matrix. To improve the training stability, we further propose the Limited-Range Rescale (LRR), which rescales the value layer within a controlled range. We argue that Cubit, as a KRR-based architecture, provides a stronger mathematical foundation than the vanilla Transformer, whose attention mechanism corresponds to Nadaraya–Watson regression. We validate this claim through comprehensive experiments. The experimental results suggest that Cubit may exhibit stronger long-sequence modeling capability. In particular, its performance gain over the Transformer appears to increase as the training sequence length grows.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Pre-Train, Model Architecutre, Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dbzFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that softmax attention performs Nadaraya-Watson kernel regression and proposes Cubit, which replaces it with kernel ridge regression. Cubit keeps the standard query-key softmax weights but first transforms the values with the inverse of a regularized similarity matrix built from a separate key-like projection. Under causal masking, this reduces to a traingular solve. A learned, bounded per-token rescaling of the values (called as Limited-Range Rescale) stabilizes training. The authors train decoder-only models at 125M, 350M and 1.3B parameters on Arxiv, Books3 and FineWeb-Edu datasets with training sequence lengths from 512 to 8192 tokens. They compare against a standard Transformer and DeltaFormer and report consistently lower validation loss, typically by 0.02-0.04 below xthe Transformer, along with slightly higher average zero-shot accuracy on seven benchmarks including ARC, HellaSwag and PIQA. They claim the advantage grows with training sequence length, and they include ablations on the separate key projection and the rescaling as well as a local linear regression variant.

Critical strengths
  1. The paper treats token mixing as a choice of regression estimator, which gives a principled reason to try a different aggregation rule and a way for comparing attention variants. The local linear regression variant in the appendix shows that this framing can produce more than one design. Cubit adds one triangular solve and a value rescaling to standard attention, so existing architectures can adopt it with little effort.

  2. Under causal masking, the triangular structure keeps the method autoregressive without computing a separate matrix inverse for each position. Cubit improves on the Transformer in every reported experiment.

Critical weaknesses
  1. The paper motivates Cubit with the guarantees of kernel ridge regression like explicit regularization, RKHS convergence rates and robustness to noise. Yet the implemented "kernel matrix" is a causally masked, row-normalized softmax matrix. That matrix is lower-triangular and therefore not symmetric, so it is not a Gram matrix of any kernel, and its quadratic form can be indefinite. It is invertible, which makes the triangular solve well defined, but no RKHS underlies it and the Representer theorem does not apply. The weighting uses a different query-key similarity from the matrix being inverted, and the ridge term starts at 1e-10, which is practically 0. The resulting operator, softmax attention over values transformed by an inverse of a triangular matrix with a per-token scale, closely resembles DeltaFormer, one of the baselines. One of the contributions claimed in the paper is a better replacement for attention. Without valid theory, the model used in experiments is an empirical variant of DeltaFormer, and the authors neither acknowledge nor analyze the differences.

  2. The gains over the Transformer are small, typically 0.02-0.04 in validation loss. The major claim that the advantage of Cubit grows with sequence length, is not an overstatement. On both Arxiv and Books3, the gap between narrows from 0.021 to 0.0187 and 0.0346 to 0.0325 respectively. Figure 3 plots training loss rather than validation loss, which would have been more informative, The paper also reports no throughput, memory or wall-clock cost for the added T×TT \times T triangular solve, so it is difficult to assess whether a small gain is a valid tradeoff.

Questions for authors
  1. What are the training throughput, peak memory usage and wall-clock time of Cubit when compared to Transformer and DeltaFormer at 1024 and 8192 tokens? Does Cubit still win when the baselines receive the same FLOPs?
  2. Both DeltaFormer and Cubit apply softmax attention to values transformed by the inverse of a lower-triangular matrix. What exactly does Cubit compute that DeltaFormer does not? Can you show an ablation that gradually changes Cubit to DeltaFormer by small changes to the matrix and design?
  3. Can you report Figure 3 in validation loss rather than training loss?
  4. How does λ\lambda change in the course of training? Does it stay near 1e-10?
2 comments
AuthorsOpenReview ↗

Response to Reviewer dbzF(Part 1/2)

Dear Reviewer dbzF,

Thank you for the support. We will address the concerns below.

Q1: Without valid theory, the model used in experiments is an empirical variant of DeltaFormer, and the authors neither acknowledge nor analyze the differences.

A1: The following is the difference between Deltaformer and Cubit.

Difference motivation The DeltaFormer also discusses how to use the inverse matrix to improve the performance, so that we can discuss the difference here. First, the motivation is different. The DeltaFormer is motivated by combining DeltaNet and Transformer, while the Cubit is motivated by the Kernel Ridge Regression.

Different Implementaion The implementation is different, as the DeltaFormer uses two different embeddings ww and kk to construct the inverse matrix, while Cubit suggests that we have to use one specific embedding (e.g., only kk or only rr) so that the matrix is invertible.

The diagonal value process Motivated by DeltaNet, the DeltaFormer sets the diagonal value to 1 to calculate the inverse matrix, while the Cubit can be sure that there exists an inverse matrix so that the diagonal value does not necessarily have to be 1. Such an error may gradually increase with the sequence length increase (empirically validated in Figure 3)

When q=kik_i, the KRR/Cubit will return viv_i, while Transformer(Nadaraya-Watson Regression) not.

The text model erros are related to associate recall. When q=kiq=k_i, we may need the perfect viv_i

TransformerTransformer=NW(q=kt,k,v)=kernel(kt,kt)kernel(kt,kt)+∑j≠tkernel(kt,kj)vt+∑i≠tkernel(kt,ki)kernel(kt,kt)+∑j≠tkernel(kt,kj)vi\text{NW}(q=k_t,k,v) = \frac{\text{kernel}(k_t, k_t)}{\text{kernel}(k_t, k_t) + \sum_{j \neq t} \text{kernel}(k_t, k_j)}v_t +\sum_{i \neq t} \frac{\text{kernel}(k_t, k_i)}{\text{kernel}(k_t, k_t) + \sum_{j \neq t} \text{kernel}(k_t, k_j)} v_i

With the context length increase, the kernel(kt,kt)kernel(kt,kt)+∑j≠tkernel(kt,kj)\frac{\text{kernel}(k_t, k_t)}{\text{kernel}(k_t, k_t) + \sum_{j \neq t} \text{kernel}(k_t, k_j)} will decreases, suggests that the the needed vtv_t will be difficult to be retrive. If the context is sufficient long, the retrived value is only noise

For KRR, we have th following

KRR(q=kt,k,v)=(qk)(kkT)−1v=(ktk)(kkT)v=vt\text{KRR}(q=k_t,k,v)=(qk)(kk^T)^-1v=(k_tk)(kk^T)v=v_t.

Q2: The gains over the Transformer are small, typically 0.02-0.04 in validation loss.

A2: For language modeling, the 0.04 loss is actually a large loss gain. For example, DeltaNet Gated Attention performance gain is only 0.018 loss gain compared to Transformer++ for 1.3B in Table 1 [1]

Reference:

[1] Yang, S., Wang, B., Zhang, Y., Shen, Y., & Kim, Y. (2024). Parallelizing linear transformers with the delta rule over sequence length. Advances in neural information processing systems, 37, 115491-115522.

Q3: The time cost

A3: The following is the time cost.

MethodCostLength 512Length 1024Length 2048Length 4096Length 8192
TransformerTime Cost (ms)326.3352.0445.3790.72078.3
Memory Cost (MB)7352.07552.08774.015980.039368.0
DeltaFormerTime Cost (ms)459.6567.3823.01766.05307.0
Memory Cost (MB)7352.07868.09542.019052.051704.0
CubitTime Cost (ms)468.2575.8835.31761.75274.5
Memory Cost (MB)7352.07448.09578.020180.051656.0
AuthorsOpenReview ↗

Response to Reviewer dbzF(Part 2/2)

Q4: From DeltaFormer gradually to Cubit

A4: The following is from DeltaFormer to Cubit.

From DeltaFormer to Cubit

MethodFormulation5k10k15k20k25k30k35k40k45k50k
DeltaFormerKernel(QK)(Kernel(WK))−1VKernel(QK)(Kernel(WK))^{-1}V and set diagnal value to be 14.13953.93613.83163.77343.71213.64823.60823.57243.52703.4705
Cubit with shared RKernel(QK)(Kernel(KK))−1VKernel(QK)(Kernel(KK))^{-1}V4.14323.93283.83133.77333.71343.64653.60583.57093.52273.4661
Cubit with Seperate RKernel(QK)(Kernel(RR))−1VKernel(QK)(Kernel(RR))^{-1}V4.13573.92933.82583.76593.70523.63943.59953.56373.51763.4597

Q5: Can you report Figure 3 in validation loss rather than training loss?

A5: The following is the validation loss for Figure 3.

Diff Length Performance: Transformer VS Cubit

MethodLength5k10k15k20k25k30k35k40k45k50k
Transformer5124.48654.26764.11824.06983.99633.93393.86103.83133.77783.7443
Cubit5124.42954.22544.07994.03193.95543.90033.82513.79613.74323.7110
Transformer10244.18293.96663.86203.80123.74273.67783.63563.59733.55273.4960
Cubit10244.13573.92933.82583.76593.70523.63943.59953.56373.51763.4597
Transformer20483.91833.78053.67733.63893.52053.46313.43873.40063.38463.3673
Cubit20483.86983.73853.63433.59673.47793.41993.39473.35843.34333.3265
Transformer40963.74553.63943.54403.46683.42353.39193.30943.31543.31143.2985
Cubit40963.67883.57763.48323.40503.36693.33523.25243.25783.25553.2430
Transformer81923.63383.45243.42603.33453.34093.27393.24393.19803.21793.1942
Cubit81923.56753.39043.36403.27433.28183.21603.18503.14123.15983.1366

Diff Length Performance: Deltaformer VS Cubit

MethodLength5k10k15k20k25k30k35k40k45k50k
Deltaformer5124.43904.23574.09014.04573.96943.91283.83913.80923.75633.7242
Cubit5124.42954.22544.07994.03193.95543.90033.82513.79613.74323.7110
Deltaformer10244.13953.93613.83163.77343.71213.64823.60823.57243.52703.4705
Cubit10244.13573.92933.82583.76593.70523.63943.59953.56373.51763.4597
Deltaformer20483.88343.75503.64853.61203.49373.43543.41313.37513.36013.3405
Cubit20483.86983.73853.63433.59673.47793.41993.39473.35843.34333.3265
Deltaformer40963.69733.59913.50613.42823.38473.35563.27323.27983.27963.2657
Cubit40963.67883.57763.48323.40503.36693.33523.25243.25783.25553.2430
Deltaformer81923.59193.41613.39183.29973.30823.24123.21133.16553.18543.1615
Cubit81923.56753.39043.36403.27433.28183.21603.18503.14123.15983.1366

Q6: How does change in the course of training? Does it stay near 1e-10?

A6: Because we use key normalziation, softmax and RoPE, the inverse matrix condition number is relatively very small and so that solution is stable and the λ\lambda could be very small. If the condition number is very large, we have to increase it.

Reviewer_fHBaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Cubit proposes a KRR-inspired token mixer that transforms values through a regularized matrix solve before attention aggregation, together with Limited-Range Rescale (LRR). Experiments evaluate language modeling and downstream performance across multiple datasets and model sizes.

Critical strengths
  1. The regression perspective motivates a concrete architectural modification. I find this an interesting design direction, whose empirical value does not depend on exact equivalence to classical KRR.
  2. Reported validation-loss improvements span several settings, with additional three-seed results in the appendix. The reference-sharing and no-LRR ablations also help examine the proposed components.
Critical weaknesses
  1. The relationship between the theoretical motivation and the implemented causal operator needs clarification. One-sided normalization, masked row-wise softmax, and triangular solving generally differ from classical symmetric KRR. Exact equivalence is not necessary for the architectural contribution to be useful, but the paper should distinguish the regression analogy from properties actually established for its implementation. My concern is whether the claimed theoretical advantages survive these modifications.
  2. The attribution of the gains could be clearer. The no-LRR ablation examines the effect of removing LRR from Cubit, but does not by itself establish whether the matrix solve provides an advantage over a baseline equipped with comparable rescaling. It would be helpful to clarify whether such a control is included or applicable. Comparisons with and without LRR for both architectures would help distinguish the contribution of the solve, the rescaling, and their interaction.
  3. The practical quality–cost trade-off remains unclear. Average downstream gains over Transformer are approximately 0.18–0.53 percentage points, but corresponding throughput, memory, and incremental decoding comparisons are missing. The additional validation-loss statistics are useful, but do not establish the uncertainty of these downstream differences. Consequently, the evidence for a practical advantage remains limited.
Questions for authors
  1. Which KRR-derived properties hold for the implemented causal operator? If an optimization interpretation is central to the theoretical claims, what objective does this operator optimize?
  2. Can LRR be applied comparably to the Transformer baseline, and is this control included in the experiments? What do the existing ablations establish about the separate contributions of the matrix solve and LRR?
  3. At matched training compute, how does Cubit compare with Transformer and DeltaFormer in quality? Please also clarify training throughput, memory use, incremental decoding performance, and the variability of the key downstream gains.
3 comments
AuthorsOpenReview ↗

Response to Reviewer fHBa(Part 1/3)

Dear Reviewer fHBa,

Thank you for the comment, we will address the concerns below.

Q1: The theoretical motivation is insufficiently connected to the provided causal implementation. One-sided normalization, masked row-wise softmax, and triangular solving generally differ from classical symmetric KRR. The claimed theoretical advantages therefore require justification for the implemented operator or more limited claims.

A1: We will explain the one-sided normalization, softmax normalization and trigular solve.

one sided normalization We could apply both query and key normalization to Cubit. The following is the comparsion.

Actually, according our training Cubit with even large scale model experience (e.g. 15B), it is better to normalize both query and key to a fixed norm (e.g. (D)\sqrt(D))。 Attention: We normalize QQ as Norm(Q)=Q∥Q∥⋅D\mathrm{Norm}(Q) = \dfrac{Q}{\lVert Q \rVert} \cdot \sqrt{D}, without any learnable scale. If a learnable scale is still desired, use a scalar rather than a vector of shape RD\mathbb{R}^{D}; the latter may cause the diagonal entry not to be the largest one, thereby increasing the condition number.

Cubit Performance with RBF kernel, following strict KRR definition, with 1.3B model size and GBS 16

Formulation5K10K15K20K25K30K35K40K45K50K
Kernel(QK)(Kernel(KNorm(K)))−1VKernel(QK)(Kernel(KNorm(K)))^{-1}V4.19863.95263.86673.74183.63673.58443.51023.50803.39863.4005
Kernel(Norm(Q)Norm(K)(Kernel(Norm(K)Norm(K)))−1VKernel(Norm(Q)Norm(K)(Kernel(Norm(K)Norm(K)))^{-1}V4.18723.94243.85913.73403.63223.57943.51133.50503.39573.3978

Softmax Normalization We have tried to apply RBF kernel (strictly follow the KRR), but the performance is worse than Softmax for the final step. We still dot not know the reason, and we sincere hope that if anyone could help solve this problem.

Cubit Performance with RBF kernel, following strict KRR definition, with 1.3B model size and GBS 16

Kernel(QK)Kernel(KK)5K10K15K20K25K30K35K40K45K50K
RBFRBF4.16403.93273.85403.73383.63223.57733.50523.50353.39513.3987
SoftmaxSoftmax4.18723.94243.85913.73403.63223.57943.51133.50503.39573.3978

The implementation of Softmax

T, B, H, D = query_layer.shape


key_layer_copy = key_layer
query_layer = query_layer.reshape(T, B, H, -1).permute(1, 2, 0, 3)  # [B*H, T, D]
key_layer = key_layer.reshape(T, B, H, -1).permute(1, 2, 0, 3)  # [B*H, T, D]
value_layer = value_layer.reshape(T, B, H, -1).permute(1, 2, 0, 3)  # [B*H, T, D]
beta = (0.5 + self.beta_ga) * F.sigmoid(
    self.b_proj(hidden_states).reshape(T, B, H, 1).permute(1, 2, 0, 3)) + 0.5 + self.alpha_zeros
k2 = key_layer_copy.reshape(T * B, H, -1)

k2 = k2.reshape(T, B, H, -1)

k2_k = (k2 / torch.norm(k2, dim=-1, keepdim=True, p=2) * math.sqrt(D)).reshape(T, B, H, -1)
query_k = (query_layer / torch.norm(query_layer, dim=-1, keepdim=True, p=2) * math.sqrt(D))

k2 = (k2).permute(1, 2, 0, 3)
k2_k = k2_k.permute(1, 2, 0, 3)
k2_k = k2_k

atten_score = torch.matmul(query_k / math.sqrt(D), (k2_k).transpose(-2, -1))
atten_score = atten_score

W = self.scale_mask_softmax(atten_score, attention_mask)

atten_score = torch.matmul(k2_k / math.sqrt(D), (k2_k).transpose(-2, -1))
atten_score = atten_score

A = self.scale_mask_softmax(atten_score, attention_mask)

with torch.cuda.amp.autocast(dtype=torch.float32):
    # solve(A, b) 
    value_layer = torch.linalg.solve_triangular(A.float(), (value_layer * beta).float(),
                                                upper=False)
    value_layer = value_layer.to(hidden_states.dtype)
context_layer = torch.einsum('bhij,bhjd->bhid', W, value_layer)

context_layer = context_layer.reshape(B, H, T, D).permute(2, 0, 1, 3).reshape(T, B, H * D)
AuthorsOpenReview ↗

Response to Reviewer fHBa(Part 2/3)

The implementation of RBF

def rbf_similarity_sqrt(Q, K, gamma=1.0, eps=1e-8):
    """
    Compute the RBF kernel similarity matrix.

    Args:
        Q: (T_q, d) query matrix
        K: (T_k, d) key matrix
        gamma: kernel width parameter (gamma > 0)
        eps: small constant for numerical stability (guard against negative zero)

    Returns:
        S: (T_q, T_k) similarity matrix, S[i,j] = exp(-gamma * ||Q[i] - K[j]||^2)
    """
    # Method 1: use torch.cdist (recommended)
    dist_sq = torch.cdist(Q, K, p=2)   # (T_q, T_k)
    # dist_sq = torch.cdist(Q, K, p=2)
    dist_sq = dist_sq**2
    dist_sq = dist_sq / (math.sqrt(16 * Q.shape[-1]))
    # Avoid negative zero (theoretically impossible, but possible due to floating-point error)
    dist_sq = torch.clamp(dist_sq, min=0.0)
    S = torch.exp(-gamma * dist_sq)
    return S

T, B, H, D = query_layer.shape


key_layer_copy = key_layer
query_layer = query_layer.reshape(T, B, H, -1).permute(1, 2, 0, 3)  # [B*H, T, D]
key_layer = key_layer.reshape(T, B, H, -1).permute(1, 2, 0, 3)  # [B*H, T, D]
value_layer = value_layer.reshape(T, B, H, -1).permute(1, 2, 0, 3)  # [B*H, T, D]
beta = (0.5 + self.beta_ga) * F.sigmoid(
    self.b_proj(hidden_states).reshape(T, B, H, 1).permute(1, 2, 0, 3)) + 0.5 + self.alpha_zeros
k2 = key_layer_copy.reshape(T * B, H, -1)

k2 = k2.reshape(T, B, H, -1)

k2_k = (k2 / torch.norm(k2, dim=-1, keepdim=True, p=2) * math.sqrt(D)).reshape(T, B, H, -1)
query_k= (query_layer / torch.norm(query_layer, dim=-1, keepdim=True, p=2) * math.sqrt(D))

k2 = (k2).permute(1, 2, 0, 3)
k2_k = k2_k.permute(1, 2, 0, 3)
k2_k=k2_k

W = rbf_similarity_sqrt(query_k.reshape(B * H, T, -1), k2_k.reshape(B * H, T, -1))
W = torch.tril(W)
W = W.reshape(B, H, T, -1)


A = rbf_similarity_sqrt(k2_k.reshape(B * H, T, -1), k2_k.reshape(B * H, T, -1))
A = torch.tril(A)
A = A.reshape(B, H, T, -1)


with torch.cuda.amp.autocast(dtype=torch.float32):
    # solve(A, b) 
    value_layer = torch.linalg.solve_triangular(A.float(), (value_layer * beta).float(),
                                                upper=False)
    value_layer = value_layer.to(hidden_states.dtype)
context_layer = torch.einsum('bhij,bhjd->bhid', W, value_layer)



context_layer = context_layer.reshape(B, H, T, D).permute(2, 0, 1, 3).reshape(T, B, H * D)

Solve trigular We apply the Cubit to train Large Language Model so that we apply casual mask. If we use it to image classification, we should remove the mask and solve the whole matrix, but that needs O(N^3) time cost.

AuthorsOpenReview ↗

Response to Reviewer fHBa(Part 3/3)

Q2: The practical quality–cost trade-off remains unclear.

The following is the time cost and memory cost.

MethodCostLength 512Length 1024Length 2048Length 4096Length 8192
TransformerTime Cost (ms)326.3352.0445.3790.72078.3
Memory Cost (MB)7352.07552.08774.015980.039368.0
DeltaFormerTime Cost (ms)459.6567.3823.01766.05307.0
Memory Cost (MB)7352.07868.09542.019052.051704.0
CubitTime Cost (ms)468.2575.8835.31761.75274.5
Memory Cost (MB)7352.07448.09578.020180.051656.0

Q3: What objective does the implemented causal operator optimize, and which KRR properties hold for it?

A3: We use casual operator because the current large scale model is decoder-only so that we add the casual mask, and you could remove it if you apply it to image classification and so on, but that needs O(N^3) time cost. The KRR properties hold for Cubit: when q=kik_i, the KRR/Cubit returns viv_i so that long context performance is betteer

The text model erros are related to associate recall. When q=kiq=k_i, we may need the perfect viv_i

TransformerTransformer=NW(q=kt,k,v)=kernel(kt,kt)kernel(kt,kt)+∑j≠tkernel(kt,kj)vt+∑i≠tkernel(kt,ki)kernel(kt,kt)+∑j≠tkernel(kt,kj)vi\text{NW}(q=k_t,k,v) = \frac{\text{kernel}(k_t, k_t)}{\text{kernel}(k_t, k_t) + \sum_{j \neq t} \text{kernel}(k_t, k_j)}v_t +\sum_{i \neq t} \frac{\text{kernel}(k_t, k_i)}{\text{kernel}(k_t, k_t) + \sum_{j \neq t} \text{kernel}(k_t, k_j)} v_i

With the context length increase, the kernel(kt,kt)kernel(kt,kt)+∑j≠tkernel(kt,kj)\frac{\text{kernel}(k_t, k_t)}{\text{kernel}(k_t, k_t) + \sum_{j \neq t} \text{kernel}(k_t, k_j)} will decreases, suggests that the the needed vtv_t will be difficult to be retrive. If the context is sufficient long, the retrived value is only noise

For KRR, we have th following

KRR(q=kt,k,v)=(qk)(kkT)−1v=(ktk)(kkT)v=vt\text{KRR}(q=k_t,k,v)=(qk)(kk^T)^-1v=(k_tk)(kk^T)v=v_t.

Reviewer_HkteOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Cubit replaces the usual attention value aggregation with a kernel-ridge-regression-inspired token mixer. It builds an additional reference similarity matrix from projected token features, adds a learned ridge parameter, and solves a causal triangular linear system for transformed values before applying ordinary query-key attention. A learned limited-range rescaling (LRR) multiplies the values entering that system. The paper motivates this as moving from a Nadaraya-Watson view of attention to a more regularized KRR view, and argues that the change should help as context length grows. The experiments compare Cubit with a Transformer and DeltaFormer on Arxiv, Books3, and FineWeb-Edu language modeling, using 125M, 350M, and 1.3B parameter settings. The paper also shows a growing training-loss gap up to length 8,192 and ablates the extra reference projection and LRR. On seven zero-shot tasks, Cubit has the highest mean in each of the three reported model/token-budget settings, although those mean differences are much smaller than the language-modeling loss differences: for example, 51.78 versus 51.53 for the Transformer at 1.3B/10B tokens.

Critical strengths
  1. The empirical pattern is reasonably consistent. Cubit is ahead of both listed baselines on Arxiv, Books3, and FineWeb validation loss at the reported endpoints, and the advantage persists at 350M and 1.3B rather than appearing only at the smallest scale. The comparison at 1,024 tokens with a doubled batch is a useful attempt to separate longer sequences from simply processing more tokens. Appendix J adds a three-seed FineWeb table, which is better than relying only on a single favorable run.

  2. I also appreciate that the ablations look at two quite different pieces of the design. The shared-reference variant still beats the Transformer in the FineWeb plot, suggesting the result is not solely an extra projection's parameter count. Removing LRR changes the final 125M FineWeb loss from 3.4597 to 3.4645, so LRR helps, but the bulk of the gain over the 3.4960 Transformer comes from the underlying mixer. That is a more informative result than a single full-model comparison.

Critical weaknesses
  1. The KRR justification does not cleanly match the operation actually implemented. Classical KRR uses a symmetric positive-semidefinite Gram matrix built from one kernel, whereas the supplied code forms the matrix to be inverted by row-wise softmax of reference-feature similarities with a causal mask and then adds a diagonal ridge term. This is generally not a symmetric Gram matrix; the separate query-key attention matrix used afterward is another learned similarity altogether. The paper itself first invokes a common feature map and then relaxes it to distinct query and key projections. That may define a useful mixer, but it is not enough to carry over KRR's convergence, regularization, or boundary-effect arguments. A careful derivation of what estimator the implemented causal solve represents is needed to sustain the main conceptual claim.

  2. The evidence for a practical long-context advantage is incomplete. The block includes a triangular solve in addition to ordinary attention, yet there are no matched throughput, memory, FLOP, or inference-latency results, even at the 8,192-token setting where the paper emphasizes a larger loss gap. Figure 3 plots training loss differences at longer lengths; it does not directly test long-range retrieval or held-out quality at lengths beyond training. The downstream improvements are modest (the seven-task mean is only 0.25 points above the Transformer at 1.3B/10B tokens), and the paper gives no uncertainty for those task means. The appendix implementation also refers to self.lrr and self.casual_mask although the constructor defines self.LRR and self.causal_mask; if this is the intended reproducibility code, it would not run as printed. None of this negates the observed loss improvement, but it weakens the claim of a scalable next-generation replacement.

Questions for authors

Can the authors specify the exact matrix solved in the experiments, including how the causal softmax is applied, and explain which of the stated KRR properties still hold for that generally nonsymmetric matrix? In particular, is there an ablation using a bona fide positive-semidefinite reference Gram matrix, or a more modest interpretation of the current operation that the authors can justify mathematically?

What are training tokens per second, peak memory, and autoregressive decoding latency for Cubit and both baselines at matched hardware and lengths 1,024, 4,096, and 8,192? A held-out long-context test, rather than only training-loss curves, would also materially change my view of the long-sequence claim. Please clarify whether the code in Appendix K is the code used for the experiments and correct the attribute-name inconsistencies if so.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Hkte (Part 1/3)

Dear Reviewer Hkte,

Thank you for the comment. We will address the concerns below.

Q1: The KRR justification does not cleanly match the operation actually implemented, with softmax kernel and reference projection.

A1: We have clearly presented the performance of RBF kernel and shared reference projection (K=R) in Figure 5, where the performance is worse。

Cubit Performance with RBF kernel, following strict KRR definition, with 1.3B model size and GBS 16

Kernel(QK)Kernel(KK)5K10K15K20K25K30K35K40K45K50K
RBFRBF4.16403.93273.85403.73383.63223.57733.50523.50353.39513.3987
SoftmaxSoftmax4.18723.94243.85913.73403.63223.57943.51133.50503.39573.3978

Cubit Performance with Shared Reference Matrix (K=R) in Figure 5

Method5k10k15k20k25k30k35k40k45k50k
Share R(K=R)4.14323.93283.83133.77333.71343.64653.60583.57093.52273.4661
Seperate R(K!=R)4.13573.92933.82583.76593.70523.63943.59953.56373.51763.4597

Q2: The evidence for a practical long-context advantage is incomplete. And the implementation question.

A2: We further provide the Multi Query Associate Recal evaluation performance.

Method102420488192Average
Hyena88.1747.6833.4356.42
Mamba97.8998.2788.7494.96
Mamba298.2098.2198.3298.24
RWKV-798.3498.5298.1398.38
Gated DeltaNet98.2998.4898.9898.58
Transformer98.6598.9098.7798.77
Cubit98.8899.1099.3999.12

The detailed implementation, with input hiddden states xx.

import math
import torch
import torch.nn as nn
import torch.nn.functional as F


def scale_mask_softmax(
    atten_score: torch.Tensor,
    attention_mask: torch.Tensor = None,
    scale: float = None,
    mask_value: float = -1e4,
) -> torch.Tensor:
    """
    Scale + mask + softmax.

    Convention: in `attention_mask`, 1 / True means KEEP, 0 / False means MASK.

    Args:
        atten_score:    [..., T, T]  raw scores (unscaled)
        attention_mask: [..., T, T] or any broadcastable shape
                        - bool:      True = keep, False = mask
                        - 0/1 value: 1 = keep, 0 = mask
        scale:          scaling factor, caller passes 1/sqrt(D)
        mask_value:     negative value filled at masked positions
                        (exp() ~ 0 in fp32, avoids -inf -> NaN)
    Returns:
        softmax result of shape [..., T, T]
    """
    if scale is not None:
        atten_score = atten_score * scale

    if attention_mask is not None:
        if attention_mask.dtype == torch.bool:
            keep = attention_mask
        else:
            # 0/1 numeric mask -> bool
            keep = attention_mask.to(torch.bool)
        # 1 keep -> unchanged; 0 mask -> fill with mask_value
        atten_score = atten_score.masked_fill(~keep, mask_value)

    return torch.softmax(atten_score, dim=-1)
AuthorsOpenReview ↗

Response to Reviewer Hkte (Part 2/3)


class Cubit(nn.Module):
    def __init__(
        self,
        hidden_size: int,
        num_heads: int,
        head_dim: int = None,
        alpha_zeros: float = 0.0,
        beta_ga: float = 1.0,
        ga_norm: float = 1.0,
        attention_dropout: float = 0.0,
        mask_value: float = -1e4,
    ):
        super().__init__()
        self.hidden_size = hidden_size
        self.num_heads = num_heads
        self.head_dim = head_dim or hidden_size // num_heads
        self.mask_value = mask_value

        self.alpha_zeros = nn.Parameter(torch.tensor(alpha_zeros))
        self.beta_ga = nn.Parameter(torch.tensor(beta_ga))
        self.ga_norm = nn.Parameter(torch.tensor(ga_norm))

        self.b_proj = nn.Linear(hidden_size, num_heads, bias=False)
        self.dropout = nn.Dropout(attention_dropout)

    def forward(
        self,
        hidden_states,
        query_layer,
        key_layer,
        value_layer,
        attention_mask=None,   # [B*H, T, T] or [B, 1, T, T]; 1/True = keep
    ):
        """
        Args:
            hidden_states: [T, B, C]
            query_layer / key_layer / value_layer: [T, B, H, D]
            attention_mask: 1/True = keep, 0/False = mask
        Returns:
            context_layer: [T, B, H*D]
        """
        T, B, H, D = query_layer.shape
        dtype = hidden_states.dtype

        # ---------- 1. reshape to [B*H, T, D] ----------
        key_layer_copy = key_layer
        q = query_layer.reshape(T, B, H, -1).permute(1, 2, 0, 3)   # [B*H, T, D]
        k = key_layer.reshape(T, B, H, -1).permute(1, 2, 0, 3)     # [B*H, T, D]
        v = value_layer.reshape(T, B, H, -1).permute(1, 2, 0, 3)   # [B*H, T, D]

        # ---------- 2. beta gating ----------
        beta = (0.5 + self.beta_ga) * torch.sigmoid(
            self.b_proj(hidden_states).reshape(T, B, H, 1).permute(1, 2, 0, 3)
        ) + 0.5 + self.alpha_zeros                                  # [B*H, T, 1]

        # ---------- 3. normalized k2 to make the condition number better ----------
        k2 = key_layer_copy.reshape(T * B, H, -1).reshape(T, B, H, -1)
        k2_k = k2 / torch.norm(k2, dim=-1, keepdim=True, p=2) * math.sqrt(D)
        k2_k = k2_k.reshape(T, B, H, -1).permute(1, 2, 0, 3) * self.ga_norm
        k2 = k2.permute(1, 2, 0, 3)                                 # [B*H, T, D]

        # ---------- 4. atten_score + custom scale_mask_softmax ----------
        scale = 1.0 / math.sqrt(D)
        atten_score = torch.matmul(q, k.transpose(-2, -1))          # [B*H, T, T]
        A = scale_mask_softmax(
            atten_score,
            attention_mask=attention_mask,
            scale=scale,
            mask_value=self.mask_value,
        )                                                           # [B*H, T, T]

        # ---------- 5. triangular solve: value = solve(A, beta * v) ----------
        with torch.cuda.amp.autocast(dtype=torch.float32):
            v_solved = torch.linalg.solve_triangular(
                A.float(), (v * beta).float(), upper=False
            )
        v_solved = v_solved.to(dtype)                               # [B*H, T, D]

        # ---------- 6. SDPA attention (q x k2) ----------
        if attention_mask is not None:
            sdpa_mask = attention_mask.to(torch.bool)
        else:
            sdpa_mask = None
        context_layer = F.scaled_dot_product_attention(
            q, k2, v_solved, attn_mask=sdpa_mask
        )
        context_layer = self.dropout(context_layer)                 # [B*H, T, D]

        # ---------- 7. restore to [T, B, H*D] ----------
        context_layer = context_layer.reshape(B, H, T, D) \
                                     .permute(2, 0, 1, 3) \
                                     .reshape(T, B, H * D)
        return context_layer
AuthorsOpenReview ↗

Response to Reviewer Hkte (Part 3/3)

Q3: Can the authors specify the exact matrix solved in the experiments, including how the causal softmax is applied, and explain which of the stated KRR properties still hold for that generally nonsymmetric matrix?

A3: For KRR, if q=kik_i, then we will have viv_i. This is still guaranteed so that the long-range retrival performance is better .

The text model erros are related to associate recall. When q=kiq=k_i, we may need the perfect viv_i

TransformerTransformer=NW(q=kt,k,v)=kernel(kt,kt)kernel(kt,kt)+∑j≠tkernel(kt,kj)vt+∑i≠tkernel(kt,ki)kernel(kt,kt)+∑j≠tkernel(kt,kj)vi\text{NW}(q=k_t,k,v) = \frac{\text{kernel}(k_t, k_t)}{\text{kernel}(k_t, k_t) + \sum_{j \neq t} \text{kernel}(k_t, k_j)}v_t +\sum_{i \neq t} \frac{\text{kernel}(k_t, k_i)}{\text{kernel}(k_t, k_t) + \sum_{j \neq t} \text{kernel}(k_t, k_j)} v_i

With the context length increase, the kernel(kt,kt)kernel(kt,kt)+∑j≠tkernel(kt,kj)\frac{\text{kernel}(k_t, k_t)}{\text{kernel}(k_t, k_t) + \sum_{j \neq t} \text{kernel}(k_t, k_j)} will decreases, suggests that the the needed vtv_t will be difficult to be retrive. If the context is sufficient long, the retrived value is only noise

For KRR, we have th following

KRR(q=kt,k,v)=(qk)(kkT)−1v=(ktk)(kkT)v=vt\text{KRR}(q=k_t,k,v)=(qk)(kk^T)^-1v=(k_tk)(kk^T)v=v_t.

Therefore, when q=ktq=k_t, we cold perfect have vtv_t, whatever the hidden size, hidden dimen and so on. The long context ability is only bounded by the computation precous

Q4: The time cost and memory cost

A4: The following is the time cost and memory cost.

MethodCostLength 512Length 1024Length 2048Length 4096Length 8192
TransformerTime Cost (ms)326.3352.0445.3790.72078.3
Memory Cost (MB)7352.07552.08774.015980.039368.0
DeltaFormerTime Cost (ms)459.6567.3823.01766.05307.0
Memory Cost (MB)7352.07868.09542.019052.051704.0
CubitTime Cost (ms)468.2575.8835.31761.75274.5
Memory Cost (MB)7352.07448.09578.020180.051656.0

Q5: The implementation code

A5: We have provided the implementation code in A2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewer

Dear Reviewers,

We address the common concerns below.

Q1: The KRR properies for Cubit and the connection to long context performance (Reviewer Hkte, dbzF)

A1: When q=kik_i, the KRR/Cubit will return viv_i, while Transformer(Nadaraya-Watson Regression) not.

The text model erros are related to associate recall. When q=kiq=k_i, we may need the perfect viv_i

TransformerTransformer=NW(q=kt,k,v)=kernel(kt,kt)kernel(kt,kt)+∑j≠tkernel(kt,kj)vt+∑i≠tkernel(kt,ki)kernel(kt,kt)+∑j≠tkernel(kt,kj)vi\text{NW}(q=k_t,k,v) = \frac{\text{kernel}(k_t, k_t)}{\text{kernel}(k_t, k_t) + \sum_{j \neq t} \text{kernel}(k_t, k_j)}v_t +\sum_{i \neq t} \frac{\text{kernel}(k_t, k_i)}{\text{kernel}(k_t, k_t) + \sum_{j \neq t} \text{kernel}(k_t, k_j)} v_i

With the context length increase, the kernel(kt,kt)kernel(kt,kt)+∑j≠tkernel(kt,kj)\frac{\text{kernel}(k_t, k_t)}{\text{kernel}(k_t, k_t) + \sum_{j \neq t} \text{kernel}(k_t, k_j)} will decreases, suggests that the the needed vtv_t will be difficult to be retrive. If the context is sufficient long, the retrived value is only noise

For KRR, we have th following

KRR(q=kt,k,v)=(qk)(kkT)−1v=(ktk)(kkT)v=vt\text{KRR}(q=k_t,k,v)=(qk)(kk^T)^-1v=(k_tk)(kk^T)v=v_t.

Q2: The time cost and memory cost (Reviewer Hkte,fHBa, dbzF)

A2: The following is the time cost and memory cost.

MethodCostLength 512Length 1024Length 2048Length 4096Length 8192
TransformerTime Cost (ms)326.3352.0445.3790.72078.3
Memory Cost (MB)7352.07552.08774.015980.039368.0
DeltaFormerTime Cost (ms)459.6567.3823.01766.05307.0
Memory Cost (MB)7352.07868.09542.019052.051704.0
CubitTime Cost (ms)468.2575.8835.31761.75274.5
Memory Cost (MB)7352.07448.09578.020180.051656.0

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.