- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates efficient training algorithms for linear Transformers with block-diagonal gating. Although linear Transformers offer favorable computational complexity for long-sequence inference, more expressive gating mechanisms can make parallel training substantially more challenging. In particular, existing chunk-wise parallelization methods may rely on matrix inversion involving gating matrices, which can limit their applicability to general block-diagonal gating structures and raise concerns regarding numerical stability and GPU execution efficiency.To address these issues, the paper proposes a generalized chunk-wise parallel algorithm that avoids matrix inversion. By combining local factorization with hierarchical tiling, the algorithm reformulates recurrent computations into matrix operations that are better suited to GPU and Tensor Core execution. Building on this algorithm, the paper introduces BGLT, which reduces the number of gating parameters through Kronecker factorization and employs an SVD-style parameterization to improve numerical stability. The paper further proposes ΔBGLT, which incorporates the Delta Rule to enhance information mixing during state updates.The experiments cover synthetic tasks, language modeling with 7B-scale models, training throughput, and activation memory consumption. The results demonstrate clear advantages on several synthetic state-tracking tasks and competitive performance on language modeling benchmarks. However, the overall gains on real-world language modeling tasks appear relatively modest, and the trade-off between performance improvements and additional computational costs requires further justification.
Critical strengths
- The problem is technically important and well motivated.The paper addresses not merely the introduction of more expressive gating mechanisms, but the fundamental challenge of efficiently parallelizing the training of more expressive recurrent structures. This problem lies at the intersection of algorithm design, numerical stability, and GPU execution efficiency. 2. The core algorithmic contribution is clearly defined.The proposed inversion-free chunk-wise parallel algorithm combines local factorization and hierarchical tiling to provide an alternative to conventional sequential recurrence for general block-diagonal gating structures. This is a meaningful algorithmic direction for improving the trainability of structured linear Transformers. 3. The model design and training algorithm are reasonably well integrated.BGLT combines Kronecker factorization to reduce gating parameter overhead with a constrained parameterization intended to improve numerical stability. Together, these choices provide a coherent design that considers model expressiveness, parameter efficiency, and hardware execution. 4. The experimental evaluation covers multiple dimensions, and the synthetic results demonstrate substantial potential.The paper evaluates synthetic tasks, 7B-scale language models, training throughput, and activation memory consumption. On synthetic benchmarks, BGLT performs strongly on tasks such as state tracking, modular arithmetic, and Copy Memory. In particular, its state-tracking performance remains high beyond the training sequence length, suggesting promising generalization to longer sequences in this setting.
Critical weaknesses
- The advantages on synthetic tasks do not yet translate into equally convincing gains on real-world language modeling, and the differences across tasks require further explanation.Although BGLT performs strongly on several synthetic tasks, Table 2 indicates that ΔBGLT achieves only a marginal difference from KDA in average language-modeling metrics, with limited improvement on the Recall task. Whether these gains are statistically robust and consistently generalize across different tasks remains unclear. Additional experiments across random seeds and task settings would help establish the reliability of the reported improvements.Furthermore, Table 3 reports an accuracy of 49.1% for BGLT on the k-HOPS task, compared with 99.2% for ΔBGLT. This substantial gap suggests that even with a more expressive block-diagonal gating structure, different tasks may benefit from different state-update mechanisms. The paper would be strengthened by a more detailed analysis of the reasons for this discrepancy and its implications for practical model selection. 2. The independent contributions of the individual components require more comprehensive ablation studies.The proposed approach incorporates several design choices, including the inversion-free chunk-wise algorithm, Kronecker factorization, SVD-style parameterization, and the Delta Rule. The current results do not fully disentangle the individual contributions of these components to the final performance.I encourage the authors to investigate the effects of varying the block size on accuracy, throughput, and memory consumption; removing Kronecker factorization; removing the SVD-based constraint and evaluating the resulting numerical stability; and incorporating the Delta Rule while measuring both capability improvements and additional computational costs. It would also be valuable to separate the gains attributable to the training algorithm from those resulting from changes to the model architecture.Without these comparisons, it remains difficult to determine whether the reported improvements primarily arise from the core algorithm, the parameterization strategy, or the additional state-update mechanism. 3. The claimed computational efficiency advantages require more comprehensive validation.Efficient training is a central motivation of this work, but more expressive gating structures may also introduce additional computational overhead. According to Figure 4, BGLT has approximately 5%–9% lower prefill throughput than GLA, while ΔBGLT incurs a further reduction of approximately 9%–11%. Figure G.4 also indicates that BGLT uses approximately 9%–13% more activation memory than KDA or GLA. Moreover, Figure G.7 reports that the forward and backward kernel runtimes of ΔBGLT are approximately 1.49× and 1.52× those of KDA, respectively, with a substantial portion of the overhead attributed to attention/Gram matrix construction.These results suggest that improvements in algorithmic parallelization do not necessarily translate into end-to-end training speedups. The current evaluation primarily reports throughput and memory consumption under specific configurations. A more comprehensive performance characterization across sequence lengths, batch sizes, and hardware platforms is still needed. It is also unclear whether the baseline implementations receive comparable levels of optimization.A more rigorous evaluation would compare end-to-end training time under consistent hardware and training configurations, clarify the implementation details of the compared methods, and quantify the practical effects of recomputation strategies on throughput and memory consumption. The throughput comparisons should also explicitly state their reference baselines and percentage-change definitions. 4. The scope of applicability and the rationale behind key design choices need further clarification.Although the proposed algorithm has potential applicability to general block-diagonal gating structures, its practical efficiency may depend strongly on block size, tiling strategy, and hardware execution characteristics. The paper would benefit from a clearer discussion of which structures and operating conditions are most suitable for the proposed method, as well as the circumstances under which its advantages may diminish.In particular, the effects of block size and hierarchical tiling on numerical stability, computational efficiency, and model performance remain insufficiently characterized. It would be useful to establish more general guidelines for selecting these design parameters and to explain how parameter efficiency, expressiveness, memory overhead, and training speed should be balanced in practice.
Questions for authors
-
Efficiency and comparison conditions. What is the intended efficiency claim: making general block-diagonal gates practically trainable, or improving end-to-end training efficiency over existing linear Transformers? Please clarify which comparisons use matched hardware, precision, batch size, sequence length, and recomputation settings, and whether the baselines receive comparable implementation optimization. Do existing measurements establish a favorable end-to-end capability–cost trade-off in any operating regime?
-
Component attribution and task dependence. Which controlled results isolate the effects of block size, Kronecker factorization, the SVD-style constraint, and the Delta Rule? In particular, what explains the 49.1% versus 99.2% k-HOPS gap between BGLT and ΔBGLT? Clarifying which conclusions are supported by existing controls, and providing targeted additional evidence where available, would affect my assessment of the architectural contribution.
-
Reliability of the language-modeling gains. Are the small differences between ΔBGLT and KDA in Table 2 consistent across independent training runs or supported by uncertainty estimates? If only single-run results are available, please clarify this limitation and whether the task-level results provide evidence of a consistent advantage.
10 comments
Q1 / W3: Kernel efficiency -- cont'd
The current evaluations primarily report throughput and memory consumptions under specific configurations. A more comprehensive performance characterization across sequence lengths, batch sizes, and hardware platforms is still needed
While due to space limitations in the main manuscript we could only report throughput timings for the configuration used for our large scale training (Fig4), we profile our kernel more in detail in AppG.4. There we provide, among other relevant information, training throughputs (FigG3) and decoding times (FigG5) for varying sequence lengths and batch sizes configurations. We conduct this analysis at different granularities, measuring runtimes needed for passes through the whole model (AppG4.1), through the single BGLT layer (AppG4.2), and through the sole BGLT recurrent cell (AppG4.3), both under torch.compile (closer to actual deployment configuration) and eager execution (to provide a more specific, per sub-kernel breakdown of runtimes, although suboptimal).
On hardware, we focus on B200 as this is the current target for large-scale training. Our kernels use specific features from the Blackwell architectures, so porting them to Hopper (or even more so to Ampere) would require ad-hoc optimizations and possibly re-implementations.
Quantify the practical effects of recomputation strategies on throughput and memory consumption
In FigG4 we already show memory consumption for the two extrema (store nothing and recompute everything vs store everything and recompute nothing), to give practitioners an idea of what the trade-off entails in terms of GPU utilisation. In practice, for large-scale deployment, GPUs are generally deployed at saturation, so it is preferable to reduce memory consumption whenever possible. For this reason, we focus our timing analysis on the same configuration we used for training, that is, full recomputation. We point out that, in terms of throughput, this is the most unfavourable scenario for us, as avoiding recomputation by storing intermediates from the forward pass reduces runtime. To get an idea of the impact of recomputing the various components on runtime, we refer to FigG7: shaded regions denote time spent recomputing quantities, which can potentially be saved at the cost of extra memory consumption.
Clarify the implementation details of the compared methods
We provide a full guide of the mathematical operations necessary for our cell application in AppF, which is what the KDA baseline also follows, for the largest part. For the specific kernel components of our cell that differ from KDA, in AppG we provide a detailed walk-through both for forward (see also Sec3.1) and backward passes, together with pseudo-code for the most relevant parts (Alg1-6), and a description of how the mathematical operations map to stages in the algorithm. We will soon provide open access to the code implemented in the course of this project, so that practitioners can directly access the implementation details. For KDA, we use its shipped implementation. We will further clarify all these points in the main text, but if the reviewer deems this insufficient, we kindly ask them to point out precisely what other details they would like outlined.
The throughput comparisons should also explicitly state their reference baselines and percentage-change definitions
The main baselines are GLA and KDA (BGLT being interpretable as an extension of the former, and BGLT as an extension of the latter), and we report this in L380 and throughout the paper. Explicit throughput numbers are directly reported in the graphs, but we are happy to include percentages differences as well.
Q2 / W2: Ablations on individual components
The proposed approach incorporates several design choices, including the inversion-free chunkwise algorithm, Kronecker factorization, SVD-style parameterization, and the DeltaRule. The current results do not fully disentangle the individual contributions
While it’s true that we introduce a number of design choices in our BGLT definition, we provide justifications for each of them in Sec3.2: these boil down to a balance between computational efficiency (kronecker factors ensure good mapping to tensor cores while limiting memory movements and parameter count), training stability (L2 normalisation via SVD is useful to prevent state/gradient blow up during training), and expressivity (4x4 blocks are the minimal requirement for solving S5). Regarding the specific ablation requests from the reviewer:
- The inversion-free chunkwise algorithm cannot impact model quality: it computes the exact quantity, only in a different manner, which is stable for block diagonal gates. It is also not something we can ablate over, as there is no viable alternative for training block-diagonal transformers at scale: sequential recurrence is too slow, parallel scans are too memory-intensive, and inverse-based factorisation cause immediate loss of numerical precision at best and NaNs at worst (see Lem3). That is precisely the reason why this algorithm is a key contribution in our paper.
- For the SVD-style parameterization, we provide an ablation result in AppC2. For training stability, it is paramount to control the gates norm, as we need to compute cumulative products thereof, spanning whole chunks of sequences, which can grow exponentially if unbounded. normalisation controls in one go both forward and backward pass, and doing so via SVD parameterisation is a natural choice which gives us direct control over singular value distribution. As an alternative, we explore also normalisation, and provide results in FigC2: works slightly better in practice.
- Regarding DeltaRule, we provide performance on all the synthetic tasks considered (AppD), and complete timing results variants (AppG) for both BGLT and its DeltaRule variant BGLT. In general BGLT performs better, which is why we prioritised the training of this version of our model at large-scale for language modeling tasks, also given the substantial GPU resources necessary to train at 7B. Nonetheless, in our response to Reviewer NgEX W3, we expand Tab2 with results from BGLT at 1.5B, spanning multiple seeds (see also discussion there).
We thank the reviewer for the thorough review. We address the raised concerns below, organised by topics
The first. The goal of our algorithm is to make block-diagonal gates practically trainable at scale: that is, we want to generalise existing algorithms so that they can be applied to more complex gate structures, and not improve the runtime of already-existing, specialised ones. Since ours is a more general algorithm, and requires more complex operations than when simpler structures are involved (like for GLA, GDN, and KDA linear transformers), it would be unreasonable to expect it could beat those specialised ones at runtime. So our goal is only to show that its runtime is comparable with that of more specialised algorithms, and not that we provide improved efficiency over those. For comparison (FigG3), ΔBGLT’s prefill overhead (+13%) is comparable to that of KDA over GLA (+11%).
We hope that our clarification above can help address the reviewer’s concerns. To further reiterate, our generalised chunkwise parallel algorithm does not aim to provide computational advantages over existing baselines: indeed, since we develop it with more complex gates in mind, it necessarily introduces additional computation. What our algorithm allows for, is efficient training, even for more complex gates (with throughputs comparable to existing baselines). The generalised chunkwise parallel algorithm is not to be intended as a substitute to other, inverse-based ones (like KDA, GDN or GLA’s) but rather as an efficient alternative when explicit factorization via inversion is not feasible (see eg L23-35, L84-87, L200-205). For diagonal gates, inversion is very much feasible (each gate component can be treated in log-space, where inversion reduces to subtraction), similarly for 2x2 orthogonal gates (each gate is uniquely parameterised by an angle, so inversion reduces to rotation by the opposite angle), so a practitioner should rely on already-existing, specialised kernels in these cases. Our algorithm comes into play in cases like general block-diagonal gates, where explicit inversion is not feasible (see discussion around Lem3).
Yes, for those tasks where the additional capability is actually needed. On A5, BGLT quickly reaches 100%, while KDA ends its full training run at 6.2% (see TabD1 in the Appendix). Similarly on copy-memory and modular arithmetic without brackets. In language modelling, ΔBGLT is ahead of KDA on recall tasks at 7B, which is a particularly arduous tasks for linear RNNs.
The main baseline we consider is the one currently deployed in Kimi K3’s KDA cell. This is a hybrid Transformer used in production-level LLMs, so we can expect the kernel implementation to be reasonably optimized. If anything, the comparison is unfavourable to our side, as our core attention assembly kernel is entirely novel and built from the ground-up by the authors in the course of this project, whereas KDA’s kernel is already mature.
All comparisons run on matched hardware (a B200 GPU, which is what the algorithms considered are more heavily optimised for) and at matched precision (generally speaking, bf16 except for anything involving operations along the sequence length, which is treated in fp32, as is the norm for linear RNNs: see also Fig captions in AppG4). Whenever we vary batch size and sequence length, we explicitly highlight this (FigG3-6). Regarding recomputation, all timing results are reported for the algorithm we actually deploy at training, which recomputes the most intermediate quantities, following the KDA recipe. In essence, we aimed at being as fair as possible throughout our comparisons, matching KDA’s configuration as closely as possible.