acceptodds
Under review as a conference paper at ICLR 2027

Removing Redundant Computation in a BDH-GPU-Derived Language Model: Exact Recurrent Training and Conditional Neuron Selection

Abstract

Dragon Hatchling (BDH-GPU) combines strict-past Q=K linear attention with a very wide positive neuronal representation. We study Arm-A, a 17.4M-parameter BDH-GPU-derived language model, and ask which apparent redundancy is actually removable. For attention, an exact packed segmented scan preserves the dense operator while avoiding cross-block score materialization. A predeclared Black- well search improves our original dense path by 1.42×, yet recurrence remains 1.52× faster at T =2048. At a fixed 131,072 positions/update, the recurrent/dense advantage grows from 1.04× at T =512 to 2.12× at T =4096. Applying the same mature executor to canonical 25M BDH still leaves recurrence below dense parity (0.964×), and a 96-update matched BF16 trajectory differs by only 7.05 × 10−4 held-out NLL despite non-bitwise parameter drift. For neuronal sparsity, causal controls show that static-support damage is directional rather than magnitude-only: preserving dense direction at the static-support norm reduces ∆NLL from 0.0906 to 0.00819 relative to the converse norm-matched intervention, with the same ordering in KL. Exact algebraic redundancy is therefore removable now; sparse neuronal computation requires conditional support selection.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
efficient training, linear attention, recurrent execution, recurrent language models, machine learning systems, activation sparsity, contextual sparsity, conditional computation, dynamic routing, BDH

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 5 reviews, above 0% of the others.
Reviewer_XM5jOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks which apparent redundancies can be removed from Arm-A, a 17.4M-parameter language model derived from the BDH-GPU architecture.

Critical strengths
  1. Transparent reporting: strengthens its own baseline and reports unfavorable results.
  2. Thorough correctness checks before any timing.
Critical weaknesses
  1. Limited novelty: Chunked linear attention with resets at document boundaries is similar to cited works like RetNet, and Mamba-2. The neuron finding, that input-dependent selection beats a fixed one, similar to Deja Vu, CATS and TEAL. What remains is engineering on one non-standard model.
  2. Presentation not self-contained:
    • The numbers in the abstract (also other places) offers little reference to gauge the significance
    • Internal names are never defined: Arm-A, MB16×4, RB512, "mature executor," "historical dense reference" ...
    • It refers to results that aren't in the paper: "the earlier canonical-BDH negative result," "the earlier logit-perturbation protocol" ... which makes it confusing to read.
  3. Supplements promised ... but no supplements provided ...: the selector results, data hashes and other-GPU results exist only "in the supplement,"
  4. Speedup claims are not convincing, for instance:
    • The dense baselines were capped at 512 token blocks; So possibly some of the 1.52x may come from block sizes ...
    • The matched comparison runs in uncompiled PyTorch. Their own compiled run is about two times faster, and there is no compiled dense baseline.
    • There is no comparison to existing chunked linear-attention kernels.
    • Generally just hard to isolate the improvements.
Questions for authors

Plz address my concerns in the weakness.

Reviewer_XWQ3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies which seemingly redundant computations in the BDH-GPU-derived language model Arm-A (17.4M parameters) can be removed while preserving the original computation. The model retains shared E, Dx, Dy, an 8-step depth recurrence, strictly causal linear attention with Q=K, and a very wide non-negative neuron space, and additionally adds a same-document coordinator and a two-layer writer.

On the attention side, the authors replace the cross-block score matrix with an exact segmented scan of block width 1024: local scores are still computed within blocks, while only the K×D state is passed between blocks and is zeroed at document boundaries. On an RTX PRO 6000, with 131072 positions fixed per step, the strengthened dense baseline (RB512) is 1.42× faster than the original dense; at T=2048 the recurrent scan is a further 1.52× faster than that baseline. With a fixed position budget, the speedup grows from 1.04× at T=512 to 2.12× at T=4096. The same executor ported to the official 25M-parameter BDH is still slower than dense (0.964×). After 96 steps of BF16-aligned training, the held-out NLL differs by only about 7×10⁻⁴, but the parameters have already drifted.

On the neuron side, although the contribution mass is concentrated, its support varies with the input. Interventions at width 320 show that keeping the directions of the static support and restoring the dense magnitude gives ΔNLL = 0.0906; keeping the dense directions and using only the smaller static magnitude lowers ΔNLL to 0.00819. A low-rank selector (rank 16/32/64) using only information before Dy outperforms the fixed support on both evaluation pools. The paper does not claim wall-clock speedup for sparse execution.

Therefore, this paper is an execution and intervention study: the exact recurrence remains faster under a stronger dense baseline, and its advantage grows with context length; the fixed neuron supports tested are harmful in direction, and input-dependent support is more faithful.

Critical strengths

The paper provides a problem worth studying, and it is sufficiently genuine in terms of experiments and explanation.

Critical weaknesses
  1. The paper lacks broad background research and good writing; it has insufficient pages and incorrect formatting.

  2. The StrtS_t r_t discussed in the paper itself belongs to the standard form of linear attention algorithms, and no substantive innovation or significance of the work is apparent; it focuses only on kernel testing.

  3. The measurement baseline used in the paper, a 17.4M model, is too small, and there is no extrapolation to larger scales.

Questions for authors

see weakness

Reviewer_JWS1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies two opportunities for reducing computation in Arm-A, a 17.4M-parameter BDH-GPU-derived language model. First, it implements an exact packed recurrent realization of strict-past linear attention and evaluates its numerical correctness, training throughput, memory usage, and sensitivity to execution geometry. Second, it investigates input-dependent neuronal support through direction/norm interventions and evaluates a prospective low-rank selector. The recurrent implementation achieves a 1.52× speedup over the strongest tested dense baseline at T=2048. The neuronal experiments show that fixed-support damage is partly directional and that conditional selection improves fidelity over the tested fixed support.

Critical strengths

The authors validate forward outputs, gradients, and packed-document isolation, and strengthen the original dense implementation by 1.42× before reporting the remaining recurrent advantage. These controls make the measured speedup more informative than a comparison against the original implementation alone.

Critical weaknesses

The robustness and specific implementation contribution of the recurrent speedup remain insufficiently established. The strongest tested dense comparator uses RB512, while the recurrent path uses B1024. Section 4.2 shows that increasing the recurrent block size from 256 to 1024 improves throughput by 37.2%, demonstrating substantial sensitivity to execution geometry. Dense RB1024 and RB2048 remain unevaluated, so their performance could materially affect the practical significance of the reported advantage. Since the underlying recurrence is established, the packed implementation is central to the contribution; the main text would also benefit from explicit segmented carry updates and the backward storage/recomputation strategy.

The incremental value of the low-rank selector is not isolated from simpler input-dependent selection. Its scores combine the already available input-dependent vector x with predicted Dy activations, but the described prospective comparison uses fixed global support. Comparisons with TopK(x) and TopK(x multiplied by the calibration-set mean of y) would establish whether low-rank prediction adds useful information beyond the existing gate. In addition, the main text reports positive improvement intervals but defers the rank-wise effect sizes to the supplement, making the practical magnitude of the selector improvement difficult to assess from the main paper.

Questions for authors

How do dense RB1024 and RB2048 perform under the same operator, precision, optimizer, and effective batch size? If they exceed memory at the primary microbatch, can you compare both paths at the largest shared feasible microbatch?

How does the low-rank selector compare with TopK(x) and TopK(x multiplied by the calibration-set mean of y)? After selection, are the selected Dy outputs computed exactly or taken from the low-rank approximation?

Could you summarize fixed-support, local-oracle, and rank-16/32/64 ΔNLL and KL results for both evaluation pools, including confidence intervals, evaluation-row counts, and the depth-consistency criterion? These results are referenced as supplementary material.

Could you provide the segmented chunk forward/backward pseudocode, including carry-read masking, final-segment state updates, row isolation, and which intermediate states are saved or recomputed?

Reviewer_NX3FOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates redundant computation in Arm-A, a model derived from BDH-GPU. It improves training throughput through an equivalent blockwise loop implementation, uses intervention experiments to analyze the degradation in predictive performance caused by fixed neuron selection, and further explores input-dependent neuron selection methods.

Critical strengths
  1. The attention implementation optimization undergoes reasonably thorough correctness checks, covering outputs, gradients, document isolation, and short-term training consistency. The paper also reports throughput for complete training updates.
  2. Through intervention experiments that distinguish update direction from magnitude, the paper shows that the harm caused by fixed neuron selection cannot be explained solely by a reduction in total contribution. This provides evidence for the need for input-dependent selection.
Critical weaknesses
  1. The paper primarily improves execution efficiency by modifying the computation process, and both components appear to be empirical and engineering-oriented optimizations. Further discussion and targeted control experiments are needed to explain why these approaches were chosen and under what conditions they are effective. The work also does not establish a cohesive new model architecture or computational approach.
  2. The experimental scope is limited. The main speedup results come from Arm-A with approximately 17M parameters, while the dense baseline search only covers block widths up to 512, leaving a limited set of comparable baselines. The experiments also lack sufficient ablations to isolate the sources of improvement, making the respective effects of neuron width, document concatenation, and architectural changes unclear.
  3. Several aspects of the PDF formatting are inconsistent with the ICLR template, and the supplementary material referenced in the main text could not be found in the submission materials currently available. This hinders verification of key results and raises concerns about the care taken in presenting the paper and organizing the supporting materials.
Questions for authors
  1. Why were the current execution and neuron selection approaches chosen? Could targeted control experiments clarify the conditions under which they are effective and the main insights beyond engineering optimization?
  2. Would dense baselines with larger block widths change the speedup conclusions? Could you further disentangle the contributions of neuron width, document concatenation, and architectural changes to the reported gains?
  3. Please clarify where the supplementary material cited in the main text can be found and check whether the PDF complies with the ICLR formatting requirements.
Reviewer_TfdqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies a 17M parameter BDH-GPU LM. BDH refers to "Dragon Hatchling" which is a different kind of language model than the standard transformer. It's more like linear attention only, with some other differences. The authors ask which apparent redundancy in the BDH-GPU LM is redundant. They find that they can speed up the model under some circumstances by carefully taking advantage of this.

Critical strengths

This paper is quite dense, and for someone like me who is not an expert in BDH models, it has a lot to say. For people who care a lot about BDH models, this might have something new to say.

Critical weaknesses

After reading this paper, I'm not convinced BDH models are good for anything, and I'm not sure what the authors did to make them better. It's a difficult paper to read. At times I was wondering if I was in the wrong field. For this to be accepted at ICLR, I expect the authors to help the reader out more.

Questions for authors

Why should we care about BDH models? What are they good for? Is this primarily a pure architecture study, with no need for quantitative results on what the model can actually do?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.