acceptodds
Under review as a conference paper at ICLR 2027

LeetDecoding: An Experimental Survey and Scalability Evaluation for Prefilling With Causal Linear Attention

Abstract

Significant progress has been made in accelerating transformer-based large language models (LLMs). Among various promising approaches, *exponentially decaying causal linear attention* has emerged as a highly effective operator, yet its existing implementations remain fragmented across the literature. In this paper, we present an experimental survey of this fundamental operator for LLM prefill inference. We also introduce `LeetDecoding`, the first comprehensive library and benchmark providing extensive computation routines and a unified testbed for scalability evaluation. `LeetDecoding` addresses multiple critical gaps in the field: the absence of rigorous complexity analyses, the lack of a systematic consolidation of existing computation methods (previously scattered as secondary contributions without direct comparison), and the dearth of extensive empirical evaluations. Our findings reveal that no single computation method universally dominates across diverse configurations, and that hardware-aware optimizations are paramount for substantially improving overall efficiency.

Primary area:
datasets and benchmarks
Keywords:
causal linear attention, computation method

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XrESOpenReview ↗
Rating
2
Confidence
2(+1)
Informativeness
1
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This work presents a library called LeetDecoding that implements a set of causal linear attention methods. The authors carefully analyze the complexity of each method and conduct a comprehensive evaluation of these methods in terms of scalability, latency, and numerical stability. Experiments show that each method has its own advantages and no single method is superior in all aspects.

Critical strengths
  1. This work implements a set of causal linear attention methods and performs a unified evaluation of these methods.
  2. The proposed library is easy to use and can be a useful tool for researchers and practitioners in the field of efficient transformers.
Critical weaknesses

Disclaimer: I am not an expert in this area, and my comments are only an educated guess based on my understanding of the paper.

  1. Under the topic of linear attention, I am wondering what is the strength of the leetdecoding library compared to other famous existing libraries such as fla [1]. The only comparison I can find is in Table 13, where the fla implementation has the similar performance as the leetdecoding implementation. But in main results I cannot find any further comparison with fla. As far as I know, fla implements most of the well-known linear attention methods and is also a highly optimized library. I am wondering if the authors can provide more discussion on the comparison with fla, and what is the strength of leetdecoding compared to it. If there is no significant strength, I do not think this work has enough contribution to be accepted as a paper in ICLR.
  2. I feel like this work does not bring much new insights to the field of linear attention. One thing that the authors emphasize is that Cdotp is faster than LA under RTX5090 with fp32 and batch size 16, with an accelerate of about 1.2x (Section C.1). I can hardly say that this is a significant contribution to the field -- this is only a narrow observation on a specific hardware and a specific setting.

[1] FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism. Yang, Songlin and Zhang, Yu

Questions for authors

See the weakness section.

4 comments

We appreciate your useful comments on the contribution of our work. By further highlighting the comparative evaluation and its practical implications, we hope to address your concerns.

W1: Strength relative to FLA

what is the strength of the leetdecoding library compared to other famous existing libraries such as fla [1]

We respectfully refer the reviewer to General Response R1 for a comprehensive discussion.

To directly answer your question regarding our core strength: our contribution is establishing a rigorous scientific benchmark across fundamentally different algorithmic paradigms, rather than merely engineering a single faster production library.

While FLA provides horizontal architectural breadth (primarily chunk-wise kernels), LeetDecoding provides vertical algorithmic depth by unifying token-wise CUDA recurrence (Cdotp), VQ-style block methods (Bb), and O(Nlog⁡N)\mathcal{O}(N\log N) recursive splitting (Recur) alongside end-to-end prefill breakdowns and zero-shot numerical evaluations. Furthermore, in Table 13, our optimized block-wise kernel (LA) performs slightly faster than FLA's retention reference (LA_ref, 351.92 s vs. 366.88 s at batch size 1), confirming that our implementation is competitive with production baselines.

W2: Limited insight

this is only a narrow observation on a specific hardware and a specific setting

We kindly refer the reviewer to General Response R2. The reviewer observes that the 1.2x advantage of Cdotp over LA is tied to a specific configuration (RTX 5090, FP32, batch size 16). However, we respectfully argue that identifying this specific regime constitutes a meaningful contribution.

Because chunk-wise implementations are assumed to be universally optimal, demonstrating a specific regime where a token-wise CUDA kernel outperforms an optimized chunk-wise Triton kernel underscores the value of context-aware benchmarking. We will revise the discussion to frame this clearly within our broader trade-off analysis.

AuthorsOpenReview ↗

Follow-up on Response

Dear Reviewer XrES,

As the discussion period draws to a close in roughly two days, we are following up to check whether our rebuttal has addressed your questions regarding our contributions relative to FLA.

Specifically, we detailed how LeetDecoding complements FLA's horizontal architectural breadth with vertical cross-paradigm depth (where our optimized kernel also outperforms FLA's retention baseline in Table 13) and explained how our findings challenge the assumption that chunk-wise methods universally dominate. We remain fully available for any further clarifications and would greatly appreciate your feedback before the interaction window closes.

Reviewer_XrESOpenReview ↗

Response to Authors

I thank the authors for their dedicated explanation. I fully understand that the contribution of this work is to propose a linear attention benchmarking library, not to provide a fast production library. The authors draw the conclusion "no single method universally dominates across operational contexts" based on a specific configuration (RTX 5090, FP32, batch size 16) and recogize this as a meaningful contribution.

I believe this work is sound and the authors faithfully report the result. But instead of "no single method universally dominates across operational contexts", the experiments actually show that in general chunk-wise implementations are indeed the optimal. This perhaps is a contribution, but I do not think this brings insight to the majority of the community and is enough to accepted as a conference paper in ICLR. After consulting experts in this area, I decide to raise the soundness score from 2 to 3 and raise confidence from 1 to 2. I respectfully suggest the authors consider publishing this paper as a workshop paper or at venues like AACL or EACL.

Dear Reviewer XrES,

We thank you for engaging in the discussion and for raising the soundness score of our work to 3, confirming the rigor of our theoretical derivations and empirical evaluations. However, we respectfully correct two critical mischaracterizations in your concluding remarks:

1. Reducing a multi-dimensional benchmark to a single hardware configuration.

Our conclusions do not rest solely on the RTX 5090 FP32 setting. Even where chunk-wise implementations hold a broad latency advantage, scientifically establishing its operational boundaries (rather than treating it as folklore) requires a controlled cross-paradigm benchmark. More importantly, focusing purely on raw latency overlooks the pillars of our study:

  • Theoretical foundation: Unifying disparate paradigms under an explicit non-iterative matrix formulation and mathematically proving the suboptimal O(Nlog⁡N)\mathcal{O}(N\log N) complexity of recursive splitting.
  • Intra-paradigm system insights: Demonstrating that hardware-aware memory prefetching and framework choices alter latency by over 50% for an identical mathematical algorithm.
  • Model-level numerical fidelity: Exposing that mathematically equivalent kernels can induce severe perplexity degradation (e.g., PPL rising from 14.02 to 333.30) and up to 92% next-token disagreement when substituted into pretrained LLMs without recovery training.

2. Misalignment of the suggested venues.

This work investigates low-level CUDA/Triton kernel engineering, hardware memory hierarchies, asymptotic time-space complexity, and floating-point numerical stability. Suggesting computational linguistics venues such as AACL or EACL reflects a fundamental mismatch with the technical nature of our contribution. Rigorous algorithmic taxonomies and hardware-aware operator benchmarks sit squarely within core machine learning and the explicit scope of the Datasets and Benchmarks track.

We appreciate the time you took to review our submission and your acknowledgment of its technical soundness.

Reviewer_oXRjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LeetDecoding provides a library and experimental survey of implementations for exponentially decaying causal linear attention. It combines recurrent, block-wise, and recursive computation methods into a single interface, performs complexity analyzes, and evaluates latency, memory, and numerical behavior. The experiments include synthetic operator benchmarks and model-level prefill measurements for TransNormerLLM-7B and RetNet-1.3B on RTX 5090 and A800 GPUs.

The primary contribution to the Datasets & Benchmarks track is the unified evaluation framework, which has the potential to benefit researchers and practitioners.

Critical strengths
  1. The benchmark focuses on a useful systems problem. Comparing implementations of the same mathematical operator can help distinguish between computational decisions and differences in model architecture or predictive quality. The shared interface and support for decaying and non-decaying masks may make these comparisons easier to replicate and extend.

  2. The evaluation includes measurements at both the operator and model levels. Latency breakdowns indicate when attention acceleration leads to end-to-end improvements and when other model components become the bottleneck. The experiments also look at different batch sizes, precisions, and two GPU architectures.

  3. The paper explicitly examines numerical behavior. Comparisons with an FP64 reference, as well as measurements of model-level perplexity and next-token disagreement, reveal significant differences that a latency-only benchmark would miss. Prefetching and precision ablations provide useful implementation-level evidence.

Critical weaknesses
  1. The benchmark's added value over existing resources is not adequately established. Existing libraries, such as Flash Linear Attention (FLA), already include optimized linear-attention implementations and benchmarking tools. Although LeetDecoding combines different computational formulations and numerical evaluations, the paper does not clearly show which additional capabilities or insights make it a significant contribution. The FLA reference provided in the RetNet evaluation is beneficial, but a more comprehensive comparison of supported methods, workloads and benchmarking capabilities is required.

  2. The limited operator coverage limits its applicability to newer linear-attention architectures. The evaluation focuses on exponentially decaying causal linear attention without explaining how its findings relate to mechanisms such as DeltaNet, gated delta-rule models, and Kimi Delta Attention. In particular, the implications of input-dependent gating and various state-update rules have yet to be investigated. These mechanisms are not interchangeable implementations of the operator studied here, so including them would necessitate broadening the benchmark's scope. Nonetheless, the paper should explain why its chosen scope remains important and which conclusions apply to these newer architectures.

  3. The comparisons do not distinguish algorithmic differences from implementation quality. PyTorch implements block-based and recursive methods, while Cdotp and Lightning Attention-2 use native CUDA or Triton implementations. These comparisons are useful for deciding between available implementations, but performance differences also account for framework overhead, kernel launches, and hardware utilization. Although the paper recognizes this distinction, its conclusions should refrain from interpreting implementation rankings as evidence of inherent algorithmic superiority.

  4. The artifact and measurement protocol require further documentation. The materials provided did not allow me to assess the repository, so I was unable to confirm the resource's usability or reproducibility. The manuscript should include software and compiler versions, upstream implementation revisions, synchronization and timing procedures, warm-up and compilation handling, and how the parameters are tuned. The paper should also explicitly state the precision used for matrix multiplication, rather than referring only to FP32 inputs and Triton’s defaults. For instance, using TF32 versus full FP32 multiplication can affect both runtime and numerical accuracy.

  5. Numerical validity is not fully integrated into performance comparisons. Table 20 shows significant discrepancies among several BF16 implementations. In comparison to native TransNormerLLM, Cdotp modifies the next-token prediction on 92% of prompts and raises perplexity from 14.02 to 16.92; Recur raises perplexity to approximately 333.30. These distinctions are relevant for implementations that aim to compute the same operator. Reporting them is useful, but latency rankings should include explicit correctness criteria or show speed-error trade-offs. Otherwise, faster implementations may appear preferable, despite producing significantly different model outputs.

  6. The workload coverage is limited when compared to the broader scalability conclusions. The principal latency sweep sets the head count to 32 and the feature dimensions to 128. Other experiments include some variation, but they do not systematically determine how performance rankings change across head counts, feature dimensions, or unequal query/key and value dimensions. Model-level evaluation is also limited to two models and 200 TREC prompts. A broader shape and shorter sweep lengths would improve the benchmark's usefulness for practical deployment.

Questions for authors
  1. What additional evaluations or insights does LeetDecoding provide beyond existing libraries like FLA, and why do these differences make a significant contribution to the community?

  2. Which findings apply to input-dependent gated linear attention and delta-rule architectures, and which are specific to exponential decay? Could the framework support DeltaNet, gated delta-rule models, or Kimi Delta Attention without a significant redesign?

  3. Do the primary performance rankings remain consistent across different head counts, feature dimensions, unequal query/key and value dimensions, and prompt-length distributions?

  4. What numerical tolerances govern whether an implementation is a valid substitute? How would performance rankings change if implementations that exceeded these tolerances were excluded?

  5. Can the benchmark possibly support practical implementation selection? Given the finding that no method consistently dominates, can the the benchmark be used to derive a selection rule based on workload, hardware, precision, and numerical tolerance, and evaluate it on configurations beyond those used to derive it?

3 comments

We thank you for your careful review and appreciate all your helpful feedback. We hope our following answers properly address your concerns.

W1/Q1: Added value over FLA

What additional evaluations or insights does LeetDecoding provide beyond existing libraries like FLA, and why do these differences make a significant contribution to the community?

We respectfully refer the reviewer to General Response R1. We appreciate your recognition that comparing implementations of the same mathematical operator separates computational choices from architectural differences. In the revision, we will make the comparison with FLA more explicit.

W2/Q2: Input-dependent gating and delta-rule architectures

Which findings apply to input-dependent gated linear attention and delta-rule architectures, and which are specific to exponential decay? Could the framework support DeltaNet, gated delta-rule models, or Kimi Delta Attention without a significant redesign?

We kindly refer the reviewer to General Response R3.

Our benchmark strictly evaluates fixed-decay causal linear attention to isolate implementation choices from mathematical changes. While our rigorous evaluation framework provides a reusable blueprint for newer operators, their distinct state updates, numerical references, and model adapters require separate treatment. We will explicitly define these scoping boundaries in the revision.

W3: Distinguishing algorithms from implementations

These comparisons are useful for deciding between available implementations, but performance differences also account for framework overhead, kernel launches, and hardware utilization. Although the paper recognizes this distinction, its conclusions should refrain from interpreting implementation rankings as evidence of inherent algorithmic superiority.

We respectfully clarify that Cdotp and LA are not limited to native CUDA or Triton in our benchmark; we also evaluate their PyTorch counterparts (Cdotp_torch and LA_torch) alongside Vanilla, Bb, and Recur (Table 2, Section 6.1). Together with our within-algorithm Triton ablations (LA (original), LA (prefetch), and LA), these comparisons explicitly examine framework overhead and implementation quality alongside our theoretical complexity analysis (Section 4, Appendix G; implementation details in Section 5), where bounds such as the recursive O(Nlog⁡N)\mathcal{O}(N\log N) factor hold independently of runtime.

Planned Revisions: As the reviewer notes, Section 6 already recognizes that sharing a programming framework does not guarantee identical kernel launch overhead or hardware utilization. We will make the Cdotp_torch and LA_torch comparisons more prominent and revise our conclusion wording to consistently describe latency rankings as results for the evaluated implementations, rather than definitive evidence of inherent algorithmic superiority.

W4: Artifact Access, Measurement Protocol, and Arithmetic Precision

The materials provided did not allow me to assess the repository, so I was unable to confirm the resource's usability or reproducibility.

Artifact Access and ICLR Guidelines:

We respectfully refer the reviewer to the Reproducibility Statement in our manuscript. In strict compliance with official ICLR guidelines (see "Q: How can we make our code available for reviewing anonymously?"), we withheld the repository link at submission, and as promised, provided the fully reproducible anonymous repository via the official comment channel directed to reviewers and ACs after the discussion forum opened. All benchmark scripts, environment configurations, and kernel implementations are directly verifiable in this repository.

1. Measurement Protocol.

Rigorous measurement procedures are already documented in the submission. For example, the manuscript specifies that single-layer latency is averaged over 15 measurements, and model-level evaluations detail the prompt aggregation and strict warm-up handling (e.g., discarding the first of four runs). To improve readability, we will consolidate these existing details along with the specific software environment versions (CUDA, Triton, PyTorch), synchronization, and autotuning procedures into a centralized appendix.

2. Arithmetic Precision.

Beyond the input/output and FP32 accumulation dtypes already detailed in Appendix A.1, the exact matrix-multiplication configurations are directly inspectable in our released codebase. To make the manuscript fully self-contained, we will update Appendix A.1 to explicitly state the exact internal matmul precision modes (e.g., IEEE FP32 vs. TF32 settings) rather than referring generally to framework defaults.

W5/Q4: Numerical Validity

What numerical tolerances govern whether an implementation is a valid substitute? How would performance rankings change if implementations that exceeded these tolerances were excluded?

We respectfully clarify that exposing these exact numerical discrepancies is a core contribution of our benchmark, not an omission. As the reviewer aptly noted in the "Strengths" section, our evaluations "reveal significant differences that a latency-only benchmark would miss."

Governing Tolerances and Ranking Changes (Answering Q4).

Table 20 evaluates zero-shot substitution into fixed pretrained weights without recovery training. Rather than applying a single arbitrary pass/fail threshold (since acceptable error margins depend heavily on the downstream task and whether a model is pretrained with that specific kernel), we report continuous fidelity metrics (PPL, hidden-state error, and next-token disagreement) to caution against blindly deploying fast implementations like Cdotp or Recur in BF16.

To directly answer how rankings change if implementations exceeding tolerances are excluded: under a practical zero-shot substitution criterion (e.g., requiring ΔPPL<0.1\Delta\text{PPL} < 0.1 or <10%<10\% next-token disagreement), Cdotp (PPL 16.92, 92% disagreement) and Recur (PPL 333.30) are disqualified in BF16, leaving LA (PPL 14.06 vs. native 14.02, 5% disagreement) as the fastest valid implementation. Conversely, in FP32, all evaluated methods maintain high fidelity and the unfiltered latency rankings hold directly.

Planned Revisions: We fully agree that speed-error trade-offs should be immediately visible alongside latency rankings. In the revised latency tables and Sections 6.1-6.2, we will add explicit visual markers (e.g., daggers/footnotes on Cdotp and Recur under BF16) and direct cross-references to Table 20, ensuring readers never interpret raw latency rankings in isolation from numerical fidelity.

W6/Q3: Workload Coverage and Benchmark Scope

Do the primary performance rankings remain consistent across different head counts, feature dimensions, unequal query/key and value dimensions, and prompt-length distributions?

We respectfully disagree with the characterization of our workload coverage as limited. Our experiments vary batch sizes, precisions, sequence lengths (up to 102,400), mask types, and GPUs (A800, RTX 5090), while covering varying feature ranks in single-layer numerical sweeps (Appendix E: ranks 32, 128, 512) and two head/dimension regimes in end-to-end prefill (TransNormerLLM-7B: h=32,r=d=128h=32, r=d=128; RetNet-1.3B: h=8,r=d=256h=8, r=d=256). Across these two model regimes, the primary model-level rankings remain consistent, while our complexity bounds in Table 1 (O(Nrd)\mathcal{O}(Nrd) vs. O(N2(r+d))\mathcal{O}(N^2(r+d))) characterize how extreme dimension shifts or unequal rr and dd affect theoretical crossover points.

Conducting a full combinatorial empirical sweep across all conceivable hyperparameters is computationally prohibitive and exceeds the realistic scope of a single foundational benchmark. Our objective is to establish a rigorous evaluation framework and expose fundamental algorithmic trade-offs across distinct computational paradigms, not to serve as an exhaustive deployment lookup table for every possible tensor shape.

Planned Revisions: We will explicitly define the boundaries of our tested parameter space in the introduction and clarify that highly specialized deployment shapes will naturally require localized profiling using our framework.

Q5: Practical Implementation Selection and Generalization

Can the benchmark possibly support practical implementation selection? Given the finding that no method consistently dominates, can the benchmark be used to derive a selection rule based on workload, hardware, precision, and numerical tolerance, and evaluate it on configurations beyond those used to derive it?

Yes, the benchmark directly supports practical selection. However, we respectfully clarify that deriving a zero-shot predictive cost model for unseen hardware and workloads is a separate compiler research problem. Within its designated scope, the benchmark already facilitates practical deployment:

  1. Practical Selection via Empirical Lookup: Practitioners can match their hardware, precision, and batch size in our tables, filter out methods exceeding their numerical tolerance (Appendices E and F), and select the fastest valid candidate.
  2. Built-in Hardware-Aware Autotuning: Our framework provides hardware-aware tile/warp configurations and lightweight profiling scripts to benchmark custom shapes on target GPUs.

Planned Revisions: We will add a brief "Deployment Guide" to the discussion to clarify how practitioners can leverage our empirical tables and built-in autotuning for implementation selection, while explicitly bounding zero-shot predictive generalization as an avenue for future compiler research.

AuthorsOpenReview ↗

Follow-up on Response

Dear Reviewer oXRj,

As the discussion period concludes in roughly two days, we are following up to check whether you have had a chance to review our detailed responses to your constructive comments.

In particular, we clarified our vertical cross-paradigm positioning relative to FLA, highlighted our matched PyTorch vs. CUDA/Triton comparisons, provided the anonymous repository link per ICLR guidelines, and directly analyzed how numerical tolerance filtering impacts latency rankings. We would sincerely appreciate your feedback or any further questions before the discussion window closes.

Reviewer_mGV3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies causal linear attention. It takes five computation methods and and benchmarks single-layer latency and end-to-end prefill on TransNormerLLM-7B and a 1.3B RetNet.

Critical strengths
  • THe paper targets multiple differenent linear attention
Critical weaknesses
  • The biggest concern is novelty, there is little to no novelty deserving a full paper with this analysis

  • I don't agree with that there is no comprehensive framework, there are kernel libraries like flash-linear-attn

Questions for authors

Look at weakness.

2 comments

We thank the reviewer for their valuable comments. By further highlighting the benchmark contribution and its relationship to existing resources, we hope to address your concerns.

W1: Novelty and contribution

The biggest concern is novelty

We respectfully refer the reviewer to General Responses R1 and R2. For an Experimental Survey and Benchmark in the Datasets and Benchmarks area, we respectfully clarify that our contribution lies in the unified taxonomy, formal complexity analyses, and cross-paradigm evaluation of token-wise, block-wise, and recursive formulations across latency, memory, and numerical fidelity. We will refine the introduction to more clearly distinguish our comparative benchmark from architectural proposals.

W2: Existing frameworks such as FLA

I don't agree with that there is no comprehensive framework

We agree that FLA is an important existing resource and kindly refer the reviewer to General Response R1. We will tighten the framework claim and explain the specific comparative questions addressed by LeetDecoding, while making the existing FLA retention baseline and the limits of that comparison more prominent.

AuthorsOpenReview ↗

Follow-up on Response

Dear Reviewer mGV3,

As the discussion period ends in about two days, we are following up to confirm whether our responses have addressed your concerns about our benchmark's novelty and its relationship to existing libraries.

We have provided a structured comparison distinguishing our cross-paradigm algorithmic evaluation (unifying token-wise, block-wise, and recursive formulations under a shared operator) from single-paradigm kernel libraries like FLA, alongside our systemic complexity and numerical fidelity insights. We remain fully available to answer any remaining questions and would appreciate your feedback before the discussion concludes.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We thank all reviewers for their constructive feedback. We first emphasize that this work is an Experimental Survey and Benchmark under the Datasets and Benchmarks area, evaluating diverse computational paradigms for a unified causal linear attention operator. Our core contributions are:

  • Algorithmic survey and complexity analysis

We unify scattered token-wise, block-wise, and recursive paradigms under an explicit non-iterative matrix formulation, providing rigorous complexity bounds (notably proving the suboptimal O(Nlog⁡N)\mathcal{O}(N \log N) complexity of recursive splitting).

  • The LeetDecoding library and benchmark

We standardize disparate algorithms with non-trivial mask/precision adaptations, custom CUDA kernels for previously unoptimized methods, and system-level enhancements (e.g., data prefetching) for operator-level and end-to-end LLM prefill evaluations.

  • Systemic empirical findings

We demonstrate that no single method universally dominates across operational contexts, that hardware optimizations can override theoretical complexity, and that model-level substitutions expose critical numerical fidelity shifts missed by latency-only benchmarks.


Below, we address shared concerns regarding our positioning relative to FLA (R1), systemic empirical insights (R2), and operator scope (R3), followed by point-by-point responses.

R1: Paradigm Depth vs. Architectural Breadth (mGV3 W1/W2, oXRj W1/Q1, XrES W1)

We thank the reviewers for highlighting FLA. The community's strong convergence on FLA's chunk-wise kernels indeed highlights the gap addressed by our work: several competitive computational paradigms for causal linear attention remain scattered across the literature without direct comparison.

1. Orthogonal Scope: Vertical Algorithmic Depth vs. Horizontal Architectural Breadth.

While FLA excels at horizontal breadth by providing standardized chunk-wise kernels across various architectures, our work pursues vertical algorithmic depth on a single foundational operator. Rather than treating the chunk-wise formulation as a settled choice, we systematically uncover, generalize, and unify methods from disparate domains, such as adapting VQ-style formulations and HyperAttention's divide-and-conquer recursive splitting. By holding the mathematical operator fixed, our benchmark separates computational choices from architectural changes. As Reviewer oXRj notes, comparing implementations of the same operator helps distinguish computational decisions from differences in model architecture or predictive quality. We develop this comparison along the following dimensions:

Core Evaluation DimensionKey Insights & Evidence in Submission
Cross-paradigm scalingSec. 3-4 & App. G: How token-wise, block-wise, and recursive formulations differ in time/space scaling (including the recursive O(Nlog⁡N)\mathcal{O}(N\log N) factor).
Intra-paradigm sensitivitySec. 5 & 6.1: How low-level scheduling and framework choices alter the latency of an identical mathematical algorithm.
Operator-to-Model prefillSec. 6.2 & App. D: How synthetic kernel speedups translate into end-to-end LLM prefill gains on TransNormerLLM-7B and RetNet-1.3B.
Numerical fidelity shiftsApp. E-F: How mathematically equivalent formulations exhibit distinct numerical deviations and perplexity shifts when swapped into pretrained weights.

2. Direct Empirical Comparison with FLA (LA_ref).

Our submission also benchmarks against FLA's retention implementation (LA_ref in Table 13, Appendix D.1). In end-to-end RetNet-1.3B prefill, our optimized block-wise kernel (LA) achieves competitive latency alongside LA_ref (351.92 s vs. 366.88 s at batch size 1; 388.90 s vs. 394.64 s at batch size 2). More importantly, our survey shows that no single method dominates across all configurations, making cross-paradigm comparisons essential for understanding configuration-dependent trade-offs.

Planned Revisions: We will refine the introduction to explicitly credit FLA for its architectural breadth, highlight LA_ref as the FLA baseline, and add a taxonomy table contrasting our cross-paradigm analytical dimensions with existing libraries.

R2: Core Empirical Insights Beyond Isolated Measurements (mGV3 W1, XrES W2)

We respectfully clarify that our primary contribution is not a single isolated speedup or proposing a novel kernel, but a systematic, cross-paradigm benchmark. As Reviewer oXRj noted, our evaluations "reveal significant differences that a latency-only benchmark would miss." Three broader insights emerge from our study:

1. Challenging the chunk-wise consensus and exposing context-dependency.

Chunk-wise kernels (e.g., FLA) are assumed to be universally optimal. Our scalability evaluation shows that no single method dominates every setting; instead, performance depends on operational context such as batch size, mask complexity, and data precision. Identifying regimes where alternative paradigms outperform optimized block-wise kernels provides practical hardware-aware guidance.

2. Engineering optimizations can outweigh theoretical complexity.

We demonstrate that system-level hardware optimizations can dictate actual efficiency independent of theoretical algorithmic complexity, thereby establishing clear, hardware-aware guidelines for future implementations.

3. The inadequacy of latency-only evaluation.

Mathematically equivalent computational choices can introduce notable numerical deviations and perplexity shifts when substituted into pretrained models, making joint numerical and end-to-end prefill evaluations necessary.

R3: Operator Scope (oXRj W2/Q2)

We agree that input-dependent gating and delta-rule updates (e.g., DeltaNet, Kimi) are important developments. However, this request highlights the core distinction established in R1: horizontal architectural breadth versus vertical algorithmic depth.

1. Fixing the Math to Isolate the Algorithm.

While frameworks like FLA scale horizontally to support many distinct attention rules, our benchmark is deliberately vertical. We target the classic causal linear attention operator to rigorously compare how different paradigms compute the identical mathematical function. Introducing delta-rules or gating changes the underlying math. TransNormerLLM and RetNet provide ideal controlled settings to swap computations while preserving the exact attention rule.

2. Reusability of the Evaluation Framework.

Because newer architectures define distinct operators, our specific complexity bounds and latency rankings do not automatically transfer to them. However, our evaluation methodology is reusable: separating algorithmic structure, profiling operator-to-model translation, and verifying zero-shot numerical fidelity provides a clear blueprint for benchmarking future architectures.

Planned Revisions: We will explicitly clarify the vertical scope of our benchmark and outline the extension boundaries for data-dependent and delta-rule models in the revised discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.