acceptodds
Under review as a conference paper at ICLR 2027

LazyFlash: Forward-Dense, Backward-Sparse Attention for Accelerated Long-Context Fine-Tuning

Abstract

As large language models (LLMs) process increasingly long sequences, attention becomes a major computational bottleneck. Sparse attention mostly targets inference, yet many methods still train densely or cannot support sparse training, and their added complexity makes sparsity hard to turn into wall-clock speedups, leaving dense attention widely used in practice. This paper introduces LazyFlash, a forward-dense, backward-sparse attention method built on the observation that attention training is dominated by the backward pass. LazyFlash keeps the forward pass exactly dense, so the trained model remains a standard dense model that existing systems such as vLLM can serve without any change. For the backward pass, LazyFlash proposes a gradient-aware block sparsification design that ranks query blocks by their attention-output gradient norms and skips those contributing negligibly to parameter updates, turning the dense backward into a block-sparse one. A gradient-mass budget adapts the amount of skipped computation to each head and training step, while importance sampling preserves the dense gradient in expectation. These designs are implemented inside the tiled FlashAttention backward kernel, where skipped blocks are never loaded or computed, translating gradient sparsity into real wall-clock acceleration. Experiments on long-context text and speech LLMs show that LazyFlash maintains downstream performance while achieving up to 5.7× attention-backward speedup over FlashAttention and 1.7× end-to-end training speedup.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Backward-Sparse Attention, Attention Acceleration, Efficient LLM Training, Long-context LLM, Speech LLM

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_yTQTOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose LazyFlash, which keeps attention dense in the forward pass but samples query blocks during backpropagation based on the output vector gradient norms. Selected blocks (rest are skipped) are reweighted to produce an unbiased gradient estimate. The method reports comparable or better fine-tuning quality, with up to 5.72× faster attention backward and 1.66× faster training steps.

Critical strengths

I like the work! the problem is relevant and the authors propose a rather cute trick to speed long-context training

  • method seems very practical: it maintains finetuning quality while resulting in speed ups in training
  • it works across model scales and context lengths
Critical weaknesses

I have a few small concerns:

  • is the hyperparameter delta -- which decides how much gradient mass to take into account for training -- the quality and speedups can be sensitive to that hyperparameter
  • long context evals are not comprehensive, maybe skipping query blocks is resulting in a drop in performance in some other long context eval
  • is their any degradation in training loss compared to the dense baseline
Questions for authors
  • what happens if you use LazyFlash for pretraining from scratch? say at 4K and at 32K? no need to train for lots of tokens, I am just interested if the loss remains similar to the dense baseline and how much wall clock speed ups one gets

  • can you show loss curves for LazyFlash during finetuning, compared to the dense baseline?

  • can you run finetuning on various delta values to check for the sensitivity of quality-speedup trade-offs

  • can you do more long context evals to be sure skipping some query blocks is not resulting in something bad in some other eval -- upto to the authors to decide what are the best evals they think can support the story that skipping query blocks backward is okay -- I don't have any particular long context evals in mind.

  • can you look at which tokens or query blocks usually get skipped or have low output vector gradient norms? are these filler tokens? or noise etc.? just curious about some interpretability -- it'd be cool that backward pass is skipped for useless queries :)

Reviewer_W4aJOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a sparse attention training method that does a full forward and a sparse backward. This results in a significant speedup during training, as most of the computation in training is in the backwards kernel.

Critical strengths
  • Attention sparsity, especially during training is overlooked by a vast majority of the sparse attention literature.

  • The proposed method results in a significant speedup.

Critical weaknesses
  • L160 says that the backward pass requires 4 quadratic operations, but I believe it should actually be 5 due to the following:

    1. Construct S in kernel for dK and dV
    2. dK computation with dS
    3. dV computation with dO
    4. Construction of S in dQ kernel
    5. Computation of dQ with dS
  • Did you ever consider calculating equation 3 as the max norm over single rows of dO? What happens if there is one large gradient in a block of dO and many small gradients?

  • If k∗k^* is the number of blocks that are dropped, then why does a small setting of δ\delta mean that there would be a large k∗k^*? Shouldn’t it be the inverse of this? If δ\delta is small, that would make the number of accepted kk terms in equation 4 smaller and therefore k∗k^* should be smaller.

  • Does the calculation of cmc_m in equation 3 decide the blocks for both dQ and dK and dV?

  • If dropping 0.1 of the gradient mass results in a large speedup, it implies there is a long tail of very small gradient norms. It would be interesting to see the distribution of gradient norms over a subset of training data. Likewise, it would also be interesting to see some statistics of how many token gradients are skipped throughout training. Is it different at the beginning and end of training? Is it wildly different per head? Answering these questions might provide some interesting insight.

  • How many tokens were trained for in the text experiment?

  • If possible, it would be interesting to see how this compares to dense attention during a small pretraining run, even on a small model. Does pretraining converge as well as fine-tuning?

  • One comparison that is missing from the work is to a sparse training method such as DSA. Most production sparse attention models are using some form of the DSA style sparse attention and therefore, anyone looking to use this method is going to naturally need to compare the training of this method to DSA.

Questions for authors

Covered above in "weaknesses."

Reviewer_Yr7oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce a grad sparsification framework to reduce the speed up the attention backward propagation in long-context LLM fine-tuning. It proposes LazyFlash, which keeps the forward pass dense while sparsifying the query-level backward computation. It first measures the importance of query blocks using their attention-output gradient norms, determines the sparsity level through a budget, and applies importance sampling to obtain unbiased gradient estimates. Experiments on long-context language and speech tasks show that proposed method maintains comparable model performance while achieving up to 5.72× attention backward speedup and 1.66× end-to-end training speedup.

Critical strengths
  1. The paper targets training efficiency through gradient sparsification and demonstrates significant wall-clock speedups, including end-to-end training acceleration. This makes the proposed approach practically valuable for long-context fine-tuning.
  2. The authors provide a well-optimized CUDA kernel implementation which integrates backward sparsification into the existing FlashAttention kernel.
Critical weaknesses
  1. It would be better to include a detailed breakdown of the computational overhead introduced by gradient importance estimation, budget determination, and importance sampling. This would help provide a clearer understanding of the actual efficiency gains.
  2. It is recommended that the authors discuss the differences between the prposed approach and BackLite, which is also adopts a forward-dense, backward-sparse attention. Providing a direct comparison can better highlight the novelty and advantages of the proposed method.
  3. In Section 3, authors use the norm of the dO as a proxy for query importance, while the quantity directly relevant to gradient approximation is g_m. However, the relationship between these two is not sufficiently analyzed. It would be helpful if the authors could provide further analysis, including potential error bounds, and conduct experiments to validate the correlation between their gradient norms across different layers and attention heads.
  4. (minor) Since the proposed method adopts top-p sparse, which can result in different sparse patterns across attention heads, it may introduce load imbalance in distributed training. It would be helpful if the authors could briefly discuss this potential issue and its impact on multi-GPU training efficiency.

[1] https://github.com/moonmath-ai/BackLite

Questions for authors
  1. Since the sparse attention mechanisms in forward pass often leverage KV sparse, could the authors clarify why sparsification is applied only along the query dimension? It would be interesting to discuss whether similar ideas could be extended to the key/value gradient computations, as well as their potential benefits and challenges.
Reviewer_QUEPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LazyFlash accelerates long-context fine-tuning by retaining dense attention in the forward pass and sparsifying attention backward computation over query blocks. It scores blocks using output-gradient norms, derives a target expected block count from a deterministic tail-energy criterion, and samples blocks with norm-dependent probabilities. Inverse-probability reweighting yields a conditionally unbiased local gradient estimator when every nonzero contribution has positive sampling probability. The method is integrated into FlashAttention backward kernels and evaluated on long-context language-model fine-tuning and speech recognition, including 4B, 8B, and 14B models and measured training-step speedups.

Critical strengths
  1. The acceleration targets a useful point in the training pipeline while preserving a standard dense inference architecture. Keeping the forward computation dense avoids requiring a new sparse inference operator after fine-tuning. The inverse-probability estimator also has a valid local unbiasedness argument under the stated conditions. Together, these properties make the method practically attractive without requiring identical optimization trajectories to dense training.
  2. The paper measures actual training benefits alongside downstream quality. At 32K, the reported attention-backward speedup is 5.72× and the complete training-step speedup is 1.66×. Table 1 reports LongBench-Pro scores of 30.24 versus 29.64 for dense fine-tuning, while the study also includes speech results and model-scale experiments. Table 3 compares alternative selectors at approximately matched backward time. These results support useful acceleration with comparable observed quality.
Critical weaknesses
  1. The specific algorithmic contribution and the source of its gains are insufficiently established. The manuscript acknowledges gradient-based sparse backward computation and standard importance sampling, so the contribution needs to be distinguished through its query-block formulation, adaptive budget and estimator. Its relationship to Back Razor and PreBackRazor is not discussed and should be clarified;

  2. Limited baseline coverage and task-level quality analysis. MoBA is the only external sparse-attention baseline evaluated. Comparisons with additional strong long-context fine-tuning methods, such as LongLoRA, are needed to establish the method's advantage in quality and training efficiency.

    • LongBench-Pro includes retrieval and dialogue-memory tasks, and the paper reports language and difficulty splits. Reporting sample counts, difficulty distributions and quality changes for representative task–length groups would clarify whether the aggregate gains extend to challenging long-context cases. Additional evaluation with longer final answers, varied input/output length ratios and ongoing multi-turn interaction would help establish how broadly quality preservation transfers.
Questions for authors
  1. Component contributions and runtime: Can targeted ablations separate the effects of (i) energy-adaptive versus fixed expected block counts, (ii) norm-dependent random sampling versus deterministic gradient-norm top-k selection, and (iii) applying versus omitting (1/q_m) reweighting? Please also report the costs of scoring, budgeting, sampling/weight preparation and sparse backward to explain the overhead of the selection pipeline.
  2. MoBA baseline tuning: Was the MoBA baseline tuned for this fine-tuning setup? Please describe the block sizes and top-k budgets explored, how the reported configuration was selected, and whether alternative settings improve quality at comparable training cost.

Comments on the paper

2 comments

Dear authors, thank you for your interesting work! We would like to highlight a closely related paper, Know What You Need: Efficient Activation Checkpointing (Ahmed & Oyallon, NeurIPS 2026), available since July 2026.

Both works use importance sampling with unbiased gradient estimates to reduce backward computation, with similar sampling strategies (KEAP, Proposition 2; LazyFlash, Section 3.3). We believe this connection would be valuable to discuss and cite in the related work, particularly as KEAP provides a theoretical analysis of optimal importance sampling for an optimization problem closely related to that addressed by LazyFlash.

Thank you for bringing KEAP to our attention and highlighting this helpful connection.

For context, HAL, an open-access research repository, records an initial deposit on July 31, 2026, while the first arXiv submission is dated November 6, 2026. Given this timing, we had not been aware of this contemporaneous work when preparing our submission. In the spirit of ICLR's guidance on contemporaneous work, we appreciate this opportunity to acknowledge the connection and will include an appropriate citation and discussion in the revised manuscript.

We acknowledge the connection through importance sampling and unbiased gradient estimation; these are supporting mathematical principles rather than claimed novelties of LazyFlash. Our contribution focuses on the attention-specific query-block formulation, adaptive gradient-energy budgeting, and sparse backward integration into FlashAttention while preserving dense forward computation. Thank you again for this constructive suggestion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.