- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose LazyFlash, which keeps attention dense in the forward pass but samples query blocks during backpropagation based on the output vector gradient norms. Selected blocks (rest are skipped) are reweighted to produce an unbiased gradient estimate. The method reports comparable or better fine-tuning quality, with up to 5.72× faster attention backward and 1.66× faster training steps.
Critical strengths
I like the work! the problem is relevant and the authors propose a rather cute trick to speed long-context training
- method seems very practical: it maintains finetuning quality while resulting in speed ups in training
- it works across model scales and context lengths
Critical weaknesses
I have a few small concerns:
- is the hyperparameter delta -- which decides how much gradient mass to take into account for training -- the quality and speedups can be sensitive to that hyperparameter
- long context evals are not comprehensive, maybe skipping query blocks is resulting in a drop in performance in some other long context eval
- is their any degradation in training loss compared to the dense baseline
Questions for authors
-
what happens if you use LazyFlash for pretraining from scratch? say at 4K and at 32K? no need to train for lots of tokens, I am just interested if the loss remains similar to the dense baseline and how much wall clock speed ups one gets
-
can you show loss curves for LazyFlash during finetuning, compared to the dense baseline?
-
can you run finetuning on various delta values to check for the sensitivity of quality-speedup trade-offs
-
can you do more long context evals to be sure skipping some query blocks is not resulting in something bad in some other eval -- upto to the authors to decide what are the best evals they think can support the story that skipping query blocks backward is okay -- I don't have any particular long context evals in mind.
-
can you look at which tokens or query blocks usually get skipped or have low output vector gradient norms? are these filler tokens? or noise etc.? just curious about some interpretability -- it'd be cool that backward pass is skipped for useless queries :)
Dear authors, thank you for your interesting work! We would like to highlight a closely related paper, Know What You Need: Efficient Activation Checkpointing (Ahmed & Oyallon, NeurIPS 2026), available since July 2026.
Both works use importance sampling with unbiased gradient estimates to reduce backward computation, with similar sampling strategies (KEAP, Proposition 2; LazyFlash, Section 3.3). We believe this connection would be valuable to discuss and cite in the related work, particularly as KEAP provides a theoretical analysis of optimal importance sampling for an optimization problem closely related to that addressed by LazyFlash.