acceptodds
Under review as a conference paper at ICLR 2027

FlashBoB-2: Algorithmic and Hardware Optimizations for Attention JVPs and Double Backward

Abstract

Jacobian–vector products (JVPs) and second-order derivatives of attention arise in training consistency models and learning through gradient updates. FlashAttention primarily optimizes attention evaluation and first-order reverse-mode differentiation, while specialized kernels such as FlashBoB and jvp_flash_attention support forward-mode or second-order operations. Building on FlashAttention-3/-4, we introduce FlashBoB-2 for attention JVPs and backward-over-backward (BoB) on Hopper and Blackwell GPUs, combining a reformulated BoB computation with schedules designed for each GPU. First, we optimize FlashBoB's computational graph, reducing the matrix-multiply count for exact BoB from eighteen to fifteen without storing intermediates and with on-chip memory independent of sequence length . Second, on Hopper, separate warpgroups build intermediate results and accumulate outputs, so each output product starts as soon as its inputs are ready. Third, on Blackwell, we reuse Tensor Memory once each value has been read for the last time and keep intermediate results in shared memory. In causal BF16 benchmarks with head dimension 128, FlashBoB-2 achieves BoB speedups of – on H100 and – on B200 over FlashBoB. For non-causal JVPs at 1K–128K against two baselines, speedups are – on H100 and – on B200.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Attention, Algorithm, Hardware, Jacobian-Vector Products, Double Backward, GPU

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nexGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper presents FlashBoB-2, an optimized implementation of the attention Jacobian-vector products (JVPs) and backward-over-backward (BoB) on modern GPUs. The authors simplify the computation graph and reduce the matrix-multiply cound from 18 to 15 by utilizing the same equation used in FlashAttention backward. The authors further develop the specialized kernels for Hopper and Blackwell GPUs. The resulting kernel achieves 2.7–3.0× speedup on H100 and 3.6–4.2× speedup on B200.

Critical strengths
  • The computation graph simplification is useful. Although the underlying math is relatively simple, it cancels redundant computation and enables a more efficient 2 pass decomposition
  • The hardware-specific optimization for the two GPUs are performed carefully. The authors carefully consider the resource constraint on specific hardware and manage to implement the kernel efficiently under these constraints.
  • The performance improvement of the kernels are significant and the paper include ablations and training workload evaluations, which supports the practical value of the optimized kernels.
Critical weaknesses
  • The presentation is difficult to follow, especially without consulting the appendix. The paper introduces many intermediate tensors and mathematical expressions when describing the BoB computation, making it difficult to understand the overall algorithm and identify the key differences from the original FlashBoB. In particular, the transition from the original 10+8 decomposition to the proposed 5+10 decomposition could be explained more intuitively. The hardware scheduling sections are also challenging to follow, as the reader needs to understand several architectural constraints and execution dependencies before the motivation for each scheduling decision becomes clear. Several useful figures, including Figures 4, 6, and 8, are only provided in the appendix. Moving some of these figures to the main paper, together with a higher-level explanation of the computation flow and scheduling decisions, would substantially improve readability.

  • The sources of the performance improvements could be analyzed more clearly. According to Table 1a, reducing the number of matrix multiplications from 18 to 15 provides a relatively modest speedup, while moving from the Triton implementation to the optimized CuTe DSL implementation accounts for most of the overall performance improvement. The paper includes ablations of several scheduling decisions, which are useful, but it is still difficult to understand which factors contribute most to the large improvement achieved by the CuTe DSL kernels. A more detailed performance breakdown, such as per-pass timings or additional profiling analysis, would help clarify the relative importance of the different hardware optimizations and better explain the reported speedups.

Questions for authors
  1. Could the authors provide a more intuitive explanation of the 5+10 decomposition and how it differs from the original 10+8 decomposition? In particular, it would be helpful to explicitly identify which computations are eliminated, which are moved between passes, and why this decomposition avoids redundant matrix multiplications. A side-by-side computation graph would make the algorithm easier to understand.
  2. Could the authors further clarify the main sources of the performance improvement from the CuTe DSL implementation? Table 1a shows that this accounts for most of the overall speedup, while Table 1b reports the benefits of several individual scheduling choices. It would be helpful to understand how much of the remaining improvement comes from more efficient Tensor Core execution, memory access and data movement, improved resource utilization, or other aspects of the implementation. Additional profiling results or a breakdown of the row-pass and column-pass execution times would be useful.
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the specific suggestions, which shaped our revision of Section 4. To make the paper easier to read, we moved the two-pass overview and the Hopper schedule from the appendix into the main text.

W1 (Q1): Intuitive account of 10+810+8 against 5+105+10, with a side-by-side graph.

A1: FlashBoB's row pass does three jobs with its ten matmuls per tile, and only the first must run there. We remove the second and move the third to the column pass, where it takes two matmuls instead of three (Figure 2).

Table R3. How FlashBoB-2 cuts FlashBoB's ten row-pass matmuls per tile to five.

Row-pass jobFlashBoB (10+810+8)FlashBoB-2 (5+105+10)
Compute JVP output d~(dO)\widetilde{d}(dO) and statistic α\alpha5 matmuls5 matmuls
Recompute dPdP and CC2 matmulsremoved (Steps 1–2)
Accumulate d~(Q)\widetilde{d}(Q)3 matmulsmoved to column pass, 2 matmuls (Step 2)
  • Step 1: what is removed. FlashAttention avoids dPdP when computing DiD_i via Di=doi⊤oiD_i=do_i^{\top}o_i. Differentiating with dOdO fixed gives Ei∘=doi⊤ d~(dO)iE^{\circ}_i=do_i^{\top}\ \widetilde{d}(dO)_i (Equation 4), one dot product per row with the output d~(dO)i\widetilde{d}(dO)_i just finished by the row pass. E∘E^{\circ} no longer needs dPdP or CC.
  • Step 2: what is moved. A row pass cannot subtract α\alpha and E∘E^{\circ} inside a tile because both complete only after the row's last tile. FlashBoB accumulates uncorrected sums and corrects them at row end, costing a third matmul. The column pass runs after both statistics exist and already forms the corrected tile d~(S)\widetilde{d}(S), so d~(Q)=τ(d~(S)K+dS UK)\widetilde{d}(Q)=\tau(\widetilde{d}(S)K+dS\ U_K) takes two matmuls (Equation 5). With d~(Q)\widetilde{d}(Q) gone, the row pass needs neither dPdP nor CC and drops from 10 matmuls to 5, the column pass goes from 8 to 10, and only 2N2N row statistics cross between passes.

W1 (continued): Figures and the scheduling sections.

A2: Former Figure 4 is now Figure 1, and condensed versions of former Figures 6 and 8 form Figure 4 (full diagrams in Appendix D.2, Figures 7 and 9). For d=128d=128, Sections 4.3–4.4 state each hardware limit before the schedule choice:

  • Hopper (Section 4.3). Building tiles (≈144\approx 144 registers per thread) and accumulating outputs (192192) exceed the 255-register thread limit together, so each gets its own warpgroup. Each intermediate tile (dS,d~(dP),d~(S)dS, \widetilde{d}(dP), \widetilde{d}(S)) gets its own barrier so output matmuls start when their input arrives.
  • Blackwell (Section 4.4). Column-pass state fills TMEM (512 columns), so partial d~(Q)\widetilde{d}(Q) reuses C⊤C^{\top}'s columns after C⊤C^{\top}'s last read. Because TMEM only feeds left operands while d~(Q)\widetilde{d}(Q) matmuls read dS,d~(S)dS, \widetilde{d}(S) as right operands, both tiles come from shared memory.
  • FlashAttention-3 and -4. Our response to Reviewer 3UER (A4) compares these limits with first-order backward's and lists what we reuse.

W2 (Q2): Where the CuTe DSL speedup comes from.

A3: On H100 at d=128d=128 from 4K to 16K, the column pass accounts for 8080–8383% of the time saved over the 15-matmul Triton kernel (Table 28).

Table R4. Per-pass times and Nsight Compute counters on H100 at 16K, d=128d=128 (new Table 2 in the main text).

MetricTriton rowCuTe rowTriton columnCuTe column
Pass time (ms)20.8–21.110.1 (2.07×2.07\times)72.7–77.924.7–24.9 (2.92.9–3.1×3.1\times)
Tensor-pipe activity41.7%94.3%26.7%75.0%
Warp instructions (10910^9)6.462.2215.596.86
Shared-memory wavefronts (10910^9)2.341.0310.214.37
  • Tensor Core execution. Both kernels run the same 15 matmuls, and ours has higher tensor-pipe activity in both passes.
  • Memory access. The Triton kernel loads inputs through registers, while ours uses TMA. Triton executes 2.32.3–2.9×2.9\times as many warp instructions and 2.3×2.3\times as many shared-memory wavefronts, with 814M bank-conflicted store wavefronts in the column pass vs. 4.0M for ours (Table 29).
  • Resource use and ablations. The Triton row pass spills registers to local memory, and ours does not (Table 29). In Table 1b, one barrier per tile gives 1.041.04–1.05×1.05\times, producer loads 1.021.02–1.03×1.03\times (row pass), two warpgroup roles 1.24×1.24\times (d=64d=64, earlier kernel), and one MMA warp on B200 1.031.03–1.05×1.05\times. Each row measures one choice, so the rows do not add up to the total.
  • Per-pass times on both GPUs. At 16K (d=128d=128), the row pass, column pass, and complete call take 12.1, 27.8, and 39.8 ms on H100 and 10.0, 17.0, and 26.8 ms on B200 (Table 20; timed back to back, where the H100 lowers its clock under its power cap, so times are longer than Table 2's single calls; Appendix G.2.1). Table 24 compares utilization with FlashAttention-4's backward on the same B200.

We hope these revisions address your concerns and that you will consider them in your final rating.

Thank you for the detailed response. The additional explanations and profiling results address my concerns. I will maintain my original rating.

AuthorsOpenReview ↗

Author Follow-up

We want to thank you for your careful and methodical review. We would also love, if time permits, to see if our reorganization of the paper helped to make it more readable. That said, we do appreciate the time you took to review our paper.

Reviewer_Nzj7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the cost of attention Jacobian–vector products (JVPs) and backward-over-backward (BoB) in training. FlashBoB-2 reorganizes FlashBoB's two-pass computation to reduce the matrix multiplications per tile from eighteen to fifteen, and implements schedules for Hopper and Blackwell GPUs in CuTe DSL. The row pass also serves as a JVP kernel, with a fused variant that returns both the attention output and its JVP. The evaluation covers kernel performance, ablations, numerical accuracy, and application workloads. At head dimension 128, the reported BoB speedups over FlashBoB are 2.7–3.0× on H100 and 3.6–4.2× on B200; a 1B-parameter GradMem workload achieves a 1.62× complete-step speedup at 16K context on B200.

Critical strengths
  1. The computation rewrite and architecture-specific schedules form a concrete technical contribution. Equation 4 obtains a required row statistic from an output already computed in the row pass. Moving the query-side BoB output to the column pass then avoids recomputation and reduces the per-tile matrix-multiply count from eighteen to fifteen. Sections 4.3 and 4.4 explain how the resulting computation fits each GPU: separate construction and accumulation warpgroups address Hopper's register budget, while reuse of Tensor Memory and placement of intermediate tiles in shared memory address Blackwell's storage and operand constraints. The paper connects these design choices to explicit computational dependencies and resource limits.

  1. The experiments demonstrate substantial kernel gains and a measurable benefit in a complete training step. At head dimension 128, FlashBoB-2 achieves 2.7–3.0× BoB speedups over FlashBoB on H100 and 3.6–4.2× on B200. The fused JVP also improves on the stronger measured JVP baseline, rCM, by 1.78–2.02× and 3.16–3.67×, respectively. Table 1 examines the computation split and scheduling choices. Table 3 demonstrates a 1.62× complete-step speedup for the 1B-parameter GradMem model at 16K context on B200. Together, these measurements establish a practical benefit for workloads where the targeted operations occupy a substantial share of execution time.
Critical weaknesses
  1. The numerical behavior in sharp-attention cases and its training impact need further validation. Appendix F identifies a nonzero query-side BoB output for a causal row attending to a single key, whose exact output is zero. The error arises from rounding in the rewritten computation of the row statistic. The appendix also identifies nearly one-hot attention as a related risk and states that deliberately constructed cases of this kind have not been measured. Aggregate RMSE can obscure errors in rows whose reference output is zero or very small. Please test these cases explicitly on both GPU architectures, report absolute errors alongside relative errors, and evaluate whether a targeted correction is needed and what performance it costs. The Sophia-H experiment in Appendix F.1 uses one initialization seed and runs for 1,908 steps, shorter than its 2,000-step learning-rate warm-up. It establishes agreement over the reported early training interval, but does not test the subsequent training regime. Extending the comparison beyond warm-up and using independent initialization seeds would strengthen the evidence about the training impact of the numerical differences.

  1. The contribution of the computation rewrite is not fully isolated from that of the GPU implementation. Table 1a reports 1.11–1.17× speedups from the fifteen-matmul rewrite in Triton on H100 and 1.04–1.07× on B200. The subsequent change to the CuTe implementation accounts for a much larger gain. Section 5.2 states that there is no equally tuned eighteen-matmul CuTe baseline. The current comparison therefore does not fully isolate the effect of the computation graph under a shared implementation framework. A representative comparison between the eighteen- and fifteen-matmul graphs in CuTe, with comparable tuning, would clarify the graph's contribution and its interaction with scheduling. The paper should also make explicit which resource constraints prevent the original graph from using the proposed schedules.
Questions for authors
  1. How does the rewritten statistic behave when attention is nearly one-hot? Please construct single-key and nearly one-hot cases and compare all four BoB outputs with FP64 and FlashBoB on H100 and B200. Report absolute errors for outputs close to zero. Can a targeted correction reduce the error identified in Appendix F, and how much runtime does that correction add?

  1. How much of the speedup depends on the fifteen-matmul graph under the same implementation framework? Can you compare the original eighteen-matmul graph and the proposed graph in CuTe at representative configurations, with comparable tuning? Please identify which proposed scheduling choices are enabled by the rewrite and which can also benefit the original graph.

  1. Do the observed training differences remain small after warm-up and across initialization seeds? Please extend the Sophia-H comparison beyond the 2,000-step warm-up into the subsequent training regime and report results across independent initialization seeds, keeping the data and optimization settings matched between backends within each seed. This would address the training impact left unresolved by the current 1,908-step, single-seed experiment.
1 comment
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the close reading of Appendix F. Your first question led to a correction that is now part of the row pass.

W1 (Q1): Sharp-attention errors, a correction, and its cost.

A1: In constructed tests on both GPUs, a row-sum correction reduces the single-key d~(Q)\widetilde{d}(Q) absolute error to ≤1.8×10−5\le 1.8\times10^{-5} (FlashBoB: 0.120.12–0.180.18). It costs 1.01.0–1.81.8% of BoB time on H100 and 1.71.7–2.62.6% on B200 (Table 19).

Cause and correction. In Equation 6, (P⊙F)V(P\odot F)V reads a BF16-rounded copy of P⊙FP\odot F, while α\alpha summed the unrounded tile. Summing the rounded tile (α′\alpha' in place of α\alpha) makes the rounding term vanish on a single-key row and shrink with 1−max⁡jPij1-\max_j P_{ij} on a nearly one-hot row (Appendix F). Computing α′\alpha' adds one row reduction per tile and no matmul.

Tests. We compare all four outputs with FP64, FlashBoB, and PyTorch's math path on H100 and B200. Set 1 puts weight 1−10−k1-10^{-k} on one key (k=1,…,6k=1,\dots,6; 2K, 8K tokens; d=64,128d=64, 128; scales 0.1,1,100.1, 1, 10). Set 2 places such rows at mask and tile boundaries. Tables 13 (Set 1, query side) and 14 (Set 2, all outputs) report absolute errors with the exact-output norm.

Table R2. Full-tensor relative ℓ2\ell_2 error against FP64 on Set 1, over both GPUs (Table 12).

OutputFlashBoB-2FlashBoBPyTorch math
d~(Q)\widetilde{d}(Q)0.0063–0.01610.0071–0.01660.0017
d~(K)\widetilde{d}(K)0.0122–0.11350.0155–0.09760.0017
d~(V)\widetilde{d}(V)0.0023–0.00290.0023–0.00240.0016–0.0017
d~(dO)\widetilde{d}(dO)0.0014–0.00440.0021–0.00870.0012–0.0017

Against FlashBoB, our error ranges are lower for d~(Q)\widetilde{d}(Q) and d~(dO)\widetilde{d}(dO), while d~(K)\widetilde{d}(K) and d~(V)\widetilde{d}(V) have higher maxima.

Nearly one-hot rows. In Set 1, the median per-bin ratio of our error to FlashBoB's is 0.660.66 (H100) and 0.670.67 (B200) on d~(Q),d~(dO)\widetilde{d}(Q), \widetilde{d}(dO). In Set 2, it is 0.640.64–0.680.68 on d~(Q)\widetilde{d}(Q) and 0.510.51–0.520.52 on d~(dO)\widetilde{d}(dO) (key-side: 1.001.00–1.231.23). Some bins exceed FlashBoB's error (largest ratio 3.78×3.78\times in Set 1, 9.1×9.1\times in Set 2). On these rows, exact d~(Q)\widetilde{d}(Q) is near zero and both BF16 kernels have large relative errors. In Set 2, the median error is 111111–113×113\times the FP32-internal math path's error for ours vs. 576×576\times for FlashBoB.

W2 (Q2): Graphs under one framework and schedule dependencies.

A2: We built the 18-matmul (10+810+8) CuTe DSL kernel on H100. Our response to Reviewer 3UER (A1, Table R1) gives the full comparison. With only the matmul graph varied, 5+105+10 is 1.161.16–1.18×1.18\times faster than 10+810+8, and the two-query-block row pass that 5+105+10 allows raises this to 1.401.40–1.45×1.45\times. On B200 the graph comparison is in Triton only (1.041.04–1.08×1.08\times, Table 1a).

Table 25 classifies each schedule choice:

  • Enabled by the rewrite. (1) Hopper's two-query-block row pass (with our register layout, 10+810+8 needs at least 68,608 registers per thread block, and a streaming multiprocessor has 65,536). (2) Keeping P,P⊙FP, P\odot F in Blackwell Tensor Memory (TMEM) during the row pass (with our layout, 10+810+8 needs 576>512576 > 512 columns). (3) Column-pass d~(Q)\widetilde{d}(Q) reduction and Blackwell reuse of C⊤C^{\top}'s TMEM columns for partial d~(Q)\widetilde{d}(Q).
  • Usable by 10+810+8. Hopper's warpgroup split, per-tile barriers, and producer TMA loads (all used by the 18-matmul kernel in Table 30), and, in principle, Blackwell's transposed column layout and single MMA warp.

Q3: Sophia-H after warm-up and across seeds.

A3: Training with α′\alpha' past warm-up and across seeds has not been run, because our cloud GPU credits are exhausted. We are working to obtain more and will then add these runs.

Nevertheless, we added two seeds trained past warm-up and a test of α′\alpha' at a trained checkpoint:

  • Two seeds past warm-up, without α′\alpha'. Two 8-GPU H100 runs (seeds 18 and 19, N=4096N=4096, matched data/optimizer per seed) use a 1,500-step warm-up and 408 steps of cosine decay. After warm-up, FlashBoB-2's evaluation loss averages 0.0090.009 and 0.0260.026 above that of PyTorch's math path (max gaps 0.013,0.0360.013, 0.036, vs. 0.0030.003 between the two math seeds; Appendix F.1).
  • Trained checkpoint, with α′\alpha'. At step 1907 of seed 18, 14.814.8% of layer-1 rows have max⁡jPij≥0.999\max_j P_{ij} \ge 0.999. There, α′\alpha' lowers the aggregate relative ℓ2\ell_2 error of the Hessian-diagonal estimate against FP32 from 30.730.7 to 5.385.38, on par with FlashBoB, the BF16 baseline (5.395.39). With shared probes, our single-GPU update direction, which uses this estimate and excludes learning rate and weight decay, differs by 0.0810.081 in relative ℓ2\ell_2 from the BF16 model's math path, vs. 0.0870.087 between two independent-probe math evaluations. This test measures the correction at fixed weights and does not establish equivalent training trajectories.
Reviewer_3UEROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents FlashBoB-2, which provides CuTe DSL kernels for attention Jacobian–vector products (JVPs) and backward-over-backward (BoB) on Hopper and Blackwell GPUs. It reformulates FlashBoB's computation to reduce the matmuls per tile from 18 to 15, without storing N×N intermediates. It also designs a schedule for each GPU. On Hopper, tile construction and accumulation are split across separate warpgroups. On Blackwell, Tensor Memory is reused and intermediate tiles are kept in shared memory. BoB's row pass also serves as a standalone JVP kernel. With causal attention and head dimension 128, the paper reports BoB speedups over FlashBoB of 2.7–3.0× on H100 and 3.6–4.2× on B200. Against existing JVP kernels, it reports 1.8–5.4× on H100 and 3.2–7.3× on B200.

Critical strengths
  1. Solid, hardware-aware engineering with a clean algorithmic reformulation. The kernels build on FlashAttention-3/4 ideas and target Hopper and Blackwell features directly (WGMMA/TMA, tcgen05/TMEM), with schedules designed around each GPU's scarcest on-chip resource. Reducing the matmuls per tile from 18 to 15 without storing N×N intermediates is backed by a minimality proof among direct two-pass splits, and the BoB row pass also serves as a standalone attention JVP kernel.

  2. Rigorous and transparent evaluation. Every design choice is ablated, the speedup is decomposed into algorithm vs. implementation (Table 1a), and numerical accuracy is checked against FP64 and PyTorch BF16. Unfavorable results are reported honestly, such as the near-1.0× speedup for TTT and the larger error on query row 0.

  3. Practical value. It enables fast attention JVPs and BoB for methods that standard kernels do not support, such as consistency/MeanFlow models and GradMem. PyTorch's double backward runs out of memory at 8K tokens on H100.

Critical weaknesses
  1. The algorithmic contribution is small; most of the speedup comes from engineering. Reducing the matmuls per tile from 18 to 15 yields only 1.11–1.17× on H100 and 1.04–1.07× on B200 when both are implemented in Triton. The rest comes from moving to CuTe DSL with schedules adapted from FlashAttention-3/4. Since there is no equally tuned 18-matmul CuTe DSL kernel, the headline 2.7–4.2× over FlashBoB largely reflects Triton vs. CuTe DSL rather than the reformulation. The JVP baselines (rCM, jvp_flash_attention) are also Triton kernels.

  2. The application domain is narrow, and the end-to-end gains are limited. BoB is mainly used in a narrow set of methods (meta-learning, test-time training, second-order optimization). End-to-end gains depend on BoB's share of the step and are modest in realistic settings: TTT with the released 8K window gains about 1.0×, and GradMem gains 1.2–1.6×. For the JVP, only the sCM forward-mode phase is measured, not a full training step.

  3. The presentation is hard to follow. The main text is heavily compressed and notation-dense, and many key arguments and results are deferred to a 35-page appendix. This makes the paper hard to follow for readers unfamiliar with FlashBoB.

Questions for authors

How much of the speedup over FlashBoB remains if FlashBoB’s 18-matmul computation is implemented in CuTe DSL with the same hardware-specific schedules? This would isolate the contribution of the 18 to 15 reformulation from the gains of moving from Triton to CuTe DSL.

Besides, I suggest trying an AI coding agent to implement and optimize these kernels, and reporting how far it can push their performance. This would show how much of the gain requires expert hand-tuning and how much current agents can already achieve.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

Thank you for the careful review and the positive assessment. We answer in three comments.

W1 (Q1): Speedup against an 18-matmul CuTe DSL kernel.

A1: We built this kernel. On H100 at d=128d=128, FlashBoB-2 is 1.401.40–1.45×1.45\times faster than FlashBoB's 18-matmul graph written in CuTe DSL (Table R1, last column). This is more than the 1.111.11–1.17×1.17\times that the rewrite gives in Triton, because the rewrite does two things: it removes matmuls, and it makes room for a faster schedule.

Table R1. Time relative to the 5+105+10 graph in one CuTe DSL kernel design on H100 (causal BF16, 12 heads, d=128d=128; Table 30). The first four columns differ only in the matmul graph. FlashBoB-2 is the final kernel, whose row pass handles two query blocks per thread block.

Batch, NN5+105+10 (15 matmuls)7+97+9 (16)8+98+9 (17)(a) 10+810+8 (18, FlashBoB)(b) FlashBoB-2(a) ÷\div (b)
8, 2K1.0001.0701.1451.1560.8201.41
4, 4K1.0001.0781.1641.1700.8371.40
1, 16K1.0001.0731.1741.1840.8181.45
4, 16K1.0001.0711.1661.1720.8271.42

Fewer matmuls: compare (a) with the 5+105+10 column. With the schedule held fixed, the 18-matmul graph takes 1.161.16–1.18×1.18\times the time of the 15-matmul graph, close to the matmul ratio 18/15=1.218/15=1.2. Fifteen is also the minimum for direct two-pass splits (Proposition C.1).

A faster row-pass schedule: compare (a) with (b). With five matmuls removed from the row pass, one thread block has room in its registers for two query blocks instead of one. With two query blocks under our register placement at d=128d=128, the 18-matmul row pass would need at least 536 registers per thread summed over its three warpgroups, or 68,608 for the thread block. One streaming multiprocessor (SM) has 65,536 (Table 25). This schedule accounts for the step from 1.161.16–1.18×1.18\times to 1.401.40–1.45×1.45\times.

A standalone JVP kernel. The rewritten row pass is a complete JVP kernel, so the graph matters more for the JVP. With both versions in Triton under the same autotuner, removing the five extra row-pass matmuls makes the JVP 2.92.9–3.3×3.3\times faster on H100 and 1.91.9–2.2×2.2\times on B200 (Table 34). rCM and jvp_flash_attention are Triton kernels, as you note, so part of our advantage over them comes from the programming model. Upstream FlashAttention's CuTe DSL and Hopper code had no JVP path as of October 2026 (Appendix G.3).

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

W2: Application domain and end-to-end gains.

A2: Both kernels serve several areas (Appendix B), and the FlashAttention versions used in our evaluation support neither operation.

  • JVP. Attention JVPs train consistency models [1] and mean-flow models [2] that generate images, audio [3], molecules [4], driving scenes [5] and robot actions [6]. Several of these methods build their own JVP kernels [7, 8] or change their loss to avoid the JVP [9]; see Table 6.
  • BoB. BoB supports methods that differentiate gradients through attention, including second-order optimizers [10], meta-learning [11], data curation (DataRater [12]), test-time training [13], gradient-written memory (GradMem [14]), gradient penalties [15] and influence functions [16].

The end-to-end gain follows the share of the step spent in attention derivatives (Table 4). A 1B-parameter GradMem model at 16K on B200 runs 1.62×1.62\times faster per step. Test-time training with its released 8K window runs 1.01×1.01\times faster, because BoB is 66% of that step.

For sCM, we timed the forward-mode and forward-backward phases separately and did not time a complete training step. The forward-mode phase at 128K runs 2.97×2.97\times faster. At 8K on B200, the two phases together give a 1.24×1.24\times estimate, which excludes the optimizer and recipe-specific teacher/target work (Section 5.4, Appendix G.4).

W3: Presentation.

A3: Section 4 now opens with a roadmap and presents the computation before the schedules. The two-pass overview and the Hopper schedule are in the main text (Figures 1 and 4), so a reader new to FlashBoB sees its two passes before the rewrite. Section 4.1 derives the graph change in two steps next to a side-by-side graph (Figure 2), and Sections 4.3–4.4 state each hardware limit before the decisions that follow from it.

References

[1] C. Lu and Y. Song. Simplifying, stabilizing and scaling continuous-time consistency models. In International Conference on Learning Representations, 2025.

[2] Z. Geng, M. Deng, X. Bai, Z. Kolter, and K. He. Mean flows for one-step generative modeling. In Advances in Neural Information Processing Systems, 2025.

[3] X. Li, J. Liu, Y. Liang, Z. Niu, W. Chen, and X. Chen. MeanAudio: Fast and faithful text-to-audio generation with mean flows. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2026.

[4] D. Rehman, T. Akhound-Sadegh, A. Gazizov, Y. Bengio, and A. Tong. FALCON: Few-step accurate likelihoods for continuous flows. In International Conference on Learning Representations, 2026.

[5] C. Jiang, D. Zhou, J. Liu, and L. Sun. VectorWorld: Efficient streaming world model via diffusion flow on vector graphs. In Proceedings of the International Conference on Machine Learning, 2026.

[6] J. Sheng, Z. Wang, P. Li, and M. Liu. MP1: MeanFlow tames policy learning in 1-step for robotic manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2026.

[7] K. Zheng, Y. Wang, Q. Ma, H. Chen, J. Zhang, Y. Balaji, J. Chen, M.-Y. Liu, J. Zhu, and Q. Zhang. Large scale diffusion distillation via score-regularized continuous-time consistency. In International Conference on Learning Representations, 2026.

[8] L. Zhou, M. Parger, A. Haque, and J. Song. Terminal velocity matching. In International Conference on Learning Representations, 2026.

[9] T. Luo, H. Yuan, and Z. Liu. SoFlow: Solution flow models for one-step generative modeling. In International Conference on Learning Representations, 2026.

[10] H. Liu, Z. Li, D. Hall, P. Liang, and T. Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In International Conference on Learning Representations, 2024.

[11] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning, 2017.

[12] D. A. Calian et al. DataRater: Meta-learned dataset curation. In Advances in Neural Information Processing Systems, 2025.

[13] A. Tandon et al. End-to-end test-time training for long context. arXiv preprint arXiv:2512.23675, 2025.

[14] Y. Kuratov, M. Kairov, A. Bulatov, I. Rodkin, and M. Burtsev. GradMem: Learning to write context into memory with test-time gradient descent. In Proceedings of the International Conference on Machine Learning, 2026.

[15] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, 2017.

[16] P. W. Koh and P. Liang. Understanding black-box predictions via influence functions. In Proceedings of the International Conference on Machine Learning, 2017.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

W1 (continued): Schedules adapted from FlashAttention-3 and -4.

A4: We build on FlashAttention-3 and -4, and Sections 4.2 and 4.3 say which parts. Their schedules were designed for first-order backward, and they do not fit the BoB column pass.

What we reuse. On Hopper, a producer warpgroup (a group of four warps) loads tiles with TMA, which copies them asynchronously, while consumer warpgroups run matmuls. A TMA reduce-add sums partial outputs, and the row pass follows FlashAttention-3's pingpong schedule. On Blackwell at d=128d=128, one MMA warp issues the column-pass matmuls.

Why their schedules do not fit at d=128d=128. The BoB column pass does twice the work per tile and keeps more state on chip.

Per tileFirst-order backward (FlashAttention-3, -4)BoB column pass (FlashBoB-2)
Matmuls510
Recomputed tiles kept on chip2 (SS, dPdP)4 (SS, FF, dPdP, CC)
State fits in one warpgroup's registers (Hopper, d=128d=128)yesno: about 144+192144+192 registers per thread, limit 255
Barriers for intermediate tiles (Hopper)13, one per tile
Intermediate tiles read as a right operand (Blackwell)none2 (dSdS, d~(S)\widetilde{d}(S))

The last three rows each lead to one scheduling decision.

1. Two warpgroups with different roles (Hopper). On Hopper, matmul results stay in registers, and a thread has at most 255. In FlashAttention-3, every consumer warpgroup both builds the tiles and accumulates the outputs, because the state of first-order backward fits in one warpgroup. Our column pass needs about 144 registers per thread to build tiles and 192 more to accumulate outputs, so one consumer warpgroup builds the tiles and the other accumulates the outputs.

2. One barrier per tile (Hopper). After this split, every intermediate tile travels through shared memory from the warpgroup that builds it to the warpgroup that uses it, and a barrier signals that the tile is ready. FlashAttention-3's backward guards its chain from PP to dSdS with one barrier. Our three intermediate tiles finish at different times. With one barrier for all three, all five output matmuls would wait for the last tile, although three of them do not need it. We give each tile its own barrier, so each output matmul starts when its input arrives. One shared barrier is 4.34.3–5.05.0% slower (Table 1b).

3. Where tiles are stored (Blackwell). On Blackwell, matmul results go to Tensor Memory, an on-chip store with 512 columns. A tile stored there can only be the left operand of a matmul, and the right operand must come from shared memory. FlashAttention-4's backward reads its intermediate tiles PP and dSdS only as left operands, so it can serve them from Tensor Memory. Our d~(Q)\widetilde{d}(Q) matmuls read dSdS and d~(S)\widetilde{d}(S) as right operands, so we write the intermediate tiles to shared memory, where one BF16 copy serves as both the left and the right operand.

The column-pass state also fills all 512 columns of Tensor Memory at d=128d=128. As FlashAttention-4 does with its own tiles, we reuse columns: the partial d~(Q)\widetilde{d}(Q) takes over the columns of C⊤C^{\top} after C⊤C^{\top} is read for the last time.

Result. On H100 from 4K tokens at d=128d=128, FlashBoB-2 sustains 6262–6464% of peak matmul throughput, FlashAttention-3's backward 5454–6262%, and FlashBoB 2525–2828% (Figure 5; each kernel is credited with its own matmul count).

Q2: AI coding agents.

A5: Thank you for this suggestion. We used AI coding agents during development, and we did not run a separate agent-only experiment. In this project, the authors supplied the decisions that span the graph, both passes and the limits of each GPU, and agents helped implement and measure them.

  • Agents: mainly implementation and profiling. They wrote kernel variants from schedules we specified and ran the benchmark and profiling sweeps. Most of the rejected alternatives in Table 1b and Appendix G.2.3 were built this way.
  • Authors: the main design decisions. We derived the 15-matmul graph. We worked out the minimality proof ourselves and had agents fill in details where needed. Most scheduling decisions, including the two warpgroup roles on Hopper and the Tensor Memory layout on Blackwell, came from our study of the two GPU architectures, and we made sure the agents' code followed them. Where the agents got stuck, an author had to step in.

This work was done several months ago, and newer agents may get further. The revised AI use statement now describes this additional use.

Summary. The revision adds the 18-matmul CuTe DSL comparison (Table 30) and reorganizes Section 4 around three main-text figures. We hope this addresses your concerns and that you will consider it in your final rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Message to all reviewers and the AC

Dear Reviewers and Area Chair,

Thank you for the careful reviews. The reviewers were broadly positive about the fifteen-matmul computation, the Hopper and Blackwell schedules, and the measured speedups. They raised four main concerns, which the revision addresses with targeted experiments. Added text in the PDF is marked in blue.

  • Graph against implementation: We wrote FlashBoB's 18-matmul graph in CuTe DSL. With the rest of the kernel held fixed on H100, it takes 1.161.16–1.18×1.18\times the time of the 15-matmul graph. The 15-matmul graph also allows a row pass that handles two query blocks per thread block, which the 18-matmul graph cannot use under our register placement at d=128d=128 (Table 25). With it, FlashBoB-2 is 1.401.40–1.45×1.45\times faster than the 18-matmul CuTe DSL kernel (Table 30).
  • Sources of the CuTe DSL gain: New Table 2 in the main text gives per-pass times and hardware counters. On H100 at d=128d=128, the column pass accounts for 8080–8383% of the gain over the 15-matmul Triton kernel (Table 28).
  • Presentation: The main text now includes the two passes (Figure 1), the new side-by-side graphs (Figure 2), and the Hopper schedule (Figure 4). Section 4.1 derives the graph change in two steps, and Sections 4.3–4.4 state each hardware limit before the schedule choice that follows from it.
  • Sharp attention: A row-sum correction in the row pass (α′\alpha' in Equation 6) brings the single-key d~(Q)\widetilde{d}(Q) absolute error to at most 1.8×10−51.8\times10^{-5} on both GPUs in our constructed tests, where FlashBoB reaches 0.120.12–0.180.18 (Tables 13 and 14). It costs 1.01.0–2.62.6% of BoB time (Table 19).

We answer each reviewer's remaining points in their own thread, and we respectfully ask that the revision be considered in your assessment.

Best,

Authors

Changes in the revised PDF

The revision mostly reorganizes the submitted material. The contribution and the claims are unchanged. We have a minor addition, which was requested by a reviewer.

  • Moved. The two-pass overview (Figure 1) and the Hopper warpgroup roles and tile timing (Figure 4) move from the appendix into the main text. To make room, the introduction's bar chart is removed (its data is in Figure 5), the related-work passages on approximate attention and kernel programming tools move to Appendix B, and the column-pass algorithm box moves to Appendix D.1.
  • Clarified. Section 4 opens with a roadmap, derives the graph change in two steps next to the new side-by-side Figure 2, and states each hardware limit before the schedule choice. The abstract names the BoB baseline ("FlashBoB's Triton kernel"), the AI use statement covers coding agents, and Limitations states the remaining BF16 limit on nearly one-hot rows. Three printed ranges now match the appendix tables (1.041.04–1.08×1.08\times B200 Triton, 1.221.22–1.28×1.28\times in Table 1b row 2, and 66–77% / 1.91.9–2.1×2.1\times for 7+97+9).
  • Trimmed. The appendix drops Proposition C.3, Section C.7, the summation-order figure, and a duplicate algorithm box. Proposition C.1 (minimality) and Table 7 remain.
  • Added (experiments in response to the reviews). The graph comparison in CuTe DSL (Table 1b row 1; Tables 25 and 30); per-pass times and hardware counters (Table 2; Tables 20, 24, 28, and 29); sharp-attention tests and the cost of α′\alpha' (Tables 12–16 and 19); two Sophia-H seeds past warm-up without α′\alpha' and a trained-checkpoint test of α′\alpha' (Appendix F); and the two-phase sCM estimate (Section 5.4, Appendix G.4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.