- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper presents FlashBoB-2, an optimized implementation of the attention Jacobian-vector products (JVPs) and backward-over-backward (BoB) on modern GPUs. The authors simplify the computation graph and reduce the matrix-multiply cound from 18 to 15 by utilizing the same equation used in FlashAttention backward. The authors further develop the specialized kernels for Hopper and Blackwell GPUs. The resulting kernel achieves 2.7–3.0× speedup on H100 and 3.6–4.2× speedup on B200.
Critical strengths
- The computation graph simplification is useful. Although the underlying math is relatively simple, it cancels redundant computation and enables a more efficient 2 pass decomposition
- The hardware-specific optimization for the two GPUs are performed carefully. The authors carefully consider the resource constraint on specific hardware and manage to implement the kernel efficiently under these constraints.
- The performance improvement of the kernels are significant and the paper include ablations and training workload evaluations, which supports the practical value of the optimized kernels.
Critical weaknesses
-
The presentation is difficult to follow, especially without consulting the appendix. The paper introduces many intermediate tensors and mathematical expressions when describing the BoB computation, making it difficult to understand the overall algorithm and identify the key differences from the original FlashBoB. In particular, the transition from the original 10+8 decomposition to the proposed 5+10 decomposition could be explained more intuitively. The hardware scheduling sections are also challenging to follow, as the reader needs to understand several architectural constraints and execution dependencies before the motivation for each scheduling decision becomes clear. Several useful figures, including Figures 4, 6, and 8, are only provided in the appendix. Moving some of these figures to the main paper, together with a higher-level explanation of the computation flow and scheduling decisions, would substantially improve readability.
-
The sources of the performance improvements could be analyzed more clearly. According to Table 1a, reducing the number of matrix multiplications from 18 to 15 provides a relatively modest speedup, while moving from the Triton implementation to the optimized CuTe DSL implementation accounts for most of the overall performance improvement. The paper includes ablations of several scheduling decisions, which are useful, but it is still difficult to understand which factors contribute most to the large improvement achieved by the CuTe DSL kernels. A more detailed performance breakdown, such as per-pass timings or additional profiling analysis, would help clarify the relative importance of the different hardware optimizations and better explain the reported speedups.
Questions for authors
- Could the authors provide a more intuitive explanation of the 5+10 decomposition and how it differs from the original 10+8 decomposition? In particular, it would be helpful to explicitly identify which computations are eliminated, which are moved between passes, and why this decomposition avoids redundant matrix multiplications. A side-by-side computation graph would make the algorithm easier to understand.
- Could the authors further clarify the main sources of the performance improvement from the CuTe DSL implementation? Table 1a shows that this accounts for most of the overall speedup, while Table 1b reports the benefits of several individual scheduling choices. It would be helpful to understand how much of the remaining improvement comes from more efficient Tensor Core execution, memory access and data movement, improved resource utilization, or other aspects of the implementation. Additional profiling results or a breakdown of the row-pass and column-pass execution times would be useful.
3 comments
Thank you for the detailed response. The additional explanations and profiling results address my concerns. I will maintain my original rating.
Author Follow-up
We want to thank you for your careful and methodical review. We would also love, if time permits, to see if our reorganization of the paper helped to make it more readable. That said, we do appreciate the time you took to review our paper.
Rebuttal by Authors
Thank you for the specific suggestions, which shaped our revision of Section 4. To make the paper easier to read, we moved the two-pass overview and the Hopper schedule from the appendix into the main text.
W1 (Q1): Intuitive account of 10+8 against 5+10, with a side-by-side graph.
A1: FlashBoB's row pass does three jobs with its ten matmuls per tile, and only the first must run there. We remove the second and move the third to the column pass, where it takes two matmuls instead of three (Figure 2).
Table R3. How FlashBoB-2 cuts FlashBoB's ten row-pass matmuls per tile to five.
W1 (continued): Figures and the scheduling sections.
A2: Former Figure 4 is now Figure 1, and condensed versions of former Figures 6 and 8 form Figure 4 (full diagrams in Appendix D.2, Figures 7 and 9). For d=128, Sections 4.3–4.4 state each hardware limit before the schedule choice:
W2 (Q2): Where the CuTe DSL speedup comes from.
A3: On H100 at d=128 from 4K to 16K, the column pass accounts for 80–83% of the time saved over the 15-matmul Triton kernel (Table 28).
Table R4. Per-pass times and Nsight Compute counters on H100 at 16K, d=128 (new Table 2 in the main text).
We hope these revisions address your concerns and that you will consider them in your final rating.