- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Strassen-style matrix multiplication saves multiplications by temporarily adding different matrix blocks together and later cancelling unwanted terms. In FP8 precision, rounding can make that cancellation imperfect. Thus, later tokens can influence earlier outputs despite the causal attention mask in language modelling.
The authors first demonstrate this failure in experimental FP8 Strassen implementations; next, they how that normal accuracy tests may not reveal it. They then construct an integer version whose output is provably identical to ordinary row-local int8 multiplication. Finally, thye reduce the theoretical block-multiplication count from 64 to 49.
Critical strengths
-
The author have made a good case on identifying a genuine correctness property that accuracy cannot certify. The causal explanation is also convincing. Rounding prevents cross-token terms from cancelling exactly. In their experiemnts, they show both bf16 and the deployed row-local FP8 kernel show zero violations as good controls
-
As a plus, author provides a mathematical certificate rather than relying only on empirical tests. Bitwise equivalence to classical int8 is a pretty clean guarantee.
-
The paper is quite well written consider the dense technical details it has to pack
Critical weaknesses
-
The proposed implementation is just not fast: one-level Strassen is about 2-3x slower than classical int8, and two-level Strassen is almost 28× slower. I know that author have made the headline 49-versus-64 saving counts multiplications, but this is not exactly latency or flops savings?
-
A more realistic profiling of efficiency would be very useful here. It seems like the current way of counting the operation excludes additional additions, scaling, overflow correction, etc? How does the claimed 49-versus-64 multiplication saving translate into actual end-to-end inference cost?
Questions for authors
-
how realistic is the problem that is posed by the authors? to my understanding, the problematic Strassen implementations are constructed by the authors?
-
I wonder also how the posed problem would behave in non-multiple choice kind of datasets or problem, where the output space is more continuous and less discrete. one concern i have is how big really is the proposed problem. Mulpitle choice tends to push things to discrete which can be seen as a worse case scenario?
4 comments
follow-up to GmGk
Dear Reviewer GmGk,
Thank you again for your review. We would be grateful to know whether our response addresses your concerns.
In short, the kernels we have written since the submission run faster than classical int8, and with every step included, the configuration we serve raises end-to-end throughput over SGLang's FP8 path by 1.2–5.6%. On realism, both existing fast kernels the paper cites fail the prefix-invariance test. Beyond multiple choice, the violations also change generated answers.
If our response has addressed your concerns, we hope this will be reflected in your final evaluation. The author–reviewer discussion closes on November 18 (AoE), and if any concern remains, we would be glad to answer it before then.
Best regards,
The authors
Thank you
Thanks for the resposnes. I find the rebuttal useful and complete. I have raised my score.
thank you
Thank you for reading our replies, and for raising your score.
reply to GmGk
Thank you for the review, and for the positive assessment of the writing.
Our new kernels run faster than classical int8. With every step included, one level of Strassen raises serving throughput over SGLang's own FP8 path by 1.2–5.6% (§5, App. E).
Point 2 of our general comment summarizes these kernels.
The problem is realistic. The two existing fast kernels the paper cites also fail the prefix-invariance test, and in batched inference one request's answer depends on another request's text (§3, App. C.6–C.8).
The failing realizations in the submission were indeed ours, so we tested both kernels. Point 1 of our general comment gives the results:
The failure also reaches users through batching. A serving engine packs many requests into one matrix, so a kernel whose rounded block sums mix token rows can make one request's answer depend on another's text. With two requests in each forward pass, FalconGEMM changes the first request's generated answer on 39.6–43.9% of NQ-open questions when only the other request's text changes.
Multiple choice is not the worst case, because a discrete choice shows only part of the change. Most of the log probabilities behind it change, and a generated answer can change at any one of its tokens (§3, App. C.4, C.6, C.7).