acceptodds
Under review as a conference paper at ICLR 2027

Beyond Accuracy: Prefix-Invariant Realizations of Low-Precision Fast Matrix Multiplication

Abstract

Fast matrix multiplication saves multiplications through exact cancellation, but rounding sums that mix token rows can leave contributions from later tokens in earlier language model outputs. This threatens prefix invariance, which multiple-choice likelihood scoring relies on: a scored likelihood must depend only on its allowed prefix. On Qwen2.5-14B-Instruct, two fast FP8 realizations repaired to ordinary-looking accuracy still change the answers chosen by likelihood on 5.83% and 10.00% of 240 OpenBookQA items when only the text after the allowed prefix is replaced with the bf16 model's own greedy continuation. Both row-local controls, the bf16 model and a deployed FP8 matrix multiplication kernel, change none. Accuracy thus does not certify prefix invariance, and the stability criteria we analyze cannot tell realizations apart: across all 512 sign variants of two-level Strassen they stay constant while teacher-forced perplexities span a range on the same model. We therefore construct certified realizations of two-level Strassen on bounded integer codes that quantize token rows independently, then mix and cancel exactly before rescaling, using 49 block multiplications instead of 64. Our certificate guarantees bitwise equality to a prescribed row-local classical int8 operator at the same quantization specification, so every certified realization inherits its prefix invariance. Certification thus turns realization choice into a pure cost decision: which certified realization runs can no longer change a single scored likelihood.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
fast matrix multiplication, low-precision computation, prefix invariance, numerical stability, integer quantization, language model inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.80 over 5 reviews, above 85% of the others.
Reviewer_GmGkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Strassen-style matrix multiplication saves multiplications by temporarily adding different matrix blocks together and later cancelling unwanted terms. In FP8 precision, rounding can make that cancellation imperfect. Thus, later tokens can influence earlier outputs despite the causal attention mask in language modelling.

The authors first demonstrate this failure in experimental FP8 Strassen implementations; next, they how that normal accuracy tests may not reveal it. They then construct an integer version whose output is provably identical to ordinary row-local int8 multiplication. Finally, thye reduce the theoretical block-multiplication count from 64 to 49.

Critical strengths
  • The author have made a good case on identifying a genuine correctness property that accuracy cannot certify. The causal explanation is also convincing. Rounding prevents cross-token terms from cancelling exactly. In their experiemnts, they show both bf16 and the deployed row-local FP8 kernel show zero violations as good controls

  • As a plus, author provides a mathematical certificate rather than relying only on empirical tests. Bitwise equivalence to classical int8 is a pretty clean guarantee.

  • The paper is quite well written consider the dense technical details it has to pack

Critical weaknesses
  • The proposed implementation is just not fast: one-level Strassen is about 2-3x slower than classical int8, and two-level Strassen is almost 28× slower. I know that author have made the headline 49-versus-64 saving counts multiplications, but this is not exactly latency or flops savings?

  • A more realistic profiling of efficiency would be very useful here. It seems like the current way of counting the operation excludes additional additions, scaling, overflow correction, etc? How does the claimed 49-versus-64 multiplication saving translate into actual end-to-end inference cost?

Questions for authors
  • how realistic is the problem that is posed by the authors? to my understanding, the problematic Strassen implementations are constructed by the authors?

  • I wonder also how the posed problem would behave in non-multiple choice kind of datasets or problem, where the output space is more continuous and less discrete. one concern i have is how big really is the proposed problem. Mulpitle choice tends to push things to discrete which can be seen as a worse case scenario?

4 comments
AuthorsOpenReview ↗

reply to GmGk

Thank you for the review, and for the positive assessment of the writing.

W1 / W2: The proposed implementation is just not fast: one-level Strassen is about 2-3x slower than classical int8, and two-level Strassen is almost 28× slower. … the headline 49-versus-64 saving counts multiplications, but this is not exactly latency or flops savings? … It seems like the current way of counting the operation excludes additional additions, scaling, overflow correction, etc? How does the claimed 49-versus-64 multiplication saving translate into actual end-to-end inference cost?

Our new kernels run faster than classical int8. With every step included, one level of Strassen raises serving throughput over SGLang's own FP8 path by 1.2–5.6% (§5, App. E).
Point 2 of our general comment summarizes these kernels.

  • Kernel time: on an NVIDIA H20, the one-level kernel, which reads block sums formed during quantization, performs 7/8 of classical int8's multiplications and takes 0.892–0.916 of classical int8's time. Complete two-level calls at code bound 31 (codes within ±31), which form the block sums themselves, perform 49/64 of the multiplications and, per layer, take 0.81–0.86 of the time of quantization followed by classical int8.
  • Additions, scaling and correction: the end-to-end timings below include the additions and the scaling, which is the same as classical int8's. We serve one level because at code bound 63 it needs no overflow correction and matches FP8's quality. Two levels avoid the correction only at code bound 31, whose coarser codes cost quality (§4.4).
  • End to end: with the served configuration, a prefill batch of each 8B model takes 0.945–0.959 of FP8's GPU time in SGLang. Throughput exceeds FP8's on four models (8B–70B) and four workloads, and on each model the excess NLL over the bf16 model is lower than FP8's.

 

Q1: how realistic is the problem that is posed by the authors? to my understanding, the problematic Strassen implementations are constructed by the authors?

The problem is realistic. The two existing fast kernels the paper cites also fail the prefix-invariance test, and in batched inference one request's answer depends on another request's text (§3, App. C.6–C.8).
The failing realizations in the submission were indeed ours, so we tested both kernels. Point 1 of our general comment gives the results:

  • FalconGEMM (Zhu et al., 2026), run unchanged in Qwen2.5-14B-Instruct, changes 4.17–7.50% of OpenBookQA answers when only the text after the allowed prefix is replaced.
  • SubCuber (Jangda, 2026), tested in a single matrix multiplication, changes 41.9% of the outputs in the first half of the rows when only the second half is replaced.

The failure also reaches users through batching. A serving engine packs many requests into one matrix, so a kernel whose rounded block sums mix token rows can make one request's answer depend on another's text. With two requests in each forward pass, FalconGEMM changes the first request's generated answer on 39.6–43.9% of NQ-open questions when only the other request's text changes.

 

Q2: I wonder also how the posed problem would behave in non-multiple choice kind of datasets or problem, where the output space is more continuous and less discrete. one concern i have is how big really is the proposed problem. Mulpitle choice tends to push things to discrete which can be seen as a worse case scenario?

Multiple choice is not the worst case, because a discrete choice shows only part of the change. Most of the log probabilities behind it change, and a generated answer can change at any one of its tokens (§3, App. C.4, C.6, C.7).

  • Multiple choice itself: under FalconGEMM, the chosen OpenBookQA answer changes on 4.17–7.50% of items, but 80.2–89.1% of the scored answer tokens change their log probability.
  • Free text: on five models, we replace everything after the first quarter of each 1,024-token window. Under our own FP8 realizations from the submission (repaired #1 and #2), the next-token distribution changes at 97.3% or more of the positions in that first quarter. Under the controls, it changes at none.
  • Generated answers: in the two-request test of Q1, on Qwen2.5-14B-Instruct, the first request asks one of 164 NQ-open questions. Under repaired #1 and #2 respectively, its first generated token changes with the other request's text on 25 and 45 of the questions, but its whole answer changes on 66 and 85. Asked "who had a baby at 100 in the bible", for example, it answers "Sarah" next to one text and "Methuselah" next to another (repaired #2).
AuthorsOpenReview ↗

follow-up to GmGk

Dear Reviewer GmGk,

Thank you again for your review. We would be grateful to know whether our response addresses your concerns.

In short, the kernels we have written since the submission run faster than classical int8, and with every step included, the configuration we serve raises end-to-end throughput over SGLang's FP8 path by 1.2–5.6%. On realism, both existing fast kernels the paper cites fail the prefix-invariance test. Beyond multiple choice, the violations also change generated answers.

If our response has addressed your concerns, we hope this will be reflected in your final evaluation. The author–reviewer discussion closes on November 18 (AoE), and if any concern remains, we would be glad to answer it before then.

Best regards,
The authors

Reviewer_GmGkOpenReview ↗

Thank you

Thanks for the resposnes. I find the rebuttal useful and complete. I have raised my score.

Thank you for reading our replies, and for raising your score.

Reviewer_hnnvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates prefix-invariance violations caused by low-precision fast matrix multiplication (FMM) in autoregressive language models. The authors show that rounding intermediate block sums in Strassen-like algorithms can prevent exact cancellation, allowing future tokens to influence earlier outputs despite causal attention masking. Experiments demonstrate that FP8 realizations with near-baseline accuracy still produce inconsistent likelihood-based predictions when future tokens are modified. Furthermore, 512 mathematically equivalent Strassen variants exhibit a 772.4× perplexity range despite having identical values under conventional coefficient-based stability criteria.

To address this issue, the authors propose certified FMM realizations using bounded integer arithmetic to ensure exact cancellation and bitwise equivalence to classical int8 matrix multiplication. The proposed construction preserves prefix invariance while reducing the theoretical number of block multiplications from 64 to 49. Theoretical guarantees are supported by numerical verification and experiments across multiple language models. However, the current implementations do not achieve practical speedups, and efficient execution of the full-range int8 variant remains an open challenge.

Critical strengths
  1. The paper demonstrates that low-precision fast matrix multiplication can violate prefix invariance even when model accuracy remains close to the baseline. The experiments provide convincing evidence across multiple models and evaluation settings. The finding that 512 mathematically equivalent Strassen variants exhibit dramatically different perplexities despite identical conventional stability criteria further highlights the limitations of existing evaluation methods.
  2. The paper provides theoretical results explaining the trade-off between prefix safety and multiplication efficiency. The proposed certified realization guarantees bitwise equivalence to classical int8 matrix multiplication while retaining the theoretical reduction from 64 to 49 block multiplications. These guarantees, supported by extensive numerical verification, provide a principled approach to preserving correctness in low-precision fast matrix multiplication.
Critical weaknesses
  1. Limited practical efficiency of the certified realizations. Although the proposed method reduces the theoretical number of block multiplications from 64 to 49, it does not demonstrate practical acceleration. The implemented certified kernels are substantially slower than classical int8, with the two-level Strassen implementation being 27.9× slower in the reported benchmark. Furthermore, the full-range int8 variant, which achieves competitive model quality, requires additional overflow correction and lacks an efficient GPU implementation. This limits the practical significance of the proposed approach for accelerating LLM inference.
  2. Unclear applicability to existing high-performance inference systems. The demonstrated prefix-invariance violations primarily arise from FP8 fast-matmul realizations constructed by the authors, while the deployed FP8 kernel used as a baseline does not exhibit these violations. Although the experiments convincingly establish that such failures are possible, it remains unclear whether they occur in existing high-performance FMM implementations used for LLM inference. This raises concerns about the practical relevance and broader impact of the identified problem.
Questions for authors
  1. The full-range int8 realization achieves competitive model quality but requires additional overflow correction. The reported multiplication count varies significantly depending on how this correction is implemented (0.79–0.82× versus 1.43–1.71× that of classical int8). Could the authors clarify which implementation strategy is practically feasible on current GPUs and whether the proposed approach can realistically retain a computational advantage over classical int8?
  2. The reported violations are primarily demonstrated using FP8 fast-matmul realizations constructed by the authors, whereas the deployed FP8 baseline does not exhibit this issue. ave been observed in existing high-performance FMM implementations for LLM inference, or whether the identified issue mainly represents a potential risk for future implementations?
3 comments
AuthorsOpenReview ↗

reply to hnnv

Thank you for the review, and for finding the evidence convincing.

W1 / Q1: The implemented certified kernels are substantially slower than classical int8, with the two-level Strassen implementation being 27.9× slower in the reported benchmark. Furthermore, the full-range int8 variant, which achieves competitive model quality, requires additional overflow correction and lacks an efficient GPU implementation. … The reported multiplication count varies significantly depending on how this correction is implemented (0.79–0.82× versus 1.43–1.71× that of classical int8). Could the authors clarify which implementation strategy is practically feasible on current GPUs and whether the proposed approach can realistically retain a computational advantage over classical int8?

Since the submission, we have written kernels that run faster than classical int8 on an H20. With them, one level of Strassen at code bound 63, which needs no correction and matches FP8's quality, is feasible today (§4.4, §5, App. D–E).
Point 2 of our general comment summarizes these kernels and the models and workloads they were measured on. Their code is in the supplementary material.

  • Kernel time: the one-level kernel, which reads block sums formed during quantization, takes 0.892–0.916 of classical int8's time. For two levels at code bound 31, the configuration of the 27.9×, complete calls now take 0.81–0.86 of the time of quantization followed by classical int8, per layer.
  • Which strategy: we use one level of Strassen at code bound 63. With an online Hadamard rotation and GPTQ weight codes, it reaches the deployed FP8 kernel's quality on all four of the paper's models without any overflow correction. Two levels also avoid the correction at code bound 31, but there the coarser codes cost quality (§4.4). We do not use the full-range variant, two levels at code bound 127, because it needs the correction. As a dense product over the rows that hold an overflow, the correction makes two levels perform 2.01–2.19× classical int8's multiplications, and even when corrected entry by entry, two levels perform 0.863–0.885 of them, about the 7/8 that one level performs with no correction (both counts over all linear layers below the output layer, not the submission's three slices).
  • Whether the advantage holds: yes, on the H20. In SGLang, one level at code bound 63 raises throughput over SGLang's own FP8 path by 1.2–5.6%, measured end to end. The advantage does depend on the GPU. On the H100, with our own int8 kernel and with NVIDIA's cuBLASLt, eight half-size products take 1.10–1.13× as long as one full-size product, so little is left of the one-eighth that one level saves. Because the certified realization's output is bit-for-bit identical to classical int8's, one can simply run the faster of the two on each GPU, classical int8 on the H100 and the certified realization on the H20.

 

W2 / Q2: … it remains unclear whether they occur in existing high-performance FMM implementations used for LLM inference. … or whether the identified issue mainly represents a potential risk for future implementations?

The violations already occur in existing kernels. Both fast kernels the paper cites fail the prefix-invariance test, and in batched inference one request's answer changes with another request's text (§3, App. C.6–C.8).

  • FalconGEMM (Zhu et al., 2026) is the kind of implementation you ask about, an FP8 Strassen kernel for LLM inference whose block sums mix token rows and are rounded to FP8 before the products. Run unchanged in Qwen2.5-14B-Instruct, it changes the chosen OpenBookQA answer on 4.17–7.50% of items when only the text after the allowed prefix is replaced. With two requests in each forward pass, it changes the first request's generated answer on 39.6–43.9% of NQ-open questions when only the other request's text changes.
  • SubCuber (Jangda, 2026), which compiles Strassen-like algorithms into fp16 kernels, has the same problem. Its released kernels do not run all of Qwen2.5-14B-Instruct's layer shapes, so we tested them directly. Replacing only the second half of the input rows changes 41.9% of the outputs in the first half.

Point 1 of our general comment also reports the same effect inside the SGLang serving engine. With an FP8 Strassen path we built like FalconGEMM's, a request's output changes with another request's text, and with our certified realization it never does.

Thank you for your responses. The results adequately address my concerns about practical relevance and efficiency. Considering the paper as a whole, I will maintain my original score.

Thank you for reading our replies, and for confirming that the new results address your concerns about practical relevance and efficiency.

Reviewer_bcVAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that low-precision fast matrix multiplication can violate prefix invariance in language models even when task accuracy remains close to the baseline. It provides sufficient conditions for integer realizations of Strassen to match a row-local classical int8 operator bit for bit, retaining the reduction from 64 to 49 block multiplications.

Critical strengths

The suffix-replacement experiments make a convincing case that accuracy alone can miss an important failure mode, with row-local controls helping isolate the effect of mixing token rows. The certificate provides a concrete remedy, and the lower bound explains why requiring every intermediate product to be prefix-safe would lose the multiplication savings.

Critical weaknesses

(1) Practical speedups remain unproven: the two-level implementation is 27.9× slower than classical int8 at the tested shape.

(2) Restricting codes to ±31 introduces a quality cost, while full-range correction can erase the multiplication savings depending on its implementation.

(3) The problematic FP8 implementations are the authors’ own compositions, leaving the prevalence of this issue in existing fast-multiplication kernels unclear.

Questions for authors
  1. Which existing fast-multiplication kernels share the cross-token mixing and rounding pattern tested here? Please distinguish the failure modes demonstrated in your implementations from those that you expect to occur in other kernels.

  2. The two-level implementation is 27.9× slower than classical int8 at the tested shape; how much of this comes from short block products, intermediate storage, and the encoding/decoding steps? A rough breakdown would help distinguish costs imposed by the construction from limitations of the current implementation.

  3. The correction requires 0.79–0.82× the classical multiplication count when counting individual nonzero entries, but 1.43–1.71× when counting affected rows and columns. How representative are the three sampled layer slices, and does the observed overflow structure suggest an efficient way to execute these corrections?

  4. The paper acknowledges prior use of exact integer arithmetic in fast matrix multiplication. Could the authors state more explicitly which parts of the certificate require new arguments, particularly the treatment of groupwise scales and the guarantee of bitwise equivalence?

2 comments
AuthorsOpenReview ↗

reply to bcVA

Thank you for the review, and for finding the suffix-replacement design convincing.

W1 / Q2: The two-level implementation is 27.9× slower than classical int8 at the tested shape; how much of this comes from short block products, intermediate storage, and the encoding/decoding steps? A rough breakdown would help distinguish costs imposed by the construction from limitations of the current implementation.

Most of the 27.9× came from forming the block sums inside the kernel, a cost of our first implementation rather than of the construction. Kernels we have written since the submission run faster than classical int8 (§5, App. E).
We re-timed the submitted kernel at the paper's 4096³ shape, adding one step at a time:

  • Short block products: the 49 products alone take 1.91× classical int8's time. The cost lies in splitting a call into short products, since the same executor takes 2.22× even for the classical algorithm's 64 short products.
  • Intermediate storage: it is not a separate step, so its cost is counted in the encoding and decoding below. The block sums and accumulators of the 16 output blocks are held in registers, and the full kernel spills 2,272 registers to local memory.
  • Encoding and decoding: adding the decoding raises the 1.91× to 2.35×, and forming the block sums inside the kernel then raises it to 27.6×, the re-timed value of the 27.9×.

The new kernels (point 2 of our general comment) form the block sums in a separate pass and run the products back to back. With one scale per token row and per output channel, two levels, including that pass, take 0.894 of classical int8's time at the same shape.

 

W2 / Q3: Restricting codes to ±31 introduces a quality cost … How representative are the three sampled layer slices, and does the observed overflow structure suggest an efficient way to execute these corrections?

The slices understate the cost, and the overflows are too scattered for an efficient dense correction. So we deploy one level at code bound 63, which needs no correction and matches FP8's quality (§4.4, §5, App. D–E).

  • The slices: they come from layers that overflow little. Over all linear layers below the output layer, two levels with the correction done entry by entry perform 0.863–0.885 of classical int8's multiplications, rather than the slices' 0.79–0.82. That is about the 7/8 that one level performs with no correction.
  • The structure: it suggests no efficient dense execution, because the overflows are spread thinly. On the median layer, the 1% of activation rows with the most overflows hold only 7.5% of them. A dense correction over the rows that hold an overflow therefore makes two levels perform 2.01–2.19× classical int8's multiplications.

 

W3 / Q1: Which existing fast-multiplication kernels share the cross-token mixing and rounding pattern tested here? Please distinguish the failure modes demonstrated in your implementations from those that you expect to occur in other kernels.

Both kernels we cite, FalconGEMM and SubCuber, share the pattern, and both fail the test (§3, App. C.6–C.7).
Both apply a fast algorithm to blocks of token rows, combining token r with r + M/2, and round the block sums to lower precision before the products. In both kernels the failure is demonstrated, not merely expected, and point 1 of our general comment gives the numbers.

  • FalconGEMM, run unchanged in Qwen2.5-14B-Instruct, fails the replacement test. With two requests in each forward pass, one request's answer also changes with the other's text.
  • SubCuber does not run all the model's layer shapes, so we tested its kernels directly. Replacing the second half of the rows changes outputs in the first half.

 

Q4: … Could the authors state more explicitly which parts of the certificate require new arguments, particularly the treatment of groupwise scales and the guarantee of bitwise equivalence?

Both parts you name need new arguments, because existing exact-integer analyses concern only the integer product. They give the overflow bounds, conditions (i) and (ii) (§4.2, App. A).

  • Groupwise scales: the integer product has no scales, while each group along the inner dimension has its own. We show that a fast algorithm's block sums must mix indices of the inner dimension (Theorem 2), and that applying independent scales after its products takes the classical number of multiplications (Lemma 1). Condition (iii) therefore keeps each call within one group, where the scales are constant and applied to the group's exact integer product, as in classical int8.
  • Bitwise equivalence: the output is formed after the integer product, by rescaling each group's product in floating point, summing over the groups and casting. Our realizations run these steps as classical int8 does, in the same order. Conditions (i)–(iii) make each group's integer product equal classical int8's. The same steps on the same values then give the same bits.
AuthorsOpenReview ↗

follow-up to bcVA

Dear Reviewer bcVA,

Thank you again for your review. We would be grateful to know whether our response addresses your concerns.

In short, most of the 27.9× came from our first kernel forming the block sums itself, and the kernels we have written since the submission run faster than classical int8. On the code bound, the configuration we serve runs one level at code bound 63, which needs no correction and matches FP8's quality. On existing kernels, FalconGEMM and SubCuber, both cited in the paper, fail the prefix-invariance test.

If our response has addressed your concerns, we hope this will be reflected in your final evaluation. The author–reviewer discussion closes on November 18 (AoE), and if any concern remains, we would be glad to answer it before then.

Best regards,
The authors

Reviewer_2Dh4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies prefix invariance, the property that a model's output at a position depends only on its allowed prefix, under low-precision fast matrix multiplication. The paper observes that fast algorithms such as Strassen form block sums that mix token rows, so rounding these sums to FP8 can let later tokens affect earlier outputs inside linear layers, where the causal mask has no effect. With two-level Strassen on Qwen2.5-14B-Instruct, FP8 realizations repaired to near-bf16 accuracy still change 5.83–10.00% of OpenBookQA answers when only the text after the allowed prefix is replaced, while the row-local controls change none. The paper further shows that coefficient-based stability criteria take a single value across all 512 sign variants of two-level Strassen, while their perplexity spans a 772.4× range. As a remedy, the method constructs certified realizations on bounded integer codes. These satisfy a coefficient-level condition (Proposition 1) under which they compute the classical int8 operator bit for bit with 49 instead of 64 block multiplications, and the paper proves that a realization keeping every product prefix-safe requires the classical count (Theorem 1).

Critical strengths
  1. The paper identifies a concrete and less-discussed failure mode of fast low-precision matrix multiplication. Mixing token rows inside a linear layer can make an output depend on later text, and in packed batches on another request's text, which accuracy does not reveal. The test design is careful: the inputs share shape and execution plan, the row-local controls (bf16 and the deployed FP8 kernel) change nothing, and repeated runs rule out run-to-run variation. The distinction from batch-invariant deterministic inference is also clearly stated.

  2. Theorems 1 and 2 give a clean characterization: a fast algorithm cannot save multiplications without products that mix rows and inner-dimension indices. This explains why prefix invariance can only be guaranteed at the level of the final output, rather than product by product.

Critical weaknesses
  1. (Significance / Originality) The certified construction obtains prefix invariance by reproducing classical int8 exactly, so the guarantee follows from exact integer arithmetic, and the remaining question of whether this path is useful is not yet supported by the results.

    a. A bilinear algorithm evaluated exactly returns the exact product, so the core of Proposition 1 is an overflow bound together with a group-alignment condition. This is close to existing bounds for exact integer Strassen (e.g., Dumas et al., 2009, which the appendix cites). Its contribution to understanding prefix invariance beyond this exactness could be clarified.

    b. Since the certified output equals classical int8, its quality is that of int8 at the admitted code bound. At code bound 31, the quantization step is about 4× coarser, and the excess NLL exceeds the deployed FP8 kernel on three of four models (e.g., 60.7 vs. 13.7 ×10⁻³ nats on Qwen2.5-14B).

    c. Restoring code bound 127 requires the overflow correction. Its multiplication count is 0.79–0.82 of classical int8 only when counting the individual overflowing entries, which is difficult to realize on dense tensor cores, and 1.43–1.71× of classical int8 when counting whole rows and columns.

    d. The current kernels take 2.04–3.17× (one level) and 27.9× (two levels) as long as classical int8. The cost decision described in the conclusion therefore currently favors classical int8, which already computes the same output bits.

  2. (Quality) The evidence for the violation rests on the authors' own FP8 compositions, and its practical consequence is not separated from ordinary quantization noise.

    a. The fast realizations are the authors' compositions of DeepGEMM routines and their own adapted repairs. The unrepaired realization reaches 18.75% on four-choice OpenBookQA, below chance. It is not fully clear how large the violations are for the fast low-precision kernels cited as motivation (Jangda, 2026; Zhu et al., 2026).

    b. For repaired #1, 9 of the 14 changed answers are wrong before and after, and accuracy is unchanged. It would strengthen the paper to show whether later text systematically biases the scored likelihood, for example toward the answer whose tokens are present. That would be the evaluation-relevant harm, rather than only sensitivity near the decision boundary.

    c. The 772.4× sign-variant spread is shown under one FP8 schedule, its mechanism remains unexplained (the outlier-ratio test in Appendix B.2 is negative), and the certified path makes all variants identical by exactness. This finding is therefore only loosely connected to the proposed solution.

Questions for authors
  1. (ref Weakness 1) Beyond exact evaluation under an overflow bound, what does the certificate add over existing exact-integer analyses of Strassen? Is there a certified realization that is row-local but not bitwise equal to classical int8, for example one with FP8 codes and exact accumulation?

  2. (ref Weakness 1) Under what hardware conditions would a certified realization be faster than classical int8, given the 49/64 ratio and the coarser codes? Can the entrywise overflow correction be realized on dense tensor cores?

  3. (ref Weakness 2) How do existing fast low-precision kernels behave on the same prefix-invariance test?

  4. (ref Weakness 2) Do the violations bias the likelihood toward the scored answer, for example when the replaced text is the true continuation versus filler, or do they act as unbiased noise?

  5. (Clarification) The packed-request result suggests cross-request dependence in batched serving. Could the authors quantify how much information about one request can be recovered from another's outputs?

  6. (Presentation, minor) The paper is dense, and several key definitions (realization, quantization specification, allowed prefix) would benefit from a short summary table. For repaired #1, the log probabilities differ across container instances (Appendix C.1), which should be noted where its determinism is discussed in Section 3.1.

6 comments
AuthorsOpenReview ↗

reply to 2Dh4 (1/3): what the certificate adds, quality and the correction (W1a–W1c, Q1, Q2)

Thank you for the detailed review, and for recognizing the test design and Theorems 1–2.

W1a / Q1: … the core of Proposition 1 is an overflow bound together with a group-alignment condition. This is close to existing bounds for exact integer Strassen … Its contribution to understanding prefix invariance beyond this exactness could be clarified. … Beyond exact evaluation under an overflow bound, what does the certificate add over existing exact-integer analyses of Strassen?

The certificate adds condition (iii), which the groupwise scales require. Existing exact-integer analyses concern only the integer product, and they give the overflow conditions (i) and (ii) (§4.2, App. A).

  • The groupwise scales: the integer product has no scales, while each group along the inner dimension has its own. We show that a fast algorithm's block sums must mix indices of the inner dimension (Theorem 2), and that applying independent scales after its products takes the classical number of multiplications (Lemma 1). The group-alignment condition (iii) therefore keeps each call within one group, where the scales are constant and are applied to the group's exact integer product, as in classical int8.
  • The result: with (i)–(iii), each group's integer product equals classical int8's, and our realizations rescale the products, sum them over the groups and cast the result as classical int8 does, in the same order. Every certified realization therefore returns classical int8's output bits on every input, and so, like classical int8, keeps prefix invariance. Without (iii), prefix invariance can fail. A control that breaks it, giving each block column of a call its own group, rounds each mixed block sum with one scale, which any of its rows can change. Changing any later row then moves every earlier output row (§4.2).

The revised §4.2 makes this distinction and cites Dumas et al. (2009).

 

Q1: Is there a certified realization that is row-local but not bitwise equal to classical int8, for example one with FP8 codes and exact accumulation?

Not under our certificate, which certifies equality with classical int8. A realization with FP8 codes and exact accumulation, block sums included, would be row-local too, since it computes the exact product of its own codes. But for every block sum of two-level Strassen to stay exact in e4m3, it could use at most 9 evenly spaced codes, whereas int8 at code bound 31 offers 63 at the same eight bits (App. D). Forming the block sums in a wider format instead would take the products off the FP8 tensor cores, so we use integer codes.

 

W1b: … At code bound 31, the quantization step is about 4× coarser, and the excess NLL exceeds the deployed FP8 kernel on three of four models (e.g., 60.7 vs. 13.7 ×10⁻³ nats on Qwen2.5-14B).

Our served configuration, one level of Strassen at code bound 63 with an online Hadamard rotation and GPTQ weight codes, has excess NLL at or below the deployed FP8 kernel's on all four models (§4.4, App. D).
Its query–key–value projections run classical int8. Excess NLL under the paper's WikiText-2 protocol, ×10⁻³ nats:

modeldeployed FP8certified, served config.submission, b = 31
Qwen2.5-14B13.7311.8560.7
Llama-3.1-8B5.944.1214.5
OLMo-2-7B2.320.3011.0
Qwen3-8B1.381.26−5.5

 

W1c / Q2: … Its multiplication count is 0.79–0.82 of classical int8 only when counting the individual overflowing entries, which is difficult to realize on dense tensor cores … Can the entrywise overflow correction be realized on dense tensor cores?

As a dense product over the rows that hold an overflow, the correction makes two levels at code bound 127 perform 2.01–2.19× classical int8's multiplications, because the overflows are spread thinly (App. E). Even done entry by entry, it would leave two levels at 0.863–0.885 of classical int8's count, about the 7/8 that one level performs with no correction (both counts over all linear layers below the output layer, rather than the submission's three slices). So we deploy one level at code bound 63, which needs no correction.

AuthorsOpenReview ↗

reply to 2Dh4 (2/3): speed, existing kernels and bias (W1d, W2a, W2b, Q2–Q4)

W1d / Q2: The current kernels take 2.04–3.17× (one level) and 27.9× (two levels) as long as classical int8. The cost decision described in the conclusion therefore currently favors classical int8 … Under what hardware conditions would a certified realization be faster than classical int8, given the 49/64 ratio and the coarser codes?

A certified realization can be faster than classical int8 on a GPU that runs half-size block products about as efficiently as full-size ones. The H20 does, and on it the kernels we have written since the submission are now faster than classical int8 (§5, App. E).
One level replaces the eight half-size block products of the classical algorithm by seven, which saves one-eighth of the multiplications. The coarser codes do not slow these products, because every block sum stays within int8 and the products still run on the int8 tensor cores. What the coarser codes cost is precision (W1b). Point 2 of our general comment summarizes the new kernels and their serving results.

  • On the H20: the one-level kernel we serve, which reads block sums formed during quantization, takes 0.892–0.916 of classical int8's time, close to the 7/8 set by its multiplications. Per layer, complete two-level calls at code bound 31, which form the block sums themselves, take 0.81–0.86 of the time of quantization followed by classical int8.
  • On the H100: with our own int8 kernel and with NVIDIA's cuBLASLt, eight half-size int8 products take 1.10–1.13× as long as one full-size product, so little is left of the one-eighth that one level saves.

Because the certified realization's output is bit-for-bit identical to classical int8's, one can simply run the faster of the two on each GPU, classical int8 on the H100 and the certified realization on the H20.

 

W2a / Q3: … The unrepaired realization reaches 18.75% on four-choice OpenBookQA, below chance. It is not fully clear how large the violations are for the fast low-precision kernels cited as motivation (Jangda, 2026; Zhu et al., 2026). … How do existing fast low-precision kernels behave on the same prefix-invariance test?

Both kernels fail the test (§3, App. C.7). Before their products, both round activation block sums that combine token r with token r + M/2, so the later token's contribution no longer cancels exactly. Run unchanged in Qwen2.5-14B-Instruct, FalconGEMM (Zhu et al., 2026) keeps accuracy near the bf16 model's, unlike our unrepaired realization, yet it changes the chosen answer on 4.17–7.50% of items. SubCuber (Jangda, 2026) does not run all of the model's layer shapes, so we tested its kernels in a single matrix multiplication. Replacing only the second half of the rows changes 41.9% of the output entries in the first half, whereas cuBLAS changes none.

 

W2b / Q4: … That would be the evaluation-relevant harm, rather than only sensitivity near the decision boundary. … Do the violations bias the likelihood toward the scored answer, for example when the replaced text is the true continuation versus filler, or do they act as unbiased noise?

We detect no bias. The changes still do harm, because they make an answer's score depend on text outside its allowed prefix, and a served answer on what else is in the batch (App. C.2, C.6–C.8).
We ran the comparison you suggest. After the allowed prefix we place either filler or one answer's own text followed by filler. Under both repaired realizations, the log probability of the answer whose text is present shifts by −0.0108 to −0.0002 nats per token relative to the other answers, with every 95% interval containing 0. The choice also moves to it about as often as to another answer chosen at random. In the paper's replacement test (Table 1), the log probabilities that change move up about as often as down (49–52% up), and accuracy does not change detectably.

A row-local kernel, such as the deployed FP8 kernel, also adds rounding noise, but that noise depends only on the allowed prefix. A kernel whose rounded block sums mix token rows adds noise that also changes with the text that follows. Under FalconGEMM, for example, 39.6–43.9% of generated answers change with the other request's text, as point 1 of our general comment reports.

AuthorsOpenReview ↗

reply to 2Dh4 (3/3): the sign variants, information across requests and presentation (W2c, Q5, Q6)

W2c: The 772.4× sign-variant spread is shown under one FP8 schedule, its mechanism remains unexplained (the outlier-ratio test in Appendix B.2 is negative), and the certified path makes all variants identical by exactness. This finding is therefore only loosely connected to the proposed solution.

We have traced the spread on Qwen2.5-14B, and the trace shows why only certification makes every variant safe (§3.3, App. B.3).

  • The mechanism: block sums combine the first token of each window, whose activations are massive, with three other rows. The FP8 rounding error of the encoded weights, multiplied by these massive activations, then reaches those three rows. Moving the massive rows to positions whose partners are padding removes all 86 catastrophic variants.
  • Another schedule, other models: rescaling each token row by a power of two before the sums, as our repaired realizations do, narrows the spread from 772.4× to 1.078×. But on Qwen2.5-3B, 2 of its 3 catastrophic variants stay catastrophic under the same rescaling, and on Qwen3-8B, neither the first token's row nor its partner carries the error behind the model's worst variant.

So which variant is safe cannot be read off in advance. The coefficient criteria take one value on all 512 variants, and a model-level statistic such as the outlier ratio does not predict which models have a catastrophic variant. Certification needs no such prediction, because all 512 sign variants pass the certificate and compute the same row-local operator, bit for bit, on every input. Which one runs then becomes a pure cost decision (§3.3).

 

Q5: … Could the authors quantify how much information about one request can be recovered from another's outputs?

We pack a request A with a request B that holds one of 32 candidate texts, and measure how much A's output reveals about which one, from 0 bits if A's output is the same for all 32 candidates to 5 bits (log₂ 32) if it differs for each (App. C.5). Under either repaired realization, A's top-5 log probabilities, rounded to 0.01, differ for every candidate and so reveal all 5 bits. A lookup table built from them in one run identifies B's text in a second, separate run in 100% of cases, and in 99.4–99.8% under FalconGEMM. A's greedy next token alone carries 0.14–0.30 bits under these kernels. Under the row-local controls, A's output is the same for every candidate and carries 0 bits, as it does by construction under a certified realization.

 

Q6: … several key definitions (realization, quantization specification, allowed prefix) would benefit from a short summary table. For repaired #1, the log probabilities differ across container instances (Appendix C.1), which should be noted where its determinism is discussed in Section 3.1.

We made both changes:

  • a table of terms (App. G), pointed to from §2;
  • in §3.1, a note that the second run of repaired #1 was in another container instance, where some log probabilities differ (App. C.1), while every chosen answer is reproduced.
Reviewer_xyDxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether low-precision fast matrix multiplication preserves prefix invariance in autoregressive language models. Its central observation is that rounding intermediate block sums that mix token rows can introduce dependence on future tokens, even when the underlying matrix multiplication algorithm is algebraically exact and task accuracy remains close to the original model. The paper demonstrates this behaviour through suffix interventions in likelihood scoring, multi-model prediction tests, and experiments with separately masked packed requests. It also shows substantial variation across sign-equivalent implementations that share the analysed coefficient-based stability criteria. On the theoretical side, the paper proves that imposing prefix safety on every intermediate bilinear product forces the classical multiplication count. It then gives sufficient conditions under which an integer implementation reproduces a specified row-local classical int8 operator bit for bit. Two-level Strassen satisfies these conditions with 49 rather than 64 block multiplications at code bound 31. An overflow correction extends the construction to full-range codes. Validation includes exactness checks, model-quality measurements, and preliminary GPU timings.

Critical strengths
  1. The prefix-invariance experiments include appropriate controls. The suffix interventions preserve the allowed prefix, input shape, and execution plan while continuing to score the original target token.

  2. The theoretical construction provides an explicit correctness specification. Theorem 1 clarifies the limitation of imposing prefix safety on every intermediate product within the stated bilinear model. The certificate then gives sufficient conditions for reproducing a specified row-local int8 operator bit for bit, including bounds on integer intermediates, compatibility with scale groups, and agreement with the reference rescaling and accumulation order.

Critical weaknesses
  1. The certified construction has no demonstrated practical advantage over the existing row-local reference. The classical int8 baseline already provides prefix invariance, and the proposed construction has not established a useful accuracy–cost advantage over it.

  2. The relevance to existing fast implementations remains insufficiently established. The failures occur in the authors’ DeepGEMM compositions, while the deployed row-local FP8 control shows no violation. Since numerical schedules can materially affect this behaviour, a direct audit or precise implementation-level comparison is needed to establish the findings’ applicability to existing fast kernels.

Questions for authors

Q1. How representative are the tested implementations of existing fast kernels? Please identify which cited implementation shares the relevant token-row mixing, quantisation, intermediate precision, and accumulation schedule with the tested DeepGEMM compositions.

Q2. How should the quality comparison be interpreted given the different scale granularities? Appendix D uses a separate weight scale for each output column within a group of 128 inner-dimension entries, whereas Appendix B.1 describes a shared scale for each (128 \times 128) weight block in the FP8 baseline. How much of the difference in Table 7 comes from this distinction, and what additional scale-storage and execution costs does it introduce? A matched-granularity comparison would help assess the claimed operating trade-off.

3 comments
AuthorsOpenReview ↗

reply to xyDx

Thank you for the review, and for recognizing the controls of our tests and the certificate's explicit correctness specification.

W1: The certified construction has no demonstrated practical advantage over the existing row-local reference. The classical int8 baseline already provides prefix invariance, and the proposed construction has not established a useful accuracy–cost advantage over it.

Since the submission, we have written kernels that make the certified realization faster than classical int8 on an H20 GPU, with classical int8's output bit for bit on the same codes (§4.4, §5, App. D–E).
Point 2 of our general comment gives more detail on these kernels and their serving results.

  • Kernel time: the one-level kernel, whose block sums are formed during quantization, takes 0.892–0.916 of classical int8's time.
  • In serving: we serve one level of Strassen at code bound 63, with an online Hadamard rotation and GPTQ weight codes. It raises throughput over SGLang's own FP8 path by 1.2–5.6% on four models of 8B–70B, and on the paper's four models its excess NLL is at or below the deployed FP8 kernel's.

 

W2 / Q1: … The failures occur in the authors’ DeepGEMM compositions, while the deployed row-local FP8 control shows no violation. Since numerical schedules can materially affect this behaviour, a direct audit or precise implementation-level comparison is needed … Please identify which cited implementation shares the relevant token-row mixing, quantisation, intermediate precision, and accumulation schedule with the tested DeepGEMM compositions.

We audited both cited kernels directly, and both fail the prefix-invariance test. FalconGEMM (Zhu et al., 2026) shares the row mixing, the quantization and the e4m3 block sums of our compositions, and with an accumulation schedule of its own it changes about as many answers as repaired #1 (§3, App. C.7).

  • Implementation-level comparison: FalconGEMM is an FP8 Strassen kernel. Like repaired #1, it runs one level of Strassen, which combines each token in the first half of the rows with one in the second half. Like all our compositions, it rounds these activation block sums to e4m3 before the products, with one scale per row for every 128 channels, and the weights' block sums with one scale per 128×128 block. It has none of our repairs and combines the products in its own kernels, in two variants that we both ran. The deployed FP8 kernel, the control that shows no violation, uses the same format and scale layout but mixes no rows.
  • Direct audit: run unchanged in Qwen2.5-14B-Instruct, FalconGEMM changes the chosen OpenBookQA answer on 4.17–7.50% of items when only the text after the allowed prefix is replaced, against 5.83% for repaired #1. SubCuber (Jangda, 2026), an fp16 Strassen kernel, changes 41.9% of the output entries in the first half of the rows of a single matrix multiplication when only the second half is replaced, whereas cuBLAS changes none.

So the failure does not depend on our schedule: every kernel we ran that rounds block sums mixing token rows fails, and no row-local control does. Point 1 of our general comment adds its effect across requests and inside the SGLang serving engine.

 

Q2: … How much of the difference in Table 7 comes from this distinction, and what additional scale-storage and execution costs does it introduce? A matched-granularity comparison would help assess the claimed operating trade-off.

At matched weight scales, int8 at code bound 127 still has a lower excess NLL than the FP8 kernel on three of the four models (App. D, new Table 13).
Table 7 (Table 10 in the revision) does give the int8 operators finer weight scales than the FP8 kernel's 128×128 blocks, so we re-measured it at matched granularity both ways: classical int8 with those blocks, and e4m3 codes with the int8 operators' per-column scales. The int8 columns are also the certified realizations' values, because each scale is constant over a group of 128 inner entries and multiplies the group's exact integer product. On one chunk per model, two-level Strassen at code bound 31 returned the output bits of classical int8 with those blocks in all 224–336 calls.

How much: on Qwen2.5-14B, the blocks raise int8's excess NLL at code bound 127 from 3.2 to 5.8 ×10⁻³ nats, against 13.7 for FP8. On OLMo-2-7B, the exception, it is 2.5 against FP8's 2.3. Matched the other way, e4m3 at per-column scales stays above int8 at 127 on three of the four models. The exception is Qwen3-8B, at −0.6 against 0.0.

Cost: per-column scales store one fp32 value for every 128 weights (3.1% of the int8 weight storage), against one for every 16,384 with blocks, and the rescaling performs the same multiplications with either. The configuration we serve (§5) uses neither. It applies one scale per token and one per output channel, once, to the int32 sum, so its 1.2–5.6% throughput gain over SGLang's FP8 path (W1) already includes every scale cost.

AuthorsOpenReview ↗

follow-up to xyDx

Dear Reviewer xyDx,

Thank you again for your review. We would be grateful to know whether our response addresses your concerns.

In short, the kernels we have written since the submission make the certified realization faster than classical int8 on an H20, with the same output bits. On existing kernels, we audited both kernels the paper cites directly, and both fail the prefix-invariance test. On scale granularity, at matched weight scales int8 at code bound 127 still has a lower excess NLL than FP8 on three of the four models.

If our response has addressed your concerns, we hope this will be reflected in your final evaluation. The author–reviewer discussion closes on November 18 (AoE), and if any concern remains, we would be glad to answer it before then.

Best regards,
The authors

AuthorsOpenReview ↗

follow-up to xyDx: serving against classical int8

Dear Reviewer xyDx,

Thank you again for your review. We know this is a busy time, and we write once more to share one result added since our last note. This result bears directly on your first weakness, the practical advantage over classical int8.

Our earlier replies showed that our kernels run faster than classical int8 on an H20 GPU. To test whether this saving carries through to serving, we compared our served configuration with classical int8 on the same codes in the SGLang serving engine on Qwen3-8B and Llama-3.1-8B (App. E; code and results in the supplementary material). Ours returns the same output bits and has higher throughput on all four of the paper's serving workloads, by 4.3–6.7% on the two prefill-heavy ones.

If our response has addressed your concerns, we hope this will be reflected in your final evaluation. The discussion closes at the end of November 18 (AoE), and if any concern remains, we would be glad to answer it before then.

Best regards,
The authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment to the AC and all reviewers: summary of the response and the revisions

We thank the five reviewers for careful and constructive reviews, which agree that the paper identifies a failure mode that accuracy does not reveal. We have replied to every point in the per-reviewer threads. Below we summarize the two concerns all five reviewers raised, then list the revisions, which include the new analyses individual reviewers asked for.

1. Realism: both existing fast kernels the paper cites fail the test, and in batched inference one request's answer depends on another request's text (§3, App. C.5–C.8).
The violating realizations in the submission were our own compositions, so we tested the two existing kernels the paper cites.

  • Both existing kernels fail the test. FalconGEMM (Zhu et al., 2026) is an FP8 Strassen kernel. Run unchanged in Qwen2.5-14B-Instruct, it changes the chosen OpenBookQA answer on 4.17–7.50% of items when only the text after the allowed prefix is replaced. SubCuber (Jangda, 2026), tested in a single matrix multiplication, changes 41.9% of the output entries of the first half of the rows when only the second half is replaced, whereas cuBLAS changes none.
  • In batched inference, one request's answer depends on another request's text. With two requests in each forward pass, FalconGEMM changes the first request's generated answer on 39.6–43.9% of NQ-open questions when only the other request's text changes. Under the SGLang serving engine's own batching, an FP8 Strassen path we built like FalconGEMM's lets a request's generated output change with the other request's text in 79 and 86 of 112 cases on two models. With our certified realization, it never changes.

The row-local controls change nothing in any of these tests.

2. Speed: the certified realization now runs faster than classical int8 on an H20 and raises serving throughput over FP8 (§4.4, §5, App. D–E).
Since the submission, we have written optimized kernels for the certified realization and measured it inside the SGLang serving engine.

  • The kernels are faster than classical int8. On an NVIDIA H20, the one-level kernel, which reads block sums formed during quantization, takes 0.892–0.916 of classical int8's time on eight projection shapes of 8B–70B models. Per layer, complete two-level calls at code bound 31 (codes within ±31), which form the block sums themselves, take 0.81–0.86 of the time of quantization followed by classical int8. Their outputs are bit-for-bit identical to classical int8's.
  • Serving is faster than FP8, end to end. In SGLang, throughput exceeds SGLang's own FP8 path by 1.2–5.6% on four models (8B–70B) and four workloads.
  • The quality is at least FP8's, without correction. The served configuration runs one level of Strassen at code bound 63 in every projection except query–key–value, and at this code bound no overflow correction is needed. With an online Hadamard rotation and GPTQ weight codes, its excess NLL is at or below the deployed FP8 kernel's on all four of the paper's models.

Revisions
In the revised PDF, new text in the main paper is in blue.

  • Abstract and §1: batched inference is named as a second setting that relies on prefix invariance, and the FalconGEMM and speed results are added.
  • §3 (all reviewers): Table 1 adds FalconGEMM, and new results cover generated answers, the identification of another request's text and SGLang. The window test now reports how often the next-token distribution itself changes (GmGk). §3.1 notes that some log probabilities of repaired #1 differ between container instances, while every chosen answer is reproduced (2Dh4). §3.3 now connects the sign-variant spread to certification (2Dh4).
  • §4.2 (2Dh4, bcVA): what the certificate adds to exact-integer overflow bounds (Dumas et al., 2009), and one level, which we deploy, beside two.
  • §4.4 (2Dh4, bcVA, hnnv): the quality of the deployed configuration.
  • §5 (all reviewers): rewritten around the new kernels, the SGLang measurements and the correction's multiplication count on every layer below the output layer.
  • §6: rewritten and shortened.
  • App. B: what sets the sign variants apart (B.3, new, 2Dh4).
  • App. C: C.2 adds the bias analysis (2Dh4) and C.5 the identification of another request's text (2Dh4). New subsections cover generated answers (C.6, GmGk), the existing kernels (C.7, all reviewers) and the serving engine (C.8).
  • App. D–G: D adds quality tables for the deployed configuration, at matched weight scales (xyDx) and for FP8 with our rotation and GPTQ (2Dh4). E adds the new kernels, their timings on an H100 (2Dh4, hnnv), the time breakdown of the submitted kernels (bcVA, GmGk), the overflow counts on every layer below the output layer (2Dh4, bcVA, hnnv), the SGLang protocol and throughput against classical int8 (2Dh4). F adds the new results to the reproducibility table, and G is a new table of terms (2Dh4).
  • Supplementary material: the code and results of the new experiments.
AuthorsOpenReview ↗

Summary of the discussion

As the author–reviewer discussion comes to a close, we thank the five reviewers for their careful reviews and the AC for handling our submission. The reviews found that the paper identifies a failure mode that accuracy does not reveal (2Dh4, bcVA, GmGk, hnnv), with controlled tests that isolate the effect (2Dh4, bcVA, GmGk, xyDx). They also found that Theorems 1–2 explain the trade-off between prefix safety and saved multiplications (2Dh4, bcVA, hnnv, xyDx), and that the certificate gives a clean guarantee of bitwise equality with classical int8 (bcVA, GmGk, hnnv, xyDx). Below we summarize the main concerns and our responses:

  • Relevance to existing kernels (all reviewers). Both fast kernels the paper cites fail the prefix-invariance test, and in batched inference FalconGEMM makes one request's answer depend on another request's text. Reviewer 2Dh4 wrote: "The results on existing fast kernels, the bias analysis, and the quantified cross-request leakage resolve my other concerns and strengthen the motivation considerably."
  • Speed (all reviewers). Since the submission, we have written kernels that run faster than classical int8 on an H20 GPU, with identical outputs. In the SGLang serving engine, the configuration we serve, one level of Strassen, raises throughput over SGLang's own FP8 path by 1.2–5.6% on four models of 8B–70B. The speedup over classical int8 depends on the GPU (2Dh4), but because the two give identical outputs, a deployment can use whichever is faster on its GPU. Reviewer hnnv wrote that the new results "adequately address my concerns about practical relevance and efficiency", and Reviewer GmGk found the rebuttal "useful and complete".
  • Quality and overflow correction (2Dh4, bcVA, hnnv, xyDx). The configuration we serve needs no overflow correction, and the revised paper now presents one level beside two (2Dh4). With an online rotation and GPTQ weight codes, its excess NLL is at or below the deployed FP8 kernel's on all four of the paper's models. In the matched comparisons the reviewers asked for, full-range int8 still has a lower excess NLL than FP8 at equal weight-scale granularity on three of the four models (xyDx), and in SGLang, FP8 with the same rotation and GPTQ still has a larger NLL increase than our configuration on the two 8B models (2Dh4).
  • What the certificate adds (2Dh4, bcVA). Existing exact-integer analyses bound overflow in the integer product only. The certificate adds the condition that per-group scales require, so every certified realization returns classical int8's output bits on every input.
  • Harm beyond multiple choice (GmGk, 2Dh4). The violations also change generated answers, not only multiple-choice scores. We detect no bias toward the scored answer, but the changes still do harm, since one request's output can leak information about another request in the same batch.

All new results are in the revised PDF, with their code in the supplementary material, and our general comment below lists each revision by location. We thank the reviewers and the AC again for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.