- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 3
Read review
Paper summary
The paper studies speculative decoding with drafters that take hidden-state features from the target model as input. This includes modern drafters such as EAGLE-3, DFlash and DSpark. The setting is a target quantized to W4A4. There, the 16-bit draft accounts for a large share of decoding time, so quantizing the draft should in principle give further speedups. However, the authors show that W4A4 drafts suffer a significant drop in acceptance length relative to a 16-bit draft, and the drop is worst at later draft positions. This offsets the speedup. They trace much of this loss to the draft's input: features from several target layers have very different value ranges but are quantized together with a single 4-bit scale per token, so the smaller-range layers lose most of their resolution.
The proposed method, FLASHLight, has two steps. It rescales each layer's features by a constant and clips extreme activation values, so layers share the 4-bit range more evenly. It learns a rotation of the draft's hidden space so that the quantized draft agrees more closely with the quantized target.
FLASHLight retains about 97% of the 16-bit draft's acceptance length and improves end-to-end throughput by about 30% over the 16-bit draft. This is a useful contribution, but it is weakened by dense writing in places and limited methodological novelty.
Critical strengths
-
Making W4A4 drafting work for state-of-the-art drafters. The paper shows that W4A4 quantization can be applied to modern feature-sharing drafters (DFlash, DSpark) while retaining over 97% of the 16-bit-draft acceptance length. This gives meaningful end-to-end speedups. Why this matters: as targets are quantized, the draft becomes a larger share of decoding cost, so this is a practically important problem.
-
Identifying the bottleneck. The paper identifies that the loss in performance when using W4A4 drafters comes largely from the target-to-draft interface, where features from different layers share one 4-bit scale. This is a useful contribution in its own right. Why this matters: it is likely relevant to other drafters that consume target hidden states.
Critical weaknesses
1. Clarity and presentation. My main concern is readability. Parts of Sections 1–3 are hard to follow, and it took me considerable effort to work out what the method actually does. Specific issues:
Terminology and notation are used before they are defined. The introduction says "With the calibrated interface parameters and fixed…" and "We use the QSD loss…" before either is explained. Similarly, "LK" appears in the introduction, "Range-only" in Section 2.2 and Figure 2c, and "sink-excluded" in Section 3.2, each before it is defined. Other terms such as "verifier-guided," "source scaling," and "basis consistency" are introduced without a plain-language explanation.
Dense phrasing. For example: “For feature-sharing drafts, accepted-prefix preservation exposes two constraints: a shared per-token A4 grid can under-resolve smaller-range sources, while conventional draft rotations are selected without verifier agreement. Because basis consistency permits an independent draft rotation through function-preserving fusion, the interface range and draft basis can be optimized separately.” As far as I can tell, this means that layers with different value ranges share one 4-bit scale, and that the draft's rotation can be chosen separately. Saying that directly would make the method much easier to follow.
Numbers in running text interrupt the argument. For example: “Aligned PTQ's sign-flip rate rises from 0.048 at to 0.467 at , where its top-1 agreement is 0.170 versus the reference's 0.254.” Another is the batch-size paragraph in Section 4.4 (“…reaching 13.7% and 13.4% at … grows from 2.1% at to 6.5% at … 3.7%, 4.1%, and 7.6% higher at …”). I would suggest putting these in tables and using the text to state the takeaway.
Missing background. The paper does not explain how DFlash or DSpark work, in particular that the draft concatenates hidden states from several target layers and projects them with . That is the central fact behind the method. A short description could help readers a lot.
Section 4 onward, and parts of Sections 1 and 2, read well.
Why this matters: the core contribution is fairly principled, but the presentation makes it hard to identify and assess. I think the paper would benefit from better signposting and explanation, plus rewrites of some paragraphs in Sections 2–3.
2. Limited methodological novelty. Most of the methodological components follow established techniques: per-source scaling, activation clipping and learned orthogonal rotations. Also, the prefix-product objective is explicitly not claimed as a new contribution. The QSD loss is also not clearly better than plain per-position TV (Table 18). I do find the diagnosis that the concatenated target-to-draft interface is an important bottleneck useful. The novelty seems to lie mainly in how existing techniques are combined and applied.
Questions for authors
1. My main issue with this paper has been the readability in places. I would like to know how the authors respond to this criticism. Do you think this concern is valid, and, if so, do you think the paper could be revised to make the method and its motivation easier to follow?
2. Do the results hold under sampling-based decoding, where acceptance depends on the full distributions rather than only on top-1 agreement?
3 comments
Additional Sampling Evidence and Clarification of FLASHLight’s Scope
Relevance to your questions: sampling validity, clarity, and practical performance. Following the earlier T=1 evaluation, we added an independent test of the method on FlatQuant W4A4 drafts, rather than only on our original SpinQuant-based draft baseline. The experiment follows a simple sequence: existing W4A4 PTQ → interface range calibration → verifier-guided QSD → matched AL/TPS evaluation.
We additionally evaluated FlatQuant W4A4 drafts with two fixed verifier configurations: FlatQuant W4A4 targets and the paper’s original SpinQuant W4A4 targets, on DFlash and DSpark. Within each target policy, the draft checkpoint, verifier, proposal setup, and evaluation conditions are matched between the FlatQuant baseline and FlatQuant + FLASHLight-QSD. Sampling AL@1024 covers six datasets and five training seeds; TPS@256 is measured end-to-end under T=1 sampling.
FlatQuant draft — T=1 sampling AL@1024
| Target / Draft | Architecture | 16-bit draft | Aligned PTQ | + FLASHLight-QSD | Gain vs aligned |
|---|---|---|---|---|---|
| FlatQuant / FlatQuant | DFlash | 2.854 | 2.777 | 2.883 | +3.82% |
| FlatQuant / FlatQuant | DSpark | 4.773 | 4.666 | 4.726 | +1.29% |
| SpinQuant / FlatQuant | DFlash | 2.097 | 2.043 | 2.129 | +4.21% |
| SpinQuant / FlatQuant | DSpark | 2.150 | 2.122 | 2.205 | +3.91% |
FlatQuant draft — T=1 sampling TPS@256 (tokens/s)
| Target | Arch | B | 16-bit | Aligned | + FLASHLight | Gain |
|---|---|---|---|---|---|---|
| FlatQuant | DFlash | 1 | 275 | 344 | 357 | +3.78% |
| 2 | 506 | 613 | 635 | +3.59% | ||
| 4 | 877 | 1,029 | 1,065 | +3.50% | ||
| 8 | 1,405 | 1,562 | 1,598 | +2.30% | ||
| 16 | 1,935 | 2,076 | 2,121 | +2.17% | ||
| FlatQuant | DSpark | 1 | 552 | 684 | 696 | +1.75% |
| 2 | 798 | 922 | 939 | +1.84% | ||
| 4 | 1,439 | 1,647 | 1,673 | +1.58% | ||
| 8 | 2,408 | 2,666 | 2,715 | +1.84% | ||
| 16 | 3,635 | 3,796 | 3,848 | +1.37% | ||
| SpinQuant | DFlash | 1 | 232 | 290 | 301 | +3.79% |
| 2 | 425 | 524 | 543 | +3.63% | ||
| 4 | 737 | 874 | 903 | +3.32% | ||
| 8 | 1,154 | 1,302 | 1,346 | +3.38% | ||
| 16 | 1,520 | 1,688 | 1,715 | +1.60% | ||
| SpinQuant | DSpark | 1 | 359 | 464 | 474 | +2.16% |
| 2 | 521 | 623 | 637 | +2.25% | ||
| 4 | 979 | 1,149 | 1,175 | +2.26% | ||
| 8 | 1,664 | 1,897 | 1,942 | +2.37% | ||
| 16 | 2,333 | 2,530 | 2,568 | +1.50% |
Under standard T=1 sampling, adding FLASHLight improves AL over FlatQuant in all four architecture–target-policy comparisons (+1.29–4.21%). The AL gains are accompanied by higher end-to-end TPS in all 20 batch configurations (+1.37–3.79%). These are incremental benefits over FlatQuant itself, not merely over the original SpinQuant-based Aligned PTQ baseline.
EAGLE-3 — optimized backend, greedy end-to-end TPS@256 (tokens/s; SpinQuant W4A4 target)
| Draft | B1 | B2 | B4 | B8 | B16 |
|---|---|---|---|---|---|
| 16-bit draft | 382 | 687 | 1,112 | 1,476 | 1,631 |
| W4A16 GPTQ (Marlin) | 499 | 856 | 1,319 | 1,584 | 1,663 |
| FLASHLight-QSD W4A4 | 512 | 897 | 1,384 | 1,693 | 1,792 |
| Gain vs W4A16 | +2.6% | +4.8% | +4.9% | +6.9% | +7.8% |
EAGLE-3 now exceeds W4A16 GPTQ (Marlin) at all five batch sizes on the optimized backend (+2.6% at B1; +7.8% at B16). This extends the deployment measurements to tree-based drafting; these EAGLE-3 results use greedy decoding, separately from the FlatQuant sampling study.
What this adds to the revised manuscript. FLASHLight complements rather than replaces existing W4A4 PTQ methods. Its acceptance improvement persists under standard T=1 sampling with both FlatQuant and SpinQuant W4A4 verifiers. The updated EAGLE-3 measurements address the remaining backend-coverage limitation without conflating greedy and stochastic results.
Response to Authors
Thank you for the detailed response and the revisions.
I think the extensive changes to wording make the paper easier to read and strengthen it overall. The stochastic sampling experiments are a nice inclusion, and the FlatQuant experiments are also a useful addition. Together, these changes make me more comfortable with my positive assessment. I will keep my score of 3, but have raised my Presentation score to 2.
Clarified presentation and results under sampling
Thank you for your suggestions on clarity and sampling evaluation. In the revised PDF, we highlight selected changes addressing these comments in blue.
Question 1: My main issue with this paper has been the readability in places. I would like to know how the authors respond to this criticism. Do you think this concern is valid, and, if so, do you think the paper could be revised to make the method and its motivation easier to follow?
We agree that the missing background and undefined terms obscured the method. Section 1 now explains how DFlash and DSpark reuse hidden states from multiple target layers and project them into the draft input. The dense interface paragraph now states directly that these features share a four-bit activation range and that we optimize the interface range and draft rotation separately.
Method sequence clarified in the revision
First, source scaling and activation clipping are calibrated with a tensor-reconstruction objective. Scaling adjusts target-layer features before they share an A4 range; this stage alone is Range-only. With calibration fixed, an independent draft rotation is learned through W4A4 forward passes. QSD couples position-wise draft–verifier overlaps through prefix products as a surrogate for accepted-prefix preservation. The deployed verifier supplies supervision; pretrained weights stay fixed. Transformations are fused into weights and quantization kernels without a separate basis-conversion operator.
Section 1 explains source scaling, Range-only, verifier guidance, and basis consistency in plain language, without the premature S⋆, c⋆, and LK references. Section 3.2 specifies the excluded attention-sink positions; Section 3.3 defines an anchor before explaining overlap, consecutive acceptance, and QSD. The same-prefix discussion now emphasizes its takeaway and refers to the figure for measurements; Figure 6 and Tables 4–8 carry the runtime comparisons.
Contribution and comparison
Our contribution is verifier-guided low-bit basis optimization after interface calibration. QSD builds on existing acceptance-aware prefix objectives. To address your concern about its small original advantage over TV, the matched sampling and runtime comparisons below test the additional value of prefix coupling.
Question 2: Do the results hold under sampling-based decoding, where acceptance depends on the full distributions rather than only on top-1 agreement?
Yes. We added standard T=1 sampling: draw x from q and accept with probability min(1,p(x)/q(x)). On DFlash/DSpark, sampling AL exceeds Aligned PTQ and Range-only in every setting. Six-dataset mean relative gains over Aligned PTQ are +7.66%/+14.61%; gains over Range-only are +5.66%/+8.65%. The 16-bit draft remains the verifier-agreement reference.
Sampling AL across six datasets
Unweighted six-dataset means of B1 AL@1024 (Table 1; W4A16 GPTQ: Appendix B.2, Tables 6 and 8).
TV already optimizes single-token acceptance, 1−TV(p,q). Under matched conditions, QSD adds +2.85%/+3.89% mean relative AL for DFlash/DSpark under sampling, versus +0.76%/+0.52% under greedy decoding. The table lists all six datasets and sampling CIs over five training seeds; Table 11 also includes greedy CIs.
Sampling QSD–TV: B1 AL@1024 gain (%) [95% CI; 5 training seeds]
Dataset-specific means and 95% CIs of paired relative AL gains across five independent training seeds (TV and QSD matched by seed). These are not intervals for the six-dataset mean or TPS. Full AL: Figure 7, Tables 10–11.
On RTX 4090, QSD delivers 1.04–5.76% higher measured TPS than TV and 4.30–12.33% higher TPS than Range-only in all 60 sampling conditions (2 architectures × 6 datasets × B1/2/4/8/16). All W4A4 methods share the optimized backend. Figure 6 and Tables 4–8 report the end-to-end gains.