acceptodds
Under review as a conference paper at ICLR 2027

FLASHLight: Preserving Accepted Prefixes in Low-Bit Speculative Decoding

Abstract

Target quantization accelerates speculative verification, making draft execution cost increasingly important. W4A4 drafting reduces this cost, but lower verifier agreement can offset the gain. On DFlash and DSpark, the conditional-acceptance gap to a 16-bit draft widens toward later positions even under basis-consistent rotation-based PTQ (Aligned PTQ). We call this phenomenon quantization-induced suffix decay. We propose FLASHLight, which reframes W4A4 draft quantization as selecting a low-bit representation for the deployed verifier rather than reconstructing the high-precision draft. With target and draft weights frozen, FLASHLight calibrates the quantization-sensitive interface, then selects an independent draft basis using verifier guidance and a prefix-coupled overlap objective for accepted-prefix preservation. On DFlash and DSpark, FLASHLight retains 97.21-97.68% of the greedy-decoding AL of the 16-bit draft paired with the same W4A4 target, improving average AL over Aligned PTQ by approximately 11.8-22.8%. Repeated measurements on a single RTX 4090 show 28.5-29.8% higher mean end-to-end throughput at batch size 1 than that reference.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
speculative decoding, large language models, post-training quantization, low-bit inference, efficient LLM inference, model compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VoBAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
3
Read review
Paper summary

The paper studies speculative decoding with drafters that take hidden-state features from the target model as input. This includes modern drafters such as EAGLE-3, DFlash and DSpark. The setting is a target quantized to W4A4. There, the 16-bit draft accounts for a large share of decoding time, so quantizing the draft should in principle give further speedups. However, the authors show that W4A4 drafts suffer a significant drop in acceptance length relative to a 16-bit draft, and the drop is worst at later draft positions. This offsets the speedup. They trace much of this loss to the draft's input: features from several target layers have very different value ranges but are quantized together with a single 4-bit scale per token, so the smaller-range layers lose most of their resolution.

The proposed method, FLASHLight, has two steps. It rescales each layer's features by a constant and clips extreme activation values, so layers share the 4-bit range more evenly. It learns a rotation of the draft's hidden space so that the quantized draft agrees more closely with the quantized target. 

FLASHLight retains about 97% of the 16-bit draft's acceptance length and improves end-to-end throughput by about 30% over the 16-bit draft. This is a useful contribution, but it is weakened by dense writing in places and limited methodological novelty.

Critical strengths
  1. Making W4A4 drafting work for state-of-the-art drafters. The paper shows that W4A4 quantization can be applied to modern feature-sharing drafters (DFlash, DSpark) while retaining over 97% of the 16-bit-draft acceptance length. This gives meaningful end-to-end speedups. Why this matters: as targets are quantized, the draft becomes a larger share of decoding cost, so this is a practically important problem.

  2. Identifying the bottleneck. The paper identifies that the loss in performance when using W4A4 drafters comes largely from the target-to-draft interface, where features from different layers share one 4-bit scale. This is a useful contribution in its own right. Why this matters: it is likely relevant to other drafters that consume target hidden states.

Critical weaknesses

1. Clarity and presentation. My main concern is readability. Parts of Sections 1–3 are hard to follow, and it took me considerable effort to work out what the method actually does. Specific issues:

Terminology and notation are used before they are defined. The introduction says "With the calibrated interface parameters S⋆S^\star and c⋆c^\star fixed…" and "We use the QSD loss…" before either is explained. Similarly, "LK" appears in the introduction, "Range-only" in Section 2.2 and Figure 2c, and "sink-excluded" in Section 3.2, each before it is defined. Other terms such as "verifier-guided," "source scaling," and "basis consistency" are introduced without a plain-language explanation.

Dense phrasing. For example: “For feature-sharing drafts, accepted-prefix preservation exposes two constraints: a shared per-token A4 grid can under-resolve smaller-range sources, while conventional draft rotations are selected without verifier agreement. Because basis consistency permits an independent draft rotation through function-preserving fusion, the interface range and draft basis can be optimized separately.” As far as I can tell, this means that layers with different value ranges share one 4-bit scale, and that the draft's rotation can be chosen separately. Saying that directly would make the method much easier to follow.

Numbers in running text interrupt the argument. For example: “Aligned PTQ's sign-flip rate rises from 0.048 at d=1d=1 to 0.467 at d=9d=9, where its top-1 agreement is 0.170 versus the reference's 0.254.” Another is the batch-size paragraph in Section 4.4 (“…reaching 13.7% and 13.4% at B=8B=8… grows from 2.1% at B=1B=1 to 6.5% at B=16B=16… 3.7%, 4.1%, and 7.6% higher at B=4,8,16B=4,8,16…”). I would suggest putting these in tables and using the text to state the takeaway.

Missing background. The paper does not explain how DFlash or DSpark work, in particular that the draft concatenates hidden states from several target layers and projects them with WcW_c. That is the central fact behind the method. A short description could help readers a lot.

Section 4 onward, and parts of Sections 1 and 2, read well.

Why this matters: the core contribution is fairly principled, but the presentation makes it hard to identify and assess. I think the paper would benefit from better signposting and explanation, plus rewrites of some paragraphs in Sections 2–3.

2. Limited methodological novelty. Most of the methodological components follow established techniques: per-source scaling, activation clipping and learned orthogonal rotations. Also, the prefix-product objective is explicitly not claimed as a new contribution. The QSD loss is also not clearly better than plain per-position TV (Table 18). I do find the diagnosis that the concatenated target-to-draft interface is an important bottleneck useful. The novelty seems to lie mainly in how existing techniques are combined and applied.

Questions for authors

1. My main issue with this paper has been the readability in places. I would like to know how the authors respond to this criticism. Do you think this concern is valid, and, if so, do you think the paper could be revised to make the method and its motivation easier to follow?

2. Do the results hold under sampling-based decoding, where acceptance depends on the full distributions rather than only on top-1 agreement?

3 comments
AuthorsOpenReview ↗

Clarified presentation and results under sampling

Thank you for your suggestions on clarity and sampling evaluation. In the revised PDF, we highlight selected changes addressing these comments in blue.

Question 1: My main issue with this paper has been the readability in places. I would like to know how the authors respond to this criticism. Do you think this concern is valid, and, if so, do you think the paper could be revised to make the method and its motivation easier to follow?

We agree that the missing background and undefined terms obscured the method. Section 1 now explains how DFlash and DSpark reuse hidden states from multiple target layers and project them into the draft input. The dense interface paragraph now states directly that these features share a four-bit activation range and that we optimize the interface range and draft rotation separately.

Method sequence clarified in the revision

First, source scaling and activation clipping are calibrated with a tensor-reconstruction objective. Scaling adjusts target-layer features before they share an A4 range; this stage alone is Range-only. With calibration fixed, an independent draft rotation is learned through W4A4 forward passes. QSD couples position-wise draft–verifier overlaps through prefix products as a surrogate for accepted-prefix preservation. The deployed verifier supplies supervision; pretrained weights stay fixed. Transformations are fused into weights and quantization kernels without a separate basis-conversion operator.

Section 1 explains source scaling, Range-only, verifier guidance, and basis consistency in plain language, without the premature S⋆S^\star, c⋆c^\star, and LK references. Section 3.2 specifies the excluded attention-sink positions; Section 3.3 defines an anchor before explaining overlap, consecutive acceptance, and QSD. The same-prefix discussion now emphasizes its takeaway and refers to the figure for measurements; Figure 6 and Tables 4–8 carry the runtime comparisons.

Contribution and comparison

Our contribution is verifier-guided low-bit basis optimization after interface calibration. QSD builds on existing acceptance-aware prefix objectives. To address your concern about its small original advantage over TV, the matched sampling and runtime comparisons below test the additional value of prefix coupling.

Question 2: Do the results hold under sampling-based decoding, where acceptance depends on the full distributions rather than only on top-1 agreement?

Yes. We added standard T=1T=1 sampling: draw xx from qq and accept with probability min⁡(1,p(x)/q(x))\min(1,p(x)/q(x)). On DFlash/DSpark, sampling AL exceeds Aligned PTQ and Range-only in every setting. Six-dataset mean relative gains over Aligned PTQ are +7.66%/+14.61%; gains over Range-only are +5.66%/+8.65%. The 16-bit draft remains the verifier-agreement reference.

Sampling AL across six datasets

MethodDFlashDSpark
Aligned PTQ1.961.94
Range-only2.002.04
W4A16 GPTQ2.082.16
FLASHLight-TV2.052.14
FLASHLight-QSD2.112.22

Unweighted six-dataset means of B1 AL@1024 (Table 1; W4A16 GPTQ: Appendix B.2, Tables 6 and 8).

TV already optimizes single-token acceptance, 1−TV(p,q)1-\mathrm{TV}(p,q). Under matched conditions, QSD adds +2.85%/+3.89% mean relative AL for DFlash/DSpark under sampling, versus +0.76%/+0.52% under greedy decoding. The table lists all six datasets and sampling CIs over five training seeds; Table 11 also includes greedy CIs.

Sampling QSD–TV: B1 AL@1024 gain (%) [95% CI; 5 training seeds]

DatasetDFlashDSpark
MT-Bench+1.44% [−0.85, +3.73]+2.59% [+0.58, +4.60]
GSM8K+2.81% [+0.42, +5.20]+3.24% [−0.50, +6.98]
HumanEval+5.52% [−1.20, +12.24]+3.00% [+0.25, +5.75]
ShareGPT+1.50% [+0.25, +2.75]+3.85% [+1.84, +5.86]
MATH-500+3.44% [+0.40, +6.48]+2.47% [+0.78, +4.16]
MBPP+2.40% [−0.95, +5.75]+8.21% [+4.12, +12.30]

Dataset-specific means and 95% CIs of paired relative AL gains across five independent training seeds (TV and QSD matched by seed). These are not intervals for the six-dataset mean or TPS. Full AL: Figure 7, Tables 10–11.

On RTX 4090, QSD delivers 1.04–5.76% higher measured TPS than TV and 4.30–12.33% higher TPS than Range-only in all 60 sampling conditions (2 architectures × 6 datasets × B1/2/4/8/16). All W4A4 methods share the optimized backend. Figure 6 and Tables 4–8 report the end-to-end gains.

AuthorsOpenReview ↗

Additional Sampling Evidence and Clarification of FLASHLight’s Scope

Relevance to your questions: sampling validity, clarity, and practical performance. Following the earlier T=1 evaluation, we added an independent test of the method on FlatQuant W4A4 drafts, rather than only on our original SpinQuant-based draft baseline. The experiment follows a simple sequence: existing W4A4 PTQ → interface range calibration → verifier-guided QSD → matched AL/TPS evaluation.

We additionally evaluated FlatQuant W4A4 drafts with two fixed verifier configurations: FlatQuant W4A4 targets and the paper’s original SpinQuant W4A4 targets, on DFlash and DSpark. Within each target policy, the draft checkpoint, verifier, proposal setup, and evaluation conditions are matched between the FlatQuant baseline and FlatQuant + FLASHLight-QSD. Sampling AL@1024 covers six datasets and five training seeds; TPS@256 is measured end-to-end under T=1 sampling.

FlatQuant draft — T=1 sampling AL@1024

Target / DraftArchitecture16-bit draftAligned PTQ+ FLASHLight-QSDGain vs aligned
FlatQuant / FlatQuantDFlash2.8542.7772.883+3.82%
FlatQuant / FlatQuantDSpark4.7734.6664.726+1.29%
SpinQuant / FlatQuantDFlash2.0972.0432.129+4.21%
SpinQuant / FlatQuantDSpark2.1502.1222.205+3.91%

FlatQuant draft — T=1 sampling TPS@256 (tokens/s)

TargetArchB16-bitAligned+ FLASHLightGain
FlatQuantDFlash1275344357+3.78%
2506613635+3.59%
48771,0291,065+3.50%
81,4051,5621,598+2.30%
161,9352,0762,121+2.17%
FlatQuantDSpark1552684696+1.75%
2798922939+1.84%
41,4391,6471,673+1.58%
82,4082,6662,715+1.84%
163,6353,7963,848+1.37%
SpinQuantDFlash1232290301+3.79%
2425524543+3.63%
4737874903+3.32%
81,1541,3021,346+3.38%
161,5201,6881,715+1.60%
SpinQuantDSpark1359464474+2.16%
2521623637+2.25%
49791,1491,175+2.26%
81,6641,8971,942+2.37%
162,3332,5302,568+1.50%

Under standard T=1 sampling, adding FLASHLight improves AL over FlatQuant in all four architecture–target-policy comparisons (+1.29–4.21%). The AL gains are accompanied by higher end-to-end TPS in all 20 batch configurations (+1.37–3.79%). These are incremental benefits over FlatQuant itself, not merely over the original SpinQuant-based Aligned PTQ baseline.

EAGLE-3 — optimized backend, greedy end-to-end TPS@256 (tokens/s; SpinQuant W4A4 target)

DraftB1B2B4B8B16
16-bit draft3826871,1121,4761,631
W4A16 GPTQ (Marlin)4998561,3191,5841,663
FLASHLight-QSD W4A45128971,3841,6931,792
Gain vs W4A16+2.6%+4.8%+4.9%+6.9%+7.8%

EAGLE-3 now exceeds W4A16 GPTQ (Marlin) at all five batch sizes on the optimized backend (+2.6% at B1; +7.8% at B16). This extends the deployment measurements to tree-based drafting; these EAGLE-3 results use greedy decoding, separately from the FlatQuant sampling study.

What this adds to the revised manuscript. FLASHLight complements rather than replaces existing W4A4 PTQ methods. Its acceptance improvement persists under standard T=1 sampling with both FlatQuant and SpinQuant W4A4 verifiers. The updated EAGLE-3 measurements address the remaining backend-coverage limitation without conflating greedy and stochastic results.

Reviewer_VoBAOpenReview ↗

Response to Authors

Thank you for the detailed response and the revisions.

I think the extensive changes to wording make the paper easier to read and strengthen it overall. The stochastic sampling experiments are a nice inclusion, and the FlatQuant experiments are also a useful addition. Together, these changes make me more comfortable with my positive assessment. I will keep my score of 3, but have raised my Presentation score to 2.

Reviewer_ahWHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reduction in acceptance length caused by W4A4 quantization of draft models in speculative decoding. FLASHLight combines range calibration with target-guided rotation optimization using a prefix-coupled objective, while keeping pretrained weights frozen. Experiments evaluate acceptance length and end-to-end throughput under different settings.

Critical strengths
  1. Using the target model to guide draft quantization is well motivated. The objective focuses on recovering acceptance length rather than only matching the full-precision draft, making it relevant to speculative decoding performance.

  2. The paper evaluates end-to-end throughput across different draft architectures and batch sizes, rather than relying on acceptance length alone. The optimized GPTQ W4A16 baseline also provides a useful reference for assessing practical gains.

Critical weaknesses
  1. The results do not demonstrate a clear advantage of W4A4 over W4A16. Throughput gains over GPTQ W4A16 are approximately 2.1%–6.5% on DFlash. DSpark shows negligible gains at small batch sizes and gains of up to 7.6% at larger batches, while improvements on EAGLE-3 remain small. Memory savings relative to W4A16 are not reported either. These results leave the need for W4A4 draft quantization insufficiently justified.

  2. The rotation and calibration techniques largely follow established approaches, with the contribution mainly lying in their adaptation to target-guided draft quantization. While target-model supervision provides measurable benefits, the additional value of the prefix-coupled objective remains unclear. In Table 18, QSD improves mean acceptance length over position-wise TV only from 4.135 to 4.158. The results therefore provide limited evidence that prefix coupling improves upon a simpler objective using the same target-model supervision.

Questions for authors
  1. Why does the native INT4 backend yield only limited end-to-end gains over GPTQ W4A16, and when does W4A4 offer a clear practical advantage?

  2. Does the prefix-coupled objective provide a meaningful practical benefit over position-wise TV with the same target-model supervision?

3 comments
AuthorsOpenReview ↗

Explaining the revised W4A4 speedups and prefix-coupling benefits

We appreciate your questions about W4A4 deployment and prefix coupling.

Question 1: Why does the native INT4 backend yield only limited end-to-end gains over GPTQ W4A16, and when does W4A4 offer a clear practical advantage?

The original B1 results showed only about +2.1% for DFlash and a near tie for DSpark, with DSpark slightly below W4A16. Those measurements used our native INT4 backend before the additional split-K optimization. The revised B1 advantage reflects this implementation improvement, which leaves the quantization method unchanged.

A separate fixed-checkpoint, four-dataset DSpark control isolates this effect. Split-K raises QSD TPS from 697.0 to 781.8 at B1 and 4735.0 to 5341.0 at B16, with output equivalence verified. With the common target-side optimization, Marlin rises from 710.0 to 772.8 and 4439.0 to 4770.0. Thus, QSD moves from slightly slower than Marlin at B1 to faster under the same workload (Appendix H.3, Table 31).

B1 gains remain limited because W4A16 already reduces weight-read costs, while target verification and other pipeline work remain. The 16-bit DFlash draft accounts for 37.3% of B1 drafting-plus-verification time. Larger batches offer more scope for low-bit computation to help (Figure 2a; Appendix H.1).

The revised six-dataset evaluation shows the clearest advantage at B16: over W4A16 GPTQ, greedy TPS improves by 10.79–21.34% and sampling TPS by 8.96–28.36%. B1 gains are 1.27–7.03% and 3.26–7.39%, respectively, across DFlash and DSpark (Figure 6; Tables 5–8).

Sampling ablation and throughput

MethodDFlash ALDFlash TPS B1DFlash TPS B16DSpark ALDSpark TPS B1DSpark TPS B16
Aligned PTQ1.96280.21,633.01.94450.72,520.3
Range-only2.00288.81,664.72.04489.32,686.0
W4A16 GPTQ2.08293.81,496.22.16509.72,638.3
FLASHLight-TV2.05302.01,746.82.14514.82,836.8
FLASHLight-QSD2.11309.31,802.82.22531.22,939.2

Unweighted six-dataset means; AL: B1 AL@1024; TPS: tokens/s (Appendix B.2, Tables 6 and 8).

All W4A4 variants use the same split-K backend, and Marlin receives the common target-side optimization. QSD improves sampling TPS by 4.30–12.33% over Range-only under this shared backend; the W4A16 comparison evaluates complete deployment configurations.

The practical benefit here is primarily throughput. Both drafts store four-bit weights, and DFlash peak allocated memory differs by only 0.03–0.04 GiB across B1–B16:

DraftB1B2B4B8B16
W4A16 GPTQ15.2215.5216.1217.3419.76
FLASHLight-QSD15.1815.4816.0917.3119.73

Peak allocated memory in GiB (Appendix H.7, Table 33).

Question 2: Does the prefix-coupled objective provide a meaningful practical benefit over position-wise TV with the same target-model supervision?

To test the practical value beyond the original 4.135-to-4.158 AL result, we added matched T=1T=1 sampling and runtime ablations. TV and QSD share verifier supervision, calibration, trainable rotations, data, budget, and backend; pretrained weights remain fixed.

TV optimizes single-token sampling acceptance, 1−TV(p,q)1-\mathrm{TV}(p,q). QSD couples acceptance across prefixes, yielding mean relative AL gains of +2.85%/+3.89% for DFlash/DSpark at T=1T=1, versus +0.76%/+0.52% under greedy decoding. Tables 10–11 report both modes; the sampling CIs below use five training seeds.

Sampling QSD–TV: B1 AL@1024 gain (%) [95% CI; 5 training seeds]

DatasetDFlashDSpark
MT-Bench+1.44% [−0.85, +3.73]+2.59% [+0.58, +4.60]
GSM8K+2.81% [+0.42, +5.20]+3.24% [−0.50, +6.98]
HumanEval+5.52% [−1.20, +12.24]+3.00% [+0.25, +5.75]
ShareGPT+1.50% [+0.25, +2.75]+3.85% [+1.84, +5.86]
MATH-500+3.44% [+0.40, +6.48]+2.47% [+0.78, +4.16]
MBPP+2.40% [−0.95, +5.75]+8.21% [+4.12, +12.30]

Each 95% CI is computed across five independent training seeds, using TV–QSD pairs matched by seed. The intervals describe dataset-specific mean relative AL gains, not timing variability. Full TPS: Tables 4–8.

QSD achieves the highest measured TPS, including ties, across all 120 architecture–dataset–batch–mode conditions. In all 60 sampling conditions, it exceeds TV by 1.04–5.76%, demonstrating a practical benefit from prefix coupling under the same backend.

Thank you for the additional experiments and clarifications. I acknowledge the measured end-to-end speedups, but the gains over GPTQ W4A16 with Marlin remain dependent on the draft architecture and batch size. The reported EAGLE-3 results show little advantage, while DSpark achieves average gains of approximately 2–4% at batch size 1 and 11–12% at batch size 16. The EAGLE-3 results still use the earlier backend, so the updated performance evaluation does not cover all three architectures.

My main concern remains the limited comparison with existing W4A4 methods. The current W4A4 evaluation primarily compares against Aligned PTQ and variants of FLASHLight. Although the ablations demonstrate improvements over this baseline, they do not establish whether a stronger existing W4A4 method could achieve comparable acceptance length and throughput. A comparison with a representative peer-reviewed method, such as FlatQuant, under the same draft, target, and evaluation settings would help clarify the additional value of FLASHLight’s specific designs.

Given the setting-dependent deployment gains and the unresolved W4A4 baseline concern, I am maintaining my rating.

AuthorsOpenReview ↗

EAGLE-3 Optimized Backend and Matched FlatQuant W4A4 Comparisons

We thank the reviewer for identifying two concrete gaps: EAGLE-3 runtime coverage and comparison against a stronger W4A4 baseline. We address both with new end-to-end measurements and matched FlatQuant experiments. The expanded evaluation covers six datasets; AL is aggregated over five training seeds.

1. EAGLE-3: optimized backend and deployment benefit

Concern: “The EAGLE-3 results still use the earlier backend” and show “little advantage.” We remeasure EAGLE-3 using the optimized native INT4 backend, retaining the SpinQuant W4A4 target and comparing against W4A16 GPTQ (Marlin) under the same greedy TPS@256 protocol.

EAGLE-3 greedy end-to-end TPS@256 (tokens/s)

DraftB1B2B4B8B16
16-bit draft3826871,1121,4761,631
W4A16 GPTQ (Marlin)4998561,3191,5841,663
FLASHLight-QSD W4A45128971,3841,6931,792
Gain vs W4A16+2.6%+4.8%+4.9%+6.9%+7.8%

The updated EAGLE-3 results close the optimized-backend evaluation gap: FLASHLight outperforms W4A16 GPTQ (Marlin) at every tested batch size (+2.6% at B1 and +7.8% at B16). While gains remain batch-dependent, EAGLE-3 no longer exhibits the near-tie seen with the earlier backend.

2. Strong W4A4 baseline: FlatQuant under matched conditions

Concern: “A comparison with a representative peer-reviewed method, such as FlatQuant, under the same draft, target, and evaluation settings.” FLASHLight complements rather than replaces FlatQuant: we compare the same pretrained drafts and quantized verifiers before and after adding interface range calibration and verifier-guided QSD. We test two target policies: (i) FlatQuant W4A4 target + FlatQuant W4A4 draft; and (ii) the original SpinQuant W4A4 target + FlatQuant W4A4 draft. Within each policy, the verifier, datasets, proposal setup, and evaluation protocol are held fixed.

FlatQuant draft — T=1 sampling AL@1024

Target / DraftArchitecture16-bit draftAligned PTQ+ FLASHLight-QSDGain vs aligned
FlatQuant / FlatQuantDFlash2.8542.7772.883+3.82%
FlatQuant / FlatQuantDSpark4.7734.6664.726+1.29%
SpinQuant / FlatQuantDFlash2.0972.0432.129+4.21%
SpinQuant / FlatQuantDSpark2.1502.1222.205+3.91%

Adding FLASHLight to FlatQuant improves sampling AL in all four matched comparisons (+1.29–4.21%), including both architectures under the original SpinQuant W4A4 verifier. This tests incremental value beyond a peer-reviewed W4A4 PTQ method, rather than relying solely on the original SpinQuant-based Aligned PTQ baseline.

FlatQuant draft — T=1 sampling TPS@256 (tokens/s)

TargetArchB16-bitAligned+ FLASHLightGain
FlatQuantDFlash1275344357+3.78%
2506613635+3.59%
48771,0291,065+3.50%
81,4051,5621,598+2.30%
161,9352,0762,121+2.17%
FlatQuantDSpark1552684696+1.75%
2798922939+1.84%
41,4391,6471,673+1.58%
82,4082,6662,715+1.84%
163,6353,7963,848+1.37%
SpinQuantDFlash1232290301+3.79%
2425524543+3.63%
4737874903+3.32%
81,1541,3021,346+3.38%
161,5201,6881,715+1.60%
SpinQuantDSpark1359464474+2.16%
2521623637+2.25%
49791,1491,175+2.26%
81,6641,8971,942+2.37%
162,3332,5302,568+1.50%

The AL improvements translate into higher end-to-end sampling TPS in all 20 matched conditions: +1.37–3.78% with FlatQuant targets and +1.50–3.79% with SpinQuant targets. These results directly address the reviewer’s primary concern: FLASHLight delivers additional acceptance and throughput gains beyond FlatQuant under matched conditions, including with the original SpinQuant verifier.

Reviewer_LajeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FLASHLight, a post-training quantization method for W4A4 draft models used with a quantized verifier in speculative decoding. The authors observe that draft quantization causes larger losses in conditional acceptance at later proposal positions, reducing the likelihood of accepting long draft prefixes. FLASHLight first calibrates source-wise feature scaling and activation clipping at the target–draft interface. With these calibration parameters fixed, it then optimizes orthogonal draft rotations under supervision from the deployed verifier, using a prefix-aware objective that accounts for acceptance across successive positions. Experiments on DFlash and DSpark across dialogue, mathematical reasoning, and code generation benchmarks show consistent recovery of accepted length under greedy decoding, with additional EAGLE-3 results supporting applicability to another drafting architecture. The paper also reports end-to-end throughput, latency, and memory measurements on an RTX 4090 to evaluate the deployment tradeoffs of the quantized drafts.

Critical strengths
  1. Quantization disproportionately degrades conditional acceptance at later draft positions.

The paper shows that the conditional-acceptance penalty caused by draft quantization becomes substantially larger at later draft positions. Under greedy decoding with the same W4A4 target, the gap between the 16-bit draft and Aligned PTQ grows from 1.5 to 18.0 percentage points for DFlash and from 6.1 to 13.3 for DSpark (Figure 1). Conditional acceptance here is measured given that all preceding draft tokens have been accepted. The DFlash same-prefix probe further shows that greater late-position degradation persists when target features and prefixes are held fixed (Figure 2c), providing evidence that rollout prefix selection alone does not explain the observation. This finding is informative for low-bit speculative decoding: preserving early-position accuracy can still leave substantial degradation at later positions and reduce the chance of accepting long prefixes.

  1. Consistent acceptance recovery with a deployable low-bit construction.

FLASHLight improves average AL from 3.72 to 4.16 on DFlash and from 3.86 to 4.74 on DSpark, with improvements across all four evaluation datasets (Table 1). The EAGLE-3 results provide additional evidence that the overall procedure can recover acceptance in another drafting architecture. The transformations can be fused into the existing execution path, and the evaluation includes actual throughput, latency, and memory measurements. These measurements help assess whether the acceptance recovery translates into practical deployment benefits.

Critical weaknesses
  1. The main recovery comes from a simple calibration adjustment, leaving the overall methodological novelty insufficiently established.

Table 1 shows that Range-only already retains 94.1%/94.5% of the 16-bit reference AL and reaches 96.3%/97.2% of the full FLASHLight result on DFlash/DSpark. Table 17 makes the concentration still clearer: source-block scaling alone supplies 0.254/0.712 of the 0.284/0.750 AL that Range-only gains over Aligned PTQ, or about 89%/95%. In my view, source-block scaling is a straightforward source-wise rescaling operation with limited methodological novelty. With much of the recovery attributable to this simple calibration adjustment, a distinct and substantial contribution from the second stage becomes central to establishing the novelty of the overall method. The subsequent verifier-guided rotation has a positive but relatively small incremental effect of 0.154/0.133 AL. More importantly, the proposed QSD prefix-product objective exceeds the matched target-TV objective by only 0.023 AL, and its three-seed 95% t interval includes zero ([-0.009, 0.055]; Table 18). The paper therefore gives strong evidence for a useful source-wise range-calibration recipe, but does not yet establish that the specialized second-stage verifier-guided objective—particularly its prefix coupling—is a substantial or necessary independent contribution. The positive rotation gains across seeds are encouraging, but do not by themselves show that QSD is needed over conventional verifier-guided TV.

  1. The practical advantage over strong deployment alternatives remains narrowly established.

At batch size 1, DFlash improves from 432.8 tokens/s with W4A16 GPTQ to 442.1 with FLASHLight, approximately 2.1%; DSpark is essentially tied at 664.1 versus 663.6 tokens/s (Table 2). EAGLE-3 is also slightly slower than W4A16 at batch size 1. Some larger-batch settings show greater benefits, but the study uses one RTX 4090, greedy decoding, and two target checkpoints. Moreover, the runtime table does not include Range-only or the matched target-TV rotation variant. Their inclusion would reveal whether the full optimization procedure materially improves the deployment tradeoff over simpler choices. Including these comparisons would help clarify whether the full method offers meaningful deployment benefits over simpler alternatives.

Questions for authors

No further questions for the authors.

2 comments
AuthorsOpenReview ↗

Contribution beyond calibration and deployment benefits

Thank you for asking us to clarify the contribution beyond calibration and its practical impact.

Response to Weakness 1: Methodological novelty beyond calibration

Calibration accounts for much of the recovery, as you note. We therefore test both the value of verifier-guided rotation beyond Range-only and the value of prefix coupling beyond matched TV. Our contribution is low-bit basis optimization after interface calibration, with pretrained weights fixed. QSD builds on prior acceptance-aware prefix objectives. The TV–QSD comparison changes only the objective; verifier, calibration, trainable rotations, data, budget, and backend are identical.

The original +0.023 AL greedy result has a three-seed interval including zero; we retain it separately in Table 12. We expanded the matched evaluation to five training seeds and six datasets, adding T=1T=1 sampling. Here TV already optimizes single-token acceptance, 1−TV(p,q)1-\mathrm{TV}(p,q), while QSD couples these probabilities across prefixes. Mean relative AL gains over TV are +2.85%/+3.89% for DFlash/DSpark under sampling, versus +0.76%/+0.52% under greedy decoding.

Sampling QSD–TV: B1 AL@1024 gain (%) [95% CI; 5 training seeds]

DatasetDFlashDSpark
MT-Bench+1.44% [−0.85, +3.73]+2.59% [+0.58, +4.60]
GSM8K+2.81% [+0.42, +5.20]+3.24% [−0.50, +6.98]
HumanEval+5.52% [−1.20, +12.24]+3.00% [+0.25, +5.75]
ShareGPT+1.50% [+0.25, +2.75]+3.85% [+1.84, +5.86]
MATH-500+3.44% [+0.40, +6.48]+2.47% [+0.78, +4.16]
MBPP+2.40% [−0.95, +5.75]+8.21% [+4.12, +12.30]

Mean paired relative AL gains and dataset-specific 95% CIs across five independent training seeds, with TV and QSD paired by seed. These are not CIs for the six-dataset mean or TPS. Tables 10–11 report full greedy and sampling AL results and CIs.

The additional benefit of prefix coupling is clearest under sampling: its mean AL gain exceeds the greedy gain in every matched architecture–dataset pair. The runtime ablation below tests whether this improves deployment over both simpler alternatives.

Response to Weakness 2: Practical advantage over strong deployment alternatives

The original B1 results you cite showed only about +2.1% for DFlash and a near tie for DSpark. The revised runtime uses an additional split-K optimization of our native INT4 backend. In a separate fixed-checkpoint DSpark control, B1 TPS rises from 697.0 to 781.8, with output equivalence verified; Marlin receives the common target-side optimization and rises from 710.0 to 772.8 (Appendix H.3, Table 31). We also added the Range-only and matched-TV runtime baselines you requested. All W4A4 variants share the optimized backend: across 60 sampling conditions, QSD improves measured TPS by 4.30–12.33% over Range-only and 1.04–5.76% over TV.

Sampling ablation and W4A16 comparison

MethodDFlash ALDFlash TPS B1DFlash TPS B16DSpark ALDSpark TPS B1DSpark TPS B16
Aligned PTQ1.96280.21,633.01.94450.72,520.3
Range-only2.00288.81,664.72.04489.32,686.0
W4A16 GPTQ2.08293.81,496.22.16509.72,638.3
FLASHLight-TV2.05302.01,746.82.14514.82,836.8
FLASHLight-QSD2.11309.31,802.82.22531.22,939.2

Unweighted six-dataset means; AL: B1 AL@1024; TPS: tokens/s. W4A16 GPTQ uses Marlin (Appendix B.2, Tables 6 and 8).

With this updated implementation, W4A16 GPTQ comparisons show greedy TPS gains of 1.27–7.03% at B1 and 10.79–21.34% at B16; sampling gains are 3.26–7.39% and 8.96–28.36%, respectively (Section 4.3, Figure 6, Tables 5–8). The expanded evidence covers both decoding modes and six datasets on the same RTX 4090 and two target checkpoints; it broadens workload coverage without establishing hardware or model-scale generalization.

AuthorsOpenReview ↗

Additional Evidence Beyond Calibration: FlatQuant and EAGLE-3

Relevance to your concern: contribution beyond calibration and existing quantization methods. Our earlier response separated Range-only, TV, and QSD under matched conditions. The new tests address a different question: does verifier-aware optimization still help when the starting W4A4 draft is a stronger, peer-reviewed PTQ method?

We additionally evaluated FlatQuant W4A4 drafts with two fixed verifier configurations: FlatQuant W4A4 targets and the paper’s original SpinQuant W4A4 targets, on DFlash and DSpark. Within each target policy, the draft checkpoint, verifier, proposal setup, and evaluation conditions are matched between the FlatQuant baseline and FlatQuant + FLASHLight-QSD. Sampling AL@1024 covers six datasets and five training seeds; TPS@256 is measured end-to-end under T=1 sampling.

FlatQuant draft — T=1 sampling AL@1024

Target / DraftArchitecture16-bit draftAligned PTQ+ FLASHLight-QSDGain vs aligned
FlatQuant / FlatQuantDFlash2.8542.7772.883+3.82%
FlatQuant / FlatQuantDSpark4.7734.6664.726+1.29%
SpinQuant / FlatQuantDFlash2.0972.0432.129+4.21%
SpinQuant / FlatQuantDSpark2.1502.1222.205+3.91%

FlatQuant draft — T=1 sampling TPS@256 (tokens/s)

TargetArchB16-bitAligned+ FLASHLightGain
FlatQuantDFlash1275344357+3.78%
2506613635+3.59%
48771,0291,065+3.50%
81,4051,5621,598+2.30%
161,9352,0762,121+2.17%
FlatQuantDSpark1552684696+1.75%
2798922939+1.84%
41,4391,6471,673+1.58%
82,4082,6662,715+1.84%
163,6353,7963,848+1.37%
SpinQuantDFlash1232290301+3.79%
2425524543+3.63%
4737874903+3.32%
81,1541,3021,346+3.38%
161,5201,6881,715+1.60%
SpinQuantDSpark1359464474+2.16%
2521623637+2.25%
49791,1491,175+2.26%
81,6641,8971,942+2.37%
162,3332,5302,568+1.50%

Across all four architecture–target-policy pairs, FLASHLight improves sampling AL over FlatQuant by +1.29–4.21%. The gains translate into higher end-to-end TPS in all 20 batch configurations (+1.37–3.79%). These are incremental benefits over FlatQuant itself, not merely over the original SpinQuant-based Aligned PTQ baseline.

EAGLE-3 — optimized backend, greedy end-to-end TPS@256 (tokens/s; SpinQuant W4A4 target)

DraftB1B2B4B8B16
16-bit draft3826871,1121,4761,631
W4A16 GPTQ (Marlin)4998561,3191,5841,663
FLASHLight-QSD W4A45128971,3841,6931,792
Gain vs W4A16+2.6%+4.8%+4.9%+6.9%+7.8%

With the optimized native INT4 backend, FLASHLight exceeds W4A16 GPTQ (Marlin) at every tested batch size, from +2.6% at B1 to +7.8% at B16. This addresses the earlier EAGLE-3 deployment limitation; gains remain batch-dependent.

FLASHLight complements rather than replaces FlatQuant. Starting from FlatQuant-quantized drafts, our interface calibration and verifier-guided QSD optimization consistently increase both accepted length and end-to-end throughput under matched deployment conditions. The improvements hold for both DFlash and DSpark and for both FlatQuant and the original SpinQuant verifiers. This demonstrates FLASHLight’s additional contribution on top of an established, strong W4A4 PTQ method. Our earlier matched TV–QSD comparison separately isolates the contribution of prefix coupling.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Rebuttal Summary: Contributions and Responses to Reviewer Concerns

We thank all three reviewers for their constructive feedback. During rebuttal, we responded to each major concern with additional experiments, controlled comparisons, and clarifications in the revised manuscript, including the follow-up requests.

Main Contributions

FLASHLight makes three contributions to low-bit speculative decoding:

  1. Problem identification and diagnosis: We identify quantization-induced suffix decay, where the conditional-acceptance gap between W4A4 and 16-bit drafts widens toward later proposal positions. We further identify the target-to-draft feature interface as a major bottleneck: features from different target layers have mismatched ranges but share a single per-token 4-bit activation scale.

  2. Verifier-aware quantization methodology: We introduce a quantization procedure that directly targets accepted-prefix preservation under the deployed W4A4 verifier, rather than solely minimizing reconstruction error. FLASHLight combines interface-specific range calibration, independent verifier-guided draft-basis optimization, and a prefix-coupled QSD objective. Pretrained weights remain frozen, and the transformations are fused into the inference path.

  3. Practical deployment: For DFlash and DSpark, FLASHLight reaches 99.0–100.2% of the six-dataset mean greedy-decoding acceptance length of the 16-bit draft paired with the same W4A4 verifier, while enabling measurable end-to-end throughput improvements through efficient W4A4 draft execution.

Responses to Reviewer Concerns

  • Laje — Methodological novelty and deployment: We added matched Range-only/TV/QSD ablations, FlatQuant comparisons, and optimized runtime measurements. Under T=1 sampling, FLASHLight improves mean relative AL over Range-only by 5.66% (DFlash) and 8.65% (DSpark). With matched calibration and verifier guidance, QSD further improves mean relative AL over TV by 2.85% and 3.89%, respectively.

  • ahWH — W4A4 benefits, prefix coupling, and stronger baselines: We completed the requested FlatQuant comparisons and optimized EAGLE-3 evaluation. Adding FLASHLight to FlatQuant improves T=1 sampling AL by 1.29–4.21% across four matched configurations and end-to-end TPS by 1.37–3.79% across 20 batch settings. Against W4A16 GPTQ (Marlin), sampling TPS improves by 3.26–7.39% at B1 and 8.96–28.36% at B16 on DFlash/DSpark. Optimized EAGLE-3 achieves 2.6–7.8% higher greedy TPS than W4A16 GPTQ across B1–B16. We also clarified the limited memory savings over W4A16.

  • VoBA — Presentation, novelty, and sampling validity: We clarified the motivation, terminology, method, and contribution framing, and added standard T=1 stochastic sampling evaluations across six datasets, together with five-training-seed robustness comparisons.

These additional experiments directly address the reviewers' main methodological and practical concerns. Together, they strengthen the evidence for FLASHLight's central contribution: identifying a quantization bottleneck in feature-sharing speculative decoding and addressing it through verifier-aware low-bit draft quantization that preserves accepted prefixes while improving deployment efficiency.

We appreciate the reviewers' time and consideration of our revised manuscript and additional evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.