- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates speculative decoding for on-demand audio understanding tasks on mobile devices. In this scenario, audio data is prefilled before the user issues a request, meaning that response latency is primarily determined by the decoding phase. The study finds that the serial draft generation accounts for a significant portion of the decoding process. However, for source-conditioned tasks such as ASR, a draft model can predict valid candidate tokens from the audio input alone, without needing to rely on the verified output prefix from the target model. To address this, the authors propose AS²D. In this method, an audio-based draft model independently generates candidate tokens and stores them in a buffer, while the target model simultaneously validates the existing candidate tokens within that buffer. When no candidate tokens are available, the target model decodes independently. If a candidate token is rejected, the system adjusts the contents of the buffer accordingly. Experiments on four Android phones show that AS²D increases ASR throughput by 42%–76% compared to target-only decoding.
Critical strengths
-
AS²D achieves consistent improvements on a real-world deployment bottleneck. On mobile phones, serial speculative decoding often fails to outperform target-only decoding, while AS²D is consistently faster than target-only decoding across four different phones and five ASR datasets, without requiring any training.
-
This paper offers a valuable insight and presents it transparently. A high acceptance rate alone does not guarantee speedup. The draft model must generate candidates before the target model requires them, and these accepted candidates must cover a substantial portion of the final output.
Critical weaknesses
-
The main results rely heavily on replay, while validation on actual devices is limited. The main comparisons on the full test set use phone-cost replay of server-generated traces, and the accuracy of this replay has only been verified to a limited extent through actual runs on a single phone (Redmi K70 Pro). One native request also shows a 45-word difference from the target-only output (Appendix D.6.1), so more native evidence would strengthen the output-preservation claim.
-
The positioning of this study relative to existing work could be further clarified. The proposed mechanism is closely related to reference-based lossless decoding such as LLMA and prompt lookup decoding. The authors may also incorporate some relevant baselines, such as asynchronous prefix-conditioned SD. This would help clarify the benefit of target-decoupled drafting.
Questions for authors
-
Can the authors provide native results (rather than replayed) on a subset of the ASR datasets beyond LibriSpeech, such as AMI, where the gains are smallest? This would help confirm whether the performance gains observed in replay also hold true for real-world device.
-
Theoretically, greedy verification should be able to reproduce the target-only output. Can the authors explain why there is a 45 words difference between AS²D's output and the target-only output at 210s in the native tests (Appendix D.6.1)?
5 comments
Response to Reviewer 3wksX (2/3): the 45-word output difference
W1 and Q2: why can greedy verification produce a different output? Single-token decoding and multi-token verification can follow different numerical execution paths and produce different target scores [2, 3, 4]. Greedy generation is sensitive to these differences: changing one selected token changes the prefix for subsequent predictions. Our controlled check at 210 s reproduced the first mismatch with identical audio and prefix: single-token decoding selected in, whereas multi-token verification selected impression. The 45-word difference mainly reflects additional continuation. The equivalence guarantee assumes verification equivalent to sequential target evaluation and correct state restoration (Proposition 1, App. C.4) [1].
The new Qwen3-ASR native evaluation gives a broader view of recognition accuracy (§5.4, Tab. 6, pp. 9–10). Across its 150 inputs and five repetitions, the difference in mean corpus WER between AS²D and target-only ranges from −0.15 to +0.05 percentage points across the five benchmarks. Per-input WER agrees in 724/750 paired measurements (96.5%), showing a small overall accuracy difference in this evaluation.
App. D.6.1 (p. 39) now includes the diagnosis and reconstructed-reference WER for the original 210 s case (target-only: 0.65%; AS²D: 29.87%). App. F.6 (pp. 44–45) expands the numerical explanation. Ensuring consistency across execution paths is a backend problem studied by LLM-42 [5] and Vosti [6]; a general mobile-backend solution is beyond this work.
References
[1] Chen et al. Accelerating Large Language Model Decoding with Speculative Sampling, 2023.
[2] Horace He and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference, 2025.
[3] Zhang et al. Deterministic Inference across Tensor Parallel Sizes That Eliminates Training–Inference Mismatch, 2026.
[4] vLLM contributors. Lossless Guarantees of Speculative Decoding, version 0.32.0, 2026.
[5] Gond et al. LLM-42: Enabling Determinism in LLM Inference with Verified Speculation, 2026.
[6] Qin et al. Vosti: Specifying, Implementing, and Verifying Deterministic LLM Inference, 2026.
Response to Reviewer wksX (3/3): relation to reference-based and asynchronous decoding
W2: how does AS²D differ from LLMA, prompt lookup, and asynchronous decoding? The revision clarifies how candidates are supplied (§2, pp. 3–4). LLMA copies continuations from available reference documents [1], while prompt lookup finds reusable spans in the prompt and generated text [2]. AS²D shares their target-verification principle, but generates candidates online from audio and the instruction through a separate producer. The target consumes ready blocks while the producer continues independently of target-prefix updates. To our knowledge, this target-decoupled form of asynchronous speculative decoding for audio LLMs has not been explored on phones.
We also added LLMA and prompt lookup comparisons on the original 688 test windows, using the same target outputs and Redmi K70 Pro cost profiles (§5.2, p. 7; App. D.4.6, Tab. 16, p. 36). Our LLMA implementation first completes the audio-conditioned draft, then copies and verifies continuations from it, charging the full drafting cost. It matches suffixes of up to three tokens and proposes up to seven candidates. Prompt lookup uses the same maximum match length and copies ten-token continuations from the prompt and generated history.
The table below repeats Tab. 16. Entries are post-prefill decoding TPS from phone-cost replay, pooling committed target tokens and decoding time; Overall includes all 688 windows.
| Method | LibriSpeech | AMI SDM | FLEURS | KeSpeech | M4Singer | Overall |
|---|---|---|---|---|---|---|
| LLMA | 20.93 | 13.25 | 19.95 | 17.96 | 16.50 | 18.14 |
| Prompt lookup | 10.51 | 10.22 | 12.65 | 12.35 | 12.44 | 11.46 |
| AS²D | 35.37 | 18.60 | 28.22 | 27.57 | 25.59 | 27.92 |
AS²D achieves 1.54× the overall TPS of LLMA and 2.44× that of prompt lookup. The sequential LLMA comparison supports the benefit of generating and verifying candidates concurrently under this audio setup. Prompt lookup accepts only 1.92% of proposed candidates, limiting the benefit of reusing text.
We hope the additional native evidence and revised explanations address the reviewer’s concerns, and would be grateful if the reviewer could reconsider the rating. We would be happy to discuss any further questions.
References
[1] Yang et al. Inference with Reference: Lossless Acceleration of Large Language Models, 2023.
[2] Saxena. Prompt Lookup Decoding, 2023.
Response to Reviewer wksX (1/3): native evaluation beyond LibriSpeech
We thank the reviewer for the careful review. We have expanded the native evaluation, investigated the output discrepancy, and added comparisons and clarifications to the revised manuscript (blue text).
W1 and Q1: do the replay gains hold on real devices beyond LibriSpeech? We added repeated native measurements on 150 selected inputs, 30 from each of the five ASR benchmarks (§5.4, Tab. 6, pp. 9–10). FLEURS pools six languages with five inputs each. All runs use a Redmi K70 Pro, a Qwen3-ASR 1.7B OpenCL target, a 0.6B CPU drafter, greedy decoding, and a draft budget of seven. Each input receives one warmup and five measured runs per method.
The table below repeats the relevant rows of Tab. 6. TPS and paired TPS ratios are five-run means ± sample standard deviation; WER is five-run mean corpus WER against human reference transcripts, using identical scoring rules.
AS²D has higher pooled TPS in all five repetitions of every benchmark. AMI, where the gain is smallest, retains a mean speedup of 1.32×, ranging from 1.26× to 1.38× across runs. Its WER is 37.42%, close to target-only’s 37.57% and below drafter-only’s 43.51%. The added native results therefore confirm decoding gains on these benchmark subsets, including AMI.