acceptodds
Under review as a conference paper at ICLR 2027

AS²D: Accelerating On-Demand Audio Understanding on Mobile Devices

Abstract

Speculative decoding accelerates autoregressive generation by using a smaller drafter to propose tokens for batched verification by a larger target. However, conventional speculative decoding couples drafting to the target's evolving verified prefix, serializing drafting and verification. We ask whether this dependency is necessary for source-conditioned generation. Our key observation is that, for audio language models, the input audio and user request can provide useful speculative candidates without following the target's evolving text prefix. Based on this observation, we propose AS²D (Audio Speculative Speculative Decoding), which enables target-decoupled drafting: an audio-conditioned drafter follows its own generation history while the target independently verifies and corrects ready candidates. Without usable candidates, the target advances alone. Thus, target feedback determines which candidates are committed but no longer determines when the drafter can make progress, enabling drafting and verification to proceed concurrently while retaining target-side verification and correction. We implement AS²D in MNN for Android and evaluate two target models across four phones, seven datasets, and three tasks covering 12.2 hours of audio. Across four phones, AS²D improves pooled ASR throughput by 42–76% over target-only decoding, while only 5.7% of evaluation windows are slower than target-only, compared with 58.1–63.0% for speculative baselines. For ASR, AS²D reaches 97.33–98.20% of a hindsight per-window oracle's pooled throughput over the evaluated drafter/budget catalog. Native on-demand execution with a 7B target achieves up to 78% higher throughput than target-only. These results show that source-conditioned audio generation can relax the conventional dependence of speculative drafting on the target's evolving output prefix, exposing substantial parallelism for efficient inference.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Speculative decoding, Audio language models, On-device inference, Asynchronous decoding, On-demand audio understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wksXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates speculative decoding for on-demand audio understanding tasks on mobile devices. In this scenario, audio data is prefilled before the user issues a request, meaning that response latency is primarily determined by the decoding phase. The study finds that the serial draft generation accounts for a significant portion of the decoding process. However, for source-conditioned tasks such as ASR, a draft model can predict valid candidate tokens from the audio input alone, without needing to rely on the verified output prefix from the target model. To address this, the authors propose AS²D. In this method, an audio-based draft model independently generates candidate tokens and stores them in a buffer, while the target model simultaneously validates the existing candidate tokens within that buffer. When no candidate tokens are available, the target model decodes independently. If a candidate token is rejected, the system adjusts the contents of the buffer accordingly. Experiments on four Android phones show that AS²D increases ASR throughput by 42%–76% compared to target-only decoding.

Critical strengths
  1. AS²D achieves consistent improvements on a real-world deployment bottleneck. On mobile phones, serial speculative decoding often fails to outperform target-only decoding, while AS²D is consistently faster than target-only decoding across four different phones and five ASR datasets, without requiring any training.

  2. This paper offers a valuable insight and presents it transparently. A high acceptance rate alone does not guarantee speedup. The draft model must generate candidates before the target model requires them, and these accepted candidates must cover a substantial portion of the final output.

Critical weaknesses
  1. The main results rely heavily on replay, while validation on actual devices is limited. The main comparisons on the full test set use phone-cost replay of server-generated traces, and the accuracy of this replay has only been verified to a limited extent through actual runs on a single phone (Redmi K70 Pro). One native request also shows a 45-word difference from the target-only output (Appendix D.6.1), so more native evidence would strengthen the output-preservation claim.

  2. The positioning of this study relative to existing work could be further clarified. The proposed mechanism is closely related to reference-based lossless decoding such as LLMA and prompt lookup decoding. The authors may also incorporate some relevant baselines, such as asynchronous prefix-conditioned SD. This would help clarify the benefit of target-decoupled drafting.

Questions for authors
  1. Can the authors provide native results (rather than replayed) on a subset of the ASR datasets beyond LibriSpeech, such as AMI, where the gains are smallest? This would help confirm whether the performance gains observed in replay also hold true for real-world device.

  2. Theoretically, greedy verification should be able to reproduce the target-only output. Can the authors explain why there is a 45 words difference between AS²D's output and the target-only output at 210s in the native tests (Appendix D.6.1)?

5 comments
AuthorsOpenReview ↗

Response to Reviewer wksX (1/3): native evaluation beyond LibriSpeech

We thank the reviewer for the careful review. We have expanded the native evaluation, investigated the output discrepancy, and added comparisons and clarifications to the revised manuscript (blue text).

W1 and Q1: do the replay gains hold on real devices beyond LibriSpeech? We added repeated native measurements on 150 selected inputs, 30 from each of the five ASR benchmarks (§5.4, Tab. 6, pp. 9–10). FLEURS pools six languages with five inputs each. All runs use a Redmi K70 Pro, a Qwen3-ASR 1.7B OpenCL target, a 0.6B CPU drafter, greedy decoding, and a draft budget of seven. Each input receives one warmup and five measured runs per method.

The table below repeats the relevant rows of Tab. 6. TPS and paired TPS ratios are five-run means ± sample standard deviation; WER is five-run mean corpus WER against human reference transcripts, using identical scoring rules.

BenchmarkLibriSpeechAMI SDMFLEURSKeSpeechM4Singer
Target-only TPS16.49 ± 0.4915.86 ± 0.5516.77 ± 0.1216.94 ± 0.8716.65 ± 1.32
AS²D TPS30.66 ± 0.9820.92 ± 0.6223.42 ± 0.6623.16 ± 0.3622.37 ± 0.58
AS²D/target-only TPS ratio1.86 ± 0.081.32 ± 0.051.40 ± 0.041.37 ± 0.081.35 ± 0.12
Drafter-only WER (%)6.6343.5163.28117.9628.29
Target-only WER (%)3.3637.5724.7825.9513.60
AS²D WER (%)3.3637.4224.8325.9513.60

AS²D has higher pooled TPS in all five repetitions of every benchmark. AMI, where the gain is smallest, retains a mean speedup of 1.32×, ranging from 1.26× to 1.38× across runs. Its WER is 37.42%, close to target-only’s 37.57% and below drafter-only’s 43.51%. The added native results therefore confirm decoding gains on these benchmark subsets, including AMI.

AuthorsOpenReview ↗

Response to Reviewer 3wksX (2/3): the 45-word output difference

W1 and Q2: why can greedy verification produce a different output? Single-token decoding and multi-token verification can follow different numerical execution paths and produce different target scores [2, 3, 4]. Greedy generation is sensitive to these differences: changing one selected token changes the prefix for subsequent predictions. Our controlled check at 210 s reproduced the first mismatch with identical audio and prefix: single-token decoding selected in, whereas multi-token verification selected impression. The 45-word difference mainly reflects additional continuation. The equivalence guarantee assumes verification equivalent to sequential target evaluation and correct state restoration (Proposition 1, App. C.4) [1].

The new Qwen3-ASR native evaluation gives a broader view of recognition accuracy (§5.4, Tab. 6, pp. 9–10). Across its 150 inputs and five repetitions, the difference in mean corpus WER between AS²D and target-only ranges from −0.15 to +0.05 percentage points across the five benchmarks. Per-input WER agrees in 724/750 paired measurements (96.5%), showing a small overall accuracy difference in this evaluation.

App. D.6.1 (p. 39) now includes the diagnosis and reconstructed-reference WER for the original 210 s case (target-only: 0.65%; AS²D: 29.87%). App. F.6 (pp. 44–45) expands the numerical explanation. Ensuring consistency across execution paths is a backend problem studied by LLM-42 [5] and Vosti [6]; a general mobile-backend solution is beyond this work.

References

[1] Chen et al. Accelerating Large Language Model Decoding with Speculative Sampling, 2023.

[2] Horace He and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference, 2025.

[3] Zhang et al. Deterministic Inference across Tensor Parallel Sizes That Eliminates Training–Inference Mismatch, 2026.

[4] vLLM contributors. Lossless Guarantees of Speculative Decoding, version 0.32.0, 2026.

[5] Gond et al. LLM-42: Enabling Determinism in LLM Inference with Verified Speculation, 2026.

[6] Qin et al. Vosti: Specifying, Implementing, and Verifying Deterministic LLM Inference, 2026.

AuthorsOpenReview ↗

Response to Reviewer wksX (3/3): relation to reference-based and asynchronous decoding

W2: how does AS²D differ from LLMA, prompt lookup, and asynchronous decoding? The revision clarifies how candidates are supplied (§2, pp. 3–4). LLMA copies continuations from available reference documents [1], while prompt lookup finds reusable spans in the prompt and generated text [2]. AS²D shares their target-verification principle, but generates candidates online from audio and the instruction through a separate producer. The target consumes ready blocks while the producer continues independently of target-prefix updates. To our knowledge, this target-decoupled form of asynchronous speculative decoding for audio LLMs has not been explored on phones.

We also added LLMA and prompt lookup comparisons on the original 688 test windows, using the same target outputs and Redmi K70 Pro cost profiles (§5.2, p. 7; App. D.4.6, Tab. 16, p. 36). Our LLMA implementation first completes the audio-conditioned draft, then copies and verifies continuations from it, charging the full drafting cost. It matches suffixes of up to three tokens and proposes up to seven candidates. Prompt lookup uses the same maximum match length and copies ten-token continuations from the prompt and generated history.

The table below repeats Tab. 16. Entries are post-prefill decoding TPS from phone-cost replay, pooling committed target tokens and decoding time; Overall includes all 688 windows.

MethodLibriSpeechAMI SDMFLEURSKeSpeechM4SingerOverall
LLMA20.9313.2519.9517.9616.5018.14
Prompt lookup10.5110.2212.6512.3512.4411.46
AS²D35.3718.6028.2227.5725.5927.92

AS²D achieves 1.54× the overall TPS of LLMA and 2.44× that of prompt lookup. The sequential LLMA comparison supports the benefit of generating and verifying candidates concurrently under this audio setup. Prompt lookup accepts only 1.92% of proposed candidates, limiting the benefit of reusing text.

We hope the additional native evidence and revised explanations address the reviewer’s concerns, and would be grateful if the reviewer could reconsider the rating. We would be happy to discuss any further questions.

References

[1] Yang et al. Inference with Reference: Lossless Acceleration of Large Language Models, 2023.

[2] Saxena. Prompt Lookup Decoding, 2023.

Reviewer_EwuXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a novel speculative decoding method for ASR tasks by decoupling the drafting process with target's prefix. Specifically, instead of sequential, interleaved drafting-verification loop, they propose a two stream pipeline that allows the drafter to run continuously in parallel to the verification and correction step, offering greater parallelism and fast speed for ASR applications. Results show that such proposed method accelerates ASR on different mobile devices.

Critical strengths
  1. The proposal work introduce a novel paradigm by challenging the convention design of SD with sequential draft-verify loop and show that for ASR tasks, they can be decoupled.
  2. The proposed framework achieve considerable speedup across different devices
Critical weaknesses
  1. Table 1's result should be contextualized in terms of concurrency/batch size/serving model. Specifically, it is not clear what is the batch size/concurrency for the observed speed up. Are the audio input streamed to the system in real time playback as discussed in Appendix D.6.2? Or are they processed offline with all requests available from the start. If latter is the case, how many requests are processed in parallel?
  2. while TPS is a good measurement additionally report end-to-end latency would provide more wholistic picture of the system-level performance accounting various overhead
Questions for authors

See weakness

  1. Please specify the exact setup, particular batch size/concurrency. Specifically, please discuss the intended use case. I image the decoupling only makes sense for no concurrency, batch size 1, single audio setting? With batch size greater than one, we have intra-batch parallelism and operation may become compute bound with little underutilized FLOPs capacity.
4 comments
AuthorsOpenReview ↗

Response to Reviewer EwuX (1/2): serving model, batch size, and concurrency

We thank the reviewer for the careful review. We have clarified the serving model, specified the batch size and concurrency, and added a native end-to-end latency example to the revised manuscript (blue text).

W1 and Q1: what are the intended use case, batch size, and concurrency? AS²D targets a personal phone serving a single user, with one locally captured audio stream and one foreground request at a time (§3, p. 4). Unlike cloud or shared edge servers, this setting does not aggregate independent requests from multiple users. Our objective is to shorten the individual user’s waiting time without waiting to form a request batch.

Table 1 uses request batch size 1 and inter-request concurrency 1. It reports Xiaomi 14 phone-cost replay with a Qwen3-ASR-1.7B OpenCL target and a 0.6B drafter on four CPU threads. Audio windows are processed offline, one request at a time, with each complete window available before decoding. Initial audio/prompt prefill is excluded from the reported decoding time. AS²D overlaps drafting and verification within that request; its proposal budget of seven permits up to eight token positions per verification, not eight requests. The revision specifies these settings in §5.1 (pp. 6–7) and the Table 1 caption (p. 8). The real-time input experiment is separate: App. D.6.2 feeds a 600-second recording as 150 four-second blocks.

We agree that larger request batches can improve target utilization and reduce the additional benefit of overlap. Our measurements concern the single-user setting and do not establish gains for larger batches. The revision now discusses this limitation in §4.4 (p. 6).

AuthorsOpenReview ↗

Response to Reviewer EwuX (2/2): end-to-end latency

Thank you for suggesting that we complement decoding TPS with end-to-end latency. We have clarified the timing boundary and included a native latency example in the revision.

W2: how does the decoding speedup translate into end-to-end latency? Our on-demand system model prepares received audio before the user requests a response. While listening, the phone can encode and prefill the audio, retaining the resulting model state. When a query arrives, the system reuses that state, processes the instruction, and generates the response. End-to-end latency runs from query arrival to completed output: preparation completed beforehand is outside this clock, while instruction prefill and any unfinished preparation are included. The revision clarifies this timing boundary in §3 (p. 4) and §5.1 (p. 7).

Because latency also depends on response length, we report a native example in which both methods produce the same output (§5.4, p. 10; App. D.6.1, p. 39). At the 120-second checkpoint on Redmi K70 Pro, using an Omni-7B target and a 0.6B drafter, both methods generate the same 173 body tokens. End-to-end latency decreases from 25.05 s to 14.07 s, a 1.78× speedup. The AS²D latency includes 0.44 s for the seven-token instruction prefill, approximately 3% of its total. Thus, in this matched-output example, the decoding acceleration translates into a substantial reduction in the user’s waiting time even after including instruction prefill.

We hope these clarifications and the added latency evidence address your concerns.

Thanks the authors for the response, my concerns have been addressed.

Thank you for your helpful feedback and for taking the time to follow up.

Reviewer_mA4zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AS2D, a speculative decoding framework for on-device audio language models. In contrast to conventional speculative decoding, where the drafter repeatedly conditions on the target model's accepted prefix, AS2D allows an audio-conditioned drafter to continue autoregressively using its own generation history. The target concurrently verifies available candidates, corrects rejected candidates, and falls back to target-only decoding when no usable candidate is available. The method additionally profiles device-specific execution costs to select a fixed speculative budget. The method is evaluated with Qwen3-ASR-1.7B and Qwen2.5-Omni-7B across four Android phones, seven datasets, and three tasks: ASR, speech translation, and spoken QA. The paper reports 42–76% higher pooled ASR throughput (TPS) than target-only decoding in phone-profile replay, as well as up to 78% higher TPS in native on-device experiments.

Critical strengths

The paper implements AS2D on Android devices and evaluates it across multiple phones, models, datasets, and audio tasks. The reported throughput improvements are substantial and demonstrate the potential practical value of the approach.

The paper provides extensive implementation details, device-specific profiling, ablation studies, and comparisons between replay-based and native execution. The authors also document cases where native execution produces different outputs from target-only decoding.

Critical weaknesses

The paper focuses on TPS without reporting WER. Although the method aims to preserve target outputs, native experiments show discrepancies, including a case with 45 word edits compared to target-only decoding. Without ground-truth WER, it is unclear whether these differences affect recognition accuracy. Additionally, the paper does not compare WER and TPS against the smaller drafter-only model, making it difficult to assess whether target verification provides enough accuracy improvement to justify its additional computational cost.

Most of the reported throughput improvements are based on phone-profile replay rather than native on-device execution. The native on-demand experiments cover a limited set of audio checkpoints, with only one formal execution per checkpoint and no run-to-run variability reported. Since the proposed speedups depend on concurrent execution and device scheduling, the replay results alone are insufficient to establish how consistently these improvements hold across real on-device workloads.

Questions for authors

1. Can the authors report ground-truth WER for AS2D and target-only decoding? In particular, how do the output differences observed in native execution affect recognition accuracy?

2. Can the authors report WER and TPS for the drafter-only model alongside target-only decoding and AS2D? This would help establish whether the accuracy improvement from target verification justifies its additional computational cost.

3. Can the authors provide additional native on-device evaluations across the tested ASR datasets, including repeated measurements? This would help establish whether the large throughput improvements observed in replay hold consistently under real device execution.

5 comments
AuthorsOpenReview ↗

Response to Reviewer mA4z (1/3): recognition accuracy

We thank the reviewer for the careful review and constructive feedback. The corresponding revisions are highlighted in blue in the updated manuscript.

W1 and Q1: ground-truth WER and native output differences. AS²D preserves target outputs under the execution assumptions of Proposition 1 (App. C.4): fixed inputs and decoding policies, verification equivalent to sequential target evaluation, and correct restoration of the accepted-prefix state. As with exact speculative sampling [1], the guarantee depends on these conditions. Native multi-token verification and single-token decoding can take different numerical paths, allowing small score perturbations to change greedy decisions [2, 3]. Such effects are also documented for vLLM [4, 5] and motivate backend determinism work such as LLM-42 and Vosti [6, 7]. In the original 45-word case (7B target, 210 s), a controlled check with identical audio and prefix reproduced the first mismatch: single-token decoding selected “in”, while multi-token verification selected “impression”. Subsequent predictions then condition on different prefixes. The revision adds this diagnosis in App. D.6.1 (p. 39) and expands the numerical explanation in App. F.6 (pp. 44–45). Backend-wide determinism is beyond this work; we assess native recognition accuracy empirically below.

WER stays close to target-only in the new native evaluation. To cover all five ASR benchmarks within the device-time budget, we evaluated 150 selected inputs, 30 per benchmark; FLEURS pools six languages with five inputs each. All three methods use a Redmi K70 Pro, a Qwen3-ASR 1.7B OpenCL target and a 0.6B CPU drafter, greedy decoding, and an AS²D draft budget of 7. Each input received one warmup and five measured runs per method. We score all outputs against human reference transcripts with identical normalization and language-specific word segmentation [8–10]. Per-input WER is identical in 724/750 paired measurements (96.5%); the largest increase is 3.33 percentage points, corresponding to three additional word errors. The revision adds this evaluation in §5.4 and Table 6 (pp. 9–10). The table below repeats its target-only and AS²D WER rows, each a five-run mean corpus WER.

Table R1.1. Native WER for target-only decoding and AS²D.

BenchmarkLibriSpeechAMI SDMFLEURSKeSpeechM4Singer
Target-only WER (%)3.3637.5724.7825.9513.60
AS²D WER (%)3.3637.4224.8325.9513.60

References

[1] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.

[2] Horace He and Thinking Machines Lab. Defeating nondeterminism in LLM inference. Thinking Machines Lab: Connectionism, 2025.

[3] Ziyang Zhang, Xinheng Ding, Jiayi Yuan, Rixin Liu, Huizi Mao, Jiarong Xing, and Zirui Liu. Deterministic inference across tensor parallel sizes that eliminates training–inference mismatch. arXiv preprint arXiv:2511.17826, version 2, 2026.

[4] vLLM contributors. Speculative decoding: Lossless guarantees of speculative decoding. vLLM documentation, version 0.32.0, 2026. Accessed November 7, 2026.

[5] Leonccaa. RFC: Batch invariance for speculative decoding needs to cover the forward pass (M=1 vs M=k+1). vLLM GitHub issue #54506, 2026.

[6] Raja Gond, Aditya K Kamath, Ramachandran Ramjee, and Ashish Panwar. LLM-42: Enabling determinism in LLM inference with verified speculation. arXiv preprint arXiv:2601.17768, 2026.

[7] Jianxing Qin, Alexander Du, Danfeng Zhang, Matthew Lentz, and Danyang Zhuo. Vosti: Specifying, implementing, and verifying deterministic LLM inference. arXiv preprint arXiv:2609.38981, 2026.

[8] Junyi Sun. jieba: Chinese text segmentation. Software, version 0.42.1, 2020.

[9] Jackson L. Lee, Litong Chen, Charles Lam, Chaak Ming Lau, and Tsz-Him Tsui. PyCantonese: Cantonese linguistics and NLP in Python. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 6607–6611, 2022.

[10] Paul McCann. fugashi, a tool for tokenizing Japanese in Python. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pp. 44–51, 2020.

AuthorsOpenReview ↗

Response to Reviewer mA4z (2/3): drafter-only comparison

W1 and Q2: the accuracy–speed trade-off of target verification. Drafter-only is faster, but AS²D substantially lowers WER on all five benchmark subsets. Under the same settings, drafter-only achieves 1.48–2.33× the TPS of AS²D. Verification reduces FLEURS WER from 63.28% to 24.83% and KeSpeech WER from 117.96% to 25.95%, with smaller reductions on the other three benchmarks. On these multilingual and dialectal inputs, the larger target makes more accurate recognition decisions and corrects disagreeing drafts. The revision adds the three-method comparison in §5.4 and Table 6 (pp. 9–10). The table below repeats the drafter-only comparison and decoding TPS for all three methods; target-only WER is given in Table R1.1.

Table R1.2. Native WER and decoding TPS (five-run means; ± denotes sample standard deviation).

BenchmarkLibriSpeechAMI SDMFLEURSKeSpeechM4Singer
Drafter-only WER (%)6.6343.5163.28117.9628.29
AS²D WER (%)3.3637.4224.8325.9513.60
Drafter-only TPS45.24 ± 2.0248.71 ± 2.9343.30 ± 2.5237.22 ± 3.0343.16 ± 3.52
Target-only TPS16.49 ± 0.4915.86 ± 0.5516.77 ± 0.1216.94 ± 0.8716.65 ± 1.32
AS²D TPS30.66 ± 0.9820.92 ± 0.6223.42 ± 0.6623.16 ± 0.3622.37 ± 0.58
AuthorsOpenReview ↗

Response to Reviewer mA4z (3/3): repeated native execution

W2 and Q3: repeated native throughput across the ASR benchmarks. AS²D achieves higher mean decoding TPS on all five benchmark subsets in repeated native execution. Using the same 150 inputs and five measured runs per method, the mean AS²D/target-only TPS ratios range from 1.32× to 1.86×. AS²D has higher pooled TPS in all five repetitions of every benchmark. The revision reports the repeated measurements and their variability in §5.4 and Table 6 (pp. 9–10). The table below repeats the paired TPS ratios as mean ± sample standard deviation over five runs.

Table R1.3. Native decoding TPS ratios (mean ± sample standard deviation over five runs).

BenchmarkLibriSpeechAMI SDMFLEURSKeSpeechM4Singer
AS²D/target-only TPS ratio1.86 ± 0.081.32 ± 0.051.40 ± 0.041.37 ± 0.081.35 ± 0.12

We hope these results and revisions address the concerns behind the rating, and we would be grateful if the reviewer would consider raising it in light of the new evidence. We are happy to discuss any remaining questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): what the revision changes

We thank reviewers mA4z, EwuX, and wksX for their careful and constructive reviews. The revised manuscript marks new and substantively revised material in blue. Part 2 maps the changes to their locations.

Repeated native results (mA4z, wksX). We added 150 selected inputs, 30 per ASR benchmark, with one warmup and five measured runs per input and method on Redmi K70 Pro. Using a Qwen3-ASR 1.7B target and 0.6B drafter, AS²D achieves mean paired throughput ratios of 1.32–1.86× relative to Target-only, with gains in all five repetitions of each benchmark. Mean corpus WER differences (AS²D minus Target-only) range from −0.15 to +0.05 percentage points. Per-input WER matches in 724/750 paired measurements (96.5%); the largest increase is 3.33 percentage points. Equal WER does not imply identical text.

Drafter-only tradeoff (mA4z). We now report WER and throughput for all three methods. The drafter alone is faster but has higher WER on all five benchmark subsets. Verification reduces FLEURS WER from 63.28% to 24.83% and KeSpeech WER from 117.96% to 25.95%.

On-demand generation and latency (EwuX). We clarify that audio context can be prepared before the request, while the requested response is generated afterward. End-to-end response latency runs from request arrival to completed output, including instruction prefill and any unfinished preparation; earlier audio preparation precedes this clock. At the 120s checkpoint, both methods generate the same 173 body tokens, while latency falls from 25.05s to 14.07s, including AS²D's 0.44s instruction prefill. This illustrates why generation dominates the response time in this case. Request batch size and inter-request concurrency are both one; drafting and verification overlap within that request.

Native output differences (mA4z, wksX). In the separate Omni-7B 210s case, outputs differ by 45 word edits. Against an audiobook reference reconstructed and automatically aligned to that window, WER is 0.65% for Target-only and 29.87% for AS²D—a substantial accuracy loss in this case. A controlled check reproduces the first differing greedy choice under single-token versus multi-token execution. Exact output preservation remains conditional on sequential-equivalent verification and correct state restoration.

Additional comparisons (wksX). On the original 688 windows, AS²D reaches 27.92 TPS, versus 18.14 for LLMA and 11.46 for prompt lookup, using matched K70 phone-cost replay. The LLMA adaptation charges the full cost of generating the audio-conditioned reference before reuse. The revision also clarifies how AS²D supplies candidates while generation continues independently of target-prefix updates.

The discussion period runs until November 18, and we will be available throughout it. We would be happy to discuss any further way to improve the paper, and we will gladly incorporate any further feedback into the revision.

AuthorsOpenReview ↗

General response (2/2): where each change is

ChangeWhere in the revised PDFRaised by
Repeated native evaluation: 150 selected inputs, five measured repetitions, WER, throughput, and variability§5.4, Table 6; App. F.5mA4z, wksX
Drafter-only comparison: accuracy and throughput alongside Target-only and AS²D§5.4, Table 6mA4z
Single-request setting: batch size and inter-request concurrency are one; conditions for useful overlap§3, §4.4, §5.1; Table 1 captionEwuX
On-demand generation and response latency: prepared audio context, request-to-complete-output timing, remaining preparation costs, and the identical-output 120s example§1, §3, §5.1, §5.4; App. D.6.1EwuX
Output discrepancy: controlled first-token check, reconstructed-reference WER for the 210s case, its accuracy cost, and clarified exactness conditions§4.2, §5.4; App. D.6.1 and F.6; existing proof in App. C.4mA4z, wksX
LLMA and prompt lookup: mechanism distinctions, matched comparisons, and full audio-reference generation cost§2, §5.2; App. D.4.6, Table 16wksX
Updated result summaries and evaluation scopeAbstract, §1, §6; App. F.5–F.6All three

New and substantively revised material is marked in blue. The main text is 10 pages, followed by the AI use statement on page 11.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.