acceptodds
Under review as a conference paper at ICLR 2027

Single Thread, Real Time: Heterogeneous Quantization for CPU Speech Recognition

Abstract

LM-based speech recognition models are among the most accurate ASR systems today, but they typically require GPUs for real-time inference, so private audio leaves the device and latency is tied to the network. We move them onto device CPUs and propose a memory-traffic model that assigns a quantized format per module. The transformer is weight-bound, so its weights are ternarized; the acoustic stack is activation-bound, so it runs entirely in INT8, which shrinks activation memory 6.2x against FP32. Progressive fake-quantization keeps training from diverging, and AVX2 and NEON intrinsics accelerate both formats. Trained and deployed this way, a 1.5B recognition model is up to 4.32x faster on a single thread than FP16 and 2.9x smaller, and transcribes faster than real time on one thread of either an Apple M4 or an Intel Core i7-13700, for a 0.43% increase in recognition error across ten corpora relative to the FP16 baseline. The allocation transfers to other recognition and generation models with comparable gains.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
quantization, on-device inference, CPU inference, speech recognition, text-to-speech, mixed-precision, quantization-aware training, ternary weights, INT8, SIMD kernels

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8cuuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a module-specific quantization strategy for running large language-model-based speech systems in real time on consumer CPUs. Based on a memory-traffic analysis, it ternarizes the weight-bound transformer while executing the activation-bound acoustic stack in INT8, supported by progressive fake quantization and optimized AVX2 and NEON kernels. For a 1.5B speech-recognition model, the approach achieves up to a 4.32x single-thread speedup and a 2.9x model-size reduction while incurring only a small increase in recognition error.

Critical strengths
  1. The paper addresses a practical deployment problem with a activation-weight ratio policy for assigning different quantization methods to the acoustic stack and transformer.
  2. The implementation demonstrates meaningful end-to-end speedups and real-time, single-thread inference on consumer CPUs using both AVX2 and NEON kernels.
  3. The efficiency gains come with only a small accuracy degradation across ten speech corpora, while also substantially reducing model size and memory usage.
Critical weaknesses
  1. While the bottleneck analysis and the resulting choice of module-specific quantization schemes are interesting, the individual methodological components appear to have limited novelty, and the overall approach can be viewed as a system-level integration of existing techniques for edge speech applications. Although the paper acknowledges several relevant lines of quantization work, its positioning relative to some closely related techniques could be strengthened through a more detailed qualitative discussion in the related work section and direct quantitative comparisons in the experiments if possible. Some examples are:

1-1. While the backbone model is not identical, there's a line of prior work on quantization in the similar domain (e.g. ASR) which should at least be acknowledged and compared methodologically. Listing a few:

1-2. Ternary quantization for LMs: This seems to be a straightforward application of the BitNet idea.

1-3. Scheduling for quantization: https://doi.org/10.24963/ijcai.2019/425


  1. The accuracy evaluation appears preliminary for several reasons:
  • The accuracy evaluation is conducted on only one model. Its generalizability is still questionable.
  • In particular, none of the TTS models are evaluated for output quality. Therefore, it is difficult to assess the effectiveness and reliability of the method for TTS.
  • The method is compared only against a BF16 baseline. It should also be compared against other quantization methods. Otherwise, it is difficult to assess how competitive the method is and whether the reported accuracy degradation is acceptable.
Questions for authors

No specific questions other than what i listed on the weakness section.

Reviewer_iXLKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a heterogeneous quantization system for CPU speech recognition. A memory-traffic analysis motivates ternary weights for the weight-dominated autoregressive transformer and INT8 weights and intermediate activations for the acoustic encoder. The implementation combines progressive quantization-aware training, operator fusion, and AVX2/NEON kernels. The evaluation uses a 1.5B VibeVoice-ASR model. On the fixed-work timing benchmark, it achieves single-thread real-time factors of 0.673 on Apple M4 and 0.863 on Intel i7-13700, with a 2.9× reduction in model-file size. Recognition error increases by an average of 0.43 percentage points across ten corpora. Ablations examine module-level speedups and activation memory. Four additional recognition and generation systems demonstrate size and execution-time reductions, but their output quality is not evaluated.

Critical strengths

A practically relevant systems contribution: Running an LM-based recognizer with competitive accuracy on a single consumer CPU thread is a worthwhile target, and the paper implements the complete inference path.

Convincing activation-memory evidence: Against identical INT8 weights, keeping intermediate tensors narrow reduces the compute arena by approximately 4.3× relative to dynamic W8A8 and improves execution time.

Useful breadth within the main evaluation: Three CPU platforms, multiple thread counts, ten recognition corpora, and module-level ablations provide complementary evidence.

Critical weaknesses

Four additional models are converted without retraining recognition accuracy, intelligibility, or generation-quality results.

Appendix B shows catastrophic failure of untrained ternary conversion for the model. Thus, the additional experiments demonstrate that the kernels execute other architectures efficiently, but do not establish that the precision allocation transfers while preserving useful behavior.

Questions for authors

Do the four converted models retain useful output quality?

Reviewer_PYDYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a methodology to develop a speech recognition model which can be run on a single thread of CPU. The authors introduce memory-traffic model that determine if a module is "weight-bound" or "activation-bound". The bits of weights and activations are assigned heterogeneously for the acoustic encoder and the decoder module. The authors proposed "progressive QAT", which anneals the model from floating-point to quantized computation. The activation traffic is further reduced with the fused kernel implementation. The accuracy of the quantized model on ASR benchmarks is comparable to the reference models, and the model can be run in real-time on a single thread of CPU.

Critical strengths

They combined various techniques ranging from quantization to inference optimization to enable a large asr model to run on a single CPU thread.

Critical weaknesses

The method proposed in this paper is not easily generalizable to other types of asr models. Many ASR models, like Whisper, Qwen3-asr, and Parakeet use mel-filterbank with 10ms stride as the input feature. As shown in Table 7 in Appendix, the activation-to-weight ratio of Qwen3-asr is much less than VibeVoice-ASR. For these type of ASR models, the throughput improvement of the proposed method is not significant compared to that of VibeVoice-ASR, as shown in Table 11.

The memory-traffic model is based on the assumption that the inference time of the asr model is bounded by memory bandwidth. However, in this case where few CPU cores are used it is unclear whether it is computation-bound or memory-bound, especially for the acoustic encoders which weights can be reused for the whole input sequences.

Questions for authors
  • Do you have any experimental (cpu or memory bandwidth util.) or theoretical analysis (e.g. theoretical peak performance) demonstrating that the model is memory traffic bounded in single thread CPU inference, considering hardware specs such as FLOP/s or memory bandwidth?

  • VibeVoice-ASR supports various speech understanding task like speaker diarization or time-stamping. How does performance on these tasks changed after quantization?

Comments on the paper

2 comments
Area_Chair_BozdOpenReview ↗

Overstatement in the introduction

The statement that "Speech recognition has converged on one design: a non-autoregressive acoustic stack of convolutional and linear layers encodes the waveform into a low-rate latent sequence, and an autoregressive transformer transcribes that sequence..." is not accurate. In many edge applications, simply using a well-trained CTC-Conformer model provides good enough transcription accuracy while requiring much less memory and computation than a speech-LM.

AC

Area_Chair_BozdOpenReview ↗

Reviewers: please respond to the authors while the discussion period is still open

Dear reviewers,

The authors have responded to your reviews and revised the paper. Please check the responses and revisions and reply to the authors while the discussion is still open.

Thanks,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.