acceptodds
Under review as a conference paper at ICLR 2027

Pruned CTC for Memory-Efficient Large-Vocabulary ASR Training

15 views

Abstract

Connectionist temporal classification (CTC) naturally supports offline and streaming speech recognition with utterance-level supervision, but conventional implementations materialize frame-by-vocabulary activations in memory, making CTC training with native LLM vocabularies prohibitively memory-intensive. A key observation is that every valid CTC alignment uses only target tokens and blank, and their union across a batch typically forms a small subset of the full vocabulary. We introduce Pruned CTC, which restricts alignment computation to this subset while retaining full-vocabulary normalization. We prove that this vocabulary reduction is exactly equivalent to full-vocabulary CTC in loss and gradients. Head-and-loss activation memory no longer scales linearly with vocabulary size. We further apply finite-beam alignment pruning. Building on Pruned CTC, we develop LLM-CTC, which adapts pretrained LLMs for non-autoregressive ASR while retaining causal attention and native vocabularies, and extend it to bounded-history streaming, avoiding chunk-level speech–text alignments. Experiments show that, with Zipformer-M encoder and 180K vocabulary, Pruned CTC reduces full-step memory by 5.1 with only 17% step-time overhead. Across three corpora, it matches standard CTC accuracy. On GigaSpeech, across six Qwen3 model sizes from 0.6B to 32B, LLM-CTC remains within 7% relative WER of LLM-CE with 7 to 10 faster recognition; when fine-tuning Qwen3-ASR for bounded-history streaming, LLM-CTC remains within 3% relative WER of matched offline models on the test set. Together, these results establish Pruned CTC as a scalable sequence objective for native-vocabulary LLM ASR across offline and streaming settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Connectionist temporal classification, Memory-efficient training, Large language models, Automatic speech recognition, Streaming ASR

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sgSKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Pruned CTC, a CTC-based method that restricts alignment computation to the subset formed by the union of target tokens and blanks of valid CTC alignments across a batch, while retaining full-vocabulary normalization. Furthermore, based on Pruned CTC, the paper also introduces LLM-CTC, which adapts pretrained LLMs for non-autoregressive ASR while retaining causal attention and native vocabularies, and extends it to bounded-history streaming. The results show that Pruned CTC can be utilized as a scalable sequence objective for native-vocabulary LLM ASR across offline and streaming settings.

Critical strengths
  • The training memory reduction and inference-time speedup over LLM-CE are significant.
  • The paper’s presentation is clear, and it has a strong theoretical foundation.
Critical weaknesses
  • Baseline models’ RTF numbers are not provided in Table 4. The missing measurements make it difficult to assess its inference efficiency relative to existing ASR systems.
  • Existing streaming LLM-based ASR systems are mentioned in the related work but are not included as baselines for LLM-CTC’s streaming comparison in Table 5.
Questions for authors
  • See Critical Weaknesses. Can the authors further provide the data and comparisons mentioned to better assess the proposed method’s effectiveness?
  • Table 1 shows additional training-time overhead for Pruned CTC compared with standard CTC. Can the authors further explain or provide a deeper analysis of this phenomenon?
  • Can the authors extend the LLM-CTC evaluation to additional datasets to further demonstrate the models’ capability?
Reviewer_V1gfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a memory-efficient implementation of CTC. Alignments are computed only over the transcript labels and blank, while normalization is still taken over the full vocabulary. The authors pair this exact reduction with alignment pruning and use the result to build non-autoregressive recognizers on top of pretrained causal LLMs. They also describe a streaming extension that trains on utterance-level transcripts and so does not need chunk-level speech–text alignments.

Critical strengths
  • The paper tackles a real practical problem: CTC becomes very expensive in memory once the output vocabulary is the size of an LLM’s. Preserving the objective and gradients while reducing activation memory makes the exact reduction potentially useful across existing CTC architectures, rather than only within the proposed LLM system. In the papers’ setting, full-step memory falls by a factor of 5.1 for a 17% increase in step time. That is a good trade, and it may let groups with modest hardware train with much larger vocabularies.

  • Using a pretrained causal LLM directly as a non-autoregressive recognizer is an interesting idea. Keeping the LLM’s native vocabulary means the output space does not have to be redesigned, and the streaming formulation performs this without the extra alignment supervision that some speech–LLM systems rely on. Results across six model sizes show that the approach is feasible for both offline and streaming recognition.

Critical weaknesses
  • According to Section 5.3, both systems are evaluated for WER with a beam size of four, while runtime is measured with greedy decoding. As a comparison of autoregressive and non-autoregressive decoding speed, this is fair. The difficulty is that the greedy WER is never reported, so we do not know what accuracy the measured speed corresponds to. The speed–accuracy tradeoff is central to the paper’s motivation, and both numbers need to come from the same decoding setup.

  • Table 4 also includes SPEAR-Xlarge CTC and RNN-T, which put CTC or RNN-T output layers on the pretrained SPEAR speech encoder [1] and act as the no-LLM baselines. Their accuracy is reported; their runtime and inference memory are not. Without those ablations, the readers cannot determine whether the accuracy gained by adding an LLM is worth what it costs at inference time. I also find the separation from earlier non-autoregressive LLM recognition to be mostly architectural. NLE [2] is discussed but never compared against experimentally, and the memory approach builds on ideas related to Cut Cross-Entropy [3]. The contribution is therefore better established as a useful implementation and combination of existing ideas than as a substantially new recognition framework.

  • Pruning is harder to evaluate. In Table 8, the unpruned reduced CTC and the pruned version have nearly the same full-step memory and runtime, so it is unclear what the approximation adds.

  • On streaming, WER rises slightly compared with the adapted offline CTC models which is acceptable. However, emission delay, hypothesis stability and streaming runtime are not computed and a two-second chunk size says little about when a word first appears or when it stops being revised. Without those computations and a comparable streaming baseline, the results show that streaming is feasible; they do not yet show a practical advantage.

REFERENCES

[1] Xiaoyu Yang, Yifan Yang, Zengrui Jin, Ziyun Cui, Wen Wu, Baoxiang Li, Chao Zhang, and Phil Woodland. “SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations.” In Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, pp. 146680–146704, 2026. https://proceedings.mlr.press/v306/yang26cv.html

[2] Avihu Dekel, Samuel Thomas, Takashi Fukada, and George Saon. “NLE: Non-autoregressive LLM-based ASR by Transcript Editing.” arXiv preprint arXiv:2603.08397, 2026. https://arxiv.org/abs/2603.08397

[3] Erik Wijmans, Brody Huval, Alexander Hertzberg, Vladlen Koltun, and Philipp Krähenbühl. “Cut Your Losses in Large-Vocabulary Language Models.” In Proceedings of the Thirteenth International Conference on Learning Representations (ICLR), 2025. https://proceedings.iclr.cc/paper_files/paper/2025/file/aa963ac256590bb7ad5fc26c68229a3a-Paper-Conference.pdf

Questions for authors

Q1. Does the speed–accuracy tradeoff still hold when WER and runtime are measured together, once with greedy decoding and once with beam-four decoding?

Q2. What are the inference time and memory of the SPEAR-Xlarge CTC and RNN-T baselines in Table 4? How does the proposed system compare with an NLE-based model [2] under matched conditions?

Q3. Could you clarify the intended benefit of finite-beam alignment pruning, given the nearly identical full-step costs of the unpruned reduced CTC and pruned CTC implementations in Table 8?

Q4. What are the emission delay, hypothesis-revision rate and runtime of the streaming model, compared with an existing streaming recognizer at similar latency?

Reviewer_ovmkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Pruned CTC, a memory-efficient training method. It improves upon the memory usage of convention CTC. Since valid CTC alignments use only target tokens and the blank symbol, i.e., a small portion of the total vocabulary, the authors maintain that CTC can be pruned to restrict alignment to the subset. Full-vocabulary is used for normalization. The authors show that pruned CTC is mathematically equivalent to conventional CTC in terms of loss and gradient. This enables the authors to integrate with LLMs for ASR. The authors also propose a streaming extension.

Critical strengths
  1. The idea is sound, with exact formulation and mathematical equivalence shown. Section 3.2 is well-written and explains the idea cleanly.
  2. The impact on memory saving is notable.
  3. LLM-CTC's design is clean and evaluated against LLM-CE across different sizes.
  4. I find the writing to be especially clean and detailed. Experimental setups are explained and all figures/tables are easy to understand.
Critical weaknesses
  1. The performance of LLM-CTC is not good enough to justify the use of large LLMs. In Table 4, the SPEAR-XLarge models achieve better performance than LLM-CTC up till 1.7B. Moreover, no RTF is given for the baselines, which would be an interesting comparison. Even with a 32B model the performance of LLM-CTC does not seem to improve significantly. These results question the motivation behind training large-vocabulary CTC that can be incorporated with LLMs.
  2. Beam pruning's contribution to the method seems unclear to me. Line 377-379 mentions that "Unpruned reduced CTC achieves nearly the same full-step memory and runtime as Pruned CTC".
Questions for authors
  1. What's the batch size used for the RTF speedup results? Have the authors tried varying the batch sizes and see how does the speed comparison change?
  2. Can the authors provide RTF results for the encoder only baselines? They could give more context to adapting LLMs.
  3. Applying LLM-CTC to Qwen3-ASR base models seem to degrade the performance for a bit (Table 4 and Table 5 offline comparison). Do the authors know of the reason?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.