acceptodds
Under review as a conference paper at ICLR 2027

Depth-Conditioned looped Transformers for Adaptive ASR Inference

Abstract

End-to-end automatic speech recognition (ASR) systems use fixed-depth encoders, so each accuracy–compute budget typically requires its own trained model. We ask whether a single compact encoder can instead make its inference budget a deployment-time choice. A natural approach is to reuse a shared Transformer block recurrently, but we find that naive looping does not fully exploit the additional recurrent compute. We introduce LARM, a depth-conditioned looped Transformer that combines sparse connectionist temporal classification (CTC) checkpoints, supervision-clock embeddings, FiLM depth conditioning, and soft-posterior feedback. These components structure the loop into recognition checkpoints separated by latent refinement phases and let shared weights specialize across recurrent steps. A single trained model can thus stop at any checkpoint within its trained loop budget . On LibriSpeech, LARM improves WER as more loops are executed within this budget, and in a common pipeline it achieves lower WER than closely related self-conditioning and recurrent CTC methods. The design also transfers to a Conformer backbone. Compared with deeper unshared encoders in the same pipeline, LARM reaches equal or better WER with far fewer parameters and less peak memory, making it well suited to memory-constrained deployment.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Test-time compute scaling, Looped Transformers, Automatic speech recognition, Iterative refinement, Depth conditioning, Connectionist temporal classification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_M2jiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LARM (Loop Audio Recurrent Model), a depth-conditioned looped Transformer for budget-adaptive ASR inference. LARM applies a shared Transformer encoder recurrently to a latent acoustic sequence, decoupling computational depth from parameter count. It combines sparse CTC supervision, supervision clock embeddings, FiLM depth conditioning, and delayed prediction feedback, where soft CTC posteriors are shifted by one frame and reinjected into the recurrent state. These components structure the loop into supervised recognition checkpoints separated by intermediate refinement steps and allow shared weights to specialize across recurrent iterations. A single trained model can stop at any supervised checkpoint within its trained loop budget. Experiments on LibriSpeech demonstrate improved greedy WER over closely related self-conditioning and recurrent CTC methods, with fewer parameters and less peak memory than deeper unshared encoders. The design also transfers to a Conformer backbone.

Critical strengths

1. Parameter efficiency. LARM achieves competitive recognition accuracy with fewer parameters.

2. Experimental evaluation. The paper provides relevant baseline comparisons, ablation studies, and an additional Conformer evaluation, supporting the effectiveness of the overall approach.

Critical weaknesses

1. Insufficient differentiation from prior work. The paper presents budget-adaptive inference and prediction feedback as major contributions, although [1] already employs recurrent CTC posterior feedback and explores varying decoding iterations, while [2] supports multiple inference depths in shared-parameter ASR models. In particular, LARM introduces a one-frame delay to the current-frame feedback used in [1], but Table 4(c) shows no statistically significant improvement (p = 0.51). Moreover, LARM lacks a direct comparison with [2], and comparisons with related recurrent ASR methods are limited to LibriSpeech 100h, where LARM and [1] achieve statistically similar WER with LM decoding. These limitations make it difficult to establish LARM's contributions over existing approaches.

2.Incomplete validation of the proposed mechanisms. In Figure 3(b), varying the CTC supervision interval also changes the supervision-clock embeddings. Therefore, the reported improvements cannot be attributed solely to sparse supervision. Moreover, the supervision-clock embedding is not independently ablated, leaving the individual contributions of sparse supervision and clock conditioning unclear.


[1] T. Komatsu, "Non-Autoregressive ASR with Self-Conditioned Folded Encoders," ICASSP, 2022.

[2] Z. Li et al., "Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models," Interspeech, 2025.

Questions for authors

Q1. What does LARM contribute beyond the recurrent feedback and flexible-depth inference already explored in [1, 2]?

Q2. How does LARM compare with existing recurrent and self-conditioned ASR methods when trained on LibriSpeech 960h? The current comparisons with these methods are limited to 100h.

Q3. What is the independent contribution of the supervision clock embedding? How does varying the CTC supervision interval affect WER when the clock embedding is disabled?

Q4. Given that [1] already uses current-frame CTC feedback, what evidence supports the benefit of LARM's proposed one-frame delay? Table 4(c) shows no statistically significant improvement, although the authors claim better results in most experiments during model design. What results support this?

5 comments
AuthorsOpenReview ↗

Response to Reviewer M2ji

We thank the reviewer for the careful review. We answer the questions below, grouped with the corresponding weaknesses.

Q1 / W1. Contribution beyond [1] and [2].

The folded encoder of Komatsu [1] is the closest method. It applies a small stack of shared blocks repeatedly, with a CTC loss and self-conditioning at every repeat, and it can vary the number of repeats at decoding time. LARM differs in three ways. First, its supervision is sparse. CTC checkpoints are placed every few loops and scheduled by a supervision clock, which leaves the loops in between free to refine the representation. These are our two new mechanisms. Second, the shared stack is conditioned on the loop depth through FiLM, whereas [1] applies the same stack identically at every repeat. Third, the goal is different. [1] targets a compact encoder used at its training depth, while LARM is designed so that each supervised checkpoint is a usable operating point.

The method of Li et al. [2] unfolds a small seed model by repeating its layers, in order to emulate a larger model. It is trained with the ASR loss on the deepest unfolded system and on the seed, together with a self-distillation term between the two. The intermediate depths receive no supervision of their own, and the shared layers receive no information about their depth. Its goal is the compression of a larger model. LARM instead trains every exit directly with CTC, without a teacher, and conditions the shared stack on its depth, with the goal of serving several inference budgets from one checkpoint.

These differences have a practical effect. At matched block passes, the exits of LARM outperform those of [1] at every budget, and most at small budgets (100h test-clean WER at 16 / 32 / 48 block passes: 15.20 / 11.57 / 11.34 for LARM vs 19.99 / 13.23 / 12.31 for [1]). We also trained the method of [2], adapted to our pipeline with the same loop (4 seed blocks, each repeated up to 12 times). At 48 block passes, it reaches 14.48 test-clean WER, against 11.34 for LARM. We also ran the base-plus-folded architecture of [1], which precedes the folded blocks with unshared base blocks. At matched block passes, LARM is significantly better under greedy decoding (11.34/31.84 vs 11.67/32.61 test-clean/test-other, p ≤ 0.014) with 31% fewer parameters, and the two are tied under LM decoding (8.66/26.28 vs 8.45/26.17).

Q2 / W1. Comparisons at 960h.

We trained [1] (R=12) and self-conditioned CTC (16 layers) at 960h with the same recipe as our 960h models. The table below reports the test-clean WER at the end of the same 50-epoch schedule.

ModelParamsTest-clean WER
Folded encoder [1], R=127.65M4.98
Self-conditioned CTC, 16 layers28.9M4.30
LARM d=3847.7M4.59
LARM d=76829.5M3.45

At the same parameter count, LARM d=384 outperforms [1] (4.59 vs 4.98), and LARM d=768 outperforms self-conditioned CTC (3.45 vs 4.30). The 100h results therefore hold at 960h.

Q3 / W2. Supervision clock.

The clock embedding is part of the sparse-supervision design. It was built to schedule the sparse checkpoints, telling the shared stack whether the current loop is a recognition checkpoint or a latent refinement step. To separate the two effects, we trained the model without the clock embedding (100h, test-clean/test-other WER, greedy).

SupervisionClockWER
Dense (c=1)off11.95/32.77
Sparse (c=4)off11.57/32.12
Sparse (c=4)on (LARM)11.34/31.84

Sparse supervision helps on its own, and the clock adds a further gain. The two mechanisms work together, and Figure 3(b) shows their combined effect.

Q4 / W1. One-frame delay.

As reported in Table 4(c), the paper already acknowledges that the delay is not significant (p = 0.51). The delay is the only such case: prediction feedback, depth conditioning and the sparse supervision schedule each bring statistically significant gains in the paper's ablations. The remark on design-phase experiments refers to a few runs and is not a claim; we will remove it to avoid any ambiguity.

We hope we have addressed the reviewer's concerns and would be happy to provide any additional details as needed. We sincerely hope the reviewer will consider revising the score in light of the clarifications provided.

[1] T. Komatsu, Non-Autoregressive ASR with Self-Conditioned Folded Encoders, ICASSP 2022.

[2] Z. Li et al., Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models, Interspeech 2025.

AuthorsOpenReview ↗

Follow-Up on Our Rebuttal Responses

Dear Reviewer M2ji,

We appreciate your feedback on our submission. In response to your comments, we have presented new experimental results and provided additional clarifications. As the discussion period is coming to an end, we would appreciate knowing whether our responses have addressed your concerns.

Best regards,

The Authors

I appreciate the authors' detailed response and additional experiments, which largely address my concerns regarding Q2 and Q3. However, my concerns regarding Q1 and Q4 remain.

Q1 / W1. Differentiation from prior work

The rebuttal clarifies that LARM's main methodological distinctions lie in sparse supervision, supervision-clock embeddings, FiLM depth conditioning, and direct supervision of intermediate exits. However, the manuscript also presents budget-adaptive inference and prediction feedback as separate main contributions. This inconsistency extends beyond the contribution list to the overall framing of the paper, suggesting that substantial revisions are needed to accurately reflect the contributions clarified in the rebuttal.

Q4 / W1. One-frame delayed feedback

My concern is not limited to the statement about design-phase experiments. The manuscript repeatedly emphasizes delayed feedback as a key mechanism throughout the Introduction, Method, and Conclusion, potentially misleading readers about its actual contribution. Removing that statement alone does not address this broader concern.

Additional Question

These issues appear to require substantial revisions to the manuscript's overall positioning and presentation of its contributions. Could the authors explain how they plan to revise the manuscript to address these concerns?

Reviewer_yKsMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LARM re-uses a small Transformer encoder stack many times: the loop index conditions the shared computation (FiLM plus a supervision clock), CTC supervision is applied only every few loops, and the model can be stopped early to trade accuracy for compute. Training and evaluation are done on LibriSpeech with all baselines re-implemented in one matched pipeline, and inference cost is reported carefully. The claim is that a single compact model can thereby serve several deployment budgets while remaining competitive with much deeper conventional encoders.

Critical strengths
  1. Careful, honest empirical methodology. All baselines re-implemented in one pipeline with matched vocab/frontend/augmentation/decoding; matched-pair bootstrap p-values; per-exit WER tables; measured GFLOPs/latency/peak memory; limitations stated plainly.
  2. The deployment story is real. Decoupling depth from parameters so one checkpoint serves multiple budgets (~10% memory variation across exits) is genuinely useful for memory-constrained serving, and it is quantified properly.
  3. Sensible, ablated design choices (sparse beats dense and final-only supervision at p<0.005; FiLM beats alternative conditioning; feedback and learned scalars each contribute) plus cross-backbone transfer.
Critical weaknesses
  1. Limited conceptual novelty: an assembly of existing parts. Looped ASR is six years old — Zhao et al.'s Universal Speech Transformer (Interspeech 2020) is a shared iterative speech encoder with dynamic depth (cited, but dismissed in one collective sentence with BiCycle and no comparison); Komatsu's folded encoders apply the stack "repeatedly for further refinement"; BiCycle (AAAI 2026) recurses a Transformer for CTC ASR (group-wise, with distillation); Tang et al. (arXiv 2023; ISNN 2024, peer-reviewed) add per-reuse adapters to a block-reusing ASR encoder; Relaxed Recursive Transformers attach per-loop LoRA. Entirely uncited weight-shared ASR lineage: Li et al. (Interspeech 2019), Gao et al. (Interspeech 2021), Bai et al. (Interspeech 2022, per-group conditioning of shared conformer blocks), Hernandez et al. (ICASSP 2023), ResidualTransformer (ICASSP 2024), Rolland & Abad (Interspeech 2025), Foldable Network (Interspeech 2025). The feedback/supervision mechanisms come from Self-Conditioned and Intermediate CTC; conditioning is FiLM; early-exit ASR exists (Shi et al. 2021; I3D; SplitFormer; Wright et al.). The novel residue is two mechanisms — sparse in-loop checkpoints (every c=4; the genuinely new element) and FiLM on loop depth (a new application of existing conditioning) — each small: the clock embedding is an auxiliary of sparse supervision (degenerate at c=1, never ablated separately from depth conditioning despite §3.2.4 calling clock and depth "two complementary signals"), and the budget framing quantifies Komatsu's decode-time repeat knob (the one-frame-delayed feedback is ablated by the paper itself: feedback per se helps — 12.31 without, 11.43 with current-frame injection — but the delay, the only twist on the self-conditioning lineage's current-frame feedback, is statistically tied: 11.34 vs 11.43, p=0.51). The ablations bound the package: the largest single contributor is feedback per se (+0.97, borrowed); among the novel mechanisms only FiLM shows a substantial gap (None 12.13 / Embedding 11.98 / MLP 11.77 vs 11.34); sparsity is significant but modest; the clock is never ablated; and the closest naive-assembly proxy, the folded encoder R=12 (12.31), ties LARM under LM decoding. The differentiation from BiCycle (the closest prior art in time) rests on a single sentence.
  2. An aging, easy regime. LibriSpeech only; 30-symbol char vocabulary; CTC + 4-gram LM; best greedy WER 3.08/8.29 at 960h vs ~1.8–2.5 for modern systems. No subword vocabulary, no neural-LM decoding, no second domain. LibriSpeech itself is small and saturated by current standards (GigaSpeech 10k h, SPGISpeech 5k h, MLS 50k h) — no large-scale or out-of-domain validation.
  3. The latency story undermines the deployment pitch. At matched FLOPs, LARM is slower than the unshared 48-block encoder (30.5 vs 24.2 ms for 10 s audio) because loops serialize; batching behavior is unaddressed. For ASR serving, wall-clock is usually the binding constraint; the memory argument alone is half the story.
  4. The headline claim is configuration-dependent. The reference d=384 LARM loses to the 48-block encoder at 960h (4.59 vs 3.87); the claim survives via d=768 + exit-at-8 (3.46 vs 3.87, 29.5M vs 85.7M). Fair, but the abstract presents only the favorable slice. Under LM decoding the separation from the strongest prior baseline disappears entirely: vs folded encoder R=12, greedy p<0.0001 but LM p=0.85/0.30 — a statistical tie (Table 8); the best LM-decoded exit is loop 7, a non-supervised loop (p<0.0001 vs the final exit at 100h).
  5. Single seed per configuration. Bootstrap over utterances addresses decode variance, not training variance (±0.1–0.3 WER), which matters for sub-point mechanism rankings.
  6. No mechanistic insight into what intermediate loops actually do; the "latent refinement" narrative is asserted, not probed.
  7. The ablation that names the contribution is missing: folded encoder equipped with LARM's mechanisms (sparse supervision + FiLM + clock + feedback), and no learned-halting baseline despite marketing adaptivity (fixed budgets only). Sharpened by a fact the paper does not disclose: the re-implemented folded encoder strips the base encoder the original method requires (Komatsu 2022's own Table 1 — the no-base variant is worse than plain CTC), and the published configuration (base+folded, 11.6–16.3M params) was never run in the matched pipeline.
  8. No method comparisons at scale, and no method control on the Conformer backbone. All method baselines (self-conditioned CTC, intermediate CTC, Align-Refine, folded encoder) are run only at 100h; at 960h LARM faces only plain encoders plus one param-matched 4-block Conformer at 12× less compute. The paper's own results show matched-FLOPs outcomes flip at 960h (beats the 48-block encoder at 100h, 11.34 vs 12.24; loses at 960h, 4.59 vs 3.87). The Conformer backbone — the strongest standard family — has no method control at any scale: no naive-looped Conformer, so the 5.48 → 3.66 gain is unattributed between LARM's mechanisms and plain weight reuse (naive looping provides most of the gain on Transformers).
Questions for authors

Major:

  1. Transplant ablation: how much of the gap over the folded encoder survives when LARM's mechanisms (sparse supervision, FiLM, clock, feedback) are moved onto it — and conversely, how far does naive looping with dense intermediate CTC and current-frame feedback (the standard assembly) go on your own loop structure?
  2. Positioning and scale: expand the one-sentence treatment of prior looped ASR (Zhao et al. 2020; BiCycle AAAI 2026) into a substantive comparison, cite the uncited weight-shared ASR lineage (Li IS2019; Gao IS2021; Bai IS2022; Hernandez ICASSP 2023; Tang ISNN 2024; ResidualTransformer ICASSP 2024; Rolland & Abad IS2025; Foldable Network IS2025), and report method baselines at 960h plus a naive-looped Conformer control.

Minor:

  1. Add one realistic configuration (BPE vocabulary and/or neural-LM rescoring, or a second corpus such as GigaSpeech/SPGISpeech/Common Voice). Do the conclusions hold?
  2. Address latency directly (batched-serving throughput vs the 48-block encoder at matched WER) and report ≥3 training seeds for the 100h ablation table.
  3. Why is learned halting out of scope? Even oracle-exit analysis would quantify the claimed adaptivity headroom; also state the held-fixed hyperparameters in the Figure 3 caption (a: K varied at c=4; b: c varied at K=12).
7 comments
AuthorsOpenReview ↗

Response to Reviewer yKsM (1/5)

We thank the reviewer for the detailed review. We answer the major and minor questions below, grouped with the corresponding weaknesses.

Major Q1 / W7 / W4 (LM). Transplant ablation and base encoder.

Transplant ablation. The folded encoder of Komatsu [19] and LARM use the same loop: 4 shared blocks applied 12 times. Both directions of the transplant are therefore already in our comparison.

  • Our loop with the standard components (a CTC loss at every loop and current-frame feedback): this is exactly our folded-encoder baseline. It reaches 12.31 test-clean WER. Without the feedback, it is the naive-recurrence row of the paper (12.70).
  • The folded encoder with LARM's mechanisms (sparse checkpoints with the clock, FiLM depth conditioning and our feedback) is LARM itself, up to implementation details (input re-injection and loss weighting). It reaches 11.34.

Therefore the gap between 12.31 and 11.34 measures the contribution of LARM's mechanisms.

Base encoder. The reviewer notes that our folded baseline omits the base encoder of the original method [19], and that the base-plus-folded configuration was not run in our pipeline. We have added the requested base-plus-folded architecture, adapted to our pipeline (3 unshared base blocks followed by 3 folded blocks). The conclusions of the paper are unchanged. LARM remains significantly better under greedy decoding. The table below reports the WER on LibriSpeech 100h (test-clean/test-other).

ModelParamsBlock passesGreedy4-gram LM
Folded encoder (no base), R=127.65M4812.31/33.308.68/26.51
Base + folded, R=611.2M2113.04/34.458.96/27.17
Base + folded, R=1511.2M4811.67/32.618.45/26.17
LARM7.7M4811.34/31.848.66/26.28

Under greedy decoding, which measures the encoder itself, LARM is significantly better than every variant (p ≤ 0.014) with fewer parameters. With the 4-gram LM, LARM is significantly better than the 21-pass variant (p ≤ 0.03) and statistically tied with the two 48-pass variants (p ≥ 0.11), as an external LM compresses the differences between encoders.

[19] T. Komatsu, Non-Autoregressive ASR with Self-Conditioned Folded Encoders, ICASSP 2022.

AuthorsOpenReview ↗

Response to Reviewer yKsM (2/5)

Major Q2 / W1 / W8. Novelty, prior work, and comparisons at scale.

We thank the reviewer for these references. We will complete our related work and clarify how our method differs from these approaches. We discuss the Universal Speech Transformer [1] and BiCycle [2] first, as requested.

The Universal Speech Transformer [1] is an attention-based encoder-decoder in which the depth is decided by the model itself, for each input position, through adaptive computation time. It has no supervised exits, so the inference budget cannot be chosen by the user, and intermediate depths are not trained to be decoded. LARM instead trains explicit CTC checkpoints inside the loop, and each of them is a usable operating point. The two works also reach opposite conclusions on depth information. The authors of [1] remove the depth embedding of the universal transformer, arguing that it dilutes the acoustic information. Our ablation shows that the form of the conditioning matters. An additive depth embedding brings little (11.98 vs 12.13 without any depth signal), whereas FiLM modulation reaches 11.34.

BiCycle [2] converts a pre-trained Conformer-CTC model into a recursive one. Its layers are divided into a phonetic group and a linguistic group, recursion is applied within each group, the pre-trained weights are transferred step by step, and group-wise feature distillation from the original model preserves its recognition behaviour. Its goal is to compress an existing model, and each BiCycle model is trained for a fixed number of recursions per group, with a CTC loss only at the output. LARM is trained from scratch with a single shared stack and no teacher, and its goal is different, namely to offer several inference budgets from one checkpoint through supervised exits.

For the other lines of work, the main differences are the following.

  • Weight-shared ASR encoders [3-7] use sharing to compress a fixed-depth encoder, and the Foldable Network [11] trains only its smallest and deepest configurations, with self-distillation between them. We use sharing instead to make the inference depth selectable within one checkpoint, with every exit trained directly by CTC and no distillation.
  • Methods that specialise reused blocks [8-10] add parameters for each reuse (adapters, LoRA modules, or individual routers and normalisation), while our depth conditioning adds no per-loop parameters, as a single FiLM network shared by all loops conditions the stack on its depth.
  • Early-exit and dynamic-depth encoders [12-15] select their depth along unshared layers, so their memory grows with depth. In LARM, all exits share the same weights.

Even the closest methods, which we reimplemented in a common pipeline, differ from LARM in several components:

MethodShared stackExitsCTC supervisionFeedbackDepth conditioning
Intermediate CTCnonofixed layersnono
Self-conditioned CTCnonofixed layersfrom lower layersno
Folded encoder [19]yesevery folddense (every fold)from previous foldno
LARMyessupervised checkpointssparse, clock-scheduledfrom previous loopFiLM on loop depth

LARM therefore differs from all these methods in both its objective and its components. In addition, it introduces two new mechanisms, sparse supervised checkpoints and the supervision clock that schedules them. These mechanisms have a practical effect. As the reviewer notes, the folded encoder can also vary its number of repeats at decoding time. In practice, however, the two behave differently. At matched block passes, LARM's exits are better at every budget, and most at small budgets (100h test-clean WER at 16 / 32 / 48 block passes: 15.20 / 11.57 / 11.34 for LARM vs 19.99 / 13.23 / 12.31 for the folded encoder), showing that our architecture is better suited to selecting the inference budget.

AuthorsOpenReview ↗

Response to Reviewer yKsM (3/5)

Clock embedding and sparse-supervision. The clock embedding is part of the sparse-supervision design. It was built to schedule the sparse checkpoints, telling the shared stack whether the current loop is a recognition checkpoint or a latent refinement step. To separate the two effects, we trained the model without the clock embedding (100h, test-clean/test-other WER, greedy).

SupervisionClockWER
Dense (c=1)off11.95/32.77
Sparse (c=4)off11.57/32.12
Sparse (c=4)on (LARM)11.34/31.84

Sparse supervision helps on its own, and the clock adds a further gain. The two mechanisms work together, and Figure 3(b) shows their combined effect.

Baselines extended to 960h. We trained the two closest method baselines at 960h with the same recipe as our 960h models: the folded encoder [19] (R=12) and self-conditioned CTC (16 layers, with the learning rate selected at 100h). Test-clean WER at the end of the same 50-epoch schedule:

ModelParamsTest-clean WER
Folded encoder [19], R=127.65M4.98
Self-conditioned CTC, 16 layers28.9M4.30
LARM d=3847.7M4.59
LARM d=76829.5M3.45

At the same parameter count, LARM d=384 outperforms the folded encoder (4.59 vs 4.98), and LARM d=768 outperforms self-conditioned CTC (3.45 vs 4.30).

The Conformer control confirms our findings. We trained a naive-looped Conformer at 100h, using the LARM-Conformer loop with the paper's naive-recurrence settings (no posterior feedback, CTC at every loop instead of clock-scheduled checkpoints). It reaches 11.74 test-clean WER, against 11.27 for LARM-Conformer and 17.24 for the parameter-matched 4-block Conformer. As for Transformers, weight reuse brings the largest gain, and LARM's mechanisms add a further improvement on top of it.

  • [1] Y. Zhao et al., Universal Speech Transformer, Interspeech 2020.
  • [2] M. H. Jang et al., BiCycle: Group-wise Recursive Transformer Based on ASR Mechanism, AAAI 2026.
  • [3] S. Li et al., Improving Transformer-Based Speech Recognition Systems with Compressed Structure and Speech Attributes Augmentation, Interspeech 2019.
  • [4] Z. Gao et al., Extremely Low Footprint End-to-End ASR System for Smart Device, Interspeech 2021.
  • [5] S. M. Hernandez et al., Sharing Low Rank Conformer Weights for Tiny Always-On Ambient Speech Recognition Models, ICASSP 2023.
  • [6] Y. Wang and J. Li, ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers, ICASSP 2024.
  • [7] T. Rolland and A. Abad, Exploring Shared-Weight Mechanisms in Transformer and Conformer Architectures for Automatic Speech Recognition, Interspeech 2025.
  • [8] H. Tang et al., Beyond Universal Transformer: Block Reusing with Adaptor in Transformer for Automatic Speech Recognition, arXiv
  • [9] S. Bae et al., Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA, ICLR 2025.
  • [10] Y. Bai et al., Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognition, Interspeech 2022.
  • [11] Z. Li et al., Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models (Foldable Network), Interspeech 2025.
  • [12] Y. Shi et al., Dynamic Encoder Transducer: A Flexible Solution for Trading Off Accuracy for Latency, Interspeech 2021.
  • [13] Y. Peng et al., I3D: Transformer Architectures with Input-Dependent Dynamic Depth for Speech Recognition, ICASSP 2023.
  • [14] M. Lasbordes et al., Splitformer: An Improved Early-Exit Architecture for Automatic Speech Recognition on Edge Devices, EUSIPCO 2025.
  • [15] G. A. Wright et al., Training Early-Exit Architectures for Automatic Speech Recognition: Fine-Tuning Pre-Trained Models or Training from Scratch, ICASSP Workshops 2024.
  • [19] T. Komatsu, Non-Autoregressive ASR with Self-Conditioned Folded Encoders, ICASSP 2022.
Reviewer_kJ7uOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposed LARM, which applies a shared stack of four Transformer blocks for up to KK recurrent loops. CTC loss is applied only at periodic checkpoints. A supervision-clock embedding marks position within the checkpoint cycle, FiLM conditions the representation on absolute loop depth, and a delayed projection of the previous loop's soft CTC posterior is fed into the recurrent state. Any supervised checkpoint within the trained loop budget can serve as an inference exit.

On LibriSpeech 100 h, the 7.7M-parameter LARM reduces greedy test-clean/test-other WER to 11.34/31.84, compared with 12.31/33.30 for a parameter- and block-pass-matched self-conditioned folded encoder and 12.24/34.06 for an 85.7M-parameter 48-block encoder. On 960 h, the 7.7M model reaches 4.59/11.75. Wider LARM models reach 3.45/9.44 at 29.5M parameters and 3.39/9.01 at 51.9M, outperforming the 85.7M 48-block reference at 3.87/10.58. LARM-Conformer reaches 3.66/9.71 versus 5.48/15.01 for its four-block parameter-matched reference.

The ablations show 12.70 WER for naive recurrence versus 11.34 for full LARM; removing feedback gives 12.31, removing depth conditioning gives 12.13, and dense or final-only supervision is worse than the periodic schedule. Cost measurements show nearly constant parameter memory across exits, while compute and latency grow with executed loops.

Critical strengths
  1. Clear technical idea. Sparse recognition checkpoints separated by latent-refinement loops are a sensible way to make repeated shared computation non-redundant.

  2. Strong common-pipeline baselines. The paper reimplements intermediate CTC, self-conditioned CTC, a self-conditioned folded encoder, Align-Refine, and fixed-depth encoders with a shared frontend, vocabulary, augmentation, and decoding setup.

  3. Compelling parameter and memory efficiency. At matched block passes, LARM substantially reduces parameter count and peak memory while matching or improving WER. The wider 29.5M LARM is particularly persuasive relative to the 85.7M fixed-depth encoder.

  4. Useful component ablations. Naive recurrence, multiple depth-conditioning mechanisms, feedback variants, learned aggregation, loop budgets, and supervision intervals are all examined.

  5. Backbone transfer. The Conformer result reduces the risk that the mechanism is specific to a plain Transformer implementation.

  6. Good cost reporting. Parameters, block passes, GFLOPs, memory, latency, greedy WER, and LM-decoded WER are reported together rather than collapsing efficiency to parameter count.

Critical weaknesses
  1. The primary efficiency gain is in parameters and memory, not computation. The reference LARM costs 47.92 GFLOPs, essentially matching the 48-block encoder at 47.78 GFLOPs, while the narrow 960 h LARM is less accurate at the final exit (4.59 versus 3.87 test-clean WER). The wider LARM's stronger result is valuable, but it reflects better parameter allocation plus recurrence rather than free gains from additional inference compute.

  2. External validity is limited to LibriSpeech. One English read-speech benchmark cannot establish behavior under noise, conversational speech, accents, code-switching, domain shift, long-form audio, or streaming constraints. The posterior-feedback mechanism may also depend on CTC calibration and the character-token setup.

  3. Exit quality is not uniformly monotonic across decoding configurations. At 100 h with the 4-gram LM, loop 7 is the best exit and is significantly better than the final loop despite per-exit LM tuning. This complicates the interpretation that later loops always refine recognition and leaves the interaction between recurrent acoustic states and external decoding unexplained.

  4. Performance not competitive. The overall performance on librispeech is not competitive compared to other CTC-based ASR works. However, the concern is minor as this is the first attempt for looped transformer as ASR encoder.

Questions for authors
  1. Under matched parameter, FLOP, and tuning budgets, how does LARM compare with a shallower, wider fixed-depth encoder, and which baseline hyperparameters were independently retuned?

  2. How would a calibrated stopping rule based on CTC confidence, posterior stability, or checkpoint agreement affect WER and expected loops per utterance?

  3. How do CTC entropy, posterior stability, substitutions, deletions, and LM corrections evolve across loops, particularly around the 100 h loop-7 optimum?

  4. Does the advantage persist on noisy or conversational speech, a non-English language, subword CTC, and long-form or streaming recognition?

3 comments
AuthorsOpenReview ↗

Response to Reviewer kJ7u (1/2)

We thank the reviewer for the careful reading. We answer the questions below, grouped with the corresponding weaknesses.

Q1/W1. Compute versus parameters; matched budgets.

LARM is designed to separate parameter count from compute: each shared parameter is used K=12 times in a forward pass. In a fixed-depth encoder, parameters and FLOPs both grow with depth x width^2, so matching both at once requires removing the loops (K=1). The paper therefore reports the two controls separately. The 4-block encoder matches LARM's parameters (7.6M vs 7.7M), and the 48-block encoder matches its FLOPs (47.78 vs 47.92 GFLOPs). All baselines share LARM's optimizer, schedule, augmentation and decoding. The learning rate of each baseline was selected separately from several values.

In our early work, we did explore other width x depth trade-offs, including the shallower, wider fixed-depth encoder the reviewer suggests (16 layers, d=768). We did not include it in the paper because this exploratory run was stopped at epoch 35 of 50. The results below compare all models at that epoch (LibriSpeech 960h, test-clean WER; GFLOPs for 10 s of audio):

Fixed-depth encoders:

  • 16 layers, d=768 (approximately 113M, approximately 61 GFLOPs): 4.71
  • 16 layers, d=384 (28.9M, 16 GFLOPs): 5.48

LARM:

  • d=384 (7.7M, 48 GFLOPs): 5.13
  • d=768 (29.5M, 180 GFLOPs): 4.00

At similar compute, the wider fixed-depth encoder is more accurate than LARM d=384 (4.71 vs 5.13), but it needs 15 times more parameters. At matched parameters, LARM d=768 is clearly better than the 16-layer d=384 encoder (4.00 vs 5.48). It also outperforms the wider encoder with 4 times fewer parameters, at the cost of more FLOPs. Together with the memory measurements in Table 2, these results again confirm LARM's parameter and memory advantages.

Q2. Stopping rules.

Early stopping is indeed a natural next step of this work. Two training-free strategies already save compute; the table compares them with fixed exits and with an oracle, for the 960h LARM d=384 model on test-clean:

StrategyAvg. loopsLoops savedWER
Full budget (exit 12)12–4.59
Fixed exit 8833%4.73
Agreement: stop at exit 8 if it matches exit 410.016%4.58
Stabilisation: stop when two consecutive loops agree5.852%4.97
Oracle (uses the reference): earliest supervised exit with no more errors than the final one5.554%4.14

The agreement rule saves 16% of the loops at no cost in WER, and the stabilisation rule saves 52% at some cost in accuracy. The oracle, which uses the reference transcript, gives an upper bound: more than half of the loops could be saved without any loss (43% on test-other). We will add these stopping rules and the oracle analysis to the final paper. A calibrated or learned stopping rule could close the remaining gap, which we leave to future work.

Q3 / W3. Behaviour across loops and the 100h loop-7 optimum.

The table follows the reference LARM (d=384, 100h) at the supervised exits (4, 8, 12) and at loop 7, with greedy decoding and with the 4-gram LM. All values are computed over the whole test-clean set.

S/D/I: substitution/deletion/insertion rates (%), which sum to the WER;

LM +/-: share of utterances that the LM improves / degrades compared with greedy decoding;

Changed: share of utterances whose greedy hypothesis differs from the previous loop.

LoopGreedy WER (S/D/I)LM WER (S/D/I)LM +/-Changed
415.20 (12.70/1.21/1.30)8.56 (6.72/0.84/1.01)69% / 1.8%92%
712.13 (10.05/0.99/1.08)8.08 (6.41/0.73/0.94)54% / 3.0%48%
811.57 (9.56/0.89/1.12)8.11 (6.43/0.71/0.98)50% / 3.2%45%
1211.34 (9.37/0.88/1.09)8.35 (6.63/0.74/0.98)46% / 2.8%30%

Greedy decoding reflects the encoder alone, and its WER decreases at every supervised exit, with substitutions, deletions and insertions all decreasing. The predictions also stabilise, and the share of hypotheses that change from the previous loop drops from 92% to 30%. The loop-7 optimum comes from the interaction with the external LM; the rise after loop 7 is mostly substitutions, and the share of utterances the LM improves keeps falling while the share it degrades stays higher than at loop 4. As the loop refines the acoustic predictions, fewer errors are left for the LM to fix, and the LM increasingly overrides correct acoustic decisions. The effect vanishes with more data. At 960h, the LM curve is flat from loop 7 (3.50-3.52; p = 0.90 and 0.68 against loop 12).

AuthorsOpenReview ↗

Response to Reviewer kJ7u (2/2)

Q4 / W2. Beyond read English speech and character-level tokens.

Non-English, conversational, long-form and streaming recognition are outside the scope of this paper. Since the loop wraps the encoder block, as the Conformer results show, chunked or streaming blocks are a natural extension that we leave to future work. We tested the other conditions raised by the reviewer. The table below compares LARM with the self-conditioned folded encoder of Komatsu [1], the closest prior method, with its base encoder (base+folded, R=6 repeats) or without it as in the paper (folded, R=12 repeats). We report results with a second tokenizer (BPE-500) on LibriSpeech test-clean/test-other, and on a second corpus, Common Voice, trained as in the paper (character tokens). WER is given with greedy decoding, with the 4-gram LM in parentheses for LibriSpeech.

SettingBaseline modelBaselineLARM
LS 100h, charactersbase+folded R=6 (21 block passes)13.04/34.45 (8.96/27.17)11.34/31.84 (8.66/26.28)
LS 100h, BPE-500base+folded R=6 (21 block passes)12.53/32.96 (9.21/26.48)10.71/29.72 (8.53/25.63)
Common Voice EN, 100hfolded R=1243.1142.74

With BPE-500, LARM remains significantly better than the baseline on all four LibriSpeech metrics (p < 0.0001), showing that its advantage is not restricted to character-level tokenization. On Common Voice (many accents and recording conditions), LARM is also better than the folded encoder (p = 0.0014).

We hope these clarifications and additional results address the reviewer's concerns, and we remain available for any further question during the discussion period.

[1] T. Komatsu, Non-Autoregressive ASR with Self-Conditioned Folded Encoders, ICASSP 2022.

AuthorsOpenReview ↗

Follow-Up on Our Rebuttal Responses

Dear Reviewer kJ7u,

We appreciate your feedback on our submission. In response to your comments, we have presented new experimental results and provided additional clarifications. As the discussion period is coming to an end, we would appreciate knowing whether our responses have addressed your concerns.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their detailed reviews. They note the careful common-pipeline comparisons, the parameter and memory efficiency of LARM, and the usefulness of serving several inference budgets from one checkpoint. We answered each reviewer individually and summarise the new results below.

Major points

Related work. We provide a detailed comparison with the Universal Speech Transformer, BiCycle and the weight-shared, block-specialisation and early-exit ASR lineages, which we will add to the paper.

Method baselines at 960h. At the same parameter count, LARM d=384 outperforms the folded encoder (4.59 vs 4.98 test-clean WER, 7.7M parameters), and LARM d=768 outperforms self-conditioned CTC (3.45 vs 4.30, 29.5M parameters). The 100h conclusions therefore hold at 960h.

Closest prior methods in our pipeline. The base-plus-folded configuration of the original folded encoder is significantly worse than LARM under greedy decoding (p ≤ 0.014) despite 31% more parameters. The unfolding method of Li et al. (Interspeech 2025), trained with the same loop, reaches 14.48 test-clean WER against 11.34 for LARM.

Supervision clock. Removing the clock embedding isolates the two mechanisms. Dense supervision without clock gives 11.95, sparse supervision without clock 11.57, and LARM 11.34 (100h test-clean). Sparse supervision helps on its own, and the clock adds a further gain.

Conformer control. A naive-looped Conformer reaches 11.74, against 11.27 for LARM-Conformer and 17.24 for the parameter-matched 4-block Conformer. Weight reuse brings the largest gain, and LARM's mechanisms add a further improvement on top of it.

Minor points

Realistic configurations. With a BPE-500 vocabulary, LARM remains significantly better than the baseline on all four LibriSpeech metrics (p < 0.0001). On Common Voice, LARM is also significantly better than the folded encoder (p = 0.0014).

Latency and throughput. At matched FLOPs, LARM d=384 has the same batched throughput as the 48-layer encoder (578 vs 591 utt/s). At batch 1, LARM d=768 at exit 8 is both faster (17.6 vs 19.0 ms) and more accurate (3.46 vs 3.87) than the 48-layer encoder, with 3 times fewer parameters.

Adaptive inference. Training-free stopping rules already save compute, for example 16% fewer loops at unchanged WER, and an oracle analysis shows that 54% of the loops could be saved without loss. We will add this analysis to the paper.

What the loops do. A per-loop analysis shows that substitutions, deletions and insertions decrease across the supervised exits and that hypotheses stabilise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.