- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LARM (Loop Audio Recurrent Model), a depth-conditioned looped Transformer for budget-adaptive ASR inference. LARM applies a shared Transformer encoder recurrently to a latent acoustic sequence, decoupling computational depth from parameter count. It combines sparse CTC supervision, supervision clock embeddings, FiLM depth conditioning, and delayed prediction feedback, where soft CTC posteriors are shifted by one frame and reinjected into the recurrent state. These components structure the loop into supervised recognition checkpoints separated by intermediate refinement steps and allow shared weights to specialize across recurrent iterations. A single trained model can stop at any supervised checkpoint within its trained loop budget. Experiments on LibriSpeech demonstrate improved greedy WER over closely related self-conditioning and recurrent CTC methods, with fewer parameters and less peak memory than deeper unshared encoders. The design also transfers to a Conformer backbone.
Critical strengths
1. Parameter efficiency. LARM achieves competitive recognition accuracy with fewer parameters.
2. Experimental evaluation. The paper provides relevant baseline comparisons, ablation studies, and an additional Conformer evaluation, supporting the effectiveness of the overall approach.
Critical weaknesses
1. Insufficient differentiation from prior work. The paper presents budget-adaptive inference and prediction feedback as major contributions, although [1] already employs recurrent CTC posterior feedback and explores varying decoding iterations, while [2] supports multiple inference depths in shared-parameter ASR models. In particular, LARM introduces a one-frame delay to the current-frame feedback used in [1], but Table 4(c) shows no statistically significant improvement (p = 0.51). Moreover, LARM lacks a direct comparison with [2], and comparisons with related recurrent ASR methods are limited to LibriSpeech 100h, where LARM and [1] achieve statistically similar WER with LM decoding. These limitations make it difficult to establish LARM's contributions over existing approaches.
2.Incomplete validation of the proposed mechanisms. In Figure 3(b), varying the CTC supervision interval also changes the supervision-clock embeddings. Therefore, the reported improvements cannot be attributed solely to sparse supervision. Moreover, the supervision-clock embedding is not independently ablated, leaving the individual contributions of sparse supervision and clock conditioning unclear.
[1] T. Komatsu, "Non-Autoregressive ASR with Self-Conditioned Folded Encoders," ICASSP, 2022.
[2] Z. Li et al., "Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models," Interspeech, 2025.
Questions for authors
Q1. What does LARM contribute beyond the recurrent feedback and flexible-depth inference already explored in [1, 2]?
Q2. How does LARM compare with existing recurrent and self-conditioned ASR methods when trained on LibriSpeech 960h? The current comparisons with these methods are limited to 100h.
Q3. What is the independent contribution of the supervision clock embedding? How does varying the CTC supervision interval affect WER when the clock embedding is disabled?
Q4. Given that [1] already uses current-frame CTC feedback, what evidence supports the benefit of LARM's proposed one-frame delay? Table 4(c) shows no statistically significant improvement, although the authors claim better results in most experiments during model design. What results support this?
5 comments
Follow-Up on Our Rebuttal Responses
Dear Reviewer M2ji,
We appreciate your feedback on our submission. In response to your comments, we have presented new experimental results and provided additional clarifications. As the discussion period is coming to an end, we would appreciate knowing whether our responses have addressed your concerns.
Best regards,
The Authors
I appreciate the authors' detailed response and additional experiments, which largely address my concerns regarding Q2 and Q3. However, my concerns regarding Q1 and Q4 remain.
Q1 / W1. Differentiation from prior work
The rebuttal clarifies that LARM's main methodological distinctions lie in sparse supervision, supervision-clock embeddings, FiLM depth conditioning, and direct supervision of intermediate exits. However, the manuscript also presents budget-adaptive inference and prediction feedback as separate main contributions. This inconsistency extends beyond the contribution list to the overall framing of the paper, suggesting that substantial revisions are needed to accurately reflect the contributions clarified in the rebuttal.
Q4 / W1. One-frame delayed feedback
My concern is not limited to the statement about design-phase experiments. The manuscript repeatedly emphasizes delayed feedback as a key mechanism throughout the Introduction, Method, and Conclusion, potentially misleading readers about its actual contribution. Removing that statement alone does not address this broader concern.
Additional Question
These issues appear to require substantial revisions to the manuscript's overall positioning and presentation of its contributions. Could the authors explain how they plan to revise the manuscript to address these concerns?
Response to Reviewer M2ji
We thank the reviewer for the careful review. We answer the questions below, grouped with the corresponding weaknesses.
Q1 / W1. Contribution beyond [1] and [2].
The folded encoder of Komatsu [1] is the closest method. It applies a small stack of shared blocks repeatedly, with a CTC loss and self-conditioning at every repeat, and it can vary the number of repeats at decoding time. LARM differs in three ways. First, its supervision is sparse. CTC checkpoints are placed every few loops and scheduled by a supervision clock, which leaves the loops in between free to refine the representation. These are our two new mechanisms. Second, the shared stack is conditioned on the loop depth through FiLM, whereas [1] applies the same stack identically at every repeat. Third, the goal is different. [1] targets a compact encoder used at its training depth, while LARM is designed so that each supervised checkpoint is a usable operating point.
The method of Li et al. [2] unfolds a small seed model by repeating its layers, in order to emulate a larger model. It is trained with the ASR loss on the deepest unfolded system and on the seed, together with a self-distillation term between the two. The intermediate depths receive no supervision of their own, and the shared layers receive no information about their depth. Its goal is the compression of a larger model. LARM instead trains every exit directly with CTC, without a teacher, and conditions the shared stack on its depth, with the goal of serving several inference budgets from one checkpoint.
These differences have a practical effect. At matched block passes, the exits of LARM outperform those of [1] at every budget, and most at small budgets (100h test-clean WER at 16 / 32 / 48 block passes: 15.20 / 11.57 / 11.34 for LARM vs 19.99 / 13.23 / 12.31 for [1]). We also trained the method of [2], adapted to our pipeline with the same loop (4 seed blocks, each repeated up to 12 times). At 48 block passes, it reaches 14.48 test-clean WER, against 11.34 for LARM. We also ran the base-plus-folded architecture of [1], which precedes the folded blocks with unshared base blocks. At matched block passes, LARM is significantly better under greedy decoding (11.34/31.84 vs 11.67/32.61 test-clean/test-other, p ≤ 0.014) with 31% fewer parameters, and the two are tied under LM decoding (8.66/26.28 vs 8.45/26.17).
Q2 / W1. Comparisons at 960h.
We trained [1] (R=12) and self-conditioned CTC (16 layers) at 960h with the same recipe as our 960h models. The table below reports the test-clean WER at the end of the same 50-epoch schedule.
At the same parameter count, LARM d=384 outperforms [1] (4.59 vs 4.98), and LARM d=768 outperforms self-conditioned CTC (3.45 vs 4.30). The 100h results therefore hold at 960h.
Q3 / W2. Supervision clock.
The clock embedding is part of the sparse-supervision design. It was built to schedule the sparse checkpoints, telling the shared stack whether the current loop is a recognition checkpoint or a latent refinement step. To separate the two effects, we trained the model without the clock embedding (100h, test-clean/test-other WER, greedy).
Sparse supervision helps on its own, and the clock adds a further gain. The two mechanisms work together, and Figure 3(b) shows their combined effect.
Q4 / W1. One-frame delay.
As reported in Table 4(c), the paper already acknowledges that the delay is not significant (p = 0.51). The delay is the only such case: prediction feedback, depth conditioning and the sparse supervision schedule each bring statistically significant gains in the paper's ablations. The remark on design-phase experiments refers to a few runs and is not a claim; we will remove it to avoid any ambiguity.
We hope we have addressed the reviewer's concerns and would be happy to provide any additional details as needed. We sincerely hope the reviewer will consider revising the score in light of the clarifications provided.
[1] T. Komatsu, Non-Autoregressive ASR with Self-Conditioned Folded Encoders, ICASSP 2022.
[2] Z. Li et al., Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models, Interspeech 2025.