- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper addresses the problem of force alignment, focusing on in-the-wild conditions (e.g., when speech is not clean and without background noise). The two main contributions are 1) the benchmark, AlignBench, which spans diverse speaker, acoustic, and text conditions, and has been manually revised; 2) the evaluation framework, including newly proposed metrics, and 3) a Transformer-based aligner, FuseAlign, able to outperform the analyzed baselines.
Critical strengths
- The paper is a very nice reading, clear and essential.
- The general problem of forced alignment is relevant in the speech community, and rarely leveraged in-the-wild data.
- The proposed evaluation framework allows for a comprehensive analysis of forced aligners.
- The FuseAlign model outperforms all the other models on the new benchmark, and partially on TIMIT and Buckeye.
Critical weaknesses
- The size of the test set is pretty small, it is less than 1-hour. I understand the effort of manually revising those data but speech test sets are, in general, at least 1-2 hours. In fact, TIMIT, which is the most largely used corpus, is more than 5 hours. Also Buckeye is above 20 hours.
- The benchmark and the aligner are tested on English only. At least, this is what can be derived from the text, since it is never explicitly written (which, in my opinion, should be clarified since the beginning, in the Abstract and Intro).
- In lines 212-3, it is written that one main weakness of existing benchmarks is that they underreperesent long-form speech. However, AlignBench is not long-form at all (see questions), therefore this weakness is not addressed with this work.
- The audio collected for the benchmark is sourced from existing corpora, the most concerning aspect is that Voxpopuli is often used as training set [1,2], since there is no test split. The risk of data contamination is high.
- I'm not fully convinced that a comparison with a highly multilingual model such as WhisperX, derived from Whisper, is a fair one as FuseAlign is trained and tested on English only. However, this aspect is never discussed.
- The ablation, besides the concerning detail about results variations under different batch sizes, is not really focused on the analyzed phenomenon but on the architectural choices, making it more a content for the Appendix rather than for the main text. Problem-focused ablations would have been those isolating different speech phenomena (e.g., presence or absence of background noise, presence or absence of difficult words), which are much more appropriate for this work. More in general, the in-the-wild conditions are presented as the main paper positioning but never appropriately analyzed.
References
[1] Rao Koluguri, N., Sekoyan, M., Zelenfroynd, G., Meister, S., Ding, S., Kostandian, S., Huang, H., Karpov#, N., Balam, J., Lavrukhin, V., Peng, Y., Papi, S., Gaido, M., Brutti, A., Ginsburg, B. (2025) Granary: Speech Recognition and Translation Dataset in 25 European Languages. Proc. Interspeech 2025, 3923-3927
[2] Marco Gaido, Sara Papi, Luisa Bentivogli, Alessio Brutti, Mauro Cettolo, Roberto Gretter, Marco Matassoni, Mohamed Nabih, and Matteo Negri. 2024. MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13934–13947, Miami, Florida, USA. Association for Computational Linguistics.
Questions for authors
- What happens if the ASR, instead of "under transcribing", hallucinates (e.g., repeats the same word or group of words)? How does FuseAlign address it?
- What is the average duration of the clips contained in the benchmark? Because, if I compute this statistic from Table 1, it does not seem to me that the this benchmark is long-form
- It is mentioned that, within the difficult words, there are also URLs in the benchmark: does this mean that URLs are spelled out in the audio? What about symbols?
- There are many audio clips released under a permissive license, why did the authors select PeopleSpeech and Voxpopuli despite being commonly used as training data?
- Why did the authors choose to collect English only? Voxpopuli contains all EU languages.
- What is the performance of FuseAlign outside English language?
- Why do the results should change in presence of a different batch size? This raises some concerns about the implementation
- What is the performance of the various models on the "difficult words" described in Table 1?
3 comments
I thank the authors for their replies, and for the corrections applied to the paper. While I understand the cost efforts, I still believe that the dataset is too small to be representative, especially when already available datasets in the literature with similar characteristics cover hours of data in the same language (e.g., TIMIT). Moreover, I find very risky to use Voxpopuli and, in general, datasets usually used to train models (e.g., NVIDIA Canary) for a test set, as even if not used for the model proposed in the paper, FuseAlign, which I find a nice contribution, these training sets will be likely used by other models, as they are intended as training sets. Since the source language was English, which is the highest resourced language, finding a suitable test set with English source is very easy (e.g., ACL 6060, MCIF) and would have been a much more sound decision for building a benchmark. Therefore, since the benchmark is the first contribution of this work and I have several concerns about its adoption, I leave my scores unchanged.
We thank the reviewer for the follow-up and for the kind words about FuseAlign. We would like to emphasize that the benchmark is one of three contributions, alongside the evaluation framework and the FuseAlign model, and we believe the latter two add value independently of AlignBench. FuseAlign, trained without any TIMIT or Buckeye data, is competitive on both. On Buckeye it has the lowest catastrophic-error rate (0.40% of words above 300 ms, against 0.59–1.98% for the baselines) and the lowest rate above 50 ms (10.52%, against 12.45–64.76%). On TIMIT, its catastrophic-error rate (0.02%) is within 0.01 points of the best system. On clean, read TIMIT speech, the HMM-based Nyra remains more precise at fine tolerances, which we report in the paper.
Contamination: We agree this deserves care and will discuss it in the revised version of the paper. However, the suggested alternatives are not contamination-free either. ACL talks are already distributed as training data: NUTSHELL [1] releases 1,172h of *ACL talk audio with an 808h training split, used as official training data in the IWSLT 2025 instruction-following track [2]. The ACL 60/60 development set has been used for ASR fine-tuning [3]. MCIF [4] avoids contamination by selecting the most recent talks available at collection time, which holds only until newer training sets include them. Both ACL 60/60 [5] and MCIF [4] are also single-presenter talks; they therefore lack the multi-speaker conditions where aligners fail most, as shown in our ablations.
Size: AlignBench has 8,072 words and 73 unique speakers, with every boundary triple-annotated. This is comparable to the suggested benchmarks (ACL 60/60 eval: 59 min, 5 speakers [5]; MCIF: 2h, 21 speakers [4]), with more speakers than both. Its representativeness comes from design: clips match the distribution of a real editing workload, and the selection was frozen before any aligner was run. System gaps are large (3 to 11× in catastrophic error), and we report results on TIMIT [6] and Buckeye [7] alongside it.
Community value: Public English corpora with hand-placed word boundaries are rare; the field still relies on TIMIT (1993) [6] and Buckeye (2007) [7]. To our knowledge, none covers accented, multi-speaker, in-the-wild speech. We release all audio, annotations, ASR transcripts and code so others can extend the benchmark. We see this as a first step, and believe any such contribution moves the community in the right direction.
References
[1] Maike Züfle, Sara Papi, Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Jan Niehues. NUTSHELL: A Dataset for Abstract Generation from Scientific Talks. IWSLT 2025, pp. 19–32. https://aclanthology.org/2025.iwslt-1.2/
[2] IWSLT 2025 Instruction-Following Speech Processing Track, data conditions. https://iwslt.org/2025/instruction-following
[3] Henry Li Xinyuan, Neha Verma, Bismarck Bamfo Odoom, Ujvala Pradeep, Matthew Wiesner, Sanjeev Khudanpur. JHU IWSLT 2023 Multilingual Speech Translation System Description. IWSLT 2023, pp. 302–310. https://aclanthology.org/2023.iwslt-1.28/
[4] Sara Papi, Maike Züfle, Marco Gaido, Beatrice Savoldi, Danni Liu, Ioannis Douros, Luisa Bentivogli, Jan Niehues. MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks. ICLR 2026. arXiv:2507.19634.
[5] Elizabeth Salesky, Kareem Darwish, Mohamed Al-Badrashiny, Mona Diab, Jan Niehues. Evaluating Multilingual Speech Translation under Realistic Conditions with Resegmentation and Terminology. IWSLT 2023, pp. 62–78. https://aclanthology.org/2023.iwslt-1.2/
[6] John S. Garofolo et al. TIMIT Acoustic-Phonetic Continuous Speech Corpus. LDC93S1, Linguistic Data Consortium, 1993.
[7] Mark A. Pitt et al. Buckeye Corpus of Conversational Speech (2nd release). Ohio State University, 2007.
We thank the reviewer for a thorough review, which exposed places where the paper was unclear or overclaimed.
W1: Test set size
We agree that AlignBench is small; annotation cost was the limit. Placing boundaries by hand takes about 50 annotator-hours per hour of audio, and every boundary was annotated three times, giving inter-annotator agreement and a median reference. Hence, our focus was on quality. AlignBench has 8,072 words. We report 95% confidence intervals and the larger TIMIT (5.38 h) and Buckeye (22.31 h).
W2: English only
Correct: all analysis, benchmark construction and training are English-only, and we now state this in the abstract, introduction and limitations.
W3: Not long-form
We thank the reviewer for catching this; "long-form" was the wrong word. Clips last 41.0 s on average (median 53.2 s, range 7.9–71.6 s). 63% of clips exceed 30 s, the input limit of most neural aligners, so they exercise chunking and long-input behavior, but they are not hour-long recordings. We have replaced "long-form" with "multi-sentence recordings longer than a typical aligner's context window" throughout.
W4: Contamination
FuseAlign's training data contains no VoxPopuli, People's Speech or AMI audio, so contamination could only favor baselines trained on them (e.g. Nyra, trained partly on AMI). The targets, human word boundaries, are new and appear in no prior corpus. Each clip's source is recorded, so aligners trained on VoxPopuli can be evaluated on the 46 other clips. We now cite [1, 2].
W5: WhisperX
We use only WhisperX's English wav2vec 2.0 alignment model (
language_code='en') with our transcript; the multilingual Whisper recognizer plays no part. We now say so in the revised text.W6: Ablations
Appendix C.3 breaks AlignBench down by accent, speaker count and difficult words: multi-speaker clips are hardest for every system, and difficult words hurt lexicon-based systems most. We will move this into the main text and the architectural ablations to the appendix, and add slices by the SNR, reverberation, bandwidth, speaking-rate and disfluency tags. On batch size, see Q7.
Q1: Hallucinations
Whisper transcripts on AlignBench contain such hallucinations, including repetitions, and training simulates insertions with words from the same chunk. FuseAlign may mark either of two identical copies as not found, but because it localizes every word independently, a hallucinated word cannot displace its neighbours. Monotonic decoders such as MFA must fit every word into one path: moving from hand-corrected to Whisper transcripts raises MFA's >300 ms rate from 3.84% to 8.41%, while FuseAlign's rises from 0.38% to 1.71%, and its 25 ms error rate is unchanged (29.82% vs 29.77%).
Q2: Duration
Clips average 41.0 s, so AlignBench is not long-form, and we have corrected the wording; see W3.
Q3: URLs and symbols
Thank you for catching this. The category list came from our tagger and overstates the content: AlignBench has no URLs or email addresses. Its 719 difficult words include 362 named entities, 277 contractions, 47 numbers, 45 hyphenated compounds, 22 acronyms and 14 abbreviations (categories overlap). Symbols appear only in written forms such as "$519,018" and "15%"; text normalization converts these to the spoken form before alignment (Appendix D). We have corrected the list in the paper.
Q4: Corpus choice
We agree this is not ideal, but it was the best option we found. We needed publicly redistributable audio without share-alike terms (People's Speech is CC BY, CC0 or public domain; VoxPopuli is CC0) that covers our target conditions: multi-speaker, accented and conversational speech. Most alternatives, such as LibriSpeech and Common Voice, are read speech, and any corpus that is free to redistribute is also likely to be used by others for training. On contamination, see W4.
Q5: Why English
Our target editing workloads are English, and a distribution-matched benchmark in another language needs that language's target distribution and calibrated annotators. Extending AlignBench is future work; we expect the architecture to extend to other languages given enough pseudo-labelled data.
Q6: Other languages
We evaluate English only and make no claims about other languages; see W2.
Q7: Batch size
Ablations train for a fixed 100k steps, so the batch size 64 run sees half the audio (about 39,000 vs 78,000 hours), and a small change is expected due to differences in the training dynamics. Every metric stays within the baseline's 95% confidence interval (17.21 ± 1.92 vs 17.16 ± 1.92 ms mean error; 0.37% vs 0.38% catastrophic), so the recipe is not sensitive to batch size.
Q8: Difficult words
Difficult words raise the >300 ms rate most for the lexicon-based Gentle (1.96% to 3.62%) and MFA (3.66% to 5.70%), while FuseAlign barely changes (0.38% to 0.42%). Appendix C.3 (Table 9) covers every system and moves to the main text (W6).