acceptodds
Under review as a conference paper at ICLR 2027

FuseAlign: Forced Alignment in the Wild

Abstract

Word-level forced alignment estimates when each transcript word occurs in an audio recording. It underpins text-based media editing, subtitling, speech-data curation, and phonetic analysis. Existing evaluations understate the difficulty of forced alignment by relying on short, clean speech, perfect transcripts, and metrics that obscure consequential alignment errors. In contrast, real-world media and data-processing pipelines operate on long and diverse recordings. Additionally, forced aligners often operate on error-prone automatic speech recognition (ASR) output. We address these gaps with improved evaluation metrics, a scoring protocol for real ASR transcripts, and AlignBench, a benchmark spanning diverse speaker, acoustic, and text conditions. We further introduce FuseAlign, a transformer-based aligner trained on large-scale pseudo-labeled speech with online label correction. FuseAlign performs joint contextualization of audio and text for the localization of coarse words. The model then refines boundaries at millisecond resolution and detects missing transcript words in the audio without lexicon-based or Viterbi decoding. On AlignBench, FuseAlign substantially outperforms all baselines and remains robust under real ASR transcripts. Ablations show that convolutional upsampling and EMA-snapshot label correction matter more than model properties such as parameter count.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
forced alignment, word-level timestamps, audio-text alignment, speech processing, speech benchmarks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7yd8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses the problem of force alignment, focusing on in-the-wild conditions (e.g., when speech is not clean and without background noise). The two main contributions are 1) the benchmark, AlignBench, which spans diverse speaker, acoustic, and text conditions, and has been manually revised; 2) the evaluation framework, including newly proposed metrics, and 3) a Transformer-based aligner, FuseAlign, able to outperform the analyzed baselines.

Critical strengths
  • The paper is a very nice reading, clear and essential.
  • The general problem of forced alignment is relevant in the speech community, and rarely leveraged in-the-wild data.
  • The proposed evaluation framework allows for a comprehensive analysis of forced aligners.
  • The FuseAlign model outperforms all the other models on the new benchmark, and partially on TIMIT and Buckeye.
Critical weaknesses
  • The size of the test set is pretty small, it is less than 1-hour. I understand the effort of manually revising those data but speech test sets are, in general, at least 1-2 hours. In fact, TIMIT, which is the most largely used corpus, is more than 5 hours. Also Buckeye is above 20 hours.
  • The benchmark and the aligner are tested on English only. At least, this is what can be derived from the text, since it is never explicitly written (which, in my opinion, should be clarified since the beginning, in the Abstract and Intro).
  • In lines 212-3, it is written that one main weakness of existing benchmarks is that they underreperesent long-form speech. However, AlignBench is not long-form at all (see questions), therefore this weakness is not addressed with this work.
  • The audio collected for the benchmark is sourced from existing corpora, the most concerning aspect is that Voxpopuli is often used as training set [1,2], since there is no test split. The risk of data contamination is high.
  • I'm not fully convinced that a comparison with a highly multilingual model such as WhisperX, derived from Whisper, is a fair one as FuseAlign is trained and tested on English only. However, this aspect is never discussed.
  • The ablation, besides the concerning detail about results variations under different batch sizes, is not really focused on the analyzed phenomenon but on the architectural choices, making it more a content for the Appendix rather than for the main text. Problem-focused ablations would have been those isolating different speech phenomena (e.g., presence or absence of background noise, presence or absence of difficult words), which are much more appropriate for this work. More in general, the in-the-wild conditions are presented as the main paper positioning but never appropriately analyzed.

References

[1] Rao Koluguri, N., Sekoyan, M., Zelenfroynd, G., Meister, S., Ding, S., Kostandian, S., Huang, H., Karpov#, N., Balam, J., Lavrukhin, V., Peng, Y., Papi, S., Gaido, M., Brutti, A., Ginsburg, B. (2025) Granary: Speech Recognition and Translation Dataset in 25 European Languages. Proc. Interspeech 2025, 3923-3927

[2] Marco Gaido, Sara Papi, Luisa Bentivogli, Alessio Brutti, Mauro Cettolo, Roberto Gretter, Marco Matassoni, Mohamed Nabih, and Matteo Negri. 2024. MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13934–13947, Miami, Florida, USA. Association for Computational Linguistics.

Questions for authors
  • What happens if the ASR, instead of "under transcribing", hallucinates (e.g., repeats the same word or group of words)? How does FuseAlign address it?
  • What is the average duration of the clips contained in the benchmark? Because, if I compute this statistic from Table 1, it does not seem to me that the this benchmark is long-form
  • It is mentioned that, within the difficult words, there are also URLs in the benchmark: does this mean that URLs are spelled out in the audio? What about symbols?
  • There are many audio clips released under a permissive license, why did the authors select PeopleSpeech and Voxpopuli despite being commonly used as training data?
  • Why did the authors choose to collect English only? Voxpopuli contains all EU languages.
  • What is the performance of FuseAlign outside English language?
  • Why do the results should change in presence of a different batch size? This raises some concerns about the implementation
  • What is the performance of the various models on the "difficult words" described in Table 1?
3 comments

We thank the reviewer for a thorough review, which exposed places where the paper was unclear or overclaimed.

W1: Test set size

We agree that AlignBench is small; annotation cost was the limit. Placing boundaries by hand takes about 50 annotator-hours per hour of audio, and every boundary was annotated three times, giving inter-annotator agreement and a median reference. Hence, our focus was on quality. AlignBench has 8,072 words. We report 95% confidence intervals and the larger TIMIT (5.38 h) and Buckeye (22.31 h).

W2: English only

Correct: all analysis, benchmark construction and training are English-only, and we now state this in the abstract, introduction and limitations.

W3: Not long-form

We thank the reviewer for catching this; "long-form" was the wrong word. Clips last 41.0 s on average (median 53.2 s, range 7.9–71.6 s). 63% of clips exceed 30 s, the input limit of most neural aligners, so they exercise chunking and long-input behavior, but they are not hour-long recordings. We have replaced "long-form" with "multi-sentence recordings longer than a typical aligner's context window" throughout.

W4: Contamination

FuseAlign's training data contains no VoxPopuli, People's Speech or AMI audio, so contamination could only favor baselines trained on them (e.g. Nyra, trained partly on AMI). The targets, human word boundaries, are new and appear in no prior corpus. Each clip's source is recorded, so aligners trained on VoxPopuli can be evaluated on the 46 other clips. We now cite [1, 2].

W5: WhisperX

We use only WhisperX's English wav2vec 2.0 alignment model (language_code='en') with our transcript; the multilingual Whisper recognizer plays no part. We now say so in the revised text.

W6: Ablations

Appendix C.3 breaks AlignBench down by accent, speaker count and difficult words: multi-speaker clips are hardest for every system, and difficult words hurt lexicon-based systems most. We will move this into the main text and the architectural ablations to the appendix, and add slices by the SNR, reverberation, bandwidth, speaking-rate and disfluency tags. On batch size, see Q7.

Q1: Hallucinations

Whisper transcripts on AlignBench contain such hallucinations, including repetitions, and training simulates insertions with words from the same chunk. FuseAlign may mark either of two identical copies as not found, but because it localizes every word independently, a hallucinated word cannot displace its neighbours. Monotonic decoders such as MFA must fit every word into one path: moving from hand-corrected to Whisper transcripts raises MFA's >300 ms rate from 3.84% to 8.41%, while FuseAlign's rises from 0.38% to 1.71%, and its 25 ms error rate is unchanged (29.82% vs 29.77%).

Q2: Duration

Clips average 41.0 s, so AlignBench is not long-form, and we have corrected the wording; see W3.

Q3: URLs and symbols

Thank you for catching this. The category list came from our tagger and overstates the content: AlignBench has no URLs or email addresses. Its 719 difficult words include 362 named entities, 277 contractions, 47 numbers, 45 hyphenated compounds, 22 acronyms and 14 abbreviations (categories overlap). Symbols appear only in written forms such as "$519,018" and "15%"; text normalization converts these to the spoken form before alignment (Appendix D). We have corrected the list in the paper.

Q4: Corpus choice

We agree this is not ideal, but it was the best option we found. We needed publicly redistributable audio without share-alike terms (People's Speech is CC BY, CC0 or public domain; VoxPopuli is CC0) that covers our target conditions: multi-speaker, accented and conversational speech. Most alternatives, such as LibriSpeech and Common Voice, are read speech, and any corpus that is free to redistribute is also likely to be used by others for training. On contamination, see W4.

Q5: Why English

Our target editing workloads are English, and a distribution-matched benchmark in another language needs that language's target distribution and calibrated annotators. Extending AlignBench is future work; we expect the architecture to extend to other languages given enough pseudo-labelled data.

Q6: Other languages

We evaluate English only and make no claims about other languages; see W2.

Q7: Batch size

Ablations train for a fixed 100k steps, so the batch size 64 run sees half the audio (about 39,000 vs 78,000 hours), and a small change is expected due to differences in the training dynamics. Every metric stays within the baseline's 95% confidence interval (17.21 ± 1.92 vs 17.16 ± 1.92 ms mean error; 0.37% vs 0.38% catastrophic), so the recipe is not sensitive to batch size.

Q8: Difficult words

Difficult words raise the >300 ms rate most for the lexicon-based Gentle (1.96% to 3.62%) and MFA (3.66% to 5.70%), while FuseAlign barely changes (0.38% to 0.42%). Appendix C.3 (Table 9) covers every system and moves to the main text (W6).

I thank the authors for their replies, and for the corrections applied to the paper. While I understand the cost efforts, I still believe that the dataset is too small to be representative, especially when already available datasets in the literature with similar characteristics cover hours of data in the same language (e.g., TIMIT). Moreover, I find very risky to use Voxpopuli and, in general, datasets usually used to train models (e.g., NVIDIA Canary) for a test set, as even if not used for the model proposed in the paper, FuseAlign, which I find a nice contribution, these training sets will be likely used by other models, as they are intended as training sets. Since the source language was English, which is the highest resourced language, finding a suitable test set with English source is very easy (e.g., ACL 6060, MCIF) and would have been a much more sound decision for building a benchmark. Therefore, since the benchmark is the first contribution of this work and I have several concerns about its adoption, I leave my scores unchanged.

We thank the reviewer for the follow-up and for the kind words about FuseAlign. We would like to emphasize that the benchmark is one of three contributions, alongside the evaluation framework and the FuseAlign model, and we believe the latter two add value independently of AlignBench. FuseAlign, trained without any TIMIT or Buckeye data, is competitive on both. On Buckeye it has the lowest catastrophic-error rate (0.40% of words above 300 ms, against 0.59–1.98% for the baselines) and the lowest rate above 50 ms (10.52%, against 12.45–64.76%). On TIMIT, its catastrophic-error rate (0.02%) is within 0.01 points of the best system. On clean, read TIMIT speech, the HMM-based Nyra remains more precise at fine tolerances, which we report in the paper.

Contamination: We agree this deserves care and will discuss it in the revised version of the paper. However, the suggested alternatives are not contamination-free either. ACL talks are already distributed as training data: NUTSHELL [1] releases 1,172h of *ACL talk audio with an 808h training split, used as official training data in the IWSLT 2025 instruction-following track [2]. The ACL 60/60 development set has been used for ASR fine-tuning [3]. MCIF [4] avoids contamination by selecting the most recent talks available at collection time, which holds only until newer training sets include them. Both ACL 60/60 [5] and MCIF [4] are also single-presenter talks; they therefore lack the multi-speaker conditions where aligners fail most, as shown in our ablations.

Size: AlignBench has 8,072 words and 73 unique speakers, with every boundary triple-annotated. This is comparable to the suggested benchmarks (ACL 60/60 eval: 59 min, 5 speakers [5]; MCIF: 2h, 21 speakers [4]), with more speakers than both. Its representativeness comes from design: clips match the distribution of a real editing workload, and the selection was frozen before any aligner was run. System gaps are large (3 to 11× in catastrophic error), and we report results on TIMIT [6] and Buckeye [7] alongside it.

Community value: Public English corpora with hand-placed word boundaries are rare; the field still relies on TIMIT (1993) [6] and Buckeye (2007) [7]. To our knowledge, none covers accented, multi-speaker, in-the-wild speech. We release all audio, annotations, ASR transcripts and code so others can extend the benchmark. We see this as a first step, and believe any such contribution moves the community in the right direction.

References

[1] Maike Züfle, Sara Papi, Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Jan Niehues. NUTSHELL: A Dataset for Abstract Generation from Scientific Talks. IWSLT 2025, pp. 19–32. https://aclanthology.org/2025.iwslt-1.2/

[2] IWSLT 2025 Instruction-Following Speech Processing Track, data conditions. https://iwslt.org/2025/instruction-following

[3] Henry Li Xinyuan, Neha Verma, Bismarck Bamfo Odoom, Ujvala Pradeep, Matthew Wiesner, Sanjeev Khudanpur. JHU IWSLT 2023 Multilingual Speech Translation System Description. IWSLT 2023, pp. 302–310. https://aclanthology.org/2023.iwslt-1.28/

[4] Sara Papi, Maike Züfle, Marco Gaido, Beatrice Savoldi, Danni Liu, Ioannis Douros, Luisa Bentivogli, Jan Niehues. MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks. ICLR 2026. arXiv:2507.19634.

[5] Elizabeth Salesky, Kareem Darwish, Mohamed Al-Badrashiny, Mona Diab, Jan Niehues. Evaluating Multilingual Speech Translation under Realistic Conditions with Resegmentation and Terminology. IWSLT 2023, pp. 62–78. https://aclanthology.org/2023.iwslt-1.2/

[6] John S. Garofolo et al. TIMIT Acoustic-Phonetic Continuous Speech Corpus. LDC93S1, Linguistic Data Consortium, 1993.

[7] Mark A. Pitt et al. Buckeye Corpus of Conversational Speech (2nd release). Ohio State University, 2007.

Reviewer_TasZOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FuseAlign, a transformer-based word-level forced aligner that jointly contextualizes waveform patches and transcript characters, then predicts independently localized word boundaries on a 1 ms grid alongside word-presence probabilities. This paper also introduces AlignBench, an English benchmark intended to reflect long-form, acoustically varied, multi-speaker media, and an evaluation protocol based on silence-aware asymmetric boundary error, word-level worst-edge error rates, and all-word scoring.

Critical strengths
  • This paper is well-motivated. The paper identifies limitations in traditional forced-alignment evaluation: clean speech, accurate transcripts, and insufficient attention to catastrophic alignment errors. These issues matter for text-based editing, subtitle generation, and speech-data preparation.

  • FuseAlign substantially reduces mean error, moderate errors, and catastrophic errors compared with several established aligners.

Critical weaknesses
  • The proposed asymmetric error metric does not always reflect editing-relevant alignment quality. The metric defines silence-tolerant intervals using only adjacent reference words, which may not correspond to actual silence in multi-speaker recordings. For example, suppose Speaker A speaks continuously from 0-3s, while Speaker B utters two words at 1-1.5s and 2-2.5s. Extending the first word's end boundary from 1.5s to 1.9s incurs zero asymmetric error, despite including 400ms of another speaker's speech. This contradicts the metric's motivation of penalizing audible editing errors, especially in overlapping speech scenarios.

  • The effectiveness of the word-presence detection head is insufficiently demonstrated. Under the proposed evaluation protocol, a trivial baseline that marks every ASR word as present achieves approximately 96.76% detection F1 on AlignBench/Whisper, nearly identical to FuseAlign's 96.83%. Similar results hold across other settings, suggesting that the reported high F1 scores largely reflect transcript accuracy rather than the effectiveness of the presence head. Moreover, the protocol treats all substituted words as absent, even when the substitution is a spelling variant or homophone that corresponds to the same acoustic segment.

Questions for authors
  • How does asymmetric boundary error account for speech from nonadjacent words during overlapping speech? Can you provide a controlled comparison against a speech-activity-aware metric or human assessments of edit quality?
1 comment

We thank the reviewer for the insightful feedback. The overlapping-speech counterexample and the trivial-baseline analysis of the presence head both identified real weaknesses, and both were concrete enough to test directly. Points shared with other reviewers, including the revised chunking setup for the baselines, are in the global response.

W1 / Q1: Asymmetric error under overlapping speech

We thank the reviewer for this counterexample. The reviewer is right that the free interval is bounded only by transcript neighbors, so in the described configuration speech from a non-adjacent word would go unpenalized. We implemented the speech-activity-aware variant the reviewer proposes. It bounds the free interval by the nearest point covered by any other reference word, and collapses the interval when another word straddles the boundary. It is identical to our definition whenever reference words do not interleave. Rescoring every system on AlignBench with hand-corrected transcripts:

SystemMean asym., all clips (ms)Mean asym., multi-speaker clips (ms)
FuseAlign17.16 → 17.3022.04 → 22.40
Gentle31.43 → 31.5447.06 → 47.31
MFA (chunked)62.31 → 62.39103.35 → 103.54
Nyra (chunked)31.36 → 31.4441.41 → 41.63
MMS-FA (chunked)52.20 → 52.3262.39 → 62.70
WhisperX49.04 → 49.1151.99 → 52.17

No threshold rate moves by more than 0.07 points and no ranking changes. Transcript neighbors in our references are almost always temporal neighbors, so the case is rare on AlignBench, but we agree it matters for heavy crosstalk and now describe the variant in the paper. We have not run a human assessment of edit quality. Our thresholds are grounded in inter-annotator agreement rather than a listening study, which we note as a limitation.

W2: Presence head effectiveness

We thank the reviewer for this analysis, which exposed a weakness we had not reported, and we agree with it. Declined words by error type on AlignBench under Whisper:

SystemInsertionsRepetitionsSubstitutionsCorrect words
FuseAlign5 / 2211 / 247 / 2468 / 7,077
Gentle133 / 22112 / 2433 / 24686 / 7,077
MFA (chunked)100 / 2218 / 243 / 246109 / 7,077
Nyra (chunked)0 / 2210 / 240 / 2460 / 7,077
MMS-FA (chunked)2 / 2210 / 240 / 2460 / 7,077
WhisperX0 / 2210 / 240 / 2460 / 7,077

Only Gentle and chunked MFA reject a substantial share of inserted words, and both pay by rejecting 86–109 correctly transcribed words.

At its 0.1 operating point FuseAlign's presence head almost never fires on real ASR errors, so its F1 reflects transcript accuracy, as the reviewer says. The paper now states this plainly. Tuning the threshold does not change this: even with the threshold tuned on AlignBench itself, F1 under Whisper rises only from 96.81% (marking every word found) to 97.10%, and under Scribe the best threshold rejects nothing.

Detection F1 is also not what matters most for editing. What matters is the final boundary error, because every transcript word still needs a span: a rejected word is either given the model's best-guess span or imputed from its neighbours, and a user cutting around it hears the result either way. Rejecting more words does not by itself produce better edits. Gentle rejects 133 of the 221 insertions under Whisper, yet its catastrophic-error rate is 3.67% against FuseAlign's 1.71%. FuseAlign's robustness to ASR transcripts comes from localizing each word independently: its 25 ms error rate on correctly transcribed words is 29.77% under Whisper against 29.82% with the hand-corrected transcript. We have removed language suggesting the presence head is the mechanism, and we list calibrating it on real ASR errors as future work. On substitutions we agree: spelling variants and homophones can occupy the same audio, so we now report insertions, which are unambiguously absent, separately from substitutions. Deciding whether a homophone substitution counts as present needs a finer measure than exact text matching; developing one, and training the presence head against it to improve robustness to ASR errors, is future work.

Reviewer_Nv8nOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

Authors analyze use of speech to text alignment metrics and make three key contributions to subfield: 1) they provide several new metrics to analyze speech alignment quality 2) they offer a new benchmark to analyze speech alignment 3) they train a novel new architecture for speech alignment. Comparative analysis against several common models are provided to show performance of model and benchmark.

Critical strengths

Speech alignment is an off neglected field of speech studies that is valuable for analysis. The authors' provision of a benchmark and metrics for analysis are well appreciated to foster research in the field. Further, the architecture is a significantly novel contribution, particularly in its reliance on pseudolabeled data for its pipeline. This alleviates resource demands in developing new models and suggests a powerful new way to approach the speech alignment problem.

Critical weaknesses

The authors' claim that "We show that Fuse-Align consistently outperforms other aligners and retains strong performance on ASR transcripts." This is a rather concerning misrepresentation of the data provided. While the model does outperform other instances on the newly provisioned AlignBench, the other datasets show a more nuanced story where varied resolutions and configurations outperform the proposed model. Indeed, looking through the appendix, it appears that MMS-FA has a stronger F1 score on AlignBench once chunking is considered. This is a facet that should be moved into the initial paper as it's critical for analysis of the strengths of the benchmark: the median length of samples (53.2s) is largely outside the training regime of most models being considered for comparison and thus would need to treat the evaluation as a long-form task. As each model has particular biases for long-form inference, they would each benefit from a different configuration, yet the paper provides the same chunking setup as for Fuse-Align. There is thus a natural bias towards Fuse-Align in the analysis that raises concerns for the comparative evaluation.

This is exacerbated by the lack of significant detail on the training data used for Fuse-Align. While I can appreciate the nature of proprietary data, the lack of description on its acquisition limitss the ability of researchers to evaluate the potential bias between training and benchmark. Is Fuse-Align a better model because of architectural design? Or is it because the data pipeline is closer to that used for the benchmark and thus biases the model towards better performance on the benchmark? There is a need for transparency - as much as is reasonable - in order to address this potential issue.

Finally, the models considered are not wholly illustrative of the scene of speech alignment. MMFS-FA and Wav2Vec-CTC are considered representative of CTC alignment despite stemming from wholly difference training regimes (with the latter being significantly dated as a general alignment model). While there are indeed resource limitations that should be taken into account, there would be some benefit into a more exhaustive consideration of the available models at hand (especially given the limited amount present).

Due to the potential oversights in research analysis, I am judging this as a weak reject. There is definitely merit to this work, but there is need of greater transparency and fair comparison against potential models for the merits of the architecture and dataset to be apparent. If authors are able to provide these more thorough comparison or assuage these concerns, I'm happy to move to an accept.

Questions for authors

Is not the long duration of the alignment samples an issue for analysis given domain variance of the models?

Why move the discussion of metrics into appendix B? This is a stronger portion of your paper and would benefit its inclusion in the main.

Is not the use of max error for word level error hiding particularly egregious alignments where both the start and end are overlapping?

What is the nature of the training data pipeline?

Why were you not able to use the Buckeye data as typically reported?

2 comments

We thank the reviewer for a careful review; the concerns about our claims and about fair long-input configurations were well founded.

W1: "Consistently outperforms" claim, and chunking

We agree that the sentence overstated the results and have rewritten it. FuseAlign is best on AlignBench at every tolerance and has the lowest catastrophic (>300 ms) rate on all three corpora under all three transcripts (tied with Nyra and MMS-FA on TIMIT). It is not uniformly best: Nyra is more precise on TIMIT at fine tolerances and in mean error, and on Buckeye at 25 ms and in mean error with hand-corrected and Scribe transcripts.

We also agree that long inputs were treated unevenly: only Gentle and Nyra window long audio internally. We now run every system on whole clips, chunked with Whisper routing, and chunked with reference routing (a ceiling), and the main table reports each baseline in its better mode: chunked for MFA, Nyra and MMS-FA, whole clips for Gentle and WhisperX. FuseAlign is best in every condition, even against baselines given perfect routing. The global response gives per-aligner numbers and explanations.

On MMS-FA's F1: with a hand-corrected transcript, F1 only counts unplaced words, so 99.99% vs 99.82% is about 2 vs 29 of 8,072. MMS-FA marks nearly every word found but places 46.06% of them more than 50 ms off, against 7.27% for FuseAlign, so its F1 advantage says little about editing quality.

W2: Training data transparency

We agree that readers need enough detail to judge this. The global response describes the corpus: about 78,000 hours of English speech from users of a media editing application, none from any corpus we evaluate on, with AlignBench finalized before any aligner was run. We acknowledge that both the target distribution and our training data reflect media-editing usage, which is the application we target. Two results suggest the gains are not specific to the benchmark. First, on Buckeye, which is lab-recorded conversation and not distribution-matched, FuseAlign has the lowest >300 ms and >50 ms rates under all three transcripts. Second, on TIMIT it ties for the lowest catastrophic rate. Fully separating architecture from data would require retraining the baselines on our data; we state this as a limitation. Within our own model, the ablations hold the data fixed and isolate the components that matter (label correction and convolutional upsampling).

W3: Baseline coverage

We agree that WhisperX's English wav2vec 2.0 model is dated. It is included because it is still a popular tool for forced alignment. We will add NeMo Forced Aligner (a modern FastConformer CTC model) and MWA (learned dynamic programming over self-supervised boundary features) as two further baselines to strengthen the study.

Q1: Long duration of the samples

Yes: 63% of clips exceed 30 s (mean 41.0 s, maximum 71.6 s), and duration affects the baselines differently. Chunking helps MFA (73.00 to 62.31 ms) and Nyra, while Gentle and WhisperX do better on whole clips. FuseAlign is best under every condition, so duration does not drive the ranking (see W1 and the global response).

Q2: Metrics in the appendix

Section 3 defines and motivates every metric; Appendix B holds the exact formulas and edge cases only due to space constraints. In the camera-ready version, with its extra page, we will move the asymmetric-error formula and the edge-case discussion into Section 3.

Q3: Max error hiding two-edge failures

No. A word fails if either edge exceeds the threshold, so two failing edges also fail, and the asymmetric error is uncapped, so a fully misplaced word pays its full displacement. We now report how many failures involve both edges: of FuseAlign's 31 words above 300 ms, 26 fail on one edge and 5 on both (MFA 196/114, Gentle 120/50, MMS-FA 138/33, WhisperX 95/19, Nyra 80/24, each in its main-table mode).

Q4: Training data pipeline

Raw audio is transcribed, force-aligned with Gentle and cut into chunks of at most 30 s. Words Gentle fails to align are dropped from supervision but kept in the text, and boundary loss is masked for collapsed labels (under 30 ms, except short function words such as 'a', 'the').

Q5: Buckeye protocol

There is no single standard Buckeye protocol. The Buckeye dataset consists of ten-minute interview sessions that include interviewer speech the transcript does not contain, and prior work segments it in different ways. The MFA 3.0 segmentation splits at silences of 300 ms or more, but its code is not released. We therefore cut sessions into interviewee-only turns using Buckeye's own tags. Appendix D reconstructs MFA's segmentation as closely as possible from the raw tiers (18.05 h against their reported 17.12 h). On those clips FuseAlign has lower mean asymmetric error than MFA (19.38 vs 24.75 ms) and about 4× fewer catastrophic errors (0.20% vs 0.89%), while MFA is slightly better at 25 ms (44.70% vs 47.81%).

Thank you once again for recognizing the strengths and weaknesses of our submission. We wanted to follow up on the additional experiments we ran in order to address some of the feedback surrounding the data, the long-form inference and some of the claims which were reported in the paper. We hope they adequately answer the questions which were raised. In case you have any further questions, we would be more than happy to address them with further clarifications/experiments.

Comments on the paper

2 comments

We thank all reviewers for their detailed feedback. This response covers points raised by more than one reviewer; each individual reply summarizes them and refers here for details.

We plan to make the following changes to the camera-ready version after incorporating the reviewer feedback:

  • The paper is scoped to English throughout (abstract, introduction, limitations), and we make no claims about other languages.
  • Overstated claims are rewritten. FuseAlign is best on AlignBench and has the lowest catastrophic (>300 ms) rate on all three corpora, but it is not uniformly best: Nyra is more precise on TIMIT at fine tolerances and on Buckeye at 25 ms. AlignBench is no longer called "long-form", and the presence head is no longer presented as the source of FuseAlign's robustness to ASR transcripts.
  • The main table reports each baseline in the chunking mode on which it performs the best, and the appendix reports all three chunking conditions below.
  • The asymmetric-error definition and the subgroup analysis (accent, speaker count, difficult words) move into the main text, and the architectural ablations move to the appendix.
  • New analyses: a speech-activity-aware variant of the metric, two-edge failure counts, presence-head declines by ASR error type, and NeMo Forced Aligner and MWA as further baselines.

Long inputs and chunking. AlignBench clips average 41.0 s (median 53.2 s, range 7.9–71.6 s), and 63% exceed 30 s, the input length most neural aligners are trained or documented for. The submission kept each baseline's original implementation, assuming it handled long audio. Gentle and Nyra do (20 s and 30 s internal windows), but MFA recommends utterances under 30 s, and MMS-FA and WhisperX apply their CTC models in windows of at most 30 s. We therefore ran every system on AlignBench (hand-corrected transcripts) in three scenarios: on whole clips; with FuseAlign's chunker (cores of at most 26 s plus 2 s of context), routing words to chunks by Whisper times as in practice; and with the same cuts, routing words by their reference times, which is a ceiling for routing.

Chunking is a tradeoff. It keeps inputs in distribution, but a word routed to the wrong chunk is displaced. Whole clips avoid routing errors but are out of distribution, increasingly so for longer recordings. We report mean asymmetric error for the three strategies: whole clips / reference routing / Whisper routing:

  • FuseAlign, chunked: n/a / 17.24 / 17.16 ms. It localizes each word independently, so routing barely matters; the change is within confidence intervals.
  • Gentle, whole clips: 31.43 / 28.79 / 101.15 ms. Chunking with reference times slightly improves it, but one misrouted word can derail its re-decoding of a whole chunk.
  • MFA, chunked: 73.00 / 54.70 / 62.31 ms. On a long recording one mismatch cascades along a single Viterbi path; chunks bound the cascade.
  • Nyra, chunked: 33.18 / 28.66 / 31.36 ms, with 29.99 / 19.38 / 19.55% of words >50 ms off. On whole clips its boundaries drift earlier by about 20 ms every 30 s; chunking resets the drift.
  • MMS-FA, roughly a tie, chunked: 48.91 / 51.67 / 52.20 ms, but fewer >300 ms errors (2.49 / 2.12 / 2.12%). Perfect routing does not recover the mean, so its loss comes from the shorter context.
  • WhisperX, whole clips: 49.04 / 49.61 / 57.35 ms. Perfect routing roughly matches whole clips, so its chunking penalty is almost entirely routing.

FuseAlign is best in every condition, including against baselines given perfect routing, so neither clip length nor the chunking setup drives the ranking.

Training data and benchmark sources. FuseAlign is trained on about 78,000 hours of transcribed English speech gathered from users of a media editing application, pseudo-labelled by Gentle forced alignment and cut into chunks of at most 30 s. None of it comes from People's Speech, VoxPopuli, AMI, TIMIT or Buckeye, the corpora we evaluate on, so any contamination from these corpora could only favor baselines trained on them. AlignBench was matched only on aggregate per-axis marginals of the target distribution and finalized before any aligner was run.

Baselines. For WhisperX we use only its English wav2vec 2.0 alignment model with our transcript; the multilingual Whisper recognizer plays no part. Because this model is dated, we add NeMo Forced Aligner (FastConformer CTC) and MWA (learned dynamic programming over self-supervised boundary features) to the list of baselines.

Robustness to ASR transcripts. FuseAlign's robustness comes from localizing each word independently, so an inserted or hallucinated word cannot displace its neighbours: its 25 ms error rate on correctly transcribed words is 29.77% under Whisper against 29.82% with the hand-corrected transcript. Its presence head rarely fires on real ASR errors, which the paper now states.

We would like to thank the reviewer for their prompt response and for raising their score. We will incorporate the additional experiments in the camera-ready version of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.