acceptodds
Under review as a conference paper at ICLR 2027

TrinityASR: An End-to-End Model for Long-Form Speech Recognition and Speaker Diarization

Abstract

In autoregressive transcription of long multi-speaker conversations, incorrect speaker labels can influence subsequent predictions and lead to persistent attribution errors. We present TrinityASR, an end-to-end model that jointly generates transcript text, speaker identities, and segment timestamps, supporting up to 90 minutes of audio in a single decoding run. To improve robustness to erroneous speaker-label histories, we propose Speaker Identity-Preserving Optimization (SIPO), which combines corrective supervised fine-tuning with reinforcement learning. Its supervised stage, Speaker Identity Decoupling (SID), perturbs historical speaker tags and trains the model to make corrective attribution decisions while accounting for the equivalence of speaker-ID renamings. Reinforcement learning then optimizes complete transcripts using a reward based on concatenated minimum-permutation character error rate (cpCER) and diarization error rate (DER). Across AISHELL-4, AliMeeting, Podcast, and Movies, TrinityASR achieves the lowest cpCER among the evaluated systems with available results. On AISHELL-4, RL initialized from SID training achieves 15.83% cpCER, compared with 24.30% for RL initialized without SID. Analysis of 32 internal two-speaker recordings further shows less persistent attribution errors and fewer long error runs for the SID-trained model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speaker Diarization, Automatic Speech Recognition, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_iefQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents Trinity ASR, an end-to-end framework for long-context Speaker-Attributed, Time-Stamped Transcription (SATS) ASR solution capable of processing up to 90 minutes of audio in a single decoding pass. The system is built on a Whisper-medium audio encoder and a Qwen3-0.6B language model backbone.

The paper focuses on mitigating autoregressive speaker-label drift over long horizons. The core methodological contribution is Speaker Identity-Preserving Optimization (SIPO), a two-stage training paradigm combining Speaker Identity Decoupling (SID) during supervised fine-tuning (SFT) with Group Relative Policy Optimization (GRPO) during reinforcement learning (RL). During SFT, SID perturbs contiguous spans of historical speaker tags and solves an online maximum-weight bipartite matching problem to assign optimal continuation labels under arbitrary one-to-one speaker ID renaming, paired with selective sub-token masking. During RL, the policy is further refined using sequence-level cpCER and Diarization Error Rate (DER) rewards.

The authors used both real in-house speech and synthesized multi-speaker speech to train the model. Empirical results show that TrinityASR outperforms several baselines across four benchmarks (AISHELL-4, Podcast, Movies, and AliMeeting).

Critical strengths
  1. The novel formulation of SID. The proposed 2-stage algorithm formulates a Speaker Identity Decoupling (SID) SFT stage and an RL stage. The SID algorithm formulates the continuation target ki†k_i^\dagger to minimize mismatches under optimal one-to-one renaming, which is an efficient and novel approach.
  2. The experiments show that the SID stage provides a synergy with the subsequent RL stage, which is an interesting finding.
Critical weaknesses
  1. Some details about the synthetic data construction are missing. The paper uses Qwen3-8B to generate speaker, time stamp template and fill in the template with speech segments. Are these speech segments from the same topic or arbitrary? How do you control it? Furthermore, what is the quality of the LLM-generated template? Some objective metrics (such as mean speaker durations, mean overlaps, etc.) and subjective evaluations should be provided here.
  2. The Error Propagation Rate (EPR) and Error Burst Rate (EBR@3) analyses is only conducted on 32 recordings from an internal two-speaker dialogue dataset. In two-speaker settings, speaker attribution reduces to binary alternation/switching, whereas the core challenge of permutation-invariant matching and new-speaker initialization (hi+1h_i + 1) manifests primarily when there are more than 3 speakers. Running EPR and EBR@3 on AISHELL-4 (5–7 speakers) or AliMeeting (2–4 speakers) alongside the RL checkpoints (Conditions C and D) would be much more informative.
Questions for authors
  1. For the RL stage, why is the reward function defined as such?
1 comment
AuthorsOpenReview ↗

Synthetic Templates, Multi-Speaker Diagnostics, and Reward Design

1. Synthetic template construction and validation.

Our templates contain only text-serialized sequences of segment start times, end times, and speaker IDs, with neither waveforms nor utterance text. We train Qwen3-8B autoregressively on timestamp–speaker sequences sampled from real training data, then generate schedules in the same format at temperature 1.0. The temporal structure is therefore learned from real conversations rather than manually prescribed. Real speech segments are inserted afterward, with a fixed one-to-one mapping between template speaker IDs and source speakers.

We also evaluate Qwen3-8B on held-out real timestamp–speaker sequences. The training–evaluation loss gap is approximately 0.1 under the same autoregressive objective, providing a diagnostic of sequence-model generalization.

2. Multi-speaker error persistence, including RL checkpoints.

Following the suggestion, we extend the original two-speaker diagnostic to AISHELL-4, AMI-SDM, and Podcast, including all four conditions. A/B denote SFT without/with SID; C/D denote RL initialized from A/B. Each result cell reports EPR / EBR@3 (%), both lower-is-better.

DatasetDuration range (min)ABCD
AISHELL-436.59–39.9022.90 / 6.1415.78 / 3.5019.90 / 5.4215.48 / 2.59
AMI-SDM13.98–49.5445.92 / 15.4437.07 / 10.5334.19 / 9.0030.57 / 7.33
Podcast25.48–78.9415.93 / 3.8810.26 / 1.278.43 / 1.227.96 / 0.92

SID reduces both metrics before and after RL on all three benchmarks. On AISHELL-4, the requested 5–7-speaker setting, final EBR@3 decreases from 5.42% to 2.59% with SID initialization. Moreover, both metrics improve along B→D on every benchmark: RL further reduces persistent attribution errors in the SID-trained model. These results extend the original finding beyond binary speaker switching to multi-speaker recordings and final RL checkpoints.

3. Rationale for the RL reward.

The reward is r = max(-5, 1 - 0.5 × e_cpCER - 0.2 × e_DER - 0.2 × f), where the error rates are expressed as fractions and f indicates an invalid output format.

where the error rates are expressed as fractions and ff indicates an invalid output format.

cpCER is the primary end-to-end objective: it jointly evaluates transcript accuracy and speaker grouping under label permutation. DER adds temporal supervision, since cpCER does not evaluate timestamps. The format penalty discourages malformed SATS outputs. The larger coefficient on cpCER reflects its primary role, while DER and format validity provide complementary constraints; the reward floor bounds extreme negative rewards.

This sequence-level objective complements SID's local corrective supervision: SID teaches continuation under perturbed speaker histories, while RL evaluates the complete generated transcription.

Reviewer_YGPbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TrinityASR jointly generates transcript text, speaker labels, and timestamps for long recordings. Its training method, SIPO, combines supervised correction under perturbed speaker-label histories with sequence-level reinforcement learning. The supervised stage, SID, computes permutation-aware corrective labels through incremental prefix matching and handles multi-token speaker tags with selective loss masking. The model supports up to 90-minute inputs and reports strong speaker-attributed transcription results on four benchmarks.

Critical strengths
  1. Substantial gains in an informative internal comparison. On AISHELL-4, SID reduces supervised cpCER from 29.39% to 22.04%. Its advantage persists after the same RL procedure: full SIPO achieves 15.83%, versus 24.30% without SID initialization. These are meaningful improvements, not merely small changes in aggregate scores.

  2. A principled treatment of speaker-label ambiguity. SID accounts for the fact that speaker IDs are recording-local and arbitrary, rather than requiring recovery of the original numeric labels under corrupted histories. The formal target definition, incremental matching solver, and treatment of multi-token tags make the method concrete and inspectable.

Critical weaknesses
  1. The distinctive methodological contribution is not isolated experimentally. The high-level recipe of history perturbation followed by metric-guided RL is straightforward; the potentially more consequential contribution is the permutation-aware corrective target. However, SID simultaneously changes the history perturbation, target construction, and token masking. Table 3 evaluates this package, while Table 5's independent-replacement control changes the perturbation itself. Neither establishes the added value of the optimal targets over ordinary supervision under the same perturbations and corruption budget. Simplicity is not itself a weakness, but without this comparison it is difficult to assess the independent advance beyond a useful augmentation recipe.

  2. The evidence for specifically long-range attribution is narrower than the system capability. The main training ablations are on AISHELL-4, while the error-persistence diagnostics use 32 internal two-speaker recordings. Cross-dataset system comparisons and support for 90-minute inputs do not isolate how much SID helps as speaker recurrence distance or recording length increases. The reported improvements are valuable, but a controlled breakdown would better connect the method to the paper's long-range attribution motivation.

Questions for authors
  1. Under identical history perturbations and training budgets, how do original-label supervision, permutation-aware corrective targets, and the selective token mask compare? Which component accounts for the SID improvement?

  2. Does the SID benefit persist across speaker-recurrence distances or recording-length groups, and on a held-out benchmark such as AliMeeting? Any available breakdown would clarify whether the benefit specifically addresses long-range attribution rather than general robustness.

1 comment
AuthorsOpenReview ↗

Cross-Benchmark Validation and Long-Form Error-Persistence Analysis

Thank you for the constructive feedback. The controlled target/masking ablations requested in Q1 are in progress and will be reported in a follow-up. We first address Q2 with additional cross-benchmark comparisons and long-form error-persistence analyses.

1. SID's benefit is not confined to AISHELL-4.

We extend the four-condition comparison in Table 3 to AMI-SDM and the specifically requested AliMeeting benchmark. A/B denote SFT without/with SID; C/D denote RL initialized from A/B. AMI-SDM uses WER/cpWER, and AliMeeting uses CER/cpCER. All values are percentages.

DatasetConditionWER/CER ↓cpWER/cpCER ↓DER ↓
AMI-SDMA: SFT without SID24.8139.6036.24
B: SFT with SID25.6738.0631.83
C: RL from A18.4726.7125.73
D: RL from B17.1917.2016.65
AliMeetingA: SFT without SID38.1247.7020.28
B: SFT with SID35.7042.1622.35
C: RL from A24.6024.7310.29
D: RL from B24.8622.177.28

AMI-SDM reproduces the AISHELL-4 finding at both training stages: SID improves cpWER and DER before and after RL, with final cpWER decreasing from 26.71% to 17.20% (35.6% relative). This directly supports the reproducibility of SID's benefit beyond AISHELL-4.

On AliMeeting (26.20–37.30-minute recordings), the final transcription gain is more modest: cpCER decreases from 24.73% to 22.17% (10.4% relative), accompanied by a DER reduction from 10.29% to 7.28%. Thus, the magnitude varies across benchmarks, but SID initialization improves both final speaker-attributed transcription and diarization on both.

2. Reduced attribution-error persistence in long-form, multi-speaker recordings.

We expand the original 32-recording, internal two-speaker diagnostic to AISHELL-4, AMI-SDM, and Podcast, including all four checkpoints. AISHELL-4 provides 5–7-speaker meetings; Podcast adds 2–11-speaker interviews with recordings up to 78.94 minutes. Each result cell reports EPR / EBR@3 (%), both lower-is-better.

DatasetDuration range (min)ABCD
AISHELL-436.59–39.9022.90 / 6.1415.78 / 3.5019.90 / 5.4215.48 / 2.59
AMI-SDM13.98–49.5445.92 / 15.4437.07 / 10.5334.19 / 9.0030.57 / 7.33
Podcast25.48–78.9415.93 / 3.8810.26 / 1.278.43 / 1.227.96 / 0.92

The pattern is consistent across all three datasets: SID reduces both metrics at SFT (A→B), and its advantage remains after RL (C→D). Moreover, RL further improves the SID-trained model itself (B→D) on both metrics in every dataset. On Podcast, for example, EPR decreases from 10.26% to 7.96% and EBR@3 from 1.27% to 0.92%.

These results strengthen the two-stage rationale in §§3.3.2 and 5.1: SID supplies local corrective supervision, while subsequent RL further reduces persistent attribution errors through sequence-level feedback. The evidence therefore extends beyond aggregate transcription gains and internal two-speaker dialogues to consistent improvements across multi-speaker meetings and long-form interviews.

Reviewer_F1aXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a training method for creating a speaker-attributed ASR system from a Whisper audio tokenizer and Qwen LLM backbone. The key components include:

  • A Speaker Identity Preserving Optimization (SIPO) training regime with two steps
  1. Speaker Identity Decoupling → Scheduled sampling with speaker ID
  2. cpCER and DER rewards in RL
  • creation of a 24k hour simulated multi-speaker dataset in addition to 7.3k real data.

There are results on AISHELL-4 and Ali Meeting and unreleased Podcast and Movies datasets. Comparisons are made against Vibe Voice and Gemini 2.5 and 3.0. Results are mostly wins against these models. The ablations show that SID and cpCER+DER are beneficial, and produce better results still combined. The ablation also shows that the synthetic data is helpful on top of the real data. There are comparisons with Gemini 2.5 and 3.0 Pro models The authors do not have a recipe for training the model or open source released version of the model.

While they are strong generalist model, don't believe Gemini models are specifically trained for the diarization task. On the other hand Vibe Voice may be a reasonable baseline, with a Qwen backend, and a specialized audio tokenizer--although Vibe Voice is meant to support 50 languages the paper doesn't state the number of languages supported, nor the mix and distribution of languages in the in-house training set or synthesized training set. All we know is Trinity ASR is likely capable of at least English, Chinese, Korean, Japanese and Cantonese.

Critical strengths

The paper has clear motivations, generally well written.

Critical weaknesses

There are a few holes, missing comparisons:

  1. Leveraging Self-Supervised Learning for Speaker Diarization ICASSP 2025 There is a claim that "modular" systems are brittle, however
  • Diarizen shows strong results on AMI, AISHELL-4 and AliMeeting
  • it is open-source with free to use models; this could be used to get comparisons on the unreleased Podcast and Movies datasets
  1. TellWhisper: Tell Whisper Who Speaks When ACL 2026 This is an End-to-end model using a Rope embedding with time and speaker information. Results are also very strong on AMI / AISHELL-4 and AliMeeting showing an alternate means of achieving good Speaker Attributed Transcripts

  2. Trinity ASR on AMI - this is highly quoted benchmark for diarization and speaker-attributed transcript; not evaluated on it can be considered a unacceptable flaw.

  3. Evaluation protocol: lack of evaluation metrics. The number of words, total length of audio, total # recordings are not reported on the evaluation sets. What are the confidence intervals, are the differences statistical significant?

  4. Reproducibility: Lack of locale splits for training (real and synthesized) and test data; lack of training recipe and open sourced code limit reproducibility. As currently described, I would judge the ability to replicate the results as low.

  5. Are the datasets that use YouTube data violating the terms of service by downloading videos from the service?

Questions for authors

What accounts for Vibe Voice achieving a superior DER (6.77%) compared to Trinity ASR (9.88%) on AISHELL-4, even though Trinity ASR dramatically wins on cpCER (15.83% vs 24.99%)?

How well does Trinity ASR scale to larger speaker counts (>12 speakers) or dense overlapping speech in highly reverberant environments?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Questions and Additional Results

Thanks for the helpful comments.

1. Open-source Availability.

In fact, we have already made it open source..

2. Additional baseline comparisons.

We evaluated DiariZen + Qwen3-ASR-1.7B as a cascaded baseline using our evaluation pipeline. Δcp = cpWER − WER.

DatasetMetricDiariZen + Qwen3-ASR-1.7BTrinityASR
AISHELL-4WER ↓16.7614.84
cpWER ↓17.8715.83
Δcp1.110.99
DER ↓7.829.88
AMI-IHMWER ↓22.9413.10
cpWER ↓22.899.48
Δcp-0.05-3.62
DER ↓22.8212.75
AMI-SDMWER ↓24.0017.19
cpWER ↓28.2817.20
Δcp4.280.01
DER ↓10.6916.65
AliMeetingWER ↓26.2024.86
cpWER ↓24.2322.17
Δcp-1.97-2.69
DER ↓7.747.28
PodcastWER ↓7.465.97
cpWER ↓9.207.37
Δcp1.741.40
MoviesWER ↓8.576.36
cpWER ↓19.3612.76
Δcp10.796.40

TrinityASR achieves lower cpWER in every evaluated condition, although DiariZen has lower DER on AISHELL-4 and AMI-SDM.

TellWhisper's weights are unavailable to us for independent evaluation. Its paper reports 32.53% cpWER on AMI-SDM, compared with our 17.20%. This is a cross-paper reference, not a controlled comparison.

3. AMI evaluation and additional baselines.

We added AMI-IHM and AMI-SDM evaluations.

DatasetMetricGemini-2.5-ProGemini-3-ProVibeVoice-ASRSoulX-TranscriberDiariZen + Qwen3-ASR-1.7BTrinityASR
AMI-IHMWER ↓18.4817.6115.2317.3622.9413.10
cpWER ↓29.5722.3414.9821.7222.899.48
Δcp11.094.73-0.254.36-0.05-3.62
DER ↓23.5446.2316.5320.1422.8212.75
AMI-SDMWER ↓22.3522.0920.5821.9224.0017.19
cpWER ↓34.7826.9131.7127.3128.2817.20
Δcp12.434.8211.135.394.280.01
DER ↓23.7943.0423.8823.3510.6916.65

The original AISHELL-4 comparison used VibeVoice's reported DER of 6.77%. Our re-evaluation yielded 17.38%, versus TrinityASR's 9.88%. DER measures time-based speaker assignment, whereas cpWER jointly evaluates transcription and attribution.

SoulX-Transcriber uses 5-minute chunks; TrinityASR processes full recordings. These settings differ in their demands on long-range speaker consistency; we include the chunked results for reference.

4. Multilingual evaluation.

We added evaluations on MLC-SLM (14 languages, 142 samples) and FLEURS (50+ languages). TrinityASR achieves 11.64% WER and 14.32% cpWER on MLC-SLM. Per-language comparisons are provided in our separate multilingual comment.

5. Evaluation set statistics.

Reference counts below use Chinese characters and English words.

  • AISHELL-4 (12.73 h, 20 recordings, 180,359 reference units): 5–6 speakers (80.0%) and 7–8 (20.0%).
  • Podcast (18.52 h, 25 recordings, 199,327 reference units): 2 speakers (22.7%), 3–4 (27.3%), 7–8 (25.6%), 9–10 (13.7%), and 11–12 (10.7%).
  • Movies (2.57 h, 800 recordings, 29,693 reference units): 1–2 speakers (30.3%), 3–4 (44.1%), and 5–6 (25.6%).
  • AliMeeting (10.78 h, 20 recordings, 209,035 reference units): 1–2 speakers (40.0%) and 3–4 (60.0%).
  • AMI-IHM-mix / AMI-SDM: 16 recordings and 89,503 words per condition; 3–4 speakers (100%).

6. YouTube data provenance.

The Podcast samples were identified through YODAS, whose dataset documentation states that source videos were restricted to Creative Commons-licensed content during collection.

AuthorsOpenReview ↗

Additional Multilingual Evaluation

We have supplemented multilingual evaluations on MLC-SLM and FLEURS to further demonstrate TrinityASR's multilingual capabilities.

MLC-SLM.

We evaluate TrinityASR on 142 samples across 14 languages from MLC-SLM, comparing it with VibeVoice-ASR. Each entry reports WER / cpWER (%).

LanguageTrinityASRVibeVoice-ASR
English7.82 / 10.689.10 / 11.88
French15.87 / 20.8019.11 / 26.59
German16.06 / 17.1320.00 / 21.25
Italian5.09 / 7.256.19 / 8.53
Japanese21.22 / 22.6825.99 / 28.14
Korean14.46 / 20.3618.23 / 23.98
Portuguese10.75 / 14.3612.48 / 16.13
Russian4.20 / 6.235.32 / 7.33
Spanish5.42 / 6.986.32 / 7.93
Tagalog10.22 / 10.9213.31 / 14.10
Thai10.27 / 12.3611.27 / 13.16
Turkish30.07 / 34.1340.32 / 44.18
Urdu13.00 / 14.5271.45 / 74.33
Vietnamese16.50 / 17.0220.64 / 21.24
Overall11.64 / 14.3216.84 / 19.83

TrinityASR achieves lower overall WER and cpWER than VibeVoice-ASR across the 14 evaluated languages.

FLEURS.

We further evaluate TrinityASR on more than 50 languages from the FLEURS test set, comparing it with Qwen3-ASR-1.7B and Whisper large-v3. The per-language WER (%) results are reported below.

LanguageTrinityASR ↓Qwen3-ASR-1.7B ↓Whisper large-v3 ↓
Arabic8.1514.6815.17
Asturian15.3348.8748.86
Bengali18.78115.6430.46
Bosnian10.4553.7013.28
Bulgarian15.5858.5612.78
Cantonese Chinese11.156.3458.02
Catalan4.0260.484.96
Cebuano12.4944.3844.32
Croatian9.4783.1811.02
Czech7.3022.8010.57
Danish12.5921.3612.35
Dutch3.137.125.50
English3.803.714.65
Estonian5.7787.2918.15
Filipino6.1124.2912.54
Finnish3.5025.197.62
French2.754.305.35
Galician6.5853.7613.76
Ganda22.61103.6928.47
German1.883.984.02
Greek8.0328.1711.51
Hindi11.9613.0740.33
Hungarian8.2733.0813.48
Icelandic16.5394.7735.62
Indonesian2.395.566.03
Italian1.202.572.20
Japanese4.625.289.82
Javanese17.9962.8225.13
Kazakh25.09106.3332.86
Korean5.444.4913.90
Kyrgyz18.58108.9127.88
Latvian14.05108.1319.00
Lingala16.2881.9018.58
Lithuanian14.97110.6724.57
Luo18.3086.7235.85
Malay4.7310.367.79
Mandarin Chinese8.507.3151.19
Maori21.6378.8938.06
Norwegian5.5258.898.16
Persian10.6228.4230.20
Polish3.4112.354.63
Portuguese2.144.153.87
Romanian7.2420.549.30
Russian2.726.214.98
Serbian21.3882.4824.03
Slovak10.0362.439.43
Slovenian13.71100.8819.09
Spanish1.263.012.71
Swahili14.3956.6134.53
Swedish7.1619.437.83
Thai7.976.966.56
Turkish2.969.356.39
Ukrainian3.9072.426.51
Vietnamese6.636.038.43
Welsh18.99100.0030.39

These results further demonstrate TrinityASR's multilingual speech recognition capability across more than 50 languages.

Thanks to the authors for the detailed rebuttal. Many of my concerns have been addressed including open-sourcing the model, multilingual capability, evaluations on AMI and evaluation set statistics. In response, I will revise my score to "Weak Accept".

Thank you for your consideration and for updating your rating. We appreciate your time and constructive feedback.

Reviewer_uLQzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an end-to-end approach to long-form speaker-attributed speech transcription. The model takes up to 90 minutes of speech as input and produces a transcription, speaker labels, and corresponding timestamps in a single pass. The paper introduces two main training innovations. First, during supervised fine-tuning, the authors perturb the speaker-label history to make subsequent predictions more robust to earlier errors. Second, they apply metric-guided consistency reinforcement learning to further improve performance. The training data consists of real in-house data and simulated synthetic multi-talker data. Experiments show substantial improvements in both transcription and speaker-attributed transcription over baseline methods, including Gemini 2.5 Pro, Gemini 3 Pro, VibeVoice, and SoulX-Transcriber.

Critical strengths

1, The paper addresses the important problem of jointly predicting transcriptions and speaker labels for long-form conversational speech, with recordings of up to 90 minutes. This is a meaningful challenge for the speech community. The experimental results demonstrate substantial improvements over the baseline methods.

2, The proposed Speaker Identity Preserving Optimization training method provides a practical and innovative approach to improving long-form speaker-attributed transcription. In particular, perturbing the speaker-label history helps mitigate the propagation of earlier speaker-label errors.

Critical weaknesses
  1. The training data is not described in sufficient detail to support reproducibility. I understand that the authors may be unable to disclose all details of the in-house data, but basic statistics would be valuable, including the languages covered, the distribution of the number of speakers, and the duration distribution of training samples. Similar statistics should also be provided for the simulated data.

  2. The experimental section would benefit from additional ablation studies isolating the contributions of the proposed training components. In addition, although the paper claims that the model can process up to 90 minutes of audio, the experiments appear to evaluate recordings of only slightly over 40 minutes. Evaluating longer recordings would help substantiate the claim. In Table 2, although SoulX-Transcriber cannot process the long recordings directly, it would be useful to include its results using chunked audio (5 minutes, or 10 minutes) for comparison.

Questions for authors
  1. How are utterances selected when generating the synthetic data? Are they chosen to form semantically coherent conversations, or are they sampled randomly?

  2. What is the total duration of the Podcast and Movies evaluation sets? For all evaluation datasets, could the authors provide the distribution of speaker counts—for example, the proportion of recordings containing each number of speakers?

  3. Which title is your final title? The title in this submission page, or the pdf file?

  4. From the content of this paper, it seems it is an updated work of MOSS-Transcribe-Diarize. For the MOSS-Transcribe-Diarize, it mentioned that the model supports 50 languages. However, the experiment in current paper only evaluates mostly Chinese and English test data. Can you explain if the model trained in the current paper is the same as the one in MOSS-Transcribe-Diarize? If so, why there is no evaluation on other languages?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Questions and Additional Results

Thanks for the helpful comments.

1. Synthetic data construction.

Speech segments are randomly sampled from the single-speaker pool and placed according to speaker-turn schedules learned from real conversations. The schedules capture speaker turns, pauses, and overlaps, while semantic coherence between sampled utterances is not explicitly enforced.

2. Evaluation set statistics and long-form evaluation.

The total duration and speaker-count distributions are as follows:

  • AISHELL-4 (12.73 h): 5–6 speakers (80.0%) and 7–8 (20.0%).
  • Podcast (18.52 h): 2 speakers (22.7%), 3–4 (27.3%), 7–8 (25.6%), 9–10 (13.7%), and 11–12 (10.7%).
  • Movies (2.57 h): 1–2 speakers (30.3%), 3–4 (44.1%), and 5–6 (25.6%).
  • AliMeeting (10.78 h): 1–2 speakers (40%) and 3–4 (60%).
  • AMI (9.06 h): 3–4 speakers (100%).

As reported in the paper, Podcast recordings range from 1,528.7 to 4,736.5 seconds (approximately 25.5–78.9 minutes), with the longest recording approaching 80 minutes. All recordings are processed in a single decoding run without chunking. We believe this is sufficient to demonstrate TrinityASR's long-form processing capability.

We additionally report TrinityASR results on AMI alongside the available baselines.

DatasetMetricGemini-2.5-ProGemini-3-ProVibeVoice-ASRTrinityASR
AMI-IHMWER ↓18.4817.6115.2313.10
cpWER ↓29.5722.3414.989.48
Δcp11.094.73-0.25-3.62
DER ↓23.5446.2316.5312.75
AMI-SDMWER ↓22.3522.0920.5817.19
cpWER ↓34.7826.9131.7117.20
Δcp12.434.8211.130.01
DER ↓23.7943.0423.8816.65

We further compare TrinityASR with SoulX-Transcriber across five benchmarks. SoulX-Transcriber is evaluated using 5-minute chunks. However, chunk-based evaluation is not directly comparable to full-recording inference, as the latter requires maintaining speaker consistency across the entire recording. Nevertheless, we provide the actual results for reference.

DatasetMetricSoulX-TranscriberTrinityASR
PodcastWER ↓6.795.97
cpWER ↓10.127.37
Δcp3.331.40
AMI-IHMWER ↓17.3613.10
cpWER ↓21.729.48
Δcp4.36-3.62
DER ↓20.1412.75
AMI-SDMWER ↓21.9217.19
cpWER ↓27.3117.20
Δcp5.390.01
DER ↓23.3516.65
AISHELL-4WER ↓14.4914.84
cpWER ↓17.3015.83
Δcp2.810.99
DER ↓8.699.88
AliMeetingWER ↓23.7524.86
cpWER ↓25.2222.17
Δcp1.47-2.69
DER ↓12.087.28

3. Paper title.

The PDF title is the final title: TrinityASR: Learning Long-Range Speaker Attribution through Identity-Preserving Optimization.

4. Multilingual evaluation.

Thanks for your interest in our work. We have supplemented multilingual evaluation results on MLC-SLM and FLEURS. Due to the character limit, we provide these results in a separate comment.

AuthorsOpenReview ↗

Additional Multilingual Evaluation on MLC-SLM and FLEURS

We have supplemented multilingual evaluations on MLC-SLM and FLEURS to further demonstrate TrinityASR's multilingual capabilities.

MLC-SLM.

We evaluate TrinityASR on 142 samples across 14 languages from MLC-SLM, comparing it with VibeVoice-ASR. Each entry reports WER / cpWER (%).

LanguageTrinityASRVibeVoice-ASR
English7.82 / 10.689.10 / 11.88
French15.87 / 20.8019.11 / 26.59
German16.06 / 17.1320.00 / 21.25
Italian5.09 / 7.256.19 / 8.53
Japanese21.22 / 22.6825.99 / 28.14
Korean14.46 / 20.3618.23 / 23.98
Portuguese10.75 / 14.3612.48 / 16.13
Russian4.20 / 6.235.32 / 7.33
Spanish5.42 / 6.986.32 / 7.93
Tagalog10.22 / 10.9213.31 / 14.10
Thai10.27 / 12.3611.27 / 13.16
Turkish30.07 / 34.1340.32 / 44.18
Urdu13.00 / 14.5271.45 / 74.33
Vietnamese16.50 / 17.0220.64 / 21.24
Overall11.64 / 14.3216.84 / 19.83

TrinityASR achieves lower overall WER and cpWER than VibeVoice-ASR across the 14 evaluated languages.

FLEURS.

We further evaluate TrinityASR on more than 50 languages from the FLEURS test set, comparing it with Qwen3-ASR-1.7B and Whisper large-v3. The per-language WER (%) results are reported below.

LanguageTrinityASR ↓Qwen3-ASR-1.7B ↓Whisper large-v3 ↓
Arabic8.1514.6815.17
Asturian15.3348.8748.86
Bengali18.78115.6430.46
Bosnian10.4553.7013.28
Bulgarian15.5858.5612.78
Cantonese Chinese11.156.3458.02
Catalan4.0260.484.96
Cebuano12.4944.3844.32
Croatian9.4783.1811.02
Czech7.3022.8010.57
Danish12.5921.3612.35
Dutch3.137.125.50
English3.803.714.65
Estonian5.7787.2918.15
Filipino6.1124.2912.54
Finnish3.5025.197.62
French2.754.305.35
Galician6.5853.7613.76
Ganda22.61103.6928.47
German1.883.984.02
Greek8.0328.1711.51
Hindi11.9613.0740.33
Hungarian8.2733.0813.48
Icelandic16.5394.7735.62
Indonesian2.395.566.03
Italian1.202.572.20
Japanese4.625.289.82
Javanese17.9962.8225.13
Kazakh25.09106.3332.86
Korean5.444.4913.90
Kyrgyz18.58108.9127.88
Latvian14.05108.1319.00
Lingala16.2881.9018.58
Lithuanian14.97110.6724.57
Luo18.3086.7235.85
Malay4.7310.367.79
Mandarin Chinese8.507.3151.19
Maori21.6378.8938.06
Norwegian5.5258.898.16
Persian10.6228.4230.20
Polish3.4112.354.63
Portuguese2.144.153.87
Romanian7.2420.549.30
Russian2.726.214.98
Serbian21.3882.4824.03
Slovak10.0362.439.43
Slovenian13.71100.8819.09
Spanish1.263.012.71
Swahili14.3956.6134.53
Swedish7.1619.437.83
Thai7.976.966.56
Turkish2.969.356.39
Ukrainian3.9072.426.51
Vietnamese6.636.038.43
Welsh18.99100.0030.39

These results further demonstrate TrinityASR's multilingual speech recognition capability across more than 50 languages.

Reviewer_uLQzOpenReview ↗

5min - evaluation results

Thanks for providing these detailed results. I wonder for the 5min - evaluation, are your results the 5min-local-segment evalution results, or the full data length evaluation results over all segments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.