acceptodds
Under review as a conference paper at ICLR 2027

Horn-ASR: A Spontaneous-Speech Evaluation Benchmark for Languages of the Horn of Africa

Abstract

Evaluation datasets for multilingual automatic speech recognition (ASR) on African languages are constructed in ways that diverge from the spontaneous speech these systems will transcribe in use: translated sentences read aloud by fewer than ten speakers per language, held-out splits of single-speaker Bible recordings, or image-prompted elicitation. FLEURS, the most widely used of these benchmarks, covers three languages of the Horn of Africa. For each of them, our audit finds that every split contains only one gender, 3 to 7 speakers per language, and every development speaker also in the test split. We introduce Horn-ASR, an evaluation benchmark of naturally-occurring spontaneous speech (broadcasts and interviews recorded for a public audience) for more realistic evaluation of Amharic, Oromo, Somali, and Tigrinya, languages spoken by more than 100 million people. It consists of 1,000 curated segments per language (15.43 h in total) from 1,710 publicly available interviews, balanced for gender and diverse across dialect and topical domain. We evaluate six multilingual ASR systems from three providers under a single evaluation protocol. Horn-ASR is harder than FLEURS and far from saturated: the best word error rate (WER) on any language is 0.275, and where a system is scored on the same language in both benchmarks, its WER is on average 0.16 higher on Horn-ASR. It supports disaggregated evaluation by gender and dialect, which single-gender read-speech datasets cannot, and reveals significant gender gaps in every language as well as dialect gaps of up to 0.22 WER that favor the dominant variety. An inter-annotator agreement study measures the reliability of the reference transcripts and labels. We release the transcripts, labels, and guidelines (CC BY-SA 4.0), the audio (under terms of use), the curation methodology, and all system outputs and scoring code.

Primary area:
datasets and benchmarks
Keywords:
speech recogntion, low resource languages, African languages

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_exfJOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces HORN-ASR, an evaluation benchmark for automatic speech recognition (ASR) in four under-resourced Afroasiatic languages of the Horn of Africa: Amharic, Oromo, Somali, and Tigrinya. It contains naturally occurring spontaneous speech (broadcasts and interviews recorded for a public audience), with 1,000 segments per language. The paper is also partly a position paper. It rightly criticises massively multilingual benchmarks such as FLEURS, which consist mostly of read speech (from translated text, also called “translationese”) with little speaker variability. The authors argue that ASR performance on such data underestimates errors on spontaneous speech and can rank systems differently, and support this by benchmarking six multilingual ASR systems on HORN-ASR (not all systems cover all four languages). They also question the extreme aggregation of scores into a single number (common practice in massively multilingual ASR eval) and advocate more detailed evaluation, for example by gender and dialect, when metadata allows.

Critical strengths
  1. The positioning is very good. The paper highlights the shortcomings of current massively multilingual ASR evaluation and suggests what could be done better.

  2. HORN-ASR is, to my knowledge, the first public ASR benchmark of naturally occurring spontaneous speech for these four languages. It is balanced for gender and diverse across dialect and domain, which makes the evaluation more realistic.

  3. The data collection and annotation methodology is described precisely.

  4. The evaluation of six current systems shows that HORN-ASR is harder than FLEURS (clearly so for Amharic and Somali) and can rank systems differently. It also reveals gender gaps (whose direction depends on the system) and dialect gaps favouring the dominant variety in Amharic, Oromo, and Tigrinya (not significant for Somali), which no existing benchmark for these languages could show before.

Critical weaknesses
  1. Rights and consent. The authors describe their copyright position as a research-use argument, not a legal determination, and say they do not claim that public availability implies consent (Ethics Statement). The paper does not mention permission from broadcasters or speakers. On what basis can the transcripts be published ungated under CC BY-SA 4.0?

  2. Repository not accessible. The anonymous repository link returned “This content can’t be shown. The repository is expired.” According to the datasheet, the dataset link is in that repository’s README, so I could not check any of the release.

Questions for authors

For FLEURS, the paper says all systems were re-run under one protocol, and checks this against the published MMS Somali number. For MMS Amharic, however, Table 10 lists 53.3 self-reported against 0.383 verbatim here. Why this difference? Sec. 6.3 also cites “0.428 on FLEURS (Table 12)”, which does not appear in that table.

Remark on venue: I appreciate bringing this discussion to the ICLR audience, but I wonder about its impact there. LREC-COLING or Interspeech would likely give the work a more relevant audience and more useful feedback. I am ambivalent about ICLR as the venue.

2 comments
AuthorsOpenReview ↗

Response to Reviewer exfJ

Weakness 1: rights and consent

Thank you for raising this; we agree it needs a precise answer, and we have improved the access model described in the paper.

Transcripts will not be published ungated. The transcripts reproduce the words spoken in the broadcasts, so the rights questions that apply to the audio apply to them too. At release, the whole dataset, audio and transcripts, is distributed behind an access agreement (an automatically approved click-through, so access stays easy for researchers in the region). The Terms of Use restrict use to evaluation and research and prohibit training on the data, speaker identification or profiling, voice cloning, and redistribution, and require deleting data after a takedown. CC BY-SA 4.0 will cover only what we created independently of the speech: the gender, dialect and domain labels and the other metadata. We will state this split on the dataset card and correct the abstract and the datasheet. During the review period the Hugging Face dataset repository stays open so that reviewers can inspect it anonymously; the gate and Terms of Use are prepared and will be switched on once the review period ends.

Basis. We rely on a research-use argument, not on a licence or permission from the broadcasters, which we did not obtain, and, as the Ethics Statement says, not on any claim that public availability implies consent. The argument rests on minimization and control: each source recording contributes at most 11 short segments (at most about 2.5 minutes of audio; a median of 1 to 2 segments, averaging 14 s each, usually from different parts of the recording rather than one continuous stretch), which cannot substitute for the original broadcast; the data is evaluation-only; and a takedown contact on the dataset card handles removal requests from rights holders and from speakers themselves.

Revision. We will add these points, the licence split, the access terms and the takedown process to the Ethics Statement and the datasheet.

Weakness 2: repository

Thank you for reporting this. The repository is live again at the same address and will remain available through the review period (see our general comment). Its README links the dataset revision used in the paper, so the release can be checked and every table reproduced from the stored system outputs.

Question: MMS Amharic and the 0.428 citation

Thank you for the careful reading. There are two reporting problems, in Table 10 and Section 6.3, and a difference in the Amharic number that we trace to the system outputs rather than the scoring.

Units. The self-reported column of Table 10 prints the published numbers as percentages (53.3, 140.3) next to our fractions. The comparable figures are 0.533 reported vs 0.383 measured for Amharic, and 0.538 vs 0.538 for Somali. We will convert the column to fractions.

Why Amharic differs. Scored verbatim against the FLEURS raw_transcription, our MMS-1B-all outputs reproduce the published Somali number exactly (0.538), but give 0.38 for Amharic. No scoring variant brings Amharic close to 0.533: the published caption's Whisper normalization gives 0.295, the FLEURS normalized transcription 0.297, and our normalization 0.282. This does not affect our conclusions: the Horn-ASR vs FLEURS gap compares our own runs of the same checkpoint on both benchmarks, with the same inference and scoring, so it does not depend on the published number.

0.428. This is the WER MMS reports for Somali with the Common-Crawl LM on FLEURS (Pratap et al., Table A1), not a number in our Table 12, which holds our Horn-ASR LM results. The citation in Section 6.3 should point to their Table A1; we will fix it.

Remark: venue

Thank you for the remark. We chose ICLR because the systems we evaluate (MMS, Omnilingual ASR, Whisper, Chirp) are built and reported in this community, mostly through aggregate FLEURS numbers, and our results show that those numbers under-estimate error and can rank systems differently for these languages. The released benchmark is open to the speech community as well.

Reviewer_exfJOpenReview ↗

acknowledgement of reviewers' response

I acknowledge the authors’ response and thank them for the clarification. I think this paper is a good example of the right way to address the challenge of low-resource languages: rather than racing for massive multilinguality, it does things properly for a few languages, evaluates the technology in realistic conditions (real conversations), and carefully documents the data collection methodology.

Reviewer_SWzYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HORN-ASR, a spontaneous-speech ASR benchmark for Amharic, Oromo, Somali, and Tigrinya. It contains 4,000 segments collected from public broadcasts and interviews, with gender, dialect, and domain annotations. The authors evaluate six multilingual ASR systems and show that performance is generally worse than on FLEURS. They also identify gender and dialect disparities and study the reliability of human transcriptions and annotations.

Critical strengths
  • The benchmark addresses an important gap in ASR evaluation. Using naturally occurring speech across four underrepresented languages.
  • The paper has detailed and useful analysis on gender and dialect differences, transcription quality, and evaluation normalization. I particularly appreciate the authors' transparency about annotation uncertainty.
Critical weaknesses
  • Independent human annotators disagree by 25–48% WER, even after normalization. Although the authors use multiple rounds of editing, the validation does not establish how accurate the final references are. For Somali, the final references agree no better with independent annotators than another independent transcription does.
Questions for authors
  • Given the substantial human transcription disagreement, how confident can we be that the reported differences between ASR systems reflect recognition quality rather than reference uncertainty?
  • The paper emphasizes spontaneous speech, but fillers and disfluencies are omitted from the references. Could this penalize models that produce more faithful, verbatim transcriptions?
1 comment
AuthorsOpenReview ↗

Response to Reviewer SWzY

Weakness and Question 1: reference accuracy

Thank you for this question, and for appreciating our reporting of annotation uncertainty. We agree that inter-annotator agreement alone does not establish how accurate the final references are, and we address the two parts separately.

Do system differences depend on the reference? We tested this directly. On the 389 items transcribed by three independent annotators (about 100 per language), we rescored every system against the released reference and, separately, against each independent transcript, with interview-clustered paired bootstrap intervals. Of the 34 system pairs, 28 are significant against the released reference, and all 28 remain significant, with the same sign, against every independent transcript. System rankings are identical under every reference in Amharic, Oromo and Somali; in Tigrinya only two systems swap, and their difference (below 0.01 WER) is not significant under any reference. For Somali in particular, all 9 of its significant pairs hold under every annotator's transcript. The smallest significant difference on these items under the homophone-aware scheme, which the agreement figures use (0.054 WER), is about a fifth of the inter-annotator disagreement and also holds under every reference. Reference uncertainty shifts absolute WER (by up to 0.08), but the comparisons between systems stay the same, so the reported differences reflect recognition quality rather than the choice of reference.

How accurate are the references, and Somali? We agree that agreement is indirect evidence of accuracy, so we measured accuracy directly. In a blind human evaluation, native-speaker expert raters listened to each of the agreement-study segments and ranked our reference against the three annotators' independent transcripts, without knowing which was which. Excluding each rater's own transcript, the reference was ranked above an annotator's transcript in 60% of pairwise comparisons for Somali (95% CI 56–64%), and in 61–68% for the other three languages (50% would mean the reference is no better than a single transcript). The Somali references evaluated are those of dataset v1.0 (see the general comment).

Revision. This strengthens what the agreement study supports. We will add the cross-reference analysis and the human evaluation to Appendix F, and revise Section 7 and Appendix F to state that inter-annotator disagreement limits how absolute WER should be read, not which system differences can be detected."

Question 2: fillers and disfluencies

Thank you for this question; we should have made a distinction clearer. The references keep lexical disfluencies of spontaneous speech: repetitions (Amharic በጣም በጣም በጣም "very very very", am_0026; Tigrinya ኣነ ኣነ "I, I", ti_0159), repeated phrases (Somali waa inuu waa inuu "he must, he must", so_0108; Oromo maatii kiyyaa maatii kiyyaa "my family, my family", om_0323), and false starts where a word is broken off and restarted (Somali joog joogay, so_0025; Oromo instiraa instiraakshinaa, om_0245; Amharic አላ አላስገቡትም, am_0357). Discourse markers are kept as words too, such as Somali yacni ("I mean", also written yacnii; 62 segments). What the guideline omits are non-lexical hesitation sounds such as uh, um and eh: while developing the guideline, we found that transcribers and editors could not identify and write them consistently across the four languages, so we left them out. The paper's wording ("fillers and disfluencies are not transcribed") is broader than the guideline, and we will correct it.

We also measured whether omitting hesitation sounds penalizes systems that transcribe them. We collected the hesitation-shaped tokens that occur in system outputs but never in the references of the full test set, stripped them from every system's output, and rescored. They are rare: at most 0.3% of a system's words, or 1.7% under a deliberately broad list that also counts single-letter hesitations. Removing them changes WER by at most 0.002 (0.019 under the broad list), leaves every system ranking unchanged in all four languages, and changes the sign of none of the 58 pairwise comparisons, under both normalization schemes. (Whisper-large-v3 on Amharic is excluded: its output is mostly repeated hesitation sounds, and most of it is empty once they are removed.) Omitting them therefore does not measurably penalize such systems.

We agree that hesitation sounds are part of spontaneous speech, and omitting them is a limitation of the current version; we plan to add them in a future release.

Revision. We will correct the wording in Sections 4.5 and 7 and Appendices A and D to "non-lexical hesitation sounds are not transcribed", add this analysis to Appendix D, release the filler lists with the guidelines, and add optional hypothesis-side filler stripping to the scoring code.

Reviewer_X9i1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Horn-ASR contains 4,000 broadcast and interview segments across Amharic, Oromo, Somali and Tigrinya, totaling 15.43 hours. The authors provide reviewed transcripts and gender, dialect and domain labels, then evaluate six multilingual ASR systems using the same scoring protocol. They find higher errors than on FLEURS, a different system ranking for Amharic, and performance differences between speaker groups.

Critical strengths
  • The recordings cover a speech setting that read-speech benchmarks miss. The accompanying labels also let researchers examine which groups a system struggles with, rather than relying on one aggregate error rate.

  • The comparison with FLEURS gives a concrete reason to use the benchmark. Across eleven shared model–language pairs, the printed results support an average WER increase of about 16 percentage points. The normalization analyses and annotation study help readers interpret those scores.

Critical weaknesses
  • The interview totals in §4.7 don't match the gender and dialect counts in Table 6. Since the bootstrap resamples interviews, the authors need to clarify which IDs define the subgroups and resampling units. This could be a reporting problem; it doesn't establish that the bootstrap itself is wrong.

  • Disagreement between human transcripts doesn't establish a 0.26–0.48 WER floor for distinguishing systems. Smaller paired differences could remain stable across references. The authors could test this directly by rescoring the existing outputs against the independent and edited references on the 389 bridged items.

Questions for authors

Can you reconcile the interview IDs, labels and bootstrap grouping? On the 389 bridged items, are system comparisons stable across independent and edited references?

1 comment
AuthorsOpenReview ↗

Response to Reviewer X9i1

Weakness 1 and Question 1: interview counts

Thank you for spotting this. The mismatch comes from the released interview identifiers; the bootstrap procedure itself is unaffected.

Which ids are used. Subgroups are defined by the per-segment gender, dialect and domain labels. Confidence intervals resample interviews: each bootstrap draw samples interview ids with replacement and keeps all of each sampled interview's segments (for a subgroup, among the interviews with segments in it). Paired system comparisons resample the same interviews for both systems.

Why the counts disagree. The released speaker_id column, which we used as the interview id, was built from interview numbers that restart in each collection batch, so different interviews sharing a number were merged under one id. From the release provenance (the source recording of every segment), the test set has 1,948 interviews (Amharic 574, Oromo 623, Somali 371, Tigrinya 380), not 1,710. Separately, Table 6 counts an interview in every subgroup in which it has at least one segment, so subgroup counts need not sum to the totals.

Effect on the results. Point estimates do not depend on the clustering. We re-ran every interval with the corrected interview ids, on the same references as the paper: the same 30 of 34 system pairs are significant, no gender gap changes significance, no subgroup changes under the minimum-size rule, and the Table 4 WER intervals widen by 0.0007 on average (largest change in any bound 0.005). The re-clustered tables are in the repository's rebuttal/ folder.

In the release. Dataset v1.0 (see the general comment) adds an interview_id column (one per source recording). The camera-ready will report the corrected counts in the abstract, Section 4.7 and Table 6 and cluster on interview_id.

Weakness 2 and Question 2: stability across references

We agree, and we ran the analysis you suggested. On the 389 bridged items (Amharic 98, Oromo 96, Somali 99, Tigrinya 96), we rescored every system output against the edited (released) reference and, separately, against each of the three independent first-pass transcripts per language, with interview-clustered paired bootstrap intervals (2,000 draws, 95%).

System comparisons are stable. Of the 34 system pairs, 28 are significant against the edited reference on these items, and all 28 remain significant, with the same sign, against every independent transcript. 33 of 34 pairs have the same sign under all four references. The exception is Tigrinya MMS-1B-all vs omniASR-LLM-7B, a difference below 0.01 WER that is not significant under any reference. The system ranking is identical under every reference for Amharic, Oromo and Somali; for Tigrinya, only those two systems swap. Results are the same under the standard and the homophone-aware normalization.

Reference disagreement affects absolute WER, not system comparisons. The smallest significant difference on these items under the homophone-aware scheme, the one the agreement figures use, is Amharic Chirp 2 vs Chirp 3 (0.054 WER). It is about a fifth of the lowest inter-annotator disagreement (0.26) and stays significant under every reference. Switching references shifts absolute WER by up to 0.08 (median 0.02), but the comparisons between systems stay the same: no significant difference changes sign. The same holds on dataset v1.0 (see the general comment). We expect v1.0 to remain stable; future versions will mainly extend coverage, for example to additional dialects.

Revision. This strengthens what the agreement study supports. We will add the analysis to Appendix F and revise Section 7 (l.458–459), Appendix F (l.1016–1018) and the conclusion (l.483–484) to state that inter-annotator disagreement qualifies the interpretation of absolute WER, not the resolution of system differences. The per-reference tables are in the evaluation repository's rebuttal/ folder.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful reviews and constructive comments.

Evaluation repository. The anonymized evaluation repository cited in the paper (https://anonymous.4open.science/r/horn-asr-baselines-C765/) had expired, as Reviewer exfJ found. We apologise. It is live again at the same address and will stay available through the review period. It contains the per-utterance outputs of every evaluated system, the scoring code and the result tables, and it regenerates the paper's tables from the pinned dataset revision. The analyses in these responses are in its rebuttal/ folder.

Dataset maintenance. As planned in the paper's maintenance commitment (Appendix G.3), we have prepared dataset v1.0, the first versioned release. It adds an interview_id column (one per source recording) and incorporates the editors' latest review round of the reference transcripts. Every main claim of the paper is supported on v1.0, and the numbers change only slightly (for example, the best WER moves from 0.275 to 0.279).

Access during review. The access gate, Terms of Use and takedown contact for the dataset are prepared. We are keeping the dataset open during the review period so that reviewers can inspect it without identifying themselves, and we will turn the gate on once the review period ends.

Reference quality. Reviewers X9i1 and SWzY both ask how reliable the references are. The agreement study measures something different from reference accuracy: each of its three annotators transcribed the audio from scratch, once, whereas each reference went through several rounds of review and editing with explicit feedback (Section 4.5). We address the question in two ways. First, system comparisons are stable whichever transcript is used as the reference (response to X9i1). Second, we ran a blind human evaluation in which native-speaker expert raters listened to each segment and ranked our reference against the annotators' independent transcripts, without knowing which was which. Excluding each rater's own transcript, the reference was ranked above an annotator's transcript in 61% of pairwise comparisons for Amharic, 68% for Oromo, 60% for Somali and 67% for Tigrinya (50% would mean the reference is no better than a single transcript; the 95% intervals exclude 50% in every language). Raters also ranked the reference above their own transcript in 59 to 72% of comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.