Horn-ASR: A Spontaneous-Speech Evaluation Benchmark for Languages of the Horn of Africa

Evaluation datasets for multilingual automatic speech recognition (ASR) on African languages are constructed in ways that diverge from the spontaneous speech these systems will transcribe in use: translated sentences read aloud by fewer than ten speakers per language, held-out splits of single-speaker Bible recordings, or image-prompted elicitation. FLEURS, the most widely used of these benchmarks, covers three languages of the Horn of Africa. For each of them, our audit finds that every split contains only one gender, 3 to 7 speakers per language, and every development speaker also in the test split. We introduce Horn-ASR, an evaluation benchmark of naturally-occurring spontaneous speech (broadcasts and interviews recorded for a public audience) for more realistic evaluation of Amharic, Oromo, Somali, and Tigrinya, languages spoken by more than 100 million people. It consists of 1,000 curated segments per language (15.43 h in total) from 1,710 publicly available interviews, balanced for gender and diverse across dialect and topical domain. We evaluate six multilingual ASR systems from three providers under a single evaluation protocol. Horn-ASR is harder than FLEURS and far from saturated: the best word error rate (WER) on any language is 0.275, and where a system is scored on the same language in both benchmarks, its WER is on average 0.16 higher on Horn-ASR. It supports disaggregated evaluation by gender and dialect, which single-gender read-speech datasets cannot, and reveals significant gender gaps in every language as well as dialect gaps of up to 0.22 WER that favor the dominant variety. An inter-annotator agreement study measures the reliability of the reference transcripts and labels. We release the transcripts, labels, and guidelines (CC BY-SA 4.0), the audio (under terms of use), the curation methodology, and all system outputs and scoring code.