- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces HORN-ASR, an evaluation benchmark for automatic speech recognition (ASR) in four under-resourced Afroasiatic languages of the Horn of Africa: Amharic, Oromo, Somali, and Tigrinya. It contains naturally occurring spontaneous speech (broadcasts and interviews recorded for a public audience), with 1,000 segments per language. The paper is also partly a position paper. It rightly criticises massively multilingual benchmarks such as FLEURS, which consist mostly of read speech (from translated text, also called “translationese”) with little speaker variability. The authors argue that ASR performance on such data underestimates errors on spontaneous speech and can rank systems differently, and support this by benchmarking six multilingual ASR systems on HORN-ASR (not all systems cover all four languages). They also question the extreme aggregation of scores into a single number (common practice in massively multilingual ASR eval) and advocate more detailed evaluation, for example by gender and dialect, when metadata allows.
Critical strengths
-
The positioning is very good. The paper highlights the shortcomings of current massively multilingual ASR evaluation and suggests what could be done better.
-
HORN-ASR is, to my knowledge, the first public ASR benchmark of naturally occurring spontaneous speech for these four languages. It is balanced for gender and diverse across dialect and domain, which makes the evaluation more realistic.
-
The data collection and annotation methodology is described precisely.
-
The evaluation of six current systems shows that HORN-ASR is harder than FLEURS (clearly so for Amharic and Somali) and can rank systems differently. It also reveals gender gaps (whose direction depends on the system) and dialect gaps favouring the dominant variety in Amharic, Oromo, and Tigrinya (not significant for Somali), which no existing benchmark for these languages could show before.
Critical weaknesses
-
Rights and consent. The authors describe their copyright position as a research-use argument, not a legal determination, and say they do not claim that public availability implies consent (Ethics Statement). The paper does not mention permission from broadcasters or speakers. On what basis can the transcripts be published ungated under CC BY-SA 4.0?
-
Repository not accessible. The anonymous repository link returned “This content can’t be shown. The repository is expired.” According to the datasheet, the dataset link is in that repository’s README, so I could not check any of the release.
Questions for authors
For FLEURS, the paper says all systems were re-run under one protocol, and checks this against the published MMS Somali number. For MMS Amharic, however, Table 10 lists 53.3 self-reported against 0.383 verbatim here. Why this difference? Sec. 6.3 also cites “0.428 on FLEURS (Table 12)”, which does not appear in that table.
Remark on venue: I appreciate bringing this discussion to the ICLR audience, but I wonder about its impact there. LREC-COLING or Interspeech would likely give the work a more relevant audience and more useful feedback. I am ambivalent about ICLR as the venue.
2 comments
acknowledgement of reviewers' response
I acknowledge the authors’ response and thank them for the clarification. I think this paper is a good example of the right way to address the challenge of low-resource languages: rather than racing for massive multilinguality, it does things properly for a few languages, evaluates the technology in realistic conditions (real conversations), and carefully documents the data collection methodology.
Response to Reviewer exfJ
Weakness 1: rights and consent
Thank you for raising this; we agree it needs a precise answer, and we have improved the access model described in the paper.
Transcripts will not be published ungated. The transcripts reproduce the words spoken in the broadcasts, so the rights questions that apply to the audio apply to them too. At release, the whole dataset, audio and transcripts, is distributed behind an access agreement (an automatically approved click-through, so access stays easy for researchers in the region). The Terms of Use restrict use to evaluation and research and prohibit training on the data, speaker identification or profiling, voice cloning, and redistribution, and require deleting data after a takedown. CC BY-SA 4.0 will cover only what we created independently of the speech: the gender, dialect and domain labels and the other metadata. We will state this split on the dataset card and correct the abstract and the datasheet. During the review period the Hugging Face dataset repository stays open so that reviewers can inspect it anonymously; the gate and Terms of Use are prepared and will be switched on once the review period ends.
Basis. We rely on a research-use argument, not on a licence or permission from the broadcasters, which we did not obtain, and, as the Ethics Statement says, not on any claim that public availability implies consent. The argument rests on minimization and control: each source recording contributes at most 11 short segments (at most about 2.5 minutes of audio; a median of 1 to 2 segments, averaging 14 s each, usually from different parts of the recording rather than one continuous stretch), which cannot substitute for the original broadcast; the data is evaluation-only; and a takedown contact on the dataset card handles removal requests from rights holders and from speakers themselves.
Revision. We will add these points, the licence split, the access terms and the takedown process to the Ethics Statement and the datasheet.
Weakness 2: repository
Thank you for reporting this. The repository is live again at the same address and will remain available through the review period (see our general comment). Its README links the dataset revision used in the paper, so the release can be checked and every table reproduced from the stored system outputs.
Question: MMS Amharic and the 0.428 citation
Thank you for the careful reading. There are two reporting problems, in Table 10 and Section 6.3, and a difference in the Amharic number that we trace to the system outputs rather than the scoring.
Units. The self-reported column of Table 10 prints the published numbers as percentages (53.3, 140.3) next to our fractions. The comparable figures are 0.533 reported vs 0.383 measured for Amharic, and 0.538 vs 0.538 for Somali. We will convert the column to fractions.
Why Amharic differs. Scored verbatim against the FLEURS
raw_transcription, our MMS-1B-all outputs reproduce the published Somali number exactly (0.538), but give 0.38 for Amharic. No scoring variant brings Amharic close to 0.533: the published caption's Whisper normalization gives 0.295, the FLEURS normalized transcription 0.297, and our normalization 0.282. This does not affect our conclusions: the Horn-ASR vs FLEURS gap compares our own runs of the same checkpoint on both benchmarks, with the same inference and scoring, so it does not depend on the published number.0.428. This is the WER MMS reports for Somali with the Common-Crawl LM on FLEURS (Pratap et al., Table A1), not a number in our Table 12, which holds our Horn-ASR LM results. The citation in Section 6.3 should point to their Table A1; we will fix it.
Remark: venue
Thank you for the remark. We chose ICLR because the systems we evaluate (MMS, Omnilingual ASR, Whisper, Chirp) are built and reported in this community, mostly through aggregate FLEURS numbers, and our results show that those numbers under-estimate error and can rank systems differently for these languages. The released benchmark is open to the speech community as well.