- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The work provides a large-scale dataset consisting of teacher and child speech, which will be accessible to the public. It contains 227 hours of audio from 130 classrooms, with annotations on transcripts and speaker roles. Utterance timestamps are also annotated for a subset. They evaluate ASR and Diarization techniques, and analyze how those errors from ASR and diarization relate to some downstream metrics, such as number of turns or mean length of utterance.
Critical strengths
The scale and diversity of the dataset (220 hours, 130 classrooms), along with ASR and diarization annotations, would be very useful for speech AI development for children.
Critical weaknesses
The following weaknesses stand out:
- They use diarization model outputs and compare them against the number of turn-taking, but I think this is not technically solid. The diarization models used are trained to diarize child and adult voices, capturing backchannels and separating pauses. The original objective of the diarization models that the authors used is not to diarize the speech into turns.
- Most of the main findings from the experiments are quite self-evident and do not provide substantial insights. For example, many prior works have shown that child speech is harder to recognize from ASR models before and after adaptation. In addition, “measure of utterance length survives recognition errors” or “measures of which words children use do not” would be quite obvious if we just break down WER from ASR into insertion, deletion, and substitution errors.
- I do not see “reframing valuation for educational use” to be a substantial contribution. The metrics picked for “educational use” are not comprehensive, and there are a lot more analyses and metrics that can be derived from ASR and diarization outputs. Redefining the ASR/Diarization evaluation around the finite set of chosen metrics for “educational use” is not very helpful.
- The paper seems substantially misaligned with the ICLR conference. The main contribution is the dataset and analysis, not machine learning or representation learning contributions. Furthermore, while the dataset is helpful for the child speech community, it does not facilitate fundamental machine learning advancements. I think there are more aligned venues (e.g., Interspeech long paper, Computer Speech & Language).
- Less critical, but the authors could also evaluate more state-of-the-art ASR models, such as Parakeet-tdt.
Questions for authors
The Cohen’s κ of 0.83 for utterance-level speaker-role labels seems a bit low (around 5% of annotations disagree), given that it is fairly easy to distinguish child vs adult speech, and some prior child-adult diarization works report ~5% speaker confusion rate. Do you have some insights on the inter-rater disagreements?