acceptodds
Under review as a conference paper at ICLR 2027

From Error Rates to Educational Measures: Benchmarking Speech AI for Teacher–Child Interaction in Preschool Classrooms

Abstract

The quality of teacher–child interaction is a central determinant of early development, yet assessing it still depends on trained experts observing classrooms in person, which cannot scale. Speech AI could automate this: transcribe classroom talk, attribute it to the teacher or children, and compute the measures that interaction assessment relies on, such as talk time, turn-taking, and vocabulary. Yet speech systems are evaluated by transcription and diarization error rates, not by whether the measures they produce match those coded by humans. In this paper, we introduce MPCS, to our knowledge the first benchmark that pairs verbatim transcripts with continuous speaker-role timelines over real preschool classroom audio: **227** hours of teacher and child speech (ages 3–5) from **130** classrooms. Evaluating 14 speech recognizers and four speaker-role models, we find that children produce 16% of the characters but 39% of the strongest recognizer's errors, a gap that model scale and in-domain fine-tuning do not close. Lower error rates also do not guarantee better measures: a speaker-role model with 1.4% diarization error still overcounts teacher–child turns by 17%. What matters is the kind of error. Classroom-adapted systems measure how much teachers and children talk to within 1%, but miscount turns and overestimate children's vocabulary. Code is available at [this anonymous repository](https://anonymous.4open.science/r/MPCS-Bench-68B2).

Primary area:
datasets and benchmarks
Keywords:
child speech recognition, classroom speech, speaker role diarization, Mandarin speech corpus

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_tdMZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The work provides a large-scale dataset consisting of teacher and child speech, which will be accessible to the public. It contains 227 hours of audio from 130 classrooms, with annotations on transcripts and speaker roles. Utterance timestamps are also annotated for a subset. They evaluate ASR and Diarization techniques, and analyze how those errors from ASR and diarization relate to some downstream metrics, such as number of turns or mean length of utterance.

Critical strengths

The scale and diversity of the dataset (220 hours, 130 classrooms), along with ASR and diarization annotations, would be very useful for speech AI development for children.

Critical weaknesses

The following weaknesses stand out:

  1. They use diarization model outputs and compare them against the number of turn-taking, but I think this is not technically solid. The diarization models used are trained to diarize child and adult voices, capturing backchannels and separating pauses. The original objective of the diarization models that the authors used is not to diarize the speech into turns.
  2. Most of the main findings from the experiments are quite self-evident and do not provide substantial insights. For example, many prior works have shown that child speech is harder to recognize from ASR models before and after adaptation. In addition, “measure of utterance length survives recognition errors” or “measures of which words children use do not” would be quite obvious if we just break down WER from ASR into insertion, deletion, and substitution errors.
  3. I do not see “reframing valuation for educational use” to be a substantial contribution. The metrics picked for “educational use” are not comprehensive, and there are a lot more analyses and metrics that can be derived from ASR and diarization outputs. Redefining the ASR/Diarization evaluation around the finite set of chosen metrics for “educational use” is not very helpful.
  4. The paper seems substantially misaligned with the ICLR conference. The main contribution is the dataset and analysis, not machine learning or representation learning contributions. Furthermore, while the dataset is helpful for the child speech community, it does not facilitate fundamental machine learning advancements. I think there are more aligned venues (e.g., Interspeech long paper, Computer Speech & Language).
  5. Less critical, but the authors could also evaluate more state-of-the-art ASR models, such as Parakeet-tdt.
Questions for authors

The Cohen’s κ of 0.83 for utterance-level speaker-role labels seems a bit low (around 5% of annotations disagree), given that it is fairly easy to distinguish child vs adult speech, and some prior child-adult diarization works report ~5% speaker confusion rate. Do you have some insights on the inter-rater disagreements?

Reviewer_aLsmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work introduces MPCS, a corpus of 227h of Mandarin preschool classroom speech from 130 classroms in China, recorded with devices worn by teachers and transcribed with teacher/child role annotations, with a smaller subset with continuous teacher/child timelines. The work additionally evaluates several models from scratch and upon finetuning and presents several findings with regards to the nuances of the setup and the data. One of the critical findings was that the CER rate for kids stays high despite finetuning and there exist several limitations to existing corpuses and models which limits transfer to improve performance.

Critical strengths
  1. The resource is novel and addresses some critical limitations in a field that is data starved and hampered by guidelines as recognized by several existing works.
  2. The out-of-domain nature of kids' speech is analyzed and addressed with care. The work shows that at fixed CER, substitutions preserve mean utterance length but that current systems' errors on children are substitution-dominated, so length-based measures survive in practice while vocabulary measures do not, and that the perturbation study quantifies how much deletions, insertions, and segment length matter at a fixed error rate. The recommendation to report the three error types by role alongside CER and DER, and to treat time, count and word-based measures differently, follows from the data.
Critical weaknesses
  1. The claim that in-domain data cannot close the gap rests on a finetuning result that may potentially arise from a poor recipe. Fine-tuning moves Whisper's child CER by 24–34 points, but moves Qwen3-ASR-1.7B's from 28.32% to 28.22% (Table 4), and in Table 5 both Qwen3 models are worse than zero-shot after cross-corpus fine-tuning. This pattern may indicate poorly calibrated setups and a sweep analysis over the SFT procedure should help make the case. It would be very interesting if the pattern survives and could speak to the limitations of the CPT procedures of existing models.
  2. The work leans on CLASS and SSTEW but not on comparison to an expert rating. The system-computed measures compared against the same measure from human transcripts leaves out the quality assessment (of turn-count for eg.) and only tracks how well the system reproduces the human's counts, not that the numbers correlate with the scores a CLASS or SSTEW observer would give the classroom, which should help the "educational measures" portion of the claims.
  3. The title and abstract don't state that the corpus is limited to Mandarin and a clarification earlier is helpful. The related work should discuss very relevant works that are missing. Benchmarks of speech foundation models on children's ASR [2, 3], the child–adult gap in conversational recordings [1, 4], and existing classroom transcript resources and measures that have been validated against instructional-quality ratings [5, 6], which bears directly on the CLASS/SSTEW framing.
  4. The work reports CER and DER on non-overlapping segments and the mean utterance lengths inherit the segmenter's granularity, which is worth clarifying along with the extent of removal.

[1] Sharma, R., Liu, D., Sun, J., Zhou, S., Qin, J., Xiong, J., & Chen, C. (2025). KidSpeak: A general multi-purpose LLM for kids' speech recognition and screening. arXiv preprint arXiv:2512.05994.

[2] Fan, R., Shankar, N. B., & Alwan, A. (2024). Benchmarking children's ASR with supervised and self-supervised speech foundation models. In Proceedings of Interspeech 2024 (pp. 5173–5177). ISCA.

[3] Ying, A., Shankar, N. B., Lin, C.-J., Shi, M., Wang, P., Shim, H., Arora, S., Van hamme, H., Alwan, A., & Watanabe, S. (2025). Benchmarking training paradigms, dataset composition, and model scaling for child ASR in ESPnet. In Workshop on Child Computer Interaction (WOCCI 2025). arXiv:2508.16576.

[4] Ashvin, A., Lahiri, R., Kommineni, A., Bishop, S., Lord, C., Kadiri, S. R., & Narayanan, S. (2025). Evaluation of speech foundation models for ASR on child–adult conversations in autism diagnostic sessions. In Workshop on Child Computer Interaction (WOCCI 2025). arXiv:2409.16135.

[5] Demszky, D., & Hill, H. (2023). The NCTE Transcripts: A dataset of elementary math classroom transcripts. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023) (pp. 528–538). Association for Computational Linguistics.

[6] Demszky, D., Liu, J., Mancenido, Z., Cohen, J., Hill, H., Jurafsky, D., & Hashimoto, T. (2021). Measuring conversational uptake: A case study on student–teacher interactions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (pp. 1638–1653). Association for Computational Linguistics.

Questions for authors
  1. How are the overlapped regions handled in the measures reported? Are they removed altogether (from the denominator too) and what is the extent of this removal?
  2. Is it possible to get inter-annotator agreement metrics over these exclusion decisions?
  3. Is it possible that if kids are excluded more often, the reported CER metric is under-stated?
  4. Why does fine-tuning move Qwen3-ASR-1.7B's child CER by 0.1 points when Whisper gains 24–34? Were learning rate and epochs swept per model, and was the encoder frozen?
  5. Class sizes and number of teachers aren't reported which could affect the overlaps and the metrics.
  6. A suggestion is to add "Mandarin" to the title so the reader is aware of the coverage of this benchmark.
Reviewer_mFZ1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents MPCS, the first benchmark for speech AI in preschool teacher–child interaction. The benchmark contains 227 hours of annotated classroom audio from 130 classrooms. It contrasts traditional transcription/diarization error rates with downstream educational interaction metrics (talk time, turn-taking, vocabulary). Evaluations on 14 ASR and 4 speaker-role models reveal children's speech drives most recognition errors, and low error rates do not guarantee reliable educational measurements. It highlights that error type matters more than overall error rate.

Critical strengths
  1. This work points out a crucial evaluation gap: speech models are judged by raw error rates, not the applied behavioral/educational metrics researchers actually care about.

  2. Real preschool classroom audio (3-to-5-year-old children) is hard to collect and annotate. 227 hours data across 130 classrooms is a very useful resource for the research community.

  3. The core takeaway ("low diarization/transcription error not equals to trustworthy downstream measures") is practically meaningful for speech and education AI.

  4. It identifies that child speech disproportionately contributes to errors, even after scaling or in-domain fine-tuning, pinpointing a key failure mode.

  5. Public benchmark and code enable follow-up researches on child speech recognition and classroom interaction measurement.

Critical weaknesses
  1. The paper mainly disucsses the problem rather than proposing a new speech/measurement model to mitigate the identified bias. It is primarily a benchmark paper without a technical solution.
  2. This work only covers talk time, turn-taking and vocabulary, without linking to validated early-childhood interaction frameworks.
  3. It focuses on preschool classrooms, and it is unclear whether the same mismatch between error rate and downstream metric holds for other child-adult speech settings.
  4. It reports aggregated gaps (e.g., 17% overcount on turns) without discussing variance across classrooms or reliability of human gold coding itself.
Questions for authors
  1. Could you elaborate on what specific error types are most damaging to turn counts versus vocabulary estimation?
  2. Why do in-domain fine-tuning and scaling fail to close the large error gap for child speech? Is it because of the limited child speech data?
  3. Would you expect your core conclusion to transfer to other child speech tasks, such as home parent–child interaction?
  4. Could you discuss how practitioners should adjust their evaluation pipelines given your findings? What alternative evaluation protocol would you recommend for classroom speech AI?
Reviewer_scv1OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Motivated by automating quality-of-instruction measures, this paper presents a benchmark dataset for evaluating in-classroom Mandarin preschool teacher & student speech. 227 hours of audio is collected from 130 classrooms (30 minute sessions). They evaluate 14 speech recognition systems and find that child speech produces the most errors, that low error rates do not correspond with downstream metrics like counting teacher-child turns.

Critical strengths

-Prior work has extensively examined the feasibility, and downstream effects of, automatically transcribing teacher and student speech from recordings of live classrooms; however, due to privacy concerns, such data is almost never publically released, which makes MPCS unique. -The paper goes beyond simple error metrics and considers downstream performance metrics as essential for understanding task feasibility---this is where the most interesting results are presented. -Authors are thorough with controlling for variables throughout their analyses and reporting evaluations of performance at substeps (e.g., agreement on a subset of transcribed-from-scratch utterances) prior to the automation for scale.

Critical weaknesses

-Timing is the biggest indicator, which arguably implies the transcripts themselves are minimally useful for the metrics being explored. This makes sense, but the authors could add additional metrics where transcripts are more meaningful, thus making the data itself more meaningful. For example, context dependent metrics like question type, depth of question (authentic versus not), whether a response was a followup, etc. would be far more interesting. -While the dataset is novel, the work does not engage with the majority of works examining ASR in classroom-discourse. For example, Donnelly et al., 2017, Southwell et al., 2022, and other works from AIED, EDM, and LAK conferences. This work is still distinct (the majority of that work is K-12, english, and conducted on datasets that are not able to be public). Much of these works have already established the findings presented here, but would still be interesting to cite as evidence that those findings are consistent cross-culture and cross-language. -Authors cite dataset composed of 130 but only 83 are used? -I am weary of the fine-tuning results wrt the claim that fine-tuning fails---this is a strong claim but it does not appear to me experiments with fine-tuning were extensive enough to support it.

Questions for authors

-What kinds of activities were taking place in classrooms when recordings were selected? Were activities limited in any way? For example, child-to-child talk during free play might be more difficult. -Were additional efforts made to fine-tune beyond what is reported in the paper? -How much teacher-child speech was excluded? -Annotated exerts of transcripts + timelines would often help with the clarity of the manuscript by visualizing the various error types

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.