- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PROCOACH, an audio-language model (ALM) framework for Mandarin and English pronunciation coaching. PRONCOACH operates directly on learner audio without a reference transcript to recover intended lexical content, infer realized pronunciation, account for acceptable contextual variations, localize pronunciation corrections, and generate feedback.
The authors described a pipeline to generate the training reasoning data for the ALM, involving intended-content recovery, realized phoneme recognition, grammar diagnosis, contextual comparison, and per-phone correction, in which several teacher ALM and supervised models are used to generate the labels. Then the Qwen3.5-Omni-based ALM was trained using supervised fine-tuning (SFT) and Reinforcement Learning with GRPO. Two variants "Thinking" and "Instruct" are investigated.
Furthermore, a pronunciation coaching benchmark PRONCOACH-BENCH is propsoed, which includes Diagnostic track and Clean-speech track. Empirical results show that the proposed model outperforms several leading ALMs.
Critical strengths
- Well-Motivated problem: Moving from transcript-dependent phone recognition/scoring to reference-free, context-aware pronunciation coaching addresses a genuine gap.
- The construction of the CoT data looks solid and reasonable: It involves several teacher ALM and strong supervised models to annotate a larger speech pool.
- The results look strong. It outperforms several ALMs on the proposed benchmark and phoneme recognition and MDD.
Critical weaknesses
- I think many important details are missing, especially in the CoT data construction pipeline. For example, for realized phoneme recognition, it is mentioned that several human annotators have annotated the realized phoneme. Why is the fine-tuned HuPER still needed? How reliable is the phoneme recognizer? What is it PER on non-native speech? What is the size of the training data and how is this model trained? I think many details about this model are missing.
- For HuPER fine-tuning, it is mentioned that "We fine-tune language-specific HuPER (Guo et al., 2026) models on expert annotations to extend pronunciation coverage and estimate candidate probabilities. Both combine canonical pronunciation with acoustic evidence from WavLM for Mandarin (Chen et al., 2022) and HuBERT for English (Hsu et al., 2021)." It is still unclear to me how exactly this architecture looks like. How does it compare to the original HuPER? This could an important technical contribution if others want to reproduce this realized phoneme recognizer.
- The metrics looks unintuitive to me. It is defined as the harmonic mean of Recall and Acc. Acc is further defined as "predicted–reference realized-pronunciation agreement at correctly localized positions", which is a bit perplexing to me. This seems to be totally ignoring the false positives, i.e. the model could claim many unrealized phonemes and it will not get penalized.
Questions for authors
- Per my understanding, the model internalizes the alignment between canonical and realized phonemes in its output. i.e. the output directly writes out the phoneme pairs that are mispronounced, and there is no need to do an explicit alignment between the canonical phonemes and realized phonemes and then compare. In fact, for the claimed transcript-free assessment case, the GT canonical phonemes are unknown. How do you define "correctly localized positions"?
- What is the motivation of the RL stage? Wouldn't the SFT model trained on the CoT data do the same job? Maybe a comparison between SFT vs. SFT+RL could be informative here.
5 comments
Response to Reviewer PqMV [2/3]
W2 Fine-tuned HuPER architecture
We use different models for Mandarin and English, rather than the same architecture with different data. The canonical pronunciation supplies "what should be pronounced at this position" as the query, the acoustic side supplies evidence from the recording through cross-attention, and the output heads predict the realized pronunciation or edit operations.
| Mandarin | English | |
|---|---|---|
| Acoustic encoder | WavLM-large, trained | HuBERT-large, frozen |
| canonical query | Sum of onset/rime/tone slot embeddings, learned positional encoding | Phone embedding, sinusoidal positional encoding |
| Acoustic interaction | 4-layer, 8-head TransformerDecoder | 8 layers, 8 heads, self + cross attention |
| Output and supervision | Three linear heads, weighted soft CE against expert vote distributions | Two heads for KEEP/DEL/SUB and insertion, hard CE |
Both use d=512, FFN 2048, and dropout 0.1. The Mandarin model additionally uses canonical-dropout 0.3: during training, the query is randomly replaced with a learned MASK, forcing the model to listen to the audio rather than copy the canonical pronunciation. The deviation weight is 2, agreement gamma is 0, and all three slot weights are 1.
Compared with the original HuPER (in which the DRRC module is responsible for "recognizing the realized pronunciation from audio and comparing it with the canonical pronunciation to identify deviations"), we made four changes: for Mandarin, we added an output structure with onset, rime, and tone slots and changed the loss from single-label supervision to fitting expert vote distributions, which is a prerequisite for retaining candidate probabilities; for English, we predict edit operations instead of directly emitting a phone sequence.
We plan to open-source the relevant realized-pronunciation model code after acceptance to help the open-source community reproduce the realized-pronunciation recognizer in this area.
W3 F1_hear
F1_hear is the harmonic mean of recall and ACC_hear, while ACC_hear is calculated only at correctly localized positions. Therefore, the number of additional corrections the model makes at positions that should not be corrected does not affect ACC_hear. We aim to evaluate these aspects separately, using the clean track to assess the model's ability to avoid unnecessary corrections. If the model corrects phones that were not actually produced, it is penalized on the clean track, as we report for Acc_c.
We will include precision in the revised manuscript: a hit is counted only when all three conditions are met—the correction is localized to a reference correction position, target is a valid canonical pronunciation at that position, and the first heard belongs to the set of valid realized pronunciations at that position.
$ P=\frac{TP}{N_p},\qquad R=\frac{TP}{N_g},\qquad F_1=\frac{2,TP}{N_g+N_p} $
For the main model in Mandarin: Np 1,634, TP 776, FP 858, and FN 542, giving P 0.4749, R 0.5888, and F1 0.5257. English F1 is 0.3277. On the Clean track, the results are 36/81 for Mandarin and 55/97 for English. Several internal configurations under the same definition are: Discrete, Mandarin 0.2531 (Clean 77/81), and Scale, 0.6786 (Clean 28/81). Discrete has the highest precision and the lowest recall, while Scale is high on both, consistent with the effect of data upscaling. A future direction may be to explore how to improve the model's ability to avoid unnecessary corrections while maintaining high recall.
Response to Reviewer PqMV [3/3]
Q1 Text alignment
The main model receives learner audio and a coaching request, recovers the content itself, and directly writes "which character was mispronounced, what it should sound like, and how it was actually pronounced." There is no explicit alignment between canonical and realized sequences at inference.
Reference-free describes the information supplied to the evaluated system, not the absence of references for the offline evaluator. Main writes its intended text and pronunciation diagnoses directly. Data construction uses explicit phonetic alignment to prepare supervision; deployment produces the answer; offline scoring localizes the stated corrections against a fixed reference transcript. These are three different procedures.
The final PronCoach-Bench includes reference text, but this text is not supplied at model inference. This also tests the large model's basic semantic reasoning and contextual understanding. Phone alignment is performed only after matching the text it generates.
The submitted localization procedure matches normalized text units and target pronunciation, with a surface fallback and deterministic handling of repeated units. Corrections that cannot be placed are recorded separately. Reference text and correction labels are not supplied to Main.
There is a point that is easy to confuse and that we did not previously explain clearly: reference-free constrains the input the user supplies to the model, not the evaluator. The model has no access to the reference transcript. The evaluator has reference text and compares the manually transcribed text with the model-predicted text to associate corrections with individual reference positions. During scoring, the text unit in a model report is located in the reference transcript, and a frozen monotonic search then makes a one-to-one assignment, ensuring that reference positions correspond uniquely to the text understood by the large model.
Q2 Motivation for the RL stage and comparison between SFT and SFT+RL
The initial motivation for RL training was to address two issues. The first was misaligned corrections. During SFT-only training, we found the following problem in the model trained on instruct data: it could produce almost correct results on the separate realized-pronunciation and target-pronunciation tasks, yet exhibited severe missed corrections or excessive corrections in the coach task—it instead made corrections at positions where the realized and target pronunciations were the same, or missed corrections at positions where the preceding two separate tasks produced different pronunciations. We attributed this to a conflict, for the model, between realized and target pronunciations in the coach task. We therefore hoped to address this through a reward that compares the two paths and rewards recall and canonical. However, during subsequent parallel development, we found that CoT data could come close to resolving this issue effectively, and that the capabilities learned from CoT data could transfer fairly well to instruct tasks. We therefore did not further design rewards to enhance this capability through RL. The second issue was json-format errors. We found occasional json parse errors in the coach_json task, so we added a format parse reward at the RL stage. These were our two initial motivations for the RL stage.
We added a comparison experiment to demonstrate the role of RL. As can be seen, with instruct-only data without RL, the metrics of correction correctness, P_loc and ACC_hear, do not decrease much, but R and FC decrease substantially. The model makes many unnecessary corrections, substantially worsening its performance.
| Model | Language | R ↑ | P_loc ↑ | F1_loc ↑ | ACC_hear ↑ | F1_hear ↑ | FC ↓ | ACC_c ↑ | Zero corrections | Overall ↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| PronCoach | Mandarin | 0.6980 | 0.5913 | 0.6402 | 0.9500 | 0.8048 | 0.2523 | 0.3457 | 36/81 | 0.7142 |
| English | 0.4967 | 0.4054 | 0.4464 | 0.8000 | 0.6129 | 0.0785 | 0.5670 | 55/97 | ||
| w/o GRPO | Mandarin | 0.5447 | 0.5695 | 0.5568 | 0.9187 | 0.6839 | 0.5842 | 0.0494 | 4/81 | 0.5865 |
| English | 0.4305 | 0.2364 | 0.3052 | 0.3846 | 0.4063 | 0.1442 | 0.4021 | 39/97 |
Response to Reviewer PqMV [1/3]
Thank you for your insightful suggestions and comments. We have revised the manuscript to address your concerns, and we provide detailed responses below.
W1a Why fine-tune HuPER?
Because the numbers of recordings they cover differ by an order of magnitude. Expert character-by-character annotations cover 16,811 Mandarin recordings and 9,900 English recordings; the speech pool used to construct the coaching data is much larger. HuPER transfers the probability-distribution-aware perception learned from expert annotations to audio without fine-grained annotations. Using multiple experts for annotation is less efficient and more expensive. The HuPER model can better represent continuous probability distributions rather than discrete vote counts alone. Over continuous probability distributions, the model can better "preserve the distribution." What is passed downstream is not HuPER's single best solution, but the candidate probabilities for each slot. The probabilities for the three Mandarin onset, rime, and tone slots are calibrated (ECE: 0.0094, 0.0076, and 0.0302, respectively), so the "dispersion of votes" itself carries information: when the ALM receives a tone distribution in which each of three candidates has a share of the probability, it knows that the character's tone is inherently ambiguous, rather than seeing a single discrete argmax. This information is lost if the distribution is compressed to an argmax before being passed downstream.
W1b Recognizer reliability
We sampled an agreement test set: 700 recordings covering seven accents, each independently transcribed by two annotators and adjudicated by one reviewer.
Calculated as per-utterance phone edit distance divided by the reference length:
The PER between the two annotators is 0.1447. In other words, when the same accented recording is transcribed by two trained people, their transcriptions differ by approximately 14%; the recognizer differs by 19% on recordings it has not been trained on.
The Mandarin component outputs character-level classification over three slots—onset, rime, and tone—rather than an IPA phone sequence. AUROC is 0.970 for onset, 0.952 for rime, and 0.782 for tone; deviation-detection recall is 0.664 / 0.553 / 0.079.
HuPER's tone predictions tend to be conservative. Across all Mandarin annotations, the three annotators disagree at only 4.7% of onset and rime positions, compared with 30.5% for tone.
W1c Training scale and training procedure
Mandarin branch: 16,811 training and 881 test supervision records, corresponding to 244,171 / 12,812 Chinese characters, or 732,513 / 38,436 onset–rime–tone slots. Training uses 8 GPUs, batch 8 per GPU, gradient accumulation 2, and at most 12 epochs. The classification-head learning rate is 6e-4 and the encoder learning rate is 1e-5. Checkpoint selection and early stopping use macro-AUROC, with patience 4; ECE is examined separately for calibration.
English branch: 9,900 / 521 supervision records, 2,256 / 128 distinct recording IDs, and 284,112 / 15,252 canonical phones. HuBERT is frozen; only the interaction layers and edit heads are trained. We use the weights after epoch 7.