acceptodds
Under review as a conference paper at ICLR 2027

Input Embeddings as a Compositional Pronunciation Interface for Frozen LLM-Based Chinese TTS

Abstract

LLM-based text-to-speech systems can misread a character even when its intended pronunciation is known. We study input embeddings as a pronunciation interface for a frozen speech model. A closed-form additive regression maps initials, finals and tones to the embeddings of common characters. To construct a target embedding, we start from real donor rows and add the regression-estimated phonological difference between donor and target. This preserves information in real rows while changing the specified reading; the TTS parameters remain frozen and no recording of the target pronunciation is required. On 120 held-out common-word items, excluding the target's entire initial–final combination in all tones from both regression fitting and donor selection, composition reaches 94% pronunciation accuracy on CosyVoice2 versus 88% for regression alone, with the same ordering on Spark-TTS and GLM-TTS. Under the same exclusion, composition repairs 77% of 193 verified rare-word errors versus 65% for regression alone. In ordinary use, a unified rule that reuses homophones when available repairs 79%, versus 70% for text rewriting. These results connect compositional control to real pronunciation repair, while revealing a trade-off between donor availability and damage to correctly read words.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
text-to-speech, pronunciation control, training-free, tokenization, embedding initialization, compositional generalization, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qjaBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free method for controlling pronunciation in frozen Chinese TTS models through input embeddings. It uses phonological features and donor-character embeddings to construct embeddings for target pronunciations. Experiments on held-out phonological combinations, rare-word correction, and human evaluation show promising results, although the benefits over simpler regression are less consistent.

Critical strengths
  1. The phonological withholding experiments and donor-only controls provide convincing evidence that the proposed embedding transformation can steer pronunciation beyond simple homophone lookup.

  2. The paper evaluates real pronunciation errors and includes human listening studies. On 193 verified errors, composition repairs 77%, compared with 65% for regression alone.

Critical weaknesses
  1. Format is wrong. Lack of novelty.

  2. The advantage over regression remains unclear. The two methods are not independently optimized, and regression even outperforms composition in one longer-context setting (98% vs. 95%). The improvement is also statistically inconclusive in the no-common-homophone human evaluation (36/45 vs. 32/45).

  3. Most sentence-level preservation experiments use homophone averaging rather than the proposed donor-based composition. Given that composition breaks 26% of previously correct words in the complete-withholding setting, its impact on surrounding speech and overall naturalness needs more evaluation.

Questions for authors
  1. Does composition still outperform regression when both methods are independently tuned with the same budget?

  2. Why are five composition outputs missing from Table 10? What would the results look like on all 50 words, including the regression fallback?

  3. Could the authors compare composition, regression, and the original model in matched natural contexts, reporting pronunciation accuracy, collateral errors, and naturalness?

Reviewer_kQb6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a compositional embedding method for pronunciation control in frozen LLM-based TTS models. It combines real donor embeddings with predicted phonological offsets to correct mispronounced Chinese characters without model retraining. Experiments on three TTS backbones show promising results: composition achieves 94% accuracy on controlled tests and repairs 77% of 193 verified errors, outperforming regression alone. However, limited comparisons with existing methods, reliance on automatic evaluation in some experiments, and collateral errors leave its generalization and practical benefits insufficiently established.

Critical strengths
  1. An interesting and lightweight intervention mechanism. The method constructs pronunciation-specific embeddings by combining real donor rows with a predicted phonological offset. The TTS model remains frozen, and no target-pronunciation recording is required. This provides a simple and interpretable alternative to methods requiring model retraining or weight editing.

  2. Controlled evidence for compositional generalization. By withholding the target's entire initial–final combination across all tones from both regression fitting and donor selection, the experiments provide evidence that the method does more than retrieve an existing homophone. On the controlled CosyVoice2 test, composition achieves 94% accuracy compared with 88% for regression alone. The ordering is also observed on Spark-TTS and GLM-TTS.

  3. Evaluation on actual pronunciation errors. The paper goes beyond embedding probes and evaluates verified rare-word errors. Under complete phonological-combination withholding, composition repairs 77% of 193 errors, compared with 65% for regression alone. It also examines collateral errors, ordinary-sentence preservation, and human listening judgments, providing a more informative assessment than controlled pronunciation accuracy alone.

Critical weaknesses
  1. Limited comparison with existing pronunciation-correction methods. The empirical comparisons primarily involve regression alone, donor-based controls, internal text rewriting, resampling, and a trained pinyin channel in one configuration. The paper does not provide an empirical comparison with several existing post-hoc editing approaches discussed in the related work. Consequently, it remains unclear how the proposed method compares with the broader state of the art under matched information and evaluation conditions.

  2. Incomplete evidence for generalization and practical benefit. The experiments cover several TTS backbones, but some secondary evaluations use only one sampling, and three evaluated systems share a tokenizer. The Korean extension is restricted to a subset selected by reference transcribability and uses automatic pronunciation labels. In addition, complete phonological-combination withholding increases damage to previously correct words from 13% under the ordinary unified method to 26%. The results therefore do not yet establish robust, architecture-independent performance or a favorable overall error trade-off across realistic deployment settings.

Questions for authors
  1. Comparison with existing methods: Can the authors provide controlled empirical comparisons against relevant post-hoc pronunciation-editing or conditioning-editing methods, using the same target readings, access to reference information, and evaluation protocol? This would clarify the method's practical advantage over existing alternatives.

  2. Reliability of pronunciation evaluation: Several results depend on ASR-based or calibrated automatic judgments, and the paper reports discrepancies between automatic labels and human listening results. How sensitive are the principal conclusions to stricter human-verified evaluation across the full set of repaired and unrepaired items?

  3. Collateral errors and deployment: Given that complete phonological-combination withholding increases the rate of damaging previously correct words, can the authors quantify the net benefit across realistic mixtures of correct and incorrect pronunciations and investigate a reliable criterion for deciding when an embedding intervention should be applied?

Reviewer_D2RbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies pronunciation control through the input embeddings of frozen LLM-based TTS models. It fits an additive map from phonological features to common-character embeddings, then constructs a target vector by adding the predicted pronunciation difference to real donor embeddings. Experiments examine both controlled pronunciation composition and repair of naturally mispronounced words, with additional Korean results. The main finding is that a frozen speech model can produce the intended reading even when that reading’s initial–final combination is excluded from the construction’s fitting data and donor pool.

Critical strengths
  1. The controlled composition experiment is well motivated. Excluding the target’s entire initial–final combination from both fitting and donor selection provides a meaningful test beyond homophone lookup. The donor-only, nearest-row and random-vector controls help establish that the phonological transformation matters.

  2. The authors calibrate automatic pronunciation judgments, compare against fresh resampling, and include several listening studies. They also disclose negative results, incomplete coverage and collateral errors. The supplementary guide documents audio pages, anonymized ratings and scripts for reproducing the listening analysis.

Critical weaknesses
  1. Novelty: McLaughlin et al. already demonstrate phonetic editing through target-minus-source directions in input embedding space. Vocab Diet also constructs embeddings from existing rows and shared transformation vectors, but is not discussed here. The speech-generation setting and withholding experiments are valuable, although it remains unclear what the proposed regression-and-donor construction adds over simpler phonological vector-offset methods. A direct comparison would help distinguish the contribution of the construction from that of the application and evaluation.

  2. What information in the donor residual is responsible for the improvement? A comparison with residuals from unrelated, frequency-matched donors would help establish whether phonological donor selection is essential.

  3. Although audio samples are included, the supplementary material is difficult to navigate. The listening pages are organized as blinded evaluations, making it cumbersome to connect individual examples to the methods and results in the paper. A simple, clearly labelled HTML demo would help: each example should show the text, intended pinyin, and side-by-side audio from the baseline, regression-only method and proposed composition, including both successful repairs and failure cases.

Questions for authors
  1. How does the proposed construction compare with direct phonetic-direction edits or offsets estimated from matched common-character pairs? Please clarify its relationship to McLaughlin et al. and Vocab Diet.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.