- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes sparse continuous representations (SCRs) as an interface for learning from unpaired speech and text, motivated by a trade-off between the information capacity of dense features and the compatibility of discrete units with text. Syllable-level speech features are represented using continuous coefficients over a small subset of a K-means dictionary, obtained through orthogonal matching pursuit. The authors incorporate SCRs into distribution matching and interleaved masked language modeling, yielding SparPUSM and SparCipher. They also develop an SCR-conditioned synthesis system trained using pseudo-transcripts. Evaluation covers unsupervised ASR in seven languages, English unsupervised TTS, and ablations on limited supervision, text-domain mismatch, tokenization, and sparsity.
Critical strengths
- A useful representation-level contribution with a relatively simple implementation. SCR offers a plausible intermediate representation between hard quantization and unrestricted dense features. The use of a fixed dictionary and sparse continuous coefficients makes the central intervention concrete. Applying it to both PUSM and a contextual decipherment model helps demonstrate that its potential usefulness is not restricted to a single downstream architecture.
- Meaningful improvements within the evaluated UASR setting. On LibriSpeech, SparPUSM improves over PUSM, and SparCipher improves over SylCipher both before and after student self-training.
Critical weaknesses
- The practical annotation-efficiency benefit remains insufficiently established. The paper motivates unpaired learning through annotation scarcity, but improvements over existing unsupervised systems do not establish whether the proposed pipeline is preferable under a small annotation budget. Figure 4(a) provides encouraging English semi-supervised results, yet it is unclear whether the pretrained encoders and resource budgets are matched. The low-resource evaluation covers only Cantonese and Taigi, without corresponding label-efficiency comparisons, while UTTS is evaluated only in English and lacks a limited-label adaptation comparison. For both tasks, a controlled comparison against the same pretrained backbone using limited paired data and conventional pseudo-label augmentation would better substantiate the practical motivation. The paired validation data used for early stopping and model selection should also be included in the annotation budget. High-resource-language benchmarks are valid methodological testbeds, but the current evidence does not sufficiently establish the claimed benefit for resource-constrained deployment.
- The UTTS results do not yet convincingly show the advantage of SCR over RVQ. Figures 6–8 compare outputs from different samples. For example, Figure 7 uses the reference for RVQ sample 1 but shows the output for RVQ sample 2, while the SCR output is correctly paired. This makes SCR appear to preserve the content better, but the comparison is not valid. Table 3 reports WER reductions of 1.5 and 2.7 percentage points over RVQ using the same SparCipher transcripts. These gains may still hold, but the supplementary SCR waveforms all have the same duration, and the inspected short example contains nonzero signal beyond the end of the reference speech. It is unclear whether this is only an export issue or also affects scoring. The authors should correct the sample pairings, explain the waveform durations, and verify that all systems are scored with consistent trimming. Until then, the UTTS evidence provides limited support for the claimed “capacity–compatibility” advantage. These concerns do not directly invalidate the UASR results.
- Inconsistent metric reporting weakens the multilingual performance claims. Table 2 states that MLS results are evaluated according to each model’s token type, yet places phone (PER), character (CER), and syllable-level(SER) error rates in the same columns and identifies the best result. These metrics are not directly comparable. Moreover, the SylCipher scores of 32.3 and 35.6 reappear in Figure 4(c), whose axis labels them as CER, creating ambiguity about the actual scoring unit. The authors should clarify the evaluation protocol, correct inconsistent labels, and use a common metric for cross-system comparisons. This does not negate improvements over comparable syllable-based baselines, but weakens the broader claim of multilingual superiority.
Questions for authors
Questions as above.
8 comments
Dear reviewer SM1S, please indicate on whether the rebuttal addressed your concerns.
Thank you for the additional experiments and clarifications. The new results provide stronger evidence of label efficiency. However, they do not fully address my main concern regarding generalization to linguistically diverse low-resource languages. The additional evaluation is limited to Cantonese, and it remains unclear whether the conclusions drawn from relatively high-resource languages generalize to languages with substantially different linguistic characteristics, such as Arabic or low-resource Southeast Asian languages. I believe broader cross-lingual evaluations are needed to substantiate the claimed generalization capabilities. So I decide to maintain my initial score.
We thank reviewer SM1S for the thoughtful and constructive feedback, and for recognizing the motivation and contributions of our approach. We address the main concerns below.
1. Low-Resource Generalization and Label Efficiency
We thank the reviewer for highlighting the need for stronger evidence of label efficiency and generalization to low-resource languages. To address these concerns, more comprehensive cross-lingual and label-efficiency evaluations. To address these concerns, we conducted two additional experiments.
Cantonese UASR with limited paired data. We extended our Cantonese evaluation to investigate the effect of limited paired supervision. As shown below, SparCipher consistently outperforms SylCipher, achieving 9.4% CER compared with 13.3% for SylCipher using 1.5 hours of paired data. These results further demonstrate the effectiveness of SCR in a low-resource linguistic setting.
UTTS adaptation with limited paired data. We further investigated the effect of introducing small amounts of paired speech-text data into UTTS training. With just one hour of paired data, WER decreases substantially from approximately 33.1% to 25.8%, while speaker similarity improves from 0.40 to 0.42. Increasing the amount of paired data to 16 hours yields a smaller additional improvement, reaching 25.2% WER and 0.43 speaker similarity.
These results demonstrate that our approach can benefit substantially from limited paired supervision, with diminishing additional gains as more paired data become available in the evaluated setting. We acknowledge that comparisons against additional supervised or pseudo-label augmentation baselines under matched resource budgets would further strengthen the practical conclusions.
We will incorporate these additional experiments into the revised manuscript.
2. Accounting for Paired Validation Data
We agree that paired validation annotations should be included in the overall annotation budget. We clarify that our proposed training objective operates entirely on unpaired speech and text, without requiring paired data for model optimization in the unsupervised setting. However, as described in the paper, a small paired validation set is used for early stopping and checkpoint selection.
We acknowledge that these validation annotations constitute additional supervision and should be explicitly accounted for. We will clarify the annotation budget and distinguish the requirements of the training objective from those of our current model-selection protocol in the revision.
3. UTTS Sample Pairing and Waveform Trimming
We thank the reviewer for carefully examining our supplementary audio examples and identifying these issues.
Sample pairing. The mismatches in Figures 6–8 resulted from an oversight when assembling the qualitative examples. Specifically, different training scripts saved validation samples in different orders, leading to incorrect reference–generation pairings. Importantly, these mismatches were confined to the qualitative presentation and did not affect the reported quantitative results. We will the correct the pairing in the revision to provide comparisons with the same utterance.
Waveform trimming. The identical durations and trailing audio in the supplementary examples resulted from batch padding during training saved without trimming. In contrast, evaluation was performed with a batch size of one, preserving the valid waveform lengths without the batch padding. Therefore, these export artifacts did not affect the reported WER or speaker-similarity scores.
We will correct the supplementary examples and clarify the evaluation procedures.
4. Consistency of Multilingual Evaluation Metrics
All MLS results except the first row of Table 2 use CER, regardless of the models' token types. The caption mistakenly suggests otherwise, which we will correct. The first row reports PER for reference only and is not a direct comparison. We will also clarify Figure 4(c)'s labeling. Thus, the main MLS comparisons use a consistent character-level metric.
We thank the reviewer again for the constructive feedback. We hope these additional experiments and clarifications address the main concerns and strengthen the empirical support for our approach. We welcome further comments and suggestions.