acceptodds
Under review as a conference paper at ICLR 2027

Sparse Continuous Representation for Unpaired Speech–Text Learning

Abstract

Learning from unpaired speech and text corpora can substantially improve sample efficiency and accessibility of spoken language systems. However, existing approaches for unpaired speech–text learning face a **capacity–compatibility tradeoff**: Dense, continuous representations preserve rich acoustic information but differ significantly with text in representational structures, resulting in low compatibility; Discrete speech units improve compatibility at the cost of information loss. In this work, we propose **sparse continuous representations (SCRs)** as an alternative that preserves rich information while maintaining compatibility. To validate our approach, we consider the task of unsupervised speech recognition (UASR) and synthesis (UTTS) as testbeds, both canonical examples of unpaired speech-text learning. Across seven languages for UASR, our SCR-based systems, SparPUSM and SparCipher, consistently outperforms the previous state-of-the-art systems, yielding an average character error rate reduction of 8-17%, with up to 17% on LibriSpeech. For UTTS, SCR-based approach also outperforms previous approaches, demonstrating the benefit beyond UASR. Code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Unsupervised speech recognition, sparse continuous representations, self-supervised speech learning, speech–text alignment, cross-modal representation learning, dictionary learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_SM1SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes sparse continuous representations (SCRs) as an interface for learning from unpaired speech and text, motivated by a trade-off between the information capacity of dense features and the compatibility of discrete units with text. Syllable-level speech features are represented using continuous coefficients over a small subset of a K-means dictionary, obtained through orthogonal matching pursuit. The authors incorporate SCRs into distribution matching and interleaved masked language modeling, yielding SparPUSM and SparCipher. They also develop an SCR-conditioned synthesis system trained using pseudo-transcripts. Evaluation covers unsupervised ASR in seven languages, English unsupervised TTS, and ablations on limited supervision, text-domain mismatch, tokenization, and sparsity.

Critical strengths
  1. A useful representation-level contribution with a relatively simple implementation. SCR offers a plausible intermediate representation between hard quantization and unrestricted dense features. The use of a fixed dictionary and sparse continuous coefficients makes the central intervention concrete. Applying it to both PUSM and a contextual decipherment model helps demonstrate that its potential usefulness is not restricted to a single downstream architecture.
  2. Meaningful improvements within the evaluated UASR setting. On LibriSpeech, SparPUSM improves over PUSM, and SparCipher improves over SylCipher both before and after student self-training.
Critical weaknesses
  1. The practical annotation-efficiency benefit remains insufficiently established. The paper motivates unpaired learning through annotation scarcity, but improvements over existing unsupervised systems do not establish whether the proposed pipeline is preferable under a small annotation budget. Figure 4(a) provides encouraging English semi-supervised results, yet it is unclear whether the pretrained encoders and resource budgets are matched. The low-resource evaluation covers only Cantonese and Taigi, without corresponding label-efficiency comparisons, while UTTS is evaluated only in English and lacks a limited-label adaptation comparison. For both tasks, a controlled comparison against the same pretrained backbone using limited paired data and conventional pseudo-label augmentation would better substantiate the practical motivation. The paired validation data used for early stopping and model selection should also be included in the annotation budget. High-resource-language benchmarks are valid methodological testbeds, but the current evidence does not sufficiently establish the claimed benefit for resource-constrained deployment.
  2. The UTTS results do not yet convincingly show the advantage of SCR over RVQ. Figures 6–8 compare outputs from different samples. For example, Figure 7 uses the reference for RVQ sample 1 but shows the output for RVQ sample 2, while the SCR output is correctly paired. This makes SCR appear to preserve the content better, but the comparison is not valid. Table 3 reports WER reductions of 1.5 and 2.7 percentage points over RVQ using the same SparCipher transcripts. These gains may still hold, but the supplementary SCR waveforms all have the same duration, and the inspected short example contains nonzero signal beyond the end of the reference speech. It is unclear whether this is only an export issue or also affects scoring. The authors should correct the sample pairings, explain the waveform durations, and verify that all systems are scored with consistent trimming. Until then, the UTTS evidence provides limited support for the claimed “capacity–compatibility” advantage. These concerns do not directly invalidate the UASR results.
  3. Inconsistent metric reporting weakens the multilingual performance claims. Table 2 states that MLS results are evaluated according to each model’s token type, yet places phone (PER), character (CER), and syllable-level(SER) error rates in the same columns and identifies the best result. These metrics are not directly comparable. Moreover, the SylCipher scores of 32.3 and 35.6 reappear in Figure 4(c), whose axis labels them as CER, creating ambiguity about the actual scoring unit. The authors should clarify the evaluation protocol, correct inconsistent labels, and use a common metric for cross-system comparisons. This does not negate improvements over comparable syllable-based baselines, but weakens the broader claim of multilingual superiority.
Questions for authors

Questions as above.

8 comments

We thank reviewer SM1S for the thoughtful and constructive feedback, and for recognizing the motivation and contributions of our approach. We address the main concerns below.

1. Low-Resource Generalization and Label Efficiency

We thank the reviewer for highlighting the need for stronger evidence of label efficiency and generalization to low-resource languages. To address these concerns, more comprehensive cross-lingual and label-efficiency evaluations. To address these concerns, we conducted two additional experiments.

Cantonese UASR with limited paired data. We extended our Cantonese evaluation to investigate the effect of limited paired supervision. As shown below, SparCipher consistently outperforms SylCipher, achieving 9.4% CER compared with 13.3% for SylCipher using 1.5 hours of paired data. These results further demonstrate the effectiveness of SCR in a low-resource linguistic setting.

MethodPaired Data (hr)CER (↓\downarrow)
SylCipher035.0
SylCipher0.513.4
SylCipher1.513.3
SparCipher031.9
SparCipher (self-train)020.6
SparCipher (self-train)0.59.9
SparCipher (self-train)1.59.4

UTTS adaptation with limited paired data. We further investigated the effect of introducing small amounts of paired speech-text data into UTTS training. With just one hour of paired data, WER decreases substantially from approximately 33.1% to 25.8%, while speaker similarity improves from 0.40 to 0.42. Increasing the amount of paired data to 16 hours yields a smaller additional improvement, reaching 25.2% WER and 0.43 speaker similarity.

Transcript ModelModelPaired Data (hr)WER (↓\downarrow)SSIM (↑\uparrow)
SparCipherSCR+CFM033.10.40
SparCipherSCR+CFM125.80.42
SparCipherSCR+CFM1625.20.43

These results demonstrate that our approach can benefit substantially from limited paired supervision, with diminishing additional gains as more paired data become available in the evaluated setting. We acknowledge that comparisons against additional supervised or pseudo-label augmentation baselines under matched resource budgets would further strengthen the practical conclusions.

We will incorporate these additional experiments into the revised manuscript.

2. Accounting for Paired Validation Data

We agree that paired validation annotations should be included in the overall annotation budget. We clarify that our proposed training objective operates entirely on unpaired speech and text, without requiring paired data for model optimization in the unsupervised setting. However, as described in the paper, a small paired validation set is used for early stopping and checkpoint selection.

We acknowledge that these validation annotations constitute additional supervision and should be explicitly accounted for. We will clarify the annotation budget and distinguish the requirements of the training objective from those of our current model-selection protocol in the revision.

3. UTTS Sample Pairing and Waveform Trimming

We thank the reviewer for carefully examining our supplementary audio examples and identifying these issues.

Sample pairing. The mismatches in Figures 6–8 resulted from an oversight when assembling the qualitative examples. Specifically, different training scripts saved validation samples in different orders, leading to incorrect reference–generation pairings. Importantly, these mismatches were confined to the qualitative presentation and did not affect the reported quantitative results. We will the correct the pairing in the revision to provide comparisons with the same utterance.

Waveform trimming. The identical durations and trailing audio in the supplementary examples resulted from batch padding during training saved without trimming. In contrast, evaluation was performed with a batch size of one, preserving the valid waveform lengths without the batch padding. Therefore, these export artifacts did not affect the reported WER or speaker-similarity scores.

We will correct the supplementary examples and clarify the evaluation procedures.

4. Consistency of Multilingual Evaluation Metrics

All MLS results except the first row of Table 2 use CER, regardless of the models' token types. The caption mistakenly suggests otherwise, which we will correct. The first row reports PER for reference only and is not a direct comparison. We will also clarify Figure 4(c)'s labeling. Thus, the main MLS comparisons use a consistent character-level metric.


We thank the reviewer again for the constructive feedback. We hope these additional experiments and clarifications address the main concerns and strengthen the empirical support for our approach. We welcome further comments and suggestions.

Dear reviewer SM1S, please indicate on whether the rebuttal addressed your concerns.

Thank you for the additional experiments and clarifications. The new results provide stronger evidence of label efficiency. However, they do not fully address my main concern regarding generalization to linguistically diverse low-resource languages. The additional evaluation is limited to Cantonese, and it remains unclear whether the conclusions drawn from relatively high-resource languages generalize to languages with substantially different linguistic characteristics, such as Arabic or low-resource Southeast Asian languages. I believe broader cross-lingual evaluations are needed to substantiate the claimed generalization capabilities. So I decide to maintain my initial score.

Reviewer_cwu4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses unpaired speech-text learning for unsupervised ASR and unsupervised TTS, arguing that existing representations face a capacity-compatibility trade-off between dense continuous features and discrete units. It proposes sparse continuous representations, in which a dense self-supervised feature is sparsely decoded over a K-means dictionary via orthogonal matching pursuit at the syllable level, and builds two unsupervised ASR systems and a flow-matching unsupervised TTS pipeline on top. Across seven languages the authors report an 8 to 17 percent relative error-rate reduction over the SylCipher and PUSM baselines in ASR and lower error than dense- and RVQ-based TTS baselines.

Critical strengths
  1. The core idea of using sparse-but-continuous codes as a speech-text interface is well-motivated and framed cleanly against the capacity-compatibility trade-off, with K-means VQ as a special case.
  2. The empirical coverage is broad and consistent across seven languages including low-resource Cantonese and Taigi, plus a semi-supervised regime and domain-shift ablations, improving over strong recent baselines in nearly every cell.
Critical weaknesses
  1. There is no ablation isolating sparse continuous codes against a matched discrete baseline on the same encoder and segmentation, so the gains cannot be attributed to sparse continuity rather than the syllable-level front end.
  2. Absolute error rates remain very high and no variance, confidence intervals, or significance tests are reported, which is below top-venue standards given margins of only 1 to 2 points in several languages.
Questions for authors
  1. Can you provide a controlled ablation whose only varied factor is sparse continuous versus a discrete code on identical syllable features, to isolate the contribution of sparse continuity?
  2. What are the variance and significance of the numbers across seeds and dictionary initializations, given that several gains are only 1 to 2 points?
  3. How sensitive are results to dictionary size and encoder choice, and is there any direct measurable evidence for the claimed text compatibility
2 comments

We thank reviewer cwu4 for the thoughtful feedback, and for recognizing the motivation and contributions of our approach. We will address the main concerns below.

1. Controlled Ablation: SCR vs. Discrete Representations

We thank the reviewer for highlighting the importance of isolating SCR's contribution. Our MLS experiments already compare sparse continuous and discrete representations using identical speech encoders and syllable segmentation, allowing us to evaluate the effect of the representation itself.

For LibriSpeech, although the original comparison with SylCipher does not use identical segmentation, our approach omits SylCipher's differentiable boundary detector, resulting in a simpler system that nevertheless achieves better performance.

To further isolate the effect of SCR, we conducted an additional controlled experiment on LibriSpeech, using the same encoder, syllable segmentation, K-means dictionary, and PUSM architecture, varying only the representation:

ModelEncoderCER (↓\downarrow)
PUSM (K-means)XEUS33.4
SparPUSM (SCR)XEUS24.4

The 9-point absolute CER improvement provides direct evidence that SCR contributes to the gains independently of the encoder, segmentation, and downstream architecture.

2. Statistical Robustness Across Random Seeds

We agree that reporting variance is important, particularly when performance differences are relatively small. We therefore conducted experiments using three random seeds and report the mean and standard deviation of CER:

MethodLanguageCER (↓\downarrow)
PUSMDutch38.8±0.438.8\pm 0.4
PUSMFrench48.3±6.948.3\pm 6.9
SparPUSMDutch34.2±1.834.2\pm 1.8
SparPUSMFrench43.2±1.543.2\pm 1.5

SparPUSM achieves lower mean CER in both languages, with absolute improvements of 4.6 and 5.1 percentage points, respectively. These results provide additional evidence that the improvements are not limited to a single random initialization.

3. Sensitivity to Dictionary Size and Encoder Choice

We further evaluated the sensitivity of SparPUSM to dictionary size and encoder choice, keeping other experimental settings unchanged.

MethodLanguageEncoderCodebook SizeCER (↓\downarrow)
SparPUSMDutchXEUS102432.8
SparPUSMDutchXEUS204839.2
SparPUSMGermanXEUS102431.5
SparPUSMGermanHuBERT102446.2

Dictionary size: Increasing the codebook size from 1024 to 2048 on Dutch increases CER from 32.8% to 39.2%. One possible explanation is that a larger dictionary yields fewer training examples per atom, making representation learning more challenging. Nevertheless, both configurations produce functioning UASR systems, indicating that SCR does not require a single specific dictionary size.

Encoder choice: Replacing XEUS with HuBERT-Large on German increases CER from 31.5% to 46.2%. This degradation may partly reflect language mismatch, as the HuBERT-Large encoder was pretrained on English. Nevertheless, SparPUSM remains functional with both encoders.

These experiments suggest that while performance depends on dictionary size and encoder quality, the approach is applicable across different configurations, rather than being restricted to one particular encoder or dictionary size.

4. Evidence for Text Compatibility

We appreciate the reviewer's question regarding direct evidence of text compatibility. We clarify that improved compatibility with text representations is a motivating hypothesis, rather than a property directly established by our experiments.

Our UASR and UTTS results provide indirect empirical support by demonstrating that SCRs effectively support learning from unpaired speech and text. However, we agree that direct measurements, such as evaluating the predictability of textual representations from speech representations, would provide stronger evidence. We will clarify this distinction and appropriately qualify our claims in the revised manuscript.


We thank the reviewer again for the constructive feedback. We hope these additional experiments and clarifications address the main concerns and strengthen the empirical support for our approach. We welcome further comments and suggestions.

Dear reviewer cwu4, please comment here on whether the rebuttal addressed any of your concerns.

Reviewer_kAK8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose sparse continuous representations (SCR) for learning from unpaired speech and text data. The proposed method is designed to give better tradeoff of expressiveness and compatibility compared to discrete or continuous representations. The authors validated the proposed representation by training and evaluation on unsupervised ASR and TTS tasks, outperforming previous methods in the g2p-free configuration.

Critical strengths

The paper is clearly written and provides sufficient details to reproduce the experiments. The experimental results are also promising compared with existing work. The method makes efficient use of unpaired speech and text, and it could benefit training of speech models for low-resource languages where publicly available labeled data are scarce, which is significant task.

Critical weaknesses

I do not see any critical weakness of this paper. I have some minor questions below.

Questions for authors

Since unpaired data is usually much more widely available than paired data, scalability with respect to the amount of unpaired training data is important for unsupervised learning. While the results show consistent improvement compared to prior work in both unsupervised setting and semi-supervised setting, the downstream performance like CER for ASR is not enough to be used in real-world setting. It would make this research more solid to present if the UASR performance improves as the amount of unpaired data increases. (e.g. increasing audio data with Libri-Light)

1 comment

Dear reviewer kAK8,

Thank you for the positive and encouraging assessment of our work, particularly for recognizing its methodological contributions and potential impact on low-resource speech learning.

We also appreciate the suggestion to investigate scalability with increasing amount of unpaired data, such as LibriLight. We agree that this is an important direction for improving the practical applicability of UASR and will consider it in future work.

Thank you again for your constructive feedback and support.

Reviewer_1wzWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes sparse continuous representations (SCRs) for speech, which aim to bridge the gap between the rich acoustic information brought by continuous representations and the higher compatibility of discrete representations to textual representations. The experiments are on two tasks, ASR and TTS, on LibriSpeech (English), AISHELL (Chinese), a small subset of MLS covering German, Dutch, and French (in addition to two low-resourced datasets, Cantonese MDCC and Taigi SuiSiann), and the proposed method proved to outperform state-of-the-art approaches.

Critical strengths
  • The paper is well motivated and the methodology section is very detailed, with small but clear supporting figures.
  • The approach is interesting and addresses a real gap between continuous and discrete speech units.
  • The proposed methods, SparPUSM and SparCipher, seem to improve over the reported baselines on both ASR and TTS tasks.
Critical weaknesses
  • The training settings are not clear, for the ASR task, only evaluation datasets are listed (maybe by wrongly saying "we evaluate" in line 312?), and since in lines 111-112, it is reported "In the task of unpaired speech–text learning, the training data consist of two independently collected corpora, a speech corpus X and a text corpus Y, without instance-level pairing", I would expect to find both an unpaired text and speech corpora. Instead, I find paired speech corpora only and a vague "To simulate domain shift, we replace 50% or 75% of the in-domain text with news or Wikipedia text." in lines 317-318 that comes out of the blue.
  • Some very relevant citations, but most importantly, state-of-the-art approaches missing, for instance:
    • X. Li, et al. "Bridging the Gap between Continuous and Informative Discrete Representations by Random Product Quantization." IEEE Transactions on Audio, Speech and Language Processing (2026).
    • X. Chang, et al., “Exploration of efficient end-to-end asr using discretized input from self-supervised learning,” in INTERSPEECH 2023.
    • X. Chang, et al., “Exploring speech recognition, translation, and understanding with discrete speech units: A comparative study,” in ICASSP 2024.
    • M. Cui, et al., "Exploring SSL Discrete Speech Features for Zipformer-based Contextual ASR". In Proc. Interspeech 2025.
  • Unclear evaluation and reporting. For instance, "Table 1: UASR results on the LibriSpeech clean subsets. [...] * indicates evaluation on dev-clean.", which seems that some models were evaluated on different dev/test sets, and it is not clear why. In Table 2, instead, we don't know which metric is reported for MLS. Also, it is not clear why speaker similarity is reported for LibriSpeech only, and other crucial speech output quality metrics, like MOS-based metrics, are missing. Most importantly, it is not clear why SylCipher results are not reported on LJSpeech in Table 3, and, similarly, wav2vec-U results are not reported for LibriSpeech.
  • Novelties and differences compared to the baselines are never discussed (see questions below), which instead is crucial to properly position the work.

Side suggestions: 1) please try to move the tables closer to the place in which they are discussed, sometimes they are 2 pages earlier than the related discussion and I found the results table even before knowing the metrics and baselines, 2) if you print the paper, the figures in Figure 4 are unreadable.

Questions for authors
  • Could the authors position their work w.r.t. the missing approaches I mentioned in the weaknesses? Especially (Chang et al, 2024) seems to be very relevant to the proposed approach.
  • Could the authors clarify how the model was trained? On which speech and text data?
  • Could the authors clarify on which sets (test, dev) was each model evaluated in the various Tables?
  • Could the authors describe the criteria used to select the baselines? How do they differ from the training and evaluation setup of the proposed system? For instance, the language coverage, it seems that JSTTI does not support Cantonese, while wav2vec does not support neither Cantonese nor Taigi. Moreover, their relevance and the differences in terms of approach are never discussed and methods' names only are just briefly mentioned in the experimental settings. I am wondering which are the novelties compared to, for instance, SylCypher, which seems to be very similar in spirit to this work.
  • Why are results not filled in for some models? Especially in Table 3.
  • How does the method generalize to different audio conditions (e.g., spontaneous speech, noise)?
3 comments

Thank you Reviewer 1wzW for the thoughtful feedback and recognition of our approach's motivation and potential. We address your concerns below.

1. Training data and unpaired setup

We agree our description of the datasets was unclear. To clarify, our main focus is the unpaired setting. Although some datasets contain paired speech and transcripts, we discard all speech-text pairings during training, treating speech and text as independent corpora. In the unmatched LibriSpeech setting, we use an independent text corpus with no corresponding speech recordings. We further introduced domain mismatch using Wikipedia and news text, following SylCipher. We will clarify that “evaluate” in line 312 refers to both training and evaluation using the respective dataset splits

2. Relevant baselines and citations

We thank the reviewer for suggesting these studies. They investigate alternative speech representations primarily in supervised ASR, whereas our work addresses unsupervised ASR with unpaired speech and text. Supervised results may not transfer to UASR, where cross-modal alignment poses additional challenges. We additionally evaluated product quantizer (PQ) and random product quantizer (RPQ) for UASR on LibriSpeech using a PUSM-based UASR system. Despite their much larger effective codebook size, these quantizers perform substantially worse than SCR:

MethodCodebook SizeCER (↓\downarrow)
PQ + PUSM2564256^472.8
RPQ (disjoint) + PUSM2564256^478.0
RPQ (overlap) + PUSM2564256^474.4
SparPUSM204824.4

These results suggest that more expressive discrete quantization does not necessarily improve unpaired speech-text learning.

3. Baseline selection and methodological novelty

Our baselines were selected to represent state of the art approaches to UASR under comparable training conditions. We include Cantonese and Taigi to evaluate low-resource languages with syllable-oriented writing systems, where UASR approaches built around phoneme or word units such as wav2vec-U and JSTTI may require additional linguistic resources such as pronunciation dictionary.

Unlike wav2vec-U, which relies on phoneme-level targets and adversarial distribution matching, our approach follows the masked language modeling framework of JSTTI and SylCipher, avoiding adversarial training and phoneme-level targets. The main novelty of SparCipher is replacing SylCipher's discrete syllable units with SCRs, retaining the underlying decipherment framework while increasing representation capacity. This allows us to isolate the benefits of sparse continuity. We will clarify these differences and our baseline selection criteria.

4. Evaluation protocol and missing results

We thank the reviewer for pointing out the ambiguity in our evaluation protocol and reporting.

Table 1: Following prior work, we include published results on different LibriSpeech subsets for completeness. We have also reproduced baselines such as wav2vec-U on the same evaluation subset as our method for direct comparisons. We will explicitly identify all evaluation splits.

Table 2: All MLS results except the first row use CER, regardless of the token type used by the model. The caption contains an error that we will correct. The first row reports PER and is included only for reference, not direct comparison.

Table 3: We have now conducted additional LJSpeech experiments and filled in the missing results, including MOS and other relevant metrics, as shown below.

DatasetTranscript ModelModelCER (↓\downarrow)WER (↓\downarrow)SSIM (↑\uparrow)UTMOS (↑\uparrow)
LibriSpeechGTDense3.19.10.492.4
SylCipherRVQ28.141.10.442.3
SparCipherRVQ23.234.60.392.5
SparCipherSCR22.133.10.402.5
LJSpeechGTDense4.46.90.392.4
SylCipherRVQ32.343.60.342.3
SparCipherRVQ27.136.80.342.5
SparCipherSCR25.934.10.342.5

Regarding the missing wav2vec-U TTS result on LibriSpeech, its original implementation targets single-speaker synthesis, whereas LibriSpeech is multi-speaker. We will clarify this in the revision.

5. Generalization to spontaneous/noisy speech

Our evaluations span English, Cantonese, and Taigi datasets with different recording conditions, showing consistent improvements over baselines. However, we acknowledge that we have not conducted controlled experiments isolating noise or spontaneity. We will clarify this limitation and identify systematic acoustic robustness evaluation as future work.


We also appreciate the presentation suggestions and will move tables closer to their discussions and improve Figure 4's print readability. We welcome further comments and suggestions.

Thanks to the authors for their reply. Regarding the missing relevant baselines, although the works I shared rely on supervised ASR, most of the proposed methods are indeed applicable to unsupervised ASR. Comparing the proposed method only to a single other work, SylCipher, which shares with this work most of the underlying approach except for "replacing SylCipher's discrete syllable units with SCRs", as mentioned by the authors themselves, does not provide, in my opinion, enough information to establish its novelty (the improvement compared to RVQ is pretty limited, 1 to 2%) and to validate SparCipher compared to the existing approaches, especially if the only few baselines present in some tables are trained on very different settings and, most importantly, do not even cover the analyzed languages, such as JSTTI and wav2vec-U. In addition, the training settings are still severely underspecified (not clear which datasets have been used in training and evaluation, how many hours of speech and tokens for text, and so on) and affects the reproducibility of the work. I believe that the work is promising, but, in my opinion, unclear settings and choices, and missing sound comparisons prevent from understanding and position the values of its contributions.

We thank the reviewer for the follow-up and for recognizing the potential of our work. We would like to further clarify our experimental settings and the rationale behind our baseline comparisons.

1. Training and Evaluation Settings

Our experimental datasets and preprocessing largely follow established UASR protocols, particularly those of wav2vec-U, JSTTI, and SylCipher. Accordingly, the original submission referenced these protocols rather than repeating their full descriptions. To make the setup more transparent and reproducible, we provide the training resources and evaluation splits below.

DatasetTraining Speech (hr)Text SourceText SizeEvaluation (hr)
LibriSpeech414LibriLM0.8B words46 (10% held-out subset)
AISHELL-363AISHELL-3 train0.7M syllables22 (standard)
MLS German100MLS German train1.1M syllables14 (standard)
MLS Dutch100MLS Dutch train1.2M syllables12 (standard)
MLS French100MLS French train1.4M syllables10 (standard)
MDCC58MDCC train0.8M syllables11 (standard)
SuiSiann3.4SuiSiann train44K syllables0.5 (standard)

For LibriSpeech, the evaluation set is a random 10% holdout from train-clean-100 and train-clean-360. For datasets where speech and text originate from the same corpus, their utterance-level correspondences are discarded during UASR training.

Our implementations also share substantial infrastructure with established UASR systems and follow their preprocessing and hyperparameter conventions wherever applicable (See Appendix A of our manuscript for more details). However, different learning objectives, particularly adversarial versus non-adversarial training, have different optimization characteristics. In particular, adversarial UASR training can be sensitive to model architecture and optimization settings, potentially leading to instability. We therefore follow each method's established configurations rather than imposing identical architectures or hyperparameters. We will clarify these settings further in the revision.

2. Baseline Comparability and Methodological Novelty

The architectural similarity between SylCipher and SparCipher is intentional, enabling a controlled evaluation of the representation-level contribution. Moreover, SCR is evaluated within two UASR frameworks, PUSM and SylCipher. Our additional controlled LibriSpeech experiment, using identical encoders, segmentation, dictionary sizes, and downstream architectures, shows that replacing K-means with SCR reduces CER from 33.4% to 24.4%.

We also evaluated stronger discrete representations, including PQ and RPQ, within PUSM, as reported in our previous response. These results provide additional evidence that the benefits of SCR extend beyond a comparison with SylCipher alone.

Regarding the relatively modest UTTS improvements over RVQ, our contribution extends beyond the WER reduction. We introduce an SCR-based speech sparsifier as an alternative to conventional discrete speech codecs and demonstrate its effectiveness in G2P-free UTTS with noisy pseudo-transcripts. These results support the applicability of SCR to both recognition and generation.

3. Language Coverage and Applicability

Regarding the reviewer’s concern that some baselines do not cover all evaluated languages, we would also like to emphasize that broader language applicability is itself an important contribution. The absence of results for wav2vec-U and JSTTI on Cantonese and Taigi does not simply reflect omitted comparisons. We attempted to adapt existing UASR approaches, including wav2vec-U, to these languages, but the tested configurations did not yield satisfactory recognition performance (See Table 2). In contrast, our approach successfully learns functional G2P-free UASR systems on both languages, demonstrating its practical applicability to additional low-resource settings. We will clarify these attempted adaptations and distinguish direct performance comparisons from evidence of broader language applicability.


We appreciate the reviewer's continued engagement and will incorporate these clarifications into the revised manuscript.

Reviewer_RQRgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Premise that transcribed "paired" data is expensive, we should move towards unsupervised training, which is very true. Much unsupervised ASR work work has been predicated on having a grapheme to phoneme (G2P) model which unavailable for most of the world's languages. However, for training speech-to-text or text-to-speech, we need a "structure" between audio (continuous) and text (discrete) modalities. UASR existing could be continuous, e.g. wav2vec or Hubert, or discrete, which hasn't found good success due to limited capacity of the discrete features to capture audio nuances. The paper proposes using sparse continuous representations (SCRs) of speech. The motivation being the few continuous representations can better model fine grain audio information, while sparsity better aligns with the text modality. The author's proposed unsupervised ASR and TTS approaches with SCR, without the use of G2P mappings, with results exceeding discrete or dense models.

Learning takes place by having a corpora of speech and a corpora of text (un-paired instances). The model then needs to learn how to map instances of speech corpora X with instances of text corpora Y. UASR has two parts: 1. find a representation that is compatible with text 2. learning to map the two modalities. The author's report strong results compared to the literature.

However, there is a lot of confusion when reporting the actual results in Table 1 and the results cited from other work. Looking at 3 of the unsupervised ASR citations, there is a mix up PER/WER (in papers) vs CER (what the results purport to be) such that it impossible to continue reading through the results due to the level of errors making difficult to trust the other work in the paper.

Critical strengths

The author's report an innovative approach of bridging the gap between discrete tokens and dense embeddings by introducing sparse dictionary coding to unpaired speech–text corpora, a way to model speech and text with these sparse continuous embeddings, and a way to map speech-to-text and text-to-speech using this representation.

Critical weaknesses

The choice of syllabic units with a fixed 2000 atom dictionary estimated through K-means appears sub-optimal. Syllables can very widely in length, have significant changes over time due to being comprised of multiple vowels and realize differently based on context. https://arxiv.org/abs/2201.05234 cites about 12,000 English syllables. The use of K-means K=2048 and using the centroids for the atoms

The numbers quoted from literature are completely misquoted:

  1. https://arxiv.org/pdf/2402.03988 (Reborn) reports in Table 1: PER/WER on LibriSpeech using 100 hours speech data. PER/WER on Librispeech dev clean is 8.3/12.5. A WER of 12.5 is about 3-4% CER.

  2. https://arxiv.org/pdf/2406.08380 (Towards Unsupervised Speech Recognition with Pronunciations, JSTTI) Curated K-word LibriSpeech clean subset, all numbers are WER. Table 5, K=4096, WER=31.60 → 8-10% CER.

  3. 2021 Baevski https://proceedings.neurips.cc/paper_files/paper/2021/file/ea159dc9788ffac311592613b7f71fbb-Paper.pdf Unsupervised Speech Recognition, the Wav2Vec-U dev-clean WER = 13.3%, which may be about 3.5-4.5% CER. In this paper, there don't appear to be experiments without G2P, however they show up in the paper in Table 1 rows 4 and 5--- these numbers are nowhere to be found in the paper at all.

Moreover, how do we interpret these results when Table 1 suggests the measurement is character error rate (CER)? CER is typically much lower than WER, maybe 3-4 times lower so the CER numbers here are way higher than what has previously been reported, e.g. estimated 3-4% CER for Reborn.

Questions for authors

How do you explain the WER / CER discrepancy and numbers being cited in Table 1?

How do we reconcile the choice of K=2048 for the number of atomic syllables, when the dictionary are raw multi syllable centroid averages rather than syllable "factors" that can sparsely compose 10,000+ syllables?

2 comments

We thank reviewer RQRg for the thoughtful feedback and for recognizing the contributions of our work. We appreciate the opportunity to clarify the evaluation protocol, comparisons with prior work, and motivation for sparse coding.

1. Clarification of Baseline Results and WER/CER Reporting

wav2vec-U and REBORN: Their published results primarily use phoneme-based representations requiring grapheme-to-phoneme (G2P) conversion, whereas our work focuses on the G2P-free setting. For a fairer comparison, we reran their publicly available implementations using characters instead of phonemes as text units, following prior G2P-free UASR practices (e.g., JSTTI). Thus, the CER values in Table 1 correspond to our character-based reproductions, not the published PER/WER values.

JSTTI: The result cited by the reviewer is obtained using a curated subset in which words outside the top-4096 vocabulary are removed through forced alignment. For our comparison, we reran the released implementation with a top-2048 word vocabulary, replacing out-of-vocabulary words with <OOV> tokens rather than removing speech segments. This avoids the forced-alignment-based utterance filtering used in their original setup. We will clarify these differences and distinguish reproduced results from published numbers.

Evaluation metric: We reported mainly CER in the paper since it allows us to perform a rigorous comparison with prior works that use a variety of linguistic units for text using a consistent metric. We did not report WERs in the paper since it will require additional postprocessing steps such as the use of a weighted finite-state transducer (WFST) and language model reweighting, which adds extra complexity to the system and makes it hard to understand whether performance gain comes from the UASR model or the later postprocessing. Nevertheless, to address the reviewer's concern, we additionally evaluated WER using a WFST decoder with a word-level 3-gram language model, applying the same decoding protocol to SparCipher and wav2vec-U:

MethodTokenWER (matched, ↓\downarrow)WER (unmatched, ↓\downarrow)
wav2vec-UCharacter60.478.5
SparCipherSyllable31.742.2

These additional results demonstrate that SparCipher also substantially outperforms the character-based wav2vec-U baseline in WER, under both matched and unmatched conditions.

We will revise Table 1 to clearly distinguish reproduced character-based CER results from previously published PER/WER results, specify the corresponding training and evaluation settings, and include WER as an additional evaluation metric.

2. Explaining the WER–CER discrepancy

The difference between CER and WER reflects both the sensitivity of word-level evaluation to character errors and the additional challenges of G2P-free recognition.

Unlike phoneme-based systems, which benefit from pronunciation dictionaries that explicitly connect acoustic realizations to words, our approach learns directly from written characters without such mappings. The mismatch between character sequences and acoustic realizations, together with imperfect syllable segmentation, can produce spelling errors that disproportionately affect WER.

Furthermore, conventional WFST decoding can exploit pronunciation lexicons and word-level language models to correct phoneme-based predictions. Such corrections are less straightforward for character-based syllable predictions, particularly when the syllable inventory is restricted. This may contribute to the remaining gap between G2P-based and G2P-free systems.

We view improved word-level decoding and spelling correction as promising directions for future work.

3. Motivation for sparse coding and dictionary size K=2048K=2048

We thank the reviewer for raising this question. We clarify that our dictionary atoms are syllabic centroids, not necessarily disentangled phonetic or sub-syllabic factors. The dictionary consists of syllabic centroids, and our motivation for sparse coding is its greater representational expressivity compared to conventional vector quantization. Specifically, vector quantization represents each syllable embedding using a single centroid, whereas sparse coding permits a linear combination of multiple centroids. The set of representations achievable by vector quantization is therefore a subset of those achievable by sparse coding, irrespective of whether the dictionary atoms correspond to linguistically meaningful factors. Consequently, K=2048K=2048 is a dictionary-capacity hyperparameter, rather than an assumption about the number of distinct syllables or underlying phonetic factors. Our method also does not require a one-to-one correspondence between dictionary atoms and syllable types.


Thank you again for the constructive feedback. We welcome any further comments or suggestions.

Dear reviewer RQRg, please comment on whether the rebuttal addressed any of your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.