- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces DyCAST, a variable frame rate speech tokenizer that uses character boundary supervision and explicit duration prediction. It supports tokenization without text or an external aligner during inference, with adjustable temporal granularity and several duration-decoding modes. Experiments cover speech resynthesis, voice conversion, discriminative tasks, and text-to-speech, demonstrating competitive performance at substantially reduced frame rates.
Critical strengths
- S1: The paper addresses an important practical problem: reducing speech sequence lengths while preserving information needed for reconstruction and downstream modeling. Combining learned boundaries with explicit duration prediction provides useful flexibility, including inference without an external aligner and controllable decoding durations.
- S2: The empirical evaluation is broad, covering multilingual and noisy resynthesis, voice conversion, recognition, speaker identification, emotion recognition, and TTS. The results demonstrate a favorable balance between temporal compression and reconstruction quality. Appendix F further supports the evaluation through boundary analysis and comparisons of duration-decoding modes.
Critical weaknesses
- W1: The experiments do not sufficiently establish why character alignment is the appropriate segmentation target. Comparisons with phoneme or syllable boundaries, or a matched-rate fixed segmentation using the same backbone and quantizer, would help distinguish the benefits of linguistic alignment from temporal compression and other architectural changes. The strongest TTS result also uses a different, non-autoregressive architecture, so its gains cannot be attributed to tokenization alone.
- W2: The behavior of the boundary and duration predictors around silence and under noise remains unclear. Section 4.1 states that silence is merged into the following non-silence token, but the consequences for token allocation and pause reconstruction are not analyzed. Table 2 already evaluates noisy speech, and Appendix F.1 examines boundary agreement, but neither directly supports the proposed explanation that duration modeling improves noise robustness by avoiding token over-fragmentation.
Questions for authors
- Q1: Can the authors provide a controlled comparison between character supervision and an alternative such as phoneme supervision or matched-rate fixed segmentation? Keeping the backbone, quantizer, and downstream architecture consistent would clarify the contribution of character alignment.
- Q2: How do predicted boundaries and durations change with pause length and noise level? A focused analysis of token allocation within silence and reconstructed pause-duration errors would clarify whether merging silence into the following token preserves temporal structure reliably.
- Q3: How does the proposed supervision relate to Voxtral TTS, whose codec uses Whisper cross-attention to align codec representations with ASR decoder states for distillation? Voxtral retains fixed-rate frames, so the mechanisms differ, but discussing this relationship would sharpen the distinction between ASR-guided representation learning and explicit boundary-guided temporal compression.
2 comments
Thank you for running these additional experiments. The matched-rate comparison is helpful, which makes a clearer case for adaptive boundaries at low frame rates. The autoregressive TTS results also help show that the intelligibility gains aren't just due to the non-autoregressive architecture. The new pause and noise analyses address my questions. Additionally, the added discussion makes the connection to Voxtral TTS clearer.
I'm still curious about the choice of characters over phonemes or syllables. The new comparison shows why adaptive boundaries help, but it leaves open which alignment unit works best. Overall, I think the revisions strengthen the paper, and I'm keeping my original rating.
We thank the reviewer for their constructive feedback. We have expanded the manuscript with additional experiments and analyses addressing the raised concerns.
Adaptive versus fixed-rate segmentation
We have added a comparison between adaptive and uniformly spaced boundaries at matched frame rates (Appendix F.6, Table 7). The experiment uses the same pretrained DyCAST model, quantizer, and decoder, changing only the segmentation strategy at inference time. At approximately 6.2 Hz, adaptive segmentation achieves 4.03 UTMOS, 8.50% dWER, and 96.6% speaker similarity, compared with 3.73 UTMOS, 21.12% dWER, and 95.1% similarity using fixed boundaries. At approximately 9 Hz, adaptive segmentation reduces dWER from 6.29% to 4.60%. This shows that content-adaptive boundary placement is particularly beneficial under aggressive temporal compression, where the placement of individual tokens becomes increasingly important.
Autoregressive and non-autoregressive TTS modeling
We have extended the TTS evaluation to include autoregressive modeling of DyCAST-CA tokens, using the same modeling architecture and training protocol employed for the other codecs (Section 5.3, Table 3). Under autoregressive modeling, DyCAST-CA achieves 3.94 UTMOS, 7.57% dWER, and 93.8% speaker similarity, compared with 4.11 UTMOS, 28.10% dWER, and 93.3% similarity for FocalCodec50. The non-autoregressive architecture further improves DyCAST-CA to 4.23 UTMOS, 3.37% dWER, and 96.2% similarity. Overall, DyCAST-CA tokens achieve strong generative performance independently of the non-autoregressive architecture, while their one-to-one correspondence with characters additionally enables effective non-autoregressive synthesis.
Influence of silence on segmentation and duration reconstruction
We have added a dedicated analysis of silence segmentation and reconstruction (Appendix F.9, Figure 8), evaluating all four DyCAST configurations on 500 LibriSpeech test-clean utterances. We first analyze the number of token boundaries within silent intervals as a function of pause duration. For intervals shorter than 200 ms, DyCAST-BP1 produces an average of 0.54 internal boundaries, decreasing to 0.34 for BP3 and 0.24 for BP5. For intervals between 200 and 500 ms, the corresponding values are 2.10, 1.09, and 1.03, respectively. These results show that token allocation within silent regions decreases as the temporal compression increases. We additionally evaluate the accuracy of reconstructed silence durations. Using encoder-derived token durations, the silence-duration mean absolute error (MAE) ranges from 4.7 to 6.1 ms across DyCAST configurations. With freely predicted durations, the MAE increases to 11.9–16.2 ms. These results show that DyCAST preserves the temporal structure of silent intervals with high accuracy when token durations are available, while the duration predictor also provides reasonably accurate pause reconstruction without explicit duration information.
Robustness of segmentation under additive noise
We have added an analysis of segmentation stability under additive background noise (Appendix F.8, Figure 7). We evaluate 500 LibriSpeech test-clean utterances corrupted with WHAM! noise at signal-to-noise ratios of 20, 10, and 0 dB, comparing the resulting boundaries against clean-speech boundaries using F1 with a 40 ms tolerance. At 10 dB SNR, DyCAST-CA achieves a boundary F1 of 0.998, while BP1, BP3, and BP5 achieve 0.934, 0.968, and 0.950, respectively. At the more challenging 0 dB SNR, boundary stability decreases, with F1 between 0.855 and 0.891 for the BP configurations, while CA maintains 0.982. We also analyze changes in token allocation under noise. At 0 dB SNR, the average token rate decreases by approximately 0.99 Hz for BP1, 0.87 Hz for BP3, and 0.56 Hz for BP5, compared with 0.24 Hz for CA. In summary, DyCAST maintains stable segmentation under moderate additive noise, with limited changes in boundary placement and token allocation.
Relationship to Voxtral TTS
We have expanded Section 2 (Related Work) to discuss Voxtral TTS and clarify its relationship to DyCAST. Voxtral TTS leverages ASR supervision through Whisper cross-attention, aligning codec representations with ASR decoder hidden states for semantic distillation. DyCAST similarly exploits linguistic supervision, but applies it to a different aspect of the tokenization process: explicitly learning character-aligned chunk boundaries to control temporal segmentation. In particular, Voxtral TTS retains a fixed frame rate, whereas DyCAST uses learned boundaries to produce variable-length tokens whose temporal granularity can be adjusted at inference time.
We hope that the additional experiments and revisions address the reviewer's concerns. We remain available to provide any further clarifications.