acceptodds
Under review as a conference paper at ICLR 2027

Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization

Abstract

Neural audio codecs are at the core of modern conversational speech technologies, converting continuous speech into sequences of discrete tokens that can be processed by LLMs. However, existing codecs typically operate at fixed frame rates, allocating tokens uniformly in time and producing unnecessarily long sequences. In this work, we introduce DyCAST, a Dynamic Character-Aligned Speech Tokenizer that enables variable-frame-rate tokenization through soft character-level alignment and explicit duration modeling. DyCAST learns to associate tokens with character-level linguistic units during training and supports alignment-free inference with direct control over token durations at decoding time. Experiments show that DyCAST achieves competitive speech resynthesis quality and downstream performance while using significantly fewer tokens than fixed-frame-rate codecs. Demo samples are available at https://anonymous.4open.science/w/dycast-web/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
neural audio codecs, speech tokenization, variable frame rate

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qeC3OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DyCAST, a variable frame rate speech tokenizer that uses character boundary supervision and explicit duration prediction. It supports tokenization without text or an external aligner during inference, with adjustable temporal granularity and several duration-decoding modes. Experiments cover speech resynthesis, voice conversion, discriminative tasks, and text-to-speech, demonstrating competitive performance at substantially reduced frame rates.

Critical strengths
  • S1: The paper addresses an important practical problem: reducing speech sequence lengths while preserving information needed for reconstruction and downstream modeling. Combining learned boundaries with explicit duration prediction provides useful flexibility, including inference without an external aligner and controllable decoding durations.
  • S2: The empirical evaluation is broad, covering multilingual and noisy resynthesis, voice conversion, recognition, speaker identification, emotion recognition, and TTS. The results demonstrate a favorable balance between temporal compression and reconstruction quality. Appendix F further supports the evaluation through boundary analysis and comparisons of duration-decoding modes.
Critical weaknesses
  • W1: The experiments do not sufficiently establish why character alignment is the appropriate segmentation target. Comparisons with phoneme or syllable boundaries, or a matched-rate fixed segmentation using the same backbone and quantizer, would help distinguish the benefits of linguistic alignment from temporal compression and other architectural changes. The strongest TTS result also uses a different, non-autoregressive architecture, so its gains cannot be attributed to tokenization alone.
  • W2: The behavior of the boundary and duration predictors around silence and under noise remains unclear. Section 4.1 states that silence is merged into the following non-silence token, but the consequences for token allocation and pause reconstruction are not analyzed. Table 2 already evaluates noisy speech, and Appendix F.1 examines boundary agreement, but neither directly supports the proposed explanation that duration modeling improves noise robustness by avoiding token over-fragmentation.
Questions for authors
  • Q1: Can the authors provide a controlled comparison between character supervision and an alternative such as phoneme supervision or matched-rate fixed segmentation? Keeping the backbone, quantizer, and downstream architecture consistent would clarify the contribution of character alignment.
  • Q2: How do predicted boundaries and durations change with pause length and noise level? A focused analysis of token allocation within silence and reconstructed pause-duration errors would clarify whether merging silence into the following token preserves temporal structure reliably.
  • Q3: How does the proposed supervision relate to Voxtral TTS, whose codec uses Whisper cross-attention to align codec representations with ASR decoder states for distillation? Voxtral retains fixed-rate frames, so the mechanisms differ, but discussing this relationship would sharpen the distinction between ASR-guided representation learning and explicit boundary-guided temporal compression.
2 comments

We thank the reviewer for their constructive feedback. We have expanded the manuscript with additional experiments and analyses addressing the raised concerns.

Adaptive versus fixed-rate segmentation

We have added a comparison between adaptive and uniformly spaced boundaries at matched frame rates (Appendix F.6, Table 7). The experiment uses the same pretrained DyCAST model, quantizer, and decoder, changing only the segmentation strategy at inference time. At approximately 6.2 Hz, adaptive segmentation achieves 4.03 UTMOS, 8.50% dWER, and 96.6% speaker similarity, compared with 3.73 UTMOS, 21.12% dWER, and 95.1% similarity using fixed boundaries. At approximately 9 Hz, adaptive segmentation reduces dWER from 6.29% to 4.60%. This shows that content-adaptive boundary placement is particularly beneficial under aggressive temporal compression, where the placement of individual tokens becomes increasingly important.

Autoregressive and non-autoregressive TTS modeling

We have extended the TTS evaluation to include autoregressive modeling of DyCAST-CA tokens, using the same modeling architecture and training protocol employed for the other codecs (Section 5.3, Table 3). Under autoregressive modeling, DyCAST-CA achieves 3.94 UTMOS, 7.57% dWER, and 93.8% speaker similarity, compared with 4.11 UTMOS, 28.10% dWER, and 93.3% similarity for FocalCodec50. The non-autoregressive architecture further improves DyCAST-CA to 4.23 UTMOS, 3.37% dWER, and 96.2% similarity. Overall, DyCAST-CA tokens achieve strong generative performance independently of the non-autoregressive architecture, while their one-to-one correspondence with characters additionally enables effective non-autoregressive synthesis.

Influence of silence on segmentation and duration reconstruction

We have added a dedicated analysis of silence segmentation and reconstruction (Appendix F.9, Figure 8), evaluating all four DyCAST configurations on 500 LibriSpeech test-clean utterances. We first analyze the number of token boundaries within silent intervals as a function of pause duration. For intervals shorter than 200 ms, DyCAST-BP1 produces an average of 0.54 internal boundaries, decreasing to 0.34 for BP3 and 0.24 for BP5. For intervals between 200 and 500 ms, the corresponding values are 2.10, 1.09, and 1.03, respectively. These results show that token allocation within silent regions decreases as the temporal compression increases. We additionally evaluate the accuracy of reconstructed silence durations. Using encoder-derived token durations, the silence-duration mean absolute error (MAE) ranges from 4.7 to 6.1 ms across DyCAST configurations. With freely predicted durations, the MAE increases to 11.9–16.2 ms. These results show that DyCAST preserves the temporal structure of silent intervals with high accuracy when token durations are available, while the duration predictor also provides reasonably accurate pause reconstruction without explicit duration information.

Robustness of segmentation under additive noise

We have added an analysis of segmentation stability under additive background noise (Appendix F.8, Figure 7). We evaluate 500 LibriSpeech test-clean utterances corrupted with WHAM! noise at signal-to-noise ratios of 20, 10, and 0 dB, comparing the resulting boundaries against clean-speech boundaries using F1 with a 40 ms tolerance. At 10 dB SNR, DyCAST-CA achieves a boundary F1 of 0.998, while BP1, BP3, and BP5 achieve 0.934, 0.968, and 0.950, respectively. At the more challenging 0 dB SNR, boundary stability decreases, with F1 between 0.855 and 0.891 for the BP configurations, while CA maintains 0.982. We also analyze changes in token allocation under noise. At 0 dB SNR, the average token rate decreases by approximately 0.99 Hz for BP1, 0.87 Hz for BP3, and 0.56 Hz for BP5, compared with 0.24 Hz for CA. In summary, DyCAST maintains stable segmentation under moderate additive noise, with limited changes in boundary placement and token allocation.

Relationship to Voxtral TTS

We have expanded Section 2 (Related Work) to discuss Voxtral TTS and clarify its relationship to DyCAST. Voxtral TTS leverages ASR supervision through Whisper cross-attention, aligning codec representations with ASR decoder hidden states for semantic distillation. DyCAST similarly exploits linguistic supervision, but applies it to a different aspect of the tokenization process: explicitly learning character-aligned chunk boundaries to control temporal segmentation. In particular, Voxtral TTS retains a fixed frame rate, whereas DyCAST uses learned boundaries to produce variable-length tokens whose temporal granularity can be adjusted at inference time.

We hope that the additional experiments and revisions address the reviewer's concerns. We remain available to provide any further clarifications.

Thank you for running these additional experiments. The matched-rate comparison is helpful, which makes a clearer case for adaptive boundaries at low frame rates. The autoregressive TTS results also help show that the intelligibility gains aren't just due to the non-autoregressive architecture. The new pause and noise analyses address my questions. Additionally, the added discussion makes the connection to Voxtral TTS clearer.

I'm still curious about the choice of characters over phonemes or syllables. The new comparison shows why adaptive boundaries help, but it leaves open which alignment unit works best. Overall, I think the revisions strengthen the paper, and I'm keeping my original rating.

Reviewer_b26gOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DyCAST (Dynamic Character-Aligned Speech Tokenizer), a framework designed to achieve variable-frame-rate speech tokenization through soft character-level alignment and explicit duration modeling. The core architecture consists of: 1) a Chunker module that identifies semantic chunk boundaries using a frozen character aligner for training supervision and a discrete-time hazard model for boundary prediction; 2) a Dechunker module that employs a negative binomial duration model to predict token durations and supports multiple decoding regimes. The authors evaluate their framework on speech resynthesis, discriminative probing tasks, and text-to-speech (TTS) generation.

Critical strengths
  1. A meaningful problem and a coherent framework. DyCAST addresses the long token sequences produced by fixed-rate speech tokenizers through character-aligned dynamic chunking and explicit duration modeling. The resulting framework is flexible and supports alignment-free inference at substantially reduced average frame rates.

  2. Broad empirical evaluation against fixed-rate codecs. The paper evaluates resynthesis, voice conversion, discriminative probing, and TTS. The results show that DyCAST can retain competitive quality at approximately 6–17 Hz, demonstrating that the proposed direction is practically promising.

Critical weaknesses
  1. The core dynamic-rate contribution is not compared with dynamic-rate baselines. The paper’s main contribution is dynamic-frame-rate tokenization, but all quantitative baselines in Tables 1–3 use fixed frame rates. Although FlexiCodec [1] and CodecSlime [2] are discussed as closely related dynamic-rate methods, they are not included experimentally. Comparisons with fixed-rate codecs demonstrate the general utility of reducing the frame rate, but do not establish whether DyCAST advances over existing dynamic-rate approaches. This missing comparison directly limits my ability to assess the paper’s main novelty and empirical contribution.

  2. The central design choices are not supported by controlled ablations. The paper shows the discrete-time hazard objective as preferable to frame-wise BCE, the negative-binomial duration model as preferable to simpler duration objectives or distributions. However, the full system is evaluated without isolating these choices. Consequently, it remains unclear whether these components are necessary for the reported performance or whether simpler alternatives would work similarly. Since these components are the main technical implementation of DyCAST, this missing evidence affects my assessment of technical soundness.

[1] J. Li, Y. Qian, Y. Hu, L. Zhang, X. Wang, H. Lu, M. Thakker, J. Li, S. Zhao, and Z. Wu, "FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates," in Proc. ICLR, 2026.

[2] H. Wang, Y. Guo, C. Shao, B. Li, and K. Yu, "CodecSlime: Temporal Redundancy Compression of Neural Speech Codec via Dynamic Frame Rate," in Proc. IEEE ICASSP, 2026.

Questions for authors
  1. Could the authors compare DyCAST with direct dynamic-frame-rate baselines such as FlexiCodec and CodecSlime at approximately matched frame rates or bitrates, using the same resynthesis metrics and evaluation data? This comparison is important for establishing the contribution relative to the closest prior work.

  2. Could the authors provide controlled ablations for the main design choices, particularly:

  • discrete-time hazard modeling versus frame-wise BCE;
  • negative-binomial duration modeling versus Poisson or L1/MSE duration prediction;

These results would affect my final assessment. If DyCAST shows competitive performance with direct dynamic-rate baselines and the ablations verify its main design choices, I would be willing to raise my rating.

2 comments

We thank the reviewer for their constructive feedback. We have expanded the experimental evaluation, including direct comparisons with variable-frame-rate codecs and controlled ablations of the main algorithmic components.

Comparisons with variable-frame-rate codecs

We have extended the evaluation to include CodecSlime and VARSTok, considering four and three operating points, respectively. Both methods are evaluated using the same datasets, metrics, and downstream evaluation protocols as DyCAST (Section 5, Tables 1–3). We additionally provide bitrate–quality curves in Appendix F.1, Figure 3. On LibriSpeech test-clean, DyCAST-BP3 achieves 4.01 UTMOS, 4.60% dWER, and 97.4% speaker similarity at approximately 9 Hz (0.57 kbps). At a comparable bitrate, CodecSlime-40 achieves 4.05 UTMOS, 7.24% dWER, and 97.3% similarity at approximately 40 Hz (0.60 kbps). VARSTok-0.8 achieves 3.74 UTMOS, 14.56% dWER, and 94.1% similarity at approximately 34.5 Hz (0.48 kbps). The downstream evaluations further highlight DyCAST ability to preserve linguistic information while maintaining high acoustic reconstruction quality. DyCAST-CA achieves 13.05% WER on ASR, compared with 27.90–38.45% for CodecSlime and approximately 48–51% for VARSTok. In autoregressive TTS, DyCAST-CA achieves 7.57% dWER, compared with 32.20–46.82% for CodecSlime and 54.92–57.84% for VARSTok, while maintaining competitive perceptual quality and speaker similarity. Overall, these results highlight DyCAST ability to maintain competitive reconstruction quality and strong downstream performance while operating at substantially lower frame rates than the evaluated variable-frame-rate codecs. In particular, the results show the suitability of character-aligned tokenization for speech understanding and generation.

Boundary prediction objective: hazard vs. BCE

We have added a controlled comparison between the hazard objective and standard frame-wise binary cross-entropy (BCE) (Appendix F.3, Figure 4, left). Both predictors are trained from scratch using the same architecture, frozen WavLM features, and character-alignment targets. Detection thresholds are calibrated on the validation set to obtain comparable token rates (14.56 Hz for hazard and 14.44 Hz for BCE, compared with a reference rate of 14.46 Hz). The hazard objective improves boundary recall for chunks longer than 40 ms, with the largest improvement of 7.89 percentage points for chunks between 160 and 320 ms. BCE achieves slightly higher overall F1 (94.89% vs. 94.42%), while the hazard objective favors boundaries following longer chunks. These results characterize the duration-dependent inductive bias introduced by the hazard formulation, which is particularly relevant for variable-length speech segmentation.

Duration prediction objective: negative binomial vs. Poisson and MSE

We have added a controlled comparison of negative binomial (NB), Poisson, and MSE-based duration prediction (Appendix F.3, Figure 4, right). All predictors share the same architecture and are trained from scratch using identical frozen quantized representations and reference character durations. We evaluate utterance-length prediction without normalization to the reference utterance length, allowing the duration models to be compared directly. On LibriSpeech test-clean, NB achieves the lowest utterance-length MAE of 402.64 ms, compared with 443.95 ms for Poisson and 418.26 ms for MSE, corresponding to a 9.3% reduction relative to Poisson. Although Poisson achieves slightly better token-level accuracy, NB better preserves overall utterance duration, an important property for speech reconstruction and generation. These results provide empirical support for the negative binomial formulation adopted in DyCAST.

We hope that the additional baseline comparisons and controlled ablations address the reviewer's concerns. We remain available to provide any further clarifications.

Thank you for the detailed rebuttal and for adding the variable-frame-rate baselines and ablation studies, which address my main concerns. One minor remaining point is that FlexiCodec—which operates in the same 6–12.5 Hz ultra-low frame-rate regime as DyCAST (unlike CodecSlime at 36–75 Hz and VARSTok at 25–34.5 Hz)—is still missing; including it in the final version would make the comparison more complete. Overall, the added experiments strengthen the paper, and I have raised my score from 2 to 3.

Reviewer_U6X7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a variable-frame-rate speech tokenizer, DyCAST, through soft character-level alignment and explicit duration modeling. This work is inspired by FocalCodec. The paper uses argmax CTC character alignments from a frozen MMS model to supervise a hazard-model boundary predictor, and groups frames into character-level chunks. A negative binomial duration predictor restores token durations during decoding. DyCAST is evaluated on speech resynthesis, voice conversion, and both discriminative (ASR, SI, SER probing) and generative (TTS) downstream tasks. The experimental results show that DyCAST achieves competitive performance with significantly fewer tokens than fixed-frame-rate codecs, and particularly, DyCAST-CA achieves superior ASR and TTS performance.

Critical strengths
  1. As clearly compared with related works, DyCAST learns to associate tokens with character-level linguistic units during training, rather than heuristic token merging or requiring transmitting token durations for reconstruction. The character-level alignment supports a fully non-autoregressive TTS model that operates E2E without requiring text and enabling direct token duration control at inference time. The work has notable and valuable novelty.

  2. DyCAST provides substantial flexibility for both encoding and decoding at inference time. The three operating modes for decoding, including Tokens+Durations, Tokens+Utterance length, Tokens-only, demonstrate the flexibility. And the corresponding results and analysis in Table 5 Librispeech Resynthesis clearly shows the tradeoff between timing accuracy, intelligibility, and decoding complexity.

  3. The evaluations are comprehensive, including speech resynthesis, voice conversion, and both discriminative (ASR, SI, SER probing) and generative (TTS) downstream tasks.

Critical weaknesses
  1. As a variable-frame-rate work, this paper missed comparing to some important variable-frame-rate baselines, leading to serious concerns on the positioning of the proposed approach:

  a. The authors only briefly discussed recent dynamic-frame-rate tokenization in Related Work, including CodecSlime (Wang et al., 2025a), the VRF codec (Zhang et al., 2025), the adaptive clustering-based codec (Zheng et al., 2026), and FlexiCodec (Li et al., 2025b), pointing out that these methods rely on heuristic frame merging or (adaptive) clustering that are weakly grounded in linguistics. However, there are no empirical comparisons to these dynamic-frame-rate codecs.

  b. The paper also very briefly mentioned codecs that require transcriptions or alignment information at inference time (Tseng et al., 2025; Hsu et al., 2025), but these baselines are also not empirically compared.

  c. As raised in the Weaknesses point below, there are not comparisons to FocalCodec (which DyCAST is built upon) at low bitrates (DyCAST is evaluated at different bitrates, but FocalCodec is only evaluated at 0.65 kbps bitrate).

  d. The TTS experiments show superior performance on UTMOS, dWER, and Sim from DyCAST-CA among all codecs compared. However, architectural and training factors could also contribute to the gain, which is confounded with the tokenizer design. It would be useful to compare to a fixed-rate codec baseline with NAR/duration, or compare to an AR model using the DyCAST-CA tokens.

  1. Some important technical details are missing:

  a. For DyCAST, each token is decomposed into 32 parallel streams, each with vocabulary size 4. The paper mentioned that tokens are flattened across the codebook dimension, but it is unclear how the tokens are embedded and predicted in an AR model.

  b. The token utilization analysis of DyCAST is missing.

  c. DyCAST by design could potentially support flexible and direct control over token durations at decoding time, as the paper kept emphasizing. However, there is no empirical validation on the controllability, e.g., speaking rate control?

  1. Key algorithmic designs are not ablated:

  a. To decouple the benefit of SSQ quantizer and the dynamic character-aligned chunking, the authors could compare to a fixed-rate alternative using the same architecture of DyCAST.

  b. A related question, please compare SSQ with BSQ and FSQ to explicitly show its benefits.

  c. There is no comparison between hazard and frame-wise BCE, so the design of the hazard formulations for boundaries is not ablated.

  d. There is also no ablation for the negative binomial duration model by explicitly modeling over-dispersed count data.

  1. The discussions of the results are not thorough and careful, with some over-claiming statements. For speech resynthesis and voice conversion results in Table 2, DyCAST-CA and -BP1 have bitrates (kbps) as 0.92 and 1.12 respectively, both higher than the most related baseline FocalCodec at 0.65 kbps bitrate, yet their dWERs are much worse than the dWER from FocalCodec. At lower bitrates of DyCAST-BP3 and -BP5 at 0.57 and 0.40 kbps, their dWERs degrade substantially, compared to FocalCodec. The claims in the paper that “DyCAST-CA and DyCAST-BP1 achieve dWER values comparable to high-rate baselines” and “for multilingual speech… a moderate increase in dWER as the frame rate is reduced” are over-claimed. It would help if a rate-distortion curve can be provided, to gain more insights into the rate-distortion efficiency.
Questions for authors
  1. Please address points under Weaknesses.

  2. The provided demo page is expired.

  3. There is no limitation discussion about the work.

4 comments

We thank the reviewer for their constructive feedback. We have revised the manuscript and added several experiments to address the concerns raised.

Comparisons with variable-frame-rate codecs

We have extended the evaluation to include CodecSlime and VARSTok, considering four and three operating points, respectively. Both methods are evaluated using the same datasets, metrics, and downstream evaluation protocols as DyCAST (Section 5, Tables 1–3). We additionally provide bitrate–quality curves in Appendix F.1, Figure 3. On LibriSpeech test-clean, DyCAST-BP3 achieves 4.01 UTMOS, 4.60% dWER, and 97.4% speaker similarity at approximately 9 Hz (0.57 kbps). At a comparable bitrate, CodecSlime-40 achieves 4.05 UTMOS, 7.24% dWER, and 97.3% similarity at approximately 40 Hz (0.60 kbps). VARSTok-0.8 achieves 3.74 UTMOS, 14.56% dWER, and 94.1% similarity at approximately 34.5 Hz (0.48 kbps). The downstream evaluations further highlight DyCAST ability to preserve both linguistic and acoustic information. DyCAST-CA achieves 13.05% WER on ASR, compared with 27.90–38.45% for CodecSlime and approximately 48–51% for VARSTok. In autoregressive TTS, DyCAST-CA achieves 7.57% dWER, compared with 32.20–46.82% for CodecSlime and 54.92–57.84% for VARSTok, while maintaining competitive perceptual quality and speaker similarity. Overall, DyCAST maintains competitive reconstruction quality and strong downstream performance while operating at substantially lower frame rates than the evaluated variable-frame-rate codecs, making its representations particularly attractive for generative speech modeling.

Additional low-bitrate FocalCodec comparisons

We have extended the FocalCodec evaluation to include 25 Hz (0.33 kbps) and 12.5 Hz (0.16 kbps) configurations, alongside the original 50 Hz (0.65 kbps) baseline (Section 5, Tables 1–3). On LibriSpeech test-clean, FocalCodec25 and FocalCodec12.5 achieve 3.30% and 7.94% dWER, respectively, confirming FocalCodec favorable bitrate–intelligibility trade-off. In contrast, DyCAST offers character-aligned representations, adaptive temporal resolution, and inference-time frame-rate control. DyCAST also achieves strong downstream performance. DyCAST-CA obtains the lowest ASR WER (13.05%) among all evaluated codecs, compared with 17.63–33.24% for FocalCodec. In autoregressive TTS, DyCAST-CA achieves 7.57% dWER, compared with 28.10–30.28% for FocalCodec, while maintaining competitive naturalness and speaker similarity.

Autoregressive and non-autoregressive TTS modeling

We have extended the TTS evaluation to include autoregressive modeling of DyCAST-CA tokens, using the same modeling architecture and training protocol employed for the other codecs (Section 5.3, Table 3). Under autoregressive modeling, DyCAST-CA achieves 3.94 UTMOS, 7.57% dWER, and 93.8% speaker similarity, compared with 4.11 UTMOS, 28.10% dWER, and 93.3% similarity for FocalCodec50. The non-autoregressive architecture further improves DyCAST-CA to 4.23 UTMOS, 3.37% dWER, and 96.2% similarity. These results show that DyCAST-CA tokens achieve strong generative performance under autoregressive modeling, while their one-to-one correspondence with characters additionally enables effective non-autoregressive synthesis.

Autoregressive modeling of factorized SSQ tokens

We have expanded Appendix E.6 to clarify how the factorized SSQ tokens are represented and modeled. Each DyCAST token consists of 32 subcodes, each taking one of four discrete values. These subcodes are flattened in frame-major order and modeled autoregressively using next-token prediction. After generation, the subcodes are regrouped into the original token representation for waveform reconstruction. The revised manuscript provides further details on the embedding and prediction procedure.

Codebook utilization

We have added a codebook utilization analysis in Appendix F.2, Table 5. All four DyCAST configurations achieve 100% utilization of the individual subcodebook values, with normalized marginal entropy ranging from 95.40% to 95.56%. These results indicate effective utilization of the factorized representation across segmentation strategies and frame rates, without evidence of marginal subcodebook collapse.

Duration controllability

We have added experiments evaluating global speaking-rate control and local duration manipulation (Appendix F.5, Figure 6). For global control, we evaluate duration scaling factors between 0.70 and 1.30 on 500 LibriSpeech test-clean utterances. The achieved duration ratios closely follow the targets, reaching 0.697 and 1.319 at the two extremes. For local control, we modify the durations associated with a selected word while keeping the discrete token sequence unchanged. On 200 TIMIT utterances, scaling factors of 0.5 and 2.0 produce average reconstructed word-duration ratios of 0.74 and 1.73, respectively. These results show that DyCAST supports both utterance-level speaking-rate adjustments and localized duration manipulation through explicit control over token durations.

Adaptive versus fixed-rate segmentation

We have added a controlled comparison between adaptive and uniformly spaced boundaries at matched average frame rates (Appendix F.6, Table 7). The experiment uses the same pretrained DyCAST model, quantizer, and decoder, varying only the segmentation strategy at inference time. This isolates the impact of adaptive boundary placement at inference time from architectural differences. At approximately 6.2 Hz, adaptive segmentation achieves 4.03 UTMOS, 8.50% dWER, and 96.6% speaker similarity, compared with 3.73 UTMOS, 21.12% dWER, and 95.1% similarity using uniformly spaced boundaries. At approximately 9 Hz, adaptive segmentation also improves dWER from 6.29% to 4.60%. This shows that content-adaptive boundary placement is particularly beneficial under aggressive temporal compression.

Quantizer ablation

We have added a controlled comparison of SSQ, BSQ, and FSQ under a matched 16-bit-per-token budget, using identical reconstruction architectures and reference chunk durations (Appendix F.3, Figure 5). SSQ achieves the lowest validation reconstruction MSE of 2.837, compared with 2.853 for BSQ and 2.883 for FSQ, corresponding to relative reductions of 0.55% and 1.60%, respectively. SSQ therefore provides competitive reconstruction accuracy while allowing the number of quantization levels to be adjusted independently of latent dimensionality.

Boundary prediction objective

We have added a controlled comparison between the hazard objective and frame-wise binary cross-entropy (BCE) (Appendix F.3, Figure 4, left). Both predictors share the same architecture and training data, with detection thresholds calibrated to comparable token rates. The hazard objective improves boundary recall for chunks longer than 40 ms, with the largest improvement of 7.89 percentage points for chunks between 160 and 320 ms. BCE achieves slightly higher overall F1 (94.89% vs. 94.42%), while the hazard objective favors boundaries following longer chunks.

Duration prediction objective

We have added a controlled comparison of negative binomial (NB), Poisson, and MSE-based duration prediction (Appendix F.3, Figure 4, right). All three predictors share the same architecture and use identical frozen quantized representations and reference duration targets. On LibriSpeech test-clean, NB achieves the lowest utterance-length MAE of 402.64 ms, compared with 443.95 ms for Poisson and 418.26 ms for MSE, corresponding to a 9.3% reduction relative to Poisson. Although Poisson achieves slightly better token-level accuracy, NB better preserves overall utterance duration, an important property for speech reconstruction and generation.

Bitrate–quality analysis and discussion

We have expanded the interpretation of the reconstruction and downstream results in Section 5 and added bitrate–quality curves in Appendix F.1, Figure 3. The curves compare DyCAST, FocalCodec, CodecSlime, and VARSTok across multiple operating points in terms of UTMOS, dWER, and speaker similarity. The revised analysis distinguishes the different strengths of these approaches: FocalCodec provides a more favorable bitrate–intelligibility trade-off, whereas DyCAST offers competitive perceptual quality and speaker similarity together with adaptive tokenization and inference-time frame-rate flexibility.

Demo page

The demo page has been restored and updated with audio samples from the additional baselines, enabling more comprehensive qualitative comparisons.

Limitations

We have added a dedicated Limitations section (Appendix A) discussing the scope of the experimental evaluation and directions for future work.

We hope that the additional experiments and revisions address the reviewer's concerns. We remain available to provide any further clarifications.

Dear Reviewer U6X7,

We hope our response has addressed your concerns. As the discussion period closes on November 18, please let us know if you have any remaining questions; we would be happy to clarify further.

Thank you for your time.

I sincerely appreciate the authors' detailed response, which I think have addressed all of my questions and concerns. The summary of the main revisions is helpful and I have gone through the revised/updated sections. The restored demo page provides reasonable samples to demonstrate the effectiveness of DyCAST compared with a wide range of baselines. I increased my rating to Weak acceptance.

Comments on the paper

2 comments

We thank the reviewers for their constructive feedback. We have revised the paper to address the raised concerns and improve the experimental evaluation. We have uploaded a revised PDF, with changes highlighted in blue.

Summary of the main revisions:

  • Variable-frame-rate baselines: Extended the evaluation to CodecSlime and VARSTok at multiple operating points (Section 5, Tables 1–3).
  • Low-bitrate FocalCodec baselines: Added FocalCodec configurations operating at 12.5 and 25 Hz, in addition to the original 50 Hz configuration (Section 5, Tables 1–3).
  • Demo page update: Fixed issues with the demo page and updated it to include audio samples from the additional baselines.
  • TTS modeling: Extended the evaluation of DyCAST-CA tokens to both autoregressive and non-autoregressive TTS models (Section 5.3, Table 3).
  • Token representation and modeling: Clarified how the factorized SSQ tokens are embedded and predicted in autoregressive models (Appendix E.6).
  • Codebook utilization: Added an analysis of codebook utilization and token diversity (Appendix F.2).
  • Duration control: Added experiments evaluating global speaking-rate control and local duration manipulation (Appendix F.5).
  • Quantizer ablation: Compared SSQ against BSQ and FSQ under a matched quantization budget (Appendix F.3).
  • Boundary prediction ablation: Compared the proposed hazard objective against frame-wise binary cross-entropy (Appendix F.3).
  • Duration prediction ablation: Compared the negative binomial model against Poisson and MSE-based duration prediction (Appendix F.3).
  • Bitrate–quality trade-offs: Added bitrate–quality curves across different operating points, analyzing perceptual quality, intelligibility, and speaker similarity (Appendix F.1).
  • Adaptive vs. fixed-rate segmentation: Added a controlled comparison using the same pretrained model to isolate the effect of adaptive boundary placement (Appendix F.6).
  • Revised results discussion: Expanded the interpretation of reconstruction and downstream results (Section 5).
  • Robustness to additive noise: Added an analysis of segmentation robustness under noisy conditions (Appendix F.8).
  • Influence of silence: Investigated how silence affects segmentation behavior and reconstruction (Appendix F.9).
  • Limitations: Added an explicit discussion of limitations and future research directions (Appendix A).
  • Related work restructuring: Reorganized the Related Work section to clearly distinguish fixed- and variable-frame-rate codecs, and expanded the discussion of text-aligned speech representations to include Voxtral TTS (Section 2).
  • Abstract, introduction, and conclusion: Revised the abstract, introduction, and conclusion to reflect the expanded experimental evaluation.

Below, we address each reviewer comments individually, referring to the corresponding sections, tables, and figures in the revised manuscript.

We thank all reviewers for their constructive feedback and for taking the time to evaluate our rebuttal and additional experiments. We appreciate their suggestions, which have helped us improve the paper. We will further refine the manuscript in the final version, incorporating their additional feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.