- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Setting: Speech generation, TTS.
Motivation: few-step generation. While it mostly works, it can fail for a minority of cases. Idea: identify them.
They use diffusion models for TTS. They measure how well the generated speech can be recognized by an ASR model. When that signal gets fed back into the TTS generation, it improves, i.e. the WER by the ASR model goes down.
They also tried to use intermediate features such as the token confidence or entropy to try to predict when it needs more steps, but all those other features cannot predict well the WER compared to using the WER from ASR.
It is a negative result against the entropy/confidence-based adaptive-scheduling for masked diffusion for TTS.
The motivation is that they can use a few diffusion steps only, and only do more where needed. But this goal is not achieved. More generation steps are always preferred.
The main finding is that entropy/confidence-based signals for masked diffusion for TTS are too weak to be useful (AUC 0.63).
Critical strengths
Interesting finding and measure about the usefullness of entropy/confidence-based signals.
Critical weaknesses
Main motivation is not achieved, to get better at few-step generation, and use more steps only when needed.
Methodology has weaknesses:
Headline numbers (AUC 0.864 vs 0.629) are only for the most family-aligned pairing (Whisper). Under cross-family labels, for Conformer, the gap is AUC 0.597 vs 0.682, so much less.
Single model. No variations of the model (e.g. training it with different random seeds, or trying other models).
No human evaluation.
More analysis on the finding should be done: why does this behave different than e.g. text diffusion LM?
Questions for authors
Two repair rounds - how does this work exactly?
Frozen round-trip threshold - what is the threshold actually?
Q1: Two repair rounds - how does this work exactly?
A: We agree that the original manuscript did not explain this procedure clearly enough. First, the generated speech is transcribed using a small ASR model, and the resulting transcript is compared with the input text. When a mismatch is detected, the corresponding token positions are remasked and regenerated using eight denoising steps, while the remaining tokens are kept unchanged. The regenerated output is accepted only if its transcript error is strictly lower than before. If mismatches remain, the procedure is repeated once more, for a maximum of two repair rounds.
Q2: Frozen round-trip threshold - what is the threshold actually?
A: The threshold is set to zero, meaning that any discrepancy between the generated speech's transcript and the input text triggers the repair process. This threshold was determined using development data and kept fixed throughout the held-out evaluation.
Regarding the preference for increasing generation steps, we would like to clarify that beyond 32 steps, only one of the four ASR evaluators demonstrates a statistically reliable improvement from extending a single generation run (Appendix C.5). Nevertheless, we agree that the original objective of using the decoder's internal signals to determine where additional generation steps should be allocated remains unmet. This limitation is central to the negative finding presented in our paper.
We also recognize the concerns regarding the dependence of our findings on the verifier–label pairing, the evaluation of only a single model, the lack of human evaluation, and the need for a more thorough investigation into the differences between speech and text diffusion language models.
We sincerely thank you for your thoughtful questions and constructive feedback. We will carefully consider these points in our future research.