acceptodds
Withdrawn from ICLR 2027

Construction Confidence Need Not Be Verification Confidence: Round-Trip Verification and Selective Repair for Masked Diffusion Speech

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Masked diffusion models can reduce computation by decoding in only a few iterative steps, but in speech generation this low-step regime can leave a minority of outputs with content failures, and which utterances fail can vary across sampling seeds. This suggests a selective strategy of spending extra computation only on outputs likely to fail. The challenge is identifying them. We study this problem in MaskGCT, a zero-shot text-to-speech system whose two stages are masked-diffusion decoders. The first maps text to semantic tokens, a discrete representation of what is said, and the second maps semantic tokens to acoustic tokens that are rendered into speech. The decoder therefore commits tokens, while the failure criterion is defined only after those tokens have been converted into a waveform. We define a content failure as rendered speech whose transcript has a word error rate (WER) above 10% against the target text, that is, speech whose content is unreliable rather than merely imperfect. This distinction gives rise to two kinds of confidence. Construction confidence comes from decoder-derived signals, such as token confidence and entropy, while the sequence is being committed. Verification confidence comes from checking the rendered speech for evidence that the output actually meets that criterion. Used as a failure detector on held-out data at an eight-step decoding budget, construction confidence falls well short of verification. A probing classifier trained on summaries of these signals (the learned probe) reaches AUC , whereas round-trip transcription with a separate small ASR verifier reaches . Round-trip transcription stays above every tested decoder-derived signal at the other decoding budgets we examined, and again in the acoustic stage. It also does so under failure labels from ASR families different from the verifier's, although the margin is smaller. Because a masked-diffusion sequence remains editable, verifier-localized spans can be re-masked and regenerated in place, whereas an autoregressive decoder would have to regenerate everything after them. This selective repair reduces transcript error relative to the unrepaired output across four ASR evaluators. These results show that, when a generator is judged only after its committed tokens have been transformed further, confidence used to construct those tokens need not be sufficient to verify the final output.

Primary area:
generative models
Keywords:
masked diffusion models, text-to-speech, selective prediction, failure detection, uncertainty estimation, output-space verification, semantic tokens, selective repair

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_TUhBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Setting: Speech generation, TTS.

Motivation: few-step generation. While it mostly works, it can fail for a minority of cases. Idea: identify them.

They use diffusion models for TTS. They measure how well the generated speech can be recognized by an ASR model. When that signal gets fed back into the TTS generation, it improves, i.e. the WER by the ASR model goes down.

They also tried to use intermediate features such as the token confidence or entropy to try to predict when it needs more steps, but all those other features cannot predict well the WER compared to using the WER from ASR.

It is a negative result against the entropy/confidence-based adaptive-scheduling for masked diffusion for TTS.

The motivation is that they can use a few diffusion steps only, and only do more where needed. But this goal is not achieved. More generation steps are always preferred.

The main finding is that entropy/confidence-based signals for masked diffusion for TTS are too weak to be useful (AUC 0.63).

Critical strengths

Interesting finding and measure about the usefullness of entropy/confidence-based signals.

Critical weaknesses

Main motivation is not achieved, to get better at few-step generation, and use more steps only when needed.

Methodology has weaknesses:

Headline numbers (AUC 0.864 vs 0.629) are only for the most family-aligned pairing (Whisper). Under cross-family labels, for Conformer, the gap is AUC 0.597 vs 0.682, so much less.

Single model. No variations of the model (e.g. training it with different random seeds, or trying other models).

No human evaluation.

More analysis on the finding should be done: why does this behave different than e.g. text diffusion LM?

Questions for authors

Two repair rounds - how does this work exactly?

Frozen round-trip threshold - what is the threshold actually?

1 comment

Q1: Two repair rounds - how does this work exactly?

A: We agree that the original manuscript did not explain this procedure clearly enough. First, the generated speech is transcribed using a small ASR model, and the resulting transcript is compared with the input text. When a mismatch is detected, the corresponding token positions are remasked and regenerated using eight denoising steps, while the remaining tokens are kept unchanged. The regenerated output is accepted only if its transcript error is strictly lower than before. If mismatches remain, the procedure is repeated once more, for a maximum of two repair rounds.

Q2: Frozen round-trip threshold - what is the threshold actually?

A: The threshold is set to zero, meaning that any discrepancy between the generated speech's transcript and the input text triggers the repair process. This threshold was determined using development data and kept fixed throughout the held-out evaluation.


Regarding the preference for increasing generation steps, we would like to clarify that beyond 32 steps, only one of the four ASR evaluators demonstrates a statistically reliable improvement from extending a single generation run (Appendix C.5). Nevertheless, we agree that the original objective of using the decoder's internal signals to determine where additional generation steps should be allocated remains unmet. This limitation is central to the negative finding presented in our paper.

We also recognize the concerns regarding the dependence of our findings on the verifier–label pairing, the evaluation of only a single model, the lack of human evaluation, and the need for a more thorough investigation into the differences between speech and text diffusion language models.

We sincerely thank you for your thoughtful questions and constructive feedback. We will carefully consider these points in our future research.

Reviewer_K5ByOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies content errors in low-step MaskGCT. It shows that decoder signals like confidence and entropy detect errors poorly, while ASR on the ouput speech works much better. It then uses ASR to find error spans, regenerates only thos spans, and reports lower WER than the unrepaired output.

Critical strengths

The paper reports experiments across various settings, including multiple decoding budgets, both generation stages, and multiple ASR evaluators.

Critical weaknesses

The repair method is essentially a standard ASR verification and partial regeneration pipeline, tested only on MaskGCT. Since ASR verification is already common practice, the contribution would be stronger with a better alternative, such as a cheaper learned verifier or improved decoder confidence. Instead, the main finding is that the tested internal signals perform worse than this existing practice, which offers little direction for improving the model itself.

Questions for authors

Decoder-derived signals are weaker failure detectors, but they reflect utterance-level information differences and are available before rendering. Did you find any distinct advantage of these signals, such as predicting difficult utterances early or complementing ASR verification?

1 comment

Q: Did you find any distinct advantage of these signals, such as predicting difficult utterances early or complementing ASR verification?

A: We appreciate this insightful question, which motivated us to conduct an additional analysis. We examined both possibilities on the held-out set but did not find a clear advantage in either case. Incorporating the learned probe into the round-trip score improved detection AUC by at most 0.016. Moreover, among outputs that passed ASR verification despite containing content errors, the probe showed limited ability to distinguish these missed failures, with AUC values ranging from 0.52 to 0.58. Regarding early prediction, although the decoder-derived signals remain relatively stable for a given utterance, failures at low step counts vary across different realizations of that utterance. This mismatch limits the signals' ability to predict such failures in advance.

We also recognize the concern that the repair pipeline largely builds on existing practices and that our findings provide limited guidance for directly improving the underlying model.

We are grateful for your constructive feedback and will take these points into consideration in our future research.

Reviewer_mstLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the problem of detecting and repairing content failures (e.g., mispronunciations or hallucinations) in masked diffusion speech generation. Using MaskGCT as a example, the authors argue that "construction confidence" (internal metrics like token probability and entropy) is significantly inferior to "verification confidence" (round-trip transcription using an external ASR model). Based on this finding, the authors propose an inference-time pipeline where an external ASR model is used to transcribe the generated audio, localize transcript mismatches, and perform remasking and regeneration on these mismatches. The method demonstrates marginal improvements in WER across different ASR evaluators.

Critical strengths

The authors apply extensive statistical rigor, utilizing 4,000-sample bootstrap intervals and Holm corrections to account for multiple comparisons, which adds credibility to the reported numbers.

The paper honestly reports the catastrophic failure of the proposed method under domain shift (e.g., on the Mandarin hard-set, the verifier flags ~95% of utterances, leading to worse performance).

Critical weaknesses

Limited scope (only tested on one model). The idea of using ASR to find errors and then redrawing those parts (inpainting) is a general trick. It could easily be used with other modern TTS architectures, like flow-matching models (e.g., F5-TTS [1]), which naturally support inpainting. By only testing this on one specific model (MaskGCT), the method feels more like a specific bug fix for MaskGCT rather than a broad, useful contribution to the field.

Poor cost-to-performance trade-off. The method adds a lot of extra computation. Running the ASR, aligning the text, and regenerating the audio increases inference time by about 24%. However, the improvement in Word Error Rate (WER) is very small, often less than 1%. In real-world scenarios, running a heavy ASR model just to get such a tiny gain is simply not worth the cost.

Making a simple idea sound too complex. The writing makes a straightforward engineering pipeline (ASR feedback + inpainting) sound much more complicated than it actually is. Inventing terms like "Construction Confidence" (which just means the model's internal probability) and "Verification Confidence" (which means checking with an external ASR) makes the paper harder to read and overstates the actual novelty.

No human listening tests. The paper entirely skips human subjective evaluations (like MOS or CMOS). While regenerating bad tokens might fix wrong words, patching speech like this often causes weird pauses, unnatural boundaries. Without real listening test on the results, we cannot know if the repaired audio actually sounds better or just trades a wrong word for a glitchy sound.

[1] Yushen Chen et al., “F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching,” ACL 2025.

Questions for authors

Since using ASR to find and fix errors is a general method, did you try applying this pipeline to other types of models, like F5-TTS? If not, how can we be sure this method is useful for TTS architectures other than MaskGCT?

How do you justify slowing down the generation by 24% for less than a 1% improvement in WER? Are there any practical use cases where this cost makes sense?

The method failed badly on the Mandarin hard-set because the ASR couldn't handle it. This shows the system is completely limited by the ASR's capability. If the ASR struggles, the TTS quality drops. How would you mitigate this issue in practice?

Why were human listening tests (MOS, CMOS) completely left out? Since regenerating parts of the audio can easily break the flow and naturalness of the voice, how do we know the "repaired" speech sounds smooth and natural to human ears?

1 comment

Q1: Did you try applying this pipeline to other types of models, like F5-TTS?

A: We have not evaluated other architectures, and we acknowledge this limitation. While the repair procedure could potentially extend to flow-matching models with inpainting capabilities, the detection analysis relies on token probabilities unavailable in continuous flow models. We therefore make no claims about generalization to other architectures.

Q2: How do you justify slowing down generation by 24% for less than a 1% improvement in WER?

A: We agree that the gain is modest relative to the computational cost. However, the relative content-failure reduction is 16–25% under Whisper-large-v3 and 4–16% under the other three evaluators. Repair compares favorably with additional denoising steps at the 32-step baseline, but not at 16 steps. For selective review, the 20% of outputs with the highest round-trip errors account for 64% of content failures in the 8-step setting.

Q3: The method failed on the Mandarin hard set because the ASR could not handle it. How would you mitigate this in practice?

A: We acknowledge that the approach depends on verifier reliability. Retrospective safeguards, such as skipping repair for extremely high transcript errors, eliminate the hard-set regression but reduce gains elsewhere, since high-error outputs often benefit most from repair. A stronger domain-adapted verifier would be a more practical solution.

Q4: Why were human listening tests left out? How do we know the repaired speech sounds natural?

A: We agree that this is an important limitation. As discussed in our general response, our evaluation focused on transcript-defined content reliability, not perceptual quality. We therefore cannot establish whether the repaired speech maintains naturalness, particularly whether regenerated spans introduce audible discontinuities.


We also acknowledge the concern that our terminology may overcomplicate the underlying pipeline.

We thank you for your detailed and constructive feedback and will take these points into account in our future work.

Reviewer_9LmaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes confidence measures in a masked diffusion model, using the zero-shot text-to-speech model MaskGCT as a case study. The authors define a content failure as an output with a word error rate (WER) above 10% and compare decoder-derived construction confidence with round-trip ASR verification. Their analysis suggests that the tested token-construction confidence measures are less effective than ASR verification at identifying failures in the final output.

Critical strengths

The paper is well written and easy to follow. The study raises an interesting question about whether token-reconstruction loss sufficiently captures the quality of the final generated speech. The authors conduct extensive experiments to support their analysis.

Critical weaknesses

The contribution of this work is limited. Although the authors provide a detailed comparison between construction confidence and ASR verification, their experiments are based on a single model architecture, MaskGCT. Therefore, it is unclear whether the conclusions generalize to other masked diffusion models, such as OmniVoice, or whether the results are partly specific to MaskGCT’s capacity or design.

The paper would make a stronger contribution for ICLR if the authors evaluated additional model architectures or proposed and evaluated a more reliable confidence metric for masked diffusion models based on their findings.

Questions for authors

The authors report objective speaker-similarity and naturalness measures, including UTMOS, but use ASR verification to decide whether to accept a repair. Have the authors considered incorporating speaker similarity and naturalness into the verification or repair-acceptance process? These properties are also important for zero-shot text-to-speech, in addition to speech intelligibility.

1 comment

Q: Have the authors considered incorporating speaker similarity and naturalness into the verification or repair-acceptance process?

A: We considered these properties only as after-the-fact checks, not as part of the acceptance decision, and we agree that is a limitation. A repaired candidate is accepted on one criterion only: its round-trip transcript error must decrease. Speaker similarity and UTMOS were computed afterwards, on outputs that had already been accepted, so they played no part in the decision. This means a repair that fixes a word while degrading the voice or the naturalness would still be accepted, and the paper cannot rule that out. Your suggestion addresses exactly this gap: an acceptance rule that also requires speaker similarity and predicted naturalness not to drop is a natural next design.

We also acknowledge the concern regarding the reliance on a single architecture.

We sincerely appreciate your valuable feedback and thoughtful suggestions, which we will carefully consider in our future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

Response to all reviewers

We thank the reviewers. We agree that the study uses a single system and one checkpoint, that no listening test was run, that the within-base repair gain is modest and evaluator-dependent, and that the two named terms should have been introduced more plainly. We do not contest these points.

On the absence of a listening test, which several reviews raise: this work began from the diffusion-language-model side, with the question of whether a masked-diffusion decoder's own confidence can verify its output, and we treated the speech system mainly as an instance of that question. We therefore checked content correctness with four ASR evaluators from different families and monitored side effects of repair with automatic measures, and limited the claim to transcript-defined content reliability. In doing so we gave too little weight to the fact that this is, in the end, a text-to-speech system, for which listening tests are a central part of evaluation. We agree the evidence is insufficient on that point, and we will make sure the follow-up work reflects it.

What the paper does establish, within that scope, is a prespecified held-out measurement. On MaskGCT, decoder-derived signals and a learned probe over them remain below the detection criterion, whereas round-trip transcription exceeds it. This gap is consistent with a variance mismatch: the signals vary across utterances, while failures at low step counts vary across realizations of the same utterance. We present repair as a consequence of that measurement rather than as a new method.

We thank all reviewers again for their careful and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.