acceptodds
Under review as a conference paper at ICLR 2027

DelexWav: An Audible Representation of Fine-Grained Prosody

Abstract

Speech carries both lexical content and prosody. Explicit acoustic features describe selected aspects of prosody, while learned vectors and tokens are heard through a downstream generator. We introduce DelexWav, a learned waveform representation that preserves fine-grained prosody while suppressing recoverable lexical content. DelexWav provides utterance-specific prosodic conditioning for speech synthesis. Its waveform form enables direct listening, acoustic comparison with the original speech, and local editing. We construct fixed-phone targets using phoneme-level conditional-flow TTS and use them as the main supervision for a neural codec that extracts DelexWav directly from speech, without transcripts or phone alignments at inference. On 1,234 LibriSpeech test-clean utterances, DelexWav preserves pitch and energy variation with correlations of 0.96 and 0.85. A phone recognizer trained on DelexWav recovers no complete source words. The English-trained model also preserves prosodic variation in cross-lingual tests without retraining. Speech synthesis experiments show that DelexWav provides fine-grained prosodic conditioning, with local edits producing corresponding changes in the generated speech. Samples are provided in the supplementary material.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
speech representation learning, prosody disentanglement, delexicalized speech, neural audio codec, controllable speech synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bUKbOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DelexWav, a codec trained to convert recordings into playable waveforms that aim to remove lexical content while preserving prosody. Extraction requires neither text nor phoneme alignment. The waveform can be edited locally and used with text to condition a separately trained speech generator. In same-text reconstruction, it achieves lower word error rates and more accurate phoneme timing than F0+Energy and FACodec, but some prosody metrics are weaker, cross-text generation is unreliable, and editing increases recognition errors.

Critical strengths
  1. Extraction requires only a recording, without text or phoneme alignment. In Table 1, energy correlation is 0.85 versus 0.67 for MBROLA m, and combined WPT correlation at the 0.125-second scale is 0.62 versus 0.32. The English-trained model also achieves F0 correlations of approximately 0.95 on Mandarin and German, supporting some cross-lingual prosody preservation.

  2. With the same backbone, data, and training schedule, same-text reconstruction achieves a WER of 7.18%, compared with 35.52% for F0+Energy and 65.91% for FACodec, with lower phoneme boundary error as well.

Critical weaknesses

(1) The downstream utility of the proposed representation remains insufficiently established. The same-text reconstruction results in Table 4 demonstrate useful improvements in phone timing and word accuracy, but primarily validate reconstruction fidelity rather than establish a practical use case for the representation. A particularly relevant application would be transferring the prosody of utterance A to a different target text B. However, the exploratory diagnostic in Appendix C.5 reports a target-text WER of 103.97% across 144 outputs from 48 A/B pairs and three seeds; only 59 outputs are closer to B than to A, while 75 are closer to A. These results indicate a severe failure to synthesize the intended target content. Although the authors acknowledge this limitation, it substantially restricts the demonstrated usefulness of DelexWav as reusable prosodic conditioning. The paper therefore leaves unclear which downstream application would materially benefit from the proposed audible representation beyond reconstructing the source utterance.

(2) The editing experiments do not establish a compelling advantage over simpler alternatives, and the edits substantially degrade content accuracy. The demonstrated operations are limited to local pitch shifting and time stretching, both of which can be implemented using explicit acoustic controls or conventional waveform processing. The paper’s own comparisons do not establish an advantage for DelexWav on these operations: Table 10 reports WERs of 20.61% and 24.82% after time scaling by 1.25 and 0.80, compared with 11.54% and 15.00% for conventional waveform stretching, which also better preserves pitch and energy contours. Furthermore, Table 5(b) shows that modest pitch shifts of +2 and −2 semitones increase WER from 10.35% to 29.95% and 46.29%, respectively. Thus, a directional response to an edit does not by itself demonstrate practically useful control. Without evidence of a distinct editing capability or a better trade-off between control and content preservation, the practical significance of introducing a learned audible waveform remains limited.

My main concern is that the practical value of the proposed audible prosody representation remains unclear. Same-text reconstruction mainly serves as a validation experiment, while the demonstrated pitch and duration edits are achievable with other existing simpler methods and introduce substantial content errors. Cross-text prosody transfer would be a more compelling application, but the reported results are highly unreliable, leaving the downstream usefulness insufficiently established.

Questions for authors
  1. What is the primary downstream use case for DelexWav beyond same-text reconstruction and local pitch/duration editing? Which reported results demonstrate a benefit of its audible waveform formulation over explicit acoustic features or conventional waveform processing?

  2. What causes the severe target-text fidelity failure in Appendix C.5: source-specific timing encoded in DelexWav, the generator’s same-text reconstruction training, or another limitation? Are controlled results available showing how the same generator handles text B without A’s DelexWav condition, or with B’s own condition, to distinguish these explanations?

  3. Given the stronger contour preservation and lower WER of conventional waveform stretching in Table 10, what practical advantage does editing DelexWav provide? Are comparable pitch-editing baseline results available to establish whether the audible representation improves the control–fidelity trade-off?

  4. Why do pitch changes of only ±2 semitones cause the large WER increases in Table 5(b)? Does the available failure analysis identify a setting in which local prosody can be meaningfully modified while maintaining content accuracy close to the unedited output?

Reviewer_ANYmOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose DelexWav, an audible waveform representation intended to preserve fine-grained prosody while suppressing lexical content. They construct synthetic targets through two stages of conditional-flow TTS, replacing the phone input with a fixed /m/ sequence during target generation. A neural codec learns to map original speech to these targets. The trained codec extracts DelexWav without transcripts or phone alignments. A separate speech generator uses its mel spectrogram, phones, and a speaker embedding for reconstruction and editing.

Experiments measure acoustic preservation, lexical recovery, cross-lingual extraction, and downstream resynthesis. DelexWav retains substantial pitch and energy variation, and the tested recognizer recovers no complete source words. It supports reconstruction and local prosody changes, although editing increases word errors and cross-text synthesis remains unreliable.

Critical strengths
  • The proposed pipeline learns to extract an audible delexicalized waveform directly from speech without transcripts or phone alignments at inference. Table 1 shows substantial pitch and energy preservation, with better energy and combined prosody-contour correlations than the tested MBROLA baseline. Table 3 provides further evidence that the English-trained extractor preserves these acoustic patterns in Mandarin and German.

  • The evaluation separates properties of the extracted waveform from its behavior in downstream synthesis. It includes original-speech recognition controls, matched downstream generators, and conventional waveform stretching. The appendix also reports unsuccessful cross-text synthesis and editing trade-offs. These results help define what the method currently supports.

Critical weaknesses
  • The added value of the learned waveform and its construction is not sufficiently established. The reported measurements mainly concern pitch, energy, voicing, and timing. No listening study establishes that fixed-phone replacement preserves perceived prominence or rhythm, or provides useful information beyond explicit features. The construction comparisons also change multiple factors. Specifically, Table 12(a) changes both synthesis and phone-replacement strategy, while Table 12(c) combines target mixing with additional training. These results leave the benefit of the proposed complexity unclear.

  • The evidence for lexical separation and reliable downstream control remains limited. The DelexWav recognizer produces a constant hypothesis, so zero recovered words does not distinguish strong suppression from an insensitive probe, despite the useful original-speech and 30%-target controls. In Table 4, the comparison systems have WERs of 35.52% and 65.91% despite receiving the correct phones, which makes the reconstruction advantage difficult to interpret. Local pitch edits also increase DelexWav WER from 10.35% to 29.95% and 46.29% (Table 5). These are substantial limitations for content-preserving control, although the authors acknowledge them.

Questions for authors
  • Could the authors clarify the constant hypothesis produced by the DelexWav recognizer and report its training and validation behavior? Evidence that predictions depend on the input, or results from an alternative decoding setup, would help interpret the negative lexical-recovery result.

  • Have the authors evaluated whether listeners perceive the source utterance’s emphasis, rhythm, and boundaries in DelexWav? Since replacing the original phones with /m/ changes the segmental context, perceptual evidence would help establish whether the preserved pitch and energy contours convey the intended prosody.

  • Could the authors provide an error breakdown or representative failures for the F0+Energy and FACodec-conditioned generators? Do available validation results indicate that both learned to follow the phone condition reliably?

  • Is a matched 350k-update codec trained only on synthetic targets available? Its acoustic and lexical-recovery results would help isolate the effect of adding original-speech targets.

Reviewer_n43LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents DelexWav, a model that takes a speech clip as input and converts it into a humming version, where every phoneme is replaced by /m/ but the pitch, loudness, timing, and other prosodic characteristics are retained. This humming /m/ version is treated as a audible prosodic representation as it presumably removes all content information but retains prosodic information, and can be listened to directly. The training targets are made in an interesting way. An F5-TTS model is trained to generate real speech given its text, a speaker embedding, and a rough prosodic guide that carries the original timing and pitch (a robotic version made with an MBROLA rule-based synthesizer). At generation time the text is replaced with /m/ sounds, so the F5-TTS model keeps the timing, pitch and roughly the voice of the speaker but says nothing intelligible.

This is done in two rounds. The first turns the robotic MBROLA outputs into more natural outputs, and the second produces the /m/ version. A DAC codec is then trained to map real speech straight to these targets. Experiments show that the DelexWav representation preserves the original clip's pitch and energy, and a recognizer trained on it recovers no words. When used as the prosody input to a separate TTS model, it gives better timing and fewer word errors than plain pitch and energy curves, but less accurate pitch and energy. The paper also edits part of DelexWav (shifting its pitch, or making it slower or faster) before passing it to the TTS model. The output changes in the right way, but its words become harder to recognize. Finally, when DelexWav from one sentence is paired with the text of a different sentence, the output does not match either sentence.

Critical strengths
  1. The approach is quite clever and definitely novel as far as I can tell. Giving a TTS model just /m/ sounds instead of the real text keeps the prosody of the sentence but removes content information. This can be thought of as a learned version of reiterant speech (a speaker repeating a sentence using only "ma"), an old approach for studying prosody (Liberman & Streeter, 1978, not cited) but likely new for the speech/ML field. Because the representation is itself audio, it can be evaluated with the same pitch and energy tools as the original audio, and these work.
  2. The complex pipeline is actually backed reasonably well by experiments. When the codec is trained on rule-based targets (the initial MBROLA ones), an off-the-shelf recognizer (Whisper) can detect some words in the output, as opposed to none with the TTS targets. The TTS targets also keep much more of the speaker's voice than rule-based. And the trained codec ends up closer to the original's pitch and energy than the targets it learned from.
Critical weaknesses
  1. The evidence that DelexWav removes the words is quite weak, and this is an important part of the paper's main claim. The main experiment (Table 2) trains small phone recognizers on real speech and on DelexWav. Because the real speech one can transcribe words but the DelexWav one can't, the paper says that proves DelexWav has successfully removed the words. But the Limitations section says the DelexWav recognizer outputs the same transcript for every input, i.e. it learned nothing. So the recognizer cannot distinguish different speech inputs; how can this tell us whether the words are gone? Off-the-shelf Whisper also fails on DelexWav, but that is expected since Whisper was trained on normal speech. Why not finetune a pretrained phone recognizer (e.g. https://arxiv.org/abs/2109.11680) on DelexWav instead of training a weak one from scratch?
  2. The TTS experiments do not show that DelexWav is a better prosody representation than plain pitch and energy curves, even though this is the paper's main application. Pitch and energy curves reproduce the original's pitch and energy better. DelexWav only wins on timing and WER, and I am not convinced about that. The pitch and energy baseline gets about 36% WER despite being given the correct transcript, which is far too high for a modern TTS model (these usually get a few percent on LibriSpeech) and makes me feel the baseline is undertrained. A TTS model given only the transcript and speaker would be a good sanity check here. Plus, the baseline gets no phone durations, so the timing comparison is not fair either.
  3. The paper is very hard to follow. It relies on its own terms (e.g. "fixed-phone targets") that are only briefly defined, and the motivation is unclear. The introduction says pitch and energy features are incomplete and learned prosody codes can only be heard through another model, and "therefore" proposes DelexWav. I don't see how this follows. DelexWav itself comes out of a learned decoder! If being audible is the main selling point, why is there is no listening test with human evals? Transferring prosody to a new sentence, which is a clear application, does not work reliably (Appendix C.5). So what is DelexWav actually for, other than re-synthesizing or editing the sentence it came from? Improving the writing and motivating the work better would help.
Questions for authors

Here are some clarification questions:

  1. How much LibriHeavy audio was converted to DelexWav to train its recognizer? What is the constant transcript it outputs, and does the pitch and energy recognizer collapse in the same way?
  2. Why are two rounds of TTS needed to build the targets? Was a single round, with the robotic MBROLA version as the guide, tried?

and here are some experiment requests:

  1. Listening tests, e.g. people writing down any words they hear in DelexWav, rating how closely its intonation and rhythm match the original, and rating the naturalness and prosody similarity of the TTS outputs.
  2. A comparison with Morrison et al. (2024) on prosody editing, since editing is one of the paper's proposed uses and that work is cited but not compared against.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.