- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes OLIVE, a speech self-supervised learning framework that jointly optimizes masked latent prediction and waveform reconstruction to support both speech analysis and synthesis. It introduces waveform view augmentations and a functional separation between early acoustic features and deeper contextual representations, and evaluates the framework on SUPERB downstream tasks and waveform reconstruction.
Critical strengths
-
The paper explores the joint optimization of speech analysis and synthesis within a unified self-supervised learning framework. Learning representations that support both discriminative downstream tasks and waveform reconstruction is a practically relevant research direction.
-
The paper evaluates OLIVE across a broad range of SUPERB downstream tasks and waveform reconstruction metrics, with comparisons against several established speech SSL baselines.
Critical weaknesses
- Limited methodological novelty.
OLIVE primarily combines existing techniques, including data2vec 2.0-style masked latent prediction, waveform augmentations, and HiFi-GAN-based waveform reconstruction, within a multi-task learning framework. Importantly, jointly optimizing masked latent prediction and waveform reconstruction has already been explored in data2vec-SG (Wang et al., 2023), which similarly aims to learn representations useful for both speech analysis and generation. The main distinction of OLIVE appears to be applying reconstruction supervision to early local encoder features rather than deeper contextual representations, together with independently augmented teacher–student views and a retained vocoder. However, the observation that early speech SSL layers preserve acoustic details while deeper layers encode more abstract information is already well established. Consequently, the proposed method appears largely incremental, and it remains unclear whether these modifications constitute a sufficiently substantial methodological contribution beyond existing analysis–synthesis SSL frameworks.
- Unclear motivation and necessity of the proposed design.
(i) The paper does not sufficiently justify why an additional explicit waveform reconstruction objective is necessary for speech SSL pre-training. Existing speech SSL methods already learn rich representations through masked prediction and reconstruction-based objectives. Prior work [1, 2] has also demonstrated that reconstruction-based pre-training can learn meaningful phonetic and linguistic representations.
(ii) Although the paper proposes a functional separation between analysis and synthesis, it does not clearly explain how the proposed training strategy establishes or enforces such a separation beyond applying two existing objectives at different encoder depths. Existing speech SSL models naturally encode different levels of acoustic and linguistic information across their layers. The paper's own results in Appendix C.3 (Table 10) show that WavLM's early features are also more suitable for waveform reconstruction than its deeper representations. Therefore, the observed hierarchical division of labor does not necessarily result from OLIVE's joint training strategy. The paper lacks convincing evidence that the proposed approach produces a more effective functional separation than conventional SSL pre-training.
(iii) The joint variants achieve limited gains on recognition tasks and mixed results on generation-related tasks, while the highest overall SUPERB score comes from an analysis-only variant without reconstruction. These findings leave the necessity of the proposed joint design insufficiently substantiated, particularly given that existing SSL representations can already support waveform reconstruction through separately trained vocoders.
[1] Chorowski et al. (2019). Unsupervised Speech Representation Learning Using WaveNet Autoencoders.
[2] Liu et al. (2020). Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders.
Questions for authors
see weakness above.
2 comments
"(iii) The joint variants achieve limited gains on recognition tasks and mixed results on generation-related tasks, while the highest overall SUPERB score comes from an analysis-only variant without reconstruction. These findings leave the necessity of the proposed joint design insufficiently substantiated, particularly given that existing SSL representations can already support waveform reconstruction through separately trained vocoders."
The joint design, in its Mask-only configuration, gives more faithful waveform reconstruction while maintaining strong recognition performance. Meanwhile, augmentations control which information the representation encodes, which shifts its usefulness across tasks. On aggregates, OLIVE-A (Mix+Gain) has the highest SUPERB$_s$ (909), and OLIVE-J (Mix) has the highest WavLM$_s$ (67.6) with a SUPERB$_s$ of 908, the best speaker and generation category scores, and the best reconstruction results.
While existing SSL representations can already support waveform reconstruction, our results show that analysis-only representations do not reach the same reconstruction faithfulness: with the same separately trained vocoder setup, PESQ is 2.21 for data2vec 2.0 and WavLM and 2.62 for OLIVE-A (Mix+Gain), versus 3.06 for OLIVE-J (Mix) (Table 4).
Models built mainly for reconstruction do not close this gap on analysis tasks either. We evaluated two neural codecs with semantic distillation, SpeechTokenizer and Mimi, on SUPERB with frozen encoders and the same default configurations:
| Model | PR PER ↓ | ASR WER ↓ | SID Acc ↑ | ASV EER ↓ | ER Acc ↑ | SE PESQ ↑ | SE STOI ↑ | VC MCD ↓ | VC WER ↓ |
|---|---|---|---|---|---|---|---|---|---|
| SpeechTokenizer | 19.7 | 16.0 | 49.6 | 13.1 | 59.7 | 2.78 | 94.2 | 8.37 | 22.2 |
| Mimi | 17.0 | 16.6 | 56.6 | 8.0 | 62.8 | 2.97 | 94.8 | 8.21 | 21.8 |
| data2vec 2.0 Base | 4.6 | 4.8 | 45.5 | 5.5 | 63.9 | 2.96 | 94.9 | 7.28 | 10.4 |
| OLIVE-J (Mask-only) | 4.7 | 4.9 | 64.6 | 5.7 | 66.9 | 2.94 | 94.8 | 7.28 | 11.1 |
| OLIVE-J (Mix) | 6.3 | 6.4 | 83.1 | 5.8 | 64.6 | 3.10 | 95.1 | 7.61 | 11.6 |
Both codecs trail the OLIVE-J variants on the downstream tasks, especially on content tasks.
We hope these clarifications address the points raised, and we will add them to the manuscript.
Thank you for your valuable feedback and comments! We answer the points raised below:
OLIVE's design indeed builds on the observation that early speech SSL layers preserve acoustic details while deeper layers encode more abstract information. The contribution is to turn this knowledge into a training design that prior analysis–synthesis models did not adopt. In fact, data2vec-SG reconstructs from deep contextual layers and discards its decoder, while UniWav conditions a flow-matching decoder on all layers, reconstructs a codec latent, and identifies reduced recognition performance as its main limitation. In the OLIVE design, waveform reconstruction is assigned to the early local features, and masked prediction shapes the contextual features. The design looks simple once stated, but it was not the choice of prior work, and it yields both the best aggregate scores on SUPERB downstream tasks among the compared Base models and the best reconstruction results, from a single-stage pre-training run. To the best of our knowledge, OLIVE is the first speech SSL model to combine single-stage joint training, direct waveform reconstruction, a retained vocoder, and competitive downstream analysis.
As we argue in Section 5, masked latent prediction has no incentive to retain signal-level detail, which is reflected for example in the SID accuracy of data2vec 2.0, the lowest among the baselines (Table 1). The explicit reconstruction objective adds the requirement to maintain more signal-level detail in the local encoder features. This is reflected in the OLIVE-J (Mask-only) results, which improve SID accuracy from 45.5 to 64.6 compared with data2vec 2.0 and reconstruction PESQ from 2.21 to 2.81 (Table 4), while ASR performance is maintained.
While the WaveNet autoencoder [1] and Mockingjay [2] use reconstruction as the representation-learning objective itself, OLIVE keeps masked latent prediction to learn contextual representations that are strong on downstream analysis tasks. This keeps the functional split between early and later layers, which proves useful for different tasks. We will clarify these points.
OLIVE leverages the separation that is typically observed, where deeper layers gradually abstract away signal-level information. It uses this separation to place the objectives: early features are encouraged to keep more signal-level detail, without constraining the deeper layers, which produce the more abstract linguistic representations useful for content tasks for example. Consistent with this, OLIVE-J features give better reconstruction than WavLM and data2vec 2.0 features under matched vocoders (Tables 3 and 4), while the contextual layers keep competitive analysis performance, with no trade-off in recognition in the Mask-only configuration.