acceptodds
Withdrawn from ICLR 2027

OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We propose Online Latent prediction with Invariant Views and rEconstruction (OLIVE), a self-supervised speech representation learning framework that jointly optimizes analysis and synthesis objectives. OLIVE couples masked latent prediction and waveform reconstruction through a functional separation within a single encoder: reconstruction constrains early encoder features to retain signal-level information, while masked latent prediction shapes later contextual representations for robust downstream performance, with waveform view augmentations defining the invariances they learn. To the best of our knowledge, OLIVE is the first speech SSL model that directly reconstructs waveforms, retains its vocoder after pre-training, and remains competitive on discriminative tasks. Adding reconstruction largely preserves the speech recognition performance of masked latent prediction alone, while improving waveform reconstruction and speaker tasks. Adding waveform mixup and gain augmentations provides a controllable way to shift the representations further toward speaker, generation, and reconstruction tasks, while remaining competitive on recognition and semantic tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech, Self-supervised Learning, Analysis-synthesis, View Augmentations, Latent Masked Prediction, Waveform Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6kiXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OLIVE, a speech self-supervised learning framework that jointly optimizes masked latent prediction and waveform reconstruction to support both speech analysis and synthesis. It introduces waveform view augmentations and a functional separation between early acoustic features and deeper contextual representations, and evaluates the framework on SUPERB downstream tasks and waveform reconstruction.

Critical strengths
  • The paper explores the joint optimization of speech analysis and synthesis within a unified self-supervised learning framework. Learning representations that support both discriminative downstream tasks and waveform reconstruction is a practically relevant research direction.

  • The paper evaluates OLIVE across a broad range of SUPERB downstream tasks and waveform reconstruction metrics, with comparisons against several established speech SSL baselines.

Critical weaknesses
  1. Limited methodological novelty.

OLIVE primarily combines existing techniques, including data2vec 2.0-style masked latent prediction, waveform augmentations, and HiFi-GAN-based waveform reconstruction, within a multi-task learning framework. Importantly, jointly optimizing masked latent prediction and waveform reconstruction has already been explored in data2vec-SG (Wang et al., 2023), which similarly aims to learn representations useful for both speech analysis and generation. The main distinction of OLIVE appears to be applying reconstruction supervision to early local encoder features rather than deeper contextual representations, together with independently augmented teacher–student views and a retained vocoder. However, the observation that early speech SSL layers preserve acoustic details while deeper layers encode more abstract information is already well established. Consequently, the proposed method appears largely incremental, and it remains unclear whether these modifications constitute a sufficiently substantial methodological contribution beyond existing analysis–synthesis SSL frameworks.

  1. Unclear motivation and necessity of the proposed design.

(i) The paper does not sufficiently justify why an additional explicit waveform reconstruction objective is necessary for speech SSL pre-training. Existing speech SSL methods already learn rich representations through masked prediction and reconstruction-based objectives. Prior work [1, 2] has also demonstrated that reconstruction-based pre-training can learn meaningful phonetic and linguistic representations.

(ii) Although the paper proposes a functional separation between analysis and synthesis, it does not clearly explain how the proposed training strategy establishes or enforces such a separation beyond applying two existing objectives at different encoder depths. Existing speech SSL models naturally encode different levels of acoustic and linguistic information across their layers. The paper's own results in Appendix C.3 (Table 10) show that WavLM's early features are also more suitable for waveform reconstruction than its deeper representations. Therefore, the observed hierarchical division of labor does not necessarily result from OLIVE's joint training strategy. The paper lacks convincing evidence that the proposed approach produces a more effective functional separation than conventional SSL pre-training.

(iii) The joint variants achieve limited gains on recognition tasks and mixed results on generation-related tasks, while the highest overall SUPERB score comes from an analysis-only variant without reconstruction. These findings leave the necessity of the proposed joint design insufficiently substantiated, particularly given that existing SSL representations can already support waveform reconstruction through separately trained vocoders.

[1] Chorowski et al. (2019). Unsupervised Speech Representation Learning Using WaveNet Autoencoders.

[2] Liu et al. (2020). Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders.

Questions for authors

see weakness above.

2 comments

Thank you for your valuable feedback and comments! We answer the points raised below:

"The main distinction of OLIVE appears to be applying reconstruction supervision to early local encoder features rather than deeper contextual representations, together with independently augmented teacher–student views and a retained vocoder. However, the observation that early speech SSL layers preserve acoustic details while deeper layers encode more abstract information is already well established."

OLIVE's design indeed builds on the observation that early speech SSL layers preserve acoustic details while deeper layers encode more abstract information. The contribution is to turn this knowledge into a training design that prior analysis–synthesis models did not adopt. In fact, data2vec-SG reconstructs from deep contextual layers and discards its decoder, while UniWav conditions a flow-matching decoder on all layers, reconstructs a codec latent, and identifies reduced recognition performance as its main limitation. In the OLIVE design, waveform reconstruction is assigned to the early local features, and masked prediction shapes the contextual features. The design looks simple once stated, but it was not the choice of prior work, and it yields both the best aggregate scores on SUPERB downstream tasks among the compared Base models and the best reconstruction results, from a single-stage pre-training run. To the best of our knowledge, OLIVE is the first speech SSL model to combine single-stage joint training, direct waveform reconstruction, a retained vocoder, and competitive downstream analysis.

"(i) The paper does not sufficiently justify why an additional explicit waveform reconstruction objective is necessary for speech SSL pre-training. [...] Prior work [1, 2] has also demonstrated that reconstruction-based pre-training can learn meaningful phonetic and linguistic representations."

As we argue in Section 5, masked latent prediction has no incentive to retain signal-level detail, which is reflected for example in the SID accuracy of data2vec 2.0, the lowest among the baselines (Table 1). The explicit reconstruction objective adds the requirement to maintain more signal-level detail in the local encoder features. This is reflected in the OLIVE-J (Mask-only) results, which improve SID accuracy from 45.5 to 64.6 compared with data2vec 2.0 and reconstruction PESQ from 2.21 to 2.81 (Table 4), while ASR performance is maintained.

While the WaveNet autoencoder [1] and Mockingjay [2] use reconstruction as the representation-learning objective itself, OLIVE keeps masked latent prediction to learn contextual representations that are strong on downstream analysis tasks. This keeps the functional split between early and later layers, which proves useful for different tasks. We will clarify these points.

"(ii) Although the paper proposes a functional separation between analysis and synthesis, it does not clearly explain how the proposed training strategy establishes or enforces such a separation [...] The paper lacks convincing evidence that the proposed approach produces a more effective functional separation than conventional SSL pre-training."

OLIVE leverages the separation that is typically observed, where deeper layers gradually abstract away signal-level information. It uses this separation to place the objectives: early features are encouraged to keep more signal-level detail, without constraining the deeper layers, which produce the more abstract linguistic representations useful for content tasks for example. Consistent with this, OLIVE-J features give better reconstruction than WavLM and data2vec 2.0 features under matched vocoders (Tables 3 and 4), while the contextual layers keep competitive analysis performance, with no trade-off in recognition in the Mask-only configuration.

"(iii) The joint variants achieve limited gains on recognition tasks and mixed results on generation-related tasks, while the highest overall SUPERB score comes from an analysis-only variant without reconstruction. These findings leave the necessity of the proposed joint design insufficiently substantiated, particularly given that existing SSL representations can already support waveform reconstruction through separately trained vocoders."

The joint design, in its Mask-only configuration, gives more faithful waveform reconstruction while maintaining strong recognition performance. Meanwhile, augmentations control which information the representation encodes, which shifts its usefulness across tasks. On aggregates, OLIVE-A (Mix+Gain) has the highest SUPERB$_s$ (909), and OLIVE-J (Mix) has the highest WavLM$_s$ (67.6) with a SUPERB$_s$ of 908, the best speaker and generation category scores, and the best reconstruction results.

While existing SSL representations can already support waveform reconstruction, our results show that analysis-only representations do not reach the same reconstruction faithfulness: with the same separately trained vocoder setup, PESQ is 2.21 for data2vec 2.0 and WavLM and 2.62 for OLIVE-A (Mix+Gain), versus 3.06 for OLIVE-J (Mix) (Table 4).

Models built mainly for reconstruction do not close this gap on analysis tasks either. We evaluated two neural codecs with semantic distillation, SpeechTokenizer and Mimi, on SUPERB with frozen encoders and the same default configurations:

ModelPR PER ↓ASR WER ↓SID Acc ↑ASV EER ↓ER Acc ↑SE PESQ ↑SE STOI ↑VC MCD ↓VC WER ↓
SpeechTokenizer19.716.049.613.159.72.7894.28.3722.2
Mimi17.016.656.68.062.82.9794.88.2121.8
data2vec 2.0 Base4.64.845.55.563.92.9694.97.2810.4
OLIVE-J (Mask-only)4.74.964.65.766.92.9494.87.2811.1
OLIVE-J (Mix)6.36.483.15.864.63.1095.17.6111.6

Both codecs trail the OLIVE-J variants on the downstream tasks, especially on content tasks.

We hope these clarifications address the points raised, and we will add them to the manuscript.

Reviewer_iq4UOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose an analysis-synthesis based speech SSL modeel, OLIVE, which builds upon data2vec2.0 by attaching a full fledged neural vocoder to the frontend encoder features. Training is done for both the masked prediction for the contextual analysis branch as well as the for the neural vocoder with its full suite of waveforms reconstruction and adversarial losses. This enables the model to retain its vocoder for later speech synthesis tasks, unlike previous works which typically discard the decoder. Experimental evaluation on the SUPERB benchmark show competitive performance.

Critical strengths
  1. Retaining the decoder as a competent vocoder is a plus point of this model.
  2. Strong performance on both synthesis and analysis tasks and analysis of the pareto fronts are useful.
Critical weaknesses
  1. While the authors do say that they do not compare with any of the other analysis-synthesis baselines because public checkpoints are not available, it is a significant drawback. Since atleast data2vec-SG is in the same architectural front, just changing the features from the early encoder layers to late contextual layers and decoder without adversarial losses, I think it should have been possible to reproduce that for a fair comparison. Currently delta from data2vec-base analysis in Appendix B is provided but it is not a fair comparison.

  2. The ablations do not consider which features are the best for reconstruction, simply stating that early encoder features is not enough, the layer analysis in appendix C3, is on a pre-trained OLIVE-A layerwise analysis. Rather it is unclear what happens if lets say Transf. 1 or 2 features are utilized for input to the vocoder during joint training.

  3. The most significant impact in performance comes from the mixup augmentation. Table 6 in Appendix B, shows that OLIVE-J with mask only has essentially no improvement on generative tasks over data2vec. Which suggests that simply having an extra synthesis branch does not lead to significant improvements in generative tasks. The mixup augmentation is the most improtant driver of the performance gains for generative tasks. Infact OLIVE-A with mixup also performs very close to OLIVE-J.

Questions for authors

See weaknesses above.

1 comment

Thank you for your helpful feedback! We answer each point below.

"While the authors do say that they do not compare with any of the other analysis-synthesis baselines because public checkpoints are not available, it is a significant drawback. Since at least data2vec-SG is in the same architectural front [...] I think it should have been possible to reproduce that for a fair comparison."

Reproducing the complete data2vec-SG run while ensuring the training is stable, the comparison is fair, and ablating configurations such as the loss weights, is challenging with our compute budget. Still, our vocoder layer ablation (Appendix C.3, Table 10) clearly shows that reconstruction quality degrades with depth, consistent with prior work (Irvin et al., 2023). Since we are interested in faithful reconstruction, this motivates decoding from the early layers. In addition, data2vec-SG discards its decoder after pre-training, whereas OLIVE retains a vocoder that reconstructs the input waveform directly, and OLIVE's design is built for this reconstruction use.

"The ablations do not consider which features are the best for reconstruction [...] it is unclear what happens if let's say Transf. 1 or 2 features are utilized for input to the vocoder during joint training."

As noted above, we base the choice of local features on our matched vocoder ablations (Appendix C.3, Table 10), and on prior work that conditioned vocoders on SSL features and found local features well suited for reconstruction (Irvin et al., 2023). This choice also offers the most versatility for downstream applications: the same local features used for synthesis can be passed through the Transformer to any layer of choice for downstream analysis.

"The most significant impact in performance comes from the mixup augmentation. Table 6 in Appendix B shows that OLIVE-J with mask only has essentially no improvement on generative tasks over data2vec. [...] simply having an extra synthesis branch does not lead to significant improvements in generative tasks."

The synthesis branch alone gives large gains on synthesis, which the SUPERB generation tasks do not measure. OLIVE-J (Mask-only) improves over data2vec 2.0 on every reconstruction metric in Tables 3 and 4 and on resynthesis speaker similarity (Appendix G). It also raises SID from 45.5 to 64.6 and ER from 63.9 to 66.9 while maintaining ASR performance.

The SUPERB generation tasks do not measure well the additional acoustic detail kept in the representation. For example, the SUPERB VC protocol is any-to-one: features are decoded to sound like one fixed target speaker. This setting therefore favors speaker-disentangled features, and retaining source-speaker detail can penalize performance.

Mixup, on the other hand, directly helps SE and SS, which reward robustness to the corruptions in their test sets. Comparing OLIVE-A (Mix) and OLIVE-J (Mix), the joint model improves on SID (66.9 to 83.1) and VC ASV acceptance (95.5 to 98.2), and its features give better reconstruction than those of OLIVE-A (Mix+Gain) (PESQ 3.06 versus 2.62, Table 4). Mixup and reconstruction therefore act on different abilities, and the gains from joint training are most clearly observed on tasks that directly benefit from the added signal-level detail.

We hope this clarifies the raised concerns, and we will make sure to explain these points clearly in the manuscript.

Reviewer_qsDfOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript introduces OLIVE. The main objective behind OLIVE is to train a self-supervised speech representation learning framework that jointly optimizes analysis and synthesis objectives. The analysis branch is modelled using data2vec 2.0, whilst the synthesis branch uses HiFi-GAN. The authors report improvements over the baselines for both discriminative and reconstruction tasks, as well as they ablate several design choices of OLIVE like the augmentations, the use of the sysnthesis branch, etc.

Critical strengths
  • The paper is well organized and relatively easy to follow.

  • OLIVE is presented in a clear way and is modular.

  • OLIVE is evaluated thoroughly: 14 SUPERB tasks, many reconstruction metrics with confidence intervals, and resynthesis WER and speaker-similarity scores.

  • I also find the paper complete enough in terms of ablations, for example the ablations to show the contribution of each branch are useful.

Critical weaknesses
  • One of the main weaknesses of the paper is that OLIVE brings incremental novelty since it combines data2vec 2.0 with a HiFi-GAN loss on CNN features, plus waveform mixup. Mixup augmentation is already in BYOL-A (Daisuke Niizumi et al., BYOL for audio: Self-supervised learning for general-purpose audio representation), and WavLM (Sanyuan Chen et al., WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing) uses utterance mixing.

  • Another critical weakness is that for the reconstruction results (Table 3), OLIVE is compared only against discriminative SSL models* (HuBERT, WavLM, data2vec), which were never trained to preserve signal-level detail, so OLIVE’s advantage is expected. The relevant comparisons with reconstruction-based models are missing. For example, what about data2vec-SG? Regarding UniWav, I understand it is trained on more data, but I think it's still worth being reported to have a comparison (you can add a column to state the number of pre-trained hours the models are trained on). What about codecs with semantic distillation (e.g., SpeechTokenizer, Mimi), which have public checkpoints? Without these, the claim of being the first SSL model to reconstruct waveforms while staying competitive on discriminative tasks is not supported.

  • All experiments use Base models on LibriSpeech 960h. Prior SSL work (e.g., WavLM) showed substantial gains from both model and data scaling, particularly for speaker and paralinguistic tasks, which are exactly OLIVE’s claimed strengths. Since mixup may compensate for the limited acoustic diversity of LibriSpeech, and reconstruction acts only on the CNN extractor, whose size is unchanged at Large scale, it is unclear whether OLIVE’s gains persist with more diverse data or larger models. A Base model trained on larger, more diverse data (e.g., Libri-Light), or a Large variant, would substantially strengthen the claims.

  • OLIVE combines a single analysis objective (data2vec 2.0) with a single synthesis objective (HiFi-GAN). However, the paper does not motivate thoroughly the choice of data2vec 2.0 (the only implicit rationale is its efficiency mentioned in the Related Work) and HiFi-GAN (the only support is precedent from prior work that trains HiFi-GAN on SSL features) over alternatives. First, can the authors motivate better why they chose to adopt these two architectures? Second, have the authors tested whether the findings hold for other choices on either branch? For analysis, this could be HuBERT- or BEST-RQ-style masked prediction. For synthesis, it could be a non-adversarial mel or STFT reconstruction loss. The modular design of OLIVE makes this natural to test, and the Small-scale setup used for the ablations would be enough. data2vec retains the least speaker information among the baselines (lowest SID in Table 1), which may leave unusually large room for reconstruction to help. Showing that recognition is preserved while speaker and generation abilities are added across objectives would considerably strengthen the paper’s general claim about joint analysis–synthesis pre-training.

  • I think the “Remains competitive” claim throughtout the paper is overstated for the Mix variants as OLIVE-J (Mix) test-other WER is 15.7 vs 11.1 for data2vec 2.0, which is worse than HuBERT and close to wav2vec 2.0. QbE also degrades for every variant.

  • Regarding the training cost, joint training (analysis + synthesis) increases pre-training time by about 67% (206 vs 123 GPU-hours per run, Appendix J), largely because of the 70.7M-parameter discriminators. Yet the retained vocoder is outperformed by a separately trained one on the same features (Table 3). I think the authors should report the cost of separate vocoder training, so that joint training can be compared against analysis-only pre-training plus a separate vocoder.

  • I believe the most important advantage of OLIVE is that it can support both reconstruction and discriminative tasks after pre-training. However, the practical benefit of a single jointly trained model over a two-stage pipeline (an SSL encoder plus a separately trained vocoder) is not clearly articulated. The best reconstruction is itself obtained with a separately trained vocoder on frozen OLIVE-J features (Table 3), and joint training costs about 67% more compute. I also feel like this advantage is not enough emphasized in the paper.

  • SOme Section 5 claims are expected by construction. Regarding the “The two components are complementary” section, this is partly contradicted by the paper’s own numbers. If we compare OLIVE-J (Mix) with OLIVE-A (Mix), i.e. adding reconstruction on top of mixup we can find that: 1) on ASR fine-tuning results on LibriSpeech OLIVE-A (Mix) is always worse (or equal in one case) than OLIVE-J (Mix) and is unchanged on SUPERB ASR (6.4 vs 6.4); 2) Emotion recognition drops (67.1 → 64.6), and speech translation BLEU drops (16.5 → 15.4). I therefore think that in multiple benchmarks the reconstruction branch does not help at all.

Questions for authors

See weaknesses as well.

  • The paper motivates reconstruction through the analysis–synthesis paradigm but gives no mechanism for why it should benefit discriminative tasks beyond speaker information; content and semantic tasks are unchanged or degraded (e.g., Table 12 or some SUPERB tasks). Can the authors explain in more detail why the additional use of a reconstruction objective should benefit the other one as well? I couldn't find convincing explanatons/motivations in the paper in favor of this extremely important claim (beyond simply reporting the results).
3 comments

Thank you for your detailed and constructive review! We respond to each point below:

"OLIVE brings incremental novelty since it combines data2vec 2.0 with a HiFi-GAN loss on CNN features, plus waveform mixup. Mixup augmentation is already in BYOL-A [...] and WavLM [...] uses utterance mixing."

The contribution is the hierarchical design that makes analysis and synthesis work together in a single training stage. OLIVE assigns waveform reconstruction to early local features and view-augmented masked prediction to later contextual features. This separation preserves the signal detail needed for synthesis without constraining the contextual representation. To the best of our knowledge, OLIVE is the first speech SSL model to combine single-stage joint training, direct waveform reconstruction, a retained vocoder, and competitive downstream analysis. Prior analysis–synthesis methods do not provide this combination. Indeed, data2vec-SG reconstructs from deep contextual layers and discards its decoder, and UniWav reconstructs an intermediate codec latent rather than the waveform and reports weaker recognition performance as its main limitation.

Regarding mixup, in this work we motivate its use within the latent prediction framework and study its effects on the representations. The analysis framework we adopt follows the joint-embedding predictive paradigm, while BYOL follows a joint-embedding paradigm, so it is still worth studying augmentations in this context. Further, the use of augmentations holds an important distinction from WavLM: WavLM mixes an interfering utterance into the input and predicts the pseudo-labels of the primary utterance, which corresponds to a form of denoising. OLIVE, on the other hand, samples independent augmented views for the student and the teacher, so the augmentation family defines the invariances of the contextual targets and gives control over these invariances. The use of established components is also a strength, ensuring a modular framework.

"For the reconstruction results (Table 3), OLIVE is compared only against discriminative SSL models, which were never trained to preserve signal-level detail, so OLIVE's advantage is expected."

Two results in Table 3 are not expected by construction. First, the analysis-only OLIVE-A (Mix+Gain), which has no reconstruction objective, also improves reconstruction over all SSL baselines on most metrics. Second, the joint models reach the best reconstruction while the OLIVE variants hold the best SUPERB aggregates (SUPERB$_s$ 909 and WavLM$_s$ 67.6), and OLIVE-J (Mask-only) does not trade off recognition performance. To the best of our knowledge, no prior SSL model reports both.

"The relevant comparisons with reconstruction-based models are missing. For example, what about data2vec-SG? Regarding UniWav [...] I think it's still worth being reported [...] (you can add a column to state the number of pre-trained hours)."

data2vec-SG and UniWav checkpoints are not open-source, which complicates this comparison, and full reproductions of these models, with proper ablations of their configurations and a fair comparison, are challenging with the compute we have available. The table below compares the models using the results reported in their papers:

ModelParamsPre-training dataReconstruction targetDecoder conditioningDecoder keptPR PER ↓SID Acc ↑SE PESQ ↑SS SI-SDRi ↑
data2vec-SG95M + 23M decoder960 hclean waveformdeep contextual layersno––2.5910.8
UniWav~500M (encoder ~300M)60k h (Libri-Light)codec latentall layers (weighted sum)yes3.793.8––
OLIVE-J (Mask-only)93.2M960 hinput waveformearly local featuresyes (vocoder)4.764.62.9410.5

data2vec-SG reports only generation tasks, and UniWav reports only PR and SID on SUPERB (with a learning-rate sweep); our results use the default S3PRL configurations without a sweep. data2vec-SG's evaluation uses different FBANK and data2vec baselines from ours, so Appendix B compares the gains of each model over the data2vec Base baseline evaluated in the same setup. UniWav uses about 60 times more pre-training data and a larger model.

"What about codecs with semantic distillation (e.g., SpeechTokenizer, Mimi), which have public checkpoints? Without these, the claim of being the first SSL model to reconstruct waveforms while staying competitive on discriminative tasks is not supported."

We evaluated both codecs on the SUPERB tasks with frozen encoders and the same default configurations. For SpeechTokenizer, we use the continuous encoder output, the output of the first RVQ layer (distilled from HuBERT), and the sum of all RVQ layer outputs. For Mimi, we use the encoder-Transformer states at 25 Hz. For SE, which requires a 10 or 20 ms frame rate, we upsample the Mimi states to 50 Hz using nearest-neighbor interpolation:

ModelPR PER ↓ASR WER ↓SID Acc ↑ASV EER ↓ER Acc ↑SE PESQ ↑SE STOI ↑VC MCD ↓VC WER ↓
SpeechTokenizer19.716.049.613.159.72.7894.28.3722.2
Mimi17.016.656.68.062.82.9794.88.2121.8
data2vec 2.0 Base4.64.845.55.563.92.9694.97.2810.4
OLIVE-J (Mask-only)4.74.964.65.766.92.9494.87.2811.1
OLIVE-J (Mix)6.36.483.15.864.63.1095.17.6111.6

Codecs are designed and trained for reconstruction and compression, not as general representation models. On SUPERB they trail both OLIVE-J variants on every task in the table, especially on content tasks. These results support the distinction between codecs and OLIVE: semantic distillation into a codec does not yield representations that compete with SSL models on discriminative tasks.

"It is unclear whether OLIVE's gains persist with more diverse data or larger models. A Base model trained on larger, more diverse data (e.g., Libri-Light), or a Large variant, would substantially strengthen the claims."

Training a Large model is not feasible with our compute. We started an ablation with Base models trained on more data: we train Base OLIVE-J (Mix) and data2vec 2.0 on LibriSpeech 960h plus 27k hours of Multilingual LibriSpeech (MLS) English (Pratap et al., 2020), which, like Libri-Light, is drawn from LibriVox. Within the discussion period, both models reached 50k updates. We perform a preliminary comparison below:

Model (50k updates, LS960 + MLS)PR PER ↓SID Acc ↑ER Acc ↑SE PESQ ↑SE STOI ↑VC MCD ↓VC WER ↓
data2vec 2.0 Base5.658.464.62.8594.77.2311.8
OLIVE-J (Mix) Base7.662.064.82.9494.87.6213.1

We will complete this ablation and add the results and corresponding interpretations to the manuscript.

We also compare Small (d=384d=384) and Base models, both trained for 400k updates on LibriSpeech 960h, to observe how the effect of view augmentation scales:

Model (400k updates, LS960)PR PER ↓SID Acc ↑ER Acc ↑SE PESQ ↑
data2vec 2.0 Small6.0262.564.12.92
OLIVE-A (Mix) Small6.7563.964.63.04
data2vec 2.0 Base4.6145.563.92.96
OLIVE-A (Mix) Base6.3666.967.13.05

For both Small and Base models, we observe the same pattern when adding mixup view augmentations: PR degrades, while SID, ER, and SE PESQ improve, with larger SID and ER gains at Base scale.

"The paper does not motivate thoroughly the choice of data2vec 2.0 [...] and HiFi-GAN [...]. First, can the authors motivate better why they chose to adopt these two architectures?"

We chose data2vec 2.0 for three reasons: its prediction of contextual targets in the latent space, which is consistent with the joint-embedding predictive (JEPA) paradigm; its teacher–student setup, which enables view augmentations and does not depend on an external target quantizer or clustering step; and its strong results and training efficiency among Base-size speech SSL models. We chose HiFi-GAN V2 because it is an efficient and widely used neural vocoder, and prior work has trained it successfully on SSL features. Further, its generator is small (1.1M parameters), so the retained vocoder adds little to the model. We will clarify this motivation in the revision.

"Second, have the authors tested whether the findings hold for other choices on either branch? For analysis, this could be HuBERT- or BEST-RQ-style masked prediction. For synthesis, it could be a non-adversarial mel or STFT reconstruction loss. [...] Showing that recognition is preserved while speaker and generation abilities are added across objectives would considerably strengthen the paper's general claim."

We ran this test at Small scale (d=384d=384, LibriSpeech 960h, no waveform augmentation, so only the synthesis branch differs within each pair). Reconstruction uses separately trained HiFi-GAN V2 vocoders on frozen local features (100k steps, test-clean, loudness-matched):

Model (Small)PESQ ↑MCD ↓STOI ↑UTMOS ↑
data2vec 2.01.615.160.8663.16
+ HiFi-GAN synthesis (OLIVE-J Mask-only)2.624.740.9183.74
+ mel/STFT synthesis, no discriminators1.934.850.8853.25
HuBERT (iteration 1, 50k updates)1.355.380.8522.92
+ HiFi-GAN synthesis (50k updates)2.474.840.9163.66

data2vec 2.0 models are compared at 100k updates, while the HuBERT pair is compared at 50k updates. The number of updates was limited by the duration of the discussion period.

The reconstruction gains hold for both analysis objectives. The non-adversarial loss also improves reconstruction from frozen features, but less than the HiFi-GAN loss. Its integrated generator produces lower-quality audio (UTMOS 1.54 versus 3.59 for the HiFi-GAN generator of OLIVE-J Mask-only Small), which supports keeping an adversarial vocoder when the decoder is retained.

"Regarding the training cost [...] I think the authors should report the cost of separate vocoder training, so that joint training can be compared against analysis-only pre-training plus a separate vocoder."

Training a separate HiFi-GAN V2 vocoder for 400k steps takes about 150 hours on one H100 MIG 3g.40gb slice after pre-training, while joint training adds 83 hours on two H100 GPUs (206 versus 123 hours). The costs of the two pipelines are therefore of the same order, but the two pipelines do not yield the same features, with joint training giving features that maintain more signal-level information. We will add more training cost details to Appendix J.

"The practical benefit of a single jointly trained model over a two-stage pipeline (an SSL encoder plus a separately trained vocoder) is not clearly articulated."

Joint training provides two benefits that a two-stage pipeline does not. First, it acts on the encoder: the reconstruction objective makes the local features retain signal detail, which is observed, for instance, in the more faithful reconstruction from OLIVE-J features compared with every analysis-only encoder we evaluated. Second, a single-stage pre-training run yields both the encoder and a usable vocoder, with no further training on the target features. We will clarify the practical benefits in the revision.

"Regarding the 'The two components are complementary' section, this is partly contradicted by the paper's own numbers. [...] in multiple benchmarks the reconstruction branch does not help at all."

We will revise this section to state more clearly that reconstruction on top of mixup improves speaker identification, several generation tasks, and waveform reconstruction, but does not necessarily improve every task.

"The paper motivates reconstruction through the analysis–synthesis paradigm but gives no mechanism for why it should benefit discriminative tasks beyond speaker information [...]. Can the authors explain in more detail why the additional use of a reconstruction objective should benefit the other one as well?"

Reconstruction encourages the encoder to maintain more signal-level detail at the early layers. Masked prediction rewards information that predicts the contextual targets and discards the rest, including signal-level detail such as voice quality and fine spectral structure. The reconstruction objective requires the local features to retain this detail, which benefits tasks that depend on it, such as speaker identification and enhancement.

Thank you again for the concrete suggestions. We will clarify these points in the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.