acceptodds
Under review as a conference paper at ICLR 2027

ReVoC: High-Fidelity and Efficient Audio Generation via Restoration Bridges and GANs

Abstract

Although reconstructing high-fidelity audio from compressed representations is an underdetermined inverse problem, the corresponding inverse transforms or decoders can recover useful acoustic structure at low computational cost. In this paper, we propose ReVoC, a few-step generative framework that combines restoration bridges with adversarial training to refine these approximate reconstructions. Starting from the outputs of Mel inversion or neural codec decoding, ReVoC learns to restore clean audio in the complex STFT domain using an efficient Vocos-style 1D backbone. To improve perceptual quality across inference budgets, we introduce multi-NFE adversarial fine-tuning, where a step-conditioned discriminator guides the joint optimization of complete one-, two-, and four-step inference trajectories. A single generator per task thus supports all three budgets with quality comparable to models fine-tuned separately for each step count. On Mel-conditioned vocoding, ReVoC-Small uses the same eight-block backbone as Vocos and improves one-step PESQ from 3.592 to 4.142. With four-step inference, ReVoC-Base achieves a PESQ of 4.433 and approximately four times the GPU throughput of four-step BridgeVoC. ReVoC also consistently improves reconstruction quality in low-bitrate neural codec restoration.

Primary area:
generative models
Keywords:
Neural vocoding, audio restoration, generative bridges, adversarial fine-tuning, few-step audio generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GB7BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed ReVoc, a high-fidelity and efficient audio generation system via bridges and GANs. The authors evaluated the performance on text to speech synthesis and shows improvement on PESQ and other scores.

Critical strengths
  • The presentation of the paper is clear.

  • The metrics from the tables indciate the effectiveness of the method.

Critical weaknesses
  • My major concern of this work is that the innovation is limited. Authors proposed bridge and GAN, but they are some marginal improvement of existing methods.

  • Bridge may not be a key point that lead to a big significance to solve the audio generation problem.

  • The proposed methodolgy part is a bit weak. Bridge methods have been research in several speech generation, and enhancement works. In addition, for a text to speech problem, I am unsure whether bridge model will outperform diffusion/flow matching-based models.

  • 4.1 proposed a mel vocoding and restoration model. This is a standard configuration of many speech audio generation works.

  • The work is about text-to-speech. Therefore, the title should be revised from audio generation to text-to-speech. Otherwise, authors should do more experiment on general audio and music generation as well.

Questions for authors
  • See critical weaknesses.
3 comments
AuthorsOpenReview ↗

Response Regarding the Scope of ReVoC and the Paper Title (Part 1)

We thank the reviewer for raising the question of scope and suggesting evaluation beyond speech. ReVoC operates at the waveform level and addresses two tasks. In mel vocoding, it synthesizes a waveform from a mel spectrogram. In codec restoration, it refines the waveform produced by a native codec decoder such as EnCodec or Mimi. In mel-based TTS, the upstream text-conditioned model predicts mel spectrograms, and ReVoC serves as the vocoder. In codec-based TTS, the upstream model predicts codec tokens, and ReVoC can refine the waveform after codec decoding. The term audio generation in the title refers to conditional waveform generation and restoration across these two tasks.

1.Zero-shot on general audio mel vocoding.

As detailed in our response to Reviewer Wzfz, we evaluated the speech-trained ReVoC-Base checkpoint in the official MUSDB18 test split. The evaluation covers mixtures and four stems. The table reports the average scores across the five categories, with PESQ-WB included as a supplementary metric.

ModelAudio-mode ViSQOL ↑PESQ-WB ↑
Vocos4.6092.110
BigVGAN-Base4.5832.349
ReVoC-Base (NFE = 1)4.6133.045
ReVoC-Base (NFE = 2)4.6253.219
ReVoC-Base (NFE = 4)4.6273.436

2.ReVoC as the waveform backend in TTS.

In our response to Reviewer BAqX, we evaluated ReVoC and Vocos on the same mel spectrograms predicted by F5-TTS Base and E2-TTS Base for 1,127 LibriSpeech-PC test-clean utterances. For each system, the acoustic model and its predicted mel spectrograms were held fixed. The table shows the results for Vocos → ReVoC-Base (NFE = 4).

Acoustic modelWER ↓SIM-o ↑UTMOS ↑MOS ↑
F5-TTS Base2.24 → 2.220.6546 → 0.66703.898 → 4.1644.175 → 4.250
E2-TTS Base2.49 → 2.480.7094 → 0.72353.572 → 3.9503.970 → 4.210

The TTS results show how ReVoC performs as a waveform backend with predicted acoustic inputs. The MUSDB18 results extend the evidence to zero-shot music waveform reconstruction at 24 kHz.

AuthorsOpenReview ↗

Response Regarding Methodological Contributions (Part 2)

We thank the reviewer for the valuable feedback. We would like to further clarify the key methodological contributions of ReVoC.

Building on prior research on bridges and adversarial learning, ReVoC investigates how to achieve high-quality waveform reconstruction with an efficient spectral backbone and a shared model across inference budgets. Its contributions concern the design of the spectral predictor, joint multi-NFE adversarial fine-tuning, and a restoration formulation applicable to both mel vocoding and codec restoration. We describe the evidence for these contributions below.

1. Spectral predictor design and reconstruction before adversarial fine-tuning

ReVoC uses a restoration bridge initialized from an acoustic estimate derived from the input. This formulation allows the model to refine the available acoustic structure through successive updates. We implement the predictor using a Vocos-style backbone, with a patch encoder that explicitly models local time–frequency structure before temporal processing.

The patch encoder makes a measurable contribution under the same restoration training setup. Our direct-flattening reference, labeled Vocos(Flow), uses the same temporal projection and backbone. After 100k Stage 1 updates, the two configurations obtain the following results using four-step EMA inference on the same 100 validation utterances.

Stage 1 predictorPESQ ↑M-STFT ↓UTMOS ↑
Vocos(Flow)3.91190.91413.4076
ReVoC-Small with patch encoder4.23940.81823.5560

The patch encoder improves PESQ by 0.3275, reduces M-STFT by 0.0959, and increases UTMOS by 0.1484. These improvements occur before GAN fine-tuning, providing direct evidence for the contribution of the spectral predictor design within the restoration framework.

2. Joint adversarial fine-tuning across inference budgets

A further methodological distinction is that ReVoC jointly fine-tunes one model for NFE = 1, 2, and 4. ReVoC makes these inference budgets available through the same checkpoint after adversarial fine-tuning.

Table 4 evaluates ReVoC-Base before and after joint fine-tuning on the same LibriTTS subset, with Stage 2 initialized from the Stage 1 checkpoint shown below.

NFEPESQ before → after ↑UTMOS before → after ↑
14.115 → 4.3573.624 → 3.718
24.375 → 4.4093.753 → 3.799
44.403 → 4.4333.783 → 3.832

Joint fine-tuning improves both metrics at every evaluated NFE, with the largest gains at NFE = 1. The resulting checkpoint supports a selectable quality–computation trade-off without retraining or switching model weights. This is the practical purpose of our multi-NFE training strategy.

(continued below)

AuthorsOpenReview ↗

Response Regarding Methodological Contributions (Part 2) - Continued

3. Quality and efficiency relative to diffusion and flow vocoders

To address the question about diffusion and flow matching, we extract the relevant comparisons from Tables 1 and 11. These include the diffusion vocoder FreGrad and the flow-based vocoders PeriodWave and RFWave.

DatasetModelNFEPESQ ↑M-STFT ↓V/UV F1 ↑UTMOS ↑ViSQOL ↑
LibriTTSRFWave104.2490.9610.9673.3834.776
ReVoC-Small24.3010.8080.9753.6464.921
ReVoC-Small44.3420.7940.9773.7204.929
LJSpeechFreGrad503.7751.3440.9433.9344.574
PeriodWave164.2351.0980.9674.3744.722
ReVoC-Base14.2900.8970.9784.1974.949
ReVoC-Base44.3670.8760.9814.3674.955

On LibriTTS, ReVoC-Small at NFE = 2 exceeds RFWave at NFE = 10 on all five metrics shown. On LJSpeech, four-step ReVoC-Base achieves higher PESQ, V/UV F1, and ViSQOL and lower M-STFT than the 16-step PeriodWave results, with a slightly lower UTMOS of 4.367 versus 4.374.

Computational efficiency is central to the design. We use a Vocos-style spectral backbone to investigate how much reconstruction quality can be obtained with inexpensive network evaluations. The following results are extracted from Table 9.

Measurements use FP32, batch size 16, and nominal one-second inputs on a single RTX 4090 GPU and eight pinned threads of an Intel Xeon Gold 6346 CPU. GMACs count neural operations per item over the complete rollout. CPU and GPU ×RT denote throughput relative to real time.

ModelNFEParams (M)GMACsCPU ×RT ↑GPU ×RT ↑
BigVGAN-base114.0232.951.3101.4
BigVGAN1112.4289.980.972.6
Vocos113.531.27233.58036.8
RFWave1018.1425.271.394.3
ReVoC-Small119.311.7883.34776.2
219.313.5557.32884.8
419.317.1035.81600.0
ReVoC-Base152.954.8852.62563.9
252.959.7531.51411.6
452.9519.5017.7738.3

ReVoC-Small and RFWave have similar parameter counts, at 19.31M and 18.14M, respectively. At NFE = 2, ReVoC-Small achieves the quality improvements shown above using 3.55 GMACs, approximately 14% of RFWave’s 25.27 GMACs. Its measured GPU throughput is approximately 30.6 times higher under this benchmark configuration.

Relative to Vocos, ReVoC introduces additional computation for iterative restoration. This computation provides measurable reconstruction gains and allows users to select the inference budget using a single checkpoint. The Small-model results demonstrate the value of the approach at a parameter scale close to existing efficient vocoders.

4. A shared restoration formulation for mel vocoding and codec restoration

Mel vocoding and codec restoration are established tasks, and Section 4.1 uses them to evaluate the restoration framework under two different types of acoustic degradation. Mel vocoding starts from an estimate derived from the mel spectrogram, while codec restoration starts from the native decoder output. Both are represented as complex STFT states and processed through the same restoration formulation.

This shared representation also supports joint training across tasks. In the supplementary experiment described in our response to Reviewer Wzfz, we continued Stage 1 training with a 1:1 mixture of the two tasks, followed by joint multi-task, multi-NFE GAN fine-tuning. The resulting unified ReVoC-Small checkpoint retained mel-vocoding performance close to the single-task model.

This experiment demonstrates that the formulation can support a single model across both tasks, while the main multi-NFE results establish that one checkpoint can retain and improve quality across inference budgets. We will make these methodological distinctions and their supporting ablations more explicit in the revision.

Reviewer_WzfzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The proposed method learns a restoration operation from degraded to clean audio, with a focus on cleaning up audio produced from common mel vocoder or neural audio codec pipelines. The method shows improvements in PESQ and mild improvement for pitch accuracy, while other measures such as ViSQOL are saturated across methods. The method shows some sensitivity to number of function evaluations.

For mel vocoding, the model first initializes with random phase, and then uses a Vocos backbone to iteratively refine the phase, with adversarial losses on each pass. This produces a clean restoration for speech. When I looked at the spectrograms on the attached page, I saw they were faithful to the input, but overshoot slightly in the 22-24khz range (e.g. the model does a little bit of unnecessary upsampling).

Critical strengths
  • Efficient method and training approach, enabling multi-step refinement for vocoders.
  • Matched model backbone to VoCos, with improvements coming from iterative refinement, and adversarial loss.
Critical weaknesses
  • No fullband (44khz or 48khz) results, and no non-speech results.
  • Codec restoration baseline is weak (DAC or Mimi would be a stronger baseline). EnCodec already has quite a few artifacts.
  • In Table 6, Vocos closes most of the gap to ReVoc, when given the EnCodec outputs.
  • Listening test in the appendix is small, but mild preference towards ReVoC (but only in high NFE).
  • In the codec restoration case, the method is expensive as it requires a codec decoding, followed by refinement of the output waveform.
  • Vocos is not retrained on the author's own dataset. It seems to be the released checkpoint of Vocos, instead of an in-house retrained version. Retraining Vocos in a like-for-like manner would clear up the exact source of improvements for ReVoC.
  • Sometimes hard to hear the differences between methods in the linked page. Some harder examples would be good to listen to, with more sibilance and fricatives to tell apart the difference.
  • Title says "audio" generation, but the method is only applied to speech.
Questions for authors
  • Requires separate checkpoints for both mel vocoding and codec restoration - could these be unified via some other approach to phase perturbation?
  • Could you retrain Vocos in the same setup, to clarify where the improvement comes from?
  • Could you add comparisons to other codecs, such as DAC and Mimi?
5 comments
AuthorsOpenReview ↗

Response Regarding a Unified Model for Mel Vocoding and Codec Restoration (Part 1)

We thank the reviewer for raising the possibility of a unified model. In mel vocoding, the input Mel spectrogram provides an informative magnitude estimate, and phase is recovered from a random initialization. In codec restoration, the decoded waveform provides an initial estimate of both magnitude and phase, and the model refines both to correct codec artifacts. We represent both initializations in the same complex STFT space, allowing a shared restoration bridge to refine either input toward clean speech.

The main experiments use task-specific checkpoints to let each model specialize within its available capacity. To evaluate a unified model, we resumed Stage 1 from a ReVoC-Small checkpoint trained for 500,000 steps and continued restoration-bridge training on a 1:1 mixture of mel-vocoding and codec-restoration examples. We then applied joint GAN fine-tuning across both tasks and multiple NFE settings. The same resulting checkpoint was evaluated on both tasks at NFE = 1, 2, and 4.

For ReVoC-Small, entries show single-task → unified at the same NFE. Vocos serves as a single-task reference.

TaskModelNFEPESQ ↑UTMOS ↑
Mel vocodingVocos13.5923.550
ReVoC-Small14.142 → 4.1073.460 → 3.459
24.301 → 4.2873.646 → 3.627
44.342 → 4.3253.720 → 3.710
Codec restoration (3 kbps)Vocos12.5753.459
ReVoC-Small12.704 → 2.5723.597 → 3.223
22.710 → 2.5563.622 → 3.236
42.721 → 2.5703.648 → 3.272
Codec restoration (6 kbps)Vocos13.1183.612
ReVoC-Small13.242 → 3.1643.814 → 3.641
23.238 → 3.1413.816 → 3.616
43.244 → 3.1433.829 → 3.638

For mel vocoding, the unified model stayed within 0.035 PESQ and 0.019 UTMOS of the single-task model across all evaluated NFE settings. For codec restoration at 6 kbps, the unified model remained competitive with single-task Vocos. At NFE = 1, it achieved 3.164 PESQ and 3.641 UTMOS, compared with 3.118 and 3.612 for Vocos. At 3 kbps, the unified model remained close to Vocos in PESQ and scored lower in UTMOS. The gap to the codec-specific ReVoC-Small was also larger at 3 kbps.

To examine whether greater capacity could improve codec restoration, we compared unified L8/C512 and L12/C768 models after 50,000 steps of multi-task Stage 1 training. Entries show L8/C512 → L12/C768.

Codec bitratePESQ ↑UTMOS ↑
3 kbps2.363 → 2.4602.972 → 3.241
6 kbps2.990 → 3.0633.467 → 3.649

The gains at both bitrates in this Stage 1 comparison suggest that increasing model capacity may improve codec restoration in the unified setting.

AuthorsOpenReview ↗

Response Regarding the Retrained Vocos Baseline (Part 2)

We thank the reviewer for suggesting this comparison. In the original submission, we used the released Vocos checkpoint for consistency with prior work. Both the released Vocos model and ReVoC were trained on LibriTTS. We retrained Vocos from scratch using its adversarial training objective, matching ReVoC-Small's training file list, segment length (65,280 samples), and global batch size (64).

We added the retrained Vocos curve to Figure 3. The figure compares the first 100k Vocos generator updates with the first 100k Stage 1 updates of ReVoC-Small with and without the patch encoder. Vocos used a single forward pass, and both ReVoC-Small variants used four-step EMA inference.

Each cell reports PESQ ↑ / M-STFT ↓.

UpdatesVocos (retrained)ReVoC-Small (no patch)ReVoC-Small (with patch)
20k3.0991 / 0.99233.2146 / 1.03243.9735 / 0.8855
40k3.1515 / 0.96383.5876 / 0.97834.1392 / 0.8453
60k3.3093 / 0.93553.7391 / 0.94664.1837 / 0.8322
80k3.3542 / 0.92973.8396 / 0.92714.2157 / 0.8249
100k3.3696 / 0.93323.9119 / 0.91414.2394 / 0.8182

Restoration-bridge training. At 100k updates, the Stage 1 ReVoC-Small variant without the patch encoder achieved higher PESQ (+0.5423) and lower M-STFT (−0.0191) than retrained Vocos, with comparable UTMOS (3.4076 vs. 3.4063). Under the inference settings in Figure 3, the fidelity gains are already visible during Stage 1, when the restoration bridge learns to recover the reference signal from degraded states.

Patch encoder. The two Stage 1 variants use the same restoration objective and four-step EMA inference. Replacing the direct frequency encoder with the patch encoder lets the model jointly encode neighboring time-frequency bins. At 100k updates, this change increased PESQ by 0.3275, reduced M-STFT by 0.0959, and increased UTMOS from 3.4076 to 3.5560.

Joint adversarial fine-tuning. We evaluated the Stage 1 ReVoC-Small checkpoint used to initialize Stage 2 and the resulting GAN-fine-tuned checkpoint on the LibriTTS test utterances. Each cell reports Stage 1 → Stage 2.

NFEPESQ ↑UTMOS ↑
13.760 → 4.1423.342 → 3.460
24.271 → 4.3013.558 → 3.646
44.329 → 4.3423.624 → 3.720

PESQ and UTMOS improved at each fixed NFE, with the largest gains at NFE = 1 (+0.382 PESQ and +0.118 UTMOS). Table 4 shows the same trend for ReVoC-Base.

Iterative refinement. In the final mel-vocoding evaluation at NFE = 1, GAN-fine-tuned ReVoC-Small achieved higher PESQ (4.142 vs. 3.592) and V/UV F1 (0.971 vs. 0.952), as well as lower M-STFT (0.843 vs. 0.870) and periodicity error (0.078 vs. 0.118), than released Vocos. Its UTMOS was slightly lower (3.460 vs. 3.550). At NFE = 1, ReVoC makes a single update from the mel-conditioned, random-phase initialization. With multiple steps, subsequent evaluations refine the intermediate STFT states produced by preceding bridge updates. Using the same ReVoC-Small checkpoint at NFE = 2 or 4 improves both PESQ and UTMOS, with UTMOS exceeding that of released Vocos at both settings.

The comparisons show fidelity gains in the Stage 1 restoration pipeline and a further benefit from the patch encoder. Joint GAN fine-tuning improves PESQ and UTMOS at fixed NFE, and additional refinement steps provide further gains with fixed model weights.

AuthorsOpenReview ↗

Response Regarding Codec Restoration and Decoding Cost (Part 3)

We thank the reviewer for suggesting additional codec comparisons and for raising concerns about the computational cost of waveform refinement after decoding.

1. EnCodec restoration and RVQ conditioning

The gains over Vocos in Table 6 are more modest for codec restoration. The native EnCodec reconstruction already provides usable estimates of magnitude and phase, so ReVoC mainly corrects residual compression artifacts. In mel vocoding, the bridge must recover phase from a random initialization, leaving more room for refinement.

Vocos maps the original EnCodec RVQ codes to their corresponding embeddings and synthesizes the waveform from these features. It uses four codebooks at 3 kbps and eight at 6 kbps. In our supplementary experiment, we used the corresponding RVQ embedding features as an additional condition for ReVoC-Small and initialized the restoration bridge with the native EnCodec-decoded waveform. All ReVoC results below use NFE = 1.

BitrateModelPESQ ↑M-STFT ↓V/UV F1 ↑Per. ↓UTMOS ↑ViSQOL ↑
3 kbpsVocos2.5751.1770.9400.1373.4594.219
ReVoC-Small2.7041.1350.9430.1323.5974.227
ReVoC-Small + RVQ2.7531.1250.9440.1323.6134.254
6 kbpsVocos3.1181.0880.9570.1093.6124.444
ReVoC-Small3.2421.0520.9600.1023.8144.423
ReVoC-Small + RVQ3.3071.0400.9600.1013.8044.458

Adding RVQ conditioning increased PESQ by 0.049 and 0.065 at 3 and 6 kbps, respectively, and reduced M-STFT at both rates. The corresponding UTMOS changes were +0.016 and −0.010.

2. Mimi codec restoration

We added a Mimi evaluation on LibriTTS test utterances. We used the standard 8-codebook setting at 1.1 kbps, with one semantic and seven acoustic codebooks. The native 32-codebook decoder at 4.4 kbps provides a higher-bitrate reference.

ModelCodebookskbpsM-STFT ↓PESQ ↑Per. ↓V/UV F1 ↑UTMOS ↑
Mimi324.41.0973.3660.1180.9533.754
Mimi81.11.3442.2370.1880.9113.472
Mimi + ReVoC-Small (NFE = 1)81.11.1722.3360.1590.9283.749
StreamFlow (NFE = 2, 175M)81.11.2172.3060.1620.9153.910

At 1.1 kbps, a single ReVoC-Small refinement step increased PESQ by 0.099 and reduced M-STFT by 0.172 relative to native Mimi decoding. It also raised UTMOS by 0.277 to 3.749, close to the 3.754 achieved by native Mimi at 4.4 kbps.

3. Decoding and restoration latency

We measured mean latency on an RTX 4090 (FP32, batch size 1) using 1000 one-second clips at 24 kHz. Starting from GPU-resident codes, we timed eager codec decoding followed by one ReVoC refinement step, run either in eager mode or CUDA Graph.

CodecCodec decoding (ms)ReVoC modeAdded latency (ms)Total / codecTotal RTF ↓
EnCodec 3 kbps3.44Eager2.571.75×0.00602
CUDA Graph0.721.21×0.00416
EnCodec 6 kbps3.57Eager2.571.72×0.00614
CUDA Graph0.721.20×0.00429
Mimi 1.1 kbps6.54Eager3.161.48×0.00970
CUDA Graph0.921.14×0.00745

CUDA Graph replay reduced the added latency of one-step ReVoC restoration from 2.57–3.16 ms in eager mode to 0.72–0.92 ms per one-second clip.

Reviewer_BAqXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents ReVoc, a few-step audio restoration framework that starts from mel-inverted or decoded codec audio. It combines a local patch encoder, a Vocos-based backbone, restoration bridge pre-training, and joint multi-NFE adversarial fine-tuning. One generator per task supports 1, 2, and 4 inference steps. Experiments include mel vocoding, OOD speech ,codec restoration, efficiency measurements, and a listening study.

Critical strengths
  • Joint multi-NFE training closely matches separately finetuned models while using only one generator. This provides practically useful quality-cost options without maintaing separate checkpoints for each NFE.
  • The model is well designed for speech. The patch encoder uses local spectral information, and the ablation studies show faster convergence in early training. Each component is not entirely new, but the combination ipmproves a good quality and speed.
Critical weaknesses
  • The practical benefit in a full TTS pipeline remains unclear. All mel-vocoding results use mel features extracted from reference audio. In practice, vocoder quality can change when the input is a mel spectrogram predicted by a TTS model, since such inputs contain different errors and artifacts. The current results therefore do not show whether the reported gains carry over to actual TTS or other speech generation systems use.
Questions for authors
  • Have you evaluated ReVoC with TTS model outputs? Vocoder results often differ between GT and predicted mel inputs. An evaluation on TTS outpus, including subjective scores such as MOS(for only sound quality), would greatly strengthen the reliabilty of the paper's claims.
5 comments
AuthorsOpenReview ↗

Response to Concerns Regarding Practical TTS Evaluation

We sincerely thank the reviewer for the constructive feedback and for recognizing the practical value of our joint multi-NFE training framework. We agree that evaluating vocoders on TTS-predicted Mel spectrograms is important, as their distributions may differ from those extracted from ground-truth speech. To address this concern, we conducted two additional experiments.

1. Vocoder Evaluation on TTS-Predicted Mel Spectrograms

We evaluated ReVoC using two TTS acoustic models, F5-TTS Base and E2-TTS Base, and compared it against Vocos. For each acoustic model, all vocoders decoded the same predicted Mel spectrograms for 1,127 utterances from LibriSpeech-PC test-clean, ensuring a consistent evaluation. Corpus WER was computed using Whisper-large-v3.

Acoustic ModelVocoderWER ↓SIM-o ↑UTMOS ↑MOS ↑
F5-TTS BaseVocos2.240.65463.8984.175
ReVoC-Base (NFE=1)2.240.66793.7644.300
ReVoC-Base (NFE=2)2.280.66794.0704.255
ReVoC-Base (NFE=4)2.220.66704.1644.250
E2-TTS BaseVocos2.490.70943.5723.970
ReVoC-Base (NFE=1)2.480.72093.5454.025
ReVoC-Base (NFE=2)2.510.72313.8464.225
ReVoC-Base (NFE=4)2.480.72353.9504.210

The results demonstrate that ReVoC's advantages extend to practical TTS-generated Mel spectrograms. In particular, the 4-step ReVoC improves UTMOS from 3.898 to 4.164 on F5-TTS and from 3.572 to 3.950 on E2-TTS, while consistently improving speaker similarity over Vocos. WER remains comparable across vocoders, indicating no substantial degradation in intelligibility.

We also conducted a listening test to assess perceived audio quality. ReVoC generally received slightly higher mean opinion scores (MOS) than Vocos. Given the small differences and potential variability across listeners, we interpret the MOS results as a complement to the objective evaluation.

2. Adapting ReVoC to TTS-Predicted Mel Spectrograms

We further examined whether ReVoC could adapt to the distribution shift from ground-truth to TTS-predicted Mel spectrograms. We used F5-TTS and E2-TTS to generate Mel spectrograms for 15,000 training utterances per model. Our joint multi-NFE framework lets the 4-step ReVoC configuration serve as a teacher for both the 1- and 2-step configurations. We used its outputs on the TTS-predicted inputs as distillation targets and mixed these training examples with ground-truth samples replayed from the original training set. We re-evaluated ReVoC after 30,000 mini-batches of GAN fine-tuning.

Metrics are reported before and after GAN fine-tuning (before → after).

Acoustic ModelNFEWER ↓SIM-o ↑UTMOS ↑
F5-TTS Base12.236 → 2.2060.6679 → 0.66933.764 → 3.865
22.275 → 2.2410.6679 → 0.66864.070 → 4.111
42.216 → 2.3000.6670 → 0.66814.164 → 4.168
E2-TTS Base12.477 → 2.4630.7209 → 0.72133.545 → 3.606
22.507 → 2.4970.7231 → 0.72303.846 → 3.860
42.482 → 2.5020.7235 → 0.72263.950 → 3.937

After joint multi-NFE fine-tuning with 4-step teacher supervision and ground-truth replay, UTMOS improved at both 1 and 2 NFE on the two TTS systems. The largest gains were at 1 NFE, with increases of 0.101 on F5-TTS and 0.061 on E2-TTS. At 4 NFE, UTMOS remained close to its pre-fine-tuning values.

Thanks the comments from the authors. I have read through it and keep my score unchanged.

Thank you for taking the time to read our response and follow up. We noticed that your recent comment may have been accidentally posted under Reviewer BAqX's discussion thread. We would therefore like to kindly follow up and ask whether our response in your original thread has sufficiently clarified your concerns regarding the novelty of our method and the scope of our title. In particular, we hope our explanation has helped clarify that our work addresses a task distinct from text-to-speech synthesis.

We sincerely appreciate your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Main Response

Main Response

We thank the three reviewers for their constructive feedback. The detailed protocols and results are available in our replies to the individual reviews.

Reviewer BAqX — Practical TTS evaluation. We compared ReVoC and Vocos on the same Mel spectrograms predicted by F5-TTS and E2-TTS, conducted a listening test, and evaluated adaptation to predicted Mel inputs.

Reviewer Wzfz — Unified training, baselines, and evaluation scope. We evaluated a single checkpoint for Mel vocoding and codec restoration and retrained Vocos under matched training data, crop length, and batch size. We also tested RVQ conditioning for EnCodec, extended codec restoration to Mimi, measured decoding and refinement latency, and evaluated zero-shot music vocoding on MUSDB18.

Reviewer GB7B — Scope and methodological contributions. We clarified that audio generation refers to conditional waveform generation for Mel vocoding and neural codec restoration. ReVoC uses a patch encoder to capture local time–frequency structure before temporal processing, jointly fine-tunes a single generator for multiple inference budgets, and refines initial estimates from Mel inversion or native codec decoding through a shared restoration formulation in the complex STFT domain. We discuss these design choices alongside the ablations and comparisons with diffusion- and flow-based vocoders.

We will incorporate these results and clarifications into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.