- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Generative-based audio restoration stands at a fundamental tension between too much freedom to a model (that might induce hallucinations) and too little (that would lead to poor restoration). This paper proposes a training time and an inference time technique to control this freedom. During training, the paper introduces Headroom-Anchored RL, which tracks rewards separately for each specific audio domain, and once RL stops yielding gains it shifts from DiffusionNFT post-training to a supervised anchor. During inference instead, the paper introduces Condition-Swap Guidance (CSG) which replaces the null of Classifier Free Guidance (CFG) with a scrambled copy of the conditioning. Out of the usage of these two techniques, the authors propose RESTORA, a 500M-parameter universal audio restoration model that surpasses the performance of state-of-the-art domain-specific audio restoration models on music, and is competitive on speech samples.
Critical strengths
- CSG is a straightforward and winning idea. It requires no manual negative condition and no retraining, and furthermore transfers to other models and domains (image restoration). This is one of the strongest contributions, as I am positive it could be useful beyond the scope of this paper.
- RESTORA is not the #1 model on all subdomains of audio, but it seems to be a pretty good all-rounder, as it excels in music and singing while still trailing closely on speech.
- The evaluation is broad and seems to be fair, as each subdomain has its own test set, over which multiple metrics are computed. Furthermore the two main music competitors are retrained on the same data.
- The guidance ablation is very much appreciated, it shows how CSG fares against CFG and other methods, showing it is best or second-best on most metrics.
- The listening test shows a clear preference for RESTORA over AnyEnhance.
Critical weaknesses
- The paper's main point is improving generative-model restoration by controlling hallucinations. Yet RESTORA's speech WER is above the noisy input itself. The authors acknowledge this in the limitations, but this deserves more discussion in the text.
- It is unclear whether the guidance strength, swapped streams and swap settings were selected on the same test sets used for the main results.
- The listening test provide neither confidence intervals nor a statistical analysis (t-test with p-values). With only 12 listeners on 10 clips per set, this could matter a lot. Furthermore, the listening test compares solely against AnyEnhance.
- Parameter counts and training costs of the compared models are not reported, making the comparison harder to judge.
- The main comparisons' numbers rest on a single training seed, with several conclusions stemming from small differences.
Questions for authors
- Can the authors state how they reconcile the faithfulness framing of the model while having speech WER remaining above the noisy input? Could the authors analyze where the additional error comes from? Maybe substituted words or other artifacts?
- Could the authors report variance across training seeds for the key Headroom comparisons (table 5 and 6)?
- How were the CSG hyperparameters selected?
- Could the authors provide confidence intervals and a statistical analysis for the listening test?
- Could the authors state the number of parameters in the compared models?
2 comments
Response Part2
Question 4 & Weakness 3: Listening test confidence intervals, statistical tests, and baseline choice
- 95% CIs and paired significance tests: Across 12 listeners and 6 test sets, we collected matched 1:1:1 triplets (1,944 ratings). Two-sided paired t-tests and Wilcoxon signed-rank tests confirm that Restora significantly outperforms AnyEnhance on every subset ( on Speech and on all other subsets; overall macro-mean gain MOS, pooled ). We have updated Section 4.2 and Table 4 in the revised PDF with these statistics:
| Category | Test set | Degraded (Mean 95% CI) | AnyEnhance (Mean 95% CI) | RESTORA (Mean 95% CI) | RESTORA AE ( CI) | Paired -test (vs. AE) | Wilcoxon (vs. AE) | |
|---|---|---|---|---|---|---|---|---|
| Voice | Singing (10) | 120 | 2.325 0.142 | 4.075 0.118 | 4.537 0.086 | +0.462 0.136 | ||
| Speech (10) | 120 | 1.975 0.136 | 4.275 0.107 | 4.500 0.092 | +0.225 0.141 | |||
| Instrumental | MUSDB18-Instr. (10) | 100 | 2.500 0.198 | 3.300 0.187 | 4.220 0.135 | +0.920 0.251 | ||
| Real music (10) | 114 | 2.974 0.145 | 3.171 0.166 | 3.750 0.146 | +0.579 0.216 | |||
| Full song | MUSDB18-Full (10) | 80 | 2.525 0.187 | 2.775 0.221 | 4.081 0.164 | +1.306 0.287 | ||
| Real full-song (10) | 114 | 3.013 0.169 | 2.974 0.165 | 3.640 0.148 | +0.667 0.216 | |||
| Overall | Macro-mean | 648 | 2.552 0.066 | 3.428 0.067 | 4.121 0.053 | +0.693 0.090 | — | — |
| Overall | Pooled | 648 | 2.547 0.071 | 3.479 0.077 | 4.129 0.058 | +0.650 0.086 |
- Why AnyEnhance was chosen: As explained in Appendix A.3 of the revised PDF, to avoid listener fatigue across the six test sets, we included one representative baseline in the listening study while reporting all 13 baselines in Tables 1–2. In Table 1, all other baselines are speech-only, whereas AnyEnhance is designed for both speech and singing; in Table 2, AnyEnhance (MERT) trained on the same music data is the strongest baseline across all aesthetic metrics (PQ, CE, and CU).
Question 5 & Weakness 4: Model parameter counts and training cost We have added the trainable backbone parameter counts of all compared models to Appendix A.2 (Table 6) of the revised PDF:
| Speech / Singing Model | Params | Music Model | Params |
|---|---|---|---|
| VoiceFixer | 117M | Mel2Mel + DiffWave | 57M |
| AnyEnhance | 363.5M | TF-Conformer | 2.5M |
| LLaSE-G1 | 1.07B | SonicMaster | 900M |
| PASE | 429M | MaskSR (retrained) | 335.7M |
| UniPASE | 545.7M | AnyEnhance (MERT) (retrained) | 335.7M |
| UniSE | 63M | — | — |
| SenSE | 1,300M | — | — |
| RE-USE | 9.6M | — | — |
| RESTORA (Ours) | 470.5M | RESTORA (Ours) | 470.5M |
- The only models that slightly beats Restora on speech perceptual metrics, SenSE (1,300M) and LLaSE-G1 (1.07B), have 2.2–2.7$\times$ our backbone parameters, and SonicMaster (900M) has nearly 2$\times$ our parameters on music. Restora covers all four domains with a single 470.5M non-autoregressive DiT (RTF = 0.141, 4.68 GiB peak memory on 1 GPU; see Appendix Table 5).
- As detailed in Appendix A.1, pre-training (300k steps) and SFT (50k steps) run on 4 80GB GPUs, while Headroom post-training takes only 1k steps on a single 80GB GPU. Both retrained music baselines (MaskSR and AnyEnhance-MERT) use the exact same music training data.
Response Part1
We thank the reviewer for the thoughtful comments and for highlighting the value and transferability of CSG. We have uploaded a revised PDF with all additions highlighted in blue.
Question 1 & Weakness 1: Faithfulness framing and WER breakdown
Whisper is trained on noisy speech and is already robust to raw noise. Generative restoration recovers speech masked by noise, cutting deletion errors nearly in half (D%: 4.96% -> 2.88%). However, where compound distortions destroy acoustic cues, the model fills in phonetically similar words that differ from the transcript, raising substitutions (S%: 11.75% -> 16.57%). CSG cuts insertion errors (hallucinated extra words) from 4.35% to 2.99% (below the noisy input's 4.38%) while also reducing substitutions.
Question 2 & Weakness 5: Statistical reliability of Headroom comparisons
Due to the compute cost of multi-stage 500M training, our main runs use a single training seed, but two sets of evidence show that the conclusions in Appendix Tables 7 and 8 (formerly Tables 5 and 6) are not due to random noise:
Question 3 & Weakness 2: How CSG hyperparameters were selected
The deployed checkpoint (1k step) and default CSG settings (swapping all three streams at ω=1.0 for speech/singing, MuQ-only at ω=0.5 for music, window [0.1, 0.7], ratio 0.5) were selected on a held-out validation split of 400 clips (100 speech, 100 singing, 100 instrumental, and 100 full-song mixtures) sampled from the training distribution. We then fixed these settings and evaluated once on the test sets. The sweeps on LibriVox and MUSDB18 in the Appendix are post-hoc ablations to show parameter sensitivity. We have clarified this selection protocol in Section 4.1 and Appendix B.2.4 of the revised PDF.