acceptodds
Under review as a conference paper at ICLR 2027

Restora: Controlling Generative Freedom for Faithful and Universal Audio Restoration

15 views

Abstract

Universal audio restoration aims to recover speech, singing, and music from diverse and compounded degradations with a single model. Generative models suit this goal because they synthesize clean content rather than simply filtering the degraded input, but this creative freedom can also produce hallucinations: plausible-sounding output unsupported by the original signal, so controlling how far generation departs from the input is essential. Two standard mechanisms govern this freedom: reward-based post-training refines the model toward perceptual quality targets, and inference-time guidance steers each sample toward the observed condition. As commonly implemented, each has a specific weakness that we address. Uniform reward optimization pushes every audio domain equally hard, so a domain that has saturated continues to receive noisy updates that degrade held-out quality; Headroom-Anchored RL instead tracks each domain on its own and, once reward signals become unreliable, shifts that domain back toward a stable supervised target. Classifier-free guidance (CFG) steers away from a null reference that discards the degraded input, the very evidence that faithful restoration relies on; Condition-Swap Guidance (CSG) replaces that null reference with a permuted copy of the input's own conditioning segments—same content, wrong timing—giving a structurally aware direction that improves perceptual quality and content accuracy together. Combining both mechanisms, Restora, a single 500M flow-matching model, raises the mean opinion score of degraded inputs from 2.55 to 4.12 across six test sets and achieves the best spectral-fidelity and aesthetic scores among compared music systems. CSG also transfers to a discrete-token speech enhancement model, showing it is not limited to diffusion guidance. Demo page: https://restora-demo.netlify.app/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio restoration, speech enhancement, flow matching, classifier-free guidance, post-training

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_UDGxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Generative-based audio restoration stands at a fundamental tension between too much freedom to a model (that might induce hallucinations) and too little (that would lead to poor restoration). This paper proposes a training time and an inference time technique to control this freedom. During training, the paper introduces Headroom-Anchored RL, which tracks rewards separately for each specific audio domain, and once RL stops yielding gains it shifts from DiffusionNFT post-training to a supervised anchor. During inference instead, the paper introduces Condition-Swap Guidance (CSG) which replaces the null of Classifier Free Guidance (CFG) with a scrambled copy of the conditioning. Out of the usage of these two techniques, the authors propose RESTORA, a 500M-parameter universal audio restoration model that surpasses the performance of state-of-the-art domain-specific audio restoration models on music, and is competitive on speech samples.

Critical strengths
  1. CSG is a straightforward and winning idea. It requires no manual negative condition and no retraining, and furthermore transfers to other models and domains (image restoration). This is one of the strongest contributions, as I am positive it could be useful beyond the scope of this paper.
  2. RESTORA is not the #1 model on all subdomains of audio, but it seems to be a pretty good all-rounder, as it excels in music and singing while still trailing closely on speech.
  3. The evaluation is broad and seems to be fair, as each subdomain has its own test set, over which multiple metrics are computed. Furthermore the two main music competitors are retrained on the same data.
  4. The guidance ablation is very much appreciated, it shows how CSG fares against CFG and other methods, showing it is best or second-best on most metrics.
  5. The listening test shows a clear preference for RESTORA over AnyEnhance.
Critical weaknesses
  1. The paper's main point is improving generative-model restoration by controlling hallucinations. Yet RESTORA's speech WER is above the noisy input itself. The authors acknowledge this in the limitations, but this deserves more discussion in the text.
  2. It is unclear whether the guidance strength, swapped streams and swap settings were selected on the same test sets used for the main results.
  3. The listening test provide neither confidence intervals nor a statistical analysis (t-test with p-values). With only 12 listeners on 10 clips per set, this could matter a lot. Furthermore, the listening test compares solely against AnyEnhance.
  4. Parameter counts and training costs of the compared models are not reported, making the comparison harder to judge.
  5. The main comparisons' numbers rest on a single training seed, with several conclusions stemming from small differences.
Questions for authors
  1. Can the authors state how they reconcile the faithfulness framing of the model while having speech WER remaining above the noisy input? Could the authors analyze where the additional error comes from? Maybe substituted words or other artifacts?
  2. Could the authors report variance across training seeds for the key Headroom comparisons (table 5 and 6)?
  3. How were the CSG hyperparameters selected?
  4. Could the authors provide confidence intervals and a statistical analysis for the listening test?
  5. Could the authors state the number of parameters in the compared models?
2 comments
AuthorsOpenReview ↗

Response Part1

We thank the reviewer for the thoughtful comments and for highlighting the value and transferability of CSG. We have uploaded a revised PDF with all additions highlighted in blue.


Question 1 & Weakness 1: Faithfulness framing and WER breakdown

  • Faithfulness framing: Discriminative filters or raw noisy inputs do not invent new words, whereas unconstrained generative restoration models often drift from the input and hallucinate content. Our goal is to control this drift within generative restoration: compared to the unguided generative baseline, CSG lowers speech WER from 0.2516 to 0.2244 (0.2175 with CSG + CFG, approaching the noisy input's 0.2109), and lowers WER on the discrete generative model AnyEnhance from 0.207 to 0.188 (Appendix Table 17). On music, Headroom slows LSD drift by 2.5×2.5\times, and CSG improves LSD from 7.032 to 6.963.
  • Where the additional WER comes from: We decomposed the LibriVox WER (500 utterances) into substitution (S%), deletion (D%), and insertion (I%) error rates (added to Section 4.4 and Appendix B.2.5, Table 16 of the revised PDF):
ConfigurationWER ↓\downarrowS% ↓\downarrowD% ↓\downarrowI% ↓\downarrow
Noisy Input0.210911.754.964.38
HR-1k (Base)0.251618.032.784.35
HR-1k + CFG0.232416.142.684.43
HR-1k + CSG0.224416.572.882.99
HR-1k + CSG + CFG0.217515.922.783.05

Whisper is trained on noisy speech and is already robust to raw noise. Generative restoration recovers speech masked by noise, cutting deletion errors nearly in half (D%: 4.96% -> 2.88%). However, where compound distortions destroy acoustic cues, the model fills in phonetically similar words that differ from the transcript, raising substitutions (S%: 11.75% -> 16.57%). CSG cuts insertion errors (hallucinated extra words) from 4.35% to 2.99% (below the noisy input's 4.38%) while also reducing substitutions.


Question 2 & Weakness 5: Statistical reliability of Headroom comparisons

Due to the compute cost of multi-stage 500M training, our main runs use a single training seed, but two sets of evidence show that the conclusions in Appendix Tables 7 and 8 (formerly Tables 5 and 6) are not due to random noise:

  • Appendix Table 8: For the CSG gain difference at the bottom of Table 8 (CSG gain on Headroom minus CSG gain on Fully-RL), the music fidelity interaction is ΔΔLSD=+0.264\Delta\Delta\text{LSD} = +0.264 with a 95% CI of [+0.219, +0.308] (excluding zero). The 95% CIs for speech NISQA (+0.038), OVRL (+0.010), SBS (+0.003), and music SSIM (+0.0022), PQ (+0.039), and CE (+0.034) all exclude zero as well (marked with * in Appendix Table 8 of the revised PDF).
  • Hyperparameters ablations: The finding that Headroom controls music LSD drift compared to Fully-RL holds across both independently trained 200M and 500M models (Appendix Table 19) and across all three independent EMA hyperparameter runs (Appendix Table 10).

Question 3 & Weakness 2: How CSG hyperparameters were selected

The deployed checkpoint (1k step) and default CSG settings (swapping all three streams at ω=1.0\omega=1.0 for speech/singing, MuQ-only at ω=0.5\omega=0.5 for music, window [0.1, 0.7], ratio 0.5) were selected on a held-out validation split of 400 clips (100 speech, 100 singing, 100 instrumental, and 100 full-song mixtures) sampled from the training distribution. We then fixed these settings and evaluated once on the test sets. The sweeps on LibriVox and MUSDB18 in the Appendix are post-hoc ablations to show parameter sensitivity. We have clarified this selection protocol in Section 4.1 and Appendix B.2.4 of the revised PDF.

AuthorsOpenReview ↗

Response Part2

Question 4 & Weakness 3: Listening test confidence intervals, statistical tests, and baseline choice

  • 95% CIs and paired significance tests: Across 12 listeners and 6 test sets, we collected N=648N = 648 matched 1:1:1 triplets (1,944 ratings). Two-sided paired t-tests and Wilcoxon signed-rank tests confirm that Restora significantly outperforms AnyEnhance on every subset (p=1.9×10−3p = 1.9\times 10 ^ {-3} on Speech and p<10−6p < 10 ^ {-6} on all other subsets; overall macro-mean gain +0.693±0.090+0.693 \pm 0.090 MOS, pooled t=14.85,p=4.0×10−43t = 14.85, p = 4.0 \times 10 ^ {-43}). We have updated Section 4.2 and Table 4 in the revised PDF with these statistics:
CategoryTest setnnDegraded (Mean ±\pm 95% CI)AnyEnhance (Mean ±\pm 95% CI)RESTORA (Mean ±\pm 95% CI)RESTORA −- AE (Δ±\Delta \pm CI)Paired tt-test (vs. AE)Wilcoxon (vs. AE)
VoiceSinging (10)1202.325 ±\pm 0.1424.075 ±\pm 0.1184.537 ±\pm 0.086+0.462 ±\pm 0.136t=6.74,p=6.0×10−10t=6.74, p=6.0\times 10 ^ {-10}p=4.9×10−9p=4.9\times 10 ^ {-9}
Speech (10)1201.975 ±\pm 0.1364.275 ±\pm 0.1074.500 ±\pm 0.092+0.225 ±\pm 0.141t=3.17,p=1.9×10−3t=3.17, p=1.9\times 10 ^ {-3}p=2.3×10−3p=2.3\times 10 ^ {-3}
InstrumentalMUSDB18-Instr. (10)1002.500 ±\pm 0.1983.300 ±\pm 0.1874.220 ±\pm 0.135+0.920 ±\pm 0.251t=7.29,p=7.9×10−11t=7.29, p=7.9\times 10 ^ {-11}p=7.8×10−10p=7.8\times 10 ^ {-10}
Real music (10)1142.974 ±\pm 0.1453.171 ±\pm 0.1663.750 ±\pm 0.146+0.579 ±\pm 0.216t=5.32,p=5.4×10−7t=5.32, p=5.4\times 10 ^ {-7}p=7.1×10−7p=7.1\times 10 ^ {-7}
Full songMUSDB18-Full (10)802.525 ±\pm 0.1872.775 ±\pm 0.2214.081 ±\pm 0.164+1.306 ±\pm 0.287t=9.06,p=7.3×10−14t=9.06, p=7.3\times 10 ^ {-14}p=6.4×10−11p=6.4\times 10 ^ {-11}
Real full-song (10)1143.013 ±\pm 0.1692.974 ±\pm 0.1653.640 ±\pm 0.148+0.667 ±\pm 0.216t=6.11,p=1.5×10−8t=6.11, p=1.5\times 10 ^ {-8}p=3.4×10−8p=3.4\times 10 ^ {-8}
OverallMacro-mean6482.552 ±\pm 0.0663.428 ±\pm 0.0674.121 ±\pm 0.053+0.693 ±\pm 0.090——
OverallPooled6482.547 ±\pm 0.0713.479 ±\pm 0.0774.129 ±\pm 0.058+0.650 ±\pm 0.086t=14.85,p=4.0×10−43t=14.85, p=4.0\times 10 ^ {-43}p=6.5×10−38p=6.5\times 10 ^ {-38}
  • Why AnyEnhance was chosen: As explained in Appendix A.3 of the revised PDF, to avoid listener fatigue across the six test sets, we included one representative baseline in the listening study while reporting all 13 baselines in Tables 1–2. In Table 1, all other baselines are speech-only, whereas AnyEnhance is designed for both speech and singing; in Table 2, AnyEnhance (MERT) trained on the same music data is the strongest baseline across all aesthetic metrics (PQ, CE, and CU).

Question 5 & Weakness 4: Model parameter counts and training cost We have added the trainable backbone parameter counts of all compared models to Appendix A.2 (Table 6) of the revised PDF:

Speech / Singing ModelParamsMusic ModelParams
VoiceFixer117MMel2Mel + DiffWave57M
AnyEnhance363.5MTF-Conformer2.5M
LLaSE-G11.07BSonicMaster900M
PASE429MMaskSR (retrained)335.7M
UniPASE545.7MAnyEnhance (MERT) (retrained)335.7M
UniSE63M——
SenSE1,300M——
RE-USE9.6M——
RESTORA (Ours)470.5MRESTORA (Ours)470.5M
  • The only models that slightly beats Restora on speech perceptual metrics, SenSE (1,300M) and LLaSE-G1 (1.07B), have 2.2–2.7$\times$ our backbone parameters, and SonicMaster (900M) has nearly 2$\times$ our parameters on music. Restora covers all four domains with a single 470.5M non-autoregressive DiT (RTF = 0.141, 4.68 GiB peak memory on 1 GPU; see Appendix Table 5).
  • As detailed in Appendix A.1, pre-training (300k steps) and SFT (50k steps) run on 4 ×\times 80GB GPUs, while Headroom post-training takes only 1k steps on a single 80GB GPU. Both retrained music baselines (MaskSR and AnyEnhance-MERT) use the exact same music training data.
Reviewer_Agb9OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RESTORA, a unified flow-matching Diffusion Transformer with 500 million parameters for universal audio restoration across speech, singing, and music. The paper introduces two main methods. First, Headroom-Anchored RL adaptively adjusts the balance between reinforcement learning and supervised fine-tuning based on reward headroom in different audio domains, aiming to mitigate reward over-optimization and fidelity drift during multi-domain training. Second, Condition-Swap Guidance (CSG) constructs negative conditions by permuting temporal segments of the input condition to enhance temporal alignment between the generated output and the input signal.

The authors evaluate RESTORA on speech, singing, and music restoration using a range of metrics for perceptual quality, content fidelity, and music quality, along with a human listening study. The results show that RESTORA performs strongly on music restoration and remains competitive on speech and singing restoration. Transfer experiments across different model representations and modalities further suggest that CSG has the potential for broader applicability.

Critical strengths
  1. The paper introduces complementary training-time and inference-time mechanisms to address content fidelity in audio restoration. Headroom-Anchored RL uses per-domain gating to mitigate reward over-optimization and performance drift in multi-domain reinforcement learning, while CSG constructs temporally misaligned negative conditions to improve the fidelity of generated outputs to the input conditions. These two mechanisms operate at the training and inference stages, respectively, providing a relatively comprehensive approach to the paper's central problem.

  2. The experiments cover multiple audio domains, and CSG demonstrates promising transferability. The authors evaluate speech, singing, and music restoration using multiple metrics for perceptual quality and content fidelity, and conduct a human listening study. The strong results on music restoration, together with the transfer of CSG to a discrete-token speech model and an image-restoration model, enhance the practical value and potential impact of the work.

Critical weaknesses
  1. The positioning of the methodological novelty remains insufficiently established. Headroom-Anchored RL is conceptually close to commonly used supervised anchoring strategies in reinforcement learning. Its main additions are per-domain gating and a control mechanism based on reward-change slopes. Meanwhile, the paper does not sufficiently discuss related guidance methods such as Dynamic Guidance and EMAG, making it difficult to fully establish the distinctive contributions of CSG relative to prior work.

  2. The validation of content fidelity and practical deployment remains insufficient. In speech restoration, WER remains higher than that of the noisy input even with CSG, indicating that the model still faces risks of content distortion or hallucination. In addition, the paper does not report NFE, RTF, or GPU memory usage, and the human listening study includes only 12 participants. These limitations constrain the assessment of the model's practical computational cost, content fidelity, and perceived quality improvements.

Questions for authors
  1. For Headroom-Anchored RL, how sensitive is the method to hyperparameters such as qq, τ\tau, βfast\beta_{\text{fast}}, βslow\beta_{\text{slow}}, and the gate floor amin⁡a_{\min}? Has training stability been evaluated under different data-shuffling orders?

  2. In speech restoration, CSG still fails to reduce WER below the noisy-input baseline. Have the authors analyzed the specific causes of this result, such as content hallucination or speech recognition errors? Is there experimental evidence that incorporating external content constraints, such as text, can improve this issue?

  3. What are the key differences between CSG and inference-time guidance methods such as Dynamic Guidance and EMAG? Beyond cross-modal transferability, could the authors provide more direct comparative experiments to further validate the independent contribution of condition-level temporal permutation to restoration fidelity?

4 comments
AuthorsOpenReview ↗

Response Part1

We thank the reviewer for the constructive feedback. We have uploaded a revised PDF with all additions highlighted in blue.


Weakness 1: Positioning relative to supervised anchoring, Dynamic Guidance, and EMAG

  • Mixing RL with supervised loss dates back to fixed-weight methods such as PPO-ptx in InstructGPT [1], which we evaluated via Fixed-0.5 in Appendix Table 7 (where music LSD still drifts). Recent methods such as WavAlign [2] adjust the mixing weight from rollout reward variance and maximum reward to filter out poor early rollouts. In multi-domain restoration, however, domains over-optimize their rewards at different rates. Headroom therefore tracks the per-domain slope of the top-qq-to-mean reward gap and shifts a domain back to SFT when outliers pull away from the mean. We have added this discussion to Section 3.3 of the revised PDF.
  • Recent dynamic guidance methods either greedily search for the CFG scale ω\omega at each step using online latent-space evaluators [3], or use an external classifier or MLLM during denoising to select and refine the positive condition from a candidate set [4]. Both approaches rely on external online evaluators and keep the negative reference unchanged. In audio restoration, the positive condition is the single given degraded recording, and CSG constructs a task-relevant negative branch directly by permuting the input condition without external evaluators. We have added this discussion to Section 3.4 of the revised PDF.
  • EMAG [5] maintains a cross-step EMA of self-attention maps AtA _ t in middle DiT layers and replaces the layer with the largest change to form a negative branch. Like PAG, SEG, and SSG, EMAG perturbs internal network states while feeding the exact same condition cc to both branches. We added an EMAG comparison on our 500M model in Section 4.4 (Table 3) and Appendix B.2.1 (Table 11) (see Question 3 below).

Weakness 2: Computational cost (NFE / RTF / GPU memory) and listening study statistics

  • Computational Cost: As noted in Appendix A.1, CSG activates the negative branch only for t∈[0.1,0.7]t \in [0.1, 0.7], taking 32 NFE per 20-step solve versus 40 NFE for CFG. We added the full inference benchmark measured on 1 GPU (batch size 1) in Appendix A.1 (Table 5) of the revised PDF:
ConfigurationStepsMeasured NFE ↓\downarrowRTF ↓\downarrowPeak GPU Mem. ↓\downarrowTime vs. Base
Base@2020200.08924.681 GiB1.00$\times$
CSG@20 (deployed)20320.14064.681 GiB1.58$\times$
CFG@2020400.16154.681 GiB1.81$\times$
CSG + CFG@2020520.21684.681 GiB2.43$\times$

CSG achieves an RTF of 0.1406 (13% faster than CFG) and adds no peak GPU memory (4.681 GiB) if the two branches run sequentially.

  • Listening test statistics: Across 12 listeners and 6 test sets, we collected N=648N=648 matched 1:1:1 triplets (1,944 ratings). We have updated Section 4.2 and Table 4 in the revised PDF with 95% CIs and paired t-tests: Restora significantly outperforms Degraded and AnyEnhance on every subset (overall macro-mean gain over AnyEnhance: +0.693±0.090+0.693 \pm 0.090 MOS, pooled t=14.85,p=4.0×10−43t = 14.85, p = 4.0 \times 10 ^ {-43}; even on the closest subset, Speech, +0.225±0.141,p=1.9×10−3+0.225 \pm 0.141, p = 1.9 \times 10 ^ {-3}; on MUSDB18-Full, +1.306±0.287,p=7.3×10−14+1.306 \pm 0.287, p = 7.3 \times 10 ^ {-14}; full table in our response to Reviewer UDGx). We compared against AnyEnhance to avoid listener fatigue across the six test sets, as AnyEnhance is the only baseline in Table 1 designed for both speech and singing, and AnyEnhance (MERT) is the strongest baseline on all aesthetic metrics (PQ/CE/CU) in Table 2.

Question 1: Sensitivity to Headroom hyperparameters and data shuffling

  • Appendix B.1.5 (Table 10) in the original submission ablates the EMA time constants. A faster βf=0.80\beta _ f = 0.80 (vs. default 0.90) better protects music later in training (2k MUSDB18 LSD: 8.075 vs. 8.722), but over-reacts to short-term fluctuations in speech, dropping the speech gate to 0.67–0.71 and lowering LibriVox NISQA. A slower βs=0.995\beta _ s = 0.995 (vs. 0.99) keeps the speech gate near 1 with metrics close to the default. The default 0.90/0.99 gives the best balance.
  • In Eq. (6), dividing sloped\text{slope} _ d by its running magnitude scaled=EMA⁡(∣sloped∣)\text{scale} _ d = \operatorname{EMA}(\lvert\text{slope} _ d\rvert) normalizes the gate input across different reward scales and top-qq magnitudes. We therefore fix q=0.10q=0.10, τ=0.2\tau=0.2, and amin⁡=0a _ {\min}=0, and control response speed through βf\beta _ f and βs\beta _ s.
  • For data shuffling, each step samples speech and music prompts 1:1, and the slow EMA (βs=0.99\beta _ s = 0.99) smooths out single-batch ordering fluctuations.
AuthorsOpenReview ↗

Response Part2

Question 2: WER breakdown and external text constraints

We decomposed the LibriVox WER (500 utterances) into substitution (S%), deletion (D%), and insertion (I%) error rates (added to Section 4.4 and Appendix B.2.5, Table 16 of the revised PDF):

ConfigurationWER ↓\downarrowS% ↓\downarrowD% ↓\downarrowI% ↓\downarrow
Noisy Input0.210911.754.964.38
HR-1k (Base)0.251618.032.784.35
HR-1k + CFG0.232416.142.684.43
HR-1k + CSG0.224416.572.882.99
HR-1k + CSG + CFG0.217515.922.783.05
  • Whisper is trained on noisy speech and is already robust to raw noise. Generative restoration recovers masked speech segments and cuts deletion errors nearly in half (D%: 4.96% -> 2.78%). However, where acoustic cues are heavily corrupted, the model synthesizes natural-sounding words that differ from the reference, increasing substitutions (S%: 11.75% -> 18.03%).
  • CSG lowers insertion errors (ungrounded word hallucinations) from 4.35% to 2.99% (below the noisy input's 4.38%) and reduces substitutions from 18.03% to 16.57%. Since CSG is stronger at suppressing insertions (2.99%) while CFG yields slightly fewer substitutions (16.14%), combining CSG + CFG further reduces substitutions to 15.92% and lowers overall WER to 0.2175.
  • For this work, Restora only takes degraded audio as input and does not use text. As noted in Section 5 (Future Work: "Text or visual prompts, such as transcripts, lyrics, or lip videos, could supply the content evidence that degraded audio lacks"), adding transcripts, lyrics, or lip motion is a promising next step when audio evidence is lost.

Question 3: Comparison with Dynamic Guidance [3, 4], EMAG [5], and condition-level temporal permutation

We implemented EMAG [5] on our 500M Headroom-1k model and added these results to Section 4.4 (Table 3), Appendix B.2.1 (Table 11), and Appendix B.2.4 of the revised PDF:

MethodPerturbation Location & MechanismSpeech NISQA ↑\uparrowSpeech WER ↓\downarrowMusic LSD ↓\downarrowSource
BaseUnguided4.3630.25167.032Table 3 / 11
EMAG [5]Internal network: cross-step Attention EMA4.4010.25787.279Table 3 / 11 (New)
SSG-tokenInternal network: hidden-state token swap4.4050.23967.031Table 3 / 11
CSG (similar)Input condition: swap similar chunks4.5550.25837.001Table 15 / B.2.4 (New)
CSG (ours)Input condition: swap dissimilar chunks4.5900.22446.963Table 3 / 11
  • CSG (ours) vs. EMAG and SSG-token: Because EMAG smooths internal attention maps while keeping the input condition unchanged, its guidance direction does not penalize ignoring the input condition. While EMAG slightly raises speech NISQA (4.401), both speech WER (0.2578) and music LSD (7.279) are worse than unguided Base. Swapping internal hidden states (SSG-token) also gives much smaller gains than CSG. This shows that perturbing the input condition is more effective for restoration fidelity than perturbing internal attention.
  • CSG (ours) vs. CSG (similar): When we permute the input condition but swap acoustically similar chunks (CSG similar, which does not break content alignment), speech WER degrades to 0.2583 and music LSD (7.001) is worse than swapping dissimilar chunks (6.963). This confirms that the fidelity gain comes from breaking the temporal alignment of the condition.

[1] Ouyang, Long, et al. "Training Language Models to Follow Instructions with Human Feedback." Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 27730-27744.

[2] Chen, Yifu, et al. "WavAlign: Enhancing Intelligence and Expressiveness in Spoken Dialogue Models via Adaptive Hybrid Post-Training." Findings of the Association for Computational Linguistics: ACL, 2026.

[3] Papalampidi, Pinelopi, et al. "Dynamic Classifier-Free Diffusion Guidance via Online Feedback." International Conference on Learning Representations, 2026.

[4] Triaridis, Kostas, et al. "Mitigating Diffusion Model Hallucinations with Dynamic Guidance." arXiv preprint arXiv:2510.05356, 2025.

[5] Yadav, Ankit, Huy Ta, and Lingqiao Liu. "EMAG: Self-Rectifying Diffusion Sampling with Exponential Moving Average Guidance." European Conference on Computer Vision, 2026.

Thanks for the author’s detailed reply and experiments. My issue is basically resolved, and I will upgrade my rating~

Pleased that our responses and experiments have resolved your concerns. Your comments are very helpful for improving our work!

Reviewer_JN1hOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Restora, a single model for restoring speech, singing, and music. It introduces Headroom-Anchored RL to balance reward-based learning with supervised training and condition-swap guidance. The proposed method achieves strong music restoration performance and competitive performance on speech and singing.

Critical strengths
  1. Restora handles several audio domains such as music, speech, and singing in a single model. Further, the listening study also favors it over AnyEnhance across all test sets.
  2. The paper provides extensive analysis, and transfer experiments further demonstrate that CSG can be applied to other restoration models without retraining.
Critical weaknesses
  1. The main proposed methods, Condition-Swap Guidance (CSG) and Headroom-Anchored RL, are based on earlier work. CSG modifies Self-Swap Guidance (SSG) by moving the swap from the network's hidden states to the input condition. Headroom-Anchored RL uses the DiffusionNFT objective and mixes RL with supervised fine-tuning, similar to WavAlign. CSG is directly compared to SSG in the paper (Table 3). However, Headroom-Anchored RL is compared only against fixed mixing schedules, not other adaptive mixing methods, so it remains unclear whether the reward-gap controller offers any advantage.
  2. If speech and music use the same weights, how does closing the music gate protect music, given that speech RL updates still affect the shared model?
Questions for authors
  1. Were the checkpoint and guidance settings selected on a separate validation split, or on the LibriVox and MUSDB18 sets used for the reported evaluation? Please clarify the selection protocol.
3 comments

We thank the reviewer for the positive evaluation and helpful questions. We have uploaded a revised PDF with all changes highlighted in blue.


Weakness 1: Comparison with adaptive mixing methods such as WavAlign

Adaptive gates in prior hybrid post-training methods such as WavAlign [1] are designed to filter out low-quality or low-variance rollouts early in training. In WavAlign (Eq. 11), the RL weight increases with the maximum reward and the group variance: λt∝σ(k(Rmax⁡−3))⋅clip⁡(Var⁡(Rt)/4,0,1)\lambda _ t \propto \sigma(k(R _ {\max}-3)) \cdot \operatorname{clip}(\operatorname{Var}(R _ t)/4, 0, 1). In multi-domain audio restoration, however, music degrades due to reward over-optimization: a few outlier rollouts exploit the reward model and get high scores, which pushes Rmax⁡R _ {\max} up and widens the group gap (see Fig. 2a and Appendix Fig. 3b). Under a WavAlign-style rule, high Rmax⁡R _ {\max} and large variance keep the RL gate fully open right when over-optimization happens. In contrast, Headroom tracks the slope of the gap between the top-qq rollouts and the mean. It keeps RL open only when the mean is catching up to the top rollouts (gap narrowing), and shifts back to SFT when outliers pull away from the mean (gap widening). Appendix Table 10 also shows that this controller maintains music fidelity across different EMA time constants. We have added this discussion to Section 3.3 of the revised PDF.


Weakness 2: How closing the music gate protects shared weights

Closing the music gate does not stop music training. Every step still samples speech and music 1:1, and amusic→0a _ {\text{music}} \to 0 switches the music update from RL to the supervised flow-matching loss on clean music (LSFT, music\mathcal{L} _ {\text{SFT, music}}).

  • In Appendix B.1.4 of the original submission, we tested removing music updates altogether (running speech RL alone with neither music RL nor music SFT). Without the music SFT anchor, shared weights drift toward speech and unguided MUSDB18 LSD degrades from 7.032 to 7.641.
  • We measured the full-model gradients (gRLg ^ {\text{RL}} and gSFTg ^ {\text{SFT}}) across all 365 trainable parameter tensors over 128 speech and 128 music rollouts:
Checkpointcos⁡(gmusicRL,gmusicSFT)\cos(g _ {\text{music}} ^ {\text{RL}}, g _ {\text{music}} ^ {\text{SFT}})Layers w/ cos⁡<0\cos < 0cos⁡(gspeechRL,gmusicSFT)\cos(g _ {\text{speech}} ^ {\text{RL}}, g _ {\text{music}} ^ {\text{SFT}})Layers w/ ∣cos⁡∣<0.1\lvert\cos\rvert < 0.1Norm ∥gspeechRL∥2\lVert g _ {\text{speech}} ^ {\text{RL}}\rVert _ 2 / ∥gmusicSFT∥2\lVert g _ {\text{music}} ^ {\text{SFT}}\rVert _ 2
SFT-50k-0.20780.3%+0.02862.7%0.221 / 0.132
Headroom-1k-0.30585.5%-0.04560.6%0.204 / 0.206

At Headroom-1k, the music RL gradient opposes the clean music SFT gradient in 85.5% of parameter layers (cos = -0.305), so closing the music gate removes updates that directly hurt the music distribution. At the same time, the music SFT gradient has the same norm as the speech RL gradient (0.206 vs. 0.204) and is nearly orthogonal to it in 60.6% of layers (∣cos⁡∣<0.1\lvert\cos\rvert < 0.1, global cos = -0.045). This keeps the shared weights anchored to clean music without interfering with speech RL. We have added this analysis to Appendix B.1.4 and Table 9 of the revised PDF


Question 1: Validation protocol for checkpoint and guidance selection

The deployed checkpoint (1k step) and default CSG settings (swapping all three streams at ω=1.0\omega=1.0 for speech/singing, MuQ-only at ω=0.5\omega=0.5 for music, window [0.1, 0.7], ratio 0.5) were selected on a held-out validation split of 400 clips (100 speech, 100 singing, 100 instrumental, and 100 full-song mixtures) sampled from the training distribution. We then fixed these settings and evaluated once on the test sets. The sweeps on LibriVox and MUSDB18 in the Appendix are post-hoc ablations to show parameter sensitivity. We have clarified this selection protocol in Section 4.1 and Appendix B.2.4 of the revised PDF.

[1] Chen, Yifu, et al. "WavAlign: Enhancing Intelligence and Expressiveness in Spoken Dialogue Models via Adaptive Hybrid Post-Training." Findings of the Association for Computational Linguistics: ACL, 2026.

Thank you for the thoughtful rebuttal and the additional results. Overall, the paper is well-structured, and the rebuttal addresses my concerns. I maintain my clear acceptance recommendation. Best of luck.

Glad to hear our response addressed your concerns. We truly appreciate your constructive feedback, which helps us improve our work. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.