acceptodds
Under review as a conference paper at ICLR 2027

SVEET: Public-Data Design and Multi-Scale Flow Matching for Singing Voice Enhancement

Abstract

Singing voice enhancement (SVE) restores studio-quality vocals from degraded recordings, but clean singing data are scarce and existing systems often rely on proprietary stems or speech models adapted with limited singing data. We study SVE from a public-data design perspective, systematically varying the composition and scale of public speech and singing corpora while constructing a controlled benchmark for both clean-input preservation and restoration. Building on this data design, we introduce SVEET, a conditional flow-matching enhancer, and SVEET-MS, which adds a degradation-adaptive multi-scale objective for spectrally coherent restoration. Both outperform public zero-shot baselines and a strong discriminative system on singing enhancement, while transferring to speech enhancement with the same checkpoint. On the real-world SingVERSE benchmark, they achieve, to our knowledge, the best perceptual quality among systems with publicly available weights. In a blind listening test, SVEET-MS is preferred over SVEET in 71-76% of decided trials despite only modest differences in objective metrics. Code, data recipes, evaluation protocols, and checkpoints are released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech enhancement, singing voice enhancement, flow matching, audio restoration, generative models, data-centric deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ndNmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates Singing Voice Enhancement (SVE) from a data-composition and generative modeling perspective. First, the authors train a discriminative baseline (Hybrid Demucs, denoted DISCSVE) across several speech-to-singing mixing ratios (3:1 to 1:3) and data volumes using public corpora. Second, they adapt a complex-STFT conditional flow-matching speech enhancer (FlowSE) to stereo 44.1 kHz singing audio (SVEET) and propose a degradation-adaptive multi-scale vector-field loss (SVEET-MS) that pools complex residuals along the frequency axis. Third, they analyze an implementation pitfall regarding mean vs. sum loss reduction under the Adam optimizer. The models are evaluated on a custom 78-clip synthetic singing benchmark, two speech enhancement test sets (DNS 2020 and EARS-WHAM), the real-world SingVERSE benchmark, and a 30-listener subjective listening test.

Critical strengths

None. While releasing data manifests and checkpoints is generally commendable in audio restoration research, the foundational premise of the paper is built on factual misrepresentations of prior work, flawed experimental protocols (such as selecting data recipes on the benchmark test set and omitting single-modality baselines), and unsubstantiated technical claims. Thus, no substantive strength supports acceptance.

Critical weaknesses

The manuscript reads heavily like an unreviewed, agent-generated draft containing factual hallucinations, methodological contradictions, and severe structural flaws. Specifically:

  1. Factual Misrepresentation of Prior Work Reduces the Core "Public-Data" Premise (Contribution 1):
  • The paper repeatedly claims that existing SVE systems like AnyEnhance (Zhang et al., 2025) rely on "21,000 hours of proprietary audio" and "train on private studio stems" (e.g., Page 1, Lines 038–040). This is factually false. As explicitly documented in Table 2 of the original AnyEnhance paper, all studio singing voice and speech corpora used to train AnyEnhance are publicly available open-source datasets (Specifically: OpenSinger, PopCS, PopBuTFy, Opencpop, M4Singer, SingStyle111, and ACESinger for singing). Claiming that prior SOTA relies on "private studio stems" to propose a "first public-data design" narrative severely reduces the paper's primary data contribution.
  1. Framing a trivial PyTorch reduction bug (mean vs sum) as a key scientific contribution (Contribution 3):
  • In the Introduction's third bulleted contribution (Page 2, Lines 063–067), Section 3.3 (Page 5, Lines 231–236), Conclusion (Page 9, Line 482), and Appendix D (Page 15), the authors claim to uncover a "loss-scale pitfall for added objectives." Section 3.3 reveals this arises from a simple implementation error: replacing the base per-example sum reduction with .mean() across all dimensions, shrinking gradients by F x T=65536 and pushing the second moment below Adam’s ϵ=10−8\epsilon=10^{-8}. Their gradient probe merely verifies a basic calculus identity. Proposing a basic coding mistake as a flagship ICLR contribution is unacceptable.
  1. Ineffectiveness of the Proposed Multi-Scale Objective (SVEET-MS) on Objective Metrics and Real-World Data (Contribution 2):
  • On the custom synthetic singing benchmark (Page 7, Table 1), SVEET-MS actually degrades SI-SDR on Acapella (Noisy) compared to the vanilla SVEET baseline (10.94 dB vs. 11.55 dB). As admitted on Page 6 (Lines 329–338), the slight overall SI-SDR increase (35.10 vs. 34.71 dB) is entirely driven by high-variance clean pass-through clips, and both SI-SDR (+0.4 dB) and PESQ (+0.03) differences are statistically insignificant (95% bootstrap CIs include zero).
  • On speech enhancement (Page 16, Table 7), SVEET-MS performs worse than vanilla SVEET across all four metrics (SI-SDR, PESQ, ESTOI, PQ) on DNS 2020, and degrades PQ on EARS-WHAM (6.25 vs. 6.34).
  • On the real-world SingVERSE benchmark (Page 17, Table 9), SVEET-MS achieves lower PQ than vanilla SVEET (6.40 vs. 6.45). The authors explicitly admit that "the perceptual gain of the multi-scale objective on our synthetic noisy conditions does not carry over to these recordings". Furthermore, in the ablation study (Page 17, Table 8), gated multi-scale {1, 2, 4} causes SingVERSE PQ to collapse from 6.45 to 6.05. Finally, both SVEET (0.747) and SVEET-MS (0.758) severely degrade speaker similarity (SIM) on SingVERSE compared to the unprocessed noisy input (0.750) and baselines like HDemucs (0.834), indicating vocal timbre distortion.
  1. Flawed Experimental Design:
  • Missing 0:1 and 1:0 Baselines in Ratio Sweeps: The paper claims to systematically study speech-to-singing ratios, yet only tests 3:1, 2:1, 1:1, 1:2, and 1:3, completely omitting speech-only (1:0) and singing-only (0:1) baselines. Without single-modality baselines, the benefit of mixing speech and singing cannot be established. Moreover, performance across the tested ratios is nearly flat (PESQ spread of 0.08).
  • Omitting Reverberation in Training Despite Testing Dereverberation: The Introduction defines SVE as removing reverberation, and both noisy benchmark conditions include synthetic room convolutions and heavy reverberation. Yet the training recipes contain zero reverberation (Page 13, Equation 6), which the authors awkwardly excuse as testing "out-of-distribution generalization."
  • Chaotic Evaluation Revisions: The exact same model (DISCSVE 1:1, 1000h) is reported with an Overall SI-SDR of 32.84 dB (15.29 dB on Singing w/ Acc Noisy) in Table 1 (Page 7), but 33.56 dB (17.78 dB on Singing w/ Acc Noisy) in Tables 12 and 13 (Page 18) due to inconsistent "earlier revisions" of the benchmark and pipeline (Page 6, Line 281).
  1. Unacceptable Presentation:
  • The main text contains zero figures and only two tables (Table 1 and Table 2). Crucial empirical evidence central to the paper's claims—data composition sweeps (Tables 12–13), loss ablations (Table 8), speech generalization (Table 7), and SingVERSE evaluation (Table 9)—are all in the Appendix.
Questions for authors
  1. Regarding AnyEnhance's training data: Can the authors explain why AnyEnhance (Zhang et al., 2025) is repeatedly characterized as being trained on "proprietary audio" and "private studio stems," when Table 2 of Zhang et al. (2025) explicitly lists public open-source singing datasets (OpenSinger, PopCS, PopBuTFy, Opencpop, M4Singer, SingStyle111, ACESinger)?
  2. Regarding the missing single-modality baselines: Why were 0:1 (singing-only) and 1:0 (speech-only) configurations omitted from a study whose primary goal is to evaluate speech-to-singing data composition?
  3. Regarding the inconsistent training degradation recipes (Page 13, Lines 695–701): Why were M4Singer, OpenCPOP, SingStyle111, and VocalSet degraded with the light speech noise profile (single layer, [-30, -14] dB) while GTSinger was degraded with 1–4 layers at [-40, -2] dB, instead of regenerating all manifests under a unified protocol?
  4. Regarding the proposed Multi-Scale Loss and the "loss-scale pitfall": Why do the authors frame an implementation bug in their initial code (accidentally replacing sum reduction with mean reduction across all time-frequency bins, scaling the loss by 1/(F*T)) as a major scientific contribution? Furthermore, once this bug is fixed, SVEET-MS degrades SI-SDR on Acapella (Noisy) (10.94 vs. 11.55 dB in Table 1), underperforms vanilla SVEET across all metrics on DNS 2020 (Table 7), and yields lower PQ (6.40 vs. 6.45) on real-world SingVERSE recordings (Table 9). Given that the objective gains are statistically insignificant or negative on unseen real-world data, how can the effectiveness of the multi-scale objective be justified?
Reviewer_W3JYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces conditional flow-matching models designed for singing voice enhancement (SVE). Operating in the complex STFT domain with a mixture-anchored prior, the approach is built upon a systematically designed public training dataset combining 126.4 hours of clean singing and 209.4 hours of speech. To improve spectral coherence, the multi-scale variant (SVEET-MS) incorporates a frequency-pooled vector-field objective with degradation-adaptive weighting. Evaluated against public zero-shot baselines and a discriminative model across synthetic singing benchmarks, real-world SingVERSE recordings, and speech tasks, the proposed systems achieve superior perceptual quality—a finding strongly corroborated by a blind best-worst-scaling listening test where SVEET-MS is heavily preferred by human evaluators.

Critical strengths
  1. Practical Contribution to a Difficult Task: The paper tackles the challenging SVE problem under the realistic constraint of limited public training data. By systematically exploring how to combine public speech and singing corpora, the authors address a significant data bottleneck in audio restoration.
  2. Thorough Perceptual Validation: You highlighted the strong combination of objective benchmarking with rigorous subjective evaluation. Rather than relying solely on metrics like SI-SDR or PESQ, the authors back up their claims with a comprehensive blind best-worst-scaling (BWS) listening test involving 30 listeners and 780 scored trials, proving that SVEET-MS achieves a genuine human-preferred perceptual gain.
  3. Commitment to Open Science and Reproducibility: You recognized that the work provides strong reproducibility value. The authors publicly release their code, data construction manifests, evaluation protocols, listening test applications, and model checkpoints, allowing other researchers to replicate and build upon their recipes.
Critical weaknesses
  1. Data-Selection Leakage: The data composition and ratio sweeps (such as the 1:1 ratio and 1000-hour budget) were tuned and selected using an earlier revision of the benchmark test set rather than an independent validation set. This introduces hyperparameter leakage and overfitting to the evaluation benchmark. Furthermore, because these sweeps were conducted exclusively on the discriminative baseline, it remains unverified whether generative flow-matching models share the same data sensitivity, making the recipe transfer questionable.
  2. Compute Mismatch and Training Fairness: There is a significant asymmetry in training optimization between the models. The discriminative baseline (DISCSVE) was trained for only 180,000 updates (8 epochs), whereas the SVEET generative variants were trained for 528,750 updates (47 epochs). Consequently, it is difficult to isolate whether SVEET's performance advantage stems from the conditional flow-matching formulation or simply from receiving nearly three times as many training updates.
  3. Metric Instability and Single-Seed Evaluation: The experiments rely heavily on single-seed runs, and the benchmark's SI-SDR scores are notoriously sensitive to a small handful of clean-input clips whose pass-through scores fluctuate wildly across checkpoints. This high variance weakens confidence in small quantitative differences between models.
Questions for authors
  1. Data Composition Sweeps and Benchmark Leakage: Why were the data ratio and volume sweeps conducted exclusively on the discriminative baseline and evaluated using an earlier version of the test benchmark rather than an independent validation set? Can the authors provide validation-set evidence demonstrating that the selected 1:1 recipe is robust and hasn't overfit to the test benchmark, and verify whether generative flow-matching models actually share the same sensitivity to data composition as discriminative architectures?
  2. Compute Mismatch and Training Fairness: How do the authors justify the nearly threefold difference in training updates between the discriminative baseline (DISCSVE at 180,000 updates / 8 epochs) and the SVEET variants (528,750 updates / 47 epochs)? Can they provide a controlled comparison or training curves showing whether extending DISCSVE to a matched compute budget closes the performance gap, or to what extent SVEET's advantage stems from the flow-matching formulation versus longer training?
Reviewer_TN5vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies 44.1 kHz stereo singing voice enhancement (SVE) trained only on public corpora. It has three parts.

  1. Data recipe. The main recipe uses 126.4 h of unique singing and 209.4 h of unique speech. It is rendered into 1,000 h of pairs with three recipes: clean pass-through, MUSAN noise, and MUSDB accompaniment. A retrained Hybrid Demucs (DiscSVE) is used to sweep the speech:singing ratio (3:1 to 1:3) and the rendered volume.
  2. Enhancer. SVEET is a mixture-anchored conditional flow-matching enhancer in the compressed complex STFT, following Lee et al.'s FlowSE (NCSN++, 65.7M parameters, 5 Euler steps). SVEET-MS adds frequency-pooled vector-field penalties at scales {1, 2, 4, 8}, whose coarse scales are weighted per pair by input SNR.
  3. Loss scale. A diagnosis of loss-reduction mismatches. A mean-reduced auxiliary next to a sum-reduced base loss contributes almost no gradient. Mean-reducing the whole objective shrinks gradients until Adam's ε dominates the updates.

Evaluation uses a 78-clip benchmark from 39 sources: two "clean" transparency conditions and two noisy ones, scored with SI-SDR, PESQ-WB and Audiobox PQ. It also covers DNS 2020 and EARS-WHAM v2 for speech, the real-recording SingVERSE benchmark (DNSMOS, PQ, WavLM SIM), and a 30-listener best–worst listening test. SVEET-MS scores 35.10 dB / 3.87 / 7.48 overall against 34.71 / 3.84 / 7.45 for SVEET. Listeners prefer SVEET-MS over SVEET in 76% of decided noisy-singing trials. On SingVERSE, among the systems evaluated under the authors' protocol, SVEET-MS has the highest DNSMOS-OVRL and SVEET the highest PQ.

Critical strengths
  • The paper reports paired and clustered bootstrap CIs and Holm-corrected contrasts. It states its limitations openly: single seeds, recipe selection on an earlier benchmark revision, and matched data but not matched compute. It excludes VoiceBank-DEMAND because of speaker overlap. It evaluates on real SingVERSE recordings in addition to synthetic degradations, and it releases manifests, checkpoints and the listening-test app. The reader can trust what is claimed and see what is not.
  • There are 30 listeners, attention probes with a pre-declared exclusion rule, and pre-declared contrasts. In the pooled contrast, SVEET-MS beats SVEET (ΔBWS +0.40, CI [0.23, 0.58]) even though the two are nearly identical on PESQ/PQ. On the speech excerpts, SVEET-MS has a 10th-percentile frame energy 3.5 dB lower. This is a plausible explanation, and a useful observation about what intrusive and aesthetic metrics miss.
Critical weaknesses
  1. The method contribution is small relative to what is inherited, and the objective gains are small.

    • Inherited parts. SVEET is FlowSE with its constants (σ_max = 0.487, t_ε = 0.03, compression 0.5/0.15, NCSN++, 5 Euler steps), retrained at 44.1 kHz stereo.
    • Size of the gains. For SVEET-MS, the reported intervals for the overall SI-SDR (+0.4 dB) and PESQ (+0.03) differences include zero. Only the PQ gain (+0.03, CI [+0.007, +0.052]) is significant, and it is small. SI-SDR on noisy acapella drops by 0.6 dB, and the noisy-condition mean drops from 13.39 to 13.14 dB. On SingVERSE, PQ is lower (6.40 vs 6.45). At validation-selected checkpoints the PESQ ordering flips (3.85 vs 3.89, Table 5).
    • Selection. SVEET-MS is the better of two seeds (chosen on validation SI-SDR), while SVEET has one. Separately, Sec. 4.4 appears to justify the reported configuration with Table 8, which is computed on the evaluation sets ("which is why it is the reported system"). Sec. 5, however, says the ablations "involve no such selection". Please clarify whether the scale set and gate were fixed before evaluation.
    • Listening test. It carries most of the argument, but contains only 7 noisy-singing clips, and the CIs bootstrap over listeners only. It is evidence of a listener preference on these fixed items, but it does not quantify uncertainty over new singing material.
  2. The data-design study does not support its stated conclusion.

    • Design. The sweeps are single-seed with no speech-only or singing-only arm. The ratio sweep is flat, which shows insensitivity to the ratio, not that mixing helps. The recipe was selected on an earlier revision of the test benchmark.
    • Main datapoint. "Composition and augmentation matter more than volume" rests mainly on the 1:2 max-data + shifts run (33.89 → 35.01 dB overall, Table 13). That gain comes entirely from the clean transparency clips (clean mean 52.66 → 55.42 dB), while the noisy-condition mean drops (15.11 → 14.60 dB).
    • Confound. Fixed epochs confound volume with the number of updates.
    • Prior result. SingVERSE (Jiang et al., 2025) already retrained VoiceFixer and AnyEnhance on mixed speech and singing and reported better singing results with no loss on speech. The new part is the finer ratio and volume grid, and that grid comes out flat.

Smaller points:

  • "Overall" SI-SDR averages 50–60 dB pass-through scores with 10–15 dB restoration scores, which makes it hard to read as a restoration measure. Sec. 4.3 does acknowledge the noisy-acapella loss, but the headline "at no cost in overall fidelity" obscures the preservation/restoration trade-off. Please add separate preservation and restoration aggregates.
  • PESQ-WB is computed at 16 kHz and discards everything above 8 kHz of 44.1 kHz singing. DNSMOS and PQ have not been validated on singing. There is no FAD and no absolute-rating test (e.g. MUSHRA).
  • Baselines:
    • SVEET itself is a same-data control for the base objective, which is good. But no generative SE model with a different architecture (e.g. SGMSE+ or StoRM) is retrained on the recipe, so we cannot tell whether the gains over the zero-shot baselines come from the data or the model family.
    • There is no modern vocal separator (BS-RoFormer / Mel-RoFormer) as a zero-shot baseline.
    • The SingVERSE paper itself evaluates LLaSE-G1, SGMSE+ and StoRM, which all have public weights, but they are missing from Table 9. That weakens the "best among public weights" claim.
  • DiscSVE gets about 180k updates at batch 32, while the flow models get 528,750 at batch 64, roughly 5.9× as many training-example presentations. The paper labels this matched-data, not matched-compute, and the comparison is fine descriptively. But it means the difference cannot be attributed to flow matching alone.
  • The loss-reduction/Adam-ε note is useful practice, but I would not count it as a research contribution.
  • Two citation problems. First, Sec. 2 cites Wang et al. (2025) together with Lee et al. for mixture-anchored flow-matching SE and calls this the starting point. Wang et al.'s FlowSE is a different model, however (mel spectrogram, DiT backbone, rectified flow); only Lee et al. matches the complex-STFT formulation of Sec. 3.2. Please clarify the scope of that citation. Second, the Introduction, Sec. 2 and the Table 9 caption say AnyEnhance was trained on "21,000 hours of proprietary audio". According to the AnyEnhance paper, those ~21k hours are a filtered subset of the public Emilia corpus; only a ~1.2k h live-noise set is proprietary. The comparison's framing (public vs proprietary data) depends on this.
  • The released checkpoints are trained on OpenCPOP (CC BY-NC-ND) and GTSinger/M4Singer (CC BY-NC-SA). Please state the checkpoint license.
Questions for authors
  1. (Could change my score.) Results for a second SVEET seed and both SVEET-MS seeds. Does the PQ and listening advantage hold with equal seed budgets?
  2. (Could change my score.) Item-level listening results, leave-one-item-out, or a mixed model with listener and item effects.
  3. (Clarification.) What was the selection protocol for the scale set {1, 2, 4, 8} and the gate parameters (ρ_c = 15 dB, τ = 5 dB)? Which data were they tuned on, and were they fixed before the benchmark runs in Table 8?
  4. (Suggestion.) Speech-only and singing-only arms at 1,000 h, with restoration and preservation reported separately.
  5. (Suggestion.) Retrain one generative SE model with a different architecture (SGMSE+ or StoRM) on your recipe.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.