acceptodds
Under review as a conference paper at ICLR 2027

Universal Mixup Loss: Training Audio Autoencoders to Decode the Positive-Weight Mixing Cone

Abstract

Waveform mixing is linear, but mixing the latent codes of an audio autoencoder rarely produces a representation that decodes to the corresponding audio mixture. This limits the use of latent arithmetic for audio editing, source separation, and generative mixing. We show that supervising a few gain and addition identities leaves large regions of the mixing space unconstrained. We introduce the Universal Mixup Loss, a decoder-side objective that instead supervises positive-weight combinations of two latent codes throughout their mixing cone. Training points are sampled in polar coordinates, decoupling an annealed radial scale from a uniformly distributed mixing angle. Paired with a scale-sensitive soft-norm bottleneck, this supervision substantially reduces mixing commutation errors while preserving reconstruction quality. The additive noise injected at the bottleneck improves tolerance to the residual codes produced by subtraction and supports decoding at extreme gains. As a result, latent subtraction can recover a source from a mixture when the codes of the interfering sources are available, and decoding remains consistent across interpolation laws absent from training. The resulting latent structure also benefits conditional rectified flow: its velocity field is easier to regress, arithmetic remains effective on generated codes, and high-quality outputs are reached with fewer sampling steps.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio autoencoders, representation learning, latent arithmetic, generative audio

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xME3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a novel loss that can supervise positive-weight combinations of two latent codes thoughout their mixing cone. It is shown that latent substraction can recover a source from a mixture when the codes of the interfering sources are available. It is also shown that the resulting latent structure also benefits several aspects of conditional rectified flow. It is a strong paper, well written despite a lenghty and far too detailed appendix.

Critical strengths

The paper has several strengths:

  • the proposed approach is novel and a clear extension of previous work
  • the method presentation is detailed, well substantiated and clear
  • the experimental plan is solid and diverse, with appropriate and reimplemented baselines
Critical weaknesses

Despite its strengths, the paper suffers from some weaknesses:

  • it includes a very lengthy appendix that contains many details, both methodological/theoretical and experimental (> 30 pages of appendix). Obviously, this appendix cannot be read in details during reviewing. Even if it may contain only related and interesting information, a more focused appendix on the major aspects would have been more appropriate.
  • The experimental settings are not totally clear, and in particular considering the training dataset (it seems that there is only a vague statement in the appendix concerning the data used for training, without the possibility to easily reproduce the experiences). Are all results provided with a system trained on the exact same training set ?
  • All results are only given using objective metrics which are not well linked to perceptual quality. There are no perceptual tests to evaluate the quality of separation and there is no demo sites where separation examples would be provided.
  • The related work section is concise and clear, but it would be interesting to also discuss other types of work which exploit parallel encoding sections depending on the type of source, inducing also a kind of additive property for source separation (see for instance the work of Bie & al, "Learning Source Disentanglement in Neural Audio Codec," ICASSP 2025).
Questions for authors
  • Can you provide more details on the exact training dataset for the different experiments ?
  • Are specific experiments demonstrate the clear out-of-domain scenario (such as for instance trained on speech, tested on music/general audio )?
  • Can you provide perceptual results or comprehensive demo page ?
2 comments

We thank the reviewer for their thorough analysis and valuable suggestions. We address each concern below.

On the Length and Structure of the Appendix (Weakness 1)

We share the reviewer's perspective: a 30+ page supplement, motivated by full transparency and reproducibility, can become cumbersome during review. To ensure the appendix is concise, focused, and immediately accessible, we will implement the following structural revisions:

  1. Focused "Core Appendix": We will restructure the appendix by clearly delineating essential methodological information (architectural specifications vital for reproducibility, the core CV-VAE theoretical synthesis, and the dataset summary table) from secondary diagnostic ablations.
  2. Condensed Derivations and Consolidated Tables: We will streamline intermediate mathematical steps in Appendix A while preserving core derivations, and merge highly granular tables (e.g., combining per-condition denoising grids and exhaustive per-stem breakdowns into compact summary tables).
  3. Structured Navigation Roadmap: We will add an upfront "Appendix Roadmap" with an explicit cross-referencing index mapping each main-text experiment to its corresponding supplementary sections.

This restructuring will reduce the page count while preserving full scientific rigor.

On the reproducibility of the experimental settings (w2, q1 and q2)

Yes, all models we trained (every autoencoder variant and their downstream flow models) were trained on the exact same training dataset, using identical preprocessing, architectures, training schedules, and random seeds. Furthermore, our headline separation and interpolation benchmarks were deliberately evaluated on datasets never used in training (MUSDB18-HQ and LJSpeech) to test zero-shot algebraic generalization. Below is a concise summary of the data split across training and evaluation:

Phase / ExperimentDataset(s)Split / DomainPurpose & Conditions
All AE Models Training (16 kHz & 44.1 kHz fine-tuning)VCTK (speech) + FSD50K (events) + FMA Small (music)Training Split (shared across all baselines)2s segments, batch size 64, 300 epochs. Identical for Minimal, Soft-Norm, Torres/Implicit, Trig, and Universal.
Rectified Flow Prior Training (Sec. 4.4)VCTK + FSD50K + FMA SmallTraining Split (synthetic mixtures)50,000 speech + noise/music mixtures. Shared identically across all four latent spaces (univ, softnorm, Implicit, Trig Mixup).
Exp. 1: Additivity & Geometry (Sec. 4.1)VCTK + FSD50K + FMA SmallIn-Domain Validation (held-out)1,000 paired clips strictly disjoint from training.
Exp. 2: Zero-Shot Stem Separation (Sec. 4.2)MUSDB18-HQ (Rafii et al., 2017)Unseen / Zero-Shot (music)100 multi-tracks (383 valid stem trials) never seen during training.
Exp. 2: Speech Denoising Extrapolation (App. D.2)LJSpeech + Synthetic NoisesUnseen / Zero-Shot (speech)10s continuous clips (5× training duration) with 5 synthetic noise types across 2 SNRs.
Exp. 3: Continuous Interpolation Laws (Sec. 4.3)MUSDB18-HQUnseen / Zero-Shot (music)600 stem pairs evaluated across 7 continuous mixing paths.
Exp. 4: Trajectory Straightness (Sec. 4.4)VCTK + FSD50K + FMA SmallIn-Domain Test (held-out)10% stems withheld from AE training (isolates flow ODE geometry without off-domain AE artifacts).

Reproducibility & Manuscript Revisions:

  • Full dataset manifests, preprocessing scripts, and training/evaluation pipelines are provided in the anonymous repository linked in Section 1.
  • In the revised manuscript, we have incorporated a clear summary table into Section 4 (Experiments) to make the dataset provenance and zero-shot test conditions immediately apparent in the main text.

On Perceptual Evaluation (Weakness 3 & Question 3)

We fully agree that listening tests are essential for evaluating audio quality. To this end, we have developed an interactive companion demo page where reviewers and readers can perceptually evaluate our proposed model across diverse configurations and experimental tasks. The listening experience takes approximately 5 to 10 minutes and is accessible at this link.

On Modular/Disentangled Codecs with Parallel Sub-Codebooks (Weakness 4)

We thank the reviewer for pointing out this relevant line of work; contrasting it with our approach strengthens the positioning of our paper.

Specifically, Bie et al. (ICASSP 2025, SD-Codec) retain a unified encoder–decoder backbone (based on DAC) but partition the discrete bottleneck into parallel domain-specific RVQ sub-codebooks assigned to predetermined semantic categories (speech, music, and sound effects), while related modular codecs employ parallel encoder branches or class-specific decoders. While effective for domain-specific tasks with fixed source classes, this paradigm differs fundamentally from our approach and incurs notable trade-offs:

  1. Category and Stem Supervision: SD-Codec requires clean, isolated stems for each fixed semantic category during training to supervise branch specialization (Ltot=∑sLg(x^s,xs)\mathcal{L}_{\text{tot}} = \sum_{s} \mathcal{L}_g(\hat{x}_s, x_s)), and cannot separate or compose arbitrary sources within the same category (e.g., two musical instruments or two speakers) or outside its predefined taxonomy.
  2. Discrete Unit-Weight Sums vs. Continuous Cone Homomorphism: Although SD-Codec trains its decoder on equal-weight sums of quantized domain embeddings (z^mix=z^speech+z^music+z^sfx\hat{z}_{\text{mix}} = \hat{z}_{\text{speech}} + \hat{z}_{\text{music}} + \hat{z}_{\text{sfx}}) at fixed LUFS loudness targets, discrete RVQ representations do not support continuous scalar rescaling (aza z) or continuous positive-weight combinations across arbitrary mixing angles θ∈[0,π/2]\theta \in [0, \pi/2] and gains a≥0a \ge 0. Indeed, for source separation, SD-Codec does not rely on direct waveform decoding of latent arithmetic, instead applying post-hoc STFT magnitude masking with the noisy mixture phase. By contrast, LUMix\mathcal{L}_{\text{UMix}} enforces continuous decoder homomorphism across the entire cone C(z1,z2)\mathcal{C}(z_1, z_2), directly enabling arbitrary crossfading laws (Sec. 4.3) and pure waveform-level latent subtraction without masks (Sec. 4.2).
  3. Bitrate Scaling and Generative Bottleneck Complexity: Partitioning the latent space into KK domain-specific RVQ streams multiplies the representation bitrate by KK (e.g., 3×3\times bitrate for three-source mixtures in SD-Codec) and fragments the bottleneck into coupled discrete hierarchies that are cumbersome for downstream continuous generative priors.

In contrast, our framework preserves a single, unified, source-agnostic continuous latent space of fixed dimensionality with no specialized sub-networks. Linearity is induced as an intrinsic geometric property of the decoder's inversion manifold via self-supervised random pairing without clean stems or category labels, natively supporting arbitrary mixing ratios, continuous crossfading, and straight-trajectory Rectified Flow generation (Sec. 4.4).

We have incorporated this discussion into Section 2 (Related Work) of the revised manuscript.

Reviewer_kN6kOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes the Universal Mixup Loss (LUMixL{UMix}), a training objective for continuous audio autoencoders that supervises positive-weight linear combinations of two latent codes (zmix=w1z1+w2z2z{mix} = w1 z1 + w2 z2 with w1,w2≥0w1, w2 \ge 0, sampled in polar coordinates) so that the learned latent space preserves additive mixture structure for downstream tasks such as latent-space source separation.

Critical strengths

None

Critical weaknesses
  1. Stripped of the heavy mathematical jargon, this paper simply tries to learn a latent space with positive, additive NMF-like properties and then use those learned latent representations as targets to train a source separation model. This idea is not new at all, and the authors fail to even mention prior studies that have done the exact same thing, specifically Two-Step Sound Source Separation [A] and the follow-up work by Jiang [B] on vector-quantized latent source separation. Those works have already demonstrated that one can learn non-negative or structured latent spaces that preserve all source characteristics and train a neural separation model directly in that latent space.

  2. The empirical reporting in the separation tables has serious issues. First, reporting a second decimal point in all SNR/SI-SDR tables shows a disconnect from what is a statistically or perceptually significant difference in the 7 to 10 dB regime (where distinguishing even a 1 dB difference is doubtful, let alone 0.01 dB). Second, Table 3 and Table 26 show questionable behavior across Euler steps (where increasing steps yields only marginal improvement or even degrades single-draw performance), and the reported standard deviations of ±0.02\pm 0.02 to ±0.04\pm 0.04 dB are unreasonably small for audio source separation models, pointing to a flawed variance calculation or a bug in the evaluation code.

[A] Tzinis, E., Venkataramani, S., Wang, Z., Subakan, C. and Smaragdis, P., 2020, May. Two-step sound source separation: Training on learned latent targets. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 31-35). IEEE.

[B] Jiang, X., 2024. Learning Discrete Latent Representations for Audio Source Separation. Master's thesis, University of Illinois Urbana-Champaign. https://www.ideals.illinois.edu/items/132083

Questions for authors

How does your two-step latent separation pipeline (Section 4.4) compare conceptually and empirically against established two-step source separation methods that learn non-negative or structured autoencoder latent targets [A, B, C] on the same dataset and parameter budget? Is there any difference between [A, B] and this proposed work except of the fact that the authors use a different encoder / decoder architecture?

[A] Tzinis, E., Venkataramani, S., Wang, Z., Subakan, C. and Smaragdis, P., 2020, May. Two-step sound source separation: Training on learned latent targets. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 31-35). IEEE.

[B] Jiang, X., 2024. Learning Discrete Latent Representations for Audio Source Separation. Master's thesis, University of Illinois Urbana-Champaign. https://www.ideals.illinois.edu/items/132083

7 comments
AuthorsOpenReview ↗

On the Scope of this Work and Fundamental Differences with [A, B] (Weakness 1 & Question 1) - Part I

We thank the reviewer for the opportunity to clarify the purpose and scope of our work. We address each concern below.

On the Scope of this Work and Fundamental Differences with [A, B] (Weakness 1 & Question 1)

We respectfully clarify a fundamental misconception: we do not propose a two-step blind source separation pipeline, nor do we learn non-negative (NMF-like) or discrete VQ latent spaces (as [A] and [B]).

Our work introduces the Universal Mixup Loss (LUMix\mathcal{L}_{\text{UMix}}), a self-supervised geometric regularization objective for general-purpose, continuous and compressed neural audio autoencoders. It enforces decoder homomorphism over the positive-weight mixing cone C(z1,z2)={w1z1+w2z2∣w1,w2≥0}\mathcal{C}(\mathbf{z}_1, \mathbf{z}_2) = \{w_1\mathbf{z}_1 + w_2\mathbf{z}_2 \mid w_1, w_2 \ge 0\}, so that linear vector algebra in the compressed latent space commutes with waveform superposition: Dϕ(w1z1+w2z2)≈w1x1+w2x2\mathcal{D}_\phi(w_1\mathbf{z}_1 + w_2\mathbf{z}_2) \approx w_1 x_1 + w_2 x_2.

AuthorsOpenReview ↗

On the Scope of this Work and Fundamental Differences with [A, B] (Weakness 1 & Question 1) - Part II

1. No "NMF-like" or Non-Negative Latent Space

The claim that we «learn a latent space with positive, additive NMF-like properties» is mathematically incorrect:

  • Unconstrained, zero-mean continuous latents: z∈RP\mathbf{z} \in \mathbb{R}^P are real-valued tensors, unconstrained in sign, and regularized toward a zero-mean, unit-variance Gaussian prior N(0,I)\mathcal{N}(\mathbf{0}, \mathbf{I}) (Sec. 3.1, 3.4, App. A.4).
  • Meaning of "Positive-Weight": the term in our title refers exclusively to the scalar acoustic mixing coefficients (w1,w2≥0w_1, w_2 \ge 0, non-negative fader gains in Eq. 1), not to the sign of the latent representations.

2. Paradigm Differences and Comparison with [A] and [B]

Works [A] and [B] (Note: [B] is X. Jiang, "Vector-Quantized Speech Separation", UIUC Senior Thesis, Dec. 2021, archived on IDEALS in 2024) address a completely different problem (task-specific, discriminative two-step blind separation) under assumptions incompatible with continuous generative modeling and latent vector algebra:

DimensionTzinis et al. [A] (ICASSP 2020)Jiang [B] (UIUC Thesis, 2021/2024)Ours (Universal Mixup Loss)
Primary ObjectiveTask-specific two-step blind source separation.Task-specific two-step speech separation over discrete VQ codebooks.Algebraic latent interface (decoder homomorphism) for continuous codecs and generative flow priors (Sec. 3, 4.4).
Latent Topology & SignNon-negative via ReLU (Sec. 3.3.1).Discrete Product-VQ codebooks, not linearly independent (Sec. 5.5.1).Unconstrained real-valued (z∈RP\mathbf{z} \in \mathbb{R}^P), scale-sensitive (no final LayerNorm), zero-mean Gaussian prior (Sec. 3.1, App. C.2).
Dimensionality / CompressionOvercomplete expansion (3.2×3.2\times to 25.6×25.6\times), no temporal compression (Sec. 3.3.1).Overcomplete pre-VQ expansion (51.2×51.2\times), quantized into K=4K=4 discrete indices (Sec. 5.3).Compressed bottleneck (4×4\times to 32×32\times reduction, App. B.1, C.3).
Step 1 (AE) SupervisionSupervised oracle separation: needs clean stems and their mixtures to train softmax masks (Sec. 2.1).Clean-source reconstruction with VQ losses; no mixing constraints (Sec. 3.4).Self-supervised random pairing in polar coordinates (a,θ)(a, \theta); no stems or oracle masks (Sec. 3.3).
Separation / Editing MechanismMultiplicative masking by a supervised TDCN separator (Sec. 2.2, 3.3.2).Codebook index classification via a supervised VQ-Separator (Sec. 4.1 to 4.3).Elementary vector algebra: zero-shot subtraction (Sec. 4.2), continuous crossfading (Sec. 4.3), latent averaging (latavg, Sec. 4.4).
Role of Separation in EvaluationCore goal.Core goal.Geometric diagnostic probe: zero-shot subtraction (Sec. 4.2) and Rectified Flow ODE straightening with conditional expectation decoding (Sec. 4.4).

3. Why [A] and [B] Cannot Be Used in Our Setting (and What Novel Questions We Address)

Beyond the differences summarized in the table above, the incompatibility is structural, not incidental. Our autoencoder contains no separation machinery: there is no separation head, no mask estimation, and no codebook classifier. In our work, the source separation experiments serve only as probes of pseudo-linearity and of how the design affects learning a Rectified Flow model (Sections 4.2 and 4.4). Plugging in the representations of [A] or [B] would not enhance this pipeline; it would remove the very property under study.

More fundamentally, neither [A] nor [B] studies how to enforce algebraic operations in the latent space: no objective constrains their decoders so that latent combinations decode to the corresponding waveform combinations, whereas our loss imposes exactly this decoder homomorphism over the positive mixing cone. Furthermore, generative Diffusion/Flow-Matching transformers require low-frame-rate, compressed, unconstrained Gaussian latents, a regime that an overcomplete non-negative basis built for multiplicative masking cannot serve efficiently.

Within this continuous, unconstrained setting, our study addresses research questions entirely absent from the two-step separation literature: how the polar vs. Cartesian parameterization of the mixing loss governs angular isotropy and prevents gain errors in attenuated regimes (App. A.1, A.3, A.7); how flat decodability can be guaranteed across unseen continuous crossfading laws (Sec. 4.3); and how conical regularity straightens Rectified Flow ODE trajectories (yielding a 9.8×9.8\times to 14.0×14.0\times inference speedup) while making both the conditional expectation E[zvoice∣zmix]\mathbb{E}[\mathbf{z}_{\text{voice}}\mid\mathbf{z}_{\text{mix}}] and the latent average of generated codes (latavg) directly decodable (Sec. 4.4). We have cited [A] in Section 2 to contextualize two-step separation.

On the Theoretical Formulation and Connections to CV-VAE / RAE (Weakness 1)

The mathematical formulation is not ornamental: it grounds our design choices in established principles of representation learning.

  1. Connection to CV-VAE and RAE: Section 3.4 and Appendix A.4 show that the soft-norm bottleneck lies at the theoretical intersection of a Constant-Variance VAE and a Regularized Autoencoder (RAE) [1], sidestepping known ELBO training pathologies while preserving sensitivity to acoustic gain.
  2. Tolerance Basin for Latent Algebra: Equations (20) to (22) prove that constant-variance stochastic injection establishes an isotropic noise floor strictly independent of the mixing angle θ\theta, so subtraction residuals zmix−zother\mathbf{z}_{\text{mix}} - \mathbf{z}_{\text{other}} decode cleanly without collapse (App. A.5).
  3. Polar Gradient Analysis: Appendix A.1 proves parameterization invariance between Cartesian and polar coordinates, with the polar formulation uniquely yielding native angular isotropy and an uncoupled radial curriculum by construction.

We will streamline intermediate formal derivations in the Appendix to enhance readability, while preserving the foundational theoretical links to CV-VAE and RAE.

On SI-SDR Precision and Cross-Seed Variance Reporting (Weakness 2)

We believe this concern stems from a misunderstanding of the reported statistical quantities: there is no bug or flawed variance calculation in our evaluation code.

1. Two-Decimal Precision Convention in Audio Evaluation

Two decimal places for aggregate SI-SDR and SNR metrics are standard practice in the audio source separation literature (Band-Split RNN [4], Demucs [2], Lin-CAE [5]) and in [B] itself. While a 0.01 dB difference is not perceptually distinguishable on a single clip, two decimals differentiate aggregate means over hundreds of evaluation instances and avoid rounding ambiguities in statistical comparisons.

2. Cross-Seed vs. Cross-Clip Variance (No Evaluation Bug)

The claim that ±0.02\pm 0.02 to ±0.14 dB\pm 0.14\text{ dB} are "unreasonably small" conflates two different statistical quantities:

  • Cross-clip variance (acoustic diversity): metric spread across diverse audio clips spans several dB (Tables 2 and 12).
  • Cross-seed variance (training stability): the contested values in Tables 3 and 26 are strictly the standard deviation across 3 independent training seeds of the DiT model on the exact same aggregate test set (Table 3 caption, App. F.3). They do not measure acoustic variance across clips; they demonstrate the optimization stability and reproducibility of the learned velocity field across random initializations.

We have made this distinction prominent in Section 4.4 and the relevant table captions to prevent any ambiguity regarding the reported statistics.

Reviewer_89zUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper Summary

This paper studies how to make the latent space of audio autoencoders support linear operations for tasks such as audio mixing, interpolation, and source separation by subtracting known interference codes. The authors propose Universal Mixup Loss, which trains the decoder using different positive combinations of two latent codes instead of only fixed mixing ratios. The method also uses a soft-norm bottleneck. Experiments show better latent mixing, source recovery by subtracting known interference codes, and faster sampling with rectified flow.

Critical strengths

The motivation is clear and the method is easy to follow. Good reconstruction alone does not mean that mixing latent codes will produce the correct audio mixture. The proposed loss addresses this directly by training on different mixing ratios and gains. It adds some training cost, but no extra parameters or computation at inference.

The experiments cover the main design choices quite well. I appreciate the comparisons with implicit linearity and Trig Mixup using the same backbone, rather than only comparing against pretrained codecs. The results on attenuated mixing paths are particularly useful, since they show cases where the simpler objectives work less well. The noise ablation also helps explain the design: better isolated reconstruction does not necessarily lead to better latent subtraction.

The rectified-flow results particularly interesting. The benefit is not limited to mixing codes obtained from the encoder. The flow model trained on the proposed representation reaches a lower validation loss, and the improvement in audio quality is especially large at one or two sampling steps. This is useful when the sampling budget is small, even though the advantage becomes much smaller at higher step counts. Also, averaging generated latent codes still produces usable audio, while the same operation performs poorly with Soft-Norm. This suggests that the learned structure remains useful when another model generates the codes, which gives the method a more direct connection to downstream audio generation.

Critical weaknesses

The sampling ablation does not fully isolate the benefit of uniform angular sampling. Table 13 shows clear improvements on attenuated mixing paths, but in Appendix A.3, the Cartesian and polar samplers differ in both angular and radial distributions. The authors also attribute some differences to radial exposure. A more direct comparison would keep the gain distribution and annealing schedule fixed, while comparing uniform and non-uniform sampling over the same range of mixing angles. Since the sampling scheme is the main new part of the loss, this distinction matters for understanding which design choice actually produces the improvement.

The flow experiments are missing the most relevant baselines for evaluating the proposed loss. Section 4.4 only compares Universal with Soft-Norm, although the matched Implicit and Trig Mixup models already achieve similar decoder commutation errors in Table 14. Evaluating the same flow model on these representations would help determine whether the few-step gains and successful averaging of generated codes come from mixing supervision in general or specifically from the full-cone objective. These downstream results are an important part of the paper’s contribution, but the current comparisons do not establish the advantage of the proposed loss over existing mixing objectives.

Questions for authors

Tables 13 and 14 already compare Universal, Implicit, and Trig Mixup at the autoencoder level. Can the authors extend this comparison to the downstream rectified-flow experiment, training a separate flow model on each representation with the same architecture and training budget? In particular, does Universal outperform the other two objectives in few-step generation and latent averaging? This would clarify whether the proposed loss brings additional downstream benefits, rather than improvements from mixing supervision in general.

How much inference speedup does the proposed representation provide at comparable audio quality when using a single generated sample? Table 26 uses six-draw waveform averaging, while the single-draw results in Table 27 use a different validation subset. Can the authors report single-draw quality and total inference time, including decoding, on the same held-out set? This would make the practical benefit clearer, since fewer sampling steps do not directly establish the end-to-end speedup.

2 comments

We thank the reviewer for their thorough and constructive feedback. Below, we address the reviewer's questions and weaknesses.

On the Sampling Ablation and Isolating Uniform Angular Sampling (Weakness 1)

As the Reviewer correctly pointed us, the Cartesian-polar comparison in Appendix A.3 does not isolate the angular axis. To provide the exact controlled comparison requested, we trained a new model with a non-uniform continuous angular density over the same range [0,π/2][0, \pi/2], sharing the identical radial schedule of Universal:

  • Uniform vs. non-uniform angle (Universal vs. Non-uniform): identical gain range (s∈[0,3]s \in [0, 3]), cosine annealing, architecture, data, and budget; the only varying factor is p(θ)p(\theta): uniform for Universal, a symmetric Beta(3,3)\mathrm{Beta}(3,3) peaked at π/4\pi/4 for the Non-uniform model (79% of angular mass in [π/8,3π/8][\pi/8, 3\pi/8], vs. 50% for uniform). This is the direct comparison the reviewer requested.
  • Continuous vs. discrete angle (Non-uniform vs. Implicit): same radial schedule and range, but Implicit supervises only two discrete rays (θ∈{0,π/4}\theta \in \{0, \pi/4\}) with zero coverage of intermediate angles.
  • Same angle, different radius (Trig mixup): keeps a≡1a \equiv 1 fixed with uniform angular coverage (no annealing); it degrades in the attenuated interior (a<0.71a < 0.71), confirming the complementarity of angular coverage and radial curriculum.
ModelAngular Distribution p(θ)p(\theta)Gain ScheduleIsolated VariablePath Mean SI-SDR
Implicit (Torres)Discrete (θ∈{0,π/4}\theta \in \{0, \pi/4\})Annealed [0,3]→(1,1)[0, 3] \to (1, 1) (Identical)Same gain, zero intermediate coverage0.47 dB
Trig mixupContinuous uniform [0,π/2][0, \pi/2]Constant a≡1a \equiv 1 (no annealing)Same angle, no radial curriculum9.39 dB
Non-uniform continuousBeta(3,3)\mathrm{Beta}(3,3) on [0,π/2][0, \pi/2], peaked at π/4\pi/4Annealed [0,3]→(1,1)[0, 3] \to (1, 1) (Identical)Same gain, non-uniform angle, the requested comparison12.70 dB
Universal (Ours)Continuous uniform [0,π/2][0, \pi/2]Annealed [0,3]→(1,1)[0, 3] \to (1, 1)Full conical reference14.21 dB

All rows report the Overall Path Metric (mean SI-SDR over the five held-out interpolation laws, 600 MUSDB18-HQ stem pairs, the protocol of Table 13).

The decomposition shows the following results:

  1. Continuous coverage is the dominant factor: moving from discrete rays to any continuous coverage recovers +12.23 dB, the decoder must see intermediate angles to decode them.
  2. Uniform density adds a measurable gain: moving from non-uniform continuous to uniform continuous yields +1.51 dB. The effect is most pronounced on the dip law of Table 13, the deepest interior trajectory where aa dips to 0.129: Universal reaches 13.96 dB vs. 8.81 dB for the non-uniform model, a 5.15 dB gap. There, the non-uniform model's reduced exposure near the axes (10% vs. 25% of angular mass per side) impairs decoding of very quiet mixtures that start and end at extreme angles.

We have integrated this controlled angular ablation into the revised manuscript; we thank the reviewer for the constructive feedback.

On the Trig-mixup and Implicit FM Experiment (Weakness 2 & Question 1)

We trained the exact same DiT architecture (16.5M parameters, 3 seeds, matched data and compute budget) on the latent spaces of Implicit (Torres et al.) and Trig Mixup.

The table below extends the evaluations from Table 3 and Table 26:

Estimatoruniv (Ours)Implicit (Torres)Trig Mixupsoftnorm
FM wavavg, N=1N=19.07 ± 0.048.73 ± 0.06 (p=1.4×10−4p=1.4\times 10^{-4})9.11 ± 0.04 (p=0.079p=0.079)0.65 ± 0.06
FM wavavg, N=2N=29.64 ± 0.109.76 ± 0.02 (p=4.0×10−4p=4.0\times 10^{-4})9.62 ± 0.06 (p=0.49p=0.49)6.91 ± 0.39
FM wavavg, N=4N=49.95 ± 0.0710.01 ± 0.01 (p=0.11p=0.11)9.94 ± 0.06 (p=0.87p=0.87)9.70 ± 0.11
FM wavavg, N=32N=329.98 ± 0.039.98 ± 0.05 (p=0.96p=0.96)9.99 ± 0.04 (p=0.75p=0.75)10.19 ± 0.08
FM latavg (K=6,N=32K=6, N=32)9.96 ± 0.038.87 ± 0.08 (p=3.7×10−9p=3.7\times 10^{-9})9.93 ± 0.03 (p=0.36p=0.36)0.54 ± 0.04

These results address the reviewer's questions directly:

  1. Few-Step Sampling (N=1N=1): Local additivity straightens the velocity field across all mixup-supervised models (∼9 dB\sim 9\text{ dB} at N=1N=1 vs. 0.65 dB0.65\text{ dB} for Soft-Norm). Still, Universal outperforms Implicit at N=1N=1 (+0.34 dB).
  2. Latent Code Averaging (latavg): Universal outperforms Implicit by +1.09 dB. Averaging K=6K=6 generated codes produces continuous intermediate angles that fall out of distribution for Implicit's two discrete supervised rays (θ∈{0,π/4}\theta \in \{0, \pi/4\}), degrading decoding.
  3. Relation to Trig Mixup: Trig Mixup matches Universal on latavg (p=0.36p=0.36) because both provide continuous angular supervision on [0,π/2][0, \pi/2]. However, as established in the paper, Trig Mixup collapses when trajectories deviate from the unit shell a≡1a \equiv 1 (Table 13: 0.00 dB0.00\text{ dB} on attenuated paths) and underperforms on algebraic subtraction (Table 19: 25× higher spectral error).

While linear supervision in general straightens flow trajectories, continuous conical coverage (Universal) is uniquely required to preserve latent algebra on generated codes (latavg) while remaining robust across non-unit gains. We have integrated this comparison into Section 4.4 in the revision.

On the Inference Speedup in Single-Draw Evaluation (Question 2)

For a single generated sample (K=1K=1), our conical representation delivers an effective end-to-end speedup between 9.8× and 14.0× at matched or superior audio quality compared to the standard baseline.

Following the reviewer's request, we report audio quality (SI-SDR) alongside measured end-to-end wall-clock latency (Ttotal=Tdit+TdecT_{\text{total}} = T_{\text{dit}} + T_{\text{dec}}, benchmarked on a single NVIDIA L40 GPU at batch=1 and batch=32) evaluated on the exact same held-out set of 128 clips used for Table 26:

Single draw (K=1K=1): Quality and end-to-end latency per 2 s clip (held-out test set n=128n=128)

Steps NNuniv (Ours)ImplicitTrig MixupsoftnormTditT_{\text{dit}} (b1)TdecT_{\text{dec}} (b1)TtotalT_{\text{total}} (b1)TtotalT_{\text{total}} (b32/clip)Speedup vs N=32N=32 (b1)
18.37 dB7.93 dB8.52 dB0.26 dB1.61 ms1.99 ms3.60 ms0.665 ms14.0×
29.13 dB9.09 dB8.90 dB5.57 dB3.14 ms1.99 ms5.12 ms0.862 ms9.8×
48.99 dB8.90 dB8.89 dB8.11 dB6.25 ms1.99 ms8.24 ms1.260 ms6.1×
88.62 dB8.55 dB8.65 dB8.46 dB12.19 ms1.99 ms14.18 ms2.027 ms3.5×
168.25 dB8.18 dB8.37 dB8.29 dB24.26 ms1.99 ms26.24 ms3.599 ms1.9×
327.99 dB7.93 dB8.13 dB8.15 dB48.29 ms1.99 ms50.28 ms6.820 ms1.0×

We report and comment the main results:

  1. Effective Speedup at Comparable Quality: softnorm needs N=8N=8 to 3232 steps (8.15 to 8.46 dB in 14.2 to 50.3 ms) to reach convergence, while univ attains higher fidelity at N=2N=2 (9.13 dB, +0.98 dB) in 5.12 ms: an effective speedup of 9.8× (batch 1) and 7.9× (batch 32), rising to 14.0× at N=1N=1 (8.37 dB, 3.60 ms vs. 50.28 ms).
  2. Decoder Overhead (TdecT_{\text{dec}}): decoder execution is a fixed overhead (1.99 ms at batch 1); ODE steps dominate total runtime for N≥2N\ge 2, so fewer steps translate directly into lower latency.
  3. Additional Multi-Draw Savings (latavg): in multi-sample workflows (K=6K=6 draws), algebraic validity allows averaging directly in latent space (latavg): at N≤4N\le 4 this cuts decoder passes from 6 to 1, further reducing end-to-end latency by 20% to 59% vs. waveform averaging (e.g., 21.58 ms to 11.64 ms at N=1N=1, batch 1).

We have incorporated this latency analysis and the matched single-draw benchmark into Appendix F of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.