acceptodds
Under review as a conference paper at ICLR 2027

Fourier is Frontier: Frequency-Aware Autoencoding for High-Fidelity Music Reconstruction

Abstract

Continuous-latent audio autoencoders form the backbone of latent music generators, and their reconstruction fidelity limits the acoustic detail available to downstream generators. At the compression rates these pipelines require, decoders commonly exhibit three recurring failure modes: high-frequency loss, phase incoherence, and stereo-image collapse. These shortcomings in high-fidelity generation share a structural root: waveform autoencoders, previously regarded as the frontier design, lack an explicit frequency axis, leaving no handle for targeted per-band correction. Among five matched-budget representations in our controlled study, the complex STFT achieves the lowest full-band and high-frequency spectral distances, providing direct access to magnitude and phase at every frequency bin. Building on this finding, we present ear-VAE2 , a complex-spectral autoencoder that exploits the explicit frequency structure and introduces cross-channel interaction between the left and right channels. First, Spec-SnakeBeta exploits the additional capacity of the frequency-domain representation by learning a periodic activation for each frequency bin, with frequency-dependent initialization in log-parameter space. This design overall outperforms the other activation variants in our controlled ablation, while using fewer parameters than the fully independent variant. Second, Duplex-Aware Refiner applies band-specific corrections to magnitude and phase. Its band allocation follows duplex theory of sound localization and corrects the decoder's reconstruction error. On the 546-track Song Describer Dataset, ear-VAE2 achieves the best point estimates on five of seven reconstruction metrics and matches the best stereo-coherence score. Adding the Duplex-Aware Refiner module further reduces Mel Distance by 19.4%, and its duplex band allocation uses ≈45% fewer residual-output dimensions than the Unconstrained Refiner version. Beyond this efficiency, the banded allocation also lowers spectral distances and targeted spatial-cue error metrics, and it receives higher mean paired ratings from professional mixing and mastering engineers. The downstream generator using ear-VAE2 latents achieves better point estimates on all 12 automatic downstream metrics. Demo page is available at: https://earvae2-2026.github.io/.

Primary area:
generative models
Keywords:
Audio representation learning, Variational autoencoders, Music reconstruction, Music generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CJZHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ear-VAE2, a 48 kHz continuous-latent music autoencoder based on the complex STFT, to address high-frequency loss, phase incoherence, and stereo-image collapse in waveform-domain audio VAEs. It introduces Spec-SnakeBeta, a frequency-dependent periodic activation function, and Duplex-Aware Refiner, which applies band-specific magnitude and phase corrections based on duplex theory of sound localization. The latter reduces residual-output dimensions by approximately 45% compared with an unconstrained refiner. Experiments show that ear-VAE2 achieves the best point estimates on five of seven reconstruction metrics, while its latents improve point estimates across all 12 downstream music generation metrics.

Critical strengths
  1. They conduct a thorough investigation on five different audio representation to identify the most suitable representation for wide-band spectral preservation. Based on this analysis, they propose a frequency-aware continuous audio VAE model operating on the complex STFT.
  2. Simple but effective Spec-SnakeBeta and Duplex-Aware Refiner to improve frequency-domain modeling and stereo reconstruction, respectively.
  3. Extensive experiments demonstrate improved reconstruction quality.
Critical weaknesses
  1. Unfair comparison with SOTA methods. The full ear-VAE2 model has ~692M parameters, and is additionally trained on 10,000 hours of unreleased, proprietary 48 kHz high-quality music. Moreover, the baselines in Tab 2 differ in sampling rates and latent frame rates (e.g., SAME-L uses only 10.8 Hz). These disparities make it difficult to attribute the reported gains to the proposed methods rather than increased model capacity, additional training data, or inconsistent compression settings.
  2. In Table 6, complex STFT exhibits substantially lower SI-SDR than Waveform Patch in the matched-budget study (−10.49 vs. 3.98 dB). Even the full ear-VAE2 achieves only 11.3 dB, below SAME-L (12.5 dB) and ear-VAE (12.4 dB). The paper does not sufficiently investigate whether this gap affects the perceptual quality of transients or phase-sensitive content.
  3. In Table 6, Waveform Patch outperforms Complex STFT on four of six metrics. Although the authors acknowledge this trade-off, the choice of complex STFT appears driven more by its compatibility with the proposed modules than by overall empirical superiority. Thus, the claim that “Fourier is Frontier” is overstated.
  4. Limited conceptual novelty. The complex-STFT backbone closely follows existing spectral codec architectures such as SpectroStream, while the band-specific magnitude/phase corrections draw on established principles from duplex theory and parametric stereo. Although Spec-SnakeBeta and the duplex-banded refiner are reasonable design choices, the overall contribution appears to be an effective integration of existing components and signal-processing priors rather than a fundamentally novel representation-learning approach.
Questions for authors

None

4 comments
AuthorsOpenReview ↗

Response to Reviewer CJZH

We thank Reviewer CJZH for the helpful comments. We have updated the revised manuscript with the controlled architecture ablation and expert listening evaluations.

Q1: On the Fairness of Comparisons with SOTA Methods

First, we added a controlled ablation in Table 9 (Appendix F.1) of the revised paper, training the external baseline architecture (LeVo 2 VAE (aligned)) on the exact same dataset with comparable model size (≈692M vs. 695M), 1920× compression, and 128-dim latent, varying only the model architecture. Under aligned conditions, εar-VAE2 outperforms LeVo 2 VAE (aligned) across spectral metrics (STFT 0.870 vs. 0.898, STFT_log 0.075 vs. 0.077, Mel 0.461 vs. 0.497, Mel_log 0.089 vs. 0.091), Spectral Pan Error (0.264 vs. 0.275), and expert listening scores (0.75 vs. 0.64). (Note on SI-SDR: the two SI-SDR discussions in our paper correspond to different regimes—SI-SDR is reliable within a moderate confidence interval such as in our small-scale operator ablations, whereas once full-scale reconstruction SI-SDR reaches 11–12+ dB, it becomes limited and distorted for quality evaluation.)

Second, we trained a smaller model variant (170.2M + 8.2M Refiner) exclusively on open-source data with a size comparable to open-source baselines (to be open-sourced soon). As shown below, this variant still outperforms open-source commercial baselines (SA-Open and LeVo 2) across all spectral, SI-SDR, and inter-channel coherence metrics—and even surpasses SAME-L on STFT, Mel-log1p, and STFT-log1p—validating our training recipe:

ModelSI-SDR ↑ (dB)Mel ↓STFT ↓Mel-log1p ↓STFT-log1p ↓Per-Ch Coh ↑CCPC (Inter-Ch Coh) ↑
SAME-L12.536 ± 0.4070.539 ± 0.0100.986 ± 0.0150.096 ± 0.0060.079 ± 0.0050.973 ± 0.0020.970 ± 0.003
SA-Open6.722 ± 0.3190.612 ± 0.0071.016 ± 0.0160.106 ± 0.0060.089 ± 0.0050.929 ± 0.0050.933 ± 0.004
LeVo 28.141 ± 0.3220.599 ± 0.0080.971 ± 0.0160.103 ± 0.0060.086 ± 0.0050.946 ± 0.0040.947 ± 0.004
Ours (Open-Source Small, w/o Refiner)9.438 ± 0.3520.773 ± 0.0061.031 ± 0.0140.1006 ± 0.00580.0846 ± 0.00490.954 ± 0.0040.966 ± 0.003
Ours (Open-Source Small, Full)8.872 ± 0.3660.551 ± 0.0050.934 ± 0.0120.0941 ± 0.00540.0789 ± 0.00460.943 ± 0.0040.953 ± 0.004

Q2: On Metric Performance of Waveform Patch Modeling and High-Frequency Underfitting

In complex music signals, SI-SDR and low-frequency STFT distance have clear limitations when a compact (~3M) model overfits energy-dominant low frequencies. Our goal at this scale is to investigate the model's ability to capture information across the full signal spectrum rather than low-frequency metric overfitting. As shown in Table 1 (§4.3) and Table 6 (Appendix A.2) of the revised paper, a finer frequency breakdown reveals that while Waveform Patch (Conv2D) fits low frequencies well (STFT_LF < 2 kHz = 0.986 vs. 1.137), it suffers from severe underfitting in high frequencies (STFT_HF > 8 kHz = 1.513, whereas Complex STFT achieves 1.178, a 22% reduction, and the best full-band STFT of 1.122 vs. 1.294). Furthermore, perceptual differences are pronounced in stereo imaging and high-frequency clarity. As reported in Table 1 & Table 6, human expert blind listening tests confirm that Complex STFT achieves the highest subjective score (0.40 vs. 0.35 for Waveform Patch, 0.26 for Waveform Conv1D, 0.21 for Magnitude-only, and 0.09 for Patch Transformer).

Q3: On Adapting the SpectroStream Backbone and Positioning Our Contributions

Our work does not focus on structural novelty in isolated generic backbone blocks; rather, within high-fidelity music reconstruction, we investigate foundational audio input-output representation modeling and introduce a series of targeted improvements built upon it. Specifically, we adopt Complex STFT as our starting point to achieve higher reconstruction quality. Complex STFT has not yet been widely adopted as a mainstream pipeline in continuous audio/music autoencoding, nor have psychoacoustic principles such as Duplex Theory been integrated into neural model training in this setting—giving our work clear frontier value and novelty in domain-knowledge transfer.

Dear Reviewer CJZH,

We hope you are doing well. As the discussion period is approaching its deadline on November 18, we would like to kindly follow up and ask whether you have had a chance to review our rebuttal.

We have provided additional evidence addressing your concerns, particularly the controlled comparison with an architecture-aligned baseline, the smaller model trained on open-source data, and expert listening evaluations supporting our choice of spectral representation.

We hope these additions have helped clarify the empirical contributions and motivation of our work. We would greatly appreciate any further feedback or questions before the discussion period ends.

Thank you again for your time and valuable suggestions.

Best regards,
The Authors

Dear Reviewer CJZH,

We hope you are doing well. As the discussion period is approaching its November 18 deadline, we wanted to briefly follow up on our previous response.

We sincerely hope that the additional experimental evidence, particularly the architecture-aligned baseline comparison, the smaller model trained exclusively on open-source data, and the expert listening evaluations, has adequately addressed your concerns regarding comparison fairness, representation choice, and the empirical contributions of our approach.

If you feel that these additional results have sufficiently addressed the concerns underlying your initial assessment, we would greatly appreciate it if you could reconsider the overall rating in light of the revised evidence and adjust it if you find it warranted.

We fully respect your independent judgment and sincerely appreciate your time and valuable feedback.

Best regards, The Authors

Reviewer_CJZHOpenReview ↗

Official Comment

I thank the authors for the detailed response, which addresses some of my concerns about the experiment results. I keep my original score of weak accept.

Reviewer_sCpVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes εar-VAE2, a complex-STFT-based variational autoencoder that compresses stereo music into 128-dimensional continuous latents at 25 Hz and reconstructs both magnitude and phase information. It introduces Spec-SnakeBeta, which learns frequency-specific nonlinear activations with parameters shared across feature channels. A Duplex-Aware Refiner further corrects the decoder’s output through phase-only refinement at low frequencies, joint magnitude-and-phase refinement at mid frequencies, and magnitude-only refinement at high frequencies. The model achieves the best point estimates on five of seven reconstruction metrics, with the refiner reducing mel distance by 19.4%. Ablation studies support the proposed components, and a blind listening test favors the Banded Refiner over its unconstrained counterpart. Downstream generation also scores higher than the LeVo 2 VAE on all 12 automatic evaluation dimensions.

Critical strengths
  1. Coherent motivation and model design

The model addresses high-frequency loss, phase incoherence, and stereo degradation through frequency-aware activations and band-specific refinement. Spec-SnakeBeta achieves the best mean on four of six ablation metrics while using approximately 128 times fewer activation parameters than the fully independent variant.

  1. Controlled representation comparison

The representation study matches model size, latent configuration, and training conditions, with complex STFT and waveform patches sharing the same Conv2D backbone. Complex STFT achieves a lower high-frequency STFT distance (1.178 versus 1.513), supporting the representation choice.

  1. Improvements across reconstruction metrics

The full model achieves the best point estimates on five of seven reconstruction metrics and ties for the best CCPC across 546 tracks. Adding the refiner reduces Mel Distance by 19.4%, with evaluation covering waveform, spectral, and stereo fidelity.

  1. Downstream and perceptual validation

Downstream generation scores higher than the LeVo 2 VAE on all 12 automatic evaluation dimensions. A blind listening test with ten professional engineers also favors the Banded Refiner over its unconstrained counterpart, with mean ratings of 0.75 versus 0.66.

Critical weaknesses
  1. Insufficient isolation of the duplex-guided allocation

The Banded and Unconstrained Refiners predict 532 and 962 residual dimensions per channel, respectively, making it difficult to separate the effect of duplex-guided allocation from potential regularization benefits. Comparisons with alternative allocations under the same output budget would better establish the contribution of the proposed frequency partitioning (Table 3; Appendix K).

  1. Limited comparability with external baselines

The external baselines use their native configurations, without matching sample rate, latent rate, model size, or training data (Table 2). Although internal ablations provide supporting evidence, the final comparison primarily demonstrates system-level competitiveness rather than isolating the advantages of the proposed architecture.

  1. Representation trade-offs and limited scaling evidence

Waveform patches outperform complex STFT on SI-SDR, CCPC, Mel Distance, and low-frequency STFT Distance, indicating trade-offs in the spectral representation (Table 6). Moreover, the representation study uses approximately 3M-parameter models, leaving it unclear whether the same rankings hold at the final system’s approximately 692M-parameter scale (Appendices A and F).

Questions for authors
  1. Does the improvement come from duplex-guided allocation or from restricting the refiner’s output dimensionality?

Please compare the proposed allocation against alternative masks with the same 532 output dimensions, including frequency-permuted masks that preserve the numbers of magnitude and phase outputs, using the same frozen decoder and training protocol.

  1. How sensitive are the results to the 1.5 kHz and 4 kHz boundaries?

A boundary-sensitivity study, ideally complemented by a learned or softly weighted allocation under a comparable parameter budget, would clarify whether the chosen partition is particularly effective or whether other frequency-dependent allocations perform similarly.

6 comments
AuthorsOpenReview ↗

Response to Reviewer sCpV

We thank Reviewer sCpV for the constructive feedback. All requested controlled ablations and expert listening evaluations have been incorporated into the revised manuscript.

Q1: On "Insufficient Isolation of Duplex-Guided Allocation" and Dimension Alignment

Regarding the dimension alignment concern between the Banded Refiner and the Unconstrained baseline: our results demonstrate that Banded Refiner achieves superior perceptual quality and objective metrics while using a ~45% smaller output dimension per channel (532 vs. 962) and lower information capacity, which further reinforces the benefit of the Duplex prior—concentrating capacity where perception needs it most.

To further isolate the dimension budget, we added a 532-dim budget-aligned unconstrained baseline (Matched-Dim Unconstrained) in Table 3 (§4.5) of the revised paper. Our Banded refiner (1.5 / 4 kHz, 532 dims/ch, 8.20M params, 4.18 GFLOPs) consistently outperforms both full Unconstrained (962 dims/ch, 8.31M params, 4.24 GFLOPs) and Matched-Dim Unconstrained (532 dims/ch): achieving lower Mel Distance (0.658 vs. 0.705 and 0.714), lower STFT Distance (1.019 vs. 1.120 and 1.122), lower binaural errors (Low IPD 0.7963 vs. 0.8155/0.8264 rad; Low ILD 0.1690 vs. 0.1840/0.1878 dB; High ILD 0.9804 vs. 1.1495/1.1536 dB), and a higher expert listening score (0.75 vs. 0.66 and 0.63).

Q2: On Comparability with External Baselines

To address the comparability concern, we added a controlled evaluation in Table 9 (Appendix F.1) of the revised paper, where we trained the external baseline architecture (LeVo 2 VAE (aligned)) under the same training dataset, comparable model size (≈692M vs. 695M), identical 1920× temporal compression, and 128-dim latent. Under this strictly aligned environment, εar-VAE2 maintains clear advantages in both objective metrics and human expert listening scores: STFT Distance 0.870 vs. 0.898, STFT_log 0.075 vs. 0.077, Mel Distance 0.461 vs. 0.497, Mel_log 0.089 vs. 0.091, Spectral Pan Error (SPE) 0.264 vs. 0.275, CCPC 0.973 (tied), and Expert Listening Rating 0.75 vs. 0.64, confirming the innovation of our architecture. (Note: Unlike low-regime operator ablations where SI-SDR lies within its discriminative confidence interval, once full-scale reconstruction SI-SDR exceeds 11 dB, SI-SDR becomes limited and distorted for quality assessment, whereas full-band spectral distances, SPE, and expert listening scores faithfully reflect perceptual quality.)

Q3: On "Waveform patches outperforms STFT" and Small-Scale Representation Capability

For complex music signals, relying solely on SI-SDR and low-frequency STFT distance has limitations, as SI-SDR can be skewed when a compact model overfits high-energy low frequencies at the expense of high frequencies. At the ~3M scale, our goal is to evaluate each representation's capacity to capture full-band information rather than low-frequency metric overfitting.

As shown in Table 1 (§4.3) and Table 6 (Appendix A.2) of the revised paper, while Waveform Patch (Conv2D) has lower low-frequency distance (STFT_LF < 2 kHz: 0.986 vs. 1.137), Complex STFT (real/imag) achieves the best full-band STFT distance (1.122 vs. 1.294) and a 22% lower high-frequency distance (STFT_HF > 8 kHz: 1.178 vs. 1.513). Consistent with the spectrogram comparisons in Figure 4 (Appendix A.3)—which show Complex STFT better preserves overall spectral energy, envelopes, and harmonic structure—our newly added human expert blind listening test in Table 1 & Table 6 confirms that Complex STFT achieves the highest subjective rating (0.40 vs. 0.35 for Waveform Patch, 0.26 for Waveform Conv1D, 0.21 for Magnitude-only, and 0.09 for Patch Transformer).

Q4: On Additional Ablations of Refiner Frequency Boundaries

We added an ablation testing alternative Refiner cutoff boundaries beyond 1.5 / 4 kHz (specifically shifting the low cutoff to the 0.5 kHz loss transition) in Table 3 (§4.5) of the revised paper. Our 1.5 / 4 kHz Duplex configuration outperforms the 0.5 / 4 kHz variant across both expert listening evaluations (0.75 vs. 0.67) and all objective metrics: SI-SDR (11.097 vs. 11.073 dB), Mel (0.658 vs. 0.674), STFT (1.019 vs. 1.027), Low IPD (0.7963 vs. 0.8171 rad), High IPD (1.2907 vs. 1.3153 rad), Low ILD (0.1690 vs. 0.1988 dB), and High ILD (0.9804 vs. 1.3489 dB).

Dear Reviewer sCpV,

We hope you are doing well. With the discussion period ending on November 18, we would like to kindly follow up and ask whether you have had a chance to review our rebuttal.

In response to your concerns, we have added controlled experiments, including a dimension-matched Refiner baseline, an architecture-aligned comparison with LeVo 2 VAE, and additional frequency-boundary ablations, along with expert listening evaluations.

We hope these new results have helped clarify the contributions of our proposed design. We would sincerely appreciate any further feedback or remaining concerns and would be glad to discuss them before the deadline.

Thank you again for your constructive review.

Best regards,
The Authors

Reviewer_sCpVOpenReview ↗

Response to rebuttal

Thank you for the additional experiments and detailed responses. The new results address most of my previous concerns, particularly regarding the controlled comparison with external baselines and the effect of output dimensionality.

However, I would appreciate further clarification on two points.

  1. Implementation of the Matched-Dim Unconstrained Refiner

Could the authors clarify how the Matched-Dim Unconstrained Refiner performs full-band magnitude and phase correction using only 532 output dimensions? Additionally, while the dimension-matched comparison is helpful, I am still interested in whether the proposed duplex-guided allocation outperforms alternative frequency-permuted masks under the same output budget. Such a comparison would help further isolate the benefit of the specific frequency allocation.

  1. Contribution of the Complex STFT Representation versus the Refiner

I acknowledge that the full εar-VAE2 outperforms the aligned LeVo 2 baseline. However, according to the originally reported reconstruction results, εar-VAE2 without the Refiner performs worse than the newly aligned LeVo 2 baseline on several metrics, including STFT and Mel Distance.

Assuming these results are directly comparable, this raises a question about the source of the final performance improvement. To what extent does the improvement come from the Complex STFT-based encoder–decoder itself, as opposed to the Duplex-Aware Refiner? Further clarification or an additional controlled comparison would help distinguish their respective contributions.

Overall, I appreciate the authors' efforts to strengthen the experimental evidence, and the additional results have addressed many of my initial concerns.

Reviewer_JveiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ε\varepsilonar-VAE2, a continuous-latent music autoencoder operating on the complex STFT domain for 48 kHz stereo music reconstruction. Based on a controlled study showing that complex STFT preserves high-frequency spectral details better than waveform representations, the authors introduce two frequency-aware designs: (1) Spec-SnakeBeta, a per-frequency-bin periodic activation parameterized in log-space, and (2) Duplex-Aware Refiner, a post-hoc module that applies band-specific magnitude and phase corrections guided by psychoacoustic duplex theory. Experiments on the Song Describer Dataset and downstream generation tasks validate the effectiveness of the pipeline.

Critical strengths
  1. Exploiting the explicit frequency axis via Spec-SnakeBeta and Duplex-Aware Refiner: Moving continuous music VAE modeling back to the complex STFT domain is well-motivated by the matched-budget study (Table 1). Designing a frequency-indexed activation (Spec-SnakeBeta) and incorporating psychoacoustic priors (Duplex-Aware Refiner) to directly address high-frequency loss and stereo phase/level errors are intuitive and domain-informed contributions.
  2. Strong empirical reconstruction and downstream generation results: The full pipeline achieves clear improvements in Mel Distance (-19.4% with the refiner, Table 2), beats the unconstrained baseline in professional listening tests (Table 3), and consistently improves all 12 metrics in downstream music generation over LeVo 2 VAE (Table 5).
Critical weaknesses
  1. Unconvincing gain vs. complexity and scalability of Spec-SnakeBeta (Table 4): In current large-scale model design, simple and scalable activations (like ELU or SiLU) are generally preferred. In Table 4, Spec-SnakeBeta-F-Log only shows marginal gains on some metrics, and actually underperforms plain ELU on STFT distance (1.315 vs. 1.313) and SiLU/ELU on Mel_log (0.125 vs. 0.123/0.124), with no standard deviations reported across the 3 seeds. Moreover, putting learnable parameters inside exp() (α=eα~,β=eβ~\alpha = e^{\tilde{\alpha}}, \beta = e^{\tilde{\beta}}) can easily cause activations or gradients to explode or vanish during scaling, hurting the survival rate of active periodic units.
  2. Rule-based band splitting in Duplex-Aware Refiner (Section 3.3, Figure 3, Table 3): The frequency cutoffs (1.5 kHz and 4 kHz) are strictly rule-based. However, Figure 3(b–c) shows notable magnitude error below 1.5 kHz and phase error plateauing around 2.5–3 kHz, which does not strictly align with the hard-coded masks (δM=0\delta M=0 below 1.5 kHz, δϕ=0\delta \phi=0 above 4 kHz). Also, saving 45% output dimensions only affects the final linear projection of the refiner, and the objective gains over the Unconstrained Refiner in Table 3 are relatively small (and slightly worse on High IPD).
Questions for authors

Clarification Requests:

  1. On Spec-SnakeBeta gain, complexity, and activation survival rate:

    • In Table 4, Spec-SnakeBeta-F-Log is slightly worse than simple ELU on STFT distance (1.315 vs. 1.313) and worse than SiLU/ELU on Mel_log. Could you provide the standard deviations across the 3 seeds to show if the gains are statistically significant? What is the actual training/inference speed overhead compared to ELU/SiLU at the full 692M scale?
    • Using exp() for αf\alpha_f and βf\beta_f risks exploding gradients or diminishing the periodic branch to zero (hurting the survival rate of active nonlinear units). Did you observe dead periodic units (exp⁡(−β~f)→0\exp(-\tilde{\beta}_f) \to 0) or activation/gradient explosion during full-scale training? Is this instability the main reason you had to use float64 losses and the Muon optimizer to suppress gradient spikes (Figure 8)?
  2. On the rule-based band splitting in Duplex-Aware Refiner:

    • In Figure 3(b–c), magnitude errors are clearly visible below 1.5 kHz and phase errors plateau before 4 kHz. How sensitive is the model to these hard-coded 1.5 kHz and 4 kHz cutoffs?
    • In Table 3, the objective gap between Banded and Unconstrained is quite narrow, and High IPD is slightly worse (1.2907 vs. 1.2825). Since the 45% dimension reduction only applies to the very last linear layer of the refiner, what is the actual FLOPs/latency saving?

Suggestions for Future Work:

  1. Learnable band allocation: Instead of a rigid rule-based split, have you tried a learnable soft-gating mechanism (e.g., learning frequency-wise gates on δM\delta M and δϕ\delta \phi, initialized with the duplex prior) so the refiner can adaptively adjust the boundaries?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Jvei

We thank Reviewer Jvei for the constructive comments. We have updated the revised manuscript with all corresponding multi-seed statistics, ablations, and inference benchmarks.

Q1: On the Metric Significance and Training Stability of Spec-SnakeBeta

In complex music reconstruction, perceptual quality depends on multiple complementary factors (SI-SDR, STFT distance, etc.). Importantly, SI-SDR has an effective confidence interval: in the foundational fitting regime of operator-level ablations (1.7–4.4 dB), SI-SDR reliably reflects waveform and phase alignment, whereas once SI-SDR reaches a high regime in full-scale models (>11 dB), further increases become limited and less indicative of perceptual quality.

As reported in Table 4 (§4.6) of the revised paper (mean ± std across three seeds), while Spec-SnakeBeta-F-Log shows modest margins on some individual metrics, it achieves the best overall balance across metrics—securing the best three-seed mean on 4 of 6 metrics: SI-SDR (4.40 ± 0.258 dB vs. 1.78–2.38 dB for ELU/SiLU/GELU and 3.40–3.51 dB for other Snake variants), STFT_log (0.102 ± 0.00143), Mel (1.070 ± 0.0110), and CCPC (0.908 ± 0.0061), while matching the best STFT (1.315 ± 0.0019). Within this regime, the SI-SDR gain directly indicates better target-reconstruction similarity in phase perception and spatial localization, showing effective multi-scale music modeling. Moreover, F-Log outperforms the channel-frequency independent CF-Log on all 6 metrics with ≈128× fewer activation parameters.

Regarding learnable activation parameters during training: with our curated data pipeline and Muon optimizer, we did not observe noticeable activation or gradient explosion or vanishing during scaling.

Q2: On Duplex-Aware Refiner Band Partitioning and Cutoff Boundaries

We have provided side-by-side audio comparisons of different band-splitting strategies on our demo page, where our Duplex-Aware strategy shows clear advantages in low-frequency phase coherence (e.g., tight "bounce" and "pump" in low-end transients) and stereo spatial depth.

Regarding why our Duplex cutoffs (1.5 / 4 kHz) differ from the prominent transition in the loss distribution (Figure 3, around 0.5 kHz), we conducted blind listening tests with 10 professional mixing/mastering engineers and added a boundary ablation in Table 3 (§4.5) of the revised paper. Splitting at the loss-transition boundary (Banded 0.5 / 4 kHz, 552 dims/ch) yields worse objective metrics and lower expert listening ratings than our Duplex-Aware 1.5 / 4 kHz configuration (532 dims/ch): SI-SDR 11.097 vs. 11.073 dB, Mel 0.658 vs. 0.674, STFT 1.019 vs. 1.027, Low IPD 0.7963 vs. 0.8171 rad, High IPD 1.2907 vs. 1.3153 rad, Low ILD 0.1690 vs. 0.1988 dB, High ILD 0.9804 vs. 1.3489 dB, and Expert Rating 0.75 vs. 0.67. Thus, partitioning at the prominent loss transition does not yield better objective or subjective results, whereas Duplex-Aware splitting performs best.

Q3: On Inference Overhead, FP64 Loss Precision, and the Muon Optimizer

  1. Inference Overhead & Multi-Seed Std: Multi-seed standard deviations are reported in Table 4 (§4.6), and full-model inference costs (NVIDIA L20Z, 1.28 s 48 kHz stereo, batch 1, FP32) are added in Table 12 (Appendix J.1). Compared to ELU/SiLU (692.040M params), Spec-SnakeBeta-F-Log (692.043M params) adds only 0.003M parameters and ~1.3 ms decoder latency (23.358 ms vs. 22.055–22.103 ms; full inference 99.262 ms vs. 94.689–94.901 ms), maintaining a 12.90× real-time speedup (RTF 0.07755) and identical peak memory (3.222 GiB vs. SiLU).

  2. FP64 Loss Precision: During training and numerical precision alignment, we noticed occasional minor fp32 precision leakage in complex STFT operations (which also default to fp64 in standard PyTorch complex routines). We therefore used float64 loss computation purely as a precautionary safeguard, and did not encounter severe numerical issues in practice.

  3. Muon Optimizer (Operator Numerical Instability vs. GAN Equilibrium Instability): We distinguish two types of instability: your concern regards operator-level numerical instability from exp during forward/backward computation, whereas our use of Muon targets GAN equilibrium instability in multi-stage training. The exp operator itself is numerically stable and not strongly coupled with Muon. Instead, because Stage 1 performs reconstruction warm-up while Stages 2–3 introduce GAN training—where generator-discriminator equilibrium is fragile and challenged by quality variance across full-scale audio data—stabilization is needed. Empirically, Muon significantly reduces adversarial gradient spikes (Figure 8, Appendix F), motivating its adoption.

AuthorsOpenReview ↗

Supplement for Activation Functions Performance Table 12 (Appendix J.1)

Inference Overhead & Multi-Seed Std: Multi-seed standard deviations are reported in Table 4 (§4.6), and full-model inference costs across activation functions (measured on NVIDIA L20Z; 1.28 s, 48 kHz stereo, batch size 1, FP32; Table 12, Appendix J.1) are shown in the table below. Compared to ELU and SiLU, Spec-SnakeBeta-F-Log adds only 0.003M parameters and ~1.3 ms decoder-only latency while maintaining a 12.90× real-time speedup and identical peak memory (3.222 GiB vs. SiLU):

Activation FunctionParams (M)Full Inference (ms)Decoder-Only (ms)Real-Time Speedup (×)RTF (Latency / Duration)Peak Memory (GiB)Conv/Linear GFLOPs
ELU692.04094.68922.05513.52×0.073983.1912334.811
SiLU692.04094.90122.10313.49×0.074143.2222334.811
Spec-SnakeBeta-F-Log (Ours)692.04399.26223.35812.90×0.077553.2222334.811

Dear Reviewer Jvei,

We hope you are doing well. As the discussion period is approaching its deadline on November 18, we would like to kindly follow up and ask whether you have had a chance to review our rebuttal.

We have provided additional experimental results and clarifications regarding your concerns, particularly the multi-seed statistics and computational overhead of Spec-SnakeBeta, training stability, and the boundary sensitivity of the Duplex-Aware Refiner.

We hope these additions have helped address your concerns. We would greatly appreciate any further feedback or questions and would be happy to provide additional clarification before the discussion period ends.

Thank you again for your time and thoughtful review.

Best regards, The Authors

Dear Reviewer Jvei,

We hope you are doing well. As the discussion period is approaching its November 18 deadline, we wanted to briefly follow up on our previous response.

We sincerely hope that our additional experimental results and clarifications, particularly the multi-seed statistics, full-scale inference benchmarks, and boundary ablations, have adequately addressed the concerns raised in your review.

If you feel that these clarifications have sufficiently resolved the issues underlying your initial assessment, we would be very grateful if you could reconsider the overall rating in light of the additional evidence and revise it if you find it warranted.

We fully respect your independent judgment and sincerely appreciate your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.