- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ear-VAE2, a 48 kHz continuous-latent music autoencoder based on the complex STFT, to address high-frequency loss, phase incoherence, and stereo-image collapse in waveform-domain audio VAEs. It introduces Spec-SnakeBeta, a frequency-dependent periodic activation function, and Duplex-Aware Refiner, which applies band-specific magnitude and phase corrections based on duplex theory of sound localization. The latter reduces residual-output dimensions by approximately 45% compared with an unconstrained refiner. Experiments show that ear-VAE2 achieves the best point estimates on five of seven reconstruction metrics, while its latents improve point estimates across all 12 downstream music generation metrics.
Critical strengths
- They conduct a thorough investigation on five different audio representation to identify the most suitable representation for wide-band spectral preservation. Based on this analysis, they propose a frequency-aware continuous audio VAE model operating on the complex STFT.
- Simple but effective Spec-SnakeBeta and Duplex-Aware Refiner to improve frequency-domain modeling and stereo reconstruction, respectively.
- Extensive experiments demonstrate improved reconstruction quality.
Critical weaknesses
- Unfair comparison with SOTA methods. The full ear-VAE2 model has ~692M parameters, and is additionally trained on 10,000 hours of unreleased, proprietary 48 kHz high-quality music. Moreover, the baselines in Tab 2 differ in sampling rates and latent frame rates (e.g., SAME-L uses only 10.8 Hz). These disparities make it difficult to attribute the reported gains to the proposed methods rather than increased model capacity, additional training data, or inconsistent compression settings.
- In Table 6, complex STFT exhibits substantially lower SI-SDR than Waveform Patch in the matched-budget study (−10.49 vs. 3.98 dB). Even the full ear-VAE2 achieves only 11.3 dB, below SAME-L (12.5 dB) and ear-VAE (12.4 dB). The paper does not sufficiently investigate whether this gap affects the perceptual quality of transients or phase-sensitive content.
- In Table 6, Waveform Patch outperforms Complex STFT on four of six metrics. Although the authors acknowledge this trade-off, the choice of complex STFT appears driven more by its compatibility with the proposed modules than by overall empirical superiority. Thus, the claim that “Fourier is Frontier” is overstated.
- Limited conceptual novelty. The complex-STFT backbone closely follows existing spectral codec architectures such as SpectroStream, while the band-specific magnitude/phase corrections draw on established principles from duplex theory and parametric stereo. Although Spec-SnakeBeta and the duplex-banded refiner are reasonable design choices, the overall contribution appears to be an effective integration of existing components and signal-processing priors rather than a fundamentally novel representation-learning approach.
Questions for authors
None
4 comments
Dear Reviewer CJZH,
We hope you are doing well. As the discussion period is approaching its deadline on November 18, we would like to kindly follow up and ask whether you have had a chance to review our rebuttal.
We have provided additional evidence addressing your concerns, particularly the controlled comparison with an architecture-aligned baseline, the smaller model trained on open-source data, and expert listening evaluations supporting our choice of spectral representation.
We hope these additions have helped clarify the empirical contributions and motivation of our work. We would greatly appreciate any further feedback or questions before the discussion period ends.
Thank you again for your time and valuable suggestions.
Best regards,
The Authors
Dear Reviewer CJZH,
We hope you are doing well. As the discussion period is approaching its November 18 deadline, we wanted to briefly follow up on our previous response.
We sincerely hope that the additional experimental evidence, particularly the architecture-aligned baseline comparison, the smaller model trained exclusively on open-source data, and the expert listening evaluations, has adequately addressed your concerns regarding comparison fairness, representation choice, and the empirical contributions of our approach.
If you feel that these additional results have sufficiently addressed the concerns underlying your initial assessment, we would greatly appreciate it if you could reconsider the overall rating in light of the revised evidence and adjust it if you find it warranted.
We fully respect your independent judgment and sincerely appreciate your time and valuable feedback.
Best regards, The Authors
Official Comment
I thank the authors for the detailed response, which addresses some of my concerns about the experiment results. I keep my original score of weak accept.
Response to Reviewer CJZH
We thank Reviewer CJZH for the helpful comments. We have updated the revised manuscript with the controlled architecture ablation and expert listening evaluations.
Q1: On the Fairness of Comparisons with SOTA Methods
First, we added a controlled ablation in Table 9 (Appendix F.1) of the revised paper, training the external baseline architecture (
LeVo 2 VAE (aligned)) on the exact same dataset with comparable model size (≈692M vs. 695M),1920×compression, and128-dim latent, varying only the model architecture. Under aligned conditions,εar-VAE2outperformsLeVo 2 VAE (aligned)across spectral metrics (STFT0.870 vs. 0.898, STFT_log0.075 vs. 0.077, Mel0.461 vs. 0.497, Mel_log0.089 vs. 0.091), Spectral Pan Error (0.264 vs. 0.275), and expert listening scores (0.75 vs. 0.64). (Note on SI-SDR: the two SI-SDR discussions in our paper correspond to different regimes—SI-SDR is reliable within a moderate confidence interval such as in our small-scale operator ablations, whereas once full-scale reconstruction SI-SDR reaches11–12+ dB, it becomes limited and distorted for quality evaluation.)Second, we trained a smaller model variant (
170.2M + 8.2M Refiner) exclusively on open-source data with a size comparable to open-source baselines (to be open-sourced soon). As shown below, this variant still outperforms open-source commercial baselines (SA-OpenandLeVo 2) across all spectral, SI-SDR, and inter-channel coherence metrics—and even surpassesSAME-LonSTFT,Mel-log1p, andSTFT-log1p—validating our training recipe:Q2: On Metric Performance of Waveform Patch Modeling and High-Frequency Underfitting
In complex music signals, SI-SDR and low-frequency STFT distance have clear limitations when a compact (~3M) model overfits energy-dominant low frequencies. Our goal at this scale is to investigate the model's ability to capture information across the full signal spectrum rather than low-frequency metric overfitting. As shown in Table 1 (§4.3) and Table 6 (Appendix A.2) of the revised paper, a finer frequency breakdown reveals that while
Waveform Patch (Conv2D)fits low frequencies well (STFT_LF < 2 kHz = 0.986vs.1.137), it suffers from severe underfitting in high frequencies (STFT_HF > 8 kHz = 1.513, whereasComplex STFTachieves1.178, a 22% reduction, and the best full-band STFT of1.122 vs. 1.294). Furthermore, perceptual differences are pronounced in stereo imaging and high-frequency clarity. As reported in Table 1 & Table 6, human expert blind listening tests confirm thatComplex STFTachieves the highest subjective score (0.40vs.0.35forWaveform Patch,0.26forWaveform Conv1D,0.21forMagnitude-only, and0.09forPatch Transformer).Q3: On Adapting the SpectroStream Backbone and Positioning Our Contributions
Our work does not focus on structural novelty in isolated generic backbone blocks; rather, within high-fidelity music reconstruction, we investigate foundational audio input-output representation modeling and introduce a series of targeted improvements built upon it. Specifically, we adopt
Complex STFTas our starting point to achieve higher reconstruction quality.Complex STFThas not yet been widely adopted as a mainstream pipeline in continuous audio/music autoencoding, nor have psychoacoustic principles such as Duplex Theory been integrated into neural model training in this setting—giving our work clear frontier value and novelty in domain-knowledge transfer.