acceptodds
Under review as a conference paper at ICLR 2027

SAGE: Semantic Audio Generative Encoder

Abstract

Audio autoencoders compress waveforms into compact latent representations that serve as the interface between raw audio and downstream models. Current systems navigate a three-way trade-off between reconstruction quality, semantic structure of the latent space, and inference speed, typically favoring one or two of these at the expense of the others. This paper introduces SAGE, Semantic Audio Generative Encoder: a compact variational autoencoder, trained solely on publicly available music, that shapes its latent by distilling embeddings from a pretrained audio-text model. This M-parameter model runs at the inference cost of Stable Audio Open and reaches the listening-test quality of SAME-L, an autoencoder larger and slower, while surpassing both on objective perceptual and distributional metrics of reconstruction. Furthermore, it sets the state of the art on all nineteen probing tasks of latent semantics, in domain and out of domain. These results establish SAGE as a lightweight audio autoencoder that strikes the best balance of the three-way trade-off among those we evaluate, combining high reconstruction fidelity, state-of-the-art semantic structure, and fast inference.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
audio, autoencoder, semantic, music generation, generative model, representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3njoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAGE is a 105M-parameter VAE for 44.1 kHz stereo music. It works on the power-compressed complex STFT with a three-stage SwinV2 encoder/decoder adapted to spectrograms (64×1 patches, 4×32 attention windows, SwiGLU, exclusive self-attention). A linear head maps the time-averaged latent to a frozen LAION-CLAP embedding under a cosine loss, following the clip-level scheme of SALAD-VAE. The head is discarded at inference. The encoder receives this gradient only after a stop-gradient warm-up of s0s_0 steps.

Training uses STFT, mel, mid/side STFT, KL and RpGAN/feature-matching losses on about 10.5k hours of FMA, MTG-Jamendo and M4Singer. A second phase fine-tunes only the decoder. The latent budget matches Stable Audio Open (×64, 64 channels at about 21.5 Hz).

All five baselines (Stable Audio Open VAE, SAME-L/S, CoDiCodec, Music2Latent) are re-run from public weights through one harness on five sets.

  • Reconstruction. SAGE has the best FAD (MERT, CLAP and PANN spaces) and CLAP similarity in 23 of the 25 set×metric cells.
  • Speed. It ties Stable Audio Open on compiled real-time factor (0.0045).
  • Listening test. A 10-excerpt MUSHRA against SAME-L, Stable Audio Open and CoDiCodec (21 listeners retained) gives SAGE 81.6 ± 2.7 and SAME-L 81.8 ± 2.6.
  • Probing. It leads all five autoencoders on 19 probing tasks built around genre, artist and instrument labels.

The semantic, discriminator and capacity studies (Sec. 3.5, App. E) use a smaller, older backbone on FMA-medium; App. B ablates the stereo loss under the full setup.

Critical strengths
  • Every baseline is run from its released weights, one call per clip, on five sets. These include each baseline's own benchmark: MusicCaps for Music2Latent/CoDiCodec and Song Describer for Stable Audio Open/SAME. Metrics are three FAD spaces, two CLAP checkpoints and SDR/STFT diagnostics. SAGE's advantage is broadly consistent across sets (23 of 25 cells, Table 2). The common harness is the main reason I believe the reconstruction claims.
  • At the same compiled RTF as Stable Audio Open (44.8 vs 45.2 ms per 10 s clip), SAGE scores about 17 MUSHRA points higher. It is roughly level with SAME-L, which has 8× the parameters, about 4× the latency and twice the latent budget (×32 vs ×64). The authors also check the obvious worry that CLAP-based metrics favour a CLAP-distilled model. In the reduced, non-adversarial teacher ablation (Table 5, lower block), CLAP distillation lowers both reconstruction CLAP similarities (0.613/0.575 vs 0.645/0.624). FAD-MERT and FAD-PANN are not tied to the teacher.
Critical weaknesses
  1. The semantic claim is supported only by genre/artist/instrument probes on the pooled latent, and never through the use it is motivated by.

    • What is probed. Lsem\mathcal{L}_{sem} aligns a projection of the time-averaged, frequency-folded latent with CLAP (Eq. 1). The probes read that same pooled latent (Sec. 3.2). The 19 tasks use several formats (classification, clustering, retrieval, reranking, pair classification), but they cover only three semantic categories: genre (10 tasks), artist (7) and instrument (2). These categories are plausibly aligned with what the CLAP teacher encodes. Winning them is not guaranteed by the loss, but it is not surprising either.
    • What is not probed. There is nothing on key, chords, tempo/beat or melody. These matter for a music generator's latent, and SAME explicitly supervises pitch-class structure through chroma targets.
    • No uncertainty. No probing uncertainty is reported. Several gains over the autoencoders are large (GTZAN accuracy 0.774 vs 0.531), but some are tiny (Jam-ALT 0.9672 vs 0.9668). The CLAP reference stays ahead on 18 of 19 tasks.
    • No downstream generator. The Introduction and Sec. 4 justify the semantic latent by its value to a downstream diffusion model ("drop-in first stage"), but the paper reports no generator trained on SAGE latents. SAME, the main baseline, does exactly this: it trains a ~1.4B DiT on its latent and shows that its semantic losses improve generation (FAD-CLAP 0.593 → 0.576, MuQ-Eval 3.34 → 3.87). As evaluated, "state-of-the-art semantic structure" means "best pooled-latent probe scores among these six autoencoders on genre/artist/instrument". That is narrower than the abstract suggests.
  2. The ablations do not isolate the method contributions.

    • Delayed schedule. Table 5 sweeps s0∈{25k,50k,100k}s_0 \in \{25k, 50k, 100k\} but never s0=0s_0 = 0, so delay versus no delay is untested. The gate opens after about 8.3k generator updates, roughly 3.5% of phase-1 training. No repeated-seed variability is reported, so the small 25k-vs-50k differences (FAD-MERT 0.240 vs 0.248, probes 0.637 vs 0.629) are hard to interpret. SALAD-VAE, whose objective SAGE adopts, already uses a curriculum: it pre-trains on reconstruction and KL and adds the CLAP, contrastive and adversarial terms "after several epochs". The new part is therefore the stop-gradient variant, in which the head trains from step 0, and that is the part left untested.
    • Architecture. The semantic, discriminator and capacity studies use a reduced backbone that predates SwiGLU and XSA, on FMA-medium. Appendix E varies width and depth (Table 11). None of the spectrogram-specific choices in contribution 1 is isolated: rectangular patches/windows, the three-stage hierarchy, XSA, the post-net.
    • Phase 2. Table 10 studies discriminator placement across phases, but there is no controlled phase-1 vs phase-2 comparison for the final model. Phase 2 is 992 epochs and about two thirds of the compute (3,043 of 4,579 GPU-h).
    • Consequence. Appendix E shows the discriminator alone moves FAD-MERT from 1.453 to 0.203. The baselines also differ in data, compression (×32 for SAME, ×128 for CoDiCodec), discriminator and training length. Table 2 therefore compares systems, and we cannot tell which part of SAGE produces the gain.

Smaller points (not decisive):

  • MUSHRA. "Statistically indistinguishable" / "matches SAME-L" rests on overlapping marginal CIs. There are 10 excerpts, 17 of 38 participants were filtered out, and no paired test or equivalence margin is reported.
  • SDR is below Stable Audio Open on all five sets (5.28 vs 5.64 on FMA, 6.31 vs 6.86 on Song Describer) and 4.7–6.2 dB below SAME-L. Treating SDR as diagnostic is defensible for an adversarially trained decoder, but it deserves a sentence next to the headline claims.
  • Missing open music autoencoders: εar-VAE (44.1 kHz stereo, open weights), the ACE-Step 1.5 VAE and Music2Latent2. The first two appear in SAME's own listening test (MUSHRA 77.6 and 76.5, against 82.2 for SAME-L), so they are the obvious next-strongest comparisons.
  • Table 11's within-sweep comparisons are controlled, but the conclusion that depth gives a larger return per parameter than width compares sweeps run with different λmel\lambda_{mel}.
  • Please state the license for the released weights; M4Singer is CC BY-NC-SA 4.0.
Questions for authors
  1. (Could change my score.) Can you add s0=0s_0 = 0 and a no-distillation run with the final backbone and the full adversarial setup? If the stop-gradient gate matters at full scale, contribution 2 stands.
  2. (Could change my score.) Train even a small latent diffusion model on SAGE latents and on Stable Audio Open latents at equal budget. If the semantic latent makes generation easier, that is the result the paper is built around.
  3. (Clarification.) You filter listeners on hidden-reference ratings. What exact threshold and participant-level rule were applied, how were the CIs constructed, and what is the paired SAGE − SAME-L difference with its CI?
  4. (Clarification.) For the track-grouped cross-validation: how many folds, and do stems inherit their parent recording's group? Does artist retrieval exclude other excerpts of the same recording?
  5. (Suggestion.) Probes for key, chords or beat (e.g. GiantSteps key) would show whether the latent is semantic in a musical sense or mainly in a tagging sense.
  6. (Clarification.) What does phase 2 add? Please give phase-1-only reconstruction and listening numbers for the final model.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 3njo (1/6): new results, delayed schedule and distillation

We thank the reviewer for the careful and precise assessment, and for stating, for each concern, the experiment that would address it and the results that could change the rating. We are glad that the common harness makes the reconstruction claims credible, that the speed/quality trade-off is judged good, and that our check that CLAP distillation does not inflate the CLAP-based metrics is acknowledged. We ran the suggested experiments and report them below.

New results since the submission

  • s0 = 0, s0 = 100k and no distillation on the final backbone with the full adversarial setup. The distillation gives the probing lead: without it, probing is 19% lower (Q1, W2).
  • Latent diffusion at equal budget on SAGE and on the SAO VAE, SAME-L and CoDiCodec latents, on two datasets and with two generator sizes. SAGE has the lowest FAD on every metric at every checkpoint. Against the SAO VAE, at the same latent shape and generator cost, its FAD is 18 to 60% lower at the end of training (Q2, W1).
  • Key, chord, pitch-class and beat probes, and emotion, singing-technique and style probes, for the eight autoencoders and, where defined, the CLAP teacher (SAGE leads the autoencoders on emotion, singing technique and style and, like its teacher, is weak on key and pitch); our recipe with chroma supervision added, which recovers much of the tonal content at proxy scale (Q5).
  • Paired listening analysis: SAGE and SAME-L are tied, both 15 to 17 points above SAO and CoDiCodec (Q3). Phase 2 improves every reconstruction metric (Q6). XSA and SwiGLU removed from the final backbone (W2). Probing re-run with paired 95% intervals and two more baselines: SAGE is significantly ahead on 17 of the 19 tasks and tied on the other two (W1).

Q1, W2 (delayed schedule and distillation on the final backbone).

The two requested runs separate the parts of contribution 2. The distillation is what makes the latent semantic: without it probing falls by 19%, on 18 of 19 tasks. The stop-gradient warm-up is not needed: s0 = 0 matches the adopted s0 = 25k. The camera-ready therefore states contribution 2 as the distillation and its measured cost (below), with the warm-up as a control. We trained the final architecture with the full adversarial setup of the paper at proxy data scale (FMA-medium, 95 epochs, phase 1 only; at full scale phase 1 alone takes 1,536 A100 hours per run), so absolute values are not those of Table 2 or 5. Each arm has two seeds (one for s0 = 100k) and the table gives their mean; reconstruction is measured on the 1,998 MoisesDB mixtures.

armprobes, mean of 19 ↑FAD-MERT ↓FAD-CLAP ↓FAD-PANN ↓SI-SDR (dB) ↑
s0 = 00.5460.960.1524.420.88
s0 = 25k (adopted)0.5450.940.1524.390.91
s0 = 100k (one seed)0.5270.940.1514.351.12
no distillation0.4420.880.1494.311.13
  • The distillation is what makes the latent semantic. Without the teacher probing falls by 19% (0.545 to 0.442), with 18 of 19 tasks significantly worse in both seeds, while reconstruction is slightly better, again in both seeds (FAD-MERT 6% lower, SI-SDR +0.2 dB), as in Table 5. The distillation also lowers the key and chord probes (Q5), and FAD-CLAP does not reward it (0.152 with, 0.149 without).
  • No warm-up is needed (s0 = 0, the requested control). With the alignment acting from the first step, probing and reconstruction match the adopted s0 = 25k within seed noise.
  • A later gate trades probing for reconstruction. At s0 = 100k probing falls by 3% (8 of 19 tasks significantly worse on the evaluation items, none better) and SI-SDR rises from 0.91 to 1.12 dB, the level of the no-distillation arm.
  • Seed variability. Two seeds of one arm differ by 0.1% (adopted) to 1.3% (s0 = 0) on the probing mean; the standard deviation of one run is about 0.3 dB in SI-SDR and 4% in FAD-MERT.
  • Prior work. SALAD-VAE adds its CLAP term after a reconstruction-only stage, and SAME trains its heads on detached latents during a warm-up, the mechanism of our gate; the s0 = 0 run shows that, in our setting, neither is needed. The camera-ready credits both; the objective follows SALAD-VAE, as the submission says.

In the camera-ready, contribution 2 becomes: "Semantic structure at a small reconstruction cost. We distill CLAP into the latent and measure, on the final backbone with the full adversarial setup, what it buys and what it costs: without it the mean of 19 probing tasks falls by 19%, while reconstruction improves only by about the spread between two training runs (without the discriminator, Table 5, the cost is larger); the cost falls on tonal content, where the key and chord probes are higher without it. No warm-up is needed: the alignment can act from the first step, and a long stop-gradient warm-up costs probing." Secs. 2.2 and 3.5 and the conclusion are reworded accordingly.

AuthorsOpenReview ↗

Response to Reviewer 3njo (2/6): generation

Q2, W1 (generation).

At equal budget, a diffusion transformer generates closer to the data on SAGE latents than on Stable Audio Open latents, the comparison the review proposes, and than on those of SAME-L and CoDiCodec: lower FAD on every metric and at every checkpoint, in each of the three settings tested (two corpora, two generator sizes). As in Table 2, these are released autoencoders, which also differ in data and recipe (W2, consequence): the study measures the latent each system gives a generator.

Setup. For each autoencoder, a fresh SiT (a diffusion transformer trained with flow matching, 12 blocks, one token per latent frame) learns to generate its frozen latents of 4-s stereo crops at 44.1 kHz, without conditioning. Only the latent changes. The data, generator, optimizer, schedule (300 epochs, 163k updates at batch 256), per-channel standardization and sampler (32 Euler steps, same seeds) are identical. Each model generates 2,048 clips, compared by FAD with 2,048 held-out validation crops in three embedding spaces. There are two corpora: FMA-medium and a Music4All subset of the same size (139k training crops each; Music4All split by artist). The generators have 130M parameters on both corpora and also 230M on FMA.

FAD at the end of training (epoch 300), lower is better:

DatasetGeneratorLatentFAD-CLAP ↓FAD-MERT ↓FAD-PANN ↓
FMA130MSAGE0.0701.436.67
FMA130MSAO VAE0.1081.749.95
FMA130MCoDiCodec0.1582.2410.62
FMA130MSAME-L0.1712.1112.52
FMA230MSAGE0.0641.196.24
FMA230MSAO VAE0.1011.609.44
FMA230MCoDiCodec0.1462.049.92
FMA230MSAME-L0.1571.9111.55
Music4All130MSAGE0.0581.425.72
Music4All130MSAO VAE0.1452.068.91
Music4All130MCoDiCodec0.1132.588.51
Music4All130MSAME-L0.2132.189.21

SAGE against Stable Audio Open. Both latents have 64 channels at 21.5 Hz, so the budget is equal in updates and in compute. On FMA, SAO is the strongest of the other latents, second on every metric at both sizes. At epoch 300, SAGE's FAD is lower by (CLAP, MERT, PANN):

  • 35, 18 and 33% on FMA (130M);
  • 37, 26 and 34% on FMA (230M);
  • 60, 31 and 36% on Music4All.

It is 18 to 60% lower at every checkpoint.

Against the other autoencoders. SAGE has the lowest FAD in every setting, on every metric and at every checkpoint. At epoch 300 it is 18 to 49% below the best other latent (18 to 34% on FAD-MERT and FAD-PANN, which are not tied to the CLAP teacher). Against SAME-L, the main baseline, the gap at epoch 300 is:

  • 32 to 38% on FAD-MERT;
  • 38 to 47% on FAD-PANN;
  • 59 to 73% on FAD-CLAP.
AuthorsOpenReview ↗

Response to Reviewer 3njo (3/6): MUSHRA and probing protocol

Q3 (MUSHRA: screening rule, intervals, paired difference).

The rule is the post-screening of ITU-R BS.1534-3. Over listeners, the intervals separate two levels: SAGE and SAME-L at the top, 15 to 17 points above SAO and CoDiCodec, with a paired SAGE - SAME-L difference of -0.3 points [-4.9, +4.4]. SAGE reaches the upper level with one eighth of SAME-L's parameters and at twice its compression.

  • Rule. A participant is excluded when they rate the hidden reference below 90 in more than 15% of the trials, i.e. in two or more of the ten. The 21 retained participants have at most one such trial; the 17 excluded have two to eight.
  • Scale. A trial is one listener rating all systems on one excerpt; our test has 210 (21 listeners × 10 excerpts), in line with or above comparable work: the MUSHRA test of SAME, whose scores the review cites for εar-VAE and ACE-Step, has 36 valid trials (12 participants, after filtering on anchor and reference ratings), and the test of DAC has 120 (10 expert listeners × 12 samples of 10 s, four of them music).
  • Intervals. Table 3 gives the mean ± 1.96 standard errors over the 210 ratings of a system, treated as independent. Since every listener rated all systems on the same ten excerpts, the unit is the listener: over the 21 per-listener means (95% t-intervals) the intervals are 1.9 to 2.4 times wider (SAGE 81.6 ± 5.2, SAME-L 81.8 ± 6.2, CoDiCodec 66.4 ± 8.4, SAO 64.6 ± 8.6).

Paired differences, computed per listener in the same way:

paired differenceMUSHRA points [95% CI]reading
SAGE - SAME-L-0.3 [-4.9, +4.4]tie
CoDiCodec - SAO+1.8 [-2.4, +6.0]tie
SAGE - SAO+17.0 [+12.3, +21.7]SAGE higher
SAGE - CoDiCodec+15.2 [+9.6, +20.7]SAGE higher

SAME-L is likewise above SAO and CoDiCodec (+17.2 and +15.4 points, both intervals excluding zero): within a level the systems are tied, and between levels they differ by 15 to 17 points. The interval of SAGE - SAME-L does not establish strict equivalence, but it excludes a difference of more than 5 points out of 100 in either direction. In the camera-ready, Table 3 reports the intervals over listeners and the paired differences, and "statistically indistinguishable" becomes: "listeners did not separate SAGE from SAME-L (paired difference -0.3 points, 95% interval [-4.9, +4.4]); both are rated 15 to 17 points above SAO and CoDiCodec." The abstract and the conclusion follow the same wording.

Q4 (probing protocol).

  • Folds. Five in every cross-validated task (NSynth uses its fixed test split).
  • Stems. Yes. MoisesDB instrument classification, the only task on stems, groups the folds by recording; in the other cross-validated tasks each recording contributes one excerpt, so no recording is split across folds.
  • Artist retrieval. On FMA and MoisesDB each recording contributes one excerpt, so there is no other excerpt to exclude. We thank the reviewer for this question, which led us to check the query itself: in the submitted evaluation it was in its own corpus and counted as a hit, for every model. The re-run removes it, and SAGE stays significantly ahead of the closest autoencoder: +0.126 [+0.111, +0.143] on FMA (SAO), +0.068 [+0.035, +0.115] on MoisesDB (SAME-L). In Jam-ALT, taken unchanged from MAEB, each clip is one lyric line and the other lines of the same song count as relevant by design; since almost every artist has a single song, this task does reward finding other excerpts of the same recording. There too we removed only the self-match, and SAGE is tied with Music2Latent, SAO and ACE-Step (+0.001 [-0.006, +0.007] against Music2Latent).

The camera-ready states this protocol in Sec. 3.2.

Reviewer_oyHFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SAGE is a variational autoencoder for stereo music. It uses a SwinV2 backbone adapted to complex spectrograms and distills a frozen CLAP encoder into its latent representation. Semantic gradients are introduced after a warm-up period. A second training phase freezes the encoder and fine-tunes the decoder to improve reconstruction. The evaluation covers five reconstruction sets and nineteen semantic probing tasks. The 105M-parameter model has a compiled encode-decode time close to Stable Audio Open and improves most perceptual and distributional reconstruction metrics. Its mean listening-test score is close to SAME-L's, with overlapping confidence intervals. SAGE also obtains the highest reported score among the compared autoencoders on all nineteen probing tasks.

Critical strengths
  1. The reconstruction results are strong for a model of this size. Table 1 reports 44.8 ms per 10-second clip for SAGE and 45.2 ms for Stable Audio Open on the same A100 setup. The mean MUSHRA scores are 81.6 and 64.6, respectively. SAGE's mean is also close to SAME-L's 81.8 despite the large difference in model size. These results make a convincing case for its practical value as an audio autoencoder.
  2. The semantic evaluation extends beyond reconstruction. The nineteen tasks include classification, clustering, and retrieval on in-domain and out-of-domain data. The gains are substantial on several tasks, particularly the upstream MAEB genre tasks. The reduced-model teacher ablation also shows that CLAP distillation improves the average probing score, from 0.495 without a teacher to 0.642 with CLAP.
Critical weaknesses
  1. The final model's improvements are not sufficiently attributed to its main design choices. Table 5 studies the semantic gate, loss weight, and teacher on a reduced backbone that predates SwiGLU and XSA. Its gate-and-strength block includes the WavTokenizer discriminator; only the teacher-comparison block omits the adversarial branch. Appendix E adds discriminator and capacity studies, while Appendix B tests the stereo loss on multi-corpus pretraining. These are useful experiments, but they do not isolate XSA, the benefit of delayed alignment over alignment from the start, or the gain from decoder fine-tuning in the final configuration. A small set of controlled comparisons on that configuration would make the methodological contribution much clearer.
  2. Some conclusions go beyond what the evaluation directly establishes. The probing results show that semantic information is accessible in SAGE's latent space. They do not show that a diffusion model learns that space faster or generates better music. Section 4 motivates this use, but no downstream generator is evaluated. I regard this as an untested application rather than a reason to dismiss the autoencoder results. The claim of leading every probing task also needs qualification: the Jam-ALT scores in Table 9 are 0.9672 for SAGE and 0.9668 for Music2Latent, with no uncertainty estimate for the difference. This concern applies to the narrow margins, not to the larger gains elsewhere in the table. The CLAP-music reconstruction metric is related to the training objective, so I would not treat it as independent evidence. However, the general-audio CLAP checkpoint, MERT and PANN metrics, and listening test address much of this concern. Likewise, overlapping listening-test confidence intervals support similar observed means, but do not establish equivalence between SAGE and SAME-L.
Questions for authors
  1. Can the authors isolate the effects of XSA, delayed semantic alignment, and decoder fine-tuning using the final model configuration? For the alignment schedule, the existing sweep compares nonzero gate times; an alignment-from-start control would directly test the proposed warm-up.
  2. What uncertainty accompanies the task-level probing scores in Table 9, especially the 0.0004 difference on Jam-ALT? Please clarify the evaluation sampling and provide paired resampling estimates or repeated-probe results where appropriate. For the listening comparison with SAME-L, is there a paired analysis beyond the separate confidence intervals in Table 3?
  3. Is there any available experiment showing a downstream generation benefit from SAGE's latent space?
7 comments
AuthorsOpenReview ↗

Response to Reviewer oyHF (1/5): new results; XSA, delayed alignment and decoder fine-tuning

We thank the reviewer for the careful and balanced assessment. We are glad that the reconstruction results are found strong for a model of this size, and that the semantic evaluation is recognized as going beyond reconstruction. We ran the requested controls and report them below.

New results since the submission

  • Ablations on the final backbone with the full adversarial setup: s0 = 0, s0 = 100k, no distillation, XSA off, SwiGLU off (Q1, W1).
  • Probing re-run with paired 95% intervals for every task, and two more baselines: SAGE is significantly ahead on 17 of the 19 tasks and tied on the other two (Q2, W2).
  • Paired listening analysis: SAGE and SAME-L are tied, both 15 to 17 points above SAO and CoDiCodec (Q2, W2).
  • Latent diffusion at equal budget on SAGE and on three baseline latents (SAO VAE, SAME-L, CoDiCodec), on two datasets and with two generator sizes. SAGE has the lowest FAD on every metric at every checkpoint, 18 to 49% below the best other latent at the end of training (Q3, W2).

Q1, W1 (XSA, delayed alignment and decoder fine-tuning on the final configuration).

On the final configuration, the distillation needs no warm-up and is what makes the latent semantic (without it probing falls by 19%); XSA is kept for spectral fidelity and SwiGLU for stereo accuracy; decoder fine-tuning improves every reconstruction metric without changing the latent.

The arms below use the final architecture and the full adversarial setup of the paper at proxy data scale (FMA-medium, 95 epochs, phase 1 only), so absolute values are not those of Table 2 or 5. Two seeds per arm (one for s0 = 100k); the tables give the mean of the two. Reconstruction on the 1,998 MoisesDB mixtures.

armprobes, mean of 19 ↑FAD-MERT ↓FAD-CLAP ↓FAD-PANN ↓SI-SDR (dB) ↑
s0 = 00.5460.960.1524.420.88
s0 = 25k (adopted)0.5450.940.1524.390.91
s0 = 100k (one seed)0.5270.940.1514.351.12
no distillation0.4420.880.1494.311.13
  • No warm-up is needed (s0 = 0, the requested control). With the alignment acting from the first step, probing and reconstruction match the adopted s0 = 25k within seed noise.
  • A later gate trades probing for reconstruction. At s0 = 100k probing falls by 3% (8 of 19 tasks significantly worse on the evaluation items, none better) and SI-SDR rises from 0.91 to 1.12 dB, the level of the no-distillation arm. The camera-ready presents the gate as this control, not as a contribution.
  • The distillation result holds on the final backbone. Without the teacher probing falls by 19%, with 18 of 19 tasks significantly worse in both seeds, while reconstruction is slightly better, again in both seeds (FAD-MERT 6% lower, SI-SDR +0.2 dB), as in Table 5, where probing falls by 23%.

Regarding XSA and SwiGLU, which the ablations of the paper predate, we removed each from the final backbone, one at a time, with data, losses and discriminator fixed. Removing either moves probing by 1% or less, so the semantic ablations of the paper carry over to the final backbone; the two act on reconstruction. We find that XSA improves FAD-MERT and the spectral and perceptual distances, with only FAD-CLAP slightly worse, and that SwiGLU trades FAD on the mono downmix for stereo accuracy.

MetricAdopted (XSA + SwiGLU)XSA offSwiGLU off
FAD-MERT ↓0.941.000.81
FAD-CLAP ↓0.1520.1400.120
FAD-PANN ↓4.394.394.00
Mel distance ↓0.4240.4500.431
CDPAM ↓0.0850.0890.095
Pan error ↓0.2410.2400.256
d_width ↓0.08440.08470.0875
ILD (low band) ↓0.9170.9080.974
  • XSA. Without it, in both seeds, FAD-MERT rises by 6%, the mel distance by 6% and CDPAM, a learned perceptual distance, by 5%; FAD-PANN is unchanged and only FAD-CLAP is better (8% lower).
  • SwiGLU. Without it the stereo image is less accurate: the pan error, the width distance of Appendix B and the low-band level difference between the channels (ILD) are 6%, 4% and 6% worse, and CDPAM is 12% higher. The three FADs, computed on the mono downmix, which removes the stereo image, are lower.

Regarding decoder fine-tuning, we compare the end of the two phases of the paper model on the MoisesDB mixtures. We find that every reconstruction metric of Table 2 improves, while the latent, and with it probing, does not change, since the encoder is frozen.

phaseFAD-MERT ↓FAD-CLAP ↓FAD-PANN ↓CLAP music ↑CLAP audio ↑SDR (dB) ↑dSTFT ↓
end of phase 10.3050.0541.6800.9500.9584.700.867
end of phase 2 (Table 2)0.2050.0261.4380.9660.9714.790.857
change, phase 1 to phase 2-33%-52%-14%+2%+1%+0.09 dB-1%
AuthorsOpenReview ↗

Response to Reviewer oyHF (2/5): probing uncertainty and the Jam-ALT margin

Q2, W2 (probing uncertainty and the Jam-ALT margin).

Paired 95% intervals for every task confirm the lead on 17 of the 19 tasks; Jam-ALT and MoisesDB genre reranking are ties.

We thank the reviewer for pointing to the Jam-ALT margin: the difference to Music2Latent is +0.001, with a 95% interval of [-0.006, +0.007], so the 0.0004 of Table 9 is not distinguishable from zero. "SAGE leads every one of the nineteen tasks" therefore becomes, in the camera-ready: "Among the eight compared autoencoders, SAGE is significantly better than every other autoencoder on 17 of the 19 probing tasks and tied with the best of them on the other two (Jam-ALT artist retrieval, MoisesDB genre reranking)." Jam-ALT is tied with Music2Latent, SAO and ACE-Step, and MoisesDB genre reranking with SAME-L, SAME-S and Music2Latent; the MoisesDB intervals are wide because the set is small (240 songs, 43 artists).

For each task: the half-width of SAGE's 95% interval, the closest of the seven other autoencoders (the five baselines of the paper, εar-VAE and the ACE-Step 1.5 VAE), and the paired difference to it. A positive difference means that SAGE is ahead; it is significant when its interval excludes zero.

taskSAGE, ±closest autoencoderSAGE - closest [95% CI]
FMA genre classification0.014CoDiCodec+0.178 [+0.163, +0.193]
FMA genre clustering0.012Music2Latent+0.027 [+0.015, +0.039]
FMA genre reranking0.007Music2Latent+0.026 [+0.020, +0.032]
FMA artist retrieval0.035SAO+0.126 [+0.111, +0.143]
FMA artist clustering0.018SAO+0.058 [+0.051, +0.066]
FMA artist pairs0.009SAO+0.053 [+0.045, +0.060]
MoisesDB genre classification0.064SAME-L+0.084 [+0.018, +0.151]
MoisesDB genre clustering0.040SAME-L+0.035 [+0.001, +0.069]
MoisesDB genre reranking0.032SAME-L+0.020 [-0.012, +0.051], tie
MoisesDB artist retrieval0.170SAME-L+0.068 [+0.035, +0.115]
MoisesDB artist clustering0.172ACE-Step+0.045 [+0.005, +0.085]
MoisesDB artist pairs0.036ACE-Step+0.041 [+0.010, +0.069]
MoisesDB instrument0.026SAME-L+0.063 [+0.037, +0.089]
GTZAN genre0.025SAME-L+0.244 [+0.213, +0.273]
GTZAN clustering0.025ACE-Step+0.117 [+0.090, +0.144]
GTZAN reranking0.045CoDiCodec+0.134 [+0.098, +0.171]
MusicGenre clustering0.020CoDiCodec+0.167 [+0.149, +0.185]
NSynth0.007CoDiCodec+0.020 [+0.013, +0.028]
Jam-ALT artist retrieval0.008Music2Latent+0.001 [-0.006, +0.007], tie
  • Reading across the tasks. The difference is positive on all 19 tasks and significant on 17: above 0.05 on 11 (up to +0.244, GTZAN genre) and between +0.020 and +0.05 on six. The narrowest of the 17 is NSynth, the only task with repeated probes: over its 10 few-shot draws SAGE is ahead of six of the seven autoencoders, and its margin over CoDiCodec is not significant over the draws; the camera-ready reports both intervals.
  • Training seed. Two seeds of the adopted configuration at proxy data scale differ by less than 0.001 on the mean of the 19 tasks and by at most 0.036 on a single task (MoisesDB genre classification, where SAGE's margin is +0.084). On each of the 17 significant tasks, the difference between the seeds is smaller than the margin.

Evaluation sampling. Each score is the MAEB metric of its task type on the frozen encoder: linear probes with 5-fold cross-validation for classification (NSynth: 10 few-shot probes on a fixed test split), the mean over queries for retrieval and reranking, k-means for clustering. The intervals are 95% bootstrap intervals over the evaluated items (over artists in artist retrieval), 1,000 replicates, with the same resamples for every model, so that differences between models are paired; for clustering, a jackknife over groups of items (whole artists in the artist tasks) together with the spread over k-means seeds. The intervals cover the evaluation items, not a new training of the encoder (for that, see the training seeds above). The re-run also tightens two details of the submitted evaluation, for every model, so its scores differ slightly from Table 9: artist retrieval no longer counts the query as its own hit, and the FMA and Jam-ALT tasks leave out the test tracks that are also in our MTG-Jamendo training data (seven FMA artists, nine Jam-ALT songs).

AuthorsOpenReview ↗

Response to Reviewer oyHF (3/5): listening test

Q2, W2 (listening test).

The paired analysis, with intervals over listeners, separates the four systems into two levels: SAGE and SAME-L at the top with similar scores, 15 to 17 points above SAO and CoDiCodec. SAGE reaches the upper level with one eighth of SAME-L's parameters, at the speed of the SAO VAE.

Every listener rated all systems on the same ten excerpts. Computed over the 21 listeners (t-intervals on the per-listener means) rather than over the 210 ratings of each system, the intervals of Table 3 are 1.9 to 2.4 times wider and still separate two levels:

levelsystemMUSHRA ↑, mean ± 95% CI
upperSAGE81.6 ± 5.2
upperSAME-L81.8 ± 6.2
lowerCoDiCodec66.4 ± 8.4
lowerSAO VAE64.6 ± 8.6

Paired differences between systems:

paired differenceMUSHRA points [95% CI]reading
SAGE - SAME-L-0.3 [-4.9, +4.4]tie
CoDiCodec - SAO+1.8 [-2.4, +6.0]tie
SAGE - SAO+17.0 [+12.3, +21.7]SAGE higher
SAGE - CoDiCodec+15.2 [+9.6, +20.7]SAGE higher

SAME-L is likewise above SAO and CoDiCodec (+17.2 and +15.4 points, both intervals excluding zero): within a level the systems are tied, and between levels they differ by 15 to 17 points. The interval of SAGE - SAME-L does not establish strict equivalence, but it excludes a difference of more than 5 points out of 100 in either direction. In the camera-ready, Table 3 reports the intervals over listeners and the paired differences, and "statistically indistinguishable" becomes: "listeners did not separate SAGE from SAME-L (paired difference -0.3 points, 95% interval [-4.9, +4.4]); both are rated 15 to 17 points above SAO and CoDiCodec." The abstract and the conclusion follow the same wording.

Reviewer_4RAQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SAGE, a ~105M VAE for 44.1 kHz stereo music. It works on the complex STFT with a SwinV2-style encoder/decoder and keeps the same compression as the Stable Audio Open (SAO) VAE. A pooled latent descriptor is distilled toward CLAP after a short warm-up, and the decoder is fine-tuned afterwards with the encoder frozen. The authors test reconstruction on five sets, run a MUSHRA test, and probe the latent on 19 tasks.

Critical strengths
  1. The speed/quality trade-off is genuinely good. At the same speed as the SAO VAE, SAGE is better on basically every FAD/CLAP number and much better in MUSHRA (81.6 vs 64.6), and it is on par with SAME-L at about 1/8 of the size.
  2. The evaluation is honest and easy to check. Every baseline goes through the same harness, the weaker SDR is reported rather than hidden, and the released MUSHRA CSV reproduces Table 3.
Critical weaknesses
  1. The title says "Generative Encoder", but nothing is ever generated. The paper calls SAGE a drop-in first stage for diffusion (p.9), yet no generator is trained on SAGE or SAO latents, and there are no latent statistics (the KL weight is only 1e-4). SAME, the closest baseline, runs exactly this experiment. Without it, the main motivation is unsupported.
  2. "State-of-the-art semantic structure" is too strong. The probes pool the latent the same way the distillation target does, so for SAGE they mostly measure how well CLAP was distilled. CLAP itself beats SAGE on 18 of 19 tasks (Table 9), and most of the margin disappears without the teacher (0.642 to 0.495, Table 5). At best this is SOTA among these five autoencoders, at clip level.
  3. Neither contribution is really new. Complex-STFT autoencoders already exist (Music2Latent and CoDiCodec, both baselines here), and ESC (EMNLP 2024) is already a hierarchical Swin encoder-decoder on the complex STFT, so the claim on p.1 that Swin is only used for audio classification isn't right. CLAP distillation into a frequency-domain VAE is SALAD-VAE, which the paper follows, and semantic alignment of music latents is what SAME does. Related audio VAEs such as εar-VAE, Semantic-VAE and STAR-VAE (ICML 2026) aren't discussed either.
  4. What is left isn't isolated. There is no s0 = 0 run for the delayed schedule. Baselines differ in data, losses and discriminator (the discriminator alone moves FAD-MERT about 7x, Table 10), and all ablations use a smaller, older backbone. So it's hard to tell how much comes from the architecture and how much from the recipe.
  5. Smaller things. SDR is below SAO on all five sets and 5-6 dB below SAME-L, so "high fidelity" should be qualified as perceptual. Table 1's "Comp." hides the frame rate and latent width, and the MUSHRA CIs treat 210 trials as independent (per-listener CIs are about twice as wide).
Questions for authors
  1. Did you train any generator on SAGE vs. SAO latents with the same budget? If not, I'd tone down the generative and SOTA claims.
  2. What happens with s0 = 0? And how does the architecture compare with data, losses and discriminator held fixed?
  3. Were JamendoLyrics/Jam-ALT tracks removed from the MTG-Jamendo training data, as the SDD tracks were? SAGE beats its own teacher on that "unseen" task, which makes me wonder.
8 comments
AuthorsOpenReview ↗

Response to Reviewer 4RAQ (1/6): new results, generation

We thank the reviewer for the careful assessment, and for reproducing Table 3 from our released data. We are glad that the speed/quality trade-off is judged good and the evaluation honest and easy to check. In the same spirit, every control on SAGE that was asked for was run and is reported below.

New results since the submission

  • Latent diffusion at equal budget on SAGE and on three baseline latents (SAO VAE, SAME-L, CoDiCodec), on two datasets and with two generator sizes: SAGE has the lowest FAD on every metric at every checkpoint, 18 to 49% below the best other latent at the end of training (Q1, W1).
  • Ablations on the final backbone with the full adversarial setup: s0 = 0, s0 = 100k, no distillation, XSA off, SwiGLU off. The distillation gives the probing lead: without it, probing is 19% lower (Q2, W4).
  • Probing re-run with paired 95% intervals and eight autoencoders: SAGE is significantly ahead on 17 of the 19 tasks and tied with the best on the other two (W2).
  • Latent statistics for every autoencoder, with SAGE's within the range of latents already used by generators (W1); two more baselines, εar-VAE and the ACE-Step 1.5 VAE, behind SAGE on all 25 FAD and CLAP cells (W3).

Q1, W1 (generation).

In each of the three settings tested (two corpora, two generator sizes), a diffusion transformer trained at equal budget on SAGE latents reaches a lower FAD, on every metric and at every checkpoint (epochs 100, 200 and 300), than on any other latent tested (SAO VAE, SAME-L, CoDiCodec): 18 to 49% below the best of them at the end of training, and 18 to 34% on FAD-MERT and FAD-PANN, which are not tied to the CLAP teacher. The submission evaluates SAGE as an autoencoder, by reconstruction and probing; this study tests the use that motivates its latent (p. 9).

Setup. For each autoencoder, a fresh SiT (a diffusion transformer trained with flow matching, 12 blocks, one token per latent frame) learns to generate its frozen latents of 4-s stereo crops at 44.1 kHz, without conditioning. Only the latent changes. The data, generator, optimizer, schedule (300 epochs, 163k updates at batch 256), per-channel standardization and sampler (32 Euler steps, same seeds) are identical. Each model generates 2,048 clips, which are compared with 2,048 held-out validation crops by FAD in three embedding spaces. There are two corpora: FMA-medium and a Music4All subset of the same size (139k training crops each; Music4All split by artist). The generators have 130M parameters on both corpora and also 230M on FMA.

FAD at the end of training (epoch 300), lower is better:

DatasetGeneratorLatentFAD-CLAP ↓FAD-MERT ↓FAD-PANN ↓
FMA130MSAGE0.0701.436.67
FMA130MSAO VAE0.1081.749.95
FMA130MCoDiCodec0.1582.2410.62
FMA130MSAME-L0.1712.1112.52
FMA230MSAGE0.0641.196.24
FMA230MSAO VAE0.1011.609.44
FMA230MCoDiCodec0.1462.049.92
FMA230MSAME-L0.1571.9111.55
Music4All130MSAGE0.0581.425.72
Music4All130MSAO VAE0.1452.068.91
Music4All130MCoDiCodec0.1132.588.51
Music4All130MSAME-L0.2132.189.21

SAGE against SAO. Both latents have 64 channels at 21.5 Hz, so the two generators also cost the same compute. On FMA, SAO is the strongest other latent, second on every metric at both sizes. At epoch 300, SAGE's FAD is lower by (CLAP, MERT, PANN):

  • 35, 18 and 33% on FMA (130M);
  • 37, 26 and 34% on FMA (230M);
  • 60, 31 and 36% on Music4All.
AuthorsOpenReview ↗

Response to Reviewer 4RAQ (2/6): latent statistics and KL weight

Q1, W1 (latent statistics and KL weight).

On these statistics, computed for all eight autoencoders, SAGE's latent raises no obstacle to a diffusion model: its scale is close to 1, its tails are near-Gaussian, all 64 channels are active and the posterior is not deterministic. Its distance from N(0, I) is within the range of autoencoders that already have generators trained on their latents (SAO and ACE-Step below it, SAME above). The KL weight, 1e-4, is the one in the public training configuration of the Stable Audio VAE with the same latent size, and at equal budget the generator above reaches a lower FAD on SAGE's latent than on SAO's.

latent (channels × Hz)std (→ 1)KL to N(0, I) ↓excess kurtosis (→ 0)
SAGE64 × 21.50.880.280.28
SAO VAE64 × 21.50.960.090.08
εar-VAE64 × 43.10.870.200.33
ACE-Step VAE64 × 250.880.100.08
SAME-L256 × 10.80.710.440.49
SAME-S256 × 10.80.680.460.47
CoDiCodec64 × 10.80.930.041.37
Music2Latent64 × 10.80.900.200.14

KL: divergence from N(0, I) of the Gaussian fit of the latent (as in FAD), in nats per dimension. Kurtosis: median over channels; it measures the tails, which the Gaussian fit does not see. All on the latent a generator is trained on (a posterior sample for the VAEs), over every frame of 1,998 MoisesDB mixtures of 10 s; Music2Latent per audio channel.

  • About 60% of SAGE's KL is per-channel offset and scale, which a per-channel standardization of the latent removes.
  • SAGE's posterior σ is 0.025, against 0.07 for SAO; εar-VAE and ACE-Step are nearly deterministic (0.001, 0.0004).

The camera-ready adds these statistics, on MoisesDB and on FMA, to the appendix.

AuthorsOpenReview ↗

Response to Reviewer 4RAQ (3/6): ablations on the final backbone; architecture and recipe

Q2, W4 (ablations on the final backbone; architecture and recipe).

On the final backbone, the distillation needs no warm-up: s0 = 0 gives the same results as the adopted s0 = 25k. The distillation itself is what matters: without it the probing mean falls by 19%. The older backbone differs from the final one only in XSA and SwiGLU, and removing either moves probing by 1% or less, so the semantic studies of the paper carry over. The baselines are compared as released systems (last paragraph).

The new runs use the final architecture at full size with the adversarial setup of the paper, on less data (FMA-medium, 95 epochs, phase 1 only; at full scale phase 1 alone takes 1,536 A100 hours per run), so absolute values are not those of Table 2 or 5. Two seeds per arm (one for s0 = 100k), mean of the two; reconstruction on the 1,998 MoisesDB mixtures.

armprobes, mean of 19 ↑FAD-MERT ↓FAD-CLAP ↓FAD-PANN ↓SI-SDR (dB) ↑
s0 = 00.5460.960.1524.420.88
s0 = 25k (adopted)0.5450.940.1524.390.91
s0 = 100k (one seed)0.5270.940.1514.351.12
no distillation0.4420.880.1494.311.13
  • No warm-up is needed. With s0 = 0, the alignment acting from the first step, probing and reconstruction match the adopted s0 = 25k within seed noise.
  • A later gate trades probing for reconstruction. At s0 = 100k probing falls by 3% (8 of 19 tasks significantly worse on the evaluation items, none better) and SI-SDR rises from 0.91 to 1.12 dB, the level of the no-distillation arm. The camera-ready presents the gate as this control.
  • The distillation result holds on the final backbone. Without the teacher probing falls by 19%, with 18 of 19 tasks significantly worse in both seeds, while reconstruction is slightly better, again in both seeds (FAD-MERT 6% lower, SI-SDR +0.2 dB), as in Table 5.
  • XSA improves spectral fidelity; SwiGLU trades FAD for stereo accuracy. Without SwiGLU the three FADs (computed on the mono downmix) are lower, but the pan error, the width distance of Appendix B and the low-band level difference between the channels (ILD) are 6%, 4% and 6% worse, and CDPAM, a learned perceptual distance, is 12% higher.

The discriminator and capacity studies (Tables 10 and 11) stay on the older backbone; the camera-ready states which backbone each result comes from.

Architecture and recipe. We compare SAGE with the existing autoencoders as released, which is the choice faced by whoever needs a latent for a generator. Retraining them with our data, losses and discriminator is beyond our budget (one training of SAGE takes 4,579 A100 hours, and there are seven baselines), and for four of them (Music2Latent, CoDiCodec, εar-VAE, ACE-Step 1.5 VAE) it is not possible: their repositories release inference code and weights but no code to train the autoencoder (ACE-Step's training code fine-tunes its generator only). Table 2 therefore compares systems; within SAGE, the ablations above separate the distillation, the gate, XSA and SwiGLU, and Table 10 attributes most of the distributional metrics to the discriminator. The camera-ready words contribution 1 as the system and its measured result. What we hold fixed is the evaluation: one harness for every system, on five sets that include each baseline's own benchmark.

Comments on the paper

2 comments
Area_Chair_DzhoOpenReview ↗

Reminder: Discussion Phase Ends November 18 (11:59 PM AoE)

Hi reviewers, a quick reminder that the discussion phase ends on November 18 at 11:59 PM AoE. Please take some time to read the author responses, engage in further discussions if you have any remaining questions or concerns, and update your reviews and scores as appropriate. Thank you!

AuthorsOpenReview ↗

Summary of new results

We thank the three reviewers for their careful and constructive reviews. Each question is answered in the thread of its review and the new results shared across them are:

  • Generation. At equal budget, a diffusion transformer trained on SAGE latents reaches the lowest FAD (lower is better) on every metric and at every checkpoint, against the SAO VAE, SAME-L and CoDiCodec latents, on two datasets and with two generator sizes: at the end of training its FAD is 18 to 49% lower than with the best other latent (18 to 34% lower on FAD-MERT and FAD-PANN).
  • Ablations on the final backbone with the full adversarial setup: the CLAP distillation gives the probing lead (19% lower without it); removing XSA or SwiGLU moves probing by 1% or less, the two act on reconstruction; decoder fine-tuning improves every reconstruction metric without changing the latent.
  • Probing with paired 95% intervals and two more baselines, εar-VAE and the ACE-Step 1.5 VAE: among eight autoencoders, SAGE is significantly ahead on 17 of the 19 tasks and tied on the other two; the two new baselines are behind SAGE on all 25 FAD and CLAP cells.
  • Listening test, analysed over listeners and paired: SAGE and SAME-L are tied (-0.3 points, 95% interval [-4.9, +4.4]), both 15 to 17 points above the SAO VAE and CoDiCodec; SAGE has one eighth of SAME-L's parameters and runs at the speed of the SAO VAE.

Tables, protocols and the camera-ready wording are in the thread of each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.