- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SAGE is a 105M-parameter VAE for 44.1 kHz stereo music. It works on the power-compressed complex STFT with a three-stage SwinV2 encoder/decoder adapted to spectrograms (64×1 patches, 4×32 attention windows, SwiGLU, exclusive self-attention). A linear head maps the time-averaged latent to a frozen LAION-CLAP embedding under a cosine loss, following the clip-level scheme of SALAD-VAE. The head is discarded at inference. The encoder receives this gradient only after a stop-gradient warm-up of steps.
Training uses STFT, mel, mid/side STFT, KL and RpGAN/feature-matching losses on about 10.5k hours of FMA, MTG-Jamendo and M4Singer. A second phase fine-tunes only the decoder. The latent budget matches Stable Audio Open (×64, 64 channels at about 21.5 Hz).
All five baselines (Stable Audio Open VAE, SAME-L/S, CoDiCodec, Music2Latent) are re-run from public weights through one harness on five sets.
- Reconstruction. SAGE has the best FAD (MERT, CLAP and PANN spaces) and CLAP similarity in 23 of the 25 set×metric cells.
- Speed. It ties Stable Audio Open on compiled real-time factor (0.0045).
- Listening test. A 10-excerpt MUSHRA against SAME-L, Stable Audio Open and CoDiCodec (21 listeners retained) gives SAGE 81.6 ± 2.7 and SAME-L 81.8 ± 2.6.
- Probing. It leads all five autoencoders on 19 probing tasks built around genre, artist and instrument labels.
The semantic, discriminator and capacity studies (Sec. 3.5, App. E) use a smaller, older backbone on FMA-medium; App. B ablates the stereo loss under the full setup.
Critical strengths
- Every baseline is run from its released weights, one call per clip, on five sets. These include each baseline's own benchmark: MusicCaps for Music2Latent/CoDiCodec and Song Describer for Stable Audio Open/SAME. Metrics are three FAD spaces, two CLAP checkpoints and SDR/STFT diagnostics. SAGE's advantage is broadly consistent across sets (23 of 25 cells, Table 2). The common harness is the main reason I believe the reconstruction claims.
- At the same compiled RTF as Stable Audio Open (44.8 vs 45.2 ms per 10 s clip), SAGE scores about 17 MUSHRA points higher. It is roughly level with SAME-L, which has 8× the parameters, about 4× the latency and twice the latent budget (×32 vs ×64). The authors also check the obvious worry that CLAP-based metrics favour a CLAP-distilled model. In the reduced, non-adversarial teacher ablation (Table 5, lower block), CLAP distillation lowers both reconstruction CLAP similarities (0.613/0.575 vs 0.645/0.624). FAD-MERT and FAD-PANN are not tied to the teacher.
Critical weaknesses
-
The semantic claim is supported only by genre/artist/instrument probes on the pooled latent, and never through the use it is motivated by.
- What is probed. aligns a projection of the time-averaged, frequency-folded latent with CLAP (Eq. 1). The probes read that same pooled latent (Sec. 3.2). The 19 tasks use several formats (classification, clustering, retrieval, reranking, pair classification), but they cover only three semantic categories: genre (10 tasks), artist (7) and instrument (2). These categories are plausibly aligned with what the CLAP teacher encodes. Winning them is not guaranteed by the loss, but it is not surprising either.
- What is not probed. There is nothing on key, chords, tempo/beat or melody. These matter for a music generator's latent, and SAME explicitly supervises pitch-class structure through chroma targets.
- No uncertainty. No probing uncertainty is reported. Several gains over the autoencoders are large (GTZAN accuracy 0.774 vs 0.531), but some are tiny (Jam-ALT 0.9672 vs 0.9668). The CLAP reference stays ahead on 18 of 19 tasks.
- No downstream generator. The Introduction and Sec. 4 justify the semantic latent by its value to a downstream diffusion model ("drop-in first stage"), but the paper reports no generator trained on SAGE latents. SAME, the main baseline, does exactly this: it trains a ~1.4B DiT on its latent and shows that its semantic losses improve generation (FAD-CLAP 0.593 → 0.576, MuQ-Eval 3.34 → 3.87). As evaluated, "state-of-the-art semantic structure" means "best pooled-latent probe scores among these six autoencoders on genre/artist/instrument". That is narrower than the abstract suggests.
-
The ablations do not isolate the method contributions.
- Delayed schedule. Table 5 sweeps but never , so delay versus no delay is untested. The gate opens after about 8.3k generator updates, roughly 3.5% of phase-1 training. No repeated-seed variability is reported, so the small 25k-vs-50k differences (FAD-MERT 0.240 vs 0.248, probes 0.637 vs 0.629) are hard to interpret. SALAD-VAE, whose objective SAGE adopts, already uses a curriculum: it pre-trains on reconstruction and KL and adds the CLAP, contrastive and adversarial terms "after several epochs". The new part is therefore the stop-gradient variant, in which the head trains from step 0, and that is the part left untested.
- Architecture. The semantic, discriminator and capacity studies use a reduced backbone that predates SwiGLU and XSA, on FMA-medium. Appendix E varies width and depth (Table 11). None of the spectrogram-specific choices in contribution 1 is isolated: rectangular patches/windows, the three-stage hierarchy, XSA, the post-net.
- Phase 2. Table 10 studies discriminator placement across phases, but there is no controlled phase-1 vs phase-2 comparison for the final model. Phase 2 is 992 epochs and about two thirds of the compute (3,043 of 4,579 GPU-h).
- Consequence. Appendix E shows the discriminator alone moves FAD-MERT from 1.453 to 0.203. The baselines also differ in data, compression (×32 for SAME, ×128 for CoDiCodec), discriminator and training length. Table 2 therefore compares systems, and we cannot tell which part of SAGE produces the gain.
Smaller points (not decisive):
- MUSHRA. "Statistically indistinguishable" / "matches SAME-L" rests on overlapping marginal CIs. There are 10 excerpts, 17 of 38 participants were filtered out, and no paired test or equivalence margin is reported.
- SDR is below Stable Audio Open on all five sets (5.28 vs 5.64 on FMA, 6.31 vs 6.86 on Song Describer) and 4.7–6.2 dB below SAME-L. Treating SDR as diagnostic is defensible for an adversarially trained decoder, but it deserves a sentence next to the headline claims.
- Missing open music autoencoders: εar-VAE (44.1 kHz stereo, open weights), the ACE-Step 1.5 VAE and Music2Latent2. The first two appear in SAME's own listening test (MUSHRA 77.6 and 76.5, against 82.2 for SAME-L), so they are the obvious next-strongest comparisons.
- Table 11's within-sweep comparisons are controlled, but the conclusion that depth gives a larger return per parameter than width compares sweeps run with different .
- Please state the license for the released weights; M4Singer is CC BY-NC-SA 4.0.
Questions for authors
- (Could change my score.) Can you add and a no-distillation run with the final backbone and the full adversarial setup? If the stop-gradient gate matters at full scale, contribution 2 stands.
- (Could change my score.) Train even a small latent diffusion model on SAGE latents and on Stable Audio Open latents at equal budget. If the semantic latent makes generation easier, that is the result the paper is built around.
- (Clarification.) You filter listeners on hidden-reference ratings. What exact threshold and participant-level rule were applied, how were the CIs constructed, and what is the paired SAGE − SAME-L difference with its CI?
- (Clarification.) For the track-grouped cross-validation: how many folds, and do stems inherit their parent recording's group? Does artist retrieval exclude other excerpts of the same recording?
- (Suggestion.) Probes for key, chords or beat (e.g. GiantSteps key) would show whether the latent is semantic in a musical sense or mainly in a tagging sense.
- (Clarification.) What does phase 2 add? Please give phase-1-only reconstruction and listening numbers for the final model.
6 comments
Response to Reviewer 3njo (2/6): generation
Q2, W1 (generation).
At equal budget, a diffusion transformer generates closer to the data on SAGE latents than on Stable Audio Open latents, the comparison the review proposes, and than on those of SAME-L and CoDiCodec: lower FAD on every metric and at every checkpoint, in each of the three settings tested (two corpora, two generator sizes). As in Table 2, these are released autoencoders, which also differ in data and recipe (W2, consequence): the study measures the latent each system gives a generator.
Setup. For each autoencoder, a fresh SiT (a diffusion transformer trained with flow matching, 12 blocks, one token per latent frame) learns to generate its frozen latents of 4-s stereo crops at 44.1 kHz, without conditioning. Only the latent changes. The data, generator, optimizer, schedule (300 epochs, 163k updates at batch 256), per-channel standardization and sampler (32 Euler steps, same seeds) are identical. Each model generates 2,048 clips, compared by FAD with 2,048 held-out validation crops in three embedding spaces. There are two corpora: FMA-medium and a Music4All subset of the same size (139k training crops each; Music4All split by artist). The generators have 130M parameters on both corpora and also 230M on FMA.
FAD at the end of training (epoch 300), lower is better:
| Dataset | Generator | Latent | FAD-CLAP ↓ | FAD-MERT ↓ | FAD-PANN ↓ |
|---|---|---|---|---|---|
| FMA | 130M | SAGE | 0.070 | 1.43 | 6.67 |
| FMA | 130M | SAO VAE | 0.108 | 1.74 | 9.95 |
| FMA | 130M | CoDiCodec | 0.158 | 2.24 | 10.62 |
| FMA | 130M | SAME-L | 0.171 | 2.11 | 12.52 |
| FMA | 230M | SAGE | 0.064 | 1.19 | 6.24 |
| FMA | 230M | SAO VAE | 0.101 | 1.60 | 9.44 |
| FMA | 230M | CoDiCodec | 0.146 | 2.04 | 9.92 |
| FMA | 230M | SAME-L | 0.157 | 1.91 | 11.55 |
| Music4All | 130M | SAGE | 0.058 | 1.42 | 5.72 |
| Music4All | 130M | SAO VAE | 0.145 | 2.06 | 8.91 |
| Music4All | 130M | CoDiCodec | 0.113 | 2.58 | 8.51 |
| Music4All | 130M | SAME-L | 0.213 | 2.18 | 9.21 |
SAGE against Stable Audio Open. Both latents have 64 channels at 21.5 Hz, so the budget is equal in updates and in compute. On FMA, SAO is the strongest of the other latents, second on every metric at both sizes. At epoch 300, SAGE's FAD is lower by (CLAP, MERT, PANN):
- 35, 18 and 33% on FMA (130M);
- 37, 26 and 34% on FMA (230M);
- 60, 31 and 36% on Music4All.
It is 18 to 60% lower at every checkpoint.
Against the other autoencoders. SAGE has the lowest FAD in every setting, on every metric and at every checkpoint. At epoch 300 it is 18 to 49% below the best other latent (18 to 34% on FAD-MERT and FAD-PANN, which are not tied to the CLAP teacher). Against SAME-L, the main baseline, the gap at epoch 300 is:
- 32 to 38% on FAD-MERT;
- 38 to 47% on FAD-PANN;
- 59 to 73% on FAD-CLAP.
Response to Reviewer 3njo (3/6): MUSHRA and probing protocol
Q3 (MUSHRA: screening rule, intervals, paired difference).
The rule is the post-screening of ITU-R BS.1534-3. Over listeners, the intervals separate two levels: SAGE and SAME-L at the top, 15 to 17 points above SAO and CoDiCodec, with a paired SAGE - SAME-L difference of -0.3 points [-4.9, +4.4]. SAGE reaches the upper level with one eighth of SAME-L's parameters and at twice its compression.
- Rule. A participant is excluded when they rate the hidden reference below 90 in more than 15% of the trials, i.e. in two or more of the ten. The 21 retained participants have at most one such trial; the 17 excluded have two to eight.
- Scale. A trial is one listener rating all systems on one excerpt; our test has 210 (21 listeners × 10 excerpts), in line with or above comparable work: the MUSHRA test of SAME, whose scores the review cites for εar-VAE and ACE-Step, has 36 valid trials (12 participants, after filtering on anchor and reference ratings), and the test of DAC has 120 (10 expert listeners × 12 samples of 10 s, four of them music).
- Intervals. Table 3 gives the mean ± 1.96 standard errors over the 210 ratings of a system, treated as independent. Since every listener rated all systems on the same ten excerpts, the unit is the listener: over the 21 per-listener means (95% t-intervals) the intervals are 1.9 to 2.4 times wider (SAGE 81.6 ± 5.2, SAME-L 81.8 ± 6.2, CoDiCodec 66.4 ± 8.4, SAO 64.6 ± 8.6).
Paired differences, computed per listener in the same way:
| paired difference | MUSHRA points [95% CI] | reading |
|---|---|---|
| SAGE - SAME-L | -0.3 [-4.9, +4.4] | tie |
| CoDiCodec - SAO | +1.8 [-2.4, +6.0] | tie |
| SAGE - SAO | +17.0 [+12.3, +21.7] | SAGE higher |
| SAGE - CoDiCodec | +15.2 [+9.6, +20.7] | SAGE higher |
SAME-L is likewise above SAO and CoDiCodec (+17.2 and +15.4 points, both intervals excluding zero): within a level the systems are tied, and between levels they differ by 15 to 17 points. The interval of SAGE - SAME-L does not establish strict equivalence, but it excludes a difference of more than 5 points out of 100 in either direction. In the camera-ready, Table 3 reports the intervals over listeners and the paired differences, and "statistically indistinguishable" becomes: "listeners did not separate SAGE from SAME-L (paired difference -0.3 points, 95% interval [-4.9, +4.4]); both are rated 15 to 17 points above SAO and CoDiCodec." The abstract and the conclusion follow the same wording.
Q4 (probing protocol).
- Folds. Five in every cross-validated task (NSynth uses its fixed test split).
- Stems. Yes. MoisesDB instrument classification, the only task on stems, groups the folds by recording; in the other cross-validated tasks each recording contributes one excerpt, so no recording is split across folds.
- Artist retrieval. On FMA and MoisesDB each recording contributes one excerpt, so there is no other excerpt to exclude. We thank the reviewer for this question, which led us to check the query itself: in the submitted evaluation it was in its own corpus and counted as a hit, for every model. The re-run removes it, and SAGE stays significantly ahead of the closest autoencoder: +0.126 [+0.111, +0.143] on FMA (SAO), +0.068 [+0.035, +0.115] on MoisesDB (SAME-L). In Jam-ALT, taken unchanged from MAEB, each clip is one lyric line and the other lines of the same song count as relevant by design; since almost every artist has a single song, this task does reward finding other excerpts of the same recording. There too we removed only the self-match, and SAGE is tied with Music2Latent, SAO and ACE-Step (+0.001 [-0.006, +0.007] against Music2Latent).
The camera-ready states this protocol in Sec. 3.2.
Response to Reviewer 3njo (1/6): new results, delayed schedule and distillation
We thank the reviewer for the careful and precise assessment, and for stating, for each concern, the experiment that would address it and the results that could change the rating. We are glad that the common harness makes the reconstruction claims credible, that the speed/quality trade-off is judged good, and that our check that CLAP distillation does not inflate the CLAP-based metrics is acknowledged. We ran the suggested experiments and report them below.
New results since the submission
Q1, W2 (delayed schedule and distillation on the final backbone).
The two requested runs separate the parts of contribution 2. The distillation is what makes the latent semantic: without it probing falls by 19%, on 18 of 19 tasks. The stop-gradient warm-up is not needed: s0 = 0 matches the adopted s0 = 25k. The camera-ready therefore states contribution 2 as the distillation and its measured cost (below), with the warm-up as a control. We trained the final architecture with the full adversarial setup of the paper at proxy data scale (FMA-medium, 95 epochs, phase 1 only; at full scale phase 1 alone takes 1,536 A100 hours per run), so absolute values are not those of Table 2 or 5. Each arm has two seeds (one for s0 = 100k) and the table gives their mean; reconstruction is measured on the 1,998 MoisesDB mixtures.
In the camera-ready, contribution 2 becomes: "Semantic structure at a small reconstruction cost. We distill CLAP into the latent and measure, on the final backbone with the full adversarial setup, what it buys and what it costs: without it the mean of 19 probing tasks falls by 19%, while reconstruction improves only by about the spread between two training runs (without the discriminator, Table 5, the cost is larger); the cost falls on tonal content, where the key and chord probes are higher without it. No warm-up is needed: the alignment can act from the first step, and a long stop-gradient warm-up costs probing." Secs. 2.2 and 3.5 and the conclusion are reworded accordingly.