- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
In this paper, the authors proposes a method for exploiting latent spectral equivariance for fine tuning VAEs in audio generation, as a mean to learn structured audio latents. More precisely, the authors propose temporal scale equivariance and spectral granularity equivariance. The principle is applied on the VAE's of two well known audio generation systems, namely Stable Audio Open and MMAudio.
Critical strengths
The main idea of the paper is interesting. In fact, aiming at building structured latents by exploiting equivariance principles, and in studying the merits of the proposed methods in audio generation (both in text-to-audio and Video-to-audio) are clear strengths of the paper.
Critical weaknesses
Nevertheless, the paper suffers from several wekanesses.
-
Presentation and readability. The paper is written in excellent Engish but the paper is still difficult to follow. Despite a rather simple mathematical framework, the mathematical notations and design choices for the presentation unecessarily complexify the reading. For instance, in equation (1) x can either be a time-domain waveform or a spectrogram... then all successive operations on x becomes more difficult to understand because each operation is different depending of the nature of x. The description of the Latent temporal transform using DFT may probably be described in simpler terms using the formalism of the scattering transform. The vocabulary chosen may also be additionaly confusing. The notion of coarseness in audio can be misleading and it is difficult for me to understand exaclty what is a coarse audio sequence (when talking of the wafevorm).. even a coarse spectrogram is not too clear what it is exactly (a spectrogram with empty spaces, with empty spaces only on the upper part ... in section 3.3, why using "leading coefficient" for mentioning the zeroth order cepstral coefficient (or alternatively first ceptral coefficient) which are much more common used terms ? Link to this, they are a number of confusing sentences such as "..low-pass filtering the waveform changes acoustic bandwidth rather than temporal modulation..(it is mostly correct but only if you asume that temporal modulations are only "low-frequency temporal modulations" ) .. or such as "For a spectrogram VAE, the input is already a log-mel spectrogram" (we are then confused between the spectrogram, the log-spectrogrm, the log-mel spectrogram). The figure 2 is not very clear. It is quite small and needs to be viewed on screen with a strong zoom to view everything. Then, the variation of the "mean log magnitude per frame" (for the temporal scale equivariance) is not intuitive (is it wrong ?) since it exibits (at some points) low values for the frames where the spectrogram has high amplitude values.
-
Experiments and evaluation: a large number of experiments are conducted. The results provided using objective metrics show the potential interest of the method proposed. Nevertheless, it is known that such objective metrics are far from being well-correlated to perception. To better assess the merit of the methods, it would have been necessary to at least provide a comprehensive demo on a companion web site or a perceptual test. Besides, it is not clear what is the advantage brought by the method by the simple fact of doing fine-tuning (or increasing learning) using augmented data (low pass filtered data with different cut-off frequencies). A comparison with a system finetuned (without equivariance) but with the same additional augmented data would have been interesting. Finally, it is not clear what is the added complexity of the fine-tuning strategy and if it is indeed marginal compared to the original training complexity.
-
Reproducibility: there are many implementation details given but this usually does not suffice for reproducibility. Distributing the code in would guarantee a higher level of reproducibility .
-
bibliography, Related work, supplementary material: The related work section is placed in the supplementary material which I think is not appropriate. The related work section is an important part of the paper since it allows the authors to well position their work in the State of the Art... and to better show what is new. This is not only additional details or evidence to suport the main text (which is what the supplementary material should be about). Besides, if numerous references are given I believe that important references in the use of equivariance principles in audio or music processing are missing (see for instance work in equivariance in self-supervised learning [1-3]) [1]: Sechaud & al. "Equivariance-based self-supervised learning for audio signal recovery from clipped measurements, Eusipco 2024 [2] Lostanlen & al. "Understanding Equivariant Self-Supervised Learning in Musical Pitch Class Space", IEEE Signal Processing Letters, 2025 [3] Riou &l . PESTO: Real‑Time Pitch Estimation with Self‑Supervised Transposition‑Equivariant Objective, Trans. of ISMIR, 2025
The formatting of the reference list can also be improved mostly those of IEEE where the year, the name of the conference, the IEEE acronyms are duplicated in the references.
Questions for authors
-
How can the readibility of the paper be improved ?
-
what is the complexity burden of the proposed approach ?
-
what is the perceptual quality of the generated audio (perceptual test or demo page) ?
-
How reproducbility can be improved ?
-
what is gain of the proposed approach compared to only performing fine-tuning with the additional augmented data ?
6 comments
Response to Reviewer oaj1 (Part 2/4)
Q1 & W1: Presentation and readability (continued)
Spectrogram, log-spectrogram, and log-mel spectrogram. Section 2 defines as the STFT magnitude spectrograms at several resolutions for the waveform VAE and as the log-mel input itself for the spectrogram VAE. The quoted sentence means that the target of the spectrogram VAE is already logarithmic, so the DCT mask of Eq. 11 is applied to it directly. We will restate this definition next to that sentence and use "log-mel spectrogram" consistently for the spectrogram VAE.
Figure 2. The figure is compact because of the page limit of the main text, and all of its details remain visible when zooming in the PDF. For convenience, both panels are also available at full size on the demo page (https://latenteq.github.io/, section "Figure 2, Enlarged"), and we are willing to include enlarged versions in the appendix of the revised paper. The curve is not wrong. It follows the low-frequency band at the bottom of each panel, which is the highest-level region of this clip, and its minima coincide with the dips of that band, as can be checked in the enlarged figure. The mid-frequency stripes look prominent on the logarithmic colour scale, but their level is well below that of the low-frequency band, and they do not coincide in time with its peaks. This is why the curve can be low at frames where these stripes appear. We will explain this in the caption.
We are willing to make all of the changes above in the revised paper.
Q2 & W2-c: Complexity of the fine-tuning strategy
We thank the reviewer for this question. The added cost of LatentEQ is confined to VAE fine-tuning, and it is small there.
Per-step cost. LatentEQ adds no parameters, no network, and no loss term, and each example is still encoded once and decoded once (Algorithm 1). The only added operations are a DFT mask on the latent and DFT and DCT masks on the target spectrograms that the reconstruction loss already computes. We measured the training throughput of both VAEs on 8 A100 GPUs with a batch size of 64:
| Backbone | FT VAE (it/s) | LatentEQ, TSE + SGE (it/s) | Throughput drop |
|---|---|---|---|
| Stable Audio Open (waveform VAE) | 1.014 | 0.982 | 3.2% |
| MMAudio (spectrogram VAE) | 1.781 | 1.718 | 3.5% |
LatentEQ lowers the throughput by less than 4% on both VAEs.
Relative to the original training. For the waveform VAE, fine-tuning for 500k steps at this throughput takes about 141 hours on 8 A100 GPUs, which is about 1.1k GPU-hours. Stable Audio Open reports 183 hours followed by 273 hours on 32 A100 GPUs for its VAE, which is about 14.6k GPU-hours. The fine-tuning therefore amounts to about 8% of the original VAE training.
Diffusion training and inference. The VAE architecture and the latent dimensions are unchanged, and no transform is applied during diffusion training or sampling (Section 3.5), so the inference cost is identical to that of the original systems. As with any change to the VAE, the diffusion model has to be trained on the new latents. It keeps its original architecture and training objective, and Figure 4 shows that the Stable Audio Open DiT converges faster on LatentEQ latents.
Compared with other latent regularization methods. The cost is in fact an advantage of LatentEQ. Semantic alignment methods such as SAME, KVAE, and MiniMax H3 need an external model during VAE training. STAR-VAE trains a new VAE architecture from scratch, for about 150 hours followed by about 70 hours on 24 H800 GPUs as reported, and it further introduces a new generation framework. LatentEQ adds no module and only fine-tunes an existing VAE, and it requires no additional design in the generation stage.
We are willing to add this analysis to Appendix C of the revised paper.
Response to Reviewer oaj1 (Part 3/4)
Q3 & W2-a: Perceptual quality (listening test and demo page)
We thank the reviewer for this point. We agree that objective metrics alone do not establish perceptual quality, and we now provide both a demo page and listening tests.
Demo page. An anonymous demo page is available at https://latenteq.github.io/. It covers the four settings evaluated in the paper: text-to-audio on AudioCaps (Table 1), video-to-audio and video-and-text-to-audio on VGGSound (Table 2), and VAE reconstruction on AudioCaps and the Song Describer Dataset (Table 3). Each example places the ground truth, baselines from the corresponding table, and our outputs side by side, together with their mel spectrograms.
Listening tests. 15 listeners rated the generated audio on a five-point scale, with all stimuli anonymized and presented in random order. For T2A, they rated the overall quality (OVL) and the relevance to the text (REL). For V2A, they rated the audio quality (MOS-Q) and the temporal synchronization with the video (MOS-T). For VT2A, they additionally rated the semantic alignment with the video and the text (MOS-A). Reconstruction quality is evaluated with a MUSHRA test, with a hidden reference and a 3.5 kHz low-pass anchor, on ten clips, five from AudioCaps (Sound) and five from the Song Describer Dataset (Music). All tables report the mean score with its 95% confidence interval.
T2A on AudioCaps (8 prompts)
| Method | OVL↑ | REL↑ |
|---|---|---|
| Ground truth | 4.34 ± 0.15 | 4.53 ± 0.15 |
| AudioLDM | 3.38 ± 0.20 | 3.38 ± 0.21 |
| Tango | 2.83 ± 0.22 | 2.98 ± 0.20 |
| MMAudio | 3.68 ± 0.20 | 3.23 ± 0.21 |
| Ours (spectrogram) | 3.96 ± 0.16 | 3.78 ± 0.18 |
| TangoFlux | 3.86 ± 0.16 | 3.82 ± 0.19 |
| AudioX | 4.07 ± 0.19 | 3.83 ± 0.20 |
| Stable Audio Open | 4.20 ± 0.15 | 3.55 ± 0.21 |
| Ours (waveform) | 4.21 ± 0.16 | 3.93 ± 0.18 |
V2A on VGGSound (6 videos)
| Method | MOS-Q↑ | MOS-T↑ |
|---|---|---|
| Ground truth | 4.06 ± 0.19 | 4.32 ± 0.18 |
| V-AURA | 3.29 ± 0.14 | 3.53 ± 0.15 |
| FRIEREN | 3.21 ± 0.26 | 3.48 ± 0.18 |
| AudioX | 3.43 ± 0.15 | 3.16 ± 0.23 |
| MMAudio | 3.49 ± 0.10 | 3.68 ± 0.18 |
| Ours | 3.76 ± 0.17 | 3.77 ± 0.14 |
VT2A on VGGSound-Omni (6 videos)
| Method | MOS-Q↑ | MOS-A↑ | MOS-T↑ |
|---|---|---|---|
| Ground truth | 4.42 ± 0.11 | 4.41 ± 0.15 | 4.41 ± 0.16 |
| ThinkSound (w/o CoT) | 3.43 ± 0.12 | 3.39 ± 0.24 | 3.71 ± 0.18 |
| HunyuanVideo-Foley | 3.61 ± 0.13 | 3.52 ± 0.15 | 3.90 ± 0.15 |
| AudioX | 3.50 ± 0.12 | 3.24 ± 0.19 | 2.99 ± 0.21 |
| MMAudio | 3.64 ± 0.11 | 3.58 ± 0.16 | 3.93 ± 0.14 |
| Ours | 3.67 ± 0.21 | 3.46 ± 0.20 | 4.10 ± 0.10 |
VAE reconstruction (MUSHRA, 0–100)
| Method | MUSHRA↑ |
|---|---|
| Hidden reference | 96.99 ± 1.18 |
| 3.5 kHz low-pass anchor | 47.12 ± 2.51 |
| AudioLDM | 67.01 ± 2.53 |
| Make-An-Audio 2 | 71.82 ± 2.80 |
| MMAudio | 76.19 ± 2.57 |
| Ours (spectrogram) | 77.09 ± 2.15 |
| HunyuanVideo Foley | 70.27 ± 2.33 |
| ar-VAE | 72.37 ± 2.34 |
| Stable Audio Open | 74.71 ± 2.74 |
| Ours (waveform) | 81.21 ± 1.72 |
Across all four listening tests, the LatentEQ models are rated the best among the evaluated systems, or above or on par with the systems they build on.
We are willing to add the listening tests to the revised paper and the link to the demo page to its abstract.
Q4 & W3: Reproducibility
We thank the reviewer for this point. We have reported the training details as completely as we could in the paper. Algorithm 1 gives the training procedure. Appendix A describes the datasets and their preprocessing, Appendix B lists the evaluation toolkits, and Appendix C specifies the cutoff schedule of TSE, the twelve pairs of SGE (Table 5), the construction of the targets for the reconstruction and adversarial terms, and the training and inference settings of the four models (Table 6). The method is implemented on top of the public code of Stable Audio Open, MMAudio, and Make-An-Audio 2 without architectural change, all training datasets are public, and all metrics are computed with public toolkits. The two paired transforms are the only additions to these code bases, so we are willing to include their reference implementation in the appendix of the revised paper, and we will also seek to release our full code and checkpoints.
Response to Reviewer oaj1 (Part 1/4)
We thank the reviewer for the detailed review, and for considering the main idea, building structured latents through equivariance and studying its merits in text-to-audio and video-to-audio generation, a clear strength of the paper. We address each question and concern below.
Main Comments:
Q1 & W1: Presentation and readability
We thank the reviewer for the detailed reading. We answer each point below and state the corresponding change to the paper where one is needed.
Two forms of x in Eq. 1. We would like to clarify that the operations of LatentEQ do not depend on the form of x. Every operation acts on the latent z and on the target spectrogram S(x), which have the same form for both VAE types. The form of x only determines S, which computes STFT magnitude spectrograms at several resolutions for a waveform and is the identity for a log-mel input. To make this explicit, we will state it at the beginning of Section 3 and summarize the differences of the spectrogram VAE in one paragraph: S is the identity, there is a single resolution, and Eq. 11 needs no logarithm or exponential.
Relation to the scattering transform. We thank the reviewer for this pointer. The connection holds on the target side: the magnitude of an STFT band followed by a temporal low-pass has the same structure as a first-order scattering coefficient, which is a band magnitude followed by temporal averaging [1]. On the latent side, the operation is a plain low-pass filter applied to the latent sequence, with no wavelet and no modulus, so the scattering formalism would not make it simpler. We will mention the connection and cite [1], and we will first describe both operations as low-pass filters along time at cutoff fc and then give the DFT masks as their implementation.
Meaning of "coarse". The two coarsenings are defined in Sections 3.3 and 3.4, and we agree that a short definition at first use would help. The waveform is never coarsened, and a coarsened spectrogram has no empty regions and no removed frequency bands. A temporally coarsened spectrogram is one in which the magnitude trajectory of every frequency band is low-pass filtered along time at fc, so that only fluctuations faster than fc are removed (Figure 2, left, top row). A spectrally coarsened spectrogram is one in which the log-magnitude spectrum of every frame is replaced by its cepstrally smoothed envelope, so that only fine structure across frequency, such as individual harmonics, is removed (Figure 2, right, top row). We will add these two definitions where the term first appears.
"Leading coefficients". The text refers to the leading coefficients in the plural, that is, the first mk cepstral coefficients starting from the zeroth, and not to the zeroth coefficient alone. To avoid this reading, we will write "the first mk (low-order) cepstral coefficients".
"Low-pass filtering the waveform changes acoustic bandwidth rather than temporal modulation". As the reviewer notes, the sentence is correct for slow modulations, and these are the only ones that TSE acts on: its cutoffs do not exceed the highest modulation frequency of the latent, which is 10.77 Hz for the waveform VAE and 21.47 Hz for the spectrogram VAE (Appendix C). We will make this range explicit by rewriting the sentence as follows: "A low-pass filter applied to the waveform removes high acoustic frequencies but leaves the slow modulations of the remaining bands, below about 20 Hz, unchanged. Since TSE targets these slow modulations, both coarsenings are defined on the spectrogram."
[1] Deep Scattering Spectrum. IEEE Transactions on Signal Processing, 2014.