acceptodds
Under review as a conference paper at ICLR 2027

Latent Spectral Equivariance

Abstract

Scalable audio generation systems typically cascade a pre-trained variational autoencoder (VAE) for audio compression and a latent diffusion model for distribution modeling. However, existing VAEs prioritize reconstruction fidelity without explicitly considering how readily the resulting latent distribution can be modeled by diffusion, potentially limiting generation quality. Recent work highlights the importance of aligning latent representations with the structure of the underlying data for diffusion modeling. Building on this direction, we introduce Latent Spectral Equivariance (LatentEQ), the first framework that systematically introduce equivariance into audio VAE optimization by pairing latent transformations with corresponding spectrogram transformations. Specifically, temporal scale equivariance applies matched low-pass filtering along time, aligning slowly varying latent components with slowly varying acoustic structure. Spectral granularity equivariance pairs latent channel prefixes with frequency-smoothed spectrograms, organizing channels to progressively capture spectral envelopes and fine harmonic detail. We instantiate LatentEQ on the VAEs of two well-known audio generation systems, Stable Audio Open and MMAudio, through a fine-tuning process without changing their architecture or introducing additional objectives. Comprehensive results demonstrate improved generation quality across waveform- and spectrogram-based VAEs and both text-to-audio and video-to-audio tasks, highlighting LatentEQ as an effective principle to learn structured audio latents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio VAE, Audio Generation, Equivariance, Signal Processing, Latent Diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oaj1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors proposes a method for exploiting latent spectral equivariance for fine tuning VAEs in audio generation, as a mean to learn structured audio latents. More precisely, the authors propose temporal scale equivariance and spectral granularity equivariance. The principle is applied on the VAE's of two well known audio generation systems, namely Stable Audio Open and MMAudio.

Critical strengths

The main idea of the paper is interesting. In fact, aiming at building structured latents by exploiting equivariance principles, and in studying the merits of the proposed methods in audio generation (both in text-to-audio and Video-to-audio) are clear strengths of the paper.

Critical weaknesses

Nevertheless, the paper suffers from several wekanesses.

  • Presentation and readability. The paper is written in excellent Engish but the paper is still difficult to follow. Despite a rather simple mathematical framework, the mathematical notations and design choices for the presentation unecessarily complexify the reading. For instance, in equation (1) x can either be a time-domain waveform or a spectrogram... then all successive operations on x becomes more difficult to understand because each operation is different depending of the nature of x. The description of the Latent temporal transform using DFT may probably be described in simpler terms using the formalism of the scattering transform. The vocabulary chosen may also be additionaly confusing. The notion of coarseness in audio can be misleading and it is difficult for me to understand exaclty what is a coarse audio sequence (when talking of the wafevorm).. even a coarse spectrogram is not too clear what it is exactly (a spectrogram with empty spaces, with empty spaces only on the upper part ... in section 3.3, why using "leading coefficient" for mentioning the zeroth order cepstral coefficient (or alternatively first ceptral coefficient) which are much more common used terms ? Link to this, they are a number of confusing sentences such as "..low-pass filtering the waveform changes acoustic bandwidth rather than temporal modulation..(it is mostly correct but only if you asume that temporal modulations are only "low-frequency temporal modulations" ) .. or such as "For a spectrogram VAE, the input is already a log-mel spectrogram" (we are then confused between the spectrogram, the log-spectrogrm, the log-mel spectrogram). The figure 2 is not very clear. It is quite small and needs to be viewed on screen with a strong zoom to view everything. Then, the variation of the "mean log magnitude per frame" (for the temporal scale equivariance) is not intuitive (is it wrong ?) since it exibits (at some points) low values for the frames where the spectrogram has high amplitude values.

  • Experiments and evaluation: a large number of experiments are conducted. The results provided using objective metrics show the potential interest of the method proposed. Nevertheless, it is known that such objective metrics are far from being well-correlated to perception. To better assess the merit of the methods, it would have been necessary to at least provide a comprehensive demo on a companion web site or a perceptual test. Besides, it is not clear what is the advantage brought by the method by the simple fact of doing fine-tuning (or increasing learning) using augmented data (low pass filtered data with different cut-off frequencies). A comparison with a system finetuned (without equivariance) but with the same additional augmented data would have been interesting. Finally, it is not clear what is the added complexity of the fine-tuning strategy and if it is indeed marginal compared to the original training complexity.

  • Reproducibility: there are many implementation details given but this usually does not suffice for reproducibility. Distributing the code in would guarantee a higher level of reproducibility .

  • bibliography, Related work, supplementary material: The related work section is placed in the supplementary material which I think is not appropriate. The related work section is an important part of the paper since it allows the authors to well position their work in the State of the Art... and to better show what is new. This is not only additional details or evidence to suport the main text (which is what the supplementary material should be about). Besides, if numerous references are given I believe that important references in the use of equivariance principles in audio or music processing are missing (see for instance work in equivariance in self-supervised learning [1-3]) [1]: Sechaud & al. "Equivariance-based self-supervised learning for audio signal recovery from clipped measurements, Eusipco 2024 [2] Lostanlen & al. "Understanding Equivariant Self-Supervised Learning in Musical Pitch Class Space", IEEE Signal Processing Letters, 2025 [3] Riou &l . PESTO: Real‑Time Pitch Estimation with Self‑Supervised Transposition‑Equivariant Objective, Trans. of ISMIR, 2025

The formatting of the reference list can also be improved mostly those of IEEE where the year, the name of the conference, the IEEE acronyms are duplicated in the references.

Questions for authors
  • How can the readibility of the paper be improved ?

  • what is the complexity burden of the proposed approach ?

  • what is the perceptual quality of the generated audio (perceptual test or demo page) ?

  • How reproducbility can be improved ?

  • what is gain of the proposed approach compared to only performing fine-tuning with the additional augmented data ?

6 comments
AuthorsOpenReview ↗

Response to Reviewer oaj1 (Part 1/4)

We thank the reviewer for the detailed review, and for considering the main idea, building structured latents through equivariance and studying its merits in text-to-audio and video-to-audio generation, a clear strength of the paper. We address each question and concern below.

Main Comments:


Q1 & W1: Presentation and readability

We thank the reviewer for the detailed reading. We answer each point below and state the corresponding change to the paper where one is needed.

Two forms of xx in Eq. 1. We would like to clarify that the operations of LatentEQ do not depend on the form of xx. Every operation acts on the latent zz and on the target spectrogram S(x)\mathcal{S}(x), which have the same form for both VAE types. The form of xx only determines S\mathcal{S}, which computes STFT magnitude spectrograms at several resolutions for a waveform and is the identity for a log-mel input. To make this explicit, we will state it at the beginning of Section 3 and summarize the differences of the spectrogram VAE in one paragraph: S\mathcal{S} is the identity, there is a single resolution, and Eq. 11 needs no logarithm or exponential.

Relation to the scattering transform. We thank the reviewer for this pointer. The connection holds on the target side: the magnitude of an STFT band followed by a temporal low-pass has the same structure as a first-order scattering coefficient, which is a band magnitude followed by temporal averaging [1]. On the latent side, the operation is a plain low-pass filter applied to the latent sequence, with no wavelet and no modulus, so the scattering formalism would not make it simpler. We will mention the connection and cite [1], and we will first describe both operations as low-pass filters along time at cutoff fcf_c and then give the DFT masks as their implementation.

Meaning of "coarse". The two coarsenings are defined in Sections 3.3 and 3.4, and we agree that a short definition at first use would help. The waveform is never coarsened, and a coarsened spectrogram has no empty regions and no removed frequency bands. A temporally coarsened spectrogram is one in which the magnitude trajectory of every frequency band is low-pass filtered along time at fcf_c, so that only fluctuations faster than fcf_c are removed (Figure 2, left, top row). A spectrally coarsened spectrogram is one in which the log-magnitude spectrum of every frame is replaced by its cepstrally smoothed envelope, so that only fine structure across frequency, such as individual harmonics, is removed (Figure 2, right, top row). We will add these two definitions where the term first appears.

"Leading coefficients". The text refers to the leading coefficients in the plural, that is, the first mkm_k cepstral coefficients starting from the zeroth, and not to the zeroth coefficient alone. To avoid this reading, we will write "the first mkm_k (low-order) cepstral coefficients".

"Low-pass filtering the waveform changes acoustic bandwidth rather than temporal modulation". As the reviewer notes, the sentence is correct for slow modulations, and these are the only ones that TSE acts on: its cutoffs do not exceed the highest modulation frequency of the latent, which is 10.77 Hz for the waveform VAE and 21.47 Hz for the spectrogram VAE (Appendix C). We will make this range explicit by rewriting the sentence as follows: "A low-pass filter applied to the waveform removes high acoustic frequencies but leaves the slow modulations of the remaining bands, below about 20 Hz, unchanged. Since TSE targets these slow modulations, both coarsenings are defined on the spectrogram."

[1] Deep Scattering Spectrum. IEEE Transactions on Signal Processing, 2014.

AuthorsOpenReview ↗

Response to Reviewer oaj1 (Part 2/4)

Q1 & W1: Presentation and readability (continued)

Spectrogram, log-spectrogram, and log-mel spectrogram. Section 2 defines S(x)\mathcal{S}(x) as the STFT magnitude spectrograms at several resolutions for the waveform VAE and as the log-mel input itself for the spectrogram VAE. The quoted sentence means that the target of the spectrogram VAE is already logarithmic, so the DCT mask of Eq. 11 is applied to it directly. We will restate this definition next to that sentence and use "log-mel spectrogram" consistently for the spectrogram VAE.

Figure 2. The figure is compact because of the page limit of the main text, and all of its details remain visible when zooming in the PDF. For convenience, both panels are also available at full size on the demo page (https://latenteq.github.io/, section "Figure 2, Enlarged"), and we are willing to include enlarged versions in the appendix of the revised paper. The curve is not wrong. It follows the low-frequency band at the bottom of each panel, which is the highest-level region of this clip, and its minima coincide with the dips of that band, as can be checked in the enlarged figure. The mid-frequency stripes look prominent on the logarithmic colour scale, but their level is well below that of the low-frequency band, and they do not coincide in time with its peaks. This is why the curve can be low at frames where these stripes appear. We will explain this in the caption.

We are willing to make all of the changes above in the revised paper.


Q2 & W2-c: Complexity of the fine-tuning strategy

We thank the reviewer for this question. The added cost of LatentEQ is confined to VAE fine-tuning, and it is small there.

Per-step cost. LatentEQ adds no parameters, no network, and no loss term, and each example is still encoded once and decoded once (Algorithm 1). The only added operations are a DFT mask on the latent and DFT and DCT masks on the target spectrograms that the reconstruction loss already computes. We measured the training throughput of both VAEs on 8 A100 GPUs with a batch size of 64:

BackboneFT VAE (it/s)LatentEQ, TSE + SGE (it/s)Throughput drop
Stable Audio Open (waveform VAE)1.0140.9823.2%
MMAudio (spectrogram VAE)1.7811.7183.5%

LatentEQ lowers the throughput by less than 4% on both VAEs.

Relative to the original training. For the waveform VAE, fine-tuning for 500k steps at this throughput takes about 141 hours on 8 A100 GPUs, which is about 1.1k GPU-hours. Stable Audio Open reports 183 hours followed by 273 hours on 32 A100 GPUs for its VAE, which is about 14.6k GPU-hours. The fine-tuning therefore amounts to about 8% of the original VAE training.

Diffusion training and inference. The VAE architecture and the latent dimensions are unchanged, and no transform is applied during diffusion training or sampling (Section 3.5), so the inference cost is identical to that of the original systems. As with any change to the VAE, the diffusion model has to be trained on the new latents. It keeps its original architecture and training objective, and Figure 4 shows that the Stable Audio Open DiT converges faster on LatentEQ latents.

Compared with other latent regularization methods. The cost is in fact an advantage of LatentEQ. Semantic alignment methods such as SAME, KVAE, and MiniMax H3 need an external model during VAE training. STAR-VAE trains a new VAE architecture from scratch, for about 150 hours followed by about 70 hours on 24 H800 GPUs as reported, and it further introduces a new generation framework. LatentEQ adds no module and only fine-tunes an existing VAE, and it requires no additional design in the generation stage.

We are willing to add this analysis to Appendix C of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer oaj1 (Part 3/4)

Q3 & W2-a: Perceptual quality (listening test and demo page)

We thank the reviewer for this point. We agree that objective metrics alone do not establish perceptual quality, and we now provide both a demo page and listening tests.

Demo page. An anonymous demo page is available at https://latenteq.github.io/. It covers the four settings evaluated in the paper: text-to-audio on AudioCaps (Table 1), video-to-audio and video-and-text-to-audio on VGGSound (Table 2), and VAE reconstruction on AudioCaps and the Song Describer Dataset (Table 3). Each example places the ground truth, baselines from the corresponding table, and our outputs side by side, together with their mel spectrograms.

Listening tests. 15 listeners rated the generated audio on a five-point scale, with all stimuli anonymized and presented in random order. For T2A, they rated the overall quality (OVL) and the relevance to the text (REL). For V2A, they rated the audio quality (MOS-Q) and the temporal synchronization with the video (MOS-T). For VT2A, they additionally rated the semantic alignment with the video and the text (MOS-A). Reconstruction quality is evaluated with a MUSHRA test, with a hidden reference and a 3.5 kHz low-pass anchor, on ten clips, five from AudioCaps (Sound) and five from the Song Describer Dataset (Music). All tables report the mean score with its 95% confidence interval.

T2A on AudioCaps (8 prompts)

MethodOVL↑REL↑
Ground truth4.34 ± 0.154.53 ± 0.15
AudioLDM3.38 ± 0.203.38 ± 0.21
Tango2.83 ± 0.222.98 ± 0.20
MMAudio3.68 ± 0.203.23 ± 0.21
Ours (spectrogram)3.96 ± 0.163.78 ± 0.18
TangoFlux3.86 ± 0.163.82 ± 0.19
AudioX4.07 ± 0.193.83 ± 0.20
Stable Audio Open4.20 ± 0.153.55 ± 0.21
Ours (waveform)4.21 ± 0.163.93 ± 0.18

V2A on VGGSound (6 videos)

MethodMOS-Q↑MOS-T↑
Ground truth4.06 ± 0.194.32 ± 0.18
V-AURA3.29 ± 0.143.53 ± 0.15
FRIEREN3.21 ± 0.263.48 ± 0.18
AudioX3.43 ± 0.153.16 ± 0.23
MMAudio3.49 ± 0.103.68 ± 0.18
Ours3.76 ± 0.173.77 ± 0.14

VT2A on VGGSound-Omni (6 videos)

MethodMOS-Q↑MOS-A↑MOS-T↑
Ground truth4.42 ± 0.114.41 ± 0.154.41 ± 0.16
ThinkSound (w/o CoT)3.43 ± 0.123.39 ± 0.243.71 ± 0.18
HunyuanVideo-Foley3.61 ± 0.133.52 ± 0.153.90 ± 0.15
AudioX3.50 ± 0.123.24 ± 0.192.99 ± 0.21
MMAudio3.64 ± 0.113.58 ± 0.163.93 ± 0.14
Ours3.67 ± 0.213.46 ± 0.204.10 ± 0.10

VAE reconstruction (MUSHRA, 0–100)

MethodMUSHRA↑
Hidden reference96.99 ± 1.18
3.5 kHz low-pass anchor47.12 ± 2.51
AudioLDM67.01 ± 2.53
Make-An-Audio 271.82 ± 2.80
MMAudio76.19 ± 2.57
Ours (spectrogram)77.09 ± 2.15
HunyuanVideo Foley70.27 ± 2.33
ϵ\epsilonar-VAE72.37 ± 2.34
Stable Audio Open74.71 ± 2.74
Ours (waveform)81.21 ± 1.72

Across all four listening tests, the LatentEQ models are rated the best among the evaluated systems, or above or on par with the systems they build on.

We are willing to add the listening tests to the revised paper and the link to the demo page to its abstract.


Q4 & W3: Reproducibility

We thank the reviewer for this point. We have reported the training details as completely as we could in the paper. Algorithm 1 gives the training procedure. Appendix A describes the datasets and their preprocessing, Appendix B lists the evaluation toolkits, and Appendix C specifies the cutoff schedule of TSE, the twelve pairs (k,mk)(k, m_k) of SGE (Table 5), the construction of the targets for the reconstruction and adversarial terms, and the training and inference settings of the four models (Table 6). The method is implemented on top of the public code of Stable Audio Open, MMAudio, and Make-An-Audio 2 without architectural change, all training datasets are public, and all metrics are computed with public toolkits. The two paired transforms are the only additions to these code bases, so we are willing to include their reference implementation in the appendix of the revised paper, and we will also seek to release our full code and checkpoints.

Reviewer_HxXmOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Latent Spectral Equivariance (LatentEQ), a method for structuring audio VAE representations to improve downstream latent diffusion modeling. It pairs temporal low-pass filtering of latent sequences with temporal smoothing of target spectrograms, and latent channel truncation with spectral smoothing. The method fine-tunes pretrained VAEs using transformed reconstruction and adversarial targets without changing the architecture. Experiments on waveform and spectrogram VAEs report improvements in reconstruction and text- and video-conditioned audio generation.

Critical strengths
  1. It addresses the distinction between preserving information for reconstruction and organizing that information for downstream generative modeling. Connecting latent transformations to interpretable acoustic transformations is a reasonable approach.
  2. The fine-tuning controls in Figure 3 are particularly valuable. Relative to fine-tuning without the proposed constraints, generation (FD_V) improves from 4.04 to 3.01 for MMAudio and from 3.01 to 1.99 for Stable Audio Open. These comparisons provide stronger evidence for the method than comparisons against official checkpoints alone.
  3. Evaluation across two representation types. Experiments with both waveform and spectrogram VAEs, together with component ablations and training-dynamics analysis, support the practical usefulness of the approach beyond a single implementation.
Critical weaknesses
  1. The general principle of making latent transformations correspond to acoustic transformations predates this work. In particular, Learning Linearity in Audio Consistency Autoencoders via Implicit Regularization (Torres et al., 2026) encourages gain equivariance and additivity in audio autoencoders. Homogeneity is itself a form of equivariance. I appreciate that Appendix D.3 discusses this work and that the introduction restricts its priority claim to audio VAEs. Nevertheless, the contribution should be framed primarily around the proposed temporal and spectral coarsening correspondences and their benefit for downstream generation. The distinction between consistency autoencoders and VAEs alone does not establish a new conceptual principle.
  2. The component ablations establish the usefulness of the proposed training procedures, but each procedure changes both the latent input and the reconstruction target. Consequently, they do not distinguish the benefit of acoustic matching from that of latent masking or truncation alone. A matched control that applies the same latent transformations while retaining the original reconstruction target would directly test this distinction. This is important because the correspondence between the two transformations is central to the paper’s contribution.
  3. Section 3.3 would be clearer if it introduced the desired correspondence before describing the DFT masks. It should explicitly distinguish acoustic frequency from temporal-modulation frequency and explain that filtering preserves clip duration and latent frame rate. Using DFT temporally and DCT spectrally is not inherently problematic; a short explanation of their respective purposes and boundary assumptions would improve readability.
Questions for authors
  1. Do you have evidence comparing matched target coarsening against the same latent masking/truncation with an unchanged reconstruction target? A limited comparison on one backbone would help establish whether acoustic matching provides a distinct benefit.
  2. How should Equation 5 be interpreted in light of the independently constructed linear-magnitude, log-magnitude, and multiresolution targets in Appendix C? Is it an idealized motivation, or is there a more precise correspondence satisfied by the implemented objective?
6 comments
AuthorsOpenReview ↗

Response to Reviewer HxXm (Part 1/4)

We thank the reviewer for the careful and constructive assessment, and for recognizing the distinction between preserving information and organizing it for generation, the fine-tuning controls in Figure 3, and the evaluation across two representation types. We address each question and concern below.

Main Comments:


Q1 & W2: Matched target coarsening versus the same latent transformation with an unchanged target

We thank the reviewer for this question. We agree that the component ablations in Figure 3 change the latent input and the reconstruction target together, so they do not separate the benefit of acoustic matching from that of latent masking or truncation alone. We would like to mention that our earliest experiments were designed exactly in this way, with the latent transformed and the original target kept. Under this design, we found that the reconstruction quality remained almost unchanged, but the performance of the downstream generation task was not improved. Since the correspondence between the two transformations is central to our contribution, we report this control below.

Control. On the Stable Audio Open backbone, we fine-tune two VAEs that apply the same latent transformations as TSE and SGE but keep the original reconstruction and adversarial targets. We then train the same DiT from scratch on the latents of each. With the original target, the decoder is asked to restore detail that the coarsened latent no longer carries, so the constraint acts on the latent alone, whereas the matched target gives each coarsened latent a defined acoustic counterpart. The other rows are those of Figure 3 (left), and reconstruction and generation are evaluated on the AudioCaps test split.

MethodReconstructionGeneration
STFT-D↓MEL-D↓SI-SDR↑FD$_V$↓FD$_V$↓KL$_P$↓IS$_P$↑FD$_P$↓CLAP↑
SAO1.230.850.083.603.472.169.9227.870.303
FT VAE1.220.870.123.483.011.859.9622.170.401
w/ TSE (orig. target)1.210.840.103.653.051.9410.0123.560.375
w/ TSE1.140.780.803.292.431.4410.6518.760.434
w/ SGE (orig. target)1.250.830.123.553.191.919.7923.930.380
w/ SGE1.220.760.743.212.351.3710.2719.880.473
Ours1.010.730.942.421.991.3511.7317.050.495

With the original target, both controls leave reconstruction almost unchanged and do not improve generation over the FT VAE. With the matched target, the same latent transformations improve both reconstruction and generation, lowering the generation FD$_V$ from 3.01 to 2.43 and 2.35. The gain therefore comes from the correspondence between the latent transformation and the target, and not from the latent transformation alone.

We will add this control to Figure 3 in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer HxXm (Part 2/4)

Q2: Interpretation of Eq. 5 given the targets in Appendix C

We thank the reviewer for this precise question. Eq. 5 is an idealized property. It acts as an equivariance constraint during training, and the trained model does not satisfy it strictly. What the implemented objective minimizes is the residual of the surrogate relation of Eq. 6, separately for each loss term. In some terms the target is given by a single well-defined TxT_x, and in others only approximately.

From Eq. 5 to Eq. 6. Eq. 5 is written for a single spectrogram operator S\mathcal{S} and a single TxT_x. Training replaces Tx(S(D(z)))T_x(\mathcal{S}(\mathcal{D}(z))) with the fixed target Tx(S(x))T_x(\mathcal{S}(x)) and retains the identity case, so it minimizes both the residual of Eq. 6 and the reconstruction error of the full latent. When TxT_x is an orthogonal projection, it is non-expansive in the ℓ2\ell_2 norm, and the deviation from Eq. 5 in this norm is bounded by these two quantities:

∥S(D(z~))−TxS(D(z))∥≤∥S(D(z~))−TxS(x)∥+∥S(x)−S(D(z))∥.\big\lVert \mathcal{S}(\mathcal{D}(\tilde{z})) - T_x \mathcal{S}(\mathcal{D}(z)) \big\rVert \le \big\lVert \mathcal{S}(\mathcal{D}(\tilde{z})) - T_x \mathcal{S}(x) \big\rVert + \big\lVert \mathcal{S}(x) - \mathcal{S}(\mathcal{D}(z)) \big\rVert.

Training reduces both quantities but does not bring them to zero, so Eq. 5 holds up to this bound and not exactly.

Terms with a single well-defined TxT_x. For the spectrogram VAE, S\mathcal{S} is the identity on the log-mel input at a single resolution, and both target transforms are orthogonal projections that act on different axes, the DFT mask along time and the DCT mask along frequency. They commute, so TxT_x is one linear projection. The same holds for the log-magnitude term of the waveform VAE at each STFT resolution. In these terms the objective implements Eq. 6 with a single well-defined TxT_x, and the bound above applies.

Terms where TxT_x is only approximately defined. The waveform VAE has three sources of approximation. (i) The linear-magnitude term has its own target. The temporal low-pass is applied to the magnitude, while SGE smooths the log-magnitude and exponentiates back, so this target is not the exponential of the log-magnitude target. A constant shift per frequency bin keeps it non-negative and changes only the temporal DC component. SGE and TSE do not commute in this term, so their order is fixed, with SGE applied first. (ii) The targets of the seven STFT resolutions are constructed independently at the same physical scales, namely the same modulation cutoff in Hz and the same spectral scale in Hz. In general, no single waveform has exactly these magnitudes at all resolutions, and the decoder output is the waveform that fits them jointly. (iii) The real input of the discriminator combines the coarsened magnitude with the original phase. For these terms, the objective minimizes a family of per-term surrogate relations that share one cutoff and one granularity, and no single TxT_x describes all of them.

Empirical check. Figure 5 measures how closely the relation holds after training, using MEL-D, which is not a term of the waveform training loss. The distance between the decoded coarsened latent and the matched target is not zero, but it stays close to or below the reconstruction error of the full latent at every cutoff and truncation depth.

Revision. We will state in Section 3.2 that Eq. 5 is an idealized property that the trained model satisfies only approximately and that Eq. 6 is imposed term by term, and we will move the per-term definitions above from Appendix C into Section 3.5.

AuthorsOpenReview ↗

Response to Reviewer HxXm (Part 3/4)

W1: Framing of the contribution relative to prior equivariance work (Torres et al., 2026)

We thank the reviewer for this comment, and we agree with it. The general principle of making latent transformations correspond to acoustic transformations predates our work, and CAE [1] already encourages gain equivariance and additivity in an audio autoencoder. We also agree that the difference between consistency autoencoders and VAEs does not by itself establish a new principle. We will no longer present the contribution as the first introduction of equivariance into audio VAEs.

Reframed contribution. The contribution is the two coarsening correspondences and their effect on downstream generation. First, TSE pairs a temporal low-pass of the latent with a low-pass of each spectrogram band at the same modulation cutoff, and SGE pairs a prefix of the latent channels with a cepstrally smoothed spectrogram at a matched granularity. Second, latents trained with these correspondences are easier for a diffusion model to learn, which we show by training the diffusion models of two systems from scratch and comparing them with plain fine-tuning controls (Figure 3 and Figure 4). The control reported in Q1 & W2 further shows that this gain comes from the correspondence itself, since the same latent transformations with the original target do not improve generation.

Relation to CAE. The two works differ in their setting and in the transformations they constrain. CAE [1] is demonstrated on the Music2Latent [2] architecture, a consistency autoencoder, that is, a diffusion autoencoder whose decoder is a consistency model. Its authors state that the approach is in theory model-agnostic, but it is applied only to this architecture. The Stable Audio Open VAE appears there only as a baseline with its public weights, and no generation model is trained on the resulting latent. As for the transformations, CAE makes the decoder linear, so that scaling and mixing latents correspond to scaling and mixing audio. These operations do not order the latent by scale. LatentEQ uses coarsenings, which progressively remove detail and are therefore ordered by temporal scale and spectral granularity. They specify which components of the latent, its low modulation frequencies and its first channels, carry each acoustic scale, and their effect is evaluated on the diffusion model trained on the latent. The two constraints are compatible.

Revision. We will revise the abstract, the Introduction, and the contribution list accordingly. In the abstract, the description of LatentEQ as "the first framework" to introduce equivariance into audio VAE optimization becomes "a framework that structures audio VAE latents by pairing temporal and spectral coarsenings of the latent with matched coarsenings of the target spectrogram". The sentence of the Introduction that reads "to our knowledge is the first framework to introduce equivariance into the training of audio VAEs" is changed in the same way. The first contribution becomes "We show that the latents of pretrained audio VAEs do not follow the temporal scales and spectral granularities of the sound, since low-pass filtering or truncating the latent does not coarsen the decoded audio accordingly". The discussion of CAE will move from Appendix D.3 into the Introduction.

[1] Learning Linearity in Audio Consistency Autoencoders via Implicit Regularization. ICASSP 2026.

[2] Music2Latent: Consistency Autoencoders for Latent Audio Compression. ISMIR 2024.

Reviewer_sqkNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I have mixed feeling regarding this paper.

The paper is inspired by the EQ-VAE paper for image generation, and tried to apply masking to the audio VAE dimensions or conducting low-pass filtering to the audio VAE features in its latent Fourier domain, to improve the "Diffusability" (compatibility with a latent diffusion model) of the audio VAE and eventually improve the generation quality.

On the one hand, the results in the table are cohereny. On the other hand, I saw several major concerns:

  1. The lack of audio samples (can come from a demo page or the supplementary file)
  2. Evaluation weakness. 2.1 The lack of DeSync metric to measure the temporal alignment in V2A and VT2A tasks 2.2 While the authors claim that latent regularization can also improve the audio quality of audio VAEs, the improvement in objective metrics is in a narrow range, so a MUSHRA test is usually required to support the claim.
  3. most important issue: contradicting description about the technical contribution
    • Line 50 says "continuous audio VAEs are still trained for reconstruction alone"
    • Line 72 - 75: the authors introduced mulple recent papers that paid attention to the regularization of the audio VAE latent space.
    • The actual contribution of this paper: on top of previous VAE regularization methods, this paper tries a new method that do not require additional loss
      • However, there is no comparison with other regularization methods... So the paper's contribution seems to be very incremental to me: We see a promising method, it works better than naive VAE training, but we don't know if it is worth to replace other other regularization methods mentioned in the introduction

Given the above 3 points, I recommend a "weak reject" to this paper.

Critical strengths

Strictly showed that LatentEQ works paper than plain finetuning of audio VAE under 2 scenarios.

Tables show that the improvement in generation quality may not necessarily be aligned with the reconstruction quality.

The evaluation is extensive and comprehensive.

Critical weaknesses

I have mentioned details in the Summary session, so I just list up items here:

  • Lack of audio samples
  • Weaknesses in evaluation metrics
  • Incoherent presentation regarding the positioning of this paper
Questions for authors

Is it possible to share audio samples?

5 comments
AuthorsOpenReview ↗

Response to Reviewer sqkN (Part 1/3)

We thank the reviewer for the constructive comments and for recognizing the controlled comparison between LatentEQ and plain fine-tuning on both VAEs, as well as the extensive evaluation. We address each question and concern below.

Main Comments:


Q1 & W1: Audio samples

We thank the reviewer for this suggestion and agree that audio samples are needed to judge the results. We have prepared an anonymous demo page at https://latenteq.github.io/. It covers the four settings evaluated in the paper: text-to-audio on AudioCaps (Table 1), video-to-audio and video-and-text-to-audio on VGGSound (Table 2), and VAE reconstruction on AudioCaps and the Song Describer Dataset (Table 3). Each example places the ground truth, baselines from the corresponding table, and our outputs side by side, together with their mel spectrograms. We will add this link to the abstract of the revised paper.


W2-a: DeSync for V2A and VT2A generation

We thank the reviewer for pointing out this missing metric. We now report DeSync, which follows MMAudio and measures the audio-visual misalignment in seconds predicted by Synchformer (lower is better), for both the V2A and VT2A settings of Table 2. The other columns are those of the paper.

V2A on the VGGSound test set

MethodFD$_V$↓KL$_P$↓IS$_P$↑FD$_P$↓IB↑DeSync↓
V-AURA2.882.4210.0814.800.280.65
ReWaS1.792.878.5117.540.151.06
FRIEREN1.342.7312.2511.450.230.85
FoleyCrafter2.512.3015.6816.240.261.23
AudioX1.553.0314.2614.980.251.21
MMAudio0.852.0914.196.560.290.48
Ours0.752.0011.445.680.300.46

VT2A on VGGSound-Omni

MethodFD$_V$↓KL$_P$↓IS$_P$↑FD$_P$↓IB↑DeSync↓
ThinkSound (w/o CoT)1.101.6011.737.410.260.53
HunyuanVideo-Foley2.361.7411.5810.020.320.57
AudioX1.181.5915.088.300.261.23
MMAudio0.811.6313.165.820.300.49
Ours0.731.6013.955.890.310.48

In both settings, our model obtains the lowest DeSync among the evaluated systems (0.46 for V2A and 0.48 for VT2A), slightly below MMAudio (0.48 and 0.49), whose diffusion architecture it keeps. Replacing the VAE with its LatentEQ counterpart therefore preserves audio-visual synchronization and slightly improves it. We will add this column to Table 2 in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer sqkN (Part 2/3)

W2-b: Subjective evaluation (MUSHRA) of reconstruction quality

We thank the reviewer for this suggestion. We agree that the differences in the objective metrics of Table 3 are small, and that a listening test is needed to support a claim about reconstruction quality. We therefore ran a MUSHRA test on our two VAEs and six of the VAE baselines of Table 3.

Protocol. Each page presents the reference and, in random order, a hidden reference, a 3.5 kHz low-pass anchor, and the eight reconstructions of one 10-second clip, and listeners rate each stimulus on a scale from 0 to 100. The test uses ten clips, five from AudioCaps (Sound) and five from the Song Describer Dataset (Music), and 15 listeners took part. The table reports the mean score with its 95% confidence interval.

MethodMUSHRA↑
Hidden reference96.99 ± 1.18
3.5 kHz low-pass anchor47.12 ± 2.51
AudioLDM67.01 ± 2.53
Make-An-Audio 271.82 ± 2.80
MMAudio76.19 ± 2.57
Ours (spectrogram)77.09 ± 2.15
HunyuanVideo Foley70.27 ± 2.33
ϵ\epsilonar-VAE72.37 ± 2.34
Stable Audio Open74.71 ± 2.74
Ours (waveform)81.21 ± 1.72

The hidden reference and the anchor receive the highest and the lowest scores, as expected. LatentEQ obtains the best subjective score both among the spectrogram VAEs and among the waveform VAEs, and each LatentEQ VAE is rated above the VAE it is fine-tuned from. This agrees with the improvement of both LatentEQ VAEs over their pretrained counterparts in Table 3.

We will add these results to the revised paper.


W3-a: Positioning of the contribution (Lines 50 and 72–75)

We thank the reviewer for pointing out this inconsistency, and we agree that the sentence at Line 50 is too broad when read next to Lines 72–75. The two passages refer to different groups of work. Line 50 describes the VAEs of the established audio LDM systems, such as AudioLDM, Stable Audio Open, and MMAudio, whose objectives contain only spectral, adversarial, and KL terms. Lines 72–75 describe recent methods that have begun to regularize the audio latent. We do not claim to be the first to regularize audio VAE latents. What we propose is a structural regularization, which differs in kind from these methods.

Positioning. Our contribution is the equivariance constraint itself: each coarsening of the latent is paired with a matched coarsening of the reconstruction target, along time (TSE) and across channels (SGE). The three groups of methods at Lines 72–75 do not impose such a correspondence. Alignment with pretrained encoder features constrains what the latent encodes, temporal-spectrum and channel masking act on the latent alone, and prior-geometry methods reshape the KL term. Requiring no additional loss term is only one aspect of this contribution, as detailed in W3-b. The Introduction states this distinction in a single clause at Lines 72–75, and the details are currently given only in Appendix D.3.

Revision. In Line 50, we will replace "continuous audio VAEs" with "the VAEs of widely used audio LDMs", so that the sentence no longer covers the recent regularization methods of Lines 72–75. We will also move the distinction above from Appendix D.3 into the Introduction.

AuthorsOpenReview ↗

Response to Reviewer sqkN (Part 3/3)

W3-b: Contribution and comparison with other latent regularization methods

We thank the reviewer for summarizing our contribution. We would like to clarify that requiring no additional loss is only one aspect of it, and we respectfully disagree that the contribution is incremental.

Our contribution. First, LatentEQ makes the latent space behave like the data space: each coarsening of the latent, along time or across channels, is paired with a matched coarsening of the target, so that the latent transforms as the sound does. None of the cited methods imposes such a relation, and the two correspondences are designed for audio, whose latent has a time axis but no frequency axis. Second, it improves downstream generation without modifying the downstream model. The diffusion model keeps its original architecture and training objective, and the latent keeps its shape, so an existing system needs no change other than training its diffusion model on the new latents. Third, it is obtained by fine-tuning a pretrained VAE, with no architectural change, no additional loss, and no external pretrained model. Fourth, these properties are verified on a waveform VAE and a spectrogram VAE, on text-to-audio and video-to-audio generation, and against plain fine-tuning controls (Figure 3), with faster convergence of the diffusion model (Figure 4) and better reconstruction (Table 3).

Essential difference from the cited methods. LatentEQ differs in kind from every cited method. These methods regularize the latent itself and leave the reconstruction target unchanged, so none of them specifies how the latent should be organized with respect to the temporal and spectral structure of the sound. SAME, KVAE, and MiniMax H3 perform semantic alignment, which is not a structural regularization: it determines what the latent encodes by pulling it toward the representation of an external pretrained model, and it places no constraint on how the latent is organized along its time and channel axes. LatentFT and LAFA are both built on a diffusion autoencoder, whose decoder is a diffusion or flow model, which changes the reconstruction paradigm, and they mask the latent while keeping the original target, so their constraint acts on the latent alone. STAR-VAE takes a different route: it reshapes the prior, ordering the latent channels by capacity through a KL weight that depends on the channel index, and it further introduces a new generation framework, STAR-Gen, to exploit this latent. LatentEQ is instead a structural regularization. It constrains the relation between the latent and the signal: a transformation applied to the latent, along time or across channels, must produce a prescribed transformation of the decoded sound. This gives each temporal scale and each channel prefix of the latent an explicit acoustic meaning (Figure 5), and it requires no external model, no additional loss, no new architecture, and no change to the downstream generation model.

Comparison with STAR-VAE. Among the cited methods, STAR-VAE is the one that shares our setting, with the same latent configuration (44.1 kHz, 21.5 Hz) and the same VAE training data as our waveform branch, which makes a direct comparison possible. The other methods differ from ours in sample rate, latent rate, architecture, or training data, and they are evaluated with their own diffusion models. For reconstruction, Table 3 already compares the two VAEs, and LatentEQ outperforms STAR-VAE on all three signal-level reconstruction metrics (STFT-D, MEL-D, and SI-SDR) on both test sets. For generation, STAR-VAE reports the Stable Audio Open DiT with its VAE on AudioCaps, using the metrics of our Table 1:

SystemFD$_O$↓KL$_S$↓CLAP↑
Stable Audio Open99.702.200.303
SAO DiT with STAR-VAE (reported by STAR-VAE)72.52.150.35
SAO DiT with LatentEQ VAE (Ours)60.261.330.495

The DiT trained on LatentEQ latents obtains better values than the one with STAR-VAE on all three metrics.

Whether LatentEQ should replace them. LatentEQ does not compete with these methods for the same role. It is a fine-tuning step that changes only the decoder input and the reconstruction target, so it is complementary to these methods and not a substitute for them. We will add the comparison above to the revised paper.


We thank the reviewer again for the constructive feedback. We hope the audio samples, the DeSync and listening-test results, the clarified positioning, and the comparison with other regularization methods address the concerns, and we would be happy to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.