acceptodds
Withdrawn from ICLR 2027

PoDAR: Power-Decoupled Audio Representation for Generative Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The performance of audio latent diffusion models is primarily governed by generator expressivity and the modelability of the underlying latent space. While recent research has focused primarily on the former, as well as improving the reconstruction fidelity of audio codecs, we demonstrate that latent modelability can be significantly improved through explicit factor decoupling. We present PoDAR (Power-Decoupled Audio Representation), a framework that utilizes a randomized power augmentation and latent consistency objective to decouple signal power from invariant semantic content. This factorization makes the latent space easier to model, which both accelerates the convergence of downstream generative models and improves final overall performance. When applied to a Stable Audio 1.0 VAE with an F5-TTS generator, PoDAR achieves about a 2x acceleration in convergence to match baseline performance, while increasing final speaker similarity by 0.055 and UTMOS by 0.22 on the LibriSpeech-PC dataset. Furthermore, isolating power into dedicated channels enables the application of CFG exclusively to power-invariant content, effectively extending the stable guidance regime to higher scales.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
representation learning, latent generative modeling, latent modelability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_G5CvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes PoDAR, a training-time regularizer for audio autoencoders that aims to make their latent spaces easier for downstream diffusion and flow models to learn. During autoencoder training, each waveform gets a random global gain in [−6, +6] dB. An L2 consistency loss on the encoder means then penalizes differences between the original and augmented latents in all channels except the first kk (k=1k=1 throughout), so that gain-induced variation is routed into a dedicated "power" channel and the remaining channels stay invariant to loudness. At inference, the authors propose "partial CFG," in which the power channel uses the unguided conditional prediction and standard classifier-free guidance is applied only to the content channels. The method is applied to two Stable Audio VAEs re-trained from scratch, VAE1 (DAC-based) and VAE2 (Oobleck-based). An F5-TTS generator is trained on each latent space using Emilia ZH-EN and evaluated on LibriSpeech-PC test-clean, Seed-TTS test-en and Seed-TTS test-zh. The paper reports that PoDAR roughly halves the training steps needed to reach baseline speaker similarity for VAE1, improves final speaker similarity in all configurations (about +0.05 for VAE1 and +0.01–0.04 for VAE2) and raises UTMOS by 0.22 for VAE1 on LibriSpeech-PC, with smaller or mixed UTMOS changes elsewhere and essentially unchanged WER. VAE1 reconstruction quality drops (ViSQOL 4.01 to 3.70). Partial CFG mainly makes quality degrade more slowly at high guidance scales. Supporting analyses include a swap test showing about 1 dB of residual gain leakage, ablations over λPoDAR\lambda_{\text{PoDAR}} and kk, a linear probe indicating that decoupling is only partial, and a latent-perturbation robustness study.

Critical strengths

Simple method with consistent gains

PoDAR needs only a gain augmentation and an L2 term, with no pretrained teacher, so it avoids the encoder-selection problem of REPA-style alignment. Speaker similarity improves in all six codec–test set configurations, and VAE1 reaches baseline SIM in roughly half the training steps (Fig. 1).

Focus on modelability over reconstruction

The paper evaluates codecs by downstream generation rather than reconstruction alone, and shows the two can diverge: VAE1-PoDAR has worse ViSQOL (4.01 → 3.70) but yields a better generator.

Critical weaknesses

Paper writing

Upon finish reading the introduction, there is still no clue what "Partial CFG" is. The current writing only states what is does in the contribution list. The intro paragraph (lines 102–107) describes the idea without naming it: guidance "can be applied selectively to the semantic subspace, while leaving the power channels governed strictly by the learned generative dynamics." The term "partial CFG" never appears there. Contribution 3 then uses the name as if it had already been introduced: "Partial CFG allows steering exclusively over the semantic subspace." A reader has to guess that this refers to the previous paragraph.


Motivation

The motivation is also asserted rather than shown. The intro claims standard CFG "may inadvertently amplify nuisance variables, such as signal power, and distort the signal." No evidence is given anywhere, such as loudness or clipping measurements of baseline outputs at high ww. Fig. 2 shows that quality metrics drop at high ww, but not that loudness amplification is the cause.


Gains are not shown to arise from power decoupling

The consistency loss also substantially regularizes the latent space (VAE1 latent scale drops from 1.482 to 0.395, Table 7), and the paper has no controls to separate this from decoupling. The evidence on mechanism is mixed. In the λ\lambda ablation, going from 0.1 to 0.5 both reduces swap gain (2.21 to 0.99 dB) and improves generation. But going from 0.5 to 0.75 improves SIM and UTMOS with no change in swap gain (0.99 vs. 0.97 dB), which suggests that something other than measured decoupling contributes, unless the swap test has a floor near 1 dB. Decoupling is also partial (linear probe R2≈0.67R^2 \approx 0.67, Table 8), and with no baseline probe or swap test, the paper doesn't show how much PoDAR improved on an unconstrained codec. Controls such as a higher-KL VAE1, augmentation without the consistency loss, or a consistency loss with no reserved channel are needed to support the central causal claim.


Lacking reference

The related work also misses the most closely related line: equivariance and invariance regularization of autoencoders to improve latent diffusability in the image domain. Examples are EQ-VAE (Kouzelis et al., 2025) and Skorokhodov et al. (2025), "Improving the Diffusability of Autoencoders." PoDAR's augmentation-consistency idea needs to be positioned against these works, and its novelty claim adjusted accordingly.

  • EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modeling, Kouzelis et al., ICML 2025
  • Improving the Diffusability of Autoencoders, Skorokhodov et al., ICML 2025
Questions for authors

If you edit channel 0 alone, does the output loudness change smoothly and predictably while the content stays intact?

Reviewer_rKJhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This presents PoDAR, a framework that can softly decouple signal power from invariant semantic content by randomized power augmentation and a latent consistency loss. Empirical results show that this decoupling makes model training two times faster and improves generation quality. In addition, separating power into a dedicated latent channel enables CFG to be applied only to the power-invariant content channels, while leaving the power channels unguided.

Critical strengths
  1. The method is simple and well-motivated. Power is a nuisance factor for many speech generation tasks and can be perturbed by a perfectly time-aligned linear transformation, making it a suitable target for consistency-based decoupling.
  2. The empirical validation is reasonably strong for the claimed speech generation setting. The method is tested on two VAE backbones and multiple benchmarks, and shows improvements not only in final quality metrics but also in convergence speed and CFG robustness.
Critical weaknesses
  1. Although the idea is reasonable and the paper provides strong results, decoupling/disentangling specific features in representations is a very common idea. For example, "Swapping Autoencoder for Deep Image Manipulation" (NeurIPS 2020) proposes an architecture that enables image manipulation by swapping parts of latent representations, and "Robust Disentangled Variational Speech Representation Learning for Zero-shot Voice Conversion" (ICASSP 2022) uses disentangled speech representations for zero-shot voice conversion. The submitted paper positions PoDAR as a factor-decoupling approach, but the related work discussion does not sufficiently distinguish it from prior disentanglement and factorized representation learning methods, including approaches based on latent swapping or augmentation consistency. I do not necessarily expect exhaustive comparisons against all prior disentanglement methods, but the paper should better situate PoDAR relative to them.
  2. Although the Limitations section explicitly states that the efficacy of the framework has been demonstrated exclusively within the speech domain, the paper should provide more comprehensive experiments and analyses, especially given that decoupling features itself is a common idea. The current evidence is promising, but I do not think it is sufficient to fully support the paper at the expected ICLR standard.
    • [2-a] Reconstruction performance should be evaluated using standard audio-codec metrics such as STFT distance, mel-spectrogram distance, or SI-SDR, in addition to ViSQOL. I understand that reconstruction fidelity is not the primary objective of this work, but reporting these metrics would still be valuable for readers, for reproducibility, and for understanding the trade-off between reconstruction quality and downstream modelability.
    • [2-b] I understand that training a TTA or TTM model is expensive, but assessing reconstruction performance on other types of audio, such as music, should be less costly, since the VAEs are trained on a mixed dataset of speech, music, and environmental audio. This would help clarify whether PoDAR improves audio latent representations more broadly or mainly benefits speech generation.
    • [2-c] The work should provide qualitative audio samples, not only quantitative metrics. For audio generation papers, listening to generated and reconstructed samples is important because metrics such as ViSQOL and UTMOS do not fully capture perceptual artifacts, naturalness, intelligibility, or failure cases. The absence of audio samples makes it difficult for reviewers and readers to independently judge the perceptual quality of the proposed method, and I consider this a significant limitation for evaluating an audio generation paper.
Questions for authors
  1. Related to Weakness 2, it would be appreciated if the authors could show speech reconstruction performance using commonly used metrics, such as STFT distance, MEL distance, or SI-SDR, in addition to ViSQOL. In addition, visualizing mel-spectrograms would help readers grasp how well the model can reconstruct audio signals.
  2. It would be appreciated if the authors could provide PoDAR's reconstruction performance of music signals in addition to speech.
  3. Section 6.2 and Figure 3 show that λPoDAR=0.75\lambda_\text{PoDAR} = 0.75 leads to better UTMOS values than λPoDAR=0.50\lambda_\text{PoDAR} = 0.50 in the generation task, but Section 5.1 and Table 1 show that λPoDAR=0.50\lambda_\text{PoDAR} = 0.50 is better at reconstruction measured by ViSQOL. I understand that generation and reconstruction performances are not always correlated, but to support the claim, it would be good to include the UTMOS of the reconstructed audio in Table 1.
  4. The current manuscript focuses on waveform autoencoders, whereas some audio generation systems use mel-spectrogram autoencoders combined with neural vocoders. Do the authors expect the same power/content decoupling behavior and downstream training benefits to hold for mel-spectrogram VAEs? If not, what aspects of the proposed method are specific to waveform autoencoders? This is more of a future-work-oriented clarification rather than a request for additional experiments during rebuttal.

Minor comment

Section 5.1 (Line 325) says, "Empirical evaluation demonstrates that the required λPoDAR\lambda_\text{PoDAR} varies by architecture." However, Appendix D provides evidence that the KL penalty affects the latent manifold. I recommend changing the statement to "Empirical evaluation demonstrates that the required λPoDAR\lambda_\text{PoDAR} varies by architecture and training objective" and referring readers to Appendix D.

Reviewer_QMysOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PoDAR, a training method for continuous audio VAEs to decouple signal power from content. During VAE training, the input audio is scaled by a random global gain in [−6,+6][-6, +6] dB, and an L2L_2 loss is applied to L−1L-1 latent channels to make them invariant to volume changes, leaving k=1k=1 channel for power. At inference, CFG is applied only to the L−1L-1 content channels (Partial CFG). Experiments on Stable Audio 1.0/2.0 VAEs and F5-TTS show faster generator convergence and improved speaker similarity/UTMOS.

Critical strengths
  1. A practical and effective training trick for audio VAEs: The proposed gain-invariance loss and Partial CFG are very simple to implement, do not require external pretrained encoders (unlike REPA or WavLM distillation), and clearly speed up F5-TTS convergence while improving high-CFG stability on the Stable Audio 1.0 VAE (Figures 1 and 2).
Critical weaknesses
  1. Limited technical contribution (Good engineering trick, but thin for an ICLR paper): The method simply scales input volume by a scalar gg and penalizes L2L_2 distance on L−1L-1 channels. As noted in Section 3.1, this only works for global gain because other attributes (pitch, speaker, rhythm) break temporal alignment. While it is a nice trick for VAE training, the methodological novelty and depth feel insufficient to support a full ICLR paper.
  2. Gains seem entangled with simple latent regularization (Table 2 & Table 7): In Table 7, adding PoDAR to VAE1 (which has a tiny KL weight of 10−610^{-6}) shrinks the latent scale (std(μ)\text{std}(\mu)) from 1.482 to 0.395, acting as a strong regularizer. On VAE2 (which already uses a larger KL weight of 10−410^{-4}), the UTMOS gains on Seed-TTS are negligible (3.87 to 3.89 on EN, and dropping from 2.98 to 2.96 on ZH in Table 2). It is unclear how much of the gain comes from power decoupling versus simply regularizing the latent scale.
  3. Side effects and incomplete decoupling (Table 1 & Table 8): Training requires two encoder forward passes, drops VAE1 reconstruction ViSQOL from 4.01 to 3.70 (Table 1), and still leaves power and content highly correlated (Linear probe Pearson r=0.83r = 0.83 in Table 8). Also, despite claiming a general "Audio Representation" and training the codec on music/environmental audio, downstream evaluation is only done on TTS.
Questions for authors

Clarification Requests:

  1. Simple explicit gain baseline: Since the method only targets global volume scaling, why not simply normalize the audio energy (or extract frame-level RMS energy as an explicit channel) before the encoder, rather than using a double-forward consistency loss that hurts ViSQOL (4.01 to 3.70) and still leaves r=0.83r=0.83 correlation in the content channels?
  2. Decoupling vs. KL regularization (Table 2 & Table 7): Since PoDAR significantly shrinks the latent scale of VAE1 (Table 7) and shows much smaller gains on the already-regularized VAE2 (Table 2), what happens if you simply increase the KL weight of VAE1 Baseline to 10−410^{-4} without PoDAR?
  3. Beyond TTS: Since the VAEs are trained on speech, music, and AudioSet, does this trick actually help on Text-to-Music or Text-to-Audio generation, or is it only tested on F5-TTS?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.