acceptodds
Under review as a conference paper at ICLR 2027

AuDiAE: Learning Audio Latents by Coupling Diffusion Priors and Decoders

Abstract

Audio autoencoders determine both the information available for waveform reconstruction and the latent target learned by a downstream generator. Standard Gaussian posterior-KL regularization limits latent information but also penalizes mismatch between the aggregate posterior and a fixed Gaussian. Under a tight KL budget, this additional distribution-matching cost leaves less capacity for reconstruction-relevant detail. Downstream conditional generation additionally depends on how much latent variation is predictable from a condition such as text, while high compression leaves recording-specific acoustics underdetermined for deterministic waveform decoders. We introduce AuDiAE, an Audio Diffusion AutoEncoder that couples a learned diffusion rate model with a generative waveform decoder through a common encoder representation. The marginal diffusion prior replaces the fixed reference with a learned density while maintaining a variational information bound. Its conditional counterpart makes condition-predictable information less costly in the training objective while the marginal rate continues to control total latent information. A flow-matching decoder trained to predict the clean waveform models residual variation and values encoded source detail through its reduction in optimal denoising risk. We evaluate AuDiAE at 2048× and its high-compression variant, AuDiAE-H, at 4096× on reconstruction and matched AudioCaps generation. AuDiAE leads every reported reconstruction metric and improves the strongest external AudioCaps FAD by 35% while attaining the highest CLAP. Matched controls support the learned rate reference and correct text–audio pairing; frozen diagnostics isolate tighter prior fit and text use without supporting a causal decoder advantage.

Primary area:
generative models
Keywords:
Audio Autoencoder, Diffusion Model, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5ksYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an audio autoencoder that combines a learned diffusion prior with a flow-matching waveform decoder to address the limitations of fixed Gaussian priors and deterministic decoding under high compression. The learned prior reduces the cost of aggregate posterior mismatch, while a text-conditional rate objective encourages the latent representation to retain information predictable from text. The flow-matching decoder models residual acoustic variation beyond the encoded latent information. Experiments at compression rates of 2048× and 4096× show that AuDiAE achieves the best results across all reported reconstruction metrics and improves AudioCaps generation, reducing FAD by 35% and achieving the highest CLAP score.

Critical strengths
  1. The paper is well-motivated. The authors combine source-faithful audio reconstruction with the learnability of latent representations for downstream conditional generation. The decomposition of the rate into mutual information and aggregate-prior mismatch provides a clear motivation for learning the rate reference.
  2. The paper distinguishes transmitted information from prior mismatch and variational slack, and explicitly acknowledges that the conditional rate is a regularizer rather than an operational bitrate or a guarantee of improved conditional generation.
  3. The proposed method achieves strong reconstruction metrics and the best reported AudioCaps FAD and CLAP scores.
Critical weaknesses
  1. The paper combines a learned diffusion prior, a text-conditional rate objective, and a flow-matching decoder. While the overall design is reasonable, it is not entirely clear what the main methodological advances and differences are from prior work, particularly existing approaches that jointly train diffusion priors and decoders.
  2. The generative decoder appears to be a key component of the proposed method, but the experiments do not clearly demonstrate its specific contribution. Since the main comparisons involve complete systems, it is difficult to determine whether the improvements come from the decoder itself or from other components. A controlled comparison with a deterministic decoder, while keeping the encoder and latent representation fixed, would provide more convincing evidence.
  3. Limited evaluation of downstream generation. The downstream generation experiments are conducted only on AudioCaps, which mainly contains sound effects. It remains unclear whether the learned representations also benefit conditional generation in other audio domains, such as music. Evaluating on additional datasets would help establish the generality of the proposed approach.
Questions for authors
  1. Can the authors provide a controlled comparison between deterministic and flow-matching decoders while keeping the encoder, latent representation, rate constraint, and training budget fixed?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 5ksY (1/2)

We sincerely thank the reviewer for the positive feedback and for recognizing that AuDiAE is "well-motivated", that our rate decomposition "provides a clear motivation for learning the rate reference", and that AuDiAE achieves "strong reconstruction metrics and the best reported AudioCaps FAD and CLAP scores." We hope our point-by-point response can fully resolve all your concerns.

  1. What are the main methodological advances over prior work, particularly approaches that jointly train diffusion priors and decoders?

Thank you for this important positioning question. AuDiAE asks how to learn compact audio latents that a downstream text-conditioned generator can model readily and that still support source-faithful waveform synthesis (Sec. 1). Joint training of a diffusion prior and a decoder [1], on which AuDiAE builds (Sec. 5), learns latents for images and video under a marginal prior, whose rate limits how much information the latent carries but not which. The main methodological advance of AuDiAE therefore lies in the rate: it decides which information the latent keeps for a text-conditioned generator, and it is accounted for as information, mismatch, and slack rather than as a bitrate. Three advances follow, (1)–(2) over joint diffusion training and (3) over audio autoencoders:

(1) The rate decides which information the latent keeps, whereas a marginal prior bounds only how much (Sec. 3.2). A text-conditioned generator must model only the latent variation that the caption leaves unexplained, I(X;Zb∣C)=I(X;Zb)−I(Zb;C)I(X;Z_b|C)=I(X;Z_b)-I(Z_b;C) (Eq. 2). Under a marginal rate, however, two latents with the same marginal distribution and the same total information receive the same rate even when only one is predictable from text (App. B.3), and [1] itself observes that text alignment degrades slightly at its lowest bitrates (Sec. 5.3 therein). AuDiAE therefore feeds the same prior either the caption or a null condition and mixes the two bounds with weight ρ\rho (Eq. 10): information predictable from text, I(Zb;C)I(Z_b;C), is charged only a fraction 1−ρ1-\rho, while ρ<1\rho<1 keeps total information bounded (Eq. 11). Text enters only this rate, never the encoder or decoder. Table 3(b) tests this against joint training with a marginal prior, i.e., the same model with the marginal rate alone: paired captions improve FAD/CLAP from 0.093/0.305 to 0.075/0.328, whereas shuffled captions, which keep the conditional branch and ρ\rho, stay at 0.095/0.303; the gain thus comes from text–audio pairing.

(2) The rate is separated into information, mismatch, and slack, whereas joint training reads its bound as the latent bitrate (Sec. 3.1). In [1], the bound is "a tight upper bound on the latent bitrate" (abstract therein). In AuDiAE, by contrast, it is the information the latent transmits plus the mismatch of its aggregate distribution to the prior plus variational slack (Eq. 8), and for a fixed encoder, learning the prior lowers only the last two. This separation determines what each component can claim: a smaller bound after learning the prior reflects a better fit, not less information, and since the text discount of (1) also contains mismatch and slack, the conditional rate is a regularizer, not a bitrate or a measure of text-predictable information (Sec. 3.2), as the reviewer notes among the strengths. AuDiAE therefore tests the learned prior with the encoder frozen, where information is fixed: its bound is tighter than that of the best fitted Gaussian (3.104 vs. 3.264 nats per element; Table 6).

(3) In audio, the rate is learned and the decoder generates what the latent leaves open, whereas audio VAEs measure the latent against a fixed Gaussian and decode a point estimate (Sec. 3.1, 3.3). Under a tight rate budget, matching a fixed Gaussian consumes rate that could carry audio detail, and at high compression (2048× and 4096× here), one latent is compatible with many waveforms, which spectral–adversarial decoders collapse into a single estimate. AuDiAE trains the diffusion rate jointly with a flow-matching waveform decoder on the same noised encoder output ZbZ_b (Eqs. 3, 12), and weighs the denoising risk the latent removes against its rate (Eqs. 14–15). In audio VAEs, the KL weight trades reconstruction against downstream generation (Sec. 1). Replacing only the Gaussian KL by the learned rate, with the encoder, channel, decoder, data, and β\beta matched, instead improves both: SI-SDR by 1.07 dB, rFAD by 17.0%, and FAD/CLAP from 0.142/0.274 to 0.093/0.305 (Table 3(a)); the complete system leads every reconstruction metric of Table 1 and AudioCaps generation (Table 2). AuDiAE is therefore an alternative to the Gaussian-KL paradigm of audio VAEs, not a refinement within it.

We have added this discussion to App. A.4 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 5ksY (2/2)

  1. The generative decoder appears to be a key component; can it be compared with a deterministic decoder while keeping the encoder, latent representation, rate constraint, and training budget fixed?

Thank you for this constructive suggestion. The aim of the paper is to study how a diffusion prior and a diffusion decoder, trained jointly around one latent, shape its learnability and its reconstruction quality; replacing the decoder with a deterministic one would change this paradigm itself rather than ablate a component of it. In AuDiAE the decoder is one side of the coupling: its denoising risk determines what encoded information is worth to the encoder (Eqs. 14–15), so a different decoder yields a different latent. A deterministic decoder would in addition require adversarial training to shape the reconstruction, as in spectral–adversarial autoencoders (Sec. 1), which AuDiAE's objective does not contain (Eq. 16). The controlled analysis therefore trains a flow decoder of the same design jointly in every row and varies only the rate objective (Table 3), and deterministic decoding is compared at the system level: εar-VAE, Stable Audio Open, and SAME-L are single-pass deterministic autoencoders, and AuDiAE improves on all three in every reconstruction metric of Table 1.

The requested comparison can, however, be made within one trained model, and we have added it to App. F of the revised manuscript (Table 12). The same latents are decoded either deterministically, as the decoder's point estimate of the waveform (the conditional mean that a regression decoder targets), or generatively, by sampling; the encoder, latent representation, rate constraint, and training budget are therefore identical. On AudioCaps, the generative decoder halves the MEL distance of the deterministic one (0.068→0.034) and lowers rFAD from 0.200 to 0.062, while SI-SDR, which favors the conditional mean, decreases from 12.49 to 10.08 dB (Table 12). The specific contribution of the generative decoder is therefore perceptual and distributional fidelity: it models the residual variation that a deterministic point estimate averages out.

References

[1] Heek et al., "Unified Latents (UL): How to Train Your Latents." arXiv:2602.17270, 2026.

AuthorsOpenReview ↗

Follow-Up on Our Responses

Dear Reviewer 5ksY,

As the public discussion period ends on November 18, we would be grateful to know whether our responses address your questions. Thank you again for your constructive suggestions, which have helped us improve the paper.

Best regards,

The Authors

Reviewer_96K3OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose to train audio latent with diffusion losses.

The model consists encoder, waveform diffusion decoder, diffusion rate model.

And the model is trained with waveform related loss, B_a, J_u, and J_c, where B_a is boundary term, and J_u and J_c are unconditional and conditional diffusion loss, respectively.

The proposed model achieves promosing results compared with previous baselines.

Critical strengths

In current version, compare to both of previous model and control version, the proposed model achieves promising results.

Critical weaknesses
  1. VAE trained with latent diffusion model is already existing idea. Both of [1] and [2] already trained the variational auto-encoder with latent diffusion. Technically, while the loss term is not very different with them, I am concerning about the novelty of the work. The only distinguishing feature I can identify is the incorporation of conditional diffusion. However without careful ablation studies, it is difficult to determine whetere this is a meaningful contribution.

  2. Boundary term looks like KL div with zero-mean Gaussian. While the manuscript mentioned that "Because ais finite, the forward law generally differs from the standard Gaussian used to initialize the reverse process.", the boundary term looks like KL divergence with zero-mean Gaussian.

  3. The controlled analyses do not appear to ensure a fair comparison between the proposed model and the baselines. Table 1,2,3,4 are the results of reconstruction, generation, but all using different dataset. So it is hard to compare control fixed-Gaussian rate model's quality compare to the baselines.

  4. Missing details of the model. The manuscript did not mention any number of parameters and detailed architecture of the encoder. Although the authors did not conduct a fair comparison with the baselines and controlled models, as noted in Comment 3, the proposed model's superior performance may be attributable to its substantially larger number of parameters.

In addition, receptive field of encoder sometime affects the model's output quality, it should be great to mention the details including receptive field.

  1. Neural Audio Codecs (NAC) are mentioned but not compared with the proposed model. Since NACs, such as DAC [3] or MOSS-audio-tokenizer [4], play critical role in audio language models, a comparison with them would be valuable. However, such a comparison is missing from the evaluation.

  2. Control models are not on the demo page It should be great to compare them on the demo page.

[1] Vahdat, Arash, Karsten Kreis, and Jan Kautz. "Score-based generative modeling in latent space." Advances in neural information processing systems 34 (2021): 11287-11302. [2] Wehenkel, Antoine, and Gilles Louppe. "Diffusion priors in variational autoencoders." arXiv preprint arXiv:2106.15671 (2021). [3] Kumar, Rithesh, et al. "High-fidelity audio compression with improved rvqgan." Advances in neural information processing systems 36 (2023): 27980-27993. [4] Gong, Yitian, et al. "Moss-audio-tokenizer: Scaling audio tokenizers for future audio foundation models." arXiv preprint arXiv:2602.10934 (2026).

Questions for authors

My two main concerns are that

  1. VAE trained with latent diffusion model is already exists.
  2. Control analyses do not appear to ensure a fair comparison.
  3. Lack of details

Therefore, it should be great to mention

  1. the technical difference with existing papers.
  2. Use the identical dataset for baseline and control analyses.
  3. Add details about the model, especially for the encoder.

Could you please revise the manuscript to address these concerns and provide additional information above?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 96K3 (1/3)

We sincerely appreciate the reviewer's recognition that AuDiAE "achieves promising results" compared with both previous models and the controlled variants. Following your comments, we have added the fixed-Gaussian control and the DAC and MOSS-Audio-Tokenizer codecs under the protocol of Table 1 (App. F, Tables 10–11), and the parameter counts and RTF of every system (Table 1).

  1. VAEs trained with latent diffusion priors already exist [1,2] and the loss is similar; beyond conditional diffusion, what is technically different, and is the conditional component carefully ablated?

We would like to clarify that in AuDiAE, the diffusion prior serves as a rate model, not as a latent generator: it determines what encoded information costs, while a waveform diffusion decoder determines what it is worth. In prior work [1,2], the diffusion prior is the latent generator of an image VAE with a likelihood decoder. This amounts to three technical differences, of which conditional diffusion is only one:

(1) In AuDiAE, the diffusion prior is the rate reference of the bottleneck, whereas in prior work it is the latent generator (Sec. 3.1). The two roles address different terms of the KL to a prior pp, Rp=I(X;Z)+KL(qϕ∥p)R_p=I(X;Z)+\text{KL}(q_\phi\|p) (Eq. 1): the information the latent transmits and the mismatch of its aggregate distribution to pp. In prior work, the prior addresses the mismatch: it is fitted to the latent distribution so that latents can be sampled from it. Limiting information is not its purpose: the latent of [1] is larger than the image (46,080 vs. 3,072 dimensions on CIFAR-10; Table 7 therein). In AuDiAE, by contrast, the prior addresses the information: the KL against it is the cost the encoder pays for what it transmits, and thus limits the information. Audio autoencoders use the KL in this way, but against a fixed Gaussian; the mismatch is then charged too and, under a tight budget, consumes rate that could carry audio detail. AuDiAE therefore learns the reference: a diffusion model trained jointly on ZbZ_b, the encoder output after a fixed noise channel (Eq. 3). Its loss still upper-bounds the information, I(X;Zb)≤RˉuI(X;Z_b)\le\bar R_u (Eq. 8), while the excess shrinks to what the model fails to fit. Hence our prior acts through the latent it shapes in training: all reported generation uses a separate generator that does not use it (Sec. 3.4; App. C.3). Table 3(a) isolates this choice: with the encoder, channel, decoder, data, and β\beta fixed, replacing the Gaussian reference by the learned one improves both reconstruction (SI-SDR +1.07 dB, rFAD −17.0%) and downstream generation (FAD/CLAP 0.142/0.274→0.093/0.305).

(2) The conditional component is a mixed rate that allocates information by text predictability, not merely a conditional prior (Sec. 3.2). A text-conditioned generator must model only the latent variation that the caption leaves unexplained, I(X;Zb∣C)=I(X;Zb)−I(Zb;C)I(X;Z_b|C)=I(X;Z_b)-I(Z_b;C) (Eq. 2), but the rate of (1) charges predictable and unpredictable information alike. AuDiAE therefore mixes it with the rate under the caption-conditioned prior, (1−ρ)Rˉu+ρRˉc=I(X;Zb∣C)+(1−ρ)I(Zb;C)+Eρ(1-\rho)\bar R_u+\rho\bar R_c=I(X;Z_b|C)+(1-\rho)I(Z_b;C)+\mathcal E_\rho (Eq. 10): text-predictable information is charged only a fraction 1−ρ1-\rho, while ρ<1\rho<1 keeps total information bounded (Eq. 11). Text enters only this rate, never the encoder or decoder, so shuffling the captions removes this discount (I(Zb;C)=0I(Z_b;C)=0) while keeping the conditional branch, text encoder, ρ\rho, and audio batches. In this ablation (Table 3(b)), shuffled captions stay at the unconditional level (FAD/CLAP 0.095/0.303 vs. 0.093/0.305), whereas paired captions reach 0.075/0.328: the gain comes from text–audio pairing, not from the added branch.

(3) The decoder determines what encoded information is worth: the latent is valued by the Bayes denoising risk it removes (Sec. 3.3). At high compression (2048× and 4096×), the latent does not determine the waveform: given ZbZ_b, a distribution of waveforms remains. AuDiAE therefore formulates decoding as conditional waveform generation: a flow-matching decoder conditioned on the same ZbZ_b models this residual variation instead of collapsing it to a point estimate (Eq. 12; Sec. 1). The decoder is trained with a weighted denoising risk (Eq. 13), and the latent's value is the part of this risk that it removes for an optimal decoder, Vw(Zb)≥0V_w(Z_b)\ge0 (Eq. 14). By the I–MMSE identity, this risk sums information increments over noise levels, each weighted by ww (App. B.4), so the value depends on which information the latent carries, not only on how much. The Bayes-optimal variational objective sets this value against the rate of (1)–(2) (Eq. 15; Sec. 3.4): information is encoded when it is worth its cost. Table 12 (App. F) confirms this: decoding the same latent to its point estimate instead of sampling raises rFAD from 0.062 to 0.200.

We have added [1,2] and this discussion to App. A.3 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 96K3 (2/3)

  1. The boundary term BaB_a looks like a KL divergence to a zero-mean Gaussian.

We would like to clarify that the latent of AuDiAE is regularized by the learned diffusion prior, not by a zero-mean Gaussian: in the rate bound Rˉu=Ba+Ju\bar R_u=B_a+J_u (Eq. 8), the learned path cost JuJ_u accounts for about 99.99% and BaB_a for 0.01%. The role of BaB_a is to account for "the mismatch at the finite-noise initialization" (App. B.2), not to limit the latent: a diffusion model starts its reverse process from N(0,I)\mathcal N(0,I), and at finite aa the noised latent "generally differs from the standard Gaussian used to initialize the reverse process" (Sec. 3.1, the sentence the reviewer quotes), so the bound must charge this gap, as every diffusion likelihood bound does [5] (LTL_T in DDPM). The gap is tiny (3×10−43\times10^{-4} nats per element in training) because BaB_a is evaluated after the latent has been noised to a=−10a=-10, where its signal weight is αa2=sigmoid(−10)=4.5×10−5\alpha_a^2=\mathrm{sigmoid}(-10)=4.5\times10^{-5} and any latent distribution is almost N(0,I)\mathcal N(0,I) (Eq. 7); it "vanishes as a→−∞a\to-\infty for fixed ν\nu, but cannot in general be discarded at a finite endpoint" (App. B.2). In contrast, a KL bottleneck applies the same expression to the latent itself (b=5b=5, signal weight 0.993; App. C.1) with no learned path, where it limits the latent; BaB_a applies it only at the noise end, where it keeps the bound valid without constraining the latent.

  1. Controls and baselines are evaluated on different datasets, so the fixed-Gaussian control cannot be compared with the baselines; please use identical data.

The fixed-Gaussian control is now compared with the baselines on identical data in both tasks: in reconstruction, we have evaluated it under the protocol of Table 1 and report it next to the baselines in App. F (Table 10) of the revised manuscript, and in generation, Tables 2 and 3 already share the AudioCaps test set and protocol (captions of both tables; App. C.3). Under the protocol of Table 1, the control obtains a sound/music SI-SDR of 9.60/13.51 dB, above every baseline (best: 8.49/13.28 dB), and an rFAD of 0.068, lower than every baseline except εar-VAE (0.054); its matched counterpart with the learned rate obtains 10.17/14.24 dB and 0.059, and AuDiAE 10.66/14.61 dB and 0.041 (Table 10). The control was not listed in Table 1 because Table 3(a) is a matched ablation of the rate reference, not a system comparison (App. C.1). In generation, the control (FAD/CLAP 0.142/0.274) is the best in neither column: its FAD lies between CoDiCodec (0.116) and Stable Audio Open (0.160), and its CLAP is below all three baselines (0.284–0.318). Replacing its Gaussian reference by the learned rate alone gives a better FAD than every baseline (0.093), and the mixed rate leads both columns (0.075/0.328).

AuthorsOpenReview ↗

Response to Reviewer 96K3 (3/3)

  1. Parameter counts and encoder details are missing; the gains may come from more parameters. Please also report the receptive field.

Thank you for requesting these details. The gains are not attributable to model size: the fixed-Gaussian control has the same network as the paired-caption model of Table 3(b), yet it trails the smaller CoDiCodec in downstream FAD (0.142 vs. 0.116), and changing only the rate objective brings it to 0.075 (Tables 2–3). Across systems, downstream quality does not follow autoencoder size either: every representation is modeled by the same generator under the same budget (App. C.3), and among the baselines of Table 2 the largest model, SAME-L (852.1M), has the worst FAD/CLAP (0.205/0.284), while the smallest, CoDiCodec (148.5M), has the best FAD (0.116). We have added the inference parameters of every system to Table 1 of the revised manuscript: AuDiAE has 259.7M and AuDiAE-H 335.3M, 70% and 61% fewer than SAME-L, yet AuDiAE is better on every metric of Table 1 and AuDiAE-H has the better downstream FAD/CLAP at the same 4096× compression (0.104/0.316 vs. 0.205/0.284); the rate prior (at most 15M) and the frozen text encoder are used only in training. The encoder, rate prior, and decoder architectures are specified in App. C.1 ("Primary architecture"), where we have also added the encoder's receptive field: each latent depends on 0.88 s of audio.

  1. Neural audio codecs such as DAC [3] and MOSS-Audio-Tokenizer [4] are mentioned but not compared.

We would like to clarify that DAC [3] and MOSS-Audio-Tokenizer [4] are discrete RVQ codecs for autoregressive audio language models, whereas AuDiAE is a continuous autoencoder for diffusion and flow generators, so they share neither a common rate axis nor our downstream protocol. App. A.1 already draws this distinction, with DAC (Improved RVQGAN) as its example: discrete codecs "quantize audio into RVQ token streams with an operational bitrate", whereas AuDiAE's variational rate "should not be interpreted as the bitrate of an entropy-coded file"; our downstream protocol, in turn, fits one continuous flow-matching generator to every representation (App. C.3), which RVQ tokens cannot enter. Reconstruction can, however, be compared at each system's own operating point under the protocol of Table 1, and we have added this comparison to App. F (Table 11) of the revised manuscript, citing [4] in App. A.1 as well. DAC (44.1 kHz, 8 kbps) obtains a sound/music SI-SDR of 6.19/10.41 dB and an rFAD of 0.051, and MOSS-Audio-Tokenizer (24 kHz, 4 kbps) 1.17/6.42 dB and 0.074, against 10.66/14.61 dB and 0.041 for AuDiAE, which is better on every objective metric (Table 11).

  1. Controlled variants are not on the demo page.

Thank you for the suggestion. We have updated the demo page to include the fixed-Gaussian control and the learned diffusion rate of Table 3(a) alongside AuDiAE and the baselines. Their effect on the complete test sets is reported in Tables 3 and 10.

References

[1] Vahdat et al., "Score-based Generative Modeling in Latent Space." NeurIPS, 2021.

[2] Wehenkel and Louppe, "Diffusion Priors in Variational Autoencoders." arXiv:2106.15671, 2021.

[3] Kumar et al., "High-Fidelity Audio Compression with Improved RVQGAN." NeurIPS, 2023.

[4] Gong et al., "MOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models." arXiv:2602.10934, 2026.

[5] Kingma et al., "Variational Diffusion Models." NeurIPS, 2021.

Reviewer_LD7tOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an audio codec design that leverages a learned diffusion prior in its continuous latent space, replacing fixed traditional priors, such as Gaussian commonly used in VAEs. Authors extend this marginal diffusion prior to the text-condition setting in order to facilitate learnability for conditional generation. To decode the waveform, a flow matching decoder is trained jointly with the prior. Experimental results demonstrate improved Pareto-optimal performance in the reconstruction vs conditional generation trade-off compared to other codecs.

Critical strengths
  • The paper is overall well written, however the wording is somewhat heavy, at parts hurting readability.
  • The motivation is clear and valid: shaping the latent space of audio AEs for the opposing objectives of effective downstream modeling and high-fidelity waveform reconstruction is an active field of research with clear gains and benefits for audio generation.
  • The proposed method is sensible and fits into the existing VAE framework.
  • The numerical results are promising, however more information is needed to clearly see the merits (detailed under Weaknesses).
Critical weaknesses
  • Even though the experimental results are promising, it would be important to provide results on inference cost: compute and latency compared to other techniques.
  • It is unclear how performance depends on the number of diffusion/flow matching steps. Is it matched with other techniques in the comparisons?
  • The paper doesn't discuss causal/streaming auto-encoding. Can this framework be adapted to such settings?
Questions for authors

How does the proposed method compare in terms of training and inference cost to other methods? Does it require more data to achieve comparable performance to other techniques due to the addition of the diffusion rate model? Is the inference cost comparable to other techniques?

Minor: can authors explain why the degradation is so mild when shuffled captions are used in Table 3 (b)? This result is counter-intuitive since in this scenario we train the model with a noise input.

7 comments
AuthorsOpenReview ↗

Response to Reviewer LD7t (1/2)

We sincerely thank the reviewer for the positive feedback and for recognizing that our "motivation is clear and valid", that the method is "sensible", and that the "numerical results are promising." We hope our point-by-point response can fully resolve all your concerns.

  1. Inference cost (compute and latency) should be compared with other techniques. How does the training cost compare, and does the diffusion rate model require more data?

Thank you for raising these practical questions. The inference cost of AuDiAE is comparable to that of single-pass autoencoders, because the diffusion rate model shapes the latent in training and is not run for encoding or decoding; it also requires no additional training data. Inference runs the encoder once and the waveform decoder four times. We have added the parameter count and the real-time factor (RTF, encoding plus decoding time divided by audio duration) of every system to Table 1 of the revised manuscript, all timed under one protocol (one NVIDIA A800 GPU, FP32, batch size 1, a 30-s 44.1 kHz clip; "Inference cost" paragraph of App. C.2). RTF is 0.018–0.019 for the single-pass autoencoders (Stable Audio Open, εar-VAE, SAME-L), 0.023 for AuDiAE and 0.026 for AuDiAE-H, and 0.059 and 0.103 for the generative autoencoders Moûsai and CoDiCodec. In training, the rate model is a small four-block DiT (App. C.1) that is fit to the latents of the same audio batches and optimized in the same updates as the encoder and the decoder (Sec. 3.4), so it adds neither a training stage nor training data. Table 3(a) tests whether it needs more data: with the same data and the same number of updates as the fixed-Gaussian control (App. C.1), the learned rate is better on all six reconstruction and generation metrics.

  1. How does performance depend on the number of diffusion/flow-matching steps, and is it matched across methods?

Thank you for raising this point. The number of decoder evaluations trades agreement with the reference waveform (SI-SDR) for realism (rFAD), as follows from how the decoder is trained and sampled; four evaluations, the setting of every AuDiAE result (App. C.2), obtain most of the realism. Across methods, each system is evaluated at the native number of decoder evaluations (NFE) of its own decoder, which we have now listed in Table 1 next to the measured inference cost. The decoder is trained to predict the clean waveform from a noisy one and the latent (Eq. 13), so its optimum is the average of the waveforms consistent with both (Sec. 3.3), and it decodes by integrating the flow ODE from Gaussian noise (App. C.2). With one evaluation the input is pure noise, so the output is the average of all waveforms that the latent "can remain compatible with" (Sec. 3.3): this average minimizes squared error, giving the highest SI-SDR (12.49 dB), but smooths out the detail in which these waveforms differ, giving the worst rFAD (0.200). Each further evaluation conditions the prediction on a less noisy waveform, so the ODE settles on one of these waveforms and restores its detail: rFAD falls to 0.062 at four evaluations and 0.053 at sixteen. This detail is the sample's own rather than the reference's, and an exact sample has, in expectation, twice the squared error of the average, i.e., an SI-SDR lower by 10log⁡102≈310\log_{10}2\approx 3 dB; the measured drop from one to sixteen evaluations is 2.96 dB. The decrease in SI-SDR is therefore the expected signature of sampling rather than a defect of the decoder, and the number of evaluations is a choice made at inference, without retraining. Four evaluations of the third-order solver give the lowest MEL distance and most of the rFAD gain at a third of the inference time of sixteen (Table 12 in App. F of the revised manuscript, reproduced below: the paired-caption model of Table 3(b) on AudioCaps, with only the number of evaluations varied).

Decoder evaluationsSI-SDR ↑MEL ↓rFAD ↓RTF ↓
112.490.0680.2000.011
211.320.0400.1040.015
4 (default)10.080.0340.0620.023
89.620.0370.0530.039
169.530.0380.0530.070

In the new NFE column of Table 1, Stable Audio Open, εar-VAE, and SAME-L use 1 evaluation, CoDiCodec 2, Moûsai 20, and AuDiAE and AuDiAE-H 4 (App. C.1). Because one evaluation costs differently in each decoder, NFE alone does not determine inference cost: CoDiCodec's two evaluations take longer than AuDiAE's four (RTF 0.103 vs. 0.023), and AuDiAE is close to the single-pass autoencoders (0.018–0.019; Response 1).

AuthorsOpenReview ↗

Response to Reviewer LD7t (2/2)

  1. Can the framework be adapted to causal or streaming autoencoding?

Thank you for raising this question. Yes. The rate objective needs no bidirectional context and both networks of AuDiAE are already largely local in time, so a streaming variant changes two components of the decoder and nothing in the objective. The rate bound (Eq. 8) depends only on the noise channel (Eq. 3) and the fit of the prior, so it holds for a causal encoder, and the prior is used only in training, so its view of the whole latent sequence adds no latency. The encoder's receptive field is 0.88 s, of which 0.44 s is look-ahead (App. C.1 of the revised manuscript), and the decoder's self-attention already uses a sliding window of 64 waveform tokens on each side. The decoder is offline only because its cross-attention to the latents spans the whole clip and its output gain is predicted from clip-level latent statistics; restricting the cross-attention to a window and predicting the gain from past latents would make it streamable after retraining the decoder under the same objective. Sampling does not limit latency (RTF 0.023 with four evaluations; Response 1); the floor is one latent frame, 46 ms at 2048× and 93 ms at 4096×, plus the look-ahead of the two networks. We have added this discussion as a new "Streaming" paragraph in App. E of the revised manuscript.

  1. Why is the degradation so mild when shuffled captions are used in Table 3(b)?

Shuffled captions are not a noisy input to the autoencoder: captions enter only the rate prior during training, while the encoder "observes only audio" and the waveform decoder receives only the latent (Sec. 3.2). The downstream generator of every row is also trained on correctly paired captions (App. C.3), so shuffling changes only the rate term of the autoencoder's objective. With paired captions, this rate "lowers the rate cost of information predictable from text" (Sec. 3.2, Eq. 10). A shuffled caption is independent of the audio and predicts nothing, so this discount vanishes and the objective reduces to the unconditional one. The shuffled model is therefore expected to match the unconditional model rather than fall below it, as Table 3(b) shows (FAD/CLAP 0.095/0.303 vs. 0.093/0.305), whereas paired captions reach 0.075/0.328: shuffling removes the entire gain of text conditioning.

Reviewer_LD7tOpenReview ↗

Response to rebuttal

I thank the authors for their reply, most of my questions have been addressed.

My remaining concerns are around causality and look-ahead in comparison methods.

In the response, authors state that there is no theoretical obstacle to causal rate prior and inference. However I don't see any experimental evidence to support this in practice. Whether the prior is still useful in a streaming setting is yet to be seen. Can authors provide a short discussion why they believe the proposed method would still perform well in a causal setting? Moreover, authors mention that the encoder uses 0.44s look-ahead. How much look-ahead is used in the models in the comparisons? AE performance strongly depends on how much the encoder/decoder can look into the future.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Main Points, New Results, and Revisions

We sincerely thank all reviewers for their constructive feedback and for recognizing AuDiAE's clear and valid motivation (LD7t, 5ksY), its separation of transmitted information from prior mismatch and variational slack (5ksY), and its promising results against prior autoencoders and the controlled variants (96K3, LD7t, 5ksY). We summarize the main clarifications and new results below; all revisions are marked in blue in the revised manuscript.

  • The diffusion prior serves as a rate model, not as a latent generator. In AuDiAE, the jointly trained diffusion prior determines what encoded information costs, and the waveform diffusion decoder determines what it is worth (Eqs. 14–15). In the diffusion-prior VAEs raised by Reviewer 96K3, the prior is instead the latent generator; relative to the joint diffusion prior–decoder training raised by Reviewer 5ksY, the rate decides which information the latent keeps, not only how much, and is accounted for as information, mismatch, and slack. The boundary term B_a is the starting-point term of the diffusion bound, not a Gaussian constraint on the latent. Replacing only the Gaussian KL improves reconstruction and generation together (Table 3(a)), and the gain of text conditioning requires paired captions (Table 3(b)): AuDiAE is an alternative to the Gaussian-KL paradigm of audio VAEs, not a refinement within it (see 96K3 R1–2; 5ksY R1).

  • The gains hold on identical data and do not come from model size. We have evaluated the fixed-Gaussian control and its learned-rate counterpart under the protocol of Table 1 (new Table 10), and generation in Tables 2 and 3 already shares one protocol. With the same network, the control does not lead downstream generation, whereas changing only the rate objective does; the parameter counts of all systems are now in Table 1 and the encoder's receptive field in App. C.1 (see 96K3 R3–4).

  • Isolated within one trained model, the generative decoder's contribution is perceptual and distributional fidelity. Decoding the same latents deterministically, as a point estimate, or generatively, by sampling, keeps the encoder, latent, rate constraint, and training budget fixed: the generative decoder halves the MEL distance and reduces rFAD by more than two thirds, at a lower SI-SDR, which favors the point estimate (new Table 12; see 5ksY R2).

  • Inference cost is comparable to single-pass autoencoders, and no additional data is needed. The rate model is trained on the autoencoder's own audio and is not run at inference; the new Params, NFE, and RTF columns of Table 1 compare all systems, four decoder evaluations obtain most of the benefit of sampling (Table 12), and a streaming variant requires changes to the decoder only (App. E; see LD7t R1–3).

  • AuDiAE also outperforms the discrete neural codecs DAC and MOSS-Audio-Tokenizer on every objective metric under the protocol of Table 1 (new Table 11; see 96K3 R5).

Revisions: Table 1 (Params, NFE, RTF); Sec. 3.4 (role of the rate prior); abstract, Sec. 4.2–4.3, and conclusion (decoder evidence of Table 12); App. A.1, A.3, A.4 (discrete codecs, diffusion-prior VAEs, joint diffusion prior–decoder training); App. C.1–C.2 (receptive field, inference cost); App. E (evaluation scope, efficiency, streaming, causal scope); new App. F (Table 10: fixed-Gaussian control under the protocol of Table 1; Table 11: discrete neural codecs; Table 12: number of decoder evaluations, from deterministic to generative decoding).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.