- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an audio autoencoder that combines a learned diffusion prior with a flow-matching waveform decoder to address the limitations of fixed Gaussian priors and deterministic decoding under high compression. The learned prior reduces the cost of aggregate posterior mismatch, while a text-conditional rate objective encourages the latent representation to retain information predictable from text. The flow-matching decoder models residual acoustic variation beyond the encoded latent information. Experiments at compression rates of 2048× and 4096× show that AuDiAE achieves the best results across all reported reconstruction metrics and improves AudioCaps generation, reducing FAD by 35% and achieving the highest CLAP score.
Critical strengths
- The paper is well-motivated. The authors combine source-faithful audio reconstruction with the learnability of latent representations for downstream conditional generation. The decomposition of the rate into mutual information and aggregate-prior mismatch provides a clear motivation for learning the rate reference.
- The paper distinguishes transmitted information from prior mismatch and variational slack, and explicitly acknowledges that the conditional rate is a regularizer rather than an operational bitrate or a guarantee of improved conditional generation.
- The proposed method achieves strong reconstruction metrics and the best reported AudioCaps FAD and CLAP scores.
Critical weaknesses
- The paper combines a learned diffusion prior, a text-conditional rate objective, and a flow-matching decoder. While the overall design is reasonable, it is not entirely clear what the main methodological advances and differences are from prior work, particularly existing approaches that jointly train diffusion priors and decoders.
- The generative decoder appears to be a key component of the proposed method, but the experiments do not clearly demonstrate its specific contribution. Since the main comparisons involve complete systems, it is difficult to determine whether the improvements come from the decoder itself or from other components. A controlled comparison with a deterministic decoder, while keeping the encoder and latent representation fixed, would provide more convincing evidence.
- Limited evaluation of downstream generation. The downstream generation experiments are conducted only on AudioCaps, which mainly contains sound effects. It remains unclear whether the learned representations also benefit conditional generation in other audio domains, such as music. Evaluating on additional datasets would help establish the generality of the proposed approach.
Questions for authors
- Can the authors provide a controlled comparison between deterministic and flow-matching decoders while keeping the encoder, latent representation, rate constraint, and training budget fixed?
3 comments
Response to Reviewer 5ksY (2/2)
- The generative decoder appears to be a key component; can it be compared with a deterministic decoder while keeping the encoder, latent representation, rate constraint, and training budget fixed?
Thank you for this constructive suggestion. The aim of the paper is to study how a diffusion prior and a diffusion decoder, trained jointly around one latent, shape its learnability and its reconstruction quality; replacing the decoder with a deterministic one would change this paradigm itself rather than ablate a component of it. In AuDiAE the decoder is one side of the coupling: its denoising risk determines what encoded information is worth to the encoder (Eqs. 14–15), so a different decoder yields a different latent. A deterministic decoder would in addition require adversarial training to shape the reconstruction, as in spectral–adversarial autoencoders (Sec. 1), which AuDiAE's objective does not contain (Eq. 16). The controlled analysis therefore trains a flow decoder of the same design jointly in every row and varies only the rate objective (Table 3), and deterministic decoding is compared at the system level: εar-VAE, Stable Audio Open, and SAME-L are single-pass deterministic autoencoders, and AuDiAE improves on all three in every reconstruction metric of Table 1.
The requested comparison can, however, be made within one trained model, and we have added it to App. F of the revised manuscript (Table 12). The same latents are decoded either deterministically, as the decoder's point estimate of the waveform (the conditional mean that a regression decoder targets), or generatively, by sampling; the encoder, latent representation, rate constraint, and training budget are therefore identical. On AudioCaps, the generative decoder halves the MEL distance of the deterministic one (0.068→0.034) and lowers rFAD from 0.200 to 0.062, while SI-SDR, which favors the conditional mean, decreases from 12.49 to 10.08 dB (Table 12). The specific contribution of the generative decoder is therefore perceptual and distributional fidelity: it models the residual variation that a deterministic point estimate averages out.
References
[1] Heek et al., "Unified Latents (UL): How to Train Your Latents." arXiv:2602.17270, 2026.
Follow-Up on Our Responses
Dear Reviewer 5ksY,
As the public discussion period ends on November 18, we would be grateful to know whether our responses address your questions. Thank you again for your constructive suggestions, which have helped us improve the paper.
Best regards,
The Authors
Response to Reviewer 5ksY (1/2)
We sincerely thank the reviewer for the positive feedback and for recognizing that AuDiAE is "well-motivated", that our rate decomposition "provides a clear motivation for learning the rate reference", and that AuDiAE achieves "strong reconstruction metrics and the best reported AudioCaps FAD and CLAP scores." We hope our point-by-point response can fully resolve all your concerns.
Thank you for this important positioning question. AuDiAE asks how to learn compact audio latents that a downstream text-conditioned generator can model readily and that still support source-faithful waveform synthesis (Sec. 1). Joint training of a diffusion prior and a decoder [1], on which AuDiAE builds (Sec. 5), learns latents for images and video under a marginal prior, whose rate limits how much information the latent carries but not which. The main methodological advance of AuDiAE therefore lies in the rate: it decides which information the latent keeps for a text-conditioned generator, and it is accounted for as information, mismatch, and slack rather than as a bitrate. Three advances follow, (1)–(2) over joint diffusion training and (3) over audio autoencoders:
(1) The rate decides which information the latent keeps, whereas a marginal prior bounds only how much (Sec. 3.2). A text-conditioned generator must model only the latent variation that the caption leaves unexplained, I(X;Zb∣C)=I(X;Zb)−I(Zb;C) (Eq. 2). Under a marginal rate, however, two latents with the same marginal distribution and the same total information receive the same rate even when only one is predictable from text (App. B.3), and [1] itself observes that text alignment degrades slightly at its lowest bitrates (Sec. 5.3 therein). AuDiAE therefore feeds the same prior either the caption or a null condition and mixes the two bounds with weight ρ (Eq. 10): information predictable from text, I(Zb;C), is charged only a fraction 1−ρ, while ρ<1 keeps total information bounded (Eq. 11). Text enters only this rate, never the encoder or decoder. Table 3(b) tests this against joint training with a marginal prior, i.e., the same model with the marginal rate alone: paired captions improve FAD/CLAP from 0.093/0.305 to 0.075/0.328, whereas shuffled captions, which keep the conditional branch and ρ, stay at 0.095/0.303; the gain thus comes from text–audio pairing.
(2) The rate is separated into information, mismatch, and slack, whereas joint training reads its bound as the latent bitrate (Sec. 3.1). In [1], the bound is "a tight upper bound on the latent bitrate" (abstract therein). In AuDiAE, by contrast, it is the information the latent transmits plus the mismatch of its aggregate distribution to the prior plus variational slack (Eq. 8), and for a fixed encoder, learning the prior lowers only the last two. This separation determines what each component can claim: a smaller bound after learning the prior reflects a better fit, not less information, and since the text discount of (1) also contains mismatch and slack, the conditional rate is a regularizer, not a bitrate or a measure of text-predictable information (Sec. 3.2), as the reviewer notes among the strengths. AuDiAE therefore tests the learned prior with the encoder frozen, where information is fixed: its bound is tighter than that of the best fitted Gaussian (3.104 vs. 3.264 nats per element; Table 6).
(3) In audio, the rate is learned and the decoder generates what the latent leaves open, whereas audio VAEs measure the latent against a fixed Gaussian and decode a point estimate (Sec. 3.1, 3.3). Under a tight rate budget, matching a fixed Gaussian consumes rate that could carry audio detail, and at high compression (2048× and 4096× here), one latent is compatible with many waveforms, which spectral–adversarial decoders collapse into a single estimate. AuDiAE trains the diffusion rate jointly with a flow-matching waveform decoder on the same noised encoder output Zb (Eqs. 3, 12), and weighs the denoising risk the latent removes against its rate (Eqs. 14–15). In audio VAEs, the KL weight trades reconstruction against downstream generation (Sec. 1). Replacing only the Gaussian KL by the learned rate, with the encoder, channel, decoder, data, and β matched, instead improves both: SI-SDR by 1.07 dB, rFAD by 17.0%, and FAD/CLAP from 0.142/0.274 to 0.093/0.305 (Table 3(a)); the complete system leads every reconstruction metric of Table 1 and AudioCaps generation (Table 2). AuDiAE is therefore an alternative to the Gaussian-KL paradigm of audio VAEs, not a refinement within it.
We have added this discussion to App. A.4 of the revised manuscript.