acceptodds
Under review as a conference paper at ICLR 2027

AudioGAR: Bridging Reconstruction and Generation in Latent Audio Generative Models

Abstract

Latent audio generative models are typically trained in two stages: an audio codec is learned first, followed by a latent generative model. This decomposition leads to a decoder train-generation mismatch: the codec decoder is trained on encoder-induced latents but deployed on generator-produced latents at inference time. Across diverse datasets and latent generative models, we observe clear reconstruction-generation gaps under both FD and FAD, showing that strong reconstruction quality does not necessarily translate into strong end-to-end generation quality. A natural remedy is to adapt the decoder on generation-produced latents, but generated latents lack correspondence with source audio and therefore cannot directly provide the paired supervision used for decoder fine-tuning. We introduce AudioGAR, which constructs intermediate latents by perturbing encoder latents and denoising them through the frozen latent diffusion model. These latents form a trajectory from reconstruction toward generation, with lower-noise latents retaining source correspondence and supporting paired decoder fine-tuning. We fine-tune only the codec decoder on these latents, while keeping the codec encoder and latent generative model frozen. When applied to AudioX, AudioGAR substantially improves generative performance. It requires only 1.5% of the original training audio hours and 0.26% of the original training cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio Generation, Latent Generative Models, Audio Codec, Latent Diffusion Models, Decoder Adaptation, Reconstruction–Generation Gap, Generation-Aware Latents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Haf4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AudioGAR, a decoder adaptation method for latent audio generative models. The paper identifies a distribution mismatch between encoder-produced latents used to train the codec decoder and generator-produced latents encountered during inference. AudioGAR constructs intermediate latents by partially noising encoder latents and denoising them using the frozen generative model. These latents preserve correspondence with the original audio while moving toward the distribution of generated latents, allowing the codec decoder to be fine-tuned with the original reconstruction objective. Experiments on AudioX and TangoMusic show improvements mainly in FD and FAD with relatively low additional training cost.

Critical strengths

The idea of improving the decoder (generation gap) is interesting.

Critical weaknesses
  1. The overall improvement is relatively limited and is mainly reflected in FD and FAD. For example, on AudioCaps, FD only improves from 11.83 to 11.62. Although larger improvements can be observed in FAD and on MusicCaps, the gains are not consistent across different evaluation metrics. Some metrics even slightly degrade after fine-tuning, such as IS and PC on AudioCaps. Therefore, it remains unclear whether AudioGAR leads to an overall improvement in generated audio quality rather than primarily improving FD/FAD.

  2. Human evaluation is missing. Since the reported improvements are mainly reflected by FD/FAD, objective metrics alone are insufficient to demonstrate whether the improvement is perceptually meaningful. A human listening study would provide important evidence that the improvements achieved by AudioGAR are actually noticeable in generated audio.

  3. The overall significance of the method appears relatively limited. AudioGAR only fine-tunes the codec decoder to better adapt to generator-produced latents, while the underlying generative model itself remains unchanged. Therefore, the method does not directly improve the generation process or the capability of the generative model, which makes the overall impact of the proposed approach less clear.

  4. The evaluation across different generative models is limited. For general audio generation, the experiments are mainly conducted on AudioX, while TangoMusic is used for music generation. Given that the proposed decoder train-generation mismatch is intended to be a general issue for latent diffusion/flow-based audio generation, experiments on more model families would strengthen the generality of the conclusions. For example, AudioLDM-based models with VAE representations and Stable Audio Open with a neural audio codec would provide useful additional test cases.

Questions for authors
  1. Can the authors provide human evaluation to verify the improvement in audio quality?
  2. Why do some metrics such as IS and PC become worse after fine-tuning?
  3. Does the method also work on other latent diffusion audio generation models, such as VAE from AudioLDM and Codec from Stable Audio Open ?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Haf4(1/4)

We appreciate that the reviewer found our idea of improving the decoder to address the generation gap interesting, and recognized that AudioGAR improves FD and FAD on AudioX and TangoMusic with relatively low additional training cost. Thank you for taking the time to review our paper and for your thoughtful feedback. We address your questions and concerns below.


R 3.1 The overall improvement is relatively limited and is mainly reflected in FD and FAD. For example, on AudioCaps, FD only improves from 11.83 to 11.62. Although larger improvements can be observed in FAD and on MusicCaps, the gains are not consistent across different evaluation metrics. Some metrics even slightly degrade after fine-tuning, such as IS and PC on AudioCaps. Therefore, it remains unclear whether AudioGAR leads to an overall improvement in generated audio quality rather than primarily improving FD/FAD.

Thank you for this comment. We agree that the FD gain of AudioX on AudioCaps is small and that IS and PC drop slightly there. We address this concern from three aspects.

First, the changes in other metrics are small. AudioGAR only changes the decoder, while the generator, and therefore the content of the generated audio, remains unchanged. KL, IS and PC mainly reflect this content, so we do not expect them to change much. On AudioX, KL slightly improves on both datasets, while IS and PC decrease by less than 4%. In comparison, FAD improves by 16.8% on AudioCaps and 31.3% on MusicCaps, and FD by 12.9% on MusicCaps.

Second, the gains are larger and broader on other models. We applied AudioGAR to two additional latent diffusion models, Stable Audio Open and AudioLDM-2-Large, using the same procedure and noise level (ηt=0.2\eta_t = 0.2), without any model-specific tuning (full results in R 3.4 and Table 3 of the revised paper). On Stable Audio Open, FD improves by 24.2% on AudioCaps and 27.7% on MusicCaps, and KL, IS and PQ also improve on both datasets. Across both models, the remaining metrics change only slightly and in both directions, consistent with the content being determined by the unchanged generator. We have added these results to Table 3 and Section 4.2 of the revised paper.

Third, FD and FAD are the main audio quality metrics in prior work. AudioLDM [1] states that "we choose FD as the main evaluation metric", ControlAudio [2] uses FAD as its "primary metric for audio fidelity" and treats IS as a supplementary metric, and BATON [3] uses FD "as the primary metric for assessing audio quality". As distribution-level measures of how close generated audio is to real audio, they are also the most sensitive to the acoustic fidelity that the decoder determines.

[1] Liu et al. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. ICML 2023.

[2] Jiang et al. ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling. ACL 2026.

[3] Liao et al. BATON: Aligning Text-to-Audio Model with Human Preference Feedback. IJCAI 2024.

AuthorsOpenReview ↗

Response to Reviewer Haf4(2/4)

R 3.2 Human evaluation is missing. Since the reported improvements are mainly reflected by FD/FAD, objective metrics alone are insufficient to demonstrate whether the improvement is perceptually meaningful. A human listening study would provide important evidence that the improvements achieved by AudioGAR are actually noticeable in generated audio.

Thank you for this suggestion. We agree that a listening study is important, and we have conducted a blind pairwise preference test. We generated 100 music clips with AudioX on MusicCaps. Each generated latent was decoded twice, once with the original decoder and once with the AudioGAR decoder, so the two versions differ only in the decoder. Three listeners heard each pair in random order, without knowing which decoder produced each version, and chose the version with better audio quality, or indicated no preference.

ListenerPrefer AudioGARNo preferencePrefer original decoder
Listener 116%70%14%
Listener 219%69%12%
Listener 313%71%16%
Mean ± std16.0 ± 3.0%70.0 ± 1.0%14.0 ± 2.0%

Listeners reported no preference for most pairs (70.0% on average). When a preference was given, the AudioGAR version was chosen slightly more often than the original (16.0% vs. 14.0%), although this difference is not statistically significant at this scale. We draw two conclusions from this result. First, AudioGAR does not lead to a noticeable degradation in perceptual quality, while improving FD and FAD. Second, FD and FAD measure differences between distributions over the whole evaluation set, which do not necessarily appear as clearly audible differences within individual pairs. We acknowledge that a larger study would be needed to establish whether the gains in FD and FAD translate into consistently noticeable improvements. We have added this evaluation and discussion to Section 4.3 and Appendix K (Table 11) of the revised paper, and updated the Ethics Statement and Limitations (Appendix M) accordingly.

R 3.3 The overall significance of the method appears relatively limited. AudioGAR only fine-tunes the codec decoder to better adapt to generator-produced latents, while the underlying generative model itself remains unchanged. Therefore, the method does not directly improve the generation process or the capability of the generative model, which makes the overall impact of the proposed approach less clear.

Thank you for this comment. We see the fact that AudioGAR leaves the generator unchanged as a strength rather than a limitation, for three reasons.

First, the decoder is part of the generation process. Every generated sample passes through it, so the final audio quality depends on the decoder as well as on the generator. Our analysis in Section 2 shows that the decoder is trained on encoder latents but used on generated latents, and that this mismatch is accompanied by a clear reconstruction-generation gap. Most prior work improves the generator; AudioGAR addresses the other side of the same pipeline.

Second, under the same budget, adapting the decoder improves FD and FAD more than further training the generator. We continued fine-tuning the AudioX generator with the same data and the same training budget as AudioGAR (10.3 hours on a single H100):

DatasetStrategyKL ↓IS ↑FD ↓FAD ↓PC ↑PQ ↑
AudioCapsAudioX (baseline)1.3112.4711.831.613.165.73
AudioCapsGenerator fine-tuning1.2712.5912.322.023.165.77
AudioCapsAudioGAR(Ours)1.2912.0611.621.343.115.73
MusicCapsAudioX (baseline)1.003.659.561.604.786.61
MusicCapsGenerator fine-tuning0.913.249.041.655.316.88
MusicCapsAudioGAR(Ours)0.993.598.331.104.756.56

Generator fine-tuning gives mixed results: it improves KL and PQ, but degrades FD and FAD on AudioCaps and gives only a modest FD gain on MusicCaps. In contrast, AudioGAR consistently improves FD and FAD on both datasets. This comparison is reported in Table 5 and Section 4.3 of the revised paper.

Third, because the generator is not changed, AudioGAR can be applied directly to released models and could in principle be combined with generator-side improvements. Indeed, it improves FD and FAD on all four models we have tested without model-specific tuning (see R 3.4). For AudioX, it needs only 1.5% of the original training audio and 0.26% of the original training cost, and it adds no inference cost. We have added this discussion to Section 4.3 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer Haf4(3/4)

R 3.4 The evaluation across different generative models is limited. For general audio generation, the experiments are mainly conducted on AudioX, while TangoMusic is used for music generation. Given that the proposed decoder train-generation mismatch is intended to be a general issue for latent diffusion/flow-based audio generation, experiments on more model families would strengthen the generality of the conclusions. For example, AudioLDM-based models with VAE representations and Stable Audio Open with a neural audio codec would provide useful additional test cases.

Thank you for this suggestion. Following it, we have applied AudioGAR to Stable Audio Open and AudioLDM-2-Large on both general audio (AudioCaps) and music (MusicCaps), using the same procedure and noise level (ηt=0.2\eta_t = 0.2) as in the paper, without any model-specific tuning.

DatasetModelAudioGARKL ↓IS ↑FD ↓FAD ↓PC ↑PQ ↑
AudioCapsStable Audio Open×2.0410.1230.363.412.696.08
AudioCapsStable Audio Open✓1.9810.2723.003.072.696.10
AudioCapsAudioLDM-2-Large×1.2210.1511.531.863.145.80
AudioCapsAudioLDM-2-Large✓1.2110.0211.061.443.155.80
MusicCapsStable Audio Open×1.562.9837.723.423.757.26
MusicCapsStable Audio Open✓1.533.1327.293.373.707.28
MusicCapsAudioLDM-2-Large×1.172.7018.842.515.387.23
MusicCapsAudioLDM-2-Large✓1.142.7517.182.495.427.19

AudioGAR improves FD and FAD for both models on both datasets, while the other metrics remain close to the baselines. The gains are most pronounced in FD on Stable Audio Open (24.2% on AudioCaps and 27.7% on MusicCaps) and in FAD on AudioLDM-2-Large for AudioCaps (22.6%), whereas the FAD gains on MusicCaps are smaller.

These additions extend our evaluation in two ways. Stable Audio Open shares its neural audio codec with AudioX but uses a different generator, so it tests whether AudioGAR transfers across generators with the same decoder. AudioLDM-2-Large uses a VAE-based latent representation, which was previously covered only for music generation by TangoMusic, and is now evaluated on both general audio and music. Overall, AudioGAR improves FD and FAD on all four models we have tested, covering both neural audio codec and VAE representations on both general audio and music generation. We have added these results to Table 3.

R 3.5 Can the authors provide human evaluation to verify the improvement in audio quality?

Thank you for this question. We have conducted a blind pairwise listening study on 100 pairs of AudioX music clips that differ only in the decoder, with three listeners. Listeners reported no preference for most pairs (70.0% on average); when a preference was given, the AudioGAR version was chosen slightly more often than the original (16.0% vs. 14.0%), although this difference is not statistically significant at this scale. The full setup, results and discussion are given in our response to R 3.2 and in Section 4.3 and Appendix K of the revised paper.

Reviewer_bXkLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper firstly identifies a mismatch in two-stage latent audio generative models, i.e., the codec decoder is trained on encoder-induced latents but deployed on generator-produced latents. Authors then propose AduioGAR to adapt the decoder. AudioGAR perturbs encoder latents with noise and partially denoises them with a frozen latent diffusion model. The decoder alone is then fine-tuned on these latents using the original codec losses. Authors show improved FAD on AudioCaps and MusicCaps using AudioX.

Critical strengths
  • I think the idea is simple and intuitive, and its adaptation is straightforward, without changing the generator and inference pipeline.

  • Authors provide clear ablations by using different noise levels to show how AudioGAR can adapt the decoder.

Critical weaknesses
  • The biggest concern I have is that AudioGAR is not compared with other existing methods adapting the decoder. There is no discussions about the difference of AudioGAR w.r.t. the literature, e.g., SDEdit, which is closely related.

  • Incremental and limited novelty. This work seems an audio adaption of Fang et al. 2026; authors should discuss the differences with it to highlight the contributions of this work.

  • AudioGAR brings improves on FD and FAD, but on the other metrics like IS, the results are mixed.

SDEdit, SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, ICLR 2022 Fang et al 2026, A Latent Distribution Perspective on Evaluating and Improving Latent Generative Models

Questions for authors

I think AudioGAR should be compared with some other decoder adaptation methods. E.g., one simple baseline would be fine-tuning on noise-perturbed encoder latents without any denoising. How would AudioGAR compare to such baseline?

4 comments
AuthorsOpenReview ↗

Response to Reviewer bXkL(1/3)

We appreciate that the reviewer recognized AudioGAR as a simple and intuitive idea with a straightforward adaptation that requires no changes to the generator or the inference pipeline. We also appreciated the recognition of our clear ablations over different noise levels, which show how AudioGAR adapts the decoder. Thank you for taking the time to review our paper and for your thoughtful feedback. We address your questions and concerns below.


R 2.1 The biggest concern I have is that AudioGAR is not compared with other existing methods adapting the decoder. There is no discussions about the difference of AudioGAR w.r.t. the literature, e.g., SDEdit, which is closely related.

Thank you for raising this. The closest prior work is GAR (Fang et al., 2026), which adapts the decoder to generated latents in visual generation. AudioGAR builds on GAR and brings this idea to latent audio generation, where we adapt each audio codec's decoder with its own training objective. To our knowledge, no existing method in audio generation adapts the codec decoder to generated latents. We therefore compare AudioGAR with alternative ways of adapting a pretrained system, all on AudioX with the same data and the same training budget (10.3 hours on a single H100):

  • Generator fine-tuning: continue training the latent generator, keeping the decoder fixed.
  • Decoder fine-tuning on zez_e: fine-tune the decoder on the original encoder latents.
  • Decoder fine-tuning on noisy latents: fine-tune the decoder on znoisytz_{\text{noisy}}^t, i.e., encoder latents with added Gaussian noise but without denoising by the generator, at the same noise level as AudioGAR (ηt=0.2\eta_t = 0.2).
DatasetStrategyFD ↓FAD ↓
AudioCapsAudioX (baseline)11.831.61
AudioCapsGenerator fine-tuning12.322.02
AudioCapsDecoder fine-tuning on zez_e11.721.55
AudioCapsDecoder fine-tuning on noisy latents12.041.62
AudioCapsAudioGAR11.621.34
MusicCapsAudioX (baseline)9.561.60
MusicCapsGenerator fine-tuning9.041.65
MusicCapsDecoder fine-tuning on zez_e8.871.38
MusicCapsDecoder fine-tuning on noisy latents9.471.54
MusicCapsAudioGAR8.331.10

AudioGAR gives the best FD and FAD among all strategies. Generator fine-tuning degrades FD and FAD on AudioCaps and gives only a modest FD gain on MusicCaps. Fine-tuning the decoder on zez_e helps somewhat; since it uses the same data as AudioGAR, its gain mainly reflects the new training data. Fine-tuning on noisy latents is worse than fine-tuning on zez_e. We attribute this to the fact that adding noise moves the latents toward an isotropic Gaussian rather than toward the generated latents, so the decoder is trained on inputs it never receives at generation time. Results for other noise levels are given in our answer to R 2.4. In the revised paper, the comparison with generator fine-tuning is in Table 5, the comparison with decoder fine-tuning on zez_e is in Table 4, and the noisy-latent baseline is in Appendix I (Table 9); all are discussed in Section 4.3

Relation to SDEdit. SDEdit perturbs an input with noise and partially denoises it with a pretrained diffusion model. It is an inference-time method for image synthesis and editing: the denoised sample is the final output, and no model is trained. In its original form, SDEdit operates on pixel-space diffusion models, so no decoder is involved. The latent construction step of AudioGAR uses the same perturb-and-denoise operation in latent space, but for a different purpose: the denoised latents are not outputs but training inputs, which remain paired with the source audio and are used to adapt the decoder to generation-time inputs. We have added the discussion of GAR to the Introduction and Section 3.3, the discussion of SDEdit to Section 3.1, and the comparisons above to Section 4.3 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer bXkL(2/3)

R 2.2 Incremental and limited novelty. This work seems an audio adaption of Fang et al. 2026; authors should discuss the differences with it to highlight the contributions of this work.

Thank you for this comment. We agree that AudioGAR builds directly on GAR (Fang et al., 2026), and we have revised the Abstract, Introduction, contribution list and Sections 3.1 and 3.3 to make this attribution explicit, together with a discussion of the differences. We kept the core idea close to GAR to test whether it transfers to latent audio generation, which was not clear in advance. Our work differs from and complements GAR in four respects:

  • First, a systematic study of the mismatch in audio (Section 2). Whether this mismatch exists in latent audio generation had not been examined. We show that encoder-induced and generator-produced latents are clearly separable, even inside the decoder, and that a clear reconstruction-generation gap exists across 29 model–dataset combinations.
  • Second, a different class of generators. GAR applies decoder adaptation to one-step flow-map models (iMF, MeanFlow, DMF) on class-conditional ImageNet. We apply it to multi-step, text-conditioned latent diffusion models (AudioX, TangoMusic, Stable Audio Open and AudioLDM-2-Large; Table 3).
  • Third, GAR trains the decoder with image-specific losses (LPIPS, a DINO-based discriminator and a Gram loss), which have no direct counterpart in audio, so its training recipe cannot be used as is. We instead adapt the decoder with each audio codec's own training objective, such as the multi-resolution spectral and adversarial losses of the Stable Audio Open codec and the HiFi-GAN objective for TangoMusic.
  • Fourth, the decoders themselves are different: GAN-trained waveform codecs, and in TangoMusic a VAE decoder followed by a HiFi-GAN vocoder.

Overall, while the core idea originates from GAR, our work shows that the mismatch is pervasive in latent audio generation and that the remedy transfers to substantially different generators, decoders and objectives.

R 2.3 AudioGAR brings improves on FD and FAD, but on the other metrics like IS, the results are mixed.

Thank you for this observation. We agree that the changes in IS are mixed, but they are small in both directions. The table below reports IS before and after AudioGAR for all four models we have evaluated.

DatasetModelIS (baseline) ↑IS (AudioGAR) ↑
AudioCapsAudioX12.4712.06
AudioCapsStable Audio Open10.1210.27
AudioCapsAudioLDM-2-Large10.1510.02
MusicCapsAudioX3.653.59
MusicCapsStable Audio Open2.983.13
MusicCapsAudioLDM-2-Large2.702.75
MusicCapsTangoMusic2.852.83

IS increases in three of the seven settings and decreases in four, and all changes are within ±5%. In contrast, AudioGAR consistently improves FD and FAD; for example, on AudioX it reduces FAD by 16.8% on AudioCaps and 31.3% on MusicCaps.

We focus on FD and FAD for two reasons. First, prior work commonly uses FD or FAD as the main measure of audio quality. AudioLDM [1] states that "we choose FD as the main evaluation metric", ControlAudio [2] uses FAD as its "primary metric for audio fidelity" and includes IS as a supplementary metric, and BATON [3] uses FD "as the primary metric for assessing audio quality".

Second, AudioGAR only changes the decoder, while the generator, and therefore the content of the generated audio, remains unchanged. KL, IS and PC mainly reflect this content, so their changes are expected to be small, whereas FD and FAD are sensitive to the acoustic fidelity that the decoder determines.

[1] Liu et al. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. ICML 2023.

[2] Jiang et al. ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling. ACL 2026.

[3] Liao et al. BATON: Aligning Text-to-Audio Model with Human Preference Feedback. IJCAI 2024.

AuthorsOpenReview ↗

Response to Reviewer bXkL(3/3)

R 2.4 I think AudioGAR should be compared with some other decoder adaptation methods. E.g., one simple baseline would be fine-tuning on noise-perturbed encoder latents without any denoising. How would AudioGAR compare to such baseline?

Thank you for suggesting this baseline. The comparison with other adaptation strategies is given in our response to R 2.1. Here we report results of the suggested baseline across noise levels on AudioX, adapting the decoder directly on the noisy latents znoisytz_{\text{noisy}}^t without the generator denoising step, using the same data and training budget. Results on AudioCaps are shown below.

Training latentsNoise levelFD ↓FAD ↓
No adaptation (AudioX)–11.831.61
zez_e0.011.721.55
znoisytz_{\text{noisy}}^t0.112.972.32
znoisytz_{\text{noisy}}^t0.212.041.62
znoisytz_{\text{noisy}}^t0.416.772.51
znoisytz_{\text{noisy}}^t0.621.293.32
znoisytz_{\text{noisy}}^t0.821.514.05
znoisytz_{\text{noisy}}^t1.019.745.26
AudioGAR (zgtz_g^t)0.211.621.34

Without the denoising step, adapting on noisy latents is worse than adapting on zez_e at every noise level, and at every level even worse than no adaptation. Prior work [4] reports a similar finding for visual generation in its Appendix A.3: post-training only the decoder of MAR-VAE with latent noise worsens FID with CFG. We attribute this to the fact that adding Gaussian noise moves the latents toward an isotropic Gaussian rather than toward the generated latents. The decoder is then trained on perturbations it never receives at generation time, while the noise also removes information about the source audio and weakens the paired supervision. At the same noise level (ηt=0.2\eta_t = 0.2), AudioGAR clearly outperforms noisy-latent adaptation, which shows that its gains do not come from noise acting as data augmentation, but from denoising with the generator, which moves the latents toward Pg\mathcal{P}_g while keeping them paired with the source audio. Both steps are therefore needed. We have added this ablation to Section 4.3 and Appendix I (Table 9) of the revised paper.

[4] Latent Denoising Makes Good Visual Tokenizers. ICLR 2026.


Please let us know if you have any remaining questions. If our clarifications and additional experiments have satisfactorily addressed your questions and concerns, we would deeply appreciate it if you considered recommending our paper to be presented at the conference.


AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer bXkL,

As the discussion period ends in a few hours, we wanted to kindly check whether our responses have addressed your concerns. We would be glad to clarify anything further before the deadline.

Thank you for your time!

Best regards,

The Authors

Reviewer_vCFGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses mismatch between encoder-induced latents and generator-produced latents and the reconstruction-generation performance gap in audio generative models. It starts with an analysis of the mismatch between the two latent distributions and the performance gap between reconstruction and generation, which motivate the proposed solution. To improve generation performance, the paper introduces AudioGAR: adapting the decoder to an augmentation of the encoder-induced latents. The augmentation involves adding noise the original encoder-induced latents, then denoise using the intended generative model. In the adaptation step, only the decoder is fine-tuned, using the standard training objectives but with the denoised latents instead of the original encoder-induced latents. On the tested models and datasets, the adaptation improves generation performance in key metrics (FD and FAD), while preserving others (KL, IS, …) compared to baselines. Ablation analyzes the trade-off in the noise level added during the augmentation process.

Critical strengths
  1. The manuscript is overall well-organized and overall easy to follow. The problem is well motivated with some empirical evidence (Sec 2).
  2. The proposed solution (to adapt the decoder to the generative latent distribution) is well-motivated and I appreciate that most design choices are mostly explained clearly.
  3. The solution design is simple and efficient, requiring no additional computation at inference time.
  4. Both Sections 2 and 4 cover provide decent empirical coverage with a large number of baselines and a variety of metrics. The additional analysis is carefully chosen to explore the impact of AudioGAR to the generation performance.
  5. The performance results of AudioX+AudioGAR are encouraging, outperforming other models and base AudioX.
Critical weaknesses
  1. AudioGAR has significant methodological overlap with one cited work for visual generative models (GAR; Fang et al., 2026). The pipeline for AudioGAR (generative model de-noising perturbed latents, decoder adaption) is mostly invented by GAR, but applied to audio models with limited innovation.
  2. How well does AudioGAR work when applied to other models besides AudioX/TangoMusic? The author acknowledges this limitation, but I believe more experiments on compatible models are needed to confirm the effectiveness/generalizability of this strategy. If this is not feasible, can the authors provide some elaboration?
  3. How does AudioGAR compare to alternative adaptation strategy (e.g., fine-tuning the generative model, among others)?
  4. The experiment reports performance gains on generation but without the impacts on reconstruction. I think this would be important information given the framing of the paper.
  5. FT the decoder on z_e (the original encoder-induced latents) already results in ~20-50% the FD/FAD gains of AudioGAR (Table 4), which slightly undermines added values of AudioGAR.

[1] Xianghong Fang, Wenjie Shu, Tongda Xu, Wenlong Mou, Dehan Kong, and Tim G. J. Rudner. Bridging reconstruction and generation: A latent distribution perspective on evaluation and improvement. ArXiv, 2026.

Questions for authors
  1. In Sec 4.1: Can the authors suggest a principled way to select the noise level ηt\eta_t? Also, can the authors elaborate on why this level of noise would be appropriate?
  2. As I understand it, AudioGAR has 2 steps: adding Gaussian noise to the encoder-induced latents, and using the generative model to denoise the noisy latents. I wonder about the impact of the second step (denoising with the generative model), maybe an ablation of adapting the decoder to the noisy latents would address this.
  3. Sec 2 provides evidence for the distribution mismatch and the performance gap, but does not establish a causal link between them (i.e., the mismatch is the cause of the gap). While the authors explicitly acknowledges this, I wonder if we can verify it empirically?
6 comments
AuthorsOpenReview ↗

Response to Reviewer vCFG (1/5)

Thank you for your thoughtful feedback. We appreciate that the reviewer found our manuscript well-organized and easy to follow, with the problem well motivated empirically and the design choices clearly explained. We also appreciate the recognition of AudioGAR as a simple, efficient solution with no extra inference cost, the broad empirical coverage across baselines and metrics, and the encouraging AudioX+AudioGAR results. We address your concerns below.


R 1.1 AudioGAR has significant methodological overlap with one cited work for visual generative models (GAR; Fang et al., 2026). The pipeline for AudioGAR (generative model de-noising perturbed latents, decoder adaption) is mostly invented by GAR, but applied to audio models with limited innovation.

Thank you for this comment. We agree that AudioGAR builds directly on GAR (Fang et al., 2026), and we have revised the Abstract, Introduction, contribution list and Sections 3.1 and 3.3 to make this attribution explicit. We kept the core idea close to GAR to test whether it transfers to latent audio generation, which was not clear in advance. Our work complements GAR in three respects:

  • First, A systematic study of the mismatch in audio (Section 2). Whether this mismatch exists in latent audio generation had not been examined. We show that encoder-induced and generator-produced latents are clearly separable, even inside the decoder, and that a clear reconstruction–generation gap exists across 29 model-dataset combinations.
  • Second, a different class of generators. GAR applies decoder adaptation to one-step flow-map models (iMF, MeanFlow, DMF) on class-conditional ImageNet. We apply it to multi-step, text-conditioned latent diffusion models (AudioX, TangoMusic, Stable Audio Open and AudioLDM-2-Large; Table 3).
  • Third, GAR trains the decoder with image-specific losses (LPIPS, a DINO-based discriminator and a Gram loss), which have no direct counterpart in audio, so its training recipe cannot be used as is. We instead adapt the decoder with each audio codec’s own training objective, such as the multi-resolution spectral and adversarial losses of the Stable Audio Open codec and the HiFi-GAN objective for TangoMusic.
  • Fourth, the decoders themselves are different: GAN-trained waveform codecs, and in TangoMusic a VAE decoder followed by a HiFi-GAN vocoder.

Overall, while the core idea originates from GAR, our work shows that the mismatch is pervasive in latent audio generation and that the remedy transfers to substantially different generators, decoders and objectives. The differences from GAR in the generator setting and training objective are discussed in the Introduction and Section 3.3.

R 1.2 How well does AudioGAR work when applied to other models besides AudioX/TangoMusic? The author acknowledges this limitation, but I believe more experiments on compatible models are needed to confirm the effectiveness/generalizability of this strategy. If this is not feasible, can the authors provide some elaboration?

Thank you for this suggestion. We have applied AudioGAR to two additional latent diffusion models, Stable Audio Open and AudioLDM-2-Large, using the same procedure and noise level (ηt=0.2\eta_t = 0.2) as in the paper, without any model-specific tuning.

DatasetModelAudioGARKL ↓IS ↑FD ↓FAD ↓PC ↑PQ ↑
AudioCapsStable Audio Open×2.0410.1230.363.412.696.08
AudioCapsStable Audio Open✓1.9810.2723.003.072.696.10
AudioCapsAudioLDM-2-Large×1.2210.1511.531.863.145.80
AudioCapsAudioLDM-2-Large✓1.2110.0211.061.443.155.80
MusicCapsStable Audio Open×1.562.9837.723.423.757.26
MusicCapsStable Audio Open✓1.533.1327.293.373.707.28
MusicCapsAudioLDM-2-Large×1.172.7018.842.515.387.23
MusicCapsAudioLDM-2-Large✓1.142.7517.182.495.427.19

AudioGAR improves FD and FAD for both models on both datasets, while the other metrics remain close to the baselines. The gains are most pronounced in FD on Stable Audio Open (24.2% on AudioCaps and 27.7% on MusicCaps) and in FAD on AudioLDM-2-Large for AudioCaps (22.6%), whereas the FAD gains on MusicCaps are smaller. Together with AudioX and TangoMusic, AudioGAR now improves all four latent diffusion models we have tested. We have added these results to Table 3.

AuthorsOpenReview ↗

Response to Reviewer vCFG (2/5)

R 1.3 How does AudioGAR compare to alternative adaptation strategy (e.g., fine-tuning the generative model, among others)?

Thank you for this question. We compare AudioGAR with continued fine-tuning of the latent generator on AudioX, using the same data and the same training budget (10.3 hours on a single H100).

DatasetStrategyKL ↓IS ↑FD ↓FAD ↓PC ↑PQ ↑
AudioCapsAudioX (baseline)1.3112.4711.831.613.165.73
AudioCapsGenerator fine-tuning1.2712.5912.322.023.165.77
AudioCapsAudioGAR(Ours)1.2912.0611.621.343.115.73
MusicCapsAudioX (baseline)1.003.659.561.604.786.61
MusicCapsGenerator fine-tuning0.913.249.041.655.316.88
MusicCapsAudioGAR(Ours)0.993.598.331.104.756.56

Under the same budget, generator fine-tuning gives mixed results: it improves KL and PQ on both datasets, but degrades FD and FAD on AudioCaps, and on MusicCaps it improves FD only modestly while degrading FAD and IS. In contrast, AudioGAR consistently improves FD and FAD on both datasets. In addition, Table 4 of the original submission compares against another decoder-side alternative: adapting the decoder on encoder latents zez_e yields smaller gains than adapting on AudioGAR latents (FAD 1.55 vs. 1.34 on AudioCaps and 1.38 vs. 1.10 on MusicCaps). Since AudioGAR leaves the generator unchanged, it could in principle be combined with generator-side improvements. We have added this comparison to Section 4.3 (Table 5) of the revised paper.

R 1.4 The experiment reports performance gains on generation but without the impacts on reconstruction. I think this would be important information given the framing of the paper.

Thank you for this suggestion. We have evaluated reconstruction before and after AudioGAR for all models. Note that AudioX and Stable Audio Open share the same codec, so their baselines are identical, whereas their adapted decoders differ because each is adapted to its own generator.

DatasetModelAudioGARrFD ↓rFAD ↓
AudioCapsAudioX×7.533.49
AudioCapsAudioX✓7.452.62
AudioCapsStable Audio Open×7.533.49
AudioCapsStable Audio Open✓8.422.68
AudioCapsAudioLDM-2-Large×3.201.31
AudioCapsAudioLDM-2-Large✓4.701.05
MusicCapsAudioX×4.310.95
MusicCapsAudioX✓4.551.23
MusicCapsStable Audio Open×4.310.95
MusicCapsStable Audio Open✓5.101.38
MusicCapsAudioLDM-2-Large×2.200.66
MusicCapsAudioLDM-2-Large✓2.470.46
MusicCapsTangoMusic×2.040.42
MusicCapsTangoMusic✓2.560.68

The impact on reconstruction is mixed. rFD increases for most models, which is expected: AudioGAR trains the decoder on generation-aware latents rather than encoder latents, moving it away from the latent distribution used in reconstruction. This is consistent with the main point of the paper: the two pathways feed the decoder different latent distributions, so improving one does not necessarily improve the other.

rFAD, however, improves in four of seven cases, including all three models on AudioCaps. One possible reason is that our adaptation data consist mainly of general-audio datasets such as AudioCaps and WavCaps (Appendix G), so fine-tuning may also benefit reconstruction in this domain. We also note that on MusicCaps, where reconstruction degrades, AudioX still obtains its largest generation improvement.

In practice, any reconstruction trade-off can be avoided. AudioGAR does not change the encoder, so the original decoder can be kept for reconstruction and the adapted decoder used for generation. We have added these results and this discussion to Section 4.3 and Appendix J (Table 10) of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer vCFG (3/5)

R 1.5 FT the decoder on z_e (the original encoder-induced latents) already results in ~20-50% the FD/FAD gains of AudioGAR (Table 4), which slightly undermines added values of AudioGAR.

Thank you for this observation. We view zez_e fine-tuning in Table 4 as a control for the adaptation data. The AudioX decoder comes from the Stable Audio Open codec, which was trained on different data from our adaptation set. Our adaptation set is instead drawn from the AudioX training data (Appendix G of the revised paper). Fine-tuning the decoder on this data can therefore help regardless of which latents are used. Since zez_e fine-tuning uses the same data and training budget as AudioGAR, its gain mainly reflects decoder fine-tuning on new data, and the added value of AudioGAR is best measured against it.

Against this control, AudioGAR further reduces FAD by 13.5% on AudioCaps and 20.3% on MusicCaps, and FD by 6.1% on MusicCaps, while the further FD gain on AudioCaps is small (0.9%). We have revised Section 4.3 to present zez_e fine-tuning as a data control. We have revised the “Effect of AudioGAR Latents” paragraph in Section 4.3 to present zez_e fine-tuning as a data control and to compare AudioGAR against it.

R 1.6 In Sec 4.1: Can the authors suggest a principled way to select the noise level? Also, can the authors elaborate on why this level of noise would be appropriate?

Thank you for this question. The choice of ηt\eta_t reflects a trade-off. A larger ηt\eta_t brings AudioGAR latents closer to generator-produced latents (Figure 4 of the original submission), so the decoder is trained on inputs closer to those it receives at generation time. However, it also removes more information about the source audio, which makes the paired reconstruction loss less reliable. At ηt=0\eta_t = 0, the decoder is still trained only on encoder latents and therefore does not adapt to the decoder train-generation mismatch. At a large ηt\eta_t such as 0.6, the latents no longer correspond well to the source audio, and performance drops (Figures 5 and 6 of the original submission). Intermediate values balance these two effects. We have noted this transfer in the Limitations (Appendix M).

In practice, the choice is not very sensitive: a range of intermediate noise levels outperforms ηt=0\eta_t = 0, with ηt=0.2\eta_t = 0.2–0.30.3 performing best. Moreover, ηt=0.2\eta_t = 0.2 transfers across models without tuning: in R 1.2, we apply it directly to Stable Audio Open and AudioLDM-2-Large and obtain consistent gains.

R 1.7 As I understand it, AudioGAR has 2 steps: adding Gaussian noise to the encoder-induced latents, and using the generative model to denoise the noisy latents. I wonder about the impact of the second step (denoising with the generative model), maybe an ablation of adapting the decoder to the noisy latents would address this.

Thank you for this suggestion. We run this ablation on AudioX, adapting the decoder directly on the noisy latents znoisytz^t_{\text{noisy}} without the generator denoising step, using the same data and training budget. Results on AudioCaps are shown below.

Training latentsNoise levelFD ↓FAD ↓
No adaptation (AudioX)–11.831.61
zez_e0.011.721.55
znoisytz^t_{\text{noisy}}0.112.972.32
znoisytz^t_{\text{noisy}}0.212.041.62
znoisytz^t_{\text{noisy}}0.416.772.51
znoisytz^t_{\text{noisy}}0.621.293.32
znoisytz^t_{\text{noisy}}0.821.514.05
znoisytz^t_{\text{noisy}}1.019.745.26
AudioGAR (zgtz^t_g)0.211.621.34

Without the denoising step, adapting on noisy latents is worse than adapting on zez_e at every noise level, and at every level even worse than no adaptation. Prior work [1] reports a similar finding for visual generation in its Appendix A.3: post-training only the decoder of MAR-VAE with latent noise worsens FID with CFG. We attribute this to the fact that adding Gaussian noise moves the latents toward an isotropic Gaussian rather than toward the generated latents. The decoder is then trained on perturbations it never receives at generation time, while the noise also removes information about the source audio and weakens the paired supervision. At the same noise level (ηt=0.2\eta_t = 0.2), AudioGAR clearly outperforms noisy-latent adaptation, which shows that its gains do not come from noise acting as data augmentation, but from denoising with the generator, which moves the latents toward Pg\mathcal{P}_g while keeping them paired with the source audio. Both steps are therefore needed. We have added this ablation to Section 4.3 and Appendix I (Table 9) of the revised paper, including the MusicCaps result at ηt=0.2\eta_t = 0.2.

[1] Latent Denoising Makes Good Visual Tokenizers. ICLR 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

Thank you to all the reviewers for their valuable feedback. We sincerely appreciate your constructive comments. We are grateful to reviewers vCFG and bXkL for highlighting the simplicity and efficiency of AudioGAR, which adapts only the decoder without changing the generator or adding inference cost, as well as the clear ablations over noise levels. We also thank reviewer vCFG for recognizing the well-motivated problem formulation, the broad empirical coverage across baselines and metrics, and the encouraging results of AudioX+AudioGAR, and reviewer Haf4 for finding our idea of improving the decoder to address the generation gap interesting. Finally, we appreciate all reviewers for recognizing the importance of the mismatch between encoder-induced and generator-produced latents in latent audio generative models.

We have revised the manuscript accordingly; changes are marked in blue. The main revisions are:

  1. Relation to GAR and prior work (vCFG, bXkL). We make the attribution to GAR explicit in the Abstract, Introduction, contribution list and Section 3.1, and explain how AudioGAR differs from GAR in the generator setting (Introduction) and the decoder training objective (Section 3.3). Section 3.1 discusses the relation to SDEdit, and Section 4.3 and Appendix I relate our noisy-latent ablation to latent denoising tokenizers.
  2. Additional models (vCFG, Haf4). We apply AudioGAR to Stable Audio Open and AudioLDM-2-Large on both AudioCaps and MusicCaps, using the same noise level (ηt=0.2\eta_t = 0.2) without model-specific tuning (Section 4.1). AudioGAR improves FD and FAD on all four models we have tested (Table 3 and Section 4.2).
  3. Comparison with alternative adaptation strategies (vCFG, bXkL, Haf4). Section 4.3 now compares AudioGAR with continued fine-tuning of the generator under the same data and budget (Table 5), presents decoder fine-tuning on encoder latents zez_e as a control for the adaptation data (Table 4), and adds a baseline that fine-tunes the decoder on noisy latents without denoising (Appendix I, Table 9).
  4. Link between latent mismatch and performance gap (vCFG). Section 3.2 adds an analysis in which only the decoder's input latent distribution changes along the AudioGAR trajectory, together with a bound relating the output-level gap to the latent mismatch (Proposition 1). Full results across five models are in Appendix E (Tables 6 and 7), and the proof is in Appendix F.
  5. Reconstruction and human evaluation (vCFG, Haf4). Section 4.3 reports the impact of AudioGAR on reconstruction (Appendix J, Table 10) and a blind pairwise listening test (Appendix K, Table 11). We have updated the Ethics Statement and the Limitations (Appendix M) accordingly. To make room for these additions, the training dynamics analysis has been moved to Appendix H.

Once again, we sincerely thank the reviewers for their thoughtful comments. We have carefully addressed each point in our individual responses to clarify the motivation, effectiveness, and generality of our proposed method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.