- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents AudioGAR, a decoder adaptation method for latent audio generative models. The paper identifies a distribution mismatch between encoder-produced latents used to train the codec decoder and generator-produced latents encountered during inference. AudioGAR constructs intermediate latents by partially noising encoder latents and denoising them using the frozen generative model. These latents preserve correspondence with the original audio while moving toward the distribution of generated latents, allowing the codec decoder to be fine-tuned with the original reconstruction objective. Experiments on AudioX and TangoMusic show improvements mainly in FD and FAD with relatively low additional training cost.
Critical strengths
The idea of improving the decoder (generation gap) is interesting.
Critical weaknesses
-
The overall improvement is relatively limited and is mainly reflected in FD and FAD. For example, on AudioCaps, FD only improves from 11.83 to 11.62. Although larger improvements can be observed in FAD and on MusicCaps, the gains are not consistent across different evaluation metrics. Some metrics even slightly degrade after fine-tuning, such as IS and PC on AudioCaps. Therefore, it remains unclear whether AudioGAR leads to an overall improvement in generated audio quality rather than primarily improving FD/FAD.
-
Human evaluation is missing. Since the reported improvements are mainly reflected by FD/FAD, objective metrics alone are insufficient to demonstrate whether the improvement is perceptually meaningful. A human listening study would provide important evidence that the improvements achieved by AudioGAR are actually noticeable in generated audio.
-
The overall significance of the method appears relatively limited. AudioGAR only fine-tunes the codec decoder to better adapt to generator-produced latents, while the underlying generative model itself remains unchanged. Therefore, the method does not directly improve the generation process or the capability of the generative model, which makes the overall impact of the proposed approach less clear.
-
The evaluation across different generative models is limited. For general audio generation, the experiments are mainly conducted on AudioX, while TangoMusic is used for music generation. Given that the proposed decoder train-generation mismatch is intended to be a general issue for latent diffusion/flow-based audio generation, experiments on more model families would strengthen the generality of the conclusions. For example, AudioLDM-based models with VAE representations and Stable Audio Open with a neural audio codec would provide useful additional test cases.
Questions for authors
- Can the authors provide human evaluation to verify the improvement in audio quality?
- Why do some metrics such as IS and PC become worse after fine-tuning?
- Does the method also work on other latent diffusion audio generation models, such as VAE from AudioLDM and Codec from Stable Audio Open ?
5 comments
Response to Reviewer Haf4(2/4)
R 3.2 Human evaluation is missing. Since the reported improvements are mainly reflected by FD/FAD, objective metrics alone are insufficient to demonstrate whether the improvement is perceptually meaningful. A human listening study would provide important evidence that the improvements achieved by AudioGAR are actually noticeable in generated audio.
Thank you for this suggestion. We agree that a listening study is important, and we have conducted a blind pairwise preference test. We generated 100 music clips with AudioX on MusicCaps. Each generated latent was decoded twice, once with the original decoder and once with the AudioGAR decoder, so the two versions differ only in the decoder. Three listeners heard each pair in random order, without knowing which decoder produced each version, and chose the version with better audio quality, or indicated no preference.
| Listener | Prefer AudioGAR | No preference | Prefer original decoder |
|---|---|---|---|
| Listener 1 | 16% | 70% | 14% |
| Listener 2 | 19% | 69% | 12% |
| Listener 3 | 13% | 71% | 16% |
| Mean ± std | 16.0 ± 3.0% | 70.0 ± 1.0% | 14.0 ± 2.0% |
Listeners reported no preference for most pairs (70.0% on average). When a preference was given, the AudioGAR version was chosen slightly more often than the original (16.0% vs. 14.0%), although this difference is not statistically significant at this scale. We draw two conclusions from this result. First, AudioGAR does not lead to a noticeable degradation in perceptual quality, while improving FD and FAD. Second, FD and FAD measure differences between distributions over the whole evaluation set, which do not necessarily appear as clearly audible differences within individual pairs. We acknowledge that a larger study would be needed to establish whether the gains in FD and FAD translate into consistently noticeable improvements. We have added this evaluation and discussion to Section 4.3 and Appendix K (Table 11) of the revised paper, and updated the Ethics Statement and Limitations (Appendix M) accordingly.
R 3.3 The overall significance of the method appears relatively limited. AudioGAR only fine-tunes the codec decoder to better adapt to generator-produced latents, while the underlying generative model itself remains unchanged. Therefore, the method does not directly improve the generation process or the capability of the generative model, which makes the overall impact of the proposed approach less clear.
Thank you for this comment. We see the fact that AudioGAR leaves the generator unchanged as a strength rather than a limitation, for three reasons.
First, the decoder is part of the generation process. Every generated sample passes through it, so the final audio quality depends on the decoder as well as on the generator. Our analysis in Section 2 shows that the decoder is trained on encoder latents but used on generated latents, and that this mismatch is accompanied by a clear reconstruction-generation gap. Most prior work improves the generator; AudioGAR addresses the other side of the same pipeline.
Second, under the same budget, adapting the decoder improves FD and FAD more than further training the generator. We continued fine-tuning the AudioX generator with the same data and the same training budget as AudioGAR (10.3 hours on a single H100):
| Dataset | Strategy | KL ↓ | IS ↑ | FD ↓ | FAD ↓ | PC ↑ | PQ ↑ |
|---|---|---|---|---|---|---|---|
| AudioCaps | AudioX (baseline) | 1.31 | 12.47 | 11.83 | 1.61 | 3.16 | 5.73 |
| AudioCaps | Generator fine-tuning | 1.27 | 12.59 | 12.32 | 2.02 | 3.16 | 5.77 |
| AudioCaps | AudioGAR(Ours) | 1.29 | 12.06 | 11.62 | 1.34 | 3.11 | 5.73 |
| MusicCaps | AudioX (baseline) | 1.00 | 3.65 | 9.56 | 1.60 | 4.78 | 6.61 |
| MusicCaps | Generator fine-tuning | 0.91 | 3.24 | 9.04 | 1.65 | 5.31 | 6.88 |
| MusicCaps | AudioGAR(Ours) | 0.99 | 3.59 | 8.33 | 1.10 | 4.75 | 6.56 |
Generator fine-tuning gives mixed results: it improves KL and PQ, but degrades FD and FAD on AudioCaps and gives only a modest FD gain on MusicCaps. In contrast, AudioGAR consistently improves FD and FAD on both datasets. This comparison is reported in Table 5 and Section 4.3 of the revised paper.
Third, because the generator is not changed, AudioGAR can be applied directly to released models and could in principle be combined with generator-side improvements. Indeed, it improves FD and FAD on all four models we have tested without model-specific tuning (see R 3.4). For AudioX, it needs only 1.5% of the original training audio and 0.26% of the original training cost, and it adds no inference cost. We have added this discussion to Section 4.3 of the revised paper.
Response to Reviewer Haf4(3/4)
R 3.4 The evaluation across different generative models is limited. For general audio generation, the experiments are mainly conducted on AudioX, while TangoMusic is used for music generation. Given that the proposed decoder train-generation mismatch is intended to be a general issue for latent diffusion/flow-based audio generation, experiments on more model families would strengthen the generality of the conclusions. For example, AudioLDM-based models with VAE representations and Stable Audio Open with a neural audio codec would provide useful additional test cases.
Thank you for this suggestion. Following it, we have applied AudioGAR to Stable Audio Open and AudioLDM-2-Large on both general audio (AudioCaps) and music (MusicCaps), using the same procedure and noise level () as in the paper, without any model-specific tuning.
| Dataset | Model | AudioGAR | KL ↓ | IS ↑ | FD ↓ | FAD ↓ | PC ↑ | PQ ↑ |
|---|---|---|---|---|---|---|---|---|
| AudioCaps | Stable Audio Open | × | 2.04 | 10.12 | 30.36 | 3.41 | 2.69 | 6.08 |
| AudioCaps | Stable Audio Open | ✓ | 1.98 | 10.27 | 23.00 | 3.07 | 2.69 | 6.10 |
| AudioCaps | AudioLDM-2-Large | × | 1.22 | 10.15 | 11.53 | 1.86 | 3.14 | 5.80 |
| AudioCaps | AudioLDM-2-Large | ✓ | 1.21 | 10.02 | 11.06 | 1.44 | 3.15 | 5.80 |
| MusicCaps | Stable Audio Open | × | 1.56 | 2.98 | 37.72 | 3.42 | 3.75 | 7.26 |
| MusicCaps | Stable Audio Open | ✓ | 1.53 | 3.13 | 27.29 | 3.37 | 3.70 | 7.28 |
| MusicCaps | AudioLDM-2-Large | × | 1.17 | 2.70 | 18.84 | 2.51 | 5.38 | 7.23 |
| MusicCaps | AudioLDM-2-Large | ✓ | 1.14 | 2.75 | 17.18 | 2.49 | 5.42 | 7.19 |
AudioGAR improves FD and FAD for both models on both datasets, while the other metrics remain close to the baselines. The gains are most pronounced in FD on Stable Audio Open (24.2% on AudioCaps and 27.7% on MusicCaps) and in FAD on AudioLDM-2-Large for AudioCaps (22.6%), whereas the FAD gains on MusicCaps are smaller.
These additions extend our evaluation in two ways. Stable Audio Open shares its neural audio codec with AudioX but uses a different generator, so it tests whether AudioGAR transfers across generators with the same decoder. AudioLDM-2-Large uses a VAE-based latent representation, which was previously covered only for music generation by TangoMusic, and is now evaluated on both general audio and music. Overall, AudioGAR improves FD and FAD on all four models we have tested, covering both neural audio codec and VAE representations on both general audio and music generation. We have added these results to Table 3.
R 3.5 Can the authors provide human evaluation to verify the improvement in audio quality?
Thank you for this question. We have conducted a blind pairwise listening study on 100 pairs of AudioX music clips that differ only in the decoder, with three listeners. Listeners reported no preference for most pairs (70.0% on average); when a preference was given, the AudioGAR version was chosen slightly more often than the original (16.0% vs. 14.0%), although this difference is not statistically significant at this scale. The full setup, results and discussion are given in our response to R 3.2 and in Section 4.3 and Appendix K of the revised paper.
Response to Reviewer Haf4(1/4)
We appreciate that the reviewer found our idea of improving the decoder to address the generation gap interesting, and recognized that AudioGAR improves FD and FAD on AudioX and TangoMusic with relatively low additional training cost. Thank you for taking the time to review our paper and for your thoughtful feedback. We address your questions and concerns below.
Thank you for this comment. We agree that the FD gain of AudioX on AudioCaps is small and that IS and PC drop slightly there. We address this concern from three aspects.
First, the changes in other metrics are small. AudioGAR only changes the decoder, while the generator, and therefore the content of the generated audio, remains unchanged. KL, IS and PC mainly reflect this content, so we do not expect them to change much. On AudioX, KL slightly improves on both datasets, while IS and PC decrease by less than 4%. In comparison, FAD improves by 16.8% on AudioCaps and 31.3% on MusicCaps, and FD by 12.9% on MusicCaps.
Second, the gains are larger and broader on other models. We applied AudioGAR to two additional latent diffusion models, Stable Audio Open and AudioLDM-2-Large, using the same procedure and noise level (ηt=0.2), without any model-specific tuning (full results in R 3.4 and Table 3 of the revised paper). On Stable Audio Open, FD improves by 24.2% on AudioCaps and 27.7% on MusicCaps, and KL, IS and PQ also improve on both datasets. Across both models, the remaining metrics change only slightly and in both directions, consistent with the content being determined by the unchanged generator. We have added these results to Table 3 and Section 4.2 of the revised paper.
Third, FD and FAD are the main audio quality metrics in prior work. AudioLDM [1] states that "we choose FD as the main evaluation metric", ControlAudio [2] uses FAD as its "primary metric for audio fidelity" and treats IS as a supplementary metric, and BATON [3] uses FD "as the primary metric for assessing audio quality". As distribution-level measures of how close generated audio is to real audio, they are also the most sensitive to the acoustic fidelity that the decoder determines.
[1] Liu et al. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. ICML 2023.
[2] Jiang et al. ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling. ACL 2026.
[3] Liao et al. BATON: Aligning Text-to-Audio Model with Human Preference Feedback. IJCAI 2024.