acceptodds
Under review as a conference paper at ICLR 2027

IAE: Interpolative Autoencoder for Improving Latent Continuity in Audio Generation

Abstract

Diffusion Transformer (DiT) based audio generation relies on a pretrained autoencoder to map audio into a latent space, where the latent quality directly influences generation performance. Recent work has significantly improved the semantic structuring of representations through distillation or downstream tasks. However, representation continuity has seen little further progress. Continuity matters because the decoder must robustly reconstruct generation-biased latent caused by imperfect training or sampling of DiT. Existing continuity methods still rely on fixed-magnitude Gaussian noise injection or early KL regularization, both of which may reduce reconstruction fidelity when decoding biased latent, thereby degrading final generation quality. In this paper, we propose the Interpolative Autoencoder (IAE), which preserves the semantic structure of the pretrained encoder while improving representation continuity through semantic reconstruction on latent-pair interpolations rather than single-point perturbation. IAE pairs each latent with its nearest neighbor in the current encoding distribution and imposes semantic reconstruction supervision at interpolation points between pairs. This design allows the noise magnitude to adapt according to the semantic structure around each sample's neighborhood, effectively filling low-density regions between neighboring samples and alleviating the problem of decoding generation-biased latent. From an optimization perspective, IAE is essentially optimizing the interpolated Fréchet Audio Distance (iFAD) indirectly. This metric has been shown in the image domain to assess the structuring and continuity of the latent space at the representation learning stage and to correlate significantly with final generation quality. Experimental results confirm this connection. Compared with existing RAE and VAE variant baselines, IAE achieves a lower iFAD while also obtaining better gFAD and WER on audio generation tasks, demonstrating that by improving representation continuity IAE can tangibly enhance generation quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio Representation Learning、Continuous Audio Tokenization、Audio Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Pq1JOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper tackles latent continuity in audio generation by introducing the Interpolative Autoencoder (IAE). To make decoders resilient against out-of-distribution or biased latents from diffusion sampling, IAE identifies nearest neighbors from an online latent queue and decodes intermediate points along their interpolation paths. Because these midpoints lack ground-truth waveforms, the authors supervise them using feature targets from a frozen semantic teacher, while using standard acoustic reconstruction and teacher-anchoring on clean encodings to prevent representation drift. The approach is evaluated across speech, music, and ambient sound datasets, demonstrating improvements in both latent interpolation quality (iFAD) and downstream generation metrics (gFAD, WER) over conventional noise-perturbed and VAE baselines.

Critical strengths
  1. IAE addresses an important mismatch between clean training encodings and imperfect generated latents. Its nearest-neighbor interpolation strategy extends decoder supervision between encoded samples, using semantic targets where waveform ground truth is unavailable.

  2. The experiments demonstrate improvements in both autoencoder reconstruction and downstream generation across multiple audio domains. Evaluation with five feature extractors, including four not used for supervision, strengthens the evidence that the gains extend beyond the training feature space. On AudioCaps, IAE reduces FAD from 5.30 for the better noise baseline to 3.84.

Critical weaknesses

Paper writing is a big problem for this work

The writing of the abstract can be improved significantly. The first sentence starts with mentioning "DiT", which is unnecessary. The problem of "latent quality directly influences generation performance" should be a common problem across "latent-diffusion-based" models, not just DiT. The second sentence brings up the term "representation" from nowhere. It's confusing at this point whether the terms "latent" and "representation" refer to the same thing. The third sentence then mentions "representation continuity", which requires background knowledge for readers to understand this term. Reading up to here, the concept jumps from latent quality to representation semantics and then to continuity, while lacking bridges between them. The fourth sentence also requires a lot of background knowledge to understand why the sampling of DiT leads to the need for "robustly reconstructing generation-biased lantents". The only reason I can still understand the abstract is because of having sufficient knowledge in literature about latent operations, such as methods like RAE. The current writing is difficult to follow for generative modeling readers who are not already familiar with methods operating in latent space.

In the "Fréchet distance optimization" section in related work, the downside of "optimizing the output distribution of a generator rather than the latent space" is unclear. The paragraph establishes a difference in optimization targets, but does not explain why that difference is a limitation. I appreciate the inclusion of the “Fréchet distance optimization” paragraph. However, it would benefit from a clearer explanation of why the distinction between IAE and methods that directly optimize FD matters.


The baselines do not isolate the proposed contribution

The comparisons do not fully isolate the contribution of neighborhood interpolation. The noise baselines use Mel reconstruction supervision, while IAE introduces semantic supervision on both clean and interpolated outputs. Table 3 tests semantic versus Mel supervision for interpolation, but does not compare interpolation against Gaussian perturbations under identical semantic objectives. Consequently, it remains unclear how much of the improvement comes from the perturbation strategy versus the additional semantic supervision.


No human evaluation or demo

The semantic loss supports semantic preservation, but alone does not establish the claim that interpolated latents decode “without distortion.” Listening tests on decoded interpolations would strengthen that claim. A demo of the generated samples with your proposed methods versus other baselines would benefit this work, too.


Temporal Misalignment

Nearest neighbors are retrieved via simple temporal-average pooling and cosine similarity, ignoring frame-level temporal alignment and fine-grained acoustic dynamics. Furthermore, when paired sequences differ in duration, the neighbor sequence is naively resampled via linear interpolation along the time axis, which risks temporal distortions or misaligned phonetic/acoustic structures when interpolating speech and music sequences.


I would consider increasing my score if the weaknesses mentioned above are addressed, especially the writing.

Questions for authors
  1. Can the authors isolate interpolation from the added semantic supervision?
  2. Can the authors provide a demo?
3 comments

We appreciate your recognition of the importance of the discrepancy between encoded and generated latent distributions. Following your writing suggestions, we have revised the Abstract, Introduction, and Related Work to clarify the conceptual transitions of terminology and the distinction from FD optimization. Additionally, we have added noise control experiments under identical semantic objectives, provided audio demos, and conducted frame-level temporal alignment diagnostics. Revisions are highlighted in blue in the revised paper, with additional diagnostic results provided below.

Question 1 (W1 Abstract & Writing & FD Optimization)

Response 1

The abstract now begins from general latent diffusion audio generation, introducing the autoencoder and latent space before clarifying that semantic structure corresponds to content organization, while continuity corresponds to reliable decoding of positions around and between encoded points. It then explains the motivation arising from the discrepancy between training latents and generated latents.

In Related Work, under the paragraph "Fréchet distance optimization," we further clarify the distinction in training stages and supervision locations. The direct FD methods discussed optimize generators based on the distributional statistics of final generated samples, where the final FD is jointly influenced by both the generator and the sampling process. In contrast, IAE directly supervises the nearest-neighbor interpolation positions of the autoencoder prior to downstream generator training. This allows the tokenizer to be trained and evaluated first, and then frozen to assess downstream benefits. This design makes local interpolation decoding capability an explicit training objective.

Question 2 (W2, Q1): Can you compare nearest-neighbor interpolation with Gaussian perturbation under identical semantic objectives?

Response 2: We have added Noise (.25/.35) + Sem-CP, which uses the same clean and perturbed output semantic objectives and weights as IAE, with no Mel reconstruction applied to the perturbed branch. We also added AE + Sem-C and Noise (.25) + Sem-P to examine the localization of semantic supervision. The complete results are presented in Table 4.

ConfigurationAudioCaps gFAD-VGGMusicCaps gFAD-VGGLibriTTS WER (clean)LibriTTS WER (other)
Noise (.25) + Sem-CP3.92812.913415.6816.84
Noise (.35) + Sem-CP4.22953.404516.1417.20
IAE3.83992.873414.6215.18

In the complete results, IAE achieves lower values across all metrics. This provides evidence for the benefits of nearest-neighbor interpolation beyond the additional semantic loss.

Question 3 (W3, Q2): Do you provide generation and interpolation demos? How does semantic supervision support content preservation?

Response 3

We have provided cross-domain listening comparisons for generation, reconstruction, and nearest-neighbor interpolation. The demo includes baseline comparisons of generation and α=0.5\alpha=0.5 interpolation outputs. https://iae26.github.io/

Question 4 (W4 Temporal Structure): Do temporal average pooling retrieval and sequence length processing cause frame-level misalignment and decoding distortion?

Response 4

Pooled retrieval is used to select high-level content-relevant nearest neighbors, while the interpolation branch preserves the source audio content through semantic objectives. To examine whether frame-level alignment yields additional benefits, we fixed the IAE tokenizer and nearest-neighbor pairs, comparing original interpolation with constrained DTW-aligned interpolation across 20 sample pairs each for speech, music, and environmental sound.

DTW improved the frame cosine similarity across all 60 pairs. However, alignment simultaneously reduced the perturbation RMS by approximately 6.6%. Therefore, we evaluated both identical α\alpha and RMS-matched settings.

The table below reports the primary decoding metrics at α=0.5\alpha=0.5, where lower values are better for all metrics.

MetricOriginal interpolation (same α\alpha)DTW (same α\alpha)Original interpolation (matched RMS)DTW (matched RMS)
Dasheng semantic loss0.553080.474200.339480.47420
Independent ATST cosine error0.117580.099880.081860.09988
Log-Mel L10.906490.827000.725570.82700

For this diagnostic, both endpoints were trimmed to the first 10 seconds of valid audio, with an encoding length of 250 frames, avoiding latent temporal resampling. Therefore, this experiment evaluates content misalignment along a common time axis. When α\alpha is fixed, DTW improves these metrics. After matching the actual perturbation magnitude, however, the original interpolation achieves lower semantic and spectral errors. This indicates that frame-level alignment improvements do not necessarily translate to decoding gains under identical perturbation magnitudes, providing empirical support for the current interpolation approach.

I thank the authors for the rebuttal and the additional experiments. The response addresses some of my concerns, and I would raise my score to 1.5 if the scale allowed it. Since the 4-point scale has no intermediate values, I keep my rating of 1, for these reasons:

  • Based on the results, I think interpolation alone adds little. The new Noise + Sem-CP controls show that most of the gain comes from semantic supervision. On AudioCaps, gFAD-VGG improves from 5.30 (Table 2) to 3.93 when the semantic objective is added to the noise baseline, while interpolation adds only about 0.09 more (3.84). Other margins are similarly small, and no seed variance is reported.
  • The “without distortion” claim is not supported. The demo is welcome, but there is no listening test. UTMOSv2 is an automatic predictor and covers speech only.
  • The temporal-alignment concern I raised is still not resolved. It trims both clips to the same length, so it never tests resampling across different durations. The experimental scale is also small (20 pairs per domain).
Reviewer_2jJLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the author propose an approach to improve latent continuity in audio generation while preserving the semantic structure of a pre-trained encoder. The experimental results exploiting objective metrics highlights the merits of the proposed approach but are not sufficient to demonstrate enhanced generation quality.

Critical strengths

The paper has some strenghts, including:

  • a novel approach for improving latent continuity in audio generation ;
  • an extended evaluation plan with improved results compared to baselines on objective metrics along with a substantial ablation study.
Critical weaknesses

The paper also suffers from some weaknesses:

  • Despite an extended evaluation plan, there are no evidences on the true perceptual quality of the audio generation: there is neither a perceptual test nor a comprehensive demo page.

  • The paper is on the overall well written although the notion of semantic is not always clear. Discussing a bit more what is this the semantic structure (and why it is semantic) would increase the readiness. Figure 1 could also include where is the semantic structure rather than referring to a teacher model (that is not discussed in the text before p4).

  • It is claimed that the proposed approach extends reliable decoding in the low density regions but it is not clear if indeed these additional samples are in low density regions. This could be further discussed.

  • It is not clear what is the complexity differences between the proposed approach and the baselines. Is there a complexity overhead ? is the evaluation conducted with models of the same size with the same amount of training data ?

  • Reproducibility: despite a number of details, the code does not seem to be distributed. This directly limits overall reproducibility.

Related work in supplementary material: The related work section is placed in the supplementary material which I think is not appropriate. The related work section is an important part of the paper since it allows the authors to well position their work in the State of the Art... and to better show what is new. This is not only additional details or evidence to suport the main text (which is what the supplementary material should be about).

Questions for authors
  • Do you paln to include some kind of peceptual evidence of the audio generation quality (perceptual test or comprehensive demo page )?
  • Do you plan to distributed the code ?
  • Can you elaborate on the complexity overhead and potential complexity differences between the proposed approach and baselines.
1 comment

We appreciate your recognition of our focus on continuity modeling, as well as the value of our cross-domain evaluations and ablation studies. In response to your comments, we have provided audio demos page, added a definition of semantic structure in the introduction, moved the related work into the main text, and included an analysis of training overhead. Revisions are highlighted in blue in the revised paper, with additional diagnostic results provided below.

Question 1 (W1, Q1): Can you provide perceptual evidence or comprehensive demos?

Response 1

We have provided a demo page featuring comparisons across speech, music, and environmental sounds: https://iae26.github.io/

Question 2 (W2): What is the specific meaning of semantic structure? How do the pretrained teachers correspond to the modules in Figure 1?

Response 2

We have revised the introduction to define semantic structure as the neighborhood geometry of the latent space, where similarity is determined by high-level pretrained features or task-relevant content. Specifically, frozen Dasheng features serve as an operational reference for this content similarity. In Figure 1, the teacher encoder utilizes encoder anchoring to preserve the pretrained structure, while the semantic encoder constrains the content of both clean and interpolated decoding outputs. These two modules originate from the same pretrained model but operate along distinct constraint paths.

Question 3 (W3): Do the newly added interpolation points lie in low-density regions?

Response 3

1. Quantitative Path Analysis (Corresponding to Figure 2)

ModelSemantic similarity (α=0\alpha=0)Semantic similarity (α=0.25\alpha=0.25)Semantic similarity (α=0.5\alpha=0.5)Clean reconstruction Log-Mel L1 ↓\downarrowClean reconstruction semantic cosine similarity ↑\uparrow
Anchored AE0.9090.8360.5750.3340.904
Anchored AE + Noise (0.25)0.9150.8810.5770.3290.912
Anchored AE + Noise (0.35)0.9150.8780.5720.3370.909
VA-VAE (128-d)0.8810.8250.5330.4130.875
IAE (Ours)0.9520.9270.7240.3100.949

At the unsupervised midpoint (α=0.5\alpha=0.5), the semantic similarity of baseline models drops sharply to 0.53–0.58, whereas IAE maintains a score of 0.724. Meanwhile, at the clean reconstruction point (α=0\alpha=0), IAE achieves the lowest Log-Mel L1 (0.310) and the highest semantic cosine similarity (0.949), demonstrating that it fills low-density regions without sacrificing standard reconstruction performance.

2. Perceptual Comparison via Demos

When sufficient constraints are lacking between encoded samples, interpolated positions are prone to audible distortion or semantic degradation. We provide audio comparisons at α=0.5\alpha=0.5 in our demos, where the midpoint decoding of baseline methods exhibits varying degrees of mechanical distortion or semantic jumps, while IAE maintains strong content intelligibility and audio quality.

Question 4 (W4, Q3): Are the model scale, training data, and computational overhead comparable?

Response 4: IAE and all Anchored AE variants share the same encoder initialization, decoder architecture, and training data. We have provided detailed data for each configuration in Appendices A.1–A.2.

ConfigurationInference parametersTraining time (ms/step)Peak reserved memory (GiB/GPU)
Anchored AE260.36M1043.3580.34
Noise (.25) + Mel260.36M1390.4586.00
Noise (.25) + Sem-CP260.36M1608.4195.45
IAE260.36M1613.07107.47

The training time of IAE increases by approximately 16.0% compared to the original Noise (.25), and differs by only about 0.3% from Noise (.25) + Sem-CP, which uses the same semantic supervision. The latter comparison more directly reflects the time overhead of the nearest-neighbor mechanism under identical semantic objectives. The primary additional VRAM usage comes from the full latent queue, which requires approximately 11.813 GiB per GPU. The queue, nearest-neighbor retrieval, frozen teacher, and discriminator are used solely during tokenizer training and are completely omitted during inference.

Question 5 (W5, Q2): Will the code be released publicly, and how will reproducibility be supported?

Response 5: Our code will be made publicly available upon acceptance of the paper, including the complete training and inference pipelines.

Reviewer_yXzzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that latent diffusion for audio needs a latent space that is not only semantically structured but also continuous, so that the decoder can decode the slightly off-manifold latents a DiT produces. It considers two existing mechanisms: fixed-RMS Gaussian noise during decoder training (RAE-style) and KL regularisation (VA-VAE). Both perturb each point without reference to its neighbours. The proposed Interpolative Autoencoder (IAE) keeps a FIFO queue of 32 768 recent latent sequences, retrieves each sample's nearest neighbour by cosine similarity of the temporally pooled latent, decodes z~=(1−α)z+αsg[z+]\tilde z = (1-\alpha) z + \alpha \mathrm{sg}[z^+] with α∼U(0,1/2]\alpha \sim U(0, 1/2] and supervises the decoded interpolant only through a feature-matching loss LinterpL_{interp} to the frozen Dasheng features of the source clip.

In experiments, IAE has the lowest rFAD and iFAD under five feature extractors. Downstream, a fixed UniFlow-Audio DiT is trained on each frozen tokenizer. IAE has best gFAD/FD/KL on AudioCaps and MusicCaps, e.g. AudioCaps FAD-VGG 3.84 vs 5.30 for the best baseline, best LibriTTS WER (14.62/15.18 vs 15.50/18.19 best baselines), and best gFAD/FD in 5 of 7 MECAT categories. Further support comes from an interpolation-path analysis on 64 pairs and ablations of the interpolation target, the α\alpha sampling, an added noise branch, the semantic supervisor, the queue encoder and the queue capacity.

Critical strengths

Well-controlled baselines and a large, consistent downstream effect. All non-VAE baselines share IAE's encoder initialisation, decoder, data, optimisation, acoustic loss and anchoring. Every DiT uses the same architecture, configuration and budget, "and only the tokenizer differs" . This is a stronger protocol than comparing against published tokenizers. Under it, IAE's gains are not marginal: AudioCaps FAD-VGG 3.84 vs 5.30 and FD 41.41 vs 47.97; MusicCaps FAD-VGG 2.87 vs 4.19; LibriTTS WER 14.62 vs 15.50 (clean) and 15.18 vs 18.19 (other); best UTMOSv2 in every speech-containing MECAT category. Also, the tokenizer ordering is stable across two independently resampled evaluation sets. For practitioners building audio DiTs on a Dasheng-like encoder, this is a useful and cheap recipe: the branch is training-only and adds no inference cost.

Critical weaknesses
  1. The claim that lower iFAD leads to better generation is asserted, not shown. The abstract states that the iFAD–generation correlation, shown in images, is confirmed here: "Experimental results confirm this connection." Section 4.2 concludes "confirming that improved representation continuity directly leads to better generation quality". The evidence is that IAE is best on both. With five tokenizers, one of which dominates everything, that is a single point of agreement. Among the four baselines, iFAD does not order generation quality. Using the mean iFAD rank over the five extractors (Table 1) against each Table 2 column for the four baselines, I find Spearman ρ\rho: 1.0 for AudioCaps FAD-VGG, 0.4–0.8 for AudioCaps AF3/Qwen/FD/KL, 0.0 for AudioCaps FAD-Dasheng, −0.2-0.2 to −0.4-0.4 for MusicCaps VGG/AF3/Dasheng, −1.0-1.0 for MusicCaps KL, and −0.8-0.8 for LibriTTS WER-clean. The clearest case is that Noise(0.25) has better iFAD than Noise(0.35) on 4/5 extractors, a sign that holds in both evaluation sets. Yet Noise(0.35) generates better on AudioCaps AF3/Qwen/Dasheng/FD/KL and on every MusicCaps column. On MECAT music categories, Noise(0.35) beats IAE (0M0 gFAD 1.67 vs 1.74, FD 15.66 vs 21.85; 0MA FD 35.88 vs 41.20). VA-VAE has the worst iFAD everywhere but better LibriTTS WER-clean than Anchored AE (15.50 vs 17.05).

  2. Novelty relative to interpolation-regularised autoencoders is not discussed. Supervising the decoder on latent interpolations so that decoded interpolants stay realistic or semantically faithful has a substantial literature: ACAI (Berthelot et al., ICLR 2019) uses a critic on decoded interpolations. GAIA (Sainburg et al., 2018), titled "generative adversarial interpolative autoencoding", trains adversarially on latent interpolations. Oring et al. (2020) shape the latent space so that interpolations stay on a locally convex manifold. Manifold Mixup (Verma et al., 2019) interpolates hidden states as a regulariser. None is cited, and the related-work section frames prior continuity work only as KL or Gaussian perturbations of "isolated encoding points". The conceptual claim that interpolation-based continuity is new relative to "single-point perturbation" needs to be repositioned.

Questions for authors
  1. What do Anchored AE + LsemL_{sem} (no interpolation branch) and Noise(0.25) with semantic instead of Mel supervision give on rFAD, iFAD and AudioCaps/LibriTTS generation? Please also state which losses are active in the Mel-target ablation rows of Tables 3(a) and D.3. These answers determine whether the gain is due to interpolation or to a Dasheng-space decoded-output loss.

  2. Why does semantic supervision at fixed α=1/2\alpha = 1/2 — the exact iFAD evaluation point — give worse iFAD than Anchored AE without any interpolation branch on VGGish, Qwen3-Omni and ATST (Table 3(b) vs Table 1)? Does a fixed α=1/4\alpha = 1/4 run (matching E[α]E[\alpha] of the default) recover the random-α\alpha result? Please also report the distribution of α∥z+−z∥F\alpha\lVert z^+ - z\rVert_F in per-sample latent-RMS units, so it can be compared with the 0.25/0.35 noise baselines.

  3. Given that Noise(0.25) has better iFAD than Noise(0.35) on 4/5 extractors in both evaluation sets, yet Noise(0.35) generates better on most AudioCaps and all MusicCaps metrics, what evidence supports "Experimental results confirm this connection" beyond IAE being best on both? Would you report rank correlations across tokenizers (ideally with additional intermediate tokenizers, e.g. different λinterp\lambda_{interp}) and at least two DiT seeds for the closest pairs?

3 comments

We appreciate your positive assessment and constructive feedback. We have added four semantic-supervision controls, extended the correlation analysis to nine tokenizers, and analyzed actual perturbation magnitudes. We have also expanded the discussion of interpolation regularization to clarify the contributions of IAE. Revisions are highlighted in blue in the revised paper, with additional diagnostic results provided below.

Question 1 (W1, Q3): Apart from IAE achieving simultaneous optimality, is there evidence supporting the link between iFAD and generation quality? How can the ranking inversion of the noise baselines be explained?

Response 1

We have added four semantic-supervision control experiments and expanded the analysis from five tokenizers to nine. To avoid directly averaging distances with different scales, we first rank models within each feature backend and then compute the average rank across the five tokenizer metrics and the downstream gFAD metrics. The correlation analysis has been added to Appendix B.2.

Downstream gFAD rankingSpearman ρ\rho with iFADpp-valueSpearman ρ\rho with rFADpp-value
AudioCaps0.86670.00450.91670.0013
MusicCaps0.85360.00540.82850.0088
Aggregate across both tasks0.91670.00130.91670.0013

Excluding IAE, the Spearman correlation coefficient between iFAD and aggregate gFAD remains 0.8810, indicating that this correlation is not solely driven by the joint optimum of IAE. rFAD also exhibits strong correlation, showing that both reconstruction capability and interpolation decoding quality are related to downstream generation.

For Noise (.25/.35), larger noise improves most AudioCaps and all MusicCaps distribution metrics under Mel supervision, whereas Noise (.25) + Sem-CP outperforms Noise (.35) + Sem-CP on most gFAD metrics and both WER splits under dual-branch semantic supervision. This may reflect the stronger acoustic constraints provided by the Mel objective, while semantic supervision provides weaker constraints on acoustic details, making content preservation harder. Thus, noise magnitude should be considered jointly with the supervision objective. This analysis has been added to Section 5.2.

Question 2 (W2): What is the contribution of IAE relative to existing interpolation regularization methods?

Response 2

We have added discussions of ACAI, GAIA, Oring et al., and Manifold Mixup to the Introduction and Related Work sections, explicitly positioning IAE as an audio representation learning method built upon interpolation regularization. IAE specifically addresses the joint requirements of adapting pretrained audio representations for latent diffusion.

Question 3 (Q1): Can you distinguish the effects of nearest-neighbor interpolation from those of additional semantic supervision? Which losses are enabled in the Mel-target ablation?

Response 3

We have added AE + Sem-C (supervising clean outputs only), Noise (.25) + Sem-P (supervising perturbed outputs only), and Noise (.25/.35) + Sem-CP (supervising both clean and perturbed outputs). The latter two Sem-CP configurations use the same clean and perturbed semantic objectives and weights as IAE, with no Mel objective applied to the perturbed outputs. The complete generation results have been added to Table 4, and the complete rFAD metrics are provided in Table C.3 of the appendix.

ConfigurationVGGish rFADVGGish iFADAudioCaps gFADMusicCaps gFADLibriTTS WER (clean/other)
AE + Sem-C0.12630.57266.45653.112119.42 / 20.91
Noise (.25) + Sem-P0.16580.59198.20264.120016.24 / 18.67
Noise (.25) + Sem-CP0.10880.58123.92812.913415.68 / 16.84
Noise (.35) + Sem-CP0.10280.57824.22953.404516.14 / 17.20
IAE0.09500.56893.83992.873414.62 / 15.18

In these control configurations, IAE achieves the lowest values across all five iFAD backends, eight generation gFAD metrics, and both WER splits. This indicates that while semantic supervision brings performance gains, selecting supervision locations based on neighborhood relationships provides additional value.

The Mel-target configuration retains clean acoustic reconstruction and encoder anchoring, uses the Mel reconstruction objective for the interpolation branch, and disables the additional semantic losses on both clean and interpolated decoding outputs. The active loss configurations for all variants have been supplemented and listed in Appendix A.2.

Question 4 (Q2): Why does training only on the midpoint perform worse than random interpolation? What are the results for fixed α=.25\alpha=.25 and actual perturbation magnitudes?

Response 4

Training exclusively on the midpoint supervises only a single fixed position, whereas random sampling covers the continuous interval between the source encoding and the midpoint. We have added a fixed α=.25\alpha=.25 experiment to further distinguish the effects of average perturbation magnitude from interval coverage.

Random sampling outperforms fixed α=.25\alpha=.25 across rFAD and iFAD on all five backends. The iFAD results for fixed α=.25\alpha=.25 have been added to Table 3(b) in blue. We conducted actual magnitude diagnostic experiments using 32,768 reference encodings and 3,000 independent queries, defining the relative magnitude as RMS⁡(Δz)/RMS⁡(z)\operatorname{RMS}(\Delta z)/\operatorname{RMS}(z).

Interpolation settingMedian relative RMSMean relative RMSP05–P95 of relative RMS
Random α∼U(0,12]\alpha\sim\mathcal{U}(0,\tfrac{1}{2}]0.28780.29050.0279–0.5784
Fixed α=0.25\alpha=0.250.29430.29020.2221–0.3396
Fixed α=0.5\alpha=0.50.58860.58040.4441–0.6792

Random sampling shares a similar average radius with fixed α=.25\alpha=.25, but provides a significantly wider coverage, which combined with the tokenizer results, supports the role of interval supervision.

Thanks for the new controls and related-work discussion. Noise(.25)+Sem-CP comes within ~2–5% of IAE on most gFAD metrics with one DiT seed, so the gain from interpolation itself is small and not yet established. rFAD correlates with gFAD as strongly as iFAD (ρ = 0.917 for both), so "Experimental results confirm this connection" is still not supported. Thus, I will keep my score.

Comments on the paper

1 comment
Area_Chair_veRROpenReview ↗

Reminder: Discussion Phase Ends November 18 (11:59 PM AoE)

Hi reviewers, a quick reminder that the discussion phase ends on November 18 at 11:59 PM AoE. Please take some time to read the author responses, engage in further discussions if you have any remaining questions or concerns, and update your reviews and scores as appropriate. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.