- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper tackles latent continuity in audio generation by introducing the Interpolative Autoencoder (IAE). To make decoders resilient against out-of-distribution or biased latents from diffusion sampling, IAE identifies nearest neighbors from an online latent queue and decodes intermediate points along their interpolation paths. Because these midpoints lack ground-truth waveforms, the authors supervise them using feature targets from a frozen semantic teacher, while using standard acoustic reconstruction and teacher-anchoring on clean encodings to prevent representation drift. The approach is evaluated across speech, music, and ambient sound datasets, demonstrating improvements in both latent interpolation quality (iFAD) and downstream generation metrics (gFAD, WER) over conventional noise-perturbed and VAE baselines.
Critical strengths
-
IAE addresses an important mismatch between clean training encodings and imperfect generated latents. Its nearest-neighbor interpolation strategy extends decoder supervision between encoded samples, using semantic targets where waveform ground truth is unavailable.
-
The experiments demonstrate improvements in both autoencoder reconstruction and downstream generation across multiple audio domains. Evaluation with five feature extractors, including four not used for supervision, strengthens the evidence that the gains extend beyond the training feature space. On AudioCaps, IAE reduces FAD from 5.30 for the better noise baseline to 3.84.
Critical weaknesses
Paper writing is a big problem for this work
The writing of the abstract can be improved significantly. The first sentence starts with mentioning "DiT", which is unnecessary. The problem of "latent quality directly influences generation performance" should be a common problem across "latent-diffusion-based" models, not just DiT. The second sentence brings up the term "representation" from nowhere. It's confusing at this point whether the terms "latent" and "representation" refer to the same thing. The third sentence then mentions "representation continuity", which requires background knowledge for readers to understand this term. Reading up to here, the concept jumps from latent quality to representation semantics and then to continuity, while lacking bridges between them. The fourth sentence also requires a lot of background knowledge to understand why the sampling of DiT leads to the need for "robustly reconstructing generation-biased lantents". The only reason I can still understand the abstract is because of having sufficient knowledge in literature about latent operations, such as methods like RAE. The current writing is difficult to follow for generative modeling readers who are not already familiar with methods operating in latent space.
In the "Fréchet distance optimization" section in related work, the downside of "optimizing the output distribution of a generator rather than the latent space" is unclear. The paragraph establishes a difference in optimization targets, but does not explain why that difference is a limitation. I appreciate the inclusion of the “Fréchet distance optimization” paragraph. However, it would benefit from a clearer explanation of why the distinction between IAE and methods that directly optimize FD matters.
The baselines do not isolate the proposed contribution
The comparisons do not fully isolate the contribution of neighborhood interpolation. The noise baselines use Mel reconstruction supervision, while IAE introduces semantic supervision on both clean and interpolated outputs. Table 3 tests semantic versus Mel supervision for interpolation, but does not compare interpolation against Gaussian perturbations under identical semantic objectives. Consequently, it remains unclear how much of the improvement comes from the perturbation strategy versus the additional semantic supervision.
No human evaluation or demo
The semantic loss supports semantic preservation, but alone does not establish the claim that interpolated latents decode “without distortion.” Listening tests on decoded interpolations would strengthen that claim. A demo of the generated samples with your proposed methods versus other baselines would benefit this work, too.
Temporal Misalignment
Nearest neighbors are retrieved via simple temporal-average pooling and cosine similarity, ignoring frame-level temporal alignment and fine-grained acoustic dynamics. Furthermore, when paired sequences differ in duration, the neighbor sequence is naively resampled via linear interpolation along the time axis, which risks temporal distortions or misaligned phonetic/acoustic structures when interpolating speech and music sequences.
I would consider increasing my score if the weaknesses mentioned above are addressed, especially the writing.
Questions for authors
- Can the authors isolate interpolation from the added semantic supervision?
- Can the authors provide a demo?
3 comments
Question 4 (W4 Temporal Structure): Do temporal average pooling retrieval and sequence length processing cause frame-level misalignment and decoding distortion?
Response 4
Pooled retrieval is used to select high-level content-relevant nearest neighbors, while the interpolation branch preserves the source audio content through semantic objectives. To examine whether frame-level alignment yields additional benefits, we fixed the IAE tokenizer and nearest-neighbor pairs, comparing original interpolation with constrained DTW-aligned interpolation across 20 sample pairs each for speech, music, and environmental sound.
DTW improved the frame cosine similarity across all 60 pairs. However, alignment simultaneously reduced the perturbation RMS by approximately 6.6%. Therefore, we evaluated both identical and RMS-matched settings.
The table below reports the primary decoding metrics at , where lower values are better for all metrics.
| Metric | Original interpolation (same ) | DTW (same ) | Original interpolation (matched RMS) | DTW (matched RMS) |
|---|---|---|---|---|
| Dasheng semantic loss | 0.55308 | 0.47420 | 0.33948 | 0.47420 |
| Independent ATST cosine error | 0.11758 | 0.09988 | 0.08186 | 0.09988 |
| Log-Mel L1 | 0.90649 | 0.82700 | 0.72557 | 0.82700 |
For this diagnostic, both endpoints were trimmed to the first 10 seconds of valid audio, with an encoding length of 250 frames, avoiding latent temporal resampling. Therefore, this experiment evaluates content misalignment along a common time axis. When is fixed, DTW improves these metrics. After matching the actual perturbation magnitude, however, the original interpolation achieves lower semantic and spectral errors. This indicates that frame-level alignment improvements do not necessarily translate to decoding gains under identical perturbation magnitudes, providing empirical support for the current interpolation approach.
I thank the authors for the rebuttal and the additional experiments. The response addresses some of my concerns, and I would raise my score to 1.5 if the scale allowed it. Since the 4-point scale has no intermediate values, I keep my rating of 1, for these reasons:
- Based on the results, I think interpolation alone adds little. The new Noise + Sem-CP controls show that most of the gain comes from semantic supervision. On AudioCaps, gFAD-VGG improves from 5.30 (Table 2) to 3.93 when the semantic objective is added to the noise baseline, while interpolation adds only about 0.09 more (3.84). Other margins are similarly small, and no seed variance is reported.
- The “without distortion” claim is not supported. The demo is welcome, but there is no listening test. UTMOSv2 is an automatic predictor and covers speech only.
- The temporal-alignment concern I raised is still not resolved. It trims both clips to the same length, so it never tests resampling across different durations. The experimental scale is also small (20 pairs per domain).
We appreciate your recognition of the importance of the discrepancy between encoded and generated latent distributions. Following your writing suggestions, we have revised the Abstract, Introduction, and Related Work to clarify the conceptual transitions of terminology and the distinction from FD optimization. Additionally, we have added noise control experiments under identical semantic objectives, provided audio demos, and conducted frame-level temporal alignment diagnostics. Revisions are highlighted in blue in the revised paper, with additional diagnostic results provided below.
Question 1 (W1 Abstract & Writing & FD Optimization)
Response 1
The abstract now begins from general latent diffusion audio generation, introducing the autoencoder and latent space before clarifying that semantic structure corresponds to content organization, while continuity corresponds to reliable decoding of positions around and between encoded points. It then explains the motivation arising from the discrepancy between training latents and generated latents.
In Related Work, under the paragraph "Fréchet distance optimization," we further clarify the distinction in training stages and supervision locations. The direct FD methods discussed optimize generators based on the distributional statistics of final generated samples, where the final FD is jointly influenced by both the generator and the sampling process. In contrast, IAE directly supervises the nearest-neighbor interpolation positions of the autoencoder prior to downstream generator training. This allows the tokenizer to be trained and evaluated first, and then frozen to assess downstream benefits. This design makes local interpolation decoding capability an explicit training objective.
Question 2 (W2, Q1): Can you compare nearest-neighbor interpolation with Gaussian perturbation under identical semantic objectives?
Response 2: We have added Noise (.25/.35) + Sem-CP, which uses the same clean and perturbed output semantic objectives and weights as IAE, with no Mel reconstruction applied to the perturbed branch. We also added AE + Sem-C and Noise (.25) + Sem-P to examine the localization of semantic supervision. The complete results are presented in Table 4.
In the complete results, IAE achieves lower values across all metrics. This provides evidence for the benefits of nearest-neighbor interpolation beyond the additional semantic loss.
Question 3 (W3, Q2): Do you provide generation and interpolation demos? How does semantic supervision support content preservation?
Response 3
We have provided cross-domain listening comparisons for generation, reconstruction, and nearest-neighbor interpolation. The demo includes baseline comparisons of generation and α=0.5 interpolation outputs. https://iae26.github.io/