- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SoundWeaver, a method that retrieves and stitches real or generated audio snippets as priors and uses them as intermediate results for diffusion models, so that the diffusion models don't have to generate from pure Gaussian noise, and thus need fewer diffusion steps, accelerating their inference. The authors use FLAM to retrieve audio snippets, use EnCodec to decide their "stitch-ability", use a graph to propose stitching patterns, and use CLAP to filter/select the stitching scheme. Then, the authors formulate the decision of the number of skipped steps as a conditional multi-arm bandit problem and train a policy. Putting everything together, on the AudioCaps dataset, when using stitched real audio segments as intermediate results, SoundWeaver improves both quality and speed.
Critical strengths
The paper is coherent and the proposed method seems to work as intended. The ablation studies answer some of the most important questions about how SoundWeaver works and its effectiveness.
Critical weaknesses
The method is quite complex with a lot of moving parts, and some decisions seem arbitrary. Some examples are listed below:
-
The authors use EnCodec to determine "stitch-ability", but because the baseline models (Stable Audio Open and AudioLDM) already come with their own embedding spaces, one could argue that some distance measures in their embedding spaces more naturally decide "stitch-ability". The authors also motivated their EnCodec-based decision based on the statement "perceptually similar sounds can differ substantially in sample-level phase, noise, and amplitude," but EnCodec doesn't solve this dilemma because the actual stitching still happens in the audio space rather than the token space. Moreover, it may make more sense to use a token sequence (not RVQ layers but frame-level token sequence) rather than a single token to determine the stitch-ability, because it's possible that a single token doesn't decide doesn't. While many of these conjectures may eventually prove wrong, ablation studies in these directions are missing, making the design seem arbitrary.
-
How well do the retrieved and stitched audio snippets align with the user prompt? While it makes sense to me to use FLAM for the retrieval, it doesn't guarantee prompt correspondence. If certain retrieved audio doesn't follow the user prompt, can the diffusion model still fix it? Intuitively, earlier diffusion steps lay the semantic backbone of the generation, while later steps refine acoustic details. If the semantic information deviates from user intent and earlier steps are skipped, can the model fix the error? I'm not claiming that this issue will bound to happen, but some experiments and explanations would greatly help.
Questions for authors
Given the complexity of the method, while the authors already present some good ablation study results, it may be hard to make the ablations water-tight. However, in the current state, it's hard to judge which parts of the design works, and what parts have room for improvement. That said, if the authors can open-source the code, this concern will mostly clear up. Considering that most, if not all, of the datasets and models used in SoundWeaver are open-source, this doesn't look like one of those projects with zero chance of open-source. So, my first question is, do you have plans to open-source?
Another question is how this method scales with dataset size. The authors offer some empirical analysis, showing that a larger cache size can result in lower latency when the data scale is a couple of thousands. However, do you expect the same trend to continue as the dataset keeps scaling? Or will the retrieval step become the bottleneck with a million-scale dataset? Will the path search become the bottleneck due to complexity? Will FLAM compute become an issue? Some theoretical complexity analysis will greatly clear this up.
Question three is regarding the choice of using Gemini-3.7-Flash as an LLM judge. How accurate is it? The authors cited some papers to motivate this choice, but I could not find where those papers tested Gemini-3.7-Flash's capabilities. That said, this is not a major issue because the authors also provided other objective and subjective metrics.
2 comments
Response to Reviewer UwVA (2/2)
W2. Can diffusion correct a semantically mismatched prior?
SoundWeaver addresses this by avoiding aggressive reuse when the prior is a poor match for the prompt. The Skip Gater conditions its decision on both the prompt and the prior. When alignment is weak, it selects a noisier warm start and runs more denoising steps, reducing dependence on mismatched cached audio.
Empirically, requests in the lowest quartile of prior–prompt CLAP similarity use 2.5× as many denoising steps as those in the highest quartile, while CLAP scores for the final audio generally improve over vanilla by 4–38% showing that our MAB provides a tailored response to each prompt, and while mismatched prompts result in more NFEs they are not as common - explaining why SoundWeaver generally shows positive improvement.
Q1. Open-source release.
Yes. We will release the code and MAB checkpoints upon acceptance.
Q2. Scaling with cache size.
Larger caches provide more diverse acoustic content, which can improve prior quality and reduce denoising time. As shown in Table 2 of the paper, SoundWeaver achieves substantial quality and latency improvements with caches of only 1K–5K recordings.
We precompute audio embeddings offline, and prompt scoring scales linearly with cache size. At 1K recordings, total retrieval and prior construction take 72 ms. At 10K recordings, we estimate a total of approximately 121 ms for the original prefix graph.
Graph density is the main scaling consideration. Dynamic programming costs , where is the path length, is the number of segments, and is the number of transitions. Bounding the number of predecessors per segment reduces this to . Our 10K-recording benchmark illustrates the computational benefit. Path search took 2.7 ms on a comparison graph with 46 nearest predecessors per eligible segment and 47.4 ms on the original prefix graph.
Thus, bounding the number of predecessors is a promising direction for scaling to larger caches. SoundWeaver already achieves substantial acceleration with modest caches and does not require million-scale retrieval to be effective.
Q3. Accuracy of the LLM judge.
Gemini 3.7 Flash, which we used in our paper, ranks second among ten audio-language models on MuLA-Bench [2] showing its capabilities in inherently understanding audio and language. To strengthen the evaluation, we upgraded to the top-ranked Gemini 3.8 Flash, expanded the evaluation from 100 prompts to all 964 AudioCaps test prompts, and added 95% confidence intervals.
The updated judge favors SoundWeaver across all four settings, with win rates of 53–59% (Table 5). We also report confidence intervals for our human study (Table 4), which support improved overall quality for both real-audio variants.
Table 4. Human study (win rate [95% CI])
| Method | OVL | REL |
|---|---|---|
| AudioLDM2, real | 0.68 [0.53, 0.80] | 0.70 [0.56, 0.82] |
| AudioLDM2, synthetic | 0.43 [0.30, 0.58] | 0.61 [0.47, 0.74] |
| SAO, real | 0.70 [0.56, 0.82] | 0.59 [0.44, 0.72] |
| SAO, synthetic | 0.30 [0.18, 0.44] | 0.43 [0.30, 0.58] |
Table 5. Gemini 3.8 Flash (win rate [95% CI])
| Method | Win / tie / loss | Win rate |
|---|---|---|
| AudioLDM2, real | 380 / 257 / 327 | 0.54 [0.50, 0.57] |
| AudioLDM2, synthetic | 360 / 279 / 325 | 0.53 [0.49, 0.56] |
| SAO, real | 477 / 161 / 326 | 0.59 [0.56, 0.63] |
| SAO, synthetic | 426 / 199 / 339 | 0.56 [0.52, 0.59] |
References: [2] Yang et al., MuLA-Bench, arXiv:2609.23416, 2026.
Response to Reviewer UwVA (1/2)
We thank the reviewer for the constructive questions. Unless otherwise noted, the new experiments use Stable Audio Open with a real-audio cache and 500 randomly selected AudioCaps test prompts.
W1. Why use EnCodec for stitchability?
EnCodec offers two advantages: efficient, backbone-independent graph construction and smoother waveform joins.
Efficiency. EnCodec prefixes group acoustically similar segment boundaries, so we compare only pairs that share a prefix. For N boundaries and E candidate pairs, indexing and enumerating the pairs takes O(N log N + E) time, whereas exhaustive comparison in a backbone’s latent space requires O(N²) pairwise comparisons. One EnCodec graph also serves any diffusion backbone. Specifically, encoding 160 recordings on the same A100 took 4.63 s with EnCodec, 7.67 s with Stable Audio Open, and 7.22 s with AudioLDM.
Join quality. We compared SoundWeaver’s EnCodec rule with selecting within the backbone’s latent space as suggested by the reviewer. We chose the segment from another recording with the smallest Euclidean distance at the boundary for this method. We then measured how abruptly the waveform changed at each join, before diffusion. We saw EnCodec produced smaller changes in both spectrum and loudness than either backbone baseline.
Spectral change measures the peak local change in the log spectrum relative to typical frame-to-frame variation. Loudness change is the difference in RMS level between the 100 ms windows on either side of a join. EnCodec produces the smoothest joins by both measures, even though the final assembly uses the original waveforms.
We also tested the suggested frame-sequence matching rule. Requiring a match across two consecutive frames reduces cross-recording segment reachability from 90% to 35% in the 1K-recording cache. Single-frame matching therefore retains substantially more compositional choices while already providing smoother joins than the tested alternatives.