acceptodds
Under review as a conference paper at ICLR 2027

SoundWeaver: Compositional Warm-Starting for Text-to-Audio Diffusion Serving

Abstract

Text-to-audio (T2A) diffusion models generate high-quality audio but require tens of neural function evaluations (NFEs), resulting in substantial inference latency. Existing acceleration methods primarily optimize each generation trajectory independently, even when reusable acoustic structure exists in prior examples. We introduce SoundWeaver, the first training-free framework for compositional cross-example acoustic reuse. Rather than generating from pure noise, SoundWeaver constructs a prompt-conditioned acoustic prior from segments distributed across cached audios and warm-starts diffusion from an intermediate noise level. We introduce an Acoustic Composition Graph (ACG), which uses neural-codec representations and segment-level identification to identify candidate cross-cache transitions and structured path search to jointly optimize prompt relevance and acoustic continuity. We further establish a warm-start error bound showing how prior quality and diffusion SNR jointly govern the admissible skip depth, motivating an adaptive contextual-bandit Skip Gater that selects the warm-start level for each request. Across T2A diffusion backbones, SoundWeaver achieves 1.48-3.74x generation speedup with only a 1K-clip cache while improving generation quality.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Text-to-Audio, Retrieval-Augmented Generation, Diffusion, Systems for Audio

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UwVAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SoundWeaver, a method that retrieves and stitches real or generated audio snippets as priors and uses them as intermediate results for diffusion models, so that the diffusion models don't have to generate from pure Gaussian noise, and thus need fewer diffusion steps, accelerating their inference. The authors use FLAM to retrieve audio snippets, use EnCodec to decide their "stitch-ability", use a graph to propose stitching patterns, and use CLAP to filter/select the stitching scheme. Then, the authors formulate the decision of the number of skipped steps as a conditional multi-arm bandit problem and train a policy. Putting everything together, on the AudioCaps dataset, when using stitched real audio segments as intermediate results, SoundWeaver improves both quality and speed.

Critical strengths

The paper is coherent and the proposed method seems to work as intended. The ablation studies answer some of the most important questions about how SoundWeaver works and its effectiveness.

Critical weaknesses

The method is quite complex with a lot of moving parts, and some decisions seem arbitrary. Some examples are listed below:

  • The authors use EnCodec to determine "stitch-ability", but because the baseline models (Stable Audio Open and AudioLDM) already come with their own embedding spaces, one could argue that some distance measures in their embedding spaces more naturally decide "stitch-ability". The authors also motivated their EnCodec-based decision based on the statement "perceptually similar sounds can differ substantially in sample-level phase, noise, and amplitude," but EnCodec doesn't solve this dilemma because the actual stitching still happens in the audio space rather than the token space. Moreover, it may make more sense to use a token sequence (not RVQ layers but frame-level token sequence) rather than a single token to determine the stitch-ability, because it's possible that a single token doesn't decide doesn't. While many of these conjectures may eventually prove wrong, ablation studies in these directions are missing, making the design seem arbitrary.

  • How well do the retrieved and stitched audio snippets align with the user prompt? While it makes sense to me to use FLAM for the retrieval, it doesn't guarantee prompt correspondence. If certain retrieved audio doesn't follow the user prompt, can the diffusion model still fix it? Intuitively, earlier diffusion steps lay the semantic backbone of the generation, while later steps refine acoustic details. If the semantic information deviates from user intent and earlier steps are skipped, can the model fix the error? I'm not claiming that this issue will bound to happen, but some experiments and explanations would greatly help.

Questions for authors

Given the complexity of the method, while the authors already present some good ablation study results, it may be hard to make the ablations water-tight. However, in the current state, it's hard to judge which parts of the design works, and what parts have room for improvement. That said, if the authors can open-source the code, this concern will mostly clear up. Considering that most, if not all, of the datasets and models used in SoundWeaver are open-source, this doesn't look like one of those projects with zero chance of open-source. So, my first question is, do you have plans to open-source?

Another question is how this method scales with dataset size. The authors offer some empirical analysis, showing that a larger cache size can result in lower latency when the data scale is a couple of thousands. However, do you expect the same trend to continue as the dataset keeps scaling? Or will the retrieval step become the bottleneck with a million-scale dataset? Will the path search become the bottleneck due to complexity? Will FLAM compute become an issue? Some theoretical complexity analysis will greatly clear this up.

Question three is regarding the choice of using Gemini-3.7-Flash as an LLM judge. How accurate is it? The authors cited some papers to motivate this choice, but I could not find where those papers tested Gemini-3.7-Flash's capabilities. That said, this is not a major issue because the authors also provided other objective and subjective metrics.

2 comments

Response to Reviewer UwVA (1/2)

We thank the reviewer for the constructive questions. Unless otherwise noted, the new experiments use Stable Audio Open with a real-audio cache and 500 randomly selected AudioCaps test prompts.

W1. Why use EnCodec for stitchability?

EnCodec offers two advantages: efficient, backbone-independent graph construction and smoother waveform joins.

Efficiency. EnCodec prefixes group acoustically similar segment boundaries, so we compare only pairs that share a prefix. For N boundaries and E candidate pairs, indexing and enumerating the pairs takes O(N log N + E) time, whereas exhaustive comparison in a backbone’s latent space requires O(N²) pairwise comparisons. One EnCodec graph also serves any diffusion backbone. Specifically, encoding 160 recordings on the same A100 took 4.63 s with EnCodec, 7.67 s with Stable Audio Open, and 7.22 s with AudioLDM.

Join quality. We compared SoundWeaver’s EnCodec rule with selecting within the backbone’s latent space as suggested by the reviewer. We chose the segment from another recording with the smallest Euclidean distance at the boundary for this method. We then measured how abruptly the waveform changed at each join, before diffusion. We saw EnCodec produced smaller changes in both spectrum and loudness than either backbone baseline.

Matching ruleSpectral change ↓Loudness change ↓
EnCodec (ours)1.28×4.92 dB
Nearest neighbor in Stable Audio Open latent space1.46×9.19 dB
Nearest neighbor in AudioLDM latent space1.42×8.19 dB
Random continuation1.74×10.63 dB

Spectral change measures the peak local change in the log spectrum relative to typical frame-to-frame variation. Loudness change is the difference in RMS level between the 100 ms windows on either side of a join. EnCodec produces the smoothest joins by both measures, even though the final assembly uses the original waveforms.

We also tested the suggested frame-sequence matching rule. Requiring a match across two consecutive frames reduces cross-recording segment reachability from 90% to 35% in the 1K-recording cache. Single-frame matching therefore retains substantially more compositional choices while already providing smoother joins than the tested alternatives.

Response to Reviewer UwVA (2/2)

W2. Can diffusion correct a semantically mismatched prior?

SoundWeaver addresses this by avoiding aggressive reuse when the prior is a poor match for the prompt. The Skip Gater conditions its decision on both the prompt and the prior. When alignment is weak, it selects a noisier warm start and runs more denoising steps, reducing dependence on mismatched cached audio.

Empirically, requests in the lowest quartile of prior–prompt CLAP similarity use 2.5× as many denoising steps as those in the highest quartile, while CLAP scores for the final audio generally improve over vanilla by 4–38% showing that our MAB provides a tailored response to each prompt, and while mismatched prompts result in more NFEs they are not as common - explaining why SoundWeaver generally shows positive improvement.

Q1. Open-source release.

Yes. We will release the code and MAB checkpoints upon acceptance.

Q2. Scaling with cache size.

Larger caches provide more diverse acoustic content, which can improve prior quality and reduce denoising time. As shown in Table 2 of the paper, SoundWeaver achieves substantial quality and latency improvements with caches of only 1K–5K recordings.

We precompute audio embeddings offline, and prompt scoring scales linearly with cache size. At 1K recordings, total retrieval and prior construction take 72 ms. At 10K recordings, we estimate a total of approximately 121 ms for the original prefix graph.

Graph density is the main scaling consideration. Dynamic programming costs O(n(N+E))O(n(N+E)), where nn is the path length, NN is the number of segments, and EE is the number of transitions. Bounding the number of predecessors per segment reduces this to O(nN)O(nN). Our 10K-recording benchmark illustrates the computational benefit. Path search took 2.7 ms on a comparison graph with 46 nearest predecessors per eligible segment and 47.4 ms on the original prefix graph.

Thus, bounding the number of predecessors is a promising direction for scaling to larger caches. SoundWeaver already achieves substantial acceleration with modest caches and does not require million-scale retrieval to be effective.

Q3. Accuracy of the LLM judge.

Gemini 3.7 Flash, which we used in our paper, ranks second among ten audio-language models on MuLA-Bench [2] showing its capabilities in inherently understanding audio and language. To strengthen the evaluation, we upgraded to the top-ranked Gemini 3.8 Flash, expanded the evaluation from 100 prompts to all 964 AudioCaps test prompts, and added 95% confidence intervals.

The updated judge favors SoundWeaver across all four settings, with win rates of 53–59% (Table 5). We also report confidence intervals for our human study (Table 4), which support improved overall quality for both real-audio variants.

Table 4. Human study (win rate [95% CI])

MethodOVLREL
AudioLDM2, real0.68 [0.53, 0.80]0.70 [0.56, 0.82]
AudioLDM2, synthetic0.43 [0.30, 0.58]0.61 [0.47, 0.74]
SAO, real0.70 [0.56, 0.82]0.59 [0.44, 0.72]
SAO, synthetic0.30 [0.18, 0.44]0.43 [0.30, 0.58]

Table 5. Gemini 3.8 Flash (win rate [95% CI])

MethodWin / tie / lossWin rate
AudioLDM2, real380 / 257 / 3270.54 [0.50, 0.57]
AudioLDM2, synthetic360 / 279 / 3250.53 [0.49, 0.56]
SAO, real477 / 161 / 3260.59 [0.56, 0.63]
SAO, synthetic426 / 199 / 3390.56 [0.52, 0.59]

References: [2] Yang et al., MuLA-Bench, arXiv:2609.23416, 2026.

Reviewer_n7rEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SoundWeaver is a training-free inference framework that accelerates text-to-audio (T2A) diffusion by reusing acoustic structure from cached audio clips. Standard T2A diffusion generates audio from pure Gaussian noise over tens of sampling steps, resulting in substantial latency. Prior image-diffusion systems accelerate inference by warm-starting from a single retrieved sample. The authors argue that full-clip reuse is limiting for audio, because semantic relevance is temporally localized and cached durations may not match the request. SoundWeaver therefore decomposes cached audios into segments and uses an Acoustic Composition Graph (ACG) with dynamic programming to assemble a prompt-tailored composite prior across multiple clips, balancing prompt relevance and boundary continuity. The prior is noised to an intermediate timestep chosen per request by a contextual-bandit Skip Gater, motivated by a warm-start KL bound relating prior quality and SNR to admissible skip depth. On AudioLDM 2 and Stable Audio Open, the authors report 1.48–3.74× speedups with a ~1K-clip cache, along with improved objective quality metrics.

Critical strengths
  • Domain-Tailored Compositional Formulation (ACG): Rather than reusing a single retrieved sample as in image warm-starting systems, the paper recognizes that audio semantics are temporally localized and builds an Acoustic Composition Graph (ACG) to synthesize a mosaic prior from multiple cached audio segments. Using dynamic programming with neural-codec prefix filtering of candidate transitions allows the system to balance semantic alignment and boundary continuity efficiently.
  • Consistent Latency Reductions with Minimal Overhead: The method achieves offline generation speedups of 1.48–3.74× on both AudioLDM 2 and Stable Audio Open with a modest 1K-clip cache. Furthermore, the prior-search and gating pipeline incurs only ≈0.1 s of per-request overhead (<3% and <1% of vanilla generation latency on AudioLDM 2 and Stable Audio Open, respectively).
Critical weaknesses

Narrow positioning, and the core motivation is never tested against alternatives

The paper positions SoundWeaver mainly against step-reduction methods and gives little attention to other methods for efficient T2A generation. These include:

  • Smaller or alternative architectures: MAR-style masked parallel decoding with a lightweight diffusion head, as in IMPACT [A] (building on MAR [B] and MAGNeT [C]), and compact distilled models such as Stable Audio Open Small, released with Novack et al. (2025a).
  • Quantization and pruning: [D–G].
  • Feature caching for diffusion transformers: [H–K]. These apply directly to Stable Audio Open, whereas DeepCache (Ma et al., 2024) was evaluated only on the U-Net-based AudioLDM2.
  • Token reduction: [L].
  • Reducing classifier-free guidance cost: [M–O].
  • Parallel inference beyond xDiT (Fang et al., 2024): [P–R].

The authors should justify why cross-example warm-starting is a worthwhile lever relative to these. Ideally they would compare against a smaller or distilled model at matched latency and show whether SoundWeaver composes with at least one complementary technique.

  • [A] Huang et al. IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling. ICML 2025.

  • [B] Li et al. Autoregressive Image Generation without Vector Quantization. NeurIPS 2024.

  • [C] Ziv et al. Masked Audio Generation using a Single Non-Autoregressive Transformer. ICLR 2024.

  • [D] Li et al. Q-Diffusion: Quantizing Diffusion Models. ICCV 2023.

  • [E] Shang et al. Post-training Quantization on Diffusion Models. CVPR 2023.

  • [F] Fang et al. Structural Pruning for Diffusion Models. NeurIPS 2023.

  • [G] Kim et al. BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusion. ECCV 2024.

  • [H] Selvaraju et al. FORA: Fast-Forward Caching in Diffusion Transformer Acceleration. arXiv 2024.

  • [I] Chen et al. Δ-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers. arXiv 2024.

  • [J] Ma et al. Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching. NeurIPS 2024.

  • [K] Zou et al. Accelerating Diffusion Transformers with Token-wise Feature Caching. ICLR 2025.

  • [L] Bolya & Hoffman. Token Merging for Fast Stable Diffusion. CVPR Workshops 2023.

  • [M] Meng et al. On Distillation of Guided Diffusion Models. CVPR 2023.

  • [N] Castillo et al. Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models. arXiv 2023.

  • [O] Kynkäänniemi et al. Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models. NeurIPS 2024.

  • [P] Li et al. DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models. CVPR 2024.

  • [Q] Fang et al. PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inference. arXiv 2024.

  • [R] Shih et al. Parallel Sampling of Diffusion Models. NeurIPS 2023.

I do not expect comparisons against all of these. However, the motivation that cross-example warm-starting is a worthwhile lever would be much better supported by comparing against one or two representative alternatives. For example, but not limited to:

  • (i) Stable Audio Open Small (https://huggingface.co/stabilityai/stable-audio-open-small) at matched latency, to show SoundWeaver on the full model beats simply running a smaller model.
  • (ii) A DiT feature-caching method such as ToCa [K] on Stable Audio Open, since DeepCache (Ma et al., 2024) was evaluated only on the U-Net-based AudioLDM2. Showing that SoundWeaver composes with one of these would further strengthen the claim that it is complementary rather than competing.

No few-step or one-step baselines, and recent fast T2A work is missing

Related work omits several recent fast T2A methods, e.g., AudioTurbo (https://arxiv.org/abs/2505.22106), AudioDEAR (https://arxiv.org/abs/2605.00329), FdAudio (https://arxiv.org/abs/2607.10421), and SwiftAudio (https://arxiv.org/abs/2606.31259). More importantly, no few-step or one-step baseline appears in the experiments, so it is unclear where SoundWeaver sits on the quality–latency frontier relative to training-based acceleration.

The related work cites several one-step/few-step T2A methods (ConsistencyTTA, AudioLCM, SoundCTM, FlashAudio, ARC, MeanAudio), but the paper neither compares against them nor discusses how SoundWeaver composes with them. Given that distilled models achieve far larger speedups, the authors should position SoundWeaver relative to them, e.g., by including one distilled model in Fig. 5 or by applying SoundWeaver on top of a few-step backbone.

The claim that SoundWeaver “improves quality” is not fully supported by the paper’s own subjective results

Human evaluation favors vanilla for the synthetic-cache variant on OVL (0.43 and 0.30) and on SAO REL (0.43). The human study is small: 22 volunteers with 12 prompts each, and no confidence intervals or significance tests are reported. The LLM Preference Score is a win rate against vanilla with ties discarded (Appendix A). Every SoundWeaver entry is ≤ 0.50 (AudioLDM2: 0.41/0.40; SAO: 0.50/0.42), meaning the judge prefers vanilla as often or more often.

Limited scope

The paper does not analyze behavior when the cache contains little relevant content (cold start or out-of-distribution prompts), which matters for real deployment.

Novelty

To the best of my knowledge, the framework is an engineering pipeline combining known ingredients: concatenative dynamic programming + EnCodec prefix indexing + SDEdit warm-start diffusion + contextual bandits. The theoretical contribution (Proposition 1) is elementary.

Track

The primary area of this submission is: infrastructure, software libraries, hardware, systems, etc. I believe there are better areas for this work, for example: "applications to computer vision, audio, language, and other modalities" or "generative models"

Typos and other issues

  • Notation alternates between x̂₀ and x̃₀ for the prior.
  • Section 4.3 states d ∈ {0,1,2,3}, while Fig. 7(b) shows d = 0–4.
  • “Stricter prefix constraints (d ≥ 1)” should read d > 1.

I know I brought up many questions in the review. I am willing to increase the score if they can be addressed.

Questions for authors

Can the authors justify the novelty of their work by explaining that it is not just an integration of multiple existing methods?

5 comments

Response to Reviewer n7rE (1/4)

We thank the reviewer for the highly detailed feedback and address each point below. We provide all results with Stable Audio Open [1] powering the SoundWeaver real audio variant on 500 randomly chosen prompts from the AudioCaps test set unless stated otherwise.

W1. Narrow positioning; core motivation not tested against alternatives

SoundWeaver is composable with every single listed method family (table below). We attribute this to the fact that SoundWeaver only modifies the initial prior, leaving the trajectory and the architecture unchanged, whereas the majority of other acceleration methods change one or the other [2,3,4,5,6,19,7,23]. We also highlight that SoundWeaver improves objective quality compared to vanilla generation, which other methods largely maintain or reduce. For methods that inherently run with few steps (e.g., distillation), although SoundWeaver’s efficiency benefits become smaller, it can still improve generation quality. We fundamentally see SoundWeaver as a work that pushes the quality-latency Pareto frontier, achieving 0.9–4.9× lower latency and 13–19% better quality, averaged over CLAP, FD-infinity and KL.

All numbers are reported with 50 steps, though Stable Audio Open Small uses 8 steps as it’s a distilled model. We report CLAP score, FD, KL, and latency. As FD [8] is known to be reliable only for large datasets, we report FD-infinity [17], which corrects for the dataset size. Due to time constraints, we do not report CCDS, as it requires multiple runs with different seeds, which is expensive.

For each domain we picked widely used representatives:

  • Smaller architectures / distilled models: Stable Audio Open Small [2]
  • Quantization and pruning: 4-bit NF4 quantization with bitsandbytes [3]
  • Feature caching: TeaCache [4], SmoothCache [5]
  • Token reduction: Token Merging [6]
  • Reducing CFG cost: limited-interval guidance [7]
  • Parallel inference: PipeFusion [23]

We run PipeFusion with 2x A100s as it’s a distributed method.

Table 1:

MethodCLAP (higher is better)FD-infinity (lower is better)KL (lower is better)Latency
Vanilla0.28044.541.95714.51 s
+ SoundWeaver0.33036.121.5893.43 s
Stable Audio Open Small [2]0.23548.512.1270.27 s
+ SoundWeaver0.26340.311.8100.30 s
4-bit NF4 quantization [3]0.27344.412.0084.47 s
+ SoundWeaver0.32935.631.7971.41 s
TeaCache [4]0.26940.531.8764.52 s
+ SoundWeaver0.32336.171.6441.74 s
SmoothCache [5]0.28143.551.91911.16 s
+ SoundWeaver0.32935.841.6592.28 s
Token Merging [6]0.25534.762.04713.32 s
+ SoundWeaver0.32335.861.7023.19 s
Limited-interval guidance [7]0.29443.901.93810.51 s
+ SoundWeaver0.32834.881.7652.97 s
PipeFusion [23]0.27542.822.0078.37 s
+ SoundWeaver0.32735.411.8042.34 s

W2. No few-step or one-step baselines; recent fast T2A work missing

We add few-step distilled baselines, each with and without SoundWeaver:

Table 2:

MethodCLAP (higher is better)FD-infinity (lower is better)KL (lower is better)Latency
AudioLCM 4 steps [9] (ACM Multimedia 2024)0.46622.631.4350.17 s
+ SoundWeaver0.47721.841.3070.19 s
SoundCTM 4 steps [10] (ICLR 2025)0.41527.371.4860.64 s
+ SoundWeaver0.41726.901.4680.64 s

We note that SoundWeaver is complementary to the distillation method, and it improves quality for few-step models. Additionally, Distilled models require retraining for each base model, whereas SoundWeaver is training-free and applies to any existing model, including distilled ones.

We also wish to point out that various downstream audio applications require many steps, such as inference time control methods which apply guidance scores across timesteps [11, 21, 22, 24, 25], thus making SoundWeaver’s NFE-reduction ability a useful tool. For instance, to the best of our knowledge, training-free guidance for V2A (video to audio) runs 25 to 30 steps [11, 20, 26], and guidance runs 16 steps even on a distilled text-to-audio model [10]. We show SoundWeaver composing with such an example at 30 steps. We use AudioLDM and report ImageBind similarity between the input video and generated audio and CLAP between the text prompt and audio following the paper [11]:

Table 3:

Methodaudio-video alignment (higher is better)CLAP (higher is better)Latency
Seeing and Hearing [11]0.3280.40313.0 s
+ SoundWeaver0.3300.4128.1 s

Response to Reviewer n7rE (2/4)

W3. "Improves quality" not fully supported by the subjective results

We first report 95% confidence intervals for all subjective results to improve clarity. The confidence intervals of the human study are large, though they still allow us to conclude that listeners prefer SoundWeaver's real-audio variant over vanilla on overall quality for both backbones; however, the study does not allow us to conclude that they prefer the synthetic variant. Reducing these intervals, however, requires more participants, which is difficult within the rebuttal period. However, we do scale up our LLM-as-judge evaluation from 100 prompts to the full 964-prompt AudioCaps test set, while upgrading the judge from Gemini 3.7 Flash to Gemini 3.8 Flash, for which SoundWeaver improves throughout.

In combination with the fact that SoundWeaver achieves better objective metrics than vanilla generation as shown in Table 1, we believe SoundWeaver, especially the real-audio variant, indeed consistently improves quality overall.

Table 4 (Human study):

Human studyOVL (quality)REL (relevance)
AudioLDM2, real0.68 [0.53, 0.80]0.70 [0.56, 0.82]
AudioLDM2, synthetic0.43 [0.30, 0.58]0.61 [0.47, 0.74]
SAO, real0.70 [0.56, 0.82]0.59 [0.44, 0.72]
SAO, synthetic0.30 [0.18, 0.44]0.43 [0.30, 0.58]

Table 5 (LLM judge):

LLM judge (new)Win / tie / lossWin rate
AudioLDM2, real380 / 257 / 3270.54 [0.50, 0.57]
AudioLDM2, synthetic360 / 279 / 3250.53 [0.49, 0.56]
SAO, real477 / 161 / 3260.59 [0.56, 0.63]
SAO, synthetic426 / 199 / 3390.56 [0.52, 0.59]

W4/Q4. Limited scope: little relevant content in the cache

With a real-audio cache there is no cold-start problem, as the cache is built offline from public recordings of which hundreds of thousands are available [12, 18]. For the synthetic variant it can be a concern; however, in our cache size ablation (Table 2 in the paper) we see strong results matching vanilla quality while reducing latency with just 1k cached clips, which greatly diminishes the cold-start concern.

W5. Novelty

We do not see SoundWeaver's contribution as merely an integration of existing techniques.

Our key novelty is enabling compositional cross-request acoustic reuse as a new mechanism for accelerating diffusion inference. SDEdit [13] and AUDIT [27] are editing methods: the user supplies the image or audio to be modified, and the goal is to remain faithful to it. Classical concatenative synthesis [14] assembles segments as final outputs. None of these addresses the fundamental challenge of constructing a reusable, prompt-aligned diffusion prior from multiple unrelated cached audios. This requires jointly solving three integral problems: (i) identifying reusable acoustic structure across examples, (ii) composing it while preserving acoustic continuity, and (iii) determining how aggressively the resulting prior can accelerate denoising.

SoundWeaver introduces a novel solution through an acoustic composition graph, globally optimized compositional retrieval, and adaptive skip selection informed by our SNR analysis. Importantly, these design choices yield measurable benefits: our ablations demonstrate the independent value of composition and adaptive gating, while the complete system achieves 1.48–3.74× acceleration across distinct diffusion backbones with similar or improved quality. Thus, the contribution is not the individual primitives, but the new inference capability arising from a novel combination.

W6. Track

The closest prior work appears at systems venues, Nirvana [15] (NSDI 2024) and MoDM [16] (ASPLOS 2026), and we also see our work more so as a systems contribution rather than a theoretical one, which is what guided our decision. But we agree that "applications to computer vision, audio, language, and other modalities" can be another good fit and are happy to move if the AC permits.

W7. Typos

We will fix all stated typos, and any additional ones that we find.

Response to Reviewer n7rE (3/4): References [1–14]

[1] Zach Evans, Julian D. Parker, CJ Carr, Zack Zukowski, Josiah Taylor, Jordi Pons. "Stable Audio Open." ICASSP 2025.

[2] Zachary Novack, Zach Evans, Zack Zukowski, Josiah Taylor, CJ Carr, Julian Parker, Adnan Al-Sinan, Gian Marco Iodice, Julian McAuley, Taylor Berg-Kirkpatrick, Jordi Pons. "Fast Text-to-Audio Generation with Adversarial Post-Training." WASPAA 2025.

[3] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer. "QLoRA: Efficient Finetuning of Quantized LLMs." NeurIPS 2023.

[4] Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, Fang Wan. "Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model." CVPR 2025.

[5] Joseph Liu, Joshua Geddes, Ziyu Guo, Haomiao Jiang, Mahesh Kumar Nandwana. "SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformers." CVPR 2025 Workshops.

[6] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, Judy Hoffman. "Token Merging: Your ViT But Faster." ICLR 2023 (oral).

[7] Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, Jaakko Lehtinen. "Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models." NeurIPS 2024.

[8] Azalea Gui, Hannes Gamper, Sebastian Braun, Dimitra Emmanouilidou. "Adapting Frechet Audio Distance for Generative Music Evaluation." ICASSP 2024.

[9] Huadai Liu, Rongjie Huang, Yang Liu, Hengyuan Cao, Jialei Wang, Xize Cheng, Siqi Zheng, Zhou Zhao. "AudioLCM: Efficient and High-Quality Text-to-Audio Generation with Minimal Inference Steps." ACM Multimedia 2024.

[10] Koichi Saito, Dongjun Kim, Takashi Shibuya, Chieh-Hsin Lai, Zhi Zhong, Yuhta Takida, Yuki Mitsufuji. "SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation." ICLR 2025.

[11] Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, Qifeng Chen. "Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners." CVPR 2024.

[12] Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, Wenwu Wang. "WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research." IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, 2024.

[13] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, Stefano Ermon. "SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations." ICLR 2022.

[14] A. J. Hunt, A. W. Black. "Unit Selection in a Concatenative Speech Synthesis System Using a Large Speech Database." ICASSP 1996.

Reviewer_VV8hOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SoundWeaver, a training-free inference acceleration framework for text-to-audio diffusion models. Instead of starting generation from Gaussian noise, the method retrieves and composes semantically relevant audio segments from a cache, constructs an "Acoustic Composition Graph" (ACG), and uses the resulting composed audio as a warm-start prior. A contextual-bandit "Skip Gater" selects the warm-start timestep. Experiments on AudioLDM2 and Stable Audio Open report 1.5-3.7 times speedups while maintaining or improving quality.

Critical strengths
  • Well wired system paper. This paper could integrate different aspects such as segment-level retrieval,graph-based composition, and adaptive timestep selection into a single framework.
  • The gains are somewhat signficant. Reported speedups of 1.5-3.7x are substantial and achieved without retraining.
  • The argument that useful audio information is localized in time rather than entire clips is reasonable. The empirical observation in Fig.1 supports this intuition.
Critical weaknesses
  • The novelty claim for "first training-free framework for compositional cross-example acoustic reuse" should be toned down as the mechanism is similar to SDEdit inference framework for audio (e.g., AUDIT: Audio Editing by Following Instructions withLatent Diffusion Models). This SDEdit by nature could also accelerate generation. From this perspective, the diffusion component itself does not seem fundamentally novel, as the retrieved-and-composed audio prior can be viewed as an alternative initialization strategy within an existing SDEdit-like formulation. The primary distinction lies in the retrieval and composition module, which could arguably be integrated into prior SDEdit-based frameworks without requiring substantial changes to the underlying diffusion process.
  • The retrieval component is heavily inspired by classical concatenative synthesis and unit-selection approaches (e.g., Hunt & Black, 1996; Schwarz, 2007), where audio segments are retrieved and composed based on target relevance and transition consistency. While the adaptation of these ideas to diffusion model acceleration is practically interesting, the underlying retrieval and composition strategy appears largely incremental rather than conceptually new. Furthermore, although the proposed Skip Gater introduces an additional module, it is built upon the well-established contextual multi-armed bandit framework, a classical decision-making approach with extensive prior literature. Taken together with the strong similarities between the diffusion inference procedure and existing SDEdit-style frameworks, the overall methodological novelty appears somewhat limited.
  • Acoustic Composition Graph appears mostly heuristic to find relevant audio for a specific segment. The continuity for somekind of mixed sound could be dififcult to interpret from the current algorithm.
  • The theoretical findings about the bound does not impact directly method design. These findings only tell when the method does not meet the conditions but it does not impact the overall algorithm design.
  • Some baselines for retrieval tasks could be considered for comparison: top-k retrieval followed by direct concatenation, beam search without graph construction,and retrieval of multiple clips with omni-LLMs (Audio included) to rank composition.
Questions for authors
  • The claim of being "first" is difficult to verify from the algorithm perspective.
  • Can the authors provide evidence that EnCodec prefix matching correlates with "warm-start"?
  • How much of the final quality improvement comes from retrieval itself Vs.diffusion refinement?
3 comments

Response to Reviewer VV8h (1/3)

We thank the reviewer for the detailed feedback and address each point below. We provide all results with Stable Audio Open [1] powering the SoundWeaver real audio variant on 500 randomly chosen prompts from the AudioCaps test set unless stated otherwise.

W1, W2. Novelty of SoundWeaver as cross-sample acoustic reuse is similar to SDEdit frameworks and the modules seem incremental

We do not see SoundWeaver's contribution as merely an integration of existing techniques.

Our key novelty is enabling compositional cross-request acoustic reuse as a new mechanism for accelerating diffusion inference. SDEdit [2] and AUDIT [3] are editing methods: the user supplies the image or audio to be modified, and the goal is to remain faithful to it. Classical concatenative synthesis [4] assembles segments as final outputs. SoundWeaver, on the other hand, uses the composed sequence only as a warm-start prior from which the diffusion model regenerates the output, rather than as the final output. Further, none of SDEdit, AUDIT, or classical concatenative synthesis addresses the fundamental challenge of constructing a reusable, prompt-aligned diffusion prior from multiple unrelated cached audios. This requires jointly solving three integral problems: (i) identifying reusable acoustic structure across examples, (ii) composing it while preserving acoustic continuity, and (iii) determining how aggressively the resulting prior can accelerate denoising. We have added this discussion to the related work.

SoundWeaver offers a novel solution through an acoustic composition graph, globally optimized compositional retrieval, and adaptive skip selection informed by our SNR analysis. To the best of our knowledge, it is the first framework to organize a diffusion generation cache as a graph for compositional prior construction, with codec-prefix indexing and prompt-aligned path search enabling this across unrelated audios. Further, our Skip Gater introduces an efficient learning method for noisy and heterogeneous feedback, where quality scores are poorly calibrated across prompts and skip sensitivity varies by nearly two orders of magnitude, by converting feedback into within-prompt percentile ranks and weighting updates by each prompt's quality variance. This focuses learning on requests where skip depth materially affects quality. All updates are performed asynchronously.

These design choices together yield measurable benefits: our ablations demonstrate the independent value of composition and adaptive gating, while the complete system achieves 1.48–3.74× acceleration across distinct diffusion backbones with similar or improved quality. Thus, the contribution is not the individual primitives, but the new inference capability arising from a novel combination.

Response to Reviewer VV8h (2/3)

W3. ACG appears heuristic; continuity for mixed sounds

We see the ACG as an explicit optimization that jointly maximizes prompt relevance and acoustic continuity (Eq. 4) by solving for the global optimum of the target and concatenation costs established in traditional audio signal processing [4] using dynamic programming. We note that the scoring models themselves are empirical choices; however, they are clearly beneficial, as replacing the ACG with whole-clip retrieval, i.e., warm-starting from the single cached audio most similar to the prompt, degrades FD from 20.53 to 22.63 on AudioLDM2 and increases latency from 3.92 s to 4.39 s on Stable Audio Open (Table 1 in the paper).

To make continuity easier to interpret, we calculate the spectral change and the loudness change across each join in the composed priors, and compare them with the same quantities inside a single recording and with joins between random segments, across the full AudioCaps test set. Spectral change is the distance between the log-magnitude spectra of consecutive 10 ms frames at the join, normalized by its median over the clip, so that 1.0× is the typical frame-to-frame change. Loudness change is the absolute difference in RMS level between the 100 ms before and the 100 ms after the join. We report medians over 1,632 ACG joins and 3,856 random joins.

Table 1:

Position in the priorSpectral change (lower is smoother)Loudness change (lower is smoother)
Inside a single recording1.09×1.2 dB
ACG joins1.18×3.7 dB
Random joins1.78×10.7 dB

The joins selected by the ACG are close to natural audio, whereas random joins are far more abrupt. We will add this analysis to the appendix.

W4. Theoretical findings do not directly impact the method design

We agree that Proposition 1 does not dictate the algorithm; we see it as theoretical intuition for why warm-starting from a better prior allows more denoising steps to be skipped, rather than as a guarantee for a particular method. A tighter, method-specific result would require characterizing Δ(x~0)\Delta(\tilde{x}_0), which depends on the unknown target distribution and cannot be observed at inference time, which is why the Skip Gater, our contextual multi-armed bandit, relies on learned alignment scores as proxies. Nevertheless, this intuition guided our design in two ways. First, since a prior closer to the target (a lower Δ(x~0)\Delta(\tilde{x}_0)) can be warm-started with less noise and thus skip more steps, we focus on improving the prior itself by composing it across multiple cached audios rather than reusing a single retrieved sample (§3.2). Second, as the mismatch varies with SNR and Δ(x~0)\Delta(\tilde{x}_0) is unobservable, the Skip Gater selects the warm-start level per request over arms spaced in log-SNR rather than using a fixed skip level (§3.3). We will clarify this role of Proposition 1 in the revised paper.

Response to Reviewer VV8h (3/3)

W5. Comparisons with other simple retrieval baselines

We add the three suggested retrieval baselines, each replacing the ACG as the warm-start prior with all other components unchanged. Top-k concatenation directly concatenates the k most relevant segments, graph-free search searches over all segments without the graph (exactly, which upper-bounds beam search), and omni-LLM ranking lets Gemini 3.8 Flash select the segments from the top-8 retrieved clips. SoundWeaver achieves the best CLAP, KL, and latency among these baselines, with comparable FD-infinity, whereas the omni-LLM call alone takes 23.4 s per request.

Table 2:

MethodCLAP (higher is better)FD-infinity (lower is better)KL (lower is better)Latency
Vanilla0.28044.541.95714.51 s
Top-k concatenation0.29536.201.7604.36 s
Graph-free search0.30234.911.6973.73 s
Omni-LLM ranking0.27937.131.78827.97 s
SoundWeaver0.33036.121.5893.43 s

References

[1] Zach Evans, Julian D. Parker, CJ Carr, Zack Zukowski, Josiah Taylor, Jordi Pons. "Stable Audio Open." ICASSP 2025.

[2] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, Stefano Ermon. "SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations." ICLR 2022.

[3] Yuancheng Wang, Zeqian Ju, Xu Tan, Lei He, Zhizheng Wu, Jiang Bian, Sheng Zhao. "AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models." arXiv:2304.00830, 2023.

[4] A. J. Hunt, A. W. Black. "Unit Selection in a Concatenative Speech Synthesis System Using a Large Speech Database." ICASSP 1996.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Rebuttal Summary

Dear Reviewers, Area Chairs, and Program Chairs,

We sincerely thank the reviewers for their thoughtful feedback and constructive suggestions. We appreciate the recognition of SoundWeaver's domain-tailored compositional formulation (n7rE), integrated system design (VV8h), and substantial 1.48–3.74× speedups with minimal overhead (VV8h, n7rE). We are also encouraged by UwVA's acknowledgment of the coherence and effectiveness of our approach.

We carefully addressed the reviewers' concerns through extensive additional experiments, comparisons, and analyses. We summarize the major concerns and our rebuttal contributions below.

  • Methodological novelty (VV8h, n7rE): We clarified SoundWeaver's distinction from SDEdit, AUDIT, and classical concatenative synthesis. Our central contribution is compositional cross-request acoustic reuse for diffusion acceleration: constructing prompt-aligned warm-start priors from multiple unrelated cached audios, rather than editing user-provided inputs or concatenating segments as final outputs. We further explained how our ACG, globally optimized path search, and adaptive Skip Gater jointly enable this new inference capability, and how our theoretical analysis motivates these design choices.

  • Broader comparisons and composability (n7rE): We introduced seven additional acceleration baselines, spanning smaller/distilled models, quantization, feature caching, token merging, guidance optimization, and parallel inference. Our results demonstrate that SoundWeaver complements these approaches, substantially reducing latency while generally improving objective quality. For example, combining SoundWeaver with 4-bit quantization reduces latency from 4.47 s to 1.41 s, while improving CLAP from 0.273 to 0.329. We further evaluated four-step distilled models (AudioLCM and SoundCTM), demonstrating additional quality improvements even when few denoising steps remain. These results show that SoundWeaver can advance the quality–latency Pareto frontier, rather than simply reducing latency at the expense of quality. We also clarified why NFE reduction remains important for downstream applications requiring many inference steps, such as inference-time control, and demonstrated this by accelerating an inference-time video-to-audio generation method by 1.6× while maintaining quality.

  • Design validation and ablations (VV8h, UwVA): We evaluated three retrieval baselines suggested by VV8h (top-k concatenation, graph-free search, and omni-LLM ranking), with SoundWeaver achieving the best CLAP, KL, and latency across all three. Additional acoustic-continuity measurements demonstrate that ACG joins are substantially smoother than arbitrary joins. We further compared EnCodec-based matching against diffusion-backbone latent-space matching, as suggested by UwVA, demonstrating that EnCodec produces smoother acoustic transitions under both spectral and loudness metrics.

  • Generation quality and robustness (n7rE, UwVA): We strengthened subjective evaluation by reporting 95% confidence intervals and expanding LLM-as-judge evaluation from 100 to 964 prompts, yielding 53–59% preference win rates across variants. Together with the objective quality improvements, the updated evaluations demonstrate that SoundWeaver improves generation quality, particularly for the real-audio variants, which also achieves favorable human preference scores.

We believe these additional results substantially strengthen SoundWeaver's methodological justification, empirical validation, and practical significance. In particular, they provide further evidence that compositional acoustic reuse is an effective, training-free, and complementary mechanism for improving the quality-efficiency frontier of audio diffusion.

We respectfully ask the Reviewers and Area Chairs to consider the new experimental evidence and clarifications presented during the rebuttal when assessing SoundWeaver's contributions. We again thank everyone for their time and thoughtful consideration.

Best regards,
Authors of Submission 54209

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.