acceptodds
Under review as a conference paper at ICLR 2027

AnyAudio: Accurate, Versatile, and Training-Free Audio Control via Steering Generation

Abstract

Text-to-audio generation systems have been capable of generating high-fidelity samples from natural language prompts with a scalable diffusion transformer (DiT). Recent advances have further allowed fine-grained controllability, which mainly spans precise timing control, long-form audio generation, as well as acoustic control (e.g., pitch and loudness). However, existing methods usually leverage conditional learning, either from scratch or fine-tuning, or training-freely explore latent manipulation, both of which hinder them to achieve versatile audio control efficiently. In this work, we present a holistic audio control framework, AnyAudio, which precisely steers the sampling trajectory to desired target, fully exploiting the advantages of pre-trained audio DiT. Specifically, we first design Timing Attention Control in the condition space, aligning each sound event with their designated time intervals for precise event timing. Furthermore, we present Latent Spectral Decoupling in the diffusion intermediate representation space, fusing overlapping segments in the frequency domain for coherent long-form generation. Then, we develop Reference Attribute Control in waveform space, matching differentiable reference attributes for fine-grained physical control. These cross-space control signals share a unified paradigm of differentiable matching loss optimized via gradient descent, which can be integrated into a diffusion sampling trajectory in an accurate, composable, and training-free manner. Extensive experiments demonstrate that AnyAudio achieves state-of-the-art performance in control of timing, long-form generation, and acoustic characteristics, independently and jointly, without training resource. Demo Page: https://audioguidance.github.io/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio Generation, Diffusion Models, Controllable Generation, Training-Free

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UgC8OpenReview ↗
Rating
3
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

AnyAudio steers a frozen audio diffusion model during sampling to control event timing, generate longer audio, and match reference attributes without task-specific training. Experiments on sound effects and music, including component ablations and joint-control evaluations, show that these controls can be combined while largely preserving generation quality.

Critical strengths
  1. The paper integrates timing, duration, and acoustic attribute control into a single generation process. These controls can be activated individually or jointly, providing a practical approach to audio generation with multiple constraints.

  2. The experiments include comparisons between individual and joint controls, long-form ablations using the same backbone, and cross-backbone evaluations of selected modules. These analyses provide useful evidence for the contribution of each module and the feasibility of combining them.

Critical weaknesses
  1. RAC operates on the final audio mixture. When multiple events overlap, matching a single loudness or pitch trajectory does not establish that the intended source has been adjusted correctly. For example, a change in overall loudness may come from the background rather than the foreground source the user intends to modify. The paper should clarify whether RAC is intended only for mixture-level attribute control and explain how its results should be interpreted in multi-source scenes.

  2. The main experiments use attribute trajectories extracted from the ground-truth recordings, where event timings and loudness or pitch changes are naturally compatible. In practice, users may not have a reference recording that matches the intended scene and may instead edit trajectories or transfer them from other recordings. It remains unclear how these trajectories should be aligned with the event intervals specified by TAC, and how the model behaves when the controls conflict. Additional experiments on environmental sounds with edited or transferred loudness and pitch trajectories would help assess whether the method can meet new attribute targets while preserving the intended sound semantics.

Questions for authors

It is unclear whether the comparison methods receive the same recaptioned descriptions as AnyAudio or only the original captions. Since recaptioning adds event information missing from the original descriptions, this choice affects the fairness of the comparison. The paper includes an ablation without recaptioning, but should also specify the text inputs used by each baseline. A small sensitivity analysis using different LLMs or alternative recaptioned descriptions would further clarify how strongly the final results depend on this preprocessing step.

4 comments
AuthorsOpenReview ↗

Response to Reviewer UgC8 (Part 1/2)

We thank the reviewer for the positive assessment and for recognizing the unified design and the thorough analyses of AnyAudio. We address each question and concern below.

Main Comments:


Q1: Text inputs of the baselines and sensitivity to recaptioning

We thank the reviewer for the careful question on the event captions. The recaption step only weaves missing event names into the caption, without any timing information. The timing-controllable baselines already receive all annotated events through their own timing inputs, so recaptioning gives AnyAudio no additional event information over them. Nevertheless, for a fairer comparison, we re-run the Table 1 baselines with public checkpoints using the same recaptioned captions. AudioComposer-S takes event descriptions and timestamps as conditions, without a global caption input, so recaptioning leaves its results unchanged.

MethodTextEb↑At↑FAD↓KL↓CLAP↑
AudioLDMOrig.9.2149.354.142.180.378
Recap.8.0550.813.692.230.401
AudioLDM 2Orig.6.3347.673.001.770.423
Recap.7.7254.842.182.040.391
TangoOrig.7.7645.522.162.110.364
Recap.6.8247.912.152.040.382
Stable Audio OpenOrig.3.3745.213.122.870.280
Recap.9.7555.122.822.800.256
TangoFluxOrig.5.9353.972.581.860.494
Recap.5.5352.522.631.880.486
AnyAudio (base)Orig.9.6466.591.901.840.430
Recap.12.5167.761.571.750.408
AudioComposer-SOrig.43.5160.834.571.990.284
Recap.43.5160.834.571.990.284
T2A-AdapterOrig.54.3668.261.871.530.307
Recap.54.4971.632.202.070.271
AnyAudioOrig.41.4569.211.861.970.392
Recap.48.6275.121.781.800.427

With the same recaptioned captions, all text-only baselines still obtain low Eb, since a caption alone cannot specify when each event occurs, and the Eb of T2A-Adapter remains almost unchanged. In contrast, AnyAudio already achieves strong timing control without recaptioning, so its timing gain comes from TAC; recaptioning further helps TAC by grounding each event to a contiguous token span.

Sensitivity to the recaption LLM. We further replace Gemini 2.5 Pro with Doubao-Seed-2.0-lite for recaptioning, and also list AnyAudio without recaptioning (Figure 2) for reference.

Recaption LLMEb↑At↑FAD↓KL↓CLAP↑
None (original caption)41.4569.211.861.970.392
Gemini 2.5 Pro48.6275.121.781.800.427
Doubao-Seed-2.0-lite43.5476.692.021.760.415

With a different LLM, AnyAudio retains its timing gain over the setting without recaptioning and even obtains a higher At, indicating that the results do not rely on a specific LLM. We will specify the text input of every baseline in the revised Table 1.


W1: Mixture-level attribute control in multi-source scenes

We thank the reviewer for this insightful point. RAC is designed as mixture-level attribute control: the extractor ϕ\phi is applied to the decoded mixture, and the reference curves in AudioCondition are likewise extracted from the mixture recordings, following the evaluation protocol of prior work. The loudness and pitch results in Table 1 should therefore be read as how well the generated mixture follows the target trajectory, rather than as source-level control. As the reviewer notes, in multi-source scenes a matched loudness curve does not guarantee that the change comes from the intended source. Source-level control fits into the same formulation by replacing ϕ\phi with a source-aware differentiable extractor, for example a text-queried separation model followed by the attribute extractor, so that the loss is computed on the target source only; this requires no change to the sampling procedure. We will state the mixture-level scope explicitly in Section 3.4 and discuss the source-level extension in the Limitations section.

AuthorsOpenReview ↗

Response to Reviewer UgC8 (Part 2/2)

W2: Edited or transferred trajectories and conflicting controls

We agree that, in practice, users often edit trajectories or transfer them from other recordings rather than using ground-truth ones.

Non-ground-truth targets. In Appendix D.1 (Table 16), we evaluate RAC on synthetic intensity curves (three shapes, two orientations, and four amplitudes, crossed with 10 genres and 10 moods), which are not extracted from any recording. RAC substantially reduces the tracking error while improving CLAP.

MethodMSE↓FAD↓CLAP↑
Default TTA63.0440.3360.323
AnyAudio5.8800.3480.353

Alignment with TAC. Event intervals and attribute trajectories are specified on the same 25 Hz latent frame grid, so users align them by placing the attribute changes of a trajectory within the corresponding event intervals.

Transferred trajectories. Following the reviewer's suggestion, we evaluate TAC + RAC on the AudioCondition test set with loudness and pitch trajectories randomly transferred from other clips, while keeping each clip's own caption and event intervals. Loudness and pitch MAE are computed against the target trajectory used for generation.

Target trajectoryEb↑At↑Loudness MAE↓Pitch MAE↓FAD↓KL↓CLAP↑
Original48.6275.125.22137.981.781.800.427
Random22.4765.054.93154.402.352.090.372

Conflicting controls. Because randomly transferred trajectories are generally misaligned with the event intervals, the Random setting also serves as a stress test for conflicting controls. RAC still follows the new trajectories, with loudness and pitch errors comparable to those of the original setting, whereas Eb and At drop: when a loudness target contradicts an event interval, for example near silence where TAC requires an event, attribute matching partially overrides the timing constraint, and the mismatched contours also lower the generation quality. The balance between the two controls can be adjusted through their relative step sizes, and the conflict is avoided when attribute changes are aligned with the event intervals, as in the original setting, where Appendix E (Tables 21–23) shows only mild interference between the controls.


We thank the reviewer again for the constructive feedback, and we would be happy to discuss any remaining questions.

Thank you for the clarifications and additional experiments. The matched-caption comparisons address my concern about evaluation fairness, while the clarification of mixture-level control and the transferred-trajectory experiments make RAC’s capabilities and limitations clearer. I encourage the authors to incorporate these explanations and results into the revised manuscript.

Thank you for your positive feedback and for acknowledging our responses. We will incorporate the matched-caption comparison, the clarification of mixture-level control, and the transferred-trajectory experiments into the revised manuscript.

Reviewer_Epx6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents a suite to steer text-to-audio generation without further training.

The suite has three major components:

  • Localized event control: using soft cross-attention masking and text prompt promotion/suppression to allow time-wise precise specification of events.

  • Duration extrapolation: using Fourier analysis to ensure the cross-segment consistency and boundary smoothness when generating multiple segments beyond the training duration.

  • Reference attribute control: using feature extractors describing desired attributes (e.g., energy curve, pitch curve) to steer denoising towards desired values.

Experiments use a custom-trained text-to-audio backbone, evaluated on general audio event generation (some music, some sound effects, some everyday sounds).

Critical strengths
  • I think both (i) softmask cross-attention and (ii) Fourier analysis-based length extrapolation are interesting and can be highly valuable in everyday use cases. I wish deeper and more focused evaluation is done on these two fronts.
Critical weaknesses
  • Overall I feel the framing of this work is a bit too broad such that it becomes a bit difficult to capture what the core contributions are, and what are mostly from prior existing literature. For example, the "Reference attribute control" part feels to me like generic inference-time optimization as explored in DITTO (Novack et al., 2024).

  • For the actual interesting aspects like fine-grained timed events and smoothening long-form generations, the evaluation could be more in-depth. For example, how well is the proposed method isolating events within the specified time frame? Or, how much are boundary effects mitigated and cross-segment coherence improved, compared to without the Fourier analysis-based adjustments?

Questions for authors
  • (Q1) For the low/high frequency dichotomy for Fourier analysis (Eq. 6 and 7), how is the cutoff frequency determined?

  • (Q2) Would it be possible to better isolate the benefits of (i) timing attention control and (ii) latent spectral decoupling by evaluating on the same pretrained backbone with vs. without the proposed techniques? This would greatly help address my 2nd point in Critical Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Epx6 (Part 1/3)

We thank the reviewer for the constructive comments and for the interest in our attention-based timing control (TAC) and frequency-decoupled long-form generation (LSD). We address each question and concern below.

Main Comments:


Q1: How is the cutoff determined?

We have included a dedicated ablation on the cutoff choice in App. C.2, where both cutoffs are selected by a sweep under the 26 s AudioCaps setting.

In Tab. 11, we sweep the low-frequency band shared across all segments. Widening the band steadily improves cross-segment coherence (I-LPIPS, I-CLAP), whereas fidelity is best at 10 bins. We therefore share 10 bins (≤ 0.9 Hz on the 25 Hz latent time axis), which covers the energy envelope and ambience, while event-level transients are left to each segment.

Sharing binsFAD↓KL↓FD↓IS↑CLAP↑I-LPIPS↓I-CLAP↑
01.941.8622.5413.270.4810.3910.850
51.721.6221.6513.550.5100.3550.880
101.591.4820.9813.760.5310.3240.905
151.811.5523.3413.200.5220.3180.911
202.171.5823.8513.190.5100.3100.918
302.841.6021.7813.220.4930.3020.924

In Tab. 12, we sweep the cutoff within the 2 s overlap, below which bins are averaged and above which they are phase-aligned. A cutoff of 5 bins performs best on nearly all metrics.

Overlap cutoff binsFAD↓KL↓FD↓IS↑CLAP↑I-LPIPS↓I-CLAP↑
01.741.5222.4513.500.5240.3360.892
51.591.4820.9813.760.5310.3240.905
101.651.5421.3213.640.5260.3250.903
151.711.5521.8513.200.5180.3280.901
201.781.5322.4312.450.5120.3270.904
261.821.5722.8411.900.5060.3260.907

These values are reused for MusicCaps and 90 s generation without retuning. We will add an explanation of this cutoff selection to Sec. 3.3.


Q2 & W2: Same-backbone ablations of TAC and LSD

We sincerely thank the reviewer for this insightful suggestion. Such same-backbone results are already included in the paper (Sec. 4.2, Tab. 1; Sec. 4.4, Fig. 2 and Tab. 5) and its appendix (App. D.3, Tabs. 18–20; App. E, Tabs. 21–23), and we will highlight them more prominently in the revised version. We address TAC and LSD separately below.

Q2 (i) & W2-a: Same-backbone ablation of TAC

Event isolation (W2-a). We measure how well events are confined to their specified intervals with two complementary SED-based metrics. At is the clip-level macro F1, which checks whether each specified event class is present in the generated clip regardless of its timing. Eb is the event-based F1, which counts a generated event as correct only if its class matches and its onset and offset fall within a tolerance of the specified interval. An event that leaks outside its interval is counted as a false positive, and one missing from its interval as a false negative. Thus, At reflects whether the events are generated, and Eb whether they stay within their intervals.

In Sec. 4.2 (Tab. 1), we compare AnyAudio with its base model, i.e., the same backbone without any control module, on the AudioCondition test set. AnyAudio substantially improves both timing metrics and the timing MOS.

MethodEb↑At↑FAD↓KL↓CLAP↑Timing MOS↑
AnyAudio (base)9.6466.591.901.840.4301.75
AnyAudio48.6275.121.781.800.4274.12

In App. E (Tab. 21), we activate TAC alone on the same test set. TAC alone yields the timing gain with comparable quality.

SettingEb↑At↑FAD↓KL↓CLAP↑
Text only9.6466.591.901.840.430
+ TAC50.1476.231.951.920.411

In App. E (Tabs. 22–23), we evaluate on our 26 s joint-control set, which contains overlapping events and events crossing segment boundaries. Adding TAC to the same LSD baseline consistently improves Eb and At on both subsets.

SubsetSettingEb↑At↑FAD↓KL↓CLAP↑
RealText only (LSD)10.4050.872.722.230.392
Real+ TAC43.2564.372.662.290.374
SimulatedText only (LSD)8.9151.223.542.480.356
Simulated+ TAC52.5171.303.632.520.348

In Sec. 4.4 (Fig. 2), we further apply TAC to public backbones, Stable Audio Open (SAO) and TangoFlux, on the same test set, and the timing gain carries over.

BackboneSettingEb↑At↑FAD↓KL↓CLAP↑
SAOText only3.3745.213.122.870.280
SAO+ TAC40.2765.273.542.610.291
TangoFluxText only5.9353.972.581.860.494
TangoFlux+ TAC38.3561.962.661.930.452
AuthorsOpenReview ↗

Response to Reviewer Epx6 (Part 2/3)

Q2 (ii) & W2-b: Same-backbone ablation of LSD

Boundary effects and cross-segment coherence (W2-b). We measure cross-segment coherence with Intra-LPIPS (I-LPIPS) and Intra-CLAP (I-CLAP), which compute the internal similarity among 10-second sliding-window crops of each generated clip: lower I-LPIPS indicates more consistent timbre and texture, and higher I-CLAP indicates more consistent semantics across the clip. Boundary effects, i.e., phase cancellation that attenuates transients and fine textures in the overlaps, degrade fidelity, which we measure with FAD, KL, IS, and CLAP.

In Sec. 4.4 (Tab. 5), we compare LSD with MultiDiffusion (overlap averaging, i.e., without Fourier-based fusion) and Swap Forward (the overlap latent swap of SaFa) on the same backbone at 26 s, and further remove each LSD component in turn. LSD outperforms both baselines on all metrics on AudioCaps and on most metrics on MusicCaps. Removing the high-frequency alignment, which targets phase cancellation at the boundaries, mainly degrades fidelity and detail (FAD, IS) while leaving coherence nearly unchanged, whereas removing the low-frequency sharing, which targets stylistic drift, causes the largest drop in coherence (I-LPIPS, I-CLAP).

DatasetMethodFAD↓KL↓IS↑CLAP↑I-LPIPS↓I-CLAP↑
AudioCapsMultiDiffusion1.841.5611.710.5040.3920.831
Swap Forward1.701.5113.180.5230.3630.874
LSD (full)1.591.4813.760.5310.3240.905
w/o HF align1.821.5711.900.5060.3260.907
w/o LF share1.941.8613.270.4810.3910.850
MusicCapsMultiDiffusion2.421.443.620.3870.3730.845
Swap Forward1.871.383.910.3940.3600.861
LSD (full)1.811.583.770.4090.3480.867
w/o HF align2.051.463.580.3820.3500.870
w/o LF share2.181.623.740.3780.3840.843

In App. D.3 (Tab. 18), we reimplement the long-form components of FreeAudio (Contextual Latent Composition and Reference Guidance) on the same backbone at 26 s. LSD outperforms it on all metrics for both sound (AudioCaps) and music (MusicCaps).

DatasetMethodFAD↓KL↓IS↑CLAP↑I-LPIPS↓I-CLAP↑
AudioCapsFreeAudio1.811.5412.160.5100.3740.862
AudioCapsLSD1.591.4813.760.5310.3240.905
MusicCapsFreeAudio2.321.653.400.3710.3650.851
MusicCapsLSD1.811.583.770.4090.3480.867

In App. D.3 (Tabs. 19–20), we further evaluate on public backbones (Stable Audio Open and TangoFlux) for 90 s generation, changing only the overlap-fusion strategy. LSD is best on all metrics on TangoFlux and on most metrics on Stable Audio Open.

BackboneDatasetMethodFAD↓KL↓FD↓IS↑CLAP↑
TangoFluxAudioCapsSeq. Outpainting3.251.4721.5210.060.452
MultiDiffusion3.141.3920.9510.500.469
LSD2.981.3520.3711.020.480
SAOAudioCapsSeq. Outpainting4.372.5347.5510.160.252
MultiDiffusion4.282.4227.4510.090.260
LSD4.252.4425.7110.180.263
SAOMusicCapsSeq. Outpainting3.351.7424.532.820.368
MultiDiffusion3.211.6424.482.850.374
LSD3.241.5222.102.870.371
AuthorsOpenReview ↗

Response to Reviewer Epx6 (Part 3/3)

W1: Core Contributions and Distinction from DITTO

We agree that the framing should distinguish our contributions from prior work more clearly. To this end, the paper compares AnyAudio with related work in several places, in particular with DITTO on its music editing and intensity-control tasks (App. D.1) and in an efficiency analysis (App. F).

Core contributions. The main technical contributions are TAC and LSD. TAC exploits the direct token-to-time correspondence of text-audio attention, combining an event-centric loss for interval-wise activation and suppression with a temporal-centric LogSumExp loss that handles two audio-specific failure modes, overlap competition and background dominance. LSD decouples overlap fusion along the latent temporal-frequency axis: low-frequency components are shared across segments against stylistic drift, and high-frequency components are phase-aligned within overlaps against phase cancellation. Neither failure mode is explicitly targeted by the latent averaging of MultiDiffusion or the latent swap of SaFa.

RAC and DITTO. We agree that RAC builds on the established principle of inference-time optimization through differentiable feature matching, which it shares with DITTO, and we will state this explicitly in Sec. 3.4. RAC serves as the waveform-space component of the cross-space framework and differs from DITTO in three respects. Optimization: DITTO optimizes the initial noise by back-propagating through the full sampling trajectory, whereas RAC directly updates the current latent at selected denoising steps through the clean-latent estimate, avoiding full-trajectory back-propagation. Scope: DITTO targets controllable music generation with an approximately 6-second native window, whereas RAC provides time-varying loudness, pitch, and localized waveform-reference control for general audio. Composability: RAC operates jointly with TAC and LSD within the same sampling process. RAC's contribution is therefore not differentiable feature matching itself, but its stepwise waveform-space control interface that composes with TAC and LSD within a single frozen generator.


We thank the reviewer again for the constructive feedback. We hope the clarified contribution hierarchy and the same-backbone evidence address the concerns regarding framing and evaluation depth, and we would be happy to discuss any remaining questions.

Reviewer_6YsdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper aims to develop training-free controlling method to steer pretrained audio generation models for timing, long-from and reference control. The timing control refers to the event occurrence time and interval. The long-form control means steer models to generate long audios that are coherent. Reference control aims to guide models to generate desired loudness, pitch and customized waveform reference within a designated time interval.

The time attention control leverages text-audio attention of the frozen backbone by increase the attention to the corresponding active segments during the designated interval. The long-form control adopts a decouple strategy. The low-frequency bank is shared across all segments, the high-frequency band is aligned in phase and fused by spectral energy within each overlapping regions. The reference audio generation restricts the decoding using a training-free manner, by guiding decoding with differentiable attribute extractors.

Critical strengths

The proposed control methods are general, applicable to different diffusion-based text-to-audio generation models.

Comparison with previous models on the AudioCondition dataset demonstrates that the proposed AnyAudio achieve better performance on the metrics related to timing, loudness, pitch and quality.

Intensive ablation studies are conducted to verify the effectiveness of the three control strategies.

Subjective evaluations are conducted to verify the coherence of generated long-form audios, which is challenging to evaluate using objective metrics.

Critical weaknesses

The approach is limited to diffusion-based models, due to the control based on denoising processing training-free guidance.

The testing of the proposed control methods to other diffusion-based text-to-audio models may be conducted to demonstrate the generalizability.

Questions for authors

During pre-training of the audio generation model, does it need to perturb the audio-text attention weights to simulate the test-time modification added by the event-centric loss?

For event-centric loss, how to determine the active events for equation 3.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6Ysd

We thank the reviewer for the positive assessment and for recognizing the generality, performance gains, thorough ablations, and evaluation protocol of AnyAudio. We address each question and concern below.

Main Comments:


Q1: Does pretraining need to perturb the attention weights?

TAC requires no attention perturbation during pretraining. The backbone is pretrained only with the standard v-prediction objective (Appendix A.3), and TAC is applied purely at inference time. TAC never modifies the attention weights: it reads the text-audio attention map AtA^t from the frozen network, computes LTAC\mathcal{L}_{\mathrm{TAC}}, and updates the latent ztz_t along −∇ztLTAC-\nabla_{z_t}\mathcal{L}_{\mathrm{TAC}}. The attention re-routes because the latent changes, while all network weights stay fixed.


Q2: How are the active events in Eq. 3 determined?

The active events are given directly by the user-specified timing annotations E={(ei,si,oi)}\mathcal{E}=\{(e_i, s_i, o_i)\}, so they are inputs rather than quantities estimated by the model. Each interval in seconds is mapped to latent frames by the frame rate F/DF/D (25 Hz in our model); for example, "a dog barking, 2–4 s" corresponds to frames 50–100. In Eq. 3, these frames are the active frames of event eie_i, and all remaining frames are inactive; if an event occurs several times, its active frames are the union of its intervals. Each event is grounded to a token span of the caption by the recaption step (examples in Appendix B.2, Table 8), and its attention profile is averaged over that span. The same annotations also define the set of active events A(j)\mathcal{A}(j) at each frame jj in Eq. 4. In our experiments, the annotations are taken from the frame-level event timestamps of the AudioCondition test set.


W1: Applicability beyond diffusion models

We thank the reviewer for pointing out the scope of applicability. AnyAudio indeed targets iterative generators over continuous latents. TAC only requires back-propagating through the denoiser along an iterative sampling trajectory, RAC additionally back-propagates through the VAE decoder, and LSD only requires overlapping windows that are denoised in parallel. Both diffusion and flow-matching models satisfy these conditions: TangoFlux, a flow-matching MM-DiT, already benefits from both TAC and LSD (see W2 below). Most recent TTA systems, such as Stable Audio Open, TangoFlux, AudioLDM 2, and Tango, belong to this family. AnyAudio does not directly apply to discrete autoregressive token models such as AudioGen and MusicGen, and we will state this scope explicitly in the Limitations section.


W2: Generalization to other text-to-audio backbones

We agree that generalization across backbones is a key property of a training-free control method. We have evaluated TAC and LSD on two public pretrained backbones without any retraining: Stable Audio Open (SAO), a DiT diffusion model, and TangoFlux, an MM-DiT flow-matching model. RAC likewise only requires back-propagation through the denoiser and the VAE decoder, both of which are differentiable in these two backbones.

In Section 4.4 (Figure 2), we apply TAC to both backbones on the AudioCondition test set, and the timing gain carries over.

BackboneSettingEb↑At↑FAD↓KL↓CLAP↑
SAOText only3.3745.213.122.870.280
SAO+ TAC40.2765.273.542.610.291
TangoFluxText only5.9353.972.581.860.494
TangoFlux+ TAC38.3561.962.661.930.452

In Appendix D.3 (Tables 19–20), we apply LSD to both backbones for 90 s generation, changing only the overlap-fusion strategy. LSD is best on all metrics on TangoFlux and on most metrics on SAO.

BackboneDatasetMethodFAD↓KL↓FD↓IS↑CLAP↑
TangoFluxAudioCapsSeq. Outpainting3.251.4721.5210.060.452
MultiDiffusion3.141.3920.9510.500.469
LSD2.981.3520.3711.020.480
SAOAudioCapsSeq. Outpainting4.372.5347.5510.160.252
MultiDiffusion4.282.4227.4510.090.260
LSD4.252.4425.7110.180.263
SAOMusicCapsSeq. Outpainting3.351.7424.532.820.368
MultiDiffusion3.211.6424.482.850.374
LSD3.241.5222.102.870.371

We thank the reviewer again for the positive assessment and helpful questions, and we would be happy to discuss any remaining questions.

Thank the authors for addressing my questions and quote the experimental results to demonstrate the generalizability of the proposed approach across text-to-audio backbones. I encourage the authors to integrate the explanation into the paper.

Thank you for your positive feedback and for acknowledging our responses. We will integrate the clarifications on TAC's inference-time mechanism and active-event determination, the scope of applicability, and the cross-backbone results into the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.