- Rating
- 3
- Confidence
- 2
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
AnyAudio steers a frozen audio diffusion model during sampling to control event timing, generate longer audio, and match reference attributes without task-specific training. Experiments on sound effects and music, including component ablations and joint-control evaluations, show that these controls can be combined while largely preserving generation quality.
Critical strengths
-
The paper integrates timing, duration, and acoustic attribute control into a single generation process. These controls can be activated individually or jointly, providing a practical approach to audio generation with multiple constraints.
-
The experiments include comparisons between individual and joint controls, long-form ablations using the same backbone, and cross-backbone evaluations of selected modules. These analyses provide useful evidence for the contribution of each module and the feasibility of combining them.
Critical weaknesses
-
RAC operates on the final audio mixture. When multiple events overlap, matching a single loudness or pitch trajectory does not establish that the intended source has been adjusted correctly. For example, a change in overall loudness may come from the background rather than the foreground source the user intends to modify. The paper should clarify whether RAC is intended only for mixture-level attribute control and explain how its results should be interpreted in multi-source scenes.
-
The main experiments use attribute trajectories extracted from the ground-truth recordings, where event timings and loudness or pitch changes are naturally compatible. In practice, users may not have a reference recording that matches the intended scene and may instead edit trajectories or transfer them from other recordings. It remains unclear how these trajectories should be aligned with the event intervals specified by TAC, and how the model behaves when the controls conflict. Additional experiments on environmental sounds with edited or transferred loudness and pitch trajectories would help assess whether the method can meet new attribute targets while preserving the intended sound semantics.
Questions for authors
It is unclear whether the comparison methods receive the same recaptioned descriptions as AnyAudio or only the original captions. Since recaptioning adds event information missing from the original descriptions, this choice affects the fairness of the comparison. The paper includes an ablation without recaptioning, but should also specify the text inputs used by each baseline. A small sensitivity analysis using different LLMs or alternative recaptioned descriptions would further clarify how strongly the final results depend on this preprocessing step.
4 comments
Response to Reviewer UgC8 (Part 2/2)
W2: Edited or transferred trajectories and conflicting controls
We agree that, in practice, users often edit trajectories or transfer them from other recordings rather than using ground-truth ones.
Non-ground-truth targets. In Appendix D.1 (Table 16), we evaluate RAC on synthetic intensity curves (three shapes, two orientations, and four amplitudes, crossed with 10 genres and 10 moods), which are not extracted from any recording. RAC substantially reduces the tracking error while improving CLAP.
| Method | MSE↓ | FAD↓ | CLAP↑ |
|---|---|---|---|
| Default TTA | 63.044 | 0.336 | 0.323 |
| AnyAudio | 5.880 | 0.348 | 0.353 |
Alignment with TAC. Event intervals and attribute trajectories are specified on the same 25 Hz latent frame grid, so users align them by placing the attribute changes of a trajectory within the corresponding event intervals.
Transferred trajectories. Following the reviewer's suggestion, we evaluate TAC + RAC on the AudioCondition test set with loudness and pitch trajectories randomly transferred from other clips, while keeping each clip's own caption and event intervals. Loudness and pitch MAE are computed against the target trajectory used for generation.
| Target trajectory | Eb↑ | At↑ | Loudness MAE↓ | Pitch MAE↓ | FAD↓ | KL↓ | CLAP↑ |
|---|---|---|---|---|---|---|---|
| Original | 48.62 | 75.12 | 5.22 | 137.98 | 1.78 | 1.80 | 0.427 |
| Random | 22.47 | 65.05 | 4.93 | 154.40 | 2.35 | 2.09 | 0.372 |
Conflicting controls. Because randomly transferred trajectories are generally misaligned with the event intervals, the Random setting also serves as a stress test for conflicting controls. RAC still follows the new trajectories, with loudness and pitch errors comparable to those of the original setting, whereas Eb and At drop: when a loudness target contradicts an event interval, for example near silence where TAC requires an event, attribute matching partially overrides the timing constraint, and the mismatched contours also lower the generation quality. The balance between the two controls can be adjusted through their relative step sizes, and the conflict is avoided when attribute changes are aligned with the event intervals, as in the original setting, where Appendix E (Tables 21–23) shows only mild interference between the controls.
We thank the reviewer again for the constructive feedback, and we would be happy to discuss any remaining questions.
Thank you for the clarifications and additional experiments. The matched-caption comparisons address my concern about evaluation fairness, while the clarification of mixture-level control and the transferred-trajectory experiments make RAC’s capabilities and limitations clearer. I encourage the authors to incorporate these explanations and results into the revised manuscript.
Thank you for your positive feedback and for acknowledging our responses. We will incorporate the matched-caption comparison, the clarification of mixture-level control, and the transferred-trajectory experiments into the revised manuscript.
Response to Reviewer UgC8 (Part 1/2)
We thank the reviewer for the positive assessment and for recognizing the unified design and the thorough analyses of AnyAudio. We address each question and concern below.
Main Comments:
Q1: Text inputs of the baselines and sensitivity to recaptioning
We thank the reviewer for the careful question on the event captions. The recaption step only weaves missing event names into the caption, without any timing information. The timing-controllable baselines already receive all annotated events through their own timing inputs, so recaptioning gives AnyAudio no additional event information over them. Nevertheless, for a fairer comparison, we re-run the Table 1 baselines with public checkpoints using the same recaptioned captions. AudioComposer-S takes event descriptions and timestamps as conditions, without a global caption input, so recaptioning leaves its results unchanged.
With the same recaptioned captions, all text-only baselines still obtain low Eb, since a caption alone cannot specify when each event occurs, and the Eb of T2A-Adapter remains almost unchanged. In contrast, AnyAudio already achieves strong timing control without recaptioning, so its timing gain comes from TAC; recaptioning further helps TAC by grounding each event to a contiguous token span.
Sensitivity to the recaption LLM. We further replace Gemini 2.5 Pro with Doubao-Seed-2.0-lite for recaptioning, and also list AnyAudio without recaptioning (Figure 2) for reference.
With a different LLM, AnyAudio retains its timing gain over the setting without recaptioning and even obtains a higher At, indicating that the results do not rely on a specific LLM. We will specify the text input of every baseline in the revised Table 1.
W1: Mixture-level attribute control in multi-source scenes
We thank the reviewer for this insightful point. RAC is designed as mixture-level attribute control: the extractor ϕ is applied to the decoded mixture, and the reference curves in AudioCondition are likewise extracted from the mixture recordings, following the evaluation protocol of prior work. The loudness and pitch results in Table 1 should therefore be read as how well the generated mixture follows the target trajectory, rather than as source-level control. As the reviewer notes, in multi-source scenes a matched loudness curve does not guarantee that the change comes from the intended source. Source-level control fits into the same formulation by replacing ϕ with a source-aware differentiable extractor, for example a text-queried separation model followed by the attribute extractor, so that the loss is computed on the target source only; this requires no change to the sampling procedure. We will state the mixture-level scope explicitly in Section 3.4 and discuss the source-level extension in the Limitations section.