- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes two post-training strategies to improve source-selection reliability in diffusion/flow-based audio editing. First, the authors show that the target source to be edited is largely determined during the initial denoising steps, and STEER-Emphasis therefore focuses more training on these critical early timesteps. Second, STEER-Recovery uses self-generated intermediate states to train the model to recover when the early trajectory deviates in the wrong direction.
Critical strengths
-
The authors conduct extensive analyses and intervention experiments to verify the importance of the initial diffusion steps. In particular, the experiments provide stronger evidence than simply observing the intermediate generation results.
-
The paper provides a more detailed understanding of how many initial steps are important for different tasks. These observations may also be useful for other diffusion-based generation, editing, or separation tasks.
-
The experiments are extensive across different tasks and models, which provides good empirical support for the observations.
-
The paper also provides detailed empirical studies of self-forcing-style training, including how the model can recover from errors at different stages of the generation trajectory.
Critical weaknesses
-
The overall novelty is limited. It is already well known and intuitive that the initial steps of diffusion are important for determining the overall generation result. The paper provides a more precise and systematic analysis for audio source selection, but the main observation itself is not particularly novel.
-
STEER-Emphasis is essentially an early-timestep reweighting/sampling strategy, while the Flow Matching objective itself remains unchanged. Therefore, the methodological novelty of this component is relatively limited.
-
STEER-Recovery is highly similar to existing self-forcing-style training, both conceptually and operationally. Both use model-generated intermediate states and ground-truth supervision to train the model to recover from its own trajectory errors. The authors also acknowledge this connection in the paper. The main difference appears to be the application to audio source selection and the specific formulation of the Flow Matching target, rather than a fundamentally new training mechanism.
-
Given the similarity to self-forcing, a direct comparison with an existing self-forcing-style method would be important. The current experiments make it difficult to determine whether the improvement comes from the specific STEER-Recovery design or more generally from training on self-generated states.
Questions for authors
- What is the substantive algorithmic difference between STEER-Recovery and existing self-forcing methods?
- Can the authors provide a direct comparison between STEER-Recovery and an existing self-forcing baseline under the same training setting and computational budget?
- For STEER-Emphasis, how much improvement comes from the task-specific identification of critical early timesteps compared with simply applying a generic early-timestep reweighting strategy?
10 comments
Rebuttal for Reviewer UgJo (Part2)
Why does this difference arise? This difference is closely related to the input and training characteristics of selective audio editing. Taking removal as an example, the reference audio contains both the target source to be removed and the non-target source to be preserved. Once the model makes an editing decision, it can minimize the training loss by generating silence for the selected source and reconstructing the remaining source directly from the reference audio, without necessarily reconsidering the instruction. In contrast, general generation must continue relying on the instruction to construct and refine the desired content. Consequently, early decisions in selective editing can exert a stronger and more persistent influence on subsequent denoising. In essence, early denoising in selective audio editing is a decision-making process, determining which sources to edit or preserve, whereas in general generation it primarily establishes a preliminary content framework for subsequent refinement. To our knowledge, this critical early source-selection behavior has not been explicitly identified or systematically investigated in prior work.
Beyond this finding, we identify and systematically investigate source-selection errors in audio editing, and propose a general approach that consistently improves selective editing across existing models. We also introduce a dedicated benchmark and evaluation criteria, and train a 1.3B-parameter unified audio editor with leading overall performance and strong selective editing capabilities. Together, these contributions provide a framework for training, analyzing, and evaluating audio editors, alongside a strong foundation model for future research.
[1] Balaji Y, Nah S, Huang X, et al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers[J]. arXiv preprint arXiv:2211.01324, 2022.
[2] Wang B, Vastola J J. Diffusion models generate images like painters: an analytical theory of outline first, details later[J]. arXiv preprint arXiv:2303.02490, 2023.
[3] Bahmani S, Skorokhodov I, Qian G, et al. Ac3d: Analyzing and improving 3d camera control in video diffusion transformers[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 22875-22889.
[4] Esser P, Kulal S, Blattmann A, et al. Scaling rectified flow transformers for high-resolution image synthesis[C]. ICML, 2024.
Rebuttal for Reviewer UgJo (Part3)
W2. Novelty of STEER-Emphasis and Comparison with Existing Reweighting Methods
As discussed in W1, selective audio editing presents a distinct challenge from general generation, requiring the model to correctly determine which sources to edit and which to preserve. To address this challenge, STEER-Emphasis is specifically designed to improve source-selection reliability. Through systematic analysis of early denoising trajectories and source-selection errors, we identify task-dependent critical ranges and increase noise-level sampling within these regions during post-training.
In contrast, existing reweighting methods primarily target other objectives and are less effective at improving selective audio editing performance. While several prior works have explored timestep resampling or reweighting strategies [5–9], they primarily target different objectives: improving image generation quality (SD3, P2), enhancing spatial control in image editing (TV-Edit), balancing motion and visual quality in video generation (TQD), or improving layer assignment in image decomposition (PixelART). These strategies do not explicitly target the short, critical early window in which audio editors must decide which reference sources to edit or preserve. Consequently, their sampling or weighting strategies are not necessarily aligned with the source-selection process, making them less effective when directly applied to selective audio editing.
We summarize representative methods below, including their original motivations and corresponding sampling or weighting strategies. Here, reweighting broadly refers to changing the timestep sampling distribution, the loss weights, or both.
| Method | Primary Objective | Reweighting Strategy | Compared |
|---|---|---|---|
| SD3 [5] | Improve high-resolution text-to-image generation quality | Yes | |
| TV-Edit [6] | Improve spatial control and structural consistency in image editing | First half: ; second half: ; loss weighted by | Yes |
| TQD [7] | Balance motion and visual quality in video generation | , where | No (requires video-specific motion and visual quality scores and ) |
| PixelART [8] | Improve layer assignment in image-to-layer decomposition | Yes | |
| P2 [9] | Improve perceptual quality in image generation | ; loss weighted by | Yes |
| STEER-Emphasis | Improve source-selection reliability in audio editing | Task-dependent sampling ranges identified through prefix interventions | Ours |
Rebuttal for Reviewer UgJo (Part1)
We sincerely thank the reviewer for these valuable suggestions, which have greatly strengthened our work. We have added these additional experiments and related discussion in the revised manuscript in Appendix I, J.
W1. Novelty: Early Denoising Plays Different Roles in Selective Audio Editing and General Generation
Previous findings. Prior studies on image and video diffusion have revealed a coarse-to-fine generation process: early denoising primarily establishes the global structure of generated content, including semantic content and spatial layout in images, and low-frequency camera motion in videos, while later steps progressively refine fine-grained details [1–3]. Consequently, early denoising has a stronger influence on the overall content and structure of generated samples than later denoising stages.
Our findings. In contrast, we find that early denoising plays a different role in selective audio editing: it determines how individual reference sources should be processed, rather than merely establishing a preliminary content framework. Specifically, within a very short early denoising window, the model must decide which sources to edit and which to preserve. Once this decision is made, subsequent denoising tends to follow the established source assignment, even if it is incorrect. By comparison, framework formation in general generation is a more gradual process that unfolds over a relatively longer denoising window. To further demonstrate this distinction, we conduct additional prefix intervention experiments comparing selective audio editing with general audio and image generation.
Experimental evidence. We conduct additional prefix intervention experiments in both settings: (1) blank interventions, which steer early trajectories toward silence or blank content, and (2) wrong-content interventions, which steer them toward content inconsistent with the original instruction. For both selective editing and general generation, we apply the same types of erroneous prefix interventions. In the blank intervention, we steer the non-target source in the removal task, which should be reconstructed from the reference audio, toward silence. Similarly, in general generation, we steer the intended generated audio toward silence. In wrong-content intervention, we steer the non-target source toward the replacement sound instead of reconstructing it, while in general audio and image generation, we steer the intended content toward semantically incorrect outputs. Thus, both settings receive comparable erroneous guidance (blank or wrong content steer), allowing us to examine how the same types of prefix corruption affect subsequent denoising in different tasks. We use SteerSound-Base, UNISON and SAM Audio for selective audio editing, SteerSound-Base for audio generation, and SD3[4] Medium for image generation. We measure the minimum number of early denoising steps that must be intervened upon to affect 80% of the outputs, denoted as k80. The results show a clear difference:
As shown in the table above, selective editing requires substantially shorter corrupted prefixes to produce persistent errors, even under similar types of incorrect guidance.