acceptodds
Under review as a conference paper at ICLR 2027

Steer Unified Audio Editing Toward Reliable Source Selection

Abstract

Unified audio editors support diverse operations, yet often struggle to modify the intended source while preserving non-target content. To strengthen this selective editing capability, we propose Source-selection Training via Early-trajectory Emphasis and Recovery (STEER), a two-stage post-training framework, and use it to develop SteerSound, a unified audio editing model supporting six common tasks. Concretely, intermediate-state observation and bidirectional prefix interventions show that early denoising states causally influence final source selection, while incorrect early decisions often persist through subsequent denoising. Guided by these findings, STEER-Emphasis emphasizes training on task-relevant early-trajectory states to reduce source-selection errors, while STEER-Recovery learns corrective denoising from self-generated rollout states to recover from early-trajectory deviations. We further introduce the Selective Audio Editing Benchmark, enabling large-scale automatic evaluation of target manipulation and non-target preservation. Experiments on both the benchmark and more realistic multi-source audio scenes show that STEER consistently improves selective editing reliability, reducing source-selection errors while improving overall editing performance. Further evaluations across different models and other selective-control tasks also support the board generalizability of STEER and its underlying early-trajectory training rationale. SteerSound achieves advanced performance among open-source unified editors.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio Editing, Flow Matching, Selective Audio Editing, Post-Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UgJoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes two post-training strategies to improve source-selection reliability in diffusion/flow-based audio editing. First, the authors show that the target source to be edited is largely determined during the initial denoising steps, and STEER-Emphasis therefore focuses more training on these critical early timesteps. Second, STEER-Recovery uses self-generated intermediate states to train the model to recover when the early trajectory deviates in the wrong direction.

Critical strengths
  1. The authors conduct extensive analyses and intervention experiments to verify the importance of the initial diffusion steps. In particular, the experiments provide stronger evidence than simply observing the intermediate generation results.

  2. The paper provides a more detailed understanding of how many initial steps are important for different tasks. These observations may also be useful for other diffusion-based generation, editing, or separation tasks.

  3. The experiments are extensive across different tasks and models, which provides good empirical support for the observations.

  4. The paper also provides detailed empirical studies of self-forcing-style training, including how the model can recover from errors at different stages of the generation trajectory.

Critical weaknesses
  1. The overall novelty is limited. It is already well known and intuitive that the initial steps of diffusion are important for determining the overall generation result. The paper provides a more precise and systematic analysis for audio source selection, but the main observation itself is not particularly novel.

  2. STEER-Emphasis is essentially an early-timestep reweighting/sampling strategy, while the Flow Matching objective itself remains unchanged. Therefore, the methodological novelty of this component is relatively limited.

  3. STEER-Recovery is highly similar to existing self-forcing-style training, both conceptually and operationally. Both use model-generated intermediate states and ground-truth supervision to train the model to recover from its own trajectory errors. The authors also acknowledge this connection in the paper. The main difference appears to be the application to audio source selection and the specific formulation of the Flow Matching target, rather than a fundamentally new training mechanism.

  4. Given the similarity to self-forcing, a direct comparison with an existing self-forcing-style method would be important. The current experiments make it difficult to determine whether the improvement comes from the specific STEER-Recovery design or more generally from training on self-generated states.

Questions for authors
  1. What is the substantive algorithmic difference between STEER-Recovery and existing self-forcing methods?
  2. Can the authors provide a direct comparison between STEER-Recovery and an existing self-forcing baseline under the same training setting and computational budget?
  3. For STEER-Emphasis, how much improvement comes from the task-specific identification of critical early timesteps compared with simply applying a generic early-timestep reweighting strategy?
10 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer UgJo (Part1)

We sincerely thank the reviewer for these valuable suggestions, which have greatly strengthened our work. We have added these additional experiments and related discussion in the revised manuscript in Appendix I, J.

W1. Novelty: Early Denoising Plays Different Roles in Selective Audio Editing and General Generation

Previous findings. Prior studies on image and video diffusion have revealed a coarse-to-fine generation process: early denoising primarily establishes the global structure of generated content, including semantic content and spatial layout in images, and low-frequency camera motion in videos, while later steps progressively refine fine-grained details [1–3]. Consequently, early denoising has a stronger influence on the overall content and structure of generated samples than later denoising stages.

Our findings. In contrast, we find that early denoising plays a different role in selective audio editing: it determines how individual reference sources should be processed, rather than merely establishing a preliminary content framework. Specifically, within a very short early denoising window, the model must decide which sources to edit and which to preserve. Once this decision is made, subsequent denoising tends to follow the established source assignment, even if it is incorrect. By comparison, framework formation in general generation is a more gradual process that unfolds over a relatively longer denoising window. To further demonstrate this distinction, we conduct additional prefix intervention experiments comparing selective audio editing with general audio and image generation.

Experimental evidence. We conduct additional prefix intervention experiments in both settings: (1) blank interventions, which steer early trajectories toward silence or blank content, and (2) wrong-content interventions, which steer them toward content inconsistent with the original instruction. For both selective editing and general generation, we apply the same types of erroneous prefix interventions. In the blank intervention, we steer the non-target source in the removal task, which should be reconstructed from the reference audio, toward silence. Similarly, in general generation, we steer the intended generated audio toward silence. In wrong-content intervention, we steer the non-target source toward the replacement sound instead of reconstructing it, while in general audio and image generation, we steer the intended content toward semantically incorrect outputs. Thus, both settings receive comparable erroneous guidance (blank or wrong content steer), allowing us to examine how the same types of prefix corruption affect subsequent denoising in different tasks. We use SteerSound-Base, UNISON and SAM Audio for selective audio editing, SteerSound-Base for audio generation, and SD3[4] Medium for image generation. We measure the minimum number of early denoising steps that must be intervened upon to affect 80% of the outputs, denoted as k80k_{80}. The results show a clear difference:

InterventionSelective Audio Editing (k₈₀ / 50)General Generation (k₈₀ / 50)
Blank / SilenceRemove: 2–4Audio: 8; Image: 7
Wrong ContentReplace: 11–13Audio: 30; Image: 35

As shown in the table above, selective editing requires substantially shorter corrupted prefixes to produce persistent errors, even under similar types of incorrect guidance.

AuthorsOpenReview ↗

Rebuttal for Reviewer UgJo (Part2)

Why does this difference arise? This difference is closely related to the input and training characteristics of selective audio editing. Taking removal as an example, the reference audio contains both the target source to be removed and the non-target source to be preserved. Once the model makes an editing decision, it can minimize the training loss by generating silence for the selected source and reconstructing the remaining source directly from the reference audio, without necessarily reconsidering the instruction. In contrast, general generation must continue relying on the instruction to construct and refine the desired content. Consequently, early decisions in selective editing can exert a stronger and more persistent influence on subsequent denoising. In essence, early denoising in selective audio editing is a decision-making process, determining which sources to edit or preserve, whereas in general generation it primarily establishes a preliminary content framework for subsequent refinement. To our knowledge, this critical early source-selection behavior has not been explicitly identified or systematically investigated in prior work.

Beyond this finding, we identify and systematically investigate source-selection errors in audio editing, and propose a general approach that consistently improves selective editing across existing models. We also introduce a dedicated benchmark and evaluation criteria, and train a 1.3B-parameter unified audio editor with leading overall performance and strong selective editing capabilities. Together, these contributions provide a framework for training, analyzing, and evaluating audio editors, alongside a strong foundation model for future research.

[1] Balaji Y, Nah S, Huang X, et al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers[J]. arXiv preprint arXiv:2211.01324, 2022.

[2] Wang B, Vastola J J. Diffusion models generate images like painters: an analytical theory of outline first, details later[J]. arXiv preprint arXiv:2303.02490, 2023.

[3] Bahmani S, Skorokhodov I, Qian G, et al. Ac3d: Analyzing and improving 3d camera control in video diffusion transformers[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 22875-22889.

[4] Esser P, Kulal S, Blattmann A, et al. Scaling rectified flow transformers for high-resolution image synthesis[C]. ICML, 2024.

AuthorsOpenReview ↗

Rebuttal for Reviewer UgJo (Part3)

W2. Novelty of STEER-Emphasis and Comparison with Existing Reweighting Methods

As discussed in W1, selective audio editing presents a distinct challenge from general generation, requiring the model to correctly determine which sources to edit and which to preserve. To address this challenge, STEER-Emphasis is specifically designed to improve source-selection reliability. Through systematic analysis of early denoising trajectories and source-selection errors, we identify task-dependent critical ranges and increase noise-level sampling within these regions during post-training.

In contrast, existing reweighting methods primarily target other objectives and are less effective at improving selective audio editing performance. While several prior works have explored timestep resampling or reweighting strategies [5–9], they primarily target different objectives: improving image generation quality (SD3, P2), enhancing spatial control in image editing (TV-Edit), balancing motion and visual quality in video generation (TQD), or improving layer assignment in image decomposition (PixelART). These strategies do not explicitly target the short, critical early window in which audio editors must decide which reference sources to edit or preserve. Consequently, their sampling or weighting strategies are not necessarily aligned with the source-selection process, making them less effective when directly applied to selective audio editing.

We summarize representative methods below, including their original motivations and corresponding sampling or weighting strategies. Here, reweighting broadly refers to changing the timestep sampling distribution, the loss weights, or both.

MethodPrimary ObjectiveReweighting StrategyCompared
SD3 [5]Improve high-resolution text-to-image generation qualityσ∼LogitNormal⁡(0,1)\sigma\sim\operatorname{LogitNormal}(0,1)Yes
TV-Edit [6]Improve spatial control and structural consistency in image editingFirst half: σ∼Beta⁡(20,2)→Beta⁡(5,2)\sigma\sim\operatorname{Beta}(20,2)\rightarrow\operatorname{Beta}(5,2); second half: σ∼Beta⁡(5,2)\sigma\sim\operatorname{Beta}(5,2); loss weighted by σ2\sigma^2Yes
TQD [7]Balance motion and visual quality in video generationσ∼Beta⁡(μκ,(1−μ)κ)\sigma\sim\operatorname{Beta}(\mu\kappa,(1-\mu)\kappa), where μ=0.5+0.5(qm−qv)\mu=0.5+0.5(q_m-q_v)No (requires video-specific motion and visual quality scores qmq_m and qvq_v)
PixelART [8]Improve layer assignment in image-to-layer decompositionσ∼0.9LogitNormal⁡(0,1)+0.1U(0.8,1)\sigma\sim0.9\operatorname{LogitNormal}(0,1)+0.1\mathcal U(0.8,1)Yes
P2 [9]Improve perceptual quality in image generationσ∼LogitNormal⁡(0,1)\sigma\sim\operatorname{LogitNormal}(0,1); loss weighted by σ2σ2+(1−σ)2\frac{\sigma^2}{\sigma^2+(1-\sigma)^2}Yes
STEER-EmphasisImprove source-selection reliability in audio editingTask-dependent sampling ranges identified through prefix interventionsOurs
Reviewer_SSWvOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies why unified audio editors often edit the wrong source in selective tasks (removal, replacement, pitch/volume). The authors apply prefix interventions on the denoising trajectory and show that source selection is decided within the first few steps. Based on this, they propose STEER: STEER-Emphasis samples noise levels uniformly in a task-dependent early range, and STEER-Recovery trains on self-rollout states with a target pointing back to the clean edit. Experiments on SteerSound, UNISON, and SAM Audio, together with a new Selective Audio Editing Benchmark, report higher success rates and fewer source-selection errors.

Critical strengths
  1. Strong diagnostic analysis. Prefix interventions causally show that source selection is decided in a few early denoising steps, and that this decisive window differs a lot across tasks (around 2 to 4 steps for removal, much longer for replacement). This task-dependent finding is the key observation, and it directly justifies the task-specific design of STEER.

  2. The method is simple, cheap, and reproducible. It only changes noise-level sampling and adds a self-rollout objective, with no extra inference cost. If released, SteerSound and the Selective Audio Editing Benchmark could also be a lasting contribution to the community.

  3. The writing is clear and the motivation is well paced, moving from failure mode to intervention study to method without wasted steps. The paper is genuinely enjoyable to read.

Critical weaknesses

Overall I do not see major problems in this paper. Below are some points that are unclear to me.

  1. I have some doubts about the evaluation. Success is decided by fixed thresholds such as CLAP > 0.08 (Table 7), but CLAP is quite noisy and the paper does not explain why these values were chosen. The criteria are also never checked against human judgments.

  2. I would like to ask whether the code, the model, and the benchmark will be open-sourced. I did not see any link or release plan in the paper. I think a model like this would be a very useful base tool for the community, so this matters a lot to me and will affect my final score.

  3. Figure 6 is a bit confusing to me. In the figure, there is one joint attention block over reference audio and noisy audio at the bottom, and then two separate blocks on top (instruction with reference audio, and instruction with noisy audio). I first read these as two separate branches, but they are marked as sharing weights.

Questions for authors

Please see Weaknesses.

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer SSWv

We sincerely thank the reviewer for these valuable suggestions, which have greatly strengthened our work. We have added these additional experiments and related discussion in the revised manuscript in Appendix K.

W1. Reliability of the Evaluation Metrics

We address these concerns from three aspects: (1) why CLAP was selected, (2) how the CLAP thresholds were determined, and (3) how our evaluation criteria were validated against human judgments.

Why CLAP? First, CLAP has been widely adopted as a primary evaluation metric in audio generation and editing [1–4]. We specifically adopt LAION's laion/larger_clap_general checkpoint (194M parameters), which covers general audio, music, and speech, and offers greater model capacity and stronger performance than smaller CLAP checkpoints. Following the reviewer's suggestion, we further evaluate the agreement between CLAP-based assessments and human judgments of audio correctness below.

How were the CLAP thresholds selected? In our evaluation, CLAP is primarily used to determine whether the intended audio is correctly generated or added. To determine the thresholds, we manually annotated 500 SteerSound-Base outputs for each of the Generate and Add tasks and evaluated the agreement between CLAP-based predictions and human judgments across different thresholds. We then selected thresholds at one-significant-figure precision that achieved near-optimal agreement, yielding 0.08 for Generate and Replace, and 0.04 for Add. The CLAP score distributions for correct and incorrect outputs are shown in Figures 43 of the revised manuscript.

Agreement between Table 7 metrics and human judgments. We further evaluated the consistency between automatic evaluations and human annotations on 500 samples per task across Generate, Add, Replace, and Remove. The results are summarized below.

TaskSamplesAgreement (%)
Generate50098.80
Add50096.00
Replace50097.80
Remove50098.80
Overall2,00097.85

The overall agreement reaches 97.85%, with a Macro-F1 of 96.48% and Cohen's κ\kappa of 0.930, supporting the reliability of our automatic evaluation protocol. For Pitch and Volume, we directly measure the corresponding attribute changes. Successful edits must achieve the requested change within a ±20% tolerance, while non-target audio must remain nearly unchanged (pitch shift < 1 semitone or volume change < 1.5 dB). Detailed analyses of threshold selection and metric reliability are provided in Appendix K of the revised manuscript.

[1] Haji-Ali M, Menapace W, Siarohin A, et al. Taming data and transformers for audio generation[J]. International Journal of Computer Vision, 2026, 134(3): 87.

[2] Ungersböck M, Grötschla F, Lanzendörfer L, et al. Sao-instruct: Free-form audio editing using natural language instructions[J]. Advances in Neural Information Processing Systems, 2026, 38: 83411-83437.

[3] Paissan F, Della Libera L, Wang Z, et al. Audio editing with non-rigid text prompts[J]. arXiv preprint arXiv:2310.12858, 2023.

[4] Li Z, Xu H, Su J, et al. UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion[J]. arXiv preprint arXiv:2605.31530, 2026.

W2. Open-Source Release

We fully agree that open-sourcing is important for advancing research in audio editing. We are committed to releasing all training code, including the online data construction and pretraining pipelines used to develop our unified audio editor, along with the evaluation code, complete training datasets, evaluation benchmarks, and model weights from all training stages. We will also provide an online inference service to facilitate practical use. We hope these resources will contribute to a more open and reproducible audio editing research community.

W3. Clarification of Figure 6

Thank you for pointing out this ambiguity. The two upper attention blocks are illustrated separately because they receive different inputs (instruction with reference audio and instruction with noisy audio), but they represent the same module with shared weights. The resulting instruction representations are then averaged. We have updated Figure 6 in the revised manuscript to improve readability.

Thanks for the rebuttal. I enjoyed reading this paper. The motivation is clear, and the method is simple but works well. My concerns are resolved: the human agreement study answers my worry about the CLAP thresholds, and I'm glad you're committing to open-source everything. I'll raise my score to 4.

AuthorsOpenReview ↗

Thank you for your positive feedback

Thank you very much for your thoughtful feedback and for taking the time to consider our rebuttal. We are glad that the human agreement study has addressed your concern regarding the CLAP thresholds. We also sincerely appreciate your encouraging assessment of the paper's motivation and method, as well as your recognition of our open-source commitment. We are grateful for your positive assessment and your decision to raise the score. Thank you again for your time and support!

Reviewer_Nnx1OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies source-selection failures in unified audio editors: when asked to remove, replace, or change the pitch or volume of one sound source, a model may leave the target unchanged, edit a non-target source instead, or edit both. By extrapolating intermediate denoising states to their estimated clean endpoints, and by replacing the first k denoising velocities with correct or deliberately corrupted ones, the authors argue that the source to be edited is largely determined within the early part of the trajectory, and that later denoising rarely corrects an early wrong choice. Based on this, they propose STEER, a two-stage post-training framework. STEER-Emphasis samples training noise levels uniformly from a task-dependent early range (the first 2 of 50 steps for removal and the first 15 for replacement, pitch, and volume), keeping the standard Flow Matching objective. STEER-Recovery then rolls the model out for k steps from noise and supervises it with a corrective velocity pointing from the self-generated state back to the clean target, increasing k with patience-based stopping. The authors also build SteerSound-Base, a 1.35B-parameter six-task unified editor, and a Selective Audio Editing Benchmark in which two temporally non-overlapping sources allow automatic categorization of failure types. STEER improves the overall selective success rate of SteerSound from 70.4% to 85.8% on this benchmark, transfers to UNISON and SAM Audio, and early-trajectory emphasis also improves transcript-conditioned target speaker removal and separation.

Critical strengths
  1. A well-defined and practically relevant problem. Isolating source selection as a distinct failure mode of unified audio editors, and splitting it into missed-target, wrong-target, and unintended non-target edits, is a useful lens for evaluating selective editing.

  2. Analysis that motivates the method. The bidirectional prefix interventions (corrective and corruptive) give more direct evidence than visualization alone, and the cumulative intervention effect is used directly to set the training ranges of STEER-Emphasis, so the analysis and the method are tightly connected.

  3. A simple and portable method. STEER-Emphasis only changes the noise-level sampling distribution, and STEER-Recovery only adds a short gradient-free rollout, so both can be applied to existing flow-matching editors with little engineering cost. The critical ranges estimated on SteerSound-Base transfer to UNISON and SAM Audio without re-estimation.

  4. Careful controls. The comparisons against Standard PT with the same number of epochs, against continued STEER-Emphasis with matched optimizer updates, and against an alternative (elbow-based) range definition help separate the effect of the proposed choices from the effect of additional training. The appendix documents data construction, metrics, hyperparameters, and validation curves in detail.

Critical weaknesses

1. The relation to closely related techniques in image and video generation should be made explicit. To my knowledge, this may be the first work to systematically analyze and exploit early-trajectory source selection in audio editing, which I consider a contribution. However, both components appear closely related to techniques that are already used in image and video generation: STEER-Emphasis re-weights training toward high-noise timesteps, and STEER-Recovery trains on self-generated states to reduce the train–inference mismatch. I would encourage the authors to discuss the relevant work from these domains and to clarify what is new in STEER relative to it.

2. Most quantitative comparisons use synthetic data from the same generator as the training data, which makes comparisons with other editors not fully fair. The Complex Audio Editing test set is produced by the same online constructor and the same source pools as the training data (only the individual source clips are held out), and the Selective Audio Editing Benchmark likewise shares its construction with the selective training set. SteerSound is therefore evaluated in-distribution, whereas Audio-Omni, MMEdit, AUDIT, PPAE, ZETA, SDEdit, SAM Audio, and UNISON in Table 1 are evaluated zero-shot. This is especially relevant for reference-based metrics such as LSD and KL-softmax, which compare outputs with targets synthesized by the same pipeline. The conclusion that SteerSound achieves advanced performance among open-source unified editors would be more convincing with results on externally constructed editing benchmarks, if suitable ones exist for these operations (for example, the evaluation set released with MMEdit, which the paper already compares against), or with baselines fine-tuned on the same training data.

Questions for authors
  1. Relation to prior work. Could you discuss how STEER-Emphasis and STEER-Recovery relate to existing timestep re-weighting and train–inference mismatch (exposure bias) techniques in image and video generation, and clarify which aspects are new?

  2. Evaluation beyond the in-house data. Could you report results on an externally constructed audio editing benchmark covering removal and replacement, or compare against baselines fine-tuned on your training data, so that the comparison in Table 1 is not confounded by in-distribution evaluation?

  3. Denoising schedule and critical ranges. Please specify the 50-step noise schedule (uniform or shifted) and the resulting values of σ2 and σ15, since the meaning of "the first 2 of 50 steps" depends on it. Please also state which data the 100 validation examples per task used to derive k = 2 and k = 15 come from (the Complex or the Selective validation set).

  4. Standard metrics for the speaker tasks. For target speaker removal and separation (Section 5.5 and Appendix G), PromptTSE provides the clean target and residual speech, so standard metrics such as SI-SDR improvement, PESQ, and STOI could be computed. Could you report these in addition to CLAP audio–audio similarity, which is less established for speech separation?

8 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer Nnx1 (Part1)

We sincerely thank the reviewer for these valuable suggestions, which have greatly strengthened our work. We have added these additional experiments and related discussion in the revised manuscript in Appendix I, J, L and M.

W1&Q1: Clarify the relation to related techniques in image and video generation.

STEER consists of two complementary stages, STEER-Emphasis and STEER-Recovery. We first clarify the differences in task objectives and challenges between selective audio editing and general image/video generation, followed by comparisons with related training approaches.

A. Differences in task objectives and challenges. General image/video generation primarily aims to produce high-quality, controllable content consistent with the instruction. In contrast, selective audio editing additionally requires the model to correctly determine which reference sources to edit and which to preserve, while maintaining high audio quality. These distinct objectives lead to different challenges during denoising. Selective audio editing requires the model to make reliable source-specific editing decisions within a very short early window and to recover from potentially incorrect decisions. In general image/video generation, early denoising primarily establishes a coarse content framework over a relatively longer initial denoising window. In other words, early denoising in selective audio editing serves as a decision-making process rather than merely framework formation, exerting a stronger and more persistent influence on subsequent denoising. We provide further experimental evidence for this distinction in Appendix I of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal for Reviewer Nnx1 (Part2)

B. Difference between existing reweighting methods and STEER-Emphasis

STEER-Emphasis is specifically designed to improve source-selection reliability, whereas existing reweighting methods primarily target other objectives and are less effective at improving selective audio editing performance. Through systematic analyses of early denoising trajectories and source-selection errors, we identify task-dependent critical ranges and specifically increase noise-level sampling in these critical early regions to guide post-training.

While several prior works have explored timestep resampling or reweighting strategies [1–5], they primarily target different objectives: improving image generation quality (SD3, P2), enhancing spatial control in image editing (TV-Edit), balancing motion and visual quality in video generation (TQD), or improving layer assignment in image decomposition (PixelART). These strategies do not explicitly target the short, critical early window in which audio editors must decide which reference sources to edit or preserve. Consequently, their sampling or weighting strategies are not necessarily aligned with the source-selection process, making them less effective when directly applied to selective audio editing.

We summarize representative methods below, including their original motivations and corresponding sampling or weighting strategies. Here, reweighting broadly refers to changing the timestep sampling distribution, the loss weights, or both.

MethodPrimary ObjectiveReweighting StrategyCompared
SD3 [1]Improve high-resolution text-to-image generation qualityσ∼LogitNormal⁡(0,1)\sigma\sim\operatorname{LogitNormal}(0,1)Yes
TV-Edit [2]Improve spatial control and structural consistency in image editingFirst half: σ∼Beta⁡(20,2)→Beta⁡(5,2)\sigma\sim\operatorname{Beta}(20,2)\rightarrow\operatorname{Beta}(5,2); second half: σ∼Beta⁡(5,2)\sigma\sim\operatorname{Beta}(5,2); loss weighted by σ2\sigma^2Yes
TQD [3]Balance motion and visual quality in video generationσ∼Beta⁡(μκ,(1−μ)κ)\sigma\sim\operatorname{Beta}(\mu\kappa,(1-\mu)\kappa), where μ=0.5+0.5(qm−qv)\mu=0.5+0.5(q_m-q_v)No (requires video-specific motion and visual quality scores qmq_m and qvq_v)
PixelART [4]Improve layer assignment in image-to-layer decompositionσ∼0.9LogitNormal⁡(0,1)+0.1U(0.8,1)\sigma\sim0.9\operatorname{LogitNormal}(0,1)+0.1\mathcal U(0.8,1)Yes
P2 [5]Improve perceptual quality in image generationσ∼LogitNormal⁡(0,1)\sigma\sim\operatorname{LogitNormal}(0,1); loss weighted by σ2σ2+(1−σ)2\frac{\sigma^2}{\sigma^2+(1-\sigma)^2}Yes
STEER-EmphasisImprove source-selection reliability in audio editingTask-dependent sampling ranges identified through prefix interventionsOurs
AuthorsOpenReview ↗

Rebuttal for Reviewer Nnx1 (Part3)

Following the reviewer's suggestion, we apply representative reweighting strategies developed for image/video generation and editing to SteerSound-Base, and evaluate their effectiveness on the Selective Audio Editing Benchmark. The results are summarized below, where Succ. denotes the success rate and SErr. denotes the proportion of cases with selective editing failures.

MethodGenerate Succ.↑Add Succ.↑Replace Succ.↑Replace SErr.↓Remove Succ.↑Remove SErr.↓Pitch Succ.↑Pitch SErr.↓Volume Succ.↑Volume SErr.↓Overall Succ.↑Overall SErr.↓
SD393.7092.9076.2016.4083.0013.4082.4416.9158.408.4081.0513.58
TV-Edit93.4094.2078.3014.8085.9010.6086.5413.0659.607.3082.8411.34
PixelART93.7093.4076.5015.8086.1010.2086.5312.9559.007.7082.3711.58
P293.7092.7075.6016.8083.7012.7083.5515.7958.508.0081.2013.16
STEER-Emphasis94.0094.6078.9014.0088.107.9087.9011.7061.606.8084.029.99

As shown above, existing image/video-oriented reweighting strategies are less effective in addressing source-selection errors in audio editing. STEER-Emphasis achieves the highest overall success rate (84.02%) and lowest selective error rate (9.99%), outperforming the strongest alternative, TV-Edit (82.84% / 11.34%), and achieving the best results across all task-wise metrics. These results indicate that directly applying reweighting strategies designed for image/video generation and editing does not adequately address the source-selection challenge in audio editing. In contrast, STEER-Emphasis explicitly emphasizes the critical early source-selection stages identified through our analysis, leading to more reliable selective editing.

[1] Esser P, Kulal S, Blattmann A, et al. Scaling rectified flow transformers for high-resolution image synthesis[C]. ICML, 2024.

[2] Xie C, Wu Y, Yi Q, et al. Text-Vision Co-Instructed Image Editing[J]. NeurIPS, 2026.

[3] Luo X, Li Q, Li Y, et al. Beyond the Golden Data: Resolving the Motion-Vision Quality Dilemma via Timestep Selective Training[C]. CVPR, 2026.

[4] Jia Z, Zhang Z, Tang Z, et al. PixelART: Image-to-Layer Decomposition without Latents or Text-to-Image Pretraining[J]. arXiv preprint arXiv:2609.22789, 2026.

[5] Choi J, Lee J, Shin C, et al. Perception prioritized training of diffusion models[C]. CVPR, 2022.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal

We sincerely thank the reviewers and the Area Chair for their time and constructive feedback.

Our work identifies the source-selection phenomenon in selective audio editing and systematically investigates its properties through carefully designed experiments. Building on these properties, we propose STEER, which reduces early selection errors (STEER-Emphasis) and trains the model to recover from them (STEER-Recovery). We also introduce a dedicated benchmark and SteerSound, a unified audio editor with leading performance among open-source editors.

The main concerns were how the importance of early denoising in selective audio editing differs from that in conventional diffusion generation (UgJo), the relation to prior techniques (UgJo, Nnx1), in-distribution evaluation (Nnx1), and evaluation reliability (SSWv). We addressed them as follows:

  1. Early denoising in selective audio editing vs. conventional generation (UgJo). In selective editing, early denoising acts as a decision-making process that determines which sources to edit or preserve, rather than merely establishing a preliminary content framework as in general generation. As a result, early denoising in selective editing exerts a stronger and more persistent influence on subsequent denoising.
  2. Relation to prior techniques (Nnx1, UgJo). Existing reweighting and self-generated-state training methods target different problems and consistently fall behind STEER when applied to audio editing.
  3. External evaluation (Nnx1). On the public MMEdit test set, SteerSound achieves the best results on all selective editing tasks, including removal, replacement, and volume change.
  4. Evaluation reliability (SSWv). Our automatic criteria show high agreement with human judgments. We commit to releasing code, data, the benchmark, and model weights.

Reviewers Nnx1 and SSWv confirmed that their main concerns were addressed, with SSWv raising the score, and Reviewer UgJo acknowledged that most of the concern regarding the relevant recovery baseline was addressed. We have also provided further clarifications and additional results in response to the reviewers' follow-up questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.