acceptodds
Under review as a conference paper at ICLR 2027

CLEF: Learning Closed-Loop Control for Text-Guided Music Editing

Abstract

In this paper, we propose CLEF, a framework that learns closed-loop control for text-guided music editing. Our key idea is to use the outcome of each editing round to adjust the guidance for the next, preserving the source audio’s structure while faithfully realizing the requested edits. Specifically, we construct two complementary feedback signals to assess unintended structural changes and insufficient adherence to the editing instruction. Temporal structural feedback uses our proposed TACons metric to evaluate how well the source audio’s temporal structure is preserved. Target semantic feedback measures alignment between the edited audio and the target text using CLAP, assessing whether the requested attributes have been faithfully realized. To translate these evaluations into control over subsequent edits, we learn a feedback-aware Compensation Controller that combines both feedback signals with the current editing state to predict the direction and magnitude of updates to the target textual conditioning representations. At inference, the updated conditioning guides the next editing round, whose output is evaluated again to inform subsequent compensation. This closes the feedback loop and allows the same trained controller to refine the conditioning over multiple rounds, adapting the editing process to the observed structural and semantic discrepancies. Moreover, CLEF can flexibly incorporate an additional reference feedback signal to assess attribute consistency with reference audio, further enhancing personalized editing. Extensive experiments demonstrate that CLEF consistently outperforms state-of-the-art baselines, achieving a 23.78% improvement in CQT1-PCC and a 4.18% gain in CLAP, along with superior subjective listening quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text-guided Music Editing, Closed-loop Control, Feedback-guided Iterative Refinement, Semantic-Structural Compensation, Target-condition Compensation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nJdSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors introduce CLEF a novel closed-loop formulation for music editing. This approach is novel in music editing as it is distinct from single-pass or training free methods. The authors evaluated based on objective evaluation metrics, in terms of various aspects distributional fidelity and source preservation. The objective and subjective (w/ 25 participants) evaluation results demonstrate improvements over baselines.

Critical strengths
  1. The approach is distinct from existing ones, which primarily have focused on single-pass or training-free approaches.
  2. The approach shows improvements in terms of objective and subjective evaluations, over existing approaches.
Critical weaknesses
  1. The authors did not provide demo, and this is a critical weakness, considering the instability of generative audio evaluation metrics.
  2. The presentation can be significantly improved. For example, it is not immediately clear which musical aspects the model edited. In addition, Line 322 (We use ... CLAP ....) is not clear. It is not clear what the CLAP metric means by reading the sentence alone. (The readers must refer to Section 4 for the context.)
  3. The paper can better contextualize their core idea relative to iterative self-refinement frameworks in other modalities. (However, music/audio editing baselines are well-covered)
  4. Inference cost is high, as the authors suggested.

Overall, given the instability of generative audio evaluation metrics, it is difficult to validate the authors' contribution without demo and code.

Questions for authors
  1. Can the authors provide demo?
  2. Any reason why the authors cannot release the code?
  3. Can the authors sharpen novelty claims within the context of self-refinement frameworks in other modalities?
Reviewer_jkXYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes CLEF, a looped text-guided music editing scheme. In order to overcome the challenge of achieving sufficient editing on the target property while preserving the rest, TAcons (DTW-based temporal alignment consistency) and CLAP are used as supportive metrics to construct feedback to the mapped conditioning embeddings. A feedback-aware compensation controller (FACC) is trained to do dynamic adjustment of conditioning embedding based on the editing result of previous step. At inference time, through the iterative adjustment of conditioning embedding respect to the feedback from the two supportive metrics, a series of candidates is generated. Lastly, the final editing result is selected by a hybrid scoring between semantic gains and structural preservation. The experiment demonstrates the effectiveness of CLEF in both text-prompted editing and personalized editing. The ablation study demonstrates the necessity of each component and the validity of the structure-to-semantics scheduling in DDS optimization.

Critical strengths
  • The idea of dynamically adjust the conditioning embedding respect to multiple targets (i.e. structural preservation and editing strength) in a looped setup. Likely applicable to other tasks as well.
  • Shows improvement over existing baselines.
Critical weaknesses
  • What if the two supportive metrics (TAcons & CLAP) do not work or conflicts to the editing prompt? For example, if the text-prompt asks to modify the melody, it's likely that CLAP cannot provide useful semantic feedback in this case. Moreover, preserving the structure does not equal to preserve the non-target properties unmentioned by the editing prompt.
Questions for authors
  • In the evaluation set, what kind of editing prompts are there? Are there anything that contradicts to the assumption? For example, prompts that ask to change temporal structure but keep the timbre and music style. In such case, how does CLEF perform compared with other baselines?

  • What are the actual regularization terms of \mathcal{L}\lambda and \mathcal{L}\delta in (5)?

  • What is \gamma in \mathcal{L}_{mag} ?

Reviewer_sYdpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper trains a lightweight adaptor that enables editing with pretrained text-to-music models.

The base editing algorithm is taken from Delta Denoising Score (Hertz et al., 2023), and this work builds on top of that by adding feedback signals that consider (i) temporal consistency (w.r.t. original output), (ii) semantic alignment (w.r.t. intended edits), and optionally (iii) reference consistency (in personalized use cases). The feedback signals are used to train the lightweight adaptor that can be iteratively invoked to update the target conditions at test time to refine the edits.

Experimental validation is done with common music editing tasks including instrument, genre, mood, and background changes. Around 10 baselines are involved in the benchmark. Ablation studies are done over {feedback signals, training objective} to show the relative optimality of the full proposed setup. Listening study is also performed to show meaningful user preference over recent strong baselines (i.e., SteerMusic and Steer Music+).

Critical strengths
  • Method is clean and elegant. Only an additional small 2-layer network is required to generate/update the target editing condition, which, if I understand correctly from line 313/314, can be trained with <500 examples. The test-time iteration is also carefully designed to balance trade-off between consistency and edit alignment.

  • Evaluations are overall comprehensive and strong. The proposed method is mostly the best performing across zero-shot and personalized editing scenarios. Ablation study is also informative.

Critical weaknesses
  • Listening examples are missing. Although I really appreciate the extension evaluation results, as a reviewer I'd really want to listen to the generated outputs to judge the qualitative characteristics, and differences from the included baselines.

  • The main method paragraph (lines 244-265) is a bit too densely packed. It is somewhat difficult to intuitively understand the detailed formulation (e.g., tanh is used for sedits_{\text{edit}} while exp is used for spress_{\text{pres}}). Consider elaborating in more detail.

Questions for authors
  • (Q1) Would it be possible to provide some listening examples? (Let me know if I missed!)

  • (Q2) Thank you for the overhead analysis in Figs 5 and 6. Is it possible to comment on the relative overhead compared to 1st-pass text-to-music generation in the main text? (E.g., To perform editing with [...] FACC iterations, CLEF takes [..]% of time, [..]% of memory relative to the initial text-to-music generation.)

  • (Q3) In Eq. 6, is it a must that the condition updates on {structure preservation, semantic alignment} must be done one after another? If so, what are some intuitive reasons the two Deltas cannot be summed together in each iteration?

  • (Q4) Did I understand correctly that the edited/updated target condition ctc_t is exactly the text embedding? If so, can the updated ctc_t be somehow decoded back to plain text for interpretability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.