acceptodds
Under review as a conference paper at ICLR 2027

SMEBench: Benchmarking Neural Music Editing with Synthesized MIDI Edits

Abstract

Neural music editing requires a system to perform a requested change while preserving the musical content that should remain unchanged. Existing evaluations often lack a paired target edit, making these requirements difficult to measure separately. We introduce SMEBench, a large-scale benchmark of controlled music edits constructed by modifying symbolic MIDI performances and rendering matched source and target audio. Each example contains a source, an instruction, a paired target, edit metadata, and complementary descriptions. The benchmark covers twelve instrumentation, articulation, dynamics, and pitch operations and includes a balanced 3,600-example test set together with a smaller real-audio complement derived from multitrack recordings. The paired targets allow source preservation, target agreement, and textual compatibility to be evaluated independently. We use these references to diagnose common audio-editing metrics and benchmark six representative editing systems, showing that the metric used to rank a system can materially change the conclusion. The dataset is available at our repository https://huggingface.co/datasets/anonymous-account-submission/SME-Bench.

Primary area:
datasets and benchmarks
Keywords:
Audio Editing, Generative Models, Music Editing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ouCxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose SMEBench, a new benchmark for music editing based on paired source and target audio. They construct the benchmark by applying different editing operations to MIDI files and synthesizing the original and edited versions into audio pairs. The benchmark covers 12 editing operations and provides ground-truth targets for evaluating music editing models. The authors also introduce SME-Real, a smaller benchmark constructed from real multitrack recordings. Experiments on several existing music editing models show that different evaluation metrics can lead to different conclusions about model performance.

Critical strengths
  1. The paper provides a relatively large benchmark with paired ground-truth targets for music editing. This is useful because obtaining paired data for real-world music editing is difficult, and the MIDI-based construction allows the authors to control the editing operations.
  2. The benchmark covers 12 different editing operations, including instrument editing, articulation, dynamics, and pitch correction. This provides relatively broad coverage of music editing tasks and may be useful for evaluating different editing models.
  3. The authors evaluate several existing music editing models and provide a detailed analysis of different evaluation metrics. The results highlight the limitations of relying on a single metric, as source similarity, target similarity, and text-audio similarity can lead to different model rankings.
Critical weaknesses
  1. The figures are difficult to follow. Some symbols and icons are not clearly explained, making it difficult to understand the overall benchmark construction pipeline. The authors should provide clearer labels and explanations.
  2. The benchmark lacks sufficient validation against human judgments. The authors demonstrate that different evaluation metrics produce different model rankings, but this does not establish whether target-based evaluation is more reliable or better aligned with human perception. In particular, there is no direct comparison between the metric scores and human judgments of editing correctness and source preservation on model-generated outputs. Such validation is important for establishing the reliability of the proposed benchmark.
  3. The benchmark construction relies heavily on manually designed rules and heuristics, with limited ablation studies. The authors make several design choices regarding instrument selection, MIDI transformations, audio filtering, and caption generation. Although these choices are reasonable, their effects on the benchmark quality and evaluation results are not sufficiently investigated. It remains unclear whether alternative construction strategies would lead to different conclusions or model rankings.
  4. The gap between synthesized MIDI audio and real-world music recordings remains a concern. MIDI-based synthesis provides precise control over editing operations, but the resulting audio may not fully represent real-world music editing scenarios. Although the authors introduce SME-Real, it contains only 416 examples and covers fewer editing operations. More systematic comparisons are needed to demonstrate whether the model rankings and metric conclusions obtained from SMEBench remain consistent on real recordings.
  5. The human validation of the generated ground-truth targets is limited. The listening study involves only 17 participants and 236 ratings, which is relatively small compared with the benchmark size and 12 editing operations. Furthermore, three operations do not reliably exceed the midpoint in instruction-alignment scores. This raises concerns about whether the generated targets consistently represent perceptually meaningful and correctly executed edits. These operations are still included in the benchmark, which may affect the reliability of the evaluation results.
  6. The reliability of the captioning pipeline is not sufficiently validated. The authors rely on extracted music features and Qwen for caption generation and filtering. Errors in the extracted features may propagate through the pipeline, while using the same Qwen checkpoint for generation and filtering may introduce correlated errors. The authors prefer Qwen because it is more conservative, but a lower acceptance rate does not necessarily indicate higher accuracy. Human validation would help justify this choice.
Questions for authors
  1. Can the authors conduct a human evaluation of the edited outputs generated by different models and report the correlation between human judgments and the evaluation metrics? In particular, do target-based metrics correlate better with human judgments of editing correctness and source preservation than source-based or text-based metrics?
  2. Can the authors provide a systematic comparison between SMEBench and SME-Real? For example, are the model rankings consistent across the two benchmarks when evaluated on the same editing operations and using the same metrics?
  3. How sensitive are the benchmark results to the construction strategies, such as instrument sampling, MIDI editing rules, and audio filtering thresholds? Can the authors provide ablation studies to justify these choices and demonstrate that the main conclusions are robust?
  4. Three editing operations do not reliably exceed the midpoint in human instruction-alignment scores. Why are these operations retained in the benchmark, and how do the authors ensure that their ground-truth targets are reliable? Would excluding these operations significantly change the evaluation results or model rankings?
  5. Can the authors provide human validation of the captioning pipeline, particularly for samples where Qwen and Mistral disagree? Since the two judges show only moderate agreement, what evidence supports the decision to use Qwen rather than Mistral for filtering?
  6. The Hugging Face dataset does not appear to display correctly online. Could the authors update the dataset viewer so that users can inspect the samples directly without downloading all the JSON files?
4 comments
AuthorsOpenReview ↗

Reply to reviewer ouCx, part 1

We thank the reviewer for recognizing the value of "a relatively large benchmark with paired ground-truth targets". Their questions on human judgments, real audio and the weak operations led to new results, reported below.

(W1) The figures are difficult to follow.

We agree to simplify them. We improved the readability of Figure 1 and made the caption more precise. We also made some changes in the revision to make the manuscript clearer.

(W2, Q1) Do evaluation and target-based metrics agree with human judgments on system outputs?

We ran an additional user study, where we deployed a MUSHRA-style listening test with 8 participants on 14 samples. Participants rated the edits of the six systems, plus two anchors (the ground-truth edit as upper anchor and the unedited source as lower anchor), from 0 to 100 on two criteria:

  • source preservation (“is everything that should not change still the same as in the source: same audio scene, same sound quality?”);

  • edit accuracy (“is the requested edit clearly audible and does it match the edit request?”).

Table R2 gives the correlation between the human rankings (anchors excluded) and each metric, at the level of the six systems and per sample (mean over samples, Wilcoxon test). Below we summarize the findings:

Table R2. Kendall τ between human ratings (Preservation and Edit Accuracy) and each metric (p-value in parentheses).

MetricPreservation (per sample)Edit Acc. (per sample)
LPAPS to target−0.57 (.001)−0.27 (.006)
LPAPS to source−0.52 (.001)−0.11 (NS)
CLAP to target0.35 (.003)0.26 (.03)
CLAP to source0.38 (.005)0.10 (NS)

Audio-to-audio metrics agree with human ratings on preservation, but only the target-based metrics have a significant per-sample correlation with edit accuracy. The text metric does not track either criterion. The number of participants is small, so we treat these results as supportive evidence.

(W3, Q3) Construction choices lack ablations.

A full ablation is not feasible, we report what the data supports.

While we agree with the reviewer that it would be preferable to ablate all the design choices, this constitutes a very large combinatorial search space, and we therefore followed choices that respect musical and machine learning principals.
We would like to also note that the system rankings agree between two independently constructed datasets (Table R1), and removing the three operations with the weakest human scores does not modify the ranking orders on any of the two (Table R2 in our reply to Reviewer EPLR).

AuthorsOpenReview ↗

Reply to reviewer ouCx, part 2

(W4, Q2) Is there a systematic comparison of SMEBench and SME-Real? Are the rankings consistent?

For this rebuttal, we have compared SMEBench and SME-Real on the same metrics and the same edit types.

On the 9 shared edit types, every metric orders the systems very similarly on both datasets. For instance on LPAPS metrics (source and target) the median ranking is the same across synthetic and real samples, with a 95% Confidence Interval of [+0.87, +1.00]. For CLAP-based metrics also, the behaviour of the baselines between synthetic and real datasets also transfer, with positive ordinal correlation: [+0.47, +0.60], [+0.33, +0.47] for text and audio respectively as well (again, 95% Confidence Interval).

(W5, Q4) Why keep the three operations that do not clear the midpoint, and would excluding them change the results?

These operations are valid edits that can be hard to hear. Excluding them doesn’t change any ranking.

The pipeline applies each requested edit exactly as specified, but some edits can be more subtle than others and it is possible for the listeners to miss these subtle edits. .

For example, when applying swap-drums, a partial selection of drums is much less audible than replacing all of them; for remove-instrument, audibility depends on the rank of the removed instrument; and hearing legato depends on the base articulation of the modified instrument. We kept the operations because they cover real editing actions. Some illustrative subtle examples, together with brief notes on them, are available at https://huggingface.co/spaces/anonymous-account-submission/SME-Bench-Companion. Removing those edits doesn’t change the rankings in evaluations.

(W6, Q5) Is the captioning pipeline validated by humans? What supports the decision to use Qwen ?

The captioning pipeline isn’t validated by humans, but the retained captions are.

We would like to note that the judge exists solely to remove bad examples.Therefore, what matters is the quality of the captions it lets through. Our goal was to measure this with participants. Raters judged captions blind: 15 came from our pipeline and 5 were written by a trained musician. The submission reported only the pipeline captions (Table 1); the revision adds the comparison.

Table R3. Percentage of ratings per level. 2 of the 236 trials have no rating.

RatingPipeline captions (%)Human-written captions (%)
Very bad (1)5.751.67
Bad (2)9.206.67
Fair (3)18.3913.33
Good (4)42.5340.00
Very good (5)24.1438.33
Number of ratings17460

We observe that 85.1% of pipeline ratings are fair or better, against 91.7% for the musician’s captions. We see that our pipeline captions are close to ones written by an expert musician. The study did not isolate the samples where Qwen and Mistral disagree, we report this as a limitation. Note that the edit instructions are deterministic: they come from a pool of human-written edit-aware captions.

(Q6) The Hugging Face viewer does not display correctly.

Thank you for noting this, we have corrected it. The dataset page now shows sme-test and sme-real samples directly, without downloading the JSON files ( https://huggingface.co/datasets/anonymous-account-submission/SME-Bench).

Thank you for the comments and additional experiments. The listening study on system outputs directly addresses my main concern about whether the metrics reflect human judgments. The comparison on shared edit types also strengthens the case that findings from SMEBench transfer to recorded music, particularly for LPAPS. I appreciate the check showing that excluding the three less audible operations leaves the rankings unchanged. Overall, the new results address my main concerns, and I am raising my score to 3 (weak accept). Good luck with the submission.

Thank you very much for your constructive review, and raising your score!

Reviewer_uxpwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SMEBench, a benchmark for neural music editing. The benchmark is constructed by modifying symbolic MIDI and rendering matched source–target audio pairs, accompanied by corresponding captions and instructions. The balanced test set contains 3,600 examples spanning 12 edit types. The authors conduct a human study to verify the targets and captions. They then evaluate six systems on source similarity, target similarity, text alignment, and audio quality, and show that different metrics can yield different system rankings.

Critical strengths

An important and clearly articulated evaluation problem. The paper identifies an important problem in music editing evaluation:

  • A high source-similarity score can reward a system that simply outputs its input unchanged.
  • A high text-alignment score does not necessarily mean that the requested edit was performed.

A practical decomposition and construction method.

  • The paper explains this problem clearly and separates source preservation, target agreement, and text compatibility. It also provides a practical method for constructing paired data.
Critical weaknesses

The experiments do not sufficiently validate how the evaluation scores should be interpreted.

Figure 3 does not establish the benchmark’s validity. Figure 3 shows only how each metric responds on SMEBench, which does not demonstrate the validity of the dataset itself. A more meaningful validation would compare against a real-world reference benchmark, such as SME-Real, as an oracle.

Figure 4 does not address the key question. Figure 4 shows the difference between each system’s metrics on SMEBench and on SME-Real, but this mainly illustrates how the systems perform on two different datasets.

  • If SME-Real serves as a real-audio reference benchmark, the key question is whether SMEBench leads to similar conclusions about relative system performance.
  • The figure does not make this clear, and the paper does not explicitly discuss which findings remain consistent across the two datasets.

The human evaluation lacks anchors. For constructing such a benchmark, I believe the most meaningful validation is human evaluation, as in Table 1. However, these scores are collected only within SMEBench, with no reference anchor (e.g., SME-Real) or lower anchor.

  • A MUSHRA-like test would be more appropriate. The simple MOS protocol used here offers limited validity in this setting, especially given that the dataset is synthetic.
Questions for authors
  • Did the listening study include anchors for each rating point?
  • Which CLAP checkpoints are used for the CLAP-based metrics? Were the same settings used for both dataset filtering and final evaluation?
  • Do the main conclusions drawn from Figure 4 remain the same when computed on SME-Real?
3 comments
AuthorsOpenReview ↗

Reply to reviewer uxpw, part 1

We thank the reviewer for describing the evaluation problem as "important and clearly articulated", and for noting that the paper separates source preservation, target agreement and text compatibility.

Below we answer the concerns,

(W1) Figure 3 does not establish the benchmark’s validity.

Figure 3 shows how the main metrics respond to each edit type, which exposes limits of these metrics used in the audio editing literature as measures of edit quality. In real audio editing datasets (e.g. ZomeBench, MedleyDB) no real target reference exists that could serve as an oracle. We use SME-Real as an independent check on the conclusions, and to establish the reality gap incurred with SMEBench. In the revision we added the SME-Real counterpart of Figure 3 (we added Figure 6 for this).

(W2, Q3) Figure 4 compares only raw values; does SMEBench lead to the same conclusions as SME-Real?

As we have also replied to reviewer EPLR, we ranked the six systems under each metric on SME-Real and on SMEBench restricted to the 9 shared edit types. Kendall's τ\tau between the two rankings (1 = same order) is +0.60+0.60 for CLAP text similarity, +0.47+0.47 and +1.00+1.00 for CLAP and LPAPS similarity to the target, and +0.33+0.33 and +1.00+1.00 for CLAP and LPAPS similarity to the source (respectively). We therefore see that the ranking correlation between SMEBench and SMEReal remain high.

All 95% bootstrap intervals are above 0; the full table is Table R1 in our reply to Reviewer EPLR. At the top of the ranking, MusicGen is first on text similarity and SteerMusic first on both target metrics, on both datasets. In the revision we states these agreements next to Figure 4.

AuthorsOpenReview ↗

Reply to reviewer uxpw, part 2

(W4, Q1) Anchors in the listening study, and a MUSHRA-like test.

As you noted, our original user study did not include anchors. We therefore extended it with a MUSHRA-style study reported in Table R2 (reply to Reviewer ouCx). Participants rated source preservation and edit accuracy of samples from the six methods and two anchors: the ground-truth edit (upper) and the unedited source (lower). On SMEBench, the unedited source is rated high on preservation (92) but very low on edit accuracy (15), while the ground-truth edit is rated high on both (91 and 84), as expected. In the revised version of the paper we added this analysis in table 5.

To summarize our findings here as well, we are sharing the table below:

ItemSMEBenchSME-Real
Unedited source (lower anchor)92 / 1591 / 10
Ground-truth edit (upper anchor)91 / 8486 / 91

Note that first numbers in each is for source preservation, and the second numbers are for the edit accuracy. We observe that the anchor ratings are very similar between SMEBench and SME-Real.

(Q2) Which CLAP checkpoint is used, and is it the same for filtering and evaluation?

We use the same checkpoint for both: music_audioset_epoch_15_esc_90.14.pt. The revision states this in the evaluation section.

Presentation.

We improved the readability of Figure 1, and added some details in the revision to make the manuscript clearer.

AuthorsOpenReview ↗

Discussion period ending, feedback

Dear reviewer,

Thank you very much for your constructive review. We were wondering if you had any feedback for our rebuttal, given that the author-reviewer discussion period is coming to a close soon.

Best regards,

Authors

Reviewer_EPLROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SMEBench is a benchmark for neural music editing that provides paired source and target audio, which existing evaluations (MedleyMDPrompts, ZoME-Bench) lack. Edits are applied at the symbolic layer. MIDI excerpts from MIDICaps are cropped to 10 seconds and modified by one of twelve operations across instrumentation (remove, replace, extract, double, on instruments or drums), articulation (staccato, legato), dynamics (crescendo, decrescendo), and pitch correction. Source and target are rendered with the same FluidSynth SoundFont. Each example has a source, an instruction, a paired target, edit metadata, and four textual views. The release has 11,160 paired edits with a balanced 3,600-example test set, plus a 416-pair real-audio complement (SME-Real) from MoisesDB stems. A listening study (17 participants, 236 trials) validates the targets (quality 4.23, source alignment 4.65, instruction alignment 3.99). The paper then shows that source-preservation, target-agreement, and text-compatibility metrics give conflicting rankings, and benchmarks six systems (SDEdit, DDIM inversion, VCI, ZETA, SteerMusic, MusicGen), finding ranking reversals (CLAP-Text favors MusicGen and VCI, target metrics favor SteerMusic and ZETA).

Critical strengths
  1. The paired-target construction is a real contribution. Modifying MIDI and rendering source and target with the same SoundFont gives controlled counterfactual pairs with known edits, plus explicit metadata and masks. This fills a genuine gap and lets the field measure preservation and edit execution separately. The dataset and code are released.

  2. The authors are upfront about the limitations. They acknowledge the synthetic-to-real gap, the correlated caption and filter errors from using the same model family, the non-bijectivity of edits, and they exclude three weakly-perceptible operations from the headline aggregate. A real-audio complement is provided to check whether the findings persist.

Critical weaknesses
  1. The benchmark is fundamentally synthetic (FluidSynth rendering), and the real-audio complement is small (416 pairs) and covers only a subset of the taxonomy (stem-based removal, extraction, doubling, swap, and dynamics, with no articulation or pitch correction). The diagnostic conclusions are therefore established mostly on synthetic audio.

  2. The validation is thin relative to the scale. 17 participants, 236 total trials, and only 20 source-target pairs were rated. Three of twelve operations (legato, remove-instrument, replace-drums) fail to clear the midpoint of the instruction-following scale (3.33, 3.27, 3.00). A balanced 3,600-example test set therefore contains a sizable fraction of edits that listeners cannot reliably attribute to the instruction. The "metrics disagree" finding itself is also well known from image editing.

Questions for authors
  1. The human study covers only 20 source-target pairs. Do you plan to scale it up and report instruction-following per operation as a first-class property, so users can subset to well-validated edits?

  2. Can you show whether the metric-diagnosis conclusions on SME-Real differ qualitatively from SMEBench, and whether the same system ranking reversal holds on the real subset?

  3. For the three weakly-perceptible operations, is there a way to redesign them (stronger interventions or more audible rendering) so the balanced test set is uniformly reliable?

3 comments
AuthorsOpenReview ↗

Reply to reviewer EPLR, part 1

We thank the reviewer for recognizing the paired-target construction as "a real contribution". Their questions on real audio and on validation led to new analyses, reported below.

(W1, Q2) The benchmark is synthetic by design. Do the conclusions on SME-Real match SMEBench?

To test whether findings persist on real audio, In Table R1, we show that every metric orders the systems the same way on both datasets.

We ranked the six systems under each metric on SME-Real and on SMEBench . Table R1 compares the two orderings according to the Kendall’s τ\tau metric (https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient ).

Table R1. Kendall's τ\tau between the system rankings on SMEBench (9 shared edit types) and on SME-Real, caption conditioning, with 95% bootstrap interval.

Metricτ\tau [95% CI]
CLAP text similarity+0.60+0.60 [+0.47+0.47, +0.60+0.60]
CLAP audio similarity to target+0.47+0.47 [+0.33+0.33, +0.47+0.47]
LPAPS distance to target+1.00+1.00 [+0.87+0.87, +1.00+1.00]
CLAP audio similarity to source+0.33+0.33 [+0.20+0.20, +0.47+0.47]
LPAPS distance to source+1.00+1.00 [+0.87+0.87, +1.00+1.00]

We observe that every metric orders the systems in the same direction on both datasets, and every interval is above 0. Absolute values differ between datasets (Figure 4), which is why we compare rankings. The revision states this next to Figure 4.

We would also like to emphasize two points to put this in context. First, an editing system that fails on clean rendered audio shows a real shortcoming. Secondly, we would like to note that MIDI sequencing and virtual instruments are part of real production workflows (Deruty et al., 2022; Bontempi et al., 2023), and rendered multitrack MIDI underlies established datasets such as Slakh2100 (Manilow et al., 2019).

AuthorsOpenReview ↗

Reply to reviewer EPLR, part 2

(Q1, W2) Is there a plan to scale up the human study and report instruction following per operation?

Yes, we extended the study to better understand the correlation between the metrics and human preference. The listening study described in our reply to Reviewer ouCx (W2, Q1) extends the main study, with a design refined in response to the reviews. We asked for two criteria: source preservation (alignment with the source), and edit accuracy (is the requested edit audible and does it match the request?).

The new round adds hidden anchors and 0-100 ratings (MUSHRA style), and the ground-truth edit is rated among the system outputs. An edit is valid only if both preservation and accuracy are high. Ground-truth edits meet both (≥ 60) in 88% of ratings, against 2% for the unedited source and 2–9% for the six systems.The new round has different samples from both sme-test and sme-real and collection is ongoing.

We would like to note that Instruction following per operation was already reported in Table 4. We do not release it as a property of each sample: it comes from ratings on specific items, so it reflects how perceivable those items were rather than the operation as a whole. The edit metadata however allows one to isolate specific edit properties: users can select the subset that suits their test, for instance well-audible edits.

(Q3) Can the three weakly audible operations be redesigned?

We would like to note that the pipeline applies each requested edit exactly as specified. However some edits are subtle (E.g. the cases where only one drum swapped out of several drum choices, or a quiet instrument removed). In these cases listeners may miss them.

We shared a listening page with Reviewer ouCx that illustrates the subtlety of these edits: https://huggingface.co/spaces/anonymous-account-submission/SME-Bench-Companion

Furthermore, note that excluding the weak operations does not change the system rankings, and the weak operations alone give the same rankings as the other nine.

References

Bontempi, P., Canazza, S., Carnovalini, F., and Rodà, A. (2023). Research in Computational Expressive Music Performance and Popular Music Production: A Potential Field of Application? Multimodal Technologies and Interaction, 7(2), 15.

Deruty, E., Grachten, M., Lattner, S., Nistal, J., and Aouameur, C. (2022). On the Development and Practice of AI Technology for Contemporary Popular Music Production. Transactions of the ISMIR, 5(1), 35–49.

Manilow, E., Wichern, G., Seetharaman, P., and Le Roux, J. (2019). Cutting Music Source Separation Some Slakh. IEEE WASPAA, 45–49.

AuthorsOpenReview ↗

Discussion period ending, feedback

Dear reviewer,

Thank you very much for your constructive review. We were wondering if you had any feedback for our rebuttal, given that the author-reviewer discussion period is coming to a close soon.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Answer

We thank the reviewers for their careful reading. All three give informativeness ("I learned something interesting or useful") and soundness ("minor or fixable concerns") the same subscores, and they describe the work as "a real contribution" that "fills a genuine gap" (EPLR), "an important and clearly articulated evaluation problem" (uxpw) and "a relatively large benchmark with paired ground-truth targets" (ouCx).

The reviewers also raised several points addressed in the revision, summarized below (Tables R1–R2 appear in the replies).

Do the findings hold on real audio? (EPLR, uxpw, ouCx).

In order to demonstrate that it does, we ranked the six systems under each metric and compared rankings with Kendall's τ\tau (+1+1 = same order, −1-1 = reversed) [1], with 95% bootstrap intervals. On the 9 edit types shared by SMEBench and the real-audio SME-Real, every metric orders the systems very similarly on both datasets (τ\tau from +0.87+0.87 to +1.00+1.00, according to the LPAPS metrics Table R1). The synthetic benchmark and the real stems lead to the same conclusions about the systems.

Do metrics agree with human judgments (uxpw, ouCx)?

We did a user study for this, and per-sample LPAPS and audio CLAP agree with humans on source preservation (Kendall ∣τ∣=0.38|\tau|=0.38 (CLAP) 0.520.52 (LPAPS). On edit accuracy, we reach a similar conclusion with Kendall $|\tau|= 0.26 (CLAP) , 0.27 (LPAPS). - Please see Table R2 for details.

Weakly audible operations (EPLR, ouCx).

Legato, remove-instrument and swap-drums score lower on instruction following because the edits are subtle, and more challenging to notice (e.g. only one drum is replaced), not because the edits are inaccurate (please see the examples at https://huggingface.co/spaces/anonymous-account-submission/SME-Bench-Companion). In our opinion, therefore excluding these edits does not change overall conclusions.

We thank the reviewers again for their constructive feedback, which helped us to improve the paper.

[1] https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.