- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose SMEBench, a new benchmark for music editing based on paired source and target audio. They construct the benchmark by applying different editing operations to MIDI files and synthesizing the original and edited versions into audio pairs. The benchmark covers 12 editing operations and provides ground-truth targets for evaluating music editing models. The authors also introduce SME-Real, a smaller benchmark constructed from real multitrack recordings. Experiments on several existing music editing models show that different evaluation metrics can lead to different conclusions about model performance.
Critical strengths
- The paper provides a relatively large benchmark with paired ground-truth targets for music editing. This is useful because obtaining paired data for real-world music editing is difficult, and the MIDI-based construction allows the authors to control the editing operations.
- The benchmark covers 12 different editing operations, including instrument editing, articulation, dynamics, and pitch correction. This provides relatively broad coverage of music editing tasks and may be useful for evaluating different editing models.
- The authors evaluate several existing music editing models and provide a detailed analysis of different evaluation metrics. The results highlight the limitations of relying on a single metric, as source similarity, target similarity, and text-audio similarity can lead to different model rankings.
Critical weaknesses
- The figures are difficult to follow. Some symbols and icons are not clearly explained, making it difficult to understand the overall benchmark construction pipeline. The authors should provide clearer labels and explanations.
- The benchmark lacks sufficient validation against human judgments. The authors demonstrate that different evaluation metrics produce different model rankings, but this does not establish whether target-based evaluation is more reliable or better aligned with human perception. In particular, there is no direct comparison between the metric scores and human judgments of editing correctness and source preservation on model-generated outputs. Such validation is important for establishing the reliability of the proposed benchmark.
- The benchmark construction relies heavily on manually designed rules and heuristics, with limited ablation studies. The authors make several design choices regarding instrument selection, MIDI transformations, audio filtering, and caption generation. Although these choices are reasonable, their effects on the benchmark quality and evaluation results are not sufficiently investigated. It remains unclear whether alternative construction strategies would lead to different conclusions or model rankings.
- The gap between synthesized MIDI audio and real-world music recordings remains a concern. MIDI-based synthesis provides precise control over editing operations, but the resulting audio may not fully represent real-world music editing scenarios. Although the authors introduce SME-Real, it contains only 416 examples and covers fewer editing operations. More systematic comparisons are needed to demonstrate whether the model rankings and metric conclusions obtained from SMEBench remain consistent on real recordings.
- The human validation of the generated ground-truth targets is limited. The listening study involves only 17 participants and 236 ratings, which is relatively small compared with the benchmark size and 12 editing operations. Furthermore, three operations do not reliably exceed the midpoint in instruction-alignment scores. This raises concerns about whether the generated targets consistently represent perceptually meaningful and correctly executed edits. These operations are still included in the benchmark, which may affect the reliability of the evaluation results.
- The reliability of the captioning pipeline is not sufficiently validated. The authors rely on extracted music features and Qwen for caption generation and filtering. Errors in the extracted features may propagate through the pipeline, while using the same Qwen checkpoint for generation and filtering may introduce correlated errors. The authors prefer Qwen because it is more conservative, but a lower acceptance rate does not necessarily indicate higher accuracy. Human validation would help justify this choice.
Questions for authors
- Can the authors conduct a human evaluation of the edited outputs generated by different models and report the correlation between human judgments and the evaluation metrics? In particular, do target-based metrics correlate better with human judgments of editing correctness and source preservation than source-based or text-based metrics?
- Can the authors provide a systematic comparison between SMEBench and SME-Real? For example, are the model rankings consistent across the two benchmarks when evaluated on the same editing operations and using the same metrics?
- How sensitive are the benchmark results to the construction strategies, such as instrument sampling, MIDI editing rules, and audio filtering thresholds? Can the authors provide ablation studies to justify these choices and demonstrate that the main conclusions are robust?
- Three editing operations do not reliably exceed the midpoint in human instruction-alignment scores. Why are these operations retained in the benchmark, and how do the authors ensure that their ground-truth targets are reliable? Would excluding these operations significantly change the evaluation results or model rankings?
- Can the authors provide human validation of the captioning pipeline, particularly for samples where Qwen and Mistral disagree? Since the two judges show only moderate agreement, what evidence supports the decision to use Qwen rather than Mistral for filtering?
- The Hugging Face dataset does not appear to display correctly online. Could the authors update the dataset viewer so that users can inspect the samples directly without downloading all the JSON files?
4 comments
Reply to reviewer ouCx, part 2
(W4, Q2) Is there a systematic comparison of SMEBench and SME-Real? Are the rankings consistent?
For this rebuttal, we have compared SMEBench and SME-Real on the same metrics and the same edit types.
On the 9 shared edit types, every metric orders the systems very similarly on both datasets. For instance on LPAPS metrics (source and target) the median ranking is the same across synthetic and real samples, with a 95% Confidence Interval of [+0.87, +1.00]. For CLAP-based metrics also, the behaviour of the baselines between synthetic and real datasets also transfer, with positive ordinal correlation: [+0.47, +0.60], [+0.33, +0.47] for text and audio respectively as well (again, 95% Confidence Interval).
(W5, Q4) Why keep the three operations that do not clear the midpoint, and would excluding them change the results?
These operations are valid edits that can be hard to hear. Excluding them doesn’t change any ranking.
The pipeline applies each requested edit exactly as specified, but some edits can be more subtle than others and it is possible for the listeners to miss these subtle edits. .
For example, when applying swap-drums, a partial selection of drums is much less audible than replacing all of them; for remove-instrument, audibility depends on the rank of the removed instrument; and hearing legato depends on the base articulation of the modified instrument. We kept the operations because they cover real editing actions. Some illustrative subtle examples, together with brief notes on them, are available at https://huggingface.co/spaces/anonymous-account-submission/SME-Bench-Companion. Removing those edits doesn’t change the rankings in evaluations.
(W6, Q5) Is the captioning pipeline validated by humans? What supports the decision to use Qwen ?
The captioning pipeline isn’t validated by humans, but the retained captions are.
We would like to note that the judge exists solely to remove bad examples.Therefore, what matters is the quality of the captions it lets through. Our goal was to measure this with participants. Raters judged captions blind: 15 came from our pipeline and 5 were written by a trained musician. The submission reported only the pipeline captions (Table 1); the revision adds the comparison.
Table R3. Percentage of ratings per level. 2 of the 236 trials have no rating.
| Rating | Pipeline captions (%) | Human-written captions (%) |
|---|---|---|
| Very bad (1) | 5.75 | 1.67 |
| Bad (2) | 9.20 | 6.67 |
| Fair (3) | 18.39 | 13.33 |
| Good (4) | 42.53 | 40.00 |
| Very good (5) | 24.14 | 38.33 |
| Number of ratings | 174 | 60 |
We observe that 85.1% of pipeline ratings are fair or better, against 91.7% for the musician’s captions. We see that our pipeline captions are close to ones written by an expert musician. The study did not isolate the samples where Qwen and Mistral disagree, we report this as a limitation. Note that the edit instructions are deterministic: they come from a pool of human-written edit-aware captions.
(Q6) The Hugging Face viewer does not display correctly.
Thank you for noting this, we have corrected it. The dataset page now shows sme-test and sme-real samples directly, without downloading the JSON files ( https://huggingface.co/datasets/anonymous-account-submission/SME-Bench).
Thank you for the comments and additional experiments. The listening study on system outputs directly addresses my main concern about whether the metrics reflect human judgments. The comparison on shared edit types also strengthens the case that findings from SMEBench transfer to recorded music, particularly for LPAPS. I appreciate the check showing that excluding the three less audible operations leaves the rankings unchanged. Overall, the new results address my main concerns, and I am raising my score to 3 (weak accept). Good luck with the submission.
Thank you very much for your constructive review, and raising your score!
Reply to reviewer ouCx, part 1
We thank the reviewer for recognizing the value of "a relatively large benchmark with paired ground-truth targets". Their questions on human judgments, real audio and the weak operations led to new results, reported below.
(W1) The figures are difficult to follow.
We agree to simplify them. We improved the readability of Figure 1 and made the caption more precise. We also made some changes in the revision to make the manuscript clearer.
(W2, Q1) Do evaluation and target-based metrics agree with human judgments on system outputs?
We ran an additional user study, where we deployed a MUSHRA-style listening test with 8 participants on 14 samples. Participants rated the edits of the six systems, plus two anchors (the ground-truth edit as upper anchor and the unedited source as lower anchor), from 0 to 100 on two criteria:
source preservation (“is everything that should not change still the same as in the source: same audio scene, same sound quality?”);
edit accuracy (“is the requested edit clearly audible and does it match the edit request?”).
Table R2 gives the correlation between the human rankings (anchors excluded) and each metric, at the level of the six systems and per sample (mean over samples, Wilcoxon test). Below we summarize the findings:
Table R2. Kendall τ between human ratings (Preservation and Edit Accuracy) and each metric (p-value in parentheses).
Audio-to-audio metrics agree with human ratings on preservation, but only the target-based metrics have a significant per-sample correlation with edit accuracy. The text metric does not track either criterion. The number of participants is small, so we treat these results as supportive evidence.
(W3, Q3) Construction choices lack ablations.
A full ablation is not feasible, we report what the data supports.
While we agree with the reviewer that it would be preferable to ablate all the design choices, this constitutes a very large combinatorial search space, and we therefore followed choices that respect musical and machine learning principals.
We would like to also note that the system rankings agree between two independently constructed datasets (Table R1), and removing the three operations with the weakest human scores does not modify the ranking orders on any of the two (Table R2 in our reply to Reviewer EPLR).