acceptodds
Under review as a conference paper at ICLR 2027

MMAE: A Massive Multitask Audio Editing Benchmark

Abstract

We introduce MMAE, a Massive Multitask Audio Editing benchmark, serving as the first comprehensive evaluation testbed designed for general-purpose instruction-based audio editing. Spurred by the shift toward intelligent creation, interactive editing has rapidly expanded from visual domains, pioneered by models like Nano-banana 2 for images and Gemini-Omni for video, into audio. However, the current evaluation infrastructure lags severely, remaining highly fragmented and restricted to specific subdomains or basic operations. Unlike existing benchmarks that are limited in scope, MMAE extends to a broad spectrum of real-world scenarios, encompassing 7 distinct audio modalities, including sound, speech, music, and their mixtures. Furthermore, we establish a comprehensive taxonomy spanning 6 levels of task complexity, from basic modifications to multi-hop reasoning and multi-round editing, 2 levels of granularity, and 8 distinct operation types. Meticulously curated through human-agent collaboration, MMAE comprises 2,000 high-fidelity samples paired with a pioneering rubric-based evaluation framework. By decomposing free-form tasks into 17,741 verifiable criteria, this robust rubric-based paradigm enables a precise, multi-dimensional assessment of both instruction following and context consistency. Our extensive evaluation of leading models reveals that current systems remain far from achieving reliable edits. Strikingly, the Exact Match Rate (EMR) consistently falls below 5% and plummets to an absolute 0% in complex, mixed-modality tasks, exposing critical bottlenecks in precise execution and structural robustness. We hope MMAE will serve as a catalyst for future advances in the intelligent creation community, providing a clear diagnostic roadmap and establishing a standardized, long-lasting evaluation paradigm for next-generation audio editing systems.

Primary area:
datasets and benchmarks
Keywords:
audio editing, benchmark, multitask, instruction following, evaluation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_7ht4OpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a large-scale audio editing benchmark, in a similar spirit to common audio understanding benchmarks such as MMSU, MMAU, etc. The data curation pipeline is rigorous. In all, it seems like a useful contribution to the audio editing and generation community. There is a suggested metric included as well, which relies heavily on LLM-as-a-judge.

Critical strengths
  • Useful and broad audio editing dataset, with baselines already included.
  • Common datasets are very important to push the field forward.
  • Well thought out data curation pipeline, with human annotation.
  • May be appropriate in a datasets and benchmarks track.
Critical weaknesses
  • I could not find a user study validating the approach for the evaluation metric.
  • LLM-as-a-judge with Qwen3Omni in my experience is not very valuable, as the quality of its instruction following is below that of frontier models. A stronger version of this paper would include a more rigorous approach to creating or proposing a judge. While the rubric is useful, it would be good to compare it against a human listener, as well as frontier audio models such as Gemini to make sure it correlates.
  • Could not find audio examples on a companion page. Would be helpful to listen in rebuttal period.
  • Some objective metrics might be good, such as CLAP score etc.
  • The setup is a bit circular - Qwen3Omni is used for both the captions as well as the judge.
Questions for authors
  • Share some audio examples so we can listen to the dataset.
  • Resolve issues for evaluation. The paper would be greatly strengthened by improved evaluation.
  • With better validation that the dataset is indeed useful, and the benchmarks and evaluation is meaningful, I would raise my rating.
Reviewer_SyMhOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces MMAE, a comprehensive benchmark for instruction-based audio editing across speech, music, sound effects, and their mixtures. The benchmark contains 2,000 samples and 17,741 evaluation rubrics, covering diverse editing operations and task complexities, including multi-hop reasoning and multi-round editing.

The key contribution is a rubric-based evaluation framework that separately measures instruction-following accuracy and content preservation. Experiments on five representative audio editing models reveal significant limitations in existing systems, particularly for complex and mixed-modality editing tasks, with overall exact-match rates below 5%.

Critical strengths
  1. Timely and well-motivated benchmark. General-purpose audio editing is an increasingly important research direction, while existing evaluation benchmarks remain fragmented. MMAE addresses a meaningful gap in the field.

  2. Comprehensive task coverage. The taxonomy spans different audio modalities, editing operations, and complexity levels. I particularly appreciate the inclusion of challenging tasks such as multi-round editing, multi-hop reasoning, and mixed-modality editing.

  3. Fine-grained and interpretable evaluation. The rubric-based framework is a valuable contribution, as it separately evaluates instruction following and content preservation rather than relying solely on conventional audio quality metrics.

  4. Thorough experimental analysis. The evaluation across representative models provides useful insights into current limitations. The detailed examples and visualizations also effectively illustrate the benchmark's diversity and evaluation design.

Critical weaknesses
  1. Reliability of the automated evaluation. The benchmark relies heavily on Qwen3-Omni as the evaluation judge. Although repeated evaluations and majority voting improve stability, additional validation against human judgments would strengthen the credibility of the proposed metrics.

  2. Fairness of model comparisons. Some models are evaluated on the full 2,000 samples, while others are evaluated on only 801 samples due to input-duration limitations. Reporting results for all models on the same subset would enable more direct comparisons.

  3. Further analysis of evaluation metrics. The extremely low Exact Match Rate (EMR) is interesting, but its sensitivity to the number of rubrics per sample deserves further investigation. Additional analysis could clarify whether low EMR primarily reflects model limitations or the strictness of the evaluation protocol.

Questions for authors

See the weaknesses.

Reviewer_xb29OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work propose MMAE, a audio editing benchmark containing 2000 samples with a set of rubric-based evaluation methods. It benchmarks Step-Audio-EditX, Ming-UniAudio, MMEdit, Audio-Omni, and SmartDJ in terms of instruction following and consistency.

Critical strengths

S1. A new benchmark dedicated on instruction-based audio editing tasks, covering different modality (speech/music/sound effect) and settings.

S2. A thorough discussion of the evaluation settings and rubric design is provided, but would benefit from more details in the main text or supplementary material.

Critical weaknesses

W1. Experiment setup. The evaluation sets differ across models, so Table 2 may give a misleading impression of an apples-to-apples comparison. MMEdit, Audio-Omni, and SmartDJ are evaluated only on <10s inputs (801 out of 2,000 samples), while Step-Audio-EditX and Ming-UniAudio are evaluated on the full set. Also, please make this distinction clear and specify whether the reference baselines use the full set or the shorter-input subset.

W2. The baseline selection and the performance. Please clarify the rationale for using Identity and Gaussian Noise as reference baselines. Are they appropriate for audio editing system? The Noise baseline’s instruction-following score appears surprisingly high for an output that preserves no meaningful content (an IFR of 32.08% vs 28.77% from Ming-UniAudio). Please explain which tasks or verification rubrics account for this result and whether it indicates a limitation of the metric. In addition, the explanation that Identity succeeds on extraction tasks or multi-round scenarios where the final output matches the original also needs clarification. It especially gets an EMR of 7.69 on Sound-Speech set. Do these cases meaningfully assess editing ability?

W3. More details for clarification. Please describe the structure of the brainstorming sessions, including the number of topics and their coverage. Also, it is recommended to clarify the licensing and usage permissions for audio collected from online videos.

Questions for authors

See Weaknesses Section.

Comments on the paper

1 comment
Area_Chair_sSwhOpenReview ↗

Reminder: Discussion Phase Ends November 18 (11:59 PM AoE)

Hi reviewers, a quick reminder that the discussion phase ends on November 18 at 11:59 PM AoE. Please take some time to read the author responses, engage in further discussions if you have any remaining questions or concerns, and update your reviews and scores as appropriate. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.