- Rating
- 4(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a large-scale audio editing benchmark, in a similar spirit to common audio understanding benchmarks such as MMSU, MMAU, etc. The data curation pipeline is rigorous. In all, it seems like a useful contribution to the audio editing and generation community. There is a suggested metric included as well, which relies heavily on LLM-as-a-judge.
Critical strengths
- Useful and broad audio editing dataset, with baselines already included.
- Common datasets are very important to push the field forward.
- Well thought out data curation pipeline, with human annotation.
- May be appropriate in a datasets and benchmarks track.
Critical weaknesses
- I could not find a user study validating the approach for the evaluation metric.
- LLM-as-a-judge with Qwen3Omni in my experience is not very valuable, as the quality of its instruction following is below that of frontier models. A stronger version of this paper would include a more rigorous approach to creating or proposing a judge. While the rubric is useful, it would be good to compare it against a human listener, as well as frontier audio models such as Gemini to make sure it correlates.
- Could not find audio examples on a companion page. Would be helpful to listen in rebuttal period.
- Some objective metrics might be good, such as CLAP score etc.
- The setup is a bit circular - Qwen3Omni is used for both the captions as well as the judge.
Questions for authors
- Share some audio examples so we can listen to the dataset.
- Resolve issues for evaluation. The paper would be greatly strengthened by improved evaluation.
- With better validation that the dataset is indeed useful, and the benchmarks and evaluation is meaningful, I would raise my rating.
Reminder: Discussion Phase Ends November 18 (11:59 PM AoE)
Hi reviewers, a quick reminder that the discussion phase ends on November 18 at 11:59 PM AoE. Please take some time to read the author responses, engage in further discussions if you have any remaining questions or concerns, and update your reviews and scores as appropriate. Thank you!