- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Review of "SED-Recipes — Understanding the Impact of Masking Strategies, Schedules and Decoders on Sound Event Detection"
This paper presents a systematic empirical benchmarking of Sound Event Detection (SED) training recipes on the DESED dataset using a fixed PaSST backbone. The authors evaluate various masking geometries, saliency-guided masks, training curricula, and modern sequence decoders, while introducing a novel evaluation protocol that does stratification by event duration and sound event polyphony.
The paper reads like a system report, where the authors ran some masking tests on the DESED dataset. No novel insights are found and the motivation is also unclear. The paper writing is clear, but due to the missing motivation and a real conclusion. As the conclusion reads: "Using a rigorous evaluation framework, we were able to show that using temporal decoders with a standard masking strategy can surpass more complicated masking strategies.", which tells the reader nothing new, i.e., using the standard masking strategy is sufficient. Thus, I lean towards a reject for ICLR.
Critical strengths
- The paper is practical, providing a comprehensive comparison of masking schemes and modern decoders, accompanied by open-source code for the DCASE/SED community.
- Transparent evaluation by using stratification to ablate duration and sound event polyphony.
Critical weaknesses
- The work is a system paper, the authors just run different masking strategies that are only applicable for the one framework they use. The paper could just be a github repository to show the most optimal masking strategy for that specific task on that specific dataset.
- The paper has no real impact or result. After running all the recipes, the conclusion does not point out any benefits of some recipes particularly and just states rather common knowledge in the field: Short overlapping events are harder to detect than long individual ones.
- Single dataset (DESED) no evidence recipe rankings transfer cross-corpus, which limits the findings.
- Results in Table 2. show that different masking strategies give different results, but the benefits are limited. No SSL baseline achieves 0.523 PSDS1 score, while the majority of results achieve ~ 0.53 a very limited gain.
- Overall, the paper is unfit for ICLR: a systems-style ablation of training recipes for one fixed framework, with nothing that generalizes beyond that pipeline — better suited to a DCASE/IEEE-style venue.
- The paper overall seems bloated: Many formulations for rather and masking approaches that overall have little impact on final performance if any.
Questions for authors
I have no real idea on any potential questions about the work
4 comments
Do not understand the response.
Dear Authors, I can't properly understand the reponse at all. What does
The revised conclusion is narrower.
mean in this context? Which conclusion? What is narrower? What does this reference to?
Inside the clip-bootstrap intervals, mask family does not rank PSDS1.
What does this mean?
The separation we can support is a metric shift: TEM and EA raise event F1 to 0.617 and 0.609 and do not lead PSDS1 by a large margin.
Whis separation are the authors referring to?
We do not claim the ranking transfers to another corpus here.
Correct, whis is a major downside of the entire paper, since its too narrow in scope.
Follow-up response
Thank you for the response.
mean in this context? Which conclusion? What is narrower? What does this reference to?
The experimental results can't support the ranking between mask types and decoder architectures as initially proclaimed based on the PSDS1 score ranking.
What does this mean?
For each PSDS1 score intervals the new mask types values are within the No-SSL and Block intervals, hence the revision questions the PSDS1 ranking favoring a "separation" argument between the PSDS1 and eF1 metrics.
Verdict
The authors did not adress my main concerns: The work is too shallow and limited in terms of its experimetns, results and datasets used. Authors only addressed in this rebuttal minor wording adjustments, not extensive novel and additional experimentation.
My verdict is still 1.
Response to Reviewer o1EC
Thank you for stating the objection plainly.
Revision: The revised conclusion is narrower. Inside the clip-bootstrap intervals, mask family does not rank PSDS1. The separation we can support is a metric shift: TEM and EA raise event F1 to 0.617 and 0.609 and do not lead PSDS1 by a large margin. Short overlapped events stay hard for every mask. The study is one frozen PaSST pipeline on DESED. We do not claim the ranking transfers to another corpus here. The method section is still long relative to the size of the gaps. We did not add further recipes in the revision.