acceptodds
Under review as a conference paper at ICLR 2027

SED-RECIPES: UNDERSTANDING THE IMPACT OF MASKING STRATEGIES, SCHEDULES AND DECODERS ON SOUND EVENT DETECTION

Abstract

Recent work has shown that sound event detection (SED) benefits from in-domain, self-supervised pretraining of the decoder. Different works propose different masking strategies, decoders and curricula, but no work has systematically compared the different recipes. In this work, we conduct one of the largest systematic evaluations of different configurations. In addition, we introduce three novel masking strategies that are inspired from the inductive biases of SED and new evaluation metrics that elucidate the impact of polyphonicity and event duration. Our findings show that shorter events remain challenging and that saliency-guided masks can improve per- formance compared to standard, randomized masking.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Deep Learning, Self-supervised Learning, Sound Event Detection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_o1ECOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

Review of "SED-Recipes — Understanding the Impact of Masking Strategies, Schedules and Decoders on Sound Event Detection"

This paper presents a systematic empirical benchmarking of Sound Event Detection (SED) training recipes on the DESED dataset using a fixed PaSST backbone. The authors evaluate various masking geometries, saliency-guided masks, training curricula, and modern sequence decoders, while introducing a novel evaluation protocol that does stratification by event duration and sound event polyphony.

The paper reads like a system report, where the authors ran some masking tests on the DESED dataset. No novel insights are found and the motivation is also unclear. The paper writing is clear, but due to the missing motivation and a real conclusion. As the conclusion reads: "Using a rigorous evaluation framework, we were able to show that using temporal decoders with a standard masking strategy can surpass more complicated masking strategies.", which tells the reader nothing new, i.e., using the standard masking strategy is sufficient. Thus, I lean towards a reject for ICLR.

Critical strengths
  • The paper is practical, providing a comprehensive comparison of masking schemes and modern decoders, accompanied by open-source code for the DCASE/SED community.
  • Transparent evaluation by using stratification to ablate duration and sound event polyphony.
Critical weaknesses
  • The work is a system paper, the authors just run different masking strategies that are only applicable for the one framework they use. The paper could just be a github repository to show the most optimal masking strategy for that specific task on that specific dataset.
  • The paper has no real impact or result. After running all the recipes, the conclusion does not point out any benefits of some recipes particularly and just states rather common knowledge in the field: Short overlapping events are harder to detect than long individual ones.
  • Single dataset (DESED) no evidence recipe rankings transfer cross-corpus, which limits the findings.
  • Results in Table 2. show that different masking strategies give different results, but the benefits are limited. No SSL baseline achieves 0.523 PSDS1 score, while the majority of results achieve ~ 0.53 a very limited gain.
  • Overall, the paper is unfit for ICLR: a systems-style ablation of training recipes for one fixed framework, with nothing that generalizes beyond that pipeline — better suited to a DCASE/IEEE-style venue.
  • The paper overall seems bloated: Many formulations for rather and masking approaches that overall have little impact on final performance if any.
Questions for authors

I have no real idea on any potential questions about the work

4 comments
AuthorsOpenReview ↗

Response to Reviewer o1EC

Thank you for stating the objection plainly.

Revision: The revised conclusion is narrower. Inside the clip-bootstrap intervals, mask family does not rank PSDS1. The separation we can support is a metric shift: TEM and EA raise event F1 to 0.617 and 0.609 and do not lead PSDS1 by a large margin. Short overlapped events stay hard for every mask. The study is one frozen PaSST pipeline on DESED. We do not claim the ranking transfers to another corpus here. The method section is still long relative to the size of the gaps. We did not add further recipes in the revision.

Reviewer_o1ECOpenReview ↗

Do not understand the response.

Dear Authors, I can't properly understand the reponse at all. What does

The revised conclusion is narrower.

mean in this context? Which conclusion? What is narrower? What does this reference to?

Inside the clip-bootstrap intervals, mask family does not rank PSDS1.

What does this mean?

The separation we can support is a metric shift: TEM and EA raise event F1 to 0.617 and 0.609 and do not lead PSDS1 by a large margin.

Whis separation are the authors referring to?

We do not claim the ranking transfers to another corpus here.

Correct, whis is a major downside of the entire paper, since its too narrow in scope.

AuthorsOpenReview ↗

Follow-up response

Thank you for the response.

mean in this context? Which conclusion? What is narrower? What does this reference to?

The experimental results can't support the ranking between mask types and decoder architectures as initially proclaimed based on the PSDS1 score ranking.

What does this mean?

For each PSDS1 score intervals the new mask types values are within the No-SSL and Block intervals, hence the revision questions the PSDS1 ranking favoring a "separation" argument between the PSDS1 and eF1 metrics.

Reviewer_o1ECOpenReview ↗

Verdict

The authors did not adress my main concerns: The work is too shallow and limited in terms of its experimetns, results and datasets used. Authors only addressed in this rebuttal minor wording adjustments, not extensive novel and additional experimentation.

My verdict is still 1.

Reviewer_x9efOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper sets out to investigate what the impact of different masking strategies are on SED. SED is inherently a challenging task because there is a high degree of superposition between sounds (polyphonicity), a high degree of variability within classes, and it has far smaller datasets than vision/language domains. A number of "recipes" are presented for SSL that aim to improve the accuracy of SED: the question is, which ones of them actually make a useful delta. This problem is made harder because sounds are highly inhomogenous - they have different durations, and different spectro-temporal extents. The paper takes a baseline transformerXL architecture and applies different masking strategies to try and disentangle the various impacts of masking strategies, from data agnostic (e.g. random) to data informed (e.g. saliency masking).

Critical strengths
  • Solid evaluation of different masking approaches: the impact of different masking approaches on transformerXL approaches was conducted well, and appears quite rigorous.
  • Useful tool: The codebase is a useful tool to the community - it will allow extended ablation tests across various architectures and datasets.
Critical weaknesses
  • The effects are minimal: this paper shows that masking strategy plays a very limited role compared with choice of decoder. Whilst this result is useful, I don't find it that suprising. Masking often acts as a small boost to improve performance, but it is often limited compared with the choice of the entire architectural stack.
  • It would be useful to have compared directly against SED strategies that explicitly introduce new masking strategies like JitTTER (although your TEM is "inspired" by it) and PMAM (which is built on a similar PaSST-CNN stack). I find it peculiar that these strategies have not been explored in a paper about masking recipes, when they themselves are masking recipes.
  • It would have been useful to explore deeper e.g. on different datasets (even synthetic) to more objectively tease out precisely when masking recipe/approach matters, and when it doesn't. The current findings are limited in their utility as the speech class dominates the mass transfer diagram (which makes sense, as it is a temporally and spectrally diverse class). Is there some point (e.g. if the classes are w-disjoint spectro-temporally, or of radically different durations) where masking strategies play an oversized role? This would help the reader better understand when a masking strategy matters and when it doesn't. This could also help inform the design of future architectures and masking strategies.
  • Reporting full benchmarks on existing methods on the same dataset would help e.g. Schmid et al. 2024 (MULTI-ITERATION MULTI-STAGE FINE-TUNING OF TRANSFORMERS FOR SOUND EVENT DETECTION WITH HETEROGENEOUS DATASETS). This would help to place these masking recipe results within the broader context.

Minor: Some of your references include the method name in the hyperlink e.g.: Mamba Gu & Dao (2024), BiMamba Gao & Yin (2025), and Hydra Hwang et al. (2024) l240: We adapt AttMask Kakogeorgiou et al. (2022)

Questions for authors
  • I would be interested in knowing when masking recipes help, and when they don't help. The general conclusion of the paper at this point is that decoders matter more. But I think the answer is more nuanced than that. I feel that there is a good presentation of recipes, but a lack of a deeper dive into why and how.
3 comments
AuthorsOpenReview ↗

Response to Reviewer kx9ef

Thank you for the review. We revised the claim, that asks, when a masking recipe helps.

Inside this pipeline, mask family does not rank PSDS1. The clip-bootstrap intervals overlap: Block is 0.531 [0.510, 0.559], AttMask 0.535, and ERB 0.533. TEM and EA do move event F1, to 0.617 and 0.609, and TEM's PSDS1 (0.521) falls below no SSL (0.523). EH-MAM (0.516) also falls below that line. So a mask helps when the metric is collar F1 and the sampler is TEM or EA. It does not help as a PSDS1 upgrade over block masking.

Decoder point estimates span a larger range, from Conformer at 0.495 to BiGRU and BiMamba at 0.540, but those models also differ in parameter count, and the intervals still overlap. They were discussed in their role of being the active reconstruction network during the masked pretraining as a complementary direction of pipeline design research. The main experiments were performed on a transformer decoder based on previous research.

On duration, short overlapped events stay hard for every mask. The appendix bins show the mask gaps at their largest on short and mid durations, not on the long classes. Speech dominates the co-occurrence graph because it has the most annotated duration and overlaps every class. We did not test a regime of spectro-temporally disjoint classes, and we did not add a second corpus. Both are still open and will be adressed in future research.

JiTTER and PMAM are in the related work as different recipes, which were the main comparison but ultimately differ in architecture and schedule so we did not include them here. JiTTER replaces masked prediction with a hierarchical shuffle and reports a 5.89% PSDS gain over MAT-SED under that protocol. The shuffle inside TEM is inspired by JiTTER. PMAM reports PSDS1 0.625 with prototype pseudo-labels on its own DESED setup. Our best score here is 0.540, against a MAT-SED reference of 0.587 from a different architecture and schedule. References:

Reviewer_x9efOpenReview ↗

A convoluted response

Thank you for taking the time to respond. Similar to the paper itself, the response is dense and not particularly clear. Your response largely repeats the last few lines of the conclusion along with some numbers - this does not really persuade me of the value/merits of your analysis/approach. I do think there are fundamental issues here - the contribution is an analysis of recipes, but incompletely done in terms of problem settings and corpora extent. The difference between all the recipes is marginal so it does not help the reader to really understand when to use a particular recipe (or whether the recipe even matters very much). Are different recipes cheaper (in terms of convergence or computational cost)?

AuthorsOpenReview ↗

Follow-up response

Thank you for the response and pointing out the flaws of the contribution.

  • "Are different recipes cheaper": The schedule is the same for all of them. Measured in pretraining hours, ERB, Random, Block, and InvBlock take 1.5 to 2.0 hours. EA takes 4.8 hours, and AttMask takes 13.4.

  • "The contribution is [...] incompletely done in terms of problem setting and corpora extent" We agree with the completeness concern. The PSDS1 ranking, lack of corpora extent, and the scope of the study do not describe an impactful enough benefit of these mask types.

Thank you again for the review.

Reviewer_kpJjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SED-Recipes studies how masking strategies, schedules, and temporal decoders affect sound event detection (SED). On DESED, it separately compares ten masking strategies and nine decoders using a PaSST-based pipeline. Decoder pretraining reconstructs masked features from a frozen PaSST encoder, and downstream finetuning adds a CNN branch. Evaluation contains PSDS, event-based F1, and duration/co-occurrence scores.

This study can be organized around four research questions:

-R1: How do geometric and saliency-guided masking strategies affect SED performance?

-R2: Does changing masking difficulty or structure during pretraining improve downstream performance?

-R3: How do temporal decoder types compare within the same training pipeline?

-R4: How does performance vary with event duration and class co-occurrence?

The corresponding methods are:

-M1: Geometric and saliency-based masking comparisons with a fixed Transformer-XL decoder, including three proposed SED-inspired strategies: temporal entropy masking (TEM), event-aware masking (EA), and energy-reweighted block masking (ERB);

-M2: Within the 10 strategy comparison, EH-MAM and HPM vary placement difficulty, but HierBlock changes block width during pretraining;

-M3: 9 decoders that contain Transformer, RNN, SSM, and xLSTM families, evaluated with standard block masking;

-M4: Duration-binned PSDS1 analyses and a directed graph of differences in correct ownership rates conditioned on class co-occurrence.

Critical strengths

Regarding the contribution and validation,

-R1: Comparing masking families within a common pipeline provides useful reference points for SED recipe design. TEM and EA report higher eF1 than fixed Block;

-R2: The curriculum comparisons are informative. EH-MAM and HierBlock underperform fixed Block on PSDS1, while HierBlock improves eF1;

-R3: The metric-dependent rankings are practically useful. BiGRU and BiMamba both achieve PSDS1 of .540, but their event-based F1 scores differ substantially;

-R4: Examining duration and class co-occurrence beyond aggregate scores is a worthwhile diagnostic direction, and the class-level results provide material for further investigation.

Critical weaknesses

-R1: Masking comparisons conflate placement, budget, corruption, and supervision. Block masks approximately 0.75 of frames versus InvBlock’s 0.25; TEM adds temporal permutation, and EA uses available strong labels. Therefore, results support recipe-level comparisons more directly than isolated masking effects. I think clip-level temporal reasoning from InvBlock also needs evidence, since nearby latents may suffice for reconstruction. The AttMask ablation interpretation conflicts with the stated sampler definition (Sec. 4.2; Appendix B.1, Table 3). Under the stated default r=0.75 and T=1000, max(rT, floor(ρT)) gives the same 750-frame pool size for ρ=0.30, 0.50, 0.75. Nevertheless, the PSDS1 drop from 0.535 to 0.520 at L=6 is attributed to pool expansion, which the stated formula does not produce. The actual sampling settings and this interpretation require clarification.

-R2: Schedule effects are not isolated from other changes. HPM realizes a mask fraction of approximately 0.35, so its comparison with Block also changes masking budget and geometry. The authors acknowledge budget limitations, but controls matching the realized budget are needed to attribute differences to the curriculum;

-R3: Decoder capacity and uncertainty limit architectural conclusions. Transformer-XL, BiMamba, and BiGRU have 12.4M, 22.6M, and 28.3M parameters, respectively. The acknowledged capacity limitation and BiMamba capacity ablation do not fully resolve this confound. Since differences between PSDS1 and eF1 alone do not identify boundary errors as their cause, the proposed interpretation needs targeted error analysis or more cautious wording. Across the study, results use seed 21, and clip-bootstrap intervals do not capture training variability. Gains of .004 PSDS1 for AttMask over Block and .009 for BiMamba over Transformer-XL need direct robustness checks; overlapping individual intervals alone do not establish nonsignificance. Appendix B reports hyperparameter sweeps on the real validation set used for scoring; separation of model selection and final evaluation needs clarification;

-R4: The duration and polyphony analyses could provide diagnostic insight, but evaluation definitions and mechanistic interpretations need stronger support. (1) Figure 6 labels its axis as median event duration, and p. 16 describes assigning classes by their median, yet every one-second bin has scores. If the validation medians in Figure 8 are used, bins such as [2,3), [3,4), and those within [5,9) contain no classes. The binning definition and reported medians therefore require reconciliation. Figure 8 also leaves handling of out-of-bin ground truth, predictions, false positives, and evaluation duration insufficiently specified; this does not establish a scoring bug. (2) Class-median comparisons cannot isolate duration from class identity, and the authors acknowledge different within-class trends. The blanket convergence claim is questionable: Figure 6’s score range across all samplers is 0.059 for [0,1) seconds and 0.103 for [9,10] seconds. (3) Table 1 reports actual temporal overlap, whereas the Appendix F graph conditions on clip-level co-occurrence. Longest-duration ownership underrepresents brief events as owners, and reduced correct ownership does not demonstrate transfer to the conditioning class. The appendix acknowledges these limitations and the lack of statistical testing; its inclusion threshold permits conditional groups as small as five clips. The graph supports exploratory associations, but does not isolate class-specific causal interference or establish the training-bias propagation claimed in the conclusion.

Questions for authors

Suggestions:

  • [Appendix E, Figures 6–8] Reconcile the binning definition with the reported medians; specify bin counts and treatment of out-of-bin predictions/ground truth and evaluation duration. Regenerate affected figures if needed and distinguish within-class duration trends from between-class differences.

  • [Section 4.2, Appendix B.1, Table 3] Report actual pool sizes and realized mask fractions; reconcile the formula, implementation settings, and ablation interpretation.

  • [Sections 4–5, Tables 2–4] Prioritize representative controls for the main claims: matched masking budgets/corruption, an EA variant using the same label-free proxy across clips, and comparable decoder capacities.

  • [Appendices A–B] Clarify hyperparameter, checkpoint, threshold, and post-processing selection, including any evaluation split held separate from selection.

  • [Section 5.5, Appendix F] Report group sizes and edge uncertainty/sensitivity. Limit conclusions to co-occurrence associations unless temporal overlap and clip composition are controlled.

  • [Minor clarification, not affecting my rating] For ERB, “silence-aware block masking” would describe the sampling rule more directly.

2 comments
AuthorsOpenReview ↗

Response to Reviewer kpJj

Thank you for the suggestion list. We went through it in that order.

On the duration figures, median-duration panels assign each class to the bin of its median, so only the bins that contain a class median are filled. True-duration panels bin individual ground-truth events, which is why a one-second axis can have a score in every bin. We do not read the median plot as a within-class duration effect. A separate rule for out-of-bin false positives is not in this revision.

On AttMask, the pool size is k = min(T, max(round(r T), round(rho T))), with budget n = round(r T). At r = 0.75, rho in {0.30, 0.50, 0.75} selects the same top-750 frames. The values 0.534 and 0.535 fit that collapse. The 0.520 at rho = 0.75 uses the same mask, so it is not pool expansion. Realized budgets stated in the paper are about 0.25 for InvBlock and about 0.35 for HPM. We did not add a realized-rate column for every row.

On the controls, we agree the tables compare recipes, not isolated factors. Block, InvBlock, and HPM still differ in budget. TEM adds a permutation. EA still uses strong labels, with spectral flux only as the unlabeled fallback. A label-free EA run, a budget-matched block run, and a capacity-matched decoder are not in this revision. The limitations section says so for the budget and for the partial BiMamba capacity check. We also have no control showing that InvBlock needs clip-level context rather than nearby frames.

On uncertainty, every reported run uses seed 21. The checkpoint is the best PSDS1 on the same real validation split we score. There is no held-out selection split. The clip bootstrap is not training variability, and an overlapping interval is not a significance test. We no longer describe the 0.004 and 0.009 PSDS1 gaps as improvements, and we do not claim that the PSDS1 / event-F1 gap is specifically a boundary error.

On the co-occurrence graph, edges are associations under clip-level co-occurrence, not the temporal overlap in Table 1. Ownership is the longest class, groups can be as small as five clips, and we report no edge intervals. We removed the training-bias sentence. The figure caption still says "stolen mass". The appendix is the wording we stand behind.

Thanks for the response and for answering each point directly. Several interpretations I questioned are now withdrawn (boundary errors, pool expansion, training-set bias), and the 0.004–0.009 gaps are no longer called improvements. I also agree with the revised statement that the masking strategies cannot be ranked on PSDS1.

My main concern remains, though. There are still no controlled comparisons (e.g., a budget-matched block mask or a capacity-matched decoder), all runs use one seed, and the checkpoint is picked on the same validation split that is reported. I understand these may not fit in a rebuttal period, but without them the remaining claims are hard to verify.

One follow-up on AttMask. If ρ = 0.30, 0.50, and 0.75 give the same mask, then 0.534 and 0.535 make sense, but I do not see where 0.520 comes from. Was anything else different in that run? If not, the difference is nearly as large as the whole spread among masking strategies in Table 2 (0.516 to 0.535).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revision summary

Thank you for the reviews! They changed the claim, and the PDF now follows them.

Inside the clip-bootstrap intervals, mask family does not rank PSDS1. TEM and EA raise event F1 to 0.617 and 0.609, and TEM's PSDS1 (0.521) falls below no SSL (0.523). AttMask (0.535) and ERB (0.533) sit inside the Block interval [0.510, 0.559]. BiGRU and BiMamba meet at PSDS1 0.540 and separate on event F1 (0.633 against 0.569).

HPM is block width 16 at nominal rate 0.75, realized near 0.35. For AttMask, rho in {0.30, 0.50, 0.75} shares one top-750 mask at r = 0.75, so that drop is not an expanded pool. We did not add budget-matched block controls. The study is still one frozen PaSST pipeline on DESED.

Point-by-point replies are posted under each review.

AuthorsOpenReview ↗

Official Comment

Thank you for the reviews and follow-ups.

The comments agree on the state of the submission. The mask study does not answer: when does a mask help, and in which data. Furthermore, mask budget and decoder capacity matching results are missing.

We accept the recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.