- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the applicability of epiplexity to audio modeling by estimating unconditional and conditional epiplexity across various audio datasets tokenized with DAC, EnCodec, and X-Codec. The authors report indicate generally weak relationships between epiplexity and model performance that depend on the tokenizer and evaluation setting, with conditional epiplexity showing more consistent positive associations with classification performance.
Critical strengths
Introducing epiplexity to audio domain
This work provides an empirical starting point for understanding how epiplexity behaves on a diverse collection of audio datasets and how it relates to model generalization.
Critical weaknesses
I assigned the lowest confidence score because of my limited familiarity with epiplexity. However, drawing on my background in audio machine learning, I identified several concerns regarding the paper’s experiments.
1. Insufficient justification of re-purposing dataset tasks and evaluation protocols
In Figure 4, the authors investigate aggregated correlations between conditional epiplexity and classification performance across datasets originally designed for substantially different tasks. For example, STARSS23 and LOCATA were developed for sound event localization and detection (SELD) and acoustic source localization/tracking, respectively. Furthermore, SynTheory datasets (Chords, Intervals, Notes, and Scales) were designed around specific music-theoretic concepts.
However, Appendix A (Table 2) and Appendix B.2 (Table 10) specify origin for STARSS23, task for LOCATA, and midi program name for the SynTheory subsets as classification targets. According to the provided implementation, STARSS23's origin refers to the recording institution (Sony or TAU: Tampere University), LOCATA's task refers to one of its six recording scenarios, and SynTheory's midi program name represents instrument identity. These classification tasks differ substantially from the datasets' original research objectives, and the manuscript does not adequately justify their use as alternative evaluation tasks.
For example, STARSS23 conventionally evaluates sound event detection and localization using metrics such as ER20, F20, localization error (LE), and localization recall (LR). In contrast, Tables 11–12 report weighted F1 and classification accuracy for recording-origin classification, which cannot be interpreted as performance on the original SELD task. Similarly, classifying LOCATA's recording scenarios does not measure acoustic source localization or tracking performance. Moreover, the provided implementation selects only the first audio channel for DAC, EnCodec, and X-Codec, thereby discarding the multichannel directional information contained in FOA recordings.
The manuscript does not adequately justify these alternative classification targets or establish whether Figure 4’s correlations are robust to task selection, limiting the support this central experiment provides for its broader conclusions about audio-model performance and generalization.
2. Limited practical implications for audio dataset selection
In the Introduction, the authors aim to investigate "What audio datasets might be the best to train on from the perspective of epiplexity and generalization?" (page 2, L61-62). While the empirical findings may provide useful insights into the behavior of epiplexity in the audio domain, their practical implications for dataset selection remain unclear. In particular, the manuscript does not establish the practical value of measured epiplexity as a criterion for selecting training datasets to improve downstream generalization.
Questions for authors
Unclear connection between spatial reasoning and the experimental evaluation
In the Introduction (page 1, L47–51), the authors suggest that improving the generalization capabilities of audio models could lead to better spatial reasoning, contrasting this with the limitations of text models. However, although the manuscript includes several spatial-audio datasets, it does not evaluate spatial reasoning or localization capabilities, such as source azimuth, elevation, or distance prediction. Also, the results and discussion do not establish how the reported epiplexity measurements relate to spatial reasoning. Could the authors clarify the intended connection between the spatial-reasoning motivation and the experimental findings?
Thank you for the response! We discussed spatial audio to motivate the work [and without saying much, lay groundwork for future work we plan to do). However, we see that we may have overemphasized it. In truth, we have no intention of replicating the SELD experiments for those datasets in this paper. We will edit the manuscript to make this point clearer and to deemphasize spatial audio in the introduction.
Now some quick small notes: