- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work proposed a method to improve the calibration of existing pretrained foundation models in bioacoustics, in particular for birds. This method aims to remove the need for labeled calibration data at specific monitoring sites to tune thresholds to accept a model's prediction. It uses pre-calibrated scores learned from a global bird dataset and fuses them with available geopriors.
Critical strengths
- the problem is very important and not investigated enough in the field. a better calibrated model to eliminate the need for location-specific threshold tuning is essential for future ecological monitoring.
- the method seems to be inexpensive, elegant and easy to apply to existing models, improving model calibration
Critical weaknesses
Small contribution, limited evaluation and lacking related work: This work only evaluated two supervised pretrained CNN models on two datasets. Since the method contribution itself is rather small (a combination of existing methods/models) and the field is pretty niche, this work requires a broader evaluation to be accepted at a top-tier conference such as ICLR in my opinion. At least a broader discussion of the field in related work would be essential (the bare minimum of different foundation models or datasets or the reason why things couldn't be used) for the general audience to get a bigger picture of this domain and put it into proper context. The related work section is very limited with only a few works mentioned. For instance, a more general evaluation on the well-established Birdset benchmark (on different geo location) with available geo information would help strengthen the contribution and better validate the generalization across different environments. The same applies for a broader evaluation of different training approaches.
While I understand that this paper is more about off the shelf models, they are also widely used with frozen embeddings (and e.g., a linear head) and it cannot be ignored entirely. An self-supervised pretrained model as an alternative representation learning technique such as BirdAVES or BirdMAE would be a good case study to strengthen the evidence for generalization across bioacoustic foundation models. Even if the evaluation use case would require some labeled data, this is pretty common in a real-world use case (e.g. for not-supported species) and the threshold-tuning is still a problem.
Also a deeper evaluation with a more specific biological use case to showcase (or discuss it for the genreal audience) the practical need for threshold tuning to accept a model's prediction would be very helpful to strengthen the contribution of this work and make the problem better understandable for non-experts.
Questions for authors
See above in weaknesses, they are also the key questions that affect my assessment. While this could theoretically also be assigned to future work, I think this is essential for the niche-topic paper to be accepted at a general top-tier conference such as ICLR