- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes PBSM, a background-modeling approach for passive acoustic vessel monitoring. Instead of learning vessel signatures, it fits a density model of the acoustic background using "no-contact" recordings (no qualifying AIS vessel present). Each approximately 11-minute no-contact recording is represented by a Welch log-PSD (0-1000 Hz), reduced to 64 PCA components and modeled with a 12-component full-covariance GMM, separately for each of eight station-seasons (4 stations, 2 seasons). One-contact recordings (exactly one AIS vessel) are split into 30-second windows, scored by GMM log-likelihood, and converted to an empirical percentile (EBAP) relative to the background scores. Windows with EBAP 5% are labeled "acoustic departures". Since the model is trained on full-length PSDs but tested on 30-second windows, held-out no-contact recordings split into 30-second windows are used as a matched control. Departures are then checked for spectral excess (at least 10 bins above the median background by 3 dB or 6 dB), and AIS-derived CPA is used after scoring to see whether departures and spectral excess change with vessel distance.
The main findings are as follow: (i) the 30-second representation alone produces very high departure rates in the control (16-100% of windows, 25-100% of recordings), so departure by itself is not detection; (ii) the enrichment of one-contact over the matched control is about in 5 of 8 station-seasons and at most (Station 5, 2016-17), and; (iii) at Station 2 (2015-16), the departure rate and the 6 dB spectral excess decrease with CPA, and spectral excess is more frequent among one-contact departures than among control departures.
Critical strengths
-
The matched 30-second no-contact control is carefully constructed. It shows that most "departures" come from the mismatch between full-duration training PSDs and 30-second test windows, not from vessels (Table 1: 99-100% of control windows depart at Station 5 (2015-16), Station 6 and Station 20). Reporting this avoids an inflated detection claim, and it is a useful cautionary result for anyone trying to apply background or OOD models to operational PAM data.
-
The evaluation does use real data from eight station-season pairs with a fixed protocol and no curation or per-site tuning. The change at Station 5 between seasons is a particularly interesting finding on how detectability depends on deployment conditions.
Critical weaknesses
There exists little to no quantitative detection evaluation nor baseline, so the value from the method cannot be accurately assessed.
The results are only prevalence rates at a single operation point. Paper reports no (AU)ROC, precision-recall, detection vs. false alarm analysis. There are also no comparisons (even simple ones like a broadband or band-limited energy detector against a running background, a density model fitted on 30-second no-contact windows (to account for the representation mismatch), standard OOD scores (k-NN distance, Mahalanobis, isolation forest, AE reconstruction), or an existing supervised ship-noise detector, etc.). This especially matters because a simple energy detector may perform just as well. The spectral-excess criterion is essentially a broadband level test: the "strong" examples in Figures 16 and 18 exceed the background by at least 6 dB in all 257 bins. At Station 2, where PBSM departure is in fact informative, the effect is small, not to mention no information carriage in the majority of station-season pairs.
2. Several design choices effectively undermine the "blind" or "independent" claim and the statistical interpretation.
(a) Section 3.3 states that PCA/GMM configurations were trialled "with their behaviour examined relative to independently derived vessel CPA" before freezing. Model selection therefore used vessel geometry, so the CPA analysis is NOT an independent validation. It is also unclear which station-seasons were used for this, and whether they overlap with the reported results.
(b) The train/test mismatch in representation is a design choice so does not constitute a property of the problem. Fitting the background on 30-second no-contact windows (with recording-level splits) is one alternative, and the paper does not explain why it was not used (or not reported).
(c) A recording counts as a departure if any of its roughly 22 dependent windows falls below 5%, so recording-level rates are inflated by multiple testing. The paper itself notes that EBAP is uncalibrated and that the windows are dependent.
(d) Low GMM likelihood is two-sided. A window can depart because it is quieter or spectrally different from the background (e.g., Figure 17: EBAP 0.000% at 0.8 km CPA, maximum excess 1.8 dB), which makes it a questionable proxy for vessel observability.
(e) Basic statistics are nowhere to be found. The number of no-contact and one-contact recordings, windows and unique vessels per station-season, the sample size per CPA bin, the sampling rate, and any confidence intervals or tests are simply not found. Without these, it is impossible to tell whether differences are actually meaningful.
Overall, the ML component (PCA, GMM likelihood and a percentile threshold) is a basic standard and close to prior background-GMM work, as the authors acknowledge. Given the limited methodological novelty and the lack of any comparative evaluation, the paper would fit better in an underwater acoustics or ocean engineering venue.
Questions for authors
-
Could you report window-level and recording-level ROC/PR curves (or AUROC) for one-contact vs. held-out no-contact for each station-season, together with at least (i) a broadband/band-level energy detector and (ii) a GMM fitted on 30-second no-contact windows? If PBSM clearly outperforms, I would reconsider my assessment.
-
Which station-seasons and data were used when trialling the PCA/GMM configurations against CPA? Were the held-out no-contact and one-contact sets involved?
-
Please give the number of recordings (no-contact train/held-out, one-contact), 30-second windows and unique MMSIs per station-season and per CPA bin, with confidence intervals for the percentages in Table 1 and Section 4.2.
-
How many of the windows with "substantive spectral excess" are broadband level shifts (most or all of the 257 bins above threshold) rather than narrowband or tonal components? How does the spectral-excess criterion perform on its own as a detector without PBSM?
-
How are vessels without AIS (small craft, etc.) handled in the no-contact set? How was the "qualifying" contact radius chosen so that no-contact recordings are actually free of audible traffic?
-
Are the CPA trends in Section 4.2 for Station 2 (2015-16) only? Do they hold after controlling for vessel type/size and speed?
Suggestions: calibrating EBAP on a separate calibration split (as mentioned by the authors) would allow the 5% threshold to be read as a false-alarm rate. The Station 5 seasonal contrast would also be more informative with the environmental and deployment covariates proposed in the Discussion.