- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes AGRI-Fidelity, a score that checks whether explanation masks from several audio models agree in time more than a cyclic-shift null would predict. It is reported next to standard masking fidelity. Experiments use poultry recordings with an injected stationary tone, raw vs denoised SmartEars, and swine coughs, with LMAC-TD, L2I, CoughLIME and AudioLIME as explainers. The main message is that fidelity and temporal reliability capture different things.
Critical strengths
Reporting fidelity and reliability separately is a good idea and the stats are done properly. Appendix H uses paired tests with Holm correction instead of just ranking explainers by mean score
Critical weaknesses
1.The score measures temporal alignment, not physiological relevance, but the paper reads it as the latter. Theorems 1 and 2 show that time-invariant masks get zero and aligned sparse masks get high reliability. A short, aligned artifact would also score high, and a long relevant sound could score low. Still, Sections 7.1 and 7.2 interpret low reliability as shortcut learning and high reliability as preserving biological signals. Without relevance labels or a control that separates duration from relevance, this goes beyond what's shown.
2.The artifact detection evaluation uses the wrong labels. Section 7.2 uses "unhealthy" as ground truth for artifacts, but Section 6 says the artifact was only detectable in about 66% of unhealthy and 12% of healthy recordings. So the confusion matrix in Table 2 doesn't really measure artifact detection. Table 2 also has 260 samples while the explainer comparison uses 52, and it's not clear how they relate.
3.The metric as implemented doesn't match the definition. Eq. 3 draws shifts uniformly from all offsets, but Table 7 uses 10 to T-100. Section 4 masks consensus regions while Appendix E.6 masks per-model regions, and E.6 uses signed fidelity while Appendix I uses absolute change. These choices can change the comparisons.
Questions for authors
1.Could you add a synthetic test with a short irrelevant cue and a long relevant cue, where the true masks are known? (suggestion)
2.Can artifact detection be evaluated against actual artifact presence? And how do the 260 samples in Table 2 relate to the 52 in Table 1 and to the train/test split? (clarification)
3.Which shift range and fidelity definition were used for each main result? Also, for healthy samples, is the sign of the binary-logit attribution flipped before keeping positive bins? (clarification)
3 comments
Thank you again for your thoughtful review. We are writing to kindly follow up on our submitted rebuttal, as we have not yet received additional feedback. When convenient, could you please review our response and let us know whether it addresses your concerns? If it changes your assessment, we would appreciate an update to your review and score as appropriate. If any concerns remain, we would be happy to provide further clarification.
Thank you for your time and consideration.
Thanks for the honest response. I appreciate that you agree the score measures temporal localization rather than physiological relevance and will reframe the claims, and the clarification on the 260 vs. 52 samples helps. The full-range shift rerun is a good sign. Still, most fixes are planned rather than done: the synthetic test, the independent artifact labels, and checking the attribution sign for Healthy samples, which could change the results. I’d need to see those before changing my view, so I’m keeping my score at 2.
Response to Reviewer NZqS: Scope of the score, artifact labels and data splits, and one unified definition
We thank the reviewer for the precise critique and for noting the separate reporting of fidelity and reliability and the paired, Holm-corrected tests. We agree with all three points. Below, "will" denotes changes to the final version; the shift rerun in item 3 is already completed.
1. Temporal alignment vs. physiological relevance (and Question 1). We agree. The score measures cross-model agreement beyond a cyclic-shift null, i.e., temporal localization of shared evidence, not physiological relevance. Both counter-cases follow from Theorems 1–2: a short artifact shared and time-aligned across models scores high, and a long relevant sound can score low. We will revise the text accordingly. In Sections 7.1–7.2, "shortcut learning" will become "temporally persistent attribution, consistent with reliance on background components", and "preserving biological signals" will become "preserving time-localized signals". The abstract and contribution (5) will be softened the same way, and Section 8 will state both counter-cases as limitations and note that relevance requires expert or event annotations.
We will add the suggested synthetic test: a short irrelevant cue and a long relevant cue injected with known masks, reporting reliability, fidelity, and overlap with the true masks. We expect it to confirm both counter-cases, which will delimit what the score can claim.
2. Artifact labels and the 260 vs. 52 samples (and Question 2). We agree that the confusion matrix in Section 7.2 used class labels as if they were artifact labels. What the data support is an association with injection status: the tone was injected only into Unhealthy files (Appendix E.3), and Near-1 FDR occurs in 81/121 injected vs. 19/139 non-injected recordings (Fisher OR 12.8, p=3.0×10−19). Because injection status is not artifact presence (an injected tone may be masked or undetectable, and some Healthy files may contain a similar component, as Section 6 notes), we will report this as an association, not as detection accuracy. Regarding Question 2, yes, detection can be evaluated against actual presence: for the final version, we will build independent presence labels (injection log plus a spectral detector of 10.2 kHz energy in the input audio) and report detection metrics against them. The detectability figures in Section 6 (≈66%/≈12%) will be replaced by these detector-based counts.
We will also clarify that FDR ≈ 1 means the observed agreement is no larger than under random shifts. This occurs when models agree on a persistent component or agree only at chance level; neither supports a reliable explanation.
Table 2 analyzes all 260 Chicken Spurious recordings with the committee. Table 1 uses the 52 recordings of the 20% held-out split (Appendix E.4); the remaining 208 are training recordings. We will state this in both captions.
3. Implementation vs. definition (and Question 3). In the submission, the reliability pipeline (Appendix E.6) used the restricted range of Table 7 (10 to T−100; 1 to T/2 for SwineCough), and fidelity was the signed change on each model's own mask; only Appendix I used the absolute change. The Section 4 description of consensus masking was inaccurate. We will use one definition throughout: