acceptodds
Under review as a conference paper at ICLR 2027

AGRI-Fidelity: Evaluating the Reliability of Listenable Explanations for Poultry Disease Detection

Abstract

Existing XAI metrics measure faithfulness for a single model, ignoring model multiplicity where near-optimal classifiers rely on different or spurious acoustic cues. In noisy farm environments, stationary artifacts such as ventilation noise can produce explanations that are faithful yet unreliable, as masking-based metrics fail to penalize redundant shortcuts. We propose AGRI-Fidelity, a reliability-oriented evaluation framework for listenable explanations in poultry disease detection without spatial ground truth. The method combines cross-model consensus with cyclic temporal permutation to construct null distributions and compute a False Discovery Rate (FDR), suppressing stationary artifacts while preserving time-localized bioacoustic markers. Across real and controlled datasets, AGRI-Fidelity provides reliability-aware discrimination for all data points versus masking-based metrics.

Primary area:
interpretability and explainable AI
Keywords:
Explainable AI, Fidelity, Poultry Vocalizations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NZqSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AGRI-Fidelity, a score that checks whether explanation masks from several audio models agree in time more than a cyclic-shift null would predict. It is reported next to standard masking fidelity. Experiments use poultry recordings with an injected stationary tone, raw vs denoised SmartEars, and swine coughs, with LMAC-TD, L2I, CoughLIME and AudioLIME as explainers. The main message is that fidelity and temporal reliability capture different things.

Critical strengths

Reporting fidelity and reliability separately is a good idea and the stats are done properly. Appendix H uses paired tests with Holm correction instead of just ranking explainers by mean score

Critical weaknesses

1.The score measures temporal alignment, not physiological relevance, but the paper reads it as the latter. Theorems 1 and 2 show that time-invariant masks get zero and aligned sparse masks get high reliability. A short, aligned artifact would also score high, and a long relevant sound could score low. Still, Sections 7.1 and 7.2 interpret low reliability as shortcut learning and high reliability as preserving biological signals. Without relevance labels or a control that separates duration from relevance, this goes beyond what's shown.

2.The artifact detection evaluation uses the wrong labels. Section 7.2 uses "unhealthy" as ground truth for artifacts, but Section 6 says the artifact was only detectable in about 66% of unhealthy and 12% of healthy recordings. So the confusion matrix in Table 2 doesn't really measure artifact detection. Table 2 also has 260 samples while the explainer comparison uses 52, and it's not clear how they relate.

3.The metric as implemented doesn't match the definition. Eq. 3 draws shifts uniformly from all offsets, but Table 7 uses 10 to T-100. Section 4 masks consensus regions while Appendix E.6 masks per-model regions, and E.6 uses signed fidelity while Appendix I uses absolute change. These choices can change the comparisons.

Questions for authors

1.Could you add a synthetic test with a short irrelevant cue and a long relevant cue, where the true masks are known? (suggestion)

2.Can artifact detection be evaluated against actual artifact presence? And how do the 260 samples in Table 2 relate to the 52 in Table 1 and to the train/test split? (clarification)

3.Which shift range and fidelity definition were used for each main result? Also, for healthy samples, is the sign of the binary-logit attribution flipped before keeping positive bins? (clarification)

3 comments
AuthorsOpenReview ↗

Response to Reviewer NZqS: Scope of the score, artifact labels and data splits, and one unified definition

We thank the reviewer for the precise critique and for noting the separate reporting of fidelity and reliability and the paired, Holm-corrected tests. We agree with all three points. Below, "will" denotes changes to the final version; the shift rerun in item 3 is already completed.

1. Temporal alignment vs. physiological relevance (and Question 1). We agree. The score measures cross-model agreement beyond a cyclic-shift null, i.e., temporal localization of shared evidence, not physiological relevance. Both counter-cases follow from Theorems 1–2: a short artifact shared and time-aligned across models scores high, and a long relevant sound can score low. We will revise the text accordingly. In Sections 7.1–7.2, "shortcut learning" will become "temporally persistent attribution, consistent with reliance on background components", and "preserving biological signals" will become "preserving time-localized signals". The abstract and contribution (5) will be softened the same way, and Section 8 will state both counter-cases as limitations and note that relevance requires expert or event annotations.

We will add the suggested synthetic test: a short irrelevant cue and a long relevant cue injected with known masks, reporting reliability, fidelity, and overlap with the true masks. We expect it to confirm both counter-cases, which will delimit what the score can claim.

2. Artifact labels and the 260 vs. 52 samples (and Question 2). We agree that the confusion matrix in Section 7.2 used class labels as if they were artifact labels. What the data support is an association with injection status: the tone was injected only into Unhealthy files (Appendix E.3), and Near-1 FDR occurs in 81/121 injected vs. 19/139 non-injected recordings (Fisher OR 12.8, p=3.0×10−19p=3.0\times10^{-19}). Because injection status is not artifact presence (an injected tone may be masked or undetectable, and some Healthy files may contain a similar component, as Section 6 notes), we will report this as an association, not as detection accuracy. Regarding Question 2, yes, detection can be evaluated against actual presence: for the final version, we will build independent presence labels (injection log plus a spectral detector of 10.2 kHz energy in the input audio) and report detection metrics against them. The detectability figures in Section 6 (≈66%/≈12%) will be replaced by these detector-based counts.

We will also clarify that FDR ≈ 1 means the observed agreement is no larger than under random shifts. This occurs when models agree on a persistent component or agree only at chance level; neither supports a reliable explanation.

Table 2 analyzes all 260 Chicken Spurious recordings with the committee. Table 1 uses the 52 recordings of the 20% held-out split (Appendix E.4); the remaining 208 are training recordings. We will state this in both captions.

3. Implementation vs. definition (and Question 3). In the submission, the reliability pipeline (Appendix E.6) used the restricted range of Table 7 (10 to T−100T-100; 1 to T/2T/2 for SwineCough), and fidelity was the signed change on each model's own mask; only Appendix I used the absolute change. The Section 4 description of consensus masking was inaccurate. We will use one definition throughout:

  • Null. Independent shifts Δ∼U{0,…,T−1}\Delta\sim\mathcal{U}\{0,\dots,T-1\} over all TT frames, as in Eq. 3 and the theorems. We already reran reliability this way on four datasets: all changes are ≤0.023 and the best method is unchanged on every dataset (values in our response to Reviewer 7gci). We will apply it to all tables, tests, and SwineCough.
  • Masked region. Fidelity masks each model's own top-kk region; consensus is used only for reliability. We will correct Section 4.
  • Fidelity sign. The signed change in the predicted-class probability, Fm=pm,c(x)−pm,c(xmasked)F_m=p_{m,c}(x)-p_{m,c}(x_{\text{masked}}). Unlike the absolute change, it does not reward masks that increase confidence; unlike a fixed-class sign, it is symmetric across classes.
  • Attribution sign for Healthy samples. Attributions are computed for the single binary logit (Table 6), and the submission does not describe a sign flip before keeping positive bins. We will verify in the code whether a flip is applied and use the predicted-class score (the negated logit for Healthy samples) for both the masks and fidelity, then regenerate the affected results.

Thank you again for your thoughtful review. We are writing to kindly follow up on our submitted rebuttal, as we have not yet received additional feedback. When convenient, could you please review our response and let us know whether it addresses your concerns? If it changes your assessment, we would appreciate an update to your review and score as appropriate. If any concerns remain, we would be happy to provide further clarification.

Thank you for your time and consideration.

Thanks for the honest response. I appreciate that you agree the score measures temporal localization rather than physiological relevance and will reframe the claims, and the clarification on the 260 vs. 52 samples helps. The full-range shift rerun is a good sign. Still, most fixes are planned rather than done: the synthetic test, the independent artifact labels, and checking the attribution sign for Healthy samples, which could change the results. I’d need to see those before changing my view, so I’m keeping my score at 2.

Reviewer_6E5cOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes AGRI-Fidelity, an evaluation metric for listenable explanations of model decisions in poultry disease detection. The authors argue that existing XAI evaluation metrics may fail in noisy farm environments (e.g., machinery and ventilation noise). These metrics assess whether an explanation supports the model’s decision but do not quantify spurious correlations driven by data noise. As a result, an explanation can be faithful to the model yet not reflect the domain-specific evidence needed for reliability.

The paper evaluates explanations against three criteria: fidelity, stability, and domain alignment. The framework measures reliability through cross-model consensus over a committee of classifiers, validated against a cyclic time-shift null. A fidelity score, computed by masking the identified salient regions, is reported alongside it. Experiments cover four explainers (LMAC-TD, L2I, CoughLIME, and AudioLIME) and four committee models (five in App. I), on poultry datasets (clean, with an injected spurious tone, and SmartEars raw and denoised) and a swine cough dataset.

Critical strengths
  • The core objective of identifying explanatory features that are consistent across models, rather than relying on a single model’s explanation, is reasonable and relevant for noisy field audio. The framework is theoretically sound, with formal results characterizing its behavior on stationary versus sparse, time-localized signals.
  • The fidelity–reliability quadrant view (Fig. 2) is a helpful way to show that explanations ranked highly by masking-based metrics can differ once cross-model agreement is considered.
Critical weaknesses

The metric does not address domain alignment, and the motivation does not establish the gap.

  1. The metric mainly measures temporal sparsity and agreement above a chance baseline. It does not inform anything about the domain, even though domain alignment is one of the three stated criteria.
  2. If all committee models rely on spurious features, the metric would still treat their agreement as reliable. The paper does not address this case.
  3. The limitations of existing metrics are discussed only at a high level, with one or two examples (ventilation, machinery). There is no quantification showing that spurious correlated features actually appear in existing XAI explanations on this data, which would help the reader judge whether the gap is real.
  4. The motivation also omits the key domain-specific features. The paper should state which criteria domain experts use for this task without AI, and what the reliable indicators are.

The results and validation do not clearly support the claims.

  1. The main experiments use four committee models and four explainers. The controlled setting injects a synthetic spurious tone, and it is unclear whether this is a standard technique for assessing spurious reliance.
  2. There is no controlled test with known-correct and known-spurious cues of both temporal types (e.g., a transient spurious cue and a stationary genuine cue).
  3. Table 1 is not easy to read, and its results are mixed, unclear, and unexplained.

Additional issues

  1. After reading the introduction, it is not clear what core quantity the authors are trying to measure.
  2. The notation for the target explanation (P*(x) in Sec. 3; E_k and E_{M*} in Sec. 4) is hard to follow, and E is overloaded across Sec. 3–4 (explainer vs. binary masks).
  3. Undefined terms: “optimal” and “near-optimal” models are not defined. Do they refer to models with higher accuracy?
  4. Unjustified hyperparameters: the selection of B (the number of permutations) is not explained.
  5. Venue fit: applicability beyond livestock audio (e.g., explanations for general audio classifiers) is not evaluated and is mentioned only briefly as future work.
Questions for authors
  1. What happens if all the committee models pick up the same spurious cue? They would agree with each other, so I’m not sure how the metric would tell that agreement apart from a genuine signal. Could you clarify how this case is handled?
  2. Domain alignment is listed as one of the three goals, but I couldn’t see where the metric actually measures it beyond temporal sparsity. What do veterinarians or domain experts listen for when diagnosing these diseases, and could those cues be used to check whether the consensus regions are meaningful?
  3. The motivation relies on examples like ventilation and machinery noise. Do you have evidence that existing XAI explanations actually highlight such noise on these datasets? Even a rough quantification would help show that the gap is real.
  4. Is injecting a synthetic tone a standard way to test for spurious reliance? Related to this, would it be possible to include a controlled test with both transient and stationary cues, where you know which ones are genuine and which are spurious?
  5. How are “near-optimal” models defined? Is there an accuracy threshold for including a model in the committee? And how was the number of permutations B chosen, and do the results change much if it varies?
  6. I found Table 1 difficult to interpret, since the metrics disagree on which explainer is best. What is the main takeaway you would like the reader to draw from it?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 6E5c: Shared spurious cues, domain alignment, controlled evidence, and clarifications

We thank the reviewer for the constructive questions and for recognizing the cross-model objective, the theory, and the quadrant view; "will" denotes changes to the final version.

1. Shared spurious cues. It depends on the cue's temporal structure. If all models share a temporally persistent cue (e.g., a stationary tone or hum), the agreement survives cyclic shifts, so the null reproduces it: FDR → 1 and reliability → 0 (Theorem 1), however many models agree. Consistent with this, Near-1 FDR occurs in 81/121 tone-injected vs. 19/139 non-injected recordings (Fisher OR 12.8, p=3.0×10−19p=3.0\times10^{-19}). If all models instead share a short, time-aligned spurious event, the shift destroys its alignment as for a genuine event, so it would be scored reliable. Our null cannot separate that case; we will state it explicitly as the main limitation in Section 8.

2. Domain alignment and what experts listen for. Respiratory disease in poultry is recognized by short events: coughs, sneezes ("snicks"), snoring, and rales (gurgling), a distinct marker in infectious-bronchitis-challenged birds (Carroll et al., IEEE GlobalSIP 2014; Rizwan et al., IEEE GlobalSIP 2016). These events are brief, whereas ventilation and machinery noise persist; hence the temporal null. We agree that the metric therefore measures domain alignment only indirectly, through temporal localization, not through expert-labeled cues. We will add these cues to the motivation, reword criterion (3) in Section 3 accordingly, and add a check of overlap between consensus regions and annotated or onset-detected cough/rale events where such annotations exist.

3. Evidence that explanations highlight background noise. On the controlled dataset, committee explanations show Near-1 FDR (agreement largely reproduced by the shift null) in 81/121 tone-injected vs. 19/139 non-injected recordings, and the Near-1 Unhealthy group has the highest fidelity (Table 3): masking those regions sharply reduces confidence. On real data, mean reliability on SmartEars rises from 0.31–0.51 (raw) to 0.91–0.97 after noise reduction and HPSS suppress stationary components (Appendix H.2), consistent with persistent barn noise dominating raw-audio attributions. As the models are retrained, this is supporting evidence. Injection status is not artifact presence, so we will add independent artifact-presence labels and report the share of attribution in low-frequency (ventilation) bands for raw vs. denoised audio. Ventilation and machinery noise will be framed as motivating examples, the tone as the controlled evidence.

4. Synthetic-tone evaluation. Injecting a known nuisance signal is a standard controlled design for testing spurious reliance (e.g., Adebayo et al., ICLR 2022), because the location and timing of the shortcut are known exactly. A stationary tone is one transparent instance, chosen to match the persistent-noise setting that motivates the work. We will add a crossed control with injected stationary and transient cues of known genuine/spurious status. We expect, and will report, that a time-aligned transient spurious cue scores as reliable, which delimits what the metric can detect.

5. Committee selection and BB. All four architectures are trained with the same protocol, and the checkpoint with the best held-out accuracy is kept. "Near-optimal" denotes comparable performance under this protocol, not a formal accuracy threshold; per-model performance is in Table 8, and we will clarify this wording. We use B=50B=50 throughout; the Monte Carlo error of the null count scales as B−1/2B^{-1/2}. We will report reliability for B∈{25,50,100,200}B\in\{25,50,100,200\}. Committee size KK is analyzed in Appendix F (theory) and Appendix I (a 5-model committee including AST).

6. Main takeaway of Table 1. Strong masking scores do not imply reliable explanations. CoughLIME leads AI and AD yet has the lowest fidelity, and LMAC-TD is the only explainer with significantly higher reliability than all others (Holm-adjusted p<0.05p<0.05). The conventional metrics disagree because they measure different properties; AGRI-Fidelity adds a property none of them measures: whether the highlighted evidence is shared across models beyond chance alignment. We will state this takeaway explicitly in Section 7.

7. Core quantity, notation, and scope. The core quantity is R=1−FDR^R=1-\widehat{\mathrm{FDR}}: how far cross-model agreement exceeds that expected under random temporal shifts, read jointly with fidelity; we will state this in the introduction. We will reserve E\mathcal{E} for the explainer, use EkE_k only for binary masks, and define the target explanation P∗(x)P^*(x) and its mask once in Section 3. The method needs only time-frequency attributions from several classifiers, so it applies to audio classifiers generally; SwineCough already goes beyond poultry, and Section 8 will discuss this.

Thank you for the detailed response. The clarification of the cyclic-shift null and its limitations addresses some of my concerns. I will increase my score.

Reviewer_7gciOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AGRI-Fidelity, an XAI reliability metric for poultry disease audio. It scores cross-model consensus against a cyclic-shift null, reporting an empirical FDR alongside standard fidelity. Experiments on five datasets compare four listenable explainers. Two theorems characterize behavior on stationary artifacts and sparse signals.

Critical strengths

The cyclic-shift null is well matched to the stated problem. Shifting preserves spectral structure while destroying temporal alignment, so stationary artifacts remain overlapping while time-localized signals do not. This is a sharper test than raw cross-model agreement.

The reproducibility materials are specified at an unusual level of detail, including per-dataset per-model training settings, and per-sample metric files.

Critical weaknesses

In the controlled spurious-injection experiment, Unhealthy samples receive high reliability, the opposite of what the metric is designed to show. The paper attributes this to CoughLIME’s local perturbations, but a reliability metric that inverts under a common explainer does not do its stated job.

Theorem 2 assumes shifts uniform over all T frames (Eq. 3), but the implementation restricts them to 10 to T−100, and to 1 to T/2 for SwineCough (Table 7).

The main quantitative table uses 52 samples, with reliability standard deviations of 0.29 to 0.40 against mean differences of 0.03 to 0.26.

Questions for authors

In Figure 2(b) the injected-artifact class receives high reliability, while Figure 2(f) shows the opposite for the same samples. What prior criterion determines which attribution source a practitioner should use, before seeing the answer?

Please report reliability under shifts uniform over all T frames, as assumed in Theorem 2, and quantify how much the 0.9 to 1.0 SwineCough reliability changes relative to the restricted 1 to T/2 range.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 7gci: Joint reading of R and F, a priori decision rule, full-support shift rerun, and paired tests

We thank the reviewer for the careful reading. Below, "will" denotes changes to the final version; the shift rerun in item 2 is already completed.

1. High reliability under CoughLIME (Fig. 2(b)) and which attribution source to use. This is the case our two-score design is meant to flag, not an inversion. In Fig. 2(b), consensus includes CoughLIME's attributions. On Unhealthy recordings, the resulting agreement is time-localized (high R) but has little effect on the prediction (low F, compared with the committee masks in Fig. 2(f)); that is, these attributions do not capture the evidence the models rely on. Our interpretation table in Appendix I defines low F with high R as a "shared region with limited prediction effect", i.e., not a trustworthy explanation. With the committee's Integrated Gradients masks instead (Fig. 2(f)), most Unhealthy recordings fall to near-zero reliability: Near-1 FDR occurs in 81/121 tone-injected vs. 19/139 non-injected recordings (Fisher OR 12.8, p=3.0×10−19p=3.0\times10^{-19}).

We agree that reliability read alone would mislead here. We will (i) move the Appendix I interpretation table into Section 4 so that R and F are always read jointly, (ii) replace the speculative mechanism sentence in Section 7.1, and (iii) document exactly how CoughLIME masks enter the consensus.

Which attribution source to use (Question 1). No source is chosen after seeing the results. AGRI-Fidelity scores whichever explainer the practitioner wants to evaluate, and the decision rule is fixed in advance: Section 4 of the submission already states that an explanation is reliable only when it combines low empirical FDR with high fidelity (see also Appendix I). Applied to both panels, this rule gives the same verdict for the injected-artifact class. In Fig. 2(b), masking changes the predicted probability by only about 0.1 (low F); in Fig. 2(f), reliability is near zero for most of them (low R). In both cases these explanations are not trusted, so the panels do not contradict each other; they fail the rule for different reasons. Moving the interpretation table to Section 4 (point (i)) will make this explicit.

2. Shift range vs. Theorem 2. The reviewer is right that the implemented range (10 to T−100T-100; 1 to T/2T/2 for SwineCough) departed from Eq. 3. We reran reliability with independent offsets Δ∼U{0,…,T−1}\Delta\sim\mathcal{U}\{0,\dots,T-1\}, keeping samples and all other settings fixed. Mean reliability, submitted → full support:

DatasetLMAC-TDL2ICoughLIMEAudioLIME
Chicken Denoised.6038→.6002.6362→.6385.5789→.5871.6283→.6223
Chicken Spurious.7445→.7508.4859→.4840.5154→.5281.5263→.5266
SmartEars.4026→.4254.3144→.3268.3738→.3892.5116→.5241
SmartEars Denoised.9472→.9564.9071→.9154.9657→.9716.9277→.9371

All changes are ≤0.023 and the best method is unchanged on every dataset. The only order change (CoughLIME vs. AudioLIME on Chicken Spurious, gap 0.0015) is between methods that were not significantly different. We will adopt the full-support null throughout the final version (Eq. 3, Table 7, all tables and tests). SwineCough has not yet been rerun (Question 2); in the final version we will report its full-support reliability next to the 1 to T/2T/2 values and quantify the change in its 0.9–1.0 range.

3. Sample size and reliability variability. The 52 recordings form one held-out split evaluated by all four explainers, so comparisons are paired. The SDs (0.29–0.40) describe variation across recordings, not uncertainty of the means, so they cannot be compared with mean differences to judge significance. Paired Wilcoxon tests with Holm correction (Appendix H.2.4, submitted values) show higher reliability for LMAC-TD than for L2I, CoughLIME and AudioLIME (adjusted pp = 0.0076, 0.0135, 0.0179), with rank-biserial correlations rr = 0.52, 0.48, 0.46. We agree that the other three explainers cannot be separated at n=52n=52 (adjusted p≥0.40p\ge0.40); on the larger SmartEars split (n=271n=271), reliability separates into three significance groups. We will recompute these tests on the full-support values and add paired bootstrap 95% CIs for the differences.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.