- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces a pipeline which involves a deep, ResNet-based feature engineering phase and classical computer vision feature extraction phase. These features are combined into a Bayesian Network, responsible for the final prediction. Each of the feature extractors also produce an uncertainty estimate, which allows (i) the same Network to produce a value of uncertainty alongside the predicted value, and (ii) to weigh the contribution of each feature extractor proportionally to its confidence, i.e., uncertain features will be weighed less in the final prediction. Finally, the authors showcase the functioning of their system on three, real-world animal disease detection datasets.
Critical strengths
The paper is addressing important concerns of trustworthiness (interpretability of predictions and estimates of uncertainty) in (animal) healthcare, and is integrating these elements into the design of their system, not as a (post-hoc) afterthought, and seem to effectively combine old-style, interpretable feature extraction with modern deep feature engineering
Critical weaknesses
-
Novelty concerns seem too narrow: while the authors are right in stating that their approach is novel in the field of animal disease detection, in my opinion, the type of conference the paper has been submitted require the novelty to be general, broader; the authors should state whether their approach towards an explainable/uncertainty-aware system for image classification is novel in the literature overall. I think, in this sense, the authors should minimally cite Chen et al. (2020) and Wasserkrug et al. (2021) as similar approaches instead of the two old references (from 1997 and 2006) they cite.
-
(important) Quality of uncertainty estimates: while I can understand the idea behind the uncertainty-based reweighting of the extracted features for producing the final prediction, this can lead to unexplainable failures in case of uncalibrated uncertainty estimates. Consider the case of a wildly overconfident extractor (as is often the case for Deep Neural Networks - DNNs). This would create an excessively-confident (and possibly wrong) output that would have a large weight in the BN, that could hence be misguided. I feel this could be a typical behavior in an out-of-distribution example, where the output of the ResNet would essentially be gibberish difficult to disentangle from in-distribution predictions. Conversely, an underconfident prediction would have minimal influence on the prediction, despite possibly being correct. In order to further validate their approach, the authors should ensure that the uncertainty estimates are well calibrated. The authors seem to be oblivious to these observations, as, in Sec. 4.3, they automatically assume distribution shift causes high uncertainty: a strong assumption which is never proven and often fails in real-life scenarios (Nguyen et al., 2015).
-
(important) No comparison with the State-of-the-Art: the paper misses a comparison of their result w.r.t. other SoTA approaches for their datasets. Without a comparison, it is difficult to contextualize the quality of the performance metrics disclosed in Table 1 (and 2 to a lesser extent). Since the focus of the paper does not reside with accuracy alone, it is not paramount that the accuracy is better, but it should at least be similar.
-
The focus on explanations adversarial attacks is not motivated: the authors fixate excessively on XAI adversarial attacks, but never motivate why this is important, nor give an example where adversarial manipulation on explanations could occur on a real-life scenario for their application. The introduction starts straight away with citing the paper from Ghorbani et al., but there is no mention as to why this specific focus on adversarial attack is chosen, and not one of the many other limitations of XAI that are reported, for instance, in Afroogh et al. (2026) or Jacobs et al. (2021).
-
The XAI evaluation relies on outdated metrics and faithfulness computation is unclear: the authors evaluate their explanations on two areas: (i) faithfulness and (ii) robustness. The metrics used are from the LIME paper for Ribeiro et al (i) and the Alvaris-Mellis paper (ii).
i. This metric requires training a surrogate model to verify the alignment w.r.t. the original model - this is not a standard approach when validating fidelity of explanations generated by means other than surrogate models; pixel flipping (Adebayo et al., 2018) and variations thereof - see Dembinsky et al. (2026) and Nauta et al. (2023) for an extensive overview of fidelity metrics. Additionally, it is unclear, even in the appendix, what is the surrogate model the authors used to verify alignment between explanation and the ResNet18.
ii. This metric has been cited as being incomplete and misleading by Agarwal et al. (2022), who then introduced three metrics (RIS, ROS, RRS), the latter two including also the model's representations/outputs in the computation. There are also more recent, manifold-based stability metrics, such as the one by Vascotto et al., (2025).
Additionally, I posit the authors should possibly include the computation of coherence (Nauta et al., 2023) in their system: one thing is verifying the explanations are faithful and robust, but they should also be validating with regards to the alignment with clinicians' knowledge; a veterinarian using the authors' system as a helper should expect an explanation to align w.r.t. their expertise.
-
Some techniques for feature importance are inadequate for images: Integrated Gradients and (possibly, depending on the implementations) SHAP are known to be unstable (Woerl et al., 2023) and sensitive to high-frequency pixels (Muzellec et al., 2023). I understand the authors use ensembling techniques to create a consensus, but possibly IG should be removed for a more reliable technique. For SHAP, I need additional info (see Question 3) before I can say something in regard.
-
A truly validated approach should include a clinical test: behind a good performance of healthcare AI system solely on benchmark datasets can hide problematic fairness issues, as famous cases highlighted by Zech et al. (2018) and Obermayer et al. (2019), or the system may be outright clinically unusable due to generic non-representativity of the dataset, as in the famous COVID papers in the early 20s (Roberts et al., 2021). This should better be reflected in (i) the limitations and (ii) in the broader impact statement - it seems, especially from the latter - that the system may be more clinically-ready than it is. Additionally, I would posit that the lack of a clinical test would void the claim as to "empirical validation" in the key contributions.
-
Missing info on Finer-CAM: Finer-CAM requires two classes to generate the feature attribution: the main class and a "contrastive" class. The paper does not state (neither in the appendices) which were the contrastive classes, nor the rationale for their selection. It is important that this is determined, since Finer-CAM is thought to require some sort of field expertise to identify similar classes on which to compute the explanation. Ideally, a clinician would reply to the question: "Why is this disease x and not disease y, whose visual symptoms look similar?"
-
Readability is hampered by poor organization of content: the paper severely limits the (formal) information introduced in the main text. My reading was hampered by the fact that searching for any detail required a consistent back-and-forth between the main text and the various appendices, even for important information. The main text includes only one formula (which, in my opinion, is not even necessary to understand the methodology) and all of the detail (incl. formulae) of the various metrics used are in the appendix, making it hard to gain any critical information on the paper from just reading the main text. Additionally, even the limitations are in the appendix, which is something I would consider quite unusual. On the other hand, I find the discourse around explanations quality and adversarial attacks rather repetitive across sections. An additional cue to poor content organization can be found in Sec. 5, where parameterization is first described, talking about (not previously introduced) datasets, while the datasets are included next. It would be more logical to talk about datasets first, then parameterization.
-
Lack of usage of specialized deep feature extractor: this is partially addressed in the limitations, but the authors should definitely consider using healthcare-specific models in their pipeline - possibly testing their performance gains (or losses?) w.r.t. the pretrained ResNet18. Their proposed system does not include any healthcare specific technique, despite the great availability of these models (e.g., the UNets for medical image segmentation) on the web.
-
Lack of clarity concerns - these are mostly minor concerns that limit the clarity of the paper:
- In Eq. 1, it should be made clearer M refers to the BN and not the feature extraction subcomponents, since previously in the paragraph the BN is called "system" and the feature extractor are called "predictive models" - this unless I misunderstood the formula, let me know if this is the case.
- In Sec. 4.1, it could be made immediately known that you're using a ResNet18.
- Consider using "predictive uncertainty" instead of "total uncertainty". I think the first term is more accurate.
- There needs to be a table (possibly in the appendix) with the accurate list of classical feature extractor methods - e.g., you only cite "color statistics such as mean Red–Green–Blue (RGB) intensities and normalized color ratios to reduce sensitivity to illumination changes" but never detail exactly the tools used.
- Clarify how the normalization of the uncertainties happen - are the uncertainty [0-1]-normalized within or between models? Within or between datasets?
References:
(Adebayo et al., 2018) Adebayo, Julius, et al. "Sanity checks for saliency maps." Advances in neural information processing systems 31 (2018).
(Afroogh et al., 2026) Afroogh, Saleh, et al. "Beyond explainable ai (xai): An overdue paradigm shift and post-xai research directions." arXiv preprint arXiv:2602.24176 (2026).
(Agarwal et al., 2022) Agarwal, Chirag, et al. "Rethinking stability for attribution-based explanations." arXiv preprint arXiv:2203.06877 (2022).
(Chen et al., 2020) Chen, Jun, et al. "Towards interpretable clinical diagnosis with Bayesian network ensembles stacked on entity-aware CNNs." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.
(Dembinsky et al., 2026) Dembinsky, David, et al. "Unifying VXAI: a systematic review and framework for the evaluation of explainable AI." arXiv preprint arXiv:2506.15408 - TMLR (2026).
(Jacobs et al., 2021) Jacobs, Maia, et al. "How machine-learning recommendations influence clinician treatment selections: the example of antidepressant selection." Translational psychiatry 11.1 (2021): 108.
(Muzellec et al., 2023) Muzellec, Sabine, et al. "Saliency strikes back: How filtering out high frequencies improves white-box explanations." arXiv preprint arXiv:2307.09591 (2023).
(Nauta et al., 2023) Nauta, Meike, et al. "From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai." ACM Computing Surveys 55.13s (2023): 1-42.
(Nguyen et al., 2015) Nguyen, Anh, Jason Yosinski, and Jeff Clune. "Deep neural networks are easily fooled: High confidence predictions for unrecognizable images." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
(Obermeyer et al., 2019) Obermeyer, Ziad, et al. "Dissecting racial bias in an algorithm used to manage the health of populations." Science 366.6464 (2019): 447-453.
(Roberts et al., 2021) Roberts, Michael, et al. "Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans." Nature Machine Intelligence 3.3 (2021): 199-217.
(Vascotto et al, 2025) Vascotto, Ilaria, et al. "When can you trust your explanations? A robustness analysis on feature importances." World Conference on Explainable Artificial Intelligence. Cham: Springer Nature Switzerland, 2025.
(Wasserkrug et al., 2021) Wasserkrug, Segev, et al. "Learning the parameters of bayesian networks from uncertain data." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 13. 2021.
(Woerl et al., 2023) Woerl, Ann-Christin, Jan Disselhoff, and Michael Wand. "Initialization noise in image gradients and saliency maps." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.
(Zech et al., 2018) Zech, John R., et al. "Confounding variables can degrade generalization performance of radiological deep learning models." arXiv preprint arXiv:1807.00431 (2018).
Questions for authors
- You cite the usage of "interpretable" feature extraction methods as aiding the interpretability of the system; however, when talking about XAI, you only seem to address explanations generated on the ResNet18 model. How can insights from the feature extraction be merged with these explanation? Clearing this point could increase the quality of the paper from the XAI point of view.
- How is the uncertainty computed on the ResNet18? Uncertainty estimation in DNNs either requires an output simplex (whereas the uncertainty is defined by the max probability value or the entropy of the distribution) or a "probabilistic" output (e.g., ensembles, dropout/dropconnect etc...); however, in your case, you have an embedding space. Is the uncertainty defined by the Mahalanobis/Euclidian distance w.r.t. one of the class centroids in the embedding space?
- Ref. Appendix C.5, is the specific computation of SHAP meaning that the "features" attributed are 128 px x 128 px in size? How does it relate to the full image size, in this case?
- The intro mentions the usage of the BN as constituting reasoning. Can you better clarify what is your definition of reasoning and how that would be the case with the BN? It seems the term reasoning may be a bit too overstated for your approach.
- What does it mean (Sec. 5) "features are reduced to 96 dimensions and combined with classical features" - how are the features reduced and how many dimensions are there before reduction?
- At page 5, you detail the uncertainty-aware adjustment P_u = (1 − u)P + u/|Y|, later detailing how u is computed, which, if I understand correctly, is a fixed mean value from training. Does this mean that the adjustment cannot be operated per-example at test time? Should this also invalidate the observations on "unreliable inputs/observations" that is made shortly after and in Appendix A's introduction?