- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes new explanation method called DECAF which consists of evidence, contradiction, and fragility scores. DECAF only requires multiple forward passes to calculate its scores unlike previous method requiring internal activation or gradient. It also reveals the direction of the model reaction change instead of simply showing the magnitude. The paper not only provides the theoretical grounding but also specific correlation, showing that DECAF’s E,C, and F is related to independent model behaviors in control experiments. DECAF is also shown valid in natural images, external attribution benchmark, etc.
Critical strengths
-
E,C, and F elements of DECAF are carefully defined and intuitive to understand.
-
DECAF solely depends on forward passes while being much faster than previous baselines.
-
Comprehensive analysis and experiments are conducted (control diagnostic, natural image, external attribution, etc), which allows the reader to assess the DECAF in various perspective.
Critical weaknesses
-
It’s a bit overclaim that DECAF is model-agnostic. L78-80 argue that DECAF can be applied to any model that returns a scalar but mainly tested on image classifier based on Resnet and ViT. Whether it also works for more advanced architecture based on LLM or VLM (such as Jev [1] or VLM embedding models [2,3]) are still uncertain.
-
Theory is actually not showing anything special. It does not prove theoretical bound when DECAF become useful compared to Abs value or explain why DECAF is better aligned with real explanation compared to other baselines. It reiterates the definition in other ways with some case studies. I would suggests to move Sec 4 (Theoretical Foundation) to the Appendix while include more details in other sections. Due to the space occupied by Sec 4, lots of details are missed in other section which highly reduce the readability.
-
The scope of experiments on natural images is very limited. Only two interventions are tested in the paper: background replacement (ImageNet-9) and patch or part deletion (Sec. 7), which is either too simple or bit artificial. No concept or attribute level counterfactual is evaluated on real images. Can author test it on more fine-grained counterfactual set based on CUB dataset (bird attribute) [4]?
-
No single experiment tests all three components together in Sec 5. Each component gets its own task, model set, and behavior, which seems bit too much for controlled experiment. So it is bit unclear whether the similar controlled setup for Evidence would also show high correlation score for Fragility and Contradiction.
[1] Deußer, T., Sparrenberg, L., & Sifa, R. (2026). Evaluating and benchmarking the system one model Jev. arXiv preprint arXiv:2609.37647.
[2] Jiang, Z., Meng, R., Yang, X., Yavuz, S., Zhou, Y., & Chen, W. (2024). Vlm2vec: Training vision-language models for massive multimodal embedding tasks. arXiv preprint arXiv:2410.05160.
[3] Li, M., Zhang, Y., Long, D., Chen, K., Song, S., Bai, S., ... & Lin, J. (2026). Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720.
[4] Bader, J., Girrbach, L., Alaniz, S., & Akata, Z. (2025, October). Sub: Benchmarking cbm generalization via synthetic attribute substitutions. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 23188-23198). IEEE.
Questions for authors
-
What is the effective strategy to choose threshold (epsilon) for distance d?
-
How the practitioner know when to use DECAF and when Abs score is sufficient?