acceptodds
Withdrawn from ICLR 2027

CHARTER: Auditing Reference Substitution in Hierarchical Compact-Evidence Evaluation for Computational Pathology

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In digital pathology, compact evidence is often used to explain or audit predictions made by whole-slide image multiple instance learning models. In hierarchical compact-evidence pipelines, candidate filtering introduces a strategy-specific candidate-conditioned prediction alongside the original full-bag prediction. If the evaluation reference changes while the intended target remains the original full-bag prediction, however, not only can the measured fidelity of the same compact evidence change, but comparisons between competing candidate strategies can also change. To make this dependence explicit, we introduce CHARTER, a reference-aware evaluation charter that asks researchers to **DECLARE** the intended target and reference, **QUANTIFY** candidate-induced prediction shift, and **AUDIT** the stability of comparative conclusions. Across the 15 comparisons in our main five-seed Random- audit, 4 showed determinate reversals; in a matched native-ranking stress test, the ACMIL comparison changed from REVERSED to PRESERVED. CHARTER turns otherwise implicit candidate-filtering and reference choices into an auditable evaluation specification, helping distinguish genuine preservation of the intended prediction from apparent gains induced by changing the prediction being explained.

Primary area:
interpretability and explainable AI
Keywords:
explainable AI, faithfulness evaluation, reference dependence, multiple instance learning, computational pathology, whole-slide imaging

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_DQcoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The compact evidence is import in digital pathology field, it can help trace the interpretable evidence. In the paper, the author makes an framework evaluation report across different tasks, MIL models and feature extractor.

Critical strengths

None

Critical weaknesses
  1. The paper is difficult to understand, especially the methodology and overall workflow. Although the authors raise a potentially meaningful research question, I find it hard to follow how the proposed method and experiments are connected to the main motivation. The writing also feels somewhat AI-generated, with repetitive descriptions and a lack of clear explanations, which makes the paper even harder to follow.
  2. Some sections are not well organized, such as the contributions in the Introduction. For example, the authors directly introduce X→C→SX \rightarrow C \rightarrow S without clearly explaining what these variables mean, making it difficult for readers to understand the contributions. Similar problems appear in other sections, where technical concepts are introduced without sufficient explanation.
  3. In my opinion, the current version of this paper does not meet the standards of ICLR. While the research question is worth investigating, the methodological contribution is relatively limited, and the experiments do not provide a sufficiently clear and convincing story. I would suggest that the authors carefully revise the manuscript, particularly its motivation, methodology, and experimental presentation, before submitting it to a more suitable conference or journal.
Questions for authors

In my view, a high-quality research paper should address a well-defined and meaningful problem with a reasonably innovative method. Such innovation does not necessarily need to be groundbreaking, but it should be closely aligned with the research problem, supported by clear motivation, and reflected in a coherent methodological design. The proposed method should then be thoroughly validated through convincing experiments, with the overall work carefully organized and refined into a logically structured and well-supported paper. However, although this paper addresses a potentially meaningful research problem, I find it difficult to identify a clear and consistent narrative throughout the methodology and experimental sections. In particular, the motivations behind individual methodological components, their relationships, and the extent to which the experimental results support the central claims are not sufficiently clear. Despite my efforts to carefully follow the technical details, I found the paper difficult to understand and struggled to develop a coherent picture of its main contributions. Given these concerns, I encourage the authors to substantially revise the manuscript by clarifying the central research narrative, strengthening the connections between methodological choices and their underlying motivations, and providing more convincing experimental evidence. I would recommend thoroughly refining the manuscript before submitting it to an appropriate conference or journal.

Reviewer_Q8uKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the evaluation of explainability methods in MIL models. It argues that filtering the patches of a slide into a candidate pool can change its prediction. Evaluating the explanation against the filtered prediction may overstate preservation of the original prediction which can lead to different conclusions when comparing methods. The authors evaluate this effect in a controlled study leading to 4 of 15 different ranking conclusions. They propose the CHARTER protocol which outlines some recommendations to circumevent this problem when evaluating explanation of MIL models in the future.

Critical strengths
  • The paper points out a flaw in evaluating the faithfulness of explanation MIL models. Looking critically at different evaluation protocols is always an important topic of research.

  • The paper proposes some guidelines on how to address the problem in the future.

Critical weaknesses
  • The paper is really hard to read. It sounds like it was heavily written with a LLM without aiming for any clarity. On a first read of the abstract and introduction, I had no clear idea what the paper is doing. Also, later throughout the paper, it was really hard to grasp the main points and after reading it several times I’m still unsure if I fully understood it. The paper needs a major rewrite for making it in any way understandable for a broader audience.

  • The paper is quite specific to the architecture used in the cited FOCI paper. I don’t think this is a commonly used architecture/protocol. So I’m unsure if this really addresses a significant problem that is widespread in the community.

  • The strategy used for subset selection in the paper is Random-K which selects the patches at random. However, the explanation method subselection procedure is likely not random (depending on the method).

  • This sentence is misleading: “Perturbation-based approaches, including xMIL, likewise assess changes in model output when evidence is removed or restricted” (line 136). xMIL is not perturbation based but uses layerwise relevance propagation (LRP) which propagates evidence from the output prediction back to the input. In the paper, patch removal is used to evaluate the faithfulness but is not part of the method/framework.

Questions for authors
  • Does your investigation uncovered a previous paper beyond the FOCI paper where the ranking would change and the results were therefore misleading?
Reviewer_oVeLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CHARTER, a reference-aware evaluation framework for hierarchical compact-evidence methods. The authors investigate how using full-bag prediction and the intermediate candidate-conditioned prediction may impact the measured fidelity of selected compact evidence. They demonstrate that changing the evaluation reference can alter fidelity measurements and sometimes reverse the rankings of candidate-selection strategies.

Critical strengths
  1. The paper is generally well organized,

  2. The authors investigate the evaluation issue across multiple MIL architectures.

  3. Experiments demonstrate that changing the reference impacts conclusions.

Critical weaknesses
  1. My primary concern is that the observation presented in this paper is quite straightforward. It is not surprising that evaluating the same prediction against different references leads to different fidelity measurements. Similarly, when different selection strategies are evaluated against different prediction targets, the comparative rankings may change. These results appear to be expected consequences of changing the evaluation target, rather than a novel methodological or scientific discovery. The authors provide extensive empirical results. But these results do not seem to reveal a completely novel insight.

  2. The practical significance of the proposed problem are not sufficiently established. If the purpose of compact-evidence selection is to preserve the original full-bag prediction, then the formulation should be focused on identifying a compact subset that minimizes the discrepancy. The resulting subset should then be evaluated against the original prediction. If an intermediate candidate-selection procedure substantially changes the prediction, then the selection procedure can be redesigned. The authors identify this distinction, but it is unclear why it constitutes a significant unresolved research problem. In particular, the paper does not convincingly establish that existing compact-evidence methods systematically use candidate-conditioned predictions while incorrectly claiming to preserve the original full-bag prediction.

  3. The candidate subset selection procedures are not motivated. The primary experiments compare candidate pools based on L2 norms and random sampling, followed by Random-K selection to get the final compact subset. The rationale for using the L2 norm as a measure of patch importance is unclear. A large embedding norm does not necessarily imply a large contribution to the MIL prediction. In addition, the main experiments use Random-K, which does not explicitly attempt to identify subsets that preserve the original prediction. The authors subsequently demonstrate that these selected subsets can produce different predictions and that using different references changes their measured fidelity. It is unclear how informative these observations are for practical compact-evidence methods that explicitly aim to identify faithful subsets.

Questions for authors
  1. Can the authors identify existing compact-evidence methods that incorrectly substitute the candidate-conditioned prediction for the original full-bag prediction?

  2. Why is L2 norm used for candidate construction? Why is it related to the predictive contribution of individual patches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.