acceptodds
Withdrawn from ICLR 2027

Don't Trust, Verify: Multi-Agent Framework with Adversarial Evidence Cross-Examination for Whole Slide Image

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Whole slide image (WSI) analysis is increasingly approached by agentic systems that emulate a pathologist’s workflow—surveying a slide and zooming into suspicious regions to accumulate evidence. We argue that this exploratory paradigm shares a fundamental blind spot: it over-trusts the perceptions of underlying multimodal large language models (MLLMs) and seeks to improve performance by gathering additional visual observations, even though navigating to different regions often cannot recover features that the model cannot reliably resolve. We introduce ReMIL-Agent, a training-free multi-agent framework operating along an orthogonal verification axis based on multiple instance learning (MIL), in which each MLLM-reported feature is treated as evidence to be actively cross-examined rather than accepted at face value. A cascade mechanism then revisits an initial prediction, subjects its supporting morphological cues to an adversarial Defender–Critic process grounded in a pathology knowledge base, and selectively re-examines key evidence across scales only when warranted, ultimately retreating to UNCERTAIN at the limits of VLM capability. Finally, a reviewer module performs multi-scale reasoning to produce interpretable and auditable pathological reports. Empirical results demonstrate that ReMIL-Agent is a generalizable workflow agent framework that can be built on top of existing MIL methods, enabling more transparent reasoning with explicit capability boundaries and further advancing trustworthiness for clinical pathologists.

Primary area:
interpretability and explainable AI
Keywords:
agentic AI, multi-agent systems, computational pathology, multiple instance learning, multimodal reasoning, evidence verification

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_F5VKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReMIL-Agent for whole-slide image classification and diagnostic explanation. The framework first obtains an initial prediction from an existing MIL model, then verifies the morphological evidence supporting that prediction and revises it only when sufficient evidence supports an alternative diagnosis. It adopts a multi-agent cascade consisting of a Predictor, a knowledge-grounded Verifier, and a planner-guided Reviewer, and evaluates the framework on multiple WSI classification benchmarks.

Critical strengths
  1. Clear motivation and a coherent verification framework. The paper addresses the problem that additional visual exploration may not resolve unreliable morphological observations in agentic pathology systems. It introduces knowledge-grounded Defender–Critic verification, cross-scale consistency checks, and conservative prediction revision. These components form a coherent workflow centered on evidence reliability, while structured evidence records improve the traceability of the diagnostic process.
  2. Consistent performance improvements. Without retraining the underlying models, the framework improves both ABMIL and Flow-MIL on all six TCGA subtyping tasks and provides an additional improvement on CAMELYON16.
Critical weaknesses
  1. The evaluation does not fully establish the added value of the multi-agent design. The quantitative evaluation is limited to classification tasks, while evidence reliability, diagnostic report quality, and auditability are primarily illustrated through case studies. Although the component ablations provide useful evidence, it remains unclear whether the full multi-agent architecture is necessary compared with simpler evidence aggregation or prediction correction approaches. More direct evaluation of the claimed verification benefits would strengthen the contribution.
  2. The accuracy–efficiency evaluation is incomplete. The classification comparisons include MIL, MLLM, and agent-based methods, but the runtime comparison is limited to PathAgent on two datasets. Although stage-wise timings are provided, the paper does not present a comprehensive comparison of accuracy and computational cost against the original MIL predictors and simpler correction pipelines under matched conditions.
Questions for authors

Can the authors provide a broader accuracy–runtime comparison that includes the original MIL predictors and additional agent-based baselines? Please clarify whether offline feature extraction and preprocessing costs are included consistently in the reported timings.

Reviewer_d83FOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReMIL-Agent, a training-free multi-agent framework for WSI analysis that integrates MIL-based verification and adversarial evidence cross-examination to improve the reliability and interpretability of MLLM-based pathology reasoning. The framework selectively re-examines morphological evidence across scales, explicitly acknowledges model limitations, and generates auditable pathology reports.

Critical strengths
  1. Novel problem formulation: The paper reframes agentic WSI analysis from iterative visual exploration to evidence verification, addressing the reliability limitations of MLLM-generated morphological observations.

  2. Modular and training-free framework: ReMIL-Agent integrates MIL-based prediction, knowledge-base-grounded adversarial verification, and selective multi-scale review, allowing it to work with existing MIL backbones without retraining.

Critical weaknesses

The idea is good but seems far away from the real applications of pathology.

  1. The SOTA of WSI cancer subtyping tasks are foundation model like UNI-2, Virchow2, Gigapath [1][2][3], where the role of MIL is relatively incremental. The implementation of baselines (different MIL) does not mention the embedding details. I doubt the baselines in the experiments are weak baselines which are far away from SOTA. What's the meaning of generative models as baselines if the FM embedding + ABMIL achieves the SOTA?
  2. The experiment setting is tricky, only with internal testing, without five-fold, without significant test. The trend of different models can be very different using different split especially in the cases here there is only internal testing and the testing #cases are very limited.

[1]Chen, R. J., et al. Towards a General-Purpose Foundation Model for Computational Pathology. Nature Medicine, 2024 [2] Zimmermann, E., et al. Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathology. arXiv preprint, 2024 [3] Xu, H., et al. A whole-slide foundation model for digital pathology from real-world data. Nature, 630, 181–188, 2024

Questions for authors

I hope the authors should compare with the real SOTA methods and experiment in a fair setting of WSI pathology tasks.

Reviewer_P8bgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReMIL-Agent, an inference pipeline that improves existing trained MIL classifiers for whole-slide pathology analysis. A confidence gate sends only uncertain cases to a Defender–Critic verifier, which compares multi-scale morphological descriptions against a pathology knowledge base. Cases that still remain unresolved are passed to a planner-guided reviewer, which looks at additional visual evidence and revises labels conservatively. The framework produces structured evidence traces and does not retrain the underlying MIL models. Experiments cover six TCGA subtyping tasks and CAMELYON16, using two MIL backbones. They also include component and knowledge-base ablations, alternative agent settings, and runtime measurements.

Critical strengths
  1. On six TCGA tasks, average BACC rises from 83.12% to 85.60% for ABMIL and from 84.62% to 87.17% for Flow-MIL; on CAMELYON16, ABMIL improves from 93.59% to 96.79%. Ablations confirm the cascade works as intended.

  2. Each decision keeps patch coordinates, quoted evidence, and knowledge-base references. A label can be flipped only with positive evidence against it. Stage-wise invocation counts and runtimes are reported in the appendix, so selective computation can be verified directly.

Critical weaknesses
  1. This work claims the system verifies evidence and knows its own capability limits, but this is not shown directly.

In Appendix A.3.2, Stage B extracts features and quotations from the VLM's own narrative. It only checks whether the text matches itself. It does not confirm that the morphology actually appears in the image. Cross-scale consistency can catch contradictions, but shared perceptual errors still pass through. The PathKB ablation shows its contribution to BACC, yet neither the ablation nor the case studies score the correctness of accepted morphological claims. The standalone benefit of the Defender–Critic debate is also not isolated.

It is unclear whether gains come from verification or further exploration. Algorithm 1 keeps the original MIL label whenever the final decision is not REVISE, including residual UNCERTAIN cases. The final uncertainty signal and its reliability are never evaluated. The evidence supports a conservative correction workflow more than a measured capability boundary.

On novelty, the paper should state how this cascade differs from WSI-Agents [1], which already combines evidence-consistency checks, pathology knowledge, and classifier-based verification.

  1. It does not fully prove its edge over non-agentic WSI predictors, and generalizability is not tested well.

UNI-based MIL and KEEP are included, but Table 1 misses a representative slide-level foundation baseline such as TITAN [2], which the paper itself discusses in Appendix A.1. The matched MIL gains average about 2.49 and 2.55 BACC points, but there are no paired uncertainty estimates or repeated runs to show robustness. ReMIL-Agent with ABMIL still trails standalone Flow-MIL on BLCA (i.e. 57.79% and 65.43%) and THCA (i.e. 72.71% and 74.69%), although the Flow-MIL cascade does improve its own baseline.

The added computation is large. Table 11 reports 0.50 to 20.67 s/WSI on BLCA and 0.41 to 45.06 s/WSI on THCA for ABMIL, under the cached-feature setting. Being faster than PathAgent does not prove a favorable trade-off against plain MIL. Routing and revision settings are left unspecified, and WSI input/aggregation protocols for locally evaluated vision-language baselines are incomplete. The seven benchmarks cover several tasks, but no fixed-configuration transfer to an independent, label-matched cohort is tested.

References

[1] WSI-Agents: A collaborative multi-agent system for multi-modal whole slide image analysis. arXiv:2507.14680, 2025.

[2] A multimodal whole-slide foundation model for pathology. Nature Medicine, 2025, 31(11): 3749–3761.

Questions for authors
  1. The evidence-verification claim rests on whether accepted morphological findings are truly visible in the tissue, yet the paper reports no direct check of this. Could the authors run an independent audit on the patches already cited in the case studies, holding the visual observations fixed and comparing findings before and after the consistency rules?

  2. The behavior after a final UNCERTAIN verdict is left unspecified. Does the system return an unresolved report flag, retain the original MIL prediction without abstaining, or abstain from prediction altogether?

  3. The routing and revision rules are described only in part. Please state the confidence thresholds that route a case to the Verifier or Reviewer, the margin required for a REVISE verdict, the evidence budget per round, and the normalization rules applied to scores across classes and scales. The data used to choose these values should also be described, together with a clear division between settings shared across tasks and settings tuned per task.

  4. The vision-language baselines run locally lack a documented input protocol. Patch selection, image count, magnification and resolution, prompts, and the rule combining per-patch outputs into one WSI label should be specified, with the standalone KEEP procedure described to the same level of detail. The daggered MLLM results come from CPathAgent's two-stage description-to-label evaluation [3], and the new baseline protocols and visual input budgets should be compared against that setup so readers can judge whether the baselines received a fair image budget.

References

[3] CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logic[J]. Advances in Neural Information Processing Systems, 2026, 38: 101673-101731.

Reviewer_oR5qOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Whole slide pathology images (WSIs) are gigapixel. Multiple instance learning (MIL) classifiers are accurate but opaque: they give a label without checkable evidence. Vision language models (VLMs) describe morphology, but their descriptions are unreliable at WSI resolutions. Recent "agentic" systems explore slides like a pathologist: survey, zoom, gather observations. This paper argues exploration cannot fix perceptual failure: if the VLM cannot resolve a feature, looking again does not add it. ReMIL Agent adds a verification axis instead.

Three stages, all training free, sit on top of an existing MIL classifier (ABMIL or Flow MIL, on UNI features). First, a Predictor gives a diagnosis, a confidence score, and three patch pools (support, counter, context) built from MIL importance scores and farthest point sampling. Second, low confidence slides go to a Verifier: a Defender agent argues for the diagnosis and a Critic argues against it, both grounded in PathKB, a knowledge base of pathognomonic features derived from WHO diagnostic criteria, with rules for scale appropriate observation and cross scale confirmation. The verdict is ACCEPT, REJECT, or UNCERTAIN. Third, UNCERTAIN slides go to a Reviewer: a planner guided multi scale re examination that maintains a belief table over all classes and revises the diagnosis only on consistent alternative evidence.

Reported results: on six TCGA cancer subtyping tasks, average balanced accuracy (BACC) rises from 83.12 to 85.60 with ABMIL and from 84.62 to 87.17 with Flow MIL. CAMELYON16 rises from 93.59 to 96.79. Ablations for Verifier, Reviewer and PathKB exist. Average time per WSI on BLCA is reported as about 21 seconds versus about 141 for the agent baseline PathAgent.

Critical strengths
  1. The framing is right. VLM described morphology is unreliable at WSI scale, so treating VLM outputs as evidence to cross examine rather than conclusions to trust is the correct instinct for clinical use. The retreat to UNCERTAIN, with explicit capability boundaries, is the responsible default.
  2. PathKB is a thoughtful hallucination suppression design. Feature strength is looked up from a curated table, not self assessed by the LLM. Every feature has a native observation scale, cross scale confirmation is required, and known mimics must be excluded first (Section A.2).
  3. Modular ablations (Verifier, Reviewer, PathKB in Tables 3 to 4) and an efficiency comparison against an exploration style agent support the cascade design.
  4. The evidence chains are the real product. The case studies (Section 4.5, for example the thyroid case) show that the framework can produce an auditable trail of what was observed, contested and revised. For clinical trust this matters more than the accuracy points.
Critical weaknesses
  1. The evidence quality does not support the claims. There are no standard deviations, no confidence intervals and no significance tests anywhere, only single run numbers on small per task test sets. Gains are uneven (THCA +2.0, RCC +1.25 with ABMIL), and "substantially improving diagnostic accuracy" is not supported as stated.
  2. The comparisons are not comparable. CPathAgent and DSMIL numbers are quoted from their papers (†), possibly under different setups. The one agent baseline the authors actually ran (PathAgent, ‡) scores implausibly low, average 55.30, far below any MIL method, which makes the about 30 point "agentic baseline" comparison meaningless. ESCA reaches exactly 100.00 BACC without explanation.
  3. "Deterministic" verdicts rest on stochastic calls. Patho R1 and Qwen3 to 4B at unspecified decoding produce the inputs that the deterministic adjudication rules consume, and the paper never measures how often verdicts flip across repeated runs of the same slide.
  4. PathKB has no human validation. It was drafted by one LLM (Gemini) and audited by another (Claude), and no pathologist review is reported. Versions are inconsistent too (Gemini 3.5 Flash in Section 3.2, Gemini 3.1 Flash in the AI use statement, the citation points to the 2023 Gemini paper). For a clinical evidence verification system this is a methodological hole.
  5. The routing gate can miss the worst errors. A confidently wrong MIL prediction never reaches the Verifier. The paper gives no calibration analysis of the MIL confidence and no false ACCEPT analysis of the gate.
  6. Presentation problems. A leftover fragment "Under review as a conference paper at" sits at the end of Section 4.2, and summary sentences overclaim ("substantially improving diagnostic accuracy, efficiency, and interpretability").
Questions for authors
  1. Please report mean and standard deviation over at least three runs (or patient level bootstrap intervals) for Tables 1 to 2, plus a per slide paired comparison against each backbone.
  2. Why does PathAgent‡ score only 40 to 61 across tasks? Is your re execution faithful to the released pipeline (prompts, magnification, evidence budget)? Can you share its logs?
  3. What happens to confidently wrong MIL predictions? What fraction of backbone errors does the confidence gate miss, and what is the cost of verifying all slides?
  4. How stable are Verifier verdicts over repeated VLM or LLM calls and prompt permutations? Please report the flip rate.
  5. Who clinically validated the PathKB entries for each task? If none, justify fully automated KB construction for clinical subtyping.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.