acceptodds
Under review as a conference paper at ICLR 2027

ES4SURV: AN EVIDENCE-SEEKING MULTIMODAL LARGE LANGUAGE MODEL FOR WHOLE-SLIDE SURVIVAL PREDICTION

Abstract

In digital pathology, accurate clinical predictions require integrating both global tissue context and local morphological details. While multimodal large language models (MLLMs) show strong potential, existing frameworks compress thousands of slide patches into a fixed set of global tokens, often discarding fine-grained, clinically decisive evidence. To overcome this limitation, we present ES4Surv, an autoregressive whole-slide image (WSI) MLLM that dynamically bridges macro- and micro-level features for survival analysis. ES4Surv first constructs a global slide representation, then uses that context to guide an adaptive, query-driven finding of high-resolution patch features aligned with predefined histologic concepts. These targeted patches are reinjected into the model, forming an interpretable reasoning pathway from global slide understanding to localized evidence and final prognosis. Across five TCGA benchmarks comprising 2,938 patients, ES4Surv consistently outperforms existing state-of-the-art slide-level models in term of macro disease-specific survival.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computational Pathology, Multimodal Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZCCfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ES4Surv, an evidence-seeking multimodal large language model for whole-slide image survival prediction. Built upon WSI-LLaVA, the method first encodes global slide context and then uses decoder-generated queries to retrieve high-resolution local patches through query-conditioned adaptive grouping. The selected patches are reintroduced into the model for final survival prediction. Experiments on five TCGA cohorts show a macro DSS C-index of 0.738, and the method additionally provides a traceable record of the selected tissue groups and local patches.

Critical strengths
  1. The proposed coarse-to-fine evidence-seeking design is well motivated by the diagnostic workflow of pathologists: it first establishes global slide context and then retrieves high-resolution local evidence for survival prediction.

  2. The method records which tissue groups and local patches are selected for each prediction, making the evidence-acquisition process traceable and providing additional cues for assessing whether a prediction is trustworthy.

Critical weaknesses
  1. The reported absolute gains are comparable to the across-fold standard deviations. Please report fold-wise paired results and appropriate paired statistical tests or confidence intervals to establish whether the improvements over WSI-LLaVA and strong non-LLM baselines are statistically reliable.

  2. ES4Surv provides only modest gains over its WSI-LLaVA backbone (0.726 to 0.738) and strong non-LLM baselines (e.g., SlotSPE: 0.722), despite introducing a 7B MLLM and additional retrieval/decoding steps. Please provide inference-cost comparisons and further justify whether the added complexity is warranted.

  3. ES4Surv builds upon the strong WSI-LLaVA baseline, but the proposed technical components do not convincingly demonstrate a statistically reliable improvement. In Table 2, slide-adaptive grouping alone yields a negligible gain over fixed grouping (0.724 vs. 0.722), and the full query-conditioned design improves the mean score to 0.738 without fold-wise paired results or significance tests. Thus, it remains unclear whether the proposed design yields a robust improvement beyond the strong baseline.

  4. While the evidence-acquisition path is traceable, its pathological validity remains insufficiently validated. The paper shows which groups and patches are selected, but does not establish whether they correspond to clinically meaningful prognostic evidence. Expert evaluation, comparison with pathology annotations, or intervention-based analyses would strengthen the claim that the retrieved evidence is trustworthy.

Questions for authors

See Weaknesses.

Reviewer_FJeVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ES4Surv, an MLLM for WSI survival prediction. It first constructs a global representation of the slide, then employs queries generated by a decoder to locate relevant local evidence. Its "QAG-Seek" module organizes patches into groups aligned with specific concepts, ranks both the groups and the patches within them, and subsequently feeds the selected high-resolution patches back into the MLLM to produce a final prediction. Evaluated across five TCGA cohorts, ES4Surv achieves an overall C-index of 0.738, compared to 0.726 for the WSI-LLaVA baseline, which relies solely on global information.

Critical strengths
  1. The closed-loop design, which re-injects selected high-resolution patches into the MLLM, addresses a limitation of the fixed compressed tokens used by existing WSI MLLMs.

  2. The selected patches are location-preserving and concept-grouped, so the evidence entering the prediction can be inspected directly.

Critical weaknesses
  1. The necessity and advantages of incorporating an MLLM have not been justified. The core of this paper lies in identifying key patches and aggregating them for survival prediction. However, the authors fail to explain why an MLLM is required for patch identification.

  2. Similarly, the paper also does not provide a rationale for why an MLLM should be used for survival prediction.

  3. The key design lacks ablation experiments. For instance, no individual ablation experiments were conducted for the four loss components. Furthermore, the number of query tokens and the number of groups have not been subjected to ablation analysis.

Questions for authors
  1. Given that both visual-language foundation models (such as [1]) and slide foundation models with attention heatmaps (such as [2], which use [3] to get attention maps) can highlight key slice regions at very low computational cost, what are the specific advantages of patch region selection methods driven by MLLMs compared to these alternatives? I have also noticed some recent work (for example, [4]) that employs a retrieval-based patch selection method for WSI analysis using MLLMs. What are the advantages of the "query token" approach used in that work compared to the method discussed here?

  2. Why is it necessary to use an MLLM for survival prediction? Many existing vision-only MIL methods, as well as various vision-language approaches, already perform very well in this task. Given that the results of ES4Surv are not particularly outstanding, what exactly are the advantages of ES4Surv?

  3. Since the proposed grouping mechanism is inherently prototype-based, what specific advantages does it offer compared to existing prototype-based methods such as PANTHER [5]? Furthermore, to what extent do query conditions actually alter the grouping results in practice?

[1] Lu, et al. "A visual-language foundation model for computational pathology." In Nature Medicine.

[2] Zhu, et al. "A unified vision-language model for precision oncology and biomarker prediction in neuroblastoma." In Nature Communications.

[3] Shao, et al. "Do multiple instance learning models transfer?." In ICML.

[4] Shao, et al. "QPath: A Question-Aware Multi-Resolution Vision Language Framework for Computational Pathology." In ICASSP.

[5] Song, et al. "Morphological prototyping for unsupervised slide representation learning in computational pathology." In CVPR.

Reviewer_SnKDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses an important limitation of multimodal large language models (MLLMs) for whole-slide image (WSI) survival prediction: existing approaches typically compress thousands of high-resolution patches into a fixed set of global visual tokens before reasoning, potentially discarding fine-grained prognostic information before the model determines which evidence is relevant. The authors propose ES4Surv, a decoding-time evidence-seeking framework intended to mimic a pathologist’s top-down workflow. Starting from 256 global GigaPath slide tokens, the model generates slide-conditioned visual queries. These queries drive the proposed Query-Conditioned Adaptive Grouping and Patch Seeking (QAG-Seek) module, which adaptively organizes the patch bank into concept-aligned groups using iterative FiLM-conditioned updates, ranks candidate groups and patches, and reinjects selected local features into a second decoding stage for survival prediction. Training combines survival prediction with auxiliary group-level supervision and concept-alignment objectives. The method is evaluated on five TCGA disease-specific survival cohorts comprising 2,938 patients. ES4Surv reports a macro C-index of 0.738 ± 0.016, compared with 0.726 ± 0.019 for a matched WSI-LLaVA baseline, and also outperforms several MIL-based methods in macro performance. The paper includes extensive ablations of adaptive grouping, query conditioning, patch-selection budgets, selection interventions, and survival readout design. Overall, the proposed decoding-time evidence acquisition paradigm is interesting and well motivated. However, the current experimental protocol and interpretation of the evidence-selection mechanism raise several concerns that should be addressed before strong conclusions regarding performance, robustness, and interpretability can be drawn.

Critical strengths
  1. Decoding-time evidence seeking is a well-motivated architectural contribution. ES4Surv addresses a fundamental limitation of existing WSI MLLMs, which compress gigapixel slides into global tokens before reasoning and may therefore discard fine-grained prognostic information prematurely. QAG-Seek instead allows the model to establish global context and then dynamically retrieve and reinject patient-specific local evidence during decoding. This is consequential because it provides a principled mechanism for integrating global and fine-grained WSI information rather than relying exclusively on a fixed precomputed representation.

  2. The matched baseline and controlled selection interventions provide meaningful evidence for the proposed mechanism. The comparison with WSI-LLaVA controls for the backbone, folds, projector, LoRA configuration, loss, and training protocol, helping isolate the contribution of evidence seeking. Moreover, under the same evidence budget, randomizing patches or groups reduces performance relative to the complete ES4Surv pipeline. These controls strengthen the paper because they indicate that the reported benefit is associated with learned evidence organization and selection rather than simply injecting additional visual tokens.

Critical weaknesses
  1. The evaluation protocol does not establish that the reported performance gain generalizes. The headline improvement over the matched WSI-LLaVA baseline is modest (0.738 vs. 0.726) relative to the reported fold-level variability, and no paired significance test or confidence interval for the difference is provided. More importantly, checkpoint selection is performed on the evaluation folds, with the better validation C-index reported, rather than using an independent held-out test estimate. Nearby hyperparameter configurations also fall at or below the matched baseline. This is consequential because it remains unclear whether the reported gain reflects a robust benefit of evidence seeking or, at least partly, model/checkpoint selection on the evaluation data. Nested cross-validation or an independent test split with paired statistical testing is needed to substantiate the central empirical claim.

  2. The concept-level interpretability claims are insufficiently validated. The 13 histologic concepts are grounded using automatically screened exemplars rather than independent pathologist annotations, and the paper does not quantitatively validate whether learned group assignments correspond to the intended tissue categories. This concern is compounded by potentially counterintuitive concept distributions and by adaptive pooling, under which selected entries may not correspond one-to-one with original high-resolution tiles. Consequently, the framework provides an inspectable evidence-selection pathway, but the stronger claim that it reliably identifies specific histologic concepts and exact prognostic patches is not yet adequately supported. This weakens one of the paper’s central claimed advantages beyond predictive performance.

Questions for authors

Can the authors report performance using a genuinely held-out evaluation protocol, such as nested cross-validation in which checkpoint and hyperparameter selection are performed exclusively on an inner validation split? Please also provide paired confidence intervals or statistical tests comparing ES4Surv with the matched WSI-LLaVA baseline. This would clarify whether the reported 0.012 C-index improvement represents a statistically reliable generalization gain rather than an effect of checkpoint/model selection.

How reliably do the learned groups correspond to the intended 13 histologic concepts? If independent tissue annotations are available, can the authors quantify group-to-concept agreement? Additionally, what fraction of slides undergo adaptive pooling to the 15,000-entry patch bank, and how are selected entries localized when one entry represents multiple original tiles? These clarifications are important for determining whether the claimed concept-level and patch-level interpretability is supported.

Independent external validation on a non-TCGA cohort would be valuable for determining whether the evidence-seeking and concept-alignment mechanisms generalize across institutions, staining protocols, scanners, and patient populations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.