- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a framework for combining predictions from multiple histopathological stains (H&E, IHC) using task-specific ensemble rules designed by an LLM agent. One base model is trained per stain using virchow2 tile features with an ABMIL model and then frozen. An LLM agent is then tasked with finding the best classifier for a given task. The methodology used by the agent is to propose a set of rules starting with a standard set, apply an optimizer to fit their combination weights and finally analyze the results to edit the set of rules and propose new ones created from learned combinations and cross-task members. The agent iterates over until it decides to stop. The method is evaluated on 4 cohorts (3 organs) both internal and external and shows an edge over fusion methods and learned ensembles.
Critical strengths
-
integrating information from multiple stains is an important problem in computational pathology. The proposed prediction-level fusion approach avoids the need for precise spatial registration between stained sections and accommodates incomplete stain panels, addressing two practical difficulties with multi-stain data.
-
separating stain-specific feature extraction and prediction from the integration procedure is appealing. Existing experts can be reused, new stains can be introduced without redesigning the entire architecture, and the final integration rule can run without an LLM. This is a potentially useful alternative to complex end-to-end fusion architectures, particularly when labeled datasets are small.
-
the evaluation covers three organs, multiple stain panels, classification and survival tasks, and an external validation cohort. The comparisons include conventional prediction-level ensembles, learned stacking approaches, and multi-stain architectures such as UNICORN and PathMoE.
-
the paper study the contribution of the candidate pool, agent-generated extensions, and search strategy and acknowledge that conventional greedy ensemble selection performs nearly as well as the agent-designed search when given the same candidate pool. This provides useful insight into where the improvements originate.
-
the evaluation separately examines patients with complete and incomplete stain panels. This helps establish that the improvements are not exclusively attributable to handling missing stains, except in the lymphoma cohort.
Critical weaknesses
-
i find that the contribution of the LLM agent is not sufficiently established. There is no ablation that replaces the LLM with other non-agentic techniques (Random Forests, SVM). Because the proposed method (the agentic part) is general (not tied to pathology) it should be validated more broadly against previously published methods such as CAAFE, FeatLLM, RuleFit, TPOT, RLIE. Also, for the same reason, 4 pathology cohorts is not sufficient to establish superiority of the approach over SOTA and established tabular benchmarks should be used.
-
the ablation is only done on 2 cohorts (both kidney), so it is hard to know if it generalizes to other organs.
-
hyperparameter tuning and optimization budgets for the baselines UNICORN and PathMoE are nor reported.
-
a concrete example is needed. The main contribution is the agentic iterative rules selection and generation but there are no example to illustrate the process (initial rules, diagnostic feedback, agent-proposed modifications, intermediate objective values, structure and weights of the final program). This makes it difficult to understand what the LLM actually invents and whether its modifications go beyond established ensemble techniques. If space is limited, it should be provided in the appendix.
-
Results are averaged over five splits without reporting standard deviations. Mean ± standard deviation, particularly for paired performance differences (figure 2), would help establish stability across splits and statistical significance.
-
On what data exactly is each stain virchow2 MIL model trained ? Is it with the same 4 datasets ? Is it with the same splits as for training the final classifier program ? I could not find this critical info.
-
How is the virchow prediction integrated over the tiles of the WSI slide ?
-
What is the complexity of training the final program with the agent ? specify number of tokens, total time.
-
The learned combiners are fitted exclusively on validation predictions rather than using out-of-fold predictions from training data. This is potentially data-inefficient, especially for small cohorts, so the authors should explain the choice. Also, combiners evaluated on one validation block are trained on other validation blocks. It should be clarified that these blocks are patient-stratified.
-
The manuscript claims that only the kidney cohorts provide multiple validation blocks, despite reporting five splits for all cohorts.
-
clarification needed: the authors describe five random 60/20/20 train/validation/evaluation splits but state that a single evolved integration program is shared across all splits. If the program is optimized jointly using the validation predictions and labels from these splits, patients used for validation in one split may also appear in the evaluation set of another. This would compromise evaluation independence. The authors must clarify the splitting protocol and demonstrate that evaluation patients have not influenced program selection.
-
The method uses within-cohort rank transformations. How does it work for inference on individual unseen patients. Please clarify.
-
MEST-C tasks are not explained nor referenced. Also it is not clear whether there's a separate ABMIL model trained for each task.
Questions for authors
see weaknesses.