acceptodds
Withdrawn from ICLR 2027

WholeSight: Learning to Think with Whole Slide Images

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reasoning over a whole slide image (WSI) requires collecting evidence from sparse tissue regions across spatial locations and magnifications. A WSI agent must decide where to look, what to inspect at higher magnification, and how new observations should guide subsequent reasoning. Existing WSI multimodal systems have improved either slide representations or active inspection, but visual evidence acquisition is typically fixed, prescribed at inference time, or learned separately from reasoning. We introduce WholeSight, which instead treats the WSI as an interactive environment in which visual search and reasoning are learned jointly as a single policy. A WSI harness exposes regional search and local inspection as executable actions, allowing the agent to acquire and interpret evidence over successive observations. Supervised fine-tuning provides a cold start for reasoning and tool use from complete interaction trajectories, after which reinforcement learning optimizes evidence acquisition and reasoning through the agent's own slide interactions. Across 2,956 questions spanning ten evaluation settings, WholeSight ranks first in eight and outperforms the strongest evaluated WSI system of comparable scale by 11.10 percentage points overall. Relative to its original multimodal large language model backbone under the same visual interface, the learned policy improves performance across every evaluation, including slide sources outside the training collection drawn from The Cancer Genome Atlas. Our code is available at https://anonymous.4open.science/r/anonymous-method-release-A53C/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
medical image analysis, computational pathology, whole slide images, visual agent, agentic reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_z5wsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WholeSight, a multimodal model that uses regional search and local zooming to answer questions about whole-slide pathology images. Training starts with supervised fine-tuning on complete interaction trajectories, followed by reinforcement learning. The model achieves 54.13% accuracy across 2,956 questions in ten evaluation settings. This is a 12.01 percentage-point gain over the original backbone with the same tool interface. The paper also examines observation strategies, tool-call budgets, and visual history.

Critical strengths

1.The comparison using the same backbone and tool interface is convincing. Accuracy rises from 42.12% to 54.13%, with gains in all ten evaluation settings. This provides direct evidence that the training pipeline improves performance. 2.The paper addresses an important problem in whole-slide analysis: finding the evidence needed to answer a question. It brings evidence acquisition and answering into a single training pipeline. The evaluation spans multiple tasks and slide sources. Overall, this is a well-developed system.

Critical weaknesses

1.The methodological contribution is still unclear. Supervised fine-tuning, tool use, and reinforcement learning are well established. The paper mainly brings these components together for whole-slide analysis. The results show that training improves the system. They do not yet establish what this interactive approach adds over other reasonable training strategies. This is my main reservation about acceptance. 2.The local-inspection experiments do not cleanly isolate the value of adaptive decisions. Fixed and adaptive cropping differ in image size, number of observations, and reasoning procedure. The trained model already reaches 50.61% using only the initial retrieved views, compared with 54.13% for the full system. Local inspection helps, but the controls do not separate the effects of location selection, crop scale, and sequential decisions. A baseline that plans all crops upfront would test the core idea more directly. It should use the same backbone and observation budget.

Questions for authors

1.What are the actual input resolutions, visual token counts, and tool-call counts for fixed and adaptive cropping? Does adaptive inspection still help when these factors are matched? 2.How does the model perform if it plans all high-magnification views upfront from the initial low-magnification images, then answers? Is there evidence that revising the plan after each observation provides an additional benefit? 3.Please provide the exact format-reward settings and visual preprocessing configuration. Could you also report confidence intervals or repeated runs for the main results? These details would help assess reproducibility and interpret the smaller performance differences.

Reviewer_USqUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed an agentic RL framework that learns "Search" and "Crop_view". The policy generates some keywords for KEEP to retrieve some coarse 5x views, and "Crop_view" can further crop detailed 20x image from the 5x coarse views. The training stage can be divided into two parts, the first part uses GPT 5.6 to generate trajectories for SFT cold start, then the second part uses DAPO to further train the policy model. On 10 evaluation settings, the authors reported 11.1% overall performance gain over base model.

Critical strengths
  1. This paper successfully adopts an established framework "thinking with images" to WSI.
  2. Reasoning with WSI is an important problem, and the authors built a useful harness framework for reading WSI.
  3. The experiments cover 10 evaluation settings, many baselines, and rich ablation studies.
Critical weaknesses

W1 The source of the performance gain is still unclear.

(a) The policy model only provide the keywords in the "Search" module, and the load-bearing part, which is selecting which coarse views to look at, is determined by a pretrained model like KEEP. The policy model never sees the WSI, it could be just reading the question, generate some keywords, and then see the coarse views directly. Important baselines, such as using the question itself as keyword or using some general pathology keywords, are missing.

(b) The "Crop" module doesn't seem to do a lot of work. Out of the 11% total performance gain, ablating the "Crop" module only harms the performance by 3.5% (Shown by Table 4). Please show "base model + 'Search'" as baseline.

W2 Missing behaviour analysis

Whether the framework can teach the model "where to look" is the key to the contribution of this paper. However, whether the model is looking at the right place is not shown. Does training changes the behaviour of the model, like how often do they crop, or where do they crop? Does the area they crop overlap with lesions?

W3 Lack of Novelty

This paper looks just SFT cold start + agentic RL, which hardly has any ML novelty. The transfer to WSI is straightforward. Most empirical performance gain likely come from the harness, which cannot constitute sufficient ML contribution. The authors also did not report the performance of the teacher model (GPT5.6) with the harness.

Questions for authors

See the above "weakness" section. I may reconsider my rating based on the authors' response.

Reviewer_a8AZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an agentic visual search and reasoning framework for gigapixel whole-slide images (WSIs). The authors primarily treat these images as interactive training environments and introduce an interaction harness strategy to facilitate multiscale evidence acquisition through regional exploration and high-magnification inspection, enabling the capture of both global and local spatial context. The proposed framework achieves an 11.10% performance improvement over the baseline and demonstrates superior performance across several evaluation scenarios involving whole-slide image visual question answering.

Critical strengths
  1. This paper presents a well-formulated approach to treating gigapixel whole-slide images as interactive training environments. In particular, this formulation enables the model to acquire visual information adaptively rather than relying solely on predetermined or fixed observations.

  2. The integration of multiscale evidence acquisition is a good choice, as it enables the model to jointly learn region selection and downstream reasoning while preserving both global and local spatial relationships for understanding complex structures.

  3. The combination of supervised fine-tuning and online reinforcement learning is a promising approach, as it facilitates adaptive learning and sequential visual exploration and reasoning through whole-slide image interactions. This enables the agent to progressively refine its evidence acquisition strategy.

Critical weaknesses
  1. Although the authors perform retrieval encoder ablations and demonstrate robustness across alternative feature representations, the reliability of the candidate region search space is not directly established. If the agent's interactions are constrained by the retrieved regions, its ability to discover diagnostically relevant evidence may depend more on the retrieval stage than on the learned policy. An analysis of candidate recall, retrieval failures, and performance under deliberately incomplete candidate sets would strengthen the claim of autonomous evidence acquisition.

  2. The authors evaluate the framework across multiple whole-slide image sources and downstream tasks to demonstrate its generalizability, these evaluations do not fully establish whether the agent can reliably identify diagnostically important regions and maintain its performance on unseen slide distributions. The authors are encouraged to provide further analyses of cross-distribution transferability, comparisons under matched observation budgets, and direct evidence grounding to better substantiate their claims.

Questions for authors
  1. Could the authors clarify how online reinforcement learning improves the visual search policy rather than primarily enhancing downstream answer prediction? The authors are encouraged to include additional experiments that evaluate the learned policy under identical visual observations to distinguish improvements in evidence acquisition from those in reasoning performance.

  2. Although the authors investigate several retrieval encoders, how sensitive is the proposed framework to the quality and completeness of the initially retrieved candidate regions? The authors are encouraged to provide additional analyses of varying candidate recall, retrieval failures, and performance under incomplete candidate sets to better assess the robustness of the evidence acquisition process.

  3. In the comparison among full candidate region inspection at 10× magnification, four fixed center crops, and adaptive inspection, are all methods evaluated under equivalent visual information and computational budgets? The authors are encouraged to provide additional performance comparisons that control for cumulative image tokens, total inspected tissue area, inference latency, and the number of tool calls to ensure a fair evaluation.

  4. Could the authors provide additional experiments to evaluate the framework's transferability to entirely unseen WSI datasets without further fine-tuning? Furthermore, add comparisons under matched observation budgets and quantitative evidence-grounding evaluations, such as measuring the relevance and recall of diagnostically important regions which can substantiate the claims of generalizable visual search and reasoning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.