acceptodds
Under review as a conference paper at ICLR 2027

Learning When to Reconsider: Adequacy-Guided Inference-Time Interventions for Histopathology VLM Agents

Abstract

Visual agentic systems that work with very large images often cannot process the whole image at once. Instead, they inspect selected regions (patches), ask a large vision-language model (LVLM) to describe what is visible, and use these descriptions to guide later navigation and reasoning. Inadequate or visually irrelevant descriptions can therefore affect the performance of the downstream task. Existing correction methods applied to LVLMs during test-time typically modify attention, decoding, or prompts to reduce visually unsupported responses, but each leaves the agentic system with a decision that cannot be resolved simply by using a stronger model: when should a frozen LVLM perceptor be corrected, and which correction should be used? We study this question in computational pathology, where agents navigate gigapixel scans of tissue specimens known as whole-slide images (WSIs) for visual question answering (VQA), using four LVLMs, three interventions, and two whole-slide image agents. Our experiments reveal two regularities. Descriptions judged relevant, sufficiently complete, and consistent with the ground truth evidence -which we call "adequate" descriptions-support higher downstream answer accuracy than descriptions judged inadequate. Secondly, intervention benefits depend on the model, question type, and baseline evidence quality of the initial description, and the same intervention can repair some descriptions while damaging others. To test whether these empirical patterns transfer to WSI VQA agentic systems and can support useful decisions by selecting one intervention family for low-confidence descriptions and accepting candidates only when their estimated adequacy increases by a fixed margin, we propose RESCUE-ROUTE-RESCORE, a simple controller. On a held-out patch-level benchmark, the proposed controller raises macro description adequacy from % to % and answer accuracy from % to %. Transferred to agentic systems that perform WSI level VQA without any WSI-level tuning, it achieves the highest point estimate of macro answer accuracy in all six dataset-agent comparisons. These findings suggest that intermediate evidence quality can serve as a transferable control signal and that visual agentic systems can improve by learning which evidence to reconsider. Code will be made available on acceptance.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
visual agentic systems, large vision-language models, test-time intervention, whole-slide imaging, computational pathology, visual question answering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kwqEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates when and how a frozen LVLM perceptor in a WSI agent should be corrected at test time. It introduces “description adequacy,” assessed offline by a reference-conditioned text-only LLM and estimated at deployment via a hidden-state probe. Using this signal, the proposed RESCUE–ROUTE–RESCORE framework intervenes only on low-confidence descriptions, selecting among three interventions (attention biasing, VCD, or review prompting) based on question type and confidence level. A revised description is accepted only if its rescored confidence exceeds the original by a fixed margin. On PathMMU, the method improves macro adequacy from 55.7% to 67.2% and reader accuracy from 56.9% to 62.3%. When transferred without further tuning to two additional WSI agents across three benchmarks, it achieves the highest macro accuracy in all six settings, with gains of 0.7–1.7 pp over the strongest fixed intervention.

Critical strengths
  1. Framing test-time correction as an instance-level decision is well motivated. The repair/damage analyses convincingly demonstrate that the effect of each intervention is conditional on the input.

  2. The experimental protocol is rigorous: image-grouped splits prevent data leakage, the controller transfers frozen across two agents and four LVLMs, and computational costs are explicitly reported.

Critical weaknesses
  1. The adequacy judge’s rubric is under-specified: a 0–4 scale is adopted across multiple dimensions, yet no concrete criteria are provided to distinguish adjacent score levels, and the judge prompt itself is not disclosed. This undermines the reliability of the adequacy assessment on which the remainder of the pipeline depends.

  2. The probe relies on the last-layer final-token hidden state across all LVLMs without ablating layers or pooling strategies, despite prior probing work [1,2] demonstrating that these choices substantially affect performance.

  3. It remains unclear how the threshold τ is determined. The paper does not report results under different threshold values, making it difficult to assess how sensitive the method is to this choice.

  4. The intervention router lacks important details. The paper does not report its routing accuracy or its consistency with MedGemma’s scoring results.

[1] LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations. ICLR'2025.

[2] INSIDE: LLMs’ Internal States Retain the Power of Hallucination Detection. ICLR'2024.

Questions for authors
  1. For the MedGemma-27B judge, what are the exact prompt and the criteria that distinguish each score level on the 0–4 scale, and how is score reliability evaluated?

  2. The adequacy probe uses the last-layer hidden state at the final answer token. Why was this strategy chosen, and was it ablated against other layers or pooling (e.g., mean over response tokens)?

  3. What are the results under different τ values, including the case without gating? How often are good descriptions wrongly filtered out, and how often are poor ones kept?

  4. How is the reliability of the intervention router evaluated? The paper does not rigorously assess routing accuracy, which has a direct impact on the overall performance of the agent system.

4 comments

Response to Reviewer 3 - Part 1

R3.1. Judge prompt rubric

We have added the text-judge and image-aware-judge prompts, the reference-free audit prompt, and the anchors distinguishing every score from 0 to 4 in Appendix A.2.1. Equation (10) in the manuscript specifies the binary adequacy score rule. We use a fixed prompt, greedy decoding and strict schema checks that all required fields are present and within range. In a development set check, re-judging a random subset of 500 samples gave 99.6% score agreement. Due to these reasons, we do not believe that the score reliabilty of the judge has a serious effect on our pipeline or our conclusions. We also assess validity separately through image-aware evaluation and blinded pathologist review, as described in R1.1 and R2.1 in the previous responses.

Revised manuscript: Appendix A.2, Equation (10), lines 814–823 (p. 16); Appendix A.2.1, prompts, score anchors and output schemas, lines 866–1022 (pp. 17–19); Appendix E.8, Table 21 and analysis, lines 1925–1943 (p. 36); Appendix E.9.3, Tables 24–26 and analysis, lines 2432–2612 (pp. 46–49).

R3.2. Layer and pooling choice

We thank the reviewer for this great question. We agree that representation choice should be tested and we had done so during the study on the validation set. We have now included our layer and pooling ablation comparing last-token and mean-pooled features at approximately 25%, 50%, 75% and final decoder depth (Table 14).

  • The final-layer last-token AUROCs are 0.7053, 0.6870, 0.6781 and 0.7794 for Patho-R1 3B, Patho-R1 7B, Quilt-LLaVA and MedGemma, respectively. This is the highest point estimate among the reported configurations for Patho-R1 3B and Quilt-LLaVA.
  • For Patho-R1 7B and MedGemma, 75%-depth mean pooling is higher by only 0.0028 and 0.0011, with paired 95% confidence intervals of [−0.018, 0.024] and [−0.015, 0.017]. Both include zero.

It is true that the literature supports testing different representation choices. However, these studies also suggest that the best choice depends on the detection task and the model. While Orgad et al. [1] and INSIDE [2] find advantages for particular token positions and intermediate layers, Ji et al. [3] find that deeper layers perform best for predicting hallucination risk from a query.

There is also precedent for using the final-layer last-token representation in various tasks. In vision-language models, Bi et al. [4] use the final-layer representation of the last prompt token in InternVL and Qwen3-VL to predict reasoning helpfulness, generation length and difficulty. They find that these signals are more linearly recoverable in the reasoning-trained variants, suggesting that the model’s training also matters for what a probe can extract. ReFL [5] separately compares layers and pooling strategies, and finds the final layer and last-token representation to perform best among the tested choices in its corrective-prompting setting.

Our results are consistent with the literature showing useful probing signals in middle-to-late layers, with the best configuration depending on the model and the task. During the study, based on results from the layer ablation in the validation set, we had decided to retain the final-layer last-token representation as a simple and consistent choice across LVLMs, without claiming that it is universally optimal.

Revised manuscript: Section 7.1, lines 368–370 (p. 7); Appendix D.1, protocol, analysis and Table 14, lines 1402–1487 (pp. 26–28).

[1] LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations, ICLR

[2] Chen, C., Liu, K., Chen, Z., Gu, Y., Wu, Y., Tao, M., Fu, Z., and Ye, J. (2024). INSIDE: LLMs’ Internal States Retain the Power of Hallucination Detection. International Conference on Learning Representations (ICLR).

[3] Ji, Z., Chen, D., Ishii, E., Cahyawijaya, S., Bang, Y., Wilie, B., and Fung, P. (2024). LLM Internal States Reveal Hallucination Risk Faced With a Query. Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 88–104.

[4] Bi, J., Song, L., Zhang, D., Liu, P., Sun, G., Wen, L. B., Cai, W., Chen, C., and Xu, C. (2026). Does Reasoning Improve Seeing? Understanding When Vision-Language Models Benefit from Thinking. Proceedings of the 43rd International Conference on Machine Learning, PMLR 306, pp. 8063–8075.

[5] Fan, C., Zhang, J., Zhang, X., Zhang, S., Lv, Z., Tao, J., and Wen, Z. (2026). ReFL: Reflective Feedback Learning for Hallucination Detection of Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers, pp. 19648–19665.

Response to Reviewer 3 - Part 2

R3.3. Threshold sensitivity and filtering errors

We selected the gate and acceptance margin on validation data before fixing the deployed policy. We now report the threshold sweep and error breakdown in our manuscript.

Threshold sensitivity. With δ=0.08\delta=0.08, validation adequacy is 63.1% at both τ=0.6\tau=0.6 and τ=0.8\tau=0.8, and 63.0% without eligibility gating. The corresponding test values are 67.2%, 67.3% and 67.1% (Table 19). Thus, the deployed τ=0.6\tau=0.6 lies at the start of the observed validation plateau.

We also noticed that acceptance is important when eligibility gating is removed. Accepting every routed candidate reduces test adequacy from 65.8% at τ=0.6\tau=0.6 to 61.7% without gating; retaining the 0.08 acceptance margin gives 67.1% without gating.

Which descriptions are harmed or left inadequate? At the deployed setting, 5.18–11.07% of originally adequate descriptions become inadequate, while 56.19–69.90% of originally inadequate descriptions remain inadequate. Equivalently, the controller preserves adequacy for 88.93–94.82% of good originals and repairs 30.10–43.81% of poor originals (Table 20). Each rate conditions on the corresponding original-adequacy group.

Acceptance errors. Across LVLMs, 12.29–36.00% of accepted candidates are inadequate, and 30.05–31.92% of rejected candidates are adequate. These denominators are all accepted and all rejected candidates, respectively. Rejecting an adequate candidate need not harm the output if the original is already adequate. Table 20 therefore also reports harmful replacements and missed repairs. We feel that these are within an acceptable range.

Revised manuscript: Appendix E.6, protocol and analysis, lines 1817–1835 (p. 34), Figure 10 and Table 19, lines 1848–1862 (p. 35); Appendix E.7, definitions, Table 20 and analysis, lines 1867–1910 (pp. 35–36).

R3.4. Reliability of intervention routing

The router predicts whether an intervention will produce an adequate description. We do not have a unique ground-truth intervention labelled as correct for each WSI decision: several interventions may produce useful descriptions, all interventions may fail and the final answer also depends on subsequent acquisition and reasoning, so it is difficult to define a 'routing accuracy'. However, we can assess whether the intervention chosen by the router produces an adequate candidate and we can also point you to the place in the manuscript where we had reported how the router is utilised in WSI runs.

  • Direct evaluation of intervention choice. We use candidates cached from all three interventions on held-out, gate-eligible PathMMU cases. We consider cases containing both adequate and inadequate candidates, and count the router’s choice as successful when its chosen intervention produces an adequate candidate under the MedGemma text-judge label. Either adequate intervention counts when two are adequate. Across the four LVLMs, the router succeeds in 61.7–66.1% of these cases, compared with 47.3–55.7% for the validation-selected best-fixed intervention. This evaluation precedes acceptance and therefore directly assesses the intervention choice.

  • Realized utilization in WSI agents. The manuscript’s policy-utilization table reports observed eligibility, acceptance and replacement rates, together with intervention shares among routed calls, for both WSI pipelines. For example, with PathAgent and Patho-R1 7B, 48.33% of calls are routed for intervention, and 40.42% of routed calls result in acceptance, giving an overall replacement rate of 19.53%. These measurements document the controller’s actual execution in WSI agents alongside the direct evaluation above.

We have added the intervention-choice experiment protocol and results to Appendix E.5.1. We are not sure if this is what you meant, but we hope this answers your concern.

Revised manuscript: Appendix E.5.1, protocol, Equations (26)–(27), analysis and Table 18, lines 1776–1811 (pp. 33–34); Appendix H, Table 31, lines 3241–3250 (p. 61).

Thank you for the additional experiments and clarifications, which address most of my concerns, so I will raise my score to 3 (weak accept). One question remains: the revision uses MedGemma-27B's outputs as the reference for evaluating the router, but it does not report how accurately MedGemma-27B judges whether a description is adequate or inadequate.

Second Response to Reviewer 3

Thank you for raising your score and for this question. As you said, our router reliabilty evaluation measures whether the router selects candidates labelled adequate by the MedGemma-27b judge. But a direct accuracy estimate of the judge might require independent human adequacy labels under the same rubric, since currently those labels come from the judge itself.

We believe that the the usefulness of the judge's supervision is supported by the indirect evidence like the image-aware evaluation, blinded pathologist assessments and downstream answer accuracy measured against benchmark answers. Although these checks support the usefulness of the learned signal, they do not replace a direct validation of the judge.

A large, independent human validation of the MedGemma 27b judge itself would therefore be valuable, and we acknowledge this as a limitation and future direction. However we wish to emphasize that the framework can also use other capable reference-conditioned text judges and is not tied to MedGemma-27B. The judge is a replaceable supervision component, although each alternative would need its own validation.

Reviewer_sszFOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates selective test-time interventions for histopathology vision-language model (VLM) agents. The authors introduce description adequacy as an intermediate supervision signal and propose Rescue–Route–Rescore, a framework that (1) estimates whether an initial patch description is adequate, (2) selects among three existing correction mechanisms based on question type and adequacy confidence, and (3) accepts a corrected description only when its estimated adequacy improves sufficiently.

The framework is trained on patch-level data and evaluated on both PathMMU and downstream whole-slide image (WSI) VQA agents. The reported results show improvements over ordinary inference and fixed intervention baselines.

Critical strengths
  • The paper addresses a relevant problem: errors in intermediate visual descriptions can propagate through multi-step agent reasoning.

  • The observation that intervention effectiveness varies across models, tasks, and initial description quality is interesting.

  • The evaluation covers multiple VLMs, datasets, and agent frameworks.

Critical weaknesses
  • The adequacy signal lacks validation. My main concern is whether the adequacy metric actually reflects visual grounding. Because the LLM judge only sees reference text instead of the images, it likely rewards answers that match references rather than claims supported by visual evidence. Since the probe and router both depend on this signal, the paper's core premise rests on shaky ground. Verifying this with multimodal judges or human expert annotations is necessary.

  • Margins for adaptive routing are small and confounded. Gains over the strongest fixed-intervention baseline are marginal (0.66 to 1.67 points) and lack error bars. More importantly, the fixed baseline does not use the rescoring acceptance check. Without this control, it is impossible to tell whether the improvement comes from adaptive routing or simply filtering candidates.

  • No fair compute-matched baseline. The extra generations, rescoring passes, and LLM calls add noticeable inference latency. Fixing the number of perceptor calls does not mean the computational budget is matched. It remains unclear whether equivalent gains could come from simpler test-time scaling, such as majority voting, longer sampling, or extra observations.

  • Overcomplicated writing obscures a modest method. The paper juggles three separate narratives (adequacy supervision, heterogeneous intervention effects, and patch-to-agent transfer) without a clear hierarchy, while key details are scattered across the text and appendix. Strip away the complex framing, and the proposed pipeline is just a small probe, logistic regression, and thresholding applied to existing tools. Because the algorithmic novelty is limited, the work requires a much clearer presentation and tighter experimental backing.

Questions for authors
  • Can authors provide a small, image-grounded audit of original versus accepted descriptions, ideally using agent-selected WSI patches? The goal is to verify whether replacements genuinely add visual evidence rather than just mimicking reference text.

  • Do any fixed-action baselines share the same acceptance gate and rescoring margin as Router+Accept? If not, please report at least one acceptance-matched baseline to isolate the router's effect, along with paired confidence intervals for WSI accuracy that account for slide-level clustering.

  • Does the gain hold against simpler alternatives at an equivalent inference budget, such as taking additional patch observations? A targeted comparison on a single representative setting is sufficient.

2 comments

Response to Reviewer 2 -Part 1

R2.1. Validation of the adequacy signal

Please see R1.1 in the previous response for our justification for using a text only judge and for brief protocols of the human/multimodal audits. We now also assess adequacy using an image-aware judge, two independent pathologist assessments and a reference-free multimodal audit:

  • Image-aware judge: Router+Accept improves macro adequacy from 50.12% to 61.48%, with all six policies retaining their ordering (Table 22). The conclusion therefore does not depend on evaluating descriptions with a text-only judge.
  • Pathologist assessment: two urological pathologists reviewed ten blinded description pairs, selected from 5 wrong-to-right and 5 unsuccessful trajectories of TCGA PRAD slides. Pathologist 1, rated 3 replacements better and seven the same; Pathologist 2 rated two better and eight the same. Neither rated a replacement worse (Tables 25–26). The comments identify clinically relevant repairs like acknowledging insufficient staging evidence in example P01 and correcting the interpretation of Gleason morphology in example P02. Please note that an equal rating ("same") does not mean that both descriptions are correct.
  • Reference-free grounding audit: Multimodal MedGemma-27B assessed 626 candidate–original pairs across 100 correctly answered and 100 incorrectly answered trajectories, without seeing the reference answer. Among accepted replacements, 67.7% improved grounding in correctly answered trajectories, compared with 12.1% in incorrectly answered trajectories (Table 23).

The expert reviews identify meaningful local repairs, the trajectory audit associates improved grounding with successful answers, and the broader image-aware evaluation preserves the policy-level conclusion. The selected human audit is qualitative and it does not estimate improvement rates across all trajectories.

Revised manuscript: B.2–B.3, lines 1219–1262 (pp. 23–24); E.8, Table 22 and analysis, lines 1945–1965 (p. 37); Appendix E.9.1, Table 23 and analysis, lines 1972–1997 (p. 37); Appendix E.9.3, Tables 24–26 and analysis, lines 2432–2612 (pp. 46–49).

R2.2. Are action selection and acceptance controlled separately?

Yes. Fixed-action, task-lookup and confidence-only policies use the same eligibility threshold (τ=0.6\tau=0.6) and acceptance margin (δ=0.08\delta=0.08) as Router+Accept. The Router comparator accepts every valid routed candidate. We apologise that the original baseline names did not make this mapping explicit and now we have clarified it in Section 6 and Appendix A.7.

The targeted WSI comparison (please see R1.3 from previous response and R2.3) also explicitly holds the gate, probe and acceptance rule fixed across Best Fixed+Accept, Task Lookup+Accept and Router+Accept. Their accuracies are 54.33%, 56.00% and 57.50%, respectively (Table 33). Router+Accept therefore has point-estimate advantages of 3.17 and 1.50 percentage points. Its GPU cost is close to task lookup: 5.050 versus 5.023 hours per 100 questions, compared with 6.010 for Best Fixed+Accept.

These comparisons support a contribution from adaptive selection when acceptance is controlled. Table 33 now also reports direct paired, slide-clustered 95% confidence intervals for Router+Accept’s accuracy gains over the acceptance-matched baselines: +3.17 pp [0.47, 5.87] versus Best Fixed+Accept and +1.50 pp [0.24, 2.76] versus Task Lookup+Accept.

Revised manuscript: Section 6, lines 325–329 (p. 7); Appendix A.7, lines 1120–1123 (p. 21); Appendix H.3, protocol and Table 33, lines 3264–3293 (p. 61).

R2.3. Equal inference budgets

We agree that inference cost needs to be considered alongside accuracy and equal call caps do not imply equal computation because the interventions have different costs. We therefore compare measured GPU time, perceptor generations and generated tokens using a single representative setting of PathAgent with Patho-R1 7B on a random subset of 600 SlideBench-TCGA questions from 491 slides(Table 33 and Figure 5).

Router+Accept reaches 57.50% accuracy at 5.050 GPU-hours per 100 questions. The 21-call ordinary policy reaches 53.50% at 5.500 GPU-hours, and Best-of-3 reaches 53.33% at 6.226 GPU-hours. Router+Accept therefore achieves higher accuracy at lower measured cost than both alternatives in this setting. This directly tests the additional-compute explanation, although it is not an exactly compute-matched comparison (It is somewhat tricky to fix the inference budget in the experiments since many of the arms cannot work with a lower budget and raising the budget of every arm to match the costliest arm is wasteful and expensive. We believe that providing metrics of accuracy versus inference cost together can help us arrive at the same conclusions as a compute-matched experiment.)

Revised manuscript: Section 7.4, lines 460–465, and Figure 5, lines 475–479 (p. 9); Appendix H.3, Table 33 and analysis, lines 3264–3301 (pp. 61–62).

Response to Reviewer 2 - Part 2

R2.4. Contribution and presentation

We agree that the intervention pipeline is very simple. But, we wish to emphasize that the intended contribution of this work is in introducing adequacy as a useful signal to trigger interventions and that conditioning even simple interventions based on this signal could improve the performance of WSI agents. Our focus was not on the interventions themselves; that is why in this paper we did not attempt an algorithmically complex intervention strategy and instead relied on combinations of existing, simple and commonly used intervention methods. The small probe, logistic router and fixed acceptance rule were deliberate experimental choices. They let us ask whether local description adequacy supplies useful information about when and how to correct the perceptor, while keeping router capacity limited.

Through this study, due to the logistic-regression based design of the router, we have managed to show that the choice of the ’best intervention’ varies across task types and initial description quality as well, suggesting the need for a novel intervention strategy that would specifically focus on improving the clinical utility and relevance of the descriptions in a targeted way. This is however, an orthogonal research direction that may use this study as a basis- but before that, we believe it is important to establish the usefulness of our signal through a study like this.

We had organised the manuscript such that we first introduce adequacy in Section 3, explore it's empirical properties relating to answer accuracy & intervention effectivness in Section 4, and try to use adequacy for WSI agent routing in Section 5 via a pipeline informed by the observations from Section 4. To make it more explicit that our contribution is the adequacy signal and its utility to WSI agents, we have simplified the abstract, Figure 1 and the wording of our manuscript, to focus less on the Rescue-Route-Rescore pipeline and more on the signal.

Revised manuscript: Abstract, lines 13–34 (p. 1); Figure 1, lines 71–78 (p. 2); Section 8, lines 501–527 (p. 10).

Reviewer_mDo8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the quality of local visual descriptions affects navigation and question answering in WSI agents for histopathology. It proposes a framework that estimates whether a description contains enough information for the task. Based on this estimate and the question type, the framework applies different strategies, including attention biasing, visual contrastive decoding, and review prompting, to improve low-confidence descriptions. A new description is used only if it achieves a score above a fixed threshold. The method is evaluated on 4 pathology LVLMs, 3 inference-time strategies, PathMMU, 2 WSI agents, and 3 WSI-VQA datasets.

Critical strengths
  1. The paper addresses a real system-level problem in WSI agents.
  2. The evaluation covers 4 pathology LVLMs, 3 interventions with different mechanisms, 2 WSI agents, and 3 WSI datasets. Both patch-level and end-to-end results show consistent positive trends.
Critical weaknesses
  1. The central premise that “sufficiency represents the quality of visual evidence” has not been sufficiently established.
  2. The mechanistic explanation that “repairing local evidence improves global decision-making” remain indirect.
  3. It is difficult to determine how much of the improvement comes from a better control strategy and how much comes from additional inference computation.
Questions for authors
  1. Under matched budgets of computation times, how does Router+Accept compare with best-fixed, generating multiple candidates and selecting the best one using the probe?
  2. Could the authors present several success and failure trajectories?
2 comments

Response to Reviewer 1 - Part 1

R1.1. Does adequacy reflect visual evidence?

We chose a text-only judge because PathMMU provides very high quality reference answers and explanations, with its validation and test questions reviewed by seven pathologists [1]. Given this supervision, we wanted to avoid introducing additional perceptual errors from a multimodal judge since there is research to suggest that even large state of the art LVLMs are not perfect at capturing details or answering questions about images that might be considered ”out of the ordinary” or uncommon[2,3]. Histopathology is one such field where the images might contain uncommon/pathological structures which can be difficult for a large multimodal LVLM trained on many different types of images to reason about. There is also evidence to suggest that even the commercial models perform poorly when it comes to histopathology images[4,5]. We have now tested whether the conclusion of our study holds when the judge also sees the image. We additionally also asked the multimodal judge to score the image/description pairs on visual grounding and compared how they differ in pairs judged adequate/inadequate by the text judge.

  • The policy conclusion is unchanged. Multimodal MedGemma-27B evaluates the same frozen policy outputs. Router+Accept improves image-aware macro adequacy from 50.12% to 61.48%, exceeding Best Fixed by 4.46 percentage points. All six policies retain their ordering (Table 22).
  • Text adequacy is associated with visual grounding. The two judges agree on 82.51–86.60% of labels across LVLMs. Descriptions labelled adequate by the text judge receive mean image-grounding scores of 3.75–3.88 out of 4, compared with 2.30–2.55 for inadequate descriptions (Table 21).
  • Text supervision remains useful for predicting reader correctness. In the separate probe-refitting experiment, text-trained probes have higher reader-correctness AUROC than image-label-trained probes across all LVLMs(Table 15).

We also added blinded assessments by two urological pathologists and a reference-free multimodal audit, discussed in R1.2. Together, these checks support the usefulness of text-supervised adequacy while assessing visual grounding separately.

Revised manuscript: Section 7.2, lines 381–385 (p. 8); Appendix B.2, lines 1219–1230 (p. 23); Appendix D.2, lines 1503–1511 (p. 28), and Table 15, lines 1513–1528 (p. 29); Appendix E.8, Tables 21–22 and analysis, lines 1917–1965 (pp. 36–37).

[1] Sun, Y., et al. PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology. ECCV, 2024. Paper.

[2] Vo, A., Nguyen, K.-N., Taesiri, M. R., Dang, V. T., Nguyen, A. T., & Kim, D. (2026). Vision Language Models Are Biased. The Fourteenth International Conference on Learning Representations (ICLR 2026). arXiv:2505.23941.

[3] Parashar, S., Lin, Z., Liu, T., Dong, X., Li, Y., Ramanan, D., Caverlee, J., & Kong, S. (2024). The Neglected Tails in Vision-Language Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12988–12997. arXiv:2401.12425.

[4] Rasheed, H., Kurpath, M. I., Ren, B., Cholakkal, H., Khan, F. S., & Khan, S. (2026). Hard Vision, Easy Vision: What GPT-6 Astra Reveals Across Computer Vision. arXiv preprint, arXiv:2609.35718.

[5] Lu, M. Y., Chen, B., Williamson, D. F. K., et al. (2024). A Multimodal Generative AI Copilot for Human Pathology. Nature, 634(8033), pp. 466–473. DOI: 10.1038/s41586-024-07618-3.

Response to Reviewer 1 - Part 2

R1.2. How does a replacement affect memory and navigation?

We now include examples of full acquisition traces, qualitative blinded assessments by two urological pathologists on 10 prostate cancer slides and a reference-free multimodal audit. For the pathologist audit, the two pathologists independently assessed ten original–replacement description pairs from five wrong-to-right and five unsuccessful trajectories. They saw the patch, question and descriptions in randomised A/B order, with arm identity and outcomes hidden; for navigation examples, the next differing patches were shown in blinded X/Y order to compare question-relevant evidence. The examples illustrate two ways in which a repaired description can affect subsequent decisions:

  • Correcting the interpretation of acquired evidence. In example P02, both the pathologists found that both descriptions captured the relevant morphology, but only the Router+Accept description correctly mapped it to the Gleason pattern corresponding to the benchmark answer. The final answer changed from incorrect B to correct C (Tables 25, 26 and 28).
  • Guiding further evidence acquisition. In example S3, the replacement acknowledged insufficient evidence for tumour extent and was followed by a different retrieved patch. Both the pathologists judged the Router+Accept patch to contain abundant malignant tissue, while the patch retrieved by the baseline arm had no evidence of malignancy. Router+Accept subsequently reached correct answer B; the baseline answered A (Table 29).

Example P01 provides a further example of acknowledging missing staging evidence rather than asserting unsupported adipose tissue invasion, followed by different high-magnification acquisitions (Table 27 and Figure 13).

For the multimodal audit, multimodal MedGemma-27B scored descriptions against their corresponding images on a 0–4 grounding scale, without seeing the reference answer. We evaluated 626 candidate–original pairs across 100 correctly answered and 100 incorrectly answered trajectories, comparing grounding changes separately for accepted and rejected candidates. The multimodal audit is consistent with the mechanisms hinted at by the pathologist audit: 67.7% of accepted replacements in correctly answered trajectories improved grounding, compared with 12.1% in incorrectly answered trajectories (Table 23). These observations connect local repairs with useful interpretation and acquisition decisions, while attributing the overall answer improvement to the complete policy.

Revised manuscript: Section 7.5, lines 484–496 (pp. 9–10); Appendix B.3, lines 1234–1262 (pp. 23–24); Appendix E.9.1, Table 23 and analysis, lines 1972–1997 (p. 37); Appendix E.9.3, Tables 24–26 and analysis, lines 2432–2612 (pp. 46–49); Appendix E.9.4, Figure 13 and Table 27, lines 2696–2765 (pp. 50–52), Table 28, lines 2888–2954 (pp. 54–55), and Table 29, lines 2970–3105 (pp. 56–58).

R1.3. Could additional observations or sampling explain the gain?

We added a direct comparison using PathAgent with Patho-R1 7B on a random subset of 600 SlideBench-TCGA questions from 491 slides. Table 33 and Figure 5 compare Router+Accept with larger ordinary-inference budgets and probe-selected sampling:

  • More observations: ordinary accuracy rises from 49.50% at a seven-call cap to 54.00% at 14 calls and 53.50% at 21 calls. Router+Accept reaches 57.50%, exceeding the 21-call policy by 4.00 percentage points while using 8.2% less GPU time.
  • More sampled descriptions: probe-selected Best-of-3 reaches 53.33%. Router+Accept is 4.17 percentage points higher while using 18.9% less GPU time.

The Router+Accept gain over ordinary inference at seven calls is 8.00 percentage points, with a paired, slide-clustered 95% confidence interval of [4.19, 11.81]. Thus, additional observations or sampling do not reproduce the same gains at the measured operating points in this representative setting.

Revised manuscript: Section 7.4, lines 460–465, and Figure 5, lines 475–479 (p. 9); Appendix H.3, protocol, Table 33 and analysis, lines 3264–3301 (pp. 61–62).

Comments on the paper

1 comment

General response

We thank the reviewers for their constructive suggestions. We have added further experiments, expert audits and clarifications to our revised manuscript to address the main concerns (in the parts below, Rx.y can be found in the response to the xth reviewer):

  • Text-only supervision and visual grounding: We had deliberately chosen a text-only judge to use PathMMU's reference supervision while avoiding additional errors from a judge's interpretation of histopathology images (please see R1.1). Under the added image-aware evaluation, Router+Accept still improves macro adequacy from 50.12% to 61.48%, and all six policies retain their ordering (Table 22). The main conclusion therefore still holds even when vision is included. We have also added separate human pathologist assessments and a reference-free multimodal audit to examine the grounding and clinical relevance of local repairs (please see R1.2, R2.1).
  • Contribution: Our contribution is introducing description adequacy as a signal for deciding when and how to intervene in WSI agents. Our work examined whether identifying and selectively correcting lower quality descriptions can help existing agents without applying the same intervention globally, retraining their perceptors or fitting the controller on WSI outcomes (please see R2.4).
  • Methodological simplicity: To make our study more focused on the adequacy signal and to study it's effects, we deliberately used existing, simple interventions and a small, interpretable router. We believe it is important to first establish the usefulness of the adequacy signal before developing a more complex intervention strategy specifically aimed at improving the clinical utility of descriptions.
  • Action selection and inference cost. We have now made it more explicit that the fixed arms in the manuscript shared both the gate and resoring margin. We have also done a performance versus cost analysis on a single representative setting with the eligibilty gate and acceptance margin deployed in the adequacy conditioned arms (please see R1.3, R2.2, R2.3, Table 33). For each arm, we report its accuracy difference from ordinary inference with B=7B=7, with a 95% confidence interval calculated using paired bootstrap resamples of slides. These comparisons support a contribution from adaptive routing beyond acceptance or additional computation alone.
  • Representation and threshold sensitivity. The layer and pooling ablations support the final-layer representation as a competitive choice across the four LVLMs(please see R3.2). With acceptance retained, performance plateaus around the validation-selected gate threshold which we had deployed in the study(please see R3.3, Tables 14 and 19).
  • Router reliability: On cases where candidate adequacy differs across the three interventions, the router selects an adequate candidate in 61.7–66.1% of cases before acceptance. This tests intervention choice selected by the router separately from the acceptance check. We also report routing, acceptance and replacement rates during WSI runs to show how the controller is used in practice (please see R3.4, Tables 18 and 31).

We thank the reviewers again for their constructive suggestions. We have made a substantial effort to carry out the requested experiments, expert audits and incorporate the additional validation and clarifications into the revised manuscript. We believe these additions address the main concerns and strengthen the evidence for description adequacy as a useful signal for improving WSI agents.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.