- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper investigates when and how a frozen LVLM perceptor in a WSI agent should be corrected at test time. It introduces “description adequacy,” assessed offline by a reference-conditioned text-only LLM and estimated at deployment via a hidden-state probe. Using this signal, the proposed RESCUE–ROUTE–RESCORE framework intervenes only on low-confidence descriptions, selecting among three interventions (attention biasing, VCD, or review prompting) based on question type and confidence level. A revised description is accepted only if its rescored confidence exceeds the original by a fixed margin. On PathMMU, the method improves macro adequacy from 55.7% to 67.2% and reader accuracy from 56.9% to 62.3%. When transferred without further tuning to two additional WSI agents across three benchmarks, it achieves the highest macro accuracy in all six settings, with gains of 0.7–1.7 pp over the strongest fixed intervention.
Critical strengths
-
Framing test-time correction as an instance-level decision is well motivated. The repair/damage analyses convincingly demonstrate that the effect of each intervention is conditional on the input.
-
The experimental protocol is rigorous: image-grouped splits prevent data leakage, the controller transfers frozen across two agents and four LVLMs, and computational costs are explicitly reported.
Critical weaknesses
-
The adequacy judge’s rubric is under-specified: a 0–4 scale is adopted across multiple dimensions, yet no concrete criteria are provided to distinguish adjacent score levels, and the judge prompt itself is not disclosed. This undermines the reliability of the adequacy assessment on which the remainder of the pipeline depends.
-
The probe relies on the last-layer final-token hidden state across all LVLMs without ablating layers or pooling strategies, despite prior probing work [1,2] demonstrating that these choices substantially affect performance.
-
It remains unclear how the threshold τ is determined. The paper does not report results under different threshold values, making it difficult to assess how sensitive the method is to this choice.
-
The intervention router lacks important details. The paper does not report its routing accuracy or its consistency with MedGemma’s scoring results.
[1] LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations. ICLR'2025.
[2] INSIDE: LLMs’ Internal States Retain the Power of Hallucination Detection. ICLR'2024.
Questions for authors
-
For the MedGemma-27B judge, what are the exact prompt and the criteria that distinguish each score level on the 0–4 scale, and how is score reliability evaluated?
-
The adequacy probe uses the last-layer hidden state at the final answer token. Why was this strategy chosen, and was it ablated against other layers or pooling (e.g., mean over response tokens)?
-
What are the results under different τ values, including the case without gating? How often are good descriptions wrongly filtered out, and how often are poor ones kept?
-
How is the reliability of the intervention router evaluated? The paper does not rigorously assess routing accuracy, which has a direct impact on the overall performance of the agent system.
4 comments
Response to Reviewer 3 - Part 2
R3.3. Threshold sensitivity and filtering errors
We selected the gate and acceptance margin on validation data before fixing the deployed policy. We now report the threshold sweep and error breakdown in our manuscript.
Threshold sensitivity. With , validation adequacy is 63.1% at both and , and 63.0% without eligibility gating. The corresponding test values are 67.2%, 67.3% and 67.1% (Table 19). Thus, the deployed lies at the start of the observed validation plateau.
We also noticed that acceptance is important when eligibility gating is removed. Accepting every routed candidate reduces test adequacy from 65.8% at to 61.7% without gating; retaining the 0.08 acceptance margin gives 67.1% without gating.
Which descriptions are harmed or left inadequate? At the deployed setting, 5.18–11.07% of originally adequate descriptions become inadequate, while 56.19–69.90% of originally inadequate descriptions remain inadequate. Equivalently, the controller preserves adequacy for 88.93–94.82% of good originals and repairs 30.10–43.81% of poor originals (Table 20). Each rate conditions on the corresponding original-adequacy group.
Acceptance errors. Across LVLMs, 12.29–36.00% of accepted candidates are inadequate, and 30.05–31.92% of rejected candidates are adequate. These denominators are all accepted and all rejected candidates, respectively. Rejecting an adequate candidate need not harm the output if the original is already adequate. Table 20 therefore also reports harmful replacements and missed repairs. We feel that these are within an acceptable range.
Revised manuscript: Appendix E.6, protocol and analysis, lines 1817–1835 (p. 34), Figure 10 and Table 19, lines 1848–1862 (p. 35); Appendix E.7, definitions, Table 20 and analysis, lines 1867–1910 (pp. 35–36).
R3.4. Reliability of intervention routing
The router predicts whether an intervention will produce an adequate description. We do not have a unique ground-truth intervention labelled as correct for each WSI decision: several interventions may produce useful descriptions, all interventions may fail and the final answer also depends on subsequent acquisition and reasoning, so it is difficult to define a 'routing accuracy'. However, we can assess whether the intervention chosen by the router produces an adequate candidate and we can also point you to the place in the manuscript where we had reported how the router is utilised in WSI runs.
-
Direct evaluation of intervention choice. We use candidates cached from all three interventions on held-out, gate-eligible PathMMU cases. We consider cases containing both adequate and inadequate candidates, and count the router’s choice as successful when its chosen intervention produces an adequate candidate under the MedGemma text-judge label. Either adequate intervention counts when two are adequate. Across the four LVLMs, the router succeeds in 61.7–66.1% of these cases, compared with 47.3–55.7% for the validation-selected best-fixed intervention. This evaluation precedes acceptance and therefore directly assesses the intervention choice.
-
Realized utilization in WSI agents. The manuscript’s policy-utilization table reports observed eligibility, acceptance and replacement rates, together with intervention shares among routed calls, for both WSI pipelines. For example, with PathAgent and Patho-R1 7B, 48.33% of calls are routed for intervention, and 40.42% of routed calls result in acceptance, giving an overall replacement rate of 19.53%. These measurements document the controller’s actual execution in WSI agents alongside the direct evaluation above.
We have added the intervention-choice experiment protocol and results to Appendix E.5.1. We are not sure if this is what you meant, but we hope this answers your concern.
Revised manuscript: Appendix E.5.1, protocol, Equations (26)–(27), analysis and Table 18, lines 1776–1811 (pp. 33–34); Appendix H, Table 31, lines 3241–3250 (p. 61).
Thank you for the additional experiments and clarifications, which address most of my concerns, so I will raise my score to 3 (weak accept). One question remains: the revision uses MedGemma-27B's outputs as the reference for evaluating the router, but it does not report how accurately MedGemma-27B judges whether a description is adequate or inadequate.
Second Response to Reviewer 3
Thank you for raising your score and for this question. As you said, our router reliabilty evaluation measures whether the router selects candidates labelled adequate by the MedGemma-27b judge. But a direct accuracy estimate of the judge might require independent human adequacy labels under the same rubric, since currently those labels come from the judge itself.
We believe that the the usefulness of the judge's supervision is supported by the indirect evidence like the image-aware evaluation, blinded pathologist assessments and downstream answer accuracy measured against benchmark answers. Although these checks support the usefulness of the learned signal, they do not replace a direct validation of the judge.
A large, independent human validation of the MedGemma 27b judge itself would therefore be valuable, and we acknowledge this as a limitation and future direction. However we wish to emphasize that the framework can also use other capable reference-conditioned text judges and is not tied to MedGemma-27B. The judge is a replaceable supervision component, although each alternative would need its own validation.
Response to Reviewer 3 - Part 1
R3.1. Judge prompt rubric
We have added the text-judge and image-aware-judge prompts, the reference-free audit prompt, and the anchors distinguishing every score from 0 to 4 in Appendix A.2.1. Equation (10) in the manuscript specifies the binary adequacy score rule. We use a fixed prompt, greedy decoding and strict schema checks that all required fields are present and within range. In a development set check, re-judging a random subset of 500 samples gave 99.6% score agreement. Due to these reasons, we do not believe that the score reliabilty of the judge has a serious effect on our pipeline or our conclusions. We also assess validity separately through image-aware evaluation and blinded pathologist review, as described in R1.1 and R2.1 in the previous responses.
Revised manuscript: Appendix A.2, Equation (10), lines 814–823 (p. 16); Appendix A.2.1, prompts, score anchors and output schemas, lines 866–1022 (pp. 17–19); Appendix E.8, Table 21 and analysis, lines 1925–1943 (p. 36); Appendix E.9.3, Tables 24–26 and analysis, lines 2432–2612 (pp. 46–49).
R3.2. Layer and pooling choice
We thank the reviewer for this great question. We agree that representation choice should be tested and we had done so during the study on the validation set. We have now included our layer and pooling ablation comparing last-token and mean-pooled features at approximately 25%, 50%, 75% and final decoder depth (Table 14).
It is true that the literature supports testing different representation choices. However, these studies also suggest that the best choice depends on the detection task and the model. While Orgad et al. [1] and INSIDE [2] find advantages for particular token positions and intermediate layers, Ji et al. [3] find that deeper layers perform best for predicting hallucination risk from a query.
There is also precedent for using the final-layer last-token representation in various tasks. In vision-language models, Bi et al. [4] use the final-layer representation of the last prompt token in InternVL and Qwen3-VL to predict reasoning helpfulness, generation length and difficulty. They find that these signals are more linearly recoverable in the reasoning-trained variants, suggesting that the model’s training also matters for what a probe can extract. ReFL [5] separately compares layers and pooling strategies, and finds the final layer and last-token representation to perform best among the tested choices in its corrective-prompting setting.
Our results are consistent with the literature showing useful probing signals in middle-to-late layers, with the best configuration depending on the model and the task. During the study, based on results from the layer ablation in the validation set, we had decided to retain the final-layer last-token representation as a simple and consistent choice across LVLMs, without claiming that it is universally optimal.
Revised manuscript: Section 7.1, lines 368–370 (p. 7); Appendix D.1, protocol, analysis and Table 14, lines 1402–1487 (pp. 26–28).
[1] LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations, ICLR
[2] Chen, C., Liu, K., Chen, Z., Gu, Y., Wu, Y., Tao, M., Fu, Z., and Ye, J. (2024). INSIDE: LLMs’ Internal States Retain the Power of Hallucination Detection. International Conference on Learning Representations (ICLR).
[3] Ji, Z., Chen, D., Ishii, E., Cahyawijaya, S., Bang, Y., Wilie, B., and Fung, P. (2024). LLM Internal States Reveal Hallucination Risk Faced With a Query. Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 88–104.
[4] Bi, J., Song, L., Zhang, D., Liu, P., Sun, G., Wen, L. B., Cai, W., Chen, C., and Xu, C. (2026). Does Reasoning Improve Seeing? Understanding When Vision-Language Models Benefit from Thinking. Proceedings of the 43rd International Conference on Machine Learning, PMLR 306, pp. 8063–8075.
[5] Fan, C., Zhang, J., Zhang, X., Zhang, S., Lv, Z., Tao, J., and Wen, Z. (2026). ReFL: Reflective Feedback Learning for Hallucination Detection of Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers, pp. 19648–19665.