acceptodds
Under review as a conference paper at ICLR 2027

See, Measure, and Reason: Learning Visually Grounded Reasoning in Pathology

Abstract

Pathological assessment relies on recognizing fine-grained visual details in histological images. Vision-language models (VLMs) increasingly support pathology interpretation, yet their ability to perceive these details remains inadequate. This weakness leads to inaccurate cellular observations that can persist even when final answers are correct. In this paper, we propose ASPECT to improve visually grounded reasoning through explicit supervision of cellular appearance and abundance. ASPECT trains intermediate visual tokens through pathology feature reconstruction, cell feature alignment, and count supervision. Three-stage supervised fine-tuning teaches the model to perceive, generate visual tokens, and reason, followed by reinforcement learning that rewards answer correctness and consistency with reported measurements. We also introduce PathoVernier, a benchmark of 759 expert-reviewed questions from five pathology datasets covering four cellular composition tasks. It evaluates both final answers and intermediate measurements to expose errors hidden by answer accuracy. On PathoVernier, ASPECT achieves relative accuracy gains of approximately 19.2% over the strongest baseline, Gemini-3.1-Pro, and 99.3% over its Qwen3-VL-8B backbone, while reducing RAWR, which measures counting errors within correct responses, by 28.1% and 42.7%, respectively. ASPECT also improves over its backbone on three external pathology benchmarks covering classification and question answering beyond cellular composition tasks.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Computational Pathology, Vision-Language Models, Visually Grounded Reasoning, Cellular Composition Analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BniSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ASPECT, a framework for improving visually grounded reasoning in pathology vision-language models. The central motivation is that correct answers to pathology questions do not necessarily indicate accurate visual observations, particularly when the answers depend on cell counts and regional cellular composition. ASPECT introduces intermediate visual tokens supervised through pathology feature reconstruction, cell feature alignment, and count prediction. The framework uses a three-stage supervised fine-tuning curriculum, followed by reinforcement learning that rewards answer correctness and consistency with the model's reported measurements. The authors also introduce PathoVernier, a benchmark containing 759 expert-reviewed questions from five pathology datasets.

Critical strengths
  • PathoVernier provides a structured evaluation of cellular composition reasoning, with reference counts and deterministic decision rules. The distinction between answer correctness and measurement accuracy is particularly useful. The benchmark may serve as a valuable resource for investigating visual grounding in pathology VLMs.

  • The paper addresses a fundamental limitation of pathology VLMs: correct predictions can be accompanied by inaccurate visual evidence. Explicitly evaluating the quantitative observations underlying model predictions is a meaningful direction for improving the reliability of medical vision-language systems.

Critical weaknesses
  • The cell-count supervision predicts image-wide cell abundances, while many benchmark questions require region-specific counting and spatial comparisons. Furthermore, the cell feature alignment objective aggregates instance features through mean pooling, which does not explicitly preserve spatial positions or individual instance identities. Although the model learns regional observations through response-text supervision, it remains unclear how much of its regional counting capability is attributable to the proposed visual-token representations rather than the generated reasoning supervision. More direct analysis of the spatial information encoded in these tokens would strengthen the methodological contribution.

  • The architectural contribution is not fully separated from the benefits of task-specific supervision. The Direct Reason baseline retains the proposed visual supervision objectives. A conventional Qwen3-VL-8B fine-tuning baseline using the same quantitative question-answer data, but without the proposed visual tokens and auxiliary objectives, would provide a clearer assessment of the contribution of the architecture itself.

Questions for authors
  • How do the proposed cell tokens encode spatial information when the cell feature alignment objective uses mean-pooled instance representations? Can the authors provide an analysis of regional counting accuracy independent of the final language-generated answers?

  • How does ASPECT compare with conventional Qwen3-VL-8B supervised fine-tuning using the same quantitative QA examples, but without the proposed visual tokens and auxiliary visual losses?

2 comments
AuthorsOpenReview ↗

Response to Reviewer BniS - Part 1/2

Thank you for your constructive review and recognition of the benchmark's evaluation of quantitative observations. We address the visual-token analysis and conventional fine-tuning comparison in two comments. Revisions are marked in purple in the revised PDF.

W1/Q1. Spatial information in visual tokens and regional counting accuracy.

A1. Mean pooling constructs the auxiliary cell-feature targets; visual-token states are computed by the decoder from the image and preceding context. Regional observations are then generated using these states together with spatial image tokens. Cell alignment and count supervision train appearance and abundance; regional observation supervision connects these representations to question-specific measurements. We test this connection through direct count-head readout, regional counting under matched text supervision, and inference-time token interventions.

Direct decoding without language generation. Using the final ASPECT model, we supply the image, question, and fixed visual-token prefix, extract the six final-layer <cell> states, add the trained role embeddings, concatenate them, and apply the existing SFT-trained count head. This directly produces counts without generating observations or answers and requires no additional training. We evaluate 553 unique patches after deduplicating the 759 benchmark questions by image.

Image-wide countCount Acc↑MAE↓Spearman ρ\rho↑
Epithelial0.4216.6670.971
Stromal0.2515.0950.886
Inflammatory0.4123.8670.961
Three-category aggregate0.3625.210—
Total nuclei, evaluated separately0.4059.8420.971

The strong correlations with annotations (ρ=0.886\rho=0.886–0.9710.971) show that cellular abundance is directly accessible from the learned <cell> states.

Regional counting with the same text supervision. We separately score all 1,892 required regional counts in generated observations from 566 spatial questions: 760 region-selection fields, 760 multi-step composition fields, and 372 region-comparison fields. The denominator includes every required field, irrespective of final-answer correctness. A count is correct when ∣c^−q∣≤max⁡(1,0.1q)|\hat c-q|\leq\max(1,0.1q); missing or unparseable counts are incorrect.

ModelRegional count accuracy↑
ASPECT0.500
ASPECT-SFT0.465
SFT without cell feature alignment0.249
SFT without count supervision0.353
Qwen3-VL-8B SFT0.411
Qwen3-VL-8B SFT+RL0.042

These regional scores are newly computed. The loss-removal variants use the paper's Table 2 settings, retaining response-text supervision, visual tokens, and curriculum. Removing cell alignment or count supervision lowers regional accuracy from 0.465 to 0.249 or 0.353, reductions of 21.6 and 11.2 percentage points. This shows a contribution from visual supervision to regional measurement under the shared text-training objective. Regional accuracy pools fields, while the paper's Count Acc averages per-question scores over all questions.

Use of visual tokens during inference. We zero or replace the KV entries of the selected token group across all 36 decoder layers, retain the original image/question and prefix token IDs, recompute subsequent prefix states, and regenerate observations and answers.

QuestionsInterventionAcc↑Regional count accuracy↑Count fields changed
759Original0.750.500—
759Zero <uni> KV0.710.42034.36%
759Zero <cell> KV0.690.39645.51%
702Original, matched subset0.740.498—
702Swap <uni> KV0.680.39726.46%
702Swap <cell> KV0.630.36849.30%

Replacement donors are different patches from the same dataset with at least a three-fold difference in reference total nuclei, encoded with the recipient question. We use seed 0 and a common donor for questions from the same image. Zeroing covers all 759 questions and 1,892 regional fields; replacement covers the 702 eligible questions and 1,708 fields. Each intervention uses its matched original-model control.

Both token groups affect reported regional counts and final answers. For example, replacing <cell> KV lowers regional accuracy from 0.498 to 0.368 and answer accuracy from 0.74 to 0.63. Together, direct readout, training ablations, and inference interventions connect quantity information in the visual tokens to improved regional measurements and their use in generation.

Manuscript changes. We clarified the count-head role in Sec. 3.2 (p. 5, lines 235–239), added spatial analyses in Sec. 4 (p. 10, lines 508–513), and included full protocols and results in Appendices E.5–E.7, Tables E.13–E.15 (pp. 32–34, lines 1714–1835).

AuthorsOpenReview ↗

Response to Reviewer BniS - Part 2/2

W2/Q2. Comparison with conventional Qwen3-VL-8B fine-tuning on the same quantitative QA data.

A2. ASPECT improves both answer and count accuracy over conventional fine-tuning on the same QA data. To test this directly, we train Qwen3-VL-8B with matched data, update budget, LoRA settings in the vision encoder/projector/LLM, and RL configuration. The conventional baseline removes visual tokens, auxiliary visual losses, and the three-stage curriculum.

MethodAcc↑CA↑RAWR↓Count Acc↑
Qwen3-VL-8B SFT0.630.710.620.37
ASPECT-SFT0.700.800.530.44
Qwen3-VL-8B SFT+RL0.570.670.970.03
ASPECT0.750.810.490.47

ASPECT-SFT improves Acc from 0.63 to 0.70 and Count Acc from 0.37 to 0.44. This shows that the proposed training design improves quantitative observations as well as final answers. RL further raises these metrics to 0.75 and 0.47. Under conventional training, the corresponding scores fall to 0.57 and 0.03, and RAWR rises to 0.97.

The paper's Table 2 provides a complementary controlled comparison: removing all visual losses while retaining the visual tokens, response format, and curriculum reduces Acc/Count Acc from 0.70/0.44 to 0.64/0.32. The conventional baseline evaluates the full ASPECT design against standard task fine-tuning, and the loss-only comparison identifies the contribution of visual supervision within that design. Together with the regional analyses in A1, these results link visual supervision to improved measurement at both image and region level.

Manuscript changes. We added the matched-training comparison in Sec. 4 (p. 8, lines 399–405) and Fig. 4(a); the loss-only ablations are in Table 2 (p. 9).

Reviewer_2Cw4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the “right answer, wrong reason” failure mode in pathology vision-language models (VLMs), where models may produce correct final answers despite inaccurate underlying cellular observations. The authors introduce ASPECT, a visually grounded reasoning framework built on Qwen3-VL-8B that explicitly supervises intermediate visual representations using three complementary signals: reconstruction of pathology features from a frozen UNI encoder, alignment with CellViT++ cell-level features, and log-space cell-count prediction. Training follows a three-stage supervised fine-tuning curriculum—Perceive, Generate, and Reason—followed by GRPO-based reinforcement learning that jointly rewards answer correctness and consistency between reported quantitative observations and final conclusions. The paper also introduces PathoVernier, an expert-reviewed benchmark containing 759 questions from 553 image patches across five nucleus-annotated pathology datasets. It evaluates four cellular-composition reasoning tasks and introduces metrics including Coherent Accuracy (CA) and Right Answer, Wrong Reason (RAWR) to assess whether correct answers are supported by accurate quantitative observations. On PathoVernier, ASPECT achieves 0.75 accuracy, 0.81 CA, and 0.49 RAWR, outperforming both its Qwen3-VL-8B backbone and strong open- and closed-source baselines. The method also improves over its backbone on PathCLS, PathVQA, and Quilt-VQA. Ablation studies examine the contributions of visual supervision, curriculum design, and reinforcement-learning reward variants, while an image-mismatch experiment provides additional evidence that performance depends on visual input. Overall, the work contributes both a training framework for improving observation–answer consistency and an evaluation paradigm for detecting perceptual errors that conventional answer-level accuracy can conceal.

Critical strengths
  1. Evaluation of visual evidence rather than final-answer accuracy alone. The strongest contribution is the PathoVernier evaluation framework and its explicit treatment of the “right answer, wrong reason” failure mode. Each benchmark item is associated with annotation-derived reference counts and a deterministic decision rule, enabling the evaluation to distinguish final-answer correctness from the fidelity of the quantitative evidence supporting that answer. In particular, Coherent Accuracy (CA) and RAWR provide complementary measures of whether reported observations support the final conclusion and whether correct answers are nevertheless based on inaccurate cellular measurements. This is consequential because conventional pathology VLM benchmarks can reward correct outputs even when the underlying cellular perception is unreliable. The reported ranking reversal—where pathology-tuned models that perform well on PathCLS perform comparatively poorly on PathoVernier—provides compelling evidence that conventional answer-level evaluation may obscure important perceptual failures.

  2. Well-motivated integration of visual supervision and consistency-aware optimization. ASPECT provides a technically coherent approach to reducing the gap between visual observation and downstream reasoning by supervising intermediate representations through pathology feature reconstruction, cell-feature alignment, and quantitative count prediction, followed by a staged Perceive→Generate→Reason curriculum and consistency-aware reinforcement learning. The ablation studies provide useful evidence that these components improve different aspects of performance: visual supervision improves count fidelity, the staged curriculum improves task performance relative to direct reasoning training, and the consistency-aware reward provides a better balance between answer correctness and observation–answer coherence than answer-only optimization. The image-mismatch experiment further supports the conclusion that the model relies meaningfully on visual information rather than exclusively exploiting textual priors. Together, these experiments make the methodological contribution substantially more convincing than improvements in answer accuracy alone.

Critical weaknesses
  1. The primary comparison does not sufficiently separate the contribution of ASPECT's proposed visual-grounding mechanisms from in-distribution task specialization. The most important limitation is that ASPECT is trained using programmatically generated questions derived from the same five source datasets, harmonized cell taxonomy, task families, and response structure used to construct PathoVernier, although the evaluation images themselves are disjoint. In contrast, the principal competing models are evaluated zero-shot. This makes the headline comparison difficult to interpret as evidence that the proposed latent visual supervision itself is responsible for the performance advantage. Notably, the ablation without visual supervision already achieves approximately 0.64 accuracy, comparable to or slightly above Gemini-3.1-Pro (0.63), whereas the complete method reaches 0.70 before RL and 0.75 after RL. Thus, a substantial fraction of the reported advantage appears attributable to task-specific supervised fine-tuning rather than uniquely to the proposed visual-token supervision.

  2. Generalization and absolute measurement fidelity remain insufficiently established. Training and PathoVernier evaluation draw from the same five nucleus-annotated datasets and use the same harmonized cell-category mapping. Although the images are disjoint, annotation conventions, staining/domain characteristics, and cellular distributions may remain correlated. Consequently, the strong PathoVernier results may partly reflect specialization to these datasets and task definitions rather than general improvement in quantitative pathology perception. The external-benchmark improvements are encouraging but do not directly test cellular counting and compositional reasoning under a genuinely unseen annotation/domain distribution; moreover, interpretation of some external gains is complicated by related data being included during SFT.

Questions for authors
  1. Matched task-training baseline: What performance does Qwen3-VL-8B achieve when trained using exactly the same SFT data and RL procedure as ASPECT, but without the proposed intermediate latent visual-token supervision? This comparison is important for determining how much of the improvement arises from the proposed visual-grounding mechanism rather than task-specific fine-tuning.

  2. Detector/rule-based baseline: What performance is obtained on PathoVernier when CellViT++ predictions are converted directly into cell counts and the benchmark's deterministic decision rules are applied (reporting at least Acc, CA, and Count Accuracy)? Please also clarify whether the CellViT++ training data contain images or closely related samples from the datasets used to construct PathoVernier.

  3. Causal use of intermediate visual tokens: Does performance change when the generated <uni> and <cell> hidden states are zeroed, perturbed, or replaced with states generated from another image while keeping the original image input unchanged? The current image-mismatch experiment establishes image dependence but does not by itself demonstrate that the proposed intermediate states causally contribute to inference.

  4. Generalization and possible training-data effects: How much of the reported improvement on PathCLS and PathVQA remains when SFT examples derived from NCT-CRC and PathVQA, respectively, are excluded? More broadly, has the quantitative counting/composition performance been evaluated on a dataset that is entirely outside the five source datasets used for both training-data construction and PathoVernier?

  5. Reward behavior: During GRPO optimization, did the authors observe evidence of reward exploitation in which the model generated internally consistent counts primarily to support a predicted answer rather than improving correspondence with the actual image? If so, how was this detected or mitigated?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2Cw4 - Part 1/3

Thank you for your constructive review. We provide matched training controls, a detector/rule baseline, token interventions, and additional generalization analyses below. Revisions are marked in purple in the revised PDF.

W1/Q1. How much of the improvement comes from task-specific training versus visual supervision?

A1. Task-specific SFT improves Qwen3-VL-8B's Acc from 0.37 to 0.63, and ASPECT provides further gains in both answers and counts. To measure this additional contribution, we train a conventional Qwen3-VL-8B baseline using the same QA data, update budget, LoRA settings in the vision encoder/projector/LLM, and RL configuration. It removes the visual tokens, auxiliary visual losses, and three-stage curriculum.

MethodAcc↑CA↑RAWR↓Count Acc↑
Qwen3-VL-8B SFT0.630.710.620.37
ASPECT-SFT0.700.800.530.44
Qwen3-VL-8B SFT+RL0.570.670.970.03
ASPECT0.750.810.490.47

ASPECT-SFT improves Acc and Count Acc by 7 percentage points each over conventional SFT. The paper's Table 2 complements this comparison with a loss-only ablation: retaining the visual tokens, response format, and curriculum while removing all visual losses lowers Acc/Count Acc from 0.70/0.44 to 0.64/0.32. This isolates the benefit of visual supervision within the same training structure.

The RL comparison also reveals a difference in count fidelity. Conventional SFT+RL raises RAWR from 0.62 to 0.97 and reduces Count Acc from 0.37 to 0.03, while ASPECT improves both metrics. The conventional model reports counts of at least 300 in 361/759 responses, although no benchmark patch contains more than 296 nuclei. These results support visually supervised training as a basis for maintaining accurate measurements during RL; A5 describes the monitoring and reward evidence.

Manuscript changes. We added the matched-training comparison in Sec. 4 (p. 8, lines 399–405) and Fig. 4(a); the loss-only ablations are in Table 2 (p. 9).

Q2. Detector/rule baseline and overlap with CellViT++ training data.

A2. CellViT++ (SAM-H) followed by the benchmark rules achieves overall Acc of 0.757, CA of 0.862, and Count Acc of 0.447. We count its predicted nuclei within each queried region and apply the same deterministic task rules. We use the Lizard-trained classifier for lymphocyte questions and the native PanNuke classifier otherwise.

SubsetMethodAcc↑CA↑RAWR↓Count Acc↑
All (759)CellViT++0.760.860.560.45
ASPECT0.750.810.490.47
PanNuke + CoNSeP (183)CellViT++0.870.920.330.65
ASPECT0.650.740.420.51
Lizard + PUMA + NuCLS (576)CellViT++0.720.840.620.38
ASPECT0.780.830.510.46

ASPECT achieves similar overall answer accuracy, with higher Count Acc and lower RAWR. On the pooled 576 questions from Lizard, PUMA, and NuCLS, ASPECT improves Acc by 5.5 percentage points and Count Acc by 7.6 points. Its RAWR is also lower, indicating more accurate counts among eligible correct responses.

Regarding the original training data, the CellViT++ segmentation model was pretrained on PanNuke, and the lymphocyte classifier used here was trained on Lizard [1]. Its performance may therefore partly benefit from exposure to these benchmark source datasets during training. The two subsets also make performance across annotation granularity visible: PanNuke and CoNSeP use a coarse inflammatory category, while the other three sources distinguish lymphocytes in the benchmark mapping.

Manuscript changes. We added the detector/rule protocol, training-source discussion, and results in Appendix E.8, Table E.16 (p. 35, lines 1836–1870).

[1] Hörst F, et al. CellViT++: Energy-efficient and adaptive cell segmentation and classification using foundation models. Computer Methods and Programs in Biomedicine. 2026;277:109206.

AuthorsOpenReview ↗

Response to Reviewer 2Cw4 - Part 2/3

Q3. Does inference causally use the intermediate visual tokens?

A3. Yes. Intervening on either <uni> or <cell> KV entries reduces answer accuracy and regional counting accuracy while retaining the original image and question.

We intervene on all eight <uni> or six <cell> token positions across all 36 decoder layers and all KV heads. We either set their Key and Value cache entries to zero or replace them with the corresponding entries from another image. After the selected block is modified, subsequent prefix states are recomputed and the complete <observe> text and final answer are regenerated. The image, question, and original prefix token IDs are retained.

QuestionsInterventionAcc↑Regional count accuracy↑Count fields changed
759Original0.750.500—
759Zero <uni> KV0.710.42034.36%
759Zero <cell> KV0.690.39645.51%
702Original, matched subset0.740.498—
702Swap <uni> KV0.680.39726.46%
702Swap <cell> KV0.630.36849.30%

For replacement, the donor is a different patch from the same dataset whose annotated total nucleus count differs by at least a factor of three. It is encoded with the recipient question, with corresponding visual-token positions. We use seed 0 and the same donor for all questions associated with one recipient image. There are 702 questions with eligible donors; their original-model results are shown as the matched control.

The regional accuracy and change-rate denominators are 1,892 fields for zeroing and 1,708 for replacement. Every required regional count is evaluated using ∣c^−q∣≤max⁡(1,0.1q)|\hat c-q|\leq\max(1,0.1q), regardless of final-answer correctness. Missing or unparseable counts are incorrect. This field-level accuracy complements the paper's Count Acc, which averages count scores over questions.

Zeroing <cell> entries reduces regional accuracy by 10.4 percentage points and changes 45.51% of the reported fields; replacement reduces accuracy by 13.0 points and changes 49.30%. The corresponding regional-accuracy reductions for <uni> are 8.0 and 10.1 points. Answer accuracy falls in all four interventions. These results show that both token groups influence the measurements and conclusions generated from the original image.

Manuscript changes. We added the main findings in Sec. 4 (p. 10, lines 508–513) and the full intervention protocol and results in Appendix E.6, Table E.14 (pp. 33–34, lines 1764–1803).

AuthorsOpenReview ↗

Response to Reviewer 2Cw4 - Part 3/3

W2/Q4. Effects of related SFT data and quantitative generalization to new sources.

A4. We mix NCT-CRC and PathVQA examples into quantitative training to maintain broad pathology classification and question-answering capabilities. After jointly removing these examples and retraining ASPECT, PathoVernier answer accuracy remains close to the full model. All three external benchmark scores decrease, with two remaining above the Qwen3-VL-8B backbone.

SettingPathoVernier AccPathoVernier Count AccPathCLS AccPathVQA AccQuilt-VQA Acc
Qwen3-VL-8B0.370.070.370.670.58
ASPECT0.750.470.540.740.67
ASPECT without NCT-CRC/PathVQA0.740.430.490.650.60

PathoVernier Acc is 0.74 after removal, compared with 0.75 for the full model. The external results support the role of the mixture in retaining broader capabilities: Quilt-VQA also decreases from 0.67 to 0.60 even though its examples are absent from SFT. PathoVernier images are disjoint from all training images.

To evaluate quantitative transfer beyond the five original source datasets, we construct 278 MoNuSAC [2] questions and 400 BRCA-M2C [3] questions using the PathoVernier generation procedure. The table reports all four metrics; CA n and RAWR n give their eligible question counts. Acc and Count Acc cover all questions in each dataset.

DatasetModelAcc↑CA↑CA nRAWR↓RAWR nCount Acc↑
MoNuSAC (278)Qwen3-VL-8B0.5110.586580.625360.070
Qwen3-VL-8B SFT+RL0.6040.6172350.9411680.043
Gemini-3.1-Pro0.7450.7442770.4432070.483
ASPECT0.6690.6032620.5071810.393
BRCA-M2C (400)Qwen3-VL-8B0.3900.4901040.783380.073
Qwen3-VL-8B SFT+RL0.5150.6083570.9761960.023
Gemini-3.1-Pro0.6050.6693930.5812420.364
ASPECT0.6280.6853810.4962460.395

Compared with matched Qwen3-VL-8B SFT+RL, ASPECT improves Acc by 6.5 percentage points on MoNuSAC and 11.3 points on BRCA-M2C. Count Acc increases by 35.0 and 37.2 points, respectively, accompanied by lower RAWR on both datasets. These gains show that improved measurement extends to sources outside those used for quantitative training. On BRCA-M2C, ASPECT also exceeds Gemini-3.1-Pro across all four metrics.

Manuscript changes. We added the data-mixture study in Appendix E.9, Table E.17 (pp. 35–36, lines 1872–1894), and OOD results in Sec. 4 and Fig. 4(b) (p. 8, lines 425–431; p. 9, lines 479–480), with complete results in Appendix E.10, Table E.18 (p. 36, lines 1896–1920).

[2] Verma R, et al. MoNuSAC2020: A Multi-Organ Nuclei Segmentation and Classification Challenge. IEEE Transactions on Medical Imaging. 2021;40(12):3413–3423.

[3] Abousamra S, et al. Multi-Class Cell Detection Using Spatial Context Representation. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021:4005–4014.

Q5. Evidence of reward exploitation and how it is monitored or mitigated.

A5. ASPECT's RL improves correspondence with the reference measurements: Count Acc rises from 0.44 to 0.47, while self-consistent responses whose counts imply a wrong conclusion remain stable (143→141 of 759). We monitor annotation-based count accuracy alongside the reward; validation count scores stay at 0.40–0.45 during ASPECT's RL, compared with a decline from 0.41 to 0.09 under conventional training. That conventional model reports counts of at least 300 in 361/759 responses, exceeding the maximum reference total of 296 nuclei. The reward ablation also clarifies the contribution of answer supervision: consistency-only RL achieves 0.98 consistency but Count Acc of 0.41, compared with 0.47 for ASPECT (paper Table 3). The answer reward checks the generated conclusion against the reference, while visually supervised SFT establishes image-grounded quantity representations before RL. The count metrics then track whether optimization preserves and improves the underlying measurements.

Manuscript reference. The matched RL comparison is in Fig. 4(a) and Sec. 4 (p. 8, lines 399–405); reward ablations are in Table 3 (p. 9) and Sec. 4 (p. 10, lines 515–520). The additional monitoring results are reported in this response.

Dear Reviewer 2Cw4,

Thank you for your thoughtful feedback. Your questions prompted us to compare ASPECT with conventional fine-tuning on the same QA data, test how visual tokens affect its predictions, and evaluate counting accuracy on two additional datasets. We have incorporated these analyses into the revised manuscript and posted a point-by-point response. We would appreciate your thoughts on the revisions when convenient and would be happy to clarify any remaining concerns.

Best regards,
The Authors

Reviewer_mb1LOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper develops a pathology multimodal large language model for cellular composition reasoning for patch level prediction. They also introduce PathoVernie, a benchmark that includes 4 tasks: region selection, region comparison, cell-type comparison and multi-step composition analysis and evaluate not only the final answer but the numerical evidence. ASPECT uses <uni> token, aligned with UNI feature reconstruction and <cell> tokens, aligned with CellViT++ pooled features and cell count prediction. The model trained using three-stage supervised training procedure and GRPO for answer correctness and count-answer agreement. The paper improve on PathoVernier and three external pathology benchmark.

Critical strengths

The model introduces the technique gaps where the current sota model fails to provide both correct numerical correct reasoning and final answer for cellular composition task. The model provides an interesting solution to guarantee cell count correctness and evident-linked to the final output. The experiment support the result

Critical weaknesses

Although the introduction motivates the cellular composition task in clinical use, the usage of the benchmark question in the clinical setting is questionable. Can we provide evidence to show the need to compare the “ratio between epithelial-like nucleus concentration present within the upper-right quadrant to that within the lower-left quadrant?” for example in pathologist workflow. Also the ratio threshold used to evaluate the counts reasoning does not have clinical evidence support.

Motivation for reconstruction through generated token: The LLM already gets all the vision token as the input, so the UNI reconstruction seems to be redundant since the generation process sees all the visual information without the UNI tokens. Although the result support the benefit of having UNI reconstruction, I am not convinced that the UNI construction token is necessary, given that the convention 2-stage LLaVA style training pipeline, training the projector then performs joint projector LLM tuning proved to work in pathology as in PathChat, Quilt-LLaVA, etc. An experiment doing it “traditionally” like training pathology language adapter may give more insight.

The link between the uni and the cell token to the final output is unclear. For example, The Cell token only learns via the alignment with the pooled feature from cellvit++ and cell count objective, so it can not contain the spatial information. How the <cell> token handles the cell spatial arrangement related task remains unclear. Technically the <cell> token is learnt via CellViT++ pool feature alignment and cell count prediction objective. It can not have the cell spatial information needed for several tasks in the benchmark. A further analysis on how we get the improvement on spatial related tasks would help.

Presentation issues: Please make it explicitly near equation 6 that C is from text generated process not from the count heads. Furthermore, since C is from the text generation, it is still hidden on whether the LLM relies on cell token to calculate C or not. Furthermore, the models already include a count prediction head during the training phase, why don't we use this result instead of autoregressive text?

Many components are poorly defined, in equation 6 what is reference counts and reference answer, can I understand it is the growth truth? Is \hat{a} is LLM generated, and what is the answer projection? An answer example and assign the component within the example may help

In equation 4, for the definition of nk and ng is unclear “Let D index categories with individually resolved counts nk , and let G contain groups G ⊆ {1, . . . , 5} with annotated combined counts nG” . Please explicitly identify nk and ng as the growth truth for individual category count and group category count. Also please state clearly that the sets D and G depend on the available annotation for each sample. This is important to understand the motivation of supervising both fine and coarse counting in the same loss. A concrete data sample right next to the equation may help

In equation 4, the need to compute in log space is not justify

In the fig2, from the arrow, it is not clear that the input for uni encoder and CellViT++ encoder is the image only or both image and text

Fig3 a is actually really hard to track the dataset statistic

Reference: Seyfioglu, Mehmet Saygin, et al. "Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

Lu, Ming Y., et al. "A multimodal generative AI copilot for human pathology." Nature 634.8033 (2024): 466-473.

Questions for authors

See above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer mb1L - Part 1/3

Thank you for your careful review and constructive suggestions. We address the clinical motivation, visual-token contribution, and presentation questions below. Revisions are marked in purple in the revised PDF.

W1. Evidence for the clinical relevance of the tasks and the choice of thresholds.

A1. The tasks assess measurement operations used in pathology workflows: identifying a cell population within a tissue region, comparing its abundance across regions, and estimating its proportion. TIL assessment requires quantifying immune infiltration within a defined tissue compartment [1]. Tumor-content assessment for molecular testing estimates the tumor-cell proportion in a selected area and compares it with the assay's requirements [2]. PathoVernier makes these operations verifiable through nucleus annotations and explicit regional boundaries. Because the compared regions have equal areas, their count ratios also give their density ratios.

We chose the numerical thresholds to make benchmark decisions reproducible. The ratio boundaries, 0.5 and 2.0, apply a symmetric two-fold comparison. Composition bands are derived from training-split tertiles for each cell type and spatial partition, then frozen before evaluation; each question states its thresholds. Appendix C.2 gives the rules and boundary-exclusion procedure. Tertile grouping also has precedent in clinical TIL research [3].

Manuscript changes. We clarified the clinical motivation and threshold construction in Sec. 3.5 (p. 6, lines 319–322; p. 7, lines 341–343).

W2. Why reconstruct UNI features when the model already receives image tokens?

A2. UNI reconstruction trains the LLM's intermediate states to retain pathology features for subsequent observation generation. The image input supplies visual evidence, and the reconstruction objective supervises how that evidence is represented inside the language model. The paper's Table 2 directly tests this objective: removing only pathology feature reconstruction lowers Count Acc from 0.44 to 0.27 while retaining the visual tokens, response format, and curriculum.

To test whether conventional fine-tuning can provide the same gains, we also train Qwen3-VL-8B on the same QA data, matching the update budget, LoRA settings in the vision encoder/projector/LLM, and RL configuration. This baseline removes the visual tokens, auxiliary visual losses, and three-stage curriculum.

MethodAcc↑CA↑RAWR↓Count Acc↑
Qwen3-VL-8B SFT0.630.710.620.37
ASPECT-SFT0.700.800.530.44
Qwen3-VL-8B SFT+RL0.570.670.970.03
ASPECT0.750.810.490.47

ASPECT-SFT improves both Acc and Count Acc by 7 percentage points over conventional SFT. RL further improves ASPECT's Count Acc from 0.44 to 0.47, while conventional SFT+RL falls from 0.37 to 0.03. In the latter model, 361 of 759 responses report counts of at least 300, exceeding the largest reference total of 296 nuclei per patch. The matched baseline shows the benefit of the full training design, and the reconstruction-only ablation identifies the contribution of UNI supervision to count fidelity.

Manuscript changes. We added the matched-training comparison in Sec. 4 (p. 8, lines 399–405) and Fig. 4(a); the loss-only ablations are in Table 2 (p. 9).

[1] Salgado R, et al. The evaluation of tumor-infiltrating lymphocytes (TILs) in breast cancer: recommendations by an International TILs Working Group 2014. Annals of Oncology. 2015;26(2):259–271.

[2] Lindeman NI, et al. Updated Molecular Testing Guideline for the Selection of Lung Cancer Patients for Treatment With Targeted Tyrosine Kinase Inhibitors: Guideline From the College of American Pathologists, the International Association for the Study of Lung Cancer, and the Association for Molecular Pathology. Archives of Pathology & Laboratory Medicine. 2018;142(3):321–346.

[3] Schuiveling M, et al. Artificial Intelligence–Detected Tumor-Infiltrating Lymphocytes and Outcomes in Anti–PD-1–Based Treated Melanoma. JAMA Oncology. 2025;11(12):1470–1478.

AuthorsOpenReview ↗

Response to Reviewer mb1L - Part 2/3

W3. How do visual tokens contribute to spatial tasks and the final output?

A3. Both visual-token groups contribute to the reported regional counts and final answers. Mean pooling constructs the cell-feature alignment targets; the decoder computes token states using the image and preceding context, and response-text supervision trains these states jointly with regional observation generation. Subsequent decoding can attend to both the visual tokens and spatial image tokens. We examine this contribution at training and inference time.

Regional counting under controlled training ablations. We score all 1,892 required regional counts from 566 spatial questions: 760 fields from region selection, 760 from multi-step composition, and 372 from region comparison. These counts are parsed from generated observations and evaluated independently of final-answer correctness. Each field is correct when ∣c^−q∣≤max⁡(1,0.1q)|\hat c-q|\leq\max(1,0.1q). Missing fields, unparseable counts, and missing responses contribute errors to the fixed denominator.

ModelRegional count accuracy↑
ASPECT0.500
ASPECT-SFT0.465
SFT without cell feature alignment0.249
SFT without count supervision0.353
Qwen3-VL-8B SFT0.411
Qwen3-VL-8B SFT+RL0.042

These are newly computed regional scores. The loss-removal models follow the paper's Table 2 protocol and retain response-text supervision, visual tokens, and the curriculum. Removing cell alignment or count supervision reduces regional accuracy from 0.465 to 0.249 or 0.353. Visual supervision therefore improves regional measurement under the same text supervision. Appendix E.1 also reports the full model's task-level results. Here regional accuracy averages fields; the paper's Count Acc averages per-question count scores over all questions.

Intervening during inference. We zero or replace the KV cache entries for all eight <uni> or six <cell> positions across all 36 decoder layers and all KV heads. The original image, question, and prefix token IDs are retained. After modifying the selected block, we recompute subsequent prefix states and regenerate the complete observations and final answer.

QuestionsInterventionAcc↑Regional count accuracy↑Count fields changed
759Original0.750.500—
759Zero <uni> KV0.710.42034.36%
759Zero <cell> KV0.690.39645.51%
702Original, matched subset0.740.498—
702Swap <uni> KV0.680.39726.46%
702Swap <cell> KV0.630.36849.30%

Replacement donors come from the same dataset but a different patch, with at least a three-fold difference in reference total nuclei. Donors are encoded with the recipient question, using corresponding token positions; seed 0 is used, and questions from the same image share a donor. All 759 questions enter the zeroing experiment; 702 have eligible donors. Regional accuracy and field-change rates use 1,892 and 1,708 fields, respectively, against matched controls.

Zeroing <cell> entries lowers regional accuracy by 10.4 percentage points; replacement lowers it by 13.0 points. The corresponding reductions for <uni> are 8.0 and 10.1 points. Answer accuracy also falls in every setting. These changes show that generation uses both visual-token groups to produce regional measurements and answers.

Manuscript changes. We added the spatial analysis in Sec. 4 (p. 10, lines 508–513), with protocols and results in Appendices E.5–E.6, Tables E.13–E.14 (pp. 32–34, lines 1714–1803).

W4. Where does CC come from, and why use text generation instead of the count head?

A4. CC is parsed from the generated <observe> text. The SFT count head predicts five image-wide category totals, whereas the benchmark asks for question-specific regional counts, comparisons, and proportions. The decoder produces these measurements by combining the image, question, and intermediate states.

The trained head also provides a direct readout of quantity information in <cell> states. On the 553 unique benchmark patches, the final ASPECT model's head-decoded counts have Spearman correlations of 0.971, 0.886, and 0.961 for epithelial, stromal, and inflammatory nuclei, and 0.971 for total nuclei. This uses the existing SFT-trained head without additional training or observation-text generation. The interventions in A3 then establish that the visual-token representations contribute to the region-specific outputs produced by the decoder.

Manuscript changes. We clarified the count-head role in Sec. 3.2 (p. 5, lines 235–239) and the source of CC in Sec. 3.4 (p. 6, lines 292–296); direct readouts are in Appendix E.7, Table E.15 (p. 34, lines 1805–1835).

AuthorsOpenReview ↗

Response to Reviewer mb1L - Part 3/3

W5. Definitions and a concrete example for Eq. 6.

A5. QQ contains ground-truth counts from the nucleus annotations, and a⋆=fu(Q)a^\star=f_u(Q) is the reference answer obtained by applying the question's rule. The generated response supplies the reported counts CC and final answer a^\hat a. The function gu(C)g_u(C) computes the task conclusion checked for consistency; πu(a^)\pi_u(\hat a) extracts the corresponding component of the generated answer. For multi-step composition, this component is the selected region.

For the example in Fig. 5, the reference lymphocyte counts in the upper-left, upper-right, lower-left, and lower-right regions are (2,0,2,7)(2,0,2,7), and the reference answer is lowerright:low. ASPECT reports regional counts C=(1,1,1,7)C=(1,1,1,7) and the same final answer. The largest reported count selects the lower-right region, so

gu(C)=πu(a^)=lowerright,Con=1.g_u(C)=\pi_u(\hat a)=\mathrm{lowerright},\qquad \mathrm{Con}=1.

ASPECT also reports the selected-region proportion 7/297/29, corresponding to the low band. The complete answer matches the reference, giving Acc=1\mathrm{Acc}=1. This example shows which quantities are annotation-derived, which are generated, and how the answer projection is used.

Manuscript changes. We added the definitions and example around Eq. 6 in Sec. 3.4 (p. 6, lines 292–309); the example is shown in Fig. 5 (p. 9).

W6. Ground-truth counts and sample-dependent category groups in Eq. 4.

A6. nkn_k is the annotated count for an individually resolved category, and nGn_G is the annotated combined count for a group of categories. The sets D\mathcal D and G\mathcal G are determined separately for each sample by the granularity of its annotations.

For PanNuke, tumor cells, normal epithelial cells, and stromal-like cells have individual labels and belong to D\mathcal D. Inflammatory cells are annotated jointly, giving

G={{lymphocyte,other inflammatory}}.\mathcal G=\{\{\mathrm{lymphocyte},\mathrm{other\ inflammatory}\}\}.

The grouped term supervises the sum of the two predicted inflammatory counts against the annotated inflammatory total. This lets fine and coarse annotations supervise the same output taxonomy.

Manuscript changes. We added the annotation-dependent definitions and PanNuke example around Eq. 4 in Sec. 3.2 (p. 5, lines 221–233).

W7. Motivation for count regression in log space.

A7. Regressing log⁡(1+n)\log(1+n) scales count errors by abundance, reducing the influence of dense patches on the training loss. For a prediction near its reference,

log⁡(1+n^)−log⁡(1+n)≈n^−n1+n.\log(1+\hat n)-\log(1+n)\approx\frac{\hat n-n}{1+n}.

Thus, the loss emphasizes error relative to the count scale, while adding 1 makes zero counts well-defined.

Manuscript reference. The existing log-count objective is defined in Sec. 3.2, Eq. 4 (p. 5, lines 216–229). The rationale above is provided in this response.

W8. Input arrows in Fig. 2 and readability of Fig. 3(a).

A8. Both UNI and CellViT++ take only the image as input. The revised Fig. 2 shows separate image-to-encoder arrows to make this explicit. The revised Fig. 3(a) uses horizontal bar charts with numerical labels for organs, data sources, magnifications, and task types. Each panel accounts for the same 759 benchmark questions.

Manuscript changes. We updated Fig. 2 (p. 4) and Fig. 3 (p. 6), including the Fig. 3 caption (lines 285–287).

Dear Reviewer mb1L,

Thank you for your thoughtful feedback. Your questions helped us clarify the clinical motivation for PathoVernier’s tasks and how ASPECT uses visual tokens to generate regional counts. Your comments on presentation also helped us clarify the notation in Eqs. 4 and 6 and improve the figures' readability. We have revised the manuscript and posted a point-by-point response. We would be grateful if you could take another look when convenient, and we welcome any further questions.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their careful handling of the paper. Hereby we summarize shared concerns and our responses. All reviewers asked how much ASPECT gains from task-specific training and how its visual tokens contribute to the counts it reports. We added three experiments to the revised manuscript in response.

  • Matched training. With the same QA data and update budget, ASPECT-SFT achieves 0.70 answer accuracy and 0.44 Count Acc, compared with 0.63 and 0.37 for conventional Qwen3-VL-8B fine-tuning. After RL, Count Acc reaches 0.47 for ASPECT and falls to 0.03 for the matched baseline (Fig. 4a).
  • Visual tokens and spatial counting. We scored all 1,892 required regional counts, regardless of final-answer correctness. With the image and question held fixed, zeroing the <uni> or <cell> KV entries reduced regional count accuracy from 50.0% to 42.0% or 39.6%, respectively. Both interventions also reduced final-answer accuracy (Appendix E.6).
  • Transfer to new datasets. On MoNuSAC and BRCA-M2C, ASPECT exceeds the matched SFT+RL baseline in both answer and count accuracy (Fig. 4b).

Our replies under each review give the full results and manuscript locations. We welcome any follow-up questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.