- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ASPECT, a framework for improving visually grounded reasoning in pathology vision-language models. The central motivation is that correct answers to pathology questions do not necessarily indicate accurate visual observations, particularly when the answers depend on cell counts and regional cellular composition. ASPECT introduces intermediate visual tokens supervised through pathology feature reconstruction, cell feature alignment, and count prediction. The framework uses a three-stage supervised fine-tuning curriculum, followed by reinforcement learning that rewards answer correctness and consistency with the model's reported measurements. The authors also introduce PathoVernier, a benchmark containing 759 expert-reviewed questions from five pathology datasets.
Critical strengths
-
PathoVernier provides a structured evaluation of cellular composition reasoning, with reference counts and deterministic decision rules. The distinction between answer correctness and measurement accuracy is particularly useful. The benchmark may serve as a valuable resource for investigating visual grounding in pathology VLMs.
-
The paper addresses a fundamental limitation of pathology VLMs: correct predictions can be accompanied by inaccurate visual evidence. Explicitly evaluating the quantitative observations underlying model predictions is a meaningful direction for improving the reliability of medical vision-language systems.
Critical weaknesses
-
The cell-count supervision predicts image-wide cell abundances, while many benchmark questions require region-specific counting and spatial comparisons. Furthermore, the cell feature alignment objective aggregates instance features through mean pooling, which does not explicitly preserve spatial positions or individual instance identities. Although the model learns regional observations through response-text supervision, it remains unclear how much of its regional counting capability is attributable to the proposed visual-token representations rather than the generated reasoning supervision. More direct analysis of the spatial information encoded in these tokens would strengthen the methodological contribution.
-
The architectural contribution is not fully separated from the benefits of task-specific supervision. The Direct Reason baseline retains the proposed visual supervision objectives. A conventional Qwen3-VL-8B fine-tuning baseline using the same quantitative question-answer data, but without the proposed visual tokens and auxiliary objectives, would provide a clearer assessment of the contribution of the architecture itself.
Questions for authors
-
How do the proposed cell tokens encode spatial information when the cell feature alignment objective uses mean-pooled instance representations? Can the authors provide an analysis of regional counting accuracy independent of the final language-generated answers?
-
How does ASPECT compare with conventional Qwen3-VL-8B supervised fine-tuning using the same quantitative QA examples, but without the proposed visual tokens and auxiliary visual losses?
2 comments
Response to Reviewer BniS - Part 2/2
W2/Q2. Comparison with conventional Qwen3-VL-8B fine-tuning on the same quantitative QA data.
A2. ASPECT improves both answer and count accuracy over conventional fine-tuning on the same QA data. To test this directly, we train Qwen3-VL-8B with matched data, update budget, LoRA settings in the vision encoder/projector/LLM, and RL configuration. The conventional baseline removes visual tokens, auxiliary visual losses, and the three-stage curriculum.
| Method | Acc↑ | CA↑ | RAWR↓ | Count Acc↑ |
|---|---|---|---|---|
| Qwen3-VL-8B SFT | 0.63 | 0.71 | 0.62 | 0.37 |
| ASPECT-SFT | 0.70 | 0.80 | 0.53 | 0.44 |
| Qwen3-VL-8B SFT+RL | 0.57 | 0.67 | 0.97 | 0.03 |
| ASPECT | 0.75 | 0.81 | 0.49 | 0.47 |
ASPECT-SFT improves Acc from 0.63 to 0.70 and Count Acc from 0.37 to 0.44. This shows that the proposed training design improves quantitative observations as well as final answers. RL further raises these metrics to 0.75 and 0.47. Under conventional training, the corresponding scores fall to 0.57 and 0.03, and RAWR rises to 0.97.
The paper's Table 2 provides a complementary controlled comparison: removing all visual losses while retaining the visual tokens, response format, and curriculum reduces Acc/Count Acc from 0.70/0.44 to 0.64/0.32. The conventional baseline evaluates the full ASPECT design against standard task fine-tuning, and the loss-only comparison identifies the contribution of visual supervision within that design. Together with the regional analyses in A1, these results link visual supervision to improved measurement at both image and region level.
Manuscript changes. We added the matched-training comparison in Sec. 4 (p. 8, lines 399–405) and Fig. 4(a); the loss-only ablations are in Table 2 (p. 9).
Response to Reviewer BniS - Part 1/2
Thank you for your constructive review and recognition of the benchmark's evaluation of quantitative observations. We address the visual-token analysis and conventional fine-tuning comparison in two comments. Revisions are marked in purple in the revised PDF.
W1/Q1. Spatial information in visual tokens and regional counting accuracy.
A1. Mean pooling constructs the auxiliary cell-feature targets; visual-token states are computed by the decoder from the image and preceding context. Regional observations are then generated using these states together with spatial image tokens. Cell alignment and count supervision train appearance and abundance; regional observation supervision connects these representations to question-specific measurements. We test this connection through direct count-head readout, regional counting under matched text supervision, and inference-time token interventions.
Direct decoding without language generation. Using the final ASPECT model, we supply the image, question, and fixed visual-token prefix, extract the six final-layer
<cell>states, add the trained role embeddings, concatenate them, and apply the existing SFT-trained count head. This directly produces counts without generating observations or answers and requires no additional training. We evaluate 553 unique patches after deduplicating the 759 benchmark questions by image.The strong correlations with annotations (ρ=0.886–0.971) show that cellular abundance is directly accessible from the learned
<cell>states.Regional counting with the same text supervision. We separately score all 1,892 required regional counts in generated observations from 566 spatial questions: 760 region-selection fields, 760 multi-step composition fields, and 372 region-comparison fields. The denominator includes every required field, irrespective of final-answer correctness. A count is correct when ∣c^−q∣≤max(1,0.1q); missing or unparseable counts are incorrect.
These regional scores are newly computed. The loss-removal variants use the paper's Table 2 settings, retaining response-text supervision, visual tokens, and curriculum. Removing cell alignment or count supervision lowers regional accuracy from 0.465 to 0.249 or 0.353, reductions of 21.6 and 11.2 percentage points. This shows a contribution from visual supervision to regional measurement under the shared text-training objective. Regional accuracy pools fields, while the paper's Count Acc averages per-question scores over all questions.
Use of visual tokens during inference. We zero or replace the KV entries of the selected token group across all 36 decoder layers, retain the original image/question and prefix token IDs, recompute subsequent prefix states, and regenerate observations and answers.
<uni>KV<cell>KV<uni>KV<cell>KVReplacement donors are different patches from the same dataset with at least a three-fold difference in reference total nuclei, encoded with the recipient question. We use seed 0 and a common donor for questions from the same image. Zeroing covers all 759 questions and 1,892 regional fields; replacement covers the 702 eligible questions and 1,708 fields. Each intervention uses its matched original-model control.
Both token groups affect reported regional counts and final answers. For example, replacing
<cell>KV lowers regional accuracy from 0.498 to 0.368 and answer accuracy from 0.74 to 0.63. Together, direct readout, training ablations, and inference interventions connect quantity information in the visual tokens to improved regional measurements and their use in generation.Manuscript changes. We clarified the count-head role in Sec. 3.2 (p. 5, lines 235–239), added spatial analyses in Sec. 4 (p. 10, lines 508–513), and included full protocols and results in Appendices E.5–E.7, Tables E.13–E.15 (pp. 32–34, lines 1714–1835).