acceptodds
Desk rejected at ICLR 2027

PathoDC-VQA: Constructing Difficulty-Controlled Pathology VQA Data from Sparse Captions

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Pathology vision–language models (VLMs) require domain-specific supervision for diagnostic reasoning, yet many public resources contain only information-sparse image–caption pairs. Naively converting these pairs into visual question answering (VQA) often produces caption paraphrases with little control over diagnostic difficulty. We introduce PathoDC, a three-stage pipeline that converts diagnostically eligible pairs into difficulty-controlled supervision. PathoDC first constructs structured Verified Diagnostic Knowledge (VDK) through automated generation and auditing. It then uses a VLM probe to organize candidates into single-image-answerable Easy cases and two evidence-sensitive hard tracks: knowledge-conditioned Hard-K and auxiliary-image-contrastive Hard-V. Finally, it generates track-conditioned rationales and applies an Easy→Hard-K→Hard-V training curriculum. This pipeline produces PathoDC-VDK with 6.3k records and PathoDC-VQA with 109k samples. Supervised fine-tuning of Qwen2.5-VL Instruct yields PathoDC-3B and PathoDC-7B. Using only about one-fifth of Patho-R1’s SFT data and no reinforcement learning, PathoDC achieves competitive performance on public pathology benchmarks. At the 7B scale, PathoDC-7B exceeds Patho-R1-7B on Quilt-VQA and performs comparably on PathVQA, while PathoDC-3B surpasses Patho-R1-3B on PathVQA. On the case-disjoint PathoDC-Bench, the two models improve their backbone weighted-average accuracy by up to 42.34 points and outperform the corresponding Patho-R1 models by 9.89 and 21.17 points, respectively. Although the gains are not uniform across all public VQA and PathMMU evaluations, the results show that evidence-aware difficulty control can improve pathology VLM supervision, with the strongest gains on diagnostic evaluations aligned with the proposed evidence taxonomy.

Primary area:
datasets and benchmarks
Keywords:
pathology VLM, visual question answering, diagnostic reasoning, difficulty-controlled supervision, supervised fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Y93BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper converts existing pathology image–caption pairs into diagnostic question–answer pairs and reasoning supervision. It first extracts explicitly named diseases from captions, then generates and automatically audits disease descriptions, differential diagnoses, and diagnostic criteria to construct Verified Diagnostic Knowledge (VDK). Candidate questions are assigned to Easy, Hard-K, or Hard-V according to a probe model’s answers: correct with the target image alone, correct after adding disease knowledge, or correct after adding an auxiliary comparison image. A teacher generates rationales using the corresponding evidence, while the student receives only the target image and question. Training follows an Easy → Hard-K → Hard-V sequence. The paper constructs approximately 110,000 training examples and a case-disjoint benchmark, with additional evaluation on public pathology VQA datasets and PathMMU.

Critical strengths
  1. The objective is clear and the pipeline is easy to understand. By transforming sparse descriptions into disease knowledge, questions, and reasoning processes, the method provides a concrete approach for reusing existing pathological image and text resources. The introduction of disease knowledge and reference images also ensures that the generated questions go beyond simple rewrites of the original descriptions.

  2. The experiments cover multiple public benchmarks, benchmarks with non-overlapping cases, two model sizes, and multiple component ablation studies. On the PathMMU test set, the 3B and 7B models achieved improvements of 10.73 and 15.50 percentage points, respectively, over the baseline model. Stage-wise results further demonstrate the contributions of different training subsets.

Critical weaknesses
  1. The explanation of the question generation process and its diversity is insufficient. The paper provides prompts for knowledge construction and review, but does not offer equally detailed explanations for initial question generation and the rewriting of difficult questions. Consequently, it remains unclear how question categories, wording, and answer formats are controlled, and whether a large number of samples follow similar question patterns. The provided category statistics are helpful but do not fully address this issue; this is particularly important for work where data construction is the primary contribution.

  2. The difficulty labels rely on the probe answering incorrectly from the target image alone and correctly after receiving additional information. However, this change in correctness does not by itself establish that the question is genuinely difficult. The source and reliability of the additional knowledge and comparison images, and how the procedure distinguishes difficult questions from problematic ones, need further clarification.

  3. Figure 1 appears to have been generated with AI, but the AI Use Statement mentions only language polishing and does not disclose its use in figure generation.

Questions for authors
  1. How does the generation process ensure diversity in the questions, rather than repeatedly producing similar questions from different images? Please clarify how the process encourages variation in question content, wording, and answer format, and provide evidence that the resulting dataset is not dominated by a small set of recurring question patterns.

  2. How do you distinguish genuinely difficult questions requiring additional knowledge or visual comparison from questions with ambiguous wording, insufficient information, or inappropriate reference answers? Beyond the criterion of failing with the target image alone and succeeding with additional information, are the final rewritten questions independently checked for answerability and reference-answer validity before admission to Hard-K or Hard-V?

  3. Figure 11(b) includes Easy (no PT), Easy (PT), Mixed, and RL, although the paper states that no RL is used and the caption describes stage variants from Table 3. Please clarify which training configuration each set of bars represents and reconcile numerical differences, including the Easy score of 77.8 in the figure versus 77.20 in Table 3.

Reviewer_uyt5OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a 3 stage pipeline PathoDC for converting pathology vqa pairs into reasoning vqa answer. Stage I constructs a diagnostic knowledge database. Stage II uses a VLM to identify easy and hard data. Stage III generates reasonings and fine-tunes Qwen. The datasets comprise approximately 6.3k knowledge records, 109,767 SFT examples, and a case-disjoint benchmark containing 1,800 questions. The contribution is primarily a data-construction and training recipe using an existing backbone and a standard SFT objective.

Critical strengths
  1. The paper proposes a useful dataset and data-construction pipeline.
  2. The results are meaningful.
Critical weaknesses
  1. The results are questionable. Fig.10 caption describes a final hard-v, whereas Tab.3 reports improvements from the preceding 7B hard-k. Fig.11 described as visualizing Tab.3, instead includes Mixed/RL absent and labels the hard-v winner as Mixed and marks the Easy as 77.8, whereas Tab.3 reports 77.20.
  2. On a 500-item 2-scored hard-k set, 72.10% implies 360.5 correct answers. On a 300-item hard-v set, 62.40% implies 187.2 correct answers.
  3. The construction probe is qwen-vl-plus. The “independent, weaker quality answerer” is also qwen-vl-plus.
  4. The proposed mechanisms and incremental novelty are not isolated by the experiments.
Questions for authors

See weaknesses.

Reviewer_9tJbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce a three-stage pipeline primarily designed to transform existing data into SFT training data with controllable difficulty. They also introduce Structured Verified Diagnostic Knowledge (VDK) generated with LLMs. Using 109K training samples, the authors train PathoDC based on Qwen2.5-VL through SFT. The experimental results show that PathoDC outperforms Patho-R1 on several benchmarks, including Quilt-VQA, PathVQA, and PathMMU, while using fewer training samples.

Critical strengths
  1. The training data generation pipeline involves multiple LLMs/VLMs, potentially leveraging the rich information contained in image-caption pairs while filtering out low-quality data.

  2. The design of structured VKD has some novelty and practical value, particularly in its application to training data generation.

  3. Transforming image-caption pairs into VQA data for SFT is meaningful, especially in the era of post-training, and could provide insights for other domains.

Critical weaknesses
  1. The paper is poorly written and appears insufficiently polished. I did not enjoy reading it, instead, I found the presentation confusing.

  2. During VQA data generation, the extraction of information directly from the images themselves is insufficient. The pipeline only uses VLMs to supplement information when the corresponding key information cannot be extracted from the captions.

Questions for authors
  1. The captions in PathCap were already generated by GPT-4V. The proposed pipeline merely processes these existing captions rather than fundamentally addressing caption generation or VQA generation itself. This form of text-information post-processing has limited value.

  2. The authors overemphasize the claim that 109K SFT samples achieve results comparable to those obtained with 550K SFT+RL samples. First, the paper does not directly compare the results obtained using 109K and 550K training samples. Second, the authors should avoid emphasizing the SFT+RL comparison. I firmly believe that RL is useful. The current results only demonstrate that PathoDC outperforms Patho-R1; they do not establish that 109K SFT samples can outperform 550K SFT+RL samples.

  3. Despite the complexity of the data generation pipeline, the performance improvements are limited. I observe only modest gains compared with Patho-R1 on Quilt-VQA, PathVQA, and PathMMU.

  4. To better demonstrate the effectiveness of the generated VQA data, the authors should include more comparisons with existing pathology VQA datasets used for SFT, including the original PathGen training data.

Reviewer_xd3bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new pathology VQA dataset, building on existing data. It includes a new pipeline to convert image captions to VQA question-answer pairs. The idea is sensible, moving from generic captions to a VQA that focuses on diagnosis, and separating easy and hard cases, depending on the evidence needed to answer a question.

Critical strengths
  1. Many models are tested.
  2. Evaluation on external VQA datasets is included.
  3. The idea is sensible: to move from generic captions to a VQA that focuses on diagnosis, and separating easy and hard cases, depending on the evidence needed to answer a question.
Critical weaknesses
  1. Results on external VQA benchmarks are mixed. The authors even say “The evidence supports transfer, not universal SOTA.” Can you explain why? What are the limitations preventing SOTA results?
  2. Can you compare your model to a model with the same architecture, trained on your dataset, but using a more standard caption to VQA conversion, and test on the external benchmarks? This would clarify the benefit of your VQA generation technology.
  3. Your abstract only includes numerical results for the testing in your own test set (IID), which can be inflated. Why not include the numerical results on other VQA datasets?
  4. Writing clarity. Text could be made clearer. Many terms are created without a clear definition when they are first mentioned. Some examples: the introduction says ‘Knowledge first, evidence-sensitive difficulty second, and track-conditioned rationales third’, without concretely explaining what this means; the text says “a context-aware store reuses disease-level rules”, without concretely defining “context-aware store” and “disease-level rules”. Text could be clarified by creating less terms and explaining concepts in plain English, with concrete examples.
  5. Can you add concrete Easy, Hard-K and Hard-V examples to the main text?
  6. Lack of human validation. The VQA seems AI-made, with no human verification results shown. At least, it would be important for a human to validate the accuracy of the models used to create the dataset, against a gold-standard baseline. For instance, many LLM/VLM help create dataset, but their accuracies are unclear: probes, audits, grounding checker, filters, etc.
  7. The choice of easy vs hard cases fully depends on the probe VLM. Therefore, would the quality and benefit of this VQA creation method and of the proposed dataset strongly depend on the VLM choice?
  8. Tab. 4: are improvements over answer-only training statistically significant? Overall, the paper lacks statistical tests.
Questions for authors

Please see Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.