acceptodds
Under review as a conference paper at ICLR 2027

PathWeaver: Enhancing Diagnostic Reasoning in Pathology Vision-Language Models with Explicit Reasoning Graphs

Abstract

Pathology vision-language models (VLMs) increasingly generate diagnostic rationales, yet these rationales are typically expressed as free-form text, making it difficult to verify or supervise individual claims and to trace the chain of evidence that leads to a conclusion. We introduce PathWeaver, a framework that makes the diagnostic reasoning behind these rationales structured, explicit, and learnable. It represents each case as a directed acyclic graph (DAG) that links morphological observations, intermediate inferences, and diagnostic conclusions through typed relations. A DAG generator is trained on graphs curated from heterogeneous pathology QA sources and refined by node-level preference optimization, in which counterfactual pairs differ in a single node so that the preference signal localizes to one claim. A graph-conditioned response model then encodes the DAG with relation-aware graph attention to guide answer generation. PathWeaver outperforms general-purpose, medical, and pathology-specialized VLMs across pathology QA formats (66.6% multiple-choice, 55.5% spatial-locating accuracy) and on PathMMU (71.0%). Their explicit structure also enables composition: training-free graph weaving integrates DAGs across patches, improving PanCancer histologic diagnosis benchmark by 10.8% over the best single-view baseline. These results show that an explicit reasoning graph can serve as a single unit of supervision, conditioning, and composition, supporting transparent and inspectable AI-assisted diagnostic reasoning and laying a foundation for more trustworthy collaboration between pathology VLMs and pathologists.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
computational pathology, vision-language models, diagnostic reasoning, multimodal learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_UxM8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PathWeaver, a framework that represents pathology reasoning using structured diagnostic evidence graphs. It combines graph generation, node-level preference optimization, and graph-conditioned answer generation, with an extension for cross-patch evidence integration.

The authors evaluate the framework on multiple pathology QA benchmarks, including PathMMU and PanCancer, reporting improvements over existing VLM baselines. However, the independent contribution of graph structure to these improvements remains unclear.

Critical strengths
  1. Structured diagnostic reasoning. The paper presents a coherent framework that uses explicit evidence graphs for fine-grained supervision, answer generation, and cross-patch integration.

  2. Promising experimental results. The framework shows improvements across multiple pathology benchmarks. In particular, the cross-patch graph integration demonstrates practical potential beyond single-image reasoning.

Critical weaknesses
  1. Insufficient validation of diagnostic evidence graphs. The paper lacks direct evidence that the generated graph nodes and relations are medically valid and faithfully grounded in pathology images. This weakens its claims about reliable and interpretable diagnostic reasoning.

  2. Unclear contribution of graph structure. The ablation results show that graph conditioning alone provides inconsistent improvements, while most gains emerge after node-level preference optimization. The authors should demonstrate whether the relational graph structure itself contributes beyond node-level textual information and additional training.

Questions for authors
  1. Could the authors clarify whether the generated diagnostic graphs have been independently evaluated for medical correctness and visual grounding?

  2. Could the authors explain what evidence supports the independent contribution of graph relations, beyond node-level textual information and preference optimization? Are any relevant ablation results already available?

The paper presents substantial work and promising results, but its central research question is somewhat diluted by the number of components and applications. I encourage the authors to focus on the most compelling contribution, particularly cross-patch evidence integration, and provide deeper analysis of why explicit graph structures are necessary and how they improve diagnostic reasoning.

Reviewer_Gv67OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents PathWeaver, a framework that represents pathology diagnostic reasoning as directed acyclic graphs (DAGs), connecting morphological observations, intermediate inferences, and diagnostic conclusions through typed relations. The framework consists of three main stages: supervised DAG generation, node-level preference optimization using counterfactual graph pairs, and graph-conditioned answer generation with relation-aware graph attention. The authors construct approximately 1.07 million pathology training examples and 414,644 node-level preference pairs. They further introduce a training-free graph-weaving procedure that merges semantically equivalent nodes to aggregate diagnostic evidence across tissue patches. PathWeaver achieves 70.95% accuracy on PathMMU, 55.46% accuracy on location-based questions, and 73.93% accuracy on PanCancer with graph weaving. Ablation studies suggest that node-level preference optimization improves downstream performance, particularly on location-based questions.

Critical strengths
  1. A coherent integration of structured reasoning across multiple stages. Rather than using graphs solely as post-hoc explanations, PathWeaver employs them as supervision targets, conditioning inputs for answer generation, and intermediate representations for cross-patch evidence aggregation. The node-level preference optimization strategy is particularly interesting because it localizes feedback to individual diagnostic claims while using global preservation and format-consistency losses to maintain the overall graph structure.

  2. Comprehensive evaluation across diverse pathology tasks. The experiments cover multiple-choice, open-ended, and location-based QA, as well as cross-patch diagnostic reasoning. The improvements in location-based QA and PanCancer are noteworthy. In particular, graph weaving improves PanCancer accuracy from 71.41% to 73.93%, outperforming graph-free patch-level majority voting (68.94%). The training-stage ablations in Table 2 also provide useful evidence of the contribution of node-level preference optimization.

Critical weaknesses
  1. Insufficient isolation of the benefits of graph-based reasoning. Although the graph-free SFT controls use the same curated training corpus, PathWeaver introduces a separate DAG generator, additional node-level preference supervision, and a graph-conditioned response model, resulting in greater model capacity and inference cost. The full 8B/8B configuration improves PathMMU accuracy only modestly over graph-free Qwen3-VL-8B SFT (70.95% vs. 70.25%). Moreover, graph conditioning without node-level preference optimization produces inconsistent gains and even degrades performance on some benchmarks (Table 2). Without compute-matched comparisons or controls that use the same node information without graph structure, it remains unclear whether the improvements primarily arise from explicit graph reasoning, preference optimization, or additional computational resources.

  2. Limited validation of the correctness and reliability of generated diagnostic graphs. Most evaluations focus on final QA performance rather than the factual correctness of observation nodes, the validity of intermediate inferences, or the medical soundness of edge relations. Consequently, the results do not directly establish whether the generated graphs provide faithful diagnostic explanations. Although the authors evaluate semantic node matching on 1,000 annotated pairs, this does not establish the correctness of complete diagnostic graphs or their merged representations. Errors introduced during LLM-assisted graph construction or cross-patch node merging could propagate into downstream predictions. A systematic, pathologist-validated evaluation of graph quality would substantially strengthen the paper's claims about interpretability and trustworthy

Questions for authors

1.Given that the graph-free SFT baselines already use the same curated training corpus, could the authors provide additional comparisons under matched model capacity, preference supervision, and inference compute? In particular, comparing the proposed method against text-based reasoning with equivalent intermediate supervision, as well as standard DPO without node-level masking, would help isolate the contributions of graph structure and node-level preference optimization.

2.Have expert pathologists independently evaluated the factual correctness of generated observation nodes, intermediate diagnostic inferences, and edge relations? If so, could the authors report these results and analyze common error types, particularly whether incorrect intermediate claims can lead to correct final diagnoses?

3.How are the negation and qualifier checks in graph weaving implemented? Are they rule-based or model-based? Additionally, how does the merging procedure handle semantically similar but clinically conflicting statements, and what safeguards prevent such conflicts from propagating into the final diagnostic conclusion?

Reviewer_qN9xOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

PathWeaver introduces case-specific, typed diagnostic evidence graphs that connect morphological observations, intermediate inferences, and diagnostic conclusions. It curates graph supervision from heterogeneous pathology data, trains a vision-language model to generate diagnostic DAGs, and refines it with node-level preference optimization (NPO). A separate response model uses relation-aware graph embeddings for answer generation, while training-free graph weaving integrates evidence across image patches. The paper evaluates the framework on pathology QA and cross-patch diagnosis, reporting consistent benefits from the complete pipeline.

Critical strengths
  1. A coherent, pathology-motivated intermediate representation. The case-specific DAG exposes how morphological evidence supports diagnostic conclusions and serves as a common representation across supervision, response generation, and evidence aggregation. This is particularly relevant to pathology, where conclusions should be traceable to individual findings.
  2. A meaningful fine-grained learning objective. Node-level counterfactuals turn individual diagnostic claims into explicit preference targets. The training-stage analysis supports the practical value of the resulting refinement, rather than treating an entire rationale as an indivisible unit.
  3. Compelling cross-patch applicability. Graph weaving reuses the learned representation to compose evidence across tissue regions without further training. Its improvements over single-view reasoning and graph-free voting provide a convincing demonstration of practical utility beyond standard pathology QA.
Critical weaknesses
  1. The necessity of explicit graph structure remains unclear. Graph conditioning alone does not consistently outperform matched graph-free fine-tuning. Without a controlled comparison against linearized evidence or ablated relations, the claimed benefit of typed graph reasoning is not convincingly distinguished from access to additional diagnostic content.
  2. The specific contribution of node-level preference optimization is not isolated. The complete NPO stage improves downstream performance, but there is no matched full-sequence DPO comparison or direct assessment of node factuality. As a result, the evidence supports preference refinement more strongly than the proposed benefit of localized credit assignment.
  3. The advantage of graph weaving over simpler matched aggregation is unresolved. The reported voting control uses a different graph-free model, rather than aggregating the same generator's patch-level predictions. This leaves the distinctive value of graph merging, relative to a simpler inference procedure with comparable information and cost, insufficiently established.
Questions for authors
  1. Under matched data, backbone, and supervision, does relation-aware graph conditioning outperform a linearized list of the same evidence nodes or a version without meaningful edge relations?
  2. Does NPO outperform conventional sequence-level DPO trained on the same preference pairs, and does it improve factual correctness at the node level?
  3. Does graph weaving outperform simpler aggregation of the same DAG generator's patch-level conclusions under comparable inference conditions?
Reviewer_PiNjOpenReview ↗
Rating
4(+1)
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PathWeaver, a framework for improving diagnostic reasoning in pathology VLMs by representing intermediate reasoning as directed acyclic graphs (DAGs). The authors leverage LLMs and medical knowledge graphs to construct a large-scale pathology dataset containing diagnostic DAGs and CoT responses. It involves three training phases. First, a DAG generator is trained to predict diagnostic graphs from pathology images and questions. Next, it is refined through node-level preference optimization to encourage accurate diagnostic claims while preserving the overall graph structure. Finally, a separate VLM is trained to incorporate diagnostic DAGs through relational graph attention for response generation. During inference, a training-free graph-weaving strategy integrates diagnostic evidence across multiple image patches. Experiments on pathology QA benchmarks demonstrate improvements over the reported baselines, particularly in multiple-choice, spatial, and cross-patch diagnostic inference.

Critical strengths
  1. The paper investigates a DAG-based representation that explicitly captures morphological observations, intermediate reasoning, and diagnostic conclusions. This representation provides a reasonable approach to structuring diagnostic evidence and could make LLM-generated conclusions more interpretable and reliable.

  2. The authors harmonize more than one million pathology examples from 15 source datasets into a consistent format. The resulting dataset and construction pipeline could be highly valuable for developing pathology models.

  3. The paper conducts extensive experiments across multiple pathology QA benchmarks and demonstrates substantial improvements over existing models in metrics such as BLEU and ROUGE-L. Notable performance gains are also observed from the training-free graph-weaving strategy on the PanCancer QA benchmark.

Critical weaknesses
  1. The contribution of DAG-based reasoning is not sufficiently isolated. PathWeaver employs a model to generate diagnostic DAGs containing both diagnostic information and relational structures, which are subsequently incorporated into the response model. However, it remains unclear whether the proposed DAG structure provides advantages over alternative reasoning paths, such as tree-based reasoning or DAGs with randomized edges.

  2. Insufficient validation of the quality and reliability of diagnostic graphs. The training pipeline relies heavily on LLM-generated findings, diagnostic graphs, and reasoning traces. However, the paper provides limited evidence regarding the medical correctness of individual nodes and the validity of their relationships. An evaluation by expert pathologists would strengthen the reliability claims.

  3. Limited comparability due to differences in model capacity and computational cost. PathWeaver employs two separate VLMs for DAG generation and response generation, whereas most baselines rely on a single model. This introduces additional parameters and inference overhead, making it difficult to isolate the benefits of the proposed graph-based reasoning framework. Comparisons under matched computational budgets would provide stronger evidence for the effectiveness of the proposed approach.

Questions for authors
  1. How much does performance change when graph edges are removed or randomized while keeping node texts and roles unchanged?

  2. What are the inference latency and computational costs of PathWeaver compared with a single graph-free VLM, especially when processing numerous patches from whole-slide images?

  3. Can the authors provide comparisons with single-VLM baselines under matched computational budgets?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary Response

For ease of reference, we refer to reviewers PiNj, qN9x, Gv67, and UxM8 as R1, R2, R3, and R4, respectively.

We sincerely thank all reviewers for their thoughtful and constructive feedback, which helped us strengthen the experimental evaluation of PathWeaver. We particularly appreciate their recognition of:

  • A coherent representation for diagnostic reasoning. Reviewers recognized the value of explicitly structuring diagnostic evidence through graphs (R1–R4), particularly their use as a common representation for supervision, answer generation, and cross-patch evidence integration (R2–R4).

  • Fine-grained supervision of diagnostic claims. Node-level counterfactual preferences provide explicit learning signals for individual claims, and the training-stage ablations provide evidence of its contribution. (R2, R3)

  • A substantial pathology data resource. The harmonization of more than one million examples provides a consistent training resource and construction pipeline for pathology models. (R1)

  • Broad evaluation and promising cross-patch applicability. The experiments cover diverse pathology QA tasks, while training-free graph weaving extends the framework to integrating diagnostic evidence across tissue regions. (R1–R4)


We have addressed the reviewers' main concerns through additional experiments and clarifications. Below, we summarize the questions and corresponding updates.

  • Expert validation. Assessing medical correctness. (R1, R3, R4)

    Addressed in Appendix R, where we collected and analyzed 1,408 ratings from seven pathologists covering nodes, edges, and final answers in single- and cross-patch settings. The analysis results support the medical validity of the generated graphs.

  • Contributions of graph content and structure. Assessing the roles of diagnostic evidence, graph connectivity, and relation semantics. (R1–R4)

    Addressed in Appendix O, Tables 20–21, through:
    (i) a linearized-text baseline using the same evidence nodes;
    (ii) node-input removal and node-order randomization;
    (iii) edge removal and random rewiring;
    (iv) random relation labels and replacement of all supports relations with contradicts.

  • Contribution of NPO. Comparing NPO with full-sequence DPO under the same preference supervision. (R2, R3)

    Addressed in Responses to R2 W2/Q2 and R3 W1/Q1, using the same 4B backbone and preference pairs. Standard full-sequence DPO suffered format degradation, adding auxiliary CE restored validity, but NPO still achieved higher validity and accuracy.

  • Model capacity and computational cost. Providing comparable single-VLM baselines and measuring inference cost. (R1, R3)

    Addressed in Response to R1 W3/Q3 and Appendix P.1, Table 22, where we added Qwen3-VL-8B SFT+DPO and InternVL3-14B SFT comparisons and measured parameter counts, FLOPs, throughput, and GPU memory.

  • Multi-patch inference latency. Assessing how inference cost scales across patches. (R1)

    Addressed in Appendix N, Tables 18–19, and Appendix P.2, Tables 23–24, comparing DAG formats in latency and downstream performance, and showing that PathWeaver's full pipeline took less than half the time of graph-free SFT processing.

  • Cross-patch aggregation. Comparing weaving with simpler aggregation of the same generator outputs. (R2, R4)

    Addressed in Section 3.4, Table 3, where we added two conclusion-voting baselines using the same patch DAGs.

  • Safeguards of graph-weaving. Explaining the rule-based checks used during grid merging. (R3)

    We clarified graph-weaving safeguards in Response to R3 Q3 and Appendix Q, Figure 7, and evaluated their effects alongside other post-merging steps in Table 25.

Together, these additions strengthen the evidence for PathWeaver’s medical validity, methodological contributions, and practical applicability, and hopefully addressed reviewers’ main concerns. We appreciate Reviewer PiNj’s and qN9x's positive reassessment and actively participating in the discussion. We sincerely thank all reviewers for their time and constructive feedback, which have helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.