- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies generalization in medical image classification when associations between disease and demographic or acquisition attributes change. The authors construct an evidence graph from PubMed literature, assigning types and clinical roles to its nodes and edges. An LLM extracts weak concept labels from reports, and logistic-regression concept readers are trained on frozen medical image embeddings. During diagnostic training, the image encoder, concept readers, and graph structure remain fixed. A graph-constrained probabilistic model combines supporting findings, competing diagnoses, latent mechanisms, and anatomical visibility, while restricting nuisance correction to permitted edges. Whether to apply correction is selected using worst-group accuracy on source-domain validation data. On ten attribute-correlation reversal tasks adopted from KnoBo, the method achieves mean OOD AUROCs of 78.5% for chest X-rays and 84.5% for skin lesions.
Critical strengths
-
The method assigns different diagnostic roles to various clinical relationships, including supporting evidence, competing explanations, anatomical visibility, and the influence of confounding factors. This represents a reasonable alternative to classifiers that arbitrarily exploit correlations among conceptual scores. The decision to enable correction is based on validation results from the source domain, and the potential loss of useful information due to correction is also taken into account.
-
The experiments cover two imaging modalities and ten existing distribution-shift tasks. Comparisons with conceptual models, linear classification heads using the same reader, graph relation role perturbations, and various correction strategies provide evidence for the effectiveness of structural constraints.
Critical weaknesses
-
Some important details regarding the construction of evidence graphs remain unclear. The paper specifies retrieval budgets and tool protocols but does not fully explain how initial evidence and queries are selected, how many results are returned each time, how they are sorted and filtered, or how to determine whether evidence coverage is sufficient. The negotiation process may also retain the original content when disagreements remain unresolved. Structural and citation checks do not prove that these disputed relationships are clinically valid, and how they contribute to the final diagnostic calculation remains to be clarified.
-
The handling of conflicts and disagreements in concept annotation requires clarification. The paper retains the original content when disagreements during graph construction remain unresolved, but it is unclear how these disputed relationships affect subsequent concept annotation and diagnosis. Furthermore, the paper does not sufficiently justify the choice of AUROC ≥ 0.65 as the criterion for retaining concept readers or examine how varying this threshold affects the number of retained concepts and final diagnostic performance.
Questions for authors
-
How are the initial evidence and search queries chosen for each diagnosis? Please clarify the number of results returned per search, the ranking and filtering rules, and the stopping criterion. The appendix limits the number of searches per agent call, but how is coverage of relevant findings and differential diagnoses assessed? Search records linking the retrieved evidence to the final graph relationships would help clarify this process.
-
When a disagreement remains unresolved, do the disputed nodes or edges still participate in concept labeling and diagnostic inference? Why are the original relationships retained, and does removing them affect performance? For concept labels, have you assessed false negatives caused by treating unmentioned or uncertain findings as absent? Please also explain how contradictory reports or inconsistent LLM labels are handled, and whether these decisions have been checked against human annotations or independent concept labels.
-
Why was AUROC ≥ 0.65 chosen as the criterion for retaining a concept reader? Please explain how this threshold was determined and whether alternative thresholds were evaluated. A sensitivity analysis showing how the threshold affects the number of retained concepts and final OOD performance would help justify this choice.
-
What temperatures and sampling settings are used by the LLMs during retrieval and graph construction? When the full process is repeated for the same disease, how much do the retrieved literature and resulting nodes and edges change? Were multiple independent retrieval and graph-construction runs performed, and were the mean and standard deviation of the resulting OOD performance reported?
6 comments
Response to Question 1(ii)
Question 1(ii): Search records that link the retrieved evidence to the final graph.
Thank you for the suggestion. As requested, we summarize the search records of the released graphs in Table R7, and we follow one record from the query to the graph.
Table R7: Evidence in the released graphs.
| Modality | Passages retrieved | Queries | Passages from template / fixed / agent / refinement queries | Passages cited in the graphs | PMC articles cited |
|---|---|---|---|---|---|
| Chest X-ray | 756 | 183 | 278 / 66 / 160 / 252 | 222 | 149 |
| Skin | 304 | 77 | 84 / 18 / 55 / 147 | 87 | 66 |
Search records are kept for every claim in the released graphs. Each node and edge lists the passages that it cites, and each passage is stored with its query, its rank and score, and its source article and section. For example, the Graph Initialization Agent wrote the query "chest radiograph atelectasis radiographic signs volume loss fissure displacement elevated diaphragm linear opacity". The passage returned at rank 3 is from the section "Atelectasis" of PMC2714572 and reads "... displacement of interlobar fissures and elevation of the hemidiaphragm towards areas of atelectasis". The Atelectasis graph cites this passage for the relationship that the finding "hemidiaphragm elevation" supports the diagnosis.
This part has been incorporated into Appendix B.2 of our revised paper, where Figure 4 shows this record together with the other passages returned by the same query. The full search records will be released together with the source code.
Response to Weakness 1(ii) and Question 2(i)
Weakness 1(ii) & Question 2(i): How nodes and edges with unresolved disagreements are used, and why they are retained.
Thank you for the constructive comment. A challenge is left unresolved when the cited evidence can neither confirm nor refute it, because the agents are shown only the first 700 characters of each retrieved passage during negotiation. In that case, the challenged node or edge is kept, because it was added with a cited passage from the medical literature, and removing it without evidence against it could remove clinically meaningful content from the graph. The kept content is then used in all later steps like any other content: nodes with reader questions are annotated and their readers are trained and audited, and nodes and edges enter diagnostic inference with their assigned roles. Empirically, in our experiments, no challenge eventually remaines unresolved. This has been incorporated into Appendix B.2 of our revised paper.
Response to Weakness 1(i) and Question 1(i)
Weakness 1(i) & Question 1(i): Retrieval and graph-construction details are unclear.
Thank you for the constructive comment. As suggested, we describe how a graph is built from start to end, and we summarize the retrieval settings in Table R6.
The corpus and the search tool follow KnoBo (NeurIPS-24) [1]. The corpus consists of articles from the PubMed Central Open Access subset, split at sentence boundaries into passages of about 1,000 characters. It is searched with BM25 as in MedRAG (ACL-24 Findings) [2]. The graph of one diagnosis is built in six steps.
In steps 2 to 5, an agent sends its candidate graph to the compiler, a deterministic program that checks the types of the nodes and edges, their roles, the required content and the citations. The agent corrects any errors and submits only a candidate that has passed. Every node and edge cites a retrieved passage.
After construction, the concepts are tested and refined (Section 3.2), and the system retrieves further passages when a concept is rewritten or replaced.
Table R6: Retrieval settings.
As shown, all retrieval settings are fixed in advance, and the coverage of findings and differential diagnoses is checked by the compiler. This part has been incorporated into Appendix B.2 of our revised paper, and [1] and [2] are now cited in Section 3.1.
[1] Y. Yang et al. A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis. NeurIPS. 2024.
[2] G. Xiong et al. Benchmarking Retrieval-Augmented Generation for Medicine. Findings of ACL. 2024.