acceptodds
Under review as a conference paper at ICLR 2027

DIAGNOSE WHAT YOU DESCRIBE: ALIGNING VISUAL EVIDENCE WITH MAMMOGRAPHY DIAGNOSIS

Abstract

Reliable predictions require agreement between outputs and supporting evidence, which is essential in high-stakes settings. However, recent Vision–Language Models often produce predictions that are not supported by the evidence they present. This issue becomes critical in medical imaging, where such inconsistencies can undermine trust in automated systems. In mammography diagnosis, for example, a model may predict malignancy while describing features that are typically associated with benign findings. To address this problem, we propose a Knowlwdge-Driven Reasoning (KDR) framework. The model generates multiple candidate predictions for each image, each paired with its supporting evidence. We evaluate each candidate by checking whether the evidence is consistent with the predicted diagnosis, which allows us to identify and discard unsupported outputs. The final prediction is selected based on this consistency criterion. A metric to measure agreement between evidence and diagnosis is introduces, which complements standard accuracy. Experiments on multiple public datasets show that the proposed framework improves classification performance and consistency.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Vision–Language Models, Knowledge-Driven, Reasoning, Consistency-Aware

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_RDMPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Knowledge Driven Reasoning, an inference time framework for improving agreement between mammographic findingsn and diagnostic assessments generated by a vlm. It samples multiple candidate interpretations, parses their desctiptions into BI-RADS-aligned features, and selects a candidate using a combination of generative likelihood and clinical-rule consistency. The paper also introduces a Reasoning Consistency Score to assess whether generated findings support the associated diagnosis.

Critical strengths
  1. Tackling inference-time dissonance between diagnostic labels and textual rationales is crucial in safety-critical medical imaging. Post-hoc symbolic filtering offers a clean, modular plug-and-play solution that avoids expensive domain finetuning.
  2. Experiments span five diverse mammography datasets covering different modalities and patient populations.
Critical weaknesses
  1. The paper is not well organized, with typos, grammer errors, very lengthy paragraphs, hard to follow, for example, the first two paragraphs of Section 3 largely repeat the same generation–parsing–verification–selection pipeline, but the clinical rules, feature parser, distance function, and feature-weight construction are not adequately specified.
  2. There is mismatch between subsection headings and their content in Section 3.4, “Iterative Hypothesis Refinement.” This subsection consists of three high-level sentences about stochastic search and expected benefits, without describing an iterative update, feedback mechanism, or stopping criterion.
  3. The main verification step is not clearly explained, the method checks whether the generated description agrees with the diagnosis, but it is unclear how it checks whether that description actually matches the image. Figure 4 introduces image-faithfulness scores without explaining how they are calculated. The clinical rules, parser, and feature weights are also not properly described, making the method difficult to understand or reproduce.
  4. For the experiment results, RCS uses the same rules that guide output selection, so a higher RCS alone does not show better visual grounding. Also, if all rule-breaking outputs are rejected, why is RCS still below 1? The paper mixes up accuracy and macro-F1, does not explain how the binary diagnosis labels are obtained, and does not clearly establish whether the baselines use the same sampling budget. These gaps make it hard to judge what the reported improvements actually demonstrate.
Questions for authors

Questions for Authors

  1. How are the FvF_v and FkF_k scores in Figure 4 calculated? Does the verifier check whether the described findings are actually present in the image? Please provide the clinical rules and a concrete example of how they work.

  2. Do candidate selection and RCS use the same rules? If invalid candidates are always rejected, why is RCS below 1? What happens when none of the candidates passes the checks?

  3. Is “ACC” accuracy or macro-F1? How are the benign/malignant labels obtained for each dataset, especially VinDr-Mammo? Please also clarify the data splits and which expert annotations are used to calculate CIG.

  4. How does KDR compare with likelihood-only selection and majority voting when they use the same sampled candidates? Is there independent evidence that the selected descriptions match the images, including cases where the diagnosis is correct but the explanation could be wrong?

Reviewer_QAa3OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Knowledge-Driven Reasoning (KDR), which samples multiple candidate reasoning trajectories from a Vision-Language Model for a mammogram, parses each into BI-RADS-aligned features and an assessment, rejects trajectories that violate the BI-RADS ontology, and selects the final report by an energy-based score that combines generative likelihood with epistemic consistency. It introduces the Reasoning Consistency Score (RCS) to measure whether the generated evidence entails the diagnosis. On five mammography datasets, KDR reports higher ACC and RCS than direct Qwen2.5-VL decoding and Human-CoT prompting.

Critical strengths
  • The problem is real. A diagnosis that contradicts the described findings is a safety problem that accuracy does not capture, and a post-hoc, model-agnostic consistency check is a reasonable, cheap direction.
  • The method is simple, applies to any VLM without retraining.
Critical weaknesses
  • The main evidence goes in a circle, and the method and the ground truth are mixed together. RCS is computed with the same BI-RADS rule check (the epistemic consistency term V_Logic) that KDR uses to reject and rank candidate trajectories. So KDR is graded by the same rule it uses to pick its answer, and a high RCS says nothing about whether the findings or the diagnosis are clinically correct. The ACC gains are also not separated from the extra compute: there is no baseline that takes a majority vote over the same k trajectories, and the "No Trade-off (lambda = 1)" row of Table 2 gives exactly the single-pass Qwen2.5-VL numbers of Table 1 (0.712 / 0.495), which needs an explanation. VinDr-Mammo and INbreast provide BI-RADS categories, not biopsy-confirmed malignancy. If "malignant" there means BI-RADS 4/5, then the label and the consistency rules are both BI-RADS, and part of the ACC gain is the rule system predicting its own label. The paper also contradicts itself on the rules: the consistency kernel in Eq. 4 gives minus infinity to any trajectory whose features do not support the assessment, and Fig. 1 rejects "spiculated mass to benign" as a rule violation, yet Case III in Fig. 4 selects "oval, circumscribed mass to malignant" as "evidence-aligned". Fig. 4(b) also ranks candidates with an "Image-Finding Faithfulness" score that is not part of the verification score S(Q, R) in Eq. 2 and cannot be computed without ground-truth findings.
  • The method and experiments are not described well enough to evaluate or reproduce. The anonymous code link only shows "The code will be available soon", so nothing can be checked there. Figures 2 and 3 are quite sparse: they take about half of page 7 for 26 numbers in total (three stacked bars, and four "normalized" curves over five values of k with no dataset named, no absolute values, and no error bars over the five runs), while the rules and the parser that the method depends on are never written down. The paper does not give: a single clinical rule from the BI-RADS ontology K; the parser that maps the generated text to features F; dist(phi_i, A) in Eq. 4; the diagnostic saliency weights w_i; lambda; the Qwen2.5-VL size; whether the model was fine-tuned (splits are mentioned, but no training is described); the Human-CoT prompt; the input unit (whole image, breast, or lesion crop); how each datase's labels were turned into benign vs malignant; who audited the 100 error cases; and any variance over the "five runs". The only baselines are the same VLM with two prompts, so a macro-F1 of 0.75 to 0.84 cannot be compared with anything.
Questions for authors
  • Please list the clinical rules in K, the parser, dist(.), w_i and lambda. How many of the k trajectories does the infinite penalty reject on average, and how does Case III (oval, circumscribed, malignant) pass it?
  • Which Qwen2.5-VL checkpoint, zero-shot or fine-tuned per dataset, and if fine-tuned, on which targets and data?
  • How were binary malignancy labels defined for VinDr-Mammo and INbreast, and what is the input unit for each dataset?
  • What do majority vote over the same k trajectories and likelihood-only selection achieve for ACC, and why is the lambda = 1 ablation identical to single-pass decoding?
  • Who performed the 100-case error audit and the RCS judgments, was a radiologist involved, and can RCS be measured by a judge independent of the selection rules? Please add standard deviations over the five runs and name the dataset used in Figures 2 and 3.
Reviewer_8ZQKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Knowledge-Driven Reasoning (KDR), an inference-time knowledge-integrated procedure for mammography malignancy prediction with a vision-language model. For each image, k candidate reasoning trajectories, each consisting of a free-text description and a diagnosis, are sampled from Qwen2.5-VL. Each trajectory is parsed into BI-RADS-style features F and an assessment A and checked against a set of clinical rules K derived from BI-RADS. Candidates that violate a rule receive an infinite penalty and are discarded, and the survivors are re-ranked by an energy score that mixes the normalized generative log-likelihood with a weighted rule-consistency term. The highest-scoring candidate is returned. The paper introduces a Reasoning Consistency Score (RCS), defined as the fraction of outputs whose extracted features entail the predicted assessment under K, and a Clinical Information Gain (CIG) metric. On CBIS-DDSM, INbreast, VinDr-Mammo, CMMD, and MIAS, KDR is reported to improve both macro-F1, which the paper calls ACC, and RCS over direct Qwen2.5-VL decoding and a Human-CoT prompting baseline (Table 1). It is also reported to shift errors from logical hallucinations to perceptual failures in a manual audit of 100 error cases (Figure 2), to improve with the number of candidates k (Figure 3), and to depend on the feature weights, the likelihood/consistency trade-off, and k in an ablation on VinDr-Mammo only (Table 2). Three qualitative CBIS-DDSM cases are shown in Figure 4.

Critical strengths
  1. The problem is real. A VLM that describes benign-looking findings and then predicts malignancy is not trustworthy even when the label happens to be right, and accuracy alone does not expose this. Separating logical from perceptual errors in Section 4.4 is a useful framing for medical VLM evaluation.

  2. A training-free verify-and-select layer that uses an external clinical ontology and leaves the backbone untouched is a reasonable design in principle, and the evaluation covers five mammography datasets with different acquisition settings.

Critical weaknesses
  1. The headline result is obtained by directly optimizing the evaluation metric. RCS and KDR's verifier are the same function.

    • RCS in Equation 6 counts an output as consistent if its extracted features entail its assessment under the rule set K. KDR's verification step in Equation 4 discards exactly the candidates that fail this test and returns one that passes. Any output KDR returns therefore satisfies the RCS criterion by construction, and KDR's RCS is close to 1 regardless of whether its outputs are clinically consistent or visually grounded. The gap between KDR and the baselines in Table 1, which is +0.38 to +0.40 RCS on every dataset, measures how often unconstrained sampling violates the authors' own rules.
    • The same applies to the Hallucination Rate in Figure 3 and to the ablations in Table 2, where removing the filter with lambda = 1 or k = 1 lowers RCS because RCS itself is the filter. The only independent evidence is the manual audit in Figure 2, which covers 100 error cases with no statement of the human audit protocol or rubrics. Consistency judged by radiologists on a random sample of outputs per method, or RCS computed against a rule set that is independent of the one used for selection, is required before any claim about reasoning quality can be made.
    • The ACC gains cannot be attributed either, because the natural baselines are missing. These include self-consistency majority voting over the k samples, which Section 2.2 cites and then does not compare to, best-of-k by likelihood alone (energy score ablation, this is conducted on only one dataset as lambda = 1), a supervised classifier on the same splits, and a fine-tuned or medical VLM, which are the common baselines available on all five datasets. Human-CoT baseline is also never clearly described.
    • The paper claims to address report generation in the Introduction and Section 3 but evaluates only binary malignancy classification with macro-F1.
  2. The design is not well motivated, and the obvious alternative use of the same knowledge is not compared.

    • KDR has access to a structured BI-RADS rule set K and uses it only after generation, to filter and re-rank k free-form samples. The most direct way to use K is to do rule-based or graph-based reasoning with it, or put it into the prompt, in the form of in-context instructions that give the BI-RADS knowledge, or as a structured output format that asks for BI-RADS-sourced concepts before the assessment, so that System 1 is guided by the knowledge rather than sampled k times and corrected post hoc. The paper never explains why knowledge should be withheld from generation and never compares to knowledge-in-prompt decoding with a single sample or with self-consistency. Even the author's own discussion in Appendix A.3 argues for guiding perception up front, since it states that when System 1 fails to extract diagnostically relevant features, System 2 has no valid hypotheses to evaluate. Without this baseline, I cannot tell whether the post-hoc filter adds anything over simply telling the model the rules, at 1/k of the inference cost.
    • The ablation results raise more questions. The lambda = 1 row in Table 2 reports 0.712 ACC and 0.495 RCS, identical to Qwen2.5-VL direct decoding on VinDr-Mammo in Table 1, although lambda = 1 should select the candidate with highest likelihood among k samples. The k = 1 row reports 0.692 and 0.441, which is worse than direct decoding, although with one candidate KDR can only return it or reject it, and the paper does not say what is returned when every candidate is rejected. Table 1 reports averages over five runs without any standard deviation or interval.
  3. The method cannot be reproduced from the paper, and the code is not available.

    • The rule set K, which is the core of the method, is never given. The paper does not list a single rule, does not say how many rules there are or where they come from, and does not explain how BI-RADS descriptors were turned into the logical constraints Logic(varphi_i -> A). This last point matters because BI-RADS descriptors are associated with probabilities of malignancy rather than with deterministic implications. A spiculated margin raises suspicion but a circumscribed margin does not exclude malignancy, so it is not obvious what a rule such as "circumscribed -> benign" would even mean.
    • The paper is also inconsistent about what kind of object K is. Equation 4 assigns an infinite penalty to any violated rule, so K is a hard filter. Section 4.7 then states that KDR does not enforce a brittle one-to-one mapping between individual BI-RADS descriptors and pathology, which seems to mean a soft score. The worked examples in Figure 4 do not resolve this. In Case I the output "oval mass with circumscribed margins, prediction malignant" is marked as low support, while in Case III the output "oval mass with circumscribed margins and heterogeneous internal density, which can still be malignant, prediction malignant" is marked as evidence-aligned. The shape and margin descriptors and the diagnosis are the same in both cases, and the only difference is the added phrase "heterogeneous internal density", so whatever rule separates the two verdicts must hinge on that phrase. The paper gives no such rule, and the next point explains why the phrase cannot come from BI-RADS. I therefore cannot infer K from the formulation or from the examples.
    • The feature vocabulary does not appear to be the BI-RADS lexicon either. The KDR outputs in Figure 4 rely on "homogeneous density" and "heterogeneous internal density" to separate benign from malignant, but these are not BI-RADS mammographic mass density descriptors. BI-RADS describes mass density as high, equal, low, or fat-containing.
    • The following are also undefined. The deterministic parsing P(F | R) could be a regex, an LLM, or a classifier. The quantity dist(varphi_i, A) in Equation 4, the saliency weights w_i and their source, the k used in Table 1, the Qwen2.5-VL size, and the sampling temperature are not stated. The classification unit and input are not stated, so I do not know whether the model sees a full image, a single view, or a lesion crop, as Figure 4 suggests. The number of test images, the class balance, or the standardized splits for INbreast, CMMD, and MIAS, which have no official splits, are not reported. Section 3.4 is titled Iterative Hypothesis Refinement and Figure 1 shows Iterative Re-ranking and Pruning, but no iteration is described anywhere, and as far as I can tell, the method is sample k, filter, and argmax. Figure 4(b) scores candidates with an image-finding faithfulness F_v and a diagnostic plausibility F_k on a 0-1 scale, and neither of these quantities appears in Section 3.
    • The Reproducibility Statement says the code is available at the anonymous link. At the time of review, the repository contains only a README stating that the code will be available soon.
  4. The figures and parts of the text appear to be under-edited, and they contain substantive errors.

    • Figure 1 contains elements with weird artifacts, including the Vision-Language Model block, the mammogram with a red arrow, and the BI-RADS Ontology document icons. Its bottom panel contrasts a Standard VLM output, "High Risk for Cancer", labeled a logical hallucination, with the KDR output "Highly Suspicious of Malignancy", labeled an epistemically consistent report. The two statements say the same thing, and the panel illustrates nothing.
    • Figure 4 overlays a colored glow on each lesion crop. These do not seem to be model outputs, since no attention or saliency method is mentioned anywhere, so they should not be presented as evidence of what each model sees. The panel (b) title reads "Candidate-level analysis forr Case I" with a typographic artifact.
    • Several citations do not support the claims they are attached to. He et al. 2022 (MAE), Wang et al. 2023 (InternImage), and Zhai et al. 2022 (scaling ViT) are cited as vision-language models or as evidence that VLM assessments contradict their evidence. D'Orsi et al. 2018, the BI-RADS atlas, is cited for "right answers for the wrong reasons". Many parts of the paper, namely Sections 3.3, 3.4, 4.5 and Appendices A.2 and A.3, contain awkward terminology from cognitive-science framing such as "BI-RADS axiomatic manifold", "decisional geometry", and "epistemic coherence". Sections 3 and 3.1 restate the same pipeline twice, and there are typos in the abstract ("Knowlwdge-Driven", "is introduces") and in section titles ("Clinic Reasoning"). The Related Work also misses the directly relevant literature, including but not limited to faithfulness and hallucination evaluation for medical report generation [1, 2], mammography vision-language models that already align images with BI-RADS descriptors and reports [3, 4], and verifier-based best-of-N selection and test-time compute scaling, of which the sample-then-select design in this paper is an instance [5, 6].
Questions for authors
  1. How is the "rule set" K constructed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.