acceptodds
Withdrawn from ICLR 2027

Retrieval-Augmented Anatomy-Aware Reasoning for Interpretable Alzheimer's Diagnosis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Accurate Alzheimer's disease (AD) diagnosis from structural MRI (sMRI) increasingly relies on deep learning, yet existing models often provide limited insight into the anatomical evidence underlying their predictions. Recent vision–language approaches improve interpretability by generating diagnostic rationales, but global image–text alignment does not guarantee that anatomy-related statements are grounded in the corresponding brain regions, leading to statement–region attention drift. We propose RAR, a retrieval-augmented anatomy-aware reasoning framework that explicitly connects anatomical evidence, diagnostic reasoning, and prediction. RAR comprises three components: an Anatomy-Aware Visual Encoder that guides visual attention toward AD-critical regions using subject-specific anatomical priors; a Retrieval-Augmented Reasoning Module that enriches the current sMRI representation with compact rationale context from relevant training cases; and an Anatomy-Grounded Rationale Generator that aligns anatomy-related rationale tokens with their supporting brain regions through token-specific anatomical priors. Experiments on ADNI, NACC, OASIS-3, and the external AIBL cohort demonstrate competitive diagnostic performance, improved rationale quality, and a 20.6-point gain in token–region grounding Dice. These results support explicitly linking regional imaging evidence to diagnostic reasoning for more interpretable AD diagnosis. Code will be made publicly available.

Primary area:
interpretability and explainable AI
Keywords:
Alzheimer's disease, structural MRI, vision--language reasoning, anatomical grounding, retrieval-augmented reasoning, interpretable diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JQcGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

RAR combines three components:

  • Anatomy-aware visual encoding: FreeSurfer-derived regional masks supervise a ViT’s attention during training.
  • Retrieval augmentation: the current MRI retrieves four embeddings of training-case rationales, which are appended to its visual tokens.
  • Anatomy-grounded generation: anatomical text segments receive ROI labels, and the generator’s cross-attention is encouraged to match the corresponding image regions.

The rationale targets are synthetic, generated using Qwen from FreeSurfer-derived regional measurements and known diagnostic labels.

Critical strengths

The motivation identifies a relevant limitation in the existing models: anatomical statements in generated rationales may not be grounded in the corresponding brain regions. The proposed combination of anatomical supervision, retrieval, and rationale generation is interesting.

Critical weaknesses
  • Literature review:

    • AD-Reasoning is a withdrawn submission in ICLR 2026 and accepted to CVPR 2026 as EMAD. The paper should discuss and distinguish itself from EMAD, a closely related approach to sentence–evidence–anatomy grounding. EMAD’s contributions include the AD-MultiSense dataset, which generates rationales similar to RAR but with expert validation. It also performs region grounding by explicitly segmenting regions, such as the hippocampus, in voxel space. Regarding the novelty, there are works like:
      • REMEMBER (ACM MM 2025): combines brain-MRI/text alignment, similar-case retrieval, evidence integration, and diagnostic explanations.
      • Region-guided radiology report generation (Interactive and Explainable Region-Guided Radiology Report Generation, CVPR 2023) explicitly connects regional features with generated descriptions
      • Reg2RG (TMI 2025): applies region-guided referring and grounding to volumetric CT report generation.

    So currently, the idea of RAR looks more like a combination of these adapted to brain MRI for Alzheimer’s classification. What is included beyond existing retrieval-guided diagnosis and region-grounded report generation is unclear.

  • Insufficient validation of patient-specific diagnostic faithfulness:

    • The targets utilized is not convincing. ADNI, AIBL, NACC, and OASIS are all well-known datasets, but without any regional abnormality ground truth. The authors use Qwen to generate this rationale without doctors in the loop for validation. This is not truly reliable because agreement with these references may not reflect clinically correct, patient-specific reasoning about abnormalities. Additionally, Dice against anatomical masks establishes correspondence with the named structure, but does not establish that the structure is abnormal or that the generated finding is correct. Therefore, these results cannot be interpreted as validation of abnormality localization. The paper only evaluates region-text grounding rather than diagnostic faithfulness, given that the target is AD diagnosis. Importantly, Torok et al. (2018)’s analysis using the scores for identifying the regions of atrophy was done on a controlled setup with age-matched groups, but that is not the case with RAR, since it pools multiple datasets with cohort differences. Consequently, its deviations and derived severity descriptions may reflect differences in the reference population rather than disease-specific abnormalities.
    • The interpretability analysis is only limited to if the regions match the regions queried, but, it is still unsure if the right atropied regions are discovered, in other words, matching a region’s name to its anatomical location does not necessarily mean the generated statement correctly describes that patient’s abnormality, or that the highlighted region drives the diagnosis. Example: Consider: “The hippocampus shows severe bilateral atrophy.” and “The hippocampus is relatively preserved.” Both can receive high dice scores, but they do not establish whether the atrophy is truly present and if the diagnosis follows it. This does not convince me that RAR can be used for patient-specific diagnostic reasoning. Also, how does anatomical grounding change with and without the retrieved text augmentation? Its unclear if the grounding is enforced by the image or anatomical and diagnostic associations from the retrieved rationale embeddings.
    • The evaluation metrics for the report generation BLEU/ROUGE, are insufficient for clinical factuality. Rather, I would be more interested to see if the right regions are identified, along with the right volumes or descriptions. Since, the model has been trained to focus on very specific regions, I wouldn’t be surprised if the model identifies the hippocampus; rather, I would be interested if the model says that “The hippocampus (HC) also shows marked bilateral atrophy(left: 34%, right: 33%)”, etc. These distinctions and analysis of the right location, severity, abnormality direction, and numerical accuracy are crucial since the paper is motivated to do so.
  • Experiments: Since the datasets considered are imbalanced, metrics like the balanced accuracy should also be reported because ACC could be misleading. Table 2 and Table 3 reports performance on the internal test set and not on the external AIBL. Also, variability across training seeds is not reported. The experiments are limited to only 3-way classification. The details of the architecture, preprocessing pipeline, hyperparameter details, etc., are missing for reproducibility.

Questions for authors

Please address the weakness.

Other questions:

  • Are the target and non-target regions defined per subject for Anatomy-Guided Attention Regularization or chosen in general based on the class label? Rel-SA (CVPRW 2025) also uses regional bias through the attention mechanism. Can this be adapted and compared with Anatomy-Guided Attention Regularization?
  • Missing details: How exactly are attention maps converted into masks for Dice? How are diagnostic probabilities and accuracy computed?
Reviewer_46U1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents RAR, a framework for Alzheimer's disease diagnosis from structural MRI. It combines anatomy-aware visual encoding, retrieval of rationale embeddings from similar training cases, and token-level anatomical grounding.

The authors use FreeSurfer to obtain anatomical masks and Qwen2.5 to generate reference rationales from regional volumetric measurements. The method is evaluated on ADNI, NACC, OASIS-3, and the external AIBL cohort. The results show improved classification accuracy over several baselines, along with higher BLEU-4 and ROUGE-L scores. Adding anatomical grounding also increases the average Dice score from 0.487 to 0.693.

However, RAR still has a lower diagnostic AUC than PPAL on both internal and external test sets. More importantly, improved attention alignment does not necessarily mean that the generated explanations are clinically reliable. The paper lacks independent clinical validation of these explanations.

Critical strengths
  • The paper addresses an important challenge in interpretable medical AI by attempting to connect generated diagnostic statements with anatomically relevant MRI regions rather than relying solely on global image–text alignment.
  • The proposed integration of subject-specific anatomical priors, retrieval-augmented rationale representations, and token-level grounding offers a coherent framework for investigating statement–region attention drift.
  • The experimental evaluation covers multiple neuroimaging cohorts, including an independent external dataset, and provides component-level ablations to examine the contributions of anatomy-aware encoding, retrieval augmentation, and grounding supervision. Specifically, the reported improvement in average token–region Dice from 0.487 to 0.693 shows that explicit anatomical supervision can substantially improve spatial attention alignment under the proposed evaluation protocol.
Critical weaknesses

There are some weaknesses for which I need more clarification from the authors:

  • A major concern is that the proposed anatomical grounding mechanism is evaluated using essentially the same anatomical priors employed during training. In Section 3.4, token-specific grounding targets are constructed by projecting Qwen2.5-generated ROI relevance scores onto FreeSurfer anatomical masks, and the model is explicitly trained to align its cross-attention distributions with these targets through a KL-divergence objective.
  • However, the principal grounding evaluation in Section 4.5 measures Dice overlap between the resulting attention maps and FreeSurfer ROI masks. Consequently, the substantial improvement in grounding Dice may primarily reflect successful optimization toward predefined anatomical targets rather than independently validated clinical interpretability. For these reasons, although this result shows some improvements in spatial alignment, it does not show that the generated explanations faithfully reflect the model's diagnostic decision process or correspond to clinically meaningful evidence.
  • The diagnostic rationales used for training and evaluation are generated automatically from FreeSurfer-derived regional volume statistics using Qwen2.5-14B-Instruct, rather than independently annotated or validated by clinical experts. Furthermore, Qwen2.5 is also employed to assign token-to-ROI relevance scores, introducing dependence between the synthetic rationale-generation and anatomical grounding supervision processes. This design raises concerns about whether the framework learns genuine clinical reasoning or primarily reproduces structured anatomical descriptions derived from predefined measurements.
Questions for authors

I have some questions for the authors:

  • As the grounding objective explicitly supervises cross-attention using FreeSurfer-derived anatomical priors, how can the paper demonstrate that improved attention–ROI Dice reflects genuine clinical interpretability rather than alignment with the training targets? Could the authors provide independent grounding evaluations, such as expert-annotated evidence regions or causal perturbation analyses?

  • How could the paper explain the differences in diagnostic AUC between RAR and PPAL, particularly on the external AIBL cohort (0.7615 vs. 0.8891)? Does the multi-objective training procedure introduce a trade-off between rationale generation and diagnostic discrimination?

  • Could the paper report retrieval-label consistency and evaluate the effects of incorrect or perturbed retrieval results on diagnosis and rationale generation?

Reviewer_C1krOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RAR combines anatomy-guided sMRI encoding, retrieval of training-case rationale embeddings, and token-specific anatomical attention regularization for NC/MCI/AD diagnosis. Reference rationales are generated by Qwen2.5 from FreeSurfer-derived regional measurements, with diagnostic labels used during target construction. The evaluation pools ADNI, NACC, and OASIS-3 for training and internal testing and reserves AIBL for external testing. The principal interpretability result is improved agreement between rationale-token attention and anatomical masks.

Critical strengths
  1. The method directly addresses statement-region attention mismatch instead of relying only on global image-text similarity. Figure 3 and Table 4 report more concentrated ROI-token attention, with mean Dice increasing from 0.487 without anatomical grounding to 0.693 with it. Masks are used for supervision and evaluation, but are not supplied at inference. This is useful evidence that the learned spatial alignment transfers to held-out scans.

  2. The validation includes external AIBL testing and informative component ablations. Table 3 shows that retrieval improves ACC from 0.7263 to 0.7458 on top of anatomy-aware encoding and rationale supervision; adding grounding then reaches 0.7743. These results support the combined design, although grounding without retrieval does not improve ACC. Section 3.3 restricts the fixed retrieval memory to training cases and excludes the paired rationale during training. Supplementary Section B also excludes AIBL and validation/test cases from reference-threshold estimation.

Critical weaknesses
  1. The spatial-alignment evidence is stronger than the evidence for faithful diagnostic explanations. Table 4 measures agreement with FreeSurfer anatomy, but overlap with the correct region does not establish that a claim about its abnormality is correct or that the claimed evidence drives the generated diagnosis. The reference texts are produced by a measurement-and-LLM pipeline, and BLEU/ROUGE/METEOR measure agreement with that corpus. Supplementary Sections C and F disclose the class-conditioned prompts, potential template biases, numerical inaccuracies, and need for further expert evaluation. Claim-level or behavioral validation is still needed to assess the generated rationales. Controlled changes to relevant and control ROIs, or an equivalent faithfulness test, would address a different question from the existing component ablations.

A related attribution issue is that the role of retrieved diagnostic information remains unclear despite the demonstrated retrieval benefit. Section 4.5 analyzes visual versus retrieval attention. However, the visual-attention fraction of 0.9946 is close to the 0.9922 uniform-allocation reference arising from 512 visual tokens versus four retrieved embeddings. Attention mass alone therefore does not establish what information influences the output. Because diagnostic class enters rationale construction, a control that removes diagnostic conclusions from the source rationales and then rebuilds their embeddings would help distinguish anatomical context from diagnostic-label cues.

  1. Table 4's statistical reporting needs reconciliation. Supplementary Table 3 lists 314 internal-test cases. For EC, Table 4 gives 0.523 +/- 0.124 without grounding, 0.718 +/- 0.092 with grounding, and p=6.57e-4 from a two-sided paired t-test. If these are subject-level means and standard deviations for all 314 paired cases, the standard deviation of paired differences cannot exceed 0.216, implying |t| >= 15.99 before accounting for rounding (at least about 15.84 even allowing rounding to three decimals). This appears incompatible with the reported p-value under those assumptions. Please clarify the actual paired sample and aggregation. Independent-run uncertainty is also not reported, limiting interpretation of small diagnostic differences.

My weak-rejection recommendation reflects the gap between improved anatomical attention alignment and evidence that the generated diagnostic claims faithfully follow the image, together with the unresolved statistical reporting. The external test and existing ablations provide a useful foundation.

Questions for authors
  1. Can you provide controlled ROI interventions, or comparable claim-level validation, showing how corresponding statements and diagnostic probabilities respond relative to control regions?

  2. For Table 4, what are the actual paired sample sizes per ROI, aggregation steps, and Dice computation, including any thresholding? How are the EC summary statistics and paired-test p-value reconciled?

  3. Does retrieval retain its benefit when diagnostic conclusions are removed before rebuilding the training-memory embeddings? Please also report any available variation across independent training runs for the main diagnostic comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.