- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Acquisition-Aware Patch Aggregation for 3D Brain MRI Report Generation provides a simple and flexible way to encode multiple acquisitions and series into a fixed size, which is then used as input to a language model for report generation.
The paper highlights two main contributions:
- MR-Weave - A method for aggregating multi acquisition, multi series MRI images into a fixed size feature.
- Evaluation protocol that uses an LLM to score AI-generated reports against the ground truth along 5 criteria: pathology detection, faithfulness, disease labeling, localization, severity.
All models are evaluated on MR-RATE dataset. MR-WEAVE outperforms all competitors on its evaluation protocol.
Critical strengths
-
This paper highlights an important problem and formulates it very well. MRI produces a lot of information and effectively compressing that information for report generation is an important task. The idea of weighting each acquisition separately at the patch level make sense since different acquisitions will be useful at different anatomical locations.
-
The weighted fusion strategy is well motivated and the ablations illustrate its advantages.
Critical weaknesses
-
The current evaluation is not convincing in its current form. While parsing the evaluation into subsections is a useful idea, LLM-as-a-judge makes interpreting those results challenging. For example, in Table 1 MedGemma 1.5 (fine-tuned) receives an average score of 1.5 and MR-Weave scored 1.8. There does not seem to be a meaningful way to interpret what an average difference of 0.3 means in this case. Without more in-depth evaluation (e.g., expanded metrics, qualitative analysis, reader study), the paper has questionable impact.
-
There is a lack of discussion on previous work in this space (specifically strategies for incorporating multi-acquisition/multi-series volumes). How does your method compare? Can you directly test against other strategies? If not, why?
Questions for authors
- Why is faithfulness the metric that other models/settings (Table 1 and 2) occasionally do better on? Is that difference substantial?
- All scores are generally low, why is that? For example, the best localization score is about 1.4. What does that mean in practice? What do the distribution of scores look like?
- What is the difference between pathology detection and disease labeling?
Thank you for your careful review and constructive suggestions. We address your questions below.
1. Validation of the evaluation — Common §1
We address this question in Section 1 of our common response.
2. Faithfulness differences across models and ablations — Common §2
We address this question in Section 2 of our common response.
3. Meaning of the Overall gain, low scores, and score distributions — Common §3
Please refer to Section 3 of our common response.
4. Additional metrics — Common §4
We address this question in Section 4 of our common response.
5. Additional Qualitative Analysis
We address this concern through an independent radiologist assessment of actual generated reports (Common response §1), complemented by qualitative examples of successful and failed reports (Appendix A.10). The reader study provides expert assessment of report quality, while the examples illustrate correctly reported findings and remaining errors, including incorrect localization and unsupported findings.
6. Prior multi-acquisition methods and feasibility of direct comparison
NeuroVFM, which we include as a baseline, already supports multiple acquisitions and repeated series for volumetric brain MRI report generation. Although MedGemma supports multiple acquisitions and repeated series, our baseline uses a single series per contrast to follow the authors’ demo code, which processes one selected series from all available series. To assess the effect of using all series with MedGemma 1.5 rather than a single series per contrast, we conducted additional ablation experiments (Common response §5).
We have also expanded the related-work discussion in Appendix D to include Cervical-RG, SpineAgent, and GateSPINE, which generate MRI reports from multiple sequences or views in other anatomical regions. These are relevant approaches to multi-acquisition integration, although they are outside the scope of direct brain MRI reporting comparisons.
7. Pathology detection versus disease labeling
Detection asks whether reference abnormalities are recognized and summarizes this on a 0–5 scale, with missed MAJOR findings penalized more heavily than missed MINOR findings. Unlike Disease labeling, Detection gives credit for recognizing an abnormality even if it is named incorrectly; the naming error is assessed under disease labeling.
Disease labeling asks whether a recognized abnormality is assigned the correct diagnosis or disease category, including the specificity supported by the reference. For example, recognizing a mass can count toward detection, while naming it as a meningioma rather than simply a “mass lesion” matters for disease labeling.
Further details on the evaluation protocol and an illustrative example are provided in Appendix B.7, with the evaluation prompts and full scoring rubric in Appendix C.