acceptodds
Under review as a conference paper at ICLR 2027

ACQUISITION-AWARE PATCH AGGREGATION FOR 3D BRAIN MRI REPORT GENERATION

14 views

Abstract

A brain MRI examination provides rich diagnostic information through multiple contrasts and series, but at the cost of substantial volumetric data. Leveraging such volumetric information for radiology report generation poses a challenge, as a vision–language model must accommodate the extensive volumetric information within a limited number of visual tokens. Existing approaches typically sample a subset of slices or compress visual representations before language generation. While necessary for tractability, these strategies may overlook where clinically relevant evidence lies: sampling can omit images containing important findings, while compression can obscure fine-grained local information. To address this, we introduce MR-Weave, a 3D brain MRI report generation model that weaves complementary evidence across acquisitions into a unified representation within a fixed visual token budget. MR-Weave groups features from all acquisitions by depth and performs acquisition-aware patch aggregation, adaptively weighting acquisition features at each patch position. We further introduce axial geometry encoding, which augments the fused tokens with physical position and scale information to support location- and size-aware descriptions. We evaluate MR-Weave on 2,000 studies from the co-registered release of MR-RATE using an LLM-as-a-judge protocol across five clinical dimensions: detection, faithfulness, disease labeling, localization, and severity. MR-Weave achieves the highest Overall score among the compared methods, outperforming the strongest baseline by 19%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
brain mri, rrg

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_18gDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Acquisition-Aware Patch Aggregation for 3D Brain MRI Report Generation provides a simple and flexible way to encode multiple acquisitions and series into a fixed size, which is then used as input to a language model for report generation.

The paper highlights two main contributions:

  1. MR-Weave - A method for aggregating multi acquisition, multi series MRI images into a fixed size feature.
  2. Evaluation protocol that uses an LLM to score AI-generated reports against the ground truth along 5 criteria: pathology detection, faithfulness, disease labeling, localization, severity.

All models are evaluated on MR-RATE dataset. MR-WEAVE outperforms all competitors on its evaluation protocol.

Critical strengths
  1. This paper highlights an important problem and formulates it very well. MRI produces a lot of information and effectively compressing that information for report generation is an important task. The idea of weighting each acquisition separately at the patch level make sense since different acquisitions will be useful at different anatomical locations.

  2. The weighted fusion strategy is well motivated and the ablations illustrate its advantages.

Critical weaknesses
  1. The current evaluation is not convincing in its current form. While parsing the evaluation into subsections is a useful idea, LLM-as-a-judge makes interpreting those results challenging. For example, in Table 1 MedGemma 1.5 (fine-tuned) receives an average score of 1.5 and MR-Weave scored 1.8. There does not seem to be a meaningful way to interpret what an average difference of 0.3 means in this case. Without more in-depth evaluation (e.g., expanded metrics, qualitative analysis, reader study), the paper has questionable impact.

  2. There is a lack of discussion on previous work in this space (specifically strategies for incorporating multi-acquisition/multi-series volumes). How does your method compare? Can you directly test against other strategies? If not, why?

Questions for authors
  • Why is faithfulness the metric that other models/settings (Table 1 and 2) occasionally do better on? Is that difference substantial?
  • All scores are generally low, why is that? For example, the best localization score is about 1.4. What does that mean in practice? What do the distribution of scores look like?
  • What is the difference between pathology detection and disease labeling?
1 comment

Thank you for your careful review and constructive suggestions. We address your questions below.

1. Validation of the evaluation — Common §1

“While parsing the evaluation into subsections is a useful idea, LLM-as-a-judge makes interpreting those results challenging.”

“Without more in-depth evaluation (e.g., …, reader study), the paper has questionable impact.”

We address this question in Section 1 of our common response.

2. Faithfulness differences across models and ablations — Common §2

“Why is faithfulness the metric that other models/settings (Table 1 and 2) occasionally do better on? Is that difference substantial?”

We address this question in Section 2 of our common response.

3. Meaning of the Overall gain, low scores, and score distributions — Common §3

“For example, in Table 1 MedGemma 1.5 (fine-tuned) receives an average score of 1.5 and MR-Weave scored 1.8.”

“There does not seem to be a meaningful way to interpret what an average difference of 0.3 means in this case.”

“All scores are generally low, why is that? For example, the best localization score is about 1.4. What does that mean in practice? What do the distribution of scores look like?”

Please refer to Section 3 of our common response.

4. Additional metrics — Common §4

“Without more in-depth evaluation (e.g., expanded metrics, …), the paper has questionable impact.”

We address this question in Section 4 of our common response.

5. Additional Qualitative Analysis

“Without more in-depth evaluation (e.g., …, qualitative analysis, …), the paper has questionable impact.”

We address this concern through an independent radiologist assessment of actual generated reports (Common response §1), complemented by qualitative examples of successful and failed reports (Appendix A.10). The reader study provides expert assessment of report quality, while the examples illustrate correctly reported findings and remaining errors, including incorrect localization and unsupported findings.

6. Prior multi-acquisition methods and feasibility of direct comparison

“There is a lack of discussion on previous work in this space (specifically strategies for incorporating multi-acquisition/multi-series volumes). How does your method compare? Can you directly test against other strategies? If not, why?”

NeuroVFM, which we include as a baseline, already supports multiple acquisitions and repeated series for volumetric brain MRI report generation. Although MedGemma supports multiple acquisitions and repeated series, our baseline uses a single series per contrast to follow the authors’ demo code, which processes one selected series from all available series. To assess the effect of using all series with MedGemma 1.5 rather than a single series per contrast, we conducted additional ablation experiments (Common response §5).

We have also expanded the related-work discussion in Appendix D to include Cervical-RG, SpineAgent, and GateSPINE, which generate MRI reports from multiple sequences or views in other anatomical regions. These are relevant approaches to multi-acquisition integration, although they are outside the scope of direct brain MRI reporting comparisons.

7. Pathology detection versus disease labeling

“What is the difference between pathology detection and disease labeling?”

Detection asks whether reference abnormalities are recognized and summarizes this on a 0–5 scale, with missed MAJOR findings penalized more heavily than missed MINOR findings. Unlike Disease labeling, Detection gives credit for recognizing an abnormality even if it is named incorrectly; the naming error is assessed under disease labeling.

Disease labeling asks whether a recognized abnormality is assigned the correct diagnosis or disease category, including the specificity supported by the reference. For example, recognizing a mass can count toward detection, while naming it as a meningioma rather than simply a “mass lesion” matters for disease labeling.

Further details on the evaluation protocol and an illustrative example are provided in Appendix B.7, with the evaluation prompts and full scoring rubric in Appendix C.

Reviewer_TbByOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MR-Weave for brain MRI report generation from multiple contrasts and series. The method groups co-registered slice features into depth bins and learns patch-level weights to combine acquisitions within a fixed language-model visual-token budget. Axial geometry encoding adds physical position and scale information. On 2,000 MR-RATE test studies, MR-Weave improves the proposed five-axis evaluation score from 1.521 to 1.811 over fine-tuned MedGemma 1.5. The supplement provides conventional report metrics and controlled perturbation tests of the evaluator.

Critical strengths
  1. Combining acquisitions at corresponding spatial locations is a sensible response to the token-budget problem in MRI reporting. The method can use complementary contrasts and repeat series while retaining a spatial token grid. The comparison with a fine-tuned model using the same backbone provides a useful practical reference.

  2. The gains have support beyond the headline evaluator. MR-Weave improves conventional metrics and RadGraph F1, while the fusion ablations favor learned soft weighting over uniform averaging. The controlled report perturbations also provide a useful sanity check that the proposed evaluator responds to localization, severity, and omission errors.

Critical weaknesses
  1. The clinical interpretation of the 19% improvement depends on a newly designed Gemini-based rubric. The perturbation experiments are useful, but do not establish agreement with radiologists on actual model outputs. Coverage weighting also makes finding omissions affect several axes. Since faithfulness is slightly lower than for fine-tuned MedGemma (2.739 versus 2.825), the robustness of the overall ranking to this weighting deserves attention.

  2. The headline comparison combines several changes. Appendix B.5 gives MedGemma only the first decodable series per contrast, whereas MR-Weave incorporates repeat series. Table 2 supports soft weighting within the patch-aggregation family, but axial reslicing and geometry encoding are introduced together. The results therefore do not fully isolate the contributions of acquisition weighting, additional input information, and spatial encoding.

Questions for authors
  1. Is the ranking stable without coverage multiplication or under another clinical evaluator? Please also provide paired uncertainty for the Overall and faithfulness differences on the existing test cases.

  2. Is there a comparison using the same candidate acquisitions and axial preprocessing for all fusion methods? In particular, how much does geometry encoding add after reslicing alone?

3 comments

Thank you for your careful assessment and for identifying the comparisons that would help clarify our results. We have organized the responses around your comments below.

1. Clinical evaluation and agreement with a radiologist — Common §1

“The clinical interpretation of the 19% improvement depends on a newly designed Gemini-based rubric.”

“The perturbation experiments are useful, but do not establish agreement with radiologists on actual model outputs.”

Please see Common response §1 for the radiologist evaluation.

2. Faithfulness trade-off — Common §2

“Since faithfulness is slightly lower than for fine-tuned MedGemma (2.739 versus 2.825), the robustness of the overall ranking to this weighting deserves attention.”

Please see Common response §2 for the faithfulness analysis.

3. Ranking stability under another clinical evaluator? — Common §4

Is the ranking stable under another clinical evaluator?

Please see Common response §4 for ranking stability across different evaluators.

4. Ranking stability without coverage weighting

“Is the ranking stable without coverage multiplication?”

We examined whether the ranking changes when coverage weighting is removed. MR-Weave ranks first both with and without coverage. Removing coverage swaps only the fourth and fifth positions.

ModelRaw Loc.Raw DisLab.Raw Sev.Overall (no cov.)Cov. Loc.Cov. DisLab.Cov. Sev.Final Loc.Final DisLab.Final Sev.Overall (with cov.)
AutoRG-Brain (zero-shot)1.3190.7150.9940.9590.1700.0640.1630.5930.2560.4790.619
MedGemma 1.5 (zero-shot)1.0921.3700.8971.2310.1300.1210.1330.4580.5690.4380.852
MedGemma 1.5 (fine-tuned)2.1582.5591.3112.0840.2890.2780.2201.1181.3270.7671.521
NeuroVFM (zero-shot)0.9401.1000.7521.2990.1370.1850.1220.4390.6270.3731.028
NeuroVFM (fine-tuned)0.9551.2110.6201.2150.1250.1210.0860.4510.5660.2920.920
MR-Weave (Ours)2.3632.9571.9512.3980.3720.3660.3481.3751.7371.2231.811

Loc.: localization; DisLab.: disease labeling; Sev.: severity; cov.: coverage. Raw and final denote scores before and after coverage weighting. Final scores are computed per study as raw × coverage and then averaged; they are not the product of the displayed column means. Overall averages the five axis means, using raw or final scores for these three axes. Bold indicates the best.

The corresponding analysis appears in Appendix A.11, Table 16 of the revised manuscript.

5. Why omissions affect multiple evaluation axes

“Coverage weighting also makes finding omissions affect several axes.”

Detection and coverage both assess how much lesion-related information in the reference is reflected in the generated report, and omitting a finding can affect both detection and coverage by design. But they serve different roles.

Detection evaluates whether reference findings are recognized, with scores reflecting the clinical importance of missed findings through MAJOR/MINOR rubric bands.

Coverage is computed separately for localization, disease labeling, and severity. It measures the proportion of eligible reference findings for which the corresponding information is reported in the generated report. A reference finding is eligible only if it specifies that information. The raw score evaluates the correctness of the reported information on each axis; coverage evaluates its completeness.

Table 21 compares two reports to illustrate how detection scores depend on the clinical importance of missed findings, while coverage accounts for omitted information. Both reports miss one of seven reference findings, but missing a MAJOR finding yields a lower detection score than missing a MINOR finding.

ReportDetection┃Localization
ReferenceDetectedMissedScore┃ReferenceReportedCoverageLocalization accuracyRaw Loc.Final Loc.
A3 MAJOR + 4 MINOR2 MAJOR + 4 MINOR1 MAJOR2┃540.80Slight imprecision in one finding4.03.2
B3 MAJOR + 4 MINOR3 MAJOR + 3 MINOR1 MINOR4┃510.20All reported locations correct5.01.0

Under localization, “Reference” counts findings with a location specified in the reference, and “Reported” counts those for which the generated report supplies a location, whether correct or not. Raw Loc. measures the accuracy of reported locations, independently of MAJOR/MINOR tier. Final Loc. = Raw Loc. × Coverage.

Report B correctly reports a location for only one of the five applicable reference findings. It therefore receives a raw score of 5, but its coverage of 0.20 reduces the final score to 1.0. Report A reports most reference locations, with slight imprecision in one finding. Although its raw score is lower at 4, its coverage of 0.80 yields a higher final score of 3.2.

Thus, a higher raw localization score does not necessarily imply a higher final score. The opposite rankings of A and B under detection score and localization coverage further illustrate that the two measure distinct aspects of report quality.

This explanation and example are included in Appendix B.7 and Table 21 of the revised manuscript.

6. Paired uncertainty for Overall and faithfulness differences

“Please also provide paired uncertainty for the Overall and faithfulness differences on the existing test cases.”

We have added paired 95% bootstrap intervals for every difference in Table 1 (Appendix A.12, Tables 17–18; 5,000 study-level resamples over the same 2,000 studies). All paired 95% confidence intervals in Table 17 exclude zero. For the two axes you asked about:

MR-Weave − baselineOverallFaithfulness
MedGemma 1.5 (fine-tuned)+0.290 [+0.227, +0.353]−0.086 [−0.153, −0.018]
NeuroVFM (zero-shot)+0.781 [+0.711, +0.852]−0.133 [−0.210, −0.055]
NeuroVFM (fine-tuned)+0.891 [+0.829, +0.955]+0.102 [+0.038, +0.168]
MedGemma 1.5 (zero-shot)+0.959 [+0.893, +1.026]+0.677 [+0.605, +0.752]
AutoRG-Brain (zero-shot)+1.192 [+1.126, +1.262]+1.934 [+1.855, +2.015]

The two faithfulness deficits have paired 95% confidence intervals that exclude zero. The ORPO-refined variant (Appendix A.6) closes the observed faithfulness gap to fine-tuned MedGemma 1.5 (Table 18, reproduced below).

MR-Weave + ORPO − systemDet.Faith.DisLab.Loc.Sev.Overall
MedGemma 1.5 (fine-tuned)+0.366 [+0.278, +0.454]+0.037 [−0.030, +0.105]+0.350 [+0.268, +0.430]+0.237 [+0.171, +0.304]+0.359 [+0.287, +0.432]+0.270 [+0.209, +0.330]

Positive values favour the ORPO-refined model. Det.: pathology detection; Faith.: faithfulness; DisLab.: disease labeling; Loc.: localization; Sev.: severity.

7. Matched-input comparisons and the contribution of geometry encoding

“Appendix B.5 gives MedGemma only the first decodable series per contrast, whereas MR-Weave incorporates repeat series.”

“Is there a comparison using the same candidate acquisitions and axial preprocessing for all fusion methods? In particular, how much does geometry encoding add after reslicing alone?”

Although MedGemma supports multiple acquisitions and repeated series, our baseline uses a single series per contrast to follow the authors’ demo code, which selects and processes only one of the available series. To isolate the effects of using all series and adding geometry encoding, we conducted a matched-input ablation under a common training and evaluation protocol. Please see Common response §5 for the matched-input ablation separating series selection, fusion strategy, axial reslicing, and geometry encoding. Results appear in §4.3, Table 2, with implementation details in Appendix B.6 of the revised manuscript. We thank you for suggesting this experiment, which makes the contribution of each component considerably clearer

Reviewer_Y9JaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MR-Weave, a method for 3D brain MRI report generation that addresses the challenge of representing multiple MRI acquisitions within a limited visual token budget. The authors introduce Acquisition-Aware Patch Aggregation, which adaptively combines features from different MRI acquisitions at corresponding spatial locations. They also propose Axial Geometry Encoding to incorporate physical position and scale information into the visual representations. The method is built upon MedGemma 1.5 and evaluated on 2,000 test studies from the MR-RATE dataset. Using an LLM-as-a-Judge evaluation protocol covering pathology detection, faithfulness, disease labeling, localization, and severity, the proposed method achieves better overall performance than the compared baselines.

Critical strengths
  1. this paper is that it addresses a clear and practically important problem in medical image report generation. A 3D brain MRI examination usually contains multiple contrasts and acquisition series, which provide complementary information about different abnormalities. Traditional slice-sampling approaches may overlook important findings in unselected images, while compressing large amounts of visual information may lead to the loss of local details. The authors attempt to integrate evidence from multiple acquisitions under a fixed visual token budget. Compared with simply increasing the number of input slices, this approach focuses on making more effective use of limited visual representation capacity, which is meaningful for practical medical vision-language applications.

  2. the method takes the characteristics of medical imaging into account. Instead of directly concatenating features from different acquisitions, MR-Weave performs patch-level feature fusion based on spatial correspondence and learned acquisition weights. This allows the model to assign different levels of importance to different MRI contrasts at different spatial locations. In addition, the authors introduce axial position and physical scale encodings to help the model generate more accurate descriptions of lesion locations and sizes. Although these components are largely based on existing feature fusion and positional encoding techniques, their integration is reasonably motivated by the requirements of multi-acquisition MRI report generation.

  3. The experimental results also provide some support for the proposed approach. The authors evaluate the method on 2,000 test studies and compare it with MedGemma 1.5, NeuroVFM, and AutoRG-Brain. Under the five-dimensional clinical evaluation protocol, MR-Weave improves the Overall Score from 1.521 for the strongest baseline to 1.811, corresponding to a relative improvement of approximately 19%. Table 2 further compares different aggregation strategies and shows that patch-wise softmax aggregation performs better than simple averaging and hard selection. Axial Geometry Encoding also provides additional improvements. Furthermore, the attention visualizations in Figure 4 offer an intuitive illustration of how the model may utilize different MRI acquisitions.

Critical weaknesses
  1. the experimental evaluation relies heavily on LLM-as-a-Judge without independent assessment by clinical experts. The authors use Gemini 2.5 Flash to compare generated reports with reference reports, but this evaluation does not necessarily guarantee accurate judgments about medical findings, lesion locations, or disease severity. Although the paper evaluates the judge's sensitivity using controlled report perturbations, this is still different from an independent assessment by radiologists in realistic clinical settings. Moreover, the Overall Score of MR-Weave is only 1.811, which remains relatively low on the evaluation scale, and its Faithfulness Score is lower than that of fine-tuned MedGemma 1.5. Therefore, the current results are not sufficient to fully demonstrate a consistent improvement in the clinical accuracy of generated reports.

  2. the discussion of computational efficiency and practical applicability remains insufficient. The authors emphasize that MR-Weave integrates multiple MRI acquisitions within a fixed visual token budget. However, fixing the number of tokens provided to the language model does not necessarily mean that the computational cost of the entire system remains constant as the number of acquisitions increases. Since the model still needs to encode candidate slices from different acquisitions, visual feature extraction may become computationally expensive for studies containing many series. Furthermore, the experiments rely on co-registered MRI data, while real clinical examinations may contain missing contrasts, registration errors, or variations in acquisition parameters. The paper does not sufficiently investigate the influence of these factors, making its robustness in realistic deployment scenarios unclear.

  3. the evidence supporting the interpretability of the proposed method is not fully convincing. In Figure 4, the authors present attention visualizations for four cases and suggest that the model focuses on relevant MRI contrasts and pathological regions. However, attention weights do not necessarily represent the causal evidence responsible for the generated diagnostic conclusions. A small number of qualitative examples is not sufficient to demonstrate reliable lesion localization or cross-acquisition reasoning. I suggest including more quantitative evaluations, such as comparing the attended regions with ground-truth lesion annotations or examining how report generation changes when specific acquisitions or lesion regions are removed. Such experiments would provide stronger support for the interpretability claims.

Questions for authors

Could the authors provide an independent evaluation by radiologists, particularly regarding missed findings, incorrect diagnoses, and localization errors? In addition, how do the authors explain why MR-Weave achieves a lower Faithfulness Score than fine-tuned MedGemma 1.5?

How does MR-Weave perform when the number of input acquisitions varies, some MRI contrasts are missing, or spatial registration errors are introduced? Could the authors also report end-to-end inference time and GPU memory consumption to better demonstrate its practical efficiency?

3 comments

1. Radiologist evaluation and clinical validity — Common §1

“the experimental evaluation relies heavily on LLM-as-a-Judge without independent assessment by clinical experts.”

“Could the authors provide an independent evaluation by radiologists, particularly regarding missed findings, incorrect diagnoses, and localization errors?”

The radiologist evaluation and its results are described in Common response §1.

2. Faithfulness relative to fine-tuned MedGemma — Common §2

“In addition, how do the authors explain why MR-Weave achieves a lower Faithfulness Score than fine-tuned MedGemma 1.5?”

The normal/pathological breakdown and the ORPO results are discussed in Common response §2.

3. Low absolute scores and clinical interpretation — Common §3

“Moreover, the Overall Score of MR-Weave is only 1.811, which remains relatively low on the evaluation scale, and its Faithfulness Score is lower than that of fine-tuned MedGemma 1.5.”

“Therefore, the current results are not sufficient to fully demonstrate a consistent improvement in the clinical accuracy of generated reports.”

The interpretation of the low absolute scores and the Overall improvement is discussed in Common response §3.

4. End-to-end computation and memory as acquisitions increase

“However, fixing the number of tokens provided to the language model does not necessarily mean that the computational cost of the entire system remains constant as the number of acquisitions increases.”

“Since the model still needs to encode candidate slices from different acquisitions, visual feature extraction may become computationally expensive for studies containing many series.”

“Could the authors also report end-to-end inference time and GPU memory consumption to better demonstrate its practical efficiency?”

The visual-token budget limits the LLM input length, while image encoding still depends on the number of candidate frames. We measured end-to-end inference time and peak GPU memory on one NVIDIA B200, comparing MR-Weave with an uncompressed variant that passes every encoded candidate-frame grid to the LLM. Each row represents one study, averaged over two runs after a warm-up. MR-Weave uses 22,070 input tokens in every tested study.

PairsFramesUncomp. input tokensEnd-to-end time (s)Peak GPU memory (GiB)
113033,6806.93 → 6.69 (−3.5%)22.0 → 22.0 (0.0%)
4600154,94015.78 → 12.52 (−20.6%)41.0 → 38.1 (−7.0%)
6 (median)742191,57620.79 → 15.86 (−23.7%)48.8 → 38.1 (−21.8%)
7893230,53429.68 → 23.01 (−22.5%)57.1 → 38.4 (−32.6%)
101,122289,61633.94 → 24.28 (−28.5%)69.6 → 38.7 (−44.3%)
141,438371,14445.64 → 30.52 (−33.1%)87.0 → 39.1 (−55.1%)

Pairs: (contrast, series) pairs. Arrows: uncompressed → MR-Weave.

At the training-set median of six acquisitions, MR-Weave reduces inference time by 23.7% and peak memory by 21.8%. For the 14-acquisition study, the reductions reach 33.1% and 55.1%, respectively. Differences in volume size and slice count mean that timing need not increase monotonically across these individual studies.

5. Robustness to acquisition variability, missing contrasts, and registration errors

“Furthermore, the experiments rely on co-registered MRI data, while real clinical examinations may contain missing contrasts, registration errors, or variations in acquisition parameters.”

“The paper does not sufficiently investigate the influence of these factors, making its robustness in realistic deployment scenarios unclear.”

“How does MR-Weave perform when the number of input acquisitions varies, some MRI contrasts are missing, or spatial registration errors are introduced?”

To answer this, we broke down the Overall scores of the 2,000 test studies by these properties. MR-RATE is a clinical dataset, so this variation is already present: 813 studies (40.7%) lack at least one of the four contrasts, and the studies come from three vendors and 13 scanner models. MR-Weave can handle this variation because its aggregation does not assume a fixed set of inputs: a single scoring network shared across all frames scores each candidate at each patch position, and a softmax normalizes these scores over whichever candidates are present. Each axial bin therefore yields one fused token grid regardless of how many acquisitions a study contains, and a missing contrast needs no imputation or zero-filled input.

PropertyStratumnNeuroVFM (ft)MedGemma 1.5 (ft)MR-Weave
Contrasts present2220.9271.2451.605
37780.9581.6211.788
41,1870.8951.4711.842
(Contrast, series) pairs1–31181.0421.6751.763
4–58110.9331.5341.835
6–78290.8971.5371.819
≥82420.8931.3541.722
Field strength1.5 T1,4180.9461.5801.795
3 T5800.8501.3751.847
VendorSiemens1,1690.9281.6121.800
Philips6150.8461.2841.801
GE2161.0791.7211.894

Bold marks the highest score in each row. ft: fine-tuned.

MR-Weave scores highest in every stratum shown. These results suggest that MR-Weave remains effective on the incomplete and varied studies that make up this clinical dataset.

The corresponding analysis is reported in Appendix A.9, Table 14 of the revised manuscript.

6. How sensitive is MR-Weave to registration errors?

We trained and evaluated MR-Weave on the co-registered data released by MR-RATE, whose alignment was performed using ANTs rigid registration. For rigid alignment between brain MRI sequences, Lin et al. (2017) reported a residual landmark error of 1.55 ± 0.46 mm. Motivated by this scale, we applied additional translations of 2, 3, 5, 10, and 20 mm to one series per study in 300 test studies (150 normal and 150 pathological). Within each study, the selected series was shifted in a direction held fixed across magnitudes.

Added translationOverallChange from 0 mm95% CI
0 mm1.868——
2 mm1.805−0.063[−0.122, −0.009]
3 mm1.828−0.039[−0.103, +0.018]
5 mm1.850−0.017[−0.081, +0.048]
10 mm1.783−0.085[−0.173, +0.000]
20 mm1.791−0.076[−0.167, +0.013]

Overall is the mean of the five axis means, excluding non-applicable scores within each axis. The confidence intervals for changes from the unshifted input use 10,000 paired study-level bootstrap resamples.

Translation perturbations reduced mean Overall by 0.017–0.085, but the reduction did not increase monotonically with displacement magnitude. Taken together, these results indicate modest average performance degradation under the tested single-series translations. The corresponding analysis is reported in Appendix A.9, Table 15 of the revised manuscript.

7. Scope of attention interpretability and quantitative validation

“However, attention weights do not necessarily represent the causal evidence responsible for the generated diagnostic conclusions.”

“A small number of qualitative examples is not sufficient to demonstrate reliable lesion localization or cross-acquisition reasoning.”

“I suggest including more quantitative evaluations, such as comparing the attended regions with ground-truth lesion annotations or examining how report generation changes when specific acquisitions or lesion regions are removed.”

We agree that attention weights alone do not establish causal evidence for diagnostic conclusions or reliable lesion localization. Our intention is to use attention to inspect which contrasts and image regions the model attends to when generating each sentence, including in failure cases, rather than to use it as a lesion detector. We acknowledge that the original wording could imply stronger claims and have revised Section 4.5 to clarify this scope. Examples are provided in Figure 4 and Appendix A.10, Figure 7.

MR-RATE does not provide voxel-level lesion annotations, so we cannot quantify attention overlap with ground-truth lesions or perform lesion-specific removal on this test set. Following your suggestion, we instead conducted a contrast-removal intervention to test whether removing a highly attended contrast changes the generated sentence more often than removing a less-attended contrast.

The analysis covered 21,041 generated sentences of at least 40 characters from studies with at least two contrasts within the 2,000-study test set. For each sentence, we separately removed the contrast with the highest or lowest attention share and regenerated the report. We counted the original sentence as retained if the regenerated report contained a sentence with a normalized SequenceMatcher similarity of at least 0.9. This intervention tests whether attention identifies inputs that influence sentence generation; it does not establish that those inputs provide clinically correct evidence.

Removed contrastRetained, all sentences (n = 21,041)Retained, finding sentences (n = 3,832)
Lowest attention82.4%63.9%
Highest attention47.7%19.3%

Removing the highest-attention contrast resulted in substantially lower sentence retention than removing the lowest-attention contrast: 47.7% versus 82.4% across all sentences, and 19.3% versus 63.9% for finding sentences. These results suggest that contrasts receiving higher attention have a greater influence on the generated content.

We therefore revised RQ3 and Section 4.5 to frame attention as a tool for inspecting model behavior, including failure analysis, rather than as evidence of reliable lesion localization. The contrast-removal results are reported in Section 4.5, Table 5, with the full protocol and failure examples in Appendix A.10.

Comments on the paper

4 comments

Responses to Common Reviewer Questions

We thank all reviewers for their constructive feedback. We are encouraged that the reviewers recognized the importance of integrating multiple MRI acquisitions under a limited visual token budget (Y9Ja, TbBy, 18gD), found the spatially aligned, acquisition-aware fusion strategy well motivated (Y9Ja, TbBy, 18gD), and noted supporting evidence from the fusion ablations and complementary evaluation metrics (TbBy, 18gD).

Revisions in the manuscript are indicated by vertical bars in the right margin. The main updates are:

  • Radiologist evaluation: Independent assessment of generated reports and agreement with the LLM judge (§4.4, Table 4, Appendix A.8).
  • Faithfulness: Analysis of normal and pathological studies, and ORPO refinement (§4.4, Table 3, Appendix A.6).
  • Score interpretation: Full score distributions and analysis of zero-score rates (Appendix A.7, Figure 5).
  • Evaluation robustness: CRG evaluation, comparisons across four LLM judges, and repeated evaluations (Appendices A.2, A.4, and A.5).
  • Matched-input ablations: A 16-variant comparison separating fusion strategy, series selection, axial reslicing, and spatial encoding (§4.3, Table 2, Appendix B.6).
  • Typo correction: Corrected the reported values for NeuroVFM (zero-shot) in Table 1; the model ranking and main conclusions are unchanged.

Shared questions are answered below, and reviewer-specific questions are addressed in the corresponding threads.


1. Does the evaluation agree with a radiologist on actual model outputs? (Y9Ja, TbBy, 18gD)

We would first clarify that the LLM judge compares generated reports against reference reports already written by radiologists, rather than interpreting MRI images itself, so concerns about its reliability should be considered in the context of this more limited role. To address concerns, we conducted an independent reader study in which a radiologist, blinded to system identity, provided 350 responses (including assessments of individual findings, rather than a single rating per report) on generated reports from 10 studies spanning the score range. We mapped these responses to report-level scores and measured their agreement with the LLM judge using Kendall’s τ, which measures how consistently two evaluators rank reports; values closer to 1 indicate stronger agreement.

Evaluation axisKendall’s τ
Pathology detection0.89
Faithfulness0.60
Disease labeling0.85
Localization0.90
Severity0.83
Overall0.85

The LLM judge shows strong rank agreement with the radiologist across the evaluation axes, with τ = 0.85 for Overall. Even the lowest correlation, τ = 0.60 for faithfulness, is at the upper end of the inter-radiologist range reported in GREEN (0.41–0.60; Ostmeier et al., 2024, §4.1). For intuition, in a comparison without tied scores, τ = 0.60 corresponds to agreeing on the ordering of 80% of report pairs. This supports substantial agreement even on our least-correlated axis.

The evaluation of these 10 studies took approximately three hours; at this annotation rate, extending the same evaluation to all 2,000 test studies would require approximately 600 radiologist-hours. These results suggest that the LLM judge can help anticipate trends in radiologists’ assessments of model performance. Further details on the evaluation procedure and results are provided in §4.4, Table 4, and Appendix A.8 of the revised manuscript.

On the 10 evaluated studies, the radiologist assigned MR-Weave the highest Overall score among the compared base models:

ModelDetectionFaithfulnessDisease labelingLocalizationSeverityRadiologist Overall
AutoRG-Brain (zero-shot)1.251.400.900.981.551.21
NeuroVFM (fine-tuned)0.623.300.620.620.621.16
MedGemma 1.5 (fine-tuned)1.253.601.171.171.141.67
MR-Weave1.753.201.521.562.142.04

2. Why is faithfulness lower despite a higher Overall score, and can it be improved? (Y9Ja, TbBy, 18gD)

MethodFaithfulnessDetectionDisease labelingLocalizationSeverityOverall
AllNormalPathological
MedGemma 1.5 (fine-tuned)2.8253.5902.1971.5691.3271.1180.7671.521
MR-Weave2.7393.2862.2901.9811.7371.3751.2231.811
MR-Weave + ORPO2.8623.4282.3981.9351.6781.3541.1261.791

Our analysis shows that MR-Weave is more faithful than fine-tuned MedGemma on pathological studies, while its lower faithfulness on normal studies is associated with more frequent mentions of findings unsupported by the reference. Improving faithfulness on pathological studies requires adapting the report to the particular lesions present, a more demanding task than reproducing a normal-report template. Normal studies account for 45.1% of the test set (901/2,000), and a single generic normal template can avoid fabricated abnormalities on these cases without demonstrating lesion-specific reporting ability. MR-Weave’s higher faithfulness on pathological studies is therefore encouraging. Its greater tendency to report abnormalities may, however, also contribute to unsupported findings on normal studies; we regard this as a possible trade-off.

We found that the ORPO-refined model therefore offers an alternative for improving faithfulness, with some reduction in performance on the other axes while largely maintaining the Overall score. (Table 3 of the revised manuscript). Further details are provided in §4.4, Table 3, and Appendix A.6 of the revised manuscript.

References

Hong, Jiwoo, Noah Lee, and James Thorne. 2024. ORPO: Monolithic Preference Optimization without Reference Model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11170–11189.


3. What does the Overall improvement mean, given the low absolute scores? (Y9Ja, 18gD)

Low mean scores largely reflect frequent zero scores from missed reference findings across models, rather than uniformly poor reports. The table below summarizes the percentage of studies receiving a score of zero on each of the four finding-based axes (lower is better). MR-Weave has the lowest zero-score rate on every axis shown.

ModelDetectionDisease labelingLocalizationSeverity
AutoRG-Brain (zero-shot)62.0%82.4%63.7%69.5%
MedGemma 1.5 (zero-shot)68.7%70.9%70.2%74.0%
MedGemma 1.5 (fine-tuned)43.7%46.1%44.3%62.5%
NeuroVFM (zero-shot)69.6%71.0%71.9%76.5%
NeuroVFM (fine-tuned)73.1%74.2%73.8%81.7%
MR-Weave36.0%37.4%36.8%44.3%

Compared with fine-tuned MedGemma, MR-Weave reduces these zero-score rates by 7.7–18.2 percentage points. This helps explain what the Overall improvement from 1.521 to 1.811 (a 19.1% relative increase) captures: more reports receive credit for recognizing and describing reference findings, although substantial omissions remain. The full score distributions are provided in Appendix A.7, Figure 5 of the revised manuscript.


4. Do the improvements hold across alternative metrics and evaluators? (TbBy, 18gD)

Beyond the radiologist assessment, we examine whether the improvements hold under alternative metrics and different LLM judges.

Additional CRG evaluation. The original evaluation already included BLEU-4, METEOR, ROUGE-L, and RadGraph F1, with MR-Weave achieving the highest score on every metric (Appendix A.1). We have now added CRG evaluation of pathology presence across the dataset’s 37 MRI pathology categories (Appendix A.2). MR-Weave also achieves the highest CRG score among the evaluated models. Higher is better.

ModelCRG
MedGemma 1.5 (zero-shot)0.3610
MedGemma 1.5 (fine-tuned)0.4152
NeuroVFM (zero-shot)0.3769
NeuroVFM (fine-tuned)0.3883
MR-Weave0.4748

Different LLM judges. We also evaluated all six systems with Gemini 2.5 Flash, GPT-5, Claude Sonnet 4.6, and Qwen3.5 on the same 500 test studies. All four judges rank MR-Weave first and fine-tuned MedGemma second. The table below reports Overall scores.

ModelGemini 2.5 FlashGPT-5Claude Sonnet 4.6Qwen3.5 122B
AutoRG-Brain (zero-shot)0.5720.7430.4250.337
MedGemma 1.5 (zero-shot)0.8180.9310.7460.696
MedGemma 1.5 (fine-tuned)1.4701.5791.4151.365
NeuroVFM (zero-shot)0.9891.1060.9530.786
NeuroVFM (fine-tuned)0.8800.9990.8390.795
MR-Weave1.7811.8931.6991.646
Margin over second best+0.312+0.314+0.283+0.282

Further details are provided in Appendix A.4, Tables 10 and 11, of the revised manuscript.


5. How do the fusion strategies compare under matched inputs? (TbBy, 18gD)

We replaced Table 2 with a 16-variant matched-input ablation separating fusion strategy, series selection, axial reslicing, and spatial encoding. Given the cost of running all 16 variants at full scale (approximately 5,460 B200 GPU-hours for training and US$340 for LLM evaluation), we use a common reduced-scale protocol: one training epoch on 12,800 studies (15.7% of the training set) and evaluation on the first 500 test studies in subject-ID order with at least six series, enabling analysis of repeated-series inputs. Results and implementation details are provided in §4.3 and Appendix B.6, respectively.

Fusion strategySeriesAxial reslicingSpatial encodingDet.Faith.DisLab.Loc.Sev.Overall
Slice samplingFirst––1.3461.8641.1500.8190.7581.187
Slice samplingAll––1.2911.8941.0760.8020.7071.154
Slice samplingAll✓–1.2231.8601.0250.7840.7711.133
Image-channel concat.First––1.0252.0520.6340.5890.6340.987
Image-channel concat.First✓–1.1002.0250.8220.6860.6991.066
Image-channel concat.All✓–1.1791.9150.8170.7520.7451.082
Latent-channel concat.First––0.0221.9940.0170.0080.0010.408
Latent-channel concat.All✓–0.3871.9180.1790.2480.3370.614
Patch-wise hard selectionAll––1.2642.0020.8760.7090.7101.112
Patch-wise hard selectionAll✓–1.4181.9091.1120.8430.8311.223
Patch-wise meanAll––1.1082.0040.8030.6170.6961.046
Patch-wise meanAll✓–1.2242.0730.9710.7220.7251.143
Patch-wise softmaxAll––1.3262.0840.9860.8070.7651.194
Patch-wise softmaxAll–RAS1.1951.9860.9220.7640.7041.114
Patch-wise softmaxAll✓–1.3832.1581.0320.8690.8531.259
Patch-wise softmax (MR-Weave)All✓Axial1.5202.1981.2000.9870.8871.358

First: the first series of each contrast; All: every series.

  • Geometry encoding adds to reslicing alone. With the same acquisitions and axial reslicing, adding axial geometry encoding raises Overall from 1.259 to 1.358 and improves all five axes.
  • Reslicing alone also helps. Axial reslicing improves all three patch-wise strategies.
  • The fusion gain holds under identical inputs. With all series and axial reslicing, patch-wise softmax (1.259) outperforms the others. Giving slice sampling every series does not close the gap, so the advantage does not come from access to more series alone.
  • MR-Weave achieves the highest Overall score among all 16 variants (1.358).

Dear AC and Reviewers,

Thank you for your careful assessment. We apologize for the late responses and the limited time remaining for discussion, as completing the additional experiments and analyses required substantial time. We highlight three findings addressing the main concerns:

Evaluation reliability. A radiologist blinded to system identity provided 350 responses, including finding-level assessments, on generated reports from 10 studies. The resulting report-level scores showed Kendall’s τ = 0.85 agreement with the LLM judge for Overall. Separately, four LLM judges consistently rank MR-Weave first and fine-tuned MedGemma second on the same 500 studies (Common response §§1, 4).

Faithfulness. MR-Weave’s faithfulness deficit relative to fine-tuned MedGemma is confined to normal studies, while it scores higher on pathological studies. We found that ORPO offers an alternative for improving faithfulness, while largely preserving Overall performance (Common response §2).

Controlled comparisons. A 16-variant ablation under a common reduced-scale training and evaluation protocol separates series selection, fusion strategy, axial reslicing, and geometry encoding. MR-Weave achieves the highest Overall score, and adding geometry encoding after axial reslicing improves Overall (Common response §5).

We have revised the manuscript to incorporate this evidence and clarify the scope and limitations of our claims, with revisions marked by vertical bars in the right margin. We respectfully ask that these updates be considered during the final discussion.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.