acceptodds
Withdrawn from ICLR 2027

Towards Perception-Centric Multimodal LLM for Multi-Sequence Breast MRI Diagnosis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) have demonstrated remarkable promise in computer-assisted diagnosis, yet their clinical efficacy remains severely constrained in multi-sequence breast MRI diagnosis. We observe that existing models can achieve better diagnostic conclusions supplied with verified perceptual findings, yet their performance degrades drastically when operating directly on raw multi-sequence MR images, even under explicit clinical guidance. This discrepancy indicates that the primary bottleneck stems from visual perception. We attribute this deficiency to two primary obstacles, specifically clinically unguided perceptual exploration alongside weak multi-sequence perceptual capability. Current MLLMs fail to organize visual interpretation along established clinical perception logic, overlooking subtle yet diagnostically critical findings. Moreover, extracted perceptual cues are weakly grounded across MRI sequences, propagating perceptual errors into downstream diagnostic reasoning. To this end, we propose PCBreast, a perception-centric MLLM for multi-sequence breast MRI diagnosis. We construct MMBreast, a comprehensive multi-center, multi-sequence breast MRI dataset paired with perceptual Chain-of-Thought (CoT) trajectories, enabling the model to internalize clinically aligned perceptual exploration. Furthermore, we introduce Perception-Sensitive Policy Optimization (PSPO), which incorporates perception-grounded rubric reward and visual-sensitive credit assignment to incentivize faithful perceptual grounding and penalize spurious visual findings. Extensive experiments demonstrate that PCBreast consistently outperforms existing MLLMs, achieving impressive improvements in diagnostic accuracy. Our code and model will be publicly available at https://anonymous.4open.science/w/PCBreast-2D2A/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Sequence Breast MRI Diagnosis, Perception-Centric Multimodal LLM, PCBreast

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tP21OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the difficulty multimodal language models face in reliably extracting diagnostic findings from multi-sequence breast MRI and proposes PCBreast. The authors construct MMBreast, a four-center dataset, and use image–report alignment and structured perceptual CoT fine-tuning to guide sequential analysis of morphology, enhancement kinetics, and signal characteristics. They then introduce PSPO, combining diagnostic accuracy and clinical description consistency rewards with visually sensitive token weighting. Evaluation covers BI-RADS classification, perceptual descriptions, component ablations, and missing-sequence experiments.

The paper presents a useful task-specific system. However, its two central contributions, the methodological advance and the claimed perceptual mechanism, currently lack sufficiently strong evidence.

Critical strengths

1. A dataset and task setup with practical value. The dataset contains over 7,000 examinations from four centers, pairing multi-sequence images with clinical findings and perceptual reasoning trajectories. It provides a valuable foundation for studying breast MRI classification and explanation generation. This is the contribution most likely to have lasting impact, although its value depends on adequate documentation of data quality, splitting protocols, and accessibility.

2. Controlled experiments provide some support for the training pipeline. Beyond the substantial gains over existing models in Table 1, Figure 7 shows incremental improvements within the same framework. Adding accuracy rewards, the perceptual rubric, and visually sensitive weighting to SFT increases accuracy from approximately 47.1% to 51.9%. These ablations support the practical benefits of the combined approach rather than relying solely on comparisons across models.

Critical weaknesses

1. The large gains in the main table do not establish a methodological advantage. The core algorithm contributes much smaller gains, and its contribution over prior work remains unestablished.

PCBreast receives alignment training, CoT supervision, and reinforcement learning on MMBreast. However, the paper does not establish that the baselines in the main comparison received comparable task-specific training. This leaves a basic question unanswered: do the gains come from the proposed method or from specialized training for breast MRI? The paper emphasizes an 18.52-percentage-point accuracy improvement over the strongest general-purpose baseline. Yet Figure 7(b) shows that visually sensitive token weighting, the most distinctive algorithmic component, increases accuracy only from 50.6% to 51.9% and Macro-F1 from 33.0% to 33.3%. These small gains do not seem sufficient to establish the effectiveness of the proposed token-weighting mechanism.

The methodological novelty is limited and insufficiently substantiated. Internalizing clinical diagnostic pathways through CoT supervision already has directly related precedents, including CogAlign. ToR uses absolute differences in token log-probabilities with and without images to identify perceptual tokens and reweight reinforcement learning signals. PGPO uses KL divergence between image-conditioned and text-only predictive distributions to reshape token-level advantages. The introduction and related work do not adequately examine the limitations of similar approaches or explain how the proposed method advances them. The experiments also lack comparisons with these closely related methods.

Furthermore, all experiments are confined to the single MMBreast test set, without an independent external cohort or an explicitly held-out hospital. The evaluation is therefore unconvincing and does not establish transferability or generalization. The per-hospital results in Appendix D cannot substitute for cross-hospital generalization testing.

2. The claimed improvement in genuine visual perception is not directly validated.

The paper does not clearly define how MRI slices, volumes, and DCE temporal information are represented in the model input. Perceptual evaluation primarily measures agreement with reference reports, and the reward even treats findings absent from the reference report as hallucinations. These results show that outputs better match reference text, but do not establish that the model identifies lesion features more accurately. The experiments in the introduction also fail to establish inadequate perception as the primary source of diagnostic errors.

Questions for authors
  1. All experiments are conducted on MMBreast, and results reported separately for the four hospitals do not establish generalization to unseen data sources. Please add an external evaluation on at least one established public breast MRI dataset independent of MMBreast, comparing PCBreast with the main baselines under consistent input and evaluation protocols. Evaluate the existing models directly, without further training or tuning on the external test data, and report diagnostic or lesion-attribute recognition performance appropriate to the available annotations. Does PCBreast retain its advantage on this independent cohort?

  2. What MMBreast training and clinical supervision did the Table 1 baselines receive? If they did not receive this training, please justify the fairness of the main comparison. Please also conduct fair evaluations on several additional established MRI benchmarks to assess generalization.

  3. What evidence independent of the current report-matching rubric demonstrates improved recognition of actual lesion attributes? Among cases with diagnostic errors, how many contain expert-confirmed perceptual errors? How many diagnoses can be corrected by correcting those attributes?

4 comments
AuthorsOpenReview ↗

Response to Reviewer tP21 [Part-1/3]

We thank the reviewer for the careful assessment and constructive suggestions. We appreciate the recognition of the practical value of MMBreast for breast MRI classification and explanation generation, and of the support provided by our controlled ablations for the combined training approach.

Response to Weakness 1, Questions 1 & 2 (Fair training comparisons, methodological novelty, and external generalization remain unestablished.)

Fair training comparisons. We agree that comparisons with off-the-shelf models alone cannot isolate methodological gains from breast MRI adaptation. We therefore add Lingshu-SFT and Qwen3.5-SFT trained on MMBreast, alongside PCBreast, as shown below.

MethodParamsAccuracyMacro F1Weighted F1Hospital-1Hospital-2Hospital-3Hospital-4
Lingshu-SFT7B0.45020.27530.43560.43120.49440.44290.3179
Qwen3.5-SFT2B0.47110.30270.45460.42200.49250.60950.2384
PCBreast2B0.51890.33330.49660.47710.52250.65710.3444

PCBreast outperforms both task-adapted SFT baselines.

Methodological novelty. We acknowledge the precedents in clinical CoT supervision (CogAlign) and visual-dependent token optimization (ToR and PGPO). Our contributions lie in a multi-sequence breast MRI dataset and the designs for breast MRI perceptual enhancement, rather than these general techniques alone. The following table clarifies the differences in the designs described by these papers.

DimensionCogAlignToRPGPOPCBreast
Research taskGastrointestinal endoscopic diagnosisGeneral multimodal reasoning, including visual mathematicsMathematical, geometric, logical, and multidisciplinary visual reasoningMulti-sequence breast MRI interpretation and BI-RADS diagnosis
Target problemClinical cognition misalignment and background shortcutsInterdependence of perception and reasoning tokensUniform credit assignment diluting visually dependent learning signalsUnguided perceptual exploration and unreliable multi-sequence MRI findings
Dataset contributionClinical reasoning annotations for images from five public endoscopy datasetsUses existing Geo3K and ViRL-39KUses existing ViRL39KFour-hospital MRI resource paired with clinical findings, BI-RADS labels, and perceptual CoT
Chain-of-Thought DesignHierarchical endoscopic reasoning from anatomical context to morphology and fine-grained mucosal featuresGeneral multimodal CoT without a prescribed clinical structureGeneral multimodal CoT without a prescribed clinical structureBreast MRI perceptual CoT organizing morphology, enhancement kinetics, and multi-sequence signal characteristics
Reward designFormat, clinical keyword coverage, and diagnostic correctnessFinal-answer correctnessFinal-answer correctness and format complianceBI-RADS correctness and clinical-reference consistency of three MRI perceptual dimensions
Visual-dependency mechanismLesion-masking counterfactual images, rather than token-level dependency scoringAbsolute image-conditioned versus text-only token log-probability difference; entropy for reasoning tokensKL divergence between image-conditioned and text-only predictive distributionsSigned token log-probability difference; selects positive MRI-sensitive tokens
Credit assignmentGRPO with counterfactual samples and clinical rewardsJoint weighting of selected perception/reasoning tokens; zero weight outside selected sets in the stated objectiveThreshold-gated suppression/boosting with normalized advantage redistributionCapped weighting of selected positive-sensitivity tokens; base weights retained for other tokens, followed by normalization
  • Data contribution: MMBreast provides over 7,000 examinations and 70,000 sequences from four hospitals, with supervision for the breast MRI findings underlying BI-RADS decisions, rather than final labels alone.
  • Algorithmic contribution: Unlike CogAlign's keyword-based reward and lesion-masking intervention, PSPO evaluates clinical-reference consistency and assigns feedback at the token level. Unlike ToR and PGPO, PSPO additionally rewards the intermediate morphological, kinetic, and signal-intensity descriptions supporting breast MRI diagnosis. Its positive signed-sensitivity selection distinguishes tokens whose likelihood increases with MRI conditioning from tokens that merely change under image removal.
AuthorsOpenReview ↗

Response to Reviewer tP21 [Part-2/3]

External generalization. We add a cross-hospital OOD evaluation, training only on Hospitals 1–3 and holding out Hospital-4 from all training stages.

MethodInternVL3.5InternVL3.5InternVL3.5InternVL3.5Qwen3.5Qwen3.5Qwen3.5Qwen3.5Qwen3.5-SFTQwen3.5-GRPO
Params1B2B4B8B0.8B2B4B9B2B2B
Accuracy0.05300.21190.19210.16560.25830.18540.21190.25170.21190.2848
MethodHulu-MedHulu-MedHulu-MedHulu-MedLingshuMedGemmaHuatuoGPT-VisionClinFusionLingshu-SFTPCBreast
Params4B7B14B32B7B4B7B8B7B2B
Accuracy0.19210.19870.18540.21190.17880.19870.10600.18540.26490.3113

PCBreast achieves the highest accuracy among the compared methods. This provides evidence of generalization to an unseen hospital.

Response to Weakness 2 (Model input and the link between perceptual deficiencies and diagnostic errors)

Model input. For each MRI sequence/volume, we feed groups of three adjacent slices into the visual encoder and concatenate the resulting visual tokens across groups and sequences. All sequences are processed in this way, including each temporal phase of DCE MRI.

Link between perceptual deficiencies and diagnostic errors. To explore how access to more reliable perceptual information affects diagnostic predictions, we provide clinical findings from medical reports alongside the MRI images and examine changes in PCBreast's previously misclassified cases. With these additional findings, 48.45% of the original errors are corrected. Combining the updated predictions with the unchanged predictions for previously correct cases yields the findings-assisted results below.

EvaluationAccuracyMacro F1Weighted F1Hospital-1Hospital-2Hospital-3Hospital-4
PCBreast51.89%33.33%49.66%47.71%52.25%65.71%34.44%
Findings-assisted correction75.20%59.93%75.22%77.06%73.03%87.62%64.24%
Change+23.31+26.60+25.56+29.36+20.79+21.90+29.80

With findings-assisted correction, accuracy increases by 23.31 percentage points, both F1 metrics improve, and accuracy rises at all four hospitals. This recovery provides supporting evidence that more reliable perceptual information can help improve MRI-based breast cancer assessment.

AuthorsOpenReview ↗

Response to Reviewer tP21 [Part-3/3]

Response to Question 3 (Evidence for improved recognition of actual lesion attributes and diagnosis correction)

1. Evidence for improved recognition of actual lesion attributes. We additionally collect a breast MRI attribute-QA dataset with 100+ examinations and 900+ QA pairs for zero-shot evaluation. Questions cover lesion detection/localization, morphology and size, sequence-specific signals and enhancement kinetics, associated findings, and breast-level characteristics. Models receive MRI images and attribute questions; performance is measured by answer accuracy.

MethodParamsAccuracy
PCBreast2B54.83%
Qwen3.5-GRPO2B54.16%
Qwen3.5-SFT2B53.94%
Lingshu7B53.39%
InternVL3.58B48.72%
HuatuoGPT-Vision7B44.73%
Qwen3.59B38.85%
Hulu-Med7B34.85%

PCBreast achieves the highest zero-shot accuracy. This separate attribute-QA evaluation provides evidence beyond the current report-matching rubric, with modest additional gains over the task-adapted baselines.

2. Diagnostic correction. As reported in the findings-assisted correction table above (Response to Weakness 2), adding clinical findings alongside MRI images corrects 48.45% of PCBreast's original diagnostic errors. Combining these updated predictions with the unchanged predictions for previously correct cases raises accuracy from 51.89% to 75.20%.

The corrected cases span all five BI-RADS categories but are concentrated in the clinically challenging BI-RADS 4 category (63.25%), highlighting the value of reliable perceptual information for these difficult diagnostic assessments. BI-RADS 4 is particularly challenging because it encompasses a broad range of malignancy risk and includes both benign and malignant lesions with overlapping imaging features. The percentages below describe the distribution of all corrected cases.

GT BI-RADS CategoryShare of All Corrected Cases
13.85%
24.27%
317.95%
463.25%
510.68%
Total100.00%

The added hospital-held-out evaluation provides useful evidence of cross-center generalization. However, all results still rely on MMBreast’s task construction, annotations, and scoring protocol. The paper has therefore yet to establish whether the proposed perception optimization transfers to independently constructed breast MRI benchmarks. Please add an evaluation on a task-matched, independent public breast MRI benchmark, comparing PCBreast with SFT and GRPO under consistent input and scoring protocols, with no overlap with training data. This evaluation is needed to determine whether the reported gains depend on MMBreast-specific supervision and evaluation design.

Reviewer_XKJwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Authors find that existing models achieve better diagnostic conclusions supplied with verified perceptual findings, but their performance degrades drastically when operating directly on raw multi-sequence MR images. Authors attribute this deficiency to two primary obstacles, specifically clinically unguided perceptual exploration and weak multi-sequence perceptual capability. Based on these observations, authors proposed PCBreast, a perception-centric MLLM for multi-sequence breast MRI diagnosis.

Critical strengths
  1. The authors point out that the primary bottleneck resides in visual perception in multi-sequence breast MR diagnosis.
  2. The authors construct MMBreast, a comprehensive multi-center,multi-sequence breast MRI dataset paired with perceptual CoT trajectories. It contains over 70,000 sequences from 7,000 examinations across centers.
  3. The authors utilize this dataset to train their model with visual alignment, SFT, and RL stages. The Perception Sensitive Policy Optimization (PSPO) is proposed for RL stage.
Critical weaknesses
  1. The distribution of the dataset is not balanced. Will this imbalance affect the experimental results?
  2. The novelty is somewhat limited. The data pipeline and PSPO are mainly technical contributions and have become common solutions in recent work.
Questions for authors
  1. As the dataset has been listed as a core contribution of the paper, will the authors release this dataset in the future?
  2. Please provide a section to discuss the limitations.
  3. The analysis of the failure case is not provided.
5 comments
AuthorsOpenReview ↗

Response to Reviewer XKJw [Part-1/3]

We thank the reviewer for the thoughtful assessment and constructive suggestions. We appreciate the recognition of our identification of the visual perception bottleneck in multi-sequence breast MRI diagnosis and the contribution of MMBreast, our multi-center, multi-sequence dataset paired with perceptual CoT trajectories.

Response to Weakness 1 (Could dataset imbalance affect the results?)

We thank the reviewer for raising this concern. We distinguish modality-count differences arising from the counting convention from BI-RADS label imbalance.

Modality counts. The apparent modality imbalance partly reflects differences in sequence labeling and grouping conventions across hospitals. Pre-contrast T1 images are recorded separately at some hospitals, whereas others include them as the first acquisition of the DCE series. In the paper, Figure 2 reflects these hospital-specific labels, so counting only separately labeled T1 sequences understates the available T1 information. After harmonizing these conventions and counting the pre-contrast first DCE acquisitions as T1, the T1 count is 6,925, rather than 798. The following table compares the original hospital-recorded counts with the harmonized counts, without double-counting acquisitions.

ModalityOriginal hospital-recorded counts (Figure 2)Harmonized counts (pre-contrast DCE acquisitions counted as T1)
T17986,925
T26,7626,762
DWI16,54516,545
ADC6,1686,168
DCE44,98338,856
Total75,25675,256

DCE also contains multiple temporal acquisitions per examination, so a larger sequence count does not directly imply a corresponding imbalance in examination-level modality availability. This counting clarification does not imply that all modality counts are equal.

BI-RADS label imbalance. Unlike the counting issue above, class imbalance can bias training toward majority categories and affect the interpretation of aggregate evaluation metrics. The uneven label distribution reflects the composition of the collected hospital cohorts. We use resampling during training to mitigate this bias and report Accuracy, Macro F1, and Weighted F1 to assess performance from complementary perspectives.

Effect of unaddressed class imbalance. In the original training set, BI-RADS 4 accounts for 60.53% of cases. The Qwen3.5-SFT model trained without resampling predicts BI-RADS 4 for 98.41% of evaluated cases and produces no predictions for the other four categories. This pronounced majority-class prediction collapse is consistent with the concern that unaddressed imbalance can adversely affect learning. The following table shows the label and prediction distributions; GT denotes the evaluation ground truth.

Distribution / ModelBI-RADS 1BI-RADS 2BI-RADS 3BI-RADS 4BI-RADS 5Invalid
Original training labels1.27%2.01%20.39%60.53%15.80%-
Original testing labels2.79%4.18%37.05%28.49%27.49%-
Prediction of Qwen3.5-SFT (without resampling)0.00%0.00%0.00%98.41%0.00%1.59%
Prediction of Qwen3.5-SFT (with resampling)5.78%3.59%52.59%16.93%20.82%0.30%
AuthorsOpenReview ↗

Response to Reviewer XKJw [Part-2/3]

Mitigation through training resampling. With resampling, predictions span all five BI-RADS categories rather than collapsing to BI-RADS 4. Accuracy improves from 27.79% to 47.11%, Macro F1 from 0.0876 to 0.3027, and Weighted F1 from 0.1248 to 0.4546. The per-hospital accuracy results also improve at all four sites. These results support the usefulness of resampling in mitigating majority-class bias in SFT, rather than suggesting that imbalance has no effect.

All performance metrics below are reported on a 0–1 scale; Hospital-1 to Hospital-4 denote per-hospital accuracy.

MethodParamsAccuracyMacro F1Weighted F1Hospital-1Hospital-2Hospital-3Hospital-4
Qwen3.5-SFT (without resampling)2B0.27790.08760.12480.38530.26780.31900.1788
Qwen3.5-SFT (with resampling)2B0.47110.30270.45460.42200.49250.60950.2384

Response to Weakness 2 (The data pipeline and PSPO have limited novelty.)

Our contributions lie in data and algorithmic designs for perception-centric breast MRI diagnosis, rather than the conventional data-generation or SFT–RL pipeline:

  • Data contribution — MMBreast: We construct a four-center resource with over 7,000 examinations and 70,000 sequences, paired with clinical findings, BI-RADS labels, and perceptual CoT trajectories. The trajectories follow the morphological, kinetic, and signal-intensity assessments used by radiologists in breast MRI interpretation, providing supervision for the imaging evidence underlying BI-RADS decisions. The contribution is this clinically structured multi-sequence resource, not CoT synthesis alone. Reviewer tP21 likewise recognizes its practical value as a foundation for breast MRI classification and explanation generation.
  • Algorithmic contribution — PSPO: PSPO targets perceptual enhancement in multi-sequence breast MRI through Perception-Grounded Rubric Reward and Visual-Sensitive Credit Assignment. The former evaluates breast MRI findings against real clinical references; the latter directs more optimization feedback toward MRI-sensitive tokens. Together, they address the reliability of the perceptual descriptions supporting BI-RADS predictions, beyond final-label correctness. Reviewer YRfW also identifies visual-sensitive training as promising for addressing textual over-reliance, while our component ablations support the practical benefit of the combined design.

Response to Question 1 (Will MMBreast be released publicly?)

Yes, we will publicly release MMBreast upon acceptance of the paper.

AuthorsOpenReview ↗

Response to Reviewer XKJw [Part-3/3]

Response to Questions 2 & 3 (The paper lacks a limitations discussion and failure case analysis.)

We thank the reviewer for these related suggestions. We will add a dedicated limitations section and the following failure-case analysis to discuss where multi-sequence breast MRI perception remains unreliable.

Limitations. Long breast MRI examinations containing multiple DCE phases and DWI b-values, as shown in the following failure-case analysis. This creates a potential bottleneck in retaining focal lesion details and integrating enhancement and diffusion evidence.

Failure-case analysis. PCBreast incorrectly classified an examination with a single approximately 20 mm breast mass as BI-RADS 1, whereas the clinical reference assigned BI-RADS 5. The input and diagnostic outcomes are summarized below.

ItemCase details
Input sequences1 T1 sequence, 5 DCE phases, 7 DWI b-values, and 1 ADC sequence
GT BI-RADS5
Predicted BI-RADS1

Clinical reference report.

Findings: Neither nipple is retracted. The skin and subcutaneous tissues are unremarkable. A mass measuring approximately 20 mm in diameter is present in the upper outer quadrant of the left breast. It shows high signal intensity on DWI, indicating restricted diffusion, and a low ADC value. Following contrast administration, the solid components and septa show marked enhancement, with a type II time–signal intensity curve. The right breast parenchyma is unremarkable. No enlarged lymph nodes are seen in either axilla or the anterior mediastinum.

Impression: The finding in the upper outer quadrant of the left breast is suspicious for breast cancer: BI-RADS 5 (highly suggestive of malignancy, greater than 95% probability).

Model output.

<think>The bilateral breast parenchyma demonstrates scattered punctate enhancing foci, each measuring no more than 5 mm in diameter, which represent physiologic background parenchymal enhancement rather than true space-occupying mass lesions or suspicious non-mass enhancement. In the upper central region of the left breast, a cystic signal space-occupying area measuring approximately 13 x 17 mm is noted, demonstrating moderate wall enhancement on post-contrast sequences, consistent with benign fibrocystic changes. The nipples, overlying skin, and subcutaneous fat planes are normal, and no enlarged axillary lymph nodes are identified.When evaluating these findings, the scattered punctate foci do not demonstrate confluent mass-like morphology or ductal distribution, and the cystic area with wall enhancement reflects background fibrocystic change without suspicious solid enhancement. Because there is no focal suspicious enhancement or discrete dominant lesion requiring higher-category assessment, this does not meet criteria for BI-RADS 2 or 3, and is appropriately categorized as negative.</think>

The patient's BI-RADS category is 1.

The generated CoT describes a benign cystic finding rather than the mass documented in the clinical report, and omits the reported diffusion restriction, low ADC, and type II enhancement kinetics. The resulting negative assessment illustrates a failure to integrate the breast MRI findings supporting the reference diagnosis. The excessively long multi-sequence input may have contributed to this failure by making it harder to preserve diagnostically relevant lesion information.

Reviewer_YRfWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed PCBreast, a perception-centric multimodal LLM for breast MRI diagnosis. The authors argue that the main bottleneck of existing medical MLLMs is insufficient visual perception rather than a mere lack of medical knowledge. The authors construct the MMBreast dataset, which includes more than 7,000 examinations from 4 medical institutions, and build clinically guided CoT reasoning trajectories along three dimensions: morphology, dynamic enhancement kinetics, and signal intensity. Methodologically, the authors propose PSPO, which combines diagnostic accuracy and imaging perception faithfulness rewards and adjusts training weights according to each token's sensitivity to visual input, in order to reduce fabricated imaging signs. In experiments, PCBreast achieves a diagnostic accuracy of 51.89%, higher than all the other compared baselines.

Critical strengths
  • The data used in this paper is at a high quality level. It has been verified by medical experts, and the evaluation data includes multi-center evaluation, which demonstrates the stability of the model's performance.
  • Previous studies have shown that medical LLMs tend to rely more on text than on images. I think the visual-sensitive token training method in this paper seems promising for addressing this problem and is a good attempt.
  • The method can align the model with doctors' diagnostic thinking, improving model performance and enriching the decision path by gradually discovering phenomena present in the images.
Critical weaknesses
  1. The training process of the model is a common method in medical LLM training area. It injects understanding of MRI images into the model through a LLaVA-like training method, and then further improves performance through SFT and RL.
  2. The paper does not provide a careful analysis of SFT and RL on the most critical benchmark (MMBreast, the one that can relect the model's diagnosis accuracy). From my overall understanding, the Base Model (Qwen3.5-2B) scores 21.71% accuracy; while in Figure 7a, after CoT SFT the model reaches 47.1% accuracy, and in Table 1 the accuracy of PCBreast is reported as 51.89%. This indicates that most of the model's capability comes entirely from the SFT stage rather than the RL stage (PSPO, which is the main contribution of this paper).
  3. The Macro F1 of PCBreast in Table 1 is only 0.3333, which does not seem to be a satisfactory number.
Questions for authors

I understand that the training methods for multimodal LLMs are now fairly standardized, so I do not want the authors to add excessive descriptions of the training methods. Instead, I would prefer to see some detailed analysises of the model's output.

  1. I hope to know what the gains and differences are between PSPO and ordinary GRPO for the model. From the results I have observed so far, a large amount of the improvement comes from SFT, which is quite intuitive, because the model must first be able to perceive before it can reason. However, I hope to see more detailed analysis of PSPO's impact on the model's chain of thought, rather than just numerical results. An analysis of changes in reasoning patterns might be a good angle, or it could be analyzed from the perspective I mentioned in Strengths 2.
  2. Could you provide more information about the changes in the model's outputs after different training stages? I noticed that there is a case study in the appendix, but perhaps it deserves to be placed in the main text and should show in more detail the changes in model outputs brought about by different training stages.
5 comments
AuthorsOpenReview ↗

Response to Reviewer YRfW [Part-1/5]

We thank the reviewer for the thoughtful assessment and constructive suggestions. We appreciate the recognition of our multi-center dataset, the promise of visual-sensitive training in addressing textual over-reliance, and the value of aligning breast MRI interpretation with radiologists' diagnostic thinking.

Response to Weakness 1 (The training pipeline follows common practices in medical multimodal modeling.)

We clarify that our primary contributions do not lie in the conventional training process, but strictly in our Data and Algorithmic designs targeted at solving the visual perception bottleneck in multi-sequence breast MRI diagnosis:

  • Data Contribution (MMBreast): We construct a large-scale multi-center dataset (7,000+ examinations, 70,000+ sequences across 4 hospitals) paired with clinical perceptual CoT trajectories, enabling reliable perceptual exploration across complementary sequences before drawing definitive diagnostic conclusions.
  • Algorithmic Contribution (PSPO): We design PSPO to enforce reliable visual perception:
    • Visual-Sensitive Credit Assignment: Quantifies token-level visual sensitivity via multimodal vs. unimodal discrepancies, mathematically steering policy gradients to visual-dependent tokens.
    • Perception-Grounded Rubric Reward: Evaluates factual consistency across real clinical dimensions against verified reports, penalizing spurious radiological findings.

Response to Weakness 2 (Most diagnostic gains come from SFT, leaving PSPO's contribution unclear.)

We agree that SFT provides an essential foundation, but SFT and PSPO (RL) serve complementary roles in multi-sequence breast MRI diagnosis. SFT teaches clinically aligned perceptual exploration through supervised CoT trajectories covering morphology, enhancement kinetics, and signal intensity. Building on this foundation, PSPO enhances perception by combining Perception-Grounded Rubric Reward, which evaluates imaging descriptions against clinical references, with Visual-Sensitive Credit Assignment, which directs optimization feedback toward MRI-sensitive tokens. This perceptual enhancement aims to further improve diagnostic performance on difficult cases and generalization to unseen scenes.

To test whether this perceptual enhancement translates into better diagnosis on challenging, unseen data, we select Hospital-4, the most challenging center for both SFT and PCBreast in the original per-hospital evaluation, as the held-out OOD cohort. This setting evaluates both performance on a challenging cohort and generalization beyond the training hospitals. We train using only Hospitals 1–3 and evaluate on Hospital-4, excluding its samples from all training stages.

MethodQwen3.5-2B (Baseline)SFTPCBreast
Accuracy on held-out Hospital-40.18540.21190.3113

On this unseen hospital, PCBreast improves accuracy from 21.19% with SFT to 31.13%, an absolute gain of 9.94 percentage points (approximately 46.9% relative improvement). This substantial OOD gain over SFT supports the effectiveness of RL-stage perceptual enhancement for multi-sequence breast MRI diagnosis under cross-hospital distribution shift.

Response to Weakness 3 (PCBreast's Macro F1 remains low.)

We thank the reviewer for pointing this out. While an absolute Macro F1 of 0.3333 may appear modest in isolation, it represents a competitive breakthrough given the intrinsic clinical characteristics of the MMBreast benchmark, for the following main reasons:

Natural Clinical Imbalance (Macro vs. Weighted F1): As shown in Figure 2 in the paper, real-world clinical cohorts naturally exhibit a skewed distribution across BI-RADS 1 to 5 (e.g., BI-RADS 1 and 2 are significantly rarer than routine categories 3, 4, and 5). Because Macro F1 weights all classes equally, lower sensitivity on these rare minority classes pulls down the unweighted average. When accounting for clinical prevalence, PCBreast achieves a strong Weighted F1 of 0.4966 (better than the best baseline's 0.2454).

Substantial Relative Gain over SOTA (+101.4%): Under this exact same rigorous setting, existing general and specialized medical MLLMs struggle severely, with the strongest baseline reaching only 0.1655 Macro F1, as shown in Table 1 in the paper. Our compact 2B PCBreast more than doubles the best baseline's Macro F1 while using 16 × fewer parameters, clearly validating the effectiveness of our perception-centric framework.

AuthorsOpenReview ↗

Response to Reviewer YRfW [Part-2/5]

Response to Question 1 & 2 (How do reasoning and outputs differ across SFT, GRPO, PSPO?)

We agree that SFT establishes the foundation for breast MRI interpretation and accounts for a substantial portion of the diagnostic improvement. To examine what changes beyond this foundation, we compare SFT, ordinary GRPO, and PSPO at three levels: internal attention allocation, perceptual descriptions within the generated CoT, and final BI-RADS predictions.

1. Internal attention allocation. As shown in the text-attention distribution table below, mean text attention during generation of the combined CoT and final response decreases from 22.9% with SFT to 22.0% with ordinary GRPO and 21.4% with PSPO. Relative to GRPO, PSPO further reduces mean text attention by 0.6 percentage points (approximately 2.7%). The proportion in the low text-attention bins (0–10%) increases from 49.65% to 51.16%, while that in the high text-attention bins (50–100%) decreases from 15.99% to 15.24%. This additional shift is consistent with reduced textual reliance and PSPO's Visual-Sensitive Credit Assignment, directly addressing the concern raised in Strengths 2.

Method0-55-1010-1515-2020-2525-3030-3535-4040-4545-5050-5555-6060-6565-7070-7575-8080-8585-9090-9595-100Mean
SFT34.65%13.23%8.47%6.29%4.95%3.96%3.55%3.04%2.58%2.23%2.18%2.11%2.14%2.23%1.86%1.41%1.42%1.05%0.90%1.76%22.9
GRPO36.36%13.29%8.51%6.05%4.78%3.88%3.24%2.99%2.66%2.23%2.19%2.20%1.99%1.66%1.58%1.54%1.27%1.00%0.77%1.79%22.0
PCBreast37.01%14.15%8.49%6.21%4.88%3.71%3.06%2.85%2.38%2.04%1.70%1.93%1.79%1.96%1.72%1.41%1.19%0.91%0.85%1.78%21.4

2. Perceptual descriptions within the CoT. We evaluate the generated CoT descriptions against clinical reports across the morphological, kinetic, and signal-intensity dimensions used in clinical breast MRI interpretation. BERT. denotes BERTScore-F1, and LLM. denotes the normalized LLM-judge score. All scores are on a 0–1 scale; higher is better, and the best scores among these three models are bolded.

MethodOverall BERT.Overall LLM.Morphological BERT.Morphological LLM.Kinetic BERT.Kinetic LLM.Signal Intensity BERT.Signal Intensity LLM.
SFT0.88260.37620.87780.31860.88980.42390.88060.3850
GRPO0.88310.38030.87820.32120.88970.42270.88200.3995
PCBreast (PSPO)0.88390.38690.87900.32370.89060.43300.88240.4058

PSPO exceeds ordinary GRPO in both metrics across all three perceptual dimensions. Overall LLM-judge alignment increases from 0.3803 to 0.3869. These results indicate that ordinary GRPO's overall improvement does not necessarily extend to every breast MRI perceptual dimension, while PSPO improves clinical-reference consistency across all three. This supports PSPO's perception-enhancement objective beyond final-label optimization.

AuthorsOpenReview ↗

Response to Reviewer YRfW [Part-3/5]

3. Final BI-RADS predictions. We also report diagnostic performance to relate these intermediate changes to the final clinical output. All metrics below are on a 0–1 scale; Hospital-1 to Hospital-4 denote per-hospital accuracy.

MethodAccuracyMacro F1Weighted F1Hospital-1Hospital-2Hospital-3Hospital-4
SFT0.47110.30270.45460.42200.49250.60950.2384
GRPO0.49800.32180.47780.40370.51870.62380.3179
PCBreast (PSPO)0.51890.33330.49660.47710.52250.65710.3444

Compared with SFT, PSPO improves Accuracy from 47.11% to 51.89%, Macro F1 from 0.3027 to 0.3333, and Weighted F1 from 0.4546 to 0.4966. Compared with ordinary GRPO, PSPO further improves Accuracy from 49.80% to 51.89%, Macro F1 from 0.3218 to 0.3333, and Weighted F1 from 0.4778 to 0.4966.

4. Case study of stage-wise output changes. To complement the aggregated results, we present the complete responses of SFT, GRPO, and PCBreast (PSPO) to the same breast MRI examination, followed by the clinical reference and an analysis of their differences.

Case analysis. PCBreast correctly describes marked enhancement, restricted diffusion, low ADC, and type II kinetics and predicts BI-RADS 4, whereas SFT misses the suspicious findings (BI-RADS 1) and GRPO overestimates the category (BI-RADS 5). SFT incorrectly interprets the enhancement as physiologic background and denies restricted diffusion, while GRPO misreports lesion morphology, location, size, and ADC value and overstates the malignancy risk as exceeding 95%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.