acceptodds
Withdrawn from ICLR 2027

PMMG: Clinical Prior-Guided Multimodal Multi-Task Cartilage Defect Grading Model for Patellofemoral Osteoarthritis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Patellofemoral osteoarthritis (PFOA) is a prevalent degenerative joint disease and a major subtype of knee osteoarthritis, characterized by irreversible cartilage defects that often present as irregular, multi-focal lesions across the patella and femur. This complexity demands early, fine-grained assessment to guide personalized, stepwise interventions. Multi-sequence Magnetic Resonance Imaging (MRI) offers superior soft-tissue visualization, yet interpreting the complex cartilage damage requires extensive radiological expertise, which makes reads subjective and time-consuming. This often leads to diagnostic delays and missed opportunities for early treatment. Existing methods are limited in that cartilage grading remains binary, without providing separate multi-level assessments for patellar and femoral cartilages. To address this, we propose PMMG, a clinical Prior-guided Multimodal Multi-task cartilage defect Grading model. Guided by clinical priors, it integrates multimodal information via multi-sequence collaborative learning with a level-wise attention. Additionally, a Cartilage-Focused Diffusion data augmentation strategy mitigates inherent class imbalance through synthetic data generation. The framework jointly performs fine-grained grading for both patellar and femoral cartilage. We construct the first large-scale multi-center multimodal PFOA dataset comprising 1,173 individuals. Extensive experiments demonstrate that PMMG (Accuracy: 0.739; AUC: 0.891) significantly outperforms state-of-the-art methods, achieves grading accuracy approaching that of senior radiologists (Accuracy: 0.767) while substantially improving the diagnostic performance of junior radiologists when used as an assistive tool, and achieves promising performance on an independent external test cohort (Accuracy: 0.705). This unified framework provides a comprehensive clinical assessment, enabling earlier and more precise stepwise interventions. Code is available at https://anonymous.4open.science/r/PMMG-2C33.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal, Multi-Task, Cartilage Defect Grading, Patellofemoral Osteoarthritis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fUMxOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PMMG for jointly grading patellar and femoral cartilage defects in patellofemoral osteoarthritis. It integrates five MRI sequences, radiology reports, grading-criteria prompts, and clinical variables through two branches with slice attention, sequence attention, and modality-level attention. The design incorporates clinical assumptions about sparse lesions, a primary diagnostic sequence, and image-led multimodal fusion. Cartilage-focused diffusion augmentation and an entropy-dependent weighted focal loss address class imbalance. Although annotations use grades 0–4, the experiments merge grades 0 and 1, yielding two four-class tasks. The study includes 1,173 patients from two centers, with 90 internal-test and 269 external-test patients. Reported mean accuracy/AUC is 0.739/0.891 internally and 0.705/0.895 externally. Validation includes baseline comparisons, component ablations, radiologist comparisons, and an assistance study.

Critical strengths
  1. A useful clinical task with a coherent modeling rationale. Separately grading patellar and femoral cartilage preserves information that a single joint-level prediction would obscure. The hierarchical design is motivated by sparse lesions and complementary MRI sequences, giving the framework a clear connection to the assessment task.

  2. Validation extends beyond an internal model comparison. The independently collected 269-patient external cohort, acquired at another center with a different scanner, provides meaningful evidence about transfer. Radiologist comparisons and an assistance evaluation also make the study relevant to its intended clinical workflow, although their interpretation requires the clarifications below.

Critical weaknesses
  1. The comparisons do not adequately isolate the proposed methodological contribution. Appendix A.3 restricts KGNet, a multi-view method and the strongest reported competitor, to sagittal images, while PMMG receives five MRI sequences plus reports and clinical variables. CoPAS also loses parts of its original fusion mechanism. The descriptions do not establish an equally informed multimodal competitor, so the reported gains conflate architecture with input availability and baseline adaptation. The ablations do not resolve this: all attention levels yield 70.6% mean accuracy in Table 3, whereas Table 9 ends at the full model’s 73.9%, without specifying how diffusion and loss weighting are controlled. Furthermore, diffusion raises accuracy by 1.6 percentage points while mean AUC decreases from 87.2 to 86.8, with no matched resampling comparison or explicit validation of synthetic grades. Consequently, the evidence for the distinct benefits of the proposed attention and augmentation remains incomplete.

  2. The claims of robust clinical grading and radiologist-level performance exceed the current evidence. External femoral accuracy falls from 81.1% to 66.2%; the merged grade-0/1 class contains 193/269 external patients, giving an always-majority prediction 71.7% accuracy. High AUC is encouraging, but it does not establish reliable four-class grading at the reported operating point. Figure 6 also shows only 2/9 correct predictions for internal femoral grade 3. The reader comparison has unresolved discrepancies: the internal senior reader’s patellar/femoral accuracies are 74.4%/78.9% in Table 2, but exact agreement with the reference is 68.9%/72.2% in Table 6. Reader independence from the reference annotators and the ordering, blinding, and washout of assisted readings are not specified. These issues directly affect the interpretation of the claimed clinical equivalence and assistance benefit.

Questions for authors
  1. Can the authors provide a comparison with a strong fusion baseline given the same MRI sequences, reports, clinical variables, preprocessing, and augmentation, with uncertainty for the paired performance difference? Please also specify which diffusion and loss components are active in each row of Tables 3 and 9.

  2. Can the authors reconcile the senior-reader accuracy/exact-agreement discrepancies between Tables 2 and 6, and the apparent junior-reader identity swaps? Were the evaluated readers independent of the reference-standard annotators? For the assistance study, what were the reading order, blinding and washout arrangements, and the paired uncertainty or significance estimates?

  3. How are the paired patellar/femoral grades assigned to the 184 synthetic subjects? Are grades inherited from source subjects or changed through conditioning, and how is consistency with cartilage defect depth, accompanying reports, and clinical variables checked? Does augmentation retain its benefit against resampling the same source subjects to an equivalent training-set size?

  4. Can the authorsthe authors provide external confusion matrices, per-grade recall, and model ordinal agreement, and explain the femoral accuracy decline relative to the 71.7% majority-class reference? How was the final prediction rule selected on validation data? These results would help establish whether the system reliably distinguishes clinically important defect grades across centers.

Reviewer_2YBYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PMMG comprehensively grades cartilage defects in the patella and femur by utilizing multiple MRI sequences, text from radiology reports, and clinical variables. The architecture employs a clinically meaningful grouping of sequences and a level-specific attention mechanism, while incorporating cartilage-focused data augmentation and a class-weighted loss function. The dataset consists of information from 1,173 patients collected across two centers, along with an independent external cohort of 269 patients. For the classification task, grades 0 and 1 were combined. The reported mean accuracy is 73.9% for internal data and 70.5% for external data. Furthermore, this study compares the model’s results with the interpretations provided by radiologists and evaluates the efficacy of model-assisted diagnostic readings.

Critical strengths
  1. Evaluating both the patellar and femoral cartilage simultaneously is a meaningful application, and the use of an independent external cohort of 269 patients represents a significant strength. Appendix Table 7 provides bootstrap confidence intervals, which are useful for assessing the uncertainty of the reported performance, rather than relying solely on point estimates.

  2. The clinical sequence construction and hierarchical fusion provide a sound inductive bias for multimodal grading. Analyses of the individual components and attention levels indicate that the fusion process accounts for a substantial portion of the observed performance gains, while the reader-study addresses real-world use cases that extend beyond standalone classification.

Critical weaknesses
  1. The comparison results fail to adequately distinguish between the proposed fusion method and the specific information provided as input to the models. While Appendix A.3 describes how inputs are adapted for the comparison models, PMMG utilizes MRI scans, report text, and clinical variables; therefore, the results alone are insufficient to demonstrate the specific advantages of an architecture designed to leverage clinical information. Section 4.1 explicitly states that the grading of cartilage and joint degeneration was removed from the reports, which is an important safeguard, but the provenance, timing, and relationship of the remaining descriptive content to the reference grades are not sufficiently clarified, making it difficult to interpret the specific contribution of the image-based component. Employing simple fusion baselines with matched inputs, alongside image, only and text/clinical, only control models, would help disentangle architectural advantages from the potential influence of auxiliary label-proximal information.

  2. There is a significant internal inconsistency in the results regarding the human readers. Table 2 reports an accuracy of 74.4% for the senior reader in grading the patella and 78.9% for the femur, whereas Appendix Table 6 reports exact agreement rates of 68.9% and 72.2%, respectively, between the senior reader and the reference standard. Given that the same cases, labels, and reference standards were used, these results should be consistent. Furthermore, while Table 5 labels the internal cohort as the "test cohort," the total count of 904 subjects matches the size of the entire internal cohort described in Section 4.1. This paper requires a clear mapping between cohorts, reading sessions, and grading definitions before any claims regarding reader performance, relative to a senior reader, or the effectiveness of reader-support tools can be evaluated. The confidence intervals in the current model fail to account for these discrepancies or to demonstrate the statistical significance of any improvements between readers.

Questions for authors
  1. How much of the gain comes from the proposed fusion architecture rather than additional clinical information?

  2. Can the internal reader-comparison results and cohort denominators be reconciled?

  3. Does the assistance study establish a paired improvement beyond reading-order effects and sampling uncertainty?

Reviewer_EvftOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PMMG for gine-grained patellar and femoral cartilage defect grading. The framework is evaluated on 1173 patient cohort.

Critical strengths

The clinical evaluation appears somewhat strong. The authors describe that they used cohort (based on multiple centers), they also used external test cohort, and comparied with radiologists. And based on this they paper suggests that model help radiologists grading accuracy, which I find interesting.

The task formulation is second point, like, instead of the PFOA binary classification, the paper formulates that patellar and femoral cartilage can exhibit different severity even in same patient.

Critical weaknesses
  • the comparison with the closest multi-view prior work isolates it technically. In the cited KGNet paper (TMI), the method represents spatially corresponding local regions for multiple MRI views, and authors state this that they implemented it only for saggital MR images. And the PMMG used additional information while having the similar network design, so does the improvment come from the architecture, or additional information.

  • The clinical 'stepwise treatment' is not sufficiently established by the presented experiments, I feel Fig. 1 associates grades with treatment cateogires bbyt the experiments evaluate cartilage grading rather than treatment selection or outcoumes. Or atleast, I could not identify a mapping from these predicted grades to the categories shown in Fig. 1.

minor: the paper can benefit from a proof read, inconcistency in the use of acronyms, some terms capitalize each word when inroducing the acronym, whereas others use lowercase wording, - Decision Curve Analysis (DCA) , confidence intervals (CI), internal test cohort (ITC), external test cohort (ETC).

Questions for authors
  • first to understand how the performance change without radiology reports, and what information do those reports contain?
  • also, is there any way that some of the terms from multi-center, multi-model, multi-modal, multi-task, multi-joint, multi-sequence, multi-focal, multi-level, can be presented alternatively,the abstract alone uses it quite a lot of times.
  • next is related to the above, the abstract may sound very convincing to a person familiar with technical and clinical terms, point being, the paper can leverage improved writing for a bit generalized readers
  • a bit clarity on the readers and radiologists, and their indpendence for reference standards?
  • some ordinal metrics are reported for readers but nor for proposed method?
  • claims use term 'significantly' (e.g., in Table 1), for gains already similar to improvments and in some areas,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.