- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents KLEAR, a model for predicting one-year T cell-mediated rejection (TCMR) and antibody-mediated rejection (ABMR) before kidney transplantation. The inputs are donor and recipient clinical data, HLA mismatch counts, and molecular mismatch features (PCA). The model has four stages: a VAE and a GMM generate extra rejection cases until the classes are balanced, XGBoost leaf indices are added as features, and a small neural network is trained with focal loss. The paper describes a 2,013-patient Mayo Clinic cohort, but every reported model uses the high-resolution HLA subset of 319 patients, with 22 TCMR and 12 ABMR cases. KLEAR reports a cross-validated AUPRC of 0.35 (TCMR) and 0.30 (ABMR), against prevalences of 0.07 and 0.04. A prospective cohort of 82 patients, with 4 TCMR and 3 ABMR cases, is also reported.
Critical strengths
- The clinical question is important. Predicting rejection risk before transplant could help with organ acceptance and treatment choices, and most existing scores (e.g., iBox) use data from after the transplant.
- The paper builds separate feature sets for TCMR and ABMR based on the biology of each (Section 3.2, Tables 7-8), and the appendix documents the features in detail.
- The authors include a later prospective cohort (Section 7) rather than stopping at cross-validation, and they report the confusion matrix (Table 4).
- AUPRC is reported next to the random baseline at each prevalence (Tables 1, 3, 4), which makes the numbers easy to read. Calibration (Figure 7) and decision curves (Figure 2) are also included.
- The abstract and ethics statement admit that the prospective cohort is too small to support deployment, and that race-based predictors need subgroup evaluation.
Critical weaknesses
-
There are too few rejection cases to support the claims. Development has 22 TCMR and 12 ABMR cases, and the prospective cohort has 4 and 3. The confidence intervals in Table 1 overlap heavily across methods (TCMR: KLEAR 0.35 [0.18-0.54] vs. logistic regression 0.26 [0.11-0.45]), and both ROC-AUC intervals for KLEAR include 0.5. There are no paired statistical tests, and Section 5.1 itself says these should be added in future revisions. The data do not show that KLEAR beats logistic regression, or that the E3 features beat clinical data alone.
-
The main hypothesis is never tested. The paper argues that high-resolution HLA typing adds value over standard typing, but the imputed (n = 1,263) and low-resolution (n = 1,340) cohorts appear in no result. Section 5.2 is titled "Resolution-dependent performance gain", yet Table 2 compares only feature subsets. Has the 2,013-patient cohort described in the paper used in any of the
-
The evaluation may leak information. Algorithm 1 fits PCA on the full dataset before cross-validation (step 3), which contradicts the reproducibility statement that feature construction is done within each fold. As described, Optuna picks hyperparameters on the same 5-fold cross-validation, with no nested loop, and early stopping uses the evaluation fold (step 20). The data used to fit the calibration is not stated. With about 20 positive cases, each of these can make the results look better than they are. If the reported numbers come from a separate held-out set, the paper should say so.
-
The prospective results are weak. AUPRC is 0.075 for TCMR and 0.077 for ABMR, only slightly above the random baseline (0.049 and 0.037, Table 4). The model catches 3 of 4 TCMR cases and all 3 ABMR cases, but it does so by also flagging 32 of 78 patients without TCMR and 48 of 79 without ABMR. The introduction still claims the model could "revolutionize donor-recipient matching", and Section 7.2 calls it "clinically superior".
-
The four stages are never ablated. The paper does not test KLEAR without oversampling, with SMOTE or class weights instead, without leaf features, or with cross-entropy instead of focal loss. Using focal loss on already balanced data corrects for imbalance twice. It is unclear which stages matter.
-
The ML novelty is limited. Feeding tree leaf indices into another model is the well-known GBDT+LR approach (He et al., 2014), which is not cited, and VAE or GMM oversampling is standard. The baselines are also weak for small tabular data (LightGBM, CatBoost or TabPFN).
-
Some results look unstable. The best TCMR setup in Table 2 has the highest AUPRC but a below-chance ROC-AUC of 0.46. XGBoost in Table 1 shows the same pattern (AUPRC 0.26, ROC-AUC 0.43). This suggests that a few positive cases rank at the top while the rest rank low, so the AUPRC may not hold on a different split. The same KLEAR setup also drops from AUPRC 0.35 on 319 patients (Table 1) to 0.264 on 285 (Table 2), and the paper does not explain why.
-
I am not convinced by the interpretation of the site analysis (Table 3). AUPRC rises with prevalence, so Florida's higher AUPRC (prevalence 0.12 vs. 0.04) is expected and does not show model robustness. Rochester contributes 443 patients to the 2,013 total but appears in none of the high-resolution analyses.
-
The top two TCMR predictors are recipient race and donor race (Figure 8a), larger than any E3 feature. This weakens the claim that molecular features are what drive the predictions. Oversampling does not address racial fairness, and there is no subgroup analysis.
-
Several method details are missing or inconsistent. The E3 framework, which produces the core features, is never cited, and how it computes them is not explained. The abstract mentions a Bayesian GMM, but Section 4.3 uses an ordinary GMM. The focal loss in Figure 1 has a (1-α) weight that Eq. (6) lacks.
Questions for authors
- Was hyperparameter tuning nested inside the cross-validation? Was PCA refit inside each fold, and what data was the calibration fitted on?
- What are the results on the imputed and low-resolution cohorts? Can you compare high-resolution, imputed and low-resolution typing on the same patients, as Section 5.2 suggests?
- Can you report paired comparisons of KLEAR against logistic regression and against clinical data alone, and show how stable the rankings are over repeated cross-validation with different seeds?
- How does performance change without oversampling, without leaf features, or without focal loss? How do LightGBM, CatBoost and TabPFN compare?
- What is the prospective ROC-AUC, and how did the bootstrap handle resamples with zero or one positive case?
- How does the TCMR model perform without race as an input, and within racial subgroups?
- What is the E3 framework, and is it publicly available?
- Why does the best TCMR setup in Table 2 have ROC-AUC 0.46 but the highest AUPRC?
2 comments
Response to Reviewer 5zoo - Part II
Q5: "What is the prospective ROC-AUC, and how did the bootstrap handle resamples with zero or one positive case?"
Answer: There were four TCMR and three ABMR events in the prospective validation cohort. Table 1 now reports prospective-cohort ROC-AUC for the corrected seed-42 models: 0.554 for TCMR and 0.679 for ABMR for KLEAR, and 0.346–0.420 and 0.430–0.620 for the other classifiers. Table 4 gives the corresponding high-resolution KLEAR AUPRC, sensitivity, specificity and confusion counts in the same cohort. Given the event counts, these are exploratory point estimates.
The original submitted AUPRC values, summarized as mean ± SD over 1,000 bootstrap replicates, remain described in Section 7.1; Table 4 now shows corrected high-resolution KLEAR point estimates and operating-point counts instead. The manuscript does not describe how replicates with zero or one positive case were handled. With so few events, the submitted operating points and decision curves support only an exploratory interpretation. The revised text reports the false-positive burden, and Section 7.2 states that clinical net benefit remains undetermined. We have removed the “revolutionize” and clinical-superiority language.
Manuscript Sections 7.1–7.2, Tables 1 and 4 and Figure 2.
Q6: "How does the TCMR model perform without race as an input, and within racial subgroups?"
Answer: The revision states that donor and recipient race were prominent model inputs and that outcome oversampling provides no evidence of demographic fairness. We agree that a matched refit removing both variables is the appropriate sensitivity analysis. This analysis and subgroup results are not included in this revision.
Manuscript Sections 6.1 and 8, and Figure 8.
Q7: "What is the E3 framework, and is it publicly available?"
Answer: Section 3.2 now explains how E3 uses typed HLA alleles and aligned donor–recipient sequences to derive residue and physicochemical mismatch features. In the high-resolution analysis, E3 generated 95 features from two-field typing across nine HLA loci. Section 3.2 distinguishes class I α1/α2-restricted summaries from full-molecule features and describes the epitope-related inputs. We cite Jucaud et al. (2020, ATC abstract 103) for earlier use of E3, and Appendix E.2 lists the model-specific inputs. An availability statement for E3 is not included in this revision.
Manuscript Section 3.2, beginning “E3 is a computational framework,” Appendix E.2 and References.
Q8: "Why does the best TCMR setup in Table 2 have ROC-AUC 0.46 but the highest AUPRC?"
Answer: A small number of high-ranked cases can raise precision at some recall levels even when the overall positive–negative ranking is weak; Section 5.2 now notes this sensitivity. In the rerun Table 2, this setting has AUPRC 0.358 and ROC-AUC 0.701. Table 2 differs from Table 1 because it uses the 285-recipient subset (21 TCMR events) without nested cross-validation, whereas Table 1 uses all 319 recipients with nested cross-validation. The similar XGBoost pattern in the submitted Table 1 (AUPRC 0.26, ROC-AUC 0.43) does not appear in the corrected Table 1 (0.083 and 0.511). We do not infer a reliable improvement from these estimates.
Manuscript Section 5.2, paragraph beginning “For TCMR, the DRC plus full E3 feature set.” Tables 1 and 2.
Additional comment: Site analysis and cohort coverage
Reviewer comment: Florida has higher TCMR prevalence than Arizona, and Rochester is absent from the reported high-resolution site analysis.
Answer: We revised the text to show the event counts alongside the AUPRC estimates and to describe Table 3 as a site-stratified development analysis, now with 95% confidence intervals. The TCMR prevalence was 13/106 in Florida and 9/213 in Arizona, with AUPRC of 0.480 (0.21–0.74) and 0.083 (0.02–0.27); the Florida ABMR estimate rests on three events. Site membership was inferred from patient identifiers and requires confirmation against source records. The AUPRC difference cannot establish transportability, and a site-held-out analysis is not included. Rochester contributes to the broader source population and was absent from this high-resolution comparison.
Manuscript Sections 3.1 and 5.3, and Table 3.
Response to Reviewer 5zoo - Part I
Q1: "Was hyperparameter tuning nested inside the cross-validation? Was PCA refit inside each fold, and what data was the calibration fitted on?"
Answer: The submitted pseudocode placed PCA before cross-validation. The corrected evaluation separates tuning and calibration from outer evaluation: Optuna selection uses inner folds, and calibration and thresholds use development-only cross-fitted predictions. Imputation, standardization and categorical encoders are fitted within the corresponding training partitions; however, the E3 principal components were precomputed rather than refit within each fold. The submitted manuscript names isotonic regression, while a companion figure script uses sigmoid calibration; we have documented this discrepancy in the Figure 7 caption. Figure 7 shows the submitted calibration plots, and calibration plots for the corrected models are not included.
Manuscript Sections 4.2 and 4.6, Appendix C and Appendix F, Figure 7.
Q2: "What are the results on the imputed and low-resolution cohorts? Can you compare high-resolution, imputed and low-resolution typing on the same patients, as Section 5.2 suggests?"
Answer: We also describe the full imputed (n = 1,263) and low-resolution (n = 1,340) HLA cohorts. Table 2 compares models in 285 recipients matched by clinical variables across all three HLA datasets, using identical folds and outcomes (21 TCMR and 12 ABMR events).
Table 2 Panel A reports the input feature-set analysis within a 285-recipient subset with shared patient folds. It includes full E3 features from high-resolution typing and partial E3 features from imputed typing; for TCMR with clinical predictors, AUPRC was 0.358 versus 0.257. Panel B compares high resolution (DRCs with full E3-MoFs) with low resolution (DRCs without HLA-derived inputs) on the same 285 recipients, endpoints and folds. Event counts now appear in the Outcome column.
Panel B now reports AUPRC increases from low to high resolution of 12.2% for TCMR and 6.7% for ABMR. Section 5.2 retains the paired absolute AUPRC differences and 95% intervals: 0.039 (−0.184 to 0.262) for TCMR and 0.022 (−0.097 to 0.140) for ABMR. The prospective results in Table 4 are for the high-resolution KLEAR model in Table 1 (AUPRC 0.065 for TCMR and 0.074 for ABMR); they are not an input-resolution comparison. These estimates are exploratory because there were four TCMR and three ABMR events. The low-resolution setting contains no HLA-derived inputs, so the development comparison estimates the added value of high-resolution molecular features over clinical characteristics and does not isolate typing resolution. A paired imputed arm and a comparison with resolution-specific HLA inputs are not included in this revision.
Manuscript Section 3.1 and Section 5.2, “this comparison estimates the added value of high-resolution molecular features over clinical characteristics and does not isolate typing resolution.” Tables 2,4.
Q3: "Can you report paired comparisons of KLEAR against logistic regression and against clinical data alone, and show how stable the rankings are over repeated cross-validation with different seeds?"
Answer: We agree that the submitted results and overlapping single-model intervals leave comparative performance unresolved. In the corrected Table 1, KLEAR’s development intervals overlap those of logistic regression and the other baselines: TCMR AUPRC was 0.143 versus 0.080, and ABMR AUPRC was 0.069 versus 0.058. The paired comparison with clinical inputs alone is Table 2 Panel B (Q2); Table 4 instead reports the high-resolution KLEAR prospective operating-point results from Table 1. This revision does not include a paired comparison of KLEAR with logistic regression or repeated-seed development results. Section 7.1 reports prospective-cohort AUPRC for four additional seeds, which does not show a stable advantage. We therefore make no claim that KLEAR outperforms simpler models.
Manuscript Sections 5.1, 5.2 and 7.1, and Tables 1, 2 and 4.
Q4: "How does performance change without oversampling, without leaf features, or without focal loss? How do LightGBM, CatBoost and TabPFN compare?"
Answer: The revision cites He et al. (2014) and frames the tree-leaf representation as prior work applied with a neural classifier in this setting. The archived companion implementation applies the same focal-loss multiplier to both classes and weights examples by prediction difficulty, and Equation 6 now matches this implementation. The abstract now describes a Gaussian mixture model rather than a Bayesian Gaussian mixture. We agree that ablations of oversampling, leaf features and focal loss, and benchmarks against LightGBM, CatBoost and TabPFN, are needed to establish the value of the combined design. These are not included in this revision.
Manuscript Abstract, Section 2, Section 4.5 and Equation 6.