- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper created a transformer&MLP based model for disease progression, that performs staging and subtyping. As opposed to SuStaIn, it uses deep-learning/transformers, continuous stages and several additional encoding steps. Validation is on simulated data with known ground truth as well as ADNI/NACC on Alzheimer's disease progression, a well known problem in medicine.
Critical strengths
Great use of transformer/deep-learning methodology for such a complicated problem involving rankings and hierarchical modeling.
Critical weaknesses
The model seems to be able to recover orderings better than SuStaIn esp with very large biomarker sets but the paper only goes up to 100. What would make the paper stronger would be to demonstrate it on real data with very large numbers of biomarkers where subtypes/heterogeneity is present. Some example datasets:
- TADPOLE: spreadsheet of ~2000 biomarkers
- UK BioBank
- applying it to brain images where each voxel can become abnormal (e.g. amyloid/tau/FDG PET images from ADNI, can reach up to ~millions of voxels, each voxel here is considered a "biomarker").
In addition, which part of the model deals with strong correlation in the biomarkers? making the i.i.d assumption with correlated biomarkers can lead to spurious results. Did the simulations simulate correlated biomarkers? How and where?
Questions for authors
Intro: amyloid → MMSE → entorhinal volume → hippocampal volume — should change ordering to reflect a more Ad consensus from jack et al: Amyloid -> MRI measures (entorhinal/hippocampus) -> cognitive tests (MMSE, etc ..). I understand that some subtypes will not follow this, so the example should reflect the current understanding of the field.
Fig 1 is not intuitive at all. Please improve it. Add diagrams/plots, graphivs, ... make it more intuitive. An LLM generated diagram like that offers little intuition for people actually trying to understand it.
R^cohort_j,b ranks patient j within biomarker b. This does not make sense. By cohort you mean sub-type? Should it not be labeled R_b,k so biomarker b within cohort subtype k? Please fix in the paper and clarify.
Section 3.4 suggests that multiple parallel branches are used (likely just 2). It would be so much useful to see the entire graphical model of the architecture (add in appendix). A figure with the full algorithm pseudocode in appendix should be further added and with the parallel branches. And any key comments to help understand it.
In Eq 10, define zeta explicitly as a map that takes a subtype k and predicts a slot m.
What is the formula for the ordering cost? It's not included.
How can eq 10 be optimized on real ADNI data where we don’t know true sub-types? Are these inferred subtypes? If yes, this raises some concerns about identifiability and local minimas. How well does sub-type inference work in practice?
Some explanation is also needed to understand why the model needs all those levels of hierarchy. [xj,b, Rwithin j,b , Rcohort j,b , dj ] -> u -> h -> t -> s -> r … what happens if you were to simplify or merge some of those steps? What is being lost?
5 comments
Thank you for the clarifications and the paper improvements. I am happy to increase the score if I see results on a large dataset of real biomarkers, such as the TADPOLE runs that are in progress. To me that is the critical missing piece that would convince someone to actually use it.
SUBTRACE results on TADPOLE
Thank you for the suggestion. We trained SUBTRACE on TADPOLE (Marinescu et al., 2018), an ADNI-derived dataset, with 239 real biomarkers. Details are in the new Appendix N of the updated PDF.
Data source
TADPOLE's 1,907 columns come from imaging blocks measured in different subsets of participants. Of 1,631 baseline participants, FDG PET covers 646, amyloid PET 791, DTI 190 and tau PET none, so adding any of them would at least halve the cohort. We use all cross-sectional FreeSurfer measures plus MMSE, ADAS13 and RAVLT: 239 biomarkers and 1,595 participants with complete data.
Training
Just like what we did for the 12 ADNI biomarkers, in TADPOLE experiment, and come from the EBM Gaussian mixture fit (Young et al., 2014), and the simulated data reproduce the measured correlation between the 239 biomarkers (mean simulated vs. measured). We trained one Exp. 9 model () on 500 simulated datasets of 1,595 participants each, in about 2 hours on one GPU. We did not run the baselines, which already take 111–368 minutes per dataset at 100 biomarkers (Table 1).
On 50 held-out synthetic datasets:
| distance | ARI | Stage MAE$/B$ | |
|---|---|---|---|
| 100 bio, Exp. 9 | 0.120 | 0.085 | 0.036 |
| TADPOLE, 239 bio, Exp. 9 | 0.287 | 0.461 | 0.041 |
Ordering error is higher because 62% of TADPOLE biomarkers barely separate CN from patients (separation below , against 8% at 12 and 9% at 100 biomarkers). On the clearly separated biomarkers, distance is .
Real TADPOLE results
1. Stage tracks diagnosis
Only CN/non-CN is input, so the order of EMCI, LMCI and AD is not imposed. Predicted stage (0–239), mean SD:
| Subtype | EMCI | LMCI | AD |
|---|---|---|---|
| 1 | |||
| 2 | |||
| 3 |
All 411 CN receive a predicted stage of 0. Stage rises monotonically within every subtype, tracking clinical severity.
2. Stage predicts conversion to AD
We repeat the ADNI survival analysis: baseline EMCI/LMCI participants, conversions from TADPOLE follow-up visits and a Cox model with stage rescaled to the ADNI 0–12 axis. Hazard ratios:
| Stage | Age | Sex | Education | APOE4 |
|---|---|---|---|---|
| (95% CI 1.25–1.38) | 0.99 | 0.83 | 1.00 |
Each 1/12 of the disease course further along at baseline raises the rate of AD diagnosis by 31%, indicating stages SUBTRACE infers carry prognostic information. Notably, subtype also predicts conversion beyond stage (likelihood-ratio ): at the same stage, patients in different subtypes convert at different rates, so the subtypes are clinically meaningful. On ADNI's 12 biomarkers, no method's subtypes did.
3. The subtypes match Murray et al. (2011)
Murray et al. define three subtypes at autopsy by the ratio of hippocampal to cortical tangle counts:
- limbic-predominant (hippocampus most affected; oldest, slowest decline)
- hippocampal-sparing (cortex most affected; youngest, fastest decline)
- typical (in between).
Each SUBTRACE subtype matches one of them on three lines of evidence: its ordering, its patients' measurements, and variables SUBTRACE never sees. The table gives the last two; volumes are mean -scores vs. CN, compared within the same diagnosis.
| Subtype 1 | Subtype 2 | Subtype 3 | ||
|---|---|---|---|---|
| Hippocampal volume, LMCI | ||||
| Hippocampal volume, AD | ||||
| Parietal volume, LMCI | ||||
| Parietal volume, AD | ||||
| Age (Murray: 86 / 73 / 79) | 74.7 | 71.8 | 74.5 | |
| Men (Murray: 31% / 63% / 45%) | 55% | 60% | 59% | |
| Conversion hazard ratio vs. Subtype 1 | 1 | 1.45 | 1.38 |
Subtype 1 (31%) matches limbic-predominant AD. Its ordering, cognition, hippocampus/amygdala entorhinal lateral temporal parietal frontal, puts hippocampus first, parietal late. Its patients have strong hippocampal and least parietal atrophy, are the oldest, have the fewest men and convert slowest.
Subtype 2 (37%) matches hippocampal-sparing AD. Its ordering, cognition parietal lateral temporal hippocampus/amygdala entorhinal frontal, puts parietal before hippocampus. Its patients have the least hippocampal atrophy at every diagnosis, are the youngest, have the most men and convert fastest.
Subtype 3 (33%) matches typical AD. Its ordering, cognition hippocampus/amygdala parietal lateral temporal entorhinal frontal, puts hippocampus and parietal early. Its patients have hippocampal and parietal atrophy and fall in between on age, men and conversion.
Response to reviewer rb6L
Thank you for the feedback. Your question about correlated biomarkers led to the most important change in our revision.
Correlated biomarkers
In the submitted version, the simulator drew each biomarker independently once a participant's stage was fixed. Real biomarkers are not like that.
We measured the correlation that remains in ADNI after stage is accounted for, and regenerated the 12-biomarker data so that each participant's biomarkers are drawn jointly with this correlation. The simulated residual correlation is 0.144 against 0.138 in ADNI, and 65 of the 66 biomarker pairs are within 0.10 of their ADNI value.
We then retrained SUBTRACE and refit every baseline (n=450 test datasets):
SUBTRACE improves on every metric while every baseline degrades.
As for which part of the model deals with correlation: no part of SUBTRACE assumes independence. The evidence Transformer attends across all of a patient's biomarkers, and the model is trained on correlated data, so it learns how much co-occurring abnormality tells it. BEBMS and SuStaIn multiply per-biomarker likelihoods, so correlated biomarkers are counted as independent pieces of evidence.
We applied this only at 12 biomarkers. ADNI has these same 12 biomarkers, so the correlation is measured rather than assumed. No real cohort exists from which to measure a 100-biomarker correlation.
Larger biomarker sets on real data
We trained and tested SUBTRACE on synthetic data with 2,000 biomarkers (Exp. 1, 50 test datasets): τ distance 0.158, ARI 0.788, stage MAE 57 out of 2,000 stages (Appendix U).
Applying SUBTRACE to TADPOLE is planned. We will post the results if we finish before the rebuttal deadline.
A real cohort with thousands of biomarkers still has no ground-truth ordering, so such an analysis would be qualitative, as in Section 5.
Introduction example
It now follows Jack et al.: amyloid → entorhinal volume → hippocampal volume → MMSE.
Figure 1
We agree and have redrawn it around the central idea: the model answers "is biomarker b abnormal for patient j?" in two ways, from the data and from its own estimates of subtype timelines and stages, and assigns each patient the subtype for which the two answers agree.
Rcohort
"Cohort" here means the whole input dataset, not a subtype. Rj,bcohort is patient j's rank among all patients on biomarker b. It cannot be computed within a subtype, because subtypes are not known yet at this step. We have clarified this in Section 3.2.
Full architecture and pseudocode
We added Appendix B with the complete architecture: the three parallel branches (router, ordering, and evidence), where they meet, and where each training loss attaches. Algorithm 1 gives the full forward pass as pseudocode.
ζ and the ordering cost
ζ is now defined as a one-to-one map that assigns each true subtype k to a predicted slot m=ζ(k). The ordering cost is now given as Eq. 10, so the matching equation is now Eq. 11.
Eq. 10 (now Eq. 11) on real data
Eq. 11 is used only during training, on simulated cohorts whose true subtypes are known. On ADNI nothing is optimized: the trained model is applied in one forward pass, so there are no per-cohort local minima. On held-out synthetic data, subtype accuracy is in Table 2: ARI 0.363 against 0.176 for the best baseline at 12 biomarkers, and 0.544 against 0.581 for SuStaIn (GMM) at 100. With five subtypes it reaches 0.415 against 0.236 (Appendix Q).
Why so many levels
We removed or simplified each step in turn (12 biomarkers, Exp. 1, n=50):
The seed-router loss and the second pass help ordering, the profile fit helps subtyping, and rank features help staging. Attention in t→s makes no measurable difference at 12 biomarkers, but at 100 it lowers stage MAE from 4.46 to 3.87 (Appendix R). u→h is only a learned embedding of the four input features. Removing h→t (no subtype pooling) gives a single ordering for everyone, as in TEMPO; Appendix H shows what that costs. s→r is only a sigmoid.
Please see Figure 1, Section 3, Section 4, Tables 2 and 3, and Appendices B, F and R in the revised PDF.