- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The goal of the paper is to assign patients to disease subtypes and disease stages, as well as reconstruct biomarkers' progression trajectories, based on both cross-sectional and longitudinal datasets. The authors compared their method, TRACE, with other baselines using real-data inspired synthetic datasets and claimed performance improvement. The authors also applied TRACE to real-world datasets.
Critical strengths
Assigning patients to different subtypes and disease stages using longitudinal data is a very important task with considerable impact on medical applications. The current paper is among a very few to solve this problem using a statistical approach. The current paper is also one of the few (maybe the first) that reconstruct biomarker trajectories from cross-sectional datasets.
Critical weaknesses
I have two main concerns over the paper: 1) incomplete and unfair benchmarking, and 2) lack of empirical support.
Incomplete and unfair benchmarking
Incompleteness
-
"Bayesian Event-Based Model for Disease Subtype and Stage Inference" (Hao & Austerweil, 2026) did exactly the same tasks as, and improved upon, SuStaIn, but was not included as one of the baselines. Considering that it performed better than SuStaIn, I don't think TRACE's results are convincing without it.
-
The papers lacked strong baselines for the special case of . For cross-sectional data, candidate baselines are
- DEBM (Venkatraghavan et al., 2019, Disease progression timeline estimation for Alzheimer’s disease using discriminative event based modeling)
- EBM, KDE (Firth et al., 2020, Sequences of cognitive decline in typical Alzheimer’s disease and posterior cortical atrophy estimated using a novel event-based model of disease progression)
- SA-EBM (Hao et al., 2025, Stage-Aware Event-Based Modeling (SA-EBM) for Disease Progression)
- TEMPO (Hao & Austerweil, 2026, TEMPO: Transformers for Temporal Disease Progression from Cross-Sectional Data)
Considering that the first three methods can only process ordinal disease stages, TEMPO, which outputs continuous disease stages as, is an important baseline to compare with in terms of Stage when . For longitudinal datasets, a strong baseline is TEBM (Wijeratne et al., 2023). For all the above, you can focus on Stage along with .
-
The trajectory reconstruction as shown in Figure 3 lacks baselines. You are just showing TRACE's results, without any comparison with results from competitive methods. As far as I know, SubLign and DPMoSt both can reconstruct biomarker trajectories using longitudinal data. I think it's important to use the metric of integrated squared error (ISE) to compare your result with these two methods.
-
LN 415-420 mentioned how TRACE identified the best possible number of subtypes for the real-world data. However, in synthetic experiments, this ability of TRACE was never tested against other baseline models. I think this is an important miss. As far as I know, both BEBMS ("Bayesian Event-Based Model for Disease Subtype and Stage Inference") and SuStaIn can do this. I recommend the authors to generate datasets with different number of subtypes and benchmark your method against BEBMS and SuStaIn using Mean Square Error (MSE) regarding estimating the number of subtypes.
-
What's the runtime and resource consumption comparison? You only had it in B.4 (Ln 901-917) but not anywhere else.
-
You used s-SuStanIn, not SuStanIn itself. Could you prove that s-SuStanIn performs at least as well as SuStaIn on your tasks? I know the s-SuStanIn paper has the benchmark but I think it's important to demonstrate it using your data as well. Also, SuStanIn has two variants (GMM and KDE) depending on the assumption of the distribution of biomarker measurements. I don't think you tested both.
Unfairness
-
Ln 690. Using uniform distribution to generate disease stage is unrealistic. For example, Donohue et al. (2014) ("Estimating long‐term multivariate progression from short‐term data") suggested that later-stage participants are underrepresented. The result might be a non-uniform empirical distribution for patient disease stages. Also, using uniform distribution may give TRACE unfair advantage in the experiment. I know SuStaIn and SubLign have the same assumption, but DPMoSt used a Gaussian prior and BEBMS had a symmetric Dirichlet prior. For fair comparisons, I recommend the authors use different priors for your synthetic datasets, for example, Gaussian, Beta, and uniform.
In sum, 1) It's a modeling leakage as you are using the same distribution in your model (LN 144) as what you used for data generation (Ln 690); 2) It's impractical as empirical distribution of disease stages might not be uniform.
-
Comparing TRACE with s-SuStaIn for longitudinal data is unfair as SuStaIn is designed for cross-sectional data. I understand the motivation, but I think it's better to make it explicit in the caption of table 1 (even though the point has been made in LN 81-83)
-
It's unfair to compare the stages predicted by SuStaIn with yours. SuStaIn can only output ordinal stages. SuStaIn interprets "stage" as the number of biomarkers that have become pathological whereas your definition of "stage" is a 0-1 float indicating the severity of the disease. They are very different concepts. And since your ground truth is continuous (Ln 690) but SuStaIn only outputs ordinal, the comparison is not apple-to-apple at all. To make the comparison fair, at least for some synthetic datasets, you have to use ordinal stages as the ground truth.
-
The experimental setting, or the data generation process, as described in Ln 677-680, is different from the underlying assumption of SuStaIn. SuStaIn assumes that all subtypes have the same ROIs, just with different orderings over ROIs. According to your data generation, I don't think the comparison is fair for SuStaIn. To make fair comparisons, I recommend the authors to ALSO test on datasets where all subtypes have the same set of ROIs.
Also, what's your rationale of assuming that different subtypes of Alzheimer's affect different regions? And how did you get the 5 shared and 10 subtype-specific regions setting? How do you prove that it's a realistic configuration?
Lack of empirical evidence
The real-world data experiments' result lack empirical support. The authors found that is the most possible number of subtypes. Any medical evidence supporting this finding?
Also, I don't think "These patterns broadly correspond to the clinical profiles (Tab. 2)" is convincing. First, Table 2 is based on your own assignment of subtypes, so it's a self-validating evaluation. Second, you are just describing the characteristics of the four subtypes you identified, without any previous medical literature supporting your findings. For example, which medical papers identified the four subtypes for Alzheimer as your finding? And how are your identified four subtypes corresponding to those subtypes?
Questions for authors
-
LN 120: "where larger values indicate greater abnormality". In reality, this might not be case. For some biomarkers, larger values indicate smaller abnormality. Is TRACE able to accommodate this? If the number of biomarkers is huge, this preprocessing is complicated and prone to errors. Therefore, I recommend the authors to make TRACE able to handle the cases where smaller values might indicate greater abnormality as well.
-
Your model assumes that biomarker values progress monotonically. However, in reality, this might not be the case. Reversion is possible (for example, Malek-Ahmadi, M. (2016). Reversion from mild cognitive impairment to normal cognition: a meta-analysis. Alzheimer Disease & Associated Disorders, 30(4), 324-330.). While your model is not able to accommodate this, SuStaIn can. This is because SuStaIn and BEBMS don't have assumptions about this monotonically. They just assume that biomarker measurements follow normal distributions. I don't think you covered this as one limitation of TRACE.
-
Why did you use logistic trajectory for biomarker progression trajectories? Why is that realistic way to model how biomarkers progress in the real world? Any prior work supporting this assumption?
-
Your method described in Section 2.2 and 2.3 is almost identical to the inference method in the paper of BEBMS. The only differences stem from how to handle disease stages: TRACE assumes continuous stages whereas both BEBMS and SuStaIn assume ordinal ones. I don't think the authors are blindly copying BEBMS's idea. If the authors used AI in the math formulation, it might be that the model got the idea from the BEBMS paper. If the authors didn't use AI in the formulation, it nonetheless indicate a lack of rigorous literature review.
-
The B.4 experiments clearly shows that the only advantage of TRACE over SuStaIn, at least with that experimental setting, is smaller runtime, without performance gains. Is that true?
-
For Section 3.2 (Semi-simulated data experiments), can the authors try different number of ROIs (in terms of total unique ROIs among all subtypes)? Say, 5, 10, and 100.
-
The synthetic experiments used only patients data, without data from controls. I think it's important to include control data for two reasons. First, it can test the models' capability to handle the edge cases. If the model cannot distinguish patients from controls, it might indicate there is something wrong. Second, practically, people might be incorrectly diagnosed and the participants with the "patient" label might be healthy and those "healthy" might be actual patients. Could the authors accommodate this suggestion in the semi-simulated experiments?
7 comments
Response to Reviewer fxRa (Part 2)
I4: Recovery of the number of subtypes was not benchmarked
We agree that recovery of the number of subtypes required a separate evaluation. Following the reviewer's suggestion, we used SuStaIn's simulator with biomarkers and true , generating five datasets for each value of . We evaluated candidate values from to and measured the mean absolute error (MAE) and mean squared error (MSE) between the selected and true numbers of subtypes.
We first applied the same CVIC-based selection rule to SuStaIn, BEBMS, and TRACE, using the rule implemented in the BEBMS package: a larger is preferred only when it reduces CVIC by more than 6.
For TRACE, we also evaluated the joint selection rule used in our real-data analysis, which formalizes the combination of held-out likelihood, assignment stability, and subtype proportions. Using five-fold cross-validation with 10 initializations per fold, we started at and accepted only when all three conditions held: (1) total held-out log-likelihood improved in at least 8 of the 10 initializations; (2) the mean pairwise ARI between out-of-fold participant assignments across initializations was at least 0.60; and (3) every subtype contained more than 10% of all participants in every initialization.
| Method and selection rule | MAE | MSE |
|---|---|---|
| BEBMS, CVIC | 1.03 | 1.70 |
| SuStaIn, CVIC | 1.33 | 2.73 |
| TRACE, CVIC | 1.40 | 2.53 |
| TRACE, joint (rule used in our real data experiments) | 0.50 | 1.07 |
Although the joint rule yielded lower MAE and MSE in this evaluation, both selection rules depend on prespecified thresholds. The CVIC rule uses a minimum improvement of 6, while our joint rule uses thresholds for likelihood improvement, assignment stability, and subtype size. Selecting these thresholds is challenging in real-world data because the true number of subtypes is unknown and no single metric can reliably identify the appropriate number of clusters across settings. This uncertainty is a broader challenge in clustering studies and and we admit it still remains as a limitation of our method.
I5: Runtime and resources comparison
We have expanded the original B.4 benchmark to report fitting time and memory usage on the main 96-ROI semi-simulated task, as well as scaling with the number of biomarkers on datasets generated using SuStaIn's simulator. We present these two evaluations below.
(a) Main 96-ROI task. We measured fitting time and memory on the main semi-simulated datasets, with each fit limited to eight CPU cores. Peak RAM is the peak resident memory of the whole process tree. The ARI and stage columns summarize subtyping and staging performance for comparison with computational cost. TRACE required approximately five minutes per fit, offering the best performance at a computational cost comparable to several of the evaluated methods.
| Method | Fit time | Peak RAM (GB) | ARI | Stage |
|---|---|---|---|---|
| TRACE | 311 13 s | 0.89 | 0.903 | 0.878 |
| s-SuStaIn | 305 30 s | 1.61 | 0.569 | 0.317 |
| DPMoSt | 73 1 s | 0.71 | 0.172 | 0.338 |
| Surreal-GAN | 435 49 s | 0.69 | 0.481 | 0.817 |
| SubLign | 105 2 s | 1.29 | 0.147 | 0.267 |
| BEBMS | ~3.7 h | 2.90 | 0.459 | 0.322 |
(b) Scaling with the number of biomarkers . We also evaluated TRACE and other event-based models on cross-sectional datasets generated using SuStaIn's simulator with increasing numbers of biomarkers. The reported slope is obtained by least-squares regression of log median fitting time on log , using the available entries from to for each method. A slope of 1 corresponds to linear growth in fitting time. TRACE's empirical slope of 0.98 indicates approximately linear scaling over this tested range, whereas the measured costs of BEBMS and SuStaIn increased more steeply.
| Method | F = 10 | F = 25 | F = 50 | F = 100 | F = 400 | Slope |
|---|---|---|---|---|---|---|
| TRACE | 19 s | 28 s | 43 s | 80 s | 6 min | 0.98 |
| DPMoSt | 13 s | 22 s | 33 s | 58 s | 3 min | 0.78 |
| Surreal-GAN | 91 s | 103 s | 2 min | 3 min | 5 min | 0.41 |
| s-SuStaIn | 3 s | 36 s | 90 s | 4 min | 23 min | 1.33 |
| BEBMS | 2 min | 8 min | 28 min | 100 min | – | 1.78 |
| SuStaIn (GMM) | 2 min | 14 min | 99 min | 9.8 h | – | 2.70 |
| SuStaIn (Z-score) | 32 min | 11.3 h | – | – | – | – |
| Dash indicates unavailable result due to its long fitting time. |
(-> continue in Part 3)
Response to Reviewer fxRa (Part 3)
I6: s-SuStaIn was used instead of original SuStaIn (GMM and KDE)
We used s-SuStaIn in the main 96-ROI benchmark because it reduces computational cost by grouping biomarkers into a smaller number of clusters. In contrast, the cost of fitting the original SuStaIn models increases steeply as the number of biomarkers grows. The runtime results in I5 illustrate this limitation. Biomarker clustering in s-SuStaIn can sacrifice information and affect recovery accuracy, so we agree that s-SuStaIn alone does not fully represent the performance of the original SuStaIn variants. Improving scalability while retaining flexible trajectory modelling is one motivation of TRACE. To provide a broader comparison, we have added benchmarks using SuStaIn's simulator that include s-SuStaIn, as well as the GMM, KDE, and z-score variants of SuStaIn, as reported in U3.
U1: The uniform stage prior is shared by the generator and the model
In the original simulation, baseline stages were sampled uniformly, while stages at subsequent visits followed participant-specific progression and were not sampled from a uniform distribution. Nevertheless, we agree that the uniform baseline distribution may favor TRACE's uniform stage prior. To ensure fairer comparison, we regenerated the 96-ROI task with three alternative baseline-stage distributions while retaining the uniform stage prior in TRACE:
- Beta(2,5): an early-skewed distribution with fewer observations at later stages.
- Truncated Gaussian: , truncated to before rescaling to the baseline-stage range.
- Symmetric Dirichlet mixture: 10 equal-width stage bins with probabilities drawn from separately for each dataset, and uniform sampling within each bin. This produces a different, irregular stage distribution in each dataset.
The results are shown as follows, with each table entry reports showing ARI / stage across datasets. TRACE with achieved higher mean ARI and stage than s-SuStaIn, DPMoSt, and BEBMS under every baseline stage distribution tested. These results suggest that its advantage is not restricted to uniformly distributed baseline stages.
| Baseline stage distribution | TRACE (=30) | TRACE (=0) | s-SuStaIn | DPMoSt | BEBMS |
|---|---|---|---|---|---|
| Uniform | 0.903 / 0.878 | 0.832 / 0.651 | 0.569 / 0.317 | 0.172 / 0.338 | 0.459 / 0.322 |
| Beta(2,5), early-skewed | 0.782 / 0.839 | 0.433 / 0.505 | 0.283 / 0.117 | 0.019 / 0.419 | 0.118 / 0.132 |
| Truncated Gaussian | 0.925 / 0.814 | 0.875 / 0.537 | 0.769 / 0.144 | 0.200 / 0.397 | 0.558 / 0.201 |
| Symmetric Dirichlet | 0.892 / 0.852 | 0.813 / 0.614 | 0.613 / 0.236 | 0.243 / 0.364 | 0.439 / 0.263 |
U2: Comparing TRACE with s-SuStaIn for longitudinal data is unfair as SuStaIn is designed for cross-sectional data.
We thank the reviewer for raising this concern. We clarify that Table 1 evaluates the method under two settings: 1) a cross-sectional setting using only one scan per participant and 2) a longitudinal setting that additionally includes scans of follow-up visits. In the cross-sectional setting, TRACE is trained without the change reconstruction loss and achieves the highest mean ARI and stage accuracy among the evaluated methods. Thus, TRACE's performance advantage is also observed under a cross-sectional comparison and is not attributable solely to its use of longitudinal data.
Our motivation was to learn from datasets containing both cross-sectional observations and sparse longitudinal follow-up, as is common in many public datasets. TRACE accommodates cross-sectional scans alone and leverages within-person changes when repeated visits are available. We view this capacity to use sparse longitudinal information as an important contribution, offering an advantage over methods that treat visits as independent observations.
(-> continue in Part 4)
Response to Reviewer fxRa (Part 1)
We sincerely thank the reviewer for the time and care devoted to this detailed review. The comments have helped us improve many aspects of our work. We appreciate the opportunity to address these concerns and strengthen the manuscript.
We organize our responses using the following labels: I for incomplete benchmarking (the first numbered list in the review), U for unfair comparisons (the second numbered list), E for empirical support for the real-data results, and Q for the reviewer's questions. We are still working on revising the manuscripts and will upload the complete version before the discussion deadline.
I1: Include BEBMS as an additional baseline
We thank the reviewer for suggesting this comparison. We have added BEBMS to our comparative evaluations using its released implementation and recommended settings. The results are presented in the relevant responses below. We will also cite BEBMS in our revised manuscript.
I2: Missing staging-only (K=1) benchmark
We agree that setting K=1 provides a useful test of staging accuracy. We therefore added benchmarks using two simulation generators: (1) our 96-ROI semi-simulated generator; and (2) SuStaIn's z-score generator, with F=10,25,50 biomarkers. We compared TRACE with SuStaIn (z-score and GMM variants), s-SuStaIn, EBM, KDE-EBM, SA-EBM, DEBM, and BEBMS, using stage ρ to assess agreement between inferred and ground-truth stages.
Results on SuStaIn's simulator.
† This entry is based on a single fit, which took 23.8 hours.
Results on our 96-ROI semi-simulated datasets with K=1.
On SuStaIn's z-score simulator, TRACE achieved staging accuracy close to that of z-score SuStaIn and higher mean accuracy than the other evaluated methods. On our 96-ROI semi-simulated datasets, TRACE with λ=30 achieved the highest mean stage ρ (0.888), followed by DEBM (0.863). Removing the change reconstruction loss reduced TRACE's mean stage ρ to 0.645, indicating that within-person changes also improve staging when there is only one subtype.
I3: Missing comparisons on the reconstructed trajectories
Figure 3 was designed to illustrate the effect of the change reconstruction loss. We appreciate the reviewer's suggestion and added comparisons with SubLign and DPMoSt on our 96-ROI semi-simulated datasets.
For each scan, we evaluated each method's fitted biomarker mean at its own inferred subtype and stage. We reversed any method-specific scaling to express all predictions in common z-score units and compared them with the noise-free ground-truth biomarker profile at that scan's true subtype and stage. We measured integrated squared error by averaging the squared differences across scans and ROIs.
As shown below, TRACE with λ=30 achieved the lowest mean reconstruction error for both affected and unaffected ROIs. Its low error on unaffected ROIs also indicates that it introduced little spurious abnormality in regions with no simulated disease effect.
(-> continue in Part 2)