- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Based on cross-sectional datasets about three modalities, i.e., Amyloid, Tau, and Neurodegeneration, CALIPER identifies subtypes and also the progression of these modality along a timeline for Alzheimer's disease.
Critical strengths
Estimating progression timeline from cross-sectional neurodegenerative disease datasets is challenging, but very useful for researchers and clinicians. Most of the time, datasets available are only sparsely longitudinal, and that's why many existing approaches such as EBMs focused on cross-sectional datasets. But most methods, using cross-sectional data, cannot output specific timelines for disease progression. This paper provides one of the few attempts (besides TEMPO) to achieve this goal.
References:
Hao, H., & Austerweil, J. L. (2026). TEMPO: Transformers for Temporal Disease Progression from Cross-Sectional Data.
Critical weaknesses
I identified two critical weaknesses of the paper.
Unclear and undefined terms
-
In the first few pages, the authors used many terms that are extremely hard to interpret. Just to name a few: "modality stages", "stage velocities", "response sensitivity", "residual variation", "local comparison", "progression direction", "statistical scale", "disease position", "calibrated local change", "latent state", "common clock", "modality-specific stage offsets", "reference regimes", "frozen representation", "endpoint-conditioned population currents", "regime responsibilities", "local stage velocities", "heteroscedastic emission model", "residual scales", "mean-response", "calibrated change", "fitted population field", "relative calibrated speeds", "development split", "shared disease clock", "disease coordinates", "local-encounter-level readout", etc.
-
LN 306-307: I also do not understand the experimental tasks: "modality-stage ordering", "relative calibrated-speed recovery", "endpoint assignment", "leading-modality accuracy". Note that I double checked, those terms have never been explained in sections before they appear and I don't think they are ever explicitly defined. Same is for terms of "modality-stage recovery", "relative information-speed recovery", and "endpoint-regime identification" in LN 324-325.
-
I also do not think "Local heteroscedasticity", and "Constant noise" are defined in the main text.
For the audiences, this makes it almost impossible to understand this paper.
Making sure the comparison is fair
The authors based their benchmarking results on synthetic datasets. However, the authors did not discuss whether the comparison with other methods (in Table 1) is fair. Many of these methods might have drastically different model assumptions than how the synthetic data is generated. The authors also failed to explain and disclose whether CALIPER "knows" any assumptions underlying data generation that are not available to the compared methods.
Questions for authors
-
What's your definition of modality? How's that different from "biomarkers"?
-
You said that previous methods didn't answer the question of "which modality is leading now?". But could you explain why this is an important question to answer in the first place? Why do we have to know this? Is it more important than knowing which biomarker will be impacted for this patient soon so we can prepare for the treatment or prevent it from being impacted?
-
Could you discuss why your synthetic data is as close to the real-world data as possible? For example, missing values.
-
Could you discuss whether it's fair to test the benchmarked methods on your datasets? Do these methods have the same underlying assumptions as your synthetic data? For example, EBM (Fonteijn et al., 2012) assumes that 1) biomarkers are either in healthy or pathological state, depending on how many biomarkers in this patient have become affected and the known biomarker-wise disease progression, and 2) participant disease stages are only nominal depending on how many biomarkers in this patient have become affected by the disease. Do these assumptions hold in your synthetic dataset generation method?
-
Relatedly, can you list assumptions that CALIPER has about the underlying synthetic data that are not known by other benchmarking methods?
References
Fonteijn, H. M., Modat, M., Clarkson, M. J., Barnes, J., Lehmann, M., Hobbs, N. Z., ... & Alexander, D. C. (2012). An event-based model for disease progression and its application in familial Alzheimer's disease and Huntington's disease. NeuroImage, 60(3), 1880-1889.
5 comments
Q2 (continued). Why this question matters clinically.
Treatment decisions already turn on which process is active. In TRAILBLAZER-ALZ 2, donanemab slowed decline in the low/medium-tau population; in a post hoc analysis of the high-tau group, the difference on the primary outcome was smaller and not statistically significant; and participants who met pre-specified amyloid-clearance criteria were switched to placebo (Sims et al., 2023). Deciding whether a person is still in the window where amyloid-directed treatment helps, and what to monitor once amyloid has been cleared, requires knowing which process is moving now, not only which marker will become abnormal next. Finally, the AT(N) framework (Jack Jr et al., 2018) characterizes individuals by their biomarker profile rather than by a single stage, and individuals at the same clinical stage can carry different profiles; two people at the same point on the disease clock can therefore differ in which process is advancing. A population sequence cannot express this; an encounter-level readout can, and that is the gap CALIPER addresses.
Relation to "which biomarker will be impacted next". The two questions use different information and serve different decisions. "Impacted next" is a population ordering that gives everyone at a given stage the same next event (Fonteijn et al., 2012); "leading now" is read from one person's snapshot, without follow-up, and is what the treatment and monitoring decisions above require. Existing snapshot methods provide the first; CALIPER adds the second.
Q3. Realism and missingness. We sampled the seven A/T/N scan combinations at frequencies that approximate those in the clinical data (Eq. 56). For example, 41.4% had MRI alone and 11.3% had all three. This captures scan availability, while the reasons a clinician orders a scan lie outside the simulation. Masks are drawn independently of stage.
Q4–Q5. Benchmark fairness and shared assumptions. All methods had access to the same snapshots, masks, available ages, and training anchors, the participants whose reference-regime labels are visible. CALIPER received no true stages, speeds, noise scales, or held-out labels. The generator does give CALIPER a structural advantage by sharing its continuous clock, bounded modality-stage offsets, degree-six regional response curves, and independent Gaussian residuals. The speed target uses the Fisher scale CALIPER is designed to estimate. In contrast, the simulated progression paths were written as closed-form curves, not generated by the Schrödinger bridge that CALIPER fits, so that part of CALIPER is not matched to the generator.
EBM infers an ordering of biomarker abnormality events. The two assumptions you list do not hold in our generator: each regional measurement changes continuously with stage rather than switching between a healthy and a pathological state, and stage is continuous rather than a count of affected biomarkers. EBM also does not output modality-specific stages or speeds, so Table 1 evaluates it only on endpoint balanced accuracy, that is, whether the correct reference regime is identified, through a classifier trained on EBM stage and age.
Among the methods listed in Table 1, CALIPER had the highest mean in five of the six comparisons on stage correlation, relative-speed correlation, and leading accuracy across the two noise settings. These metrics and settings are explained under Weakness 1 (Part 3). In the sixth, stage correlation under constant noise, CALIPER reached 0.955 versus 0.960 for MoPoE-VAE. A within-model ablation isolates the contribution of calibration: the fitted model is kept fixed and only the scale used to compare modalities is changed. Under local heteroscedasticity, leading accuracy was 0.775 with the local Fisher scale, 0.737 with a fixed scale, and 0.633 without calibration. On 525 development-split real-data anchors, CALIPER's endpoint balanced accuracy was 0.748 versus 0.677 for same-anchor logistic regression (Appendix H.2).
Weakness 2: fairness of the comparison. Our answers to Q3–Q5 above state what all methods share, where the generator favors CALIPER, and how EBM was evaluated. The paper describes the simulation as assumption-matched (Section 4.2; Appendix D.1) and gives the shared baseline protocol in Appendix D.4. The dashes in Table 1 mark readouts that a method does not provide.
Reference not in the submission (Jack Jr et al., 2018 and Fonteijn et al., 2012 are cited as in the paper)
- Sims JR, Zimmer JA, Evans CD, et al. Donanemab in early symptomatic Alzheimer disease: the TRAILBLAZER-ALZ 2 randomized clinical trial. JAMA. 2023;330(6):512–527. doi:10.1001/jama.2023.13239
Weakness 1: undefined terms. We agree that the submission left many of these terms undefined and used more than one name for the same quantity. In the revised paper we made the following changes.
- Introduction. It now defines modality, the disease clock and modality stage. We also replaced several of the terms you listed with plain descriptions. For example, “response sensitivity and residual variation” is now “how strongly the predicted regional means change with stage and how much the measurements vary around those means”; “local stage velocities” is now “each modality’s rate of stage change along the estimated population path”; “heteroscedastic emission model” is now “an observation model with stage- and group-dependent residual variance”; and “a statistical scale” is now “units of residual standard deviation”.
- Section 4.1. We replaced the list of task names with definitions of the four measures: stage correlation, relative-speed correlation, endpoint balanced accuracy and leading-modality accuracy.
- Section 4.2. We added definitions of local heteroscedasticity and constant noise, and removed the second list of task names (lines 324–325 of the submission).
The notation is listed in Appendix A. Below we define every term you listed. A term in parentheses is another name that the submission used for the same quantity.
Where a person is.
- Shared disease clock (common clock, disease position, disease coordinate): one number that places each snapshot on a common timeline of the disease.
- Modality-specific stage offset: how far one modality runs ahead of or behind the clock. It is bounded.
- Modality stage: clock plus offset, that is, how far that modality has progressed.
- Latent state: a learned summary of the available scans, from which the clock and the stages are computed.
- Frozen representation: the fitted clock, stages and measurement model, which are held fixed in the later steps.
Which way the population moves.
- Reference regime: a clinically defined endpoint group. Anchors are the participants whose clinical labels define these groups.
- Endpoint-conditioned population current (fitted population field, progression direction): the average direction in which snapshots move from the early group toward one endpoint.
- Regime responsibility: how strongly a snapshot belongs to each regime.
- Stage velocity (local stage velocity): how fast a modality's stage changes along the current.
How modalities are compared.
- Heteroscedastic emission model: the measurement model. It gives each measurement's average at a stage (mean response), how strongly that average changes with stage (response sensitivity), and its noise (residual scale, residual variation), which may vary with stage.
- Statistical scale (Fisher scale): sensitivity relative to noise, that is, how well a small change in stage can be told apart from noise. Local means that it is computed at the person's own stage.
- Calibrated change (calibrated local change, calibrated speed): stage velocity times this scale, so that each modality's change is in units of its own noise.
- Leading modality: the observed modality with the largest calibrated speed. Relative calibrated speeds are each modality's share of the total.
- Encounter-level readout (local encounter-level readout, local comparison): this summary for one visit.
Experimental tasks. In the submission, lines 306–307 and 324–325 named the same four tasks in two ways.
- Stage correlation (modality-stage ordering, modality-stage recovery): are participants ranked correctly by each modality's true stage?
- Endpoint balanced accuracy (endpoint assignment, endpoint-regime identification): is the correct reference regime identified?
- Relative-speed correlation (relative calibrated-speed recovery, relative information-speed recovery): are the modalities' shares of calibrated speed recovered?
- Leading accuracy (leading-modality accuracy): is the leading modality identified?
These tasks use simulated data because only there is the truth known.
Noise settings and data split. Under local heteroscedasticity, each region's noise changes with stage and reference regime. Under constant noise, it is fixed and everything else is identical. All methods are compared under both. The development split is the set of real participants kept out of training and used during method development.
We hope our answers resolve your questions. If anything remains unclear, please let us know; we would be very happy to continue the discussion.
We thank the reviewer for the careful and rigorous reading and for helping us identify the problems in our presentation. To answer the concerns in short: a modality is one imaging stream (amyloid PET, tau PET, or MRI-derived cortical volume), and a biomarker is one measured indicator within it; "which modality is leading now" asks which observed modality is changing the most relative to its own noise at the person's current position, not which biomarker will be impacted next; scan combinations in the simulation approximate the clinical frequencies, all methods received the same data and anchors, and we state where the generator favors CALIPER; finally, we revised the Introduction and Sections 4.1–4.2 to define the key terms, and we define every listed term under Weakness 1 (Part 3). The details follow, with the five questions first and then the two weaknesses.
Q1. Modality versus biomarker. We use modality for one stream of regional imaging measurements: amyloid PET, tau PET, or MRI-derived cortical volume as a proxy for neurodegeneration (the A, T, and N of the A/T/N framework; 31 regions each in the shared view). A biomarker is one measured indicator; regional tau uptake is one biomarker within the tau modality. An event-based model treats each biomarker as a separate event, whereas CALIPER assigns each modality one stage and compares modalities, not individual regional biomarkers.
Q2. What "which modality is leading now?" asks. The leading modality is the observed modality whose measurements move the most along the population current, the direction in which snapshots move as the disease advances, in units of that modality's own noise at its current stage (Eq. 10 and the paragraph after it in Section 3.5). It is a population-level quantity read at the person's position, not the person's per-year rate (Table 3 caption). Each term is defined under Weakness 1 (Part 3).
Consider a person midway along the disease clock whose amyloid is far advanced (near its plateau), tau moderately advanced, and cortical volume still close to normal. Along the population current, a small step forward implies the following (illustrative numbers in the spirit of the simulation):
Four questions have four different answers. Which modality is most advanced? Amyloid; this is stage, or burden, which staging methods report. Which has the largest raw change? Cortical volume, but the units are not comparable, and dividing by one pooled SD per modality gives an answer that depends on the pooled SD chosen, because noise itself varies with stage. Which will become abnormal next? Neurodegeneration; this is the ordering question of event-based models, a population sequence shared by everyone at this stage. Which is leading now? Tau: its movement is the largest relative to its own noise at this stage. Amyloid is most advanced but has stopped moving; volume moves a lot in raw units but within its noise.
Why this question matters clinically. Our framing follows the dynamic-biomarker model cited in Section 1 (Jack et al., 2010; 2013): amyloid, tau, and neurodegeneration markers become abnormal in sequence along sigmoid-shaped trajectories, so each marker's rate of change depends on where a person is in the course, and amyloid markers approach a plateau while tau and neurodegeneration markers are still changing. Two consequences follow. First, at a given visit the most abnormal marker is often no longer the one that is changing, so "which process is moving now" is the per-person form of the question that model answers for the population. Second, a marker carries prognostic information while it is still changing: in 1,431 participants from eight cohorts, baseline tau PET predicted subsequent cognitive decline better than amyloid PET or MRI (Ossenkoppele et al., 2021). A leading-modality readout generalizes this instead of assuming that tau is always the informative marker: it reports which observed modality is informative at this person's position, which is also the modality worth re-measuring at the next visit.
Reference not in the submission (Jack et al., 2010; 2013 are cited as in the paper)