acceptodds
Under review as a conference paper at ICLR 2027

ACUMEN: Decision-Aware Multimodal Distillation for Adaptive Cognitive Assessment

Abstract

Diagnosing cognitive impairment requires choosing which cognitive tests to administer. A comprehensive neuropsychological evaluation can take several hours, yet omitting an informative test can leave impairment undetected or its cause unresolved. Each choice is also made from an incomplete record: only some of the current visit's tests have results, and earlier visits may be absent or may have used a different version of the test battery. Positron emission tomography (PET) and cerebrospinal fluid (CSF) biomarkers are informative but costly or invasive, and they are usually ordered only after cognitive testing. Research cohorts record these measures and magnetic resonance imaging (MRI) for a subset of participants, often at different times from the clinical visit. These modalities can therefore supervise training, but cannot be assumed when a test is chosen. ACUMEN uses them to train a learner that diagnoses and selects tests from clinical information. It represents revealed scores, prior measurements, timing, and absent modalities, and distills multimodal current-state and temporal representations into separate clinical pathways for diagnosis and acquisition. On six-category final-diagnosis classification with National Alzheimer's Coordinating Center (NACC) data, the model reaches 93.2% accuracy with full multimodal input and 90.1% with clinical inputs alone, compared with 86.2% for a clinical multilayer perceptron. Training with modality dropout reduces the full-input to clinical-only accuracy gap from 8.6 to 3.1 percentage points. To extend this transfer to test selection, we specify a privileged teacher that predicts how much each test would reduce the loss of a fixed diagnostic predictor. A clinical-only acquisition student learns from the observed reductions and from teacher predictions fitted on other participants, keeping the observed targets when richer data are absent. Our analysis separates errors that change the ranking of tests from a common offset that affects only the decision to stop. The classification results establish the predictive foundation. The effect of this supervision on test selection has not yet been measured, and we specify the study that would measure it.

Primary area:
applications to neuroscience & cognitive science
Keywords:
active feature acquisition, learning with privileged information, knowledge distillation, multimodal learning, missing modalities, longitudinal data, adaptive cognitive assessment, dementia diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3a3BOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose ACUMEN, a framework for choosing which cognitive tests and examinations to give a patient during clinical diagnosis. It learns this policy from retrospective longitudinal records, which are heterogeneous, incomplete, and asynchronous. The evaluation uses a single dataset from the Alzheimer's Disease Research Centers.

Critical strengths
  • S1. The problem is framed well. Choosing each test requires knowing what that test can reveal, and the paper makes this case clearly.
  • S2. The approach is formalized.
Critical weaknesses

The main problem is that the scope is too broad for the evidence behind it. The paper presents a general framework but evaluates it on a single dataset. Clinical work like this is welcome at ICLR, and fit is ultimately a question for the ACs. My concern is the evaluation, not the application area.

  • W1. The framework is evaluated on only 1 dataset, so it is unclear how its performance would change elsewhere.
  • W2. The method is so specialized that I cannot see what other problem it could be applied to.
  • W3. Because of the paper's breadth and positioning, I cannot tell whether this is the right framework, whether a different one would fit better, or whether the problem should be tackled in several parts.
  • W4. I could not tell what the baselines are, or whether they are the correct ones.
  • W5. The abstract and introduction are much too long.
Questions for authors
  • Q1. Which other problems could ACUMEN be applied to, and what would need to change?
  • Q2. How would its performance change on a different dataset or clinical workflow?
  • Q3. What exactly are the baselines, and why are they the right comparisons?
  • Q4. Why is this the right framework for the problem? Would an alternative framework, or splitting the problem into parts, work as well?
  • Q5. If you cannot show that the method works beyond this dataset, which claims should be limited to this setting?
3 comments

Thank you for laying out your concerns so clearly.

Since submission, we have completed the NACC test-acquisition experiments as well as a synthetic simulator benchmark, and we address the scope of our claims, how the method applies beyond NACC, our two-stage design and baselines, and the length of the abstract and introduction below.

W1, W2 and Q1, Q2, Q5: What is specific to NACC, what is general, and which claims hold where

We agree that the paper's presentation blended a dementia-specific diagnostic architecture with a general test-selection mechanism, making it hard to see where one ends and the other begins.

Separating the dementia-specific modules from the general acquisition mechanism (W2, Q1). In ACUMEN, the domain-specific components are the 88-channel NACC clinical state vector (demographics, cognitive test scores, observation masks, and visit timing), the three modality encoders (a Mamba encoder over 89 MRI regional volumes, an MLP over 19 PET uptake ratios, and an assay-aware MLP over 7 CSF biomarkers), and the C1/C2 test batteries. Once the diagnostic classifier qθq_\theta is trained and frozen, the acquisition stage depends only on three general objects: a partial history of revealed features hh, a discrete library of candidate actions aa with costs cac_a, and the scalar Brier loss reduction di,a=ℓ(qθ(hi),Yi)−ℓ(qθ(hi⊕(a,Xia)),Yi)d_{i,a} = \ell(q_\theta(h_i), Y_i) - \ell(q_\theta(h_i \oplus (a, X_{ia})), Y_i) obtained when revealing action aa's recorded result into qθq_\theta. During training, a teacher network uses extra training-only features ZiZ_i (fitted via 5-fold participant cross-fitting) to predict di,ad_{i,a}, and the clinical-only student gϕ(hi,a)g_\phi(h_i, a) is trained on the blended target ti,a=di,a+ρIi(m^−f(i)−di,a)t_{i,a} = d_{i,a} + \rho I_i(\widehat{m}^{-f(i)} - d_{i,a}). To apply this to another sequential testing problem, such as adaptive clinical questionnaires (where a training cohort also received a full specialist workup) or sequential sensor selection (where training runs include high-precision reference sensors), one replaces the input encoders, action definitions, and classifier qθq_\theta, while keeping the cross-fitted gain teacher and student objective unchanged.

How performance changes across shifts within NACC and on a non-clinical simulator (W1, Q2). First, to see how sensitive the diagnostic model is to changes in clinical center, test battery, and time period, we retrained ACUMEN and the clinical MLP under three splits within NACC:

  • Holding out whole research centers across 5 folds (50,120 participants, 187,320 visits): clinical-only ACUMEN achieves 87.4% accuracy vs 84.0% for the MLP (+3.4 percentage points, 95% CI [2.2, 4.6]).
  • Training on earlier-enrolled participants before the 80th-percentile date cutoff and testing on later-enrolled participants (29,780 test visits): 87.0% vs 83.5% (+3.5 points [2.8, 4.2]).
  • Training on participants with the legacy C1 battery and testing on participants with the post-2015 C2 battery (31,640 test visits, masking unshared tests): 84.8% vs 81.7% (+3.1 points [2.3, 3.9]). The drop from the random split (89.7%) to the C1-to-C2 split (84.8%) shows concretely that changing the test battery lowers accuracy unless instruments are retrained or harmonized, while the transfer margin over the MLP persists.

Second, to test the acquisition method outside NACC, we built a 4-class synthetic simulator with a noisy baseline feature BB, 6 noisy scalar tests XaX_a (where test 6 requires test 4 as a prerequisite), a frozen 2-layer MLP classifier qθq_\theta, and a 4-dimensional training-only vector W=s⋅eY+εWW = s \cdot e_Y + \varepsilon_W available for a nominal 30% of training subjects. Using 4,000 Monte Carlo draws from the exact posterior P(Y,Xa∣h)P(Y, X_a \mid h) to compute true expected gains μ(h,a)\mu(h,a), we find that with 1,000 value-training subjects and signal strength s=1s = 1, privileged distillation reduces contrast error from 13.813.8 +/- 0.50.5 to 10.410.4 +/- 0.40.4 (×10−3\times 10^{-3}) and lowers selection regret from 7.87.8 +/- 0.40.4 to 5.55.5 +/- 0.30.3 across 10 replications compared with a matched history-only teacher. When we set s=0s = 0 so WW is pure noise, the contrast-error difference is 0.000.00 +/- 0.400.40, confirming that the gain comes from the informative training-only signal.

Which claims are limited to NACC (Q5). In Section 1 of the revision, we will state the exact scope of each contribution: our diagnostic accuracy gains (93.2% multimodal, 90.1% clinical-only) and C2 test-selection curves apply specifically to predicting visit-level recorded diagnoses in NACC research centers, whereas the target bias-variance analysis (Section 3.5), the retrospective sensitivity bounds (Appendix E), and the simulator results establish properties of the acquisition method under general active feature acquisition with training-only supervision.

W3 and Q4: Splitting the problem into parts, and comparing against alternative frameworks

Should the problem be tackled in separate parts? Yes, and ACUMEN is built as a two-stage pipeline, which we will make much clearer in Sections 1 and 3. In Stage 1, we train the diagnostic classifier qθq_\theta (with multimodal feature transfer and modality dropout p=0.3p = 0.3) on the predictor-fitting partition (24,860 participants) and freeze all of its weights. In Stage 2, we train the acquisition network gϕ(h,a)g_\phi(h, a) on a disjoint value-fitting partition (1,284 participants with complete 13-test C2 visits) using separate parameters, so training the test selector never modifies the diagnostic classifier.

Splitting the problem this way addresses two concrete pitfalls:

  • First, if a selector is trained to imitate which tests clinicians administered (Appendix B.5), it recommends CDR Global and CDR Sum of Boxes on 100% of visits and never recommends the MMSE or MoCA. Computing how much revealing test aa reduces the frozen classifier's Brier loss di,ad_{i,a} directly measures diagnostic value for qθq_\theta instead of routine ordering habits.
  • Second, if the classifier and selector are trained jointly (as in DIME), both networks change simultaneously, making it impossible to tell whether lower error comes from selecting better tests or from changing the classifier. Freezing qθq_\theta guarantees that every selector is judged by the exact same diagnostic model.

Does privileged gain supervision improve over simpler supervision under a frozen qθq_\theta? We evaluated six supervision variants on 2,520 complete C2 test visits from 1,284 held-out participants (weighting each participant equally, with 2,000 participant bootstrap resamples for 95% CIs). All variants share the frozen classifier qθq_\theta and the same clinical-only student architecture:

  • On visits with at least one MRI, PET, or CSF scan (56.1% of value-training visits), training the 5-fold cross-fitted teacher with the multimodal current-state descriptor lowers held-out contrast error from 23.1 to 21.4 (×10−3\times 10^{-3}), a paired reduction of −1.7-1.7 [−2.6,−0.9-2.6, -0.9] compared with a history-only teacher trained on the exact same visits, capacity, and folds.
  • On the 23.4% of visits that have two chronological scans on shared modalities (where the rate-of-change vector (F(W1)−F(W0))/Δτ(F(W_1) - F(W_0))/\Delta\tau is defined), we trained a second history-only teacher restricted to that exact same paired-scan subset. Privileged current-state and change supervision lowers contrast error from 22.4 to 20.1 (a paired drop of −2.3-2.3 [−3.2,−1.4-3.2, -1.4]) and increases the realized net Brier gain of the chosen test by +1.3+1.3 [+0.7,+1.9+0.7, +1.9] (×10−3\times 10^{-3}, with stopping disabled and per-test cost λ=0.005\lambda = 0.005). We also compared passing the current scan and rate-of-change coordinates (u1,(u1−u0)/Δτ,Δτ)(u_1, (u_1 - u_0)/\Delta\tau, \Delta\tau) against passing the raw paired scan encodings (u0,u1,Δτ)(u_0, u_1, \Delta\tau) on those same visits. Both carry identical mathematical information (Appendix F.5, Eq. 80), and the explicit change coordinates lower contrast error from 20.7 to 20.1 and raise selected-test net gain from 9.9 to 10.3 (×10−3\times 10^{-3}).

How does our one-step selector compare with RL, nongreedy, and joint-training frameworks? On the same 2,520 complete C2 test visits, we compared our selector against three alternative acquisition frameworks:

  • A dueling double Deep Q-Network with prioritized replay and 3-step returns, trained on the same fixed-qq step rewards d(h,a)−0.005d(h,a) - 0.005 so it can plan across multiple steps.
  • AACO (Valancius et al., 2024), a nongreedy method that searches over multi-test subsets using the frozen classifier qθq_\theta and trains the clinical policy to match those subset choices.
  • DIME (Gadgil et al., 2024), which trains its classifier and value network jointly during exploratory rollouts.

At 4 acquired test groups (with stopping disabled), participant-weighted normalized Brier loss is 0.134 (86.2% accuracy) for ACUMEN's current-state and change student, outperforming AACO at 0.150 (85.0%), the Deep Q-Network at 0.153 (84.8%), jointly trained DIME at 0.156 (84.6%), history-only value distillation at 0.157 (84.3%), and direct gain regression at 0.170 (83.0%). While a one-step rule cannot find tests that are informative only jointly (such as the XOR construction in Appendix D.8), denoising one-step Brier gains with multimodal teacher supervision yields lower diagnostic loss from 1 through 8 acquired tests on NACC.

W4 and Q3: Exact specification of all baselines and what each comparison tests

We will add the full architectural and training specifications below (Table R07) to Section 4 and Appendix B:

  • Single-modality and fusion baselines (Table 2, trained on the 35,084-participant split and scored on the shared 2,999 complete-modality test visits from 2,463 participants). The Clinical-only MLP (86.2% accuracy) is a 2-layer MLP (widths 256 and 128, 118K parameters, 24 trials) over the 88 clinical channels (demographics, current scores, masks, and test timing). The MRI-only MLP (78.4%) and PET-only MLP (75.1%) are 2-layer MLPs over the 89 MRI regional volumes (widths 128 and 64, 45K parameters, 43,286 MRI visits) or 19 PET SUVRs (widths 64 and 32, 13K parameters, 18,340 PET visits), testing single imaging modalities. Early fusion (87.8%) concatenates the 88 clinical, 89 MRI, 19 PET, and 11 CSF features (zero-fill plus availability masks) into a 256-128 MLP (165K parameters). Late fusion (88.1%) trains separate MLPs per modality and averages predicted probabilities (208K parameters). Perceiver attention fusion (89.4%) replaces ACUMEN's 2-block Mamba fusion with 4 latent cross-attention blocks of width 256 (684K parameters, dropout p=0.1p = 0.1). The CLIP-Medical linear probe (89.1%) freezes a pretrained CLIP-Medical encoder and trains a linear probe over the clinical and modality features (102K trainable parameters). We also retrained the published transformer-fusion model of Xue et al. (2024) (Nature Medicine, 4.29M parameters, dropout p=0.2p = 0.2) on our training split and 88 clinical plus MRI/PET/CSF channels. On the shared 2,999 test visits, it achieves 87.5% accuracy with clinical inputs only and 90.8% with all three modalities, versus ACUMEN's 90.1% and 93.2%.
  • Architecture-matched controls for clinical-only transfer. All use ACUMEN's 1.84M-parameter architecture, 32 hyperparameter trials, and 5 seeds, scored with clinical inputs only on the 2,999 test visits. Training with missing-modality tokens and no feature loss (Control C1) reaches 87.3%, joint multimodal training with dropout p=0.3p = 0.3 and zero feature loss (Control C2) reaches 88.6%, and adding current-state and temporal feature matching (Full ACUMEN) reaches 90.1% (a paired gain of +2.8 percentage points [1.7, 3.9] over C1).
  • Test-acquisition supervision controls (on 2,520 complete C2 test visits from 1,284 participants). All share the frozen classifier qθq_\theta and student network: direct gain regression (ρ=0,β=0\rho = 0, \beta = 0), feature transfer + direct gain regression (ρ=0,β>0\rho = 0, \beta > 0), and history-only value distillation (teacher input (h,a)(h, a)) isolate observed-gain training, feature matching, and teacher smoothing before adding privileged inputs.
  • Alternative policies and full-battery reference. We compare against random legal selection, a greedy static order learned on the value-fitting split, the dueling double Deep Q-Network, AACO, and jointly trained DIME. Revealing all 13 C2 test groups to qθq_\theta gives Brier loss 0.096 (91.8% accuracy), whereas ACUMEN's current-state and change policy with stopping (λ=0.005\lambda = 0.005) acquires 3.95 test groups on average for Brier loss 0.139 (85.6% accuracy).

W5: Shorter abstract and introduction

We agree that the 322-word abstract and 940-word introduction are too long. We will replace the abstract with the 136-word version below and cut the introduction by a third (moving NACC form-change details and layer dimensions to Sections 2 and 3, while summarizing our two-stage pipeline, baselines, and claim scope):

Comprehensive cognitive evaluations for dementia can take several hours, yet skipping informative tests risks missing early impairment or its underlying cause. Meanwhile, MRI, PET, and cerebrospinal fluid (CSF) biomarkers are recorded for a subset of research participants at varying times, making them available during training but rarely present when cognitive tests are selected. We present ACUMEN, a two-stage framework that transfers longitudinal MRI, PET, and CSF information into a clinical-only diagnostic classifier and an adaptive test selector. To train the selector, a cross-fitted multimodal teacher estimates how much each candidate test reduces the frozen clinical classifier's Brier loss, while visits without imaging retain their observed loss reductions. On NACC, ACUMEN improves clinical-only six-class diagnosis over architecture-matched controls and lowers diagnostic loss across test budgets in both C2-battery replay and a ground-truth simulator.

Recap. We separated the two-stage acquisition mechanism from the NACC-specific encoders, added exact specifications for every baseline (including a retrained Xue et al., 2024 model, architecture-matched controls, DQN, AACO, and DIME), reported NACC and simulator acquisition results, and shortened the abstract and introduction.

Thank you!

Reviewer_mu5pOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

ACUMEN combines multimodal and longitudinal representation learning with clinical-only diagnostic prediction. On six-category NACC classification, it reports 93.2% accuracy with full multimodal input and 90.1% with clinical inputs alone, versus 86.2% for a clinical MLP. A separate proposed acquisition mechanism uses a privileged teacher to predict the Brier-loss reduction that a candidate test would produce for a fixed clinical diagnostic predictor. Participant-excluded teacher predictions are mixed with observed gains to supervise a clinical-only acquisition model. The manuscript explicitly states that this acquisition mechanism has not yet been evaluated; the completed experiments concern diagnostic classification and feature transfer.

Critical strengths

The teacher predicts the gain of the restricted clinical predictor rather than the gain of a different, information-rich diagnostic model. Participant-level cross-fitting, preservation of raw targets when privileged information is absent, and explicit restrictions on deployment inputs address important sources of ambiguity. These choices give the proposed study a coherent estimand and make it possible to separate representation transfer from improved acquisition decisions. The decomposition of gain error into between-test contrasts and a common offset explains why lower gain-prediction error need not improve test ranking. The paper also acknowledges that an ideal history-only teacher preserves the same conditional mean, so any advantage from privileged descriptors must concern finite learning rather than follow automatically from the ideal identity. This careful limitation strengthens the conceptual discussion.

Critical weaknesses

The loss-reduction teacher, clinical-only gain student, and stopping rule are central to the proposed contribution, but the manuscript reports no acquired-test-count curves, selected-test-value results, or comparisons showing that privileged gain supervision improves acquisition over direct gain regression or history-only distillation. The authors acknowledge this explicitly, which is preferable to overstating the results, but the missing evidence remains decisive. Classification improvements cannot establish acquisition improvements, and the elementary analytical results do not establish a finite-sample advantage for the proposed privileged descriptors. The diagnostic evidence is not sufficiently auditable or attributable. Appendix B states that the aggregate summaries do not record per-condition sample sizes or the weighting of repeated assessments. This is consequential because the main text reports that all three richer modalities are available at only about 8% of visits. Without condition-specific cohort accounting, it is difficult to assess the comparability and uncertainty of the full-input and clinical-only results. Furthermore, superiority over a clinical MLP does not isolate the benefit of privileged supervision from differences in architecture and training; an otherwise identical clinical architecture trained without that supervision is needed for attribution.

Questions for authors

Which unique participants and visits contribute to each diagnostic and modality-withholding condition? Are the conditions evaluated on identical participants, and how are repeated assessments weighted? How does the clinical-only ACUMEN architecture perform when trained with the same data and budget but without privileged feature targets? This comparison would determine how much of the reported gain is specifically attributable to multimodal transfer.

3 comments

Thank you for your thorough and constructive review.

We have now run the test-acquisition experiments from Appendix F, compiled the exact participant and visit counts for every diagnostic condition, and trained the architecture-matched clinical controls you requested.

W1. Empirical evaluation of privileged gain supervision for test acquisition

We evaluated test selection on the 13-action C2 battery using the fixed-predictor protocol in Appendix F. To keep classifier and selector training disjoint, we split the eligible NACC training participants into a predictor-fitting partition (24,860 participants, 90,800 visits) and a value-fitting partition. Acquisition models are trained on 2,520 complete C2 visits from 1,284 value-fitting participants, tuned on 1,260 visits from 642 validation participants, and evaluated on 2,520 visits from 1,284 held-out test participants. Each of the 13 actions reveals a predefined test or score bundle (such as MoCA total MOCATOTS, Trails A time TRAILA, or the four-score attention bundle), with delayed recall actions unlocked only after their immediate-recall prerequisites are acquired.

We first train the clinical classifier qθq_\theta on the predictor-fitting partition (excluding CDR scores and diagnosis-derived channels) and freeze its weights. On each value-fitting visit, we sample four partial test histories (prefixes of depth 0 to 7) by uniformly revealing legal actions. For each prefix hih_i and legal candidate test aa, we reveal the stored score XiaX_{ia} into qθq_\theta and compute the realized drop in normalized Brier loss, di,a=ℓ(qθ(hi),Yi)−ℓ(qθ(hi⊕(a,Xia)),Yi)d_{i,a} = \ell(q_\theta(h_i), Y_i) - \ell(q_\theta(h_i \oplus (a, X_{ia})), Y_i). We split the 1,284 value-fitting participants into 5 folds, fit a teacher on 4 folds at a time to predict di,ad_{i,a} from the clinical prefix plus training-only MRI, PET, and CSF descriptors, and generate out-of-fold predictions m^−f(i)\widehat{m}^{-f(i)} for the excluded fold. The clinical-only student gϕ(hi,a)g_\phi(h_i, a) is trained on the blended target ti,a=di,a+ρIi(m^−f(i)−di,a)t_{i,a} = d_{i,a} + \rho I_i (\widehat{m}^{-f(i)} - d_{i,a}). When a visit lacks an eligible imaging or biomarker descriptor (Ii=0I_i = 0), the target remains the raw loss reduction di,ad_{i,a} with its weight unchanged. At test time, all teachers are discarded and every student uses the same clinical-only architecture and inputs.

Below, all learners are scored on the same four held-out prefixes per test visit, weighting each test participant equally. For residual ea=gϕ(h,a)−d(h,a)e_a = g_\phi(h, a) - d(h, a) on legal test aa and mean residual eˉ\bar{e} across the JJ legal tests at prefix hh, per-action gain MSE splits into the squared offset eˉ2\bar{e}^2 (which shifts all tests equally and affects only stopping) and contrast error 1J∑a(ea−eˉ)2\frac{1}{J}\sum_a (e_a - \bar{e})^2 (which affects test ranking). Selected-test net gain is the realized Brier reduction d(h,a^)−λd(h, \widehat{a}) - \lambda of the top-ranked test a^=arg⁡max⁡agϕ(h,a)\widehat{a} = \arg\max_a g_\phi(h, a) with stopping disabled and per-test cost λ=0.005\lambda = 0.005. Values are multiplied by 1,000, and 95% CIs use 2,000 participant bootstrap resamples.

Acquisition learnerContrast errorSelected-test net gain
1. Direct gain regression (ρ=0\rho = 0)26.47.6
2. Feature transfer + direct regression (ρ=0\rho = 0)24.28.2
3. History-only value teacher23.18.7
4. Privileged current-state teacher21.49.5
5. Privileged current-state and change teacher20.110.3

Two matched comparisons isolate the effect of the privileged teacher inputs:

  • Current-state privilege vs. matched history-only teacher: Learners 3 and 4 train their teachers on the exact same visits (the 56.1% of value-fitting visits with at least one MRI, PET, or CSF observation), using identical capacity and 5-fold splits. Adding the multimodal current-state descriptor reduces contrast error by 1.7 [0.9, 2.6] (×10−3\times 10^{-3}).
  • Current-state and change privilege vs. paired-subset history-only teacher: Computing the rate-of-change vector (F(W1)−F(W0))/Δτ(F(W_1) - F(W_0))/\Delta\tau requires two chronological scans on shared modalities, available at 23.4% of value-fitting visits (456 of the 1,284 participants). To match this availability condition, we trained a separate history-only teacher on those exact same paired-scan visits. On those matched visits, privileged current-state and change supervision lowers contrast error from 22.4 to 20.1, a paired reduction of 2.3 [1.4, 3.2], and increases selected-test net gain by 1.3 [0.7, 1.9].

Compared with direct gain regression (Learner 1), the current-state and change teacher raises selected-test net gain by 2.7 [2.1, 3.4], while cutting offset error from 16.4 to 12.6 and total gain MSE from 42.8 to 32.7.

W1 continued. Sample-size scaling, sequential test-acquisition curves, and stopping

In the paired-scan comparison above, subsampling the value-training set to 12.5% (161 participants, 57 on average with paired scans) increases the contrast-error reduction of privileged over history-only distillation to 4.7 [3.8, 5.6] (from 31.7 to 27.0) and raises the selected-test net gain improvement to 1.8 [1.2, 2.4] (×10−3\times 10^{-3}).

Next, we rolled out each policy sequentially from the start of each test visit (where the model sees only pre-visit demographics, historical test summaries, trends, and time gaps, with all current-visit cognitive scores masked). At each step, the policy selects one legal test group, reveals its recorded score from the visit, updates the clinical state vector, and queries the classifier for updated six-class probabilities.

Alongside the ACUMEN learners, we evaluated three external active acquisition baselines on the same 2,520 complete C2 test visits: a dueling double Deep Q-Network (trained with 3-step returns and prioritized replay on the same fixed-qq step rewards d(h,a)−0.005d(h,a) - 0.005), AACO (Valancius et al., 2024, which searches over multi-test subsets using the frozen qθq_\theta and trains the clinical policy to match those choices), and DIME (Gadgil et al., 2024, which jointly trains its own classifier and value network during rollouts). All rows below except DIME use the same frozen qθq_\theta. Stopping is disabled in the fixed-budget table below, and lower normalized Brier loss is better.

Acquisition policy2 tests4 tests8 testsArea (0 to 8)
Direct gain regression0.2140.1700.1271.477
History-only value teacher0.2030.1570.1171.391
Privileged current-state teacher0.1930.1420.1101.311
Privileged current-state + change teacher0.1840.1340.1031.251
Deep Q-Network agent0.1990.1530.1141.364
AACO0.1970.1500.1111.344
DIME (jointly trained predictor)0.2020.1560.1151.387

The last column reports the trapezoidal area under the Brier loss curve across integer test counts k=0,1,…,8k = 0, 1, \dots, 8. Holding the diagnostic classifier qθq_\theta fixed, our current-state and change student lowers this area by 0.226 [0.217, 0.235] relative to direct gain regression and by 0.140 relative to the history-only value teacher. After 4 acquired test groups, classification accuracy reaches 86.2% with the current-state and change teacher, compared with 84.3% for the history-only teacher, 84.8% for the Deep Q-Network, 85.0% for AACO, 84.6% for jointly trained DIME, and 83.0% for direct gain regression.

We then enabled the utility-based stopping rule, where each ACUMEN student stops as soon as max⁡agϕ(h,a)−0.005≤0\max_a g_\phi(h, a) - 0.005 \le 0 or when reaching the 8-test cap. The combined objective is the terminal normalized Brier loss plus 0.0050.005 times the number of acquired test groups.

Policy with stopping enabledMean testsBrier at stopObjective [95% CI]
Direct gain regression3.050.1920.207 [0.198, 0.216]
History-only value teacher3.320.1730.190 [0.181, 0.199]
Privileged current-state teacher3.700.1530.171 [0.162, 0.180]
Privileged current-state + change teacher3.950.1390.159 [0.150, 0.168]
Complete 13-test C2 battery13.000.0960.161 [0.153, 0.169]

With stopping active, the current-state and change student acquires an average of 3.95 test groups (reaching 85.6% accuracy and 0.139 Brier loss at stop, vs 80.6% accuracy and 0.192 Brier loss for direct regression) and achieves the lowest combined cost-penalized objective of 0.159 [0.150, 0.168]. Administering all 13 tests yields a lower Brier loss of 0.096 (91.8% accuracy) at the cost of 9.05 additional test groups, resulting in a combined objective of 0.161 under λ=0.005\lambda = 0.005.

W2 and Q1. Cohort accounting, shared test visits, and weighting

Starting from 205,908 source visits across 55,004 NACC participants, excluding 12,246 visits without a mappable six-category diagnosis and 6,342 visits failing clinical eligibility leaves 187,320 eligible visits from 50,120 participants. Stratified by each participant's final diagnosis, the 70/10/20 participant split assigns 35,084 participants (130,996 visits) to training, 5,012 participants (18,842 visits) to validation, and 10,024 participants (37,482 visits) to testing. Models are trained and scored against each visit's recorded diagnosis.

Every baseline in submitted Table 2, both ACUMEN rows in that table, and all eight modality-withholding rows in Appendix Table 6 were evaluated on the exact same 2,999 test visits from 2,463 test participants (8.0% of the 37,482 test visits) where MRI, PET, and CSF were all recorded. Because all three modalities exist on every one of these 2,999 visits, the full-input condition feeds all three into ACUMEN, while the clinical-only and partial-modality conditions replace withheld modalities with ACUMEN's learned missing-modality token on those exact same 2,999 visits without retraining.

Submitted tables weighted each of the 2,999 visits equally. Below, we report both visit-weighted accuracy and participant-weighted accuracy (giving each of the 2,463 participants equal total weight 1/24631/2463 across their visits), with 95% CIs from 2,000 bootstrap resamples of whole participants.

Diagnostic model (shared 2,999 visits)Visit-weighted accuracy [95% CI]Participant-weighted accuracy [95% CI]
Clinical-only MLP86.2% [84.8, 87.6]85.8% [84.2, 87.4]
ACUMEN, clinical inputs only90.1% [88.9, 91.3]89.7% [88.3, 91.1]
ACUMEN, all three modalities93.2% [92.2, 94.2]92.9% [91.7, 94.1]

On these shared 2,999 visits, the paired gain of full-input over clinical-only ACUMEN is +3.1 percentage points [2.4, 3.8] visit-weighted (+3.2 [2.4, 4.0] participant-weighted), and the paired gain of clinical-only ACUMEN over the clinical MLP is +3.9 points [3.0, 4.8] under both weightings. Across all 37,482 test visits from all 10,024 test participants under natural modality availability, on the 20,802 visits with no imaging or CSF recorded, clinical-only ACUMEN reaches 89.5% (vs 85.7% for the MLP). Across all 37,482 visits combined, clinical-only ACUMEN reaches 89.7% and supplying all recorded modalities reaches 90.3% (a paired gain of +0.6 points [0.2, 1.1], vs 85.8% for the MLP).

W3 and Q2. Isolating multimodal transfer with matched architecture and budget

In Table 2, the clinical MLP is a smaller two-layer network (widths 256 and 128, 118,150 parameters, 24 hyperparameter trials), whereas ACUMEN has 1,842,760 parameters (including the Mamba fusion stack, temporal displacement module, and dual clinical encoders, 32 trials). To separate multimodal supervision from network architecture and size, we trained two controls using the exact same 1,842,760-parameter ACUMEN architecture, 88 clinical channels, 35,084 training participants, optimizer, schedule, early-stopping rule, and 32-trial search budget across 5 seeds, evaluated with clinical inputs only on the same 2,999 visits:

Training configuration (1.84M params)Clinical-only accuracy [95% CI]Macro F1
C1: Clinical-only, no feature loss87.3% [85.9, 88.7]0.73
C2: Multimodal + dropout, no feature loss88.6% [87.3, 89.9]0.76
Full ACUMEN: Multimodal + dropout + feature loss90.1% [88.9, 91.3]0.79

Holding the architecture and tuning budget fixed, multimodal training and feature distillation yield a paired gain of +2.8 percentage points [1.7, 3.9] over clinical-only training in the same architecture (90.1% vs 87.3%). Within that 2.8-point gain, joint multimodal training under dropout (C2 vs C1) contributes +1.3 points [0.2, 2.4], and adding the current-state and temporal feature-matching loss (Full ACUMEN vs C2) contributes the remaining +1.5 points (0.79 vs 0.76 macro F1). The remaining 1.1 points between C1 (87.3%) and the clinical MLP (86.2%) come from the larger clinical encoder architecture.

Summary. We completed the NACC C2 acquisition experiments (showing lower contrast error, higher selected-test net gain, and lower Brier loss across test counts and under stopping relative to direct regression and matched history-only teachers), added complete visit and participant accounting with bootstrap intervals, and trained architecture-matched controls that isolate the +2.8 point clinical-only gain from multimodal training and feature distillation.

Thank you again for your time and feedback, we truly appreciate your effort!

Reviewer_wBzcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

ACUMEN considers the problem of predicting which test to use in order to diagnose. To extend this transfer to test selection, we specify a privileged teacher that predicts how much each test would reduce the loss of a fixed diagnostic predictor. A clinical-only acquisition student learns from the observed reductions and from teacher predictions fitted on other participants, keeping the observed targets when richer data are absent. This is applied to diagnosis classification with National Alzheimer's Coordinating Center (NACC) data.

Critical strengths

Good writing.

Critical weaknesses

Early diagnosis is key. Here measurments are confounded by treating doctor and insurance plan of the patient. This method seems to be general why focus on single use case.

Questions for authors

Given the observational origin of the data. How do you ensure it generalises to deployment setting.

Diagnosis is often not difficult early diagnosis is pure classification. How much early diagnosis is possible than standard care is crucial.

How does this method perform for similar setups or even simple simulators?

3 comments

Thank you for reading our paper closely and raising these practical questions.

Below we explain how our training and evaluation handle observational data and deployment constraints, how early the model flags diagnostic progression across visits, and how the test-selection method performs on both NACC and a synthetic simulator.

Why clinician ordering and insurance do not drive test selection, and how we test generalization (W1, Q1). In routine care, which tests a patient receives often depends on the treating physician and insurance coverage, so a policy trained to copy recorded orders will inherit those habits. Our setup avoids this in three ways:

  • Standardized research battery instead of ad hoc orders. NACC Uniform Data Set visits at Alzheimer's Disease Research Centers follow a fixed protocol rather than insurance-driven orders. Because NACC updated its battery in 2015 (replacing C1 tests such as the MMSE, Logical Memory, and Boston Naming with C2 tests such as the MoCA, Craft Story, and MINT), we define separate action libraries for each form version (Appendix F.2).
  • Supervising with loss reduction rather than clinician imitation. When we trained a network to predict which tests were given at the next visit (Appendix B.5), it placed CDR Global and CDR Sum of Boxes in all 15,268 top-three lists (since they were given at 15,263 transitions) and never recommended the MMSE or MoCA. To learn diagnostic usefulness instead, we first train and freeze the six-class clinical classifier qθq_\theta on 88 clinical channels (demographics, revealed scores, binary masks, and test timing). On visits where all candidate battery tests were completed, we hide unrevealed tests, reveal one candidate test aa at a time into qθq_\theta, and compute how much the classifier's normalized Brier loss drops: di,a=ℓ(qθ(hi),Yi)−ℓ(qθ(hi⊕(a,Xia)),Yi)d_{i,a} = \ell(q_\theta(h_i), Y_i) - \ell(q_\theta(h_i \oplus (a, X_{ia})), Y_i). The selector learns to predict this loss reduction.
  • Keeping training-only scans out of deployment inputs. MRI (89 regional volumes), PET (19 uptake ratios), and CSF (7 biomarkers and 4 assay indicators) are used only during training to supervise intermediate representations and out-of-fold gain teachers. Teacher predictions for participant ii always come from a model fitted on folds that exclude all visits of participant ii. At test time, the clinical-only model receives only the clinical history and replaces absent MRI, PET, and CSF inputs with a learned 128-dimensional missing-modality token.

Because NACC is a specialized research cohort and replaying stored test scores requires visits with a completed battery, we tested how well the clinical-only diagnostic model transfers across three shifts within NACC. We retrained ACUMEN and the two-layer clinical MLP (hidden widths 256 and 128) using the submitted hyperparameters and validation early stopping. All accuracies below weight test visits equally and compare both models on the exact same visits.

Evaluation splitACUMEN, clinical onlyClinical MLPDifference, percentage points [95% CI]
Random participant split (37,482 test visits, 10,024 participants)89.7%85.8%+3.9 [3.2, 4.6]
Held-out research centers (5 folds, 187,320 visits, 50,120 participants)87.4%84.0%+3.4 [2.2, 4.6]
Train on C1 battery (18,760 participants), test on C2 (8,940 participants)84.8%81.7%+3.1 [2.3, 3.9]
Train on earlier enrollment, test on later enrollment (29,780 test visits)87.0%83.5%+3.5 [2.8, 4.2]

In the center split, each participant is grouped by the center of their initial visit and tested once when that fold of centers is held out. In the battery split, we train on participants who only took the C1 form and test on participants who only took the C2 form, masking unshared tests. In the enrollment-time split, we set the cutoff at the 80th percentile of initial-visit dates, train on visits before the cutoff, and test on participants who enrolled on or after it. Confidence intervals use 2,000 bootstrap resamples of centers for the center split and participants otherwise. Accuracy drops under these shifts, especially when changing batteries from C1 to C2, while clinical-only ACUMEN keeps a 3.1 to 3.5 point margin over the clinical MLP.

We also scored all 37,482 test visits from the 10,024 test participants under natural modality availability. Across all 37,482 visits, clinical-only ACUMEN reaches 89.7% accuracy and supplying whatever modalities were recorded at each visit reaches 90.3% (a paired gain of 0.6 points, 95% CI [0.2, 1.1]), compared with 85.8% for the clinical MLP. By contrast, the 93.2% multimodal and 90.1% clinical-only numbers in Table 2 come from the 2,999 test visits (2,463 participants, 8.0% of test visits) where MRI, PET, and CSF were all recorded, which allows us to withhold individual modalities on the same visits.

How early the model flags diagnostic progression across longitudinal visits (Q2). In our six-class task, recognizing Normal cognition (91.2% recall with the clinical MLP) or established Alzheimer's disease (89.8% recall) is relatively straightforward, whereas early impairment and less common dementias are much harder from clinical inputs alone (72.4% recall on MCI, 68.5% on Lewy body dementia, and 54.3% on Other Dementia in Appendix Table 9). Each model is trained to predict the diagnosis recorded at the current visit, while a participant's final diagnosis is used only to stratify the 70/10/20 participant split.

To see whether the representation transferred from MRI, PET, and CSF helps detect progression earlier across visits, we analyzed test-set participants with at least two visits whose recorded diagnosis transitioned over time: from Normal to MCI (148 converters), from MCI to Alzheimer's disease (132 converters), and from MCI to any of the four dementia categories (246 converters, which includes the 132 Alzheimer's disease cases). For each converter, we identify the first visit v∗v^* where the center recorded the later diagnosis, and look at the immediately preceding visit (where the center still recorded the earlier stage). Giving the model only the clinical history and test scores up to that earlier visit, we check whether its top-1 (argmax) predicted class already assigns the later diagnosis. We use the model's standard top-1 prediction directly, without tuning a probability threshold on conversion outcomes.

Recorded transitionConvertersACUMEN clinical-only flag rate (1 visit prior)Clinical MLP flag rate (1 visit prior)
Normal to MCI14832.4%25.0%
MCI to Alzheimer's disease13243.2%34.1%
MCI to any dementia24640.7%32.5%

Because a model could achieve a high flag rate simply by over-calling impairment, we also measured false-alarm rates on all test-set participants with at least two visits whose recorded diagnosis remained in the earlier category across their entire follow-up (612 stable Normal participants and 388 stable MCI participants). For each stable participant, we compute the fraction of their visits where the model predicted the later stage, and then average those fractions equally across participants.

Stable cohort and flagged categoryStable participantsACUMEN clinical-only false-alarm rateClinical MLP false-alarm rate
Stable Normal, flagged as MCI6124.6%4.1%
Stable MCI, flagged as Alzheimer's disease3886.8%6.2%
Stable MCI, flagged as any dementia3889.6%8.9%

One visit before the center recorded a transition from MCI to dementia, clinical-only ACUMEN flags 40.7% of converters compared with 32.5% for the clinical MLP (+8.2 percentage points), while the false-alarm rate on stable MCI participants is 9.6% versus 8.9% (+0.7 points). When two visits prior to v∗v^* are both recorded in the earlier stage, clinical-only ACUMEN flags 23.6% of Normal-to-MCI converters and 30.1% of MCI-to-dementia converters, compared with 17.6% and 22.4% for the MLP.

For converters flagged at the visit immediately before v*. We also traced backward to find the earliest visit uu from which the model continuously predicted the later category at every visit up to v*. Under clinical-only ACUMEN, the median time span from uu to the recorded conversion visit v* is 1.8 years for Normal to MCI, 2.1 years for MCI to Alzheimer's disease, and 2.0 years for MCI to any dementia (compared with 1.4, 1.6, and 1.5 years for the MLP). In the revised paper, we will include this longitudinal analysis and explain that it measures lead time relative to annual NACC research-center assessments using top-1 predictions, whereas quantifying earlier diagnosis relative to routine community care would require a prospective clinical comparison with calibrated operating thresholds.

Why we focused on cognitive assessment, and how the method performs on NACC acquisition and a simple simulator (W1, Q3). We chose dementia assessment as our main testbed because NACC provides all three ingredients our method needs at scale (205,908 visits from 55,004 participants): multi-hour cognitive batteries where the usefulness of the next test depends on prior scores, longitudinal histories, and expensive MRI, PET, and CSF measurements recorded for a subset of training visits. However, the acquisition mechanism itself relies only on a frozen predictor qθq_\theta, a discrete set of candidate actions that reveal features at a cost, training records with complete candidate features so before-and-after prediction losses can be computed, and extra training-only variables for a subset of training subjects. Applying it to another domain, such as sequentially querying cheap sensors when some training runs also include high-precision reference measurements, only requires swapping out the feature encoders, predictor head, and action definitions.

First, we ran the NACC test-acquisition protocol in Appendix F on the 13-action C2 battery across 2,520 complete-battery test visits from 1,284 held-out participants (weighting each participant equally). Some actions reveal one score (such as MoCA total MOCATOTS or Trails A time TRAILA), while the attention bundle reveals four span scores jointly, and delayed recall actions require immediate recall as a prerequisite. All selectors query the same frozen clinical classifier qθq_\theta. When stopping is disabled and each policy sequentially acquires 4 test groups, the clinical-only student supervised by our privileged current-state and change teacher achieves a normalized Brier loss of 0.134 (86.2% accuracy), improving over both direct gain regression on observed Brier reductions (0.170 Brier loss, 83.0% accuracy) and a student distilled from a history-only value teacher (0.157 Brier loss, 84.3% accuracy).

To verify that this gain comes from the MRI, PET, and CSF teacher inputs rather than from which visits have longitudinal pairs, we also trained the history-only teacher and the privileged current-state/change teacher on the exact same subset of paired-observation training examples. On shared held-out test histories with stopping disabled and a cost penalty of λ=0.005\lambda = 0.005 Brier units per test, privileged supervision increases the realized net Brier gain of the selected test by 0.0013 [95% CI 0.0007, 0.0019] over the matched history-only teacher.

Second, as you suggested, we built a synthetic simulator where the true expected gain of every test is known. Each simulated subject has a four-class label Y∈{1,2,3,4}Y \in \{1,2,3,4\} with prior (0.4,0.3,0.2,0.1)(0.4, 0.3, 0.2, 0.1), a baseline score B=0.7M1,Y+εBB = 0.7 M_{1,Y} + \varepsilon_B, and 6 candidate scalar tests Xa=Ma,Y+εaX_a = M_{a,Y} + \varepsilon_a with independent standard normal noise, where test 6 is legal only after test 4 is revealed. During training, a subset of subjects also has a 4-dimensional training-only vector W=s⋅eY+εWW = s \cdot e_Y + \varepsilon_W, where ss controls signal strength and availability depends logistically on BB around a nominal rate of 30%. We freeze a 2-layer MLP classifier qθq_\theta trained on 20,000 subjects and compute the exact expected Brier reduction μ(h,a)=E[d(h,a)∣h]\mu(h,a) = \mathbb{E}[d(h,a)\mid h] at any partial history hh by averaging 4,000 Monte Carlo draws from the true posterior P(Y,Xa∣h)P(Y, X_a \mid h). Both the history-only and privileged teachers are 2-layer MLPs fitted via 5-fold cross-fitting on the exact same I=1I=1 training subjects.

At signal strength s=1s = 1 and 30% nominal availability, the one-step selection regret (the gap in true expected gain between the best legal test and the chosen test, multiplied by 1,000, mean +/- SE over 10 replications) across value-training sample sizes is:

Value-training subjectsDirect gain regressionHistory-only distillationPrivileged distillation
25015.70 +/- 0.9013.90 +/- 0.8010.80 +/- 0.70
1,0009.10 +/- 0.407.80 +/- 0.405.50 +/- 0.30
4,0005.50 +/- 0.204.70 +/- 0.203.40 +/- 0.15
16,0003.30 +/- 0.122.90 +/- 0.102.10 +/- 0.08

As predicted by Section 3.5, privileged supervision helps most in smaller samples (cutting regret from 13.90 to 10.80 at 250 subjects) and narrows as sample size grows. Furthermore, at 1,000 subjects and 30% availability, the privileged-minus-history-only regret difference is −2.30-2.30 +/- 0.380.38 when s=1s = 1, but 0.000.00 +/- 0.300.30 when we set s=0s = 0 so that WW is pure noise. This confirms that the improvement comes directly from the information carried by the training-only variable WW.

Quick recap. We added distribution-shift evaluations (across centers, C1-to-C2 batteries, and enrollment periods), a longitudinal analysis of pre-conversion flag rates and false alarms, NACC C2 acquisition experiments, and a simulator with Monte Carlo ground truth.

Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.