- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose ACUMEN, a framework for choosing which cognitive tests and examinations to give a patient during clinical diagnosis. It learns this policy from retrospective longitudinal records, which are heterogeneous, incomplete, and asynchronous. The evaluation uses a single dataset from the Alzheimer's Disease Research Centers.
Critical strengths
- S1. The problem is framed well. Choosing each test requires knowing what that test can reveal, and the paper makes this case clearly.
- S2. The approach is formalized.
Critical weaknesses
The main problem is that the scope is too broad for the evidence behind it. The paper presents a general framework but evaluates it on a single dataset. Clinical work like this is welcome at ICLR, and fit is ultimately a question for the ACs. My concern is the evaluation, not the application area.
- W1. The framework is evaluated on only 1 dataset, so it is unclear how its performance would change elsewhere.
- W2. The method is so specialized that I cannot see what other problem it could be applied to.
- W3. Because of the paper's breadth and positioning, I cannot tell whether this is the right framework, whether a different one would fit better, or whether the problem should be tackled in several parts.
- W4. I could not tell what the baselines are, or whether they are the correct ones.
- W5. The abstract and introduction are much too long.
Questions for authors
- Q1. Which other problems could ACUMEN be applied to, and what would need to change?
- Q2. How would its performance change on a different dataset or clinical workflow?
- Q3. What exactly are the baselines, and why are they the right comparisons?
- Q4. Why is this the right framework for the problem? Would an alternative framework, or splitting the problem into parts, work as well?
- Q5. If you cannot show that the method works beyond this dataset, which claims should be limited to this setting?
3 comments
Part 2
W3 and Q4: Splitting the problem into parts, and comparing against alternative frameworks
Should the problem be tackled in separate parts? Yes, and ACUMEN is built as a two-stage pipeline, which we will make much clearer in Sections 1 and 3. In Stage 1, we train the diagnostic classifier (with multimodal feature transfer and modality dropout ) on the predictor-fitting partition (24,860 participants) and freeze all of its weights. In Stage 2, we train the acquisition network on a disjoint value-fitting partition (1,284 participants with complete 13-test C2 visits) using separate parameters, so training the test selector never modifies the diagnostic classifier.
Splitting the problem this way addresses two concrete pitfalls:
- First, if a selector is trained to imitate which tests clinicians administered (Appendix B.5), it recommends CDR Global and CDR Sum of Boxes on 100% of visits and never recommends the MMSE or MoCA. Computing how much revealing test reduces the frozen classifier's Brier loss directly measures diagnostic value for instead of routine ordering habits.
- Second, if the classifier and selector are trained jointly (as in DIME), both networks change simultaneously, making it impossible to tell whether lower error comes from selecting better tests or from changing the classifier. Freezing guarantees that every selector is judged by the exact same diagnostic model.
Does privileged gain supervision improve over simpler supervision under a frozen ? We evaluated six supervision variants on 2,520 complete C2 test visits from 1,284 held-out participants (weighting each participant equally, with 2,000 participant bootstrap resamples for 95% CIs). All variants share the frozen classifier and the same clinical-only student architecture:
- On visits with at least one MRI, PET, or CSF scan (56.1% of value-training visits), training the 5-fold cross-fitted teacher with the multimodal current-state descriptor lowers held-out contrast error from 23.1 to 21.4 (), a paired reduction of [] compared with a history-only teacher trained on the exact same visits, capacity, and folds.
- On the 23.4% of visits that have two chronological scans on shared modalities (where the rate-of-change vector is defined), we trained a second history-only teacher restricted to that exact same paired-scan subset. Privileged current-state and change supervision lowers contrast error from 22.4 to 20.1 (a paired drop of []) and increases the realized net Brier gain of the chosen test by [] (, with stopping disabled and per-test cost ). We also compared passing the current scan and rate-of-change coordinates against passing the raw paired scan encodings on those same visits. Both carry identical mathematical information (Appendix F.5, Eq. 80), and the explicit change coordinates lower contrast error from 20.7 to 20.1 and raise selected-test net gain from 9.9 to 10.3 ().
How does our one-step selector compare with RL, nongreedy, and joint-training frameworks? On the same 2,520 complete C2 test visits, we compared our selector against three alternative acquisition frameworks:
- A dueling double Deep Q-Network with prioritized replay and 3-step returns, trained on the same fixed- step rewards so it can plan across multiple steps.
- AACO (Valancius et al., 2024), a nongreedy method that searches over multi-test subsets using the frozen classifier and trains the clinical policy to match those subset choices.
- DIME (Gadgil et al., 2024), which trains its classifier and value network jointly during exploratory rollouts.
At 4 acquired test groups (with stopping disabled), participant-weighted normalized Brier loss is 0.134 (86.2% accuracy) for ACUMEN's current-state and change student, outperforming AACO at 0.150 (85.0%), the Deep Q-Network at 0.153 (84.8%), jointly trained DIME at 0.156 (84.6%), history-only value distillation at 0.157 (84.3%), and direct gain regression at 0.170 (83.0%). While a one-step rule cannot find tests that are informative only jointly (such as the XOR construction in Appendix D.8), denoising one-step Brier gains with multimodal teacher supervision yields lower diagnostic loss from 1 through 8 acquired tests on NACC.
Part 3
W4 and Q3: Exact specification of all baselines and what each comparison tests
We will add the full architectural and training specifications below (Table R07) to Section 4 and Appendix B:
- Single-modality and fusion baselines (Table 2, trained on the 35,084-participant split and scored on the shared 2,999 complete-modality test visits from 2,463 participants). The Clinical-only MLP (86.2% accuracy) is a 2-layer MLP (widths 256 and 128, 118K parameters, 24 trials) over the 88 clinical channels (demographics, current scores, masks, and test timing). The MRI-only MLP (78.4%) and PET-only MLP (75.1%) are 2-layer MLPs over the 89 MRI regional volumes (widths 128 and 64, 45K parameters, 43,286 MRI visits) or 19 PET SUVRs (widths 64 and 32, 13K parameters, 18,340 PET visits), testing single imaging modalities. Early fusion (87.8%) concatenates the 88 clinical, 89 MRI, 19 PET, and 11 CSF features (zero-fill plus availability masks) into a 256-128 MLP (165K parameters). Late fusion (88.1%) trains separate MLPs per modality and averages predicted probabilities (208K parameters). Perceiver attention fusion (89.4%) replaces ACUMEN's 2-block Mamba fusion with 4 latent cross-attention blocks of width 256 (684K parameters, dropout ). The CLIP-Medical linear probe (89.1%) freezes a pretrained CLIP-Medical encoder and trains a linear probe over the clinical and modality features (102K trainable parameters). We also retrained the published transformer-fusion model of Xue et al. (2024) (Nature Medicine, 4.29M parameters, dropout ) on our training split and 88 clinical plus MRI/PET/CSF channels. On the shared 2,999 test visits, it achieves 87.5% accuracy with clinical inputs only and 90.8% with all three modalities, versus ACUMEN's 90.1% and 93.2%.
- Architecture-matched controls for clinical-only transfer. All use ACUMEN's 1.84M-parameter architecture, 32 hyperparameter trials, and 5 seeds, scored with clinical inputs only on the 2,999 test visits. Training with missing-modality tokens and no feature loss (Control C1) reaches 87.3%, joint multimodal training with dropout and zero feature loss (Control C2) reaches 88.6%, and adding current-state and temporal feature matching (Full ACUMEN) reaches 90.1% (a paired gain of +2.8 percentage points [1.7, 3.9] over C1).
- Test-acquisition supervision controls (on 2,520 complete C2 test visits from 1,284 participants). All share the frozen classifier and student network: direct gain regression (), feature transfer + direct gain regression (), and history-only value distillation (teacher input ) isolate observed-gain training, feature matching, and teacher smoothing before adding privileged inputs.
- Alternative policies and full-battery reference. We compare against random legal selection, a greedy static order learned on the value-fitting split, the dueling double Deep Q-Network, AACO, and jointly trained DIME. Revealing all 13 C2 test groups to gives Brier loss 0.096 (91.8% accuracy), whereas ACUMEN's current-state and change policy with stopping () acquires 3.95 test groups on average for Brier loss 0.139 (85.6% accuracy).
W5: Shorter abstract and introduction
We agree that the 322-word abstract and 940-word introduction are too long. We will replace the abstract with the 136-word version below and cut the introduction by a third (moving NACC form-change details and layer dimensions to Sections 2 and 3, while summarizing our two-stage pipeline, baselines, and claim scope):
Comprehensive cognitive evaluations for dementia can take several hours, yet skipping informative tests risks missing early impairment or its underlying cause. Meanwhile, MRI, PET, and cerebrospinal fluid (CSF) biomarkers are recorded for a subset of research participants at varying times, making them available during training but rarely present when cognitive tests are selected. We present ACUMEN, a two-stage framework that transfers longitudinal MRI, PET, and CSF information into a clinical-only diagnostic classifier and an adaptive test selector. To train the selector, a cross-fitted multimodal teacher estimates how much each candidate test reduces the frozen clinical classifier's Brier loss, while visits without imaging retain their observed loss reductions. On NACC, ACUMEN improves clinical-only six-class diagnosis over architecture-matched controls and lowers diagnostic loss across test budgets in both C2-battery replay and a ground-truth simulator.
Recap. We separated the two-stage acquisition mechanism from the NACC-specific encoders, added exact specifications for every baseline (including a retrained Xue et al., 2024 model, architecture-matched controls, DQN, AACO, and DIME), reported NACC and simulator acquisition results, and shortened the abstract and introduction.
Part 1
Thank you for laying out your concerns so clearly.
Since submission, we have completed the NACC test-acquisition experiments as well as a synthetic simulator benchmark, and we address the scope of our claims, how the method applies beyond NACC, our two-stage design and baselines, and the length of the abstract and introduction below.
W1, W2 and Q1, Q2, Q5: What is specific to NACC, what is general, and which claims hold where
We agree that the paper's presentation blended a dementia-specific diagnostic architecture with a general test-selection mechanism, making it hard to see where one ends and the other begins.
Separating the dementia-specific modules from the general acquisition mechanism (W2, Q1). In ACUMEN, the domain-specific components are the 88-channel NACC clinical state vector (demographics, cognitive test scores, observation masks, and visit timing), the three modality encoders (a Mamba encoder over 89 MRI regional volumes, an MLP over 19 PET uptake ratios, and an assay-aware MLP over 7 CSF biomarkers), and the C1/C2 test batteries. Once the diagnostic classifier qθ is trained and frozen, the acquisition stage depends only on three general objects: a partial history of revealed features h, a discrete library of candidate actions a with costs ca, and the scalar Brier loss reduction di,a=ℓ(qθ(hi),Yi)−ℓ(qθ(hi⊕(a,Xia)),Yi) obtained when revealing action a's recorded result into qθ. During training, a teacher network uses extra training-only features Zi (fitted via 5-fold participant cross-fitting) to predict di,a, and the clinical-only student gϕ(hi,a) is trained on the blended target ti,a=di,a+ρIi(m−f(i)−di,a). To apply this to another sequential testing problem, such as adaptive clinical questionnaires (where a training cohort also received a full specialist workup) or sequential sensor selection (where training runs include high-precision reference sensors), one replaces the input encoders, action definitions, and classifier qθ, while keeping the cross-fitted gain teacher and student objective unchanged.
How performance changes across shifts within NACC and on a non-clinical simulator (W1, Q2). First, to see how sensitive the diagnostic model is to changes in clinical center, test battery, and time period, we retrained ACUMEN and the clinical MLP under three splits within NACC:
Second, to test the acquisition method outside NACC, we built a 4-class synthetic simulator with a noisy baseline feature B, 6 noisy scalar tests Xa (where test 6 requires test 4 as a prerequisite), a frozen 2-layer MLP classifier qθ, and a 4-dimensional training-only vector W=s⋅eY+εW available for a nominal 30% of training subjects. Using 4,000 Monte Carlo draws from the exact posterior P(Y,Xa∣h) to compute true expected gains μ(h,a), we find that with 1,000 value-training subjects and signal strength s=1, privileged distillation reduces contrast error from 13.8 +/- 0.5 to 10.4 +/- 0.4 (×10−3) and lowers selection regret from 7.8 +/- 0.4 to 5.5 +/- 0.3 across 10 replications compared with a matched history-only teacher. When we set s=0 so W is pure noise, the contrast-error difference is 0.00 +/- 0.40, confirming that the gain comes from the informative training-only signal.
Which claims are limited to NACC (Q5). In Section 1 of the revision, we will state the exact scope of each contribution: our diagnostic accuracy gains (93.2% multimodal, 90.1% clinical-only) and C2 test-selection curves apply specifically to predicting visit-level recorded diagnoses in NACC research centers, whereas the target bias-variance analysis (Section 3.5), the retrospective sensitivity bounds (Appendix E), and the simulator results establish properties of the acquisition method under general active feature acquisition with training-only supervision.