acceptodds
Under review as a conference paper at ICLR 2027

HINT-AD: Hierarchical and Interpretable Transformer for Speech-Based Alzheimer's Disease Staging and Marker Discovery

Abstract

Accurate staging within the mild cognitive impairment (MCI) continuum is challenging because progression from early MCI (eMCI) to late MCI (lMCI) manifests through subtle, interacting changes in speech, language, and cognition. Can eMCI and lMCI stages be distinguished from connected speech by explicitly modeling the hierarchy of cognitive-linguistic disruption while retaining interpretable evidence for each prediction? We introduce HINT-AD, a hierarchically interpretable Transformer that integrates phonological, syntactic, semantic, and cognitive-linguistic markers through marker-guided Transformer stages for eMCI–lMCI staging. Rather than treating heterogeneous markers as independent features, HINT-AD progressively models their interactions while retaining links from marker summaries to supporting acoustic representations and transcript spans. Across 420 English and Chinese picture-description recordings from 218 participants, using MCI-stage labels transferred from the Alzheimer's Disease Neuroimaging Initiative (ADNI) reference cohort, with an additional 89 Spanish passage readings used for auxiliary training only, HINT-AD achieves 80% and 79% ROC-AUC and 69% and 74% average precision (AP) in English and Chinese, respectively. Compared with the strongest baselines, these results correspond to relative improvements of 8.1% and 5.3% in ROC-AUC and 7.8% and 10.4% in AP, respectively. Joint English–Chinese training improves ROC-AUC by 1.4% and 8.3% relative to language-specific training, while joint training with Spanish auxiliary data yields relative gains of 14.3% and 9.7% over the same baselines, respectively. Marker-group ablations further show that masking any linguistic level reduces performance in both languages, with the largest ROC-AUC reductions arising from semantics in English and cognition in Chinese. SHAP analysis identifies semantic markers as the strongest positive contributors in both languages, including role identification and lexical use, with additional language-specific contributions from scene integration and contrast use in English and thematic coherence and verb use in Chinese. Together, these findings provide evidence that hierarchical modeling of connected-speech markers improves fine-grained staging within the MCI continuum while exposing interpretable cognitive-linguistic patterns associated with stage. HINT-AD reframes speech-based within-MCI staging from flat classification over heterogeneous biomarkers to evidence-linked hierarchical representation learning, offering a clinically motivated and interpretable framework for computational modeling of subtle neurocognitive change.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Alzheimer’s disease, speech assessment, linguistic markers, hierarchical transformer, SHAP analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_29gQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

the goal is to model cognitive and linguistic speech markers to improve early versus late MCI staging across languages.

Critical strengths

Clinically meaningful problem

Hierarchical but not flat modeling

Potential for interpretability

Critical weaknesses

The method is incremental. It combines BERT layers grouped into stages, marker-query cross attention, concatenated summaries, and extracted markers.

No ablation on the hierarchy itself. So the claimed benefit of "hierarchical" modeling is unclear.

Missing baselines: e.g., CogniVoice https://arxiv.org/abs/2407.13660 (Interspeech 2024) is a multimodal and multilingual model tested on the same TAUKADIAL data.

The eMCI/lMCI labels are not clinical. They come automatically created using only age, sex, and MMSE, so the model may be learning a proxy for those variables.

no confidence intervals, seeds, or significance tests.

poor metric choice for imbalanced test data.. use AUC-PRC instead, see this paper for details: https://ftp.cs.wisc.edu/machine-learning/shavlik-group/davis.icml06.pdf

Evaluation covers only one task and one condition.

Questions for authors

see above.

Reviewer_JUiKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed the evaluation of a range of speech processing models for the detection of Alzheimer's disease from speech, on 3 corpora: Pitt, TAUKADIAL, and PROCESS-2, in English and Chinese. The authors propose a new architecture (HINT-AD) based on mBERT where 'sentence-level audio embeddings' are projected into various layers of mBERT. This new proposed architecture shows results closed to the state of the art in AD detection from speech, with ROC-AUC values of 0.80 on English and 0.79 on Chinese. An additional Shapley values analysis is given to support the correlation of the model's predictions with explainable features.

Critical strengths

The writing is generally understandable, most metrics are explained in the article, and the datasets are open source.

Critical weaknesses

The language-wise evalutation makes the results difficult to compare to existing works, it would have been preferred to keep the evaluation sets split per dataset. The related works are missing a large part on similar existing works. "Zokaeinikoo, M., Kazemian, P., & Mitra, P. (2023). Interpretable hierarchical deep learning model for noninvasive Alzheimer’s disease diagnosis. INFORMS Journal on Data Science, 2(2), 183-196. " shows a 0.99 ROC-AUC on Pitt. "Agbavor, F., & Liang, H. (2024). Multilingual prediction of cognitive impairment with large language models and speech analysis. Brain sciences, 14(12), 1292." shows a 0.88 F1 score on the English subset of TAKAUDIAL and 0.95 F1 score on the chinese set of TAKAUDIAL "Pahar, M., Illingworth, C. H., Mirheidari, B., Elghazaly, H., Peters, F., Young, S., ... & Christensen, H. (2026). PROCESS-2: A Benchmark Speech Corpus for Early Cognitive Impairment Detection. arXiv preprint arXiv:2605.14888. ", the paper that introduced the PROCESS-2 dataset, shows a 0.85 F1-score on linguistic components only.

Questions for authors

Could you add F1 score metrics and dataset-wise results for clear comparison with existing works?

Reviewer_VxioOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes HINT-AD, a multimodal, hierarchically structured Transformer for staging early vs. late Mild Cognitive Impairment (eMCI vs. lMCI) from picture-description speech recordings in English and Chinese (with Spanish reading passages used as auxiliary training data). The pipeline extracts 12 sentence-level acoustic/phonological features via forced alignment and uses an LLM to score 15 higher-level linguistic markers (5 each for syntax, semantics, and cognition, along with supporting transcript spans). These 4 marker levels are progressively routed into pairs of mBERT layers via cross-attention (layers 0–1 for phonology, 2–3 for syntax, 4–5 for semantics, 6–7 for cognition), followed by multimodal fusion with Qwen2-Audio embeddings in layers 8–11. Across 420 recordings from 218 participants (Pitt, TAUKADIAL, PROCESS-2), HINT-AD reports 0.80 and 0.79 ROC-AUC in English and Chinese, alongside marker-group ablations and KernelSHAP analyses.

Critical strengths

Strengths Thoughtful clinical and linguistic framing: Organizing multimodal speech-and-text fusion around a four-stage linguistic hierarchy (phonology → syntax → semantics → discourse/cognition) is intuitive, and linking LLM-extracted marker scores to specific transcript spans and pause intervals makes the model's intermediate steps much easier for clinicians to inspect than a black-box pooling layer.

Cross-linguistic evaluation and interpretability checks: Testing joint English–Chinese training alongside KernelSHAP attributions (Table 4) and deterministic sanity checks on repetition/omission counts (Table 7) is a nice effort toward understanding what the classifier actually relies on.

Critical weaknesses

The ground-truth "eMCI vs. lMCI" labels are synthetically imputed via 5-NN from Age, Sex, and MMSE (Fatal Soundness Flaw): In Section 4.1 (Page 5, Line 250), the authors reveal that the speech datasets (Pitt, TAUKADIAL, PROCESS-2) do not actually have eMCI vs. lMCI ground-truth labels. Instead, the authors trained a 5-nearest-neighbor (5-NN) classifier on an external reference cohort (ADNI) using only three scalar variables: standardized Age, Sex, and MMSE score, and used it to assign synthetic eMCI/lMCI labels to the speech cohorts

Also the paper uses tiny hold-out test set, and not having any standard deviation / variance indications.

Questions for authors

On Dataset Labels and 5-NN Transferability: Since the eMCI/lMCI labels on the speech datasets are purely synthetic (generated by a 5-NN classifier based only on Age, Sex, and MMSE), is HINT-AD merely learning to predict the MMSE bracket of a patient? What is the performance of a simple baseline that classifies these pseudo-labels using only the patients' ground-truth metadata (Age, Sex, and MMSE)?

Given that the held-out Chinese test set contains only ~6 eMCI participants, how did you control for high evaluation variance? Can you provide 95% bootstrap confidence intervals or repeated-fold cross-validation standard deviations for the main results in Tables 1-3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.