- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
NarraSpace constructs source-linked features for Alzheimer’s dementia classification from picture-description speech. It aligns transcript words with speech intervals and fifteen predefined scene anchors, retrieves utterances for seven scene events, and computes seventy narrative-spatial descriptors. A frozen Qwen2-7B model scores retrieved event text under Alzheimer’s disease and healthy-control exemplar contexts drawn from training participants. These measurements are combined with acoustic, linguistic, timing, and image–text features; sixty inputs are selected for a support vector classifier.
On ADReSS’s official 108-participant training and 48-participant test split, the corrected system achieves 80.88% mean accuracy across fifty seeds, with aggregated accuracy of 81.25% and AUC of 91.32%. Validation includes feature removals, targeted controls, participant-level bootstrap intervals, decision-change analyses, and a limited anchor-mapping audit. The principal contribution is traceability from measurements to their source speech, rather than demonstrated classifier explanation or clinical utility.
Critical strengths
Explicit and inspectable links between measurements and source speech. Paper representation retains utterances, speech intervals, anchor matches, and extraction records alongside event scores. This is useful for identifying extraction errors and understanding what a numerical feature measures. The authors appropriately distinguish this source attribution from explaining a classifier’s decision or improving clinical judgment. Such distinction makes the proposed contribution more precise.
Careful disclosure and analysis of experimental limitations. The authors disclose earlier fitting errors and test exposure, separate historical results from corrected analyses, and distinguish seed variation from participant-sampling uncertainty. The paired comparisons and complete account of changed decisions are particularly useful: adding the likelihood stream corrects five decisions and worsens five others, with different effects across diagnostic groups. This is more informative than reporting aggregate accuracy alone.
Critical weaknesses
The incremental predictive value of the proposed representation remains uncertain, without an independent confirmatory evaluation. Adding the twelve event-score descriptors to coverage, surprisal, and missingness features improves mean accuracy by only 0.71 percentage points, with a paired 95% interval of [−6.42, 7.92], while aggregated AUC decreases by 0.69 points (Table 18). Removing the entire likelihood stream leaves aggregated accuracy unchanged and slightly increases mean accuracy (Table 2). The narrative-spatial family’s mean-accuracy benefit is also uncertain: 2.67 points, with interval [−3.67, 9.00].
These results do not establish that the components are ineffective, but they leave their benefit unresolved. A matched comparison between event-local and whole-transcript scoring using the same frozen model and evaluation procedure is absent. Furthermore, the 48 test participants were examined during earlier development, and correcting fitting boundaries cannot restore an untouched test set. The authors acknowledge this appropriately, but an independent evaluation is still needed to substantiate generalisation.
The main source-linked evidence contribution lacks validation at the event level. The reported 96.04% precision concerns accepted phrase-rule and prototype anchor assignments reviewed by one annotator. It does not measure missed concepts, event-retrieval precision or recall, semantic relations, or timing accuracy (Appendix G). Event retrieval accepts any relevant lexical or anchor match, does not resolve negation or actor-action relations, and shares approximately 79–82% of retrieved utterances across multiple events. Overlap is not inherently erroneous, but it makes the semantic specificity of the event-indexed measurements an important unanswered question.
Traceability shows where a measurement came from; it does not establish that the retrieved text adequately represents the named event. Because the paper’s primary contribution is this evidence organization, direct event-level annotation or a comparison of inspection accuracy and effort against simple retrieval would provide more relevant validation than classification results alone. Neither is currently available.
Questions for authors
-
Matched scoring comparison: Can you compare event-local and whole-transcript scoring while holding the Qwen model, eligible exemplar participants, transcript processing, downstream fitting, and threshold-selection procedure fixed? This would help determine whether event indexing adds predictive information beyond contextual language-model scoring.
-
Event-level validity: Is there an independently annotated subset measuring event-retrieval precision and recall, including paraphrases, negated mentions, incorrect relations, and shared utterances? The existing accepted-anchor audit does not resolve these questions.
-
Independent evaluation: Is there an untouched participant cohort or other evaluation set on which the complete fixed procedure can be assessed? Please distinguish any new validation from further analyses of the already-examined official test participants.
-
Verification of corrected context construction: Appendix C states that permitted fold-specific scores already existed, but the original scoring records lack exemplar identifiers. Can the exact prompts and exemplar membership be reconstructed and a sample independently rescored to verify both fold exclusion and agreement with the cached likelihood values? This is a request to verify the correction, not an assertion that leakage remains.
-
Reproducibility: What materials can an authorized researcher obtain to reproduce the corrected results? Appendix H describes a separate reference implementation and states that the saved predictions, participant order, and bootstrap indices required by its statistical reproducer are not included. Clarifying access to the experimental pipeline and required artifacts would materially improve confidence in reproducibility.
2 comments
This paper is weak by design. Need a fundamental rethink of the methodology.
I appreciate the authors for the matched scoring comparison and the independent, blinded annotation, which directly address my questions.
Both, however, weaken the central claims. Event-local scoring does not outperform whole-transcript scoring, and adding event scores to the whole-transcript score leaves log loss essentially unchanged.
NarraSpace retrieval achieves lower precision than simple keyword matching, and moderate inter-annotator agreement suggests the events are loosely defined.
With only 48 previously examined test participants, differences of two to three participants cannot be resolved, and no untouched cohort has been evaluated. The paper has very limited reproducibility. I maintain my original weak-rejection rating.
Thank you for the specific questions. Our answers follow your numbering.
Q1 (matched scoring). Done. Both arms add one score to the same 253 candidates (216 inherited features, two coverage counts and 35 availability flags), and each fit selects up to 60 inputs. The frozen Qwen2-7B, transcript processing, eligible training participants per fold, classifier specification and training-only threshold rule are the same in both arms. The exemplar texts that are drawn and the fitted thresholds depend on the scoring unit.
Both scores increase AUC over the common pool: +3.47 points (paired 95% CI [-1.04, +9.55]) for the event score and +4.17 ([-0.35, +9.72]) for the whole-transcript score. Event minus whole-transcript is -0.69 AUC points ([-3.47, +1.74]) and -4.17 accuracy points ([-12.50, +4.17], McNemar p = 0.625). In controls with matched exemplar counts the result is 39 vs. 42 correct, and with the same retrieved text regrouped it is 38 vs. 41, so the gap between the two units is two to three participants in all settings.
We also tested combining the two. Keeping the whole-transcript score and adding no event scores, seven matched event scores, seven mismatched event scores, or one event summary gives 41/48, 41/48, 40/48 and 42/48. With the seven matched scores, log loss changes from 0.3096 to 0.3080 (difference -0.0016, 95% CI [-0.0361, +0.0337]).
Q2 (event-level validity). Done on all 48 test transcripts. Two university students, who were not involved in developing NarraSpace, independently went through all 590 utterances and selected the related, supporting and negating utterances for each event (336 participant-event records per annotator). Diagnosis and system outputs were hidden. Since they read the whole transcript, their selections include utterances that retrieval missed.
Recall of NarraSpace is above 91% for E1-E5 against both annotators, and it is 6.61 and 7.00 points higher than keyword retrieval. The keyword and semantic rows are the comparison with simple retrieval. F1 of NarraSpace is 58.43% and 63.75%. The positive agreement between the annotators is 69.52% (414 shared assignments out of 605 and 586), and Cohen's kappa on event state is 0.434.
Regarding the cases you mention: paraphrases and partial mentions count as relevant in the protocol, so they are included in the numbers above. Shared utterances are frequent in both annotations (170/352 and 169/372 selected utterances relate to more than one event). Negated mentions are rare: the annotators marked zero and two negating assignments in the 48 transcripts. The annotators disagree most on E7 (global kitchen context), where they share 19 assignments out of 103 and 70. System recall on E7 is 39.81% and 70.00%.
Q3 (independent evaluation). We have not evaluated a new cohort. All new analyses use the official test participants that were examined during development. The corrected evaluation kept the specified SVM parameters and uses fold-specific training references and thresholds selected on training data. English Lu and the Pitt participants outside ADReSS are candidate Cookie Theft cohorts for evaluating the fixed procedure, after checking diagnoses and overlap with all 156 ADReSS participants.
Q4 (context construction). Partly verified. When we reaggregate the fold-keyed records, we get exactly the accepted features (maximum absolute difference 0.0). In the new comparison we saved exemplar IDs and prompt hashes, and for every record we checked fold exclusion, target exclusion, prompt reconstruction and score aggregation. The historical rows contain fold, class, seed and draw, and membership can be reconstructed from these with the frozen pools and sampling rules. They do not contain the exemplar list. The per-record membership check you describe was done on the new scoring records, and we did not rescore the historical cache.
Q5 (reproducibility). ADReSS can be obtained by authorized researchers from DementiaBank. The anonymous supplement has a separate reference implementation with synthetic examples, and a statistical reproducer that we checked against the seven result rows and eight paired contrasts of Appendix I (15,486 checks). The archived predictions, participant order and bootstrap indices that the reproducer reads are not in the anonymous package (see Appendix H).