acceptodds
Under review as a conference paper at ICLR 2027

NarraSpace: Multimodal Scene-Grounded Evidence for Picture-Description Dementia Classification

Abstract

Picture-description dementia classification uses a shared visual scene, but a transcript-level prediction alone does not identify the scene evidence underlying its inputs. NarraSpace links words and speech times to named picture elements and retrieves utterances for predefined scene events. Narrative-spatial descriptors summarize coverage and temporal organization. A frozen language model contrasts each event's text under examples from Alzheimer's disease and healthy-control training participants, alongside transcript surprisal and coverage measurements. Retained source records link event scores to their source utterances, speech intervals, and lexical matches. On the official 48-participant test set of the Alzheimer's Dementia Recognition through Spontaneous Speech benchmark, a fixed support vector classifier achieves 80.88% ± 2.56% mean accuracy over fifty seeds; averaged probabilities yield 81.25% accuracy and 91.32% area under the receiver operating characteristic curve. NarraSpace makes the construction of scene-linked classification inputs explicit and inspectable by connecting event-indexed language-model measurements to their source utterances, speech timing, and extraction records. Such source-linked measurements could support clinician review of picture descriptions by presenting quantitative summaries alongside the original speech evidence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Alzheimer's disease detection, picture description, multimodal learning, scene grounding, narrative-spatial representations, language-model likelihood, source attribution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_au2fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NarraSpace constructs source-linked features for Alzheimer’s dementia classification from picture-description speech. It aligns transcript words with speech intervals and fifteen predefined scene anchors, retrieves utterances for seven scene events, and computes seventy narrative-spatial descriptors. A frozen Qwen2-7B model scores retrieved event text under Alzheimer’s disease and healthy-control exemplar contexts drawn from training participants. These measurements are combined with acoustic, linguistic, timing, and image–text features; sixty inputs are selected for a support vector classifier.

On ADReSS’s official 108-participant training and 48-participant test split, the corrected system achieves 80.88% mean accuracy across fifty seeds, with aggregated accuracy of 81.25% and AUC of 91.32%. Validation includes feature removals, targeted controls, participant-level bootstrap intervals, decision-change analyses, and a limited anchor-mapping audit. The principal contribution is traceability from measurements to their source speech, rather than demonstrated classifier explanation or clinical utility.

Critical strengths

Explicit and inspectable links between measurements and source speech. Paper representation retains utterances, speech intervals, anchor matches, and extraction records alongside event scores. This is useful for identifying extraction errors and understanding what a numerical feature measures. The authors appropriately distinguish this source attribution from explaining a classifier’s decision or improving clinical judgment. Such distinction makes the proposed contribution more precise.

Careful disclosure and analysis of experimental limitations. The authors disclose earlier fitting errors and test exposure, separate historical results from corrected analyses, and distinguish seed variation from participant-sampling uncertainty. The paired comparisons and complete account of changed decisions are particularly useful: adding the likelihood stream corrects five decisions and worsens five others, with different effects across diagnostic groups. This is more informative than reporting aggregate accuracy alone.

Critical weaknesses

The incremental predictive value of the proposed representation remains uncertain, without an independent confirmatory evaluation. Adding the twelve event-score descriptors to coverage, surprisal, and missingness features improves mean accuracy by only 0.71 percentage points, with a paired 95% interval of [−6.42, 7.92], while aggregated AUC decreases by 0.69 points (Table 18). Removing the entire likelihood stream leaves aggregated accuracy unchanged and slightly increases mean accuracy (Table 2). The narrative-spatial family’s mean-accuracy benefit is also uncertain: 2.67 points, with interval [−3.67, 9.00].

These results do not establish that the components are ineffective, but they leave their benefit unresolved. A matched comparison between event-local and whole-transcript scoring using the same frozen model and evaluation procedure is absent. Furthermore, the 48 test participants were examined during earlier development, and correcting fitting boundaries cannot restore an untouched test set. The authors acknowledge this appropriately, but an independent evaluation is still needed to substantiate generalisation.

The main source-linked evidence contribution lacks validation at the event level. The reported 96.04% precision concerns accepted phrase-rule and prototype anchor assignments reviewed by one annotator. It does not measure missed concepts, event-retrieval precision or recall, semantic relations, or timing accuracy (Appendix G). Event retrieval accepts any relevant lexical or anchor match, does not resolve negation or actor-action relations, and shares approximately 79–82% of retrieved utterances across multiple events. Overlap is not inherently erroneous, but it makes the semantic specificity of the event-indexed measurements an important unanswered question.

Traceability shows where a measurement came from; it does not establish that the retrieved text adequately represents the named event. Because the paper’s primary contribution is this evidence organization, direct event-level annotation or a comparison of inspection accuracy and effort against simple retrieval would provide more relevant validation than classification results alone. Neither is currently available.

Questions for authors
  1. Matched scoring comparison: Can you compare event-local and whole-transcript scoring while holding the Qwen model, eligible exemplar participants, transcript processing, downstream fitting, and threshold-selection procedure fixed? This would help determine whether event indexing adds predictive information beyond contextual language-model scoring.

  2. Event-level validity: Is there an independently annotated subset measuring event-retrieval precision and recall, including paraphrases, negated mentions, incorrect relations, and shared utterances? The existing accepted-anchor audit does not resolve these questions.

  3. Independent evaluation: Is there an untouched participant cohort or other evaluation set on which the complete fixed procedure can be assessed? Please distinguish any new validation from further analyses of the already-examined official test participants.

  4. Verification of corrected context construction: Appendix C states that permitted fold-specific scores already existed, but the original scoring records lack exemplar identifiers. Can the exact prompts and exemplar membership be reconstructed and a sample independently rescored to verify both fold exclusion and agreement with the cached likelihood values? This is a request to verify the correction, not an assertion that leakage remains.

  5. Reproducibility: What materials can an authorized researcher obtain to reproduce the corrected results? Appendix H describes a separate reference implementation and states that the saved predictions, participant order, and bootstrap indices required by its statistical reproducer are not included. Clarifying access to the experimental pipeline and required artifacts would materially improve confidence in reproducibility.

2 comments

Thank you for the specific questions. Our answers follow your numbering.

Q1 (matched scoring). Done. Both arms add one score to the same 253 candidates (216 inherited features, two coverage counts and 35 availability flags), and each fit selects up to 60 inputs. The frozen Qwen2-7B, transcript processing, eligible training participants per fold, classifier specification and training-only threshold rule are the same in both arms. The exemplar texts that are drawn and the fitted thresholds depend on the scoring unit.

InputCorrect / 48Mean accuracy (%)AUC (%)
Common pool4284.4690.63
+ event score3981.6394.10
+ whole-transcript score4184.8394.79

Both scores increase AUC over the common pool: +3.47 points (paired 95% CI [-1.04, +9.55]) for the event score and +4.17 ([-0.35, +9.72]) for the whole-transcript score. Event minus whole-transcript is -0.69 AUC points ([-3.47, +1.74]) and -4.17 accuracy points ([-12.50, +4.17], McNemar p = 0.625). In controls with matched exemplar counts the result is 39 vs. 42 correct, and with the same retrieved text regrouped it is 38 vs. 41, so the gap between the two units is two to three participants in all settings.

We also tested combining the two. Keeping the whole-transcript score and adding no event scores, seven matched event scores, seven mismatched event scores, or one event summary gives 41/48, 41/48, 40/48 and 42/48. With the seven matched scores, log loss changes from 0.3096 to 0.3080 (difference -0.0016, 95% CI [-0.0361, +0.0337]).

Q2 (event-level validity). Done on all 48 test transcripts. Two university students, who were not involved in developing NarraSpace, independently went through all 590 utterances and selected the related, supporting and negating utterances for each event (336 participant-event records per annotator). Diagnosis and system outputs were hidden. Since they read the whole transcript, their selections include utterances that retrieval missed.

RetrievalA: precision / recall (%)B: precision / recall (%)
NarraSpace44.90 / 83.6448.45 / 93.17
Keyword only53.20 / 77.0257.65 / 86.18
Semantic ranking, matched utterance count46.58 / 86.7847.29 / 90.96

Recall of NarraSpace is above 91% for E1-E5 against both annotators, and it is 6.61 and 7.00 points higher than keyword retrieval. The keyword and semantic rows are the comparison with simple retrieval. F1 of NarraSpace is 58.43% and 63.75%. The positive agreement between the annotators is 69.52% (414 shared assignments out of 605 and 586), and Cohen's kappa on event state is 0.434.

Regarding the cases you mention: paraphrases and partial mentions count as relevant in the protocol, so they are included in the numbers above. Shared utterances are frequent in both annotations (170/352 and 169/372 selected utterances relate to more than one event). Negated mentions are rare: the annotators marked zero and two negating assignments in the 48 transcripts. The annotators disagree most on E7 (global kitchen context), where they share 19 assignments out of 103 and 70. System recall on E7 is 39.81% and 70.00%.

Q3 (independent evaluation). We have not evaluated a new cohort. All new analyses use the official test participants that were examined during development. The corrected evaluation kept the specified SVM parameters and uses fold-specific training references and thresholds selected on training data. English Lu and the Pitt participants outside ADReSS are candidate Cookie Theft cohorts for evaluating the fixed procedure, after checking diagnoses and overlap with all 156 ADReSS participants.

Q4 (context construction). Partly verified. When we reaggregate the fold-keyed records, we get exactly the accepted features (maximum absolute difference 0.0). In the new comparison we saved exemplar IDs and prompt hashes, and for every record we checked fold exclusion, target exclusion, prompt reconstruction and score aggregation. The historical rows contain fold, class, seed and draw, and membership can be reconstructed from these with the frozen pools and sampling rules. They do not contain the exemplar list. The per-record membership check you describe was done on the new scoring records, and we did not rescore the historical cache.

Q5 (reproducibility). ADReSS can be obtained by authorized researchers from DementiaBank. The anonymous supplement has a separate reference implementation with synthetic examples, and a statistical reproducer that we checked against the seven result rows and eight paired contrasts of Appendix I (15,486 checks). The archived predictions, participant order and bootstrap indices that the reproducer reads are not in the anonymous package (see Appendix H).

Reviewer_au2fOpenReview ↗

This paper is weak by design. Need a fundamental rethink of the methodology.

I appreciate the authors for the matched scoring comparison and the independent, blinded annotation, which directly address my questions.

Both, however, weaken the central claims. Event-local scoring does not outperform whole-transcript scoring, and adding event scores to the whole-transcript score leaves log loss essentially unchanged.

NarraSpace retrieval achieves lower precision than simple keyword matching, and moderate inter-annotator agreement suggests the events are loosely defined.

With only 48 previously examined test participants, differences of two to three participants cannot be resolved, and no untouched cohort has been evaluated. The paper has very limited reproducibility. I maintain my original weak-rejection rating.

Reviewer_rx81OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

NarraSpace is a method for using spoken descriptions of an image to detect Alzheimers. It connects words to objects and events in the image and creates a number of features, including probability scores from language models, which are then used in a binary classifier. With this method, it aims to better attribute predictions to specific features.

Critical strengths
  1. Developing interpretable ML methods for clinical use is an important area.
  2. The authors are transparent and thorough in their experiments. They acknowledge uncertainty and limitations in their results.
Critical weaknesses
  1. The likelihood measurements with an LLM do not improve the model performance according to table 2. This suggests the performance is driven mostly by the simpler features. The authors acknowledge this but it is a major limitation of the method they present nonetheless.
  2. The test set is small (n=48) and the authors say it had already been used previously in their experiments, where it may have informed other modelling decisions.
  3. The paper's main claim is the benefit of attributing predictions to specific features for explainability, but there is no evidence presented that this attribution is working or helpful in a clinical context. All the results center around classification performance.
Questions for authors
  1. Does measuring likelihood over the whole transcript rather than per event help? How does the method compare to simple zero shot classification with an LLM? If an LLM is asked to ground its predictions based on features of the transcript, do its claimed top features align with those of NarraSpace?
  2. Are the authors able to evaluate how useful the feature attribution of NarraSpace is in a clinical context?
2 comments

Thank you for the comments and for suggesting the baselines.

Claim of the paper. The paper claims source attribution, i.e. every event score can be traced to the utterances and extraction matches it was computed from (Fig. 3). This is a claim about the measurements. It is different from attributing the decision of the classifier to features. We now have human evidence for this claim in addition to the classification results. Two independent annotators marked which utterances are related to each event in all 48 test transcripts. The event records recover 83.64% and 93.17% of their selections, with precision of 44.90% and 48.45%. A reader study with clinicians is the next step, as described in Section 6.

Likelihood measurements. The likelihood stream does affect the fitted system. Without it, aggregated AUC drops from 91.32% to 88.89% (difference +2.43, paired 95% CI [-4.86, +9.90]) and ten of the 48 decisions change (five in each direction), while both systems classify 39/48 correctly. In the new same-model comparison, adding one likelihood score to a common input pool increases AUC from 90.63% to 94.10% or 94.79%.

Whole-transcript vs. per-event likelihood. We ran this with the same frozen model and the same fitting and threshold rules. The common inputs alone give 42/48 correct and 90.63% AUC. With a whole-transcript score we get 41/48 and 94.79%, and with an event score 39/48 and 94.10%. Event minus whole-transcript is -0.69 AUC points ([-3.47, +1.74]) and -4.17 accuracy points ([-12.50, +4.17]). The point estimates are higher for whole-transcript scoring. The advantage of event scoring is that the scored unit is explicit in the event record.

Zero-shot LLM. We used the Qwen2-7B base model, which is the frozen backbone of our likelihood measurements, to keep the model fixed. Each cleaned transcript is given to the model with no examples. The normalized probabilities of the two answer labels are averaged over both answer orders. We tested two label formats and report both.

MethodCorrect / 48AUC (%)
Zero-shot, label without preceding space3678.30
Zero-shot, label with preceding space3676.74
Submitted NarraSpace3991.32

The AUC difference between NarraSpace and zero-shot is +13.02 points ([-0.69, +28.47]) and +14.58 ([+0.95, +30.21]), and the accuracy difference is +6.25 points ([-6.25, +18.75]) for both formats. The zero-shot counts are at the fixed 0.50 cutoff. With cutoffs selected on training data they are 34/48 and 36/48. Note that NarraSpace is a complete system that uses training labels and several input families, while the baseline is a single prompt to the same backbone.

Agreement with the LLM's stated evidence. We asked the LLM to quote the passages of the transcript that support its answer, and measured how much they overlap with the source text of NarraSpace. 44 of the 48 responses contained a parseable list of quotations. Of the 92 quotations, 84 are verbatim spans of the transcript, and 61 of these 84 (72.62%) are fully inside the utterances retrieved by NarraSpace. These retrieved utterances cover 83.10% of the transcript words. So most of the passages cited by the LLM are already stored by NarraSpace as source records, where each one is linked to a named event and a score.

Thank you for the detailed response and for running the additional analyses. They clarified some points, and I appreciate how openly the results are reported. However, my main concerns remain, and I am keeping my score.

  • The new results do not show that the event-level likelihood stream contributes. The AUC difference from removing it has a CI that includes zero
  • Annotator precision of about 45–48% means most retrieved utterances were not judged related to the event. Retrieved utterances also cover 83% of transcript words, so high recall is close to guaranteed and the attribution is not very specific.
  • All new comparisons were again run on the 48-participant test set.
Reviewer_QVVkOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NarraSpace, a multimodal framework for Alzheimer's disease classification from picture-description speech. It combines scene-anchor trajectories, event-level language-model likelihood contrasts, acoustic and linguistic features, and source-linked evidence records. On the ADReSS benchmark, it achieves 80.88% mean accuracy over 50 seeds and 81.25% aggregated accuracy. The results show limited and statistically inconclusive predictive gains from the proposed components, while demonstrating a framework for tracing constructed features back to source utterances.

Critical strengths
  1. Scene-grounded evidence representation. The method connects 15 scene anchors, seven predefined events, word-level timestamps, and event-conditioned language-model scores. This provides a structured representation of both scene coverage and narrative organization.

  2. Traceability of extracted evidence. The framework preserves source utterances, timestamps, anchor assignments, and lexical matches for each event. This allows users to inspect how individual measurements were constructed, rather than relying only on transcript-level classification scores.

  3. Detailed ablations and uncertainty analysis. The evaluation includes six feature-family removals, targeted representation controls, 50 random seeds, participant-level bootstrap confidence intervals, and McNemar tests. These analyses expose the uncertainty behind the reported performance differences.

  4. Transparent evaluation and implementation limitations. The authors explicitly identify earlier validation leakage and correct the feature-construction boundaries. They also disclose prior test-set exposure, train/test reference-pool differences, and limitations of their reproducibility package.

Critical weaknesses
  1. Evaluation is too limited. The paper only use 1 dataset for evaluation, with 108 / 43 train / test samples. There are other demential benchmarks, DementiaBank, TAUKADIAL challenge, PREPARE challenge.

  2. The method provides limited improvements. Removing the entire scene-grounded likelihood stream increases mean accuracy from 80.88% to 81.63%. The targeted control in Table 17 shows that adding the twelve event-score features improves mean accuracy by only 0.71 percentage points, with a 95% confidence interval of [-6.42, 7.92]. Aggregated AUC decreases by 0.69 points in that comparison. Removing the narrative-spatial family reduces mean accuracy by 2.67 points, but its confidence interval also includes zero. Moreover, removing speech-timing features improves aggregated accuracy from 81.25% to 85.42%. These results do not establish a reliable predictive advantage for the full representation.

  3. The event retrieval mechanism does not sufficiently isolate event-specific semantic evidence. An utterance is assigned to an event whenever any associated lexical field or anchor matches. The mechanism does not verify actor–action relationships, negation, or semantic entailment. More importantly, Table 21 reports that approximately 79-82% of retrieved utterances are shared across multiple events. Consequently, the likelihood contrasts for different events may depend heavily on overlapping text. This weakens the premise that they capture distinct event-level information. The paper also lacks a matched comparison between event-level and whole-transcript scoring using the same frozen language model, exemplar construction, and evaluation protocol. The historical comparison changes multiple factors simultaneously and cannot isolate the benefit of localization.

  4. The semantic correctness of the scene-grounding pipeline is insufficiently validated. The reported 96.04% mapping precision is based on 404 accepted assignments reviewed by one annotator. The audit excludes rejected assignments, deterministic lexical mappings, and event-level semantic correctness. Thus, it does not establish recall or the accuracy of identifying scene events. The absence of inter-annotator agreement also limits confidence in the audit. Furthermore, word-level timestamps and event retrieval accuracy are not independently evaluated. These errors can propagate into the event likelihoods and narrative-spatial descriptors.

  5. The narrative-spatial representation contains substantial redundancy and potentially misleading temporal measurements. Appendix reports that graph node count, linguistic anchor count, and narrative anchor count are exactly identical across all participants. Normalized coverage is also nearly a deterministic rescaling of these counts. All four coordinates are selected in every final classifier fit. For an RBF SVM, including multiple standardized copies of the same feature effectively increases its weight in the kernel distance. Therefore, the reported contribution of the 70-feature family may partly reflect redundant feature weighting. In addition, consecutive mentions of the same anchor form a single visit even across arbitrarily long pauses and intervening unassigned words. The resulting dwell duration does not necessarily measure time spent describing that scene element.

  6. The contribution of actual visual and acoustic grounding remains unclear. The framework uses a single shared Cookie Theft image and manually specified scene anchors. Removing the CLIP-based image-text feature family increases mean accuracy from 80.88% to 81.25%. This raises questions about whether image embeddings add useful information beyond the predefined scene vocabulary. The 88 acoustic descriptors are also extracted from the full enhanced recording rather than participant-only speech. This creates a potential confound from examiner speech or recording composition. No experiment isolates this effect. Finally, the evaluation does not test whether the scene-grounding procedure transfers to unseen pictures or different narrative tasks.

Questions for authors

see above

1 comment

Thank you for the detailed technical comments. We reply point by point.

1. Evaluation scope. The official ADReSS split has 108 training and 48 test participants and is balanced for age and gender. The other datasets you mention are for different tasks. TAUKADIAL contains English Cookie Theft descriptions, but the labels are MCI vs. normal cognition. The acoustic track of PREPARE has Control/MCI/ADRD labels and clips of at most 30 seconds. For AD vs. healthy control, English Lu and the Pitt participants outside ADReSS are candidate cohorts, after checking diagnoses and participant overlap.

2. Predictive contribution. Among the six families, narrative-spatial is the only one whose removal decreases mean accuracy (80.88% to 78.21%, full minus removal +2.67 points, [-3.67, +9.00]). It also has the largest change in AUC (91.32% to 87.67%, +3.65, [-2.78, +10.59]). The compact six-feature alternative gives 77.96% mean accuracy and 89.06% AUC, which are both lower than with the full family.

For the likelihood features, in the new same-model comparison one likelihood score increases AUC over a common input pool from 90.63% to 94.10% (event) or 94.79% (whole transcript). In the submission, AUC changes from 91.32% to 88.89% when the full stream is removed, and adding the 12 event-score descriptors gives +0.71 points in mean accuracy ([-6.42, +7.92]). Without speech timing, mean accuracy is 83.25% and 41/48 are correct. We fixed the full system as primary before running these removals and did not use them to choose a different system.

3. Event specificity and matched scoring. Retrieval is meant to be a high-recall candidate step, and the complete utterance stays visible in the record. Two independent annotators labeled all 590 utterances for the seven events. Retrieval recovers 83.64% and 93.17% of their assignments (above 91% for E1-E5), which is 6.61 and 7.00 points more than keyword retrieval. Precision is 44.90% and 48.45%. The keyword and semantic controls are in our reply to Reviewer au2f. Human annotators also assign utterances to several events: 170/352 and 169/372 of their selected utterances relate to more than one event. Negated mentions are rare (zero and two negating assignments in 48 transcripts).

We have run the matched comparison you asked for. The model, exemplar participants, common inputs and fitting rules are shared. Common inputs alone give 42/48 and 90.63% AUC. With an event score it is 39/48 and 94.10%, and with a whole-transcript score 41/48 and 94.79%. Event minus whole-transcript is -0.69 AUC points ([-3.47, +1.74]) and -4.17 accuracy points ([-12.50, +4.17]).

4. Grounding and timestamps. In the audit, 388 of 404 accepted anchor assignments were correct (96.04%; 100% for phrase rules, 92.45% for prototype matches). The new annotation provides event-level precision and recall and inter-annotator agreement (positive agreement 69.52%, kappa 0.434 on event state). Word times are obtained by forced alignment of the official human transcripts, so the words are already transcribed and are only placed on the audio. Direct word matching was recorded for all 156 participants and the fallback was never used. Event retrieval and all likelihood scores are computed from text only and do not use timing. The aligned times are used in the dwell, pause and timing-space descriptors.

5. Redundancy and dwell. This is tested in Table 17. We removed graph node count and normalized coverage (two of the four equal coordinates) and refitted. The result is 80.96% mean accuracy, 39/48 correct and 92.19% AUC, compared with 80.88%, 39/48 and 91.32% for the full fit. Performance stays the same when half of the duplicated coordinates are removed.

Dwell is defined in Section 3.2 as the interval from the first onset to the last offset within a visit to the same anchor. It measures the span of the visit, including pauses and unmapped speech while the speaker stays on that anchor.

6. Visual and acoustic inputs, transfer. Because the picture is shared, all participants can be measured against one schema. The image embedding is the same for everyone, but its similarity to the utterances differs per participant. Removing the image-text family changes aggregated accuracy from 81.25% to 79.17% and mean accuracy from 80.88% to 81.25%. Without the 88 acoustic descriptors, mean accuracy is 81.08% and AUC is 91.15% (80.88% and 91.32% for the full system), so the reported performance is also obtained without the acoustic family. The schema with 15 anchors and seven events is defined for Cookie Theft, which is the picture used in ADReSS, Pitt and the English part of TAUKADIAL. For another picture, the anchors and events have to be defined for that picture, and the construction of the records stays the same.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all three reviewers for their detailed comments. In the Discussion we listed two analyses that the submission was missing: human event-level annotation, and a comparison of event-level and whole-transcript scoring with the same model. We have run both. We also ran the zero-shot LLM baseline that Reviewer rx81 asked for. The main results are summarized here and the details are in the individual replies.

Contribution. NarraSpace is a scene-indexed measurement representation that can be inspected. A scene event determines which utterances are scored, how the class-specific reference texts are organized, and which source record is stored with the score (Fig. 3).

Event-level annotation (new). We asked two university students, who were not involved in developing NarraSpace, to annotate all 48 test transcripts (590 utterances) independently. They did not see the diagnosis or any system output. Retrieval recovers 83.64% and 93.17% of the event-utterance assignments made by the two annotators. For E1-E5, recall is above 91% against both. Compared with keyword-only retrieval, recall is higher by 6.61 and 7.00 points (paired 95% CIs [4.89, 8.36] and [4.26, 9.87]). Precision is 44.90% and 48.45%, and F1 is 58.43% and 63.75%. For comparison, the agreement between the two annotators on the same measure is 69.52%. Both annotators also assigned nearly half of the utterances they selected to more than one event (170/352 and 169/372), so the cross-event sharing in Table 21 shows up in the human annotations as well.

Event vs. whole-transcript scoring (new). We kept the frozen Qwen2-7B, the eligible training participants, the classifier specification and the training-only threshold rule fixed, and added a single likelihood score to a common input pool. AUC goes from 90.63% to 94.10% with the event score (+3.47, 95% CI [-1.04, +9.55]) and to 94.79% with the whole-transcript score (+4.17, [-0.35, +9.72]). The difference between the two scoring units is 0.69 AUC points ([-3.47, +1.74]). Only the event score comes with a named-event source record. Please note that three different systems in the table below classify 39/48 participants correctly.

SettingCorrect / 48AUC (%)
Submitted full system3991.32
Submitted system, likelihood stream removed3988.89
New comparison: common inputs alone4290.63
New comparison: common inputs + event score3994.10
New comparison: common inputs + whole-transcript score4194.79

Zero-shot LLM baseline (new). Prompting the same frozen Qwen2-7B base model without examples gives 36/48 correct, with AUC of 78.30% or 76.74% depending on the answer-label format. The submitted system gives 39/48 and 91.32%, i.e. +13.02 AUC points ([-0.69, +28.47]) and +14.58 ([+0.95, +30.21]).

All new analyses are on the official 48-participant test set. Intervals are paired participant-bootstrap intervals, and AUC is computed from probabilities averaged over 50 seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.