- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces EviDial, a benchmark that asks whether a depression prediction is supported by the dialogue, rather than simply whether it matches a recorded clinical label. For example, a model might correctly predict “no sleep disturbance” even though sleep was never discussed. The authors call this textually unsupported correctness. They analyze 382 dialogues from three datasets, using 2,744 participant–symptom pairs for the main reference. LLM judges classify evidence as absent, partial, conflicting, or sufficient, with explicit rules for whose symptoms are described, the relevant time window, negation, and retracted statements. A latent-class model combines the judgments. The central contribution is an evaluation framework that separates getting the label right from having enough evidence to justify it. Main findings:
- 82.8% of symptom assessments lack sufficient evidence under the automated reference, with substantial differences between datasets.
- Across three models and four prompting strategies evaluated on 87 development cells, 69.2–96.9% of label-correct predictions lack supporting evidence.
- Evidence-focused prompting does not consistently solve the problem; abstention reduces unsupported predictions at the cost of answering fewer cases.
- A 120-cell human validation reportedly preserves the main finding, although automated judges perform poorly on controlled evidence perturbations.
Critical strengths
The paper identifies an important limitation of depression-dialogue evaluation: a prediction can match a clinical label without sufficient support in the model-visible transcript. Separating these properties helps prevent benchmark accuracy from being over interpreted as evidence-grounded assessment capability.
Critical weaknesses
- The evidence reference is not sufficiently validated to support the headline estimates. No automated judge condition passes the perturbation-validation gate, and latent-class fusion performs worse than plurality on the available anchor comparisons while producing no sufficient-negative or conflicting assignments. The reported human validation is encouraging, but lacks essential details about sampling, pre-adjudication agreement, and overlap with the model audit. Because the central results depend on this reference, these uncertainties materially weaken confidence in the benchmark’s validity.
- The empirical evidence supports narrower conclusions than the paper’s framing suggests. The model audit comprises only 87 development cells, with single runs and no completed held-out evaluation. Meanwhile, E-DAIC dominates the reference pool and underwent LLM-based reformatting without independent verification of evidence preservation. Consequently, the submission does not adequately establish how broadly the reported gap generalizes or h
Questions for authors
- Clarification: human validation: How were the 120 cells sampled, what were the annotators’ qualifications and pre-adjudication agreement, and how many cells overlapped the model audit? Please provide exact denominators and uncertainty estimates for the human-rescored results. These details are essential to assessing whether human validation supports the headline conclusions.
- Clarification: reference robustness: Given fusion’s worse anchor performance than plurality and its absence of sufficient-negative or conflicting assignments, what justifies selecting it as the primary reference? Do the evidence-gap and unsupported-correctness estimates remain similar under plurality and human labels when evaluated on matched cells?