- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DepressionBench, a unified multimodal benchmark for depression recognition and reasoning analysis. It curates six public datasets covering text, audio, vision, and audiovisual inputs across four scenarios and three languages, with 34,973 participants in total. It evaluates twelve open-weight LLMs under zero-shot, few-shot, SFT, and RL settings on classification and severity estimation. It further uses an analyst agent to rate model-reported reasoning quality and to diagnose failure types and root causes. Three clinical experts reviewed 50 sampled assessments in a blinded protocol.
Critical strengths
Strength 1: The paper evaluates only open-weight models of at most 30B parameters. Most existing depression-recognition benchmarks rely on proprietary cloud models, so their results do not show how well locally deployable models perform. By restricting evaluation to this model class, the paper measures the setting that privacy-sensitive use actually requires: models that can run on a user's device without sending mental-health records to a cloud service. Strength 2: The paper examines model-reported reasoning in depression recognition. An analyst agent rates each explanation along four dimensions without seeing the reference label, then identifies failure types and root causes in clinically wrong predictions. Existing clinical reasoning benchmarks assess reasoning from textual medical questions or clinical cases. Here, a model listening to speech should not reason as though it had received only a transcript, and any claim about tone, pauses, or facial behavior should be supported by the recording.
Critical weaknesses
Weakness 1:The paper proposes a single unified score to compare all models, but never explains what that score is actually measuring. Its own results show that performance differs a lot across datasets, languages, and modalities. If performance differs that much, it is unclear whether combining everything into one score is still meaningful. Weakness 2:The paper treats reasoning quality and prediction correctness as two separate things, but only states this without giving a reason. Both come from the same model answering the same input. If they come from the same source, treating them as two independent evaluation dimensions needs justification. Weakness 3:The failure taxonomy is presented as exhaustive and mutually exclusive without a theoretical rule. Multiple failure types can plausibly co-occur in one wrong prediction.
Questions for authors
(1)The paper presents a unified benchmark for comparing models across datasets, languages, and modalities. But performance varies sharply across these dimensions. If the variation comes from these dimensions themselves, it is unclear what the benchmark compares. Justify why a unified benchmark is necessary here. (2)Explain why the failure taxonomy is exhaustive and mutually exclusive, and state the rule that assigns a failure to one type. (3)Clarify whether the four quality dimensions are theoretically independent or correlated by construction. (4) Clarify whether severity estimation requires a different theoretical mechanism from classification. (5) Explain why explanation quality is a valid proxy for clinical knowledge.