acceptodds
Withdrawn from ICLR 2027

DepressionBench: A Multimodal Benchmark for Depression Recognition and Reasoning Analysis with Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Depression is a common and serious mental health condition, but its signs can be difficult to recognize in what people say, write, and do across everyday contexts. Large language models (LLMs) could help identify patterns in the language and behavior of people experiencing depression and explain the basis for their assessments. However, it remains unclear how reliably LLMs recognize signs of depression from text, audio, and video collected in different everyday contexts. We introduce DepressionBench, a curated large cohort datasets consists of 34,973 participants and a unified benchmark for depression recognition and analysis of model-reported reasoning. It covers text, audio, vision, and audiovisual inputs across four source scenarios and three languages. To assess depression-recognition performance and whether post-training improves models’ ability to identify signs of depression across modalities and everyday contexts, we evaluate twelve open-weight models under zero-shot and few-shot prompting, supervised fine-tuning, and reinforcement learning. Also, to examine whether the models’ explanations are consistent with the evidence in the inputs and to identify where their predictions fail, we assess the quality of their reported reasoning and analyze common prediction errors. DepressionBench provides a foundation for future research on multimodal depression recognition, including methods that generalize across everyday contexts. Our project page is available at: https://anonymous.4open.science/r/DepreesionBenchmark-C669

Primary area:
datasets and benchmarks
Keywords:
depression recognition, multimodal benchmark, large language models, mental health

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TitUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DepressionBench, a unified multimodal benchmark for depression recognition and reasoning analysis. It curates six public datasets covering text, audio, vision, and audiovisual inputs across four scenarios and three languages, with 34,973 participants in total. It evaluates twelve open-weight LLMs under zero-shot, few-shot, SFT, and RL settings on classification and severity estimation. It further uses an analyst agent to rate model-reported reasoning quality and to diagnose failure types and root causes. Three clinical experts reviewed 50 sampled assessments in a blinded protocol.

Critical strengths

Strength 1: The paper evaluates only open-weight models of at most 30B parameters. Most existing depression-recognition benchmarks rely on proprietary cloud models, so their results do not show how well locally deployable models perform. By restricting evaluation to this model class, the paper measures the setting that privacy-sensitive use actually requires: models that can run on a user's device without sending mental-health records to a cloud service. Strength 2: The paper examines model-reported reasoning in depression recognition. An analyst agent rates each explanation along four dimensions without seeing the reference label, then identifies failure types and root causes in clinically wrong predictions. Existing clinical reasoning benchmarks assess reasoning from textual medical questions or clinical cases. Here, a model listening to speech should not reason as though it had received only a transcript, and any claim about tone, pauses, or facial behavior should be supported by the recording.

Critical weaknesses

Weakness 1:The paper proposes a single unified score to compare all models, but never explains what that score is actually measuring. Its own results show that performance differs a lot across datasets, languages, and modalities. If performance differs that much, it is unclear whether combining everything into one score is still meaningful. Weakness 2:The paper treats reasoning quality and prediction correctness as two separate things, but only states this without giving a reason. Both come from the same model answering the same input. If they come from the same source, treating them as two independent evaluation dimensions needs justification. Weakness 3:The failure taxonomy is presented as exhaustive and mutually exclusive without a theoretical rule. Multiple failure types can plausibly co-occur in one wrong prediction.

Questions for authors

(1)The paper presents a unified benchmark for comparing models across datasets, languages, and modalities. But performance varies sharply across these dimensions. If the variation comes from these dimensions themselves, it is unclear what the benchmark compares. Justify why a unified benchmark is necessary here. (2)Explain why the failure taxonomy is exhaustive and mutually exclusive, and state the rule that assigns a failure to one type. (3)Clarify whether the four quality dimensions are theoretically independent or correlated by construction. (4) Clarify whether severity estimation requires a different theoretical mechanism from classification. (5) Explain why explanation quality is a valid proxy for clinical knowledge.

Reviewer_RAEQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces DepressionBench, a multimodal benchmark for evaluating depression recognition and model-reported reasoning across text, audio, vision, and audiovisual inputs. The benchmark curates six public datasets comprising 34,973 participants across four source scenarios and three languages, covering both binary classification and depression-severity estimation. Twelve open-weight models are evaluated using zero-shot and few-shot prompting, supervised fine-tuning, and reinforcement learning, with results showing substantial variation across datasets and modalities and no consistently effective adaptation strategy. The paper also introduces an analyst agent that evaluates whether model explanations are grounded in the available evidence and analyzes common failure types and underlying causes in incorrect predictions.

Critical strengths
  • The idea of consolidating multiple depression-recognition datasets under a common evaluation framework is valuable. Existing studies use different datasets, task definitions, and evaluation protocols, making their findings difficult to compare directly.

  • The benchmark considers diverse forms of evidence, including text, audio, vision, and audiovisual inputs. This broad coverage reflects the fact that depression-related signals may appear across multiple aspects of language and behavior.

  • The analysis extends beyond prediction performance to model-reported reasoning. Evaluating whether explanations are grounded in the available evidence is particularly important in a sensitive application such as depression recognition.

Critical weaknesses
  • The six datasets are fundamentally heterogeneous, making the meaning of a unified benchmark unclear. They differ not only in modality but also in source scenario, language, participant population, sample size, input representation, clinical instrument, and labeling procedure. Applying a common evaluation protocol does not by itself resolve these differences or establish that the resulting scores measure comparable tasks.

  • Most cross-dataset comparisons cannot isolate the effect of modality. Differences between text, audio, and vision results may arise from dataset, language, scenario, population, class balance, or label quality rather than modality itself. AVEC2014 enables a useful controlled comparison using the same participants and recordings, but this advantage does not extend to most comparisons across the six datasets.

  • The harmonization of clinical targets requires stronger justification. The benchmark combines BDI-II, PHQ-8, SDS, binary annotations, and self-reported diagnostic markers. It is unclear whether mapping these different targets to dataset-specific clinical bands provides sufficient comparability for a unified analysis of classification and severity estimation.

  • The benchmark’s novelty beyond aggregating existing public datasets is not sufficiently clear. Because no new participant data are collected, the main contribution should come from a principled harmonization methodology, controlled cross-dataset comparisons, reusable annotations, or new analyses that could not be obtained from the individual datasets. The paper should more clearly identify which benchmark components constitute new research contributions.

  • The reported total size may overstate the scale of the multimodal evidence. Of the 34,973 participants, 32,570 come from the WU3D social-media text dataset, while the remaining datasets are substantially smaller. The paper should avoid implying that all modalities are supported by a comparably large cohort.

Questions for authors
  • What specific harmonization methodology makes results comparable across datasets with different scenarios, languages, populations, clinical scales, and labeling procedures?

  • Which analyses support controlled conclusions about modality, and which should be interpreted only as dataset-specific evaluations?

  • What evidence supports the comparability of severity labels derived from BDI-II, PHQ-8, and SDS?

  • Beyond aggregating and reevaluating existing public datasets, what new benchmark assets or methodological contributions will DepressionBench provide?

  • Could the authors report and discuss the dataset-level distribution of participants more prominently to avoid conflating the overall sample size with the scale of the multimodal subsets?

Reviewer_VdmeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a benchmark for depression detection. It combines multiple existing depression detection datasets, along with the evaluations from multiple LLMs.

Critical strengths

The motivation of the work, that due to various external factors, such as how data is collected, labeled, therefore a unified benchmark is needed for both performance and reasoning, is well justified.

Critical weaknesses

1. The starting of the abstract is a bit of contradicted. The models use what people speak, say, and do for depression detection, but the authors started with claiming that its signs can be difficult to recognize. I believe better phrases can be used here.

2. Under label quality, authors claim that they exclude datasets that used indirect labels. Can you provide a few concrete examples of such datasets?

3. Under section 3.2, you mentioned that accuracy is not related to LLM size. First, such conclusion should not only be discussed in Appendix; second, this result is driven by text-based performance, which should be clearly stated.

4. All the figures are hard to read due to small font size and compact structure.

5. So even the benchmark building is multimodal, (I assume) due to the capabilities of different LLMs used in this work, their performance is still restricted under one modality? This makes the title of the work a bit confounding.

Questions for authors

As a researcher working on speech-based depression detection, for the past a few years, by all means, I have never able to access the raw data from AVEC 2014 or D-Vlog. I am curious how the author of this work obtained access to these datasets.

Also, I assume it is granted by the author of each of these datasets, that you can combine them together for a benchmark purpose?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.