- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces MentaBench, a benchmark for text-based mental-health signal detection with small language models (SLMs). It aggregates 21 public datasets into 40 classification tasks spanning stress, depression, suicidal risk, mental-health symptoms, affective states, interpersonal-risk factors, wellbeing, and empathy. The paper evaluates 11 SLMs under zero-shot, few-shot, and LoRA fine-tuning regimes, with BACC as the main cross-task metric, and compares against TF-IDF/logistic regression, BERT, and Gemini Flash/Pro baselines. It also reports deployment-oriented metrics such as latency, throughput, and RAM on an iPhone 15 Pro Max. The main conclusions are that fine-tuned SLMs substantially improve over zero/few-shot SLMs, that Gemma4-E4B has the strongest predictive performance, and that smaller LLaMA models offer a better deployment-efficiency tradeoff for on-device mental-health monitoring.
Critical strengths
-
The paper makes a useful effort to consolidate scattered mental-health text classification resources under a common evaluation protocol. This is consequential because the area currently contains many small, heterogeneous datasets and narrow single-task evaluations; a well-constructed benchmark could help researchers compare models and understand task-level variation more systematically.
-
The paper includes deployment-oriented measurements rather than only predictive scores. This is relevant to the stated use case because latency, throughput, and memory matter for any realistic low-resource or on-device mental-health monitoring system. This makes the empirical setup more practically motivated than a purely accuracy-based benchmark.
Critical weaknesses
-
The core motivation and contribution are not sufficiently convincing. I am not convinced that SLMs need a dedicated benchmark in the way the paper argues, nor that smaller models solve the privacy problem. From a user's perspective, very few people can deploy and maintain models locally on their own phones; if phone manufacturers deploy models on-device, users still have to trust the vendor, operating system, app, logging/telemetry, and surrounding data pipeline. On-device inference may reduce some server-transfer risks, but it does not by itself solve privacy. Relatedly, the paper's comparison against LLMs is not strong enough to support broad claims that SLMs outperform LLMs: Gemini2.5-Flash-Lite has an undisclosed parameter count and is positioned as a cost-efficient, low-latency Lite model, while newer and stronger Gemini Flash models are already available. Overall, the work reads to me more like large-scale dataset integration, label mapping/standardisation, and evaluation than a benchmark with a clearly new task definition, new clinical grounding, or new methodological contribution. This directly weakens the paper's significance and originality.
-
The benchmark validity and evaluation design do not support the strength of the conclusions. Many included tasks appear to be proxy/source-label tasks or generic affect/sentiment/empathy/wellness tasks rather than clinically meaningful mental-health signal detection. Some suicide-risk labels are based on subreddit/source membership, and high performance may therefore reflect platform or community cues rather than robust mental-health inference. In addition, the fine-tuning protocol appears to use within-dataset random splits with exact duplicate removal, but without clear user-level, thread-level, temporal, near-duplicate, or pretraining-contamination controls. This matters because the paper claims cross-dataset robustness, deployment readiness, and real-world mental-health signal detection, but the current evidence could still be driven by dataset artifacts and leakage-like shortcuts.
Questions for authors
-
What is the intended contribution beyond aggregating existing datasets and running model evaluations? In particular, why does SLM-based mental-health signal detection require a dedicated benchmark rather than being evaluated with the same benchmark under additional efficiency/deployment metrics?
-
What leakage and artifact controls were applied beyond exact duplicate removal? Were train/validation/test splits user-level, thread-level, source-level, or temporal where applicable? Was near-duplicate overlap or possible pretraining contamination of public Reddit/Twitter/Hugging Face/Kaggle examples checked?
-
Why are the LLM baselines limited to relatively weak or outdated Lite models such as Gemini2.5-Flash-Lite, rather than stronger and newer Gemini Flash models? Are the LLM baselines given comparable supervision/adaptation budgets to the fine-tuned SLMs? The answer would materially affect whether I believe the claim that SLMs outperform LLMs.
-
What exactly will be released: raw text, processed text, dataset pointers, scripts, or model checkpoints? How do you address consent, contextual integrity, and re-identification risks for sensitive suicide/depression-related posts, especially since the appendix says no additional formal anonymization was applied beyond excluding usernames/account metadata?
6 comments
Part 2/3
W1/Q3 llm baseline models
In addition to the Gemini Flash-Lite baselines, we have already evaluated Gemini 3.1 Pro across all 40 tasks, as reported in Appendix Table 6, as in the following table:
| Model | ACC | BACC | F1 | AUPRC |
|---|---|---|---|---|
| Gemini 3.1 Pro | 0.70 | 0.67 | 0.63 | 0.60 |
| Gemini 2.5 Flash-Lite | 0.79 | 0.77 | 0.74 | 0.69 |
| Gemma4-E4B (fine-tuned) | 0.82 | 0.76 | 0.76 | 0.70 |
Our result shows that the stronger baseline did not perform better under our evaluated protocol. Therefore, we describe SLMs as achieving competitive performance rather than consistently outperforming LLMs. We have revised the conclusion to state that fine-tuned SLMs achieve competitive performance under our evaluated settings, rather than uniformly outperforming LLMs. We will restrict conclusions to the evaluated models, tasks, and adaptation settings and add clarification.
We have added the following clarification to Appendix C.1: “Fine-tuned Gemma4-E4B achieved higher ACC, F1, and AUPRC, whereas Gemini 2.5 Flash-Lite achieved slightly higher BACC (0.77 versus 0.76). These results indicate that SLMs achieve competitive performance rather than consistently outperforming LLMs.”
Q1 What is the intended contribution…why require a dedicated SLM benchmark…
While language models have been widely applied, and being employed in the field of mental health, in both clinical intervention and general monitoring [5-7], there is a gap in establishing a systematic framework for assessing the predictive capabilities and deployment feasibility of SLMs across diverse mental health-related tasks. Our research question concerns how different SLMs and adaptation strategies perform across these tasks, and how predictive performance relates to practical deployment costs.
Note that MentaBench does not introduce a new clinical task, a new annotation methodology, or a new model architecture. Instead, its intended contribution is a unified evaluation resource that makes it possible to compare model behaviour across heterogeneous mental-health-related tasks, adaptation regimes, and deployment constraints.
Previous studies have demonstrated the potential of SLMs for mental health classification [8]. However, these evaluations often cover fewer datasets, use different experimental protocols, or focus on predictive performance without examining mobile deployment. MentaBench addresses these gaps by bringing together 21 datasets and 40 tasks, evaluating 11 SLMs under zero-shot, few-shot, and LoRA fine-tuning settings, and incorporating measurements of latency, throughput, and memory on an actual mobile device. Therefore, a more systematic evaluation is needed to assess how effectively SLMs perform mental health and wellbeing tasks and how their predictive performance relates to practical deployment costs.
Reference:
- Minding Your Mind: Get involved with research. Accessed 2026.
- Enabling Research and Clinical Use of Patient-Generated Health Data (the mindLAMP Platform): Digital Phenotyping Study. 2022.
- New Tools for New Research in Psychiatry: A Scalable and Customizable Platform to Empower Data Driven Smartphone Research. 2016.
- Assessment of the Data Sharing and Privacy Practices of Smartphone Apps for Depression and Smoking Cessation. 2019.
- Randomized Trial of a Generative AI Chatbot for Mental Health Treatment. 2025.
- Large Language Models and Text Embeddings for Detecting Depression and Suicide in Patient Narratives. 2025.
- Detection of Suicidality from Medical Text Using Privacy-Preserving Large Language Models. 2025.
- Beyond Scale: Small Language Models Are Comparable to GPT-4 in Mental Health Understanding. 2025.
- Wysa: Frequently Asked Questions. Accessed 2026. Website.
- Slingshot AI: Ash—AI for Mental Health. Accessed 2026. Website.
Our response continues in the next comment (Part 3/3).
Part 3/3
W2/Q2 What leakage and controls were applied…? Were train/validation test split in multilevel applicable? Was near-duplicate overlap checked?
Data independence and leakage. We removed 31,507 exact text–label duplicates. Near-duplicates, paraphrases, and examples from shared users or threads may remain. User-, thread-, source-, and temporal separation have not been verified across all tasks because the necessary metadata are not consistently available in the anonymised source datasets.
Pretraining contamination and future controls. Public benchmark examples may have appeared in model pretraining corpora, although this remains unverified. Future versions should document split construction and label provenance, assess near-duplicate overlap, and apply metadata-aware separation where feasible. Newly collected, independently annotated cohorts would strengthen evidence of generalisation but fall outside this benchmark’s scope of evaluating models across mental health and wellbeing tasks and deployment settings.
We have added these in Appendix A.1 (Benchmark Construction).
The train split information is listed in line 218 according to the different tasks of the datasets. Specifically, we used 72% for training, 8% for validation, and 20% for testing across all settings. Our evaluation is designed to use fixed, disjoint training, validation, and test partitions, with a consistent held-out test set across adaptation settings. Few-shot demonstrations were sampled only from the training partition, using fixed seeds and the same support examples across models. We will also reconcile the main-text split specification with the training-size ablations in Appendix Tables 11–15 and provide the exact sampling procedure, split sizes, random seeds, and few-shot selection rules. The exact sampling procedure, split sizes, random seed and few-shot selection rules are now described in Appendix B.
Q4 what will be released… how to address consent, contextual integrity, and re-identification risks…
We thank the reviewer for raising these important concerns. Our study uses previously released datasets, and Appendix A documents their sources and available licensing information. Our current preprocessing retains the text and task labels while removing available usernames, account identifiers, and other metadata. Our release will include code and benchmark documentation, with trained model checkpoints released subject to applicable licences, data-use restrictions, and privacy considerations. We will not release original datasets or raw or processed source text.
We will clarify the following release policy:
Benchmark resources. We will release task definitions, label mappings, source dataset references, preprocessing and evaluation scripts, model configurations, and reproducibility information, subject to the conditions of the original datasets.
Sensitive text. All text analysed comes from publicly available datasets. No raw or processed source text will be redistributed.
Model artifacts. Checkpoint release will be assessed separately against model licenses, underlying data-use restrictions, and potential privacy risks.
Privacy and consent limitations. Our study involves secondary analysis of existing public datasets, without participant recruitment or new data collection. We therefore do not propose obtaining additional participant consent.
Deployment safety. MentaBench evaluates text classification and computational efficiency. It does not evaluate clinical diagnostic accuracy, intervention outcomes, crisis-response safety, or acceptability to patients and clinicians. Figure 13 illustrates a potential downstream workflow, but the intervention components are out of the scope of our study.
We relied on the original dataset providers’ documented anonymisation or de-identification procedures where available. Our benchmark excludes usernames and account metadata and retains only the post text and task labels. No additional formal anonymisation was applied. These exclusions reduce identification risks. We have included this clarification in Appendix A.4 (Release Policy).
We appreciate the reviewer's emphasis on these issues. We hope our detailed responses have adequately addressed your concerns, and we would be very grateful if you might consider raising your score in light of these updates.
Part 1/3
We appreciate the reviewer's comments. We agree that several claims in the current manuscript require clearer boundaries. We address each concern below.
While language models have been widely applied, and being employed in the field of mental health, in both clinical intervention and general monitoring [5-7], there is a gap in establishing a systematic framework for assessing the predictive capabilities and deployment feasibility of SLMs across diverse mental health-related tasks. Our central research question is whether, and how well, SLMs can perform these tasks, and how their predictive performance relates to practical deployment costs.
Few people can deploy and maintain models locally on their phones. However, our intended users are research and clinical development teams, rather than individuals expected to deploy and maintain models themselves. Such teams already maintain smartphone tools for mental health monitoring and support, including the Minding Your Mind research app, mindLAMP, Beiwe, Wysa, Ash, and Therabot [1–3, 5, 9-10]. These platforms illustrate an existing setting in which developers could evaluate compact language models for analysing user-authorised text. MentaBench supports this process by comparing predictive performance, adaptation strategies, and deployment costs.
On-device inference does not eliminate all privacy tasks, that it does not address risks arising from application providers, device software, local storage, or the underlying data collection process. However, it can reduce the need to transmit raw user text to a remote inference service. We have clarified these limitations in Appendix C.3 (Limitation and Future Direction) as follows: “On-device inference can reduce the transmission of sensitive user text to remote inference services, but it does not eliminate privacy risks associated with application providers, device software, local storage, logging, or data collection practices. Our evaluation does not assess end-to-end privacy or security. Local inference should therefore be considered one component of a broader privacy strategy, requiring system-level safeguards, informed consent, and transparent data governance in future user studies.”
Regarding privacy, local inference can reduce external transmission of sensitive text, but privacy still depends on the application, operating system, storage, telemetry, and surrounding data pipeline. Documented discrepancies between mental health apps’ privacy disclosures and actual data-sharing behaviour reinforce this distinction [4].
Reference:
Our response continues in the next comment (Part 2/3).