acceptodds
Under review as a conference paper at ICLR 2027

MentaBench: Benchmarking Mental Health Signal Detection from Text with Small Language Models

Abstract

Text-based mental health signals can enable just-in-time adaptive interventions (JITAIs), but practical deployment requires models that are accurate, efficient, low-latency, and privacy-preserving. While large language models (LLMs) perform well on mental health prediction tasks, their computational cost and reliance on centralized inference limit their applicability in sensitive or resource-constrained settings. Small language models (SLMs) offer a promising alternative; however, their reliability for mental health signal detection remains underexplored, with existing evaluations fragmented across datasets, tasks, and adaptation settings. We introduce **MentaBench**, a unified benchmark for evaluating SLMs on text-based mental health signal detection. We curated a large-scale mental health corpus which comprises 21 public datasets spanning 40 tasks and diverse text sources, and evaluates 11 representative SLMs under zero-shot, few-shot, and fine-tuning regimes. We benchmarking results show that fine-tuned SLMs on mental health datasets can outperform traditional machine learning and encoder-based baselines as well as LLMs. Evaluating on both predictive performance and deployment-related metrics, we further find that Gemma4-E4B achieves the strongest predictive performance, while LLaMA3.2-1B and LLaMA3.2-3B reach a more balanced trade-off for real-world deployment scenarios. **MentaBench** provides a standardized foundation for assessing the generalization, adaptability, and deployability of efficient language models for mental health monitoring and future JITAI systems. Code is available at: **[MentaBench Code](https://anonymous.4open.science/r/MentaBench_Code-65F9)**.

Primary area:
datasets and benchmarks
Keywords:
benchmark, Small Language Models, mental health, social media text, mental wellbeing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hcVPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MentaBench, a benchmark for text-based mental-health signal detection with small language models (SLMs). It aggregates 21 public datasets into 40 classification tasks spanning stress, depression, suicidal risk, mental-health symptoms, affective states, interpersonal-risk factors, wellbeing, and empathy. The paper evaluates 11 SLMs under zero-shot, few-shot, and LoRA fine-tuning regimes, with BACC as the main cross-task metric, and compares against TF-IDF/logistic regression, BERT, and Gemini Flash/Pro baselines. It also reports deployment-oriented metrics such as latency, throughput, and RAM on an iPhone 15 Pro Max. The main conclusions are that fine-tuned SLMs substantially improve over zero/few-shot SLMs, that Gemma4-E4B has the strongest predictive performance, and that smaller LLaMA models offer a better deployment-efficiency tradeoff for on-device mental-health monitoring.

Critical strengths
  1. The paper makes a useful effort to consolidate scattered mental-health text classification resources under a common evaluation protocol. This is consequential because the area currently contains many small, heterogeneous datasets and narrow single-task evaluations; a well-constructed benchmark could help researchers compare models and understand task-level variation more systematically.

  2. The paper includes deployment-oriented measurements rather than only predictive scores. This is relevant to the stated use case because latency, throughput, and memory matter for any realistic low-resource or on-device mental-health monitoring system. This makes the empirical setup more practically motivated than a purely accuracy-based benchmark.

Critical weaknesses
  1. The core motivation and contribution are not sufficiently convincing. I am not convinced that SLMs need a dedicated benchmark in the way the paper argues, nor that smaller models solve the privacy problem. From a user's perspective, very few people can deploy and maintain models locally on their own phones; if phone manufacturers deploy models on-device, users still have to trust the vendor, operating system, app, logging/telemetry, and surrounding data pipeline. On-device inference may reduce some server-transfer risks, but it does not by itself solve privacy. Relatedly, the paper's comparison against LLMs is not strong enough to support broad claims that SLMs outperform LLMs: Gemini2.5-Flash-Lite has an undisclosed parameter count and is positioned as a cost-efficient, low-latency Lite model, while newer and stronger Gemini Flash models are already available. Overall, the work reads to me more like large-scale dataset integration, label mapping/standardisation, and evaluation than a benchmark with a clearly new task definition, new clinical grounding, or new methodological contribution. This directly weakens the paper's significance and originality.

  2. The benchmark validity and evaluation design do not support the strength of the conclusions. Many included tasks appear to be proxy/source-label tasks or generic affect/sentiment/empathy/wellness tasks rather than clinically meaningful mental-health signal detection. Some suicide-risk labels are based on subreddit/source membership, and high performance may therefore reflect platform or community cues rather than robust mental-health inference. In addition, the fine-tuning protocol appears to use within-dataset random splits with exact duplicate removal, but without clear user-level, thread-level, temporal, near-duplicate, or pretraining-contamination controls. This matters because the paper claims cross-dataset robustness, deployment readiness, and real-world mental-health signal detection, but the current evidence could still be driven by dataset artifacts and leakage-like shortcuts.

Questions for authors
  1. What is the intended contribution beyond aggregating existing datasets and running model evaluations? In particular, why does SLM-based mental-health signal detection require a dedicated benchmark rather than being evaluated with the same benchmark under additional efficiency/deployment metrics?

  2. What leakage and artifact controls were applied beyond exact duplicate removal? Were train/validation/test splits user-level, thread-level, source-level, or temporal where applicable? Was near-duplicate overlap or possible pretraining contamination of public Reddit/Twitter/Hugging Face/Kaggle examples checked?

  3. Why are the LLM baselines limited to relatively weak or outdated Lite models such as Gemini2.5-Flash-Lite, rather than stronger and newer Gemini Flash models? Are the LLM baselines given comparable supervision/adaptation budgets to the fine-tuned SLMs? The answer would materially affect whether I believe the claim that SLMs outperform LLMs.

  4. What exactly will be released: raw text, processed text, dataset pointers, scripts, or model checkpoints? How do you address consent, contextual integrity, and re-identification risks for sensitive suicide/depression-related posts, especially since the appendix says no additional formal anonymization was applied beyond excluding usernames/account metadata?

6 comments

We appreciate the reviewer's comments. We agree that several claims in the current manuscript require clearer boundaries. We address each concern below.


W1.1 SLMs need a dedicate benchmark

While language models have been widely applied, and being employed in the field of mental health, in both clinical intervention and general monitoring [5-7], there is a gap in establishing a systematic framework for assessing the predictive capabilities and deployment feasibility of SLMs across diverse mental health-related tasks. Our central research question is whether, and how well, SLMs can perform these tasks, and how their predictive performance relates to practical deployment costs.


W1.2 privacy problem

Few people can deploy and maintain models locally on their phones. However, our intended users are research and clinical development teams, rather than individuals expected to deploy and maintain models themselves. Such teams already maintain smartphone tools for mental health monitoring and support, including the Minding Your Mind research app, mindLAMP, Beiwe, Wysa, Ash, and Therabot [1–3, 5, 9-10]. These platforms illustrate an existing setting in which developers could evaluate compact language models for analysing user-authorised text. MentaBench supports this process by comparing predictive performance, adaptation strategies, and deployment costs.

On-device inference does not eliminate all privacy tasks, that it does not address risks arising from application providers, device software, local storage, or the underlying data collection process. However, it can reduce the need to transmit raw user text to a remote inference service. We have clarified these limitations in Appendix C.3 (Limitation and Future Direction) as follows: “On-device inference can reduce the transmission of sensitive user text to remote inference services, but it does not eliminate privacy risks associated with application providers, device software, local storage, logging, or data collection practices. Our evaluation does not assess end-to-end privacy or security. Local inference should therefore be considered one component of a broader privacy strategy, requiring system-level safeguards, informed consent, and transparent data governance in future user studies.”

Regarding privacy, local inference can reduce external transmission of sensitive text, but privacy still depends on the application, operating system, storage, telemetry, and surrounding data pipeline. Documented discrepancies between mental health apps’ privacy disclosures and actual data-sharing behaviour reinforce this distinction [4].


Reference:

  1. Minding Your Mind: Get involved with research. Accessed 2026.
  2. Enabling Research and Clinical Use of Patient-Generated Health Data (the mindLAMP Platform): Digital Phenotyping Study. 2022.
  3. New Tools for New Research in Psychiatry: A Scalable and Customizable Platform to Empower Data Driven Smartphone Research. 2016.
  4. Assessment of the Data Sharing and Privacy Practices of Smartphone Apps for Depression and Smoking Cessation. 2019.
  5. Randomized Trial of a Generative AI Chatbot for Mental Health Treatment. 2025.
  6. Large Language Models and Text Embeddings for Detecting Depression and Suicide in Patient Narratives. 2025.
  7. Detection of Suicidality from Medical Text Using Privacy-Preserving Large Language Models. 2025.
  8. Beyond Scale: Small Language Models Are Comparable to GPT-4 in Mental Health Understanding. 2025.
  9. Wysa: Frequently Asked Questions. Accessed 2026. Website.
  10. Slingshot AI: Ash—AI for Mental Health. Accessed 2026. Website.

Our response continues in the next comment (Part 2/3).

W1/Q3 llm baseline models

In addition to the Gemini Flash-Lite baselines, we have already evaluated Gemini 3.1 Pro across all 40 tasks, as reported in Appendix Table 6, as in the following table:

ModelACCBACCF1AUPRC
Gemini 3.1 Pro0.700.670.630.60
Gemini 2.5 Flash-Lite0.790.770.740.69
Gemma4-E4B (fine-tuned)0.820.760.760.70

Our result shows that the stronger baseline did not perform better under our evaluated protocol. Therefore, we describe SLMs as achieving competitive performance rather than consistently outperforming LLMs. We have revised the conclusion to state that fine-tuned SLMs achieve competitive performance under our evaluated settings, rather than uniformly outperforming LLMs. We will restrict conclusions to the evaluated models, tasks, and adaptation settings and add clarification.

We have added the following clarification to Appendix C.1: “Fine-tuned Gemma4-E4B achieved higher ACC, F1, and AUPRC, whereas Gemini 2.5 Flash-Lite achieved slightly higher BACC (0.77 versus 0.76). These results indicate that SLMs achieve competitive performance rather than consistently outperforming LLMs.”


Q1 What is the intended contribution…why require a dedicated SLM benchmark…

While language models have been widely applied, and being employed in the field of mental health, in both clinical intervention and general monitoring [5-7], there is a gap in establishing a systematic framework for assessing the predictive capabilities and deployment feasibility of SLMs across diverse mental health-related tasks. Our research question concerns how different SLMs and adaptation strategies perform across these tasks, and how predictive performance relates to practical deployment costs.

Note that MentaBench does not introduce a new clinical task, a new annotation methodology, or a new model architecture. Instead, its intended contribution is a unified evaluation resource that makes it possible to compare model behaviour across heterogeneous mental-health-related tasks, adaptation regimes, and deployment constraints.

Previous studies have demonstrated the potential of SLMs for mental health classification [8]. However, these evaluations often cover fewer datasets, use different experimental protocols, or focus on predictive performance without examining mobile deployment. MentaBench addresses these gaps by bringing together 21 datasets and 40 tasks, evaluating 11 SLMs under zero-shot, few-shot, and LoRA fine-tuning settings, and incorporating measurements of latency, throughput, and memory on an actual mobile device. Therefore, a more systematic evaluation is needed to assess how effectively SLMs perform mental health and wellbeing tasks and how their predictive performance relates to practical deployment costs.


Reference:

  1. Minding Your Mind: Get involved with research. Accessed 2026.
  2. Enabling Research and Clinical Use of Patient-Generated Health Data (the mindLAMP Platform): Digital Phenotyping Study. 2022.
  3. New Tools for New Research in Psychiatry: A Scalable and Customizable Platform to Empower Data Driven Smartphone Research. 2016.
  4. Assessment of the Data Sharing and Privacy Practices of Smartphone Apps for Depression and Smoking Cessation. 2019.
  5. Randomized Trial of a Generative AI Chatbot for Mental Health Treatment. 2025.
  6. Large Language Models and Text Embeddings for Detecting Depression and Suicide in Patient Narratives. 2025.
  7. Detection of Suicidality from Medical Text Using Privacy-Preserving Large Language Models. 2025.
  8. Beyond Scale: Small Language Models Are Comparable to GPT-4 in Mental Health Understanding. 2025.
  9. Wysa: Frequently Asked Questions. Accessed 2026. Website.
  10. Slingshot AI: Ash—AI for Mental Health. Accessed 2026. Website.

Our response continues in the next comment (Part 3/3).

W2/Q2 What leakage and controls were applied…? Were train/validation test split in multilevel applicable? Was near-duplicate overlap checked?

Data independence and leakage. We removed 31,507 exact text–label duplicates. Near-duplicates, paraphrases, and examples from shared users or threads may remain. User-, thread-, source-, and temporal separation have not been verified across all tasks because the necessary metadata are not consistently available in the anonymised source datasets.

Pretraining contamination and future controls. Public benchmark examples may have appeared in model pretraining corpora, although this remains unverified. Future versions should document split construction and label provenance, assess near-duplicate overlap, and apply metadata-aware separation where feasible. Newly collected, independently annotated cohorts would strengthen evidence of generalisation but fall outside this benchmark’s scope of evaluating models across mental health and wellbeing tasks and deployment settings.

We have added these in Appendix A.1 (Benchmark Construction).

The train split information is listed in line 218 according to the different tasks of the datasets. Specifically, we used 72% for training, 8% for validation, and 20% for testing across all settings. Our evaluation is designed to use fixed, disjoint training, validation, and test partitions, with a consistent held-out test set across adaptation settings. Few-shot demonstrations were sampled only from the training partition, using fixed seeds and the same support examples across models. We will also reconcile the main-text split specification with the training-size ablations in Appendix Tables 11–15 and provide the exact sampling procedure, split sizes, random seeds, and few-shot selection rules. The exact sampling procedure, split sizes, random seed and few-shot selection rules are now described in Appendix B.


Q4 what will be released… how to address consent, contextual integrity, and re-identification risks…

We thank the reviewer for raising these important concerns. Our study uses previously released datasets, and Appendix A documents their sources and available licensing information. Our current preprocessing retains the text and task labels while removing available usernames, account identifiers, and other metadata. Our release will include code and benchmark documentation, with trained model checkpoints released subject to applicable licences, data-use restrictions, and privacy considerations. We will not release original datasets or raw or processed source text.

We will clarify the following release policy:

Benchmark resources. We will release task definitions, label mappings, source dataset references, preprocessing and evaluation scripts, model configurations, and reproducibility information, subject to the conditions of the original datasets.

Sensitive text. All text analysed comes from publicly available datasets. No raw or processed source text will be redistributed.

Model artifacts. Checkpoint release will be assessed separately against model licenses, underlying data-use restrictions, and potential privacy risks.

Privacy and consent limitations. Our study involves secondary analysis of existing public datasets, without participant recruitment or new data collection. We therefore do not propose obtaining additional participant consent.

Deployment safety. MentaBench evaluates text classification and computational efficiency. It does not evaluate clinical diagnostic accuracy, intervention outcomes, crisis-response safety, or acceptability to patients and clinicians. Figure 13 illustrates a potential downstream workflow, but the intervention components are out of the scope of our study.

We relied on the original dataset providers’ documented anonymisation or de-identification procedures where available. Our benchmark excludes usernames and account metadata and retains only the post text and task labels. No additional formal anonymisation was applied. These exclusions reduce identification risks. We have included this clarification in Appendix A.4 (Release Policy).


We appreciate the reviewer's emphasis on these issues. We hope our detailed responses have adequately addressed your concerns, and we would be very grateful if you might consider raising your score in light of these updates.

Reviewer_AgcFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper benchmarks mental health prediction tasks for small language models (1B-7B parameters). The paper argues that existing datasets and benchmarks are either task-specific and data-limited or focus on particular ML techniques. In addition, small language models are suitable for applications in just-in-time adaptive interventions. The paper proposes MentaBench, which consolidates many datasets and tasks and includes evaluation protocols for zero-shot, few-shot, and fine-tuned settings. The experimental results show that fine-tuned LLaMA3.2-3B offers the best trade-off between accuracy and on-device throughput (iPhone 15 Pro Max). In addition, the fine-tuned Gemma4-E4B can achieve comparable or better accuracy than LLMs (Gemini2.0-Flash-Lite and Gemini2.5-Flash-Lite) with lower inference time.

Critical strengths

The paper consolidates datasets and tasks into a single benchmark and reports accuracy and on-device runtimes. The benchmark would help researchers advance mental health classification.

Critical weaknesses

While the dataset collection is helpful, the position on SLM specifics is unclear, and the evaluation protocol is underspecified as a benchmark.

  1. The dataset collection consists only of text classification tasks. While the evaluation protocol includes zero-shot, few-shot, and LoRA settings, nothing in the benchmark specifically requires SLMs. In addition, from a JITAI standpoint, given runtime and memory constraints, a specialized small model would be a more suitable goal.
  2. The training split for few-shot learning is missing. Since we need a support set to run this experiment, the benchmark should either provide a support set for each test sample or provide training data for other approaches to example selection (other than random, as in the paper).
  3. The choice of the device is questionable for future comparisons. The paper benchmarks on the iPhone 15 Pro Max. While this is helpful, it does not generalize well to future devices. The runtime methodology would need more analysis. For example, a controlled analysis of parameter count versus throughput, latency, memory, and predictive accuracy would strengthen the generality of its conclusions.
  4. The conclusion and the experiment results are somewhat inconsistent. The recommended models are LLaMA. However, the comparison against the LLMs is concluded using Gemma4-E4B. In addition, the LLM runtimes (via API calls) are missing.
  5. For all settings, how the prompts or instructions are constructed is not clear in the paper (prompts are available in the code). The paper also says its prompts are leaner than Mental-LLM's. The paper provides no comparison results (e.g., template lengths).
Questions for authors
  1. Is the benchmark intended to use the same train/val/test splits across all settings?
  2. If we compare LLaMA-based models' balanced accuracies with Gemini-based models, can you still conclude that the fine-tuned SLMs outperform LLMs?
5 comments

We thank the reviewer for the detailed feedback and for recognizing the usefulness of the consolidated benchmark and on-device evaluation. We address the main concerns below.


W1 Data are only text…[why] requires SLMs…would a specialized small model be more suitable?

We agree that the text classification tasks and evaluation settings used in MentaBench are not inherently restricted to SLMs or language models. However, while language models have been widely applied, and being employed in the field of mental health, in both clinical intervention and general monitoring [1-3], there is a gap in establishing a systematic framework for assessing the predictive capabilities and deployment feasibility of SLMs across diverse mental health-related tasks. Our research question concerns how different SLMs and adaptation strategies perform across these tasks, and how predictive performance relates to practical deployment costs.

We further discuss deployment costs. Our fine-tuned SLMs are single models: for each SLM, one LoRA adapter is trained on the combined training splits of all 40 tasks, and the resulting model serves every task through task-specific prompts. In contrast, specialized single-task models would require 40 separate models and 40× the storage. For a JITAI application, we adopt 10 seconds as the latency threshold, a widely used usability limit for holding a user's attention during a wait [6]. The following table (to be included as Table 23) reports latency from our iPhone 15 Pro Max measurements at 500 input tokens (cf. Table 20), together with the accuracy gained from fine-tuning.

ModelPer-task latency (s)Latency for all 40 tasks (s)BACC (zero-shot)BACC (fine-tuned)
LLaMA3.2-1B5.06202.40.430.69
TinyLLaMA-1.1B6.20248.00.470.56
Falcon-1.3B6.61264.40.480.56
Gemma4-E2B9.86394.40.610.70
Qwen2.5-3B11.31452.40.580.69
LLaMA3.2-3B11.55462.00.500.74
StableLM-3B11.64465.60.470.60
Gemma4-E4B12.10484.00.650.76
Qwen3-4B12.36494.40.460.70
Qwen3.5-4B13.16526.40.440.70
Phi4-mini-3.8B17.41696.40.560.71

Per-task latency on iPhone 15 Pro Max at 500 input / 128 output tokens (Tables 20–21). Latency for all 40 tasks is 40 × per-task latency, which is the same whether one model or 40 specialized models are used. BACC from Table 2.

Regarding specialised compact classifiers, our comparisons with BERT, Logistic Regression, and LIWC-based models provide initial baselines, and their accuracy are not as good as our evaluated language models. Additionally, generative language models may offer complementary capabilities, such as producing contextual explanations alongside classification outputs, which could be valuable for real-world mental health applications requiring both signal detection and interpretable outputs [4, 5]. While language models have been widely applied in mental health, including clinical intervention and general monitoring [7-9], it is necessary to focus on SLMs in their predictive performance relates to practical deployment costs.


Reference:

  1. Randomized Trial of a Generative AI Chatbot for Mental Health Treatment. 2025.
  2. Large Language Models and Text Embeddings for Detecting Depression and Suicide in Patient Narratives. 2025.
  3. Detection of Suicidality from Medical Text Using Privacy-Preserving Large Language Models. 2025.
  4. Explainable Depression Symptom Detection in Social Media. 2024.
  5. MentalGLM Series: Explainable Large Language Models for Mental Health Analysis on Chinese Social Media. 2025.
  6. Times, R. (1993). The 3 important limits. Режим доступу: https://www.nngroup.com.
  7. Using Language in Social Media Posts to Study the Network Dynamics of Depression Longitudinally. 2022.
  8. Just-in-Time Adaptive Interventions (JITAIs) in Mobile Health: Key Components and Design Principles for Ongoing Health Behavior Support. 2018.
  9. Journaling with Large Language Models: A Novel UX Paradigm for AI-Driven Personal Health Management. 2025.

W2/Q1… training split for few-shot learning…

The train split information is listed in line 218 according to the different tasks of the datasets. Specifically, we used 72% for training, 8% for validation, and 20% for testing across all settings. Our evaluation is designed to use fixed, disjoint training, validation, and test partitions, with a consistent held-out test set across adaptation settings. Few-shot demonstrations were sampled only from the training partition, using fixed seeds and the same support examples across models. We will also reconcile the main-text split specification with the training-size ablations in Appendix Tables 11–15 and provide the exact sampling procedure, split sizes, random seeds, and few-shot selection rules.


Our response continues in the next comment (Part 2/3)


W3.1 …iPhone 15 Pro Max not generalize well to future devices…

The iPhone 15 Pro Max (released in 2023) has 8 GB of RAM, making it a relevant lower-end device compared with, for example, the iPhone 18 Pro (released in 2026), which has 12 GB of RAM. As shown in Appendix Tables 20–21, larger SLMs can easily fit on the iPhone 18 Pro compared with the iPhone 15 Pro Max. For example, Gemma4-E4B requires 4.25 GB of RAM for inference. Therefore, we expect the framework and our model to fit easily on future, more powerful devices. We will include this in Appendix C.5.


W3.2 controlled runtime experiment

We thank the reviewer for this suggestion. We added a controlled analysis at a fixed input length of 500 tokens (ref Table 2, 20-21), holding the device, runtime, prompts and output length (128 tokens) constant. The following table (we will include it as Table 24) relates parameter count to throughput, latency, memory and predictive accuracy.

Table 24. Efficiency at 500 input tokens (Tables 20–21). BACC is averaged over the 40 tasks (Table 2); Gemma results are from the deployed setting.

ModelParamsOTPS (throughput) ↑Latency (s) ↓RAM (GB) ↓BACC zero-shotBACC fine-tuned
LLaMA3.2-1B1B31.205.061.950.430.69
TinyLLaMA-1.1B1.1B27.456.201.900.470.56
Falcon-1.3B1.3B23.806.612.200.480.56
Gemma4-E2B2B16.209.862.880.610.70
LLaMA3.2-3B3B13.8311.553.030.500.74
StableLM-3B3B14.2011.643.630.470.60
Qwen2.5-3B3B14.5511.313.780.580.69
Phi4-mini-3.8B3.8B10.3917.413.300.560.71
Qwen3-4B4B13.5512.363.300.460.70
Qwen3.5-4B4B12.8513.163.450.440.70
Gemma4-E4B4B13.1012.104.250.650.76

We can see that LLaMA achieves the best trade-off in terms of throughput, latency, and accuracy, while Gemma 4-E4B achieves the best accuracy within mobile memory (e.g., iPhone 15 Pro Max, 8GB RAM) constraints.


W4.1 …conclusion inconsistent.. best is LLaMA or Gemma4-E4B?

As shown in Table 2, 20 and 21. LLaMA3.2-1B achieves the highest throughput and lowest latency. Gemma4-E4B achieves the best accuracy on all six metrics within on-device memory constraints (4.25 GB). We therefore recommend LLaMA when latency and throughput matter most and Gemma when accuracy is the priority, and we have revised the paper accordingly.

ModelACCBACCF1AUPRCPrec.Rec.OTPS ↑Latency (s) ↓
LLaMA3.2-1B0.770.690.690.630.720.6931.205.06
LLaMA3.2-3B0.800.740.740.670.770.7413.8311.55
Gemma4-E2B0.760.700.710.650.730.7116.209.86
Gemma4-E4B0.820.760.760.700.790.7613.1012.10

W4.2 …LLM runtimes are missing…

SLM runtimes results are discussed in Tables 20–21. Gemini models were accessed through cloud APIs and were included in predictive comparisons only. Their response times would additionally depend on network conditions and remote infrastructure, so directly treating these timings as equivalent to local execution would be misleading.


W5 …prompts only in code.. different prompt?..

Thanks for the suggestion. We discussed prompt strategy in line 1199, page 23, Appendix C.1. Our prompting strategy uses task-specific instructions and explicit label spaces while maintaining a consistent structure across evaluated models. We compared our approach in Appendix Table 7, with the Mental-LLM prompt template, reporting broadly similar BACC values across four representative SLMs. For example, Gemma4-E4B achieves BACC 0.65 using both templates, while LLaMA3.2-3B obtains 0.50 and 0.51, respectively.

The comparison results are included in Appendix C.1, under “Prompt Strategies” (Table 7), which compares our prompt template with the Mental-LLM template across four SLMs. We will also include the actual templates in Appendix C.4, Prompt Template, clarify the differences between zero-shot and few-shot instructions, and report the tokenised prompt lengths: 113 tokens for our template versus 127 tokens for Mental-LLM.


Our response continues in the next comment (Part 3/3)


Q2 fine-tuned LLaMA3.2 and compare Gemini; is the conclusion hold?

We compare fine-tuned LLaMA3.2-1B and 3B with all Gemini baselines evaluated in the paper (Table 3 and Appendix Table 6). Gemini models are evaluated zero-shot (temperature 0, mean of three runs); SLMs are LoRA fine-tuned. The results are below:

ModelTypeAdaptationACCBACCF1AUPRC
Gemini 2.0 Flash-LiteLLM (API)zero-shot0.700.660.630.59
Gemini 2.5 Flash-LiteLLM (API)zero-shot0.790.770.740.69
Gemini 3.1 ProLLM (API)zero-shot0.700.670.630.60
LLaMA3.2-1BSLMLoRA fine-tuned0.770.690.690.63
LLaMA3.2-3BSLMLoRA fine-tuned0.800.740.740.67
Gemma4-E4BSLMLoRA fine-tuned0.820.760.760.70

As we can see, LLaMA3.2-3B outperforms Gemini 2.0 Flash-Lite and Gemini 3.1 Pro on ACC, BACC, F1 and AUPRC. However, it does not outperform Gemini 2.5 Flash-Lite, which achieves higher BACC (0.77 vs. 0.74) and AUPRC (0.69 vs. 0.67) with equal F1 (0.74); LLaMA3.2-1B is below Gemini 2.5 Flash-Lite on all four metrics. Among the SLMs, only Gemma4-E4B is on par with Gemini 2.5 Flash-Lite (higher ACC, F1 and AUPRC; slightly lower BACC). Therefore, we describe SLMs as achieving competitive performance rather than consistently outperforming LLMs.


Thank you again for these suggestions. We hope our detailed responses have adequately addressed your concerns, and we would be very grateful if you might consider raising your score in light of these updates.

Reviewer_h6n5OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

MentaBench evaluates small language models for text-based mental health signal detection across 21 public datasets and 40 classification tasks. It compares 11 SLMs under zero-shot, few-shot, and task-specific fine-tuning settings, alongside BERT, logistic regression, and larger language models.

Critical strengths
  1. Broad evaluation across tasks and adaptation settings. The benchmark brings together diverse datasets and evaluates three adaptation settings, allowing systematic comparisons beyond a single dataset or training setup.

  2. Evaluation on actual mobile hardware. Measuring latency, throughput, and memory on an iPhone, including their variation with input length, adds practical value beyond predictive-performance comparisons.

Critical weaknesses
  1. The connection between some benchmark tasks and mental-health signal detection needs stronger justification.

The benchmark includes Reddit community classification (T8), general sentiment classification (T28), and empathy prediction (T25), alongside symptom and risk detection. These tasks may be relevant to mental health, but identifying a post’s community or sentiment does not necessarily demonstrate an ability to detect distress or risk. Appendix B acknowledges these distinctions, yet it remains unclear what a high overall benchmark score tells us about mental-health signal detection specifically. The concern is not the inclusion of diverse tasks, but whether success across this collection supports the paper’s intended conclusions.

  1. The connection between the mobile evaluation and the JITAI motivation needs clarification.

The mobile evaluation provides useful evidence about computational efficiency, but the paper does not specify the monitoring scenario or response-time requirements underlying its JITAI motivation. This makes it difficult to assess how the reported latency differences should influence model selection alongside predictive performance. A concrete example connecting the relevant benchmark tasks, available text input, and required prediction time would clarify what these measurements imply for JITAI deployment and support the paper’s practical recommendations.

Questions for authors
  • How should performance on community classification, sentiment, and empathy contribute to an assessment of mental-health signal detection? What conclusions should readers draw from the overall score, beyond performance on the individual tasks?
  • Could the authors describe a representative JITAI scenario, specify its latency requirements, and explain how the reported accuracy–latency trade-offs would guide model selection in that scenario?
  • Beyond Scale [r1] already compares SLMs with larger models on mental-health classification tasks under zero-shot and few-shot settings. Could the authors discuss this predecessor and clarify which additional findings emerge from MentaBench’s broader task coverage, fine-tuning experiments, and mobile evaluation?

[r1] Jia, Hong, et al. "Beyond scale: Small language models are comparable to gpt-4 in mental health understanding." arXiv preprint arXiv:2507.08031 (2025).

5 comments

We thank the reviewer for the constructive feedback and for recognizing the value of our broad evaluation and on-device measurements. We hope the detailed responses below address your concerns and strengthen our submission.


W1/Q1 …connection between some benchmark tasks and mental-health signal detection…

Thank you for raising this. We agree that more explanation to our rationale can be provided in the paper, therefore have added the following paragraphs in Appendix A.3:

Mental wellbeing and psychiatric symptoms represent distinct but interrelated dimensions of mental health [1], with longitudinal evidence suggesting that changes in positive mental health are associated with subsequent risk of mental illness [2]. Previous studies also have demonstrated that naturally occurring language contains meaningful signals associated with clinically documented depression [3], population-level subjective wellbeing [4], and within-person changes during depressive episodes [5]. These findings highlight the importance of recognising changes in everyday psychological functioning, which could provide complementary information about emerging mental health difficulties before they are identified through conventional clinical assessments. Therefore, mental health monitoring should extend beyond the detection of diagnosed psychiatric conditions to encompass everyday wellbeing (T32-T38), emotional experiences (T14-T24,T26-T28, T39-T40), and social functioning (T30-T31, T35), particularly among individuals who have not received a clinical diagnosis.

Accordingly, MentaBench incorporates wellbeing-related tasks, including sentiment and emotion analysis (T14–T24, T26–T28, T39–T40), empathy detection (T25), wellness assessment (T32--T38), and mental health-related topic classification (T8), alongside clinical symptom and suicide-risk detection. These tasks capture complementary aspects of mental health-relevant language: sentiment and emotion analysis characterise expressed affect, wellness assessment examines dimensions of everyday functioning, topic classification identifies concerns and experiences discussed in online communities, and empathy detection assesses the interpersonal support conveyed in responses. Importantly, these capabilities are relevant not only to individuals with established mental health conditions but also to broader populations whose wellbeing may fluctuate without meeting diagnostic criteria. Evaluating these capabilities may support the early identification of emerging mental health difficulties, facilitate ongoing wellbeing monitoring, and contribute to a better understanding of individuals' mental health.


Reference:

  1. Mental Illness and/or Mental Health? Investigating Axioms of the Complete State Model of Health. 2005.
  2. Change in Level of Positive Mental Health as a Predictor of Future Risk of Mental Illness. 2010.
  3. Facebook Language Predicts Depression in Medical Records. 2018.
  4. Estimating Geographic Subjective Well-Being from Twitter: A Comparison of Dictionary and Data-Driven Language Methods. 2020.
  5. Using Language in Social Media Posts to Study the Network Dynamics of Depression Longitudinally. 2022.
  6. Just-in-Time Adaptive Interventions (JITAIs) in Mobile Health: Key Components and Design Principles for Ongoing Health Behavior Support. 2018.
  7. Journaling with Large Language Models: A Novel UX Paradigm for AI-Driven Personal Health Management. 2025.
  8. Detecting Receptivity for mHealth Interventions in the Natural Environment. 2021.
  9. Just-in-Time Adaptive Interventions for Suicide Prevention: Promise, Challenges, and Future Directions. 2022.
  10. Beyond Scale: Small Language Models Are Comparable to GPT-4 in Mental Health Understanding. 2025.
  11. Response Times: The 3 Important Limits. 1993.

Our response continues in the next comment (Part 2/3).

W2/Q2 …connection between mobile evaluation and the JITAI motivation…

Thank you for raising this. We agree that further strength between mobile evaluation and the JITAI motivation can further support our motivation, and have added the following paragraphs in Appendix C.1 (From MentaBench to JITAIs):

JITAI frameworks adapt support to an individual’s changing state, needs and receptivity at defined decision points [6, 8]. As an illustrative use case, we consider an opt-in mobile journaling application. With explicit consent, a user submits a short free-text entry, and an on-device SLM screens it for distress-related signals. The decision point is the submission of an entry. The tailoring variables can be five binary core-tier tasks: stress (T1), depression (T2), suicidal ideation (T4), thwarted belongingness (T30) and perceived burdensomeness (T31). The input can be an entry of 100–200 tokens, similar in length to the posts in these tasks (85–138 tokens on average, Table 1). With the task prompt, the input stays within 500 tokens. Predictions can inform an in-app supportive message, such as showing support resources, or a later check-in or human-reviewed support workflow. In either case, the entry is not transmitted to a remote inference server.

The required prediction time depends on which of these responses the application delivers. We distinguish two working cases. In-session feedback must appear before the user leaves the app. For this case, we adopt 10s, a widely used usability limit for holding a user’s attention during a wait [11]. Check-ins and human-reviewed workflows act minutes to hours later, so a difference of a few seconds in inference time does not affect them.

The following table provides end-to-end latency from our iPhone 15 Pro Max (Note that different device may have different latency) measurements at 500 input tokens (cf. Table 20).

ModelMean BACC across 5 TasksLatency per inference (s)
LLaMA3.2-1B0.785.06
TinyLLaMA-1.1B0.636.20
Falcon-1.3B0.626.61
Gemma4-E2B0.759.86
Qwen2.5-3B0.7311.31
LLaMA3.2-3B0.8011.55
StableLM-3B0.6711.64
Gemma4-E4B0.8012.10
Qwen3-4B0.7412.36
Qwen3.5-4B0.7513.16
Phi4Mini-3.8B0.7817.41

Within the 10 s limit for in-session feedback, LLaMA3.2-1B offers the best trade-off, with a mean BACC of 0.78 at 5.06 s per inference. For check-ins and human-reviewed workflows, where latency is not binding, LLaMA3.2-3B or Gemma4-E4B offer higher mean (BACC 0.80) and can be preferable.


Reference:

  1. Mental Illness and/or Mental Health? Investigating Axioms of the Complete State Model of Health. 2005.
  2. Change in Level of Positive Mental Health as a Predictor of Future Risk of Mental Illness. 2010.
  3. Facebook Language Predicts Depression in Medical Records. 2018.
  4. Estimating Geographic Subjective Well-Being from Twitter: A Comparison of Dictionary and Data-Driven Language Methods. 2020.
  5. Using Language in Social Media Posts to Study the Network Dynamics of Depression Longitudinally. 2022.
  6. Just-in-Time Adaptive Interventions (JITAIs) in Mobile Health: Key Components and Design Principles for Ongoing Health Behavior Support. 2018.
  7. Journaling with Large Language Models: A Novel UX Paradigm for AI-Driven Personal Health Management. 2025.
  8. Detecting Receptivity for mHealth Interventions in the Natural Environment. 2021.
  9. Just-in-Time Adaptive Interventions for Suicide Prevention: Promise, Challenges, and Future Directions. 2022.
  10. Beyond Scale: Small Language Models Are Comparable to GPT-4 in Mental Health Understanding. 2025.
  11. Response Times: The 3 Important Limits. 1993.

Our response continues in the next comment (Part 3/3)

Q3 …additional findings merge from MentaBench…

Thank you for highlighting Beyond Scale: Small Language Models are Comparable to GPT-4 in Mental Health Understanding [10]. We agree that this work provides important prior evidence of SLM capabilities in mental health classification. However, its evaluation covered a limited range of tasks and did not include fine-tuning or deployment evaluation. We have added this discussion to Appendix A.1.

MentaBench extends this line of research in three ways:

  • It consolidates 21 datasets into 40 classification tasks covering a broader range of psychological constructs and annotation paradigms.

  • It systematically compares zero-shot prompting, few-shot prompting, and task-specific LoRA fine-tuning (fine-tuning not evaluated in [10]) to assess the effect of supervised adaptation.

  • It integrates predictive evaluation with real-device latency, throughput, and memory measurements, enabling deployment-oriented model comparisons.

Our findings compared with the previous paper include:

  • Fine-tuned SLMs can be competitive with larger LLMs on mental health classification tasks.

  • Predictive performance and computational efficiency do not necessarily improve together, revealing important accuracy–efficiency trade-offs for practical mental health applications, with LLaMA3.2-1B or -3B offering the most balanced combinations of predictive accuracy and inference efficiency among the evaluated models.

  • SLM performance varies substantially across mental health constructs, with certain explicit suicidal-risk detection tasks achieving relatively strong classification performance after fine-tuning, while other tasks remain more challenging, highlighting the importance of evaluating model reliability across diverse mental health dimensions rather than relying on aggregate performance alone.

Our contribution is therefore a broader, reproducible evaluation framework rather than the first demonstration that SLMs can perform mental health classification.


Reference:

  1. Mental Illness and/or Mental Health? Investigating Axioms of the Complete State Model of Health. 2005.
  2. Change in Level of Positive Mental Health as a Predictor of Future Risk of Mental Illness. 2010.
  3. Facebook Language Predicts Depression in Medical Records. 2018.
  4. Estimating Geographic Subjective Well-Being from Twitter: A Comparison of Dictionary and Data-Driven Language Methods. 2020.
  5. Using Language in Social Media Posts to Study the Network Dynamics of Depression Longitudinally. 2022.
  6. Just-in-Time Adaptive Interventions (JITAIs) in Mobile Health: Key Components and Design Principles for Ongoing Health Behavior Support. 2018.
  7. Journaling with Large Language Models: A Novel UX Paradigm for AI-Driven Personal Health Management. 2025.
  8. Detecting Receptivity for mHealth Interventions in the Natural Environment. 2021.
  9. Just-in-Time Adaptive Interventions for Suicide Prevention: Promise, Challenges, and Future Directions. 2022.
  10. Beyond Scale: Small Language Models Are Comparable to GPT-4 in Mental Health Understanding. 2025.
  11. Response Times: The 3 Important Limits. 1993.

We appreciate the reviewer's suggestions. These clarifications will help us better distinguish what MentaBench establishes empirically from what remains to be validated for real-world mental health applications. We hope our detailed responses have adequately addressed your concerns, and we would be very grateful if you might consider raising your score in light of these updates.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank all reviewers for their thoughtful and constructive feedback, which has helped us further improve the paper. We have uploaded a revised manuscript, with all changes highlighted in blue.

The main revisions, their locations and what they show are summarised below:

  1. SLM–LLM comparison: Abstract; Section 1; Section 5.1; Appendix C.1, Table 6 (Reviewers AgcF, hcVP). We aligned the wording with our unchanged results: fine-tuned SLMs outperform traditional baselines and are competitive with LLMs, including the added stronger baseline Gemini 3.1 Pro.
  2. Deployment evaluation protocol: Section 5.2; Appendix D (Reviewer AgcF). We added a hardware-agnostic protocol with a budget-based selection rule, which supports our model recommendations and can be reused on future devices.
  3. Efficiency analysis and device generalisation: Appendix C.1; Table 25 (Reviewer AgcF). We show that accuracy does not scale with model size and that all evaluated SLMs fit on current smartphones.
  4. JITAI use case: Appendix C.1 (Reviewer h6n5). We show how our results guide model selection in a concrete JITAI scenario.
  5. Reproducibility: Appendix B; Prompts 1–3 (Reviewers AgcF, hcVP). We document the data splits, few-shot sampling and prompt templates. Our code is available via the anonymized link in the Abstract. We will publicly release our data annotations, data identifiers with retrieval scripts, and annotation protocols, and host a public leaderboard.
  6. Task scope and prior work: Appendix A.1; Appendix A.3 (Reviewer h6n5). We justify the inclusion of wellbeing and affective tasks and clarify how MentaBench extends Beyond Scale.
  7. Limitations, privacy and release: Appendix A.1; Appendix A.4; Appendix C.3 (Reviewer hcVP). We clarify the scope of our conclusions, the privacy limits of on-device inference, and what will be released.

Detailed responses to each reviewer are provided in the individual threads. We hope these revisions address your concerns, and we are happy to clarify any remaining questions during the discussion period.

Dear Area Chair,

We thank you for handling our submission and the reviewers for their constructive feedback, which has helped us improve the paper. Below we briefly summarise our contributions and how we addressed the reviewers' concerns during the rebuttal.


Contributions

  • Benchmark: MentaBench unifies 21 public datasets into 40 mental-health text classification tasks, covering stress, depression, suicidal risk, symptoms, affective states, interpersonal risk, wellbeing and empathy, with standardised label spaces and splits.
  • Systematic evaluation: 11 SLMs are evaluated under zero-shot, few-shot and fine-tuning settings against traditional baselines and LLMs. Fine-tuned SLMs outperform traditional baselines and are competitive with LLMs, and fine-tuning yields far larger gains than few-shot prompting.
  • Deployment evaluation: Latency, throughput and memory are measured on a real smartphone, together with a hardware-agnostic, budget-based protocol for selecting models under on-device constraints.
  • Key findings and open resource: Fine-tuned SLMs outperform traditional baselines and achieve accuracy competitive with LLMs, with fine-tuning yielding far larger gains than few-shot prompting. Gemma4-E4B achieves the highest accuracy, while LLaMA3.2-1B/3B offer the best accuracy–efficiency trade-off; accuracy does not scale with model size, and performance varies substantially across mental-health constructs. Code, task definitions, split identifiers, retrieval scripts and a public leaderboard support reproducible comparison.

Rebuttal

  • Claims and LLM baselines (AgcF, hcVP): We aligned the SLM–LLM claim with our results ("competitive"), added Gemini 3.1 Pro, and provided per-task BACC for both Gemini models.
  • Deployment and model selection (AgcF, h6n5): We added a reusable deployment protocol, an efficiency analysis and a JITAI use case showing how the results guide model selection.
  • Reproducibility and scope (AgcF, h6n5): We documented data splits, few-shot sampling and prompt templates, justified the inclusion of wellbeing and affective tasks, and clarified the relation to prior work.
  • Validity, privacy and release (hcVP): We clarified leakage and contamination limitations, the privacy limits of on-device inference and the release policy, and are replacing the majority-class fallback with a rule that counts invalid outputs as errors.

We thank you again for your time and consideration, and we hope this summary helps in assessing our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.