acceptodds
Withdrawn from ICLR 2027

SSLLM: Auditable Multimodal Reasoning for Stress Evaluation from Social and Wearable Signals

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Depression-related deterioration unfolds continuously, but assessment remains episodic and largely self-reported. Existing wearable models capture physiological dynamics yet often provide limited interpretability, while language models produce fluent explanations that may be weakly grounded. We present SSLLM, a novel social-sensing framework for depression-oriented stress evaluation that turns wearable signals, behavioral summaries, and social text into an auditable reasoning pipeline. SSLLM combines time-series-to-language reprogramming, multimodal fusion, Tab-CoT reasoning as structured Observation–Reasoning–Evaluation evidence atoms, constrained by a diagnostic knowledge graph, and a drafter–auditor–refiner loop that retains only evidence-supported conclusions. Across five benchmark slices, SSLLM achieves an average macro-F1 of 0.92, compared with 0.59 for Gemini 3.0 Pro and 0.29 for base Qwen-3. Component ablations further suggest complementary contributions from physiological alignment, social context, structured reasoning, and auditing. Qualitative review of example outputs suggests that the resulting rationales can be traced to depression-relevant cues including sleep disruption, reduced activity, sentiment shifts, and social withdrawal. These results provide early evidence that multimodal health signals can be transformed into an inspectable reasoning artifact rather than only a scalar prediction. SSLLM evaluates stress levels, not major depressive disorder, and is intended as a research framework for evidence-grounded, depression-aware review rather than a clinically validated diagnostic system. As a next step toward real-world deployment, we develop a privacy-preserving mobile app that combines on-device multimodal sensing with federated learning, enabling richer longitudinal data collection while sharing only model updates and privacy-minimized evidence with the cloud-based SSLLM.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Reasoning, LLM, Multi-modal Learning, Fine-Tuning, Representation Learning, Neuro-Symbolic

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bg4bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a framework that combines reasoning over signals from wearables and social texts [sic!] in the context of mental wellbeing assessments, specifically for stress evaluations. The authors combine modality specific representations (extracted using state of the art foundation models) and use these to construct a structured reasoning apprach that follows an observation-reasoning-evaluation direction. Essentially, this reasoning backend is a graph-based method that not only provides numerical assessments (of mental stress in this case) but also, as claimed by the authors, provides an “evidence-based” auditing trail. The authors claim that such an interpretable approach is more valuable for practical applications of computational psychiatry, which is the field this paper targets. An experimental evaluation on a range of five existing datasets is used to demonstrate the effectiveness of the presented approach, albeit seemingly limited to quantitative results mainly. In their discussion the authors outline next steps in terms of plans to design a mobile app that would be used for actual deployments rather than evaluations on existing datasets.

Critical strengths

This paper operates in an interesting and relevant area. Mental stress is a growing concern worldwide and the analysis of a combination of both signals from wearables, which would be used to capture measurable physiological indicators that might be correlated with stress, and social signals that directly link to mental wellbeing and stress is an interesting and promising idea. I applaud the authors for their system-level approach that combines established methods for uni-modal representation extraction and multi-modal integration into a reasoning framework thatm, as per the description provided by the authors, not only outputs individual stress scores but also aims for their explanations through auditable traces, which would support practitioners and patients alike.

Critical weaknesses

While I appreciate the system focus of this paper (the individual components for both processing the signals of various modalities as well as their integrations are fairly standard) because it has great potential for substantial impact, the paper does not fully deliver what it promises. I fully agree with the authors that automated stress assessments should not be reduced to some seemingly arbitrary numbers but rather should provide explanations for these — given the significance of such assessments and their outcomes. Yet, the paper does not actually dive into these critical insights. Table 3 seems to be the only indication for results beyond quantitative scores, yet no actual details are given.

I am also concerned about the impact this paper could have in its current form. Again, the work targets system level, which is great. Yet, the authors only evaluate it (again, mainly at the quantitative level that they wanted to go beyond) on existing datasets. The authors hint at an app and an actual deployment but this is left for future work. I would argue that such an app and an actual deployment is mandatory for this paper such that a reader can learn about the actual practical impact this work could have.

The presentation of the quantitative results has some issues: Why are the data points in figure 4 connected? This is just plain wrong because it suggests some sort of connection that simply is not there. Also: The baselines seem to weak and thus result in an unfair comparison. Why would one expect those models out of the box function well? I would argue that they would need to be finetuned etc. to make this a meaningful comparison. Without it, the gap between SOTA and the proposed framework seems inflated.

Questions for authors
  1. Why are there no results beyond mere quantitative assessments?
  2. Why are the data points in figure 4 connected?
  3. Why did you use SOTA baselines as is, rather than adapt them to the domain?
Reviewer_EeGfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a social sensing framework for depression-oriented stress evaluation that turns wearable signals, behavior summaries, and social text into an auditable reasoning pipeline. The authors motivate it by arguing that stress is not just a change in heart rate or sleep, and that when it is sustained and paired with withdrawal, circadian disruption, and negative self-expression, it becomes part of a depression-relevant trajectory. The system encodes physiological time series into compact summaries, projects them into prompt tokens for a fine-tuned Qwen3 model, and has the model produce a table of observations, reasoning, and evaluations. They curated a diagnostic knowledge graph to constrain reasoning, and included a drafter, an auditor, and a refiner to check and finalize the output. The authors evaluate on five benchmark slices from three public datasets and compare against a base Qwen3 and Gemini 3 Pro. The task is stress level prediction, not a depression diagnosis, and the authors present the system as a research framework rather than a clinically validated diagnostic tool.

Critical strengths
  1. The idea of turning multimodal health signals into an inspectable reasoning artifact, instead of only a single prediction score, is interesting.
  2. The authors constrain the LLM’s reasoning with a knowledge graph built from established diagnostic frameworks and the stress literature, instead of letting it reason freely.
  3. The authors ran ablations that add each component one at a time.
Critical weaknesses
  1. The jump from the baselines to the proposed system feels suspicious. I feel like there might be some sort of leakage. In the ablation, performance jumps when the authors add social media input, and this is where leakage could happen, for example if people write that they are stressed. Adding chain of thought makes it a little worse, and then the auditor loop pushes accuracy to almost perfect, so the auditor loop might also be introducing leakage. The authors also use public datasets, so the LLM may have already seen them during training.
  2. It’s not clear what the authors give as input to the baselines, what prompt they use, or how their system differs from those baselines, since their model is also a fine-tuned Qwen3. It may as well be that it performs better just because of the fine-tuning. The authors also don’t say what data they use to fine-tune the model or what the fine-tuning task is.
  3. The benchmark and the task are stress level evaluation, while the reasoning layer is oriented toward depression. The authors never actually measure depression, because the label is stress.
  4. The method is hard to follow. It’s not clear whether the latent vector that goes into the language model is text or not, and the paper doesn’t define several items, like the number of streams, devices, or the time dimension. It’s also not clear what the mental-focus queries are, where the social media activity comes from, whether any text comes from interviews or patient self-reports, and whether the drafter, the auditor, and the refiner are humans or models. Overall, I don’t think there is enough detail in this paper to evaluate it properly.
  5. The authors evaluate the quality of the reasoning with a separate automated judge model. If this judge has access to the same data, its scores would naturally align with what the system produced. This is not really a quality assessment by a human.
Questions for authors
  1. What is the verifier that maps entities and relations to the knowledge graph and blocks or downweights unsupported links? Is it an LLM or a human?
  2. If it's an LLM, how accurate is it, how often does it block or downweight links, and do you have any assessment of whether this mechanism works properly?
  3. Do you also pass the social media queries to the baselines? Could they contain information that reveals the stress level, which would be a form of leakage?
  4. What data do you use to fine-tune the Qwen3 model, and what is the fine-tuning task?
  5. What input and prompt do you give the baselines?
  6. Are the drafter, the auditor, and the refiner humans or LLMs?
  7. What do the streams, devices, and time dimension correspond to in your formulation?
Reviewer_WQb6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses the opacity of predictions from wearable-signal models and the lack of evidence constraints in LLM-generated mental health explanations. It reframes stress assessment from directly predicting labels to first constructing a multimodal evidence chain, then auditing it and producing a conclusion. The proposed method first uses Prompt-as-Prefix to map physiological time series into the language model's representation space, integrates contextual information, and organizes evidence in an Observation–Reasoning–Evaluation format. It then uses a diagnostic knowledge graph and a drafter–auditor–refiner loop to filter out unsupported or temporally inconsistent inferences, improving alignment between physiological signals and linguistic context while making the basis for stress judgments traceable and inspectable. Across five benchmark datasets, the average Macro-F1 is 0.92, compared with 0.29 for Base Qwen-3 and 0.59 for Gemini 3.0 Pro.

Critical strengths

The research direction is well motivated. Combining wearable-device data from continuous monitoring with social context and providing traceable, verifiable grounds for judgments has potential clinical and public health value.

Critical weaknesses
  1. The organization is overly fragmented: the paper is divided into many subsections, but each contains limited substantive content.
  2. Reliability and completeness are insufficiently established. Converting continuous monitoring data and social context into discrete evidence atoms may lose temporal dependencies, cross-modal relationships, and stochasticity. The paper needs to clearly demonstrate how this conversion maps back to the original records, how observations are distinguished from inferences, and how omissions and extraction errors are detected.
Questions for authors

How do you validate the fidelity and completeness of evidence atoms with respect to the original data, particularly the preservation of temporal dependencies, cross-modal relationships, and uncertainty? Could you provide a manual verification or annotation-based evaluation against the original records, reporting omissions of key evidence, extraction errors, and cases where inferences are presented as observations, and analyze how these errors affect downstream judgments and results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.