SSLLM: Auditable Multimodal Reasoning for Stress Evaluation from Social and Wearable Signals

Depression-related deterioration unfolds continuously, but assessment remains episodic and largely self-reported. Existing wearable models capture physiological dynamics yet often provide limited interpretability, while language models produce fluent explanations that may be weakly grounded. We present SSLLM, a novel social-sensing framework for depression-oriented stress evaluation that turns wearable signals, behavioral summaries, and social text into an auditable reasoning pipeline. SSLLM combines time-series-to-language reprogramming, multimodal fusion, Tab-CoT reasoning as structured Observation–Reasoning–Evaluation evidence atoms, constrained by a diagnostic knowledge graph, and a drafter–auditor–refiner loop that retains only evidence-supported conclusions. Across five benchmark slices, SSLLM achieves an average macro-F1 of 0.92, compared with 0.59 for Gemini 3.0 Pro and 0.29 for base Qwen-3. Component ablations further suggest complementary contributions from physiological alignment, social context, structured reasoning, and auditing. Qualitative review of example outputs suggests that the resulting rationales can be traced to depression-relevant cues including sleep disruption, reduced activity, sentiment shifts, and social withdrawal. These results provide early evidence that multimodal health signals can be transformed into an inspectable reasoning artifact rather than only a scalar prediction. SSLLM evaluates stress levels, not major depressive disorder, and is intended as a research framework for evidence-grounded, depression-aware review rather than a clinically validated diagnostic system. As a next step toward real-world deployment, we develop a privacy-preserving mobile app that combines on-device multimodal sensing with federated learning, enabling richer longitudinal data collection while sharing only model updates and privacy-minimized evidence with the cloud-based SSLLM.