- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors proposed a Reason-then-Assess framework for depression risk detection, in which the model first generates a structured multi-dimensional analysis and a downstream classifier performs the final risk detection. For the first step, the authors proposed a token-wise reward modeling approach and used it at inference time to guide and improve the model reasoning.
Critical strengths
The approach successfully improves inference-time alignment across datasets and backbone model families, compared to existing approaches including GenARM, SFT, and DPO. The ablations further provide evidence supporting the token-wise reward modeling. The Reason-then-Assess framework also provides valuable methodological insights for LLM-based mental health analysis.
Critical weaknesses
The token importance is estimated based on the final-position max logit, which measures tokens' influence on the model's prediction, but not their clinical relevance. It remains unclear whether important tokens indicate clinically meaningful evidence, and whether they reflect real risk signals of depression. Moreover, the clinical validity of both the generated reports and the final risk labels is unclear, since the labels in the datasets were not clinical diagnoses.
Questions for authors
Can the authors provide analysis or evidence supporting the clinical meaning of the token importance? Can the authors justify the clinical validity of the LLM-generated risk prediction and reports?
2 comments
Thanks for your reply. Your clarifications do address some of my concerns.
Thanks for the positive assessment and for raising the important question of clinical validity. We want to be precise about what our current evidence supports. Below we clarify what the token weights measure, explain why decision-relevant tokens can still carry clinical meaning, discuss the validity of the labels and generated reports, and list the revisions we will make.
1. What ωt measures and what it does not.
We agree that ωt measures a token's influence on the prediction, not its clinical relevance. This is by design: the preference direction comes entirely from criteria-based preference pairs, and ωt only acts as a positive scaling factor on each gradient term (App. B, Eq. 21–25). It can change how strongly a position is updated, but it cannot reverse the preference direction (see also Eq. 33). We therefore do not claim that gradient saliency equals clinical evidence. We claim that focusing a clinically defined preference signal on the most decision-relevant positions makes supervision more efficient. The w/o Weight ablation supports this: performance drops consistently on all three datasets, and the drops are larger on F1/MCC, which reflect high-risk detection (Table 2).
2. Why decision-relevant tokens can carry clinical meaning
Explanations from models grounded in clinical symptom frameworks have been found to match expert-annotated evidence more closely than those of models trained on diagnostic labels alone [1]. In the suicide-risk setting, attribution-derived evidence has likewise been shown to map onto clinically defined risk-factor categories, with these mappings validated against expert annotations and the resulting explanations rated helpful by practitioners [2, 3]. Across tasks and model architectures, gradient-based attribution has further emerged as the most reliable family of explanation methods in systematic comparisons [4]. AlignDx follows the same principle: weights are computed over reports organized by DSM-5/PHQ-9-grounded dimensions, so decision-relevant tokens sit inside clinically defined slots.
3. Validity of the labels and the generated reports
The dataset labels are not clinical diagnoses, and we do not claim that they are. Like most social-media mental-health benchmarks, our datasets rely on proxy labels. In eRisk, positive users are identified through self-reported diagnoses that were manually screened under strict rules [5]. WU3D and SWDD are built from user-level annotation protocols. Prior work notes that all such proxy signals have limited external validity [6]. This is why we use the term "risk detection," why the generation prompt states that "the label indicates depression risk rather than a clinical diagnosis" (App. D.1), and why the Ethics Statement excludes automated individual decision-making. Accordingly, we position AlignDx as a risk-screening framework, not a diagnostic tool.
The reports themselves, however, are anchored in clinical criteria: each dimension maps to DSM-5 criteria and validated instruments (PHQ-9, HAM-D), must cite verbatim quotes from the posts with fabrication explicitly prohibited, and is scored by the preference judge on grounding and accuracy (App. D.1–D.2). This design follows prior work showing that grounding model outputs in clinical symptom frameworks yields more meaningful explanations [1], and that social-media language contains clinically valid signal [7]. Supporting evidence appears in the case studies, where the aligned model correctly attributes speakers and context (Fig. 4, App. E), and the gains replicate across two downstream classifiers (App. H), indicating they stem from report content rather than any particular model.
4. Revisions.
We will make three revisions:
1.Change the wording in Appendix F.2 to "decision-relevant tokens, which in our examples coincide with symptom-bearing evidence."
2.Add the literature above to the Related Work section.
3.List expert validation of token weights as future work in the Limitations section.
We thank the reviewer again. These comments have helped us make our claims more precise, and we are happy to provide further clarification.
[1] Zirikly & Dredze, Explaining Models of Mental Health via Clinically Grounded Auxiliary Tasks, CLPsych 2022.
[2] Munz et al., Dual-Mode Explanations for Suicide Risk Detection in Online Mental Health Support, Information Systems Frontiers, 2026.
[3] Nguyen et al., Conceptualizing Suicidal Behavior: Utilizing Explanations of Predicted Outcomes to Analyze Longitudinal Social Media Data, 2023.
[4] Atanasova et al., A Diagnostic Study of Explainability Techniques for Text Classification, EMNLP 2020.
[5] Losada & Crestani, A Test Collection for Research on Depression and Language Use, CLEF 2016.
[6] Ernala et al., Methodological Gaps in Predicting Mental Health States from Social Media, CHI 2019.
[7] Eichstaedt et al., Facebook language predicts depression in medical records, PNAS 2018.