- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper observes that generic post-training (SFT or standard RL) does not reflect how mental-health professionals actually assess — which has two salient properties the authors want to build into LLM reasoning. Experiments use Qwen3-8B as the base for all methods, jointly trained on the combined train+validation splits of 8 public datasets. This is competent, well-organised incremental work: GRPO plus a stage-dependent entropy schedule plus a staged output format, applied carefully to eight public mental-health datasets with consistently positive results. The breadth of the comparison set is a genuine strength, and the appendix-level documentation of the eight datasets and their splits is useful.
Critical strengths
- This is an interesting idea. Mental-health assessment is a plausible setting in which how a model reasons determines whether subtle internal cues survive against salient external functioning. 2.More analysis than is typical for this kind of submission. The schedule study is a well-built control. Combined with the M × τ grid, this is a real attempt to explain the mechanism rather than just report scores. 3.All datasets are public; split protocols are documented including the sensible decision to align DepSeverity partitions with Dreaddit because they share a corpus, and the de-duplication of RSD to one post per individual. This addresses a real problem in this subfield.
Critical weaknesses
1.The paper's novelty is that its staged structure is justified by cognitive appraisal theory ("theory-grounded", "cognition-aligned", "inspired by and formalised from appraisal theory"). But nothing distinguishes "appraisal-theory stages" from any five-stage reasoning scaffold. 2.The reasoning-demand subset is selected by GPT-5, and then Mental-R1 is compared against GPT-5 on exactly those samples, yielding the paper's second headline number. 3."Reasoning demand" has no human or clinical validation — it is an unvalidated LLM judgement used to construct an evaluation set that favours reporting large gains. 4.The abstract promises "theory-grounded interpretable inference," and Section 1 lists interpretable inference as an outcome. There is no evaluation of whether the generated appraisals are clinically coherent, whether they are faithful, or whether a practitioner would find them useful. 5.Only weighted F1 is reported. Several datasets are imbalanced and several are ordinal; since the paper follows prior work in using weighted F1 it is defensible, but weighted F1 can mask minority-class collapse. 6.Maximum completion length 512 tokens with ≤4096 thinking budget elsewhere; at 5 stages plus the answer this may constrain some datasets. Worth reporting truncation rates.
Questions for authors
1.Please add a structured-CoT prompting-only baseline and an SFT-on-structured-traces baseline, to separate the staged output scaffold from the RL objective. 2.Was the M/τ grid evaluated on the same test splits used for Table 2 and Table 3? If so, please re-select on validation and report both. 3.Even a small annotation study comparing CRPO traces with GRPO traces for coherence and usefulness would substantiate the interpretability claim. If none is possible, please drop the claim.
3 comments
Response to Reviewer smnb
Q1. Structured-CoT prompting and structured-trace SFT baselines
We added both requested baselines. Since the original training datasets do not provide reasoning traces, we constructed synthetic structured traces using the original, frozen Qwen3-8B model. For each training prompt, we sampled four candidate completions with the five-stage prompt and retained at most one trace, randomly selected from candidates that satisfied the format requirements and whose final answer matched the training label. The selected traces were then used for supervised fine-tuning.
| Method | DATD | RSD | DepSeverity | LT-EDI | SDCNL | Dreaddit | FIG | LID | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Frozen Qwen3-8B, five-stage prompt | 53.62±2.45 | 30.75±1.91 | 59.66±1.45 | 35.77±1.83 | 49.42±2.29 | 70.74±0.72 | 78.62±0.42 | 21.08±1.57 | 49.96 |
| SFT on structured traces | 54.93±2.39 | 25.39±1.78 | 66.50±1.00 | 31.45±1.71 | 53.15±1.46 | 73.22±1.86 | 81.27±1.94 | 17.62±3.36 | 50.44 |
| CRPO | 63.62±1.03 | 50.14±4.72 | 78.86±1.16 | 57.14±0.97 | 61.75±1.40 | 80.56±0.57 | 91.57±0.61 | 38.39±1.66 | 65.25 |
CRPO achieves 65.25 average weighted F1, compared with 49.96 for five-stage prompting and 50.44 for structured-trace SFT. It outperforms both baselines on all eight datasets, with a 14.81-point average gain over structured-trace SFT. These results support the contribution of the CRPO training procedure beyond the tested prompting-only and supervised-learning alternatives. The baseline construction and results are provided in Appendix D and Table 5.
Q2. Was the M/τ grid evaluated on the test splits?
The original Figure 5 presented the parameter-sensitivity results on the test sets. We used the separate training, validation, and test partitions described in Appendix C, and selected parameters using the validation set. We have added the corresponding validation-set plot in Appendix E (Figure 6) and report the validation results below.
| M | τ = 2.5 | τ = 3.5 | τ = 4.5 |
|---|---|---|---|
| 0.04 | 61.42 | 64.78 | 62.61 |
| 0.06 | 62.96 | 65.84 | 62.19 |
| 0.08 | 61.73 | 63.52 | 62.25 |
The validation results show consistent overall trends with the test-set results: τ = 3.5 performs best for each tested value of M, and M = 0.06, τ = 3.5 is the best configuration on both splits. This validation-selected configuration achieves 65.84 on validation and 65.25 on test.
Dear Reviewer,
We appreciate the time and effort you have taken to review our submission and provide valuable feedback. We have carefully addressed the comments raised during the first round and submitted our responses and revision accordingly. As the discussion period is ending soon, we would greatly appreciate any further feedback you might have. Please let us know if there are any remaining concerns or questions, so we can address them in time.
Thank you again for reviewing our paper.
Best regards,
Authors
Response to Reviewer smnb
Thank you for the detailed assessment and constructive suggestions on the experimental controls, evaluation protocol, and analysis. We have uploaded a revised manuscript incorporating the additional experiments and clarifications described below.
To examine the contribution of the appraisal-inspired stage definitions, we trained a CRPO variant using five generic reasoning stages: observation, analysis, interpretation, evaluation, and conclusion. This variant used the same SER schedule and parameters (M = 0.06 and τ = 3.5) as CRPO.
CRPO achieves 65.25 average weighted F1, compared with 62.18 for the generic-stage variant, a gain of 3.08 points, with higher scores on all eight datasets. These results support the usefulness of the appraisal-inspired stage definitions under the same entropy schedule. This comparison is included in Table 2 as the w/ GS ablation and discussed in Section 4.2; the generic-stage definitions and configuration are described in Appendix C.1.
We repeated the subset selection using Gemini 4, which is outside our baseline set. The top-20% subsets selected by GPT-5 and Gemini 4 have an overlap of 78.4%. Mental-R1’s gain in average weighted F1 over the strongest baseline is 15.3 points on the Gemini-selected subsets, compared with 15.6 points on the GPT-5-selected subsets. These results show that the reported advantage persists across the two selectors. The second-selector analysis has been added to Section 4.4, and the scoring prompt is provided in Appendix H.
We conducted a human evaluation comparing CRPO and GRPO on 80 test inputs, with 10 randomly sampled from each dataset's test split. Two mental health professionals independently selected the better output or a tie on each criterion, yielding 160 judgments per criterion. The outputs were presented without model or method identifiers.
Raw inter-annotator agreement ranged from 78.8% to 81.3%, with Cohen’s κ = 0.63–0.69. CRPO received more preference votes than GRPO on all three criteria: 62 versus 21 for coherence, 76 versus 23 for usefulness, and 83 versus 14 for closeness to human professionals’ reasoning. These results are reported in Section 4.6 and Figure 5, with the evaluation criteria and agreement statistics in Appendix I.1.
We added macro-F1, which assigns equal weight to each class. CRPO achieves an average macro-F1 of 56.99, compared with 45.37 for GRPO, a gain of 11.62 points, with higher scores on all eight datasets. The gains on RSD, DepSeverity, and LT-EDI are 15.07, 13.11, and 14.22 points, respectively. The per-dataset macro-F1 results are included in Table 6 of the revised manuscript.
The maximum completion length was set to 512 tokens as a practical trade-off given the model size and available GPU memory. Across the CRPO test outputs, 2.7% of completions reached this limit. These details are reported in Appendix C.