- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The useful part is that they do not stop at “the representations separate.” They intervene and look at behavior, and they do it on more than one model family. That is a better starting point than another output-only comparison.
The pairs come from actual training books, and they do try to kill the obvious alternative explanations. Style is matched on the surface, and they also run a version where both sides are LLM-written. AUC stays very high after that, so I do not think this is just human textbook prose versus model prose.
Critical strengths
The useful part is that they do not stop at “the representations separate.” They intervene and look at behavior, and they do it on more than one model family. That is a better starting point than another output-only comparison.
The pairs come from actual training books, and they do try to kill the obvious alternative explanations. Style is matched on the surface, and they also run a version where both sides are LLM-written. AUC stays very high after that, so I do not think this is just human textbook prose versus model prose.
Critical weaknesses
I am not convinced the judge is counting skills. Telling a textbook answer from a synthetic misstep is a different task from labeling individual skills in a free generation. Figure 1 is the example that stuck with me: a short acknowledgment and a clarification question are given ten skills. Several of those look like the same move under different names. Unless someone who trains or practices this work has checked the generations, a higher count does not tell me the response is a better intervention.
The steering result is also thinner than the writeup suggests. The gains come from searching layers and strengths on twelve inputs per setting, and it is not clear that this search was frozen before the numbers being reported. There is no therapy-prompt baseline, no generic empathy or helpfulness direction, and no random direction matched on norm. Right now I cannot tell whether the axis is about psychotherapy skill or about sounding warmer and more supportive.
Questions for authors
Have clinicians or trainers scored the generated responses, both for whether each skill is really there and for whether it fits the moment? For the response in Figure 1, were the ten labels checked one by one, and how did you treat skills that describe the same utterance?
Do you have a comparison to a therapy prompt, to an empathy or helpfulness direction, or to random directions with the same norm? I would want that before reading the gains as specific to the Healing Axis.