- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces RELATE, a framework for measuring inward-facing (IF) and outward-scaffolding (OS) language in multi-turn emotional-support conversations. It evaluates seven LLMs across 76 situations and three simulated user styles, using sentence-level automated judgments and a smaller human-annotated subset.
Critical strengths
- The distinction between support quality and relational orientation is potentially useful, and allowing IF and OS to co-occur also captures a potentially important aspect of supportive dialogue.
- The experimental protocol is documented in detail, and the authors transparently report discrepancies between human and automated evaluations.
Critical weaknesses
- Concerns about the paper's motivation. In the abstract, the authors motivate their work by concerns about the displacement of real-world relationships and the risk of dependence on AI, and propose using IF and OS to audit and steer model behavior. This framing may lead readers to expect that the two metrics clearly indicate desirable or undesirable behavior. However, the results analysis fail to clarify. This ambiguity raises questions about the paper’s motivation and limits the practical guidance it offers as an evaluation framework. In my view, IF combined with OS over the short term may not pose a risk of relationship substitution, whereas including OS in every turn may instead resemble a formulaic response pattern.
- The steering claim is not empirically demonstrated. The abstract presents RELATE as providing a signal for both auditing and steering supportive LLMs. Although the paper performs descriptive auditing, it reports no intervention that uses IF/OS feedback to modify model behavior. A limited demonstration, including whether support quality is preserved, would strengthen this claim. Otherwise, the abstract should identify steering as a potential application rather than an empirically varified capability.
- Human–judge disagreement undermines the validity of the automated IF measurements. Table 17 on page 28 reports human–judge Cohen’s κ values of only 0.01–0.17 for IF, with Gwet’s AC1 also near zero. Human–human agreement is substantially higher (κ = 0.88), and human and automated annotations yield opposite turn-level trends on the shared validation subset (Table 18). The authors acknowledge these discrepancies, but they still directly weaken the interpretation of the main empirical findings.
- The relationship between IF/OS, response quality, and real human user reliance is not evaluated. Without such evidence, it remains unclear whether IF and OS conflict with response quality, and whether IF captures the risk of AI dependence.
Questions for authors
- Table7 reports sentences with IF=0 and OS=0 as "Neither". Could you provide representative, human-annotated examples and explain their communicative functions? Do these sentences include useful support that does not direct the user toward either the assistant or other people?
- Potential typo on page3, line135: "The second is where it directs..." should be "The second is whether it directs...".