- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces SleekAware, a benchmark for evaluating LLMs’ ability to assess compliance with social, legal, ethical, empathetic, and cultural norms in conversations. The benchmark consists of 700 Reddit conversations across therapy, PhD supervision, and elderly care, along with 24 expert-defined rules and over 6,800 human annotations. It evaluates rule applicability, compliance judgments, evidence extraction, and omission-based violation detection. Experiments with five LLMs and four prompting strategies reveal gaps in prediction accuracy and coverage, particularly in identifying implicit norms and grounding judgments in conversational evidence.
Critical strengths
The paper addresses a relevant problem and introduces a benchmark with human-annotated conversational evidence. However, I do not find these contributions sufficient to support acceptance given the significant methodological concerns. Please see the Critical Weaknesses and Questions section for details.
Critical weaknesses
-
The benchmark's definition of SLEEC-awareness is not well justified. Many rules treat the absence of an explicit mention of a norm as a violation, even when the response itself may be perfectly appropriate. The benchmark also covers only 24 rules across three domains, which seems rather limited given the broad claims made in the paper.
-
The way ground truth is constructed raises concerns. The authors keep conflicting human judgments as equally valid references and use best-match scoring (Section 3.6). This makes it hard to tell whether the reported scores reflect actual model performance or disagreement among annotators.
-
The evidence evaluation is not entirely convincing. Token-level IoU can penalize valid evidence simply because it differs from the human-selected span. The authors even acknowledge this issue in Appendix A.3, yet still conclude that models struggle to ground their judgments in conversational evidence.
-
The experiments lack some important validation. There is no systematic analysis of inter-annotator agreement, sensitivity to evaluation choices, or statistical uncertainty. Without these checks, it is difficult to know how reliable the reported findings are.
-
The paper needs considerable improvement in writing and presentation. Citations are inconsistently formatted, some tables extend beyond the page margins, and parts of the methodology are not clearly explained. Section and appendix references also lack working hyperlinks. These issues make the paper feel insufficiently polished for a conference submission.
Questions for authors
- How do the authors justify treating the absence of an explicitly stated norm as a violation? A response may be appropriate without mentioning a specific expectation. Was this assumption validated by independent experts, particularly given that Reddit respondents are not necessarily professionals?
- How much disagreement was there among human annotators? Since conflicting judgments are retained as ground truth, would the results change if these annotations were aggregated or adjudicated instead of using best-match scoring?
- How sensitive are the evidence-grounding results to the choice of evaluation metric? Tables 11 and 12 suggest that valid evidence can be penalized by token-level IoU. Do the main conclusions still hold with different thresholds or semantic evidence matching?
- What evidence supports the generalizability of the benchmark? With only 24 rules across three domains, how well does it capture the five dimensions of SLEEC-awareness? Can the authors provide a breakdown of performance across these dimensions?