acceptodds
Under review as a conference paper at ICLR 2027

SleekAware: Benchmarking Social, Ethical, Empathetic, and Cultural Awareness of LLMs

Abstract

Large Language Models (LLMs) are increasingly used in education and healthcare, where they influence human experiences and decisions, and as judges of other models' responses. These roles require adherence to Social, Legal, Ethical, Empathetic, and Cultural (SLEEC) norms. Yet it remains unclear whether LLMs can reliably recognize and evaluate compliance with these norms. We introduce SleekAware, a benchmark for evaluating LLMs' SLEEC reasoning in natural multi-turn conversations. It comprises three parts: (i) real-world Reddit conversations across therapy, PhD supervision, and elderly care, (ii) expert-defined SLEEC rules for each domain, and (iii) over 6,800 crowdsourced human annotations assessing conversation compliance with these rules. SleekAware evaluates LLMs along two dimensions: (i) how closely their final judgments match human judgments, and (ii) how closely the conversation excerpts they select to support their decisions match those selected by humans. Together, these dimensions go beyond agreement with human judgments to probe the evidence underlying LLM decisions, providing a deeper assessment of SLEEC-awareness. We evaluate five state-of-the-art LLMs using strategies ranging from zero-shot to structured reasoning prompts. Models show only partial SLEEC-awareness, struggling with implicit normative reasoning and emotionally nuanced interactions. While some identify rule compliance well, none reliably ground their judgments in conversational evidence. To advance LLMs' SLEEC capabilities, we release our benchmark and annotation framework at https://sleech.net/benchmark.

Primary area:
datasets and benchmarks
Keywords:
Ethics, Bias, and Fairness, Human-Centered NLP and Human-AI Interaction, Interpretability and Analysis of Models for NLP, Dialogue and Interactive Systems, Language Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_HGDUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SleekAware, a benchmark for evaluating LLMs’ ability to assess compliance with social, legal, ethical, empathetic, and cultural norms in conversations. The benchmark consists of 700 Reddit conversations across therapy, PhD supervision, and elderly care, along with 24 expert-defined rules and over 6,800 human annotations. It evaluates rule applicability, compliance judgments, evidence extraction, and omission-based violation detection. Experiments with five LLMs and four prompting strategies reveal gaps in prediction accuracy and coverage, particularly in identifying implicit norms and grounding judgments in conversational evidence.

Critical strengths

The paper addresses a relevant problem and introduces a benchmark with human-annotated conversational evidence. However, I do not find these contributions sufficient to support acceptance given the significant methodological concerns. Please see the Critical Weaknesses and Questions section for details.

Critical weaknesses
  1. The benchmark's definition of SLEEC-awareness is not well justified. Many rules treat the absence of an explicit mention of a norm as a violation, even when the response itself may be perfectly appropriate. The benchmark also covers only 24 rules across three domains, which seems rather limited given the broad claims made in the paper.

  2. The way ground truth is constructed raises concerns. The authors keep conflicting human judgments as equally valid references and use best-match scoring (Section 3.6). This makes it hard to tell whether the reported scores reflect actual model performance or disagreement among annotators.

  3. The evidence evaluation is not entirely convincing. Token-level IoU can penalize valid evidence simply because it differs from the human-selected span. The authors even acknowledge this issue in Appendix A.3, yet still conclude that models struggle to ground their judgments in conversational evidence.

  4. The experiments lack some important validation. There is no systematic analysis of inter-annotator agreement, sensitivity to evaluation choices, or statistical uncertainty. Without these checks, it is difficult to know how reliable the reported findings are.

  5. The paper needs considerable improvement in writing and presentation. Citations are inconsistently formatted, some tables extend beyond the page margins, and parts of the methodology are not clearly explained. Section and appendix references also lack working hyperlinks. These issues make the paper feel insufficiently polished for a conference submission.

Questions for authors
  1. How do the authors justify treating the absence of an explicitly stated norm as a violation? A response may be appropriate without mentioning a specific expectation. Was this assumption validated by independent experts, particularly given that Reddit respondents are not necessarily professionals?
  2. How much disagreement was there among human annotators? Since conflicting judgments are retained as ground truth, would the results change if these annotations were aggregated or adjudicated instead of using best-match scoring?
  3. How sensitive are the evidence-grounding results to the choice of evaluation metric? Tables 11 and 12 suggest that valid evidence can be penalized by token-level IoU. Do the main conclusions still hold with different thresholds or semantic evidence matching?
  4. What evidence supports the generalizability of the benchmark? With only 24 rules across three domains, how well does it capture the five dimensions of SLEEC-awareness? Can the authors provide a breakdown of performance across these dimensions?
Reviewer_MBBXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for LLM evaluation on recognizing social, legal, ethical, empathetic, and cultural expectations in conversations. It contains Reddit conversations from therapy, PhD supervision, and elderly-care, together with expert-defined rules and human annotations.

Critical strengths

S1) The paper is well-written, and evaluating both the final judgment and the supporting conversational evidence is a useful idea. S2) The benchmark could become a useful source for evaluating LLM judges and broader LLM capabilities if it is expanded to cover more domains, rules, and contexts.

Critical weaknesses

W1) The contribution is limited for the main ICLR track. In the current form, I believe that the work may be better fitting for a dataset/benchmark track or a relevant workshop. The paper mainly introduces a small, domain-specific dataset and evaluates existing models using different prompts, without proposing a new learning method or offering technical insight.

W2) The experiments & evaluation is limited to expert designed prompt guidance, although there are more technical ways to improve or guide model behavior. Evaluating a fine-tuned or aligned SLEEC-aware model would be particularly interesting and could make the work much stronger.

Questions for authors
  • Could the SLEEC rules and annotations be used for supervised fine-tuning or preference-based alignment?
  • How would such a SLEEC-aware model compare with the prompt-guided baselines?
Reviewer_hmwyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a benchmark for LLMs ability in identifying applicable rules, judging response compliance, and locating supporting evidence. It tests five LLMs in three domains with crowdsourced annotations.

Critical strengths

The evaluation of LLM-as-judge is a critical problem. The rule-level judgements with supporting evidence spans can be useful resources.

Critical weaknesses
  1. The critical problem is that the annotators are general crowdworkers, not domain experts. Judging which rules apply and whether they are satisfied in these three domains requires domain expertise. It is difficult to interpret a comparison between an LLM’s understanding of therapy and a non-clinician’s understanding of therapy. Non-clinicians’ judgments may be incorrect or differ substantially from clinicians’ judgments.

  2. Similarly, the rules were formulated by an “ethics expert” and reviewed by a “requirements engineering researcher,” with no reported involvement of experts in the application domains. Domain experts are needed to validate both the rules and the boundaries of their applicability.

  3. The incremental contribution over existing benchmarks is unclear. For example, HealthBench already includes conversations about therapy and care for older adults and compares LLM and expert judgments (not general crowdworkers). Moreover, each conversation has its own specific rubric written by clinicians.

  4. No human-human agreement is reported for rule applicability, compliance judgments, or evidence selection.

Questions for authors
  1. Were crowdworker judgments validated against domain experts, and what agreement was observed?
  2. Did application-domain experts validate the rules and their applicability boundaries?
  3. What does SleekAware add beyond HealthBench’s clinician-written rubrics and model–expert comparisons?
  4. What is the human–human agreement for rule applicability, compliance judgments, and evidence selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.