- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces HR Simulator, a game consisting of five workplace communication scenarios, to study how humans and LLMs write and evaluate professional emails. The authors compare human-written, LLM-written, and LLM-rewritten emails and analyze their tone and evaluation outcomes. They further examine agreement across 23 LLM judges and investigate how communication preferences relate to model capability, using descriptors such as empathy, specificity, and scenario-specific tact. The main findings are that LLMs generally produce more formal and empathetic emails, LLM judges tend to favor LLM-written emails, and higher-capability judges show greater agreement and potentially different preferences regarding tact.
Critical strengths
- A relevant research question.
Examining LLMs as both writers and evaluators of workplace communication is worthwhile, particularly as AI-mediated communication becomes more common. The paper draws attention to potentially important differences between human and model preferences.
- Extensive comparisons across models and evaluation conditions.
The authors examine multiple writing conditions, 23 LLM judges for agreement analysis, and 10 judges for detailed preference analysis. The inclusion of human annotations and additional controls is valuable.
Critical weaknesses
- The paper's novel scientific contribution and broader takeaway remain unclear.
The main observations—that LLMs tend to produce more formal and empathetic text, that LLM judges exhibit systematic stylistic preferences, and that their evaluations differ from human judgments—largely overlap with established findings in LLM generation and evaluation research. Although the paper demonstrates these patterns in workplace email scenarios, it is unclear what substantially new knowledge this setting provides.
The potentially more distinctive finding concerns capability-dependent judge preferences, but its implications are not sufficiently developed or validated. The paper raises the possibility that LLMs may reshape workplace communication norms, yet it does not investigate actual changes in communication behavior or demonstrate consequences of the observed preference differences.
The methodological contribution is also limited. HR Simulator consists of five manually designed scenarios using prompted LLMs to simulate recipients and evaluate outcomes. While this is a reasonable experimental setup, the paper does not establish why it constitutes a reusable research instrument rather than a task-specific collection of simulated scenarios. There is little evidence that the framework produces insights that generalize beyond its predefined situations and evaluation criteria.
Overall, despite the substantial experimental effort, I found it difficult to identify a sufficiently novel and consequential takeaway for the broader ICLR community.
- The evidence supporting the most distinctive findings is not sufficiently convincing.
The claim that higher-capability judges exhibit increasingly shared communication preferences relies on relatively modest correlations between judge agreement and model capability (r = .34 for parameter count and r = .39 for Arena Elo). Judge pairs are not independent because individual models appear in multiple pairs, and the analysis does not adequately establish whether the trend generalizes beyond the particular models tested.
The analysis of tact raises further concerns. The tact criteria were partly derived from the judges' own rationales, making the construct insufficiently independent of the preferences it is intended to explain. Although the regression reports a significant capability interaction, the analysis includes only 10 judge models, and clustering by email pair does not fully address dependence at the judge level. The judge-level correlation between tact preference and capability is not statistically conclusive (r = .54, p = .10), and the parameter-count sensitivity analysis does not reproduce the interaction.
Moreover, adding tact improves held-out prediction accuracy only from 78.6% to 79.1%. The authors acknowledge several of these limitations, but the current evidence still supports an exploratory observation rather than a robust finding about how communication preferences change with model capability.
These concerns matter because the capability-dependent analysis appears to be the paper's strongest candidate for a novel contribution. Without more convincing evidence, the remaining results are primarily descriptive observations of already familiar LLM behaviors.
Questions for authors
-
What is the central novel finding of this work beyond previously documented differences in human and LLM writing and LLM-as-judge preferences? How does HR Simulator enable scientific insights that could not be obtained using existing evaluation setups?
-
What evidence supports the generalizability and reusability of HR Simulator beyond its five predefined scenarios? Can the authors demonstrate that the observed preferences persist under different scenario designs or evaluation criteria?
-
Do the capability-dependent convergence and tact findings remain robust under judge-level uncertainty estimation, leave-one-judge-out analyses, or independently defined tact annotations? These results would substantially affect my assessment of the paper's main empirical contribution.
2 comments
Thank you for the detailed response. I appreciate the clarification that specificity and empathy are the main preference findings, while tact is intended as an exploratory analysis. The examples of how weaker and stronger judges evaluate the same communication choices differently are interesting, and the additional robustness checks are helpful. I also agree that directly observing long-term changes in workplace communication norms is beyond the scope of this study.
However, my main concern remains the novelty and broader scientific contribution of the findings. While HR Simulator provides a useful setting for studying LLM-mediated communication, I am still not fully convinced that the results offer sufficiently new insights beyond previously documented LLM writing tendencies and evaluation biases. The capability-dependent differences are potentially interesting, but their broader significance remains unclear, particularly given that the evidence comes from five predefined scenarios and their evaluation criteria.
I recognize the substantial experimental effort and the value of examining this problem in a workplace context. Nevertheless, the response has not sufficiently changed my assessment of the paper's scientific contribution or broader implications. I therefore maintain my original score.
Thank you for your comments. Please see our responses below.
Re: Novelty of contributions and broader takeaway.
We agree that LLMs’ stylistic tendencies and self-preference have been documented before. Our contribution shows that they persist in high-stakes workplace communication, a setting where LLMs increasingly read and reply to email on users’ behalf. Here, these tendencies are not only evaluation artifacts but can shape which messages get rewarded.
HR Simulator’s novel contribution is using a text-based game to study open-ended communication, where success depends on how the reader judges the message. Humans and LLMs write for the same scenarios under the same incentives, which makes their communication comparable. We designed a few scenarios carefully by hand rather than generating many noisy ones. Each scenario targets a general communication skill (declining a request, resolving conflict, reversing a commitment, seeking information, and persuasion) rather than situation-specific knowledge.
The judge rationales show the same pattern across scenarios: weaker judges reward concrete detail and warmth, while stronger judges attend more to strategic choices. For example, in scenario 1, weaker judges reward mentioning the recipient’s higher salary as a concrete appeal, while stronger judges see it as transactional (Table 2). In scenario 2, weaker judges reward prescribing specific creative decisions, while stronger judges warn that this may create resentment (Tables 11–14).
Observing how norms change as LLMs are adopted would take time beyond the scope of a single study. However, people already use LLMs to improve their emails. The open question is whether the quality metric is defined by human readers or by LLMs. Our results show that these definitions can differ, and that LLMs increasingly converge on their own definition as they become more capable.
Re: Robustness of capability-dependent findings.
Our main preference findings concern specificity and empathy rather than tact. Both have the largest associations with preference, with odds ratios above 1.5 for every judge, and both are weighted less by more capable judges (specificity: 2.51 to 1.55, empathy: 2.78 to 1.88). Specificity is defined independently of judge rationales. Prior work shows LLM judges favor concrete detail (Park et al., 2024) and we show that this preference weakens with capability.
Tact is a post-hoc characterization of how weaker and stronger judges disagree, used to test which effect remains beyond specificity. We therefore derived it from judge rationales by design and label it exploratory in the paper. A small gain is what we would expect if tact accounts for only the remaining differences after specificity and empathy.
Specificity serves as an independently defined check and shows the same capability trend as tact. Tact scores are also consistent across annotators: Gemini 3 Flash, Grok 4 Fast, and GPT-5.2 mostly correlate above 0.7 (Appendix B). Excluding Gemini 3 Flash, which supplied the annotations, as a judge leaves the tact interaction nearly unchanged (0.145 vs. 0.148).
References
Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. OffsetBias: Leveraging Debiased Data for Tuning Evaluators. Findings of the Association for Computational Linguistics: EMNLP 2024. https://arxiv.org/abs/2407.06551