- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CARE, a benchmark for evaluating how well LLMs reproduce reactions from online communities. The authors collect Reddit comments about COVID-19 news from 207 communities and compare them with simulated reactions from Gemini models. They evaluate both general attitudes and more specific tones, such as sarcasm or frustration.
Critical strengths
I think this is an interesting research question, especially given how often LLMs are used to simulate human behavior and opinions.
I also appreciate the data that consists of 207 Reddit communities. The distinction between overall community behavior and reactions to specific events is interesting.
Critical weaknesses
The authors use the five most upvoted comments to represent a community's reaction to a news article. However, I'm not sure these comments necessarily represent the community as a whole. Highly upvoted comments may reflect popular opinions, but they may also be influenced by timing, moderation, or Reddit voting behavior. (UPD: I still see no answer from authors, so again I highlight the weakness is that only highly upvoted comments are taken into account)
I'm also concerned that the authors generate only one simulated comment per news article. Real communities can have very different reactions to the same event. I wonder whether generating multiple responses and comparing their distributions would be more appropriate.
The authors use GPT-5 to annotate tones and then map them to broader attitudes. I appreciate that they include a human validation study. However, I'm not sure that assigning a fixed attitude to each tone is always appropriate.
For example, an assertive or sarcastic comment can express either a positive or negative opinion depending on the context. I wonder how much this mapping affects the results.
I also noticed that the community-informed CoT condition receives more information than the regular community-informed condition. It includes the subreddit description, while the other condition only provides the subreddit name. This makes it difficult to tell whether the differences come from reasoning or simply from having more information.
UPD: I still stand by these points and believe that the idea of the paper is great, but for clear acceptance, the points I mentioned must be addressed.
Questions for authors
Why did you choose only the five most upvoted comments to represent each community? Have you considered using a more diverse sample of reactions?
Have you tried generating multiple comments per article? I'm curious whether this would change the instance-level results.
How do you handle tones that can express different attitudes depending on context? For example, sarcasm is not necessarily negative.
UPD: For changing my score to 3, I still wait for answers
Reply to avQd
Thank you for following up. Below are answers to each point, with pointers to the revised PDF.
1. Why the five most upvoted comments? (W1, Q1) Votes are the community's own signal of which reactions it endorses. A member who would say the same thing tends to upvote an existing comment, or reply under it, rather than post another top-level comment, so the votes on a top-level comment also carry the agreement of members who did not write it. We use top-level comments because nested replies tend to drift from the article (§3.1). The cap of five came from inspecting the raw data: even among posts with at least ten comments, many have only a few top-level comments (in the final set, 24% of posts have fewer than five), so a larger cap would mostly shrink the pool of eligible posts or leave more posts below it. We agree that votes are also shaped by timing and moderation: the five comments are the reactions the community promoted, not a sample of the whole thread, and we will state this scope explicitly in the final version.
2. Multiple comments per article? (W2, Q2) Yes, for Gemini 3.1 Pro: two additional full runs of every condition on all 2,194 posts (App. D, Table 10). The instance-level results barely change: across the three runs, the standard deviation is at most 0.005 for AttHit and 0.010 for per-post Spearman ρ and EM_syn, and the orderings between conditions hold in all 3 × 3 pairings of runs. On comparing distributions: the paper evaluates LLM simulation as it is commonly used, one simulated reaction per prompt at the provider's default temperature. How many voices should represent a community, and how many reactions a prompted LLM should produce, are separate design choices; we see distribution-level simulation as an open question the benchmark invites, and the coverage design of AttHit is what lets a single reaction be scored against a plural community.
3. Tones whose attitude depends on context (W3, Q3) The tone-to-attitude mapping is context-independent by design: each tone receives one attitude, so the attitude label never depends on a second, contextual judgment (§4.2). Assigning one of three values (positive +1, negative −1, neutral/mixed 0) to 364 tones is necessarily coarse, which is why attitude is only the coarsest layer of our hierarchical evaluation. The tone-level results (EM, EM_syn, JSD, sarcasm share, tonal diversity and structure) work on the tones and synonym groups themselves and do not use the mapping at all. For sarcasm, if one value must be chosen, we believe −1 fits best: people rarely sound sarcastic when their reaction is truly positive, and at most a sarcastic comment may read as mixed (Contempt Mockery & Wit → Sharp Sarcasm, Table 4), unlike playful (Humor Playfulness & Sincerity → Amusement & Wit, Positive, Table 5).
Assertive, on the other hand, raises a point well taken. Its attitude was debated among the authors when we built the taxonomy, and your comment helped us finally decide: assertive and its synonyms emphatic and confident (the Strong Conviction subgroup) are now mapped to Mixed instead of Positive (App. B, Table 7), and all attitude results use the updated mapping. The change moves AttHit by at most 0.005 and reverses no comparison between conditions in the paper.
4. Comm-Inf.-CoT receives more information (W4) That is right: the condition adds context (the subreddit description and the 2020 framing) and written reasoning at once. We separated the two (App. G): a full 2 × 2 on Claude Sonnet 5.5 (all posts) and a preliminary context-only run on Gemini 3.1 Pro (1,000 posts). For Sonnet, the reasoning step drives the change (AttHit +0.03, EM_syn +0.035; context alone: no clear change); for Gemini 3.1 Pro, the added context drives it (EM_syn −0.034, sarcasm share −8.2 points; reasoning on top: no clear change). Since the answer depends on the model, the main text now attributes these effects to the Comm-Inf.-CoT condition as a whole, and §4.1 describes everything it adds. The new prompts are in App. H.4–H.5.
We hope these clarifications and new analyses address your concerns, and we would be grateful if you would take them into account in your final assessment. We are happy to clarify anything further.