- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that LLM evaluation on social media looks at what a post is about and rarely at what the poster is socially doing.
- Taxonomy: 3 classes and 7 subclasses, motivated by sociology (Granovetter, Katz, Habermas, Goffman).
- Dataset (Reddit-Scene): 6,003 posts from 22 subreddits with reply threads, labeled by three trained annotators (κ 0.78–0.84).
- Task 1 (SceneBench): 8 LLMs reach at most 54.2% 7-way accuracy (Table 1). On a 500-post subset they identify the source community at 87–90%, a difference the paper calls the "Topic–Role Gap" (Table 8a). Humans reach 78.3% and a fine-tuned RoBERTa-base 76.2% (Tables 3, 9). From this the paper concludes that LLMs fail specifically at role inference. Adding the reply thread lowers accuracy (Table 2).
- Task 2: in an LLM multi-agent simulation, adding social-function labels to posts raises engagement (Tables 4, 14).
Critical strengths
The question is a good one, and the paper illustrates it well: whether a model understands what someone is doing with a post, not just its topic. The Task 1 experiments also have an unusually thorough set of controls, including different amounts of thread context, length-matched random text, few-shot and CoT prompting, a human baseline and fine-tuned encoders. With a cleaner label set, this design would be very reusable.
Critical weaknesses
The labels are confusing, so I can't tell what the headline result measures.
- It is unclear whether the label is the poster's intent or what the thread ended up doing. The prompt asks the model for "the poster's hidden intention" (L848), but the gold label is defined by "the post's own framing and the thread's response pattern" (L775).
- The categories overlap. "Information Seeking" is defined as requesting knowledge in the main text, but as "providing or requesting factual information" in the prompt, and its examples are news headlines. In the paper's own same-topic example (Fig. 4), a news headline is labelled Information Seeking while a question about the same event is labelled Public Event Discussion.
- This matters because the models' biggest error is exactly here: 62% of Information Seeking posts are predicted as Public Event Discussion. Two other classes (family, workplace) describe a setting rather than a function, and posts with several functions are forced into one label. I'd suggest picking one target, giving each class a single consistent definition, and reporting per-class agreement.
The LLM-vs-human comparison isn't fair.
- The annotators had training, a calibration set and explicit rules for the boundary cases, including the one behind the main confusion. The LLMs got one-line category names. So the low LLM accuracy may reflect the prompt rather than a lack of social understanding. That a fine-tuned encoder can fit the labels shows they are consistent, not that they match the definitions the LLMs were given. Re-running the LLMs with the full annotation guidelines, and adding a fine-tuned LLM baseline, would make the "LLMs can't infer social roles" claim much more convincing.
Questions for authors
-
Is the gold label the poster's intent or the function the thread ended up serving? How many labels change when annotators see the full thread?
-
How do the LLMs do when given the full annotation guidelines and at least one example per class? And what about a fine-tuned LLM on the encoder's split?
-
Were all 6,003 posts evaluated by every model? Some numbers (e.g., "244 of 396" vs the class sizes in Fig. 3) suggest a subsample. For the subreddit task, what were the prompt and the answer options? A per-subreddit breakdown of labels would show whether the model can shortcut from topic to label.
-
In the simulation, each agent "selects one action" (L348), but likes plus comments exceed 100 per post. What exactly is the behavioural score, and which LLM drives the agents?
5 comments
Response to Reviewer L59P: (W1) Taxonomy Consistency, Category Overlap, and Annotation Reliability
We acknowledge an inconsistency in how the Information Seeking boundary was explained across the paper. Our taxonomy and evaluation guidelines define this category as providing or requesting factual information and knowledge. However, some boundary-rule discussions describe it more narrowly as requesting information, potentially obscuring why factual news reports also belong to this category. We clarify that both information provision and information requests fall under Information Seeking, whereas Public Event Discussion centers on expressing or soliciting evaluations and public debate.
1. Clarifying category definitions and boundaries.
We acknowledge that the original main-text definition of Information Seeking as “requesting knowledge” was narrower than the annotation guideline, which also includes providing factual information. We clarify the intended distinctions as follows:
| Category | Operational Criterion |
|---|---|
| Information Seeking | Requesting or conveying factual information, including questions, news, and factual updates |
| Public Event Discussion | Expressing positions, evaluating public issues, or inviting debate |
| Interest Exchange | Sharing personal interests, experiences, or opinions to facilitate exchange |
| Family Daily Life | Performing relational functions within family interactions |
| Workplace Collaboration | Coordinating activities or responsibilities within workplace interactions |
For example, in Figure 4, a news headline about a public event is labeled Information Seeking because its dominant framing is factual information delivery. A question inviting opinions about the same event is labeled Public Event Discussion because its communicative role is to initiate evaluation or debate.
Thus, the distinction depends on communicative framing and interaction evidence, not topic or sentence form alone.
We also clarify that Family Daily Life and Workplace Collaboration are relational functions rather than merely topic labels. Family relationships and workplace coordination are constitutive of the corresponding social interactions, although we acknowledge that these categories are partly defined by relational settings.
2. Handling overlapping functions and ambiguous posts.
We agree that a post may serve multiple communicative purposes. Assigning one dominant function is an operational choice for benchmark evaluation, not a claim that real-world communication is inherently single-functional.
Annotators identify the function that best characterizes the post's primary communicative role within the observed interaction. When no defensible dominant label can be established, the post is excluded.
During quality control, 41 posts with unresolved labels and 19 posts without interaction were excluded, leaving 6,003 posts from 6,063 candidates. These represent distinct exclusion reasons; not all 60 posts were ambiguous multi-function cases. 3. Category-level annotation reliability.
To assess the reliability of these distinctions, three annotators independently re-annotated 100 posts under Post-only and Full P–R–T conditions.
Overall Fleiss' κ was 0.866 and 0.849, respectively. Category-level one-vs-rest Cohen's κ, averaged across annotator pairs, is reported below:
| Social Function | Post-only κ | Full P–R–T κ |
|---|---|---|
| Interest Exchange | 0.82 | 0.73 |
| Information Seeking | 0.76 | 0.74 |
| Transaction & Cooperation | 0.96 | 0.96 |
| Public Event Discussion | 0.78 | 0.79 |
| Content Creation & Diffusion | 1.00 | 0.98 |
| Family Daily Life | 1.00 | 1.00 |
| Workplace Collaboration | 1.00 | 1.00 |
Agreement is generally high, but Information Seeking, Public Event Discussion, and Interest Exchange exhibit relatively lower reliability, consistent with their overlapping boundaries. Estimates for less frequent categories should be interpreted cautiously given the 100-post sample.
4. Do explicit definitions resolve the category confusion?
We further evaluated Qwen3.6-27B with the complete annotation codebook, including seven definitions, six boundary rules, and one example per category, on the same 500 posts.
| Category | Original Recall | Full Codebook Recall | Δ |
|---|---|---|---|
| Information Seeking | 27.2% | 32.9% | +5.7 |
| Public Event Discussion | 66.0% | 64.0% | −2.0 |
| Interest Exchange | 51.0% | 66.7% | +15.7 |
Explicit definitions improve some categories, particularly Interest Exchange, but substantial errors remain for Information Seeking. This suggests that definitional unfamiliarity contributes to the confusion but does not fully explain it. Applying overlapping boundaries and inferring the dominant communicative role remain challenging.
We therefore do not claim that all seven categories are equally distinguishable or that single-label annotation captures every secondary function.
Response to Reviewer L59P: (W2&Q2):
We thank the reviewer for questioning the fairness of comparing zero-shot LLMs with trained human annotators. We agree that human annotators received training and boundary-case calibration unavailable to zero-shot models. To assess this difference, we conducted two additional experiments: a Full Codebook Control and a supervised LLM baseline.
1. Does providing complete annotation guidelines improve zero-shot performance?
The original zero-shot prompt included brief definitions of all seven categories and three analysis hints, rather than category names alone. However, it did not include the complete annotation guidelines used for human calibration, particularly the six boundary rules and worked examples.
We therefore evaluated the same 500 posts with identical P–R–T inputs and decoding settings, adding the full codebook: seven category definitions, six boundary rules, and one worked example per category.
| Model | Original Prompt | Full Codebook | Δ |
|---|---|---|---|
| Qwen3.6-27B | 45.8% | 47.6% | +1.8 pp |
| DeepSeek-V4.1-Flash | 50.2% | 56.2% | +6.0 pp |
| Qwen3.8-Max (n=202) | 55.7% | 54.5% | −1.2 pp |
The effect is model-dependent. DeepSeek improves significantly (+6.0 points, McNemar p = .0005), whereas Qwen3.6-27B shows only a small, statistically insignificant improvement (+1.8 points, p = .31).
Thus, incomplete access to annotation guidelines contributes to some errors, but providing the full codebook does not eliminate the substantial zero-shot performance gap.
2. Can a supervised LLM learn the taxonomy?
We additionally fine-tuned Qwen3-4B using LoRA on the same training data, text fields, and test split as the supervised RoBERTa baseline. All conditions below use root-only inputs on the same 500-post evaluation subset.
| Model / Condition | Accuracy |
|---|---|
| Qwen3-4B, Zero-shot | 29.2% |
| Qwen3-4B, Full Codebook | 33.8% |
| Qwen3-4B, LoRA Fine-tuned | 74.6% |
| RoBERTa-base, Fine-tuned | 76.2% |
Providing the full codebook improves Qwen3-4B by only 4.6 points, whereas supervised fine-tuning improves accuracy by 45.4 points, approaching RoBERTa's 76.2%.
This demonstrates that LLMs can learn the seven-category classification task through supervision, while explicit definitions alone provide substantially smaller improvements.
3. Interpretation .
We acknowledge that zero-shot LLMs and trained human annotators do not receive equivalent preparation. Therefore, the human–LLM performance gap should not be interpreted as a controlled comparison of intrinsic social understanding.
Our experiments support a narrower conclusion: complete annotation guidelines only partially improve zero-shot performance, while supervised adaptation substantially increases in-distribution classification accuracy. High supervised accuracy, however, does not establish transferable social-function understanding.
Response to Reviewer L59P: (W1a + Q1)
We thank the reviewer for identifying the inconsistency between our annotation definition and evaluation prompt. We agree that inferring a poster's hidden intention and identifying the social function a post performs within its thread are not equivalent tasks. We clarify the intended evaluation target and provide new human re-annotation results.
1. What exactly does the gold label represent?
SceneBench evaluates the dominant social function a post performs within community interaction, rather than the poster's unobservable private intention.
The label is determined from two sources of evidence:
Post framing provides primary evidence, while thread responses help resolve ambiguous boundaries.
We acknowledge that the original evaluation prompt, "infer the poster's hidden intention," was inconsistent with this definition. We will replace this wording with instructions to "infer the social action the post performs" and "observe the comment section", explicitly considering how the post is framed and how others respond. This aligns the prediction target with the annotation criteria.
2. How often do human labels change when the full thread is provided?
We conducted an independent re-annotation study involving 100 posts, three annotators, and two conditions: Post-only and Full P–R–T.
Accuracy is calculated against the original reference labels; the new human judgments constitute an independent measurement rather than replacements for the gold labels.
Human accuracy decreases by only 2–3 percentage points per annotator after adding replies, and individual labels change in just 3–4% of cases. On the same 100 posts, model accuracy decreases by 15.0 percentage points (58.0% → 43.0%).
Among the ten annotator-level label changes, only one resolves a previous disagreement with the reference label; eight introduce new disagreements, and one changes between two non-reference labels.
These findings demonstrate that the model exhibits a substantially larger context-related accuracy decline than human annotators.
3. Does label stability contradict the interaction-level definition?
No. Interaction-grounded annotation does not imply that adding replies must change the dominant function. For each annotator, 96–97% of judgments remain unchanged between Post-only and Full P–R–T conditions. Thread evidence can nevertheless be important for resolving particular boundary cases.
We do not claim that the 100-post study establishes the proportion of all 6,003 posts whose labels would change under a thread-blind annotation protocol.