acceptodds
Under review as a conference paper at ICLR 2027

Beyond Topic Recognition: Social-Function Understanding in Large Language Models

Abstract

Large Language Models (LLMs) have made significant progress in social media mining, yet existing evaluations predominantly focus on surface-level topics and content categories, leaving the communicative functions of online interactions comparatively underexplored. To address this gap, we introduce a theory-informed framework for social-function understanding in social media. Guided by Granovetter’s tie strength theory, Katz’s uses-and-gratifications theory, and Habermas’s public sphere theory, we construct a social-function taxonomy with three top-level categories (Relational, Instrumental, Public) and seven fine-grained subcategories. On this basis, we make three contributions: (1) Reddit-Scene, a multimodal dataset of 6,003 annotated samples from 22 Reddit communities, organized in a three-tier Post–Pillar Comment–Thread (P-R-T) structure with five micro-sociological features; (2) SceneBench, a benchmark evaluating eight frontier LLMs on social-function inference, where the best model reaches only 54.2% seven-class accuracy while a controlled community-identification probe reaches 90.1%, revealing a substantial topic–role gap that is not reliably resolved by additional conversational context; and (3) a controlled multi-agent behavioral probe showing that social-function information has an observable influence on downstream agent responses. A fine-tuned encoder reaches 76.2%, and a three-annotator human baseline reaches 78.3% on social-function inference but 84.7% on community identification, further suggesting that the main difficulty lies in social-role inference rather than surface community recognition.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Large Language Models, Social Function

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_L59POpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that LLM evaluation on social media looks at what a post is about and rarely at what the poster is socially doing.

  • Taxonomy: 3 classes and 7 subclasses, motivated by sociology (Granovetter, Katz, Habermas, Goffman).
  • Dataset (Reddit-Scene): 6,003 posts from 22 subreddits with reply threads, labeled by three trained annotators (κ 0.78–0.84).
  • Task 1 (SceneBench): 8 LLMs reach at most 54.2% 7-way accuracy (Table 1). On a 500-post subset they identify the source community at 87–90%, a difference the paper calls the "Topic–Role Gap" (Table 8a). Humans reach 78.3% and a fine-tuned RoBERTa-base 76.2% (Tables 3, 9). From this the paper concludes that LLMs fail specifically at role inference. Adding the reply thread lowers accuracy (Table 2).
  • Task 2: in an LLM multi-agent simulation, adding social-function labels to posts raises engagement (Tables 4, 14).
Critical strengths

The question is a good one, and the paper illustrates it well: whether a model understands what someone is doing with a post, not just its topic. The Task 1 experiments also have an unusually thorough set of controls, including different amounts of thread context, length-matched random text, few-shot and CoT prompting, a human baseline and fine-tuned encoders. With a cleaner label set, this design would be very reusable.

Critical weaknesses

The labels are confusing, so I can't tell what the headline result measures.

  • It is unclear whether the label is the poster's intent or what the thread ended up doing. The prompt asks the model for "the poster's hidden intention" (L848), but the gold label is defined by "the post's own framing and the thread's response pattern" (L775).
  • The categories overlap. "Information Seeking" is defined as requesting knowledge in the main text, but as "providing or requesting factual information" in the prompt, and its examples are news headlines. In the paper's own same-topic example (Fig. 4), a news headline is labelled Information Seeking while a question about the same event is labelled Public Event Discussion.
  • This matters because the models' biggest error is exactly here: 62% of Information Seeking posts are predicted as Public Event Discussion. Two other classes (family, workplace) describe a setting rather than a function, and posts with several functions are forced into one label. I'd suggest picking one target, giving each class a single consistent definition, and reporting per-class agreement.

The LLM-vs-human comparison isn't fair.

  • The annotators had training, a calibration set and explicit rules for the boundary cases, including the one behind the main confusion. The LLMs got one-line category names. So the low LLM accuracy may reflect the prompt rather than a lack of social understanding. That a fine-tuned encoder can fit the labels shows they are consistent, not that they match the definitions the LLMs were given. Re-running the LLMs with the full annotation guidelines, and adding a fine-tuned LLM baseline, would make the "LLMs can't infer social roles" claim much more convincing.
Questions for authors
  • Is the gold label the poster's intent or the function the thread ended up serving? How many labels change when annotators see the full thread?

  • How do the LLMs do when given the full annotation guidelines and at least one example per class? And what about a fine-tuned LLM on the encoder's split?

  • Were all 6,003 posts evaluated by every model? Some numbers (e.g., "244 of 396" vs the class sizes in Fig. 3) suggest a subsample. For the subreddit task, what were the prompt and the answer options? A per-subreddit breakdown of labels would show whether the model can shortcut from topic to label.

  • In the simulation, each agent "selects one action" (L348), but likes plus comments exceed 100 per post. What exactly is the behavioural score, and which LLM drives the agents?

5 comments

Response to Reviewer L59P: (W1a + Q1)

We thank the reviewer for identifying the inconsistency between our annotation definition and evaluation prompt. We agree that inferring a poster's hidden intention and identifying the social function a post performs within its thread are not equivalent tasks. We clarify the intended evaluation target and provide new human re-annotation results.

1. What exactly does the gold label represent?

SceneBench evaluates the dominant social function a post performs within community interaction, rather than the poster's unobservable private intention.

The label is determined from two sources of evidence:

  • Post framing: What communicative action does the post perform, such as requesting information, sharing experiences, or initiating discussion?
  • Interaction evidence: How do replies engage with the post, such as answering questions, exchanging experiences, or debating public issues?

Post framing provides primary evidence, while thread responses help resolve ambiguous boundaries.

We acknowledge that the original evaluation prompt, "infer the poster's hidden intention," was inconsistent with this definition. We will replace this wording with instructions to "infer the social action the post performs" and "observe the comment section", explicitly considering how the post is framed and how others respond. This aligns the prediction target with the annotation criteria.

2. How often do human labels change when the full thread is provided?

We conducted an independent re-annotation study involving 100 posts, three annotators, and two conditions: Post-only and Full P–R–T.

EvaluatorPost-only AccuracyFull AccuracyLabel Change Rate
Annotator A74.0%72.0%4.0%
Annotator B71.0%68.0%3.0%
Annotator C75.0%73.0%3.0%
Qwen3.6-27B (same posts)58.0%43.0%—

Accuracy is calculated against the original reference labels; the new human judgments constitute an independent measurement rather than replacements for the gold labels.

Human accuracy decreases by only 2–3 percentage points per annotator after adding replies, and individual labels change in just 3–4% of cases. On the same 100 posts, model accuracy decreases by 15.0 percentage points (58.0% → 43.0%).

Among the ten annotator-level label changes, only one resolves a previous disagreement with the reference label; eight introduce new disagreements, and one changes between two non-reference labels.

These findings demonstrate that the model exhibits a substantially larger context-related accuracy decline than human annotators.

3. Does label stability contradict the interaction-level definition?

No. Interaction-grounded annotation does not imply that adding replies must change the dominant function. For each annotator, 96–97% of judgments remain unchanged between Post-only and Full P–R–T conditions. Thread evidence can nevertheless be important for resolving particular boundary cases.

We do not claim that the 100-post study establishes the proportion of all 6,003 posts whose labels would change under a thread-blind annotation protocol.

Response to Reviewer L59P: (W1) Taxonomy Consistency, Category Overlap, and Annotation Reliability

We acknowledge an inconsistency in how the Information Seeking boundary was explained across the paper. Our taxonomy and evaluation guidelines define this category as providing or requesting factual information and knowledge. However, some boundary-rule discussions describe it more narrowly as requesting information, potentially obscuring why factual news reports also belong to this category. We clarify that both information provision and information requests fall under Information Seeking, whereas Public Event Discussion centers on expressing or soliciting evaluations and public debate.

1. Clarifying category definitions and boundaries.

We acknowledge that the original main-text definition of Information Seeking as “requesting knowledge” was narrower than the annotation guideline, which also includes providing factual information. We clarify the intended distinctions as follows:

CategoryOperational Criterion
Information SeekingRequesting or conveying factual information, including questions, news, and factual updates
Public Event DiscussionExpressing positions, evaluating public issues, or inviting debate
Interest ExchangeSharing personal interests, experiences, or opinions to facilitate exchange
Family Daily LifePerforming relational functions within family interactions
Workplace CollaborationCoordinating activities or responsibilities within workplace interactions

For example, in Figure 4, a news headline about a public event is labeled Information Seeking because its dominant framing is factual information delivery. A question inviting opinions about the same event is labeled Public Event Discussion because its communicative role is to initiate evaluation or debate.

Thus, the distinction depends on communicative framing and interaction evidence, not topic or sentence form alone.

We also clarify that Family Daily Life and Workplace Collaboration are relational functions rather than merely topic labels. Family relationships and workplace coordination are constitutive of the corresponding social interactions, although we acknowledge that these categories are partly defined by relational settings.

2. Handling overlapping functions and ambiguous posts.

We agree that a post may serve multiple communicative purposes. Assigning one dominant function is an operational choice for benchmark evaluation, not a claim that real-world communication is inherently single-functional.

Annotators identify the function that best characterizes the post's primary communicative role within the observed interaction. When no defensible dominant label can be established, the post is excluded.

During quality control, 41 posts with unresolved labels and 19 posts without interaction were excluded, leaving 6,003 posts from 6,063 candidates. These represent distinct exclusion reasons; not all 60 posts were ambiguous multi-function cases. 3. Category-level annotation reliability.

To assess the reliability of these distinctions, three annotators independently re-annotated 100 posts under Post-only and Full P–R–T conditions.

Overall Fleiss' κ was 0.866 and 0.849, respectively. Category-level one-vs-rest Cohen's κ, averaged across annotator pairs, is reported below:

Social FunctionPost-only κFull P–R–T κ
Interest Exchange0.820.73
Information Seeking0.760.74
Transaction & Cooperation0.960.96
Public Event Discussion0.780.79
Content Creation & Diffusion1.000.98
Family Daily Life1.001.00
Workplace Collaboration1.001.00

Agreement is generally high, but Information Seeking, Public Event Discussion, and Interest Exchange exhibit relatively lower reliability, consistent with their overlapping boundaries. Estimates for less frequent categories should be interpreted cautiously given the 100-post sample.

4. Do explicit definitions resolve the category confusion?

We further evaluated Qwen3.6-27B with the complete annotation codebook, including seven definitions, six boundary rules, and one example per category, on the same 500 posts.

CategoryOriginal RecallFull Codebook RecallΔ
Information Seeking27.2%32.9%+5.7
Public Event Discussion66.0%64.0%−2.0
Interest Exchange51.0%66.7%+15.7

Explicit definitions improve some categories, particularly Interest Exchange, but substantial errors remain for Information Seeking. This suggests that definitional unfamiliarity contributes to the confusion but does not fully explain it. Applying overlapping boundaries and inferring the dominant communicative role remain challenging.

We therefore do not claim that all seven categories are equally distinguishable or that single-label annotation captures every secondary function.

Response to Reviewer L59P: (W2&Q2):

We thank the reviewer for questioning the fairness of comparing zero-shot LLMs with trained human annotators. We agree that human annotators received training and boundary-case calibration unavailable to zero-shot models. To assess this difference, we conducted two additional experiments: a Full Codebook Control and a supervised LLM baseline.

1. Does providing complete annotation guidelines improve zero-shot performance?

The original zero-shot prompt included brief definitions of all seven categories and three analysis hints, rather than category names alone. However, it did not include the complete annotation guidelines used for human calibration, particularly the six boundary rules and worked examples.

We therefore evaluated the same 500 posts with identical P–R–T inputs and decoding settings, adding the full codebook: seven category definitions, six boundary rules, and one worked example per category.

ModelOriginal PromptFull CodebookΔ
Qwen3.6-27B45.8%47.6%+1.8 pp
DeepSeek-V4.1-Flash50.2%56.2%+6.0 pp
Qwen3.8-Max (n=202)55.7%54.5%−1.2 pp

The effect is model-dependent. DeepSeek improves significantly (+6.0 points, McNemar p = .0005), whereas Qwen3.6-27B shows only a small, statistically insignificant improvement (+1.8 points, p = .31).

Thus, incomplete access to annotation guidelines contributes to some errors, but providing the full codebook does not eliminate the substantial zero-shot performance gap.

2. Can a supervised LLM learn the taxonomy?

We additionally fine-tuned Qwen3-4B using LoRA on the same training data, text fields, and test split as the supervised RoBERTa baseline. All conditions below use root-only inputs on the same 500-post evaluation subset.

Model / ConditionAccuracy
Qwen3-4B, Zero-shot29.2%
Qwen3-4B, Full Codebook33.8%
Qwen3-4B, LoRA Fine-tuned74.6%
RoBERTa-base, Fine-tuned76.2%

Providing the full codebook improves Qwen3-4B by only 4.6 points, whereas supervised fine-tuning improves accuracy by 45.4 points, approaching RoBERTa's 76.2%.

This demonstrates that LLMs can learn the seven-category classification task through supervision, while explicit definitions alone provide substantially smaller improvements.

3. Interpretation .

We acknowledge that zero-shot LLMs and trained human annotators do not receive equivalent preparation. Therefore, the human–LLM performance gap should not be interpreted as a controlled comparison of intrinsic social understanding.

Our experiments support a narrower conclusion: complete annotation guidelines only partially improve zero-shot performance, while supervised adaptation substantially increases in-distribution classification accuracy. High supervised accuracy, however, does not establish transferable social-function understanding.

Reviewer_iAbEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether LLMs can identify the social function of social media posts, meaning the communicative purpose a post serves in an interaction. The authors propose a taxonomy with three broad categories and seven subcategories, and introduce Reddit-Scene, a dataset of 6,003 posts. The experiments show that the evaluated models perform much better at community identification than social function classification. Adding real comments does not consistently improve classification and often reduces accuracy. Providing social function labels also changes the responses of simulated agents.

Critical strengths
  1. The task and dataset are useful and could support research on social agents.

  2. Several control experiments help assess alternative explanations. For example, the difference between real threads and random text of the same length in Table 2 shows that input length alone cannot fully explain the performance drop.

Critical weaknesses
  1. The distinction from intent and dialogue act recognition could be clearer.

  2. Choosing the dominant function of posts with multiple purposes still seems subjective. For example, how is the label chosen when a post both provides factual information and invites opinions? More examples and statistics on excluded ambiguous posts would help.

  3. RoBERTa reaches 76.2% accuracy after supervised training. Could the low zero shot results partly reflect misalignment with the annotation conventions that training can correct? Testing on unseen communities or later time periods would help show whether training on this dataset improves generalizable social function understanding.

  4. The error analysis does not clearly separate misunderstanding the label definitions from difficulty inferring a post’s social function. A more systematic analysis of these possibilities would strengthen the conclusions, especially for categories with very low accuracy.

Questions for authors
  1. Since all posts were independently annotated by three annotators, why is agreement reported only on 100 reannotated posts? Could you also report agreement across the full set of initial annotations?
4 comments

Response to Reviewer iAbE: W1

We thank the reviewer for noting that the distinction from existing work on dialogue acts, speech acts, and communicative intent needs to be clearer. We agree that SceneBench builds on established research on communicative behavior; our contribution is not a new theory of communication, but a distinct evaluation target and benchmark.

1. Different units and evaluation targets.

Dialogue-act recognition typically classifies communicative acts at the utterance or turn level, while speech-act research characterizes actions performed through language. Communicative-intent recognition focuses on the goals a speaker seeks to achieve. SceneBench instead evaluates the dominant social function of a platform post within its community interaction.

To make this distinction explicit, we will revise §2.1 to define social function as "the communicative action [a post] performs within its thread, as evidenced by the post's own framing and the response it elicits—not the poster's private intention, which is neither observable nor annotatable." This definition will clarify that our target is an observable interaction-level function, rather than an attempt to recover the author's internal intention.

2. Different operationalization and annotation evidence.

SceneBench determines social function from the post's communicative framing together with observable interaction evidence. Its theory-informed taxonomy draws on relational ties (Granovetter, 1973), uses and gratifications (Katz et al., 1973), and public discourse (Habermas, 1962), motivating three broad classes—Relational, Instrumental, and Public—and seven fine-grained categories.

For example, posts with similar question-like linguistic forms may serve different social functions: requesting factual information (Information Seeking), exchanging experiences (Interest Exchange), or inviting discussion of a public issue (Public Event Discussion). Conversely, the same social function can be expressed through different linguistic forms. Thus, dialogue-act and social-function labels may overlap, but are not interchangeable.

3. Complementary evaluation scope and prompt alignment.

Beyond label prediction, SceneBench examines social-function inference under different conversational-context conditions and whether inferred social-function cues influence downstream simulated interactions. This connects function recognition, context sensitivity, and behavioral consequences within one benchmark.

We will make three corresponding revisions in the manuscript: (1) explicitly define social function in §2.1 using observable interaction evidence rather than private intention; (2) strengthen §5 to distinguish dialogue-act, speech-act, and communicative-intent research, clarifying that "our unit is the post as a public act, labeled by its framing and by how the thread responds"; and (3) align the evaluation prompt in Appendix B.1.2 with this definition by replacing the previous "hidden intention" wording with "infer the social action the post performs."

We therefore position SceneBench as complementary to existing work: it evaluates what a post socially does within community interaction, rather than replacing established theories of communicative acts or intent.

Response to Reviewer iAbE: W2

We thank the reviewer for highlighting the potential subjectivity of assigning a single dominant function to posts serving multiple purposes. We agree that a post may simultaneously provide factual information and invite opinions. Our goal is not to assume that each post has only one function, but to identify its dominant function within the observed interaction.

1. Explicit boundary rules and dominant-label adjudication.

We have established four explicit category-boundary rules in Appendix A.4.3 to distinguish potentially overlapping categories: (i) Information Seeking vs. Interest Exchange, (ii) Information Seeking vs. Public Event Discussion, (iii) Transaction & Cooperation vs. Interest Exchange, and (iv) Relational vs. Instrumental. Appendix A.4.4 further specifies that the dominant function is determined based on both the post's communicative framing and the observed thread responses. When multiple functions remain equally plausible after applying these rules, the post is excluded rather than forced into a single category.

For the reviewer's example of a post that both provides factual information and invites opinions, Information Seeking is appropriate when obtaining specific information is the primary communicative function, whereas Public Event Discussion is appropriate when factual information mainly provides context for expressing or soliciting opinions on a public issue. Thus, factual content alone does not determine the label; the decision depends on the post's framing and how the thread responds.

2. Concrete examples of boundary decisions.

To make the annotation decisions more transparent, we will retain the representative examples for all seven categories in Table 7, together with their corresponding decisive criteria, and expand Appendix A.4.5 with an additional interaction-level boundary case. Specifically, we will include a post that both provides factual information and invites opinions, illustrating how its framing and observed thread responses distinguish Information Seeking from Public Event Discussion. We will explain why the function that primarily organizes the interaction is selected as dominant, rather than treating every expressed purpose as an equally valid label.

3. Annotation reliability.

The reproducibility of these annotation decisions is supported by our independent reliability assessment on a 100-post re-annotated subset. Pairwise Cohen's κ ranges from 0.78 to 0.84 for the seven-class taxonomy and from 0.81 to 0.87 for the three-category grouping. We acknowledge that these aggregate estimates do not establish equal reliability across all seven fine-grained categories, but provide evidence that the annotation protocol can yield consistent judgments beyond an individual annotator's interpretation.

4. Excluded ambiguous posts.

During quality control, 41 posts were excluded because annotators could not assign a label (empty or irreducibly ambiguous), and 19 were excluded because they contained no interaction. If confirmed by the original data-processing records, these 60 exclusions represent approximately 1.0% of the 6,063 candidate posts. The original records do not support a reliable further breakdown of the 41 posts into empty and irreducibly ambiguous cases. We therefore retain the aggregate count rather than introduce unsupported estimates. We will clarify this limitation in Appendix A.4 and explain that exclusion is reserved for cases where the available content and interaction evidence do not support a reliable dominant-function decision.

Overall, SceneBench does not assume that posts serve mutually exclusive purposes. Instead, it operationalizes dominant-function annotation through explicit category-boundary rules, observable interaction evidence, and a principled exclusion procedure. These revisions will make the annotation decisions and their limitations more transparent and reproducible.

Response to Reviewer iAbE: W3

We thank the reviewer for raising two questions: whether low zero-shot accuracy reflects unfamiliarity with annotation conventions, and whether supervised training generalizes beyond observed communities. We conducted three complementary experiments.

1. Full Codebook Control: Can explicit guidelines close the gap?

We evaluated the same 500 posts with identical inputs and decoding settings, adding seven category definitions, six boundary rules, and one example per category.

ModelOriginalCodebookΔ
Qwen3.6-27B45.8%47.6%+1.8
DeepSeek-V4.1-Flash50.2%56.2%+6.0
Qwen3.8-Max (n=202)55.7%54.5%−1.2

The improvements are limited and model-dependent. Annotation conventions explain some errors, but cannot fully account for low zero-shot performance.

2. Supervised Fine-tuning: Does training help beyond definitions?

Using the same root-only inputs and 500-post test split:

ConditionAccuracy
Qwen3-4B Zero-shot29.2%
Qwen3-4B + Codebook33.8%
Qwen3-4B + LoRA74.6%
RoBERTa-base (supervised)76.2%

Supervised training produces much larger gains than providing definitions alone. However, high in-domain accuracy does not establish transferable understanding.

3. Cross-Community Transfer: Does supervision generalize?

We held out seven communities entirely from training and evaluated both conditions on the same 196 posts.

Training ConditionAccuracyMacro-F1
All 22 communities69.4%47.9%
15 communities; 7 unseen46.9%24.3%
Difference−22.5−23.6

The 69.4% control differs from the original 76.2% because of the evaluation subset and training configuration; both transfer conditions use identical settings except community coverage.

The substantial decline confirms that supervised performance partly relies on community-specific patterns. We therefore do not interpret RoBERTa's in-domain accuracy as evidence of fully generalizable social-function understanding.

4. Interpretation and revision.

Together, these results show that (i) annotation misalignment explains only part of zero-shot errors, (ii) supervised learning captures useful predictive structure, but (iii) cross-community generalization remains limited.

This limitation concerns supervised transfer; our main zero-shot findings, including the Topic–Role Gap and context effects, remain supported by their matched evaluation comparisons. Temporal generalization remains future work.

We will incorporate these experiments and clarify the distinction between annotation alignment, supervised learnability, and generalization in the revised manuscript.

Response to Reviewer iAbE: W4

We thank the reviewer for highlighting the need to distinguish misunderstanding of label definitions from difficulties in social-function inference. We address this through a Full Codebook Control and a systematic per-category recall analysis.

1. Full Codebook Control.

We evaluated the same 500 posts with identical P–R–T inputs and decoding settings, adding the complete annotation guidelines: seven category definitions, six boundary rules, and one example per category.

ModelOriginalFull CodebookΔ
Qwen3.6-27B45.8%47.6%+1.8
DeepSeek-V4.1-Flash50.2%56.2%+6.0

The limited, model-dependent improvements indicate that unfamiliarity with label definitions contributes to errors but does not fully explain the low accuracy.

2. Per-category error decomposition.

We further compared category-level recall under the two conditions for Qwen3.6-27B.

Social FunctionOriginal RecallCodebook RecallΔ
Interest Exchange51.0%66.7%+15.7
Information Seeking27.2%32.9%+5.7
Transaction & Cooperation92.6%85.2%−7.4
Public Event Discussion66.0%64.0%−2.0
Content Creation & Diffusion31.1%22.6%−8.5
Family Daily Life54.5%63.6%+9.1
Workplace Collaboration72.0%80.0%+8.0

The codebook produces heterogeneous effects. Interest Exchange improves substantially (+15.7 points), suggesting that clearer definitions help resolve some classification errors.

However, Information Seeking remains low (27.2% → 32.9%), while Content Creation & Diffusion deteriorates (31.1% → 22.6%). These findings suggest that providing definitions alone is insufficient for several difficult categories.

3. Interpretation.

The results distinguish two contributing factors: definitional unfamiliarity, reflected in improvements after supplying explicit rules, and residual classification difficulty, reflected in errors that persist despite those rules.

We emphasize that this is an indirect decomposition: persistent errors may also involve ambiguous category boundaries or difficulty applying the definitions, rather than social inference alone.

Response to Reviewer iAbE: Q

We thank the reviewer for identifying this ambiguity. We apologize that our original description incorrectly suggested that every post was independently annotated by three annotators. The reported agreement statistics come from a separate 100-post re-annotation study, not from three independent labels for the entire corpus.

  1. Clarification of the annotation protocol

The original annotation process produced one final label per post through annotation and quality control. Three independent judgments were not retained for every post. Therefore, full-corpus inter-annotator agreement cannot be reconstructed from the original records.

To assess annotation reliability, three annotators independently re-annotated 100 posts under two conditions: Post-only and Full P–R–T.

2. Agreement results.

Agreement MetricPost-onlyFull P–R–T
Fleiss' κ (3 annotators)0.8660.849
Pairwise Cohen's κ (range)0.812–0.8990.812–0.900

We also calculated category-level one-vs-rest Cohen's κ, averaged across annotator pairs:

Social FunctionPost-only κFull P–R–T κ
Interest Exchange0.820.73
Information Seeking0.760.74
Transaction & Cooperation0.960.96
Public Event Discussion0.780.79
Content Creation & Diffusion1.000.98
Family Daily Life1.001.00
Workplace Collaboration1.001.00

The results show high overall agreement, with relatively lower reliability for Information Seeking and Public Event Discussion. Category-level estimates for less frequent classes should be interpreted cautiously given the 100-post sample size.

Reviewer_wSKvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SceneBench, a benchmark for evaluating social-function understanding in LLMs. It constructs Reddit-Scene, containing 6,003 annotated posts with a three-level social-function taxonomy and conversational context. Experiments with eight LLMs reveal a gap between community recognition and social-function inference, and show that additional conversational context can reduce accuracy. A multi-agent simulation further examines how social-function information affects agent responses.

Critical strengths
  1. The paper studies an interesting aspect of social understanding that goes beyond topic classification. The distinction between content and communicative function is well motivated.

  2. Reddit-Scene provides a useful dataset with conversational structures and theory-informed annotations. The annotation agreement analysis also supports its reliability.

  3. The evaluation is relatively comprehensive, including multiple LLMs, human and supervised baselines, context ablations, and controlled behavioral simulations.

Critical weaknesses
  1. The seven-category taxonomy includes potentially overlapping communicative functions. Assigning a single dominant label may oversimplify posts serving multiple purposes, while category-level annotation reliability is not reported.

  2. The novelty of social-function understanding is not sufficiently distinguished from existing research on dialogue acts, speech acts, and communicative intent.

  3. The Topic–Role Gap compares community identification with social-function classification, which differ in task difficulty and label semantics. The accuracy difference alone does not establish a specific deficiency in social reasoning.

  4. The interpretation of context misuse needs further justification. Since the gold labels are assigned using conversational context, the higher Post-Only accuracy raises questions about how context-dependent social functions are defined and evaluated.

  5. The behavioral analysis relies entirely on LLM-based simulation. The Predicted and Oracle conditions perform similarly despite substantial differences in label accuracy, and the gain over shuffled labels is relatively small. The practical significance of these behavioral differences remains unclear.

Questions for authors
  1. Clarify the distinction from dialogue-act and communicative-intent research.

  2. Discuss overlapping categories and the use of single dominant labels.

  3. Provide stronger support for the Topic–Role Gap and context misuse conclusions.

  4. Clarify the interpretation and practical significance of the simulation results.

5 comments

Response to Reviewer wSKv: (W1 + Q2)

We thank the reviewer for raising concerns about overlapping communicative functions and the subjectivity of assigning a single dominant label. We agree that a post may serve multiple purposes. Our single-label design is an operational choice for benchmark evaluation, not an assumption that communication is inherently single-functional. We address this through explicit decision criteria, independent annotation analysis, and transparent reporting of unresolved cases.

1. How is the dominant function determined?

The dominant function is determined from the post's communicative framing and observed interaction, rather than its topic or linguistic form alone. When a post serves multiple purposes, annotators identify the function that best characterizes its primary communicative role in the thread.

For example, a post about public policy that provides factual information and invites opinions is classified according to its primary purpose: requesting specific facts or information corresponds to Information Seeking, whereas soliciting positions, evaluations, or public debate corresponds to Public Event Discussion. Similarly, questions seeking concrete answers or solutions are distinguished from those inviting personal experiences or opinions (Interest Exchange).

Thus, classification depends on the communicative role performed in context. Cases without a defensible dominant function after applying these criteria are excluded rather than forcibly labeled.

2. Are annotation disagreements widespread or concentrated?

Our independent re-annotation analysis shows that disagreements concentrate around semantically adjacent categories. Of 68 observed disagreement instances, 48 (70.6%) occurred between Information Seeking and Public Event Discussion, while another 14 involved Information Seeking and Interest Exchange. This indicates that the main difficulty lies in distinguishing specific neighboring functions, rather than applying the taxonomy as a whole.

The same boundary challenges models. Under the full-codebook condition, providing all seven definitions, six boundary rules, and one example per category reduces the Information Seeking to Public Event Discussion confusion from 63% to 54%, but does not eliminate it. This suggests that explicit definitions alone are insufficient; interpreting communicative framing and interaction evidence remains challenging.

3. How reliable are the annotations?

The original corpus annotation retained one final label per post, rather than three independent judgments for every post. Therefore, full-corpus inter-annotator agreement cannot be reconstructed from the original records.

We independently re-annotated a stratified sample of 100 posts with three annotators under post-only and full post–reply–thread (P–R–T) conditions. The results are:

Reliability metricPost-onlyFull P–R–T
Fleiss' κ0.8660.849
Pairwise Cohen's κ range0.812–0.8990.812–0.900

Category-level one-vs-rest Cohen's κ, averaged across annotator pairs:

Social functionPost-only κFull P–R–T κ
Interest Exchange0.820.73
Information Seeking0.760.74
Transaction & Cooperation0.960.96
Public Event Discussion0.780.79
Content Creation & Diffusion1.000.98
Family Daily Life1.001.00
Workplace Collaboration1.001.00

These results indicate substantial overall agreement, although reliability varies across categories. Information Seeking and Public Event Discussion show relatively lower agreement, consistent with the disagreement pattern. High κ values for other categories should be interpreted cautiously given their limited representation in the 100-post sample. We therefore do not claim that all category boundaries are equally unambiguous.

4. How are ambiguous and excluded posts handled?

Quality control excluded 41 posts that could not be labeled (empty or irreducibly ambiguous) and 19 without interaction, leaving 6,003 of 6,063 candidate posts. These represent distinct exclusion reasons; we do not treat all 60 posts as ambiguous multi-function cases.

We acknowledge that single-label annotation cannot capture every secondary communicative purpose. Quantifying multiple simultaneous functions would require dedicated multi-label annotation, which is beyond the current design.

Response to Reviewer wSKv: (W2 + Q1)

We thank the reviewer for raising this important concern. We agree that social-function understanding is closely related to dialogue-act recognition, speech-act theory, and communicative-intent classification. Our contribution is not a new theory of communication, nor do we claim that these concepts are mutually exclusive. Rather, SceneBench operationalizes a different evaluation target: identifying the dominant social function of a post within community interaction.

1. Distinction from prior approaches.

Following the speech-act tradition (Austin; Searle) and established dialogue-act frameworks such as ISO 24617-2, we acknowledge that communicative acts can be context-dependent. The distinction is therefore not whether prior approaches use context, but what they classify and how the labels are defined.

DimensionDialogue ActsCommunicative IntentSceneBench
Primary unitUtterance/dialogue turnTask-defined interactionPost within community interaction
Prediction targetLocal communicative moveSpeaker's task-related goalDominant social function
Label sourceDialogue-act annotation schemesApplication-defined intent categoriesSociologically motivated taxonomy
EvidenceUtterance/dialogue contextTask/interaction contextPost framing and observed thread responses
Interaction structureOften dialogue-turn-basedApplication-dependentAsynchronous, community-embedded, extended reply structures

SceneBench's seven functions are organized into three sociologically motivated dimensions—relational, instrumental, and public—drawing on theories of social ties, uses and gratifications, and the public sphere. Unlike conventional local dialogue-turn settings, social media interactions are asynchronous, community-embedded, and unfold through extended reply structures. Accordingly, SceneBench evaluates the social role a post performs within this broader interaction.

Importantly, the gold label is determined from the post's communicative framing and observed interaction evidence, rather than equated with a speaker's private intention.

2. Why are these not simply renamed dialogue-act or intent labels?

A dialogue-act label describes a communicative move; a SceneBench label describes the post's dominant role in community interaction.

For example, a question may be classified as a Question under a dialogue-act scheme. In SceneBench, however, it may be Information Seeking when requesting a concrete answer, Interest Exchange when inviting personal experiences, or Public Event Discussion when soliciting positions on a public issue. These distinctions depend on communicative framing and interaction evidence, not simply linguistic form.

We acknowledge that some functions overlap with existing dialogue-act or intent categories. A post can simultaneously have a dialogue-act label and a social-function label; these variables may correlate without being identical. Our novelty lies in the interaction-level operationalization and benchmark design, not in claiming that individual communicative categories are exclusive to SceneBench.

3. Empirical evidence for the evaluation target.

SceneBench reveals a substantial gap between recognizing a post's topical community and identifying its social function:

ModelCommunity RecognitionSocial-Function RecognitionGap
Qwen3.6-Max-Preview90.1%52.0%38.1 pp
Qwen3.6-27B87.4%50.5%36.9 pp

A supplementary 20-option community-recognition probe achieves 73.5% accuracy, compared with 45.9% on seven-class social-function recognition. Across eight evaluated models, social-function accuracy ranges from 29.8% to 54.2%.

These findings demonstrate the benchmark's diagnostic value: strong community recognition does not necessarily translate into accurate social-function inference. We do not interpret these accuracy differences alone as proof of conceptual non-equivalence, since the tasks differ in label spaces and difficulty.

Furthermore, SceneBench evaluates whether inferred social functions affect downstream agent responses (Task 2), extending evaluation beyond label prediction to observable behavioral consequences. This measures behavioral sensitivity rather than response optimality.

Response to Reviewer wSKv: (W3+ Q3)

We thank the reviewer for questioning whether the Topic–Role Gap reflects difficulty in social-function understanding or merely differences in classification difficulty and label semantics. We agree that an accuracy gap alone establishes no mechanism. We therefore provide three complementary controls: a human comparison under matched task conditions, a larger candidate-set probe for community recognition, and a same-topic pair analysis.

1. Human control: Is the asymmetry model-specific?

Three annotators evaluated the same 500 posts using seven candidate labels per task and the same P–R–T evidence available to the models (1,500 judgments per task).

GroupCommunity RecognitionSocial-Function RecognitionGap
Humans (3 annotators)84.7%78.3%6.4 pp
Qwen3.6-Max-Preview90.1%52.0%38.1 pp
Qwen3.6-27B87.4%50.5%36.9 pp
DeepSeek-V4-Flash86.6%40.6%46.0 pp

The corresponding human gap was 6.4 percentage points, compared with 36.9–46.0 percentage points across the models. Moreover, all three models matched or exceeded human accuracy on community recognition while performing substantially worse on social-function recognition. This suggests that the asymmetry is substantially more pronounced in the evaluated models than in human evaluators.

2. Larger candidate-set probe: Is the gap caused by fewer candidate labels?

To test whether the gap can be explained by the smaller number of community-recognition options, we introduced a 20-option community-recognition probe, with candidates drawn from the same pool and shuffled for each post. We evaluated the same Qwen3.6-27B checkpoint in a matched re-run that also included the seven-option social-function task.

TaskNumber of OptionsAccuracyChance Level
Community Recognition2073.5%5.0%
Social-Function Recognition745.9%14.3%

Even after increasing the community candidate set to 20 options, community recognition remained substantially higher than social-function recognition (73.5% vs. 45.9%). Thus, candidate-set size alone does not explain the asymmetry.

The absolute scores in this control differ from those in the main evaluation because this control was conducted in a rebuilt inference environment. We therefore use the main run for the primary comparison and this re-run only for within-control comparisons. Within the control, the community and social-function conditions were evaluated using the same checkpoint, prompt templates, decoding settings, and serving configuration. The 20-option probe is not intended to make the two tasks perfectly equivalent; rather, it tests whether the original seven-option community task benefits solely from having fewer candidate labels.

3. Same-topic pairs: Can models distinguish social functions when topic is held constant?

We further analyzed 51 validated pairs of posts that shared the same gold topic but had different gold social-function labels.

Under both Full P–R–T and Post-only conditions, predictions assigned the same social-function label to both posts in 96.1% of pairs (Figure 4).

This provides direct evidence that distinguishing social functions remains difficult even when topical content is held constant. The result is consistent with models relying heavily on topical cues or frequently predicted categories, although it does not by itself identify which mechanism causes the errors.

4. Interpretation and limitations

Together, the three controls support the following conclusions:

  • Human–model asymmetry: Humans exhibit a substantially smaller Topic–Role Gap on matched evaluation items.
  • Candidate-set robustness: Increasing community-recognition options from seven to twenty does not eliminate the gap.
  • Topic-controlled evidence: Models frequently fail to distinguish different social functions among posts sharing the same topic.

These findings are also consistent across the broader evaluation: social-function accuracy ranges from 29.8% to 54.2% across eight models.

We acknowledge that the results do not establish a unique causal mechanism. Familiarity with Reddit community names, differences in label semantics, class-frequency biases, and other task-specific factors may contribute. Moreover, the 20-option probe does not make the two tasks intrinsically equivalent, and the same-topic analysis does not identify which cues drive the predictions.

We therefore characterize the Topic–Role Gap as a robust empirical performance asymmetry, rather than definitive evidence that models lack social reasoning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.