acceptodds
Under review as a conference paper at ICLR 2027

Knowledge, Mechanisms, and Context: Diagnosing and Improving Social UGC Understanding — @SITU

Abstract

Understanding social user-generated content (UGC) requires interpreting evolving expressions and integrating meaning across posts and interactions. Existing benchmarks largely evaluate downstream task labels or individual phenomena, leaving these abilities difficult to assess together. We introduce SITU (*Situated Interpretation of Textual UGC*), a bilingual benchmark built from Chinese and English social-media posts and discussions. It contains 89,199 open-ended interpretation probes from three platforms, including 9,522 difficulty-filtered probes on which frontier models remain well below perfect performance. Nine coverage-oriented probe targets span expression-centered and context-integrative challenges. To study improvement, we analyze item knowledge, reusable expression mechanisms, and contextual integration through retrieval, expression oracles, supervised fine-tuning (SFT), and training-data scaling. Retrieval helps most when it supplies answer-critical knowledge; SFT produces broader gains, and their combination performs best overall. A full-stream Chinese exposure audit finds persistent gains on mechanism-transferable expressions after excluding detected same-item knowledge. Matched downstream training further shows benefits for translation and homophone-robust toxicity detection, with substantial variation across tasks and backbones. These results establish SITU as a testbed for evaluating social UGC understanding and identifying which improvements can transfer beyond interpretation. We will release the benchmark and evaluation code.

Primary area:
foundation or frontier models, including LLMs
Keywords:
social media, user-generated content, LLM, language model evaluation, knowledge acquisition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qzGNOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a bilingual benchmark for evaluating understanding of social UGC. It uses 3 data sources, one in Chinese and two in English, and creates a regular set and a hard set to evaluate how existing models perform.

Critical strengths
  • The paper covers a very important problem and exposes limitations of even frontier models in dealing with contextual user-generated context. Having more benchmarks like this is very useful for evaluating LLM progress on this complex problem.
  • Analysis covers most scenarios such as hard cases, retrieval, fine-tuning on this data, unseen cases, etc. Comprehensive enough to be useful for multiple folks.
Critical weaknesses
  • A big part of this problem is the temporal aspect, and this is only mentioned in the related work. I would have liked an analysis of the dataset's time span, as it might explain the gap between some of the models -- very likely, the frontier models have some data leakage here.
  • Conflating Reddit and YouTube might have made the results hard to parse. Both platforms are very different, and instead of putting both under "English", it would have been better to analyze them separately.
Questions for authors
  • Will you release the dataset?
  • Could you add an analysis breakdown between youtube and reddit? Would those be separated when you release the dataset?
  • Could you provide some temporal analysis for the tasks? Even better, for the open source models, if we could check how they compare on data before they were trained and after.
1 comment
AuthorsOpenReview ↗

# Response to Reviewer qzGN

Manuscript section, figure, table, and page references below use the revised PDF.

We apologize for the delayed response. We sincerely thank you for your encouraging assessment and thoughtful suggestions, and particularly appreciate your recognition of the importance of social UGC understanding and the value of SITU's coverage and intervention analyses.

Q1: Will the dataset be released?

  • Yes. As mentioned in the abstract, we will release SITU and the evaluation code. The release will preserve Xiaohongshu-zh, Reddit-en, and YouTube-en as separate subsets rather than pooling Reddit and YouTube into one English subset.

W2/Q2: Could Reddit and YouTube be analyzed and released separately?

  • We agree that Reddit and YouTube differ substantially and that reporting them separately makes the results easier to interpret. The original tables combined them for space (Tables 2 and 3, pp. 5 and 7); the revised appendix reports each platform separately for both the twelve-model evaluation and the Foundation/RAG/SFT intervention analysis.
  • The split shows that the twelve-model rankings on Reddit and YouTube are highly consistent (Spearman ρ=0.972\rho=0.972), although absolute scores are higher on YouTube. The best SITU-Hard model scores are 1.60/2 on Reddit and 1.74/2 on YouTube, compared with 1.52/2 on Xiaohongshu; none of the three settings is saturated. The full release contains 31,158 Reddit probes, 26,845 YouTube probes, and 31,196 Xiaohongshu probes.
  • Revision: We added platform-specific versions of the main model and intervention tables to Appendix B.4.2 (pp. 30–31; Tables 15–16, p. 31) and will preserve the three platform/language subsets in the release.

W1/Q3: What time span does SITU cover, and can it support analysis around model-training cutoffs?

  • We agree that temporal coverage is important for understanding the benchmark and interpreting model performance. The statement that records were sampled in 2026 referred to the collection process, not the publication year of every retained record. We recovered source-record publication timestamps and added their distributions. Xiaohongshu spans June 2, 2023–May 13, 2026, with 2,094/2,200 records (95.18%) from April–May 2026. Reddit and YouTube have broader multi-year coverage:
PlatformPublication-date rangeMedianPublished in 2025–2026Largest single-month share
Reddit2010-02-11–2026-08-052024-09-1943.32%2.82%
YouTube2007-01-12–2026-08-082024-08-0443.37%3.14%
  • Each SITU reference is tied to the meaning and pragmatic function in its dated occurrence and retained context. Controlled cutoff-specific and diachronic evaluation remains future work.
  • Revision: We clarified the distinction between collection time and publication time in Section 3.2 (p. 3) and added the platform date ranges and temporal distributions to Appendix A.1 (p. 15).

Sincerely,
Authors of Submission 58854

Reviewer_CxkbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SITU, a bilingual benchmark for evaluating how large language models interpret expressions and conversational context in social user-generated content. Built from Chinese Xiaohongshu and English Reddit and YouTube records, SITU contains 89,199 open-ended interpretation probes across nine categories, including 9,522 difficulty-filtered probes. The authors evaluate twelve models using an LLM-based scoring framework supported by calibration and targeted human validation.

Beyond benchmark construction, the paper investigates the roles of item knowledge, reusable expression mechanisms, and contextual integration in social-media understanding. Through retrieval, expression-oracle interventions, SFT, and training-data scaling, the authors find that retrieval is particularly effective when relevant knowledge is available, while SFT produces broader improvements. Combining retrieval and SFT achieves the strongest overall performance. An additional exposure analysis examines whether improvements extend to expressions without detected same-item training exposure.

Finally, the authors evaluate whether interpretation-focused training benefits downstream applications under matched training conditions. Experiments on social-media translation and toxicity detection show improvements in several settings, particularly translation and homophone-robust toxicity detection, although the benefits vary across tasks and model backbones.

Critical strengths

The paper introduces a large-scale bilingual benchmark containing 89,199 open-ended interpretation probes across nine categories. By evaluating both expression-centered and context-integrative phenomena under a common task format, SITU provides a useful framework for comparing abilities that have largely been studied separately. Its scale and coverage make it a potentially valuable resource for future research on social-media language understanding.

Beyond benchmark construction, the paper examines how retrieval, expression-oracle interventions, SFT, and training-data scaling affect different interpretation challenges. The training-exposure analysis and matched downstream experiments further investigate whether these improvements extend beyond individual benchmark items. This broader experimental scope provides useful empirical evidence about the complementary effects of retrieval and supervision, as well as their potential benefits for downstream applications.

Critical weaknesses

The paper aims to measure social UGC understanding, but it remains unclear whether performance on open-ended interpretation probes reliably reflects the ability to understand expressions in their social and conversational contexts. SITU evaluates generated explanations against frozen reference answers, without systematically testing whether models distinguish different meanings of the same expression across communities, contexts, or time. This is particularly important because the meaning and pragmatic function of social-media expressions can change substantially depending on their use.

The reliability of the evaluation framework also raises concerns. Appendix B.3 reports only 59.5% exact agreement between the LLM judge and the author on 126 candidate answers, while the agreement for C-E/C-I diagnostic labels is relatively low. These diagnostic distinctions are central to the paper's analysis of knowledge, expression mechanisms, and contextual integration. Without stronger evidence that the evaluation reliably measures these intended abilities, the reported performance differences are difficult to interpret as improvements in social UGC understanding rather than improvements in generating reference-compatible explanations.

A central contribution of the paper is its investigation of whether SFT improves reusable expression mechanisms beyond memorized item knowledge. However, the evidence does not convincingly establish this distinction. In Section 5.4 and Appendix C.7, the E-M improvement remains positive after excluding detected same-item exposure, but decreases to +0.075 with a 95% confidence interval of [−0.117, +0.279] when component knowledge is also excluded. Moreover, the exposure audit examines the additional SFT data rather than the backbone's pretraining history. The paper does not evaluate systematically constructed novel expressions or controlled changes in temporal and community context, which would provide more direct evidence of generalization beyond previously encountered linguistic knowledge.

The external evaluations further question whether the improvements represent a broadly transferable understanding ability. Appendix D.8 reports performance decreases on Chumor 2.0, CHIME open-ended meaning explanation, and SNS-Bench after SUU training. These results suggest that improvements within SITU do not consistently translate into better performance on related understanding tasks. Consequently, the evidence does not sufficiently distinguish learning reusable interpretation mechanisms from adapting to the benchmark's explanation format or improving the use of previously acquired knowledge.

These concerns directly affect the paper's two main claims: that SITU provides a reliable measure of social UGC understanding and that its supervision improves generalizable interpretation abilities. Addressing them would require stronger evaluation validation and controlled generalization experiments, rather than only additional analysis of the existing results.

Questions for authors
  1. In Sections 3.2–3.3 and Appendix A.1, the authors state that the data were sampled in 2026. What are the actual publication-date ranges of the collected posts, and how does SITU account for changes in expression meanings over time?

  2. In Section 3.2 and Table 1, Chinese data come exclusively from Xiaohongshu, while English data come from Reddit and YouTube. How can the authors distinguish language-specific understanding from platform-specific and cultural differences under this design?

  3. In Section 3.2 and Appendix A.1, all source records undergo privacy-preserving rewriting. How was it verified that this process preserves the original expressions, pragmatic intentions, community-specific meanings, and ambiguities? Was this preservation independently validated by human annotators?

  4. In Sections 3–4, SITU measures understanding through open-ended explanations evaluated against frozen reference answers. What independent evidence establishes that this evaluation measures situated understanding rather than familiarity with particular expressions or the ability to generate reference-compatible explanations?

  5. In Appendix B.3, Table 10 reports 59.5% exact agreement between the LLM judge and the author, with QWK = 0.642. How robust are the main evaluation results and model comparisons to human scoring or alternative judges, particularly for context-dependent interpretations with multiple plausible answers?

  6. In Appendix B.3, 10 of the 76 author-reviewed reference packages required core repairs, and three were unanswerable. Given that the reviewed sample was enriched for uncertain cases, can the authors estimate the remaining reference-error rate using an independently sampled, representative subset of the final benchmark?

  7. In Section 5.1 and Appendix B.3, the agreement for C-E/C-I labels is k = 0.444, while the mechanism-transferability ratings achieve QWK = 0.603. How sensitive are the diagnostic findings and intervention comparisons to uncertainty in these labels?

  8. In Section 4 and Appendix B.4, the Hard subset is selected using Gemma-4-31B-it, which also serves as the main evaluation judge. How would the model rankings and intervention results change under an independently constructed difficulty subset, particularly given the performance regressions on Chinese Easy probes after SFT?

  9. In Section 5.4 and Appendix C.7, the E-M improvement decreases from 0.204 to 0.075 after excluding both same-item and component knowledge exposure. What evidence supports the acquisition of reusable interpretation mechanisms rather than the improved use of previously acquired lexical or compositional knowledge?

  10. In Section 5.4, generalization is evaluated using naturally occurring expressions absent from the inspected training stream. Could the authors test systematically constructed novel expressions, including previously unseen word formations and phonetic variations, to distinguish reusable mechanism learning from memorization and prior knowledge?

  11. In Section 3.2 and Section 5.4, how well does SITU capture expressions whose meanings change across communities or social contexts? Could the authors evaluate controlled examples in which the same expression has different meanings or pragmatic functions, rather than relying primarily on distinct expressions with fixed reference interpretations?

  12. In Section 6.1, the SUU-Task models receive an additional interpretation-training stage compared with Foundation-Task. How much of the reported translation improvement is attributable specifically to SUU supervision rather than additional training, and would the advantage persist against a compute-matched baseline trained on general instruction-following data?

  13. In Section 6.2 and Appendix D.5–D.6, SeTox evaluates harmful neologisms, but the CounterContext experiments do not establish a reliable improvement when conversational context is provided. What evidence shows that SUU training improves the ability to distinguish derogatory, neutral, ironic, quoted, or counterspeech uses of the same expression based on its context?

  14. In Appendix D.8, SUU training reduces performance on Chumor 2.0 by 9.82 percentage points, CHIME open-ended meaning explanation by 1.71 points, and SNS-Bench by 2.05 points. How do the authors reconcile these regressions with the improvements observed on SITU, and what evidence distinguishes generalizable understanding improvements from adaptation to SITU's specific interpretation format?

  15. Also, I discovered that the reference in the SMILE paper was not written correctly, so please fix them.

7 comments
AuthorsOpenReview ↗

# Response to Reviewer Cxkb (1/6)

Manuscript section, figure, table, and page references below use the revised PDF.

We apologize for the delayed response. Thank you for the careful review. We have revised the paper and added evidence addressing the four Critical Weaknesses:

  1. Situated construct validity (Q4/Q11): A same-surface/different-sense test covers 2,154 pairs and a 448-pair diagnostic subset. Restricting context lowers Foundation accuracy by 9.2 points on that subset.
  2. Evaluation reliability (Q5–Q8): Independent DeepSeek rejudging preserves the main model ranking (Spearman ρ = 0.993); Qwen rejudging preserves positive intervention gains. We also test sensitivity to diagnostic labels and difficulty selection.
  3. Generalization beyond memorized items (Q9/Q10): We retain the uncertainty of the strict 157-probe exposure subset and add 554 independently constructed, SFT-unseen phonetic variants, with a two-seed exact-recovery gain of +6.1 [3.5, 8.9] points.
  4. Transfer beyond SITU's explanation format (Q12–Q14): Phun-Bench [1] homophonic-idiom recovery improves by +8.42 [6.24, 10.66] points; we also report the mixed results on its other task and the remaining external benchmarks.

Two additional human-review studies are underway: an audit of final reference answers and a paired audit of privacy-rewrite preservation. We will complete both within two days and report the results in a follow-up response.

Situated, context-conditioned understanding

First Critical Weakness; Q4 and Q11: Does SITU measure situated interpretation rather than reference-compatible explanation, including meaning changes across contexts, communities, and time?

  • As specified in the submitted manuscript (Section 3.2, p. 3), each SITU question and reference is anchored to one expression occurrence or one complete record, including the bounded preceding context needed to interpret that occurrence. The intended answer is the meaning, referent, or pragmatic function in that record, instead of a context-free dictionary definition.
  • We agree that testing different meanings of the same expression provides a more direct check of context-sensitive understanding. We therefore added a fixed-choice evaluation from independently collected expression data. Each pair holds the surface string fixed while two natural occurrences support different established senses. The model selects the occurrence-specific sense from two fixed descriptions. We compare the complete occurrence context with a local window around the target surface.

Each pair therefore contains two occurrence-level decisions. Per-occurrence accuracy scores them separately, while pair accuracy requires both to be correct. Full-context gain is the difference between per-occurrence accuracy with the complete context and with the local window.

Evaluation setModelPer-occurrence accuracy, full contextPair accuracy, both contexts correctPer-occurrence accuracy, local windowFull-context gain
All 2,154 pairsFoundation92.5%85.4%90.6%+1.9 [1.2, 2.6]
All 2,154 pairsSFT seeds A/B92.8% / 92.7%86.1% / 85.9%91.0% / 91.1%+1.9 / +1.6
448-pair diagnostic subsetFoundation64.2%29.7%55.0%+9.2 [6.0, 12.3]
448-pair diagnostic subsetSFT seeds A/B69.5% / 69.6%41.1% / 41.7%62.6% / 63.5%+6.9 / +6.1
  • The 448-pair diagnostic subset was selected using Foundation behavior before either SFT seed was inspected: it retains pairs where Foundation misses at least one occurrence under full context or changes its choice when the input is restricted to the local window. On this subset, the two-seed SFT gain over Foundation is +5.4 [3.3, 7.5] points. On the complete set, the gain is +0.2 [−0.3, 0.7], reflecting its near-ceiling baseline.
  • This test supplies two forms of construct evidence. The same surface requires different answers in two occurrence contexts, so surface familiarity alone is insufficient. The Foundation full-minus-local gap further shows that context beyond the immediate local window affects the selected interpretation.
  • Revision: We added the full and diagnostic-subset results to Section 5.4 (pp. 8–9) and the complete construction, ablation, and scoring protocol to Appendix C.4.2 (pp. 38–39; Table 23, p. 39). Part 6 addresses temporal coverage (Q1) and the platform and community scope of the comparison (Q2).

References

[1] Yue, Shen, and Lu. Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese. arXiv:2606.07300, 2026.

AuthorsOpenReview ↗

# Response to Reviewer Cxkb (2/6)

Response to Reviewer Cxkb (2/6)

Evaluation, reference, label, and difficulty reliability

Second Critical Weakness; Q5–Q8: Are candidate scores, references, diagnostic labels, and difficulty selection reliable enough to support the reported model and intervention comparisons?

  • The submitted manuscript already separated reference construction from candidate scoring (Section 4, pp. 4–5; Appendix A.1, pp. 15–16): a Gemini-based complete auditor constructs and freezes each reference answer and supporting explanation before Gemma scores anonymized candidate answers. It also reported two validation checks (now Appendices B.2.1 and B.3, pp. 26–28). On 500 frozen answers from five actor models, Gemma has 82.2% exact agreement with the Gemini-3.1-Pro strong-model calibration anchor, with QWK 0.837 [0.785, 0.882], mean bias −0.026, and eight 0↔2 reversals.
  • The second submitted check compared Gemma with blinded author review on a deliberately difficult, model-triaged sample concentrated on disagreements, low-confidence cases, and reference flags. Across 126 answers attached to usable references, exact agreement is 59.5% and QWK is 0.642; all 51 disagreements are adjacent on the 0–2 scale, with no 0↔2 reversal (Table 12, p. 28).
  • Main-evaluation scores: We re-scored all 114,264 answers from the 12 models (9,522 per model) with DeepSeek-V4.1-Flash-1017 (High thinking), using the same references, judge prompts, and 0–2 rubric, with model identities hidden. The table below compares mean scores on the 0–2 scale, ordered by DeepSeek-V4.1 Flash score. We have added this comparison to Appendix B.2.2, Table 10 (p. 27).
Evaluated modelGemma-4-31B-it judgeDeepSeek-V4.1 Flash judgeRank (Gemma → DeepSeek)
Claude Opus 51.5781.5971 → 1
Gemini-3.8-Flash1.5571.5842 → 2
GPT-5.6-Sol1.5121.5723 → 3
Kimi-K31.4981.5494 → 4
DeepSeek-V4-Pro1.4651.5095 → 5
GLM-5.21.3811.4146 → 6
Qwen3.5-397B-A17B1.2631.2737 → 7
Qwen3.5-27B1.2311.2218 → 8
Gemma-4-31B-it1.1081.13310 → 9
Gemma-4-E4B-it1.1421.0709 → 10
Qwen3.5-9B0.9841.01211 → 11
Qwen3.5-4B0.8610.89312 → 12

Overall model-rank correlation is Spearman ρ = 0.993; Claude Opus 5 remains first, and only the two Gemma models exchange ninth and tenth place. Chinese and English rank correlations are 0.958 and 0.993. Across individual answers, the judges agree exactly on 78.3%, with QWK 0.784. On the original 500-answer calibration set, DeepSeek-V4.1 Flash achieves 81.8% agreement and QWK 0.824 against the Gemini-3.1-Pro anchor.

  • Intervention scores: We re-evaluated a frozen 21,414-probe slice outside SITU-Hard, defined by probes on which Qwen3.5-9B or Qwen3.5-27B received less than full credit under the original Gemma evaluation. Qwen3.6-27B independently rejudged every Foundation, SFT, and Two-round RAG answer under the unchanged 0–2 rubric. Gemma and Qwen agree exactly on 94.3%/94.6% of Foundation/SFT answers, with QWK 0.853/0.780. Under the independent judge:
Slice selected by Qwen actor resultsNSFT − FoundationTwo-round RAG − Foundation
Either Qwen actor difficult21,414+0.062 [0.053, 0.071]+0.068 [0.060, 0.076]
Both Qwen actors difficult4,992+0.116 [0.091, 0.141]+0.122 [0.100, 0.144]
  • Both intervention gains remain positive under a different judge family. The same-surface, constructed-phonetic, and Phun-Bench evaluations provide complementary checks using exact outcomes.
  • Revision: Appendix B.2.2 reports DeepSeek-V4.1 Flash rejudging of the main evaluation (Table 10, p. 27) and Qwen rejudging of the interventions (Table 11, p. 28), together with the original calibration (Appendix B.2.1, pp. 26–27) and blinded author comparison (Appendix B.3, pp. 27–28).

Q6: What is the remaining final-reference error rate?

  • We agree that an independent sample is needed alongside the previously reviewed 76 risk-enriched packages. We froze 300 final reference packages, stratified by platform and Hard/Easy difficulty, for author review of the record, question, reference answer, and explanation. The result is:
Human judgmentCountSample percentage
Valid27090.0%
Minor repair113.7%
Core repair186.0%
Unanswerable10.3%
  • Core-repair and unanswerable cases total 19/300 (6.3%). After accounting for the sampling weights, their estimated prevalence in SITU (full) is 2.4%. Confirmed repair cases will receive source-grounded repair and renewed complete audit, with difficulty re-evaluated for semantically changed probes.
AuthorsOpenReview ↗

# Response to Reviewer Cxkb (3/6)

Diagnostic-label and difficulty-selection sensitivity

Q7: How sensitive are the diagnostic findings to uncertainty in the C-E/C-I and E-M/E-K annotations?

  • We agree that uncertainty in these labels needs to be tested against the diagnostic conclusions. The diagnostic labels stratify already frozen model scores; they do not determine overall scores, rankings, or aggregate intervention gains.
  • For C-E/C-I, we build on the 43 valid author-reviewed labels reported in the submitted appendix (now Appendix B.3, Table 12, p. 28). The contrast is defined as the intervention gain on C-E minus its gain on C-I. Replacing the production labels on these cases with the blinded author's labels changes this contrast only from .197 to .199 for Two-round RAG, .209 to .211 for Agentic RAG, and .257 to .260 for the expression oracle. We also simulated corpus-wide relabeling using the observed production-to-author label transitions. Across 5,000 draws, the contrasts remain positive: .097 [.030, .153], .103 [.033, .159], and .126 [.045, .194], respectively. SFT remains beneficial in both C-E (+.223 [.199, .246]) and C-I (+.231 [.218, .244]).
  • For E-M/E-K, the SFT-versus-Agentic-RAG interaction remains positive for mechanism-transferability cutoffs 2–5 (+.153 to +.388, with all 95% intervals above zero) and when the original continuous scores are used directly (+.151 [.104, .197]). Only the very permissive cutoff at score 1 collapses the distinction. Label uncertainty therefore attenuates the diagnostic contrasts without changing the reported intervention pattern.

Q8: How does Gemma-conditioned Hard subset (SITU-Hard) selection affect rankings and intervention results?

  • We now describe SITU-Hard explicitly as a Gemma-conditioned stress subset and report SITU (full) alongside it (Appendix B.4.1, pp. 28–30). Across five Foundation models, the SITU-Hard-versus-full rank correlation is ρ=.700\rho=.700; Gemma moves from third on SITU-Hard to first on SITU (full), confirming that the stress-set ranking depends partly on its selector.
  • The Qwen-selected slice in Part 2 evaluates interventions outside the Gemma-conditioned subset. Under independent Qwen rejudging, both SFT and RAG retain positive gains, including on the stricter intersection where both Qwen actors receive less than full credit.
  • The submitted manuscript already reported (now Appendix B.4.1, Table 14, p. 29) that SFT changes the Chinese Easy and Hard splits by −0.0542 and +0.3420, respectively. We now add the corresponding SITU (full) change of +0.0008. The improvement is concentrated on difficult probes, with a small loss on Easy probes that Foundation already solved.
Reviewer_gYeFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a bilingual Chinese and English benchmark, SITU (Situated Interpretation of Textual User-generated content (UGC)), which contains 89k open-ended interpretation probes built from social media posts. The probes have two types: (1) expression-centered probes that cover new surface, new meaning, word variation, and wordplay and (2) context-integrative probes which contain Q/A on the topic, content, pragmatics, reference, and integration. In their diagnostic analysis, they examine whether 12 models' ability on understanding social content according to their target types. Six judges (Claude, GPT, Gemini, Kimi, Deepseek, GLM, Qwen) are used to evaluate models' answers. Then they also examine whether they can transfer SUU (Social UGC Understanding) to downstream tasks such as machine translation problems on social media dataset and toxicity detection. Findings show that finetuning models on their dataset help improve performance while the transferability of this task to downstream tasks vary.

Critical strengths

A new dataset for the task of social user-content generated understanding.

Critical weaknesses
  • The probe generation process and categorization is not clear. How did you generate the probes? Are they human-written or generated by LLMs? The decision of splitting probes into nine targets is not clear and well-grounded on prior study or necessity on real practice. As the paper lacks concrete examples of the probes and how to operationalize this in their experiment setting, it becomes unclear about what they really test the models.
  • The paper uses the same model family to be LLM as judge (only explained in the Appendix) while running the experiments. This makes the validity of the experiment result questionable.
  • There are too many abbreviations within abbreviations that reader may forget such as SUU, SITU, UGC, E-K, E-C and so on and this makes the paper hard to read.
Questions for authors
  • The Mandarin text in Figure 2 bottom left consists of Q and A, but the English translation is just one sentence, making the setting of context-integrative probe confusing. Is it a sentence probe or a question-answering pair with context?
  • How good are the LLMs as judge in line 200-208?
  • What is the motivation of section 5.1? I don't understand the need of intervention-oriented diagnostic aspects. The paper will be more clear if you add concrete examples of E-M, E-K, C-E, and C-I. In the same section, you mention solver and annotator? Are they all LLM-based solvers and annotators?
  • In section 5.2, is your goal to compare whether retrieval and SFT for improving the model for this SUU task?
  • What is Foundation in Section 6?
9 comments
AuthorsOpenReview ↗

# Response to Reviewer gYeF (1/4)

Manuscript section, figure, table, and page references below use the revised PDF.

We apologize for the delayed response. Thank you for recognizing the value of a benchmark for social user-generated content (UGC) understanding.

We recognize that the original presentation did not make the probe-construction pipeline and category grounding (W1), judge design and validation (W2/Q2), abbreviation system (W3), task interface and translation (Q1), or diagnostic terminology and intervention paths (Q3–Q5) sufficiently clear. We have revised these parts to connect the construction process, concrete examples, model inputs, scoring contract, and diagnostic analyses more directly, and we have added an independent-family judge sensitivity analysis to address the evaluation-validity concern.

Probe construction and task presentation

W1a: How did you generate the probes? Are they human-written or generated by LLMs?

  • The submitted manuscript already described the multi-stage blind extraction, complete audit, adjudication, and difficulty-filtering pipeline (Appendix A.1, pp. 15–16). We now make its provenance explicit: the probes are QA-style test cases generated through an LLM-based pipeline with targeted human review, rather than manually written from scratch. A blind LLM extractor proposes candidate probes from each rewritten record. A Gemini-based complete auditor then checks coverage, answerability, source support, reference answers, and supporting explanations, using web search when external verification is needed. Flagged cases undergo further model adjudication or targeted human review, and every semantic repair is re-audited. Finally, five closed-book Gemma attempts determine the empirical difficulty label.

  • Revision: We now state this provenance explicitly in Section 3.2 (p. 3) and Appendix A.1 (pp. 15–16), and have refined the construction flowchart (Figure 6, p. 15) to distinguish the blind LLM extractor, complete auditor, and targeted human review.

  • Examples: The submitted appendix already contained 54 bilingual examples—six per category—in Appendix A.4 (Table 8, pp. 19–25), in addition to the four illustrations in Figure 2 (p. 4). We agree that the original signposting did not connect these examples to the executable task interface clearly enough. We have therefore revised the formatting and translations and strengthened the cross-references among Figure 2 (p. 4), Appendix A.4 (pp. 19–25), and the inference and judging prompts in Appendix E (Figures 9–11, pp. 46–48).

  • How this is operationalized in evaluation: The inference and judging prompts already provided in the submitted manuscript (Appendix E, pp. 46–48) implement this interface. Each category guides the extraction of an atomic question from a benchmark record. The evaluated model receives the rewritten record, its bounded and ordered post/comment/reply context, and the question, and produces an open-ended answer. The judge compares that answer with a frozen reference answer, supporting evidence, and a target-specific scoring contract on the 0–2 scale. We then report both aggregate and category-stratified results.

W3: Too many abbreviations

  • We agree that the density of abbreviations made the paper harder to follow. UGC denotes user-generated content; SUU denotes social UGC understanding, the ability studied in this paper; SITU denotes our benchmark, Situated Interpretation of Textual UGC; and E-M and E-K denote the Mechanism-Transferable Expression Aspect and Item-Knowledge-Dependent Expression Aspect introduced in Section 5.1 (p. 6). The four diagnostic aspects and their examples are presented together under Q3 in Part 4. We now distinguish these aspects explicitly from the nine coverage-oriented probe categories.
  • Revision: We added abbreviation notes to tables involving the nine probe categories (e.g., Table 2, p. 5) or the four diagnostic aspects (e.g., Table 3, p. 7). We also renamed several probe categories and revised the surrounding text for clarity.

Q1: Confusing English translation in the example in Figure 2. Is it a sentence probe or a question-answering pair with context?

  • You are right that the presentation in Figure 2 (p. 4) was inconsistent: the Mandarin example retained a Q/A structure, whereas the English line compressed the question and answer into one sentence. We revised the figure and all 54 examples in Appendix A.4 (Table 8, pp. 19–25) so that the English presentation follows the same structure as the source example.
  • It is a context-integrative probe in the form of question answering over a record and its relevant context, not a sentence-only probe. The tested model receives the post or video text, the bounded and ordered comment context, and one atomic question. The frozen reference answer and supporting evidence are provided only to the judge.
AuthorsOpenReview ↗

# Response to Reviewer gYeF (2/4)

Probe-category grounding

W1b: Why use nine probe targets (categories), and how are they grounded and operationalized?

  • We organize social UGC understanding into two scopes: expression-centered probes, which recover the contextual form or meaning of a localized expression, and context-integrative probes, which integrate semantic, referential, pragmatic, and interactional evidence across a post or thread. This distinction is grounded in work on social-media expressions and variation [1, 2, 3] and on social-media discourse, pragmatics, reference, and discourse relations [7, 8, 9]. We operationalize these scopes through nine coverage-oriented categories designed to capture recurring interpretation demands with minimal systematic omission.
    • The four expression-centered probe categories each extend an established line of NLP work. New Term covers neologisms and emerging expressions in social media [1]. Contextual Meaning covers slang interpretation and disambiguation in its occurrence context [2]. Form Variation (previously Word Variation) covers lexical normalization and phonetic or orthographic variation [3, 4]. Wordplay covers pun explanation and culturally specific humor [5, 6]. New, shifted, and varied expressions are prominent in social UGC. The submitted manuscript already situated SITU against benchmarks that study these phenomena separately (Table 7, p. 18); the revision makes the category-level grounding explicit (Appendix A.3, p. 17). SITU aggregates and generalizes these lines into four coverage targets for the major localized-expression demands in our scope.
    • The context-integrative probe categories connect established NLP tasks to social UGC interpretation. Domain and Topic covers topic, relevance, and taxonomy classification, while Semantic Content covers semantic similarity, question answering, and summarization [10, 11]. Attitude and Pragmatics brings together sentiment, stance, irony, intent, and implicature [8, 10]. Reference or Discourse Link covers reference, deixis, and discourse-relation resolution [7, 9]. Integrated Reading retains a defining structure of social UGC that these component-level tasks often abstract away: information is distributed across a reply tree, so the answer may require combining a post, a comment, and one or more replies while preserving who responds to whom [7].

The nine categories serve concrete roles in SITU: they give the extractor fixed coverage slots, let the auditor check omissions and validity by category, specify what each reference package must preserve, and support category-stratified analysis. This use of an explicit inventory follows coverage-oriented behavioral testing [12]. The categories may overlap when one record supports multiple answer-critical phenomena; they are an operational coverage scheme, instead of a proposed canonical nine-part ontology.

References

[1] Rossini and van der Plas. From 124 Million Tokens to 1,021 Neologisms: A Large-Scale Pipeline for Automatic Neologism Detection. Workshop on Neology and Large Language Models at LREC, 2026.

[2] Sun, Zemel, and Xu. Semantically Informed Slang Interpretation. NAACL, 2022.

[3] van der Goot et al. MultiLexNorm: A Shared Task on Multilingual Lexical Normalization. W-NUT, 2021.

[4] Ma et al. Reasoning or Memorization? Investigating LLMs' Capability in Restoring Chinese Internet Homophones. Knowledgeable Foundation Models at ACL, 2025.

[5] Sun et al. ExPUNations: Augmenting Puns with Keywords and Explanations. EMNLP, 2022.

[6] He et al. Chumor 2.0: Towards Benchmarking Chinese Humor Understanding. 2024.

[7] Benamara, Inkpen, and Taboada. Introduction to the Special Issue on Language in Social Media: Exploiting Discourse and Other Contextual Information. Computational Linguistics, 2018.

[8] Sravanthi et al. PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities. Findings of ACL, 2024.

[9] Braud et al. DISRPT: A Multilingual, Multi-domain, Cross-framework Benchmark for Discourse Processing. LREC-COLING, 2024.

[10] Antypas et al. SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research. Findings of EMNLP, 2023.

[11] Guo et al. SNS-Bench: Defining, Building, and Assessing Capabilities of Large Language Models in Social Networking Services. ICML, 2025.

[12] Ribeiro et al. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. ACL, 2020.

AuthorsOpenReview ↗

# Response to Reviewer gYeF (3/4)

Judge quality and family dependence

W2: Does using a Gemma-family judge compromise evaluation validity?

  • We agree that judge-family overlap warrants an independent check. The overlap has different implications in the main benchmark evaluation and the Section 5 intervention experiments. In Table 2 (p. 5), the Gemma-4-31B-it judge scores two Gemma actors and ten actors from other model families. The same Gemma-4-31B-it checkpoint also defines SITU-Hard by selecting probes it does not solve reliably, so this selection acts against its own actor row; we mark that row as selector-conditioned and also report SITU (full). Gemma-4-E4B-it shares the judge family but was not used as the difficulty filter. The submitted manuscript already reported calibration on frozen outputs from five actor models (now Appendix B.2.1, pp. 26–27) and a separate comparison with blinded author judgments (Appendix B.3, pp. 27–28); we summarize those results under Q2.
  • Main evaluation: We re-scored all twelve models' answers with DeepSeek-V4.1-Flash-1017 (High thinking), using the same references and judge prompts: 9,522 answers per model, or 114,264 in total. The table below compares mean scores on the 0–2 scale, ordered by DeepSeek-V4.1 Flash score. We have added this comparison to Appendix B.2.2, Table 10 (p. 27).
Evaluated modelGemma-4-31B-it judgeDeepSeek-V4.1 Flash judgeRank (Gemma → DeepSeek)
Claude Opus 51.5781.5971 → 1
Gemini-3.8-Flash1.5571.5842 → 2
GPT-5.6-Sol1.5121.5723 → 3
Kimi-K31.4981.5494 → 4
DeepSeek-V4-Pro1.4651.5095 → 5
GLM-5.21.3811.4146 → 6
Qwen3.5-397B-A17B1.2631.2737 → 7
Qwen3.5-27B1.2311.2218 → 8
Gemma-4-31B-it1.1081.13310 → 9
Gemma-4-E4B-it1.1421.0709 → 10
Qwen3.5-9B0.9841.01211 → 11
Qwen3.5-4B0.8610.89312 → 12

Model-rank correlation between the two judges is Spearman ρ = 0.993. Claude Opus 5 remains first; only the two Gemma models exchange ninth and tenth place. The similar scores and rankings support the stability of the overall model comparison across judge families.

  • Section 5 interventions: Reference construction is separated from candidate scoring: a Gemini-based auditor constructs and freezes the reference answer and supporting explanation before Gemma scores anonymized candidate answers. We added an independent Qwen3.6-27B re-evaluation of the Section 5 comparisons (Appendix B.2.2, Table 11, p. 28). We froze the union of probes from the Easy portion of SITU on which Qwen3.5-9B or Qwen3.5-27B originally received less than full credit under the original Gemma judge, and then rejudged all Foundation, SFT, and Two-round RAG answers with Qwen3.6-27B under the unchanged 0–2 rubric.
SliceNSFT − FoundationTwo-round RAG − Foundation
Either Qwen actor difficult21,414+0.062 [+0.053, +0.071]+0.068 [+0.060, +0.076]
Both Qwen actors difficult4,992+0.116 [+0.091, +0.141]+0.122 [+0.100, +0.144]
  • Brackets give source-record-clustered bootstrap 95% confidence intervals. Both intervention gains remain positive under a different judge family, including on the stricter slice where both Qwen actors receive less than full credit.
  • Revision: Appendix B.2.2 reports DeepSeek-V4.1 Flash rejudging of the main evaluation (Table 10, p. 27) and Qwen rejudging of the interventions (Table 11, p. 28), alongside the original judge calibration (Appendix B.2.1, pp. 26–27) and blinded author comparison (Appendix B.3, pp. 27–28).

Q2: How reliable is the LLM judge used in lines 200–208?

  • As reported in the submitted manuscript (now Appendix B.2.1, pp. 26–27; Table 9, p. 27), we selected the operational judge through a frozen calibration rather than assuming its quality. Gemini-3.1-Pro served as a strong-model calibration anchor, not as human gold. On 500 frozen candidate answers, Gemma-4-31B-it performed best among six candidate judges, with 82.2% exact agreement, QWK 0.837 [0.785, 0.882], and mean bias -0.026.
  • The submitted appendix also compared the frozen Gemma judgments with blinded author review on a deliberately difficult, model-triaged sample concentrated on disagreements, low-confidence cases, and reference flags. Across 126 usable comparisons, exact agreement was 59.5% and QWK was 0.642; all 51 disagreements were adjacent on the 0–2 scale, with no severe 0↔2 reversal (Appendix B.3, Table 12, p. 28).
  • On the new 21,414-probe analysis (Appendix B.2.2, pp. 26–28), Gemma and the independent Qwen judge agree exactly on 94.3%/94.6% of Foundation/SFT answers, with QWK 0.853/0.780. Together, the strong-model calibration, blinded author comparison, and independent-family re-evaluation support the aggregate Section 5 intervention comparisons.

Comments on the paper

2 comments
Area_Chair_ghe7OpenReview ↗

Reminder: Discussion deadline approaching

Dear Reviewer,

Thank you for your time and effort in reviewing Submission 58854. As the discussion deadline is approaching, could you please read the authors’ response and any additional evidence, reply in the discussion to indicate whether your concerns have been addressed, and update your evaluation accordingly as soon as possible? Thank you for your contribution to the review process.

Best regards, Area Chair

AuthorsOpenReview ↗

# General Response to the Reviewers and Area Chair

Hello!

We sincerely thank the reviewers and the Area Chair for their constructive feedback, which helped us strengthen the validation and presentation of our work.

We summarize our main clarifications, additional evidence, and manuscript revisions below, with detailed results and protocols in our reviewer-specific responses.

Our work introduces SITU\texttt{SITU}, a bilingual benchmark for social user-generated content (UGC) that brings expression-centered and context-integrative understanding into a common evaluation setting, and investigates how retrieval and supervised fine-tuning improve performance across different interpretation requirements. It also explores whether training on social UGC understanding transfers to downstream translation and moderation through matched experiments, identifying both benefits and task-dependent boundaries.

1. Clearer categories and diagnostic motivation (gYeF, W1/Q3–Q4 and follow-up)

We now illustrate all nine categories with context, questions, and answers in the main text and strengthen their grounding in prior NLP work. In Section 5.1, we distinguish what questions ask (categories) from the requirements for answering them (diagnostic aspects), define “bottleneck,” and clarify that annotators assign labels independently of evaluation outcomes. We thus make the analysis’s purpose explicit: to understand which requirements benefit from different interventions.

2. Benchmark construction and evaluation reliability (gYeF; Cxkb)

During rebuttal, we added a blinded author audit of a stratified random sample to directly assess reference quality: 270/300 packages were valid, 11 needed minor repair, and 19 needed core repair or were unanswerable (gYeF, follow-up; Cxkb, Q6). This complements targeted review with independently sampled evidence. We also supplied construction-stage coverage and audit counts.

We added a 100-record privacy audit and found that 131/148 original difficulty anchors retained their meaning or mechanism; 15/210 final Hard probes targeted newly introduced content (Cxkb, Q3). These findings show that rewriting largely preserves the original challenges while sometimes introducing new targets.

We added independent DeepSeek-V4.1 Flash and Qwen re-evaluations: we find nearly unchanged rankings across 114,264 DeepSeek-scored answers (Spearman ρ = 0.993) and positive SFT and retrieval gains under Qwen judging on 21,414 probes outside the Gemma-conditioned Hard subset (gYeF, W2/Q2; Cxkb, Q5/Q8). We report these checks in Appendix B.2.2, supporting stability across judge families and beyond the original difficulty subset.

3. Generalization and context-sensitive understanding (Cxkb)

We added a controlled test of 554 constructed phonetic variants absent from the SFT stream. We find a 6.1-percentage-point gain [3.5, 8.9] in exact recovery across two seeds, supporting generalization to unseen forms beyond detected training-item exposure (Cxkb, Q9/Q10).

For context sensitivity, we added a test of 2,154 same-expression/different-sense pairs. On the 448-pair diagnostic subset defined by the original model’s behavior, we find that restricting context lowers its accuracy by 9.2 points, demonstrating that occurrence context contributes to sense selection (Cxkb, Q4/Q11). We report these tests in Section 5.4 and Appendix C, including the near-ceiling full-set results.

4. Transfer beyond SITU’s explanation format (Cxkb, Q12–Q14)

In the submitted paper, we already demonstrated selective translation and moderation benefits under matched task training. We now add a Phun-Bench evaluation and find an 8.42-point gain [6.24, 10.66] in exact homophonic-idiom recovery, showing that improvements extend beyond SITU-style explanations. We retain the mixed external results in Section 6 and Appendix D to clarify where transfer helps.

5. Platform coverage, temporal scope, and release (Cxkb; qzGN)

We added separate Reddit/YouTube results and find highly consistent rankings (ρ = 0.972) despite different absolute difficulty, supporting comparisons within each setting (Cxkb, Q2; qzGN, Q2). We added publication-date distributions and clarified that controlled temporal evaluation remains future work (Cxkb, Q1; qzGN, Q3). We will release SITU and evaluation code with separate platform subsets (qzGN, Q1).

We mark first-round revisions in blue and second-round revisions addressing Reviewer gYeF’s follow-up comments in dark green. Thank you again for helping us improve the paper.

Sincerely,
Authors of Submission 58854

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.