- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes a bilingual benchmark for evaluating understanding of social UGC. It uses 3 data sources, one in Chinese and two in English, and creates a regular set and a hard set to evaluate how existing models perform.
Critical strengths
- The paper covers a very important problem and exposes limitations of even frontier models in dealing with contextual user-generated context. Having more benchmarks like this is very useful for evaluating LLM progress on this complex problem.
- Analysis covers most scenarios such as hard cases, retrieval, fine-tuning on this data, unseen cases, etc. Comprehensive enough to be useful for multiple folks.
Critical weaknesses
- A big part of this problem is the temporal aspect, and this is only mentioned in the related work. I would have liked an analysis of the dataset's time span, as it might explain the gap between some of the models -- very likely, the frontier models have some data leakage here.
- Conflating Reddit and YouTube might have made the results hard to parse. Both platforms are very different, and instead of putting both under "English", it would have been better to analyze them separately.
Questions for authors
- Will you release the dataset?
- Could you add an analysis breakdown between youtube and reddit? Would those be separated when you release the dataset?
- Could you provide some temporal analysis for the tasks? Even better, for the open source models, if we could check how they compare on data before they were trained and after.
# Response to Reviewer qzGN
Manuscript section, figure, table, and page references below use the revised PDF.
We apologize for the delayed response. We sincerely thank you for your encouraging assessment and thoughtful suggestions, and particularly appreciate your recognition of the importance of social UGC understanding and the value of SITU's coverage and intervention analyses.
Q1: Will the dataset be released?
W2/Q2: Could Reddit and YouTube be analyzed and released separately?
W1/Q3: What time span does SITU cover, and can it support analysis around model-training cutoffs?
Sincerely,
Authors of Submission 58854