- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces VeriBench-Wild, a benchmark of 27,158 social-media records (multilingual but English-dominant) built from X Community Notes and the PolitiFact/GossipCop records in UPFD, released with frozen ~80/10/10 train/validation/test splits. The primary task (T1) classifies a claim, given its Community Note, into six non-exclusive misinformation mechanisms (factual error, manipulated media, outdated, missing context, unverified, satire), using the tags chosen by note writers as labels; five further task views are released as references or diagnostics. The authors use the resource to run three audits of how evaluation settings affect scores. (1) Information: on T1, removing, swapping, or masking the note shows that the note alone nearly matches full-input performance, while a wrong note erases the lead of API models over open-weight models. (2) Source and adaptation: on a binary-verdict transfer task between UPFD and Community Notes, QLoRA shows a much larger home-source advantage than 16-shot in-context learning, winning within source but losing across sources, and matching the source pools reduces but does not remove this gap. (3) Scoring: eight LLM judges, each with its parser, accept the same 476 generated-claim judgments at rates from 15% to 63%; judges that accept more agree less with three human raters, and the parser choice matters mainly for a judge that ignores the requested format. The paper concludes that results should report information inputs, source and adaptation settings, and evaluator and parser configuration.
Critical strengths
- Interesting question to investigate how you can determine veracity of claims from a "semi-trusted" source of information like community notes.
Critical weaknesses
- Limited novelty of the conclusions.
- The labels come from the same author as the note, so the note alone carrying the task is expected. The authors acknowledge this. Is T1 then a good primary task? I still do not see how this is any different than CommunityFact.
- T-Match is confounded with collection tier, T-Prop is almost 1, T2-HP uses keyword-derived labels, and T-Author partly predicts collection tier. Calling this a “benchmark” with six task views overstates it.
- The paper has many issues in writing with even random ?-marks in the paper.
Questions for authors
Check weaknesses.