acceptodds
Under review as a conference paper at ICLR 2027

VERIBENCH-WILD: A BENCHMARK AND MEASUREMENT TESTBED FOR MISINFORMATION EVALUATION

Abstract

We introduce VERIBENCH-WILD, a benchmark and measurement testbed for studying how evaluation choices shape misinformation benchmark scores. A higher score need not mean better detection; it also depends on the information a model receives, its source and adaptation setting, and how its outputs are scored. VERIBENCH-WILD provides 27,158 multilingual but English-dominant social-media records with frozen splits, a primary mechanism-typology task, five further task views, public and full representations, and reproducible reference baselines. Across five evaluated models, removing the fact-check note reduces macro-F1 by 12.7–21.3 points, while note-only inputs remain close to full-input performance. The cross-source gap changes substantially with the adaptation regime, and restricting training and evaluation to common-support examples reduces but does not eliminate it. For generated claims, eight automatic evaluation pipelines, each a judge with its parser, accept the same outputs at rates from 15.3% to 63.4%. These choices change rankings, not only scores: a wrong fact-check note erases the lead of API models over open-weight models, QLoRA fine-tuning beats in-context learning within source but loses to it across sources, and ranking judges by acceptance nearly reverses their ranking by agreement with three blinded human raters. Across all eight judges, the parser matters only when a judge ignores the requested format, which moves one judge from 0.4% to 31.9%. Scores on \veribench should therefore state their information inputs, source and adaptation settings, and evaluator and parser configuration, so that comparisons are interpreted under explicit conditions.

Primary area:
datasets and benchmarks
Keywords:
Misinformation detection, Fact verification, Benchmark evaluation, Large language models, Multilingual analysis, Community Notes

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_v5zeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces VeriBench-Wild, a benchmark of 27,158 social-media records (multilingual but English-dominant) built from X Community Notes and the PolitiFact/GossipCop records in UPFD, released with frozen ~80/10/10 train/validation/test splits. The primary task (T1) classifies a claim, given its Community Note, into six non-exclusive misinformation mechanisms (factual error, manipulated media, outdated, missing context, unverified, satire), using the tags chosen by note writers as labels; five further task views are released as references or diagnostics. The authors use the resource to run three audits of how evaluation settings affect scores. (1) Information: on T1, removing, swapping, or masking the note shows that the note alone nearly matches full-input performance, while a wrong note erases the lead of API models over open-weight models. (2) Source and adaptation: on a binary-verdict transfer task between UPFD and Community Notes, QLoRA shows a much larger home-source advantage than 16-shot in-context learning, winning within source but losing across sources, and matching the source pools reduces but does not remove this gap. (3) Scoring: eight LLM judges, each with its parser, accept the same 476 generated-claim judgments at rates from 15% to 63%; judges that accept more agree less with three human raters, and the parser choice matters mainly for a judge that ignores the requested format. The paper concludes that results should report information inputs, source and adaptation settings, and evaluator and parser configuration.

Critical strengths
  • Interesting question to investigate how you can determine veracity of claims from a "semi-trusted" source of information like community notes.
Critical weaknesses
  • Limited novelty of the conclusions.
  • The labels come from the same author as the note, so the note alone carrying the task is expected. The authors acknowledge this. Is T1 then a good primary task? I still do not see how this is any different than CommunityFact.
  • T-Match is confounded with collection tier, T-Prop is almost 1, T2-HP uses keyword-derived labels, and T-Author partly predicts collection tier. Calling this a “benchmark” with six task views overstates it.
  • The paper has many issues in writing with even random ?-marks in the paper.
Questions for authors

Check weaknesses.

Reviewer_Xss6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces VERIBENCH-WILD, which is a collection of community notes from X and older PolitiFact and GossipCop news claim, that can serve as a measurement testbed for misinformation analysis. The main evaluation task is to predict the misleading-reason tags chosen by the community note writer, and the authors also perform additional ablations to see how the score change with the change in input features. While the dataset is multilingual, it's primarily English-dominant.

Critical strengths

None

Critical weaknesses

It is very difficult to evaluate this paper because of how it presents the findings.

First, the paper is very, very hard to read and very difficult to understand. For example: A lot of findings, results, and phrases are condensed together in one sentence, terms are introduced in abstract and introduction that the readers have no idea about, and the task definitions themselves are hard to understand. The paper needs significant rewriting to improve clarity and reduce verbosity.

Further, the experiments themselves don't make much sense to me. The model is provided with fact-check note and asked to predict the misleading-reason tag, so the answer is already in the input. For example: in the test set provided by authors in the anonymous repo, "record_id": "vw1-430ad9ee0b7f8ba2", which is the first test sample, the note says, “The image depicted is AI generated and not a factual representation of the user’s description”, and the only positive tag is manipulated media, whose description in the prompt includes AI-generation. As such, this is more akin to text matching than misinformation analysis. The authors also write that the T1 scores only reflect evidence-conditioned recognition. Moreover, in the ablations, adding posts to input don't add anything, and the community-notes are enough by themselves for the task.

The dataset itself is also not documented properly. The construction description of dataset is very vague, which says that it joins Community Notes to accessible post text. However, it is never described what these post texts are, and how are they obtained, and how the posts were sampled into four collection batches? While multilingual, the dataset is also primarily English-dominant.

Questions for authors

Suggestions for Authors:

I request and recommend that the authors take time and rewrite the paper from scratch. The paper is too difficult to read. While the authors have been transparent about their usage of AI for manuscript organization and language polishing, the current paper needs a significant rewriting to meet academic standards of publishing.

Reviewer_mEHJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VeriBench-Wild, a misinformation evaluation benchmark containing 27,158 records (mostly in English) from X Community Notes and UPFD, with a fixed data split. The main task (T1) is to classify misinformation into six types based on a claim and its fact-check annotation. The benchmark also includes five auxiliary or diagnostic tasks. The authors introduce the concept of a "measurement contract" and use three controlled experiments to show that different evaluation settings can change model rankings.

Critical strengths
  1. The evaluation audits include human judgments, and the findings are practically useful. Three blind reviewers independently annotated 476 judgments and found that higher acceptance rates are associated with lower agreement between LLM judges and humans. This is an important finding for research using LLM-as-judge and is the most convincing part of the paper.

  2. The experiments are well controlled and statistically rigorous. Each audit clearly specifies the controlled variables and uses multiple random seeds, paired confidence intervals, and bootstrapping to report uncertainty. The authors are also transparent about the limitations.

Critical weaknesses
  1. The novelty is limited, and the validity of the main task is questionable. The three main findings (label leakage from evidence, differences between fine-tuning and ICL across domains, and instability in LLM evaluation) have already been reported in previous work, and the authors themselves acknowledge that these findings are not new. The T1 labels were selected by the annotators themselves, so the high performance achieved by providing the annotations is not surprising. Human agreement was evaluated on only 50 samples, and the most common "missing context" category had a κ of just 0.08. This raises concerns about the reliability of T1 as the main task and its use for model ranking.

  2. Some key comparisons are not well controlled, several conclusions are overstated, and important data have not been released. QLoRA uses thousands of training samples, while ICL uses only 16, making the comparison difficult to interpret. The paper also lacks comparisons with equal numbers of labeled samples and calibration analyses. The confidence interval for the claim that incorrect annotations eliminate the advantage of API models includes zero. The parser-related conclusions are based on only one model, and the issue may come from contradictory instructions in the prompt itself. In addition, the human annotations and original responses from the evaluation models have not been released, limiting reproducibility and reuse. The authors acknowledge these limitations but provide few additional experiments to address them. The claims in the abstract are also stronger than those supported by the main text.

Questions for authors
  1. Could the authors release the audit samples, original model responses, and individual human annotations after the paper is accepted? This would affect my assessment of the paper's resource contribution.

  2. If QLoRA is trained with the same number of labeled samples as ICL (e.g., 16, 64, or 256), how would the cross-domain gap G change?

  3. The CN-trained QLoRA predicts only about 5–9% of UPFD samples as SUPPORTED, while the actual proportion is 55.7%. How much of the cross-domain gap remains after prior or threshold calibration?

  4. After excluding categories with low agreement, such as outdated or missing context, does the correlation of ρ = −0.81 still hold? Could the authors also provide a confidence interval?

  5. How were the binary SUPPORTED/REFUTED labels constructed for CN records?

Reviewer_p7z2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the reliability of evaluation protocols in misinformation detection and investigates how different experimental settings affect reported model performance. The authors propose VERIBENCH-WILD, a benchmark containing 27,158 multi-source social media records, with frozen data splits and six task views. The paper conducts systematic audits of information access, cross-source generalization, and automatic evaluation pipelines. The results show that fact-checking notes can substantially influence model performance, QLoRA fine-tuning improves in-source performance but can lead to worse cross-source generalization than in-context learning, and different automatic judges produce substantially different acceptance rates and levels of agreement with human evaluations. The paper highlights the importance of explicitly reporting evaluation settings to improve the reliability and comparability of misinformation detection benchmarks.

Critical strengths

1. Rigorous experimental design and solid resource contribution All three audits follow strict variable-controlled protocols: the information audit fixes test sets and model weights, the generalization audit holds the base model constant, and the scoring audit uses frozen generated outputs. All results are accompanied by proper statistical intervals and paired tests. The full release of the 27k-record benchmark, frozen prompts, parser rules, and reproducible scripts provides a valuable standardized diagnostic resource for the misinformation detection community.

2. Practically actionable findings with real-world selection implications The study quantifies the magnitude of evaluation protocol effects and reveals several counter-intuitive ranking reversals beyond general intuition. Specifically, the finding that incorrect evidence harms more than missing evidence, that QLoRA underperforms few-shot ICL in cross-source generalization, and the strong negative correlation between judge acceptance rate and human alignment, offer direct guidance for model selection and evaluation design in real-world deployment scenarios.

Critical weaknesses
  • The main findings are largely consistent with existing observations in the broader NLP evaluation literature. The effects of input information, domain shifts, and evaluation metrics on model performance have been extensively studied. Although the paper investigates these issues in misinformation detection, it does not introduce a substantially new evaluation methodology or theoretical framework. The concept of a measurement contract is also not sufficiently distinguished from conventional evaluation protocol specifications.

  • The comparison between QLoRA fine-tuning and in-context learning is not fully controlled. QLoRA uses substantially more labeled training examples than the 16-shot ICL setting, and the two approaches also differ in parameter updates and numerical precision. Therefore, it is difficult to attribute the observed cross-source generalization gap specifically to the adaptation strategy.

  • The benchmark evaluation is based on a limited set of data sources and task definitions. In particular, the automatic judge audit uses a relatively small set of generated samples, and human agreement varies considerably across different misinformation mechanisms. This raises questions about the representativeness of the evaluation results and their generalizability to broader real-world misinformation detection scenarios.

Questions for authors
  1. What does the proposed measurement contract framework offer beyond existing studies of evaluation protocol sensitivity?

  2. Could the authors provide a controlled comparison between QLoRA and ICL with matched supervision budgets?

  3. How representative are the selected data sources and human evaluation samples of real-world misinformation detection scenarios?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.