- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Some AI systems now write whole research papers, code included. FabScore uses an AI coding assistant to take each number in such a paper and check it against the paper's code, by reading the code and, where it can, re-running it. Each number is marked as matching, not matching, or impossible to check. Across 144 AI-written papers, about one number in five did not match, mostly because the code does not do what the paper describes. AI reviewers at a conference for AI-written papers raised about one in ten of these mismatches. People checked the tool's verdicts on only ten papers, and could see those verdicts while checking.
Critical strengths
AI-written papers now reach conferences, and a reviewer who reads only the text cannot see that a headline number was typed into the evaluation script rather than measured. Checking every reported number against the released code is a useful idea, and the main finding is easy to grasp: about one number in five could not be reconciled with the code, most often because the code does not do what the paper describes, with hardcoded or simulated values the largest group (Table 2). That gives builders of AI research systems concrete failure modes and organisers a possible screening tool. The study covers 144 papers, and the appendix gives the full prompts and both assistants' error matrices.
Critical weaknesses
-
The instrument's validity and reliability are not established. FabScore is a prompted coding agent used as a measuring instrument, so the question is how well its labels agree with the truth and with themselves. The only validation is 10 papers, labelled by annotators in an interface that displays the tool's "verdict labels, supporting explanations". If they saw these first, the 100% precision measures agreement with the tool, not correctness. The same 10 papers were used to choose Claude Code over Codex; there is no inter-annotator agreement, no interval, and no statement of whether the prompts were developed on them. Reliability is not reported at all: no re-runs, although a stochastic agent re-executes stochastic code, and the choice of assistant changes the share of claims flagged on the same papers. The labels also have no tolerance: any difference between a reported and a re-run number is "result fabrication".
-
"Fabrication" is the wrong name for what is measured, and identifiable submissions carry the label. In research-integrity usage, fabrication means making up data or results and excludes honest error. Here it covers any mismatch, and Table 2 counts "formula or metric implementation produces incorrect values" and "code implementation bug" as experiment fabrication. The paper's neutral definition is not in the abstract, while the headline that over 70% of conference submissions "contain fabrications" recurs throughout.
-
The comparisons between sources are not interpretable as they stand.
-
The comparison with AI reviewers is built to favour the tool. §3.5 does not say whether the reviewers saw the code; if not, low coverage is guaranteed. Coverage is decided by an unchecked LLM judge that sees only the review and a list of claims
-
The writing and positioning claim more than the evidence. The abstract reports "high precision and label accuracy" without saying this rests on 10 papers checked with the verdicts on screen and one assistant, and the six verdicts are said to "capture all possible verification outcomes" although the prompts produce others. One source (FARS) is described only by a blog link, percentages carry decimals that 50 detections cannot support (i.e. spurious precision)
Questions for authors
- Did annotators see FabScore's verdicts before assigning their own labels, and was any claim labelled by more than one annotator?
- Were the prompts developed on the 10 validation papers, and how many of the 65 human-identified mismatches did neither assistant flag?
- Does "result fabrication" allow any tolerance for rounding or seed variation, and was FabScore run more than once on any paper?
- Of the 1,174 experiment fabrications, how many are hardcoded, simulated or mock values rather than implementation errors?
- Did the AI reviewers in the RQ4 comparison have access to the code? Were you aware of the venue's code audit of submission 220, and can you explain the difference from Fig. 4?
- Were the authors of the named Agents4Science submissions informed?
- How were the 27 rejected submissions and the 30 papers per system sampled?
2 comments
Response to Reviewer HH8c (Part 2/2)
Q1: Did annotators see FabScore's verdicts before assigning their own labels, and was any claim labelled by more than one annotator?
Yes, we provided a visualization interface for the annotators to review the papers, where they could view FabScore's verdicts and explanations, only for reference. Ultimately, the annotators were required to label each claim based on their own independent judgment by inspecting the paper and code. Specifically, each claim was annotated by two annotators.
Q2: Were the prompts developed on the 10 validation papers, and how many of the 65 human-identified mismatches did neither assistant flag?
No, the prompts did not develop on the 10 papers. In our human evaluation, all the 65 human-annotated fabrications are flagged by either Claude Code or Codex.
Q3: Does "result fabrication" allow any tolerance for rounding or seed variation, and was FabScore run more than once on any paper?
Yes, in our prompt defining result fabrication, we explicitly instruct the agent to exclude minor discrepancies such as rounding differences. When the reported and reproduced numbers are very close, they are generally not classified as result fabrications. We only run FabScore once on each paper.
Q4: Of the 1,174 experiment fabrications, how many are hardcoded, simulated or mock values rather than implementation errors?
Of the 1,174 experiment-fabrication claims, 524 (44.63%) were classified as involving simulated or hardcoded values that replace an experimental component.
Q5: Did the AI reviewers in the RQ4 comparison have access to the code? Were you aware of the venue's code audit of submission 220, and can you explain the difference from Fig. 4?
We are not sure if AI reviewers have access to the code since it is handled by the OpenReview system. We manually re-inspected the code and logs of submission 220 and confirmed that our finding in Figure 4 is grounded in the actual code.
Q6: Were the authors of the named Agents4Science submissions informed?
We did not inform any author.
Q7: How were the 27 rejected submissions and the 30 papers per system sampled?
We randomly selected 27 rejected submissions with code from OpenReview. For each system, we also randomly selected 30 papers.
Response to Reviewer HH8c (Part 1/2)
We thank Reviewer HH8c for finding our work provides a useful idea and the main finding is easy to grasp. Below, we address all the key points raised in the review and hope our responses could resolve the concerns.
W1: The instrument's validity and reliability are not established.
Thanks for this comment. We agree that our human evaluation process needs clearer reporting.
Our human validation was conducted by six experts, organized into three pairs. Each pair was assigned the same subset of papers, which the two experts annotated independently. Each expert first examined the code and logs, rerunning experiments only when the evidence was insufficient to reach a judgment. In practice, most cases could be resolved by inspecting the code and logs, without extensive reruns. We then aggregated the annotations to identify disagreements, which the paired experts discussed to reach a final consensus.
W2: "Fabrication" is the wrong name for what is measured, and identifiable submissions carry the label.
Thank you for this comment. Our intention is to study hallucinations in AI-generated papers. We acknowledge that “fabrication” might suggest intent and will revise the terminology to describe the observed inconsistencies more precisely.
W3: The comparisons between sources are not interpretable as they stand.
Thank you for this comment. We acknowledge that comparisons across paper sources are difficult to interpret. Our analysis aims to offer explanations on the potential causes to facilitate further discussion.
W4: The comparison with AI reviewers is built to favour the tool. Coverage is decided by an unchecked LLM judge that sees only the review and a list of claims.
Thanks for this comment. We conducted a human validation for the LLM judge in Figure 5 (a), and hope our validation would strengthen the reliability of these analyses.
Specifically, we randomly sampled 30 FabScore-detected fabrication claims from 10 papers. Our human annotators assessed whether the corresponding AI reviews identified the same underlying issues. Their judgments agreed with the LLM’s coverage judgments in 25 of 30 cases (83.3%).
W5: The writing and positioning claim more than the evidence.
Thanks for the suggestion. We will revise our description as suggested in this comment in our revision.