acceptodds
Under review as a conference paper at ICLR 2027

FabScore: Fine-Grained Evaluation of Fabrications in Automated AI Research

Abstract

Automated AI research shows great promise for accelerating scientific discovery, but ensuring the integrity of AI-generated research remains a critical challenge. In this work, we introduce FabScore, a new framework for fine-grained evaluation of fabrications in automated AI research. Given a research paper and its associated code, FabScore extracts numerical results and figure labels as individual claims, employs a coding agent to evaluate each claim through static analysis and code execution, and ultimately assigns one of six verdict categories covering fabricated, reproducible, and unverifiable outcomes. Human evaluation shows that FabScore, using Claude Code as the coding agent, achieves high precision and label accuracy, with a recall of 76.9%, in fabrication detection. Applying FabScore to 144 papers from various sources, we find the overall claim-level fabrication rate to be 21.2%. Notably, over 70% of evaluated AI-authored conference submissions contain fabrications, with accepted submissions still reaching a paper-level fabrication rate of 59.3%. Experiment fabrication is the most prevalent type, indicating that AI research systems often struggle to correctly implement the experiments described in the paper. Finally, over 85% of FabScore-detected fabrications are missed by AI reviewers, suggesting that our framework can serve as a valuable complementary tool to existing AI review processes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Automated Research, Fabrication Evaluation, AI Scientist

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HH8cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Some AI systems now write whole research papers, code included. FabScore uses an AI coding assistant to take each number in such a paper and check it against the paper's code, by reading the code and, where it can, re-running it. Each number is marked as matching, not matching, or impossible to check. Across 144 AI-written papers, about one number in five did not match, mostly because the code does not do what the paper describes. AI reviewers at a conference for AI-written papers raised about one in ten of these mismatches. People checked the tool's verdicts on only ten papers, and could see those verdicts while checking.

Critical strengths

AI-written papers now reach conferences, and a reviewer who reads only the text cannot see that a headline number was typed into the evaluation script rather than measured. Checking every reported number against the released code is a useful idea, and the main finding is easy to grasp: about one number in five could not be reconciled with the code, most often because the code does not do what the paper describes, with hardcoded or simulated values the largest group (Table 2). That gives builders of AI research systems concrete failure modes and organisers a possible screening tool. The study covers 144 papers, and the appendix gives the full prompts and both assistants' error matrices.

Critical weaknesses
  1. The instrument's validity and reliability are not established. FabScore is a prompted coding agent used as a measuring instrument, so the question is how well its labels agree with the truth and with themselves. The only validation is 10 papers, labelled by annotators in an interface that displays the tool's "verdict labels, supporting explanations". If they saw these first, the 100% precision measures agreement with the tool, not correctness. The same 10 papers were used to choose Claude Code over Codex; there is no inter-annotator agreement, no interval, and no statement of whether the prompts were developed on them. Reliability is not reported at all: no re-runs, although a stochastic agent re-executes stochastic code, and the choice of assistant changes the share of claims flagged on the same papers. The labels also have no tolerance: any difference between a reported and a re-run number is "result fabrication".

  2. "Fabrication" is the wrong name for what is measured, and identifiable submissions carry the label. In research-integrity usage, fabrication means making up data or results and excludes honest error. Here it covers any mismatch, and Table 2 counts "formula or metric implementation produces incorrect values" and "code implementation bug" as experiment fabrication. The paper's neutral definition is not in the abstract, while the headline that over 70% of conference submissions "contain fabrications" recurs throughout.

  3. The comparisons between sources are not interpretable as they stand.

  4. The comparison with AI reviewers is built to favour the tool. §3.5 does not say whether the reviewers saw the code; if not, low coverage is guaranteed. Coverage is decided by an unchecked LLM judge that sees only the review and a list of claims

  5. The writing and positioning claim more than the evidence. The abstract reports "high precision and label accuracy" without saying this rests on 10 papers checked with the verdicts on screen and one assistant, and the six verdicts are said to "capture all possible verification outcomes" although the prompts produce others. One source (FARS) is described only by a blog link, percentages carry decimals that 50 detections cannot support (i.e. spurious precision)

Questions for authors
  1. Did annotators see FabScore's verdicts before assigning their own labels, and was any claim labelled by more than one annotator?
  2. Were the prompts developed on the 10 validation papers, and how many of the 65 human-identified mismatches did neither assistant flag?
  3. Does "result fabrication" allow any tolerance for rounding or seed variation, and was FabScore run more than once on any paper?
  4. Of the 1,174 experiment fabrications, how many are hardcoded, simulated or mock values rather than implementation errors?
  5. Did the AI reviewers in the RQ4 comparison have access to the code? Were you aware of the venue's code audit of submission 220, and can you explain the difference from Fig. 4?
  6. Were the authors of the named Agents4Science submissions informed?
  7. How were the 27 rejected submissions and the 30 papers per system sampled?
2 comments
AuthorsOpenReview ↗

Response to Reviewer HH8c (Part 1/2)

We thank Reviewer HH8c for finding our work provides a useful idea and the main finding is easy to grasp. Below, we address all the key points raised in the review and hope our responses could resolve the concerns.


W1: The instrument's validity and reliability are not established.

Thanks for this comment. We agree that our human evaluation process needs clearer reporting.

Our human validation was conducted by six experts, organized into three pairs. Each pair was assigned the same subset of papers, which the two experts annotated independently. Each expert first examined the code and logs, rerunning experiments only when the evidence was insufficient to reach a judgment. In practice, most cases could be resolved by inspecting the code and logs, without extensive reruns. We then aggregated the annotations to identify disagreements, which the paired experts discussed to reach a final consensus.


W2: "Fabrication" is the wrong name for what is measured, and identifiable submissions carry the label.

Thank you for this comment. Our intention is to study hallucinations in AI-generated papers. We acknowledge that “fabrication” might suggest intent and will revise the terminology to describe the observed inconsistencies more precisely.


W3: The comparisons between sources are not interpretable as they stand.

Thank you for this comment. We acknowledge that comparisons across paper sources are difficult to interpret. Our analysis aims to offer explanations on the potential causes to facilitate further discussion.


W4: The comparison with AI reviewers is built to favour the tool. Coverage is decided by an unchecked LLM judge that sees only the review and a list of claims.

Thanks for this comment. We conducted a human validation for the LLM judge in Figure 5 (a), and hope our validation would strengthen the reliability of these analyses.

Specifically, we randomly sampled 30 FabScore-detected fabrication claims from 10 papers. Our human annotators assessed whether the corresponding AI reviews identified the same underlying issues. Their judgments agreed with the LLM’s coverage judgments in 25 of 30 cases (83.3%).


W5: The writing and positioning claim more than the evidence.

Thanks for the suggestion. We will revise our description as suggested in this comment in our revision.

AuthorsOpenReview ↗

Response to Reviewer HH8c (Part 2/2)

Q1: Did annotators see FabScore's verdicts before assigning their own labels, and was any claim labelled by more than one annotator?

Yes, we provided a visualization interface for the annotators to review the papers, where they could view FabScore's verdicts and explanations, only for reference. Ultimately, the annotators were required to label each claim based on their own independent judgment by inspecting the paper and code. Specifically, each claim was annotated by two annotators.


Q2: Were the prompts developed on the 10 validation papers, and how many of the 65 human-identified mismatches did neither assistant flag?

No, the prompts did not develop on the 10 papers. In our human evaluation, all the 65 human-annotated fabrications are flagged by either Claude Code or Codex.


Q3: Does "result fabrication" allow any tolerance for rounding or seed variation, and was FabScore run more than once on any paper?

Yes, in our prompt defining result fabrication, we explicitly instruct the agent to exclude minor discrepancies such as rounding differences. When the reported and reproduced numbers are very close, they are generally not classified as result fabrications. We only run FabScore once on each paper.


Q4: Of the 1,174 experiment fabrications, how many are hardcoded, simulated or mock values rather than implementation errors?

Of the 1,174 experiment-fabrication claims, 524 (44.63%) were classified as involving simulated or hardcoded values that replace an experimental component.


Q5: Did the AI reviewers in the RQ4 comparison have access to the code? Were you aware of the venue's code audit of submission 220, and can you explain the difference from Fig. 4?

We are not sure if AI reviewers have access to the code since it is handled by the OpenReview system. We manually re-inspected the code and logs of submission 220 and confirmed that our finding in Figure 4 is grounded in the actual code.


Q6: Were the authors of the named Agents4Science submissions informed?

We did not inform any author.


Q7: How were the 27 rejected submissions and the 30 papers per system sampled?

We randomly selected 27 rejected submissions with code from OpenReview. For each system, we also randomly selected 30 papers.

Reviewer_qmmJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FabScore which is a framework for finding out fabrications in AI generated research papers. A coding agent (which could be Claude Code or Codex) extracts numerical claims from tables, results section and figures. It checks each claim by static code analysis and code execution. It assigns one of these 6 verdicts -> verified, data fabrication, experiment fabrication, result fabrication, no code files, or insufficient evidence.

The authors validate the framework with a human study on a small sample (~10) of papers. They then apply it to a collection of AI generated papers from several automated research systems and AI authored conference submissions. They report that fabrications are common across all sources with experiment fabrication the most common one and also argue that their system complements AI based peer review.

My main concern with the evaluation is the age of the corpus. A lot of it comes from older systems (AI Scientist - 2024, MLR-Agent - 2025, Agents4Science - 2025), and the more recent systems (FARS) score noticeably lower on fabrication.

Critical strengths

The paper addresses an important problem -> Automated systems are now producing papers that get submitted to and accepted at conferences but whether their results match what the code actually does remains an open question. This paper addresses that gap by showing how code based checks can catch issues that text only reviews miss. Its claim level analysis and categorization of data, experiment, and result mismatches offer a useful starting point for the community.

Critical weaknesses
  1. The evaluated corpus is outdated and the validator is a much newer model, so the current results reflect a model gap and tell less about the framework itself -> Much of the 144 paper eval corpus comes from older systems (the 30 AI Scientist papers are from 2024, while the 30 MLR Agent papers come from a 2025 benchmark built around models such as Claude 3.7 Sonnet and o4-mini). FabScore however uses Claude Sonnet 4.6 or GPT-5.4. Evaluating the outputs of older models with a newer stronger model is the easier case here and does not show how well the framework tackles current systems. FARS, one of the more recent systems has the lowest fabrication rate out of all the sources evaluated. This suggests that as generators improve, both the fabrication rates FabScore detects and its advantage over existing approaches may decrease. The paper does not show how well it performs against current systems.
  1. The contribution is limited and closely related work is not discussed -> The method is a standard agentic framework (extract claims, read code, run it and compare). The 6 way taxonomy is sensible but incremental and the main contribution is the empirical study whose reliability is limited by weakness (1). Also several related works are missing -> Concurrent claim2code auditing pipelines - ReAgent [1], Studies of silent failures and hallucinations in AI scientist output: "The More You Automate, the Less You See" [2], PaperRecon [3], ResearchArena [4], and Beel et al. on the AI Scientist [5], Surveys: a survey of the verification gap in AI scientists [6].

Given these, the claim that prior work "lacks fine grained evaluation" is overstated, and the paper includes no direct comparison.

References ->

[1] Shen et al. "Beyond the Text: Verifying That Agent-Written Papers Are Backed by Their Artifacts" (ReAgent). arXiv:2609.22111, Aug 2026.

[2] Luo, Kasirzadeh, Shah. "The More You Automate, the Less You See." arXiv:2509.08713, NeurIPS 2025.

[3] Miyai et al. "PaperRecon" (Paper Reconstruction Evaluation). arXiv:2604.01128, Apr 2026.

[4] "How Far Are We From True Auto-Research?" (ResearchArena). arXiv:2605.19156, May 2026.

[5] Beel, Kan, Baumgart. "Evaluating Sakana's AI Scientist." arXiv:2502.14297, Feb 2025.

[6] Ding et al. "Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap." arXiv:2608.05179.

Questions for authors
  1. Can you re run the analysis on a newer corpora of AI generated papers (in 2026)? -> Newer public sets with code where available include the CAISc 2026 accepted papers [1], ResearchArena's frontier coding-agent papers [2] and example papers from systems such as the DeepScientist [3], AI-Researcher [4].
  2. How were the 30 papers per source chosen (for AI Scientist, MLR-Agent, and FARS)?
  3. Were annotators blind to FabScore verdicts and explanations? Please report inter annotator agreement as well.

Clarifications and suggestions

  1. How does FabScore compare with the concurrent ReAgent (arXiv 2609.22111) and the "More You Automate, the Less You See" study (arXiv 2509.08713)?
  2. Please fix the small inconsistencies -> the Table 2 percentages (experiment row) that sum to 94.7%.

References

[1] CAISc 2026 accepted papers. caisc2026.github.io (reviews on OpenReview, CAISc/2026/Conference).

[2] "How Far Are We From True Auto-Research?" (ResearchArena). arXiv:2605.19156.

[3] DeepScientist. github.com/ResearAI/DeepScientist (arXiv:2509.26603).

[4] AI-Researcher (HKUDS). github.com/HKUDS/AI-Researcher

2 comments
AuthorsOpenReview ↗

Response to Reviewer qmmJ

We thank Reviewer qmmJ for appreciating our work addresses an important problem and for recognizing our analysis offers a useful starting point for the community. Below, we address all the points raised and hope our responses adequately resolve the concerns.


W1&Q1: The evaluated corpus is outdated and the validator is a much newer model. Can you rerun the analysis on a newer corpora of AI generated papers (in 2026)?

Thanks for this suggestion. We agree that evaluating more recent papers would make our findings more informative.

ResearchArena [2], DeepScientist [3], and AI-Researcher [4] are all excellent works. However, we have not yet been able to find the open-source code or example papers for ResearchArena, and DeepScientist and AI-Researcher were both released in 2025, no later than the papers we have already evaluated. Therefore, we instead evaluate two papers from CAISc 2026 to provide the most up-to-date assessment. We will add these works to our references. Table A shows the results.

Table A: Claim verification results for the two CAISC26 papers.

Submission IDPaper TitleTotal ClaimsVerifiedData FabricationExperiment FabricationResult FabricationNo Code FilesInsufficient Evidence
114Recombining Local Optima in Conway’s Strongly Regular Graph Problem137900022718
175A Forced-Structure Reduction and Verifiable Bounds for Conway’s 99-Graph121100001

Table A shows that most claims in both papers were verified. However, Submission 114 had two claims classified as result fabrication, while approximately 33% of its claims could not be verified due to missing code files or insufficient evidence.


W2: The contribution is limited and closely related work is not discussed.

Thank you for this comment.

  • The contribution is limited.

We agree that our method follows a standard agentic framework, and we do not intend to claim technical novelty as our main contribution. Instead, our contributions include providing a tool that automatically detects fabricated experimental results in AI-generated papers and an empirical analysis based on this tool. Our human evaluation shows that the tool provides a conservative lower bound for fabrication detection.

Although we cannot cover many recent papers in this work, our evaluation of 144 papers already reveals a surprising finding: experimental fabrication is widespread in AI-generated papers. Moreover, we find that FabScore can identify issues that current AI reviewers often miss, so our tool can also serve as a complementary reference for AI reviewers.

  • Closely related work is not discussed.

Thanks for pointing out these papers. We will add them to our references.


Q2: How were the 30 papers per source chosen (for AI Scientist, MLR-Agent, and FARS)?

We randomly selected 30 papers from each source.


Q3: Were annotators blind to FabScore verdicts and explanations?

No, our annotators were not blind to them. In our human evaluation, we provided annotators with a visualization interface for reviewing each paper, in which they could see the verdicts and explanations produced by different coding agents. These outputs served only as references, and annotators reached their final conclusions based on their own judgment.


Clarification 1: How does FabScore compare with the concurrent ReAgent (arXiv 2609.22111) and the "More You Automate, the Less You See" study (arXiv 2509.08713)?

Thanks for mentioning these two great works. We acknowledge that the evaluation frameworks of ReAgent and FabScore share some similarities, and the papers they evaluate partially overlap (e.g., both evaluate papers generated by the FARS system). However, the two works differ mainly in their focus. ReAgent focuses on designing a reliable evaluation framework, and most of its discussion centers on the reliability of the tool and how to use it. In contrast, beyond tool design, FabScore analyzes the evaluation results in depth and reveals the possible causes of fabricated results in current AI-generated papers. Moreover, FabScore compares its detection results with those of AI reviewers and shows that it can identify issues that AI reviewers fail to detect.

“More You Automate, the Less You See” also evaluates and analyzes potential risks in AI scientist systems. However, this paper was published in 2025, and its evaluation framework does not use the latest coding agents. We will add both papers to our references.


Clarification 2: Please fix the small inconsistencies -> the Table 2 percentages (experiment row) that sum to 94.7%.

Thanks for this suggestion. Please note that Table 2 lists only the most common possible reasons for each fabrication type. Since other reasons also exist, the percentages listed do not necessarily sum to 100%.

I would like to thank the authors for their detailed response.

  1. It seems like ResearchArena's papers are open source at www.github.com/YouAreSpecialToMe/ResearchArena (the paper doesn't give the link so thats why it might be hard to find), so ideally the authors should have run on these papers. I completely understand they don't have time for these at the moment. The 2 CAISC's papers seem like a useful start but both of them address a verifiable problem track instead of open ended ML experiments in the rest of their study.

  2. My second concern in W1 wasn't addressed i.e. whether detection of false claims is an attribute of the pipeline/harness or simply from the fact that newer models are reviewing papers generated from older models.

  3. Since the annotators saw the pipeline's decision, this would bias their own view. The authors mention that "the final conclusions were based on their own judgment" but that is after conducting a biased experiment. I advise the authors to redo the annotator verification experiment without showing them their own system's decisions.

Given these gaps, I will keep my current score of Weak Reject, but I appreciate the author's responses. I believe that the problem the authors are trying to address is extremely crucial for the community as we need better systems to critique auto research agents and their generations, and such systems will only help us improve the capability of autonomous research and ultimately towards RSI.

One suggestion that is separate from my assessment of the paper and has no effect on my score - since the area moves quickly, the authors might consider maintaining a public leaderboard of fabrication rates for new AI-research systems. It would let the community track progress over time and keep the tool useful even as the systems change.

Reviewer_7rfuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

With AI systems capable of writing research papers end to end, it is becoming imperative to find ways to validate the results in this papers. The number of AI generated papers is increasing and it is already unfeasible for humans to check them properly, thus there is an urgent need for an automated and reliable verification process. The paper proposes FabScore, which is a prompt-based pipeline in which a coding agent extracts every numerical claim in a paper and check it agains the accompanying code. It assigns one of the six verdicts: verified, two unverifiable categories or “fabrications” (data, experiment or result). The validation of FabScore is performed against human evaluations for 10 different papers, claiming that (with Claude Code) FabScore reaches 100% precision and 76.9% recall. When applied to 144 AI generated papers it finds that 21.2% of claims are fabricated. Notably, it finds that 70% of AI-authored Agents4Science submissions contain at least one fabrication, and that AI reviewers mention only 11% of those.

Critical strengths

(1) This paper addresses a relevant problem, since AI-authored papers are reaching main venues. Checking claims against code is necessary and rarely done. The validation (which in my opinion is a central matter) can be audited, every fabrication verdict comes with an explanation (Section 3.2), which can be checked by a human. This makes the tool useful even with an imperfect detector.

(2) The paper observes that the evaluated AI reviewers (of AI-authored papers) focused on contradictions within the text and almost never on mismatches between code and text. This is an important evidence to keep in mind when adopting AI reviewers in venues.

Critical weaknesses

(1) The validation is too small, all accuracy claims depend on 10 papers, annotated by “6 human experts with research experience”. Moreover, it is not clearly reported how this validation worked. Section 3.2 seems to contradict itself about who extracted the claims and wrote the verdicts: annotators "extract claims … and assign a ground-truth verdict", yet the interface displays the agents' "extracted claims, verdict labels, supporting explanations". If annotators saw the agents' output, the ground truth may be anchored to the system, inflating both precision and recall. It is also not reported whether annotators ran the code, how long they spent per paper, how many annotators labeled each paper, or how often they agreed. Without this, it is difficult to interpret the validation.

(2) The method is prompt-based plus a six-label taxonomy. No design choice is tested. The cost of running the pipeline (time, tokens, compute per paper) is not reported either. This makes the study feel incomplete.

(3) Table 2 (reasons) and Figure 5 (AI-review coverage) are produced by LLM judges, and no human validation of these judges is reported. This effectively adds a new layer of human validation needed.

(4) The definition of “fabrication” is broad and suggests intent. However, it includes wrong formulas and code bugs. I believe this needs a refinement, otherwise is not easy to interpret what “over 70% of AI-authored conference submissions contain fabrications”.

(5) Several aspects of the human validation are not specified. For example, the time taken to review each paper by a human annotator.

(6) Estimating cost of performing these reviews is not reported.

Questions for authors
  1. Did annotators label the claims before or after seeing the agents' extracted claims, verdicts and explanations? Of the 65 human-marked fabrications, how many were flagged by neither Claude Code nor Codex?
  2. Did annotators read or run the code? How long did they spend per paper? How many annotators labeled each paper, and how often did they agree? What is their expertise beyond "AI research experience"?
  3. If FabScore is run twice on the same paper, how often do the verdicts agree?
  4. Design choices and baselines. How do the results change if only static analysis is used, without code execution? How does FabScore compare with existing reproducibility checkers (e.g. Bai et al., 2026) on the same papers?
  5. Cost. What is the time, token and compute cost of running FabScore on one paper?
  6. LLM judges. Were the reason labels (Table 2) and the AI-review coverage decisions (Figure 5) checked by humans on a sample? If so, how often did the humans agree?
  7. AI reviewers. Did the Agents4Science AI reviewers have access to the code? If not, it is expected that they rarely flag mismatches between code and text.
  8. What counts as fabrication. How are ordinary run-to-run differences on re-execution distinguished from result fabrication? How would the headline rates change if wrong formulas and code bugs were counted separately from hardcoded or simulated values?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 7rfu (Part 1/3)

We thank Reviewer 7rfu for recognizing that our work addresses a relevant problem and for appreciating that our work provides a useful tool. We are also glad that the reviewer finds our presentation is clear and easy to follow. Below, we address all the points raised and hope our responses adequately resolve the concerns.


W1 & Q1 & Q2: The validation is too small, all accuracy claims depend on 10 papers, annotated by “6 human experts with research experience”. Did annotators label the claims before or after seeing the agents' extracted claims, verdicts and explanations? Of the 65 human-marked fabrications, how many were flagged by neither Claude Code nor Codex?

Thanks for this comment. We address your concerns separately below.

  • The validation is too small.

We agree that more human evaluation data would strengthen our findings. Table B shows the results of human evaluation on 20 sampled tasks.

Table B: Human evaluation of claim extraction and fabrication detection using Claude Code and Codex on 20 sampled tasks, with 4 papers sampled from each data source. Bold indicates the higher percentage for each metric.

MetricClaude CodeCodex
Claim Extraction
# Total claims489489
# Extracted claims471470
Extraction coverage (%)96.3%96.1%
Fabrication Detection
# Human-annotated fabrications145145
# Detected fabrications99180
# Human-confirmed detections94130
# Label-agreed detections90114
Precision (%)94.9%72.2%
Recall (%)64.8%89.7%
Label accuracy (%)95.7%87.7%

As shown in Table B, Claude Code achieves higher precision (94.9%) and label accuracy (95.7%), while Codex achieves higher recall (89.7% vs. 64.8%). These findings are similar to our earlier human validation on 10 papers. We will update our human evaluation results in the revision.

  • Moreover, it is not clearly reported how this validation worked. It is also not reported whether annotators ran the code, how long they spent per paper, how many annotators labeled each paper, or how often they agreed.

Our human validation was conducted by six experts, organized into three pairs. Each pair was assigned the same subset of papers, which the two experts annotated independently. Each expert first examined the code and logs, rerunning experiments only when the evidence was insufficient to reach a judgment. In practice, most cases could be resolved by inspecting the code and logs, without extensive reruns.

We then aggregated the annotations to identify disagreements, which the paired experts discussed to reach a final consensus. As shown in Table B, the three pairs evaluated 20 papers in total, with each pair assigned six or seven papers. The annotation process took approximately 25–30 minutes per paper.

  • Did annotators label the claims before or after seeing the agents' extracted claims, verdicts and explanations? Of the 65 human-marked fabrications, how many were flagged by neither Claude Code nor Codex?

Yes, we provided experts with a visualization interface for reviewing each paper, in which annotators could see the verdicts and explanations produced by different coding agents. These outputs served only as references, and annotators reached their final conclusions based on their own judgment. In our human evaluation, all human-annotated fabrications are flagged by either Claude Code or Codex.

AuthorsOpenReview ↗

Response to Reviewer 7rfu (Part 2/3)

W2&W5&W6&Q3-5: No design choice is tested. The cost of running the pipeline (time, tokens, compute per paper) is not reported either.

Thanks for this suggestion. We address each question separately below.

  • Costs of running the pipeline.

We agree that printing the cost of running the pipeline would make our study more complete. Table C presents the evaluation costs and runtimes for 144 papers.

Table C: Evaluation costs and runtimes for 144 papers using Claude Code as the FabScore backbone.

SourcePapersTotal Cost (USD)Average Cost per Paper (USD)Cumulative Runtime (Hours)Average Runtime per Paper (Minutes)
AI-Scientist30$95.20$3.1713.0726.13
Agents4Sci (acc)27$156.42$5.7914.7832.85
Agents4Sci (rej)27$149.52$5.5415.4834.41
FARS30$40.04$1.334.378.75
MLR-Agent30$35.32$1.183.356.71
Total144476.503.3151.0621.27

As shown in Table C, evaluating one paper with FabScore took an average of 21.27 minutes and cost 3.31 USD using Claude Code as the backbone.

  • If FabScore is run twice on the same paper, how often do the verdicts agree?

Thank you for this question. We have not conducted repeated evaluations of the same papers and therefore cannot report a measured agreement rate. We would, however, like to clarify FabScore’s intended scope: it aims to provide a conservative lower bound on fabrication. As FabScore relies on an LLM backbone, variation across runs is possible. We acknowledge this as a limitation and will clarify it in the revised manuscript.

  • How do the results change if only static analysis is used, without code execution?

Using only static analysis, FabScore detects 1,135 fabrication claims across 144 papers, accounting for 76.74% of those detected by the full pipeline.

  • How does FabScore compare with existing reproducibility checkers (e.g. Bai et al., 2026) on the same papers?

Thank you for the suggestion. MechEvalAgent (Bai et al., 2026) and FabScore differ fundamentally in their evaluation granularity and focus.

MechEvalAgent assesses a research output as a whole for coherence, reproducibility and generalizability, expects standardized research outputs, and is instantiated for mechanistic interpretability.

In contrast, FabScore takes a paper and its code as they are, first extracts all reported numerical results as individual claims, and then verifies each claim one by one to assign a fine-grained verdict, allowing us to quantify exact fabrication rates and their underlying causes.

As the two frameworks target different objectives and operate at different granularities, a direct empirical comparison on the same papers cannot yield a fair conclusion.

AuthorsOpenReview ↗

Response to Reviewer 7rfu (Part 3/3)

W3&Q6: Table 2 (reasons) and Figure 5 (AI-review coverage) are produced by LLM judges, and no human validation of these judges is reported. Were the reason labels (Table 2) and the AI-review coverage decisions (Figure 5) checked by humans on a sample? If so, how often did the humans agree?

Thank you for this comment. We agree that human validation would strengthen the reliability of these analyses. For Table 2, we randomly sampled 30 fabrication claims and manually checked their assigned reason labels. Human judgments agreed with the LLM-assigned labels in 28 of 30 cases (93.3%).

For Figure 5, we also randomly sampled 30 FabScore-detected fabrication claims for human validation. Our human annotators assessed whether the corresponding AI reviews identified the same underlying issues. Their judgments agreed with the LLM’s coverage judgments in 25 of 30 cases (83.3%).


W4 & Q8: The definition of “fabrication” is broad and suggests intent. However, it includes wrong formulas and code bugs. I believe this needs a refinement, otherwise is not easy to interpret what “over 70% of AI-authored conference submissions contain fabrications”

Thank you for this comment. We acknowledge that our definition of fabrication is relatively coarse-grained. However, our primary goal in this study is to provide a broad initial estimate of the issue.

The finding that “over 70% of AI-authored conference submissions contain fabrications” reflects that over 70% of the evaluated Agents4Sci submissions contain at least one fabricated claim. We agree that a more fine-grained taxonomy of fabrications would offer richer interpretability, but given the scope of this paper, we plan to explore this in future work.


Q7: Did the Agents4Science AI reviewers have access to the code? If not, it is expected that they rarely flag mismatches between code and text.

We are not sure, but we suspect that the AI reviewers may not have inspected the code.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers and AC,

We sincerely thank all reviewers for your time and constructive feedback. We are encouraged that the reviewers recognize the following strengths of our work:

  • Important and timely problem. As AI-generated papers increasingly reach conferences, verifying whether reported results are supported by the accompanying code is a necessary but rarely performed step (7rfu, qmmJ, HH8c).
  • Code-based verification reveals issues that text-only review misses. Our finding that AI reviewers focus on textual inconsistencies and rarely detect code–paper mismatches provides important evidence for venues adopting AI reviewers (7rfu, qmmJ, HH8c).
  • Auditable, claim-level analysis. Every verdict comes with an explanation that humans can check (7rfu), the claim-level taxonomy offers a useful starting point for the community (qmmJ), and the findings expose concrete failure modes for builders of AI research systems (HH8c).
  • Clear presentation. (7rfu, qmmJ).

Below, we summarize the main shared concerns and how we have addressed them.

1. Strengthening the human validation (7rfu, qmmJ, HH8c)

We agree with the reviewers that our original human validation on 10 papers was limited in scale and lacked important details. We have strengthened it in three ways:

  • Expanded human validation.

We doubled the validation set from 10 to 20 papers (four papers from each source), covering 489 claims and 145 human-annotated fabrications. On this larger set, Claude Code achieves 94.9% precision and 95.7% label accuracy, while Codex achieves 89.7% recall, and both cover over 96% of the claims during extraction. These results are consistent with our original findings on 10 papers, which we hope addresses your concern about the reliability of our validation.

  • Detailed annotation protocol.

We would also like to clarify how the validation was conducted. Six experts, organized into three pairs, annotated each claim independently by inspecting the paper, code and logs, and reran experiments when the evidence was insufficient. Each pair then discussed their disagreements to reach a final consensus. Annotation took about 25–30 minutes per paper. In addition, the prompts were not developed on the validation papers.

  • Transparency on agent outputs.

As several reviewers asked, annotators could view the coding agents' verdicts and explanations during annotation, but only as references. Each annotator labeled every claim based on their own inspection of the paper and code. Notably, all human-annotated fabrications were flagged by at least one of the two coding agents.

2. Reliability of the LLM judges in Table 2 and Figure 5 (7rfu, HH8c)

We added human validation on randomly sampled claims. Human judgments agreed with the LLM-assigned reason labels (Table 2) in 28 of 30 cases (93.3%), and with the LLM's AI-review coverage judgments (Figure 5) in 25 of 30 cases (83.3%).

3. Cost and design choices (7rfu)

We thank Reviewer 7rfu for raising the questions about cost and design choices, and we agree that reporting them makes our study more complete.

  • Cost.

On average, evaluating one paper with FabScore (using Claude Code as the backbone) takes about 21 minutes and costs about 3.31 USD. Since the process is fully automated, we believe this cost is practical for screening AI-generated papers at scale.

  • Static analysis vs. code execution.

To understand the role of code execution, we also ran FabScore using static analysis only. In this setting, FabScore still detects 1,135 fabrication claims across the 144 papers, which is 76.74% of those detected by the full pipeline. In other words, static analysis alone catches most fabrications, while code execution uncovers the remaining ones that cannot be identified by reading the code alone.

4. The definition of "fabrication" (7rfu, HH8c)

We agree that "fabrication" may suggest intent, and we will revise this word to describe the observed inconsistencies more precisely. "Hallucination" might be a better choice.

5. Evaluation on recent papers (qmmJ)

We agree with Reviewer qmmJ that evaluating more recent papers would make our findings more informative. We therefore evaluated two papers from CAISc 2026, the most recent AI-generated papers with code that we could find.

6. Contribution and related work (qmmJ, 7rfu)

Our main contribution is not technical novelty in the agentic framework, but (1) a tool for claim-level verification of AI-generated papers against their code, and (2) the empirical findings. We have clarified how FabScore differs from ReAgent, "The More You Automate, the Less You See", and MechEvalAgent, and we will add all suggested references with a discussion in the related work section.

We thank the reviewers again for the valuable feedback, which has substantially improved our work!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.