acceptodds
Under review as a conference paper at ICLR 2027

SciAudit: Benchmarking MLLMs for Scientific Error Detection

Abstract

The rapid growth of scientific output makes it increasingly difficult for peer reviewers to examine every paper in depth. As a result, substantive errors or inconsistencies may be missed, leading to monetary, reputational, or scientific costs. Multimodal large language models (MLLMs) could assist scientific review, but existing error detection benchmarks remain limited in data scale, evaluation granularity, and task openness. We introduce SciAudit, a benchmark for open-ended scientific error detection in full papers. SciAudit comprises two complementary subsets spanning 16 scientific domains: Gold contains 2938 human-verified issues from 293 problematic papers, while Silver contains 5441 controlled errors constructed from 527 high-quality papers through a multi-model Draft–Check–Refine pipeline. Each error is annotated with its location, supporting evidence, reasoning, and type. These annotations support quality-aware metrics that assess error detection and the quality of the intermediate reasoning process. We evaluate 13 MLLMs under thinking and non-thinking settings. Results show that current MLLMs remain limited in open-ended error detection, with even the strongest model achieving less than 50% under our quality-aware F1 evaluation. Additional thinking generally improves performance, but errors requiring composite capabilities or multi-step reasoning remain challenging. Furthermore, we find that models can correctly verify 52.5%–81.3% of previously missed errors when explicitly prompted about them. This suggests that many detection failures stem from limitations in error discovery and reasoning, rather than insufficient knowledge alone. We hope SciAudit can provide a rigorous testbed for evaluating MLLMs and developing more reliable agent systems for scientific auditing in the future.

Primary area:
datasets and benchmarks
Keywords:
Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qagVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SciAudit tests whether models can read a full paper and find errors without being told where to look. Gold contains 2,938 issues from 293 papers, checked by human experts. Silver contains 5,441 errors that the authors introduced into 527 papers using models. Together, the two sets cover 16 scientific fields. Each error has a location, evidence, an explanation, and a type label.

The authors test 13 multimodal models. Their scores measure both whether a model finds an annotated error and how well it locates and explains it. They also ask models to check previously missed errors after giving their locations. Experts check samples of the annotations, and the authors compare results under different scoring models and ordinary precision, recall, and F1.

Critical strengths
  1. Gold contains a useful collection of real errors. The authors collected 2,938 issues, had experts check them, and recorded where each issue occurs, why it is wrong, and what evidence supports that judgment. Models must read the full paper and find problems themselves. This resembles the error-finding part of human peer review. This dataset is my main reason for supporting acceptance.

  2. The authors check both the annotations and the scoring. Experts review samples from Gold and Silver. The authors also change the model used for scoring, compare with ordinary F1, and check whether models are better at finding errors they generated themselves. The scoring-model and ordinary-metric comparisons give broadly similar results. These checks make the findings more convincing.

Critical weaknesses

I find the scoring rules broadly reasonable. I have two minor concerns, but I do not currently expect them to substantially affect the overall scores.

  1. Splitting the same evidence may cost points. Suppose the reference puts two supporting facts in one evidence item, while the model lists them separately. Both contain the same information, with no repetition. Under Eq. (2), p. 6, the evidence score can still be about 0.707 out of 1. This is an example derived from the formula; how often it occurs in actual answers is unknown. The authors could clarify whether the scoring code merges such items before calculating the score. The broadly similar comparisons under ordinary F1 also reduce my concern about the effect of this detail.

  2. Finding a real error missing from the references may lower Precision and F1. The audit on p. 9 found 3 such cases among 57 apparent false positives. However, the paper already reports Recall and Q-Recall. Adding an unmatched prediction does not directly lower those recall scores when the existing matches and their scores stay unchanged. This concern therefore applies mainly to how Precision and F1 treat valid new discoveries. The audit does not establish that the effect on overall scores is large.

These are details I would like the authors to clarify. They do not change my current recommendation of weak acceptance.

Questions for authors
  1. Before applying Eq. (2), does the scoring code merge evidence items that contain the same information but are split differently?

  2. After adding the valid errors missed by the annotations, how much do ordinary and quality-weighted Precision and F1 change? Do the model rankings change?

5 comments

W1. Splitting the same evidence may cost points.

Our scoring code does not merge evidence items before evaluation. Therefore, splitting the same information across multiple items can lower the evidence score.

To minimize this issue, our prompt requires models to output concise, non-redundant evidence items with distinct support.

To quantify this effect, we sampled 50 papers, merged prediction evidence items that jointly matched one ground-truth evidence item, and recomputed all metrics. The table reports the corrected scores and their absolute increases over the original scores for "Thinking / Non-thinking" settings.

DetectorQ-F1 (%)Q-Recall (%)Q-Precision (%)What (%)
Grok-4.320.30 (+0.18) / 7.59 (+0.04)12.93 (+0.11) / 5.38 (+0.03)57.20 (+0.53) / 18.10 (+0.08)64.83 (+2.25) / 24.64 (+0.47)
Claude-Sonnet-4.621.00 (+0.09) / 13.64 (+0.06)13.50 (+0.06) / 10.61 (+0.04)59.74 (+0.28) / 26.09 (+0.10)62.48 (+1.40) / 43.83 (+0.94)
Gemini-3.1-Pro28.71 (+0.11) / 23.90 (+0.08)21.77 (+0.07) / 17.21 (+0.06)47.22 (+0.22) / 44.42 (+0.12)62.90 (+0.99) / 62.34 (+0.88)
Qwen-3.6-Plus29.15 (+0.11) / 18.79 (+0.05)21.54 (+0.09) / 12.68 (+0.03)50.93 (+0.19) / 48.32 (+0.16)67.61 (+0.99) / 56.63 (+0.87)
Doubao-Seed-2.0-Pro29.96 (+0.17) / 30.33 (+0.08)21.51 (+0.13) / 24.18 (+0.06)56.16 (+0.28) / 46.14 (+0.12)62.11 (+2.08) / 64.74 (+0.73)
GPT-5.445.28 (+0.33) / 35.32 (+0.27)37.85 (+0.26) / 30.92 (+0.25)61.01 (+0.46) / 45.31 (+0.34)75.99 (+2.22) / 71.27 (+2.08)
GLM-4.6V5.66 (+0.03) / 5.84 (+0.02)3.56 (+0.02) / 3.36 (+0.01)20.59 (+0.13) / 27.91 (+0.14)19.90 (+0.60) / 23.41 (+0.54)
Mistral-Small-412.80 (+0.05) / 6.61 (+0.04)13.10 (+0.05) / 7.14 (+0.03)13.85 (+0.06) / 7.32 (+0.04)45.88 (+0.88) / 30.26 (+0.59)
Intern-S2-Preview22.50 (+0.06) / 8.54 (+0.07)16.11 (+0.04) / 5.53 (+0.05)43.02 (+0.12) / 22.36 (+0.19)61.98 (+0.51) / 37.71 (+1.48)
Gemma-4-31B-it28.26 (+0.16) / 13.89 (+0.04)21.02 (+0.12) / 8.73 (+0.03)51.61 (+0.28) / 40.47 (+0.14)63.41 (+1.55) / 45.42 (+0.90)
Qwen3.5-122B-A10B27.48 (+0.23) / 16.03 (+0.14)20.11 (+0.16) / 11.46 (+0.10)49.74 (+0.48) / 33.42 (+0.33)73.77 (+2.39) / 54.21 (+2.05)
Qwen3.5-27B26.72 (+0.16) / 20.36 (+0.14)20.81 (+0.13) / 14.00 (+0.10)42.88 (+0.24) / 43.41 (+0.27)64.29 (+1.73) / 61.85 (+2.13)

We draw the following conclusions:

  • Evidence splitting has a limited impact. After merging, What increases by 0.47–2.39 percentage points (pp), while Q-Precision, Q-Recall, and Q-F1 increase by at most 0.53, 0.26, and 0.33 pp.
  • Model rankings and key conclusions remain stable. GPT-5.4 remains the strongest detector, and the overall model rankings remain stable.

W2. Finding a real error missing from the references may lower Precision and F1.

We agree that valid but unannotated errors are counted as false positives, possibly affecting Precision and F1. To quantify this effect, we added the three missed errors to the ground-truth errors set and recomputed the metrics for both models on the same 30 papers.

DetectorMetricOriginalAfter Adding Missing ErrorsChange
GPT-5.4Precision0.76360.7939+0.0303
Q-Precision0.62870.6473+0.0186
Recall0.47730.4870+0.0097
Q-Recall0.39290.3989+0.0060
F10.58740.6037+0.0163
Q-F10.48360.4936+0.0100
Doubao-Seed-2.0-ProPrecision0.78300.8019+0.0189
Q-Precision0.63160.6482+0.0166
Recall0.31440.3160+0.0016
Q-Recall0.25360.2546+0.0010
F10.44860.4533+0.0047
Q-F10.36190.3656+0.0037

We draw two conclusions:

  • Metric changes are limited. F1 and Q-F1 increase by at most 1.63% and 1.00%, while Precision and Q-Precision increase by at most 3.03% and 1.86%.
  • The main conclusion remains unchanged. GPT-5.4 still outperforms Doubao-Seed-2.0-Pro across all four metrics.

Q1. Does the scoring code merge evidence items that contain the same information but are split differently?

Thank you for raising this question. We address it in our response to "W1".

Q2. Impact of valid errors missed by the annotations.

We appreciate this concern. Please refer to "W2", where this issue has already been addressed.

Reviewer_Z4vzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SciAudit, a benchmark for open-ended detection of objectively verifiable errors in full scientific papers by MLLMs, spanning 16 domains. Gold contains 2938 errors in 293 real papers that drew public criticism (PubPeer, Retraction Watch, OpenReview). 3 frontier MLLMs mined these errors from each paper and its reviews, and human experts then verified them. Silver contains 5441 errors injected into 527 high-quality 2026 papers through a multi-model Draft–Check–Refine pipeline. Each error is annotated with its location, evidence, reasoning and one of 9 types.

For evaluation, GPT-5.4 matches predictions to the ground truth, and each match is weighted by 4 quality scores (Where/What/Why/Which) to give Q-Precision, Q-Recall and Q-F1. The authors validate the data and scoring with human audits and judge-robustness checks. Across 13 MLLMs, the best model (GPT-5.4) reaches 38.7 Q-F1 on Gold and 47.6 on Silver. Thinking generally helps, and when given the location of an error they missed, models can verify 52.5 to 81.3% of those errors.

Critical strengths
  1. The real-error data is larger, broader and better curated than in prior open-ended benchmarks. Gold has 2938 human-verified errors in 293 real papers across 16 domains. SPOT, the only prior benchmark in Table 1 that is both open-ended and built from real errors, has 91. Experts rejected 27% of the LLM candidates, revised 1154 and added 395. Independent double-rated audits score at least 94% of Gold errors and 93% of Silver errors as valid.

  2. The finding that frontier MLLMs miss most known errors holds across three LLM judges, which give identical rankings with ICC ≈ 0.97. It also holds across Silver subsets split by the generating model, with no model gaining an advantage on its own errors (Table 11), and under conventional P/R/F1 (Table 13).

Critical weaknesses
  1. The paper does not show that its scores measure detection quality rather than agreement with references the evaluated models helped write. GPT-5.4 decides every match (true vs. false positive) and also supplies the inputs to the What and Why scores. The only validation is agreement among 3 LLM judges, which shows reliability, not validity, and no human–judge agreement is reported. The quality dimensions also reward similarity to a single reference rather than quality. Where is string overlap, so it is sensitive to formatting and OCR. What and Why penalize extra or alternative evidence and reasoning even when it is valid. Which depends on an ambiguous nine-way taxonomy. Because conventional metrics give the same conclusions, the value of the Q-weighting is not demonstrated. In addition, 86.6% of Gold errors began as candidates from GPT-5.4, Gemini-3.1-Pro or Qwen-3.6-Plus, and each paper got a single expert pass. There is no provenance analysis and no estimate of how complete Gold is, so valid errors outside this pool count as false positives.

  2. The claim that failures stem from discovery rather than knowledge rests on Table 4. Models are told the location of a known error, but are never shown locations without errors as a control (e.g., the 939 rejected Gold candidates). LLMs tend to confirm problems suggested to them, so the 52.5–81.3% "accuracy" cannot be interpreted. The claim that composite errors are hardest does not follow from Table 3. Composite deviations stay within 2.6 points with no uncertainty estimates, 8 of 12 models below their average is consistent with chance (sign test p at around 0.39), and Cognition errors are slightly harder on average. Table 3 and Figure 5 also pool Gold and Silver, although every model scores 1.2–1.6× higher on Silver.

Questions for authors
  1. On a random sample of gold–prediction pairs, how often do humans agree with GPT-5.4's match decisions? Do S1–S4 track human quality ratings better than binary matching does?
  2. How do models respond when given equally specific locations that contain no error (e.g., discarded Gold candidates, or Silver passages before injection)? Please report specificity and define "accuracy".
  3. How many Gold errors came from each mining model, and how many trace back to the public reviews? What is each detector's recall on the 395 expert-added errors? Please also break both subsets down by source and by the nine error types.
6 comments

W1. The paper fails to measure detection quality rather than agreement with references the evaluated models helped write.

(1) Human–judge agreement.

We randomly sampled 20 papers (10 Gold and 10 Silver) containing 171 ground-truth errors. Human Expert independently evaluated the predictions of four detectors, and we separately computed detector performance based on human and GPT-5.4 judgments. The relevant metrics are defined as follows:

  • GT-side match agreement: Human–GPT-5.4 agreement on matching results for each GT error.

  • Pre-side match agreement: Human–GPT-5.4 agreement on matching results for each model prediction.

  • Recall, Precision, F1: Detector performance based on human and GPT-5.4 judgments, reported as “human / GPT-5.4”.

DetectorGT-side match agreementPre-side match agreementRecallPrecisionF1
GPT-5.40.9880.9840.310 / 0.3180.411 / 0.4250.353 / 0.364
Doubao-Seed-2.0-Pro0.9880.9740.229 / 0.2420.506 / 0.5360.315 / 0.333
Qwen3.5-27B0.9830.9500.127 / 0.1430.380 / 0.4150.190 / 0.213
Qwen-3.6-Plus0.9650.8720.083 / 0.1140.275 / 0.3500.128 / 0.172

We draw the following conclusions:

  • GPT-5.4’s matching judgments are highly consistent with human judgments. The four detectors achieve 96.5%–98.8% GT-side agreement and 87.2%–98.4% prediction-side agreement.
  • The ranking of models stays the same under human evaluation. The rankings of Recall, Precision and F1 remain identical under both human and GPT-5.4 matching results.

W1. The paper fails to measure detection quality rather than agreement with references the evaluated models helped write.

(2) The quality dimensions reward similarity to a single reference rather than quality.

To test whether our four quality dimensions reflect quality, we conducted blinded human evaluations for "Where", "What", "Why", and "Which". Four volunteers, all unfamiliar with our evaluation metrics, were each assigned one dimension and evaluated randomly shuffled cases. The results show strong human–metric alignment across all four dimensions, with agreement increasing as score differences become larger.

For “Where,” “What,” and “Why,” each volunteer compared two matched model predictions, A and B, and selected the one closer to the ground truth without seeing their metric scores. The comparison was restricted to the relevant fields: error_text and error_location for Where, error_evidence for What, and error_reasoning for Why.

We sampled 100 comparisons per dimension (50 Gold and 50 Silver), stratified by score gap into low (<0.1), medium (0.1–0.2), and high (>0.2) groups containing 34, 33, and 33 cases, respectively. The results are as follows:

Score-gap groupHuman chose higher Where scoreHuman chose Higher What scoreHuman chose Higher Why score
Low79.4%58.8%55.9%
Medium87.9%72.7%69.7%
High97.0%81.8%84.8%
Overall88.0%71.0%70.0%

For "Which", the volunteer independently reassigned one of the nine error types to 100 ground-truth errors (50 Gold and 50 Silver) without seeing the original labels. The reassigned labels matched the benchmark labels in 87% of cases.

These results support two conclusions:

  • All four quality dimensions align with independent human judgment. Volunteers preferred the higher-scoring prediction in 88% of Where, 71% of What, and 70% of Why comparisons, while Which achieved 87% reannotation agreement.

  • Larger score differences indicate clearer quality differences. Where (79.4% → 97.0%), What (58.8% → 81.8%), Why (55.9% → 84.8%). This confirms valid metric ordering, with larger gaps corresponding to stronger human preference.

In response to your question, we offer the following clarifications about our metric design:

  • Where: The score uses the normalized longest common substring between error locations. Because all models use the same OCR text and must quote the erroneous segment, OCR and formatting effects are limited. The 88% human preference rate supports this measure.
  • What and Why: These are reference-grounded quality measures, not exhaustive measures of all valid evidence or reasoning. As shown in the preceding experimental results, higher scores align well with human preferences. This design also follows established evaluation practices, including ScholScan [Ref1], ROSCOE [Ref2], and FActScore [Ref3].
  • Which: Semantically matched errors receive a base score of 0.5, with another 0.5 awarded for an exact type match. The 87% reannotation agreement supports the reliability of the taxonomy.
  • Quality weighting: Weighting is important when models have similar detection rates. For example, Mistral-Small-4 has 1.3% higher Recall than Intern-S2-Preview (Table 13) but 0.4% lower Q-Recall (Table 2), indicating that it detects slightly more errors but with lower overall quality.

[Ref1] Li, Rongjin, et al. "Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning." The Fourteenth International Conference on Learning Representations. 2026.

[Ref2] Golovneva, Olga, et al. "Roscoe: A suite of metrics for scoring step-by-step reasoning." arXiv preprint arXiv:2212.07919 (2022).

[Ref3] Min, Sewon, et al. "FActScore: Fine-grained atomic evaluation of factual precision in long form text generation." Proceedings of the 2023 conference on empirical methods in natural language processing. 2023.

W1. The paper fails to measure detection quality rather than agreement with references the evaluated models helped write.

(3) The provenance analysis of Gold Errors.

We categorized all Gold errors by their source, with the source types defined below:

  • GPT, Qwen, and Gemini: Errors detected by GPT-5.4, Qwen-3.6-plus and Gemini-3.1-pro. These results are generated via multi-turn reasoning on papers and review comments, then verified by experts.
  • Review comments: Errors that can be traced back to the review comments.
  • Expert-added: Errors introduced by experts during the review process.

These source labels are non-exclusive since one error may come from multiple mining models or public reviews. The counts are shown in the table below.

GPTQwenGeminiReview commentsExpert-added
183013691412755395

The table below shows the Q-Recall (%) of each detector, reported as "thinking / non-thinking".

DetectorGPTQwenGeminiReview commentsExpert-added
Grok-4.312.21 / 5.7815.37 / 6.2113.19 / 5.4611.63 / 5.635.33 / 3.21
Claude-Sonnet-4.612.91 / 12.0215.56 / 13.6114.20 / 11.6011.16 / 10.643.64 / 7.04
Gemini-3.1-Pro17.81 / 15.7919.97 / 17.7430.30 / 26.7516.98 / 15.777.92 / 6.13
Qwen-3.6-Plus19.86 / 12.2034.21 / 19.5121.17 / 12.6517.88 / 10.517.68 / 6.40
Doubao-Seed-2.0-Pro20.78 / 19.6922.99 / 22.7319.99 / 19.7516.68 / 17.158.61 / 9.75
GPT-5.445.57 / 32.5734.38 / 25.9134.00 / 23.4430.93 / 22.1912.55 / 9.10
Llama-4-Maverick- / 1.57- / 2.10- / 1.75- / 2.09- / 0.11
GLM-4.6V4.34 / 3.875.21 / 4.414.93 / 4.024.97 / 4.832.04 / 1.83
Mistral-Small-413.42 / 9.6714.36 / 9.7313.07 / 8.4513.60 / 10.147.46 / 7.09
Intern-S2-Preview14.23 / 6.4817.12 / 7.5514.91 / 6.9813.20 / 7.866.62 / 3.82
Gemma-4-31B-it18.17 / 9.3520.85 / 10.8919.76 / 9.8117.47 / 8.938.70 / 3.57
Qwen3.5-122B-A10B19.12 / 12.6321.30 / 15.2018.21 / 12.8715.39 / 10.429.10 / 4.07
Qwen3.5-27B20.98 / 12.8423.91 / 16.4420.53 / 12.2717.42 / 11.3710.83 / 7.40

We provide the following clarifications:

  • Detectors often work better on errors they help find. For example, GPT-5.4 reaches 45.57% Q-Recall on errors mined by GPT, outperforming its results on errors from other sources. From the perspective of error distribution, this issue is inevitable: each model has unique detection preferences, and the errors it mines naturally follow the distribution it is good at identifying.
  • Even so, we need models to help collect errors. Finding and labeling all errors by hand is expensive. We aim to collect many expert-checked errors. This is a common approach for data construction. For instance, AutoBencher [Ref1] uses GPT-4 to make QA data and tests GPT-4-Turbo; SEED-Bench [Ref2] uses ChatGPT/GPT-4 to create multimodal questions before human review.
  • All other detectors perform steadily, except GPT-5.4, Qwen-3.6-plus and Gemini-3.1-pro. Therefore, this bias does not affect our evaluation of the remaining models.

[Ref1] LI, XIANG, et al. "Autobencher: Towards declarative benchmark construction." International Conference on Learning Representations. Vol. 2025.

[Ref2] Li, Bohao, et al. "Seed-bench: Benchmarking multimodal large language models." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

Reviewer_BHomOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SciAudit, a benchmark for open-ended scientific error detection in full papers across 16 scientific domains, with a Gold subset of human-verified issues mined from real papers and their public reviews and a Silver subset of controlled injected errors. Predictions are scored with a quality-aware evaluation framework. Across diverse frontier MLLMs the strongest model stays under 50% Q-F1, and the paper argues that failures stem from discovery rather than insufficient knowledge.

Critical strengths

S1. The task setup is solid. Auditing a whole paper with no predefined inspection direction is much closer to real reviewing than question–answer formats, and the scale (8,379 annotations from 820 papers) is well beyond the other real-error benchmarks in Table 1.

S2. The framework is well described. Construction and evaluation prompts in Appendix A and two-evaluator quality audits for both subsets.

S3. The evaluator sensitivity check is solid. Three evaluator models give a consistent Q-F1 ICC of 0.970 and the same detector ranking.

Critical weaknesses

W1. My biggest concern is that gold's error selection may favor the models used to mine candidates. Three evaluated models propose the initial issues (Section 3.2). Experts verify and revise these candidates and add 395 missed errors, which provides valuable quality control. However, the paper does not show whether the mining models perform better on their own proposed errors. A breakdown by candidate source, including expert-added errors, would help assess this possible bias.

W2. The Table 4 tests only known errors, without matched non-error locations. The explanation scores provide useful evidence, but verification accuracy could still be inflated by a tendency to agree with the cue. Adding non-error cases would strengthen the claim that models can verify errors they fail to discover.

Questions for authors

Q1. Section 3.1 organizes its 9 error types following ScholScan. Could you clarify the novelty of SciAudit's construction and quality-aware evaluation relative to ScholScan?

Q2. Section 3.2 states that Gold candidates are mined by GPT-5.4, Gemini-3.1-Pro and Qwen-3.6. Could you report each detector's Gold Q-Recall split by mining model?

Q3. How scalable is SciAudit to larger evaluation sets, measured by expert review hours for Gold and model inference cost per paper?

6 comments

W1. Gold's error selection may favor the models used to mine candidates.

We categorized all Gold errors by their source, with the source types defined below:

  • GPT, Qwen, and Gemini: Errors detected by GPT-5.4, Qwen-3.6-plus and Gemini-3.1-pro. These results are generated via multi-turn reasoning on papers and review comments, then verified by experts.
  • Review comments: Errors that can be traced back to the review comments.
  • Expert-added: Errors introduced by experts during the review process.

These source labels are non-exclusive since one error may come from multiple mining models or public reviews. The counts are shown in the table below.

GPTQwenGeminiReview commentsExpert-added
183013691412755395

The table below shows the Q-Recall (%) of each detector, reported as "thinking / non-thinking".

DetectorGPTQwenGeminiReview commentsExpert-added
Grok-4.312.21 / 5.7815.37 / 6.2113.19 / 5.4611.63 / 5.635.33 / 3.21
Claude-Sonnet-4.612.91 / 12.0215.56 / 13.6114.20 / 11.6011.16 / 10.643.64 / 7.04
Gemini-3.1-Pro17.81 / 15.7919.97 / 17.7430.30 / 26.7516.98 / 15.777.92 / 6.13
Qwen-3.6-Plus19.86 / 12.2034.21 / 19.5121.17 / 12.6517.88 / 10.517.68 / 6.40
Doubao-Seed-2.0-Pro20.78 / 19.6922.99 / 22.7319.99 / 19.7516.68 / 17.158.61 / 9.75
GPT-5.445.57 / 32.5734.38 / 25.9134.00 / 23.4430.93 / 22.1912.55 / 9.10
Llama-4-Maverick- / 1.57- / 2.10- / 1.75- / 2.09- / 0.11
GLM-4.6V4.34 / 3.875.21 / 4.414.93 / 4.024.97 / 4.832.04 / 1.83
Mistral-Small-413.42 / 9.6714.36 / 9.7313.07 / 8.4513.60 / 10.147.46 / 7.09
Intern-S2-Preview14.23 / 6.4817.12 / 7.5514.91 / 6.9813.20 / 7.866.62 / 3.82
Gemma-4-31B-it18.17 / 9.3520.85 / 10.8919.76 / 9.8117.47 / 8.938.70 / 3.57
Qwen3.5-122B-A10B19.12 / 12.6321.30 / 15.2018.21 / 12.8715.39 / 10.429.10 / 4.07
Qwen3.5-27B20.98 / 12.8423.91 / 16.4420.53 / 12.2717.42 / 11.3710.83 / 7.40

We provide the following clarifications:

  • Detectors often work better on errors they help find. For example, GPT-5.4 reaches 45.57% Q-Recall on errors mined by GPT, outperforming its results on errors from other sources. From the perspective of error distribution, this issue is inevitable: each model has unique detection preferences, and the errors it mines naturally follow the distribution it is good at identifying.
  • Even so, we need models to help collect errors. Finding and labeling all errors by hand is expensive. We aim to collect many expert-checked errors. This is a common approach for data construction. For instance, AutoBencher [Ref1] uses GPT-4 to make QA data and tests GPT-4-Turbo; SEED-Bench [Ref2] uses ChatGPT/GPT-4 to create multimodal questions before human review.
  • All other detectors perform steadily, except GPT-5.4, Qwen-3.6-plus and Gemini-3.1-pro. Therefore, this bias does not affect our evaluation of the remaining models.

Thank you for this suggestion. We will include this experiment in the final manuscript to help readers better interpret our findings.

[Ref1] LI, XIANG, et al. "Autobencher: Towards declarative benchmark construction." International Conference on Learning Representations. Vol. 2025.

[Ref2] Li, Bohao, et al. "Seed-bench: Benchmarking multimodal large language models." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

W2. The Table 4 tests only known errors, without matched non-error locations.

We add non-error cases and reran the verification experiment (Table 4 in the paper) on the same 50 papers. For Gold, non-error cases were drawn from discarded candidate errors and passages manually verified as correct. For Silver, non-error cases were taken from the original text before error injection and then manually verified. We sampled 8 non-error cases per paper, for a total of 400 cases.

In this experiment, erroneous cases are the positive class and non-error cases are the negative class. The relevant metrics are defined as follows:

  • True-positive rate (TPR): The proportion of erroneous cases correctly identified as errors.

  • True-negative rate (TNR): The proportion of non-error cases correctly identified as non-errors.

  • “Which,” “What,” and “Why”: The dimension scores calculated only for true-positive predictions.

ModelError casesNon-error casesTPR (%)TNR (%)Which (%)What (%)Why (%)
GPT-5.432740080.1291.5086.2774.9968.17
Doubao-Seed-2.0-Pro39440053.8195.0085.3871.1964.50
Qwen-3.6-Plus40040052.5092.7585.0368.7868.95
Qwen3.5-27B40440056.4494.7585.9468.3960.76

The models rejected 91.50%–95.00% of non-error cases while verifying 52.50%–80.12% of previously missed errors. This indicates that many missed errors are within the models’ ability to recognize but are easily overlooked during autonomous detection.

We appreciate this suggestion. We will add this experiment to the final manuscript to help readers better interpret our findings.

Q1. The novelty of SciAudit's construction and quality-aware evaluation relative to ScholScan.

Thank you for this suggestion. We will clarify the differences between SciAudit and SholScan by examining their data construction and evaluation frameworks.

(1) Data Construction

SciAudit provides full-text error annotation. SciAudit targets open-ended scientific error detection across full papers, using a prompt such as: Please help me detect scientific errors for the paper. Therefore, SciAudit provides as detailed an annotation as possible for the various types of errors found throughout a paper. In comparison, ScholScan performs error detection within predefined scopes of a paper, using a prompt such as: Assess the Methods section for Measurement & Operationalization issues. Its annotations are bounded by pre-defined task scopes and do not cover full papers.

SciAudit is larger and more diverse. SciAudit adopts a two-subset design: the Gold contains genuine errors from original papers, while the Silver includes supplementary controlled injected errors, and the two subsets complement each other. ScholScan comprises only a single expert-reviewed subset. Meanwhile, Table 1 in paper shows that SciAudit features a larger scale and more diverse data sources.

(2) Evaluation Framework

SciAudit extends quality assessment to open-ended detection tasks. ScholScan uses a QA-style evaluation that scores errors, evidence and reasoning in single answers. This metric does not fit open-ended detection tasks, as its hyperparameters and penalties are built for QA and cannot be directly reused. Thus, SciAudit adopts a new fine-grained scoring formula based on recall, precision and F1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.