- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SciAudit tests whether models can read a full paper and find errors without being told where to look. Gold contains 2,938 issues from 293 papers, checked by human experts. Silver contains 5,441 errors that the authors introduced into 527 papers using models. Together, the two sets cover 16 scientific fields. Each error has a location, evidence, an explanation, and a type label.
The authors test 13 multimodal models. Their scores measure both whether a model finds an annotated error and how well it locates and explains it. They also ask models to check previously missed errors after giving their locations. Experts check samples of the annotations, and the authors compare results under different scoring models and ordinary precision, recall, and F1.
Critical strengths
-
Gold contains a useful collection of real errors. The authors collected 2,938 issues, had experts check them, and recorded where each issue occurs, why it is wrong, and what evidence supports that judgment. Models must read the full paper and find problems themselves. This resembles the error-finding part of human peer review. This dataset is my main reason for supporting acceptance.
-
The authors check both the annotations and the scoring. Experts review samples from Gold and Silver. The authors also change the model used for scoring, compare with ordinary F1, and check whether models are better at finding errors they generated themselves. The scoring-model and ordinary-metric comparisons give broadly similar results. These checks make the findings more convincing.
Critical weaknesses
I find the scoring rules broadly reasonable. I have two minor concerns, but I do not currently expect them to substantially affect the overall scores.
-
Splitting the same evidence may cost points. Suppose the reference puts two supporting facts in one evidence item, while the model lists them separately. Both contain the same information, with no repetition. Under Eq. (2), p. 6, the evidence score can still be about 0.707 out of 1. This is an example derived from the formula; how often it occurs in actual answers is unknown. The authors could clarify whether the scoring code merges such items before calculating the score. The broadly similar comparisons under ordinary F1 also reduce my concern about the effect of this detail.
-
Finding a real error missing from the references may lower Precision and F1. The audit on p. 9 found 3 such cases among 57 apparent false positives. However, the paper already reports Recall and Q-Recall. Adding an unmatched prediction does not directly lower those recall scores when the existing matches and their scores stay unchanged. This concern therefore applies mainly to how Precision and F1 treat valid new discoveries. The audit does not establish that the effect on overall scores is large.
These are details I would like the authors to clarify. They do not change my current recommendation of weak acceptance.
Questions for authors
-
Before applying Eq. (2), does the scoring code merge evidence items that contain the same information but are split differently?
-
After adding the valid errors missed by the annotations, how much do ordinary and quality-weighted Precision and F1 change? Do the model rankings change?
5 comments
W2. Finding a real error missing from the references may lower Precision and F1.
We agree that valid but unannotated errors are counted as false positives, possibly affecting Precision and F1. To quantify this effect, we added the three missed errors to the ground-truth errors set and recomputed the metrics for both models on the same 30 papers.
| Detector | Metric | Original | After Adding Missing Errors | Change |
|---|---|---|---|---|
| GPT-5.4 | Precision | 0.7636 | 0.7939 | +0.0303 |
| Q-Precision | 0.6287 | 0.6473 | +0.0186 | |
| Recall | 0.4773 | 0.4870 | +0.0097 | |
| Q-Recall | 0.3929 | 0.3989 | +0.0060 | |
| F1 | 0.5874 | 0.6037 | +0.0163 | |
| Q-F1 | 0.4836 | 0.4936 | +0.0100 | |
| Doubao-Seed-2.0-Pro | Precision | 0.7830 | 0.8019 | +0.0189 |
| Q-Precision | 0.6316 | 0.6482 | +0.0166 | |
| Recall | 0.3144 | 0.3160 | +0.0016 | |
| Q-Recall | 0.2536 | 0.2546 | +0.0010 | |
| F1 | 0.4486 | 0.4533 | +0.0047 | |
| Q-F1 | 0.3619 | 0.3656 | +0.0037 |
We draw two conclusions:
- Metric changes are limited. F1 and Q-F1 increase by at most 1.63% and 1.00%, while Precision and Q-Precision increase by at most 3.03% and 1.86%.
- The main conclusion remains unchanged. GPT-5.4 still outperforms Doubao-Seed-2.0-Pro across all four metrics.
Q1. Does the scoring code merge evidence items that contain the same information but are split differently?
Thank you for raising this question. We address it in our response to "W1".
Q2. Impact of valid errors missed by the annotations.
We appreciate this concern. Please refer to "W2", where this issue has already been addressed.
W1. Splitting the same evidence may cost points.
Our scoring code does not merge evidence items before evaluation. Therefore, splitting the same information across multiple items can lower the evidence score.
To minimize this issue, our prompt requires models to output concise, non-redundant evidence items with distinct support.
To quantify this effect, we sampled 50 papers, merged prediction evidence items that jointly matched one ground-truth evidence item, and recomputed all metrics. The table reports the corrected scores and their absolute increases over the original scores for "Thinking / Non-thinking" settings.
We draw the following conclusions: