- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper defines scientific slop as recurring breakdowns in how scientific reasoning connects across a paper, and measures their occurrence. The paper proposes SciSlopBench, which pairs 390 AI-generated papers with accepted human-written papers, and reports that the aggregate scientific slop score separates the pairs better than Binoculars and automated reviewers (PairAcc 0.859 vs. 0.687). It also proposes SciSlopHarness, a revision method that reports a 63% smaller gap to the human means than the strongest revision baseline.
Critical strengths
S1. Evaluating the scientific quality of papers produced by LLM agents is timely and important.
S2. The benchmark construction is documented with care (pairing rules H1-H4 and the retrieval audit in Table 5).
S3. Reviewing each SciSlopHarness change against experiment outputs and code is a sensible guard against gaming the metric. Table 4 and Appendix F make the failure modes of slop-aware revision concrete.
Critical weaknesses
W1. All human anchors are accepted papers read from LaTeX, while 199 of 247 Agents4Science papers were rejected and 229 were reconstructed from PDF (Table 6). A relationship with paper quality is expected for a slop measure, but these differences make the AI-authorship detection result harder to interpret.
W2. The revisions are evaluated with the six measures used to guide them, although the editor receives no target score. Table 12 reports unchanged aggregate Argument graph and Figure exposition values for all four baselines. The preservation checks and examples are useful, but I could not find an independent assessment of overall revision quality. The 63% metric-gap reduction therefore does not by itself establish that the revised papers are better.
W3. Appendix B.4 and Table 8 explain how this benchmark differs from earlier datasets. However, Kim et al. [1] also detect generated text using discourse structure and are neither cited nor compared. Their features differ from the scientific reasoning measures here, but a comparison would help establish the added value of the work.
[1] Zae Myung Kim, Kwang Hee Lee, Preston Zhu, Vipul Raheja, Dongyeop Kang, "Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motif", ACL 2024.
Questions for authors
Q1. Could you clarify the novelty of STRUCTURE and ARGUMENT by identifying failures they capture beyond Kim et al.'s discourse features?
Q2. Could you report PairAcc using quality-matched human anchors, with the same extraction procedure applied to both groups?
Q3. Could blinded human evaluators compare SciSlopHarness revisions with the strongest baseline for reasoning quality and preservation of scientific content?
6 comments
Response to Reviewer WwHb (2/2)
[Weakness 3 / Q1] Added Value over Discourse Features
We thank the reviewer for pointing us to this valuable and relevant work. We will cite Kim et al. [1] in §2 and Appendix B.4. Below, we clarify how STRUCTURE and ARGUMENT differ from their discourse features and report the available comparison.
(1) Our detection signals identify specific scientific connections to inspect. Both approaches distinguish AI-generated from human-written text. Kim et al. use the frequency and depth of rhetorical patterns [1]. Our measures are defined by the scientific role of the connections being examined , giving the detection signal a specific interpretation.
| Kim et al. | STRUCTURE | ARGUMENT | |
|---|---|---|---|
| Focus | Rhetorical patterns | Section connections | Argument connections |
| Signals | Frequency, depth | Repetition, references | Support order, citations |
| Tracked units | Motifs | Sentences, objects | Claims, citing sentences |
Kim et al. model hierarchical rhetorical relations and use recurring discourse motifs to distinguish human and generated text. Our definition goes beyond the frequency and depth of these motifs to recurring failures in the connections needed to develop and support scientific claims. STRUCTURE and ARGUMENT make these failures measurable, linking authorship detection to the interpretation and repair of scientific reasoning connections. We will clarify this distinction in §2 and Appendix B.4.
(2) Additional comparison with discourse-based detection. We evaluated both released ToS checkpoints on all 390 SciSlopBench pairs using the Table 2 protocol. Plain uses text alone. Motif also uses discourse motifs extracted with the original pipeline's DMRST parser.
| Detector | PairAcc | AUROC |
|---|---|---|
| ToS-Longformer-Plain | 0.485 | 0.502 |
| ToS-Longformer-Motif | 0.323 | 0.327 |
| Binoculars | 0.687 | 0.683 |
| Scientific slop (ours) | 0.859 | 0.854 |
Scientific slop achieves the highest detection performance in this comparison. This supports the value of explicitly measuring scientific connections when detecting AI-generated papers. The ToS results evaluate transfer from short HC3/MAGE passages to full scientific papers. We will add both checkpoints to Table 2 and the comparison to Appendix B.4.
Reference
[1] Kim et al. “Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifs.” ACL 2024.
Response to Reviewer WwHb
Dear Reviewer WwHb,
We would like to sincerely thank you for the time to review our manuscript and for providing valuable comments and suggestions.
As the rebuttal phase will end soon, we just wanted to kindly check if our clarifications have fully resolved the points raised, or if any further discussion is needed from our side.
We would be truly grateful if you could take a moment to let us know your thoughts.
Thank you again for your support and your time.
Warm regards,
The Authors
Response to Reviewer WwHb (1/2)
We appreciate the careful review and the recognition of our benchmark construction and revision against experiment records. We address each point below.
[Weakness 1 / Q2] Manuscript Quality and Extraction Procedure
We respectfully clarify that the AI-human separation holds beyond the original manuscript-selection and extraction choices. We present alternative-reference and common-extraction comparisons below to support this conclusion.
(1) Manuscript Quality: Separation persists with rejected human references.
To assess whether manuscript quality associated with human-paper selection explains the separation, we kept all 390 AI papers fixed and replaced their references with nearest-title matches from 120 rejected ICLR 2026 papers screened at Pangram ≤ 0.05. Both comparisons use the same four deterministic measures. Fully AI-generated papers accepted at top-tier main tracks with author-disclosed provenance are not publicly available as a collection, so this control varies acceptance status on the human side. We will explicitly acknowledge this collection constraint and the potential manuscript-quality bias in Section 3.2.
Separation persists when the human references are rejected papers. The accepted status of the human references therefore does not by itself explain the separation.
(2) Extraction Procedure: Detection remains nearly unchanged under the same extraction procedure.
To isolate extraction effects, we additionally compared the original processing with PDF reconstruction applied to both papers in the Agents4Science subset. Each row uses the same pairs with valid scores under both routes.
PairAcc changes by at most 0.009 across all four measures. The separation remains nearly unchanged under a common extraction procedure, supporting its robustness to the LaTeX–PDF processing difference. We will add these controls to Appendices B.1 and B.3.
[Weakness 2 / Q3] Revision Quality and Preservation of Scientific Content
We assess revision quality by whether edits strengthen scientific reasoning while preserving scientific content, beyond reducing the guiding measures. The blinded human evaluation below tests these criteria, followed by an examination of the unchanged baseline scores.
(1) [Additional Human Evaluation] Experts prefer SciSlopHarness for argumentation and flag fewer unsupported changes.
To strengthen the evidence beyond the preservation checks and examples the reviewer found useful, we recruited ten experts through Prolific. They assessed 28 flagged passages across all six slop types, comparing the original with SciSlopHarness and slop-aware prompting. They judged claim–evidence connections and unsupported changes with method names hidden. Results use majority vote.
The expert judgments provide aggregate evidence of stronger claim–evidence relationships and fewer unsupported changes. This supports revision quality on the evaluated passages independently of the six guiding measures. We will add this evaluation to §5.4 and Appendix F.
(2) [Additional Figure-Editing Control] Baselines improve with figure access, and SciSlopHarness retains the lowest reported score.
The original baselines edited text but not figures. To isolate the contribution of figure-editing access, we gave both baselines the same figure-editing tool. We ran three rounds with text revisions fixed.
All methods start at 0.289. The human mean is 0.025. SciSlopHarness achieves the lowest Figure exposition score even with equal tool access. This strengthens the evidence that its revision framework contributes beyond providing figure-editing capabilities. We will add this comparison to Appendix D.