acceptodds
Under review as a conference paper at ICLR 2027

Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers

Abstract

AI-generated content, often called AI slop, is increasingly common everywhere, particularly in academia. Slop in AI-generated scientific papers, however, has more complex patterns that cannot be easily detected by existing token-based AI detectors such as Pangram. Each part of such a paper looks plausible while the scientific reasoning that connects the parts breaks down, which can mislead how readers assess the work. We benchmark these failures as scientific slop through six measures across Structure, Argument, and Artifacts. We construct \benchmarkName with 390 AI-generated papers, mostly in computer science but spanning the life, social, and natural sciences, each paired with a human-written paper matched by research problem and contribution type. Our measures identify the AI paper in each pair with 85.9% accuracy, compared with 68.7% for Binoculars. Higher scientific slop accompanies lower ICLR ratings and distinguishes rejected from accepted papers above chance in every year from 2017 to 2025. Reducing these patterns, however, is not as simple as directly optimizing the measures. We therefore propose \MitigateName, a harness-level framework that guides a fixed LLM to revise slop only where the experiment records support the change. While standard revisions leave residual slop and direct slop-aware prompting triggers reward hacking, \MitigateName reduces the remaining AI–human gap by 63% over the strongest revision baseline without requiring human reference targets. Overall, we demonstrate that AI-generated scientific papers leave fundamental traces in their global reasoning, and that responsible mitigation demands strict evidentiary grounding rather than mere prose refinement.Our code and datasets are available at https://anonymous.4open.science/r/ScientificSlop-01EE

Primary area:
datasets and benchmarks
Keywords:
scientific slop, AI-generated paper detection, benchmark, agent skill file, iterative revision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WwHbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines scientific slop as recurring breakdowns in how scientific reasoning connects across a paper, and measures their occurrence. The paper proposes SciSlopBench, which pairs 390 AI-generated papers with accepted human-written papers, and reports that the aggregate scientific slop score separates the pairs better than Binoculars and automated reviewers (PairAcc 0.859 vs. 0.687). It also proposes SciSlopHarness, a revision method that reports a 63% smaller gap to the human means than the strongest revision baseline.

Critical strengths

S1. Evaluating the scientific quality of papers produced by LLM agents is timely and important.

S2. The benchmark construction is documented with care (pairing rules H1-H4 and the retrieval audit in Table 5).

S3. Reviewing each SciSlopHarness change against experiment outputs and code is a sensible guard against gaming the metric. Table 4 and Appendix F make the failure modes of slop-aware revision concrete.

Critical weaknesses

W1. All human anchors are accepted papers read from LaTeX, while 199 of 247 Agents4Science papers were rejected and 229 were reconstructed from PDF (Table 6). A relationship with paper quality is expected for a slop measure, but these differences make the AI-authorship detection result harder to interpret.

W2. The revisions are evaluated with the six measures used to guide them, although the editor receives no target score. Table 12 reports unchanged aggregate Argument graph and Figure exposition values for all four baselines. The preservation checks and examples are useful, but I could not find an independent assessment of overall revision quality. The 63% metric-gap reduction therefore does not by itself establish that the revised papers are better.

W3. Appendix B.4 and Table 8 explain how this benchmark differs from earlier datasets. However, Kim et al. [1] also detect generated text using discourse structure and are neither cited nor compared. Their features differ from the scientific reasoning measures here, but a comparison would help establish the added value of the work.

[1] Zae Myung Kim, Kwang Hee Lee, Preston Zhu, Vipul Raheja, Dongyeop Kang, "Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motif", ACL 2024.

Questions for authors

Q1. Could you clarify the novelty of STRUCTURE and ARGUMENT by identifying failures they capture beyond Kim et al.'s discourse features?

Q2. Could you report PairAcc using quality-matched human anchors, with the same extraction procedure applied to both groups?

Q3. Could blinded human evaluators compare SciSlopHarness revisions with the strongest baseline for reasoning quality and preservation of scientific content?

6 comments
AuthorsOpenReview ↗

Response to Reviewer WwHb (1/2)

We appreciate the careful review and the recognition of our benchmark construction and revision against experiment records. We address each point below.

[Weakness 1 / Q2] Manuscript Quality and Extraction Procedure

We respectfully clarify that the AI-human separation holds beyond the original manuscript-selection and extraction choices. We present alternative-reference and common-extraction comparisons below to support this conclusion.

(1) Manuscript Quality: Separation persists with rejected human references.

To assess whether manuscript quality associated with human-paper selection explains the separation, we kept all 390 AI papers fixed and replaced their references with nearest-title matches from 120 rejected ICLR 2026 papers screened at Pangram ≤ 0.05. Both comparisons use the same four deterministic measures. Fully AI-generated papers accepted at top-tier main tracks with author-disclosed provenance are not publicly available as a collection, so this control varies acceptance status on the human side. We will explicitly acknowledge this collection constraint and the potential manuscript-quality bias in Section 3.2.

Human referencesPairAcc
Original accepted papers0.877
Rejected ICLR 2026 papers0.797

Separation persists when the human references are rejected papers. The accepted status of the human references therefore does not by itself explain the separation.

(2) Extraction Procedure: Detection remains nearly unchanged under the same extraction procedure.

To isolate extraction effects, we additionally compared the original processing with PDF reconstruction applied to both papers in the Agents4Science subset. Each row uses the same pairs with valid scores under both routes.

MeasureOriginal PairAcc (same subset)Both-PDF PairAcc
Cross-section references0.8700.869
Citation isolation0.6980.691
Evidence gap0.7590.768
Macro redundancy0.5770.577

PairAcc changes by at most 0.009 across all four measures. The separation remains nearly unchanged under a common extraction procedure, supporting its robustness to the LaTeX–PDF processing difference. We will add these controls to Appendices B.1 and B.3.

[Weakness 2 / Q3] Revision Quality and Preservation of Scientific Content

We assess revision quality by whether edits strengthen scientific reasoning while preserving scientific content, beyond reducing the guiding measures. The blinded human evaluation below tests these criteria, followed by an examination of the unchanged baseline scores.

(1) [Additional Human Evaluation] Experts prefer SciSlopHarness for argumentation and flag fewer unsupported changes.

To strengthen the evidence beyond the preservation checks and examples the reviewer found useful, we recruited ten experts through Prolific. They assessed 28 flagged passages across all six slop types, comparing the original with SciSlopHarness and slop-aware prompting. They judged claim–evidence connections and unsupported changes with method names hidden. Results use majority vote.

Expert assessmentSciSlopHarnessSlop-aware prompting
Revision preference ↑96%4%
Unsupported changes ↓3%74%

The expert judgments provide aggregate evidence of stronger claim–evidence relationships and fewer unsupported changes. This supports revision quality on the evaluated passages independently of the six guiding measures. We will add this evaluation to §5.4 and Appendix F.

(2) [Additional Figure-Editing Control] Baselines improve with figure access, and SciSlopHarness retains the lowest reported score.

The original baselines edited text but not figures. To isolate the contribution of figure-editing access, we gave both baselines the same figure-editing tool. We ran three rounds with text revisions fixed.

MethodFigure exposition ↓
Slop-aware prompting + same tool0.169
Claude Code + same tool0.280
SciSlopHarness (Table 12 / Figure 5)0.114

All methods start at 0.289. The human mean is 0.025. SciSlopHarness achieves the lowest Figure exposition score even with equal tool access. This strengthens the evidence that its revision framework contributes beyond providing figure-editing capabilities. We will add this comparison to Appendix D.

AuthorsOpenReview ↗

Response to Reviewer WwHb (2/2)

[Weakness 3 / Q1] Added Value over Discourse Features

We thank the reviewer for pointing us to this valuable and relevant work. We will cite Kim et al. [1] in §2 and Appendix B.4. Below, we clarify how STRUCTURE and ARGUMENT differ from their discourse features and report the available comparison.

(1) Our detection signals identify specific scientific connections to inspect. Both approaches distinguish AI-generated from human-written text. Kim et al. use the frequency and depth of rhetorical patterns [1]. Our measures are defined by the scientific role of the connections being examined , giving the detection signal a specific interpretation.

Kim et al.STRUCTUREARGUMENT
FocusRhetorical patternsSection connectionsArgument connections
SignalsFrequency, depthRepetition, referencesSupport order, citations
Tracked unitsMotifsSentences, objectsClaims, citing sentences

Kim et al. model hierarchical rhetorical relations and use recurring discourse motifs to distinguish human and generated text. Our definition goes beyond the frequency and depth of these motifs to recurring failures in the connections needed to develop and support scientific claims. STRUCTURE and ARGUMENT make these failures measurable, linking authorship detection to the interpretation and repair of scientific reasoning connections. We will clarify this distinction in §2 and Appendix B.4.

(2) Additional comparison with discourse-based detection. We evaluated both released ToS checkpoints on all 390 SciSlopBench pairs using the Table 2 protocol. Plain uses text alone. Motif also uses discourse motifs extracted with the original pipeline's DMRST parser.

DetectorPairAccAUROC
ToS-Longformer-Plain0.4850.502
ToS-Longformer-Motif0.3230.327
Binoculars0.6870.683
Scientific slop (ours)0.8590.854

Scientific slop achieves the highest detection performance in this comparison. This supports the value of explicitly measuring scientific connections when detecting AI-generated papers. The ToS results evaluate transfer from short HC3/MAGE passages to full scientific papers. We will add both checkpoints to Table 2 and the comparison to Appendix B.4.

Reference

[1] Kim et al. “Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifs.” ACL 2024.

AuthorsOpenReview ↗

Response to Reviewer WwHb

Dear Reviewer WwHb,

We would like to sincerely thank you for the time to review our manuscript and for providing valuable comments and suggestions.

As the rebuttal phase will end soon, we just wanted to kindly check if our clarifications have fully resolved the points raised, or if any further discussion is needed from our side.

We would be truly grateful if you could take a moment to let us know your thoughts.

Thank you again for your support and your time.

Warm regards,

The Authors

Reviewer_QQLxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper organized several recurring issues in naturally generated papers into identifiable indicators, and integrated the results of this identification into an evidence-constrained revision process. It defines “scientific slop”, through six measures covering document structure, argumentation, and artifacts including SciSlopBench and SciSlopHarness.

Critical strengths
  1. The paper makes several recurring manuscript weaknesses explicit and inspectable. Its whole-paper evaluation captures structural information that prose-only detectors discard.
  2. SciSlopHarness provides a concrete mechanism for constraining revision by available evidence. Localized findings guide the editor, while change-level review can reject invented examples and preserve citations or reported results.
Critical weaknesses
  1. The measures are not sufficiently validated as indicators of “scientific reasoning quality”. A table can be fully explained within its own section, and a claim can legitimately precede its supporting explanation, but both may be penalized. Conversely, adding cross-references can improve a score without strengthening the evidence. Controlled tests separating harmless presentation changes from actual reasoning defects are needed.
  2. The detection benchmark confounds authorship with manuscript quality. Human references are accepted top-tier papers, while the AI corpus includes many rejected submissions. Matching topics and contribution types does not resolve this difference, so the reported accuracy may partly reflect manuscript maturity.
  3. The mitigation evaluation does not establish that the reported gains translate into better scientific reasoning. The 63% reduction measures distance to human feature means, but proximity to those means is not a validated measure of revision quality. The examples demonstrate useful repairs, while aggregate evidence of improved claim–evidence relationships remains limited.
Questions for authors
  1. What independent evidence establishes that the flagged instances are reasoning defects? It is suggested to add instance-level expert judgments or controlled comparisons that preserve scientific content while changing sentence order, refs or figure annotations.
  2. How do the discrimination results change when both sides use the same document-processing route and the analysis controls for manuscript quality? Please report results for the strictly matched subset, since Appendix B.3 relaxes the matching conditions for 62 Agents4Science papers.
  3. Why is acknowledging an evidence gap assigned 0.5 while human anchors retain the original scoring rule? How much of the reported improvement remains under a common scoring rule, and what independent evidence shows that retained edits improve reasoning while preserving correctness?
5 comments
AuthorsOpenReview ↗

Response to Reviewer QQLx (1/3)

We sincerely appreciate the careful review and the recognition of our inspectable whole-paper measures and evidence-constrained revision as meaningful contributions. We address each point below.

[Weakness 1 / Q1] Relationship Between the Measures and Scientific Reasoning Quality

We appreciate the opportunity to make clear that the measures’ relevance to scientific reasoning quality extends beyond presentation alone. We substantiate this through existing research, human review outcomes, and the additional instance-level expert judgments below.

(1) Section 3.1 already provides the rationale for the measures as indicators of scientific reasoning quality.

Section 3.1 already provides a research-based rationale for each of the six measures as an indicator of scientific reasoning quality. To make this rationale more explicit for the cross-section references raised by the reviewer, prior studies show that repairing missing textual connections and signaling text structure improved readers’ recall [1, 2]. We will add these studies [1, 2] and clarify the research foundations of all six measures in Section 3.1.

(2) Figure 4 already supports the measures’ relevance to scientific quality.

Scientific slop is the only evaluated method with significant negative correlations across all four ICLR 2026 review dimensions, including presentation, soundness, contribution, and overall rating (p < 0.05, Figure 4b). It also separates accepted from rejected papers above chance in every year from 2017 to 2025 (Figure 4c). These results support relevance to scientific soundness and contribution as well as presentation.

(3) Table 4 already shows revisions that improve argumentation while preserving scientific content.

SciSlopHarness verifies each edit against the manuscript and experiment records (§4.2, Appendix C.3). Unsupported changes are reverted, so retained revisions must preserve scientific content and use the available evidence. Table 4 illustrates how this process improves the scientific argument. These examples support the value of the repairs beyond reducing diagnostic scores.

(4) [Additional Expert Evaluation] Experts confirm that sampled flagged issues hinder scientific argumentation.

To independently assess whether the flagged issues represent reasoning problems, we recruited ten experts through Prolific. Each expert assessed 25 instances across all six slop types in a blinded evaluation. We used the majority vote. Expert judgments agreed with the audit’s problem/no-problem labels on 24 of 25 instances (96%). This independent evaluation strengthens the evidence that our diagnostics distinguish actual reasoning defects from harmless presentation changes in the sampled cases. We will add this evaluation to Section 5.4 and Appendix F.1.

References

[1] Britton and Gülgöz, "Using Kintsch's Computational Model to Improve Instructional Text: Effects of Repairing Inference Calls on Recall and Cognitive Structures." Journal of Educational Psychology 1991.

[2] Lorch et al., "Effects of Signaling Topic Structure on Text Recall." Journal of Educational Psychology 1993.

AuthorsOpenReview ↗

Response to Reviewer QQLx (2/3)

[Weakness 2 / Q2] Manuscript Quality and Document Processing

We respectfully clarify that the AI-human separation holds beyond the original manuscript-selection and processing choices. We present common-processing comparisons and strict-matching below to support this conclusion.

(1) Manuscript Quality and Matching.

(1-1) [Additional Selection Control] Separation persists with rejected human references.

To assess whether manuscript quality associated with human-paper selection explains the separation, we kept all 390 AI papers fixed and replaced their references with nearest-title matches from 120 rejected ICLR 2026 papers screened at Pangram ≤ 0.05. Both comparisons use the same four deterministic measures. Fully AI-generated papers accepted at top-tier main tracks with author-disclosed provenance are not publicly available as a collection, so this control varies acceptance status on the human side. We will explicitly acknowledge this collection constraint and the potential manuscript-quality bias in Section 3.2.

Human referencesPairAcc
Original accepted papers0.877
Rejected ICLR 2026 papers0.797

Separation persists when the human references are rejected papers. Together with the accepted-pair result, this shows that the signal extends across acceptance-status comparisons.

(1-2) [Strictly Matched Subset] Detection remains nearly unchanged under strict matching.

To address the matching concern, we restricted Agents4Science to the 185 pairs satisfying the strict criteria.

Agents4Science subsetPairAccAUROC
All 247 pairs0.8300.818
Strictly matched 185 pairs0.8320.816

Strict matching leaves detection performance nearly unchanged. The relaxed matches therefore do not drive the reported separation.

(2) Document Processing: Three measures retain separation under the same PDF reconstruction route.

To isolate processing effects, we compared the original processing with PDF reconstruction applied to both papers in the Agents4Science subset. Each row uses the same pairs with valid scores under both routes, rather than the full benchmark of Table 3.

MeasureOriginal PairAcc (same subset)Both-PDF PairAcc
Cross-section references0.8700.869
Citation isolation0.6980.691
Evidence gap0.7590.768
Macro redundancy0.5770.577

PairAcc changes by at most 0.009 across all four measures. The AI–human separation remains nearly unchanged when both papers undergo the same PDF reconstruction, supporting its robustness to document processing. We will add these controls to Appendices B.1 and B.3.

AuthorsOpenReview ↗

Response to Reviewer QQLx (3/3)

[Weakness 3 / Q3] Revision Quality and Claim Preservation

We present evidence that SciSlopHarness improves scientific argumentation while preserving claims, beyond reducing slop scores. We substantiate these improvements through the blinded human evaluation and common-scoring analysis presented below.

(1) [Additional Human Evaluation] Experts prefer SciSlopHarness for argumentation and flag fewer unsupported changes.

To strengthen the evidence beyond the repair examples the reviewer found useful, we recruited ten experts through Prolific to assess argumentation and claim preservation across 28 flagged passages covering all six slop types. They examined the original and both revisions, judging claim–evidence connections and unsupported changes. Method names were hidden, and results use majority vote.

Expert assessmentSciSlopHarnessSlop-aware prompting
Revision preference ↑96%4%
Unsupported changes ↓3%74%

The expert judgments provide aggregate evidence of stronger claim–evidence relationships and fewer unsupported changes. This supports SciSlopHarness’s revision quality beyond the reported reduction in distance to human-paper means. We will add this evaluation to §5.4 and Appendix F.

(2) [Common-Scoring Analysis] The improvement persists without disclosure credit.

To make the scoring comparison explicit, we applied the same disclosure rule to AI and human papers. Acknowledging missing evidence receives partial credit (0.5), while providing an example scores 0 and leaving the gap unstated scores 1. All 143 FARS human scores remain unchanged. We also recalculated the Table 12 improvement over Claude Code without disclosure credit.

Scoring ruleGap reduction
Reported rule63%
No disclosure credit for either method41%
Evidence gap excluded61%

The advantage persists when neither method receives disclosure credit and when Evidence gap is excluded. The scoring analysis establishes robustness to the 0.5 rule, while the expert evaluation directly supports argumentation and claim preservation. We will add the scoring analysis to Appendix C.5.

Reviewer_VtxmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces six diagnostics covering structural, argumentative, and artifact-related problems, evaluated on 390 AI–human paper pairs. It also proposes an editing pipeline that uses localized findings and experiment records, with a reviewer deciding whether individual edits should be retained or reverted.

Critical strengths

The diagnostics identify specific passages or artifacts, making them actionable. Using experiment records to constrain revisions and checking edits before retaining them are sensible attempts to prevent unsupported modifications.

Critical weaknesses

Several metrics operationalize stylistic conventions whose relationship to scientific quality is insufficiently validated. For example, cross-section references or concrete examples are not uniformly necessary across paper types. Differences in source-processing pipelines and paper selection may also influence AI–human comparisons. Furthermore, moving aggregate scores toward human-paper averages does not establish improved scientific quality. Unequal access to figure-editing capabilities complicates comparisons with the mitigation baselines.

Questions for authors

Can blinded human evaluation confirm that the edits improve scientific argumentation while preserving claims? Does the mitigation advantage persist when baselines receive comparable editing capabilities?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Vtxm (1/3)

We sincerely appreciate the careful review and the recognition of our localized diagnostics and record-grounded edit review as meaningful contributions. We address each point below.

[Weakness 1] Relationship Between the Measures and Scientific Quality

We respectfully clarify that the measures’ relationship to scientific quality is supported beyond stylistic conventions by the rationale in Section 3.1 and human review outcomes in Figure 4. Our additional paper-type analysis and expert judgments further assess whether the flagged issues matter for scientific argumentation in context.

(1) Section 3.1 already explains the measures’ relevance to scientific quality.

Section 3.1 grounds the measures in connections that help readers follow scientific reasoning and inspect its supporting evidence [1-3]. To make this rationale more explicit, research relevant to cross-section references shows that repairing missing textual connections and signaling text structure improved readers’ recall [4, 5]. For evidence gap, concrete text improved comprehension and recall across four text types [6]. These findings connect the measures to understanding and assessing scientific arguments beyond stylistic conventions. We will expand the rationale already presented in Section 3.1 with these additional studies [4-6].

(2) Figure 4 already supports the measures’ relevance to scientific quality.

Figure 4 already shows that scientific slop is the only evaluated method with significant negative correlations across all four ICLR 2026 review dimensions, including presentation, soundness, contribution, and overall rating (p < 0.05, Figure 4b). It also separates accepted from rejected papers above chance in every year from 2017 to 2025 (Figure 4c). These results link the measures to scientific soundness and contribution as well as presentation.

(3) [Additional Paper-Type Analysis] AI-human differences persist within matched contribution types.

To clarify whether differing needs across paper types explain the observed gaps, we additionally compare pairs of the same contribution type. This builds on our pairing criteria, which already account for research problem and contribution type (Appendix B.1).

Paper typeCross-section references, Human / AIEvidence gap, Human / AI
Method0.799 ± 0.009 / 0.958 ± 0.0050.396 ± 0.038 / 0.905 ± 0.023
Analysis0.758 ± 0.018 / 0.980 ± 0.0060.346 ± 0.095 / 0.962 ± 0.038
Framework0.789 ± 0.044 / 0.954 ± 0.0200.250 ± 0.250 / 0.750 ± 0.250
Benchmark0.761 ± 0.042 / 0.952 ± 0.0160.000 ± 0.000 / 0.500 ± 0.500

AI scores remain higher on both measures within each of the four contribution types. Thus, the difference persists when AI and human papers are compared within the same type, accounting for the reviewer’s concern about differing needs across types. We will add this comparison to Appendix B.

(4) [Additional Expert Evaluation] Experts confirm that sampled flagged issues hinder scientific argumentation.

We recruited ten experts through Prolific to assess whether the identified issues hinder scientific argumentation. Each expert assessed 25 instances in a blinded evaluation. We used the majority vote. Expert judgments agreed with the audit’s problem/no-problem labels on 24 of 25 instances (96%). This supports the measures’ ability to distinguish impediments to scientific argumentation from acceptable presentation choices in the sampled cases. We will add this to Section 5.4 and Appendix F.1.

References

[1] Mann and Thompson, "Rhetorical Structure Theory: Toward a Functional Theory of Text Organization." Text 1988.

[2] Hyland, "Metadiscourse." In Conducting Genre-Based Research in Applied Linguistics, Routledge 2023.

[3] Lipton and Steinhardt, "Troubling Trends in Machine Learning Scholarship." ACM Queue 2019.

[4] Britton and Gülgöz, "Using Kintsch's Computational Model to Improve Instructional Text: Effects of Repairing Inference Calls on Recall and Cognitive Structures." Journal of Educational Psychology 1991.

[5] Lorch et al., "Effects of Signaling Topic Structure on Text Recall." Journal of Educational Psychology 1993.

[6] Sadoski et al., "Engaging Texts: Effects of Concreteness on Comprehensibility, Interest, and Recall in Four Text Types." Journal of Educational Psychology 2000.

AuthorsOpenReview ↗

Response to Reviewer Vtxm (2/3)

[Weakness 2] Source Processing and Paper Selection

We appreciate the opportunity to clarify that our findings hold beyond the original source-processing and paper-selection choices. The processing and selection controls presented below support this conclusion.

(1) [Additional Processing Control] Mean scores remain stable after PDF reconstruction.

To test whether PDF reconstruction changes the scores, we scored the same human papers using their original LaTeX and versions reconstructed from PDF. The table reports the average slop scores.

MeasureLaTeX meanPDF mean
Citation isolation0.4070.426
Evidence gap0.3840.377
Macro redundancy0.0040.007
Cross-section references0.8020.823

Average scores change by no more than 0.021. The human reference averages therefore remain stable after PDF reconstruction.

(2) [Additional Selection Controls] Separation persists under alternative paper selection.

To test dependence on paper selection, we replaced matched human references with random or rejected papers and separately restricted Agents4Science to strictly matched pairs. Rejected references were nearest-title matches from 120 ICLR 2026 papers screened at Pangram ≤ 0.05, evaluated with the same four deterministic measures on both sides.

Selection controlOriginal PairAccControl PairAcc
Random human references, 1,000 draws0.8590.750
Rejected ICLR 2026 references, four measures0.8770.797
Strictly matched Agents4Science pairs, 247 → 185 pairs0.8300.832

AI papers remain distinguishable even when the human references are rejected papers. Random references also retain above-chance separation, while strict matching leaves accuracy nearly unchanged. Thus, the result does not depend on selecting accepted human papers or including loosely matched pairs. We will add the processing and selection controls to Appendix B.

[Weakness 3 / Q1] Revision Quality and Claim Preservation

We present the existing evidence and additional blinded evaluations below to show the relationship between revision quality and claim preservation.

(1) Section 5.4 already separates valid repairs from score reduction alone.

Section 5.4 already tests whether a lower score reflects a valid repair. In Table 4, slop-aware revision reaches a zero Macro redundancy score by deleting the repeated sentence together with four citations. SciSlopHarness removes only the repetition and keeps all four citations. Appendix F shows the same distinction for the remaining measures. A lower score therefore counts as an improvement in our evaluation only when the content is preserved. Accordingly, each edit is checked against the original manuscript and experiment records, with unsupported changes reverted (§4.2, Table 11), and the human means are never given as revision targets (§5.3).

(2) [Additional Human Evaluation] Experts prefer SciSlopHarness for argumentation and flag fewer unsupported changes.

To assess whether the edits improve scientific argumentation while preserving claims, we recruited ten experts through Prolific to evaluate 28 flagged passages across all six slop types. They examined the original and both revisions, judging claim-evidence connections and unsupported changes. Method names were hidden, and results use majority vote.

Expert assessmentSciSlopHarnessSlop-aware prompting
Revision preference ↑96%4%
Unsupported changes ↓3%74%

Experts judged the revised text directly, without slop scores. Their judgments support improved scientific argumentation with fewer unsupported changes on the evaluated passages, beyond movement toward human-paper averages.

AuthorsOpenReview ↗

Response to Reviewer Vtxm (3/3)

[Weakness 4 / Q2] Comparable Figure-Editing Capabilities

This is a valid concern, and we conducted a controlled experiment to answer the question. To isolate the contribution of figure-editing access, we gave both baselines the same figure-editing tool. We ran three rounds with text revisions fixed.

MethodFigure exposition ↓
Slop-aware prompting + same tool0.169
Claude Code + same tool0.280
SciSlopHarness (Table 12 / Figure 5)0.114

All methods start at 0.289. The human mean is 0.025. SciSlopHarness achieves the lowest Figure exposition score even with equal tool access. This strengthens the evidence that its revision framework contributes beyond providing figure-editing capabilities. We will add this comparison to Appendix D.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely appreciate the thoughtful feedback and insightful questions provided by all reviewers. We are particularly encouraged by the recognition of our work’s timely focus on scientific quality in AI-generated papers (Reviewer WwHb), the value of making scientific slop explicit and actionable through whole-paper diagnostics (Reviewers Vtxm, QQLx), and our evidence-constrained revision framework that checks individual changes against scientific records to guard against unsupported edits and metric gaming (Reviewers Vtxm, QQLx, WwHb).

Should there be any additional questions or clarifications needed, please let us know. If we have successfully addressed your concerns, we would greatly appreciate your consideration in improving your assessment of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.