acceptodds
Under review as a conference paper at ICLR 2027

Lazy Literature Review: Quantifying Sloppiness in LLM Research Agents

Abstract

LLM agents are increasingly trusted to do research: this includes compiling evidence from a corpus of literature and making recommendations or decisions about further steps. A recognised weakness of such agents is sloppiness, work that is left incomplete but reported as finished. In evidence synthesis this is hard to notice, because a fluent, well-cited answer looks the same whether or not the evidence supports it. We set out to quantify this sloppiness, and chose one failure mode that can be verified exactly: treating results as comparable when they are not. We build synthetic corpora of literature in which the studies bearing on a contested claim measure different things, so their numbers cannot be combined and the right conclusion is known by construction. We score whether an agent reaches a conclusion without noticing the incompatibility and, separately, whether it uses the incompatible numbers in its conclusion at all. Across twelve frontier models, most notice the problem and pool the numbers anyway: some name the incompatibility in over 90% of answers and still conflate in a third to a half. Only the strongest model, GPT-6 Astra, avoids the error on our synthetic literatures. On real arXiv papers whose results rest on incompatible protocols, every model, Astra included, still ranks by the incompatible numbers. Beyond the results, we propose the construction and scoring methodology as a general recipe for studies of this kind, and release the protocol and example corpora.

Primary area:
foundation or frontier models, including LLMs
Keywords:
autoresearch, agents, slop, conflation, synthetic, benchmark, lazyness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bUvEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work studies LLMs/agents' sloppiness, i.e., work that is left incomplete but reported as finished, with a focus on one type of failure mode in literature analysis where the LLM/agent draws a conclusion by mixing numerical results that are not directly comparable. To quantify LLMs/agents' behavior on this failure mode, this paper synthesizes a literature corpus where pairs of papers arrive at contradicting conclusions about a claim because their measurements and results are different by definition. With additional literature sourced from real papers, the experiment on 12 frontier models reveals that most of the tested models exhibit sloppiness in the manner that although they are able to identify the differences in the measurement but still mixes/pools/compares them.

Critical strengths
  1. The paper studies a timely and important research topic on LLM/agent sloppiness.
  2. The authors propose an interesting and valuable angle for controllable and quantitative study of the sloppiness problem through literature analysis of contested claims with different measurements.
  3. The data synthesis pipeline provides a scalable scaffold for automatic construction of testbeds for sloppiness in literature review.
Critical weaknesses
  1. Some parts of the presentation are confusing. For instance, it is hard to follow Section 3.2 and gain a clear understanding of how the literature generation process is conducted. And for Section 3.4, there is a lack of elaboration on how the findings from Fable experiments can be extended to a general setting for challenging task construction.
  2. Lack of constructive experiments and findings. While the analysis is thorough, they mostly only describe how different models fail under this setup. There is a lack of proposals for constructive approaches with corresponding experiments on what might be useful methods that can help mitigate this kind of sloppiness issues.
Questions for authors

Please see the weaknesses above.

1 comment

Thank you for the review and for recognising the controlled angle and the generation pipeline. We address the two weaknesses below.

Presentation of Sec. 3.2 (literature generation). The generation process, step by step (we have clarified the wording of Sec. 3.2 along these lines):

  1. Topic specification. A claim about an intervention (e.g. "text-message reminders reduce wasted clinic appointments"), eight sites with their effect values, and two defensible ways of counting the outcome (e.g. no-shows vs. slots written off as unused). Four sites report a large effect and four almost none, and this split coincides with the counting split.
  2. Writing the site papers. A generator model writes each site's evaluation in full, leaving a placeholder for the one sentence in Methods that states how the outcome was counted.
  3. Rendering two mirrored versions. The placeholder is filled in two ways: in one version the large-effect sites use the first counting rule, in the other the second. Nothing else differs, so a model that follows the surface numbers is wrong in one of the two versions.
  4. Supporting papers. Eight methodology papers show that ordinary nuisance factors (reporting delays, boundary rules, record linkage) cannot produce the gap; in the agentic setting 32 adjacent documents on related programmes are added, giving a 48-document folder.
  5. Quality control. Automatic checks that no definition appears anywhere else in a paper, a model review of every site paper with regeneration on failure, and a final audit.

Sec. 3.4 and how it generalises to building hard tasks. The general lesson, which we now state in one sentence at the end of Sec. 3.4: adding more documents did not make the task hard for frontier models; making the wrong step the natural route did. Two ingredients were needed: (i) no single source is decisive on its own (wide confidence intervals in each site paper), and (ii) the material presents the wrong step, here pooling the eight results, as the obvious next move. The package experiment (Sec. 4.3) tests these separately: Fable 5.1 never conflates with narrow intervals, even when pooling is invited, and conflates in 6 of 12 answers when both are present; Opus 5 is already pushed by the pooling prose alone (7 of 12) and fails in 11 of 12 with both. For other evaluations of a "notice and respect" skill, the recipe is: plant one checkable flaw, make the shortcut that ignores it necessary to finish the task, and confirm on separate packages that strong models fail on the combination, not on each part alone.

Constructive findings. The paper's aim is to measure this failure and show that noticing a problem does not mean acting on it, so we did not set out to engineer a mitigation. The results nevertheless already identify levers:

  • More reasoning helps some models and not others. In the agentic setting, high effort cuts conflation for Opus 5.5 from 31 to 3 of 60 answers, for GPT-5.6 Sol from 28 to 12 and for Gemini 3.8 Flash from 24 to 15, and leaves Fable 5.1, GPT-5.5, GLM 5.3 and Kimi K3 unchanged or worse (Sec. 4.2, effort figure).
  • Allowing "not established" protects some models. When the question permits the answer that the evidence does not settle the claim, conflation falls for GPT-5.6 Sol from 70 to 23 of 120 answers, for GPT-5.5 from 77 to 44 of 119 and for Opus 5 from 20 to 7 of 101, with no effect on the others (Appendix question modes).
  • Framing that invites aggregation is a risk factor. The package experiment shows which pressures cause the failure, which tells both task designers and users what to avoid.
  • Model choice matters but is not enough. GPT-6 Astra never conflates on our synthetic literatures, yet on real papers every model, Astra included, still ranks by incompatible numbers.
Reviewer_nppFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether research agents combine findings measured under incompatible outcome definitions. It evaluates 12 models using synthetic literature collections, including mirrored versions that preserve numerical results while swapping measurement definitions, and extends the evaluation to real-paper collections. Its central finding is that recognizing an incompatibility does not necessarily prevent an agent from using the incompatible evidence in its final recommendation.

Critical strengths

The distinction between identifying a methodological caveat and actually respecting it is useful and practically relevant. The mirrored design provides strong experimental control, while the real-literature experiments broaden the evidence beyond synthetic examples.

Critical weaknesses

Topics were selected partly through pilot failures, so the reported rates should be interpreted as performance on a targeted challenge set rather than estimates of everyday research-agent reliability. Evaluation also relies heavily on LLM judgments. Although a small human audit is provided, broader independent validation would strengthen confidence, particularly for ambiguous conflation cases and the real-literature experiments.

Questions for authors

Can the authors provide independent human validation of the real literature judgments, including cases excluded because automated judges disagreed?

1 comment

Thank you for the careful review. We address the two concerns in turn and have added the new results to the revised PDF (Appendices K and L).

Topic selection. We agree that the rates describe a challenge set, as Sec. 3.1 states. To be concrete about how the topics were chosen:

  1. Several frontier models generated candidate claims, each with two ways of counting its outcome.
  2. We screened the candidates inline with Opus 4.8, the most capable model in our pilots at the time, and ranked them by how often it missed the counting split.
  3. To avoid tuning the benchmark on its own test topics, we took the top-ranked topics as a development set and built the agentic setup on them.
  4. We then dropped the development set entirely, returned to the next best-ranked topics, and kept those that trapped Fable 5.1, the strongest model we could still trap at the time, in short agentic pilots.

Our goal was a challenging benchmark for a failure that is otherwise hard to pin down, so we curated the cases to surface it clearly. The rates show which agents fall for it and how often, which is worth keeping in mind when using an agent for a literature review and when choosing one. These comparisons do not depend on our choice: when we instead took the topics on which a different model (GPT-5.6 Sol) failed most often and ran all models on them, the models ranked in the same order (rank correlation 0.93).

Human review. We added a human review of each place where the paper relies on model judgments.

1. Real-literature reference answers (17 setups reviewed). In the real-literature setting (Sec. 4.5) each setup is a set of 12 real arXiv papers and a buyer's question; in 17 of the 21 setups our reference answer states that no approach can be declared best, because the headline numbers rest on different evaluation protocols (e.g. different FID sample counts or reference statistics). Each of these 17 reference answers was checked by hand against the papers, using a per-paper table of the relevant evaluation settings in which every entry links to the supporting passage. All 17 were confirmed.

2. Real-literature judgments, including the excluded answers (42 answers reviewed). Each of the 170 answers in those 17 setups (10 models ×\times 17) is graded on three questions: did it name the specific protocol mismatch, did it put forward a winner, and did its recommendation rest on numbers from incompatible protocols. Two model judges from different families (GPT-6 Astra and Claude Fable 5.1) grade every answer, and the paper reports a label only where both agree, which excluded 4, 13 and 17 answers respectively (28 distinct answers). Human review labelled all 28 on the question the judges disagreed on, and 14 answers on which they agreed (one or two per model) on all three questions. On the agreed answers it matched the judges on 39 of 42 labels. Replacing each excluded label by the human label gives rates over all 170 answers:

Paper (agreed answers only)All 170 answers, excluded cases labelled by human review
Named the protocol mismatch133/166 (80%)133/170 (78%)
Put forward a winner151/157 (96%)161/170 (95%)
Relied on numbers from incompatible protocols133/153 (87%)146/170 (86%)

The conclusions of Sec. 4.5 are unchanged.

3. Ambiguous synthetic cases (30 answers reviewed). In the synthetic setting every answer is labelled for conflation by an answer-only model judge and independently by a blind GPT-6 Astra review; where they disagree, a tiebreak decides (Sec. 3.6). These disagreements are the ambiguous cases you mention: 86 of the agentic answers. We sampled 30 of them across models for human review of whether each conclusion uses evidence across the two counting rules. Human review agreed with our final label on 22 of 30. Of the 8 disagreements, 6 were answers our labels had counted as clean and human review counted as conflation, and 2 the reverse, so on the ambiguous cases our reported conflation rates are, if anything, slightly conservative.

These checks come in addition to the existing hand check of 100 synthetic answers from four models (Sec. 4.4: the author agreed with 99 of 100 conflation labels).

Reviewer_myBQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper incorporates a checkable flaw in synthetic literature: studies that count the outcome under incompatible definitions. Most of the twelve frontier models notice the incompatibility and then pool the numbers anyway. Only GPT-6 Astra avoids this on synthetic data, and no model avoids it on real arXiv papers.

Critical strengths
  1. The experiments are done rigorously.
  2. The scoring (Section 3.5) is a valuable contribution.
Critical weaknesses
  1. Generators and judges come from the same family of LLMs.
  2. The selection of 15 topics was biased.
Questions for authors
  1. How were the real-literature ground truths annotated? Was there human agreement on them, and do the results change on topics that were not selected from pilot failures?
1 comment

Thank you for the positive assessment and for highlighting the scoring. We answer the two weaknesses and the question below; the new results are in two new appendices (K and L).

Generators and judges from the same family. Two model families are crossed at every stage, so no family grades only itself:

  • Corpora. Every topic is written twice, once by Opus 4.8 (Anthropic) and once by GPT-5.6 Sol (OpenAI), and every model answers both versions. Reported rates therefore average over the two families (per-generator results are in the appendix).
  • Synthetic labels. An Anthropic judge labels every answer, and GPT-6 Astra (OpenAI) reviews every answer blind to the model and to the first labels, including all answers from Anthropic models; disagreements go to a tiebreak.
  • Real-literature labels. GPT-6 Astra and Claude Fable 5.1 grade every answer independently, and we report a label only where both agree. Neither favours its own family: Fable marks Astra's answers as relying on incompatible numbers in 11 of 17 setups, against Astra's own 14, and both judges give Fable 15.
  • Human checks. Human review of the cases where the automatic judges disagreed, in both settings, agrees with the reported numbers (details in our response to Reviewer nppF).

Selection of the 15 topics. The topics were chosen in four steps: candidate claims generated by several models; an inline screen ranking them by how often Opus 4.8 missed the counting split; a development set taken from the top of that ranking to build the agentic setup and then dropped entirely, so the benchmark was not tuned on its own test topics; and, from the next best-ranked topics, those that trapped Fable 5.1 in short agentic pilots. Our goal was a challenging benchmark for a failure that is otherwise hard to pin down, so we curated the cases to surface it clearly.

Question: how were the real-literature reference answers made, was there human agreement on them, and do the results change on topics not selected from pilot failures?

  • How they were made. For each setup, the authors of that setting, with agent assistance, read the 12 papers and wrote a reference answer with three parts: the decision (in 17 of 21 setups, that no approach can be declared best), the reasons, i.e. which evaluation settings differ between the papers' headline numbers, and what an answer may and may not conclude.
  • Human agreement. All 17 no-winner reference answers were checked by hand against the papers by another author, using a per-paper table of the relevant evaluation settings in which every entry links to the supporting passage. All 17 were confirmed.
  • Topics not selected from pilot failures. We repeated the selection with a different model: we took the topics on which GPT-5.6 Sol, instead of Opus 4.8, failed most often, and ran all models on them. The models rank in the same order as on our topics (rank correlation 0.93). Absolute rates change with the topics; the comparison between models does not.

Comments on the paper

1 comment

We thank the reviewers. In response we added:

  1. Human review of the real-literature reference answers, of the real-literature judgments including the answers previously excluded because the two model judges disagreed, and of the ambiguous synthetic cases. The reported results hold.
  2. A selection check: on topics chosen by a different model's failures, the models rank in the same order.
  3. Clarifications of how the literatures are generated (Sec. 3.2) and of the general lesson for building hard tasks (Sec. 3.4).

The new results are in Appendices K and L; all changes in the revised PDF are highlighted in green.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.