- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work studies LLMs/agents' sloppiness, i.e., work that is left incomplete but reported as finished, with a focus on one type of failure mode in literature analysis where the LLM/agent draws a conclusion by mixing numerical results that are not directly comparable. To quantify LLMs/agents' behavior on this failure mode, this paper synthesizes a literature corpus where pairs of papers arrive at contradicting conclusions about a claim because their measurements and results are different by definition. With additional literature sourced from real papers, the experiment on 12 frontier models reveals that most of the tested models exhibit sloppiness in the manner that although they are able to identify the differences in the measurement but still mixes/pools/compares them.
Critical strengths
- The paper studies a timely and important research topic on LLM/agent sloppiness.
- The authors propose an interesting and valuable angle for controllable and quantitative study of the sloppiness problem through literature analysis of contested claims with different measurements.
- The data synthesis pipeline provides a scalable scaffold for automatic construction of testbeds for sloppiness in literature review.
Critical weaknesses
- Some parts of the presentation are confusing. For instance, it is hard to follow Section 3.2 and gain a clear understanding of how the literature generation process is conducted. And for Section 3.4, there is a lack of elaboration on how the findings from Fable experiments can be extended to a general setting for challenging task construction.
- Lack of constructive experiments and findings. While the analysis is thorough, they mostly only describe how different models fail under this setup. There is a lack of proposals for constructive approaches with corresponding experiments on what might be useful methods that can help mitigate this kind of sloppiness issues.
Questions for authors
Please see the weaknesses above.
Thank you for the review and for recognising the controlled angle and the generation pipeline. We address the two weaknesses below.
Presentation of Sec. 3.2 (literature generation). The generation process, step by step (we have clarified the wording of Sec. 3.2 along these lines):
Sec. 3.4 and how it generalises to building hard tasks. The general lesson, which we now state in one sentence at the end of Sec. 3.4: adding more documents did not make the task hard for frontier models; making the wrong step the natural route did. Two ingredients were needed: (i) no single source is decisive on its own (wide confidence intervals in each site paper), and (ii) the material presents the wrong step, here pooling the eight results, as the obvious next move. The package experiment (Sec. 4.3) tests these separately: Fable 5.1 never conflates with narrow intervals, even when pooling is invited, and conflates in 6 of 12 answers when both are present; Opus 5 is already pushed by the pooling prose alone (7 of 12) and fails in 11 of 12 with both. For other evaluations of a "notice and respect" skill, the recipe is: plant one checkable flaw, make the shortcut that ignores it necessary to finish the task, and confirm on separate packages that strong models fail on the combination, not on each part alone.
Constructive findings. The paper's aim is to measure this failure and show that noticing a problem does not mean acting on it, so we did not set out to engineer a mitigation. The results nevertheless already identify levers: