acceptodds
Under review as a conference paper at ICLR 2027

Quantifying Claim Smoothing by LLM Research Agents

Abstract

Deep research, or any use of large language models (LLMs) to read a large document collection and produce insights, requires synthesizing information from different sources. In this work, we find that LLMs produce long-form research reports that read as confident syntheses but which fail to accurately reflect the underlying documents. We call this failure smoothing. In collapsing diverse and potentially conflicting sources into a single narrative, the model discards the evidential "jaggedness" of its corpus and asserts conclusions more decisive than the citations can support. We introduce a claim level scoring pipeline, SmoothScore, to quantify this effect. Our pipeline extracts key synthesis claims from model generated reports, then finds evidence relevant to each claim and clusters that evidence along high level aspects. Finally, each claim is scored with respect to the evidence to quantify the degree of smoothing done by the agent. Across 5 systems and 4 datasets, we find that models consistently produce stronger claims than warranted.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Deep Research, Evaluation, Claim Verification, Synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_YkA7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on “smoothing” failure of LLMs - where the models create confident syntheses of long documents but fail to faithfully reflect the underlying documents. The authors propose a claim-level scoring method, SMOOTH-SCORE, to measure this effect and find that models consistently produce stronger claims than they are supposed to.

Critical strengths
  • The research question is well motivated and could be helpful for critical fields that involve multiple document aggregation.

  • The evaluation experiments are conducted on a good number of datasets and systems, covering both open-source and proprietary models.

  • This paper is generally clearly written and easy to follow.

Critical weaknesses
  • Smoothing is an interesting failure but moderately narrow in scope for an ICLR paper. I think this paper could be much strengthened by additionally having either one of the following: 1. more in-depth analysis of why smoothing happens (i.e., a stronger Section 6); 2. solutions or even just practical takeaways that can help reduce smoothing failures.

  • Cross-system SmoothScores are not very comparable: Each claim is judged against the documents that the system cited, and different systems cite very different numbers of documents. Since smoothing scales with number of documents, the numbers are not comparable across systems, making the evaluation less useful for cross-system comparison.

  • Human annotators are used in the experiments (Table 3), but the details of human annotators (e.g., demographics, expertise) are not disclosed anywhere.

  • Some textual claims don’t match the tables or are inaccurate.

      * “Gemini smooths severely on 26–43% of claims, roughly 2–3× the rate of the open-source agents” - but on DRB-II, QUEST gets a total of higher score of >=4 than Gemini, so the “2-3x” doesn’t hold.
    
      * “we find that roughly halves the smoothing σ = 1.65, σ ≥3 down 11 points.” - the drops (1.950>1.65 and 29.3->18.6) are much smaller than “halves”
    
      * Table 1 - is the 95% CI computed over each claim? So this would require the assumption that the claims are independent of each other, but claims can come from the same report and thus will be dependent on each other.
    
Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Part 1 of the response to Reviewer YkA7

Thank you for your comments!

We are posting a general update to the main paper based on added experiments requested by the reviewers. These updates add (1) an updated analysis of the relationship between cited documents and SmoothScore, (2) an extended validation of the SmoothScore pipeline using human annotations, additional judges and a common document pool experiment and (3) a more thorough and improved analysis of engagement of research systems with documents. Additionally, we also conducted a post hoc analysis of reports generated and added 2 extra systems (Sol and Opus). Any additional experiments or clarifications are present in the appendix.

As requested by the reviewers, we also report the answer coverage using the gold rubrics presented by the different benchmarks. However, we retain the claim that SmoothScore tests a failure distinct from coverage. This is added as an external evaluation of the quality of generation to ensure that any changes do not compromise performance of the generation in any other axes.

Addressing the weaknesses below:

 

more in-depth analysis of why smoothing happens…solutions or even just practical takeaways that can help reduce smoothing failures

Our interventions (in Section 6) are practical fixes to reduce smoothing. We find that providing the full documents in context or giving instructions to not smooth claims, reduces smoothing. For instance, with all documents in context, SmoothScore for Sonnet reduces from 2.48 to 2.38, and with instructions alone it reduces from 2.48 to 2.16. Combining the two gives the largest reduction (1.52 on Luna and 1.78 on Sonnet) showing that smoothing reflects both limited engagement with sources and how conclusions are presented.

Another way of reducing smoothing (post-hoc) is to revise an existing report against its cited documents with our intervention instruction (to not smooth claims). This allows the model to revise the original report, keeping the original content and making fixes where needed. Revision reduces smoothing and the share of materially smoothed claims, while rubric coverage is retained or improved.

ModelConditionLengthClaimsSmoothScore [95% CI]% 1% 2% 3% 4% 5Coverage
LunaWebSearch (draft)1.0×4161.95 [1.84, 2.07]37.333.426.03.40.068.5
Revise + docs + instr2.2×4061.46 [1.37, 1.55]67.520.910.11.50.068.6
Sonnet 5WebSearch (draft)1.0×4192.48 [2.39, 2.57]14.329.849.66.20.064.3
Revise + docs + instr2.6×4001.92 [1.83, 2.00]39.033.824.03.20.071.8

The full table is provided in Section 6, Table 5. Our aim is to release our pipeline as a harness for any search-based agent to use (during generation or post-hoc). We leave training-based methods for improvement to future work.

 

Cross-system SmoothScores are not very comparable

We provide results for this in the comments for reviewer e41a. Our findings indicate a weak correlation between SmoothScore and the number of cited documents across systems (ρ = 0.35, p = .064), with a near-zero pooled within-system correlation (ρ = 0.04, p = .26; Appendix B, Table 11), and we have updated our observations accordingly (Section 4.1, line 296). We also ran an additional shared corpus experiment of 10 documents per question to isolate synthesis (Details in Appendix E). Comparing both conditions on the same questions, Luna has the lowest SmoothScore and Gemini the highest SmoothScore overall. Gemini’s mean score changes little (3.25 → 3.18), despite receiving roughly a quarter as many documents as it normally cites. Only QUEST is the outlier showing the largest overall reduction (2.77 → 2.28), reaching a SmoothScore comparable to Sonnet’s (2.29).

 

Human annotators are used in the experiments

The two annotators of the pipeline are graduate researchers in Computer Science with experience in LLM evaluation and in reading literature in the evaluated domains. All annotations were provided independently using the same rubric as the judge without any access to system identity or judge scores.

AuthorsOpenReview ↗

Part 2 of the response to Reviewer YkA7

Paper Errors

Thank you for pointing this out! We provide updates and details of the changes in the new draft for the updated claims:

Point 1: We agree the previous values were too broad and have removed this blanket comparison and now report Gemini’s severe-smoothing range directly: 26–43% across datasets (Section 4.1, line 266).

Point 2: We have removed the incorrect “roughly halves” wording and the revised Section 6 reports the expanded experiments and conclusions.

Point 3: For the 95% confidence intervals in Table 1 we use a cluster bootstrap over reports, not individual claims. For each system-dataset pair, we resample whole reports with replacement, keeping each report’s claims together, and recompute the mean SmoothScore over all claims in the sampled reports. This uses 10,000 resamples and we report the 2.5th and 97.5th percentiles. We have clarified this procedure in the revised paper.

 

Note on the revised PDF. In the paper, we have highlighted the portions with updates. Where an entire table, figure, or paragraph is new or rewritten, we have highlighted its caption or bold paragraph heading; where an entire appendix section is new, we have highlighted the section heading.

To keep changes easy to verify, we left the paper's structure unchanged and placed most new experiments in the appendix. For the final version, we will integrate the new experiments and results into the main text.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you again for your thoughtful review. As the discussion period is nearing its end, we would like to kindly follow up on our rebuttal and ensure that we have adequately addressed your concerns. We would appreciate any further feedback and please let us know if there is anything we can clarify!

Reviewer_e41aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates claim smoothing, where LLM research agents produce conclusions that are broader, more certain, or more decisive than their cited evidence warrants. It introduces SmoothScore, a pipeline that extracts synthesis claims, collects supporting and countervailing evidence, organizes evidence by aspect, and assigns a five-point severity score. Experiments across five systems and four datasets suggest that smoothing is common. An additional experiment shows that providing full documents and instructions to acknowledge disagreements reduces measured smoothing.

Critical strengths

Important and well-motivated problem. The paper highlights a meaningful limitation of citation-based factuality evaluation: a conclusion can have supporting citations while still misrepresenting the qualifications and disagreements in the underlying evidence.

Thoughtful evaluation design. The pipeline attempts to preserve claim qualifications, distinguish genuine counterevidence from already acknowledged exceptions, and avoid treating evidence counts as votes. The severity rubric provides more nuance than binary support judgments.

Useful examples and practical implications. The examples of unsupported composition, scope expansion, and premature closure make the phenomenon concrete. The document-access intervention also suggests a practical direction for improving research agents.

Critical weaknesses

Limited validation of the proposed metric. Human evaluation covers only 53 claims. Judge–annotator exact agreement is 55%, while agreement within one point is 91%. However, the main prevalence estimates depend on the boundary between scores 2 and 3, making one-point disagreements consequential. Only 67% of claims flagged as ≥3 by the judge also receive that classification from Annotator 1. These results leave substantial uncertainty about the reported prevalence. It is also unclear whether annotators independently inspected the original reports and documents or relied on pipeline-generated evidence.

System comparisons conflate synthesis with evidence selection. Each system is evaluated against its own cited documents. A system citing a larger or more heterogeneous collection may face more qualifying evidence, while a system citing a narrow collection may receive a favorable score despite omitting relevant external counterevidence. The rankings therefore do not isolate synthesis quality. A shared-corpus comparison, together with coverage and informativeness measurements, would make the interpretation stronger.

Missing baselines, ablations, and evaluator robustness checks. The paper does not establish whether aspect induction and separate support/skeptic passes improve agreement with humans over simpler alternatives. A direct judge baseline and component ablations are needed. Because preprocessing and final judging use models from the same family, replication with independent evaluator families would also strengthen confidence in the findings.

The document-count analysis does not isolate the proposed mechanism. Figure 3 compares aggregate system–dataset measurements, with document count varying alongside system identity, retrieval strategy, and report length. Larger collections may also allow the evaluator to discover more counterevidence. The analysis therefore supports an association, but does not establish that additional documents make a fixed generator less faithful.

The intervention’s interpretation is stronger than its evidence. Providing full documents changes the generation setup, increases report length by approximately 1.6×, and reduces the number of extracted claims. The improvement may therefore partly reflect changes in claim selection or presentation. The experiment covers only one system and one dataset. Additionally, describing the reduction as “roughly halves” is imprecise: the ≥3 rate falls from 29.3% to 18.6%, approximately a 37% relative reduction.

Questions for authors

How were the 53 human-evaluated claims selected? Did annotators independently read the full reports and source documents? Can the authors provide threshold-level confusion matrices and agreement with both annotators?

How does SmoothScore compare with a direct claim-and-documents judge? What improvement comes specifically from aspect organization and separate evidence-extraction passes?

Do the system rankings persist when all systems receive the same evidence collection and are evaluated for both smoothing and answer coverage?

How stable are scores under changes in judge model, semantically equivalent paraphrases, and attribution-preserving claim extraction?

In the document-access intervention, are important conclusions retained, or does part of the improvement come from omitting difficult claims?

4 comments
AuthorsOpenReview ↗

Part 1 of the response to Reviewer e41a

Thank you for your review!

We are posting a general update to the main paper based on added experiments requested by the reviewers. These updates add (1) an updated analysis of the relationship between cited documents and SmoothScore, (2) an extended validation of the SmoothScore pipeline using human annotations, additional judges and a common document pool experiment and (3) a more thorough and improved analysis of engagement of research systems with documents. Additionally, we also conducted a post hoc analysis of reports generated and added 2 extra systems (Sol and Opus). Any additional experiments or clarifications are present in the appendix.

As requested by the reviewers, we also report the answer coverage using the gold rubrics presented by the different benchmarks. However, we retain the claim that SmoothScore tests a failure distinct from coverage. This is added as an external evaluation of the quality of generation to ensure that any changes do not compromise performance of the generation in any other axes.

We address the questions and weaknesses raised by you below. All section/line number references in the responses refer to the updated draft:

 

How were the 53 human-evaluated claims selected?

For the human study, we randomly select 12 reports across three systems, and consider all claims from these reports for annotations. The prompts were selected at random from the ResearcherBench set of queries (details in Appendix D, line 1255).

Two annotators are given the query, claim, source documents, and the retrieved evidence from the skeptic and the support agents. They independently look at the claims to give their judgment on the same rubric (as the LLM Judge) regarding the degree of smoothing.

Since the paper submission, we updated our human evaluation from 53 to 73 claims (in a similar manner as above) and the new results are reflected in the updated paper draft. The inter-annotator agreement is now α = 0.76 (from 0.64), and the judge agreement with annotator 1 is high as well (0.68 from 0.62). If we threshold at >=3, the agreement between the annotators is 89% (65 of 73 claims; Cohen's κ = 0.77).

Annotator 1 \ Annotator 212345
1126000
2518100
3041640
403211
Annotator 1 \ Judge1234
115120
247112
315135
40016
Annotator 2 \ Judge1234
19260
210696
31486
40131
50010

Threshold level matrix :

Annotator 1 \ Annotator 2<3≥3
<3411
≥3724
Annotator 1 \ Judge<3≥3
<32715
≥3625
Annotator 2 \ Judge<3≥3
<32721
≥3619

On average the judge is close to both annotators: its mean score is 2.45, compared with 2.27 for Annotator 1 and 2.21 for Annotator 2, and it is within one point of Annotator 1 on 93% of claims.

 

How does SmoothScore compare with a direct claim-and-documents judge?

We run ablations starting from the direct claim baseline, and iteratively adding components to see how important they are to the scoring pipeline. Ablations tested: (a) Direct Claim (b) Direct Claim + Documents (c) Direct Claim + Extracted Evidence (d) Direct claim + Aspect-Organized Evidence

The set up is given the same scoring rubric and is run on 73 claims to see agreement with annotators.

Judge inputα vs annotator 1
Direct Claim0.37
Direct Claim + Documents*0.54
Direct Claim + Extracted Evidence0.57
Direct claim + Aspect-Organized Evidence / SmoothScore0.68

*7 of the 12 document pools exceed the judge’s context and were truncated to 220k tokens.

We see that organized evidence agrees most closely with human annotations. Against the direct claim-and-documents judge, SmoothScore improves agreement from 0.54 to 0.68. Evidence extraction accounts for 0.54 → 0.57 and aspect organization for 0.57 → 0.68. Details in Appendix D, line 1315 of the updated draft.

AuthorsOpenReview ↗

Part 2 of the response to Reviewer e41a

The document-count analysis does not isolate the proposed mechanism.

After we extended our evaluation to seven systems, we examined whether the cross-system trend in Figure 3 reflects document count (Appendix B, Table 11). We found a weak correlation with the number of cited documents across systems (ρ = 0.35, p = .064), which is near zero in the pooled within-system analysis (ρ = 0.04, p = .26). Figure 3 shows a cross-system association, not a mechanism (Section 4.1, line 296).

 

Do the system rankings persist when all systems receive the same evidence collection

We run an experiment where we give all systems the same set of documents to generate from. To create a common set of documents, we first randomly select a subset of questions (40 total, 10 from each dataset), pool all papers cited across all systems, rank the cited papers by how many systems cite them, and then select the top papers. We then provide papers from the same top 10 pool in context (and disable web-access), subject to each system’s context limit. We then calculate SmoothScore on the generated reports and measure coverage using rubrics provided by benchmarks where available. Under the shared-document setup, some runs did not yield usable reports. We retained the common 29 questions (907 claims) with usable reports from every system and compared shared-document results with original own-retrieval results on the same subset of questions from all five systems.

SmoothScore

SystemDRB-II orig.DRB-II sharedScholarQA-CS2 orig.ScholarQA-CS2 sharedResearchRubrics orig.ResearchRubrics sharedResearcherBench orig.ResearcherBench sharedAll orig.All shared
Luna1.461.531.731.782.012.151.961.771.811.82
Sonnet2.312.142.462.292.502.362.432.342.432.29
QUEST2.932.402.622.373.162.092.352.292.772.28
DR Tulu*2.082.202.732.522.792.612.542.342.542.42
Gemini3.162.973.173.123.423.373.213.223.253.18

Answer coverage

SystemDRB-II orig.DRB-II sharedResearchRubrics orig.ResearchRubrics sharedResearcherBench orig.ResearcherBench sharedAll orig.All shared
Luna49.451.663.057.061.065.858.158.4
Sonnet51.143.255.551.262.062.056.452.5
QUEST36.040.651.445.751.355.246.747.4
DR Tulu*34.531.854.847.048.660.446.447.0
Gemini51.345.258.953.367.345.759.548.2

*At its native context limit, DR Tulu received a median of two papers per question

We observe:

  • On CS2-Expert, the system order with shared documents is identical to Table 1. Gemini has the highest SmoothScore on all four datasets in both settings.
  • On ResearcherBench and DRB-II, the only changes are between adjacent systems in the middle, each by a small margin: Sonnet and DR Tulu become tied on ResearcherBench (both 2.34, against 2.43 and 2.54 originally), and Sonnet moves ahead of DR Tulu on DRB-II (2.14 vs 2.20).
  • Only QUEST on ResearchRubrics changes by more than one position: it moves from fourth (3.16) to first with shared documents (2.09, against Luna’s 2.15).

Pooled across the datasets, we find that Luna scores significantly lower and Gemini significantly higher than every other system (Holm-adjusted p ≤ .008). The middle three systems remain close (pairwise differences ≤ 0.14), consistent with their small differences under their own retrieval on the full benchmarks (≤ 0.17).

Overall coverage is essentially unchanged for Luna, QUEST and DR Tulu (within 1 point), while it falls for Sonnet (56.4 → 52.5) and most for Gemini (59.5 → 48.2), the latter driven by ResearcherBench (67.3 → 45.7). We provide additional details of the setup in Appendix E of the updated draft.

AuthorsOpenReview ↗

Part 3 of the response to Reviewer e41a

How stable are scores under changes in judge model, semantically equivalent paraphrases, and attribution-preserving claim extraction?

We address each point under this question:

Judge model. We evaluate six alternative judges on the 73 human-annotated claims. All seven judges preserve the same system ranking as Annotator 1 (Luna < DR Tulu < Gemini) (Details provided in Appendix D, line 1274).

Judgeα vs. Annotator 1
GPT-50.68
Gemini 4 Argon0.63
gpt-6.1-terra0.53
Opus 5.50.52
gpt-6.1-sol0.48
Gemini 2.5 Pro0.41
Claude Sonnet 4.50.11
SystemAnnotator 1GPT-5Gemini 2.5 ProSonnet 4.5Gemini 4 Argongpt-6.1-terragpt-6.1-solOpus 5.5
Luna1.961.712.332.881.711.581.542.00
DR Tulu2.162.602.963.322.562.882.882.96
Gemini Deep Research2.713.043.834.422.883.253.463.29
All 73 claims2.272.453.043.532.382.582.632.75

We additionally re-score claims from the five original systems across all four benchmarks using gpt-6.1-terra, gpt-6.1-sol, Opus 5.5, and Gemini 4 Argon, keeping the prompt and evidence fixed. Rankings are identical to GPT-5’s in 9 of 16 judge–dataset comparisons; the remaining differences involve adjacent-system swaps (Kendall’s τ = .60–1.00). Luna scores lowest in 15 of 16 comparisons and Gemini highest in 14 of 16. These results show broad ranking stability, while showing that absolute scores and some pairwise orderings depend on the judge. Full results and scoring coverage are provided in Appendix D, line 1295.

 

Paraphrases. Paraphrases of claims that preserve the takeaway, including its hedges and scope, are judged similarly (Prompt D.2 in Appendix D.1). For 25 ResearcherBench queries, we have Luna and Sonnet generate 3 different paraphrases of their respective claims, keeping hedges, scope and numbers intact. The report-level SmoothScore remained mostly unchanged (Luna 2.25 → 2.19; Sonnet 5 2.82 → 2.78) with 64% of Luna and 72% of Sonnet’s paraphrases receiving an identical score. Across both, 3% or fewer move more than a single level and the score distribution is preserved (SD 0.88 → 0.88; 0.67 → 0.69) [Appendix D.1, line 1347 in the updated draft]

 

Attribution-preserving extraction. For claim extraction, we pick claims directly from the report to ensure that the hedges, qualifiers and scope are intact (Appendix C.1). We don't do atomic decomposition, because it can separate a conclusion from its qualifications. Across the total judged claims (5,696), 91.4% appear verbatim in their report (after removing citation and markdown markup), and mean 4-gram overlap between each claim and its report is 0.99. [Appendix D, line 1342 in the updated draft]

 

In the document-access intervention, are important conclusions retained

The document-intervention experiment is to see if engagement with source documents affects smoothing. In our update, we revised the claims to make them more precise and also updated the analysis with two model families.

To check retention of important conclusions, we check coverage using gold rubrics on ResearcherBench and find that regenerating the report with documents and instructions reduces coverage (Luna −7.4; Sonnet −4.0 percentage points) and as you observed, also reduces the number of claims generated. Drop in coverage can be an indication of the intervention leading to dropping conclusions. To check this further, we look at the alternative revision approach described in Section 6 of the updated draft. Here coverage is retained for Luna (68.5% → 68.6%) and increases for Sonnet (64.3% → 71.8%). Additionally, we map each claim from the original report to the claims of the revised report. We use GPT-5 to label a claim as retained if a claim from the revised report carries the same takeaway, even if hedged or narrowed. Using this, we find that 91% of Luna’s and 86% of Sonnet 5’s original claims are retained. Among retained claims that were originally smoothed (score ≥3), mean SmoothScore fell from 3.10 to 1.78 (Luna) and from 3.12 to 2.04 (Sonnet 5). These results suggest that the improvement when revising comes mainly from how conclusions are stated, rather than from omitting them (Details in Appendix H, line 1822).

 

Note on the revised PDF. In the paper, we have highlighted the portions with updates. Where an entire table, figure, or paragraph is new or rewritten, we have highlighted its caption or bold paragraph heading; where an entire appendix section is new, we have highlighted the section heading.

To keep changes easy to verify, we left the paper's structure unchanged and placed most new experiments in the appendix. For the final version, we will integrate the new experiments and results into the main text.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you again for your thoughtful review. As the discussion period is nearing its end, we would like to kindly follow up on our rebuttal and ensure that we have adequately addressed your concerns. We would appreciate any further feedback and please let us know if there is anything we can clarify!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.