acceptodds
Under review as a conference paper at ICLR 2027

Auditing and Redesigning Deep-Research Report-Writing Pipelines for Faithful Reports

Abstract

Deep-research (DR) systems are commonly evaluated with report-level rubrics, which can obscure whether their writers omit retrieved evidence, introduce unsupported claims, or attach citations to the wrong sources. We introduce CLAIMPROBE, a claim-level audit that aligns report claims with the evidence collected by a DR system. Applied to three open DR systems, CLAIMPROBE reveals substantial differences in faithfulness and evidence coverage among reports with similar rubric-based scores. These findings motivate CLAIMWRITER, an evidence-linked writer that organizes atomic source facts before drafting report sections and preserves their links to the resulting prose. Across the evaluated host and judge configurations, replacing each system's native writer with a ClaimWriter variant lowers measured hallucination by 7 to 21 percentage points and raises necessary-fact recall by 7 to 30 points, while changing overall rubric scores only modestly. Reports can become stale when the sources they summarize change, yet rerunning the research pipeline can be costly and may unnecessarily rewrite unaffected content. CLAIMWRITER instead converts source additions, removals, and replacements into fact-level edits and maps them to the sections that use those facts. In controlled experiments where a subset of source documents changes, CLAIMWRITER's structured revision achieves the highest rate of incorporating new facts while removing outdated values among the evaluated replay and text-edit methods.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Deep Research, Factuality and hallucination, Evidence-grounded generation, Trustworthy AI Systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uYMWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study proposes a source auditing framework for evaluating source-report alignment in deep research reports. The paper also introduces an evidence-based report generation framework. Experiments across 3 deep research systems show improvements in evaluating report hallucination, misattribution, and fact recall compared with other pipelines. The paper also show that their approach reaches higher report update success and lower token costs than other pipelines.

Critical strengths
  1. The overall experimental design is relatively comprehensive. The authors also assess robustness across different LLM judges.
  2. The writing is clear and easy to follow, though the motivation of this study needs further explanation.
  3. The proposed evidence-based writing framework is sound and necessary. This is a useful extension beyond one-time report generation.
Critical weaknesses
  1. Based on the Abstract, it is unclear to me why the new ClaimProbe approach is necessary to for report evaluation. What’s the limitation of prior evaluation metrics? Add one or two sentences after the first sentence to describe why the motivation is necessary. It is also vague how modest the overall scores are changed. It would be better to describe the results more specifically.
  2. Table 1: The human-to-human agreement for hallucination and misattribution is pretty low, as well as LLM judge and human agreement. I’m concerned about the reliability of those metrics given the main experiments are built upon them. Additionally, disagreement analysis is needed.
  3. The paper is missing validation of the fact retrieval part. Specifically, if a claim misses supporting evidence during retrieval but the claim itself is correct, it will be labeled as hallucinated. How to deal with such a situation? Hallucinations can be either non-factual or factual. Currently, the paper does not consider factual hallucinations. Those errors should be quantified. Also, the necessary-fact recall depends on automatically extracted facts and relevance labels, but the accuracy of the extraction is not evaluated.
  4. The experiment is missing an ablation study of individual components of ClaimWriter. For example, which component contributes the most improvement gain over the native writers?
  5. The efficiency of ClaimWriter is one of the biggest concerns I have. The paper needs further analysis regarding the trade-offs between additional costs and quality.
Questions for authors
  1. How is top-k=20 derived? Is it a random choice, or is there a supporting ablation study for it?
  2. Experiments are missing confidence intervals or statistical significance tests.
  3. What is the performance of the English and Chinese tasks in DeepResearch Bench separately?
Reviewer_PFKwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents the problem of evaluating claim-level factuality of deep research report outputs. Current evaluation approaches focus largely on report-level rubrics which may conflate claim-level factuality which is important for verifying the outputs of DR systems. The paper uses an LLM as a Judge to extract claims and evidence from a report and then computes four metrics: hallucination, misattribution, relevance and neccessary fact recall. Based on this evaluation framework, the paper evaluates 3 existing DR systems and finds that they struggle with claim-level faithfulness. To mitigate this, the paper introduces two claim writer approaches that separate evidence from response generation and use it to replace the native writer used in the original report generation. The claim writers show improvement compared to the native writers on factuality metrics, though the improvements come with small drops in report-level rubrics such as RACE. The paper also compares whether the claim writers are robust to source changes and shows better performance than the native writers in handling changes in source information. The main contributions of this work are 1) an evaluation framework for claim-level in DR systems, 2) an empirical evalaution of 3 DR systems and 3) a claim writer approach to improve faithfulness.

Critical strengths
  1. The paper addresses an important problem of claim-level faitfhulness in DR outputs and provides an evaluation framework, an empirical analysis of 3 DR systems and approaches to improve faitfhulness of current DR systems.
  2. The paper tests the claim writer's robustness to source changes which mirrors challenges DR systems face in the real-world providing insights into the failures of current systems.
Critical weaknesses

1. A thorough qualitative and quantitative comparison with existing deep research report faitfhulness evaluation metrics is lacking and should be addressed.

A comparison of how your faitfhulness metric differs from existing approaches or how it adopts them to deep research reports is lacking, making it hard to scope the novelty of the evaluation framework itself. The paper decomposes faitfhulness evaluation into four metrics: hallucination, misattribution, necessary fact recall and Uncited-but-supported. However, the evaluation approach itself has been widely applied in long-form factuality evaluation ( FACTScore, VeriScore, etc) and more recently in deep research report benchmarks (DeepResearchBench, LiveResearchBench, LiveDRBench).

For example, the DeepResearchBench framework introduces the FACT metric (Factual Abundance and Citation Trustworthiness), alongside the RACE metric that you use as a report-level rubric. The FACT metric is presented as a claim-level probe which extracts statement-URL pairs from a report and computes factual and citation accuracy. I extracted this verbatim from the DeepResearchBench paper to help in comparison with your approach: “We employ a Judge LLM to extract discrete statements from reports generated by DRAs with their corresponding cited source URLs. Then, the Judge LLM examines the pairs to identify cases where multiple statements associated with the same URL describe exactly the same fact. In such cases, only one representative Statement-URL pair is retained, ensuring each unique factual claim is represented only once.”

LiveResearchBench also introduces a claim-level faitfhulness metrics i.e. citation accuracy and citation traceability which similarly uses an LLM as a judge to check what statements are not cited and what citations do not support the claims made. It also provides a report-level faitfhulness metric. e.g. Missatribution appears similar to citation accuracy.

I provide my list of concerns and weaknesses with regards to comparison and distinction to existing metrics:

  1. First, the intro (L040 - 046) mentions that current DR benchmarks primarily score reports with holistic rubrics. However, I believe that this is a misrepresentation of existing metrics and approaches, DeepResearchBench does provide a claim-level metric i.e. FACT, and LiveResearchBench has a similar metric. It would be great for the authors to clarify this difference beyond just mentioning them in the relevant works section. I also took note that the paper uses the RACE metric from DeepResearchBench, however, makes no mention or comparison to FACT, which is a claim-level metric from the same work. To better scope this work, I would recommend to the authors to provide a clear qualitative comparison beyond a mention in the related works, that compares each faitfhulness metric in prior deep research report evaluations and shows the differences or similarities.

  2. Second, beyond just providing a qualitative comparison, a quantitative analysis of the extent which the metrics proposed correlate with existing faitfhulness metrics is missing, and as such is unclear whether they offer new diagnosing information. For the paper to justify its evaluation framework, it would be important to see if their metrics tell us anything new compared to existing metrics. An experiment that would be interesting to see is to report the FACT metric alongside the 4 metrics the paper proposes. This could help us understand 1) Do the new metrics tell us something new? 2) How do the metrics differ from existing deep research faitfhulness report metrics? 3) Are the improvements made by the CLAIM writer also observed in the other metrics?

What would help address these weakeness?

  • A qualitative analysis that compares the metrics you propose and other metrics from other deep research report benchmarks i.e. what metrics are similar and what metrics capture new evaluation points.
  • A quantitative analysis between existing metrics identified above e.g. FACT and your metric to see if they are correlated and 2) whether the improvements made by the Claim writer are seen in other similar metrics.

W2: High levels of subjectivity in the judge-human calibration remains unexplained and the proportion of task items which human annotators all agree on is not provided.

In L149 - L150, the reported human agreement on the task is lower than what's expected for a metric to be reliable (>0.6). From the numbers reported, humans seem to have very low agreement on some of the tasks e.g. hallucination. What does this mean for the metric itself and judge reliability? Can we still trust the metric if human annotators seem to disagree? What drives this disagreement, is the task inherently ambigous?

While the agreement with the LLM judge is higher on the unanimous cases, what proportion of task items do all human annotators agree on? Please provide clearer numbers on the distribution of labels and what the proportion of task items that have agreement. This would help us better understand whether the 0.484 reported is on a small or large proportion of unanimous human labels. Please provide details of the annotators, are they authors of the paper? If not, are they compensated?

W3: Claim writers (TD and BU) add computational costs to existing DR systems warranting a more competitive baseline to justify the method

Both approaches require multiple steps i.e. reconciling the evidence in the top-down and bottom-up, and then a ClaimWriter to output the final report, in addition to the steps in the original pipeline that generates the first report. It is clear that this extra steps adds computational and latency costs to the pipeline. I believe this work could substantiate the need for this extra complexity better, for example, did you compare the improvements to a naive rewriter that is not tuned to perform top-down or bottom-up extraction of evidence. If the goal is to separate between response and evidence generation, it would be expected to add a simple comparison to an "untuned" evidence extractor or claim rewriter. This would help to better justify the need for extra complexity in the approach.

W4: Distinction between the methodological significance of the different claim writer approaches (TD vs BU) is unclear

The paper introduces two claim writers that help address the issues identified in the baseline writers: a top-down approach and a bottom-up approach with the results showing differences across both approaches but improvements over the baseline. Beyond the scores provided and token cost comparisons presented in Table 4, it is still unclear what the significance of each approach and in what scenarios one approach would be better than the other. This could be better clarified in a discussion section as both methods seem to have different tradeoffs and none seems to perform better than the other.

Questions for authors
  1. Do you do any ablations to test the reliability of the embedding models? Do you think the design choice of the embedding model potentially impacts the metric? Should we expect similar results when we change the embedding model?
  2. What could explain the drop in perfromance between the original reports and rewritten versions as reported using RACE? Did you do any qualitative annotations to understand what the claim writer does that makes it drop performance on RACE? Do you think there's a trade-off between optimizing for claim level faitfhulness vs holistic report rubrics?
  3. To what extent does the LLM used as the backbone for the deep research system impact the evaluation? In the paper, you only used GPT 5.5 for the evaluation, do we expect similar observations to hold when using other models. Did you run any ablations on the report writer e.g. swap it for a different model?
Reviewer_Zo7QOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces ClaimProbe for claim-level auditing and ClaimWriter for evidence-linked report generation and revision. With retrieved evidence held fixed, ClaimWriter improves measured faithfulness and fact coverage across three deep-research systems, but reduces readability and incurs substantial initial generation costs.

Critical strengths
  • Holding retrieved evidence and the generation model fixed provides a useful controlled comparison of report-writing pipelines.

  • The improvements in measured faithfulness and coverage are substantial and persist across judge configurations. Comparisons using the same initial report also support the usefulness of fact-to-section links for updating.

Critical weaknesses
  • The methodological novelty is limited. The contribution primarily combines claim-level auditing, evidence-organized writing, and source-linked revision; component ablations and stronger, simpler baselines are needed to establish the value of the specific design.

  • The overall quality benefit remains unclear. RACE scores decline slightly and readability consistently worsens, leaving the practical value of the faithfulness–readability trade-off unresolved. Similar aggregate scores could reflect offsetting quality changes and do not alone establish that rubric-based evaluation is insensitive to faithfulness.

  • The reliability of hallucination evaluation remains uncertain. Human agreement is low (κ = 0.173), as is judge agreement with the human majority (κ = 0.299). Consistent rankings across judges support the relative improvements but do not fully validate the reported hallucination rates.

  • The computational benefits are conditional. Initial report construction is expensive, and update-time savings are substantial against native replay but inconsistent against text editing. Comparisons under matched computational budgets are needed to establish whether the additional complexity is justified.

Questions for authors
  • Do the update gains extend to a broader set of tasks and source changes, including fact additions, removals, and conflicting information? Can the authors also report update success for facts already present in the initial report, to separate update accuracy from initial coverage?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.