Auditing and Redesigning Deep-Research Report-Writing Pipelines for Faithful Reports
Abstract
Deep-research (DR) systems are commonly evaluated with report-level rubrics, which can obscure whether their writers omit retrieved evidence, introduce unsupported claims, or attach citations to the wrong sources. We introduce CLAIMPROBE, a claim-level audit that aligns report claims with the evidence collected by a DR system. Applied to three open DR systems, CLAIMPROBE reveals substantial differences in faithfulness and evidence coverage among reports with similar rubric-based scores. These findings motivate CLAIMWRITER, an evidence-linked writer that organizes atomic source facts before drafting report sections and preserves their links to the resulting prose. Across the evaluated host and judge configurations, replacing each system's native writer with a ClaimWriter variant lowers measured hallucination by 7 to 21 percentage points and raises necessary-fact recall by 7 to 30 points, while changing overall rubric scores only modestly. Reports can become stale when the sources they summarize change, yet rerunning the research pipeline can be costly and may unnecessarily rewrite unaffected content. CLAIMWRITER instead converts source additions, removals, and replacements into fact-level edits and maps them to the sections that use those facts. In controlled experiments where a subset of source documents changes, CLAIMWRITER's structured revision achieves the highest rate of incorporating new facts while removing outdated values among the evaluated replay and text-edit methods.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.