1. A thorough qualitative and quantitative comparison with existing deep research report faitfhulness evaluation metrics is lacking and should be addressed.
A comparison of how your faitfhulness metric differs from existing approaches or how it adopts them to deep research reports is lacking, making it hard to scope the novelty of the evaluation framework itself. The paper decomposes faitfhulness evaluation into four metrics: hallucination, misattribution, necessary fact recall and Uncited-but-supported. However, the evaluation approach itself has been widely applied in long-form factuality evaluation ( FACTScore, VeriScore, etc) and more recently in deep research report benchmarks (DeepResearchBench, LiveResearchBench, LiveDRBench).
For example, the DeepResearchBench framework introduces the FACT metric (Factual Abundance and Citation Trustworthiness), alongside the RACE metric that you use as a report-level rubric. The FACT metric is presented as a claim-level probe which extracts statement-URL pairs from a report and computes factual and citation accuracy. I extracted this verbatim from the DeepResearchBench paper to help in comparison with your approach: “We employ a Judge LLM to extract discrete statements from reports generated by DRAs with their corresponding cited source URLs. Then, the Judge LLM examines the pairs to identify cases where multiple statements associated with the same URL describe exactly the same fact. In such cases, only one representative Statement-URL pair is retained, ensuring each unique factual claim is represented only once.”
LiveResearchBench also introduces a claim-level faitfhulness metrics i.e. citation accuracy and citation traceability which similarly uses an LLM as a judge to check what statements are not cited and what citations do not support the claims made. It also provides a report-level faitfhulness metric. e.g. Missatribution appears similar to citation accuracy.
I provide my list of concerns and weaknesses with regards to comparison and distinction to existing metrics:
-
First, the intro (L040 - 046) mentions that current DR benchmarks primarily score reports with holistic rubrics. However, I believe that this is a misrepresentation of existing metrics and approaches, DeepResearchBench does provide a claim-level metric i.e. FACT, and LiveResearchBench has a similar metric. It would be great for the authors to clarify this difference beyond just mentioning them in the relevant works section. I also took note that the paper uses the RACE metric from DeepResearchBench, however, makes no mention or comparison to FACT, which is a claim-level metric from the same work. To better scope this work, I would recommend to the authors to provide a clear qualitative comparison beyond a mention in the related works, that compares each faitfhulness metric in prior deep research report evaluations and shows the differences or similarities.
-
Second, beyond just providing a qualitative comparison, a quantitative analysis of the extent which the metrics proposed correlate with existing faitfhulness metrics is missing, and as such is unclear whether they offer new diagnosing information. For the paper to justify its evaluation framework, it would be important to see if their metrics tell us anything new compared to existing metrics. An experiment that would be interesting to see is to report the FACT metric alongside the 4 metrics the paper proposes. This could help us understand 1) Do the new metrics tell us something new? 2) How do the metrics differ from existing deep research faitfhulness report metrics? 3) Are the improvements made by the CLAIM writer also observed in the other metrics?
What would help address these weakeness?
- A qualitative analysis that compares the metrics you propose and other metrics from other deep research report benchmarks i.e. what metrics are similar and what metrics capture new evaluation points.
- A quantitative analysis between existing metrics identified above e.g. FACT and your metric to see if they are correlated and 2) whether the improvements made by the Claim writer are seen in other similar metrics.
W2: High levels of subjectivity in the judge-human calibration remains unexplained and the proportion of task items which human annotators all agree on is not provided.
In L149 - L150, the reported human agreement on the task is lower than what's expected for a metric to be reliable (>0.6). From the numbers reported, humans seem to have very low agreement on some of the tasks e.g. hallucination. What does this mean for the metric itself and judge reliability? Can we still trust the metric if human annotators seem to disagree? What drives this disagreement, is the task inherently ambigous?
While the agreement with the LLM judge is higher on the unanimous cases, what proportion of task items do all human annotators agree on? Please provide clearer numbers on the distribution of labels and what the proportion of task items that have agreement. This would help us better understand whether the 0.484 reported is on a small or large proportion of unanimous human labels. Please provide details of the annotators, are they authors of the paper? If not, are they compensated?
W3: Claim writers (TD and BU) add computational costs to existing DR systems warranting a more competitive baseline to justify the method
Both approaches require multiple steps i.e. reconciling the evidence in the top-down and bottom-up, and then a ClaimWriter to output the final report, in addition to the steps in the original pipeline that generates the first report. It is clear that this extra steps adds computational and latency costs to the pipeline. I believe this work could substantiate the need for this extra complexity better, for example, did you compare the improvements to a naive rewriter that is not tuned to perform top-down or bottom-up extraction of evidence. If the goal is to separate between response and evidence generation, it would be expected to add a simple comparison to an "untuned" evidence extractor or claim rewriter. This would help to better justify the need for extra complexity in the approach.
W4: Distinction between the methodological significance of the different claim writer approaches (TD vs BU) is unclear
The paper introduces two claim writers that help address the issues identified in the baseline writers: a top-down approach and a bottom-up approach with the results showing differences across both approaches but improvements over the baseline. Beyond the scores provided and token cost comparisons presented in Table 4, it is still unclear what the significance of each approach and in what scenarios one approach would be better than the other. This could be better clarified in a discussion section as both methods seem to have different tradeoffs and none seems to perform better than the other.