- Rating
- 3(+2)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors proposed a deep research framework, Evidence-Grounded Residual Completion (EGRC), which is claimed to recover information omitted from the findings and produce a better report. Experiments were conducted on DRB2, and the proposed method achieved higher scores for "quality" and "recovery."
Critical strengths
-
The paper proposed a method that collects omitted information from findings to improve the quality of the report.
-
The experimental results include human annotation that compares the proposed method with baseline(s).
Critical weaknesses
-
The abstract and introduction are terrible. A lot of undefined terms (e.g., targeted/untargeted completion, omission, selector, flag, etc.) appear throughout, making it very difficult to understand the paper's main claim and the task the authors attempted to focus on.
-
The methodology section (Section 3 & 4) missed important details. (Please see questions for the details.)
-
The experiment involved designs that are not justified. Metrics are not well defined. The generalizability and practicality of EGRC were not addressed.
Questions for authors
Presentation
- The paper is hard to follow. Terms without definitions appear early in the paper, (e.g., targeted/untargeted completion, omission, selector). What's worse is that after reading the paper, I am still uncertain about what these terms mean.
Methodology
-
The methodology part is handwavy. The authors mentioned a selector proposes candidate omissions using the "request, findings, and report" (line 143), but how? From the latter part of the paper (Section 5), the selectors seem to be generative models, so how to prove the models will not omit useful information, as if they did when proposing the initial draft, and lead to false positives?
-
How to justify the correctness of M(t,e) (line 147)?
Experiment
-
The most frequently used metrics in the paper, Quality and Recovery are not defined properly. I suppose quality is DRB2’s overall score, but the authors need to include a definition for the uncommon metrics before discussing experimental results based on the metrics.
-
The experiment did not contain results using different generative models as backbones. Thus, it is unclear whether EGRC is generalizable.
-
Another important factor for deep research is the cost. Considering EGRC needs a selector to validate all of the "findings", the token usage is probably larger than the baseline in the paper. Considering the improvement is not huge on "Quality", it is necessary to provide the cost (as token usage or converted to USD) to better understand the practicality of EGRC.
13 comments
Responses to Reviewer tCrN (2/6)
2. Omission selection (W2; Methodology question 1).
Section 3.1 of the original submission defines candidate omissions and specifies the selector's inputs: the request, acquired findings, and initial report.
We prompt an LLM to act as the selector. It receives the user's request, the complete initial report, and summaries of all acquired findings, with their identifiers and source links. The prompt asks which findings contain useful information that the report has not fully expressed, including missing facts, corrections, quantities, comparisons, and qualifications needed to answer the request. The model returns finding identifiers, not another report. The program checks that every identifier is in the supplied collection and appears only once. These selections form the original omission set, R, which remains fixed during revision. The selector receives neither reference answers nor benchmark criteria.
You ask whether the selector can miss useful information or make incorrect selections, just as the model can make mistakes during initial writing. Writing requires the model to choose and organize information into an answer. The selector has a different task: it compares the completed report with the acquired findings to identify relevant information that is not fully expressed.
We considered these possible errors when designing the revision step. EGRC therefore treats the selections as candidate omissions, not as instructions to insert every item. During revision, the writer must identify a finding's existing expression, propose an edit, or explain why source, relevance, or space constraints prevent inclusion. Proposed edits undergo source review before they are applied.
Furthermore, we do not assume that the omission list is complete or error-free. To assess the selector's reliability, two independent human annotators checked a sample of 128 findings, as reported in the original submission (Section 3.2 and Appendix G). Both annotators confirmed source support, relevance, and absence for 40 of 64 selected findings, compared with 4 of 64 unselected findings. In this sample, the selected set thus concentrates human-confirmed omissions at ten times the rate of the unselected set, supporting the selector's ability to identify supported, relevant omissions.
Appendix E of the original submission also checks recovery using only omissions identified by both Qwen and DeepSeek. The two selectors agree on 57 omissions across 23 test tasks. We keep each method's reports unchanged and recalculate recovery using these findings. Recovery is 71.6% for EGRC, 31.2% for Generic, and 49.1% for Two-pass. The advantage therefore remains when both selectors must agree on the omissions being evaluated.
Section 3.1 of the revised manuscript now explains the selector's inputs, instructions, and outputs beside the definition of R. We have also added the original prompt and an input/output example to the supplementary material. The human comparison demonstrates the selector's ability to identify supported, relevant omissions. The two-selector check shows that EGRC's recovery advantage persists when evaluation uses only omissions identified by both Qwen and DeepSeek.
We have also clarified the revision procedure in Section 4, including how the writing LLM organizes the answer and decides what to do with each original omission. We now explain the separate LLM calls for checking edits against the sources and the program checks performed before edits are applied. These changes make the roles of the LLMs and the program explicit in the method description.
Responses to Reviewer tCrN (3/6)
3. Expression judgment M (W2; Methodology question 2).
Thank you for raising this question. M determines whether an omitted finding is counted as recovered, so its reliability is central to our evaluation.
For a text t and a finding e, an evaluation LLM determines M(t,e): 1 if the text fully expresses the finding, and 0 otherwise. In the main experiments, we use GLM-5.2 as the evaluation LLM. It receives the complete report, the findings being assessed, and their supporting source passages. It uses an evaluation prompt separate from the revision prompt. A positive judgment must quote text from the report that expresses the finding. A finding is counted as recovered only if the report itself fully expresses it. The writing LLM's own decisions do not determine the score.
The evaluation LLM checks the details essential to each finding's meaning. In the policy example in Figure 1, both the employee threshold and the subsidiary exclusion are needed to determine which firms qualify. A report stating only the threshold is therefore incomplete. The report can express these details in different words or across more than one passage. For example, the threshold can appear in one paragraph and the exclusion in another.
We validate these judgments against human annotations and a second assessment model. On 129 sampled report–finding pairs with valid model judgments, the evaluation LLM agrees with the two independent readers in 96.1% and 94.6% of cases, respectively (Appendix G.1). The readers also confirm a 34.4-percentage-point recovery advantage over Two-pass on the sampled DRB2 findings.
We also reassess the existing reports with GPT-5.5, comparing EGRC with List-only. Both receive the same organized answer and omission list, but List-only chooses which entries to address. Under this second assessment, EGRC's recovery is 17.3 percentage points higher across all 36 test tasks.
The original submission defined M in Section 3.1 and included the evaluation prompt in the supplementary package. We have expanded Section 3.1 to explain the evaluation inputs and judgment requirements more directly. Section 5.5 reports the human agreement, and Appendix C.2 gives the detailed expression criteria. Together, these results support the reliability of M and show that EGRC's recovery advantage is also observed in human judgments and with a different evaluation LLM.
Responses to Reviewer tCrN (1/6)
Dear Reviewer tCrN,
Thank you for reviewing our paper and for recognizing the human evaluation as a strength. Below, we clarify the task, method, and evaluation, including the definitions and experimental evidence already provided in the original submission. We also present an additional experiment with GPT as the writer and a more complete cost analysis. We have revised the manuscript to make these explanations easier to follow.
Responses to weaknesses and questions
1. Task and terminology (W1; Presentation question).
We study how Deep Research agents use acquired evidence when writing long-form research reports. An agent may have found the information needed to answer a user's request, yet leave important details out of the report. We call this loss evidence-to-answer attrition. EGRC revises the report to recover useful missing details and improve the answer to the user's request. It uses the evidence already collected, without conducting another search.
In the illustrative policy example in Figure 1, a source document retrieved by the agent states that a policy applies to firms with fewer than 50 employees and excludes subsidiaries. A report saying only that eligibility depends on firm size is supported, but it does not tell the reader which firms qualify. A finding is a specific statement from a source, including the conditions needed to understand it. We prompt a large language model (LLM) to compare the acquired findings with the initial report and identify relevant information that is not fully expressed. We call this model the selector. Its selections are candidate omissions, and “flag” simply means marking a finding as a candidate omission.
The paper combines an analysis of evidence-to-answer attrition with a method for reducing it. Section 3 examines how often reports omit acquired findings and uses completion experiments to study what the writer selects and fully expresses. For these experiments, the original report stays unchanged. In untargeted completion (Generic), the writing model receives summaries of all acquired findings and chooses useful additions. In targeted completion, it receives the same summaries plus the omission list and is asked to address the listed findings. Both use the same acquired findings, writing model, and addition budget. The Two-pass baseline first asks the writer to find its own omissions and then writes additions in a separate call.
Based on this analysis, EGRC (Section 4) organizes the answer around the user's request and requires the writer to make a decision about every original omission. Source-reviewed local edits integrate the missing details into the relevant passages. Section 5 evaluates EGRC's effects on recovery and report quality. It also examines the contribution of its key design choices and tests the method across writing models and research tasks.
We have revised the abstract to describe what may be missing or incompletely expressed in a report, how additions are targeted, and what the writer does during revision. The introduction explains the main terms before discussing the completion results, and Section 3.3 introduces the writing conditions before comparing them. These changes clarify the connection between the problem analysis, the revision method, and its evaluation.