acceptodds
Under review as a conference paper at ICLR 2027

Incremental Open-Ended Deep Research with Structured Harness

Abstract

Existing Open-Ended Deep Research (OEDR) systems primarily generate reports from scratch, making them inefficient for scenarios where research reports need to be continuously maintained as new information emerges. We introduce Incremental Open-Ended Deep Research (Incremental-OEDR), a research setting that treats a report as an evolving research state and incrementally updates it by preserving valid knowledge, revising outdated or incomplete content, and incorporating newly available information. To support this setting, we propose Structured Harness, which represents reports as structured collections of outlines, sections, and supporting evidence, and provides structured retrieval, a persistent structured evidence pool, and structured generation for selective report updating and evidence reuse. We further establish a temporal evaluation framework spanning ten years, with Single-Step Task and Long-Chain Task to evaluate incremental updates over both individual transitions and long-term update chains. Extensive Experiments on DeepResearch Bench and DeepConsult under both the Open-source Configuration (OC) and Proprietary Configuration (PC) show that Incremental-OEDR maintains competitive report quality while substantially improving report continuity and reducing research costs. As shown in Figure 1, it achieves up to 0.40 higher content-level ROUGE-L F1, 0.51 higher outline-level EM F1, 33% lower token consumption, and 61% fewer search calls than OEDR on DeepResearch Bench. We will release our code to facilitate future OEDR research in the community.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open-Ended Deep Research, Agent Harness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tux8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Existing Open-Ended Deep Research (OEDR) systems often rely on a less efficient approach: generating reports from scratch. To address this, the authors propose "Incremental Open-Ended Deep Research" (Incremental-OEDR). This method enables the incremental updating of reports by retaining valid knowledge, revising outdated or incomplete content, and integrating newly acquired information. The authors introduce a "Structured Harness," which represents a report as a structured collection comprising an outline, sections, and supporting evidence. This framework facilitates selective report updates by providing structured retrieval, a persistent pool of structured evidence, and a structured generation mechanism. A temporal evaluation over 2016–2025 on DeepResearch Bench and DeepConsult with both single-step and long-chain tasks shows comparable or slightly improved judged report quality, markedly higher continuity, and large reductions in token and search costs relative to one-off OEDR runs.

Critical strengths
  1. Framing OEDR as a continual, stateful process is timely and practically important; the notion of treating the report and its evidence as persistent state is a useful shift from one-shot pipelines. The Structured Harness design (structured representation + retrieval + persistent evidence pool + constrained generation) is a coherent systems contribution, clearly targeting the incremental-update pain point. The evidence-pool refresh with content-similarity gating and claim-level diff analysis is a pragmatic mechanism for focusing updates where sources have changed.
  2. A temporal evaluation protocol (2016–2025) with two tasks (SST and LCT) is well-motivated and exposes year-to-year variation that typical “latest-only” evaluations miss. Results cover two benchmarks and two LLM backends, and report three complementary axes: report quality, continuity, and cost. The LCT vs SST analysis helps attribute efficiency gains to persistent evidence reuse. The update-interval study (10y/5y/1y/1m) provides insight into when incremental maintenance is most cost-effective.
  3. The work addresses a real deployment gap for deep research assistants: keeping long-form research up-to-date at lower cost with higher continuity. If widely adopted, the approach could reduce operational costs and stabilize report structures in industry and research monitoring workflows.
Critical weaknesses
  1. The key technical details of the method are not sufficiently developed. While the principles of structured retrieval, evidence change detection, and partial updates are outlined, the specific mechanisms that ensure the system’s correctness remain largely undefined. Equation (4) defines three components but fails to fully specify the structural relationships between them within the main text. This is insufficient to guarantee that changes to evidence are accurately propagated to all conclusions that rely on that evidence. For instance, consider a report with three separate sections that all cite the same annual statistical report. If that statistical report is revised, the system must be able to determine whether all three sections require updates. Simply storing the URL, the original content, and the text of the sections does not automatically resolve this dependency-propagation issue.

  2. Error propagation risk: preserving prior sections may entrench outdated or incorrect claims; there is no explicit evaluation of factual drift or error accumulation across long chains. The evidence-change detector (char-level threshold + LLM diff) may be brittle to formatting or templated content changes; no robustness or ablation is provided (e.g., threshold sensitivity, false positives/negatives). The method does not incorporate explicit claim verification or span-grounded attribution in the update loop, which would be natural for guarding against stale or unsupported content.

  3. Missing ablations for the Structured Harness components (remove evidence pool; disable structured retrieval; unconstrained generation-only updates) to quantify each part’s contribution. No “naïve incremental” baseline (e.g., prompt an LLM to update the prior report with fresh search, without structured tooling). This makes it hard to isolate the value of the proposed structure vs simply editing the prior report. Continuity metrics (ROUGE/BLEU to prior) can reward inertia; there is no complementary measure of “timeliness” or “update-recall” that the system actually integrates the new/changed facts. Costs are reported as tokens/searches but not wall-clock time or variability; latency and throughput are important in practice.

  4. Decision policy for preserve vs revise vs extend at the section level is not formalized beyond tooling; clearer heuristics/criteria or prompt snippets would improve reproducibility. Outline-level EM F1 and its computation (matching policy across variable-length, hierarchical headings) is under-specified. Minor formatting glitches in Section 4.3/Table 4 make normalization definitions unclear.

Questions for authors
  1. How does the system algorithmically decide whether a section is preserved vs revised vs extended? Can you provide explicit prompts/criteria and any quantitative error rates (e.g., human audit) for these decisions?
  2. What is the empirical precision/recall of your evidence-change detector (τ=0.9 + LLM diff) on a labeled set of updated web pages? How sensitive are results to τ?
  3. Can you add a “naïve incremental” baseline (LLM edits prior report with new search, no structured harness) and an ablation study per component (disable evidence pool, disable structured retrieval, unconstrained generation) to isolate contributions?
  4. How do you mitigate or measure error propagation across LCT (e.g., a prior unsupported claim persisting)? Would integrating span-grounded verification (e.g., FullCite/ENOKI-style) into Structured Generation be feasible, and have you tried it?
  5. Continuity can reward inertia. Can you report an “update coverage” metric (e.g., proportion of newly emergent, externally verifiable facts between t−1 and t that are incorporated) and an “outdated fact removal” rate?
  6. Did you observe cases where the LLM’s parametric knowledge leaked post-cutoff facts despite Tavily end_date filtering? If so, how often, and how did you detect/prevent it?
  7. Beyond tokens and searches, what are the wall-clock time and variance per report under SST and LCT? Are there failure modes (dead links, redirects, high-churn sites) that cause large tail latencies?
  8. How is outline-level EM F1 computed over hierarchical headings—what is considered a match (exact text, normalized text, order-sensitive), and how are insertions/deletions scored?
  9. The methodology section describes the role of structured retrieval, yet it fails to provide a complete retrieval algorithm—neither in Equation (4) nor in the subsequent explanation. For instance:
    • Does the process rely on keyword matching against section headings, or does it employ vector similarity?
    • Is the retrieval based on research tasks, or on evidence of change?
    • How many sections are retrieved? Is there a Top-kk truncation?
    • How are causal relationships spanning multiple sections, shared evidence, and conflicting conclusions handled?
    • How can one ensure that critical historical knowledge—which may have low textual similarity—is not overlooked?
Reviewer_7bFnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work tackles an important problem setting of Incremental-OEDR, where deep research reports are an evolving state and need to be incrementally updated. The authors propopose Structured Harness for this setting, representing reports as structured collections of outlines, sections and supporting evidence. The Structured Harness consists of the Structured Representation of the report along with three components: Structured Retrieval, Structured Evidence Pool and Structured Generation. The authors then propose a evaluation framework to evaluate Incremental-OEDR, consisting of Single-step Tasks and Long-Chain Tasks, and metrics focused on three key dimensions: Report Quality, Report Continuity, and Research Cost. Overall, the authors tackle a timely and important problem, but the design of the continuity metrics appears flawed.

Critical strengths
  1. Practical and relevant problem setting. The authors study and important problem setting that is overlooked in prior deep research benchmarks. Reports that must be maintained over time represent a realistic and timely problem setting, where re-generating from scratch is wasteful and treating the state as evolving context is well-motivated.
  2. Demonstrated cost savings. The results are promising, demonstrating similar RACE scores with fewer searches and fewer tokens on two benchmarks and two model configurations.
Critical weaknesses
  1. Design of continuity metrics. The Researcher and Reporter prompts explicitly prompt the agent to preserve structure, so high ROUGE-L between consecutive reports is a direct consequence of the instruction, not evidence that incremental research works. High continuity is also what a failed update may look like when a system that copies R_{t-1} verbatim scores 1.0 on continuity and near-zero on cost. There is no metric for update correctness i.e., whether facts that changed between t-1 and t were revised, whether stale claims were removed, or whether new post-t-1 developments were incorporated. Without a freshness-recall metric, the three reported axes (quality, continuity, cost) cannot distinguish the proposed method from simply not updating the report.

  2. No component ablation. It would be interesting to understand how structured retrieval, the evidence pool, and structured generation individually contribute to performance and whether all three components are needed by ablating the design of each one.

  3. Results do not report variance. There are no standard deviations, confidence intervals, or number of runs, and LLM-judge variance reported on RACE, which makes it difficult to interpret whether the performance differences are significant or noise.

Questions for authors
  1. How many runs per query and timestamp did the benchmark include?
  2. What are the standard deviations or confidence intervals on the RACE and DeepConsult scores?
Reviewer_D3MbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a Deep Research harness that address the problem of maintaining a research report as new information emerges with time. To test this, the paper introduces an evaluation framework to evaluate incremental updates to a research report. Experiments on DeepResearch Bench, DeepConsult show that their harness improves or maintains the report quality with time.

Critical strengths
  1. The paper tackles a very important and useful problem in open ended deep research. Updating research reports has significant practical value as well.
Critical weaknesses
  1. Overall technical contribution of this paper is very weak. The work introduces some new tools along with a new prompt (at phase 0). This is an engineering contribution and does not provide any new scientific insight to the community.

  2. It is unclear if correct updates are being rewarded. The existing continuity metric measures the ROUGE-L F1 and BLEU-4 scores which only check the similarity between reports at different timesteps. These scores can be very low if major updates are made or very high even if incorrect small updates dominate.

  3. This paper doesnt report the DeepResearch Bench FACT metric which would better capture resused or stale evidence.

  4. Qwen3.5, GPT, etc have parametric knowledge of events from 2015. Simply setting "current date" in the prompt would not prevent implicit parametric memory leakage. A claim level dating would be a slight better measure.

  5. The authors state "our temporal evaluation relies on Tavily’s temporal search restriction, which controls the set of webpages accessible at each timestamp, but cannot simulate how the content of an existing webpage evolves over time". In my view this a major limitation that can cause serious leakage and undermines the entire paper's claims. A fix could be using WayBackMachine perhaps.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.