- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Dr-CiK, a benchmark for evaluating whether deep research agents can actively find useful context for time-series forecasting. Dr-CiK contains 279 tasks and 10342 documents across domains such as economics, environment, transportation, healthcare, and system observability. The benchmark distributes forecasting-relevant evidence across multiple supporting documents and introduces distractors involving incorrect entities, variables, time windows, irrelevant information, and counterfactual histories. It evaluates document retrieval, evidence extraction, distractor filtering, and downstream forecasting. The experiments show that correct supporting evidence can substantially improve forecasting accuracy. However, existing deep research methods recover only a small portion of the key evidence and frequently cite distractor documents, causing forecasting performance to remain similar to or even worse than forecasting without context. These results suggest that the main bottleneck is not whether forecasting models can use textual information, but whether agents can reliably find and synthesize evidence that is truly useful for prediction.
Critical strengths
-
Dr-CiK connects deep research with downstream time-series forecasting. Instead of evaluating retrieval only through relevance or citation quality, it measures whether retrieved evidence actually improves forecasts. This reveals that misleading context can perform worse than using no context.
-
The benchmark supports detailed failure diagnosis. Its evidence, document, distractor, and future-value annotations help separate failures in retrieval, evidence extraction, distractor filtering, and forecasting.
Critical weaknesses
-
The benchmark relies on a predefined reference evidence chain. Supporting documents are generated from this chain, and Evidence Recall is evaluated against it. However, real forecasting may involve multiple valid evidence paths, so the current evaluation may underestimate agents that make accurate forecasts using alternative evidence.
-
The comparison between deep research methods is not fully fair. CODEX uses GPT-5.5 with high reasoning effort, while the other methods mainly use Gemini-3 Flash and may have different search and computational budgets. The experiments therefore cannot clearly separate the contribution of the agent design from that of the underlying model and computational resources.
Questions for authors
-
How does the benchmark handle valid evidence that is not included in the predefined reference chain? Could the evaluation support multiple evidence paths or use human assessment to recognize alternative evidence?
-
Could the authors compare the DR agents using the same underlying model, tool configuration, and search budget to better isolate the contribution of each agent framework?
-
Could the authors conduct additional experiments using real web search or a frozen web snapshot? Some evaluated agents may be specifically optimized for open-web environments, and the local synthetic corpus may not accurately reflect their capabilities or relative rankings.
-
What are the token usage, number of tool calls, runtime, and overall cost of each method? These details are important for evaluating both comparison fairness and practical usefulness.
Thank you for your response. I have already given a positive score and I decided to maintain my score.