acceptodds
Under review as a conference paper at ICLR 2027

Dr-CiK: A Benchmark testing Deep Research for Context-aided Forecasting

Abstract

Time series forecasting in real-world settings often depends not only on historical observations, but also on external context that must be actively discovered from noisy, heterogeneous information sources. Yet existing context-aided forecasting benchmarks typically assume that the supporting context is already provided, leaving open whether agents can identify it on their own. Therefore, we introduce Dr-CiK, a benchmark for evaluating whether agents can retrieve forecasting-relevant supporting context from a document corpus, filter out distractors, distill the retrieved context into forecast-useful evidence, and generate forecasts supported by that evidence. Through context ablations and evaluations of state-of-the-art deep research and forecasting methods paired together, we show that high-quality context substantially improves forecasting performance in Dr-CiK. However, existing DR methods recover limited ground-truth supporting evidence, are frequently misled by distractors, and can cause forecasters to perform worse with retrieved context than without context. Our results motivate research on agents that search for the right context to predict the future.

Primary area:
datasets and benchmarks
Keywords:
Context-aided Forecasting, Deep Research, Agents, Benchmark, Dataset

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5C2mOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Dr-CiK, a benchmark for evaluating whether deep research agents can actively find useful context for time-series forecasting. Dr-CiK contains 279 tasks and 10342 documents across domains such as economics, environment, transportation, healthcare, and system observability. The benchmark distributes forecasting-relevant evidence across multiple supporting documents and introduces distractors involving incorrect entities, variables, time windows, irrelevant information, and counterfactual histories. It evaluates document retrieval, evidence extraction, distractor filtering, and downstream forecasting. The experiments show that correct supporting evidence can substantially improve forecasting accuracy. However, existing deep research methods recover only a small portion of the key evidence and frequently cite distractor documents, causing forecasting performance to remain similar to or even worse than forecasting without context. These results suggest that the main bottleneck is not whether forecasting models can use textual information, but whether agents can reliably find and synthesize evidence that is truly useful for prediction.

Critical strengths
  1. Dr-CiK connects deep research with downstream time-series forecasting. Instead of evaluating retrieval only through relevance or citation quality, it measures whether retrieved evidence actually improves forecasts. This reveals that misleading context can perform worse than using no context.

  2. The benchmark supports detailed failure diagnosis. Its evidence, document, distractor, and future-value annotations help separate failures in retrieval, evidence extraction, distractor filtering, and forecasting.

Critical weaknesses
  1. The benchmark relies on a predefined reference evidence chain. Supporting documents are generated from this chain, and Evidence Recall is evaluated against it. However, real forecasting may involve multiple valid evidence paths, so the current evaluation may underestimate agents that make accurate forecasts using alternative evidence.

  2. The comparison between deep research methods is not fully fair. CODEX uses GPT-5.5 with high reasoning effort, while the other methods mainly use Gemini-3 Flash and may have different search and computational budgets. The experiments therefore cannot clearly separate the contribution of the agent design from that of the underlying model and computational resources.

Questions for authors
  1. How does the benchmark handle valid evidence that is not included in the predefined reference chain? Could the evaluation support multiple evidence paths or use human assessment to recognize alternative evidence?

  2. Could the authors compare the DR agents using the same underlying model, tool configuration, and search budget to better isolate the contribution of each agent framework?

  3. Could the authors conduct additional experiments using real web search or a frozen web snapshot? Some evaluated agents may be specifically optimized for open-web environments, and the local synthetic corpus may not accurately reflect their capabilities or relative rankings.

  4. What are the token usage, number of tool calls, runtime, and overall cost of each method? These details are important for evaluating both comparison fairness and practical usefulness.

1 comment

Thank you for your response. I have already given a positive score and I decided to maintain my score.

Reviewer_GP2kOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Dr-CiK, a benchmark for evaluating deep research agents in context-aided time-series forecasting. It contains 279 forecasting tasks and 10,342 documents, including supporting evidence and five types of distractors. The benchmark evaluates agents' ability to retrieve relevant evidence, avoid distractors, and synthesize information for downstream forecasting. Experiments with different deep research agents and forecasting models show that ground-truth context improves forecasting accuracy, while retrieved context often provides limited benefits or degrades performance due to incomplete evidence recovery and distractor contamination.

Critical strengths
  1. The paper focuses on the gap between context-aided forecasting with provided context and forecasting that requires actively retrieving relevant information. This setting reflects an important challenge not directly evaluated by existing forecasting benchmarks.
  2. The benchmark includes labeled supporting evidence and multiple types of distractors, allowing separate evaluation of evidence retrieval, distractor avoidance, and downstream forecasting performance. The context ablations also help identify how retrieval quality affects forecasting accuracy.
Critical weaknesses
  1. The benchmark mainly extends existing context-aided forecasting tasks by introducing document retrieval and distractors. The fundamental novelty beyond combining existing retrieval and forecasting benchmarks remains unclear.
  2. The benchmark relies on a fixed, largely synthetic document corpus rather than realistic information sources. It is unclear whether the task truly evaluates deep research capabilities or primarily measures document retrieval and filtering.
  3. Supporting evidence is derived from predefined contexts rather than independently validated causal relationships. The generation and verification procedures ensure consistency but do not necessarily establish genuine predictive relevance.
  4. Most distractors are constructed using predefined entity, temporal, or historical inconsistencies. Given that 6,583 of 6,975 distractors pass explicit rule-based distinguishability checks, it remains unclear whether rejecting them requires substantial forecasting-oriented reasoning.
  5. The multi-hop structure is created by artificially expanding evidence into linear reasoning chains. The declining recall with additional hops may reflect increased retrieval burden rather than genuinely more complex reasoning.
  6. Codex uses GPT-5.5 with high reasoning effort, while the other DR methods use Gemini-3 Flash. Without controlling backbone capabilities and computational budgets, the performance differences cannot be attributed to DR methodologies alone.
  7. The correlation between evidence coverage and forecasting improvement is relatively weak (Spearman = 0.29). Additionally, the poor performance with raw supporting documents suggests that context utilization and synthesis may be major bottlenecks.
  8. In Appendix A.1.1, annotators construct prospective evidence with access to future time-series observations. This raises concerns about whether the evidence reflects information available at prediction time or retrospectively constructed explanations.
Questions for authors
  1. How do the authors ensure that the constructed supporting evidence is genuinely predictive rather than retrospectively aligned with known future outcomes? In particular, how is hindsight bias addressed when annotators have access to future observations?
  2. What evidence demonstrates that Dr-CiK evaluates genuine deep research and multi-hop reasoning capabilities rather than conventional document retrieval, rule-based distractor filtering, and increased retrieval complexity? Could the authors provide stronger baselines to validate these distinctions?
  3. Could the authors provide comparisons under matched LLM backbones and computational budgets? Given the relatively weak correlation between evidence recall and forecasting improvement, how do the authors establish that retrieval is the primary bottleneck rather than evidence synthesis or context utilization?
Reviewer_pypgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a benchmark to evaluate whether deep research agents can discover and synthesize external information to improve time-series forecasting. Experiments with various deep research agents and forecasting models reveal a substantial gap between the benefits of ground truth evidence and agent retrieved context. These findings highlight evidence discovery, distractor rejection, and context synthesis as critical bottlenecks in developing reliable deep research agents for forecasting.

Critical strengths

The research question itself is novel and cracial to explore, and the data generation pipeline is solid and thorough, with human verification. The paper provide comprehensive experiments to test how the current deep serach agent fails in the context augmented forecasting task.

Critical weaknesses

How to make sure that the distractors will not affect the forecasting outcome? For example, any side effect or butterfly effect of the distractors may affect the final result. However, in the paper, the reasoning chain is generated before adding distractors, and the authors' haven't verified whether the distractors are going to affect the reasoning chain.

How does the synthesized environment align with real-world time-series forecasting? For example, how does the distractors come from, are they defined by human or distilled from real-world scenarios? Providing this can better illustrate the usefulness of the benchmark.

Questions for authors

According to the weaknesses, can the aurthors provide any analysis or experiment showing the soundess of the synthesized environment, and the distractors are independent from the forecasting reasoning chain or the final outcome?

Open Question: Have the authors considered studying deep search in event forecasting? Can the proposed method generalize to event forecasting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.