acceptodds
Under review as a conference paper at ICLR 2027

RSI-forecaster: A Self-Improving Agent for Event Forecasting

16 views

Abstract

Accurate event forecasting across real-world domains requires task-specific strategies whose development has long relied on substantial human expertise and effort. We formulate autonomous forecasting strategy development as a recursive selfimprovement (RSI) process, where an LLM iteratively executes, evaluates, and revises its forecasting strategy. Yet forecasting creates a fundamental feedback deadlock: the target outcome remains unknown when a prediction is made. We empirically show that resolving this deadlock is non-trivial: naive historical tasks and imperfect backtesting provide little target improvement, and existing RSI methods improve historical performance in backtesting but struggle to reliably transfer gains to the unresolved targets. We introduce RSI-FORECASTER, an end-to-end RSI system that constructs target-relevant historical proxy tasks, replays them in a leakage-controlled open-web environment, and uses predictive and process feedback to hierarchically improve strategies, followed by held-out selection and transfer the resulting gains to the unresolved target. Across three months of evaluation on two popular living forecasting benchmarks, RSI-FORECASTER maintains leading performance and ranks second among 400 submissions on ForecastBench’s preliminary cross-round leaderboard as of the paper submission deadline. Moreover, it substantially outperforms forecasts from the general public and prediction-market crowds, while performing on par with human superforecasters. RSI-FORECASTER consistently improves Brier Index across five LLM backbones, up to 15.54 points.

Primary area:
learning on time series and dynamical systems
Keywords:
Forecasting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_BfQ5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a recursive self-improvement framework, named RSI-Forecaster, that executes, evaluates, and revises the forecasting strategy of LLMs. The framework constructs target-relevant historical proxy tasks, replays them in a leakage-controlled open-web environment, and uses predictive and process feedback to hierarchically improve forecasting strategies. Experiments on two living forecasting benchmarks, ForecastBench and FutureX, demonstrate strong performance against existing methods, with consistent improvements across five LLM backbones. Ablation studies further validate the effectiveness of the key components.

Critical strengths
  1. The research problem in this paper is meaningful. In my view, the most valuable aspect of this paper is that it clearly identifies a core challenge in applying the RSI to event prediction: the actual outcome of a target event is not yet available, making it impossible to provide direct feedback for strategy optimization.
  2. The experiment is complete. It involves different kinds of experiments to evaluate different aspects of the method that support the conclusions.
Critical weaknesses
  1. The definition of Feedback Deadlock is exaggerated. This paper defines the lack of real-world feedback for the predicted target as the core difficulty of RSI in forecasting, but backtesting and policy optimization using historical events are already common practices in forecasting. Therefore, the contribution of this paper is not likely to fundamentally solve a problem where feedback was previously unavailable.
  2. The computational overhead and the fairness of the comparisons are not sufficiently clear. Although the authors set the number of iterations to 8 in Table 2, the actual computational cost per iteration may vary between different methods. The paper lacks an analysis of computational overhead and comparative experiments conducted under an equivalent computational budget.
  3. The gains from RSI have not yet been disentangled from those resulting from the search for and selection of strategies. RSI-Forecaster encompasses recursive strategy improvement, historical task evaluation, held-out selection, and the final assembly of a strategy portfolio. Although the paper includes ablation studies, it lacks a direct comparison against a non-recursive approach that generates multiple independent strategies and filters them via historical backtesting. It remains unclear to what extent the performance improvements are specifically attributable to the recursive improvement mechanism itself.
  4. Strong reliance on the quality of historical proxy tasks. RSI-Forecaster depends heavily on identifying historical tasks that share similar predictive structures with the target problem. The paper employs a setup using 12 historical proxy tasks per target. However, it lacks a sensitivity analysis regarding the number, quality, and distributional diversity of these proxy tasks. Consequently, the robustness of the method in scenarios where historical data is scarce or the underlying distribution shifts remains unclear.
  5. The significance of some results is not yet sufficient. Although RSI-Forecaster achieved the highest average F1 score on FutureX, this evaluation was based on only 73 questions, and the margin over the second-best-performing model was narrow (64.97 vs. 64.32). It remains uncertain whether this advantage possesses sufficient statistical stability.
Questions for authors
  1. What fundamentally distinguishes the proposed feedback deadlock from the challenges already addressed by conventional historical backtesting?
  2. Could the authors report the computational costs of each method in Table 2 and provide comparisons under matched computational budgets?
  3. How does RSI-Forecaster compare against a non-recursive baseline that independently generates and selects strategies under the same computational budget?
  4. How does forecasting performance vary with the number, quality, and distributional similarity of historical proxy tasks?
  5. Could the authors provide confidence intervals or statistical significance tests for the performance gap between RSI-Forecaster and the second-best method on FutureX?
4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer BfQ5

We thank Reviewer BfQ5 for the thoughtful review. We particularly appreciate the recognition of (i) the core challenge of RSI for event forecasting: the target outcome is not yet available, so direct feedback is impossible; and (ii) the completeness of our experiments.

For W1 and Q1: Feedback deadlock vs. conventional backtesting

"Backtesting and policy optimization using historical events are already common practices in forecasting." / "What fundamentally distinguishes the proposed feedback deadlock from the challenges already addressed by conventional historical backtesting?"

We thank the reviewer for this fundamental question, and we agree that historical backtesting itself is standard practice for forecasting.

We would like to clarify that the core difference lies in the forecasting targets. Conventional backtesting often targets a fixed problem, such as a stock-price time series, based on curated point-in-time datasets. Our work instead aims to build general forecasting agents that target arbitrary event questions.

This goal raises two fundamental challenges, which we refer to as the feedback deadlock (Sec. 3): data, i.e., automatically constructing historical proxy tasks for each target, and environment, i.e., high-fidelity backtesting on the open web. Thus, conventional backtesting solutions are either inapplicable or require substantial manual curation. Our RSI-Forecaster aims to address both challenges in an autonomous framework, which is also non-trivial (Table 1): naive proxy tasks and archived backtesting raise Target BI only from 46.33 to 47.35 and 47.32, respectively, versus 52.65 with our design.

In the final version, we will narrow the concept of the feedback deadlock and expand the related work to discuss conventional backtesting, which inspired our solution.

For W2 and Q2: Computational cost

"Although the authors set the number of iterations to 8 in Table 2, the actual computational cost per iteration may vary between different methods." / "Could the authors report the computational costs of each method in Table 2 and provide comparisons under matched computational budgets?"

We thank the reviewer for this suggestion. In the original submission, we used the same number of iterations (Sec. 3), since computational costs naturally vary across autonomous agents due to their adaptive reasoning and tool use.

Following the suggestion, we first compared the LLM API costs. With the same Gemini-3.5-Flash backbone, RSI-Forecaster costs USD 2.88 per target question, versus USD 2.06 on average for the five baselines in Table 2, about 1.4×. The difference comes mainly from reading forecasting traces to guide revisions (process assessment and workflow diagnosis) and from the strategy selection and target adaptation stage (Sec. 4.3).

Further, to verify that the gain does not come from this extra cost, we newly reran every baseline with 12 iterations instead of 8 (8 × 1.4, rounded up) but kept RSI-Forecaster at 8 iterations, which approximately matches the token cost:

MethodIterationsTarget BI
A-Evolve12 (newly added)56.17
Adaptive Auto-Harness12 (newly added)54.92
GEPA12 (newly added)47.29
Meta-Harness12 (newly added)52.58
Continual Harness12 (newly added)49.77
RSI-Forecaster (ours)865.70

At approximately matched cost, RSI-Forecaster still leads the strongest baseline by 9.5 points. We will add this discussion to the final version.

For W3 and Q3: Non-recursive variant

"It lacks a direct comparison against a non-recursive approach that generates multiple independent strategies and filters them via historical backtesting." / "How does RSI-Forecaster compare against a non-recursive baseline that independently generates and selects strategies under the same computational budget?"

We thank the reviewer for suggesting this baseline. To isolate the recursive improvement mechanism, we newly added a non-recursive variant that keeps all other components fixed: the same proposer generates the same number of candidates independently, without historical feedback, followed by the same historical evaluation, held-out selection and final assembly (adaptation + selector):

VariantTarget BI
Non-recursive version60.49
RSI-Forecaster65.70

With all other components held fixed, recursive improvement adds 5.2 points, showing the advantage of our recursive design over the same number of independent proposals. We will include this result in the final version.

For W4 and Q4: Sensitivity to proxy tasks

"It lacks a sensitivity analysis regarding the number, quality, and distributional diversity of these proxy tasks." / "How does forecasting performance vary with the number, quality, and distributional similarity of historical proxy tasks?"

We thank the reviewer for this constructive suggestion.

(1) Number. We use twelve proxy tasks to balance cost and performance (Sec. 6). Following the suggestion, we expanded the set to eighteen:

Proxy tasks (research + selection)Target BI
12 (8 + 4)65.70
18 (12 + 6) (newly added)65.94

Performance is improved with more proxy tasks.

(2) Quality. Figure 3 compares three ways of constructing proxy tasks:

Proxy-task constructionTarget BI
Rule-based retrieval54.09
Embedding similarity57.99
Data agent (ours)65.70

(3) Scarcity. On the 10% of targets whose proxy tasks are least similar to them in text embedding (App. F), RSI-Forecaster still improves Target BI from 59.18 to 62.63.

For W5 and Q5: Statistical significance on FutureX

"This evaluation was based on only 73 questions, and the margin over the second-best-performing model was narrow (64.97 vs. 64.32)."

We agree that the 0.65-point difference alone does not establish statistically significant superiority. Because per-question predictions of the competing submission are unavailable to us, we cannot perform the requested paired comparison. We will describe FutureX as evidence that RSI-Forecaster is competitive with the leading systems.

To provide further evaluation beyond ForecastBench and FutureX, we have newly started a live evaluation on the Kalshi prediction market. On the 29 questions resolved so far, our solution outperforms the market price with p value = 0.007. We kindly ask for more time to collect living results and will include them in the final version.

We thank Reviewer BfQ5 again for the constructive review, and we are very happy to discuss anything that remains open.

Thank you to the authors for the detailed response and additional experiments. The rebuttal has addressed most of my concerns. I appreciate the authors' efforts and clarifications. I maintain my weak acceptance decision.

We sincerely thank Reviewer BfQ5 for acknowledging that our rebuttal has addressed most of the concerns. We welcome any further discussion or clarification if needed.

Reviewer_xSnFOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper improves executable forecasting strategies using resolved historical questions chosen for their relevance to an unresolved target. It combines historical replay, iterative strategy revision, held-out selection, and target adaptation. Evaluation includes retrospective ablations and forecasts submitted before outcomes were known, testing whether improvements transfer to new events.

Critical strengths
  • Submitting forecasts before outcomes are known provides meaningful evidence for the complete system. The within-round comparisons use the same questions, making them more informative than retrospective results alone (Tables 3–4; Appendix K).
  • The method includes useful information controls: historical tasks must be eligible at the target cutoff, selection data are withheld during search, and process assessment is outcome-blind. These address several obvious routes to optimistic evaluation (Sections 4.1–4.3; Appendix H).
Critical weaknesses
  • Repeated strategy search on eight research tasks, followed by selection on four tasks, leaves substantial room for overfitting the proxies. The live results support the system’s usefulness, but do not isolate how well the proposed design controls this risk. Similar average replay and live scores also do not establish that replay identifies the best strategies (Appendix H; Table 14).
  • The controls do not clearly separate iterative improvement from additional search, selection, and averaging. Equal outer-round counts need not imply comparable compute or starting strategies. This limits what the results establish about the proposed improvement mechanism (Table 2; Figure 3).
Questions for authors
  1. How many candidate strategies are evaluated per target? How stable are the gains across alternative chronological proxy splits that keep related event families together?
  2. Which live rounds used a pipeline fixed before evaluation? Were the 200 retrospective targets, including those behind the iteration curve in Table 8, used to develop components or tune hyperparameters?
  3. At the same total compute budget, how does RSI compare with independently proposed strategies using the same historical selection procedure? How does the final selector compare with averaging the same three shortlisted forecasts?
  4. In the existing 15-strategy replay experiment, does backtesting preserve the ranking observed in live evaluation?
2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer xSnF

We thank Reviewer xSnF for the careful and constructive review. We particularly appreciate the recognition that (i) our live submissions provide meaningful evidence for the complete system, and (ii) our information controls block several routes to optimistic evaluation.

For W1-1: Overfitting to the proxy tasks

"Repeated strategy search on eight research tasks, followed by selection on four tasks, leaves substantial room for overfitting the proxies. The live results support the system's usefulness, but do not isolate how well the proposed design controls this risk."

We agree that overfitting to the proxies is the central challenge of RSI for forecasting. Sec. 3 shows that existing RSI methods underperform (Table 2) due to overfitting. Our design addresses it through three mechanisms: outcome-blind process assessment, a withheld selection set, and target adaptation (Sec. 4.3). In Figure 3, we provided initial empirical evidence: removing process assessment lowers Target BI from 65.70 to 60.55, and selecting by held-out rather than research-set score raises it from 60.78 to 62.85. Our new experiment further shows that removing target adaptation lowers Target BI from 65.70 to 63.33. These results indicate that all three designs are effective.

We would like to further clarify that we use twelve proxy tasks to balance cost and performance (Sec. 6). In our new experiment, expanding the proxies from 12 to 18 raises Target BI from 65.70 to 65.94. We expect larger proxy sets to further improve generalization. We will add this discussion to the final version.

For W1-2 and Q4: Strategy rankings of backtesting vs. live evaluation

"Similar average replay and live scores also do not establish that replay identifies the best strategies." / "In the existing 15-strategy replay experiment, does backtesting preserve the ranking observed in live evaluation?"

We thank the reviewer for this insightful question. In Table 14, we have shown that our replay reproduces the mean live score (54.1 vs. 56.2), whereas no leakage control inflates it to 83.4. Following the suggestion, we further computed the agreement for individual strategies on the same 15-strategy replay:

EnvironmentSpearman ρ with live rankingPairwise ranking agreement
No leakage control−0.36736.9%
Ours0.668 (p < 0.01)71.4%

Our environment therefore largely preserves the ranking of individual strategies, whereas without leakage control the ranking breaks down. This is also consistent with our competitive living results (Sec. 5). We will include this analysis in the final version.

For W2-1 and Q3-1: Recursion vs. independent proposals

"The controls do not clearly separate iterative improvement from additional search, selection, and averaging." / "At the same total compute budget, how does RSI compare with independently proposed strategies using the same historical selection procedure?"

We thank the reviewer for this important question. To isolate the recursive improvement mechanism, we added a non-recursive variant that keeps all other components fixed: the same proposer generates the same number of candidates independently, without historical feedback, followed by the same historical evaluation, held-out selection and final assembly (adaptation + selector):

VariantCandidatesTarget BI
Non-recursive version9 independent proposed60.49
RSI-Forecaster9 recursive65.70

With all other components held fixed, recursive improvement adds 5.2 points. Regarding the budget, RSI-Forecaster after two rounds already outperforms the non-recursive version. We will add these results to the final version.

For W2-2 and Q3-2: Selector vs. averaging

"The controls do not clearly separate iterative improvement from additional … averaging." / "How does the final selector compare with averaging the same three shortlisted forecasts?"

We want to clarify that the selector can pick one shortlisted candidate or combine their forecasts with nonnegative weights, so equal-weight averaging is one of its options (App. H). Following the suggestion, we compared the selection vs averaging using the same experimental settings:

Final stepTarget BI
Equal-weight average63.01
LLM selector65.70

The selector adds 2.7 points over averaging. We will include this result in the final version.

For W2-3: Compute and starting strategies across RSI methods

"Equal outer-round counts need not imply comparable compute or starting strategies. This limits what the results establish about the proposed improvement mechanism (Table 2; Figure 3)."

In the original submission, we used the same number of iterations (Sec. 3), since computational costs naturally vary across autonomous agents due to their adaptive reasoning and tool use.

Following the suggestion, we first compared the LLM API costs. With the same Gemini-3.5-Flash backbone, RSI-Forecaster costs USD 2.88 per target question, versus USD 2.06 on average for the five baselines in Table 2, about 1.4×. The difference comes mainly from reading forecasting traces to guide revisions (process assessment and workflow diagnosis) and from the strategy selection and target adaptation stage (Sec. 4.3).

Further, to verify that the gain does not come from this extra cost, we reran every baseline with 12 iterations instead of 8 (8 × 1.4, rounded up) but kept RSI-Forecaster at 8 iterations, which approximately matches the token cost:

MethodIterationsTarget BI
A-Evolve1256.17
Adaptive Auto-Harness1254.92
GEPA1247.29
Meta-Harness1252.58
Continual Harness1249.77
RSI-Forecaster (ours)865.70

At approximately matched cost, RSI-Forecaster still leads the strongest baseline by 9.5 points. We further used RSI-Forecaster's initial strategy (after adaptation) as A-Evolve's starting strategy; its Target BI changes only from 56.17 to 57.02. We will add this discussion to the final version.

For Q1-1: Number of candidate strategies

"How many candidate strategies are evaluated per target?"

We thank the reviewer for this question. The three best on the held-out set are shortlisted for target adaptation and final selection (App. H).

For Q1-2: Proxy tasks splits

"How stable are the gains across alternative chronological proxy splits that keep related event families together?"

We thank the reviewer for this insightful question. We would first like to clarify that related events do not affect the target question itself: the target is never a proxy task, and a proxy is eligible only if its outcome was available before the target's forecast cutoff (Sec. 4.1, App. E). The concern therefore applies only to the split between research and selection tasks. We would further clarify that proxy tasks from related event families are of two kinds:

(a) Earlier instances of a recurring event, such as earlier Starship launches for a target on the next launch, are high-quality proxies. Research uses earlier instances and selection later ones, which tests the forward transfer the target requires.

(b) Contracts on the same event, such as different thresholds of one market, are near-duplicates that make the proxy set redundant and lower its quality. As shown in Sec. 4.1, the data agents are designed to group them by event and audits the selected set for redundant event families. To address this concern, we further scanned all target questions for near-duplicates across research and selection tasks using Claude Opus 5.5. We found this case in 8 targets. Then we moved the near-duplicates into the research set and added new selection tasks:

8 targets with near-duplicates across the splitTarget BI
Original split61.72
Same-event contracts on one side62.03

We will add this quality check as a separate step of the data agent for better performance. We again thank the reviewer for this constructive suggestion.

For Q2: Living rounds and development use

"Which live rounds used a pipeline fixed before evaluation? Were the 200 retrospective targets, including those behind the iteration curve in Table 8, used to develop components or tune hyperparameters?"

We thank the reviewer for this question. We would like to clarify that the pipeline was largely fixed across all living rounds; our main ongoing improvement was extending the data agent to cover more resolved questions (Sec. 4.1). The 200 retrospective targets were not used for development; we constructed a separate pool of prediction questions (about five thousand) for development. We will state these explicitly in the final version.

We thank Reviewer xSnF again for the specific and constructive review, and we are very happy to discuss anything that remains open.

Reviewer_AcP5OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses forecasting (as a task) and the development of strategies to do so better as a way to measure recursive self-improvement (RSI). The paper's multi-step process uses a carefully constructed backtesting environment to restrict it to news from before the event resolves. Then, using Gemini-3.5-Flash, they find that a scaffolded approach (with a 'hierarchical loop') creates a strategy (automated forecaster) that performs quite well on a public (live) benchmark of forecasting accuracy (ForecastBench).

Critical strengths

Three main strengths of this paper are: (1) the authors evaluated this system on live benchmarks of forecasting accuracy, which is rare in this literature and provides significant evidence of the practical value of their proposed system for automated forecasting; and (2) the paper introduces a novel way to give the model, as its training and hypothesis-generating, feedback on accuracy from similar questions (grouped by type) in a way that feels novel to me in this literature.

Critical weaknesses

My main concerns about this paper are as follows:

  1. The gain over the basic ensemble of almost 15 points in Brier Index strikes me as implausible. The baseline ensemble is no better than a coin flip, so it is strange to measure improvements in accuracy and process relative to that baseline. Instead, if we look at a general-purpose LLM and coding framework (like Gemini CLI), which seems to provide high levels of accuracy, the pattern switches, and the novel scaffolding approach here does not seem to improve on basic use of the Gemini CLI.

  2. This paper is framed as being about RSI. But the definition of RSI is murky at best, and I recommend the authors describe their method as a more constrained approach that improves model/system performance on a complex task (forecasting real-world events). If the authors want to make claims about generalizability and RSI, I would expect at least two other applications of this method in non-forecasting domains which show similar improvements and results.

Questions for authors

I think a careful addressing of the two critical weaknesses I identified would convince me about the value of this paper. I have no further questions.

1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer AcP5

We thank Reviewer AcP5 for the careful review. We particularly appreciate the reviewer's recognition that (i) evaluating on live forecasting benchmarks is rare and provides significant evidence of practical value, and (ii) learning from accuracy feedback on similar questions is novel. We also thank the reviewer for stating clearly what would be convincing. We address both points below.

For W1: The improvement over coding agent

"if we look at a general-purpose LLM and coding framework (like Gemini CLI) … the novel scaffolding approach here does not seem to improve on basic use of the Gemini CLI."

We thank the reviewer for this concern. We would like to clarify that RSI-Forecaster still improves on strong coding agents by about 5 points, close to the 6-point gap between superforecasters and the general public (Figure 1c):

BackboneCoding agent (Table 9)RSI-Forecaster (Table 5)Gain
Gemini-3.5-FlashGemini CLI: 60.7665.70+4.94
Claude-Sonnet-5Claude Code: 61.0566.27+5.22

This substantial gain over strong coding agents validates our main idea: feedback from resolved similar questions lets RSI-Forecaster develop target-specific strategies that general-purpose agents lack.

It is also expected that coding agents are stronger baselines than the direct-forecast ensemble and existing RSI methods: (1) forecasting is a complex task involving multi-step search and computation (Sec. 1), and (2) coding agents such as Claude Code have strong general capabilities from sustained investment by frontier labs.

In the final version, we will switch to reporting the gains relative to strong coding agents in the abstract and introduction.

For W2: Scope of the RSI claim

"The definition of RSI is murky at best, and I recommend the authors describe their method as a more constrained approach that improves model/system performance on a complex task (forecasting real-world events)."

We agree, and we will adopt the constrained description and make no claim beyond event forecasting. Specifically, we will clarify RSI as "recursive strategy improvement for forecasting" and scope it explicitly to: an LLM repeatedly executes, evaluates and revises its own executable forecasting strategy for target questions.

We thank Reviewer AcP5 again, and we are very happy to discuss anything that remains open.

Reviewer_WKg1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RSI-Forecaster, a self-improving agentic framework for future-event forecasting. The framework uses resolved historical forecasting questions as proxy tasks to provide feedback when the target outcomes are still unknown. It combines historical task construction, leakage-controlled backtesting, and recursive strategy improvement to develop forecasting strategies. The authors evaluate the framework on ForecastBench and FutureX, showing competitive performance on living forecasting benchmarks. Additional experiments compare different RSI methods and examine the contributions of the proposed components.

Critical strengths
  1. The framework's design is clear and well motivated. Historical task construction, backtesting environments, and recursive strategy improvement are very well-combined into a system.

  2. The comparison between existing RSI methods and RSI-Forecaster suggests that the design of the improvement process matters for transferring historical experience to future forecasting tasks.

  3. The evaluation on living forecasting benchmarks provides useful evidence of the framework's effectiveness beyond historical backtesting.

  4. The ablation studies covered important components, including historical task retrieval, backtesting environments, process assessment, and hierarchical strategy improvement

Critical weaknesses
  1. It may still need ablation study for Target Adaptation and the final LLM-based Strategy Selector. Otherwise it will be difficult to understand which mechanism contributes to the final performance improvement.

  2. The paper compares several RSI methods using the same number of iterations, the actual computational costs are not clearly compared.

  3. In Appendix N, the proposed backtesting environment achieves an average F1 score of 54.1, which is close to the 56.2 obtained in living evaluations. However, this does not necessarily show whether individual forecasting strategies perform consistently. For example, a strategy that performs well in historical backtesting may not necessarily achieve similarly strong results in living evaluations, even if the overall average scores remain similar.

  4. In Table 13, the proposed leakage detection method achieves a recall of 95.4%, meaning that some snippets containing future information are still not detected. The webpage leakage removal experiment shows that the method successfully removes leakage from 25 out of 26 contaminated pages (96.2%). The leakage detection and removal mechanisms are not completely reliable.

Questions for authors
  1. Could the authors provide separate ablation studies for Target Adaptation and the final LLM-based Strategy Selector to clarify their individual contributions?

  2. How do the computational costs, including token usage, API calls, and runtime, compare across the evaluated RSI methods?

  3. Would it be possible to report the correlation between individual strategy scores or rankings in historical backtesting and living evaluations to further validate the reliability of the backtesting environment?

  4. Given the imperfect leakage detection and removal rates reported in Appendix N, to what extent might undetected future information affect historical evaluations and subsequent strategy improvement?

  5. Table 1 does not include Naive RSI Data combined with Archived and Leaky environments. Is there a reason these combinations were excluded, and how might their inclusion help clarify potential interactions between the components?

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer WKg1

We thank Reviewer WKg1 for the positive assessment and the concrete suggestions. We particularly appreciate the reviewer's recognition that (i) the framework is clearly designed and well motivated, (ii) the comparison with existing RSI methods is informative, and (iii) the living evaluation is useful and the ablations are comprehensive. We address each suggestion below and mark results added during the rebuttal as (newly added).

For W1 and Q1: Separate ablations of target adaptation and the selector

"It may still need ablation study for Target Adaptation and the final LLM-based Strategy Selector. Otherwise it will be difficult to understand which mechanism contributes to the final performance improvement."

We thank the reviewer for this helpful suggestion. In Figure 3, we compared the full adaptation and selection stage with two simpler variants that select a single strategy by research-set or held-out performance (60.78 and 62.85 vs. 65.70 BI). Following the reviewer’s suggestion, we now provide separate ablations of target adaptation and the LLM selector, using the same experimental settings:

Equal-weight averageLLM selector
Without target adaptation61.34 (newly added)63.33 (newly added)
With target adaptation63.01 (newly added)65.70

Combining the two components gives the best result, raising Target BI from 61.34 to 65.70. Adaptation also helps the selector more than averaging (+2.37 vs. +1.67), because the selector judges candidates by their target execution traces, which adaptation repairs. We will include these results in the final version.

For W2 and Q2: Computational cost

"The paper compares several RSI methods using the same number of iterations, the actual computational costs are not clearly compared." / "How do the computational costs, including token usage, API calls, and runtime, compare across the evaluated RSI methods?"

We thank the reviewer for this suggestion. In the original submission, we used the same number of iterations (Sec. 3), since computational costs naturally vary across autonomous agents due to their adaptive reasoning and tool use.

Following the suggestion, we first compared the LLM costs. With the same Gemini-3.5-Flash backbone, RSI-Forecaster costs USD 2.88 per target question, versus USD 2.06 on average for the five baselines in Table 2, about 1.4×. The difference comes mainly from reading forecasting traces to guide revisions (process assessment and workflow diagnosis) and from the strategy selection and target adaptation stage (Sec. 4.3).

Further, to verify that the gain does not come from this extra cost, we reran every baseline with 12 iterations instead of 8 (8 × 1.4, rounded up) but kept RSI-Forecaster at 8 iterations, which approximately matches the token cost:

MethodIterationsTarget BI
A-Evolve12 (newly added)56.17
Adaptive Auto-Harness12 (newly added)54.92
GEPA12 (newly added)47.29
Meta-Harness12 (newly added)52.58
Continual Harness12 (newly added)49.77
RSI-Forecaster (ours)865.70

At approximately matched cost, RSI-Forecaster still leads the strongest baseline by 9.5 points. We will add this discussion to the final version.

For W3 and Q3: Strategy rankings of backtesting vs. live evaluation

"A strategy that performs well in historical backtesting may not necessarily achieve similarly strong results in living evaluations, even if the overall average scores remain similar." / "Would it be possible to report the correlation between individual strategy scores or rankings in historical backtesting and living evaluations?"

We thank the reviewer for this insightful question. In Table 14, we have shown that our replay reproduces the mean live score (54.1 vs. 56.2), whereas no leakage control inflates it to 83.4.

Following the suggestion, we further computed the agreement for individual strategies on the same 15-strategy replay:

EnvironmentSpearman ρ with live rankingPairwise ranking agreement
No leakage control−0.36736.9%
Ours0.668 (p < 0.01)71.4%

Our environment therefore largely preserves the ranking of individual strategies, whereas without leakage control the ranking breaks down. This is also consistent with our competitive living results (Sec. 5). We will include this analysis in the final version.

For W4 and Q4: Residual leakage

"Given the imperfect leakage detection and removal rates reported in Appendix N, to what extent might undetected future information affect historical evaluations and subsequent strategy improvement?"

We thank the reviewer for this thoughtful question. We would first like to clarify that it is almost impossible for an open-web backtest to be perfect; our environment keeps leakage low while retaining information sufficiency (Sec. 4.2): it detects 95.4% of outcome-revealing snippets (Table 13) and cleans 25 of 26 leaking pages while discarding only 2.0% of clean pages (App. N). Two end-to-end results further support the effectiveness of our environment:

(1) Our replay stays close to live performance (54.1 vs. 56.2 F1), whereas no leakage control inflates it to 83.4 and archived replay lowers it to 45.0 (Table 14).

(2) Strategies improved in our environment transfer best to target questions (65.70 Target BI), versus 55.35 with archived replay and 51.65 with prompt-only leakage control (Figure 3). Moreover, our living results are free from leakage and remain competitive (Sec. 5). We will discuss residual leakage in the limitations of the final version.

For Q5: Missing experimental results

"Table 1 does not include Naive RSI Data combined with Archived and Leaky environments. Is there a reason these combinations were excluded, and how might their inclusion help clarify potential interactions between the components?"

We thank the reviewer for this suggestion. In the original submission, Table 1 varies one component at a time to isolate each gap (Sec. 3). We agree that the full grid further clarifies their interaction and have run the two remaining cells:

RSI data \ RSI environmentArchivedLeakyOurs
Naive (rule-based)46.93 (newly added)47.11 (newly added)47.35
Ours47.3247.8952.65

The conclusions are consistent: fixing only the data or only the environment adds at most 0.8 points, whereas fixing both adds 5.5–5.7 points, so the two components are complementary. We will add these cells to Table 1 in the final version.

We thank Reviewer WKg1 again for the constructive suggestions, and we are very happy to discuss anything that remains open.

Thank you for the response. The additional ablations and strategy-ranking analysis address several of my concerns. However, the effect of residual leakage on historical scores and recursive strategy improvement remains unquantified. The live results support the system’s effectiveness, but do not establish whether repeated optimization exploits leakage in the backtesting feedback. The extended baseline runs are also useful, although average baseline costs do not establish per-method budget matching. These remaining uncertainties prevent me from strengthening my recommendation, so I will maintain my rating of 3.

AuthorsOpenReview ↗

Follow-up Response to Reviewer WKg1

We thank Reviewer WKg1 for the follow-up discussion. We are glad that the additional ablations and strategy-ranking analysis addressed most of the concerns, and we would like to take this opportunity to address the two remaining ones.

Effect of residual leakage.

We would like to clarify that, in addition to the leakage control in the environment (Sec. 4.2), our RSI method is also designed to mitigate interference from leakage: the outcome-blind process assessment and the LLM selector consider the soundness of the forecasting process (Sec. 4.3, App. H), rather than relying on predictive scores which might be leakage-inflated. We further verify this with an additional experiment. For each target question, we took a strategy evolved from the leaky environment (Sec. 3, App. D), which scores well in backtesting by using leakage, and inserted it into the shortlist of our system. We find that, on every target question, the LLM selector neither selects the inserted leaky strategy nor gives it any weight in the final forecast. We will add this discussion to the final version.

Per-method budget matching.

We thank the reviewer for this suggestion. Following the suggestion, we used the cost of our RSI-Forecaster as the budget and let each baseline keep evolving until its cumulative cost reached or exceeded this budget:

MethodTarget BI with matched budget
A-Evolve56.12
Adaptive Auto-Harness54.98
GEPA46.88
Meta-Harness52.75
Continual Harness49.61
RSI-Forecaster (ours)65.70

The conclusion is consistent with our original results.

We thank Reviewer WKg1 again for the careful review and constructive suggestions, and we are happy to address any remaining concerns.

Comments on the paper

1 comment

Dear Reviewers,

The author-reviewer discussion phase is ending soon. If you have not done so yet, please read the author rebuttals and respond.

Please recall that, as a reviewer, you have agreed to engage in discussion with the authors to clarify and improve the paper. Failure to do so may result in penalties on your own submissions, as per the official ICLR policy. Given the high volume of submissions, your participation is crucial.

Thank you for your time and help.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.