- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a recursive self-improvement framework, named RSI-Forecaster, that executes, evaluates, and revises the forecasting strategy of LLMs. The framework constructs target-relevant historical proxy tasks, replays them in a leakage-controlled open-web environment, and uses predictive and process feedback to hierarchically improve forecasting strategies. Experiments on two living forecasting benchmarks, ForecastBench and FutureX, demonstrate strong performance against existing methods, with consistent improvements across five LLM backbones. Ablation studies further validate the effectiveness of the key components.
Critical strengths
- The research problem in this paper is meaningful. In my view, the most valuable aspect of this paper is that it clearly identifies a core challenge in applying the RSI to event prediction: the actual outcome of a target event is not yet available, making it impossible to provide direct feedback for strategy optimization.
- The experiment is complete. It involves different kinds of experiments to evaluate different aspects of the method that support the conclusions.
Critical weaknesses
- The definition of Feedback Deadlock is exaggerated. This paper defines the lack of real-world feedback for the predicted target as the core difficulty of RSI in forecasting, but backtesting and policy optimization using historical events are already common practices in forecasting. Therefore, the contribution of this paper is not likely to fundamentally solve a problem where feedback was previously unavailable.
- The computational overhead and the fairness of the comparisons are not sufficiently clear. Although the authors set the number of iterations to 8 in Table 2, the actual computational cost per iteration may vary between different methods. The paper lacks an analysis of computational overhead and comparative experiments conducted under an equivalent computational budget.
- The gains from RSI have not yet been disentangled from those resulting from the search for and selection of strategies. RSI-Forecaster encompasses recursive strategy improvement, historical task evaluation, held-out selection, and the final assembly of a strategy portfolio. Although the paper includes ablation studies, it lacks a direct comparison against a non-recursive approach that generates multiple independent strategies and filters them via historical backtesting. It remains unclear to what extent the performance improvements are specifically attributable to the recursive improvement mechanism itself.
- Strong reliance on the quality of historical proxy tasks. RSI-Forecaster depends heavily on identifying historical tasks that share similar predictive structures with the target problem. The paper employs a setup using 12 historical proxy tasks per target. However, it lacks a sensitivity analysis regarding the number, quality, and distributional diversity of these proxy tasks. Consequently, the robustness of the method in scenarios where historical data is scarce or the underlying distribution shifts remains unclear.
- The significance of some results is not yet sufficient. Although RSI-Forecaster achieved the highest average F1 score on FutureX, this evaluation was based on only 73 questions, and the margin over the second-best-performing model was narrow (64.97 vs. 64.32). It remains uncertain whether this advantage possesses sufficient statistical stability.
Questions for authors
- What fundamentally distinguishes the proposed feedback deadlock from the challenges already addressed by conventional historical backtesting?
- Could the authors report the computational costs of each method in Table 2 and provide comparisons under matched computational budgets?
- How does RSI-Forecaster compare against a non-recursive baseline that independently generates and selects strategies under the same computational budget?
- How does forecasting performance vary with the number, quality, and distributional similarity of historical proxy tasks?
- Could the authors provide confidence intervals or statistical significance tests for the performance gap between RSI-Forecaster and the second-best method on FutureX?
4 comments
For W4 and Q4: Sensitivity to proxy tasks
"It lacks a sensitivity analysis regarding the number, quality, and distributional diversity of these proxy tasks." / "How does forecasting performance vary with the number, quality, and distributional similarity of historical proxy tasks?"
We thank the reviewer for this constructive suggestion.
(1) Number. We use twelve proxy tasks to balance cost and performance (Sec. 6). Following the suggestion, we expanded the set to eighteen:
| Proxy tasks (research + selection) | Target BI |
|---|---|
| 12 (8 + 4) | 65.70 |
| 18 (12 + 6) (newly added) | 65.94 |
Performance is improved with more proxy tasks.
(2) Quality. Figure 3 compares three ways of constructing proxy tasks:
| Proxy-task construction | Target BI |
|---|---|
| Rule-based retrieval | 54.09 |
| Embedding similarity | 57.99 |
| Data agent (ours) | 65.70 |
(3) Scarcity. On the 10% of targets whose proxy tasks are least similar to them in text embedding (App. F), RSI-Forecaster still improves Target BI from 59.18 to 62.63.
For W5 and Q5: Statistical significance on FutureX
"This evaluation was based on only 73 questions, and the margin over the second-best-performing model was narrow (64.97 vs. 64.32)."
We agree that the 0.65-point difference alone does not establish statistically significant superiority. Because per-question predictions of the competing submission are unavailable to us, we cannot perform the requested paired comparison. We will describe FutureX as evidence that RSI-Forecaster is competitive with the leading systems.
To provide further evaluation beyond ForecastBench and FutureX, we have newly started a live evaluation on the Kalshi prediction market. On the 29 questions resolved so far, our solution outperforms the market price with p value = 0.007. We kindly ask for more time to collect living results and will include them in the final version.
We thank Reviewer BfQ5 again for the constructive review, and we are very happy to discuss anything that remains open.
Thank you to the authors for the detailed response and additional experiments. The rebuttal has addressed most of my concerns. I appreciate the authors' efforts and clarifications. I maintain my weak acceptance decision.
We sincerely thank Reviewer BfQ5 for acknowledging that our rebuttal has addressed most of the concerns. We welcome any further discussion or clarification if needed.
Rebuttal to Reviewer BfQ5
We thank Reviewer BfQ5 for the thoughtful review. We particularly appreciate the recognition of (i) the core challenge of RSI for event forecasting: the target outcome is not yet available, so direct feedback is impossible; and (ii) the completeness of our experiments.
For W1 and Q1: Feedback deadlock vs. conventional backtesting
We thank the reviewer for this fundamental question, and we agree that historical backtesting itself is standard practice for forecasting.
We would like to clarify that the core difference lies in the forecasting targets. Conventional backtesting often targets a fixed problem, such as a stock-price time series, based on curated point-in-time datasets. Our work instead aims to build general forecasting agents that target arbitrary event questions.
This goal raises two fundamental challenges, which we refer to as the feedback deadlock (Sec. 3): data, i.e., automatically constructing historical proxy tasks for each target, and environment, i.e., high-fidelity backtesting on the open web. Thus, conventional backtesting solutions are either inapplicable or require substantial manual curation. Our RSI-Forecaster aims to address both challenges in an autonomous framework, which is also non-trivial (Table 1): naive proxy tasks and archived backtesting raise Target BI only from 46.33 to 47.35 and 47.32, respectively, versus 52.65 with our design.
In the final version, we will narrow the concept of the feedback deadlock and expand the related work to discuss conventional backtesting, which inspired our solution.
For W2 and Q2: Computational cost
We thank the reviewer for this suggestion. In the original submission, we used the same number of iterations (Sec. 3), since computational costs naturally vary across autonomous agents due to their adaptive reasoning and tool use.
Following the suggestion, we first compared the LLM API costs. With the same Gemini-3.5-Flash backbone, RSI-Forecaster costs USD 2.88 per target question, versus USD 2.06 on average for the five baselines in Table 2, about 1.4×. The difference comes mainly from reading forecasting traces to guide revisions (process assessment and workflow diagnosis) and from the strategy selection and target adaptation stage (Sec. 4.3).
Further, to verify that the gain does not come from this extra cost, we newly reran every baseline with 12 iterations instead of 8 (8 × 1.4, rounded up) but kept RSI-Forecaster at 8 iterations, which approximately matches the token cost:
At approximately matched cost, RSI-Forecaster still leads the strongest baseline by 9.5 points. We will add this discussion to the final version.
For W3 and Q3: Non-recursive variant
We thank the reviewer for suggesting this baseline. To isolate the recursive improvement mechanism, we newly added a non-recursive variant that keeps all other components fixed: the same proposer generates the same number of candidates independently, without historical feedback, followed by the same historical evaluation, held-out selection and final assembly (adaptation + selector):
With all other components held fixed, recursive improvement adds 5.2 points, showing the advantage of our recursive design over the same number of independent proposals. We will include this result in the final version.