- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposed an evolutional framework that updates the skills library for LLM forecasting tasks. Unlike approaches that update model parameters or modify the agent harness, Future-SE keeps both fixed and evolves reusable forecasting procedures. The framework collects the trajectory used by agent to perform forecasting, and integrate multi-step correction including modify, create, and consolidate, to enhance the skill catalog. Based on the evaluation experiments, the proposed method is able to improve the forecasting alpha.
Critical strengths
The paper proposed an intuitive idea to separate forecasting execution and skill upgrade. The paper is well organized and easy to follow.
Critical weaknesses
Insufficient statistical evidence to support the robustness of the improvement, and no ablation study to show whether the improved forecasting power is driven by which part of the skill evolution. The proposed evolution process is closely coupled to a particular multi-agent architecture, and the evidence for scalability and generalization is incomplete.
Questions for authors
- How statistically robust are the reported forecasting improvements?
- Are the comparisons with existing skill evolution methods controlled for agent architecture? Can it be generally applied to any forecasting agent?
3 comments
3. Architecture control and applicability. Revised Table 2 compares Future-SE, Trace2Skill, and SkillOpt under the same ReAct framework and execution model. All three methods independently evolve their skill libraries under ReAct for 20 epochs using the same forecasting questions and the same evolution-data volume (Section 4.2).
Under ReAct, Alpha is 0.0039 for Future-SE, 0.0017 for Trace2Skill, and -0.0006 for SkillOpt, preserving the point-estimate ordering. Future-SE's ReAct skills are independently evolved, demonstrating that its evolution procedure is not restricted to the main agent + workers architecture.
Moreover, our experiments integrate Future-SE's delayed-feedback infrastructure into Trace2Skill and SkillOpt while preserving their core skill-update mechanisms, demonstrating that this infrastructure can be incorporated into existing skill-evolution methods. Other aspects of Future-SE, such as catalog-level skill management and the main agent + workers design, primarily support the study of forecasting itself by making skill use and revision observable and separating judgment from evidence acquisition.
Thank you for your constructive feedback. We look forward to hearing your thoughts on our response. Please let us know if any concerns remain or if you have further questions; we are happy to address them.
We thank the reviewer for the constructive feedback. We have added confidence intervals, evolution-mechanism ablations, and comparisons under a shared agent framework.
1. Statistical robustness. Revised Table 2 reports 95% bootstrap confidence intervals for Alpha, the matched no-skill baseline, and ΔAlpha. The full Future-SE configuration improves over its matched no-skill baseline under both agent frameworks:
Both intervals lie above zero, supporting improvements over the corresponding no-skill baselines in this evaluation. The gains are also observed across all three execution models on the external FutureX benchmark, with score improvements of 6.91–10.09 points.
2. Evolution-mechanism ablations. Appendix B.7 adds ten-epoch ablations using DeepSeek-V4.1-Flash. All variants start with empty skill libraries and no memory or merit records, retain the same per-update data volume, and are evaluated with frozen libraries on the same 180 Polymarket questions. Direct evolution uses the same selected resolved trajectories and their outcomes to directly revise the skill library, without the staged Observe–Modify–Create–Consolidate procedure.
Full Future-SE achieves the highest point estimate.
Table 8 in Appendix B.7 examines diagnosis and a single evolution step from a common epoch-19 skill library using existing DeepSeek trajectories. It evaluates the immediate effects of skill edits through stepwise replay, complementing Table 9's forecasting evaluation after ten epochs of evolution. We independently replay every step of the archived trajectories from its original context, including skill selection and instruction execution. Selection steps use updated routing descriptions, while execution steps use updated skill instructions. Historical evidence and tool observations are reused, and regenerated responses are not propagated to later steps. We assess whether the response repairs an original fault, retains an identified merit, or introduces a new fault or merit. Counts are normalized per prediction, repeated labels within a prediction are deduplicated, and no-skill reference trajectories are excluded from the denominator.
Table 8(a): Fault diagnosis and repair.
Table 8(b): Merit diagnosis and retention.
Full Future-SE achieves the highest total fault-repair rate (53.7%, versus 39.5–42.2% for the ablations) and introduces the fewest new faults (1.80 per prediction, versus 1.97–2.37). It also identifies the most merits (3.15 per prediction) and produces the most new merits (1.05 per prediction). Removing Consolidate slightly increases merit retention from 94.2% to 95.7%, but lowers total fault repair to 39.5% and new merits to 0.82 per prediction, indicating a tradeoff between preserving existing mechanisms and repairing catalog boundaries.