acceptodds
Under review as a conference paper at ICLR 2027

Live Skill Evolution for Forecasting Agents with Delayed Outcome Feedback

Abstract

Human forecasters can improve their predictions by reflecting on and learning from past experience. However, for forecasting agents, systematically accumulating such experience and distilling it into reusable forecasting skills remains challenging. To address this challenge, we introduce Future-SE, a live skill-evolution framework that updates an explicit natural-language skill library while keeping the base model and agent harness fixed. The library separates thinking skills for judgment from working skills for evidence acquisition. At prediction time, Future-SE records matched skill-enabled and no-skill trajectory groups. Once outcomes resolve, it uses outcome feedback, prediction-time market baselines, and trace evidence to refine the library. Experiments with DeepSeek-V4-Flash, GLM-5.2, and Kimi-K2.7-Code on two dynamic forecasting tracks and FutureX show that evolved skills improve forecasting performance over matched no-skill agents. Role ablations show larger gains from thinking-only than from working-only configurations. Analysis of the evolved libraries and their revision histories identifies reusable procedures for question interpretation, prior construction, and structured evidence preparation, while revealing conditional evidence weighting as a persistent difficulty.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Forecasting Agents, Skill Evolution, Continual Learning, LLM Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Q3AMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed an evolutional framework that updates the skills library for LLM forecasting tasks. Unlike approaches that update model parameters or modify the agent harness, Future-SE keeps both fixed and evolves reusable forecasting procedures. The framework collects the trajectory used by agent to perform forecasting, and integrate multi-step correction including modify, create, and consolidate, to enhance the skill catalog. Based on the evaluation experiments, the proposed method is able to improve the forecasting alpha.

Critical strengths

The paper proposed an intuitive idea to separate forecasting execution and skill upgrade. The paper is well organized and easy to follow.

Critical weaknesses

Insufficient statistical evidence to support the robustness of the improvement, and no ablation study to show whether the improved forecasting power is driven by which part of the skill evolution. The proposed evolution process is closely coupled to a particular multi-agent architecture, and the evidence for scalability and generalization is incomplete.

Questions for authors
  1. How statistically robust are the reported forecasting improvements?
  2. Are the comparisons with existing skill evolution methods controlled for agent architecture? Can it be generally applied to any forecasting agent?
3 comments

We thank the reviewer for the constructive feedback. We have added confidence intervals, evolution-mechanism ablations, and comparisons under a shared agent framework.

1. Statistical robustness. Revised Table 2 reports 95% bootstrap confidence intervals for Alpha, the matched no-skill baseline, and Δ\DeltaAlpha. The full Future-SE configuration improves over its matched no-skill baseline under both agent frameworks:

Agent frameworkΔ\DeltaAlpha over no skill95% bootstrap CI
Main agent + workers+0.0105[0.0042, 0.0188]
ReAct+0.0098[0.0051, 0.0141]

Both intervals lie above zero, supporting improvements over the corresponding no-skill baselines in this evaluation. The gains are also observed across all three execution models on the external FutureX benchmark, with score improvements of 6.91–10.09 points.

2. Evolution-mechanism ablations. Appendix B.7 adds ten-epoch ablations using DeepSeek-V4.1-Flash. All variants start with empty skill libraries and no memory or merit records, retain the same per-update data volume, and are evaluated with frozen libraries on the same 180 Polymarket questions. Direct evolution uses the same selected resolved trajectories and their outcomes to directly revise the skill library, without the staged Observe–Modify–Create–Consolidate procedure.

VariantAlpha95% bootstrap CI
Full Future-SE0.0059[0.0019, 0.0099]
w/o no-skill control0.0035[-0.0011, 0.0081]
w/o variance selection0.0011[-0.0025, 0.0047]
w/o historical-version review-0.0010[-0.0110, 0.0090]
w/o Consolidate0.0005[-0.0067, 0.0077]
Direct evolution-0.0046[-0.0107, 0.0015]
No skill-0.0065[-0.0117, -0.0013]

Full Future-SE achieves the highest point estimate.

Table 8 in Appendix B.7 examines diagnosis and a single evolution step from a common epoch-19 skill library using existing DeepSeek trajectories. It evaluates the immediate effects of skill edits through stepwise replay, complementing Table 9's forecasting evaluation after ten epochs of evolution. We independently replay every step of the archived trajectories from its original context, including skill selection and instruction execution. Selection steps use updated routing descriptions, while execution steps use updated skill instructions. Historical evidence and tool observations are reused, and regenerated responses are not propagated to later steps. We assess whether the response repairs an original fault, retains an identified merit, or introduces a new fault or merit. Counts are normalized per prediction, repeated labels within a prediction are deduplicated, and no-skill reference trajectories are excluded from the denominator.

Table 8(a): Fault diagnosis and repair.

VariantContent faults / predictionRouting faults / predictionTotal faults / predictionContent repair rateRouting repair rateTotal repair rateNew faults / prediction
Full Future-SE2.852.955.8035.1%71.8%53.7%1.80
w/o no-skill control2.831.674.5026.5%67.0%41.5%2.20
w/o variance selection2.551.283.8328.8%68.8%42.2%2.37
w/o historical-version review2.731.604.3329.9%60.4%41.2%2.33
w/o Consolidate2.882.855.7328.9%50.3%39.5%1.97

Table 8(b): Merit diagnosis and retention.

VariantMerits / predictionRetention rateNew merits / prediction
Full Future-SE3.1594.2%1.05
w/o no-skill control2.3289.2%0.68
w/o variance selection1.9387.9%0.70
w/o historical-version review2.0886.4%0.95
w/o Consolidate3.0795.7%0.82

Full Future-SE achieves the highest total fault-repair rate (53.7%, versus 39.5–42.2% for the ablations) and introduces the fewest new faults (1.80 per prediction, versus 1.97–2.37). It also identifies the most merits (3.15 per prediction) and produces the most new merits (1.05 per prediction). Removing Consolidate slightly increases merit retention from 94.2% to 95.7%, but lowers total fault repair to 39.5% and new merits to 0.82 per prediction, indicating a tradeoff between preserving existing mechanisms and repairing catalog boundaries.

3. Architecture control and applicability. Revised Table 2 compares Future-SE, Trace2Skill, and SkillOpt under the same ReAct framework and execution model. All three methods independently evolve their skill libraries under ReAct for 20 epochs using the same forecasting questions and the same evolution-data volume (Section 4.2).

Under ReAct, Alpha is 0.0039 for Future-SE, 0.0017 for Trace2Skill, and -0.0006 for SkillOpt, preserving the point-estimate ordering. Future-SE's ReAct skills are independently evolved, demonstrating that its evolution procedure is not restricted to the main agent + workers architecture.

Moreover, our experiments integrate Future-SE's delayed-feedback infrastructure into Trace2Skill and SkillOpt while preserving their core skill-update mechanisms, demonstrating that this infrastructure can be incorporated into existing skill-evolution methods. Other aspects of Future-SE, such as catalog-level skill management and the main agent + workers design, primarily support the study of forecasting itself by making skill use and revision observable and separating judgment from evidence acquisition.

Thank you for your constructive feedback. We look forward to hearing your thoughts on our response. Please let us know if any concerns remain or if you have further questions; we are happy to address them.

Reviewer_bTKuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Future-SE updates an explicit forecasting skill library from delayed real-world outcomes while keeping the base model and agent harness fixed. It separates thinking skills for the main agent from working skills for evidence-gathering workers, storing matched skill/no-skill trajectories and prediction-time market probabilities. After enough events resolve, observers diagnose faults and useful mechanisms, followed by Modify, Create, and Consolidate stages. Experiments cover three execution models, two dynamic forecasting tracks, and external FutureX evaluation, with analyses of skill roles, cross-model transfer, and revision histories.

Critical strengths
  1. The setting is valuable and the treatment of delayed feedback is concrete. Live forecasting requires acting before answers are known and learning after resolution. The method retains immutable prediction-time records and skill versions, distinguishes historical execution versions from the current catalog, and checks whether an old fault still applies before editing. This makes delayed attribution and revision inspectable and provides more information than evaluating a static skill library alone (Sections 3.3–3.4; Appendix A.4).

  2. Multiple models and external evaluation support the practical value of the libraries. All three models improve over their matched no-skill controls on FutureX. Role ablations and cross-model transfer provide complementary observations. Revision histories identify reusable procedures for checking resolution conditions, constructing priors, and preparing evidence, while acknowledging that weighting conflicting evidence remains unresolved. These findings are useful for forecasting-agent design (Table 1; Section 6; Appendices B.3 and C).

Critical weaknesses
  1. Comparisons with existing skill-evolution methods do not adequately control the agent framework. In Table 2, Future-SE uses a main-agent-plus-workers framework, whereas Trace2Skill and SkillOpt use ReAct. Subtracting each method's own no-skill baseline describes improvement, but does not eliminate interactions between skills and the harness. Matching actual model calls, retrieval, and token consumption across frameworks is also not established. The comparison supports the usefulness of the complete system more clearly than the superiority of its evolution method (Sections 4.2 and 5.3).

  2. Evidence for the specific evolution design and the stability of its gains remains limited. Thinking-only/working-only ablations test roles of the resulting skills, rather than isolate the value of matched trajectories, variance-based selection, or the update stages. Live evaluation also changes with question composition and time: each Polymarket run spans one seven-day window, and FutureX contains 68 events. Although figures include bootstrap intervals, event-level paired uncertainty for the key method contrasts is not clearly reported. Exact temporal boundaries between evolution and evaluation, and treatment of repeated events, also need clarification to assess generalization (Section 4.2; Figures 3–5; Tables 1–2).

Questions for authors
  1. Can Future-SE, Trace2Skill, and SkillOpt be compared under the same harness, evolution data, and actual inference budget? If such results are available, do they preserve the ordering in Table 2?

  2. Relative to simple batch reflection and skill revision using the same resolved trajectories, what gains come from variance-based selection, matched skill/no-skill traces, and the staged update procedure? Which existing results isolate these effects?

  3. Please specify evolution and evaluation dates, catalog-freezing times, and event-overlap rules. Can you provide paired confidence intervals for the main method contrasts and explain how dependence within events or adjacent dates is handled?

Skill learning from live delayed feedback is a meaningful problem, and the cross-model and external results indicate that the resulting libraries can be useful. However, framework confounding and limited validation of the specific evolution design and stability of the gains leave the methodological contribution insufficiently established. Comparisons under a shared harness, together with a clearer temporal protocol and paired uncertainty estimates, could materially change my assessment.

2 comments

We thank the reviewer for the constructive feedback. We address the controlled comparisons, evolution mechanisms, and evaluation protocol below.

1. Shared framework and computational cost. Revised Table 2 compares Future-SE, Trace2Skill, and SkillOpt under the same ReAct framework and execution model. All three independently evolve their skill libraries for 20 epochs using the same forecasting questions and evolution-data volume (Section 4.2).

MethodAlphaBaseline AlphaΔ\DeltaAlpha
Future-SE (ReAct)0.0039 [0.0009, 0.0066]-0.0059 [-0.0112, -0.0014]0.0098 [0.0051, 0.0141]
Trace2Skill (ReAct)0.0017 [-0.0005, 0.0043]-0.0059 [-0.0112, -0.0014]0.0076 [0.0043, 0.0120]
SkillOpt (ReAct)-0.0006 [-0.0052, 0.0038]-0.0059 [-0.0112, -0.0014]0.0053 [0.0025, 0.0086]

We acknowledge that Future-SE's advantage over the baselines under ReAct is not statistically established. Nevertheless, completing 20 epochs in approximately 10 days with Trace2Skill and SkillOpt already depends on Future-SE's delayed-feedback infrastructure. Without it, historical timestamp replay projects approximately 163 days under a serial schedule (Appendix B.2). Beyond making live evolution feasible, Future-SE provides a more observable skill-evolution framework for both performance improvement and problem diagnosis.

Table 10 (Appendix B.8) separates forecasting and skill-update token usage. Evolution usage excludes trajectory generation; round totals cover 60 questions with three trajectories each, all using the listed configuration, plus one update. Counts include input and visible output tokens.

Table 10(a): Per-prediction token usage.

ConfigurationInput (K)Visible output (K)
no skill (ReAct)45.130.66
Trace2Skill (ReAct)53.071.35
SkillOpt (ReAct)49.140.95
Future-SE (ReAct)51.421.20
no skill (Main agent + workers)265.335.38
Future-SE (Main agent + workers)318.335.94

Table 10(b): Per-update token usage and per-round totals.

ConfigurationEvolution input (M)Evolution visible output (K)Total input (M)Total visible output (K)
Trace2Skill (ReAct)1.1411.7310.69254.73
SkillOpt (ReAct)0.918.909.76179.90
Future-SE (ReAct)1.4232.3010.68248.30
Future-SE (Main agent + workers)18.60216.0775.901285.27

Under ReAct, Future-SE uses more tokens per skill update, but trajectory generation dominates overall consumption. Total usage is therefore comparable: approximately 10.92M, 10.95M, and 9.94M tokens for Future-SE, Trace2Skill, and SkillOpt, respectively.

The main agent + workers configuration uses more tokens but separates thinking skills for judgment from working skills for evidence acquisition, enabling diagnosis of their respective contributions to forecasting. Moreover, even small Alpha improvements are valuable in forecasting.

2. Contributions of the evolution mechanisms. Appendix B.7 adds ten-epoch ablations using DeepSeek-V4.1-Flash. All variants start with an empty skill library and no memory or merit records, retain the per-update data volume in Section 3.3, and are evaluated with frozen libraries on the same 180 Polymarket questions. Direct evolution uses the same selected resolved trajectories and their outcomes to directly revise the skill library, without the staged Observe–Modify–Create–Consolidate procedure.

VariantAlpha95% bootstrap CI
Full Future-SE0.0059[0.0019, 0.0099]
w/o no-skill control0.0035[-0.0011, 0.0081]
w/o variance selection0.0011[-0.0025, 0.0047]
w/o historical-version review-0.0010[-0.0110, 0.0090]
w/o Consolidate0.0005[-0.0067, 0.0077]
Direct evolution-0.0046[-0.0107, 0.0015]
No skill-0.0065[-0.0117, -0.0013]

Full Future-SE has the highest Alpha point estimate. Direct evolution yields only a small point-estimate improvement over no skill (+0.0019).

Table 8 (Appendix B.7) evaluates diagnosis and a single evolution step from a common epoch-19 skill library using existing DeepSeek trajectories. We independently replay every step of the archived trajectories from its original context, including skill selection and instruction execution. Selection steps use updated routing descriptions, while execution steps use updated skill instructions. Historical evidence and tool observations are reused, and regenerated responses are not propagated to later steps. We assess fault repair, merit retention, and new faults or merits. Counts are per prediction, with repeated labels deduplicated and no-skill reference trajectories excluded from the denominator.

Table 8(a): Fault diagnosis and repair.

VariantContent faults / predictionRouting faults / predictionTotal faults / predictionContent repair rateRouting repair rateTotal repair rateNew faults / prediction
Full Future-SE2.852.955.8035.1%71.8%53.7%1.80
w/o no-skill control2.831.674.5026.5%67.0%41.5%2.20
w/o variance selection2.551.283.8328.8%68.8%42.2%2.37
w/o historical-version review2.731.604.3329.9%60.4%41.2%2.33
w/o Consolidate2.882.855.7328.9%50.3%39.5%1.97

Table 8(b): Merit diagnosis and retention.

VariantMerits / predictionRetention rateNew merits / prediction
Full Future-SE3.1594.2%1.05
w/o no-skill control2.3289.2%0.68
w/o variance selection1.9387.9%0.70
w/o historical-version review2.0886.4%0.95
w/o Consolidate3.0795.7%0.82

Full Future-SE has the highest total fault-repair rate (53.7%, versus 39.5–42.2%) and the fewest new faults (1.80 per prediction, versus 1.97–2.37), while identifying and introducing the most merits. Removing Consolidate slightly improves merit retention (95.7% versus 94.2%) but reduces fault repair and new merits, revealing a tradeoff between preserving existing mechanisms and repairing catalog boundaries.

3. Temporal separation and event sampling. Evolution runs on August 8–17, 2026, followed by Polymarket Run 1 on August 20–26 and Run 2 on August 28–September 3 (Section 4.2). Libraries are frozen on August 17 and unchanged during evaluation. Skill-update events resolve before freezing; evaluation events remain unresolved at prediction time, with no event overlap between the two sets.

For the additional ablations (Appendix B.7), evolution runs on November 6–10; November 10 updates use only questions introduced by November 9 and resolved before updating. Frozen libraries are then evaluated on 180 new questions introduced and forecast on November 10, resolving on November 11–12. These questions and outcomes are excluded from evolution.

We collect 30 evaluation questions once daily, targeting resolution 24–48 hours later and retaining at most one market per event (Section 4.2). This prevents within-round duplication of an event, while the daily shift in the eligible resolution window reduces repetition across adjacent dates.

Reviewer_SpVTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Future-SE, an online skill-evolution framework for future-event forecasting. It keeps the base model and agent harness fixed while evolving an explicit natural-language skill library, split into thinking skills used by the main agent and working skills used by workers. Daily live rollouts produce matched skill/no-skill trajectory groups for the same question. After delayed outcomes resolve, Future-SE performs trace-grounded diagnosis and applies bounded updates through Observe–Modify–Create–Consolidate, while retaining version attribution, merit records, and watch items. Experiments on DeepSeek-V4-Flash, GLM-5.2, and Kimi-K2.7-Code show improvements on FutureX and Polymarket over matched no-skill agents. Analysis of evolved libraries reveals a shared forecasting workflow across models, but also a persistent unresolved difficulty: conditional evidence weighting—how to balance conflicting yet individually valid signals.

Critical strengths

1.Novel and important problem setting: First systematic study of how delayed live outcomes can maintain and update a forecasting skill library, with useful version-attribution and delayed-fault review mechanisms.

2.Convincing external results: On FutureX, full-skill agents outperform matched no-skill agents across all three execution models, with score gains of +6.91 to +10.09 and rank improvements of 22–37 places.

3.Analysis beyond aggregate scores: Merit ledgers, role ablations, and revision histories reveal reusable forecasting procedures and persistent failure modes; the finding that conditional evidence weighting remains unresolved is particularly informative.

4.Reasonably auditable protocol: Same-question group construction, variance-based batch selection, four-stage updates, persistent state, and prompt templates are documented in the appendix.

Critical weaknesses

1.Imperfectly matched baselines: Trace2Skill and SkillOpt are compared under ReAct, whereas Future-SE uses a main-agent-plus-workers framework; the comparison does not isolate the skill-evolution method itself.

2.Moderate methodological novelty: The four-stage skill editing is close to existing skill-library work; the main novelty lies in the combination of delayed outcomes, prediction markets, and version attribution.

3.Selection–execution gap remains open: Skills help only when selected and followed correctly, which the library itself cannot guarantee; live evaluation also makes deterministic reproduction difficult.

Questions for authors

see Critical Weaknesses

1 comment

We thank the reviewer for the positive assessment and constructive feedback. We address the concerns below.

1. Baseline fairness. We have added the Future-SE ReAct result to Table 2. All three methods independently evolve their skill libraries under ReAct for 20 epochs using the same forecasting questions and the same evolution-data volume (Section 4.2).

MethodAlphaBaseline AlphaΔ\DeltaAlpha
Future-SE (ReAct)0.0039 [0.0009, 0.0066]-0.0059 [-0.0112, -0.0014]0.0098 [0.0051, 0.0141]
Trace2Skill (ReAct)0.0017 [-0.0005, 0.0043]-0.0059 [-0.0112, -0.0014]0.0076 [0.0043, 0.0120]
SkillOpt (ReAct)-0.0006 [-0.0052, 0.0038]-0.0059 [-0.0112, -0.0014]0.0053 [0.0025, 0.0086]

For the ReAct comparison, the shared no-skill baseline is -0.0059. The ordering of the point estimates is preserved under the same framework; Future-SE's ReAct skills are independently evolved, not transferred from its main agent + workers setup.

2. Methodological novelty. Our contribution has two main aspects. First, asynchronous collection and reuse of delayed feedback makes live skill evolution practical. Integrating this infrastructure into Trace2Skill and SkillOpt was necessary to make their skill evolution feasible within a practical experimental timeframe. With this infrastructure, both baselines complete 20 epochs in approximately 10 days while retaining their core skill-update mechanisms; without it, repeated waits for outcomes under a serial schedule would make these experiments impractical. Historical timestamp replay estimates approximately 163 days for 20 epochs under a serial prediction–feedback–evolution loop. Even forecasting 180 questions per batch and using only the first 60 with complete feedback—discarding two-thirds of the predictions—still gives an estimated 37.5 days (Appendix B.2).

Second, Future-SE is designed both to improve forecasting performance and to study forecasting itself through skill evolution. Unlike Trace2Skill and SkillOpt, which primarily optimize a consolidated skill directory and a single external skill document, respectively, Future-SE maintains multiple skills, distinguishing thinking skills for judgment from working skills for evidence acquisition. Per-skill memory, fault and merit records, together with versioned invocation logs and observer records, let us trace capability use, failure sources, and revision rationales. Thus, our main contribution is a more observable skill-evolution framework that supports both performance improvement and problem diagnosis.

3. Selection, execution, and reproducibility. We agree that skills cannot guarantee correct selection or execution. We view this as a general limitation of external natural-language skills: their effectiveness depends on the underlying model selecting and following them, making this gap difficult to fully resolve through skill edits alone while keeping the model fixed. Nevertheless, Future-SE explicitly identifies and attempts to repair these issues during skill evolution. Observe labels missed or unnecessary skill selections as routing_error.underselected or routing_error.overselected, and failures to follow or carry out skill instructions as content_error.non_executable (Appendix A.3). Modify uses these diagnoses to revise routing descriptions or instruction bodies. These efforts mitigate the problem, but cannot guarantee correct skill use on every trajectory, as acknowledged in Section 6.3 and Limitations. Regarding reproducibility, live forecasting depends on changing questions and evidence, which makes exact replication difficult. To make the experimental protocol reproducible, we specify the evaluation schedule and frozen-skill protocol in Section 4.2 and provide execution procedures and prompt templates in the appendices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.