acceptodds
Under review as a conference paper at ICLR 2027

FORESIGHT-9: Prospective and Process-Aware Evaluation of Adaptive Trading Agents

Abstract

Trading-agent evaluation can fail at three successive levels: a ranking on realized history may not transfer reliably to alternative future paths, prospective returns may not exceed simple non-learning controls, and a favorable terminal outcome may persist after adaptation itself has stopped. We introduce FORESIGHT-9, a prospective and process-aware benchmark with nine auditable counterfactual worldlines branching from a common July 2026 information boundary. A com- mon contract fixes the market panel, cadence, constraints, costs, and account writes while preserving framework-specific research and proposal mechanisms. Across 36 long-horizon runs from two frameworks and two model backbones, Native-Policy Retrospective Replay yields a historical ranking whose median Spearman correlation with prospective rankings is −0.8; its winner remains first in only 1/9 worldlines. Prospectively, periodic equal weighting and a causal inverse- volatility rule each outperform 31 of 36 agent–worldline runs, showing that pos- itive agent returns need not establish incremental value from online adaptation. Finally, in a high-return WL8 run the live research library collapses while the declared ensemble persists and executed holdings converge toward equal weight: the run keeps earning after meaningful self-evolution has ceased. FORESIGHT- 9 therefore evaluates future robustness, incremental value over simple controls, and whether the intended adaptive mechanism remains functional. We release the worldlines, trajectories, audit traces, and deterministic regeneration scripts.

Primary area:
datasets and benchmarks
Keywords:
agent evaluation, foundation-model agents, adaptive trading agents, datasets and benchmarks, counterfactual worldlines, retrospective replay, process-aware evaluation, financial agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uGd9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FORESIGHT-9, a benchmark for evaluating adaptive financial trading agents under multiple synthetic future market scenarios. The benchmark constructs nine deterministic counterfactual worldlines from a shared July 2026 information boundary and standardizes market observations, portfolio constraints, execution, and transaction costs. It evaluates two adaptive trading frameworks with two model backbones, yielding 36 long-horizon runs. The study examines whether retrospective rankings transfer to alternative future scenarios, whether adaptive agents outperform simple non-learning strategies, and whether their internal adaptation mechanisms remain functional. The authors report weak retrospective-to-prospective rank transfer, frequent underperformance relative to periodic equal weighting, and cases where portfolio returns remain positive despite degradation of the underlying research process.

Critical strengths

S1. Important Evaluation Perspective. The paper makes a useful distinction between retrospective performance, incremental value over simple trading rules, and the continued functionality of an agent's adaptation process. This process-aware perspective addresses a meaningful limitation of outcome-only evaluations.

S2. Auditable Evaluation Infrastructure. FORESIGHT-9 provides deterministic scenario regeneration, a shared information boundary, standardized execution constraints, and persistent records of factor libraries, proposed portfolios, and executed holdings (Sections 3–4). These design choices improve transparency and facilitate the diagnosis of failures that terminal performance metrics alone may miss.

Critical weaknesses

W1. Limited Validity and Representativeness of the Synthetic Evaluation Environment (Fundamental). The central findings rely on nine manually specified, agent-assisted macro-financial scenarios whose trajectories are generated through anchored interpolation and Brownian-bridge perturbations (Section 3). The authors acknowledge that these paths are not sampled from a calibrated joint return distribution and that cross-asset covariance is not empirically calibrated. Moreover, the evaluation uses synthetic yield-level exposures and combines local-currency returns without FX conversion (Section 3.1). These assumptions substantially limit financial realism. More importantly, the unusually strong performance of periodic equal weighting may reflect the repeated re-anchoring of the synthetic paths, as the authors acknowledge in Section 6. Consequently, the finding that simple non-learning strategies outperform adaptive agents may be specific to the scenario construction rather than representative of realistic future market conditions. Independent scenario validation and sensitivity analyses are needed to support the benchmark's broader conclusions.

W2. Insufficient Evidence for General Claims About Retrospective Rankings and Adaptive-Agent Failure (Fundamental). The retrospective-to-prospective analysis compares only four agent configurations, each evaluated with one run per worldline (Sections 4–5). Furthermore, Native-Policy Retrospective Replay evaluates a frozen final policy on its own historical training period, whereas prospective evaluation permits continued adaptation. The resulting rank disagreement therefore combines distribution shift with a change in learning protocol and cannot isolate why retrospective rankings fail. The median Spearman correlation of -0.80 is descriptive over only four configurations and nine constructed scenarios. Similarly, the process-collapse example in WL8 is informative but does not establish the prevalence or generality of such failures across adaptive-agent architectures. The lack of multi-seed replication and stronger comparisons between frozen and adaptive prospective policies weakens the evidence for the benchmark's central claims.

Questions for authors
  1. Scenario validity: What evidence supports the financial realism of the nine constructed worldlines? Are there sensitivity analyses using different scenario anchors, bridge seeds, or empirically calibrated cross-asset dependence that assess whether the baseline advantages and agent rankings remain stable?

  2. Rank-transfer interpretation: Can the authors clarify how much of the retrospective-to-prospective ranking difference may result from comparing a frozen retrospective policy with continuously adapting prospective agents? Are frozen-policy prospective results available for a more directly matched comparison?

  3. Robustness and process validity: Are repeated runs or additional process-failure summaries available across configurations and worldlines? In particular, how frequently do research-state collapse or inconsistencies between declared and executed portfolios occur, beyond the WL8 example?

Reviewer_gXojOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces nine synthetic market continuations from a shared July 2026 information boundary and evaluates two trading-agent frameworks with two model backbones, producing 36 long-horizon runs. It combines retrospective ranking diagnostics, prospective comparisons with simple allocation rules, and audits of persistent research and execution state. Reported findings include a median retrospective-to-prospective rank correlation of −0.80, equal-weight and inverse-volatility controls each outperforming 31/36 agent runs, and a profitable run whose factor library and executed policy have diverged.

Critical strengths
  1. The separation of return, incremental value over simple controls, and functioning adaptation is useful. The WL8 audit demonstrates why a profitable account can conceal a failed research-to-execution mechanism.

  2. The benchmark specifies temporal access, costs, constraints, deterministic scenarios, and factor-lifecycle records. These make its negative findings inspectable rather than leaving agent performance as an unexplained terminal score.

Critical weaknesses

1.The generator changes the information and trading environment substantially: cross-asset covariance is not empirically calibrated, forward volume is constant, and VWAP becomes the close (Section 3.3; Appendix C). These choices can disable historically learned factors and favor rebalancing. The limitations acknowledge this, but the reported rankings do not establish robustness to those construction choices.

  1. Prospective execution is not uniformly causal. Appendix B states that AC observes the decision-day close but fills at that day’s open. Its overlap audit is informative, but a summarized overlap adjustment does not by itself establish the results of a complete causal rerun, including subsequent portfolio and gate changes.
Questions for authors
  1. Assessment-relevant validation: How stable are the rank reversal and 31/36 baseline-win counts across additional agent seeds and bridge realizations, and under a generator retaining joint dependence and informative auxiliary fields? Agarwal et al., “Deep Reinforcement Learning at the Edge of the Statistical Precipice” (NeurIPS 2021), not cited in the manuscript, explains why few-run rankings need uncertainty analysis. Here the relevant uncertainty should distinguish agent randomness from the fixed scenario set.

https://papers.nips.cc/paper/2021/hash/f514cec81cb148559cf475e7426eed5e-Abstract.html

  1. Clarification: Does Appendix B’s statement that rankings do not change follow from a complete next-open rerun, or from the additive same-day-overlap calculation? Please provide the corresponding causal outcome matrix, including the baseline-win counts.
Reviewer_aMN7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that evaluating LLM trading agents only by their outcome on realized history fails in three ways: 1) Rankings from historical backtests don’t carry over to alternative futures. 2) Positive returns don’t show that adaptation adds value beyond simple rules. 3) Good terminal returns can continue after the adaptive mechanism has stopped working.

To test this, the authors build FORESIGHT-9: 1) Worldlines. Nine synthetic “worldlines” (2026-07-17 to 2035-12-31) branch from a shared information boundary (2026-07-15). 2) Generator. Paths are made by log-linear interpolation between multi-asset stage anchors, with seeded Brownian-bridge noise added in between. 3) Panel. There are 15 tradable series (equities, commodities, crypto, and sovereign yields traded as yield-level exposures) plus 5 observation-only series. 4) Execution contract. All agents share the same panel, a 10-day decision cadence, long-only fully invested portfolios, a 3 bps cost, and a migration gate that only executes a proposal if expected edge exceeds 3τ.

Critical strengths
  1. Single-path, outcome-only evaluation of LLM trading agents is a real and widespread weakness. A benchmark that varies the environment and audits internal state is well motivated.

  2. Buy-and-hold, then EW-15, then inverse-vol cleanly separates passive exposure, mechanical rebalancing, and a causal risk rule from “adaptive intelligence.” Reporting that agents mostly fail to beat EW-15 is a healthy negative result, and the field needs more of these.

  3. Separating research state, declared state, and executed state, and logging admissions, evictions, and gate decisions, made the WL8 failure detectable at all. Outcome-only benchmarks would miss it.

  4. RR is explicitly labeled in-sample, there are no p-values on n=4 rankings, and “apparent excess return” is used instead of “alpha.”

Critical weaknesses
  1. The synthetic environment likely produces the headline results, which therefore may not reflect agent behavior. In detailed, each stage is a deterministic log-linear drift plus a pinned Brownian bridge. The bridge noise is independent across assets and is re-anchored at every stage. In addition, the EW-15 vs. buy-and-hold gap is implausibly large for real markets. The authors concede in the Limitations that the generator “may favor constant-mix rebalancing.” Also, yet “periodic equal weighting beats 31/36 agents” is presented as the second of three core findings about agents.

  2. Feature degeneration may confounds the rank-transfer result. In the forward panel, open = previous close, vwap = close, volume is constant, and PE/PS/PB/DYR are undefined. Several warmup factors therefore collapse. Examples are intraday_vwap_pressure (close − vwap ≡ 0), cumulative_amount_per_share, and the volume-acceleration and volume-residual factors. So the prospective test measures more than a regime shift. It also measures a change in feature semantics between training and test. This is especially concerning for FM–DS. It drives most of the negative correlation. In addition, it admits zero factors online and runs only on its 8 warmup factors, at least one of which is volume/amount-based.

  3. Contract choices weaken the claim to “preserve native mechanisms.”

  4. Some writing issue. Lack the recent LLM financial-agent benchmarks, including InvestorBench, StockBench, When Agents Trade, DeepFund. Table 17 inconsistency. dec + reb exceeds the 245 decision windows in several cells (e.g., AC-terra WL3: 294 + 134; AC-DS WL6: 42 + 244). Multiple proposals per cycle are never explained.

Questions for authors
  1. See in "Critical Weaknesses".

  2. What fraction of each configuration’s ensemble weight depends on volume, vwap, amount, or open fields that degenerate after the boundary? Does RR with the degenerated schema still rank FM–DS first?

Comments on the paper

1 comment

We thank the reviewers for their thoughtful feedback and for recognizing the value of evaluating financial agents through both their outcomes and their internal processes.

Financial markets are demanding and unstable environments. Regimes change, relationships between assets evolve, and strategies must operate through uncertainty and loss. As a group of newcomers working in quantitative finance, we began this project with a practical ambition: to develop agents capable of sustainable long-term profitability. Before pursuing higher returns, we needed to understand where our strategies might fail, how much they could lose, and whether their apparent success still reflected a functioning decision process. FORESIGHT-9 grew out of this need to understand risk.

Our focus is on diagnosing vulnerabilities rather than introducing another trading architecture. FORESIGHT-9 combines controlled alternative market scenarios, simple allocation references, and persistent records connecting research state, declared decisions, and executed holdings (§§3–4). Together, these allow us to examine what an agent earns, the risks it encounters, and whether its research continues to influence its actual trades. We regard this connection between evaluation and actionable diagnosis as the central contribution.

Two observations illustrate its practical value. In WL9, FM–terra achieves a cumulative return of +109.4% while experiencing a maximum drawdown of 31.4% (Appendix B, Tables 3 and 5). Examining the losses along the path alongside the terminal return matters when the objective is to sustain a strategy over time. In WL8, AC–DS earns +86.3% while its live factor library becomes empty, its declared ensemble continues to reference three factors, and its executed holdings converge toward equal weighting (§5.4; Appendix G). The audit traces a concrete disconnect between research and execution, including divergent paths for writing and reading ensemble artifacts. This gives developers a specific failure mechanism to investigate and repair. WL8 is valuable as a documented failure case; broader process audits are needed to characterize its prevalence.

We agree that the generator, observation schema, and learning protocol affect the interpretation of the numerical comparisons. Repeated anchoring and uncalibrated cross-asset dependence may favor rebalancing; constant volume and VWAP equal to close can disable learned signals. Comparing frozen retrospective policies with adapting prospective agents also combines environmental changes with learning-protocol differences. Accordingly, the rank correlations and baseline-win counts are descriptive findings under the current setup. The relevant validations include measuring ensemble-weight exposure to altered fields, matched-schema replay, frozen-policy prospective comparisons, scenario sensitivity, and repeated runs separating agent and generator variability. These checks serve our risk-diagnosis objective: the evaluation should distinguish strategy vulnerabilities from artifacts of its own design.

On execution timing, Appendix B reports an additive audit of same-day exposure, rather than a complete causal next-open rerun. The latter remains necessary and must propagate changed fills through subsequent holdings, gate decisions, and agent behavior. The proposal counts in Table 17 require reconciliation with the underlying decision windows and records. The description of preserved framework mechanisms should also explicitly identify the shared admission and execution constraints. We appreciate these specific concerns, together with the suggestions for broader related-work coverage.

These issues identify how the evaluation tool needs to improve. Its existing process records nevertheless make a practically important problem inspectable: an account can remain profitable while the research-to-execution mechanism has deteriorated. For researchers trying to build reliable trading agents, discovering and diagnosing such failures is part of the path toward sustainable performance. Our purpose is to make risks visible early enough to guide development, rather than to treat a favorable terminal score as the end of the evaluation.

As powerful technologies become easier to build and use, we believe financial-agent research should give long-term reliability and risk control the same attention as new capabilities. An impressive return during one favorable period is a much shorter test than maintaining profitability through years of changing conditions while controlling drawdowns and keeping the decision process functional. Our long-term goal is to develop FORESIGHT-9 into a shared, auditable foundation for evaluating financial-agent drawdown risk and sustained reliability. This work is an initial step toward that goal: understanding how strategies fail so that we can build agents better equipped to survive, adapt, and remain profitable over time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.