acceptodds
Under review as a conference paper at ICLR 2027

AlphaDreamer: Model-Predictive Prefix Planning for Formulaic Alpha Discovery

Abstract

Discovering predictive alpha factors is fundamental to quantitative trading. However, evaluating a formula’s predictive quality requires completing its construction, leaving intermediate decisions without direct performance feedback. We introduce AlphaDreamer, a model-based reinforcement learning framework that learns predictive models from evaluated formulas to guide construction through prefix-level planning. Its Model-Predictive Prefix Planner (MPPP) couples exact grammar transitions with learned predictions of prefix representations and completion quality. A learned reference model directly scores candidate prefixes to complement predictive lookahead in action selection. During prefix planning, the planner executes only the first action of each selected plan before replanning from the updated prefix. A shared policy learns both to generate formulas from scratch and to complete planner-provided prefixes. Empirical results show that AlphaDreamer outperforms the evaluated baselines in predictive accuracy and portfolio performance. Prefix-level planning improves both discovery quality and evaluation efficiency, while the full planner consistently outperforms direct prefix scoring alone.

Primary area:
reinforcement learning
Keywords:
formulaic alpha discovery, model-based reinforcement learning, model-predictive planning, symbolic formula generation, quantitative finance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_thh9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the delayed-feedback problem in formulaic alpha discovery, where the predictive quality of a factor can only be evaluated after its formula has been fully constructed, leaving intermediate construction decisions without direct quality feedback.

To address this problem, the authors propose AlphaDreamer, a model-based reinforcement learning framework centered on Model-Predictive Prefix Planning (MPPP). The method learns prefix representations, successor-state predictions, and completion-quality estimates from evaluated formulas. MPPP uses these predictions, together with grammar-constrained search and a separate reference model, to guide formula construction through stepwise planning and replanning.

During training, the framework combines scratch rollouts generated by a shared PPO policy with planner-guided rollouts, where MPPP selects a limited number of construction actions before the same policy completes the formula. Evaluated formulas provide feedback for training the predictive models and improving the policy.

Experiments on CSI300, CSI500, and S&P 500 show improvements over the evaluated baselines in predictive performance and portfolio metrics. Additional experiments demonstrate improvements in valid-formula discovery efficiency, examine different planning horizons, and show that the full MPPP outperforms reference-only scoring.

Critical strengths
  1. A well-motivated and insightful approach to delayed feedback in alpha discovery. The paper addresses an important challenge in formulaic alpha discovery: the lack of direct quality feedback during intermediate formula construction. AlphaDreamer provides a coherent approach by using learned predictions of prefix consequences and completion quality to guide construction decisions. The integration of predictive lookahead, direct reference evaluation, and replanning is technically interesting and offers a meaningful approach to improving formula search.

  2. Meaningful experimental evidence supporting the effectiveness of MPPP. The paper provides a reasonably comprehensive evaluation across three equity markets, along with targeted experiments on planning depth and search efficiency. In particular, Table 2 shows improvements in both predictive quality and the number of unique valid formulas discovered per evaluator call. Figure 4 further demonstrates that the full MPPP outperforms reference-only scoring across six paired seeds. These results provide meaningful evidence that the proposed planning framework improves alpha discovery.

Critical weaknesses
  1. The contribution of MPPP and its division of labor with the shared policy could be better understood. Although the experiments demonstrate the effectiveness of the overall framework, the specific pathway through which MPPP contributes to the final results remains unclear. Since training includes both scratch and planner-guided rollouts, it would be informative to distinguish whether MPPP primarily improves the shared policy through training, directly discovers higher-quality formulas, or contributes through both mechanisms. In addition, the choice to limit MPPP to at most eight executed actions is not sufficiently justified, and the paper does not investigate how the performance changes when MPPP controls more of the formula construction process.

  2. Some implementation details are insufficient for independent reproduction. While the overall framework and training objectives are described, several neural network components lack detailed architectural specifications. In particular, the implementations of the forward and backward models, quality prediction heads, reference model, prefix ranker, and shared actor-critic would benefit from further clarification. These details are important for faithfully reproducing the proposed method and understanding how the components interact.

Questions for authors
  1. Planning budget and policy completion.

The proposed framework limits MPPP to at most (K_{\mathrm{plan}}=8) executed actions before handing the remaining formula construction to the shared policy. What motivates this choice? Have the authors examined how performance changes with different values of (K_{\mathrm{plan}}), particularly when MPPP is allowed to complete the entire formula without policy completion?

  1. Treatment of duplicate formulas in search-efficiency evaluation.

Table 7 shows that MPPP generates a noticeable proportion of duplicate formulas, while only first occurrences invoke the evaluator. Could the authors provide a comparison that also accounts for duplicate completions as evaluation attempts, using a consistent protocol for MPPP and Policy-only?

  1. Implementation details of the model components.

While the paper describes the relational graph encoder and training objectives, some architectural details remain unclear. Could the authors provide more specific implementation details for the forward and backward models, quality prediction heads, reference model, prefix ranker, and shared actor–critic policy?

  1. Direct and indirect contributions of MPPP to the final factor pool.

Table 2 provides evidence that the overall MPPP-guided framework outperforms Policy-only generation. However, since AlphaDreamer uses both scratch and planner-guided rollouts during training, it remains unclear whether MPPP primarily contributes by directly discovering higher-quality formulas or by improving the shared PPO policy through planner-guided training. Could the authors clarify or experimentally distinguish the direct contribution of MPPP-guided search from its indirect contribution through improved policy training?

Reviewer_vNsFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents AlphaDreamer, a model-predictive planning framework for formulaic alpha discovery in quantitative finance. The method learns latent representations of partially constructed alpha expressions and uses learned transition, quality prediction, and value models to evaluate potential formula extensions. A disagreement-based penalty between model predictions and a reference estimator is introduced to improve the reliability of planning, while a PPO-based policy guides formula generation.

The main contribution is the integration of learned latent dynamics and model-predictive planning into the sequential construction of formulaic alpha factors, aiming to improve search efficiency and factor quality. The authors evaluate AlphaDreamer on CSI300, CSI500, and S&P 500 datasets against several traditional and learning-based baselines. Experimental results suggest improvements in predictive performance and the efficiency of discovering valid alpha expressions, with ablation studies examining the effects of planning horizons and selected model components.

Critical strengths
  1. Promising empirical performance across multiple financial datasets. AlphaDreamer demonstrates competitive or improved predictive performance on CSI300, CSI500, and S&P 500 compared with several established baselines. The evaluation across different markets provides useful evidence that the proposed framework can be effective beyond a single dataset. This is the strongest argument in favor of acceptance, particularly given the practical challenges of formulaic alpha discovery.

  2. Improved search efficiency with a practically motivated framework. The proposed approach incorporates predictive scoring into formula generation to prioritize promising candidates. The experiments show that AlphaDreamer discovers more unique valid formulas while requiring fewer evaluator calls than the policy-only baseline. This is a meaningful practical advantage because evaluating candidate alpha expressions can be computationally expensive, although the overall efficiency gains still need to be established under matched end-to-end computational budgets.

Critical weaknesses
  1. Limited methodological novelty. The proposed AlphaDreamer framework primarily integrates existing techniques, including relational graph representations, learned latent dynamics, model-predictive planning, and PPO-based policy optimization. While the combination is practically motivated, the paper does not introduce a fundamentally new architecture, representation learning principle, or optimization algorithm. The methodological contributions beyond existing model-based reinforcement learning and search-based alpha discovery remain insufficiently distinguished.

  2. Insufficient theoretical justification. The paper provides limited theoretical analysis of the proposed planning framework. In particular, it lacks an analysis of prediction error propagation, the relationship between planning horizon and decision quality, or performance guarantees for the disagreement-based scoring mechanism. Consequently, the effectiveness of the proposed approach relies almost entirely on empirical observations.

  3. Unclear benefits of model-predictive planning. The planning-horizon ablation reveals that one-step planning (H=1H=1) achieves the best performance, while increasing the horizon generally degrades prediction quality and substantially increases computational cost. This raises concerns about whether the learned latent dynamics model provides meaningful advantages over simpler one-step value estimation or direct prefix scoring.

  4. Insufficient experimental controls and computational fairness. Although the paper reports improvements under matched evaluator-call budgets, it does not provide a comprehensive comparison of total computational resources, including GPU-hours, training time, model inference overhead, and search costs. Furthermore, the main experiments omit closely related search-based approaches such as Alpha2 and AlphaCFG. These limitations make it difficult to determine whether the improvements arise from the proposed methodology or additional computational resources.

  5. Incomplete validation of individual components and experimental robustness. The paper lacks direct evaluations of latent transition prediction accuracy, prefix-quality estimation, and the effectiveness of the disagreement penalty. More comprehensive component-wise ablations are needed to establish the necessity of each module. Additionally, the main results are averaged over only three random seeds, and the use of absolute-valued correlation metrics may obscure prediction-direction errors. These issues weaken the reliability and interpretability of the empirical conclusions.

Questions for authors
  1. The proposed framework combines several established techniques, including model-based planning, graph representations, and reinforcement learning. Could the authors clarify what the key methodological contribution is beyond their integration, particularly in comparison with existing search-based approaches such as Alpha2 and AlphaCFG?

  2. An interesting observation is that H=1H=1 consistently performs best, while deeper planning often leads to worse results. Could the authors explain why this happens? It would also be helpful to evaluate the accuracy of the learned transition and quality prediction models, and clarify whether the additional complexity of latent dynamics and multi-step planning is necessary compared to simpler one-step value estimation.

  3. Could the authors provide more evidence for the robustness and computational efficiency of the proposed method? In particular, comparisons under matched total computational budgets (e.g., GPU-hours or wall-clock time), rather than evaluator calls alone, would be valuable. Additional random seeds, statistical significance tests, and results using signed IC/RIC would also help strengthen the empirical conclusions.

Reviewer_YBNNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

AlphaDreamer models formula construction as sequential decision-making with exact grammar transitions. Learned prefix representations, transition models, and completion-quality predictors guide search. The planner combines predicted continuation quality with a separately parameterized direct prefix evaluator, while a shared policy generates formulas and completes planned prefixes. The approach is plausible and empirically promising. The central unresolved question is whether its gains specifically establish the value of model-predictive planning rather than stronger prefix scoring and additional learned components.

Critical strengths
  1. Separating exact syntactic transitions from learned quality prediction is a sensible design.

  2. The main evaluation covers three markets using chronological annual retraining and validation.

  3. The paper examines discovery quality at fixed evaluator-call budgets and includes a paired comparison against direct reference scoring.

  4. Horizon experiments transparently show that deeper planning adds cost without consistently improving performance.

  5. The appendix supplies additional training objectives, planning details, and experimental specifications, which strengthen reproducibility.

Critical weaknesses
  1. The strongest results do not establish a benefit from multi-step planning. The main benchmark uses H=1, and longer horizons do not consistently improve quality (Table 3). This is an informative result, but it narrows the demonstrated contribution toward repeated one-step predictive selection rather than deeper model-based search.

  2. Planning gains are not fully separated from additional scoring capacity. The method uses separately parameterized predictive and reference models and combines them through a discrepancy penalty. The full-versus-reference comparison is useful, but does not isolate latent-transition learning from the benefit of combining multiple estimators. Since symbolic successors can be constructed exactly, an exact-successor evaluator and a matched ensemble of direct evaluators would be informative controls.

  3. The closest search-based comparisons are not represented in the main benchmark. Alpha^2, AlphaCFG, and RiskMiner are discussed as related approaches, but the main comparison focuses on other discovery methods and predictive models. Comparisons with suitable established search methods, or an explanation of why they are not directly comparable, would strengthen the planning contribution. Missing comparisons with contemporaneous or exclusively preprint work should not by themselves determine rejection.

  4. The robustness of the planning advantage requires more direct evidence. In Table 2, policy-only IC is 8.69±0.608.69\pm0.60, versus $9.53\pm2.17$ for one-step planning. These summaries alone do not establish a stable paired improvement. The six-seed full-versus-reference result on p. 9 addresses a related but different comparison. Per-seed, per-year, and cross-market planning ablations would clarify consistency.

Questions for authors
  1. Does the method outperform exact-successor value estimation and two direct evaluators under matched capacity and computation?

  2. Which component accounts for the gain: latent-transition prediction, discrepancy regularization, prefix initialization, or their combination?

  3. Is the policy-only versus planning improvement consistent across paired seeds, individual years, and all three markets?

  4. Can the authors provide a direct comparison with an appropriate established search-based alpha-discovery method?

  5. How does end-to-end discovery time compare when model training and planning overhead are included, beyond evaluator calls and per-decision planning time?

1 comment

Thanks for the additional results. My main concern was whether the improvement actually came from the learned transition model rather than the extra scoring components. The exact-successor comparison helps address this, and the other ablations are useful as well.

Overalll, I will increase my rating

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.