- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the delayed-feedback problem in formulaic alpha discovery, where the predictive quality of a factor can only be evaluated after its formula has been fully constructed, leaving intermediate construction decisions without direct quality feedback.
To address this problem, the authors propose AlphaDreamer, a model-based reinforcement learning framework centered on Model-Predictive Prefix Planning (MPPP). The method learns prefix representations, successor-state predictions, and completion-quality estimates from evaluated formulas. MPPP uses these predictions, together with grammar-constrained search and a separate reference model, to guide formula construction through stepwise planning and replanning.
During training, the framework combines scratch rollouts generated by a shared PPO policy with planner-guided rollouts, where MPPP selects a limited number of construction actions before the same policy completes the formula. Evaluated formulas provide feedback for training the predictive models and improving the policy.
Experiments on CSI300, CSI500, and S&P 500 show improvements over the evaluated baselines in predictive performance and portfolio metrics. Additional experiments demonstrate improvements in valid-formula discovery efficiency, examine different planning horizons, and show that the full MPPP outperforms reference-only scoring.
Critical strengths
-
A well-motivated and insightful approach to delayed feedback in alpha discovery. The paper addresses an important challenge in formulaic alpha discovery: the lack of direct quality feedback during intermediate formula construction. AlphaDreamer provides a coherent approach by using learned predictions of prefix consequences and completion quality to guide construction decisions. The integration of predictive lookahead, direct reference evaluation, and replanning is technically interesting and offers a meaningful approach to improving formula search.
-
Meaningful experimental evidence supporting the effectiveness of MPPP. The paper provides a reasonably comprehensive evaluation across three equity markets, along with targeted experiments on planning depth and search efficiency. In particular, Table 2 shows improvements in both predictive quality and the number of unique valid formulas discovered per evaluator call. Figure 4 further demonstrates that the full MPPP outperforms reference-only scoring across six paired seeds. These results provide meaningful evidence that the proposed planning framework improves alpha discovery.
Critical weaknesses
-
The contribution of MPPP and its division of labor with the shared policy could be better understood. Although the experiments demonstrate the effectiveness of the overall framework, the specific pathway through which MPPP contributes to the final results remains unclear. Since training includes both scratch and planner-guided rollouts, it would be informative to distinguish whether MPPP primarily improves the shared policy through training, directly discovers higher-quality formulas, or contributes through both mechanisms. In addition, the choice to limit MPPP to at most eight executed actions is not sufficiently justified, and the paper does not investigate how the performance changes when MPPP controls more of the formula construction process.
-
Some implementation details are insufficient for independent reproduction. While the overall framework and training objectives are described, several neural network components lack detailed architectural specifications. In particular, the implementations of the forward and backward models, quality prediction heads, reference model, prefix ranker, and shared actor-critic would benefit from further clarification. These details are important for faithfully reproducing the proposed method and understanding how the components interact.
Questions for authors
- Planning budget and policy completion.
The proposed framework limits MPPP to at most (K_{\mathrm{plan}}=8) executed actions before handing the remaining formula construction to the shared policy. What motivates this choice? Have the authors examined how performance changes with different values of (K_{\mathrm{plan}}), particularly when MPPP is allowed to complete the entire formula without policy completion?
- Treatment of duplicate formulas in search-efficiency evaluation.
Table 7 shows that MPPP generates a noticeable proportion of duplicate formulas, while only first occurrences invoke the evaluator. Could the authors provide a comparison that also accounts for duplicate completions as evaluation attempts, using a consistent protocol for MPPP and Policy-only?
- Implementation details of the model components.
While the paper describes the relational graph encoder and training objectives, some architectural details remain unclear. Could the authors provide more specific implementation details for the forward and backward models, quality prediction heads, reference model, prefix ranker, and shared actor–critic policy?
- Direct and indirect contributions of MPPP to the final factor pool.
Table 2 provides evidence that the overall MPPP-guided framework outperforms Policy-only generation. However, since AlphaDreamer uses both scratch and planner-guided rollouts during training, it remains unclear whether MPPP primarily contributes by directly discovering higher-quality formulas or by improving the shared PPO policy through planner-guided training. Could the authors clarify or experimentally distinguish the direct contribution of MPPP-guided search from its indirect contribution through improved policy training?
Thanks for the additional results. My main concern was whether the improvement actually came from the learned transition model rather than the extra scoring components. The exact-successor comparison helps address this, and the other ablations are useful as well.
Overalll, I will increase my rating