- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces an agentic framework DASH (Diagnostic Agentic Search with Hypotheses) for the financial forecasting task (alpha mining and optimizing predictive models on such alphas). Specifically, the agent not only proposes updates to the current candidates, but also formulates hypotheses for the expected effects of such changes. External skills drive the evaluation and verification of these hypotheses and only move the ones where the actual evaluation outcome agrees with the expected outcome. Mechanisms to control and balance the diversity and quality of proposed alphas/models are also employed. The proposed framework finds more robust alphas and models, achieving better performance comparing to other financial forecasting agentic framework, for both predictive power (IC, Rank IC, ICIR) and portfolio performance (IR, annual returns, MDD).
Critical strengths
Overfitting is a tricky problem to solve in the field of stock trend forecasting. Even with a validation set available, former frameworks often overfit on the validation set indirectly. If the agent just attempts random modifications and select variants that improved the performance on the validation set, the resulting model will be prone to the aforementioned overfitting problem. With the proposed method, the agent is expected to predict the outcome correctly before applying a modification to the prediction model, which discourages the agent to do random explorations, and consequently enhancing the overall robustness of the resulting model.
Critical weaknesses
The major concern about agentic frameworks for alpha mining is the potential risk of data leak. Since financial market data is of low signal-to-noise ratio, any introduction of future data might significantly influence the results. Since the knowledge cutoff of the LLMs used in this work are in 2024, certain patterns of future market (e.g. the prevalent regime) regarding the testing window might be leaked to the LLMs.
Questions for authors
- Concerns about potential data leak. Please refer to the "Weaknesses" section.
- Minor: The paper does not specify the actual form of the alphas and the model architectures it finds. Case study on some concrete examples would be nice to have.
- Minor: The experiment setup is a bit weird. Since top-50-drop-5 is used, the expected holding period of each ticker is 10 days, yet the prediction target is 5-day returns.
7 comments
Table R2. PIT factor–model search on S&P 500 (2025 OOD evaluation).
| Budget | Method | IC | ICIR | Rank ICIR | NAV |
|---|---|---|---|---|---|
| 100K | AutoResearch | ||||
| 100K | R&D-Agent | ||||
| 100K | AIRA | ||||
| 100K | ML-Master 2.0 | ||||
| 100K | DASH | ||||
| 400K | AutoResearch | ||||
| 400K | R&D-Agent | ||||
| 400K | AIRA | ||||
| 400K | ML-Master 2.0 | ||||
| 400K | DASH |
Table R3. Factor-only search on NASDAQ-100 (2025 OOD evaluation, 400K-token budget).
| Aggregation | Method | IC | Rank IC | ICIR | Rank ICIR |
|---|---|---|---|---|---|
| Best factor | AlphaAgent | ||||
| Best factor | R&D-Agent | ||||
| Best factor | FactorMiner | ||||
| Best factor | DASH | ||||
| Top-5 equal weight | R&D-Agent | ||||
| Top-5 equal weight | AlphaAgent | ||||
| Top-5 equal weight | FactorMiner | ||||
| Top-5 equal weight | DASH | ||||
| Top-20 equal weight | R&D-Agent | ||||
| Top-20 equal weight | AlphaAgent | ||||
| Top-20 equal weight | FactorMiner | ||||
| Top-20 equal weight | DASH | ||||
| Top-40 equal weight | R&D-Agent | ||||
| Top-40 equal weight | AlphaAgent | ||||
| Top-40 equal weight | FactorMiner | ||||
| Top-40 equal weight | DASH |
Table R4. Factor-only search on S&P 500 (2025 OOD evaluation, 400K-token budget).
| Aggregation | Method | IC | Rank IC | ICIR | Rank ICIR |
|---|---|---|---|---|---|
| Best factor | AlphaAgent | ||||
| Best factor | FactorMiner | ||||
| Best factor | R&D-Agent | ||||
| Best factor | DASH | ||||
| Top-5 equal weight | FactorMiner | ||||
| Top-5 equal weight | AlphaAgent | ||||
| Top-5 equal weight | R&D-Agent | ||||
| Top-5 equal weight | DASH | ||||
| Top-20 equal weight | AlphaAgent | ||||
| Top-20 equal weight | FactorMiner | ||||
| Top-20 equal weight | R&D-Agent | ||||
| Top-20 equal weight | DASH | ||||
| Top-40 equal weight | AlphaAgent | ||||
| Top-40 equal weight | FactorMiner | ||||
| Top-40 equal weight | R&D-Agent | ||||
| Top-40 equal weight | DASH |
Response to Q2:
We thank the reviewer for this suggestion. We will include additional factor and model case studies, along with their implementations, in the revised manuscript. To illustrate DASH's concrete search outputs, we provide a factor–model joint-search case study on NASDAQ-100. Table R5 presents ten generated factors, including their names, simplified code, and interpretations. Table R6 details the proposed model specification including model architecture and training configuration.
Table R5. Representative factors discovered by DASH.
| Factor Name | Simplified Code | Interpretation |
|---|---|---|
market_rel_close_20d_cs | -cs_z(hist_pct(close / market_close, 20)) | Market-relative price reversal based on a stock's historical relative-price percentile. |
momentum_5d_cs_rank | -cs_rank(ret(5)) | Cross-sectional short-term reversal based on 5-day returns. |
cross_sectional_demeaned_return_5d | -(logret(5) - cs_mean(logret(5))) | Return reversal relative to the cross-sectional average. |
momentum_20d_low_volatility_filter | -roll_sum(ret(1), 20).mask(vol20 > cs_q(vol20, 0.75)) | Volatility-filtered reversal, excluding relatively high-volatility stocks. |
cs_neg_3d_return_sum_zscore | -cs_z(roll_sum(ret(1) - cs_mean(ret(1)), 3)) | Standardized 3-day reversal relative to the market-wide cross-sectional mean. |
ADV_10_CS_RANK | -cs_rank(roll_mean(close * volume, 10)) | Cross-sectional trading activity and liquidity-related signal. |
neg_5d_return_zscore | -cs_z(logret(5)) | Cross-sectionally standardized 5-day log-return reversal. |
regime_vol20_quantile | cs_rank(roll_std(logret(1), 20)) | Relative volatility regime based on historical return dispersion. |
regime_bull | (logret(60) > cs_median(logret(60))).float() | Indicator of above-median 60-day stock performance. |
regime_bear | (logret(60) < cs_median(logret(60))).float() | Indicator of below-median 60-day stock performance. |
Notes: ret(h) and logret(h) denote per-stock simple and log returns over h observations. roll_* denotes per-stock rolling statistics, cs_* denotes cross-sectional operations, and hist_pct computes the historical percentile.
Response to Q1:
We thank the reviewer for highlighting this important concern. We agree that pretrained LLMs may contain information about future market conditions, potentially causing information leakage in financial evaluations, especially given the low signal-to-noise ratio of financial data.
Existing mitigation strategies. Our original framework incorporates two measures to reduce the potential impact of this concern. First, we do not disclose the specific calendar dates of the OOD evaluation period to the LLM agents. Second, we use market-beta-adjusted return targets to reduce the potential predictive advantage from broad market trends and regimes that may be encoded in LLM pretraining data, thereby placing greater emphasis on stock-specific signals. Nevertheless, we acknowledge that these measures cannot fully eliminate information leakage from pretrained model weights. This is a common challenge in evaluating agentic research frameworks using pretrained frontier LLMs. We will explicitly discuss this limitation in the revised manuscript.
Additional OOD evaluation starting in 2025. To further investigate this concern, we adjust the original OOD evaluation period to start on January 6, 2025, a period entirely after the reported 2024 knowledge cutoffs of the evaluated LLMs. We report the results for PIT factor–model search in Tables R1–R2 and factor-only search in Tables R3–R4. For PIT factor–model search, DASH achieves the highest mean IC and ICIR across both NASDAQ-100 and S&P 500 under both the 100K- and 400K-token budgets. It also achieves the highest NAV across all four settings. Although DASH does not lead in Rank ICIR on S&P 500, its overall predictive and portfolio performance remains competitive. For factor-only search, DASH achieves the highest mean IC, Rank IC, ICIR, and Rank ICIR for both the Top-20 and Top-40 equal-weight ensembles on NASDAQ-100 and S&P 500 under the 400K-token budget. Results for individual factors and Top-5 ensembles are more mixed, but the larger ensembles exhibit consistent advantages. Additionally, the cryptocurrency experiments reported in Appendix G.3 of the original manuscript already use an OOD evaluation period starting in 2025, providing complementary evidence of DASH's performance on a different asset class.
Table R1. PIT factor–model search on NASDAQ-100 (2025 OOD evaluation).