acceptodds
Under review as a conference paper at ICLR 2027

DASH: Diagnostic-Guided Quality-Diversity Search for Financial ML Agents

Abstract

Automated LLM research agents iteratively propose, evaluate, and refine machine-learning configurations using empirical feedback. In financial forecasting, low signal-to-noise ratios and nonstationarity make apparent improvements uncertain and potentially transient, allowing search to reinforce spurious or regime-specific gains. We introduce DASH (Diagnostic Agentic Search with Hypotheses), a diagnostic-guided search framework that evaluates configurations across temporal regimes to construct condition-specific, multi-axis behavioral profiles. Before refining a candidate, the agent formulates hypotheses with pre-specified expected outcomes and excludes those refuted by computed evidence. A quality-diversity archive organizes evaluated states by behavioral descriptors while separately accounting for predictive strength and redundancy when retaining and selecting alternatives. Empirically, DASH achieves strong held-out OOD predictive and portfolio performance under matched search budgets. In joint factor–model search mode, it improves IC over the strongest evaluated agentic baseline by 8.7% on NASDAQ-100 and 38.2% on S&P 500, while achieving the highest portfolio IR on both markets. Factor-only analyses show a high concentration of strong OOD factors with low predictive redundancy in the discovered factor pools. In a separate NASDAQ-100 ablation, replacing externally computed evidence with LLM-only judgments consistently degrades predictive performance across two backbones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LM research agents, diagnostic-guided refinement, quality-diversity search, automated machine learning, financial forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_sfa4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an agentic framework DASH (Diagnostic Agentic Search with Hypotheses) for the financial forecasting task (alpha mining and optimizing predictive models on such alphas). Specifically, the agent not only proposes updates to the current candidates, but also formulates hypotheses for the expected effects of such changes. External skills drive the evaluation and verification of these hypotheses and only move the ones where the actual evaluation outcome agrees with the expected outcome. Mechanisms to control and balance the diversity and quality of proposed alphas/models are also employed. The proposed framework finds more robust alphas and models, achieving better performance comparing to other financial forecasting agentic framework, for both predictive power (IC, Rank IC, ICIR) and portfolio performance (IR, annual returns, MDD).

Critical strengths

Overfitting is a tricky problem to solve in the field of stock trend forecasting. Even with a validation set available, former frameworks often overfit on the validation set indirectly. If the agent just attempts random modifications and select variants that improved the performance on the validation set, the resulting model will be prone to the aforementioned overfitting problem. With the proposed method, the agent is expected to predict the outcome correctly before applying a modification to the prediction model, which discourages the agent to do random explorations, and consequently enhancing the overall robustness of the resulting model.

Critical weaknesses

The major concern about agentic frameworks for alpha mining is the potential risk of data leak. Since financial market data is of low signal-to-noise ratio, any introduction of future data might significantly influence the results. Since the knowledge cutoff of the LLMs used in this work are in 2024, certain patterns of future market (e.g. the prevalent regime) regarding the testing window might be leaked to the LLMs.

Questions for authors
  • Concerns about potential data leak. Please refer to the "Weaknesses" section.
  • Minor: The paper does not specify the actual form of the alphas and the model architectures it finds. Case study on some concrete examples would be nice to have.
  • Minor: The experiment setup is a bit weird. Since top-50-drop-5 is used, the expected holding period of each ticker is 10 days, yet the prediction target is 5-day returns.
7 comments

Response to Q1:

We thank the reviewer for highlighting this important concern. We agree that pretrained LLMs may contain information about future market conditions, potentially causing information leakage in financial evaluations, especially given the low signal-to-noise ratio of financial data.

Existing mitigation strategies. Our original framework incorporates two measures to reduce the potential impact of this concern. First, we do not disclose the specific calendar dates of the OOD evaluation period to the LLM agents. Second, we use market-beta-adjusted return targets to reduce the potential predictive advantage from broad market trends and regimes that may be encoded in LLM pretraining data, thereby placing greater emphasis on stock-specific signals. Nevertheless, we acknowledge that these measures cannot fully eliminate information leakage from pretrained model weights. This is a common challenge in evaluating agentic research frameworks using pretrained frontier LLMs. We will explicitly discuss this limitation in the revised manuscript.

Additional OOD evaluation starting in 2025. To further investigate this concern, we adjust the original OOD evaluation period to start on January 6, 2025, a period entirely after the reported 2024 knowledge cutoffs of the evaluated LLMs. We report the results for PIT factor–model search in Tables R1–R2 and factor-only search in Tables R3–R4. For PIT factor–model search, DASH achieves the highest mean IC and ICIR across both NASDAQ-100 and S&P 500 under both the 100K- and 400K-token budgets. It also achieves the highest NAV across all four settings. Although DASH does not lead in Rank ICIR on S&P 500, its overall predictive and portfolio performance remains competitive. For factor-only search, DASH achieves the highest mean IC, Rank IC, ICIR, and Rank ICIR for both the Top-20 and Top-40 equal-weight ensembles on NASDAQ-100 and S&P 500 under the 400K-token budget. Results for individual factors and Top-5 ensembles are more mixed, but the larger ensembles exhibit consistent advantages. Additionally, the cryptocurrency experiments reported in Appendix G.3 of the original manuscript already use an OOD evaluation period starting in 2025, providing complementary evidence of DASH's performance on a different asset class.

Table R1. PIT factor–model search on NASDAQ-100 (2025 OOD evaluation).

BudgetMethodICICIRRank ICIRNAV
100KAutoResearch0.0167±0.01340.0167 \pm 0.01340.1006±0.07870.1006 \pm 0.07870.1058±0.09070.1058 \pm 0.09071.0985±0.11061.0985 \pm 0.1106
100KR&D-Agent0.0061±0.01420.0061 \pm 0.01420.0514±0.11560.0514 \pm 0.11560.0427±0.10380.0427 \pm 0.10381.0936±0.06601.0936 \pm 0.0660
100KAIRA0.0288±0.0128‾\underline{0.0288 \pm 0.0128}0.1923±0.0985‾\underline{0.1923 \pm 0.0985}0.1778±0.0556‾\underline{0.1778 \pm 0.0556}1.1634±0.1121‾\underline{1.1634 \pm 0.1121}
100KML-Master 2.00.0252±0.01530.0252 \pm 0.01530.1534±0.08300.1534 \pm 0.08300.1515±0.09190.1515 \pm 0.09191.0756±0.01601.0756 \pm 0.0160
100KDASH0.0404±0.0081\boldsymbol{0.0404 \pm 0.0081}0.2939±0.0494\boldsymbol{0.2939 \pm 0.0494}0.1882±0.0436\boldsymbol{0.1882 \pm 0.0436}1.2239±0.0351\boldsymbol{1.2239 \pm 0.0351}
400KAutoResearch0.0151±0.01710.0151 \pm 0.01710.0881±0.10200.0881 \pm 0.10200.0729±0.08990.0729 \pm 0.08991.0734±0.05901.0734 \pm 0.0590
400KR&D-Agent0.0107±0.00920.0107 \pm 0.00920.0785±0.07220.0785 \pm 0.07220.0851±0.02140.0851 \pm 0.02141.1450±0.0734‾\underline{1.1450 \pm 0.0734}
400KAIRA0.0298±0.0154‾\underline{0.0298 \pm 0.0154}0.1913±0.1056‾\underline{0.1913 \pm 0.1056}0.1743±0.0535‾\underline{0.1743 \pm 0.0535}1.1433±0.11751.1433 \pm 0.1175
400KML-Master 2.00.0231±0.01920.0231 \pm 0.01920.1433±0.10930.1433 \pm 0.10930.1426±0.11860.1426 \pm 0.11861.0973±0.02561.0973 \pm 0.0256
400KDASH0.0434±0.0057\boldsymbol{0.0434 \pm 0.0057}0.3021±0.0381\boldsymbol{0.3021 \pm 0.0381}0.2276±0.0367\boldsymbol{0.2276 \pm 0.0367}1.1624±0.0419\boldsymbol{1.1624 \pm 0.0419}

Table R2. PIT factor–model search on S&P 500 (2025 OOD evaluation).

BudgetMethodICICIRRank ICIRNAV
100KAutoResearch0.0108±0.0033‾\underline{0.0108 \pm 0.0033}0.1066±0.0266‾\underline{0.1066 \pm 0.0266}0.0659±0.0305\boldsymbol{0.0659 \pm 0.0305}1.0656±0.0319‾\underline{1.0656 \pm 0.0319}
100KR&D-Agent0.0009±0.01370.0009 \pm 0.01370.0034±0.15030.0034 \pm 0.1503−0.0061±0.1534-0.0061 \pm 0.15341.0378±0.05211.0378 \pm 0.0521
100KAIRA0.0080±0.00310.0080 \pm 0.00310.0815±0.02970.0815 \pm 0.02970.0120±0.04220.0120 \pm 0.04221.0322±0.02571.0322 \pm 0.0257
100KML-Master 2.00.0080±0.00550.0080 \pm 0.00550.0739±0.05480.0739 \pm 0.05480.0394±0.04460.0394 \pm 0.04461.0647±0.07311.0647 \pm 0.0731
100KDASH0.0140±0.0011\boldsymbol{0.0140 \pm 0.0011}0.1477±0.0168\boldsymbol{0.1477 \pm 0.0168}0.0450±0.0150‾\underline{0.0450 \pm 0.0150}1.1142±0.0179\boldsymbol{1.1142 \pm 0.0179}
400KAutoResearch0.0092±0.0046‾\underline{0.0092 \pm 0.0046}0.0926±0.0508‾\underline{0.0926 \pm 0.0508}0.0540±0.0095\boldsymbol{0.0540 \pm 0.0095}1.0926±0.0525‾\underline{1.0926 \pm 0.0525}
400KR&D-Agent0.0015±0.01440.0015 \pm 0.01440.0066±0.15530.0066 \pm 0.1553−0.0430±0.1205-0.0430 \pm 0.12051.0671±0.06901.0671 \pm 0.0690
400KAIRA0.0088±0.00310.0088 \pm 0.00310.0874±0.02900.0874 \pm 0.02900.0271±0.02530.0271 \pm 0.02531.0390±0.02191.0390 \pm 0.0219
400KML-Master 2.00.0080±0.00550.0080 \pm 0.00550.0739±0.05480.0739 \pm 0.05480.0394±0.04460.0394 \pm 0.04461.0647±0.07311.0647 \pm 0.0731
400KDASH0.0126±0.0060\boldsymbol{0.0126 \pm 0.0060}0.1363±0.0541\boldsymbol{0.1363 \pm 0.0541}0.0456±0.0295‾\underline{0.0456 \pm 0.0295}1.1461±0.0245\boldsymbol{1.1461 \pm 0.0245}

Table R3. Factor-only search on NASDAQ-100 (2025 OOD evaluation, 400K-token budget).

AggregationMethodICRank ICICIRRank ICIR
Best factorAlphaAgent0.0230±0.0224\boldsymbol{0.0230 \pm 0.0224}0.0262±0.0310‾\underline{0.0262 \pm 0.0310}0.1837±0.1764\boldsymbol{0.1837 \pm 0.1764}0.1355±0.1608‾\underline{0.1355 \pm 0.1608}
Best factorR&D-Agent0.0168±0.01060.0168 \pm 0.01060.0039±0.00200.0039 \pm 0.00200.1126±0.07230.1126 \pm 0.07230.0268±0.01500.0268 \pm 0.0150
Best factorFactorMiner0.0003±0.01760.0003 \pm 0.0176−0.0007±0.0183-0.0007 \pm 0.0183−0.0028±0.1161-0.0028 \pm 0.1161−0.0049±0.1198-0.0049 \pm 0.1198
Best factorDASH0.0214±0.0343‾\underline{0.0214 \pm 0.0343}0.0320±0.0425\boldsymbol{0.0320 \pm 0.0425}0.1243±0.1960‾\underline{0.1243 \pm 0.1960}0.1495±0.2199\boldsymbol{0.1495 \pm 0.2199}
Top-5 equal weightR&D-Agent0.0296±0.0177‾\underline{0.0296 \pm 0.0177}0.0237±0.02440.0237 \pm 0.02440.1856±0.1041\boldsymbol{0.1856 \pm 0.1041}0.1377±0.1284‾\underline{0.1377 \pm 0.1284}
Top-5 equal weightAlphaAgent0.0199±0.01730.0199 \pm 0.01730.0253±0.0316‾\underline{0.0253 \pm 0.0316}0.1277±0.10970.1277 \pm 0.10970.1334±0.19440.1334 \pm 0.1944
Top-5 equal weightFactorMiner−0.0092±0.0061-0.0092 \pm 0.0061−0.0120±0.0092-0.0120 \pm 0.0092−0.0556±0.0434-0.0556 \pm 0.0434−0.0696±0.0592-0.0696 \pm 0.0592
Top-5 equal weightDASH0.0317±0.0417\boldsymbol{0.0317 \pm 0.0417}0.0351±0.0447\boldsymbol{0.0351 \pm 0.0447}0.1624±0.2181‾\underline{0.1624 \pm 0.2181}0.1623±0.2206\boldsymbol{0.1623 \pm 0.2206}
Top-20 equal weightR&D-Agent0.0262±0.0229‾\underline{0.0262 \pm 0.0229}0.0212±0.01480.0212 \pm 0.01480.1904±0.1658‾\underline{0.1904 \pm 0.1658}0.1473±0.0981‾\underline{0.1473 \pm 0.0981}
Top-20 equal weightAlphaAgent0.0199±0.01730.0199 \pm 0.01730.0253±0.0316‾\underline{0.0253 \pm 0.0316}0.1277±0.10970.1277 \pm 0.10970.1334±0.19440.1334 \pm 0.1944
Top-20 equal weightFactorMiner0.0075±0.01280.0075 \pm 0.01280.0093±0.01800.0093 \pm 0.01800.0400±0.06690.0400 \pm 0.06690.0482±0.08870.0482 \pm 0.0887
Top-20 equal weightDASH0.0560±0.0101\boldsymbol{0.0560 \pm 0.0101}0.0571±0.0118\boldsymbol{0.0571 \pm 0.0118}0.2889±0.0347\boldsymbol{0.2889 \pm 0.0347}0.2810±0.0385\boldsymbol{0.2810 \pm 0.0385}
Top-40 equal weightR&D-Agent0.0235±0.0221‾\underline{0.0235 \pm 0.0221}0.0205±0.01590.0205 \pm 0.01590.1747±0.1660‾\underline{0.1747 \pm 0.1660}0.1446±0.1072‾\underline{0.1446 \pm 0.1072}
Top-40 equal weightAlphaAgent0.0199±0.01730.0199 \pm 0.01730.0253±0.0316‾\underline{0.0253 \pm 0.0316}0.1277±0.10970.1277 \pm 0.10970.1334±0.19440.1334 \pm 0.1944
Top-40 equal weightFactorMiner0.0196±0.01040.0196 \pm 0.01040.0219±0.01460.0219 \pm 0.01460.1031±0.05540.1031 \pm 0.05540.1113±0.07590.1113 \pm 0.0759
Top-40 equal weightDASH0.0528±0.0155\boldsymbol{0.0528 \pm 0.0155}0.0552±0.0137\boldsymbol{0.0552 \pm 0.0137}0.2742±0.0783\boldsymbol{0.2742 \pm 0.0783}0.2747±0.0596\boldsymbol{0.2747 \pm 0.0596}

Table R4. Factor-only search on S&P 500 (2025 OOD evaluation, 400K-token budget).

AggregationMethodICRank ICICIRRank ICIR
Best factorAlphaAgent0.0359±0.0207‾\underline{0.0359 \pm 0.0207}0.0266±0.01130.0266 \pm 0.01130.2200±0.12150.2200 \pm 0.12150.1523±0.06370.1523 \pm 0.0637
Best factorFactorMiner0.0342±0.01340.0342 \pm 0.01340.0306±0.0138‾\underline{0.0306 \pm 0.0138}0.2687±0.1170\boldsymbol{0.2687 \pm 0.1170}0.2165±0.1060\boldsymbol{0.2165 \pm 0.1060}
Best factorR&D-Agent−0.0060±0.0129-0.0060 \pm 0.0129−0.0046±0.0104-0.0046 \pm 0.0104−0.0971±0.1644-0.0971 \pm 0.1644−0.0647±0.1054-0.0647 \pm 0.1054
Best factorDASH0.0362±0.0029\boldsymbol{0.0362 \pm 0.0029}0.0364±0.0027\boldsymbol{0.0364 \pm 0.0027}0.2241±0.0276‾\underline{0.2241 \pm 0.0276}0.2067±0.0251‾\underline{0.2067 \pm 0.0251}
Top-5 equal weightFactorMiner0.0428±0.0043\boldsymbol{0.0428 \pm 0.0043}0.0381±0.0061‾\underline{0.0381 \pm 0.0061}0.2836±0.0223‾\underline{0.2836 \pm 0.0223}0.2319±0.0360‾\underline{0.2319 \pm 0.0360}
Top-5 equal weightAlphaAgent0.0332±0.01880.0332 \pm 0.01880.0264±0.01120.0264 \pm 0.01120.2118±0.11500.2118 \pm 0.11500.1575±0.06860.1575 \pm 0.0686
Top-5 equal weightR&D-Agent−0.0038±0.0058-0.0038 \pm 0.0058−0.0085±0.0061-0.0085 \pm 0.0061−0.0414±0.0638-0.0414 \pm 0.0638−0.0796±0.0678-0.0796 \pm 0.0678
Top-5 equal weightDASH0.0422±0.0029‾\underline{0.0422 \pm 0.0029}0.0399±0.0030\boldsymbol{0.0399 \pm 0.0030}0.3405±0.0228\boldsymbol{0.3405 \pm 0.0228}0.2886±0.0242\boldsymbol{0.2886 \pm 0.0242}
Top-20 equal weightAlphaAgent0.0332±0.0188‾\underline{0.0332 \pm 0.0188}0.0264±0.0112‾\underline{0.0264 \pm 0.0112}0.2118±0.1150‾\underline{0.2118 \pm 0.1150}0.1575±0.0686‾\underline{0.1575 \pm 0.0686}
Top-20 equal weightFactorMiner0.0242±0.00470.0242 \pm 0.00470.0253±0.00580.0253 \pm 0.00580.1362±0.03940.1362 \pm 0.03940.1305±0.04380.1305 \pm 0.0438
Top-20 equal weightR&D-Agent0.0082±0.00350.0082 \pm 0.00350.0060±0.00610.0060 \pm 0.00610.0665±0.01620.0665 \pm 0.01620.0377±0.04280.0377 \pm 0.0428
Top-20 equal weightDASH0.0435±0.0054\boldsymbol{0.0435 \pm 0.0054}0.0368±0.0045\boldsymbol{0.0368 \pm 0.0045}0.2794±0.0241\boldsymbol{0.2794 \pm 0.0241}0.2215±0.0208\boldsymbol{0.2215 \pm 0.0208}
Top-40 equal weightAlphaAgent0.0332±0.0188‾\underline{0.0332 \pm 0.0188}0.0264±0.0112‾\underline{0.0264 \pm 0.0112}0.2118±0.1150‾\underline{0.2118 \pm 0.1150}0.1575±0.0686‾\underline{0.1575 \pm 0.0686}
Top-40 equal weightFactorMiner0.0233±0.00310.0233 \pm 0.00310.0246±0.00320.0246 \pm 0.00320.1254±0.02130.1254 \pm 0.02130.1208±0.02090.1208 \pm 0.0209
Top-40 equal weightR&D-Agent0.0096±0.00680.0096 \pm 0.00680.0074±0.00270.0074 \pm 0.00270.0794±0.05400.0794 \pm 0.05400.0551±0.00870.0551 \pm 0.0087
Top-40 equal weightDASH0.0443±0.0044\boldsymbol{0.0443 \pm 0.0044}0.0374±0.0041\boldsymbol{0.0374 \pm 0.0041}0.2815±0.0215\boldsymbol{0.2815 \pm 0.0215}0.2235±0.0194\boldsymbol{0.2235 \pm 0.0194}

Response to Q2:

We thank the reviewer for this suggestion. We will include additional factor and model case studies, along with their implementations, in the revised manuscript. To illustrate DASH's concrete search outputs, we provide a factor–model joint-search case study on NASDAQ-100. Table R5 presents ten generated factors, including their names, simplified code, and interpretations. Table R6 details the proposed model specification including model architecture and training configuration.

Table R5. Representative factors discovered by DASH.

Factor NameSimplified CodeInterpretation
market_rel_close_20d_cs-cs_z(hist_pct(close / market_close, 20))Market-relative price reversal based on a stock's historical relative-price percentile.
momentum_5d_cs_rank-cs_rank(ret(5))Cross-sectional short-term reversal based on 5-day returns.
cross_sectional_demeaned_return_5d-(logret(5) - cs_mean(logret(5)))Return reversal relative to the cross-sectional average.
momentum_20d_low_volatility_filter-roll_sum(ret(1), 20).mask(vol20 > cs_q(vol20, 0.75))Volatility-filtered reversal, excluding relatively high-volatility stocks.
cs_neg_3d_return_sum_zscore-cs_z(roll_sum(ret(1) - cs_mean(ret(1)), 3))Standardized 3-day reversal relative to the market-wide cross-sectional mean.
ADV_10_CS_RANK-cs_rank(roll_mean(close * volume, 10))Cross-sectional trading activity and liquidity-related signal.
neg_5d_return_zscore-cs_z(logret(5))Cross-sectionally standardized 5-day log-return reversal.
regime_vol20_quantilecs_rank(roll_std(logret(1), 20))Relative volatility regime based on historical return dispersion.
regime_bull(logret(60) > cs_median(logret(60))).float()Indicator of above-median 60-day stock performance.
regime_bear(logret(60) < cs_median(logret(60))).float()Indicator of below-median 60-day stock performance.

Notes: ret(h) and logret(h) denote per-stock simple and log returns over h observations. roll_* denotes per-stock rolling statistics, cs_* denotes cross-sectional operations, and hist_pct computes the historical percentile.

Reviewer_5dirOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DASH, a diagnostic-guided quality-diversity search framework for financial ML agents. To address noisy validation feedback and nonstationarity, it constructs behavioral profiles across temporal splits, asks the LLM to formulate hypotheses with expected outcomes before additional measurements are computed, and removes hypotheses contradicted by the resulting evidence. A quality-diversity archive retains strong but behaviorally distinct candidates. The evaluation covers NASDAQ-100 and S&P 500, factor-only and joint factor-model search, two LLM backbones, and additional cryptocurrency and point-in-time equity-universe experiments.

Critical strengths
  1. The paper addresses an important failure mode of agentic ML search: noisy validation feedback can shape the entire search trajectory, not just final model selection. Requiring expected evidence before measurement is an interesting and clearly motivated way to constrain post-hoc explanations in financial forecasting.

  2. The system-level evaluation is broad and encouraging. The reported gains span two equity universes, two backbones, factor-only and joint search, and iteration- and token-matched comparisons. The additional cryptocurrency and point-in-time experiments, together with the stated separation of the OOD period from search and leakage safeguards, strengthen the empirical case for the complete system.

Critical weaknesses
  1. The evidence does not sufficiently isolate the central methodological contribution. Table 6 shows that externally computed diagnostic evidence is more useful than LLM-only judgments, but this does not distinguish the benefit of richer diagnostic information from the benefit of pre-specified expectations and explicit hypothesis refutation. A control receiving exactly the same diagnostic metrics and computed evidence, with the pre-commitment and refutation steps removed, is needed to support that attribution. Moreover, Tables 5 and 6 do not appear to report variability across independent runs, although they provide much of the evidence for the proposed mechanism. These issues limit how confidently the system-level gains can be attributed to the claimed contribution.

  2. The efficiency evidence is narrower than a claim about total search compute. Token-matched comparisons are useful, but DASH also performs diagnostic computations and may evaluate multiple refinements. Candidate counts, model-training compute, or wall-clock measurements are needed to establish the corresponding computational cost. Overall, the system-level results are promising, but the unresolved mechanism attribution and uncertainty in the key ablations lead me to recommend weak rejection. A matched-diagnostic control and repeated central ablations could materially change my assessment.

Questions for authors
  1. Can the authors compare DASH with a variant receiving the same externally computed diagnostic evidence but without pre-specified expected outcomes or explicit hypothesis refutation? This is the most important comparison for my assessment of the methodological contribution.

  2. Are Tables 5 and 6 based on single runs? Please clarify the number of independent runs and, where available, report variability for the central mechanism ablations.

  3. Under the token-matched setting, how do the methods compare in the number of evaluated configurations and total training or diagnostic compute? This would clarify the scope of the efficiency claim.

10 comments

We thank the reviewer for recognizing the promise of the system-level results and for the constructive suggestions. We have added a matched-evidence verification ablation, repeated the central ablations across three repeated runs, and expanded the search-efficiency analysis with execution statistics and a wall-clock-matched comparison.

Response to Weakness 1: Verification mechanism and uncertainty

Matched-evidence verification ablation. To distinguish the value of richer diagnostic feedback from that of pre-commitment and explicit refutation, we add No Verdict + Evidence. It retains the externally computed diagnostic metrics and evidence available to DASH, but does not specify expected metric outcomes in advance or assign verdicts to hypotheses. Instead, the agent proposes refinements directly from the hypotheses and computed evidence. For completeness, LLM Verdict makes LLM-only judgments without externally computed measurements, whereas No Verdict performs direct refinement without external evidence or a verdict. Table R1 clarifies the variants.

Table R1. Verification-ablation configurations.

MethodExternal metricsExplicit verdict
DASHYesYes
LLM VerdictNoYes
No VerdictNoNo
No Verdict + EvidenceYesNo

We ran each variant three times using GPT-4.1 and GPT-5-mini on NASDAQ-100 and S&P 500. Table R2 reports mean ± sample standard deviation (SD). DASH has the highest mean IC and ICIR, as well as the highest portfolio IR and least severe MDD, in all four universe–backbone settings. In particular, DASH outperforms No Verdict + Evidence in mean IC across all settings, supporting the contribution of the verification protocol beyond access to external evidence.

Table R2. Verification ablations on NASDAQ-100 and S&P 500 (three repeated runs; mean ± sample SD).

UniverseLLM backboneVariantICICIRRank ICIRIRMDD
NASDAQ-100GPT-4.1DASH0.0312 ± 0.00130.2243 ± 0.00880.2089 ± 0.02041.0428 ± 0.0417-0.0887 ± 0.0470
NASDAQ-100GPT-4.1No Verdict0.0268 ± 0.00310.1675 ± 0.01910.1096 ± 0.00750.5466 ± 0.2579-0.1621 ± 0.0359
NASDAQ-100GPT-4.1LLM Verdict0.0276 ± 0.00510.1706 ± 0.02290.1119 ± 0.01160.8418 ± 0.4251-0.1323 ± 0.0334
NASDAQ-100GPT-4.1No Verdict + Evidence0.0228 ± 0.00170.1441 ± 0.00770.1035 ± 0.01680.5346 ± 0.1887-0.1577 ± 0.0097
S&P 500GPT-4.1DASH0.0175 ± 0.00050.1377 ± 0.00100.0731 ± 0.00340.3150 ± 0.2342-0.0994 ± 0.0131
S&P 500GPT-4.1No Verdict0.0092 ± 0.01260.0692 ± 0.09480.0266 ± 0.0640-0.0180 ± 0.3184-0.1285 ± 0.0397
S&P 500GPT-4.1LLM Verdict0.0148 ± 0.00330.1263 ± 0.00770.0531 ± 0.04080.0396 ± 0.7486-0.1617 ± 0.0902
S&P 500GPT-4.1No Verdict + Evidence0.0144 ± 0.00090.1130 ± 0.00750.0564 ± 0.0046-0.0240 ± 0.0287-0.1379 ± 0.0033
NASDAQ-100GPT-5-miniDASH0.0301 ± 0.00510.2116 ± 0.02360.1056 ± 0.05030.9801 ± 0.5754-0.1010 ± 0.0331
NASDAQ-100GPT-5-miniNo Verdict0.0263 ± 0.00180.1705 ± 0.01000.1220 ± 0.00840.7431 ± 0.3166-0.1458 ± 0.0099
NASDAQ-100GPT-5-miniLLM Verdict0.0283 ± 0.00650.1700 ± 0.04190.1256 ± 0.03470.6799 ± 0.5389-0.1697 ± 0.0668
NASDAQ-100GPT-5-miniNo Verdict + Evidence0.0237 ± 0.00240.1522 ± 0.00880.1128 ± 0.01060.4938 ± 0.3031-0.1570 ± 0.0106
S&P 500GPT-5-miniDASH0.0246 ± 0.00570.1996 ± 0.03810.1478 ± 0.05791.1214 ± 0.1819-0.0871 ± 0.0180
S&P 500GPT-5-miniNo Verdict0.0205 ± 0.00300.1537 ± 0.01720.0959 ± 0.00420.1013 ± 0.6214-0.1419 ± 0.0505
S&P 500GPT-5-miniLLM Verdict0.0176 ± 0.00450.1300 ± 0.02860.0737 ± 0.0190-0.0453 ± 0.5268-0.1622 ± 0.0832
S&P 500GPT-5-miniNo Verdict + Evidence0.0161 ± 0.00160.1212 ± 0.01550.0659 ± 0.0022-0.1640 ± 0.3601-0.1577 ± 0.0530

Parent-level analysis. To further isolate the contribution of explicit hypothesis verification, we randomly sampled 50 parent states from the NASDAQ-100 search traces. For each parent, both branches share the same hypotheses and externally computed diagnostic evidence but follow different refinement procedures: (1) DASH assigns explicit verdicts, excludes refuted hypotheses, and proposes refinements based on the remaining hypotheses; (2) No Verdict + Evidence skips the verdict steps and directly proposes refinements using the same hypotheses and evidence. We random sample three candidate refinements per branch and evaluate children states against their shared parent. Table R3 reports the mean Δ (average child-minus-parent metric difference) and improving fraction (proportion of evaluated children outperforming their parent). Across all four predictive metrics, the full verification branch achieves larger average improvements and higher improving fractions. These results provide additional evidence that pre-specified expectations and verdict-based filtering helps guide more effective refinements under matched diagnostic evidence.

Table R3. Parent-to-child refinement outcomes by hypothesis-verification status on NASDAQ-100 (50 sampled parent states).

MetricDASH Mean ΔNo Verdict + Evidence Mean ΔDASH Improving FractionNo Verdict + Evidence Improving Fraction
IC0.002210.001310.660.37
Rank IC0.003360.001670.550.46
ICIR0.007200.003690.510.42
Rank ICIR0.019120.009340.610.48

Repeated component ablations. We additionally add the component ablations from manuscript Table 5 over three repeated runs (Table R4). DASH achieves the highest mean IC, ICIR, Rank IC, and Rank ICIR in both universes. The NASDAQ-100 w/o hybrid splits variant attains a slightly higher mean portfolio IR, so the portfolio effects are not uniform across metrics. Overall, the repeated results provide further support for the complementary roles of temporal diagnostics, diagnostic-guided refinement, and the quality-diversity archive in predictive performance.

Table R4. Three-run component ablations corresponding to manuscript Table 5, using GPT-4.1 (mean ± sample SD).

UniverseVariantICICIRRank ICRank ICIRIRMDD
NASDAQ-100DASH0.0312 ± 0.00130.2243 ± 0.00880.0275 ± 0.00280.2089 ± 0.02041.0428 ± 0.0417-0.0887 ± 0.0470
NASDAQ-100w/o hybrid splits0.0297 ± 0.00770.1806 ± 0.04370.0204 ± 0.00360.1302 ± 0.02101.1029 ± 0.0555-0.1319 ± 0.0224
NASDAQ-100w/o archive0.0210 ± 0.01710.1263 ± 0.09710.0174 ± 0.00630.1141 ± 0.03700.7384 ± 0.8171-0.1416 ± 0.0497
NASDAQ-100w/o diagnostics0.0293 ± 0.00200.1859 ± 0.01980.0201 ± 0.00180.1308 ± 0.01490.7311 ± 0.1240-0.1554 ± 0.0070
NASDAQ-100w/o LLM Rank0.0304 ± 0.00160.1802 ± 0.00360.0181 ± 0.00200.1150 ± 0.01151.0292 ± 0.0975-0.1380 ± 0.0092
S&P 500DASH0.0175 ± 0.00050.1377 ± 0.00100.0101 ± 0.00080.0731 ± 0.00340.3150 ± 0.2342-0.0994 ± 0.0131
S&P 500w/o hybrid splits0.0167 ± 0.00310.1298 ± 0.02230.0099 ± 0.00260.0715 ± 0.01890.0665 ± 0.3147-0.1267 ± 0.0195
S&P 500w/o archive0.0170 ± 0.00200.1299 ± 0.01260.0100 ± 0.00250.0708 ± 0.01580.0562 ± 0.3790-0.1328 ± 0.0300
S&P 500w/o diagnostics0.0164 ± 0.00200.1287 ± 0.01140.0091 ± 0.00350.0645 ± 0.0236-0.0215 ± 0.2061-0.1380 ± 0.0376
S&P 500w/o LLM Rank0.0174 ± 0.00220.1304 ± 0.01020.0085 ± 0.00060.0602 ± 0.00590.1098 ± 0.2063-0.1336 ± 0.0157

Response to Weakness 2: Search efficiency and computational cost

Figures 3 and 10 of the manuscript report parent-to-child improvement rates and positive gains at different token budgets; bubble size indicates the number of evaluated transitions. These figures demonstrate that, under matched token budgets, DASH evaluates fewer candidate configurations while achieving higher parent-to-child improvement rates and larger gains among successful refinements. These results highlight DASH's ability to prioritize more promising search directions. To further quantify the associated computational costs, we provide additional comparisons of evaluated candidate counts, LLM API calls, search wall-clock time, and model-fitting time under 200K and 400K token budgets (Tables R5a and R5b), along with a breakdown of DASH's token usage.

Token-matched execution statistics. Tables R5a and R5b report completed evaluated candidate configurations, LLM API calls, search wall-clock time, and search-stage model-fitting elapsed time at the 200K- and 400K-token checkpoints. All experiments use the same per-run resource allocation of 8 CPU cores and 24 GB of GPU memory across DASH and the baselines. We also report two post hoc parent-to-child OOD IC summaries across generated children: the improving-transition rate (percentage of evaluated children with higher OOD IC than their parent) and mean relative ΔOOD IC (average parent-to-child percentage change). Under matched token budgets, DASH evaluates fewer candidate configurations while achieving stronger parent-to-child improvement statistics and requiring less measured search-stage model-fitting time. However, it generally uses more LLM API calls and does not consistently achieve the shortest wall-clock time. These results suggest that DASH prioritizes more promising search directions, while incurring additional LLM interaction costs.

Table R5a. PIT NASDAQ-100: token-matched search execution and model-fitting time (three repeated runs).

MethodToken budgetEvaluated candidatesLLM API callsWall-clock (h)Search model fitting (h)Improving transitions (%)Mean relative ΔOOD IC (%)
DASH200K5.67 ± 0.5848.67 ± 5.690.80 ± 0.160.01 ± 0.0060.673.60
AutoResearch200K12.00 ± 1.0028.67 ± 3.211.48 ± 0.500.06 ± 0.0227.27-3.73
AIRA200K11.00 ± 2.6522.67 ± 1.531.71 ± 0.410.04 ± 0.0137.50-3.48
ML-Master 2.0200K9.67 ± 0.5824.00 ± 2.002.29 ± 0.480.38 ± 0.0631.25-0.69
R&D-Agent200K8.00 ± 3.6142.33 ± 3.511.19 ± 0.360.02 ± 0.0018.75-11.41
DASH400K11.00 ± 1.0075.00 ± 7.213.83 ± 1.680.03 ± 0.0080.008.12
AutoResearch400K21.67 ± 4.6254.00 ± 5.292.84 ± 1.130.31 ± 0.3242.11-0.70
AIRA400K17.67 ± 4.7338.67 ± 2.313.24 ± 1.170.08 ± 0.0425.81-3.39
ML-Master 2.0400K20.00 ± 1.0042.67 ± 3.794.67 ± 1.860.45 ± 0.0825.000.81
R&D-Agent400K16.67 ± 4.7378.00 ± 7.212.39 ± 0.470.06 ± 0.0119.44-10.19

Table R5b. PIT S&P 500: token-matched search execution and model-fitting time (three repeated runs).

MethodToken budgetEvaluated candidatesLLM API callsWall-clock (h)Search model fitting (h)Improving transitions (%)Mean relative ΔOOD IC (%)
DASH200K6.67 ± 0.5838.33 ± 9.242.46 ± 0.380.05 ± 0.0175.0051.80
AutoResearch200K12.67 ± 0.5827.67 ± 1.537.27 ± 2.410.17 ± 0.0433.335.42
AIRA200K12.33 ± 0.5822.67 ± 2.086.20 ± 0.440.17 ± 0.0721.056.99
ML-Master 2.0200K8.67 ± 1.1523.00 ± 1.009.06 ± 3.262.06 ± 3.0243.754.49
R&D-Agent200K8.00 ± 1.7336.67 ± 1.534.33 ± 0.820.08 ± 0.0350.004.82
DASH400K11.67 ± 1.1585.33 ± 5.864.24 ± 0.360.10 ± 0.0244.4418.59
AutoResearch400K23.33 ± 3.7949.67 ± 3.0615.86 ± 8.110.37 ± 0.0825.712.98
AIRA400K21.00 ± 2.0037.33 ± 3.2111.62 ± 1.000.36 ± 0.1813.334.43
ML-Master 2.0400K16.67 ± 2.5240.00 ± 2.0018.82 ± 7.002.42 ± 2.9444.122.20
R&D-Agent400K14.33 ± 3.2168.67 ± 2.529.03 ± 2.210.19 ± 0.0629.171.90
Reviewer_WyS2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DASH, an LLM agent that searches for factors and model configurations to predict daily stock returns. Each candidate is tested on a fixed reference split and two random diagnostic splits, using six metrics covering predictive performance, market exposure, stability, generalization, regime sensitivity, and prediction quality. Before making changes, the agent proposes hypotheses about a candidate's weaknesses, checks them using code-based metrics, and uses the supported hypotheses to guide its next edits.

DASH also maintains a MAP-Elites-style archive to preserve diverse candidates, ranking them by IC, Rank IC, and a prediction-correlation penalty. Experiments cover joint factor–model and factor-only search on the NASDAQ-100 and S&P 500, using 2005–2022 for search and 2023–2025 for forward testing, with additional tests on cryptocurrency and point-in-time universes. The paper compares DASH with four agentic ML systems, three factor-mining agents, and standard ML models, and includes ablations of its main components.

Critical strengths
  1. Grounding hypotheses in numerical evidence. DASH requires the agent to specify which metric and data split would support a hypothesis before computing the result. This helps reduce post-hoc explanations of noisy validation gains. The metric-grounding ablation on NASDAQ-100 (Table 6) provides some evidence for this: replacing computed metrics with LLM-only judgments reduces IC from 0.0319 to 0.0189 with GPT-4.1, although the statistical reliability of this result remains unclear.

  2. Careful leakage controls and broad evaluation. The agent cannot modify the evaluation harness, data splits, or targets. Generated factors also undergo static checks and causal truncation tests (Appendix H), while OOD results are kept out of prompts. The experiments cover two search modes, token-matched and iteration-matched budgets, cryptocurrency data, and point-in-time universes. These choices make the evaluation more thorough and easier to assess than much of the existing LLM factor-mining work.

Critical weaknesses
  1. The main comparisons are selective, and the gains lack statistical support.

    • Baseline comparisons: The abstract selects the best DASH backbone for each market. With GPT-4.1 fixed, DASH trails R&D-Agent on S&P 500 IC (0.0175 vs. 0.0178) and AutoResearch on IR (0.315 vs. 0.518). On NASDAQ-100, XGBoost with Alpha158 also achieves higher IC (0.0323 vs. 0.0312) and ARR (0.1068 vs. 0.0569).
    • Statistical reliability: Only three runs are reported, with IR standard deviations up to 0.58 for DASH and 0.87 for baselines. No paired tests or HAC corrections for overlapping 5-day returns are provided.
    • Ablations: The full-method reference in Tables 5, 6, and 12 (IC 0.0319, IR 1.5581) differs substantially from Table 1 (IC 0.0312, IR 1.0428 ± 0.0417). On S&P 500, removing the diagnostic loop changes IC by only −0.0013, while removing hybrid splits improves ARR and IR.
    • Compute fairness: Table 21 reports identical baseline results across some methods and token budgets (100K–400K), suggesting limited actual search. The iteration-matched comparison also omits token and LLM-call counts, making efficiency claims difficult to assess.
  2. Factor-quality claims and diagnostic validity need stronger evidence.

    • Factor diversity: Tables 17–18 contradict the claimed low redundancy. On NASDAQ-100, DASH's diversity is 0.8403 versus 0.8741 for FactorMiner and 0.9041 for R&D-Agent, while redundancy is 2.5% versus FactorMiner's 1.0%.
    • Selection bias: DASH filters factors using ∣IC∣≥0.01|\mathrm{IC}| \ge 0.01, potentially inflating the reported concentration of strong OOD factors. Table 3 also appears to report single-run best-factor results.
    • Survivorship bias: Main experiments appear to use current constituents. On PIT S&P 500 at 100K tokens, DASH's Rank ICIR (0.0307) trails AutoResearch (0.0469) and ML-Master 2.0 (0.0453).
    • Data leakage: The 5-day forward-return targets lack purge or embargo periods between training (ending 2018-12-31), validation (starting 2019-01-05, ending 2022-12-31), and testing (starting 2023-01-05).
    • Hypothesis verification: The LLM judges hypotheses rather than mechanically comparing zjz_j with πj\pi_j, and it sees PiP_i beforehand. In Figure 2, declining SHAP interactions (0.142 to 0.018) and Rank IC (0.031 to 0.009) lead to greater model complexity, although they may indicate overfitting. No rejection rates or rule-based verification comparisons are reported.
Questions for authors
  1. Please report the main comparison with the backbone held fixed in both markets, and include XGBoost-Alpha158 in the headline comparison. Can you provide paired tests (e.g., Diebold–Mariano on daily IC differences, with HAC standard errors) for DASH against the strongest baseline?
  2. Are the ablations in Tables 5–6 and the backbone results in Table 12 single runs? Why does the full-method NASDAQ-100 IR (1.5581) differ so much from the Table 1 mean (1.0428 ± 0.0417)? Please repeat the ablations over the same three seeds used in Table 1.
9 comments

We thank the reviewer for the detailed assessment. Below we clarify the backbone-controlled comparisons, report repeated ablations and search-resource measurements, and address factor selection, point-in-time evaluation, label maturity, and hypothesis verification.

Questions

Q1. Backbone-controlled main comparisons, XGBoost-Alpha158, and paired tests

We thank the reviewer for this suggestion and agree that our main performance claims should be more precisely stated. Under backbone-controlled comparisons with strong agentic baselines, DASH achieves higher mean OOD IC in three of the four market–backbone settings, with the exception of GPT-4.1 on S&P 500 (0.0175 vs. 0.0178 for R&D-Agent). Against the strong non-agentic baseline XGBoost-Alpha158, DASH achieves higher ICIR and portfolio IR on NASDAQ-100 despite a slightly lower IC, and higher IC on S&P 500 under both backbones. We will revise the abstract and main-results discussion to emphasize these specific advantages.

Statistical significance analysis. To further assess DASH's predictive improvements, we conduct paired Newey–West HAC tests on daily OOD IC and Rank IC differences against strong baselines, using a lag length of four to account for temporal dependence induced by overlapping five-day prediction targets. We prioritize backbone-controlled comparisons between agentic methods and additionally include XGBoost-Alpha158 and MLP-OHLCV as non-agentic baselines. Tables R1a–R1c report the corresponding confidence intervals, with significance interpreted at nominal levels. On NASDAQ-100, DASH achieves a statistically significant Rank IC improvement over XGBoost-Alpha158 at the 10% level. Compared with R&D-Agent under matched backbones, DASH achieves a significant Rank IC improvement with GPT-4.1 and a significant IC improvement with GPT-5-mini, both at the 5% level. On S&P 500, DASH with GPT-5-mini significantly outperforms R&D-Agent in both IC and Rank IC, and MLP-OHLCV in IC, at the 5% level. Overall, these results provide further statistical support for DASH's predictive advantages over strong baselines, while highlighting that the magnitude and significance of these improvements vary across markets, LLM backbones, and evaluation metrics.

Table R1a. Paired HAC tests for the main comparisons on NASDAQ-100 (three repeated runs; Newey–West lag = 4).

ComparisonMetricMean ΔHAC SE95% CI90% CI
DASH GPT-4.1 vs. XGBoost-Alpha158IC-0.00110.00762[-0.01603, +0.01383][-0.01363, +0.01143]
DASH GPT-4.1 vs. XGBoost-Alpha158Rank IC+0.00450.00269[-0.00077, +0.00977][+0.00008, +0.00892]
DASH GPT-4.1 vs. R&D-Agent GPT-4.1IC+0.00250.00312[-0.00361, +0.00861][-0.00262, +0.00762]
DASH GPT-4.1 vs. R&D-Agent GPT-4.1Rank IC+0.00890.00286[+0.00330, +0.01450][+0.00420, +0.01360]
DASH GPT-5-mini vs. R&D-Agent GPT-5-miniIC+0.00500.00246[+0.00019, +0.00981][+0.00096, +0.00904]
DASH GPT-5-mini vs. R&D-Agent GPT-5-miniRank IC-0.00620.00417[-0.01437, +0.00197][-0.01306, +0.00066]

Table R1b. Paired HAC tests for the main comparisons on S&P 500 (three repeated runs; Newey–West lag = 4).

ComparisonMetricMean ΔHAC SE95% CI90% CI
DASH GPT-4.1 vs. R&D-Agent GPT-4.1IC-0.00030.00272[-0.00563, +0.00503][-0.00478, +0.00418]
DASH GPT-4.1 vs. R&D-Agent GPT-4.1Rank IC-0.00040.00275[-0.00579, +0.00499][-0.00492, +0.00412]
DASH GPT-5-mini vs. R&D-Agent GPT-5-miniIC+0.00830.00205[+0.00428, +0.01232][+0.00493, +0.01167]
DASH GPT-5-mini vs. R&D-Agent GPT-5-miniRank IC+0.01200.00190[+0.00828, +0.01572][+0.00887, +0.01513]
DASH GPT-5-mini vs. MLP-OHLCVIC+0.01280.00390[+0.00516, +0.02044][+0.00639, +0.01921]
DASH GPT-5-mini vs. MLP-OHLCVRank IC+0.00650.00402[-0.00138, +0.01438][-0.00011, +0.01311]

To assess the statistical significance of DASH's improvements in the factor-only search setting, we conduct paired HAC tests under the 400K-token budget, comparing DASH against R&D-Agent. We select R&D-Agent as the baseline because it achieves the second-highest IC across all four evaluated settings. As shown in Table R1c, DASH consistently outperforms R&D-Agent in both IC and Rank IC across the Top-20 and Top-40 equal-weight ensembles. The improvements are statistically significant at the 5% level in all four NASDAQ-100 comparisons and at the 10% level in all four S&P 500 comparisons, with the Top-40 IC improvement on S&P 500 also reaching the 5% level.

Table R1c. Paired HAC tests for factor-only search at the 400K-token budget (three repeated runs; Newey–West lag = 4).

UniverseAggregationMetricDASH (Mean ± SD)Baseline (Mean ± SD)95% CI90% CI
NASDAQ-100Top-20IC0.0349 ± 0.00140.0183 ± 0.0123[+0.00592, +0.02728][+0.00764, +0.02556]
NASDAQ-100Top-20Rank IC0.0324 ± 0.00460.0148 ± 0.0089[+0.00713, +0.02807][+0.00882, +0.02638]
NASDAQ-100Top-40IC0.0337 ± 0.00500.0170 ± 0.0100[+0.00619, +0.02721][+0.00788, +0.02552]
NASDAQ-100Top-40Rank IC0.0334 ± 0.00220.0161 ± 0.0092[+0.00719, +0.02741][+0.00881, +0.02579]
S&P 500Top-20IC0.0211 ± 0.00240.0127 ± 0.0107[-0.00034, +0.01714][+0.00106, +0.01574]
S&P 500Top-20Rank IC0.0149 ± 0.00080.0069 ± 0.0041[-0.00023, +0.01623][+0.00109, +0.01491]
S&P 500Top-40IC0.0218 ± 0.00160.0126 ± 0.0127[+0.00020, +0.01820][+0.00165, +0.01675]
S&P 500Top-40Rank IC0.0149 ± 0.00070.0065 ± 0.0036[-0.00046, +0.01726][+0.00097, +0.01583]

Q2. Single-run ablations versus three-run main results

The original manuscript Tables 5 and 6 report single-run results, whereas Table 1 reports mean ± SD across three repeated runs. We now repeated the main verification and component ablations three times and report the updated results in Tables R2 and R3. In Table R2, No Verdict directly proposes refinements without supplemental external evidence or explicit hypothesis verification; LLM Verdict assigns verdicts without externally computed measurements; and No Verdict + Evidence retains external diagnostic evidence but removes both pre-specified expected outcomes and explicit verdict/refutation, proceeding directly to refinement. Across both markets and LLM backbones, DASH achieves the highest mean IC and ICIR among these verification variants. Table R3 further shows that DASH leads the reported component variants in all four predictive metrics under GPT-4.1 on both universes. These results provide additional support for the contribution of the overall verification and search framework.

Table R2. Verification ablations on NASDAQ-100 and S&P 500 across three repeated runs.

UniverseLLM backboneVariantICICIRRank ICIRIRMDD
NASDAQ-100GPT-4.1DASH0.0312 ± 0.00130.2243 ± 0.00880.2089 ± 0.02041.0428 ± 0.0417-0.0887 ± 0.0470
NASDAQ-100GPT-4.1No Verdict0.0268 ± 0.00310.1675 ± 0.01910.1096 ± 0.00750.5466 ± 0.2579-0.1621 ± 0.0359
NASDAQ-100GPT-4.1LLM Verdict0.0276 ± 0.00510.1706 ± 0.02290.1119 ± 0.01160.8418 ± 0.4251-0.1323 ± 0.0334
NASDAQ-100GPT-4.1No Verdict + Evidence0.0228 ± 0.00170.1441 ± 0.00770.1035 ± 0.01680.5346 ± 0.1887-0.1577 ± 0.0097
S&P 500GPT-4.1DASH0.0175 ± 0.00050.1377 ± 0.00100.0731 ± 0.00340.3150 ± 0.2342-0.0994 ± 0.0131
S&P 500GPT-4.1No Verdict0.0092 ± 0.01260.0692 ± 0.09480.0266 ± 0.0640-0.0180 ± 0.3184-0.1285 ± 0.0397
S&P 500GPT-4.1LLM Verdict0.0148 ± 0.00330.1263 ± 0.00770.0531 ± 0.04080.0396 ± 0.7486-0.1617 ± 0.0902
S&P 500GPT-4.1No Verdict + Evidence0.0144 ± 0.00090.1130 ± 0.00750.0564 ± 0.0046-0.0240 ± 0.0287-0.1379 ± 0.0033
NASDAQ-100GPT-5-miniDASH0.0301 ± 0.00510.2116 ± 0.02360.1056 ± 0.05030.9801 ± 0.5754-0.1010 ± 0.0331
NASDAQ-100GPT-5-miniNo Verdict0.0263 ± 0.00180.1705 ± 0.01000.1220 ± 0.00840.7431 ± 0.3166-0.1458 ± 0.0099
NASDAQ-100GPT-5-miniLLM Verdict0.0283 ± 0.00650.1700 ± 0.04190.1256 ± 0.03470.6799 ± 0.5389-0.1697 ± 0.0668
NASDAQ-100GPT-5-miniNo Verdict + Evidence0.0237 ± 0.00240.1522 ± 0.00880.1128 ± 0.01060.4938 ± 0.3031-0.1570 ± 0.0106
S&P 500GPT-5-miniDASH0.0246 ± 0.00570.1996 ± 0.03810.1478 ± 0.05791.1214 ± 0.1819-0.0871 ± 0.0180
S&P 500GPT-5-miniNo Verdict0.0205 ± 0.00300.1537 ± 0.01720.0959 ± 0.00420.1013 ± 0.6214-0.1419 ± 0.0505
S&P 500GPT-5-miniLLM Verdict0.0176 ± 0.00450.1300 ± 0.02860.0737 ± 0.0190-0.0453 ± 0.5268-0.1622 ± 0.0832
S&P 500GPT-5-miniNo Verdict + Evidence0.0161 ± 0.00160.1212 ± 0.01550.0659 ± 0.0022-0.1640 ± 0.3601-0.1577 ± 0.0530
Reviewer_4yjVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary The diagnostic loop is a sensible idea for this setting: a good validation score alone says little about why a financial model works or fails. DASH adds temporal diagnostics, checks proposed explanations against computed evidence, and keeps several configurations in an archive. The additional market and point-in-time experiments are useful. I am less sure that the current ablations pin down the value of verification itself, and the market purity descriptor needs more justification.

Critical strengths

Strengths • The motivation fits the application well. Looking at when a configuration fails is useful in a setting where an aggregate validation score can hide substantial variation across market conditions. Specifying the expected outcomes of measurements in advance also makes the reasoning behind a proposed edit easier to inspect. • The joint-search results include three independent runs and two main LLM backbones, with additional comparisons under matched token budgets. The cryptocurrency and point-in-time universe experiments are useful additions. On S&P 500, for example, the best DASH variant reaches an IC of 0.0246 compared with 0.0178 for the strongest agentic baseline in Table 2. • The paper gives useful detail on evaluation safeguards. The separation of search and forward evaluation, restrictions on code edits, and causal truncation checks for generated factors address important failure modes in this type of system.

Critical weaknesses

Weaknesses

  1. Table 6 does not isolate hypothesis verification: LLM Verdict also removes computed evidence, and No Verdict is underspecified. A control receiving the same measurements but refining directly would better establish the gain beyond the feedback mechanisms in R&D-Agent and AlphaEvolve [1, 2].
  2. The purity descriptor in Appendix C.4 changes when a date-specific constant is added to all predictions, although daily IC, Rank IC, and TopK rankings stay unchanged. Its relation to portfolio exposure needs justification. Table 5 also leaves the contribution of behavioral binning unclear.
  3. Tables 5 and 6 and the factor-only results do not report run counts or uncertainty estimates. Paired repetitions would help establish whether the component gains are stable, accounting for temporal dependence where relevant.
  4. Portfolio metrics exclude transaction costs, leaving net performance unclear. The five-day target also requires an explicit rule for labels whose realization crosses training and validation boundaries, particularly for diagnostic splits without a gap. [1] Li et al. (2025). R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization. [2] Novikov et al. (2025). AlphaEvolve: A coding agent for scientific and algorithmic discovery.
Questions for authors
  1. Does No Verdict receive exactly the same supplemental evidence as DASH?
  2. Are predictions normalized before computing market purity?
  3. How many runs underlie Tables 5 and 6, and how are labels crossing split boundaries handled?
  4. Does the portfolio advantage remain after transaction costs, and what turnover do the selected strategies produce?
9 comments

We thank the reviewer for the careful assessment and constructive suggestions. We have clarified the verification ablations, added repeated experiments and a behavioral-binning control, and report paired HAC tests and transaction-cost-adjusted portfolio results below.

Weakness 1: Isolating hypothesis verification

In LLM Verdict, the LLM assigns a hypothesis verdict without externally computed supplemental metrics; No Verdict omits both external metric queries and the verdict step, proceeding directly from the hypotheses to refinements. We additionally evaluate No Verdict + Evidence, which retains the same external metric-computation protocol as DASH but removes explicit verdict-based filtering. This provides a more direct comparison of the contribution of the verdict mechanism while controlling for access to external evidence, although the realized search trajectories may differ across variants.

Table R1. Verification-ablation configurations.

MethodExternal metricsExplicit verdict
DASHYesYes
LLM VerdictNoYes
No VerdictNoNo
No Verdict + EvidenceYesNo

We evaluate the verification variants on NASDAQ-100 and S&P 500 with both GPT-4.1 and GPT-5-mini, reporting the mean and sample standard deviation across three repeated runs in Table R2. DASH achieves higher mean IC, ICIR, and portfolio IR and a less severe MDD than all three variants in each market–backbone setting. Rank ICIR is not uniformly best for DASH (in particular, for GPT-5-mini on NASDAQ-100). These comparisons support the utility of the full verification pipeline, especially the explicit verdict step when external evidence is available.

Table R2. Verification ablations on NASDAQ-100 and S&P 500 using GPT-4.1 and GPT-5-mini (mean ± sample SD across three repeated runs).

MarketLLM backboneVariantICICIRRank ICIRIRMDD
NASDAQ-100GPT-4.1DASH0.0312 ± 0.00130.2243 ± 0.00880.2089 ± 0.02041.0428 ± 0.0417-0.0887 ± 0.0470
NASDAQ-100GPT-4.1No Verdict0.0268 ± 0.00310.1675 ± 0.01910.1096 ± 0.00750.5466 ± 0.2579-0.1621 ± 0.0359
NASDAQ-100GPT-4.1LLM Verdict0.0276 ± 0.00510.1706 ± 0.02290.1119 ± 0.01160.8418 ± 0.4251-0.1323 ± 0.0334
NASDAQ-100GPT-4.1No Verdict + Evidence0.0228 ± 0.00170.1441 ± 0.00770.1035 ± 0.01680.5346 ± 0.1887-0.1577 ± 0.0097
S&P 500GPT-4.1DASH0.0175 ± 0.00050.1377 ± 0.00100.0731 ± 0.00340.3150 ± 0.2342-0.0994 ± 0.0131
S&P 500GPT-4.1No Verdict0.0092 ± 0.01260.0692 ± 0.09480.0266 ± 0.0640-0.0180 ± 0.3184-0.1285 ± 0.0397
S&P 500GPT-4.1LLM Verdict0.0148 ± 0.00330.1263 ± 0.00770.0531 ± 0.04080.0396 ± 0.7486-0.1617 ± 0.0902
S&P 500GPT-4.1No Verdict + Evidence0.0144 ± 0.00090.1130 ± 0.00750.0564 ± 0.0046-0.0240 ± 0.0287-0.1379 ± 0.0033
NASDAQ-100GPT-5-miniDASH0.0301 ± 0.00510.2116 ± 0.02360.1056 ± 0.05030.9801 ± 0.5754-0.1010 ± 0.0331
NASDAQ-100GPT-5-miniNo Verdict0.0263 ± 0.00180.1705 ± 0.01000.1220 ± 0.00840.7431 ± 0.3166-0.1458 ± 0.0099
NASDAQ-100GPT-5-miniLLM Verdict0.0283 ± 0.00650.1700 ± 0.04190.1256 ± 0.03470.6799 ± 0.5389-0.1697 ± 0.0668
NASDAQ-100GPT-5-miniNo Verdict + Evidence0.0237 ± 0.00240.1522 ± 0.00880.1128 ± 0.01060.4938 ± 0.3031-0.1570 ± 0.0106
S&P 500GPT-5-miniDASH0.0246 ± 0.00570.1996 ± 0.03810.1478 ± 0.05791.1214 ± 0.1819-0.0871 ± 0.0180
S&P 500GPT-5-miniNo Verdict0.0205 ± 0.00300.1537 ± 0.01720.0959 ± 0.00420.1013 ± 0.6214-0.1419 ± 0.0505
S&P 500GPT-5-miniLLM Verdict0.0176 ± 0.00450.1300 ± 0.02860.0737 ± 0.0190-0.0453 ± 0.5268-0.1622 ± 0.0832
S&P 500GPT-5-miniNo Verdict + Evidence0.0161 ± 0.00160.1212 ± 0.01550.0659 ± 0.0022-0.1640 ± 0.3601-0.1577 ± 0.0530

Weakness 2: Market-purity descriptor and behavioral binning

As defined in Appendix C.4, market purity is a prediction-level market-alignment proxy, not a direct measure of portfolio beta or market neutrality. It complements daily cross-sectional IC and Rank IC by characterizing the temporal co-movement between the cross-sectional mean of raw predictions and recent benchmark returns. The descriptor is inexpensive to compute and does not require selecting a portfolio-construction rule. We agree with the reviewer that a date-specific common shift in predictions can change this value without changing daily IC, Rank IC, or TopK rankings. We will explicitly state this shift sensitivity and consistently refer to the descriptor as a proxy, rather than a portfolio-exposure measure.

To isolate behavioral binning from the rest of the archive mechanism, we add a w/o behavioral binning control. It preserves archive admission but selects parents globally rather than sampling them through behavioral-cell quotas. Table R3 reports the three-run comparison. DASH achieves higher average IC, ICIR, Rank ICIR, and portfolio IR in both universes.This provides descriptive evidence for the value of cell-based parent selection.

Table R3. Behavioral-binning ablation with GPT-4.1 (mean ± sample SD across three repeated runs).

MarketLLM backboneVariantICICIRRank ICRank ICIRIRMDD
NASDAQ-100GPT-4.1DASH0.0312 ± 0.00130.2243 ± 0.00880.0275 ± 0.00280.2089 ± 0.02041.0428 ± 0.0417-0.0887 ± 0.0470
NASDAQ-100GPT-4.1w/o behavioral binning0.0274 ± 0.00470.1649 ± 0.01460.0184 ± 0.00100.1159 ± 0.00830.7951 ± 0.1730-0.1400 ± 0.0200
S&P 500GPT-4.1DASH0.0175 ± 0.00050.1377 ± 0.00100.0101 ± 0.00080.0731 ± 0.00340.3150 ± 0.2342-0.0994 ± 0.0131
S&P 500GPT-4.1w/o behavioral binning0.0173 ± 0.00010.1309 ± 0.00390.0101 ± 0.00110.0702 ± 0.00930.0556 ± 0.0607-0.1275 ± 0.0234

Weakness 3: Run counts, uncertainty, and temporal dependence

The component and metric-grounding ablations in manuscript Tables 5 and 6 were originally reported from one run per setting. We have now conducted three repeated runs per setting for the central verification and component ablations (Tables R2 and R6) and the factor-only evaluation at a 400K-token budget (Tables R4 and R5). All repeated-run tables report mean ± sample standard deviation across runs. The factor-only repeats reveal an important distinction between individual factors and ensembles. On NASDAQ-100, the best-factor and Top-5 results vary substantially across runs, while DASH has the highest mean predictive metrics for the Top-20 and Top-40 equal-weight ensembles. On S&P 500, DASH has the highest mean IC, Rank IC, ICIR, and Rank ICIR across the displayed aggregation settings (Tables R4–R5).

Table R4. Factor-only OOD results on NASDAQ-100 at a 400K-token budget (mean ± sample SD across three repeated runs).

AggregationMethodICRank ICICIRRank ICIR
Best factorDASH0.0106 ± 0.01450.0173 ± 0.01550.0706 ± 0.09090.0942 ± 0.0856
Best factorAlphaAgent0.0099 ± 0.01060.0066 ± 0.01530.0846 ± 0.08820.0336 ± 0.0923
Best factorR&D-Agent0.0098 ± 0.00260.0039 ± 0.00720.0684 ± 0.02190.0255 ± 0.0459
Best factorFactorMiner-0.0072 ± 0.0047-0.0039 ± 0.0057-0.0470 ± 0.0313-0.0252 ± 0.0365
Top-5 equal-weightDASH0.0177 ± 0.01830.0215 ± 0.01690.1026 ± 0.10610.1149 ± 0.0899
Top-5 equal-weightR&D-Agent0.0171 ± 0.00950.0098 ± 0.01580.1131 ± 0.05780.0572 ± 0.0940
Top-5 equal-weightAlphaAgent0.0136 ± 0.00530.0179 ± 0.01670.0996 ± 0.02600.1080 ± 0.1011
Top-5 equal-weightFactorMiner-0.0099 ± 0.0014-0.0055 ± 0.0031-0.0607 ± 0.0124-0.0313 ± 0.0177
Top-20 equal-weightDASH0.0349 ± 0.00140.0324 ± 0.00460.1989 ± 0.01540.1781 ± 0.0197
Top-20 equal-weightR&D-Agent0.0183 ± 0.01230.0148 ± 0.00890.1405 ± 0.09050.1099 ± 0.0674
Top-20 equal-weightAlphaAgent0.0136 ± 0.00530.0179 ± 0.01670.0996 ± 0.02600.1080 ± 0.1011
Top-20 equal-weightFactorMiner0.0024 ± 0.00390.0037 ± 0.00330.0146 ± 0.02290.0208 ± 0.0182
Top-40 equal-weightDASH0.0337 ± 0.00500.0334 ± 0.00220.1921 ± 0.02590.1849 ± 0.0121
Top-40 equal-weightR&D-Agent0.0170 ± 0.01000.0161 ± 0.00920.1319 ± 0.07520.1197 ± 0.0688
Top-40 equal-weightAlphaAgent0.0136 ± 0.00530.0179 ± 0.01670.0996 ± 0.02600.1080 ± 0.1011
Top-40 equal-weightFactorMiner0.0122 ± 0.00450.0102 ± 0.00350.0736 ± 0.02850.0565 ± 0.0211

Note: Top-N ensembles are constructed from up to N eligible factors selected using validation performance. When fewer than N eligible factors are available, all eligible factors are used, which may result in identical performance across different ensemble sizes.

Comments on the paper

2 comments

Dear Authors,

Thank you for your response. However, the current rebuttal on OpenReview contains too much content across many nested layers, which makes it very difficult for us to locate the actual changes to the tables.

Please do the following in your revised manuscript:

  1. Update the tables directly in the manuscript file.
  2. Highlight/mark all changed cells or rows (e.g., bold, color, or a "changed" column) so the modifications are visible at a glance.
  3. In your response, provide a short summary list (e.g., "Table 2: updated rows 3–5; Table 4: added column X") rather than repeating the full content.

Dear Reviewer,

Thank you for your feedback and suggestions.

We have updated and uploaded the revised manuscript as requested, with the changes to the tables clearly highlighted for easier reference.

We would greatly appreciate it if you could take a look at the revised manuscript and let us know whether these changes address your concerns. We would be happy to provide any further clarification if needed.

Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.