- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces EvoQuantLab, a training-free framework for automated quantitative factor discovery. The proposed system combines an inner genetic programming loop for symbolic factor evolution with an outer diagnosis loop that modifies research protocols based on development-set evidence. It also incorporates a multi-stage validation pipeline intended to reduce backtest overfitting and improve the reliability of discovered factors.
The authors evaluate the framework against four LLM-based quantitative research agents on CSI500, S&P500, and hourly cryptocurrency datasets. They report improvements in predictive metrics and portfolio returns, along with component ablations, development-to-test transfer analyses, and computational efficiency comparisons.
Critical strengths
-
Relevant problem and systematic workflow. The paper addresses adaptive overfitting in automated quantitative research, which is an important practical problem. Separating factor search, development validation, and held-out evaluation is a sensible design principle.
-
Broad evaluation scope. The evaluation includes multiple markets and frequencies, several agent baselines, component ablations, and development-to-OOS diagnostics. These are useful ingredients for studying automated factor discovery.
Critical weaknesses
- The experimental comparisons are insufficiently controlled, and the reported efficiency claims are not adequately supported.
The main experiments compare EvoQuantLab against four agent-based baselines, but the computational budgets differ substantially. Table 5 reports that EvoQuantLab requires 22 hours and 8 minutes, whereas the baselines finish within approximately 19 minutes to 2 hours. EvoQuantLab also achieves the lowest reported throughput, at 0.95 L5 factors per hour.
Although the authors emphasize reduced LLM token consumption, this does not establish overall computational efficiency. The substantial additional runtime and genetic programming search effort are not normalized against the competing methods.
Moreover, the paper does not establish performance under matched total computational budgets or matched numbers of evaluated factor candidates. Without these controls, it is difficult to distinguish the benefit of the proposed framework from the effect of additional search and validation effort.
The comparison is further complicated by the downstream evaluation pipeline. Appendix C.2 states that the daily equity strategies employ a LightGBM model trained on selected factor pools. Therefore, the final performance reflects both the quality of discovered factors and the downstream supervised learning and selection procedures. Additional controls are needed to isolate the contribution of the proposed discovery algorithm.
- The central claims about overfitting prevention and realistic trading performance are not convincingly established.
The paper repeatedly positions its contribution as a solution to adaptive backtest overfitting. However, the experimental evidence is limited to relatively short OOS periods: approximately three months for the daily equity experiments and slightly over two months for cryptocurrency.
Such evaluation windows are insufficient to establish robust generalization across different market regimes, especially for a framework that makes strong claims about preventing overfitting and improving research reliability. The results are also largely reported as single performance estimates without sufficiently comprehensive repeated-run uncertainty analysis.
The development-to-OOS transfer analysis does not fully resolve this concern. The methods retain substantially different numbers of candidate factors, ranging from single-factor baselines to dozens of candidates for EvoQuantLab. Consequently, comparing sign-survival rates and winner performance across these heterogeneous candidate pools is not a controlled demonstration of reduced selection bias.
There is also an important inconsistency in the trading evaluation. The paper describes its experiments as execution-aware and incorporating transaction costs. However, Appendix C.2 explicitly states that the reported CSI500 and S&P500 performance reflects gross returns before execution fees and slippage. The cryptocurrency evaluation includes a fixed 5 bps trading cost but excludes perpetual funding costs.
These omissions weaken the paper's claims about executable and economically robust factor discovery. The evidence does not convincingly establish that the reported advantages would persist under realistic trading costs and implementation constraints.
Questions for authors
Can the authors provide matched-budget comparisons using identical total computational resources, wall-clock limits, and numbers of evaluated factor candidates? How much improvement remains under these controls?
Why does the manuscript characterize the daily equity backtests as execution-aware when Appendix C.2 explicitly reports gross returns before transaction costs? Can the authors provide net-of-cost results with turnover, fees, slippage, and liquidity constraints?
Can the authors provide repeated-seed evaluations and statistical confidence intervals for the main OOS performance results? How sensitive are the results to the selected test periods?
How much of the performance improvement is attributable specifically to the outer diagnosis loop rather than additional GP search, multi-stage filtering, or the downstream LightGBM pipeline? Can the authors provide a more controlled attribution analysis?
How are the different candidate-pool sizes and selection budgets accounted for when comparing development-to-OOS sign survival and selection-bias robustness?
6 comments
(2/3) Response by authors
W2-a, Q3: OOS window length and uncertainty.
Why ~70 days.
- Short-horizon signals. Our factors are short-horizon price-volume and order-flow signals. Comparable formulaic alphas hold positions for 0.6–6.4 days on average (Kakushadze, 2016), and published return predictors lose much of their efficacy over time (McLean & Pontiff, 2016), so such signals are re-discovered and re-validated on a rolling basis. A ~70-day frozen window spans dozens of independent holding periods.
- Statistical sufficiency (Crypto). By the Minimum Track Record Length criterion (Bailey & López de Prado, 2012), confirming an hourly IR of 3.709 at 95% confidence requires ~1,725 hourly bars, slightly more than our original window (1,680 bars). This motivated the forward extension below.
Forward extension (new). We evaluated the frozen, content-hashed artifacts on data after the original OOS windows, without re-running any search (CSI 500: 2026-08-18 to 09-30; S&P 500: 2026-08-17 to 09-30; Crypto: 2026-06-10 to 08-31, re-collected as a complete snapshot and checked for coverage before evaluation):
| Market | Original OOS IR | Extension-only IR | Combined OOS IR | Best baseline (combined) |
|---|---|---|---|---|
| CSI 500 | 1.613 | 1.587 | 1.602 | 1.224 (AlphaAgent) |
| S&P 500 | 1.614 | 1.488 | 1.567 | 1.356 (FactorMiner) |
| Crypto | 3.709 | 4.921 | 4.343 | 1.195 (FactorMiner) |
Over the combined crypto window (3,672 bars), the Probabilistic Sharpe Ratio reaches 0.998. For daily equities, establishing an IR of ~1.6 on a single series would require roughly a year of daily data, so our equity evidence rests on the paired comparison with baselines over the same period and on consistency across seeds.
Repeated campaigns (new). Three independent 11-cycle campaigns from scratch per method (OOS IR, mean ± std, n = 3):
| Method | Crypto | CSI 500 | S&P 500 |
|---|---|---|---|
| EvoQuantLab | 3.65 ± 0.19 | 1.65 ± 0.08 | 1.64 ± 0.07 |
| Best baseline | 0.93 ± 0.04 (FactorMiner) | 1.32 ± 0.04 (AlphaAgent) | 1.50 ± 0.07 (FactorMiner) |
EvoQuantLab ranks first in every seed on all three markets. We will add a Limitations section noting that multi-month OOS windows do not span multi-year macroeconomic cycles.
W2-b, Q5: Heterogeneous candidate pools in the transfer audit.
We agree that sign survival over pools of unequal size is descriptive rather than controlled, and we will present it as such in App. E. We add two controlled checks.
- Matched top-K audit. For each method we rank its own candidates by Dev IR only and report the mean OOS IR of the top-K, so that EvoQuantLab's larger pool cannot benefit from averaging over more factors. We compare against the strongest baseline of each market in the main results (Table 2 / Fig. 4):
| Market | Method (M) | Top-1 | Top-3 mean | Top-5 mean |
|---|---|---|---|---|
| Crypto | EvoQuantLab (12) | 4.055 | 4.120 | 3.075 |
| Crypto | FactorMiner (1) | 0.952 | – | – |
| CSI 500 | EvoQuantLab (36) | 1.041 | 1.725 | 1.642 |
| CSI 500 | AlphaAgent (4) | 0.798 | 1.216 | 1.328 (all 4) |
| S&P 500 | EvoQuantLab (49) | 2.466 | 1.928 | 1.802 |
| S&P 500 | FactorMiner (1) | 1.382 | – | – |
EvoQuantLab's top-3 and top-5 factors remain strong out of sample in all three markets, and it is higher than the strongest baseline at every K where the comparison is defined.
- Trial-budget-matched selection correction. Selection bias scales with the number of evaluated trials rather than the retained pool size . Eq. 2 compares each cycle's winner with block-permuted null searches under the same trial budget; in all 11 Crypto cycles the real winner exceeded every null maximum (, the smallest attainable value). We will also correct App. E: extreme-value noise grows as , not "exponentially".
References
- Bailey, D. H. and López de Prado, M. The Sharpe Ratio Efficient Frontier. Journal of Risk, 15(2):3–44, 2012.
- Kakushadze, Z. 101 Formulaic Alphas. Wilmott, 2016(84):72–81, 2016.
- McLean, R. D. and Pontiff, J. Does Academic Research Destroy Stock Return Predictability? The Journal of Finance, 71(1):5–32, 2016.
(3/3) Response by authors
W2-c, Q2: Costs, funding, and execution conventions.
We thank the reviewer for identifying this inconsistency. The error is in the description, not in the reported results: our Qlib evaluation configs confirm that the "gross, dollar-neutral" wording in Sec. 4.1, Eq. 11, the IR definition (App. C.1) and App. C.2 came from an early prototype and does not describe the protocol used for any reported number. We will correct all four.
Equities (Tables 2, 3 and 8). All equity results use the long-only TopkDropout(50, 5) protocol of AlphaAgent (top-50 stocks, at most 5 replacements per day), measured as excess return over the equal-weighted universe.
- Costs. 5 bps buy / 15 bps sell on CSI 500 (including stamp duty); 0 bps buy / 5 bps sell on S&P 500, approximating zero-commission buys and regulatory/transaction fees on sales.
- Execution and liquidity. Orders are filled at the daily close under Qlib's standard
SimulatorExecutor(aligned with next-day forward returns), which blocks buys at limit-up, sells at limit-down, and trades on suspended or zero-volume days. Being long-only, the portfolio respects A-share short-sale constraints. - Slippage. We do not model slippage separately. The cost sensitivity reported below for crypto (break-even ≈ 62 bps one-way) indicates the margin available to absorb it.
All reported equity figures (e.g., CSI 500: AR 18.34% / IR 1.613; S&P 500: AR 21.30% / IR 1.614) are therefore net of transaction costs.
Crypto (17-factor equal-weight portfolio, frozen OOS window).
- Execution lag and turnover. Signals formed at hour are filled at the close of bar , so the price move between signal formation and execution is not earned. Each bar's one-way turnover is , where is the previous hour's target weight (without intraday drift re-weighting), and costs are charged on at every bar. For readability, we report the sum over the 24 bars of each day averaged across OOS days: 0.150 / day ( per year; this aggregation is for reporting only and does not affect PnL). Turnover stays low because of L4 turnover gating and 72-hour rolling position smoothing.
- Perpetual funding. Historical funding rates are available for 67% of notional exposure (covering major perpetual pairs with complete 8-hour funding snapshots; the remaining 33% correspond to assets lacking archived funding-rate history). Their net effect is +0.14%/yr, because dollar neutrality cancels the market-wide funding component, giving IR 3.725 / AR 38.05% net of fees and funding. Under an adversarial assumption for the uncovered 33% (longs pay the 90th-percentile rate, shorts receive the 10th), the additional drag is at most −3.40%/yr (AR ≈ 34.6%).
- Cost sensitivity. Re-evaluating the same portfolio at {0, 5, 10, 20} bps one-way cost gives OOS IR 4.036 / 3.709 / 3.395 / 2.734 (AR 41.16% / 37.91% / 34.53% / 27.92%). The extrapolated break-even cost is ≈ 62 bps one-way (≈ 12.4× the 5 bps assumed in the paper).
(1/3) Response by authors
We agree that LLM-token savings do not establish overall efficiency and will narrow Sec. 5.3: our claim is only that LLM cost does not scale with search breadth (~2,560 candidates with 50 LLM calls). Three clarifications on Table 5:
Matched candidate budget (Crypto, identical L0–L6 gates, ~2,560 evaluations):
No Loop B)Conventional GP reaches ~37% of the full system's IR, and adding LLM proposals without diagnosis lowers it further. This indicates that the gain comes from the outer loop rather than from additional search.
Matched wall-clock (Crypto, cumulative L5 portfolio OOS IR):
(Note: This trajectory tracks our primary benchmark campaign ending at 3.709; Fig. 5 wrongly plotted an earlier evaluation script with part of missing exchange records, which we will align in the revision.)
At a comparable wall-clock budget (~2.75h), EvoQuantLab already exceeds every baseline's full-budget result on Crypto.
Without LightGBM, EvoQuantLab's factor pools still outperform the best baseline under the same parameter-free combination on both markets. The advantage therefore comes from the discovered factors rather than from the downstream model.