- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FactorBench evaluates 4,914 factors generated by nine mining systems across five equity markets, with three runs per method–market pair. Its common evaluation pipeline follows factors through executability, temporal generalization, exposure overlap, pool distinctness, and after-cost portfolio performance. The principal finding is that newer mining paradigms do not consistently outperform earlier approaches across this evaluation chain.
Critical strengths
-
The benchmark connects discovery performance to economically meaningful outcomes. Training-fixed factor orientation preserves out-of-sample sign reversals, while exposure neutralization, pool comparisons, and portfolio evaluation distinguish several sources of apparent progress. These distinctions materially change interpretation: R&D-Agent’s composite IC decreases from 0.0253 to 0.0011 after neutralization, whereas AlphaQCM retains 0.0175 of its original 0.0191. The evaluation therefore provides insight beyond a predictive-score leaderboard.
-
The experimental coverage is substantial and the conclusions are appropriately restrained. The paper includes 135 mining runs, market- and seed-level results, alternative similarity measures, and sensitivity to transaction costs, rebalancing, and signal combination. It explicitly limits its conclusions to the evaluated implementations and settings. This supports its central observation that no method consistently dominates without claiming a universal ranking of paradigms.
Critical weaknesses
-
Portfolio eligibility appears to depend on future test-period availability.
- Section 3.3 and Appendix C.1 require at least 100 scorable dates in every split, including testing. Appendix E then starts portfolio selection from factors satisfying these common validity and coverage requirements. Consequently, future missingness or degeneracy can influence the pool available before testing.
- A common cohort is reasonable for paired temporal diagnostics, but prospective portfolio evaluation needs eligibility determined entirely from training and validation, together with a predefined test-time failure policy. This is not demonstrated return-label leakage, and the economic effect may be small; however, the benchmark should establish that its conclusions survive a prospective eligibility rule.
-
Important details of the common evaluation contract remain insufficiently specified.
- The target uses overlapping 20-day forward returns, but the manuscript does not explain boundary purging or temporal buffers, despite referring readers to Appendix A for them. Signal formation and execution timing also need clarification. For generated Python factors, the described execution contract does not establish how future input observations are prevented from influencing earlier outputs.
- The portfolio specification additionally omits the ridge preprocessing and regularization search details and the numerical short-financing assumption. Released configurations may resolve these omissions, but these are central auditability requirements for a benchmark comparing heterogeneous executable systems.
Questions for authors
- Does portfolio construction use the test-conditioned scorability mask? How do the results change when eligibility depends exclusively on training and validation?
- How are overlapping forward-return labels handled at split boundaries, and when are signals formed relative to trade execution?
- What causal checks prevent generated Python factors from accessing future observations—for example, prefix-invariance tests?
- Please identify the exact ridge preprocessing, missing-value policy, regularization search, and short-borrow assumptions.
2 comments
Response to Reviewer 9Q3v
Q4. Ridge preprocessing, missing values, regularization, and short borrowing
These are now specified in Appendix E:
- Preprocessing. On each date, every selected factor is converted to its cross-sectional percentile rank among index members minus 0.5 and multiplied by its training-fixed sign, so all inputs are bounded, centred, and consistently oriented.
- Ridge fit. The 20-day forward return is regressed on the transformed factors over all stock–date observations of the training period, with an intercept and no sign constraint on the weights; only observations on which every selected factor is available enter the fit. The penalty is chosen from {0.01, 0.1, 1, 10, 100, 1000} by the mean daily cross-sectional correlation between the fitted composite and realized returns in the validation period, and the weights are rescaled to unit L1 norm. If fewer than 50 complete training or validation observations are available, ridge falls back to equal weights.
- Missing values in the composite. A factor that is missing for a stock on a date contributes nothing to that stock's score; the composite is undefined only if every selected factor is missing.
- Costs and financing. Trading costs are 10 basis points per dollar traded, with turnover measured against the price-drifted portfolio so that drift is not billed as trading. Long–short portfolios pay a short-financing cost of 50 basis points per year on the short notional, with no rebate.
Response to Reviewer 9Q3v
We thank the reviewer for the detailed and constructive review, and for recognizing that training-fixed orientation, exposure neutralization, and portfolio evaluation change the interpretation of apparent progress. Each of the reviewer's auditability concerns has led to a concrete change in the revision, which also reruns the full benchmark on point-in-time index membership. Please see our revised manuscript.
Q1. Portfolio eligibility and the test-conditioned scorability mask The reviewer is right: in the submission, portfolio eligibility inherited the factor-level requirement of 100 scorable dates in every split, including testing. The revision separates the two rules:
Effect of the change. We isolated it by running the same point-in-time data under both rules. The prospective rule changes the selected pools of only four of the ten methods (Alpha101, AlphaForge, AlphaSAGE, and QuantaAlpha), adding on average fewer than one factor per pool. Under the default ridge combination, long-only Sharpe ratios change by at most 0.02 and long–short Sharpe ratios by at most 0.01 for the mined methods; the largest change is Alpha101's long–short Sharpe ratio, from −0.33 to −0.41. The other six methods are unaffected. All conclusions survive the prospective rule, and all reported portfolio results use it.
Q2. Label overlap at split boundaries and execution timing
Purging. Training runs from 1 January 2017 to 30 November 2021, validation from 1 January 2022 to 29 November 2023, and testing from 1 January 2024 to 31 October 2025. Training and validation end early so that the 20-day forward-return label of their last date is realized before the next period begins; the trading days in between are unused, and testing ends in October 2025 so that every test date has a complete label (Section 3.2, Appendix A). Because the horizon is measured on each market's own trading calendar, an automated test verifies for every market that the label of the last training and validation date ends before the following period starts.
Execution timing. A signal is formed from data up to the close of day t, the portfolio is traded on day t+1, and the first return it earns is from the close of t+1 onward, so a signal never earns a return that overlaps the information used to form it (Section 3.3). We verified this convention directly: a synthetic signal equal to the next day's return earns no excess performance in the backtest, while one equal to the return two days ahead, the first return a position can earn, is fully captured.
Q3. Causal checks for generated Python factors
The revision adds exactly such a prefix-invariance test, applied to every delivered factor, including all 273 Python programs generated by R&D-Agent (Appendix C.2):
Results. Seven of the 7,087 mined factors are flagged, including 3 of the 273 R&D-Agent programs, and none of the 71 Alpha101 formulas; every flag reproduces at all three cuts (Appendix C.2, Table 7). Consistent with evaluating methods as released, flagged factors are reported as causality failures rather than repaired. The test detects dependence on later market data under the tested perturbations and cut dates.