acceptodds
Under review as a conference paper at ICLR 2027

FactorBench: A Portfolio-Aware Benchmark for Automated Factor Mining

14 views

Abstract

Factor mining seeks to discover signals from financial data that predict future asset returns and guide portfolio construction. Automated factor mining now spans genetic programming, reinforcement learning, generative models, and large language model agents. Yet it remains unclear whether advances across these paradigms yield more generalizable, distinct, and economically useful financial signals. We introduce FactorBench, a portfolio-aware benchmark comparing roughly five thousand mined factors from nine automated mining methods across five equity markets. A shared data and evaluation contract supports both symbolic expressions and executable Python factors, connecting heterogeneous discovery algorithms to common signal combination and portfolio construction procedures. FactorBench traces the outputs of mining systems across three levels: factor validity, temporal generalization, and predictiveness beyond measured risk and style exposures; within- and across-method pool distinctness, including similarity to the benchmark Alpha101; and composite-signal quality and after-cost long-only and long–short portfolio performance. After systematically assessing whether advances in factor mining translate into signal quality and portfolio performance, FactorBench finds that no paradigm consistently dominates.

Primary area:
datasets and benchmarks
Keywords:
Factor Mining, Financial Machine Learning, AI in Finance, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9Q3vOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

FactorBench evaluates 4,914 factors generated by nine mining systems across five equity markets, with three runs per method–market pair. Its common evaluation pipeline follows factors through executability, temporal generalization, exposure overlap, pool distinctness, and after-cost portfolio performance. The principal finding is that newer mining paradigms do not consistently outperform earlier approaches across this evaluation chain.

Critical strengths
  • The benchmark connects discovery performance to economically meaningful outcomes. Training-fixed factor orientation preserves out-of-sample sign reversals, while exposure neutralization, pool comparisons, and portfolio evaluation distinguish several sources of apparent progress. These distinctions materially change interpretation: R&D-Agent’s composite IC decreases from 0.0253 to 0.0011 after neutralization, whereas AlphaQCM retains 0.0175 of its original 0.0191. The evaluation therefore provides insight beyond a predictive-score leaderboard.

  • The experimental coverage is substantial and the conclusions are appropriately restrained. The paper includes 135 mining runs, market- and seed-level results, alternative similarity measures, and sensitivity to transaction costs, rebalancing, and signal combination. It explicitly limits its conclusions to the evaluated implementations and settings. This supports its central observation that no method consistently dominates without claiming a universal ranking of paradigms.

Critical weaknesses
  • Portfolio eligibility appears to depend on future test-period availability.

    • Section 3.3 and Appendix C.1 require at least 100 scorable dates in every split, including testing. Appendix E then starts portfolio selection from factors satisfying these common validity and coverage requirements. Consequently, future missingness or degeneracy can influence the pool available before testing.
    • A common cohort is reasonable for paired temporal diagnostics, but prospective portfolio evaluation needs eligibility determined entirely from training and validation, together with a predefined test-time failure policy. This is not demonstrated return-label leakage, and the economic effect may be small; however, the benchmark should establish that its conclusions survive a prospective eligibility rule.
  • Important details of the common evaluation contract remain insufficiently specified.

    • The target uses overlapping 20-day forward returns, but the manuscript does not explain boundary purging or temporal buffers, despite referring readers to Appendix A for them. Signal formation and execution timing also need clarification. For generated Python factors, the described execution contract does not establish how future input observations are prevented from influencing earlier outputs.
    • The portfolio specification additionally omits the ridge preprocessing and regularization search details and the numerical short-financing assumption. Released configurations may resolve these omissions, but these are central auditability requirements for a benchmark comparing heterogeneous executable systems.
Questions for authors
  • Does portfolio construction use the test-conditioned scorability mask? How do the results change when eligibility depends exclusively on training and validation?
  • How are overlapping forward-return labels handled at split boundaries, and when are signals formed relative to trade execution?
  • What causal checks prevent generated Python factors from accessing future observations—for example, prefix-invariance tests?
  • Please identify the exact ridge preprocessing, missing-value policy, regularization search, and short-borrow assumptions.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 9Q3v

We thank the reviewer for the detailed and constructive review, and for recognizing that training-fixed orientation, exposure neutralization, and portfolio evaluation change the interpretation of apparent progress. Each of the reviewer's auditability concerns has led to a concrete change in the revision, which also reruns the full benchmark on point-in-time index membership. Please see our revised manuscript.

Q1. Portfolio eligibility and the test-conditioned scorability mask The reviewer is right: in the submission, portfolio eligibility inherited the factor-level requirement of 100 scorable dates in every split, including testing. The revision separates the two rules:

  • Factor-level reporting keeps the common cohort with coverage in all three periods, because paired train-to-test statistics require a test IC for every factor (Appendix C.1).
  • Portfolio eligibility now requires at least 100 scorable dates in the training and validation periods only, a finite validation IC, and successful execution; test-period coverage plays no role (Section 3.3, Appendix E).
  • Test-time failure policy. A selected factor that is missing for a stock on a test date, or degenerates during the test period, simply contributes nothing to that stock's composite score; the composite is formed from the remaining selected factors and is undefined only if all of them are missing (Appendix E).

Effect of the change. We isolated it by running the same point-in-time data under both rules. The prospective rule changes the selected pools of only four of the ten methods (Alpha101, AlphaForge, AlphaSAGE, and QuantaAlpha), adding on average fewer than one factor per pool. Under the default ridge combination, long-only Sharpe ratios change by at most 0.02 and long–short Sharpe ratios by at most 0.01 for the mined methods; the largest change is Alpha101's long–short Sharpe ratio, from −0.33 to −0.41. The other six methods are unaffected. All conclusions survive the prospective rule, and all reported portfolio results use it.

Q2. Label overlap at split boundaries and execution timing

Purging. Training runs from 1 January 2017 to 30 November 2021, validation from 1 January 2022 to 29 November 2023, and testing from 1 January 2024 to 31 October 2025. Training and validation end early so that the 20-day forward-return label of their last date is realized before the next period begins; the trading days in between are unused, and testing ends in October 2025 so that every test date has a complete label (Section 3.2, Appendix A). Because the horizon is measured on each market's own trading calendar, an automated test verifies for every market that the label of the last training and validation date ends before the following period starts.

Execution timing. A signal is formed from data up to the close of day t, the portfolio is traded on day t+1, and the first return it earns is from the close of t+1 onward, so a signal never earns a return that overlaps the information used to form it (Section 3.3). We verified this convention directly: a synthetic signal equal to the next day's return earns no excess performance in the backtest, while one equal to the return two days ahead, the first return a position can earn, is fully captured.

Q3. Causal checks for generated Python factors

The revision adds exactly such a prefix-invariance test, applied to every delivered factor, including all 273 Python programs generated by R&D-Agent (Appendix C.2):

  • For each market, we choose one cut date in each period (28 June 2019, 30 December 2022, 31 December 2024). After the cut, open, high, low, close, and volume are multiplied by independent noise drawn uniformly from [0.5, 1.5), derived fields are recomputed, and all rows up to the cut are verified to be unchanged.
  • Each factor is executed on the original and the perturbed panel in separate processes, and its values on the 60 trading days before the cut must be identical up to floating-point tolerance. This detects forward shifts, centred windows, back-fills, and normalizations over the whole history.

Results. Seven of the 7,087 mined factors are flagged, including 3 of the 273 R&D-Agent programs, and none of the 71 Alpha101 formulas; every flag reproduces at all three cuts (Appendix C.2, Table 7). Consistent with evaluating methods as released, flagged factors are reported as causality failures rather than repaired. The test detects dependence on later market data under the tested perturbations and cut dates.

AuthorsOpenReview ↗

Response to Reviewer 9Q3v

Q4. Ridge preprocessing, missing values, regularization, and short borrowing

These are now specified in Appendix E:

  • Preprocessing. On each date, every selected factor is converted to its cross-sectional percentile rank among index members minus 0.5 and multiplied by its training-fixed sign, so all inputs are bounded, centred, and consistently oriented.
  • Ridge fit. The 20-day forward return is regressed on the transformed factors over all stock–date observations of the training period, with an intercept and no sign constraint on the weights; only observations on which every selected factor is available enter the fit. The penalty is chosen from {0.01, 0.1, 1, 10, 100, 1000} by the mean daily cross-sectional correlation between the fitted composite and realized returns in the validation period, and the weights are rescaled to unit L1 norm. If fewer than 50 complete training or validation observations are available, ridge falls back to equal weights.
  • Missing values in the composite. A factor that is missing for a stock on a date contributes nothing to that stock's score; the composite is undefined only if every selected factor is missing.
  • Costs and financing. Trading costs are 10 basis points per dollar traded, with turnover measured against the price-drifted portfolio so that drift is not billed as trading. Long–short portfolios pay a short-financing cost of 50 basis points per year on the short notional, with no rebate.
Reviewer_HT8kOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FactorBench, an open benchmark standardizing the evaluation of automated alpha factor discovery systems across five major equity markets: S&P 100, Hang Seng, CSI 300, FTSE 100, and Nikkei 225 from 2017 to 2025. Implementing a shared execution contract to assess approximately 5,000 mined factors from nine systems spanning genetic programming, reinforcement learning, generative models, and LLM-based agents, the benchmark evaluates three progressive tiers: factor-level validity and style neutralization, pool-level redundancy against Alpha101, and cost-adjusted long-only and long-short portfolio performance. Empirical results reveal that complex mining paradigms do not reliably improve downstream returns: once controlled for style exposures and trading frictions, LLM agents and generative methods degrade substantially, failing to outperform classical formula libraries or reinforcement learning baselines.

Critical strengths
  1. The three-tier evaluation framework connects factor-level predictive validity and temporal generalization, pool-level redundancy, and downstream portfolio performance under a shared evaluation protocol. By incorporating transaction costs and short-borrow costs and examining performance after neutralizing five measured style exposures, it provides a more comprehensive assessment of automated factor mining than predictive-score-based evaluation alone.
  2. The evaluation spans five equity markets and 135 method–market–run combinations, covering 4,914 factor instances delivered by nine automated mining methods, alongside 71 Alpha101 reference formulas. Results show that no paradigm consistently dominates across factor validity, pool distinctness, style-neutralized predictiveness, and cost-adjusted portfolio returns. Under the default portfolio setup, most automated methods fail to maintain positive net long–short returns after neutralizing five style exposures, demonstrating that algorithmic complexity does not necessarily translate into genuine economic value.
Critical weaknesses
  1. The benchmark retrospective evaluation applies fixed contemporary constituent baskets across 2016–2025 without reconstructing point-in-time index memberships, leaving long-only portfolio returns susceptible to survivorship and universe-selection biases. Because the prediction target spans 20 trading days, the absence of documented purging or embargo protocols at temporal split boundaries prevents independent verification that target windows do not cross into out-of-sample data.
  2. Across the evaluated systems, wall-clock runtimes vary from 0.14 hours for GP to 27.9 hours for QuantaAlpha alongside a fifteenfold difference in delivered factor counts, while specific agent implementations rely on next-day feedback rather than directly targeting the shared 20-day horizon. While native evaluations reflect practical off-the-shelf utility, the lack of compute-normalized curves and aligned search objectives limits the degree to which downstream performance gaps can be attributed to intrinsic algorithmic paradigms rather than resource or objective disparities.
  3. Although wealth curves plot baseline references, the primary performance tables omit matched benchmark-relative excess returns, obscuring how much long-only performance derives from stock selection versus broad market movement. In addition, dispersion across three random seeds fails to capture temporal dependence or shifting market regimes, leaving cross-method comparisons without dependence-aware inference, paired confidence intervals, or formal accounting for multiple-testing effects.
Questions for authors
  1. Given the 20-trading-day forward target, what explicit boundary-exclusion or purging protocols were applied to prevent forward-return windows from overlapping across training, validation, and test splits?
  2. Does the evaluation rely on point-in-time index constituents or retrospective contemporary baskets, and to what extent might resulting survivorship or selection biases alter the reported long-only portfolio returns?
  3. How stable are the relative rankings across discovery paradigms when evaluated under comparable computational and candidate budgets, and with internal search objectives aligned to the 20-day horizon?
  4. Could matched benchmark-relative excess returns (against equal-weighted universes and market total-return indices) be tabulated in the main results, alongside dependence-aware confidence intervals addressing multiple-testing effects?
2 comments
AuthorsOpenReview ↗

Response to Reviewer HT8k

We thank the reviewer for the clear summary and for recognizing the value of connecting factor validity, pool redundancy, and cost-adjusted portfolio performance under one protocol. The reviewer's concerns about survivorship, label overlap, horizon alignment, and statistical inference have all led to changes in the revision, including a full rerun of the benchmark on point-in-time index membership. We address each point below; please see our revised manuscript.

W1 / Q1. Purging at split boundaries

The splits are now explicitly purged. Training runs from 1 January 2017 to 30 November 2021, validation from 1 January 2022 to 29 November 2023, and testing from 1 January 2024 to 31 October 2025. Training and validation end early so that the 20-day forward-return label of their last date is realized before the next period begins; the trading days in between are unused. Testing ends in October 2025 so that every test date has a complete label (Section 3.2, Appendix A). Because the horizon is measured on each market's own trading calendar, the boundaries are verified per market by an automated test in the released code, which checks that the label of the last training and validation date in every market ends before the following period starts. No additional embargo is needed, since information flows only forward from one period to the next.

W1 / Q2. Point-in-time constituents and survivorship

We agree that this was the most important limitation of the submission, and we have rerun the full benchmark, including all 135 mining runs and every evaluation stage, on point-in-time universes:

  • Membership. We reconstruct daily index membership from 31 December 2016 to 31 October 2025 for all five indices from index-provider announcements, archived constituent lists, and weekly constituent snapshots (Appendix A, Table 4). Stocks later removed or delisted are included while they were members.
  • Mining sees only rows on which a stock is an index member, and stocks that first joined an index after the validation period are excluded from mining entirely, so test-period membership cannot inform discovery.
  • Evaluation and portfolios score, select, and trade a stock only while it is a member, and the equal-weight benchmark holds that day's members.

Effect on the results. Survivorship had inflated the submitted portfolio results. Long-only Sharpe ratios fall from 1.03–1.31 to 0.90–1.09, and long–short Sharpe ratios from −0.09–0.68 to −0.41–0.19 (Table 2). The qualitative conclusions are unchanged and, with the new benchmark-relative tests, stronger: pooled across markets, no method's return establishes an advantage over the equal-weight universe (Table 3).

W2 / Q3. Compute, candidate budgets, and objective alignment

Equal budgets are a deliberate non-goal of FactorBench. The benchmark is designed to test each method faithfully, as its authors released it, and at the operating point its authors intended: every method runs from its official implementation with its released configuration, search procedure, and budget, so that each is evaluated at the strength its authors designed it for. Equalizing compute or candidate counts would require shrinking some methods' budgets and inflating others' away from their published settings. The benchmark would then test benchmark-specific variants rather than the methods themselves, and would risk understating methods whose performance depends on their intended scale. What FactorBench equalizes instead is everything outside the search: markets, point-in-time universes, input fields, the 20-day target, purged splits, execution, and all downstream evaluation. The revision states this scope explicitly: conclusions apply to the evaluated implementations and configurations, not to intrinsic paradigm capabilities, and we make no claim about computational efficiency (Section 5).

Resulting budgets. Under this design, budgets differ substantially, and we report them rather than normalize them: in the rerun, mean wall-clock time ranges from 0.02 hours per run for GP to 43.8 hours for QuantaAlpha, and delivered pools from 123 to 3,605 factors; 13 of the 15 QuantaAlpha runs reach the 48-hour limit (Section 4.1). Two observations indicate that the results are not simply driven by these differences:

  1. Larger budgets do not translate into better outcomes. QuantaAlpha has the largest budget and pool, and AlphaForge the second-largest pool, yet neither leads at the portfolio level, while AlphaQCM, with 123 factors, has the highest long-only Sharpe ratio (Table 2).
  2. Rankings do not follow paradigm boundaries. Methods within the same paradigm differ as much as methods across paradigms, for example AlphaGen and AlphaCFG in long–short Sharpe ratio (0.19 vs. −0.18).

A compute-matched study would answer a complementary question about efficiency, and we list it as future work.

AuthorsOpenReview ↗

Response to Reviewer HT8k

W3 / Q4. Benchmark-relative returns and dependence-aware inference

This is now Table 3 in the main text:

  • Benchmark-relative returns. Long-only performance is reported as the annualized net return in excess of an equal-weight portfolio of the same day's index members. This benchmark is matched to the strategies in both constituents and return basis, since it is built from dividend-adjusted prices. Long–short portfolios are dollar-neutral and are tested against zero.
  • Dependence-aware intervals. 95% intervals come from a circular block bootstrap with 20-day blocks, equal to the rebalancing interval. Each draw resamples the same calendar blocks in all five markets, preserving both serial dependence and cross-market dependence during the common test period. Estimates are pooled exactly as in Table 2 (Appendix E.2).
  • Multiple testing. p-values are adjusted with the Holm procedure across the ten methods.

Result. Pooled across markets, no method's long-only return establishes an advantage over the equal-weight universe; the largest estimate, AlphaQCM's 4.3% per year, has an interval of [−1.9%, 10.1%]. No long–short Sharpe ratio is distinguishable from zero after Holm adjustment. We note that intervals containing zero do not establish that methods are equivalent to the benchmark; they show that the evidence does not support an advantage. Market-level intervals, which are larger in both directions, are reported in Appendix E.2 without multiple-testing adjustment.

Reviewer_Zv5mOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FactorBench, a benchmark for automated factor mining. It compares nine methods across five equity markets and evaluates them at the factor, pool, and portfolio levels. The main conclusion is that newer mining paradigms do not consistently lead to better out-of-sample signals or portfolio performance.

The problem is relevant and the experimental effort is substantial. However, I have concerns about whether the benchmark design supports the paper's main cross-paradigm conclusion.

Critical strengths

1.The paper studies an important evaluation problem in automated factor mining.

2.The factor-to-pool-to-portfolio evaluation pipeline is useful and more informative than reporting IC alone.

3.The experiments cover multiple markets and a broad set of recent methods.

4.The exposure-neutralization and long-short results provide useful analysis of where apparent performance com

Critical weaknesses

1. The cross-paradigm comparison is not well controlled

The compared methods differ in many dimensions beyond the mining algorithm itself, including search space, search budget, number of generated factors, pool size, validation usage, and compute. The paper explicitly states that candidate evaluations and compute are not equalized.

Therefore, it is difficult to interpret the results as evidence about GP, RL, generative, or LLM-based paradigms themselves. At most, the experiments compare several specific systems under their respective settings.

This is important because the main conclusion is framed at the paradigm level: newer mining approaches do not consistently translate into better downstream performance. I do not think the current design is controlled enough to support that conclusion.

2. Some methods are substantially modified for the benchmark

Several methods are not evaluated exactly as originally proposed. For example, AlphaForge uses a different prediction target, its native combination stage is removed, and an additional stopping rule is introduced. AlphaSAGE also does not use its optional adaptive combination stage.

These modifications may be necessary for standardization, but they make it unclear whether the benchmark is comparing the original methods or benchmark-specific variants.

3. The portfolio evaluation uses a static stock universe

The same constituent set is used throughout the full historical period, rather than using point-in-time index membership. The paper acknowledges that this may introduce survivorship bias.

This is a significant issue for a portfolio-aware benchmark because future index membership can affect both factor discovery and backtest performance. A point-in-time universe would be needed to support stronger claims about economic usefulness.

Questions for authors

1.How should the cross-paradigm results be interpreted given that the methods use different search spaces, search budgets, output sizes, validation feedback, and compute? What evidence supports attributing the observed differences to the mining paradigms rather than to these system-level differences?

2.Several methods are modified to fit the common benchmark protocol. How sensitive are the conclusions to these adaptations, especially for methods whose native targets or combination procedures are changed or removed?

3.Have you evaluated the benchmark using point-in-time index constituents? Since the current experiments use a fixed universe throughout the sample, how much could survivorship bias affect the factor and portfolio-level conclusions?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Zv5m

We thank the reviewer for the careful reading and for recognizing the value of the factor-to-pool-to-portfolio pipeline and of the exposure-neutralization and long–short analyses. The most substantive concern, the static universe, has been addressed by rerunning the entire benchmark on point-in-time index membership. We respond to each point below; please also see our revised manuscript.

W1 / Q1. Interpreting cross-paradigm results when systems differ in budget, search space and validation use.

We agree that FactorBench compares systems, not paradigms in isolation, and the paper states this explicitly (§5, “How far do these conclusions generalize?”). Our design deliberately runs each method with its authors’ released configuration, so every system operates as intended rather than as a reduced variant. What we hold fixed is everything outside the search: the same markets, point-in-time universes, input fields, 20-day target, purged chronological splits, execution contract and evaluation.

Our claim is therefore not that one paradigm is intrinsically better or worse. It is that among representative, state-of-the-art systems run as released, newer paradigms do not consistently deliver better downstream performances. Establishing this pattern does not require identifying which individual system characteristics explain it.

W2 / Q2. Method modifications and sensitivity to them

These adaptations are deliberate consequences of FactorBench’s evaluation design. Our aim is to assess the factor pools delivered by heterogeneous discovery systems under a shared financial task and common downstream procedures. Accordingly, we preserve method-specific discovery mechanisms where possible while standardizing the prediction target and the subsequent selection, combination, and portfolio evaluation. Without this standardization, differences in prediction targets and downstream procedures would confound comparisons at the factor, pool, and portfolio levels.

All adaptations are documented in Appendix B.2, and each is limited to what the shared protocol requires:

  • AlphaForge's target is changed from its delayed VWAP-to-VWAP return to the common 20-day close-to-close return. This is the quantity on which every method is evaluated; optimizing a different target would disadvantage AlphaForge rather than favor it. The native targets of AlphaGen and AlphaSAGE already coincide with the benchmark target, and the LLM agents' feedback is computed on it.
  • AlphaForge's stopping rule. The released mining loop has no termination condition when the factor zoo cannot reach its target size. We impose a maximum of 50 rounds so that every run terminates; runs that reach the target size are unaffected.
  • Native combination stages of AlphaForge and AlphaSAGE are not used. FactorBench evaluates the delivered factor pools under one shared selection and combination procedure, so that combination is held constant across methods; allowing each method its own combiner would confound discovery quality with combination quality. AlphaForge's per-day combination weights also cannot be expressed as a fixed factor pool.

Because combination is the principal adaptation, the paper varies the combination rule directly, comparing equal weighting, validation-IC weighting, and ridge regression (Appendix E.4). No single rule dominates for every method, and several long–short outcomes change sign across rules, but the absence of a consistent ordering across paradigms holds under each rule.

W3 / Q3. Static universe and survivorship bias

We agree that this was the most important limitation of the submission, and we have rerun the full benchmark, including all 135 mining runs and every evaluation stage, on point-in-time universes:

  • Membership. We reconstruct daily index membership from 31 December 2016 to 31 October 2025 for all five indices from index-provider announcements, archived constituent lists, and weekly constituent snapshots (Appendix A, Table 4). Stocks later removed or delisted are included while they were members.
  • Mining sees only rows on which a stock is an index member, and stocks that first joined an index after the validation period are excluded from mining entirely, so test-period membership cannot inform discovery.
  • Evaluation and portfolios score, select, and trade a stock only while it is a member, and the equal-weight benchmark holds that day's members.

Effect on the results. Survivorship had inflated the submitted portfolio results. Long-only Sharpe ratios fall from 1.03–1.31 to 0.90–1.09, and long–short Sharpe ratios from −0.09–0.68 to −0.41–0.19 (Table 2). The qualitative conclusions are unchanged and, with the new benchmark-relative tests, stronger: pooled across markets, no method's return establishes an advantage over the equal-weight universe (Table 3).

Thank you for the thoughtful response and the additional experiments. I appreciate the authors’ efforts to address my concerns. These revisions are helpful, but some concerns regarding the comparisons and method adaptations remain. So I will maintain my rating.

AuthorsOpenReview ↗

Reply to Zv5m

Thank you for reading our response and for acknowledging the revisions. We would like to point out that these two remaining concerns pull in opposite directions, and no benchmark can fully satisfy both:

  • There is no neutral common budget. The methods spend effort in incommensurable units: PPO steps, MCTS games, generator epochs, evolutionary rounds, LLM calls. Wall-clock time depends on hardware and API latency, and candidate counts mean different things for a GP that scores thousands of expressions internally and an agent that proposes a few. Equalizing any one of these unequalizes the others, and the result would depend on which arbitrary unit was chosen.

  • Any equalization is itself an adaptation. Matching budgets requires truncating some searches, extending others, and retuning their configurations, which turns each method into a benchmark-specific variant: exactly the reviewer's second concern, in stronger form.

FactorBench therefore resolves this trade-off explicitly: it runs each method as released, limits adaptations to the minimum any shared comparison needs (a common target, a terminating loop, a common combination stage; Appendix B.2), reports every budget transparently, and scopes its conclusions to the evaluated systems (Section 5). Our central finding is also robust to this choice: no system, including those with the largest budgets, establishes an advantage over an equal-weight portfolio of the same stocks (Table 3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.