acceptodds
Under review as a conference paper at ICLR 2027

Can LLMs Design Strategies for Investor Goals? Goal-Oriented Quantitative Strategy Generation

Abstract

Developing quantitative strategies requires searching over factors, trading rules, and parameters through iterative backtesting and refinement to meet investment objectives. LLMs could assist this process using financial knowledge and backtesting feedback, but their ability to find strategies satisfying specified goals remains insufficiently evaluated. To address this gap, we introduce AlphaGoal, a benchmark for goal-directed strategy search with limited backtest evaluations. It provides 175 validated atomic strategies and 2,221 strategy-construction tasks, each specifying an investment goal in natural language. Grounded in up to 11.5 years of daily data across 5,941 stocks, these tasks cover strategy generation and parameter optimization. Each goal is empirically reachable, with success verified programmatically under a unified historical backtesting protocol. We also introduce AlphaLoop, a reference baseline combining factor information and backtesting feedback to construct and refine strategies. Across six LLMs, AlphaLoop improves overall pass rates over Vanilla, while hard strategy-generation tasks remain challenging.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Quantitative Strategy Generation, Goal-Directed Search, Backtesting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZtZNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AlphaGoal, a benchmark for goal-directed quantitative strategy search under a limited backtest budget. It contains 175 validated atomic strategies and 2,221 strategy-construction tasks derived from historical backtests over up to 11.5 years of daily data across 5,941 stocks, with 1,360 strategy-generation and 861 parameter-optimization tasks. Each task specifies an investment goal in natural language, and success is verified programmatically by a unified backtesting engine rather than LLM judgment. The paper also proposes AlphaLoop, an LLM agent that combines historical IC, atomic-strategy documentation, and backtest feedback to construct and refine strategies. Across six LLMs, AlphaLoop improves overall pass rates over a Vanilla baseline, reaching 47.37% overall on GPT-5.6-Sol, while hard strategy-generation tasks remain challenging.

Critical strengths
  1. The benchmark is carefully designed to measure goal-satisfaction search ability rather than code executability or aggregate performance, using a unified engine, programmatic success verification, natural-language task generation with peer review, and hidden reference feasibility witnesses. This is a meaningful advance in how LLM quant-strategy tasks can be evaluated.
  2. The paper provides unusually complete documentation of the strategy space, backtest protocol, transaction costs, metric definitions, and limitations, including an explicit statement that the evaluation measures retrospective goal satisfaction and search efficiency, not out-of-sample generalization.
Critical weaknesses
  1. The paper's central deliverable is a quant backtest benchmark, and quantitative-strategy papers are notoriously difficult to reproduce: small differences in data cleaning, delisting/survivorship handling, transaction rules, benchmark alignment, fee schedules, and engine semantics can change pass rates materially. This paper does not allow that standard reproducibility burden to be discharged. The anonymous AlphaGoal URL returns HTTP 401 not_connected, no code or data supplement is shipped, and I could not inspect the engine, 175 strategy implementations, or the 155M backtest pipeline. The paper's numbers therefore currently rest entirely on the authors' private implementation.
  2. The evaluation is retrospective and in-sample by design, and the targets are selected from the upper tail of approximately 155 million sampled backtests. Without a held-out period or statistical correction for selection-on-the-best, the benchmark cannot distinguish a model that finds robust strategies from one that overfits the same fixed historical interval. This limits the external validity of the headline pass rates.
  3. The main results are single-run without confidence intervals or significance tests, and the feasibility audit for all 2,221 tasks is not reported. Several AlphaLoop-vs-Vanilla gains are small (e.g., 2.43 pp overall for Gemma4-31B) and could be sampling noise.
Questions for authors
  1. What exactly are the "twelve model-task combinations" in "ten of twelve"? Does this refer only to SG and PO, and can the wording be aligned with Table 1?
  2. For the main Table 1 difficulty ordering, is the Random Search hard number 7.50% a single-run value and 6.63 ± 0.88 the three-run mean? Please label which run is quoted in the main text.
  3. How many candidate tasks were generated before the final 2,221? What is the audit distribution of nominal versus realized threshold pass proportions, and what fraction of target thresholds had a satisfying reference after all filters?
  4. Can you provide per-task-type single/dual counts for parameter optimization so that Table 11 All columns are reconstructible?
  5. Are any headline AlphaLoop-vs-Vanilla differences statistically significant after repeated runs? Table 1 has no error bars.
  6. Does hidden-reference construction introduce a selection-on-the-best bias that interacts with the five-backtest budget? Are hard tasks hard because random candidates cannot satisfy them, or because reference strategies were selected by multiple criteria?
  7. How were the 14 securities dropped from 5,955 to 5,941 filtered? Is there a survivorship-bias risk in the Yahoo Finance coverage?
  8. Please provide an accessible code/data package, not only an anonymous URL. It must include the unified engine, all 175 atomic strategy implementations, the task-generation and threshold-audit pipeline, and a runnable minimal example so reviewers can independently verify at least one reported pass rate.
  9. How sensitive are pass rates to API/nondeterminism for the API-served models, given the paper says serving hardware is not controlled?
  10. Would the main conclusions hold if success required a one-year held-out check after the search interval?
4 comments

Thank you for recognizing the benchmark’s goal-satisfaction formulation and the documentation of its execution protocol. We address access to the released implementation first, then provide the construction audit, repeated-run evidence, and clarifications requested in your questions. Where the submission already specifies a design choice or limitation, we give its page and line references below.

W1/Q8. Reproducibility.

Thank you for reporting the access failure. Code and data were released with the submission at the anonymous AlphaGoal repository, which is accessible from our side. We apologize that the reported 401 not_connected response prevented your inspection and would appreciate another attempt to access the release.

As described in the submitted Reproducibility Statement [p.10, ll.486–496], the released materials include:

  • The complete 2,221-task dataset, including references for auditing that remain hidden from evaluated agents.
  • The market data and a PostgreSQL importer.
  • The unified backtesting Engine and all 175 atomic-strategy implementations.
  • The execution rules and market-specific fees, taxes, and lot sizes.
  • AlphaLoop, its prompts and configuration, and the script that computes pass rates from evaluation runs.
  • Docker deployment instructions and a minimal one-request backtest example.

These materials support two complementary checks: the minimal example verifies Engine setup and execution, while the evaluation pipeline and summary script allow pass rates to be recomputed from a run. The construction procedure, including percentile and tie handling, reference filtering, and task sampling, is documented in Appendix D. We additionally provide candidate and retention counts in Q3 below.

The implementation and data are therefore part of the released benchmark, rather than evidence available only through the paper’s numerical summaries.

W2. Selection-on-the-best and the historical protocol.

The construction does not take a single extreme strategy from a pooled collection of 155M backtests and use its performance as the target. Thresholds are per-stock empirical percentiles, based on approximately 26,108 strategies for that stock [p.5, ll.220–233]. They span P1–P35 depending on task cardinality and are fixed before the reference is selected.

As specified in the submission, the reference is selected only after the target conditions have been defined and serves as evidence of reachability [p.5, ll.237–241]. Evaluated methods do not have to recover it, match its returns, or satisfy its reference-ranking criteria. Any admissible strategy meeting the task conditions passes [p.5, ll.253–256].

The historical protocol is likewise part of the submitted problem definition: each query specifies a backtest request [p.3, ll.145–147]. The Introduction locates the research question on a designated historical interval [p.1, ll.039–043], and Appendix A explicitly identifies the reported outcome as retrospective goal satisfaction rather than future profitability [p.12, ll.637–639].

The benchmark therefore measures the ability to solve a historical construction problem with verified feasibility. Adding a held-out requirement would evaluate a further capability, as discussed in Q10.

W3/Q5. Uncertainty and feasibility.

The submission already includes three repeated strategy-generation runs for Qwen3.8-27B and Gemma4-31B (Table 9, [p.38, ll.2007–2017]); for Qwen3.8-27B, AlphaLoop reaches 33.68 ± 0.49% versus 24.26 ± 0.55% for Vanilla.

For Gemma4-31B, the model you identify as having a small main-table gain (+2.43 pp overall), we additionally repeated parameter optimization twice, giving three complete runs over all 2,221 tasks (mean ± sample SD, pass rate %; strategy-generation values are from Table 9):

Gemma4-31BVanillaAlphaLoopGain (pp)
Strategy generation29.36 ± 1.1831.86 ± 1.18+2.50 ± 0.67
Parameter optimization53.31 ± 0.2355.83 ± 0.86+2.52 ± 0.64
Overall (both task types)38.65 ± 0.7741.15 ± 0.86+2.51 ± 0.26

AlphaLoop outperforms Vanilla in every run, on both task types and overall. The overall gain reproduces the main-table +2.43 pp. We report run-to-run variability rather than formal significance tests.

Feasibility is enforced during construction. A target is retained only if a sampled strategy satisfies all its conditions and passes the reference-quality checks [p.5, ll.237–241; p.22, ll.1182–1187]. Every final record undergoes schema and reference validation, with an additional structural audit for parameter optimization [p.28, ll.1509–1512].

The reference is a witness of reachability, not a unique correct answer. Its parameters remain hidden; in parameter optimization, only its rule structure is supplied [p.23, ll.1201–1206]. Scoring checks the submitted strategy against the target conditions rather than comparing it with the reference. The candidate-retention audit in Q3 makes the feasibility filtering explicit.

Q1. What are the twelve model–task combinations?

They are six LLMs × two task types: strategy generation and parameter optimization, matching Table 1 [p.7]. The two exceptions to improvement are Qwen3.8-27B on parameter optimization and DS-V4-Flash on strategy generation, as stated in the main results [p.7, ll.372–374].

Q2. Are 7.50% and 6.63% different runs or summaries?

Yes. 7.50% is the Random Search hard-task result from the single run in Table 1 [p.7]. 6.63 ± 0.88% is the three-run mean and sample standard deviation in Table 9 [p.38, ll.2007–2010]. We will make this distinction explicit wherever these values are quoted.

Q3. Candidate counts, reference retention, and realized proportions.

A candidate threshold specification is a stock–metric or stock–metric-pair–level combination that passes the applicable business floors and Buy-and-Hold requirement. These eligibility conditions are specified in Appendix D.4 [p.22, ll.1149–1168].

Construction stageSingle-metric candidatesDual-metric candidates
Eligible threshold specifications236,00461,742
Retaining a satisfying reference after evidence filters235,99129,958
Retention fraction99.99%48.5%

The evidence filters require annualized return above 3%, at least 12 completed trades, and no forced-liquidation artifact or other abnormal backtesting outcome [p.22, ll.1182–1187]. The final 2,221 tasks are sampled from retained candidates under balanced coverage quotas.

For strategy-generation difficulty, the realized proportions are:

DifficultySingle-metric realized proportion: rangeDual-metric realized joint proportion: median
Hard0.98–3.00%0.33%
Medium3.99–6.00%2.82%
Easy6.98–10.00%13.56%

Joint proportions are measured directly rather than inferred by multiplying marginal proportions, consistent with the construction procedure [p.22, ll.1142–1145]. The labels describe ordered scarcity bands within each cardinality, as defined in Appendix D.6 [p.23, ll.1208–1216].

Q4. Single/dual counts for each task type.

Task typeSingle metricDual metricTotal
Strategy generation8874731,360
Parameter optimization643218861
Total1,5306912,221

These counts are consistent with the submitted dataset composition and single-metric breakdown [p.30, Tables 6–7] and provide the weights needed to reconstruct the “All” columns in Table 11 [p.40].

Q6. Does reference selection make hard tasks hard under the five-backtest budget?

For strategy generation, the target conditions and difficulty label are fixed before the reference is chosen. Ranking satisfying references by trading activity, rule count, and return does not change the set of strategies that meet those conditions [p.5, ll.237–241]. The evaluated method can find a different, simpler strategy and is not scored against the reference’s trade count, complexity, or return [p.5, ll.253–256].

The reference therefore establishes that a solution exists; the empirical pass proportion characterizes the scarcity of satisfying candidates. The reference-ranking criteria do not become additional hidden evaluation constraints.

For parameter optimization, the selected reference also determines the supplied rule structure [p.23, ll.1201–1206]. Difficulty therefore concerns the parameter space conditional on that structure. This is precisely why parameter-optimization tasks are not assigned the easy/medium/hard labels derived from the full strategy pool, as already stated in the main text [p.6, ll.270–272] and Appendix D.6 [p.23, ll.1215–1216]. All methods receive the same structure and budget, while successful reference parameters remain hidden.

Thus, the reported hard-task results concern strategy-generation scarcity, rather than an obligation to reconstruct a specially selected reference.

Q7. Excluded securities and survivorship.

The 14 securities, approximately 0.2% of the collected universe, were excluded before strategy sampling because of insufficient history, incomplete required metadata, or non-executable Engine requests [p.19, ll.1023–1025]. These are data and execution checks, rather than filters based on the profitability of strategies subsequently searched by the evaluated methods.

The collected universe does not include delisted stocks. Our comparisons are conditional on the released benchmark universe: each task concerns a specified stock, its thresholds are derived from that stock’s strategy distribution, and every method is evaluated on the same tasks and execution settings [p.5, ll.223–236; p.7, ll.353–356].

This supports a controlled comparison within AlphaGoal. The paper’s conclusions concern search performance on those tasks, rather than an estimate of market-wide portfolio profitability or a claim that the same ranking must hold for every alternative security population.

Q9. Sensitivity to API nondeterminism.

The two API-served models were evaluated once per task in the main table; we therefore do not report a repeated-run sensitivity estimate for those models. The single-run main-table protocol and the selected repeated-run analysis are explicitly distinguished in the submission [p.7, ll.358–360].

The API models’ observed overall gains are +9.55 pp for GPT-5.6-Sol and +18.96 pp for GLM-5.3-Flash [p.7, Table 1]. Improvements are also observed for the four self-hosted models, and the Qwen3.8-27B and Gemma4-31B strategy-generation gains persist in the three-run analysis [p.38, Table 9]. The empirical evidence therefore spans multiple model families and includes repeated results beyond the API-served configurations.

We keep these forms of evidence distinct: repeated self-hosted results support repeatability for those configurations, while the API results remain single-run observations. The serving setups and externally controlled API hardware are documented in Appendix G.2 [p.38, ll.2044–2047]. Task records retain candidate evaluations, tool interactions, stopping reasons, and runtime information for auditing [p.37, Appendix E.7, ll.1983–1985].

Q10. Would the conclusions hold if success also required a one-year held-out check?

Adding that requirement would change the success criterion. The current conclusion concerns whether AlphaLoop improves the ability to find strategies satisfying specified historical goals under a fixed budget. The submitted manuscript already distinguishes this capability from temporal robustness [p.12, ll.637–639; p.13, ll.677–689].

There is a positive methodological reason for retaining the long-history protocol. Historical strategy development evaluates and revises trading rules through backtesting [p.1, ll.032–035; p.13, ll.698–702]. Long-history evaluation is well established in technical-strategy research: Brock et al. (1992) use 1897–1986 data, Lo et al. (2000) study 1962–1996 data, and Hurst et al. (2017) extend trend-following evidence back to 1880. Our data span January 2015–June 2026 [p.19, ll.1013–1014], including contrasting market conditions.

A single subsequent year supplies one additional realized price path for each stock. Requiring the same return, drawdown, or exposure targets to hold in that year would make the passing decision depend on those particular conditions as well as historical search performance. Appendix A.1 already discusses this issue [p.13, ll.663–668].

It would also require a new feasibility analysis: a target known to be reachable over the designated research interval is not automatically reachable over the subsequent year. This consequence is explicitly discussed in the submission [p.13, ll.669–676].

AlphaGoal’s central design makes failure interpretable by combining reliable execution, verified feasibility, and a controlled search budget. The historical protocol preserves those properties. A temporal extension can be valuable, but it should be defined and calibrated as an additional evaluation rather than treated as a necessary condition for the validity of the current benchmark.

We hope the construction audit, repeated-run evidence, and references to the submitted evaluation definition address the main methodological concerns and support reconsideration of the benchmark’s contribution.

Reviewer_f2JvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AlphaGoal, a benchmark for assessing whether LLMs can construct trading strategies that meet explicit investment goals within a limited backtesting budget. It comprises 2,221 strategy-generation and parameter-optimization tasks built from 175 validated atomic strategies. Targets are derived from historical backtests, with hidden reference strategies establishing feasibility and a unified execution engine verifying constraint satisfaction. The accompanying AlphaLoop baseline integrates historical factor information coefficients, strategy documentation, and backtesting feedback to guide iterative search. Experiments involving six LLMs report higher overall pass rates for AlphaLoop than for a minimally guided LLM baseline under the default budget of five valid backtests, with the best configuration reaching 47.37%. The study also includes ablation studies, selected repeated runs, and budget-sensitivity analyses.

Critical strengths
  1. AlphaGoal combines a validated strategy library, a standardized backtesting engine, and hidden feasibility witnesses for each task. This design reduces ambiguity about whether failure reflects implementation errors, infeasible targets, or unsuccessful search. Accepting any strategy that meets the specified constraints also avoids treating a single reference solution as uniquely correct. These features provide a reproducible framework for evaluating goal-directed strategy search, which represents the paper’s principal contribution.

  2. The experiments span six LLMs, two task types, multiple difficulty levels, and varying backtesting budgets. Component ablations and selected repeated runs help clarify when IC guidance and detailed feedback are useful, while the results indicate that strategy generation remains challenging. These analyses complement aggregate rankings and provide a basis for studying how agents use limited backtesting opportunities to satisfy quantitative constraints.

Critical weaknesses
  1. The added value over simpler informed search is not sufficiently established. In the component controls reported in Appendix G.3, Vanilla + IC achieves a strategy-generation pass rate of 32.13% with Qwen3.8-27B, compared with 33.46% for the complete AlphaLoop system. The detailed-feedback condition also combines explicit constraint checks with trajectory access and terminal analysis, leaving their individual contributions unresolved. A non-LLM baseline using the same IC information, strategy library, and backtesting budget would help assess the workflow's incremental benefit and provide context for the benchmark's remaining difficulty.

  2. Difficulty calibration requires further empirical validation, particularly for dual-constraint tasks. As described in Appendices D.3 and D.6, difficulty labels are assigned using nominal marginal percentiles, while realized joint pass proportions are recorded only as audit statistics. Metric dependence and tied outcomes can yield different joint pass proportions even for tasks with similar marginal percentiles. The aggregate decline in performance from easy to hard supports the intended ordering, but does not establish comparable difficulty within tiers or across tasks with different numbers of constraints. Reporting the distributions of realized joint pass proportions and comparing matched task subsets would provide more direct evidence for the calibration. More extensive uncertainty reporting would also help assess the reliability of smaller performance differences across ablations and models.

  3. AlphaLoop uses absolute ICs computed over the full evaluation interval to guide factor selection, exposing the agent to evaluation-period return information. This design means the benchmark measures retrospective search with full-sample information rather than discovery on unseen data.

  4. AlphaGoal constructs tasks from historical backtests and retains a successful reference strategy to ensure feasibility. While suitable for controlled search evaluation, this does not demonstrate generalization to unseen periods.

  5. AlphaGoal's pass rate measures compliance with specified task constraints, which may not capture overall risk and trade quality. For example, the strategy in Appendix G.5 has a maximum drawdown of 58.54% and a profit-to-loss ratio of 0.93, illustrating that passing a task may not imply practical acceptability.

Questions for authors

Please refer to Critical Weaknesses for the questions, especially those on the comparison with informed search and difficulty calibration.

4 comments

Thank you for recognizing the benchmark’s principal contribution: a controlled, executable evaluation that separates search failure from implementation errors and infeasible goals. Following your suggestions, we added an IC-informed non-LLM baseline and report additional difficulty-audit results below. We also point to the relevant passages in the submitted manuscript to clarify the historical evaluation protocol.

W1. Added value over informed non-LLM search.

We added IC-Random, which receives the same IC information, strategy library, and five-backtest budget as AlphaLoop. It samples the buy rule from the Top-K factors ranked by absolute IC, and samples the sell rule and parameters at random. We evaluate K = 5, 10, and 20 on strategy generation. Random Search and IC-Random values below are three-run means; AlphaLoop ranges summarize the six main-table model configurations, each run once per task.

Strategy generation pass rate (%)EasyMediumHardAll
Random Search (3 runs, Table 9)40.2621.186.6320.81
IC-Random, Top-5 (3 runs)33.4218.426.3817.89
IC-Random, Top-10 (3 runs)36.3218.127.3819.00
IC-Random, Top-20 (3 runs)34.7418.056.3818.14
AlphaLoop (6 LLMs, Table 1)44.21–57.3725.06–33.1116.70–21.7627.13–35.44

Providing the IC ranking does not by itself reproduce AlphaLoop’s performance. All six AlphaLoop configurations exceed the strongest IC-Random setting by 8.13–16.44 percentage points overall, and each exceeds every tested IC-Random setting in every difficulty tier. None of the three IC-Random settings improves on Random Search’s overall mean, although Top-10 is slightly better on hard tasks.

These results directly address the suggested informed-search comparison. Restricting random search to high-|IC| factors is insufficient to obtain AlphaLoop’s performance: selecting a compatible entry–exit combination and configuring its parameters remain substantive parts of the task.

The Vanilla+IC result answers a complementary question. For Qwen3.8-27B, adding IC raises Vanilla from 24.19% to 32.13%, as reported in Appendix G.3 [p.39, ll.2093–2094], while the complete AlphaLoop reaches 33.46%. This identifies IC-guided construction as the main source of this model’s gain, with a further 1.33-point improvement from the combined detailed-feedback workflow. This interpretation is already stated in the submitted ablation analysis [p.39, ll.2093–2097]. AlphaLoop is a reference baseline for the benchmark; its usefulness does not require every component to contribute equally.

We agree that the detailed-feedback condition groups constraint checks, trajectory access, and terminal analysis. The current ablation measures their combined effect and does not attribute an independent gain to each. Under the default configurations in Table 1, hard-task pass rates remain at most 21.76% [p.7, Table 1; p.8, ll.402–404], even with full IC guidance, leaving substantial headroom for better search methods.

W2. Difficulty calibration and uncertainty.

The submitted manuscript explicitly states that single- and dual-metric tasks are calibrated separately [pp.5–6, ll.268–272]. Their difficulty labels describe ordered scarcity bands within each constraint cardinality, rather than asserting identical search difficulty for every task carrying the same label.

For a single metric, level P_p selects an observed threshold whose realized pass proportion is as close as possible to, but does not exceed, p%; ties are explicitly handled and the realized proportion is recorded [p.22, ll.1138–1141]. For two metrics, the actual joint proportion depends on their dependence and is no greater than either marginal proportion [p.22, ll.1142–1145]. We therefore use broader marginal levels for dual-metric tasks and audit their joint proportions directly, without assuming independence.

The realized proportions are:

DifficultySingle-metric realized proportion: rangeDual-metric realized joint proportion: median
Hard0.98–3.00%0.33%
Medium3.99–6.00%2.82%
Easy6.98–10.00%13.56%

The empirical ordering also holds within single-metric tasks and within dual-metric tasks separately: pass rates decrease from easy to medium to hard for Random Search and for both Vanilla and AlphaLoop with all six LLMs. Thus, the aggregate ordering is not merely an artifact of mixing different numbers of constraints.

The labels provide a reproducible stratification of each task family; they are not intended to make single- and dual-metric tasks with the same label probability-matched. This distinction is already reflected in Figure 2 [p.8, ll.409–411] and the separate cardinality results in Table 11 [p.40; Appendix G.4, ll.2157–2160]. Every method is nevertheless evaluated on the same tasks and constraints, so task composition does not give AlphaLoop an advantage over its baselines.

Appendix D.13 also explains the finite-budget interpretation: for independent draws from a fixed sampler with single-candidate success probability q, success within B attempts is 1−(1−q)B1-(1-q)^B [p.31, ll.1625–1634]. This is a reference interpretation of solution scarcity, while actual method performance is measured empirically.

For uncertainty, the submitted Table 9 already reports three-run means and sample standard deviations [p.38, ll.2007–2017]:

ModelVanillaAlphaLoopMean gain
Qwen3.8-27B24.26 ± 0.55%33.68 ± 0.49%+9.42 pp
Gemma4-31B29.36 ± 1.18%31.86 ± 1.18%+2.50 pp

Gemma4-31B is the model with the smallest main-table overall gain, so this comparison specifically examines a small-effect case. Its positive mean improvement persists across repeated runs. We distinguish this repeated-run evidence from the smaller single-run component differences, which are interpreted as descriptive ablation results.

W3/W4. Full-interval IC and unseen periods.

The historical evaluation setting is stated in the submitted manuscript, rather than introduced in response to the reviews. The Introduction specifies that we study the question “over a designated historical interval” [p.1, ll.039–043]. Each task specifies a historical backtest request [p.3, ll.145–147], and Section 4 explicitly describes full-interval IC as “retrospective, in-sample guidance” [p.6, ll.286–289].

This setting evaluates a meaningful stage of quantitative research: translating return and risk objectives into executable rules, evaluating them, and revising them using historical evidence. That workflow is described in the Introduction [p.1, ll.032–035] and Appendix B [p.13, ll.698–702].

Long-history evaluation has established precedents in technical-strategy research. Brock et al. (1992) examine trading rules over 1897–1986; Lo et al. (2000) evaluate technical patterns over 1962–1996; and Hurst et al. (2017) extend trend-following evidence back to 1880. Our data span January 2015–June 2026 [p.19, ll.1013–1014], covering contrasting market conditions. Requiring the same targets to hold in a subsequent year would introduce an additional temporal-robustness criterion, as discussed in Appendix A.1 [pp.12–13, ll.646–689].

Within the declared protocol, IC is an available research statistic. It informs factor selection, while the executed strategy remains a fixed rule whose signals use only information available at each decision time [pp.18–19, ll.966–986]. The same-IC non-LLM control in W1 directly examines whether access to that statistic alone accounts for AlphaLoop’s results.

The interpretation of the pass rate is therefore consistent throughout the paper: historical goal satisfaction and search efficiency. The submitted Limitations explicitly state that this does not establish out-of-sample generalization or future profitability [p.12, ll.637–639].

W5. Pass rate and practical acceptability.

Pass rate measures whether the agent satisfies the investor’s specified combination of objectives under a standardized execution protocol. The submitted evaluation definition accepts any admissible strategy meeting all stated constraints [p.5, ll.253–256]. This task-dependent criterion is central to the benchmark: return, risk, market exposure, and trade quality are distinct objectives, and investors can request different combinations.

The Appendix G.5 example illustrates this distinction explicitly. Its alpha and beta meet the requested thresholds, while its 58.54% maximum drawdown and 0.93 profit-to-loss ratio reveal weaknesses on unrequested dimensions. The submitted case study reports these values and explains why task success does not imply broad strategy quality [p.42, Appendix G.5, ll.2234–2237]. Adding an unstated drawdown threshold would change that task’s requested optimization problem. Where drawdown or trade quality is requested, it is checked directly as part of the passing conditions.

The tasks also incorporate financial grounding: return-oriented thresholds must exceed Buy-and-Hold on the requested metric [p.5, ll.234–236; p.22, ll.1159–1163], and the Engine models commissions, stamp duty, minimum fees, and market-specific lot sizes [p.16, ll.850–859]. The remaining execution limitations, including omitted slippage and market impact, are explicitly stated [p.12, ll.633–635].

These choices make the historical search problem economically interpretable while preserving an objective, task-specific success criterion. We hope the added informed-search comparison and difficulty evidence address the main empirical concerns, and that the cited passages clarify the evaluation scope already established in the submission.

Dear Reviewer f2Jv,

Thank you again for your constructive suggestions. Following them, we added an IC-informed non-LLM baseline under the same library and budget, and reported the realized difficulty proportions for single- and dual-metric tasks. As the discussion ends on November 18, we would be grateful to know whether these additions address your concerns, or whether there is anything further we could clarify.

With sincere thanks,

The Authors

Reviewer_PSKFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AlphaGoal, a benchmark that tests whether LLMs can build a trading strategy meeting a stated investment goal using only a few backtests. Examples of goals are “annualized return ≥ X” or “alpha ≥ X and |beta| ≤ Y.” The search space. Models choose from a fixed library of 175 expert-verified atomic strategies (87 buy, 88 sell, from 122 factors drawn from Alpha158, BacktestBench, and technical/candlestick patterns). They combine these by same-side conjunction and set their parameters.

The paper involves 2,221 tasks that split into 1,360 strategy-generation tasks and 861 parameter-optimization tasks. The authors also propose AlphaLoop, a ReAct/Reflexion-style agent with four tools. Six LLMs were tested with a budget of 5 valid backtests. AlphaLoop beats a minimal “Vanilla” agent for every model.

Critical strengths
  1. The paper separates three reasons a model might fail: incorrect implementation, an infeasible goal, and weak search. Using a shared audited library plus a feasibility witness for every task is a principled way to isolate search ability.

  2. Per-stock empirical percentiles avoid thresholds that are trivial for one stock and impossible for another. The random-search pass rates confirm the easy > medium > hard ordering.

  3. Candid reporting. The authors report failure cases, non-monotonic search trajectories, configurations where Vanilla falls below Random Search, and a limitations section that acknowledges the in-sample design.

  4. The Vanilla / Vanilla+IC / AlphaLoop−IC / AlphaLoop 2×2 is a well-designed decomposition. Its finding, that the IC prior drives most of AlphaLoop’s gain for Qwen, is informative.

Critical weaknesses
  1. The benchmark rewards in-sample overfitting, which works against its motivation. The paper frames the task as meeting investor goals, but an investor cares about future performance. AlphaGoal scores pure in-sample goal satisfaction: search, IC estimation, and verification all use the same interval; many search methods are compared on the same history. However, Iin quantitative finance, this is what the backtest-overfitting literature warns against.

  2. AlphaLoop’s main ingredient is information about the evaluation period itself. The IC prior is a Spearman correlation between factor values and next-day returns over the very interval used for scoring. Each trade is causal, but the factor selection uses future returns from the test period.

  3. Difficulty is defined relative to random sampling, which makes the random-search results partly circular.

  4. Figure 1 contains leftover placeholder text (“BUY Lorem ipsum”).

Questions for authors
  1. See in Critical Weaknesses.

  2. How do passing strategies perform out-of-sample compared with random strategies that pass and with Buy-and-Hold?

7 comments

Thank you for recognizing the value of separating implementation correctness, goal feasibility, and search ability, as well as the informative IC/feedback decomposition. We believe the central issue is the interpretation of the evaluation target. Below, we clarify that target with references to the submitted manuscript and provide a new same-IC non-LLM comparison.

W1. Historical goal satisfaction and the motivation of the benchmark.

AlphaGoal’s historical objective is explicit in the submitted problem formulation. The Introduction states that the research question is studied “over a designated historical interval” [p.1, ll.039–043]. Each task specifies the stock, backtest start date, initial capital, and performance conditions [p.3, ll.145–147]. Search and verification then operate under that historical protocol.

Investors’ eventual interest in future performance does not make the intermediate task of constructing and evaluating strategies on historical data uninformative. An agent must still translate objectives into trading rules, configure parameters, interpret backtest evidence, and identify a satisfying strategy under a limited budget. This iterative workflow is described in the submitted Introduction [p.1, ll.032–035] and Appendix B [p.13, ll.698–702], and automated quantitative backtesting is also studied as an evaluation problem by BacktestBench (Wang et al., 2026).

AlphaGoal isolates this capability using audited execution and a hidden feasibility witness for every task [p.1, ll.018–021; p.5, ll.237–241]. The large construction pool is not a set of candidate answers given to the agent. The evaluated agent must find its own solution within five valid backtests, without access to the hidden reference parameters [p.7, ll.353–356; p.23, ll.1201–1206].

The relevant inference is whether the agent solves the specified historical problem. The concern about interpreting a selected backtest as evidence of future profitability would apply to a different conclusion from the one reported here. The submitted Limitations already state that performance measures “retrospective goal satisfaction, not out-of-sample generalization or future profitability” [p.12, ll.637–639].

Long-history evaluation also has established precedents. Brock et al. (1992) examine trading rules over 1897–1986; Lo et al. (2000) study technical patterns over 1962–1996; and Hurst et al. (2017) extend trend-following evidence back to 1880. Our historical data span January 2015–June 2026 [p.19, ll.1013–1014]. This supports evaluation across contrasting market conditions, consistent with the long-history strategy-research setting.

The task differs from evaluating learned return predictors, as in Gu et al. (2020), or predictive-model workflows supported by Qlib (Yang et al., 2020). The distinction concerns the capability being evaluated: AlphaGoal measures historical strategy construction under a controlled search budget. Appendix A.1 explains this choice and its relationship to temporal robustness [pp.12–13, ll.646–689].

W2. IC uses returns from the designated historical interval.

The submitted manuscript explicitly states that IC is computed over the task’s full historical interval and provides “retrospective, in-sample guidance” [p.6, ll.286–289]. That interval is the agent’s declared historical research environment; it is not designated as an unseen forecasting test set.

The paper also distinguishes the information used during research from the information available during execution. IC helps prioritize factors, but the submitted strategy neither reads IC nor queries future returns. Each simulated signal uses only information available at its decision time [pp.18–19, ll.966–986]. This distinction is also reiterated in Appendix A.1 [p.13, ll.685–689].

We added a direct control to examine whether access to the ranking alone explains the results. IC-Random receives the same IC, strategy library, and B = 5 budget. Its three-run SG-All means are 17.89%, 19.00%, and 18.14% for Top-5/10/20, compared with 20.81% for Random Search. AlphaLoop achieves 27.13–35.44% across the six main-table models [p.7, Table 1]. Even the strongest IC-Random setting therefore trails every AlphaLoop configuration by at least 8.13 percentage points overall.

Together with the existing Vanilla/Vanilla+IC/AlphaLoop−IC/AlphaLoop decomposition [pp.39–40, Appendix G.3], this separates two findings: historical IC is useful for guided construction, and supplying the same ranking to a simple non-LLM sampler does not reproduce the LLM workflow’s performance. Configuring complete strategies remains a substantive search problem.

W3. Difficulty calibration relative to random sampling.

The relation to random sampling is an explicit calibration choice. Targets are per-stock empirical percentiles [p.5, ll.231–233], and Random Search measures what uninformed search can achieve under a finite budget. Appendix D.13 explains this role directly [p.31, ll.1622–1638]. Agreement between the random baseline and the scarcity ordering is a calibration check, rather than independent evidence of LLM superiority.

The method comparisons themselves are empirical. The construction does not force an LLM to beat Random Search or AlphaLoop to beat Vanilla. Indeed, the paper reports configurations where Vanilla falls below Random Search and task types where AlphaLoop declines [p.7, ll.372–377]. AlphaLoop’s overall improvement across the six models is therefore an observed result, not a consequence of the percentile definition.

The random baseline was also selected competitively. We tested K_max ∈ {2, 3, 4, 5}, covering the construction range of up to five rules [p.5, ll.218–220]. The reported K_max = 2 setting has the highest three-run mean in every difficulty tier [p.38, Table 9; ll.2031–2038]. It samples rules and parameters from the same library and legal domains used in construction [pp.6–7, ll.323–347].

Its hard-task mean is 6.63 ± 0.88%, compared with 16.70–21.76% for the six default AlphaLoop configurations [p.38, Table 9; p.7, Table 1]. Moreover, easy > medium > hard holds for all twelve LLM configurations, including separately within single- and dual-metric tasks. GPT-5.6-Sol with AlphaLoop, for example, achieves 57.37%, 33.11%, and 21.76% [p.8, ll.402–404].

Thus, scarcity provides an interpretable difficulty scale, while the observed results determine which methods use the finite budget effectively.

W4. Figure 1.

Thank you for catching this. It came from an unfilled text box left in the drawing. The box kept its default placeholder text, which is invisible to the human eye in the compiled PDF. We have located and removed it.

Q2. Out-of-sample comparison.

That comparison would evaluate the temporal robustness of the selected strategies. The submitted manuscript already distinguishes that question from historical goal-directed search [p.12, ll.637–639; p.13, ll.677–689].

There is a methodological reason for retaining the long-history protocol. Historical strategy development evaluates and revises trading rules using backtest evidence [p.1, ll.032–035; p.13, ll.698–702]. Long samples allow evaluation over contrasting conditions, as illustrated by Brock et al. (1992), Lo et al. (2000), and Hurst et al. (2017). Our data span January 2015–June 2026 [p.19, ll.1013–1014], including the 2015 Chinese market crash, the 2020 crash and rebound, and the 2022 U.S. bear market.

Requiring the same targets to hold in a single subsequent year would make success depend jointly on historical search and that year’s market regime and stock-specific price path. Appendix A.1 already explains why a strategy can satisfy a historical target but miss it in another period without changing its implementation or the search procedure [p.13, ll.663–668]. It also explains that feasibility on the original interval does not establish feasibility on the new interval [p.13, ll.669–676].

Within the stated historical task, the passive comparison is already built into target construction: every return-oriented threshold must strictly exceed Buy-and-Hold on the same stock under the same capital, fee, and execution settings [p.5, ll.234–236; p.22, ll.1159–1163]. A passing strategy therefore clears this hurdle on the requested metric. Random Search and AlphaLoop are evaluated against exactly the same conditions.

We respectfully ask that the paper be assessed on this stated contribution: controlled, finite-budget strategy search with verified feasibility and programmatic evaluation. The new same-IC comparison further demonstrates that the benchmark distinguishes effective strategy construction from simply receiving a historical factor ranking.

References cited in the responses

  • Brock, W., Lakonishok, J., and LeBaron, B. Simple Technical Trading Rules and the Stochastic Properties of Stock Returns. Journal of Finance, 1992.
  • Lo, A. W., Mamaysky, H., and Wang, J. Foundations of Technical Analysis: Computational Algorithms, Statistical Inference, and Empirical Implementation. Journal of Finance, 2000.
  • Hurst, B., Ooi, Y. H., and Pedersen, L. H. A Century of Evidence on Trend-Following Investing. Journal of Portfolio Management, 2017.
  • Wang et al. BacktestBench: Benchmarking Large Language Models for Automated Quantitative Strategy Backtesting. KDD, 2026. Already cited in the submitted paper.
  • Gu, S., Kelly, B., and Xiu, D. Empirical Asset Pricing via Machine Learning. Review of Financial Studies, 2020.
  • Yang et al. Qlib: An AI-oriented Quantitative Investment Platform. arXiv:2009.11189, 2020. Already cited in the submitted paper.

Comments on the paper

1 comment
Area_Chair_hBPNOpenReview ↗

Engaging with the authors

Dear reviewers,

The authors have posted several replies to your review. Please have a look and interact with the authors as needed. Reviewer PSKF has left a comment and how about Reviewer f2Jv and ZtZN?

Thank you, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.