- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces AlphaGoal, a benchmark for goal-directed quantitative strategy search under a limited backtest budget. It contains 175 validated atomic strategies and 2,221 strategy-construction tasks derived from historical backtests over up to 11.5 years of daily data across 5,941 stocks, with 1,360 strategy-generation and 861 parameter-optimization tasks. Each task specifies an investment goal in natural language, and success is verified programmatically by a unified backtesting engine rather than LLM judgment. The paper also proposes AlphaLoop, an LLM agent that combines historical IC, atomic-strategy documentation, and backtest feedback to construct and refine strategies. Across six LLMs, AlphaLoop improves overall pass rates over a Vanilla baseline, reaching 47.37% overall on GPT-5.6-Sol, while hard strategy-generation tasks remain challenging.
Critical strengths
- The benchmark is carefully designed to measure goal-satisfaction search ability rather than code executability or aggregate performance, using a unified engine, programmatic success verification, natural-language task generation with peer review, and hidden reference feasibility witnesses. This is a meaningful advance in how LLM quant-strategy tasks can be evaluated.
- The paper provides unusually complete documentation of the strategy space, backtest protocol, transaction costs, metric definitions, and limitations, including an explicit statement that the evaluation measures retrospective goal satisfaction and search efficiency, not out-of-sample generalization.
Critical weaknesses
- The paper's central deliverable is a quant backtest benchmark, and quantitative-strategy papers are notoriously difficult to reproduce: small differences in data cleaning, delisting/survivorship handling, transaction rules, benchmark alignment, fee schedules, and engine semantics can change pass rates materially. This paper does not allow that standard reproducibility burden to be discharged. The anonymous AlphaGoal URL returns HTTP 401 not_connected, no code or data supplement is shipped, and I could not inspect the engine, 175 strategy implementations, or the 155M backtest pipeline. The paper's numbers therefore currently rest entirely on the authors' private implementation.
- The evaluation is retrospective and in-sample by design, and the targets are selected from the upper tail of approximately 155 million sampled backtests. Without a held-out period or statistical correction for selection-on-the-best, the benchmark cannot distinguish a model that finds robust strategies from one that overfits the same fixed historical interval. This limits the external validity of the headline pass rates.
- The main results are single-run without confidence intervals or significance tests, and the feasibility audit for all 2,221 tasks is not reported. Several AlphaLoop-vs-Vanilla gains are small (e.g., 2.43 pp overall for Gemma4-31B) and could be sampling noise.
Questions for authors
- What exactly are the "twelve model-task combinations" in "ten of twelve"? Does this refer only to SG and PO, and can the wording be aligned with Table 1?
- For the main Table 1 difficulty ordering, is the Random Search hard number 7.50% a single-run value and 6.63 ± 0.88 the three-run mean? Please label which run is quoted in the main text.
- How many candidate tasks were generated before the final 2,221? What is the audit distribution of nominal versus realized threshold pass proportions, and what fraction of target thresholds had a satisfying reference after all filters?
- Can you provide per-task-type single/dual counts for parameter optimization so that Table 11 All columns are reconstructible?
- Are any headline AlphaLoop-vs-Vanilla differences statistically significant after repeated runs? Table 1 has no error bars.
- Does hidden-reference construction introduce a selection-on-the-best bias that interacts with the five-backtest budget? Are hard tasks hard because random candidates cannot satisfy them, or because reference strategies were selected by multiple criteria?
- How were the 14 securities dropped from 5,955 to 5,941 filtered? Is there a survivorship-bias risk in the Yahoo Finance coverage?
- Please provide an accessible code/data package, not only an anonymous URL. It must include the unified engine, all 175 atomic strategy implementations, the task-generation and threshold-audit pipeline, and a runnable minimal example so reviewers can independently verify at least one reported pass rate.
- How sensitive are pass rates to API/nondeterminism for the API-served models, given the paper says serving hardware is not controlled?
- Would the main conclusions hold if success required a one-year held-out check after the search interval?
4 comments
W3/Q5. Uncertainty and feasibility.
The submission already includes three repeated strategy-generation runs for Qwen3.8-27B and Gemma4-31B (Table 9, [p.38, ll.2007–2017]); for Qwen3.8-27B, AlphaLoop reaches 33.68 ± 0.49% versus 24.26 ± 0.55% for Vanilla.
For Gemma4-31B, the model you identify as having a small main-table gain (+2.43 pp overall), we additionally repeated parameter optimization twice, giving three complete runs over all 2,221 tasks (mean ± sample SD, pass rate %; strategy-generation values are from Table 9):
| Gemma4-31B | Vanilla | AlphaLoop | Gain (pp) |
|---|---|---|---|
| Strategy generation | 29.36 ± 1.18 | 31.86 ± 1.18 | +2.50 ± 0.67 |
| Parameter optimization | 53.31 ± 0.23 | 55.83 ± 0.86 | +2.52 ± 0.64 |
| Overall (both task types) | 38.65 ± 0.77 | 41.15 ± 0.86 | +2.51 ± 0.26 |
AlphaLoop outperforms Vanilla in every run, on both task types and overall. The overall gain reproduces the main-table +2.43 pp. We report run-to-run variability rather than formal significance tests.
Feasibility is enforced during construction. A target is retained only if a sampled strategy satisfies all its conditions and passes the reference-quality checks [p.5, ll.237–241; p.22, ll.1182–1187]. Every final record undergoes schema and reference validation, with an additional structural audit for parameter optimization [p.28, ll.1509–1512].
The reference is a witness of reachability, not a unique correct answer. Its parameters remain hidden; in parameter optimization, only its rule structure is supplied [p.23, ll.1201–1206]. Scoring checks the submitted strategy against the target conditions rather than comparing it with the reference. The candidate-retention audit in Q3 makes the feasibility filtering explicit.
Q1. What are the twelve model–task combinations?
They are six LLMs × two task types: strategy generation and parameter optimization, matching Table 1 [p.7]. The two exceptions to improvement are Qwen3.8-27B on parameter optimization and DS-V4-Flash on strategy generation, as stated in the main results [p.7, ll.372–374].
Q2. Are 7.50% and 6.63% different runs or summaries?
Yes. 7.50% is the Random Search hard-task result from the single run in Table 1 [p.7]. 6.63 ± 0.88% is the three-run mean and sample standard deviation in Table 9 [p.38, ll.2007–2010]. We will make this distinction explicit wherever these values are quoted.
Q3. Candidate counts, reference retention, and realized proportions.
A candidate threshold specification is a stock–metric or stock–metric-pair–level combination that passes the applicable business floors and Buy-and-Hold requirement. These eligibility conditions are specified in Appendix D.4 [p.22, ll.1149–1168].
| Construction stage | Single-metric candidates | Dual-metric candidates |
|---|---|---|
| Eligible threshold specifications | 236,004 | 61,742 |
| Retaining a satisfying reference after evidence filters | 235,991 | 29,958 |
| Retention fraction | 99.99% | 48.5% |
The evidence filters require annualized return above 3%, at least 12 completed trades, and no forced-liquidation artifact or other abnormal backtesting outcome [p.22, ll.1182–1187]. The final 2,221 tasks are sampled from retained candidates under balanced coverage quotas.
For strategy-generation difficulty, the realized proportions are:
| Difficulty | Single-metric realized proportion: range | Dual-metric realized joint proportion: median |
|---|---|---|
| Hard | 0.98–3.00% | 0.33% |
| Medium | 3.99–6.00% | 2.82% |
| Easy | 6.98–10.00% | 13.56% |
Joint proportions are measured directly rather than inferred by multiplying marginal proportions, consistent with the construction procedure [p.22, ll.1142–1145]. The labels describe ordered scarcity bands within each cardinality, as defined in Appendix D.6 [p.23, ll.1208–1216].
Q4. Single/dual counts for each task type.
| Task type | Single metric | Dual metric | Total |
|---|---|---|---|
| Strategy generation | 887 | 473 | 1,360 |
| Parameter optimization | 643 | 218 | 861 |
| Total | 1,530 | 691 | 2,221 |
These counts are consistent with the submitted dataset composition and single-metric breakdown [p.30, Tables 6–7] and provide the weights needed to reconstruct the “All” columns in Table 11 [p.40].
Q6. Does reference selection make hard tasks hard under the five-backtest budget?
For strategy generation, the target conditions and difficulty label are fixed before the reference is chosen. Ranking satisfying references by trading activity, rule count, and return does not change the set of strategies that meet those conditions [p.5, ll.237–241]. The evaluated method can find a different, simpler strategy and is not scored against the reference’s trade count, complexity, or return [p.5, ll.253–256].
The reference therefore establishes that a solution exists; the empirical pass proportion characterizes the scarcity of satisfying candidates. The reference-ranking criteria do not become additional hidden evaluation constraints.
For parameter optimization, the selected reference also determines the supplied rule structure [p.23, ll.1201–1206]. Difficulty therefore concerns the parameter space conditional on that structure. This is precisely why parameter-optimization tasks are not assigned the easy/medium/hard labels derived from the full strategy pool, as already stated in the main text [p.6, ll.270–272] and Appendix D.6 [p.23, ll.1215–1216]. All methods receive the same structure and budget, while successful reference parameters remain hidden.
Thus, the reported hard-task results concern strategy-generation scarcity, rather than an obligation to reconstruct a specially selected reference.
Q7. Excluded securities and survivorship.
The 14 securities, approximately 0.2% of the collected universe, were excluded before strategy sampling because of insufficient history, incomplete required metadata, or non-executable Engine requests [p.19, ll.1023–1025]. These are data and execution checks, rather than filters based on the profitability of strategies subsequently searched by the evaluated methods.
The collected universe does not include delisted stocks. Our comparisons are conditional on the released benchmark universe: each task concerns a specified stock, its thresholds are derived from that stock’s strategy distribution, and every method is evaluated on the same tasks and execution settings [p.5, ll.223–236; p.7, ll.353–356].
This supports a controlled comparison within AlphaGoal. The paper’s conclusions concern search performance on those tasks, rather than an estimate of market-wide portfolio profitability or a claim that the same ranking must hold for every alternative security population.
Q9. Sensitivity to API nondeterminism.
The two API-served models were evaluated once per task in the main table; we therefore do not report a repeated-run sensitivity estimate for those models. The single-run main-table protocol and the selected repeated-run analysis are explicitly distinguished in the submission [p.7, ll.358–360].
The API models’ observed overall gains are +9.55 pp for GPT-5.6-Sol and +18.96 pp for GLM-5.3-Flash [p.7, Table 1]. Improvements are also observed for the four self-hosted models, and the Qwen3.8-27B and Gemma4-31B strategy-generation gains persist in the three-run analysis [p.38, Table 9]. The empirical evidence therefore spans multiple model families and includes repeated results beyond the API-served configurations.
We keep these forms of evidence distinct: repeated self-hosted results support repeatability for those configurations, while the API results remain single-run observations. The serving setups and externally controlled API hardware are documented in Appendix G.2 [p.38, ll.2044–2047]. Task records retain candidate evaluations, tool interactions, stopping reasons, and runtime information for auditing [p.37, Appendix E.7, ll.1983–1985].
Q10. Would the conclusions hold if success also required a one-year held-out check?
Adding that requirement would change the success criterion. The current conclusion concerns whether AlphaLoop improves the ability to find strategies satisfying specified historical goals under a fixed budget. The submitted manuscript already distinguishes this capability from temporal robustness [p.12, ll.637–639; p.13, ll.677–689].
There is a positive methodological reason for retaining the long-history protocol. Historical strategy development evaluates and revises trading rules through backtesting [p.1, ll.032–035; p.13, ll.698–702]. Long-history evaluation is well established in technical-strategy research: Brock et al. (1992) use 1897–1986 data, Lo et al. (2000) study 1962–1996 data, and Hurst et al. (2017) extend trend-following evidence back to 1880. Our data span January 2015–June 2026 [p.19, ll.1013–1014], including contrasting market conditions.
A single subsequent year supplies one additional realized price path for each stock. Requiring the same return, drawdown, or exposure targets to hold in that year would make the passing decision depend on those particular conditions as well as historical search performance. Appendix A.1 already discusses this issue [p.13, ll.663–668].
It would also require a new feasibility analysis: a target known to be reachable over the designated research interval is not automatically reachable over the subsequent year. This consequence is explicitly discussed in the submission [p.13, ll.669–676].
AlphaGoal’s central design makes failure interpretable by combining reliable execution, verified feasibility, and a controlled search budget. The historical protocol preserves those properties. A temporal extension can be valuable, but it should be defined and calibrated as an additional evaluation rather than treated as a necessary condition for the validity of the current benchmark.
We hope the construction audit, repeated-run evidence, and references to the submitted evaluation definition address the main methodological concerns and support reconsideration of the benchmark’s contribution.
Thank you for recognizing the benchmark’s goal-satisfaction formulation and the documentation of its execution protocol. We address access to the released implementation first, then provide the construction audit, repeated-run evidence, and clarifications requested in your questions. Where the submission already specifies a design choice or limitation, we give its page and line references below.
W1/Q8. Reproducibility.
Thank you for reporting the access failure. Code and data were released with the submission at the anonymous AlphaGoal repository, which is accessible from our side. We apologize that the reported 401 not_connected response prevented your inspection and would appreciate another attempt to access the release.
As described in the submitted Reproducibility Statement [p.10, ll.486–496], the released materials include:
These materials support two complementary checks: the minimal example verifies Engine setup and execution, while the evaluation pipeline and summary script allow pass rates to be recomputed from a run. The construction procedure, including percentile and tie handling, reference filtering, and task sampling, is documented in Appendix D. We additionally provide candidate and retention counts in Q3 below.
The implementation and data are therefore part of the released benchmark, rather than evidence available only through the paper’s numerical summaries.
W2. Selection-on-the-best and the historical protocol.
The construction does not take a single extreme strategy from a pooled collection of 155M backtests and use its performance as the target. Thresholds are per-stock empirical percentiles, based on approximately 26,108 strategies for that stock [p.5, ll.220–233]. They span P1–P35 depending on task cardinality and are fixed before the reference is selected.
As specified in the submission, the reference is selected only after the target conditions have been defined and serves as evidence of reachability [p.5, ll.237–241]. Evaluated methods do not have to recover it, match its returns, or satisfy its reference-ranking criteria. Any admissible strategy meeting the task conditions passes [p.5, ll.253–256].
The historical protocol is likewise part of the submitted problem definition: each query specifies a backtest request [p.3, ll.145–147]. The Introduction locates the research question on a designated historical interval [p.1, ll.039–043], and Appendix A explicitly identifies the reported outcome as retrospective goal satisfaction rather than future profitability [p.12, ll.637–639].
The benchmark therefore measures the ability to solve a historical construction problem with verified feasibility. Adding a held-out requirement would evaluate a further capability, as discussed in Q10.