acceptodds
Desk rejected at ICLR 2027

AutoScientist-Quant: Self-Evolving Coding Agents for Automatic Research in Quantitative Investment

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model agents can discover alphas, yet current methods have three weaknesses. The search cannot adapt during the run, automation usually ends at alpha generation while library selection and model choice stay manual, and alpha discovery can read the test window through loop feedback or code problems. We present AutoScientist-Quant, a self evolving search process that regards quantitative research as one budgeted search problem. A single controller conditions every decision on the remaining budget, choosing at each round whether to improve, combine, pivot, or stop, which node to expand, how many alphas to generate, and how to retrieve past trajectories from the shared memory. The same core then selects from the library and tunes the model, closing the loop from hypothesis to deployable strategy. We also review the evaluation pipeline reused from prior work, fix two lookahead problems, and keep the feedback window disjoint from the held out test window, so every comparison tests true generalization. On CSI universes, the framework attains the best value of nearly every metric in every setting, and these conclusions hold across several backbones and markets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Agent, Quantitative Investment, Self-Evolving, Automatic Research

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WThfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper models quantitative research as a shared-memory, budget-constrained dynamic search in which one LLM controller coordinates alpha discovery, alpha filtering, and predictor/model search. It separates the feedback and test windows, fixes two look-ahead issues in the reused evaluation code, and reports results across Chinese and US markets and two LLM backbones.

Critical strengths
  • The method unifies alpha generation, library selection, and predictive-model search in a resource-aware closed loop with a clear engineering motivation.
  • Separating the feedback and test windows and correcting the look-ahead issues materially improve the credibility of the evaluation protocol.
  • Experiments cover multiple markets, backbones, information metrics, portfolio metrics, and component ablations. The recorded hypotheses, revisions, feedback, and stopping reasons provide useful traceability for analyzing alpha evolution.
Critical weaknesses
  • The controller is primarily an untrained prompt-based LLM decision maker, but the paper does not report full search token usage, call counts, wall-clock time, or monetary cost under a strictly matched compute budget.
  • R&D-Agent-Quant is excluded from comparison, while CSI uses index-relative returns and US experiments use absolute returns, and the simplified daily backtest may not justify the stronger deployable-strategy claims.
Questions for authors
  • Can the authors provide a unified LLM-call, token, wall-clock, and monetary-cost comparison and test sensitivity to controller prompts, temperature, and retry behavior under matched budgets?
  • Can the authors rerun R&D-Agent-Quant on the corrected pipeline and validate the conclusion with unified return definitions, realistic transaction costs, turnover/liquidity constraints, and rolling out-of-sample evaluation?
Reviewer_RJAhOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work looks at the use of Autoresearch agents for alpha generation in quantitative finance, or the discovery of stocks/bundles with higher expected future returns. It provides a harness that allows an agent to adjust time/resources spent in analysis and react to results over time. It also process for evaluation of such agents that is claimed to be trustworthy, along with a library of discovered alphas.

I find the work to lack clarity especially in the methods section. I find it lacks significant details that are important to the understanding of the paper. It seems to be fundamentally flawed due to the leakage of data through llm training.

Critical strengths
  1. The paper does a good job providing an introduction on alpha ranking and auto-research loops. I found it the most readable part of the paper. The motivations the paper lists out in this section are clear.
Critical weaknesses

The paper suffers from organizational clarity. I find it more difficult than usual to understand the details and objectives of the paper. A significant rewriting focused on clarity may help. Individual sentences, especially in sections 2 and 3, are hard to understand. An example from section 2: "Discovery returns a library whose members are individually validated but jointly redundant."

There are significant items I found missing from the paper that seem critical, for example, it is not clear to me what the actions of the LLM are within the context of the research agent - this should be easily understandable within a standard reading of the paper.

It is not clear to me how the training data for the LLM's does not provide significant leakage into the training set. GPT-OSS-120B and GLM-5 both would contain significant training data from the test set of June 2023 - May 2026. I am not sure how you can trust results generated by these models when they likely know how each of these universes will evolve.

Questions for authors

It is not clear to me what the actions of the LLM are within the context of the research agent. What exactly is being provided to the llm in context and what is it returning?

It is not clear to me how the training data for the LLM's does not provide significant leakage into the training set. GPT-OSS-120B and GLM-5 both would contain significant training data from the test set of June 2023 - May 2026. I am not sure how you can trust results generated by these models when they likely know how each of these universes will evolve. What are the arguments that this is not an issue?

Reviewer_sAMBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

LLM agents can now discover stock-picking signals (“alphas”), but existing systems follow a fixed script, stop once the alphas are generated, and, as the authors show, accidentally leaked test data into the search. AutoScientist-Quant uses a single LLM controller with a budget that decides each round whether to refine an idea, combine ideas, try something new, or stop. It reuses that same loop to trim the alpha library and choose the prediction model. With the leaks fixed, earlier methods lost money after trading costs on Chinese stocks, while this one earned modest positive returns (up to ~3.5% a year), and the lead held across two LLMs and several markets.

Critical strengths
  • Their method turns predictive signal into positive excess returns after trading costs, while the baselines lose money despite reasonable IC, and this holds across two LLM backbones and six Chinese and US markets.
  • The authors identify and fix two concrete lookahead bugs in the evaluation code shared by prior agentic alpha-mining work, keep the feedback window strictly separate from the test window, and re-run all baselines on the corrected pipeline.
  • With their method, each discovered alpha comes with a readable history of the hypothesis, revisions, and evidence behind it, which makes the results more interpretable than black-box factor mining.
Critical weaknesses
  • The controller, which is the central contribution, is underspecified. The paper gives no prompts, no description of how options, parents, and retrieval strategies are chosen. So the method is hard to verify or reproduce.
  • Regarding the results, the gains are small and noisy. Run-to-run standard deviations are about as large as the mean ARR (e.g., 1.8% ± 1.3%), and with only three runs per setting, the significance tests have little power.
  • The test window (2023–2026) largely overlaps the backbones’ pretraining periods, and the post-release check covers only a few hand-picked settings, so possible memorization of market outcomes is not fully ruled out.
  • The budget counts only generated alphas, not LLM tokens, wall-clock time, or compute. The cost of the extra filtering and model-search stages is never reported or matched for the baselines.
Questions for authors
  • Could you release the controller prompts and code, and report how often each option (IMPROVE, COMBINE, PIVOT, STOP) is chosen across runs?
  • What is the total cost of each method in LLM tokens and wall-clock time, including the filtering and model-search stages? How do the baselines perform if they get the same downstream filtering and model search, or a compute-matched budget?

Comments on the paper

1 comment

Thank you to the authors for responding carefully to my review. I will read carefully through the changes but I wanted to respond to this specific item quickly to if possible get response on in from the authors, as it is in my opinion the most pressing item to my review score.

You mention the llm's cannot see specific stock information, but that "The only market information it sees consists of aggregate metrics computed on the feedback window"

This seems to my like it would not anonymize the stock information? The LLMs were certainly trained on aggregate metrics of stocks, including the ones used in this paper, as well as what models preformed well historically in those contexts. Even if the LLM could not identify the specific stock, the LLM would update the distribution of stocks it could be interacting with. Given the nature of LLM training, it seems they would fundamentally learn historical relationships between any constructed metrics and historical behavior of stocks, and thus the only way to properly evaluate such models would be on stock data that was from after model releases?

Having worked in quantitative finance, I have seen many times where models exposed in any way to historical data would lead to biased evaluations even when engineers went though pains to anonymize said data. I am extremely skeptical of evals when models have in any way been effected by stocks in the eval.

Thanks again for responding carefully to my comments, I look forward to the author response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.