- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper models quantitative research as a shared-memory, budget-constrained dynamic search in which one LLM controller coordinates alpha discovery, alpha filtering, and predictor/model search. It separates the feedback and test windows, fixes two look-ahead issues in the reused evaluation code, and reports results across Chinese and US markets and two LLM backbones.
Critical strengths
- The method unifies alpha generation, library selection, and predictive-model search in a resource-aware closed loop with a clear engineering motivation.
- Separating the feedback and test windows and correcting the look-ahead issues materially improve the credibility of the evaluation protocol.
- Experiments cover multiple markets, backbones, information metrics, portfolio metrics, and component ablations. The recorded hypotheses, revisions, feedback, and stopping reasons provide useful traceability for analyzing alpha evolution.
Critical weaknesses
- The controller is primarily an untrained prompt-based LLM decision maker, but the paper does not report full search token usage, call counts, wall-clock time, or monetary cost under a strictly matched compute budget.
- R&D-Agent-Quant is excluded from comparison, while CSI uses index-relative returns and US experiments use absolute returns, and the simplified daily backtest may not justify the stronger deployable-strategy claims.
Questions for authors
- Can the authors provide a unified LLM-call, token, wall-clock, and monetary-cost comparison and test sensitivity to controller prompts, temperature, and retry behavior under matched budgets?
- Can the authors rerun R&D-Agent-Quant on the corrected pipeline and validate the conclusion with unified return definitions, realistic transaction costs, turnover/liquidity constraints, and rolling out-of-sample evaluation?
Thank you to the authors for responding carefully to my review. I will read carefully through the changes but I wanted to respond to this specific item quickly to if possible get response on in from the authors, as it is in my opinion the most pressing item to my review score.
You mention the llm's cannot see specific stock information, but that "The only market information it sees consists of aggregate metrics computed on the feedback window"
This seems to my like it would not anonymize the stock information? The LLMs were certainly trained on aggregate metrics of stocks, including the ones used in this paper, as well as what models preformed well historically in those contexts. Even if the LLM could not identify the specific stock, the LLM would update the distribution of stocks it could be interacting with. Given the nature of LLM training, it seems they would fundamentally learn historical relationships between any constructed metrics and historical behavior of stocks, and thus the only way to properly evaluate such models would be on stock data that was from after model releases?
Having worked in quantitative finance, I have seen many times where models exposed in any way to historical data would lead to biased evaluations even when engineers went though pains to anonymize said data. I am extremely skeptical of evals when models have in any way been effected by stocks in the eval.
Thanks again for responding carefully to my comments, I look forward to the author response.