- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes CosmosAlpha, an LLM-driven formulaic alpha-mining framework that organizes previously evaluated factors as a multi-relational graph rather than only through generation lineage. The graph contains generation, expression-structure, and empirical exposure-correlation relations among complete factors. The method defines three search utilities, namely Quality, Fertility, and Diversity, propagates these utilities through the graph using Graph Flow, and then applies Pareto-guided parent selection before LLM-based factor mutation.
The method is evaluated on CSI 300 and CSI 500 using a common downstream LightGBM and portfolio evaluation pipeline. The main comparisons include handcrafted factors, genetic programming, several recent formulaic alpha-mining methods, multiple LLM-based systems, and a controlled CosmosAlpha-MCTS baseline. The paper also provides component and relation ablations, LLM-backbone sensitivity experiments, and several independent runs. The reported results show that CosmosAlpha-Graph achieves strong predictive and portfolio metrics relative to the considered baselines.
Critical strengths
-
The paper proposes a clear and reasonably well-motivated search representation. Instead of preserving only generation lineage, CosmosAlpha connects complete evaluated factors using generation, expression-structure, and empirical exposure-correlation relations. The combination of Graph Flow with Pareto-guided selection over Quality, Fertility, and Diversity gives the paper a concrete algorithmic contribution rather than relying only on prompt engineering or a generic LLM search loop.
-
The experimental section is relatively comprehensive. The authors compare against a broad set of formulaic alpha-mining and LLM-driven baselines, include a controlled CosmosAlpha-MCTS comparison, provide relation- and utility-level ablations, test multiple LLM backbones, and report independent runs. In particular, the ablation study is useful because removing Graph Flow, individual objectives, or individual relation types consistently degrades the reported metrics.
Critical weaknesses
-
I am not yet sufficiently convinced that the reported improvement is robust to overfitting or adaptive search effects. The paper uses a chronological train/validation/test protocol, which is a meaningful strength, and the test period is held out from factor-expression search. However, formulaic alpha mining evaluates a large number of candidate hypotheses and search decisions, while the headline evidence is still based primarily on one temporal split and two closely related A-share universes. This leaves open the possibility that some of the reported advantage is specific to the chosen market regime, temporal partition, or experimental design. I would like to see additional robustness evidence, such as alternative chronological cutoffs, rolling or walk-forward evaluation, subperiod results, or another genuinely untouched market/period. This is important because the central claim is not simply that the search procedure fits the training period well, but that it discovers factors with robust out-of-sample predictive value.
-
The economic evaluation is incomplete without a clear transaction-cost sensitivity analysis. The paper reports substantial differences in ARR, IR, and MDD, but these metrics alone do not establish that the advantage remains economically meaningful after trading frictions. This matters especially for automatically mined factors because different factor pools may induce very different turnover. I would like the authors to report turnover and net performance under reasonable transaction-cost and slippage assumptions, preferably across several cost levels. If CosmosAlpha retains its relative advantage after such costs, this would significantly strengthen the empirical contribution.
As for writing, the paper is generally understandable and the high-level method is reasonably well organized. However, I found several presentation issues that noticeably reduce readability:
Most importantly, Figure 1 is visually poor in the submitted PDF. The text and graphical elements appear heavily rasterized/compressed, and some details are difficult to read at normal zoom. I have no objection to the use of AI-assisted figure generation or design tools, but the final paper should provide a publication-quality figure, preferably as vector graphics, with text and diagram elements that remain sharp and readable in the PDF.
The layouts of Tables 1 and 2 could also be improved. The large number of metrics, narrow columns, and multi-line headers make the tables visually crowded and harder to scan than necessary. Reformatting the predictive and portfolio metric groups, improving spacing, and reducing header clutter would substantially improve presentation quality.
Questions for authors
- Could the authors provide stronger evidence that the reported improvement is not specific to the current temporal split or the particular A-share evaluation setting?
I appreciate that the 2021--2025 test period is held out from factor search. However, given the highly adaptive nature of formulaic alpha mining, a single chronological split does not completely resolve my concern about search overfitting or regime-specific discovery. Could the authors report at least one additional robustness experiment, for example:
- alternative chronological train/test cutoffs,
- rolling or walk-forward evaluation,
- performance broken down by test subperiod or market regime,
- or evaluation on an additional genuinely untouched market or period?
I would also find it useful to see whether the relative ranking between CosmosAlpha and the strongest baselines remains stable across these splits. Convincing additional evidence here would materially improve my assessment.
- How does the portfolio advantage change after realistic trading costs?
Please report the turnover of the resulting strategies and net portfolio performance after commissions/slippage. A simple sensitivity analysis using several reasonable one-way cost assumptions, for example 5, 10, and 20 bps, would already be informative. The important question for me is not the exact cost model, but whether CosmosAlpha's improvement in ARR and IR remains after accounting for differential turnover across methods.
A convincing transaction-cost robustness result would materially increase my confidence in the practical significance of the reported portfolio improvements.
- Could the authors improve the camera-ready presentation of Figure 1 and Tables 1--2?
Figure 1 is difficult to read in the current PDF because the visual appears strongly rasterized/compressed. Please consider regenerating it as a vector figure with readable text at normal zoom. Tables 1 and 2 would also benefit from cleaner spacing and layout, as the current multi-level headers and dense metric columns make them unnecessarily difficult to scan. This is primarily a presentation concern rather than a technical one.
3 comments
Thanks for the additional experiments. The new ablations and cost-matched comparison address several of my earlier concerns, particularly regarding component contributions and evaluation fairness. The development-quota experiment is also interesting, as more feedback improves dev IC but hurts test IC.
My main remaining concerns are the lack of repeated runs for the ablations and the limited evidence for scaling efficiency under a fixed compute budget. I also think the paper could be more careful in distinguishing positive test IC from statistically significant discoveries. Overall, the rebuttal has improved my assessment
Thank You for Your Constructive Follow-up
Dear Reviewer,
Thank you for your thoughtful follow-up and for taking the time to carefully reconsider our work in light of the additional experiments. We sincerely appreciate your constructive feedback and your recognition of the improvements made during the rebuttal.
We acknowledge your remaining concerns regarding the reproducibility of the ablation results and the limited evidence for scaling efficiency across different compute budgets. These are important considerations for assessing the robustness and generalizability of the proposed search framework. We will clarify the scope and limitations of our current experimental evidence in the revised manuscript.
We also appreciate your suggestion to distinguish positive out-of-sample IC from statistically significant factor discoveries. We agree that this distinction is particularly important in adaptive factor mining, and we will refine the relevant statements to ensure that our conclusions accurately reflect the strength of the statistical evidence.
Thank you again for your valuable feedback and constructive suggestions. They have been very helpful in improving the rigor and clarity of our work.
Response to W1–W3/Q1–Q3: Rolling Robustness, Transaction Costs, and Presentation
Response to Reviewer 3
Rolling out-of-sample robustness (W1; Q1)
Thank you. We added a five-window CSI 300 rolling evaluation against Alpha158, MCTS and Chain. Both factor mining and LightGBM training roll forward annually. The first window trains on April 2010–May 2019, validates on June 2019–December 2020 and tests on 2021. All three segments then move forward one year, through the 2025 test year. In each window, Graph, MCTS and Chain mine factors using only that window's training segment, which also trains LightGBM; expressions are then fixed for validation and testing. Alpha158 retains its predefined expressions. LightGBM hyperparameters are shared and fixed; validation selects the boosting iteration.
These supplemental rolling results are reported separately from the original Table 1 runs.
Annual test RankIC. Bold indicates the highest value within each year.
Annual test ARR (%), before transaction costs.
Graph's RankIC exceeds MCTS and Chain in all five years and Alpha158 in four. Its ARR exceeds each baseline in four of five years. Graph ranks first in RankIC in four years and ARR in three. These results support the temporal robustness of both factor mining and downstream integration across different chronological cutoffs.
Turnover and transaction-cost sensitivity (W2; Q2)
Thank you. We replayed saved-model CSI 300 predictions over 2021–2025, separately from the rolling study above. We charge symmetric proportional costs of 0/5/10/20 bps on both buys and sells as combined commission/slippage scenarios. Predictions and strategy settings are fixed; portfolios are replayed at each cost level.
Daily turnover is total buy-plus-sell notional divided by previous account value; annualized turnover is its daily mean times 252. ARR and IR use the common Qlib annualization; signed MDD is drawdown of cumulative summed daily returns. The uncharged CSI 300 price-index reference covers the same 1,212 dates.
Graph has the lowest turnover and leads all factor baselines on ARR, IR and MDD at every cost level. At 10 bps, its net ARR is 4.01% (IR 0.2485). At 20 bps, all four net ARRs are negative. Even at this highest tested cost, Graph still outperforms the uncharged CSI 300 index in ARR (-0.53% vs. -0.78%), IR (-0.0330 vs. -0.0445) and MDD (-46.00% vs. -56.00%).
Presentation (W3; Q3)
We will provide a vector version of Figure 1 and improve the spacing, headers and metric grouping in Tables 1–2 in the revised manuscript.