- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GAMMA, an LLM-based framework for automated alpha factor mining that addresses two limitations of existing approaches: ungrounded economic hypothesis generation and misaligned credit assignment. GAMMA introduces Active Empirical Grounding (AEG) to extract empirically supported market observations and uses them to generate economic hypotheses. Each hypothesis is implemented through multiple alpha factors, which are evaluated and clustered according to their ranking behavior to guide subsequent exploration. Experiments on the CSI 300 demonstrate strong portfolio performance against 16 baselines, while ablation studies and evaluations on additional markets provide further evidence of the framework's effectiveness.
Critical strengths
-
Clear Motivation and Well-Structured Framework. The paper identifies meaningful limitations in existing LLM-based alpha mining approaches, particularly the difficulty of distinguishing the validity of an economic hypothesis from the quality of its individual implementations. Generating multiple factor specifications from a single hypothesis and evaluating their ranking behavior at the group level is a reasonable approach. The integration of AEG and groupwise feedback provides a coherent framework with clearly defined components.
-
Strong Empirical Performance and Meaningful Ablation Results. GAMMA achieves an annualized excess return of 13.29% and an information ratio of 1.471 on the CSI 300, outperforming the reported baselines in portfolio performance. The ablation results in Table 3 show that both AEG and groupwise feedback contribute to performance improvements, with their combination achieving the strongest results. Removing behavioral clustering leads to substantial reductions in performance and search efficiency. These findings provide meaningful evidence of the framework's practical effectiveness.
-
Detailed Implementation and Evaluation Protocol. The paper provides relatively detailed descriptions of factor construction and validation, including DSL-based expression generation, AST validation, and restrictions on using future information. The chronological data splits and embargo periods are appropriate for financial time series evaluation. The evaluation includes predictive metrics, portfolio performance, factor transfer to the CSI 500, and independent mining experiments in additional markets. These design choices strengthen the empirical evaluation.
Critical weaknesses
- Insufficient Validation of Economic Hypotheses and Core Mechanisms.
The paper identifies ungrounded hallucination and misaligned credit assignment as major limitations of existing methods. While GAMMA introduces mechanisms intended to address these issues, the experiments do not sufficiently establish whether the underlying problems are directly mitigated.
AEG statistically validates market observations before hypothesis generation, but the validity of the resulting economic hypotheses is not directly evaluated. It remains unclear whether these hypotheses are consistently supported by the empirical evidence or whether GAMMA reduces the generation of ungrounded hypotheses compared with existing approaches.
A similar concern applies to Merge. The method combines economic mechanisms associated with factors that exhibit low behavioral similarity, but low correlation alone does not establish that the resulting hypothesis has a sound economic basis. The empirical validity of these merged hypotheses remains insufficiently examined.
The ablation studies demonstrate the practical contributions of AEG and groupwise feedback, but they do not directly establish that hypothesis-level credit assignment becomes more accurate. The independent contribution of Merge is not isolated through a dedicated ablation. Additional evidence connecting the proposed mechanisms to the claimed improvements would strengthen the paper.
- Unclear Role of Independent Validation in Factor Selection.
According to Appendix D.2, candidate factors must satisfy predictive performance thresholds on the training period and the combined training and validation period. The selection procedure does not appear to require candidates to satisfy the same thresholds on the validation period independently.
Since the training period is substantially longer than the validation period, a candidate with strong training performance may satisfy the combined threshold even when its predictive performance deteriorates or changes direction on the validation period. This raises questions about how effectively the selection procedure identifies factors that generalize beyond the training data.
Using the combined training and validation period is not inherently inappropriate, and the paper maintains a separate test period. This concern does not imply test data leakage. Nevertheless, the rationale for this selection criterion and its influence on the final factor library would benefit from clarification. A comparison with validation-only screening would help assess the sensitivity of the reported results to this design choice.
- Insufficient Control of Search Budgets in Baseline Comparisons.
The paper controls the number of selected factors across automated factor mining methods, using a target library size of 50 factors. However, an equal number of selected factors does not necessarily imply an equal search budget.
Search efficiency is presented as an important contribution of GAMMA, yet the comparisons do not clearly establish whether the major LLM-based baselines operate under comparable token budgets or candidate evaluation budgets. Differences in resource consumption could affect the interpretation of the reported performance improvements.
Table 2 primarily reports results from representative runs, while variability across independent runs is mainly presented for the internal ablation experiments. It is consequently difficult to determine whether the performance advantages over the strongest competing methods remain consistent across different mining runs. Comparisons under matched search budgets, or analyses of performance relative to computational cost, would provide stronger evidence for the claimed efficiency improvements.
- Limited Justification of Design Choices and Generalization Claims.
The selection of the thresholds for behavioral clustering and Merge is not sufficiently justified. It remains unclear whether the reported performance and search efficiency are robust to different combinations of these thresholds.
The handling of redundant factors also requires clarification. When a newly generated factor exhibits better predictive performance than an existing library member but is rejected as redundant, the paper does not clearly specify whether the existing factor can be replaced. This design choice may introduce dependence on the order in which factors are discovered.
The cross-market experiments provide encouraging results on the CSI 500. However, the S&P 500 strategy achieves a lower cumulative return than its benchmark even before transaction costs are considered. The evidence therefore supports the applicability of GAMMA across markets more clearly than it supports a claim of consistently strong portfolio performance across different market structures.
Questions for authors
-
How do the authors verify that the economic hypotheses generated by AEG are supported by empirical market observations? Is there direct evidence that GAMMA reduces ungrounded hypothesis generation relative to existing methods? Does the same validation apply to ? I would also appreciate clarification on whether Merge contributes independently to the reported improvements and whether groupwise feedback leads to more accurate hypothesis-level credit assignment.
-
What motivates the use of performance thresholds on the training period and the combined training and validation period rather than requiring independent validation performance? Have the authors examined how validation-only screening affects the selected factor library and the final test performance?
-
How do the search budgets of GAMMA and the major LLM-based baselines compare in terms of token consumption and evaluated candidates? Are the reported improvements maintained under comparable search budgets? It would also be helpful to know whether the strongest baselines were evaluated across multiple independent mining runs and how much their results vary.
-
How were the thresholds for behavioral clustering and Merge selected, and how sensitive are the results to their values? When a newly generated factor outperforms an existing library member but is classified as redundant, does GAMMA allow the existing factor to be replaced?
3 comments
Response to Reviewer JZ8Y
3. Repeated mining runs and search-budget comparability
Weakness 3: Equal factor counts do not imply equal resource budgets, and single-run comparisons leave mining variability unclear.
Related Question 3: Are the gains maintained across independent runs and under comparable token or candidate budgets?
Response:
-
Added independent mining runs.
Appendix B.1/Table 4 adds three independent CSI300 runs for each of the five LLM methods, reporting means and sample standard deviations:
Method RankIC Net AER (%) IR Vanilla LLM 0.0270 ± 0.0021 3.02 ± 0.47 0.359 ± 0.033 AlphaBench-EA 0.0529 ± 0.0054 9.97 ± 0.44 1.012 ± 0.066 RD-Agent 0.0594 ± 0.0032 3.63 ± 0.63 0.431 ± 0.059 AlphaAgent 0.0506 ± 0.0053 1.51 ± 0.10 0.179 ± 0.027 GAMMA 0.0632 ± 0.0004 12.31 ± 1.03 1.332 ± 0.131 -
Performance across repeated runs.
GAMMA retains the highest mean AER and IR, with a 2.34-point mean AER advantage over AlphaBench-EA. The advantage therefore extends beyond the primary-run comparison.
-
Search efficiency at a common library target.
Table 3 and Appendix B.4/Figure 6 measure resources to reach the common library target. Full GAMMA uses 653 candidates and achieves 7.69% admission, the best among the ablations. Fixed-token or fixed-candidate comparisons against all external baselines are not reported.
4. Thresholds, replacement, and cross-market interpretation
Weakness 4: Threshold robustness, redundant-factor handling, and the generalization claim need clarification.
Related Question 4: How were clustering and Merge thresholds justified, and can a stronger redundant factor replace an existing member?
Response:
-
Threshold roles and sensitivity.
Clustering uses similarity ≥0.60; Merge considers pairs with similarity ≤0.40 (Appendix D.3). Reflection consolidates similar implementations, while Merge explores complementary parents. These are fixed settings; a joint sensitivity grid is not included.
-
Replacement of stronger redundant factors.
Yes: a stronger redundant candidate can replace an existing factor. The implementation retains the higher absolute pooled training–validation RankIC and records the replacement. This prevents an earlier, weaker duplicate from automatically blocking a stronger candidate.
-
Added cross-market evidence and its scope.
Figure 4/Table 5 adds cross-market baselines. GAMMA leads these methods in AR and terminal wealth: AR is 11.08% on CSI500 and 9.34% on S&P500. This supports comparative discovery performance across markets; Figure 4(b) separately shows the S&P500 index shortfall.
Discussion Follow Up
Thank you again for your valuable feedback. We have carefully addressed your concerns in our rebuttal, including additional experimental results and clarifications.
As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal.
We have provided detailed responses to your concerns, along with additional experimental evidence. We would sincerely appreciate your feedback on whether these responses have resolved your main concerns.
Response to Reviewer JZ8Y
Thank you for your detailed review and constructive questions.
We appreciate your recognition of three strengths that are central to our contribution:
Clear motivation. You recognized the distinction between an economic hypothesis and its implementations. GAMMA develops multiple measurements and organizes their outcomes through behavioral feedback.
Strong performance and informative ablations. You highlighted the primary CSI300 AER of 13.29% and IR of 1.471. The added Table 4 also places GAMMA first in mean AER and IR across three runs per LLM method.
Inspectable implementation. You recognized the DSL constraints, AST checks, chronological splits, and embargoes. Appendices C–D document the research trace and evaluation procedures.
GAMMA also uses behavioral similarity to organize reflection and evolutionary branching, extending its role beyond deduplication. Figure 4/Table 5 adds cross-market comparisons.
We address each weakness together with its related question below, highlighting the mechanism design and additional experimental evidence:
1. Economic hypotheses, Merge, and credit assignment
Weakness 1: The experiments do not directly validate economic hypotheses, isolate Merge, or establish more accurate hypothesis-level credit assignment.
Related Question 1: How are original and merged hypotheses supported, and what directly demonstrates the proposed mechanisms?
Response:
What each evaluation stage checks.
AEG screens observations, the semantic checker checks hypothesis–specification–expression consistency, and factor admission evaluates predictive strength and redundancy. Appendices D.1 and D.3 distinguish descriptive grounding from predictive evaluation, with verdicts assigned to tested implementations.
Evidence from the case study.
Appendix C.1/Table 6 traces one hypothesis through different implementations and their feedback. It makes the evidence flow inspectable and complements the aggregate ablations.
How merged hypotheses are evaluated.
Merge selects less-correlated parents and proposes a combined rationale. New implementations must pass their own predictive evaluation; they do not inherit admission from their parents. Section 3.4 and Appendix D.3 specify this procedure.
What the mechanism experiments establish.
Table 3 shows complementary gains: full GAMMA reaches mean AER of 12.31%, versus 5.02% for AEG-only and 5.39% for group-only. The clustering ablation further tests feedback organization. Separate no-Merge and direct hallucination/attribution-accuracy measurements are not included.
2. Independent validation and the pooled admission rule
Weakness 2: The longer training period can dominate pooled statistics and mask weak validation performance.
Related Question 2: What motivates the pooled gate, and how does validation-only screening affect the library and portfolio?
Response:
Rationale and role of validation.
A bull or bear market can persist across much of a short validation window. With less than two years of validation data, an independent hard gate can therefore make admission overly dependent on that window's market regime. Our design treats train + validation as the full in-sample history, with an additional training-only strength requirement, to reduce this regime dependence (Appendix D.2). Validation also informs redundancy checks and clustering. This pooling does not itself introduce test-data leakage: the stated protocol reserves test data for final evaluation, excluding them from factor selection and Ridge fitting.
Validation-only screening results.
Validation-only thresholds would retain 26/50 factors; 24 fail at least one threshold, while two reverse RankIC direction. This distinguishes signal magnitude from direction. A validation-filtered portfolio rerun is not included.
Reported diagnostics of predictive stability.
Table 2 reports IC sign-flip rates; Figure 3(a) reports absolute RankIC across splits. Mean absolute RankIC is 0.0313/0.0223/0.0218 for train/validation/test, and 43 factors preserve their RankIC direction throughout. These provide additional stability evidence alongside the admission rule.
Further validation comparisons.
In future work, we will compare different validation settings under a common evaluation protocol, including pooled screening, an additional independent validation gate, and directional-consistency requirements, examining both the selected library and downstream portfolio performance.