- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AlphaGEAR is an LLM-based framework for constructing factor libraries for an equal-weight rank predictor. It combines library-relative residual feedback, a state-dependent program-evolution controller, statistical admission, development-window refinement, and a query-limited certification interface.
Critical strengths
The paper makes a useful distinction between standalone factor performance, incremental evidence, and improvement to the deployed combination. The frozen forward evaluation and fixed-candidate construction experiments are valuable. Appendix C.2 also provides paired temporal bootstrap analyses with multiplicity correction.
Critical weaknesses
1. The residual statistic does not establish incremental predictive information.
In Eq. (1), linear residualization ensures orthogonality to the library ranks, but the subsequent Spearman calculation reranks the residuals and can destroy this property. For example, let the library and return ranks be x = y = (1, ..., 24), and candidate ranks be z = (1, ..., 8, 21, ..., 24, 9, ..., 20). The residual–return Spearman correlation is approximately 0.44696, despite the library already perfectly predicting returns. The candidate–library correlation is 0.66609, below the 0.7 redundancy threshold. Thus, a positive residual statistic need not indicate incremental predictive information, although downstream marginal filtering may reject such candidates.
2. The ablations do not establish the central mechanisms’ added value.
In Table 4, the full method achieves a forward RankIC of 0.0430, compared with 0.0436 without residual-guided search, 0.0471 without residual-based admission, and 0.0451 with library-family BH. Removing the marginal filter or LOO pruning yields 0.0429 and 0.0438, respectively. With only three seeds, these results do not establish that the components are harmful, but they provide insufficient evidence of improved forward prediction. Benefits in compactness or reliability require matched-size comparisons or direct validation.
Questions for authors
How should the positive residual Spearman statistic in the counterexample above be interpreted as evidence beyond the existing library? Does an alternative partial-rank statistic change the experimental findings?
How does Full compare with direct greedy marginal-RankIC selection and backward elimination under identical initialization, orientation, and candidate budgets?
What measurable benefit does certification provide relative to no certification or equally query-limited continuous feedback? Can its false-promotion behavior be evaluated in a controlled setting?
3 comments
Response to Reviewer 3 (2/3) — Partial-rank reruns and matched-size selection
R3-Q1
The alternative partial-rank statistic preserves positive mean RankIC across all evaluation windows in complete closed-loop reruns.
Appendix C.4 reports three complementary checks.
- Fixed-library diagnostics. Holding each candidate's reference library and decision threshold fixed, 278 of 303 admission verdicts agree. The remaining 25 comprise 22 pass-to-reject and 3 reject-to-pass changes.
- Candidate-stream replay. Replaying the recorded candidate streams with unchanged thresholds, while allowing the reference library to evolve, preserves two of the three libraries exactly. The third library's development RankIC changes from 0.0547 to 0.0523. The union grows from 10 to 13 members, and its development RankIC changes from 0.0584 to 0.0565.
- Complete closed-loop reruns. We reran the search with the corrected partial-rank statistic in online fitness, admission, refinement, and certification, using three independent CSI 300 runs and 100 iterations per run, against three matched runs with the original statistic. Development RankIC is 0.0507 ± 0.0012 versus 0.0494 ± 0.0029, and historical RankIC is 0.0303 ± 0.0031 versus 0.0309 ± 0.0074. In the post-submission supplementary evaluation, all six frozen libraries have positive forward RankIC (Newey–West t ≥ 5.0 for each run). Forward RankIC is 0.0469 ± 0.0009 with the corrected statistic and 0.0419 ± 0.0020 with the original statistic. Values are means ± sample standard deviations across three runs. The forward difference of +0.0050 is not statistically significant (95% interval [−0.0013, +0.0115]).
The main results retain the submitted residual-Spearman implementation. Table 13 reports the separate closed-loop evaluation of the corrected partial-rank implementation.
R3-Q2
With the same candidate pools and matched library sizes, direct greedy marginal-RankIC selection achieves higher forward RankIC than Full on AlphaGEAR’s collections, while Full scores higher than backward elimination on R&D-Agent(Q)’s collections.
We compared the methods on the same 12 frozen candidate collections, including the shared initial programs and the available candidates rejected by the full construction policy. All three methods retain native signal orientations. Both direct selectors optimize development RankIC of the same equal-weight combination and match Full’s final library size in each run. The full construction policy and greedy forward selection start from an empty library. Backward elimination starts from the entire pool and removes members.
On the forward window, greedy selection beats the full construction policy on our candidate collections by +0.0066 (pointwise 95% interval [0.0022, 0.0114], Holm-adjusted p = 0.029). Backward elimination is also higher on these collections by +0.0050, with an adjusted p = 0.103. The full construction policy beats backward elimination on the R&D-Agent(Q) collections by +0.0076 (pointwise 95% interval [0.0025, 0.0125], adjusted p = 0.021). On the historical window, the policy beats both selectors on the AlphaAgent collections by +0.0026 and +0.0030 (adjusted p = 0.028 and 0.005). After Holm correction, two of the eight forward contrasts and two of the eight historical contrasts reached statistical significance (Appendix C.5). The bootstrap intervals quantify temporal uncertainty conditional on the observed runs.
The number of development-window objective evaluations can differ across selectors. The direct selectors optimize the combination score without the policy’s statistical screen or redundancy cap, so the contrasts measure the effect of changing the construction policy as a whole. The full construction policy’s admission rule makes the statistical evidence and redundancy criteria for accepting a candidate explicit.
Response to Reviewer 3 (3/3) — Certification and false promotion
R3-Q3
Under matched search budgets, the forward RankIC benefit of certification remains uncertain, and controlled simulations show that the corrected statistic reduces false promotion.
Certification under matched search budgets. We compared certification with no certification using the original residual statistic (Appendix C.6). In the post-submission supplementary evaluation of frozen libraries, mean forward RankIC was 0.0419 with certification and 0.0451 without it. The enabled-minus-disabled difference was −0.0032 (95% interval [−0.0097, +0.0030]).
The interface caps queries, restricts each response to a finite alphabet, and logs the feedback returned to search (Appendix D). Whether discrete feedback reduces adaptive overfitting remains unresolved pending a comparison with equally query-limited continuous feedback.
Per-query false promotion under the two statistics. The synthetic experiment evaluates the submitted residual-Spearman statistic and the corrected partial-rank statistic on panels with known truth (360 days × 300 names, 2,000 replicates per setting), using the certification service’s margin and threshold for a PROMOTE verdict. The partial-rank statistic reduces false-promotion rates in all 36 library-dependent settings without incremental information, covering linear, quadratic, and rearrangement-based candidate families. In the primary noiseless-candidate setting of a synthetic family based on the reviewer’s rearrangement, the rate falls from 99.8% to 10.4% with one library member and from 95.3% to 9.2% with five members. Here the labels contain independent noise, distinguishing this simulation from the exact counterexample discussed above.
For candidates with genuine incremental information, the two statistics yield similar observed detection rates in 20 of 24 settings. The corrected statistic yields rates 2.45–4.20 percentage points lower in the remaining four weak-signal settings correlated with the library. Pure-noise promotion rates range from 1.9% to 3.0%, near the 2.31% finite-sample reference rate.
Calibration varies across settings. Across the 36 library-dependent settings, corrected rates range from 0% to 10.4%. In 29 of the 36 settings, the entire 95% Wilson interval lies above the finite-sample reference rate of 2.31% (P(t₃₅₉ ≥ 2)). The statistic measures linear partial association in rank space, so nonlinear dependence on the library can remain after projection. These simulations characterize per-query behavior under the specified synthetic models (Appendix C.7). The comparison with equally query-limited continuous feedback and the effect on adaptive overfitting across a search remain open.
The certification verdicts reported in the submission were produced with the original statistic and are reported as such.
Response to Reviewer 3 (1/3) — Residual statistic and ablations
R3-W1
The counterexample identifies a re-ranking artifact, and the corrected partial-rank statistic returns zero under our degeneracy convention.
We thank the reviewer for the counterexample. Re-ranking the projected candidate residual can reintroduce association with the library ranks. The submitted statistic gives 0.44696 in this example, which should not be interpreted as evidence beyond the library. We correct Eq. (1) by projecting both candidate and label ranks on the library ranks and correlating the two residuals directly, without re-ranking. The corrected statistic measures linear partial association in rank space. In the counterexample, the label residual is zero, so the statistic returns zero under our degeneracy convention [Sec. 3.2, Appendix C.4].
Adding the candidate also lowers the equal-weight RankIC from 1.0000 to 0.8982, so it fails the separate marginal-contribution check during refinement [Sec. 3.3, Appendix C.4]. R3-Q1 reports the empirical impact of the correction, and R3-Q3 reports the controlled false-promotion experiments.
R3-W2
On the same candidate streams, Full retains fewer factors than the variants without LOO pruning, without residual-based admission, or with library-family BH. Table 4 leaves the added value of residual-guided search and residual-based admission for forward prediction unresolved.
Removing either gives a higher observed forward mean (0.0436 and 0.0471, respectively, versus 0.0430 for the full construction policy). Removing the marginal filter changes the observed forward mean only slightly, from 0.0430 to 0.0429 (Table 4).
On the same recorded candidate streams, the full construction policy retains 6.3 members on average, compared with 7.7 without leave-one-out pruning, 8.0 without residual-based admission, and 8.3 with library-family BH. All three larger variants have higher observed forward means, by 0.0008 to 0.0041. The full construction policy produces smaller libraries and higher observed forward RankIC than simple screening on all four fixed candidate collections, with gains for two sources surviving Holm correction (Tables 3 and 10).
Greedy selection outperforms the full construction policy on our candidate collections (+0.0066 forward RankIC, Holm-adjusted p = 0.029), while the policy outperforms backward elimination on the R&D-Agent(Q) collections (+0.0076, Holm-adjusted p = 0.021). Appendix C.5 reports all matched-size comparisons. In the fixed-size replacement diagnostic, the historical RankIC improvement rate for replacements was below 50% in six of nine Full runs, short of the prespecified minimum of eight runs (Appendix C.8).