- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies continual LLM-based alpha mining, with a particular focus on the failure modes that arise when an autonomous research agent repeatedly searches over factor hypotheses for extended periods. The authors argue that existing alpha-mining systems may repeatedly fit the development window, rediscover near-duplicate factors, and report production counts that overstate genuine out-of-sample discoveries.
To address these issues, the paper proposes ULTRALPHA, a host-controlled long-horizon research harness. The framework combines metered access to development feedback, a persistent verified research state reconstructed from host-side execution records, an opportunity atlas for organizing and deduplicating previously explored factor directions, structured research memory across epochs, and parallel research lanes.
The main experiments are conducted on the CSI300 universe using three LLM backbones and several recent LLM-based and symbolic alpha-mining baselines. The reported results show high retention of deduplicated factors on an unseen test window, positive out-of-sample Rank IC, and continued growth in the number of surviving factors as research width and depth increase. The paper additionally reports a transfer experiment on minute-level cryptocurrency perpetual markets and an analysis of differences in research behavior across LLM backbones.
Critical strengths
-
The paper identifies an important and under-emphasized evaluation problem in long-horizon alpha mining. The frozen-panel diagnostic is useful because it shows that raw production count can substantially overstate genuine discovery: development performance may not transfer to unseen data, many generated factors can be near-duplicates, and a non-trivial fraction of produced candidates do not retain positive out-of-sample signal. I find this problem formulation important for evaluating autonomous research agents that repeatedly interact with historical financial data.
-
The empirical evaluation is comparatively extensive for this line of work. The authors evaluate multiple LLM backbones, compare against several recent agentic and symbolic alpha-mining baselines, analyze both research width and depth, study model behavior through host-side logs, and additionally test transfer to minute-level cryptocurrency perpetual markets. The consistency of the reported results across several backbones is a meaningful strength of the submission.
Critical weaknesses
-
My main concern is that the methodological novelty is not yet sufficiently established relative to an increasingly crowded literature on LLM-based alpha mining and autonomous quantitative research. The paper itself cites closely related systems that already explore iterative research loops, regularized exploration, evolutionary search over research trajectories, multi-agent factor construction, and research memory across mining cycles. ULTRALPHA introduces a host-controlled ledger, metered development feedback, persistent verified state, an opportunity atlas, and multi-epoch research continuation, but I am not yet convinced that the paper clearly identifies which of these mechanisms constitutes a genuinely new technical contribution rather than a careful integration of several existing ideas into a stronger research harness. Given the density of recent work in this area, I believe the paper needs a substantially sharper comparison with the closest prior systems and stronger evidence that the proposed framework enables capabilities that those systems do not.
-
The current experiments show that the complete ULTRALPHA system performs well, but they do not sufficiently isolate the source of these gains. Several important mechanisms are introduced simultaneously, including development-query budgeting, host-side verification, persistent memory, deduplication, opportunity-atlas routing, and multi-epoch continuation. Without component-level ablations, it is difficult to determine whether the improvement comes from a new search methodology, a stricter evaluation protocol, better deduplication, or simply the combined effect of multiple sensible safeguards. In addition, the width/depth scaling study increases total research compute and is explicitly not compute-matched, so it demonstrates that additional compute produces additional surviving factors under ULTRALPHA, but does not establish that ULTRALPHA uses research compute more effectively than competing systems.
Questions for authors
-
Could the authors provide a more explicit component-by-component comparison between ULTRALPHA and the closest existing LLM-based alpha-mining systems, particularly AlphaAgent, RD-Agent(Q), QuantaAlpha, and recent memory-based alpha-research systems? For each major component of ULTRALPHA, such as metered development feedback, verified research state, the opportunity atlas, persistent cross-epoch memory, and parallel research lanes, I would like to understand whether a comparable mechanism already exists in prior work and what the authors consider to be the main genuinely new technical contribution. A convincing answer to this question would materially affect my assessment of novelty.
-
Can the authors provide component-level ablations that independently remove or modify the opportunity atlas, verified research state, metered development feedback, and persistent research memory? In particular, I would like to know how the number of surviving factors, retention, and unseen-test IC change when each mechanism is removed. Such evidence would help determine whether ULTRALPHA's gains are driven by a specific methodological innovation or by the joint effect of several engineering and evaluation safeguards.
-
Can the authors provide a compute-matched comparison for the research-scaling claim? The current width and depth experiments intentionally increase the total research budget. It would be informative to compare ULTRALPHA with the strongest baselines under approximately matched model calls, tokens, wall-clock time, or development-query budgets, and report quantities such as surviving novel factors per unit of research compute. This would help distinguish "more compute produces more factors" from the stronger claim that ULTRALPHA provides a more scalable research process.
4 comments
Rebuttal (2/2)
Concern 3: The scaling claim requires comparisons at matched research budgets and surviving factors per unit of compute.
To address your concern, we compared ULTRALPHA with the baselines at approximately $30 per method and reanalyzed historical widths at common token caps. These comparisons examine test quality and factor yield under comparable model budgets. We report surviving factors per dollar alongside test quality. All methods in the cost comparison use Kimi-K3 and four lanes. The ULTRALPHA row is the additional Full campaign. A common token price schedule determines complete request prefixes from the recorded runs. Baselines retain their native workflows, and candidates delivered before the cutoff receive the same offline evaluation.
| Method | Model cost / USD | Elapsed time / h | Kept | Top-10 Test IC | All-dedup Test IC |
|---|---|---|---|---|---|
| ULTRALPHA | 29.98 | 3.56 | 21 | 0.0321 | 0.0359 |
| RD-Agent(Q) | 30.00 | 4.02 | 17 | 0.0225 | 0.0176 |
| AlphaAgent | 29.92 | 4.67 | 35 | 0.0166 | 0.0148 |
| QuantaAlpha | 29.87 | 19.21 | 95 | 0.0241 | 0.0126 |
ULTRALPHA achieves Top-10 Test IC of 0.0321, compared with 0.0225 for RD-Agent(Q) and 0.0166 for AlphaAgent, and also has higher mean Test IC. Kept/USD is 0.70 for ULTRALPHA, 0.57 for RD-Agent(Q), 1.17 for AlphaAgent, and 3.18 for QuantaAlpha. The comparison supports stronger average test signals at similar model expenditure, while AlphaAgent and QuantaAlpha produce more positive representatives per dollar.
ULTRALPHA's recorded research time is 3.56 h, compared with 4.02–19.21 h for the baselines. Time runs from the first accepted model request to the cost cutoff, including API waiting and recovery but excluding common offline evaluation. Model cost is the matched resource. These durations describe the separate observed runs.
To examine resource allocation within ULTRALPHA, we also compare historical DeepSeek widths at common total token caps across all lanes. The table below counts representatives from candidates whose development evaluations finished before each cutoff.
| Total token cap / M | W1 Kept | W2 Kept | W4 Kept | W8 Kept |
|---|---|---|---|---|
| 100 | 0 | 2 | 9 | 4 |
| 200 | 0 | 2 | 10 | 6 |
| 400 | 3 | 5 | 20 | 7 |
| 582.336 | 8 | 9 | 28 | 7 |
At the largest cap, W4 retains 28 factors and W8 retains seven. Eighteen of W4's factors come from later epochs, whereas W8's seven come from its first epoch. Greater width does not monotonically increase output under a common budget. W8 has higher mean Test IC than W4, 0.0322 versus 0.0254. W4 uses a different model endpoint, and development usage differs across runs. These differences limit attribution to parallelism alone.
The added evidence separates predictive quality at a common model cost from the quantity obtained by different research allocations. It supports the protocol's test quality at the reported expenditure and characterizes width choices within the recorded runs. A full comparison of scaling efficiency requires results across multiple budgets.
We have added the cost comparison in Appendix E.4, referenced from Section 4.1, and the width reanalysis in Appendix E.5, discussed in Section 5.1.
References
- [1] Tang et al. AlphaAgent: LLM-Driven Alpha Mining with Regularized Exploration to Counteract Alpha Decay. KDD, 2025.
- [2] Li et al. R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization. arXiv:2505.15155, 2025.
- [3] Han et al. QuantaAlpha: An Evolutionary Framework for LLM-Driven Alpha Mining. arXiv:2602.07085, 2026.
- [4] Liu et al. XAlpha: A Memory-Driven AI Quant Researcher for Hypothesis-to-Code Alpha Discovery. arXiv:2607.08332, 2026.
Thanks for the additional experiments. The new ablations and cost-matched comparison address several of my earlier concerns, particularly regarding component contributions and evaluation fairness. The development-quota experiment is also interesting, as more feedback improves dev IC but hurts test IC.
My main remaining concerns are the lack of repeated runs for the ablations and the limited evidence for scaling efficiency under a fixed compute budget. I also think the paper could be more careful in distinguishing positive test IC from statistically significant discoveries.
Overall, the rebuttal has improved my assessment, although I still have some reservations about the novelty and scaling claims.
Thank you
Dear Reviewer WrMe,
Thanks for your follow-up comment and positive feedback on the new experiments.
Rather than adding another memory module, Ultralpha specifies a joint protocol in which host-verified execution records govern research continuation while new development feedback is separately budgeted. To our knowledge, no prior alpha mining system connects verified state, autonomous planning, and a shared evaluation allowance in this way.
In the next version, we will add repeated ablation runs with variance, and extend the fixed-budget scaling analysis.
Best regards,
The Authors of Ultralpha
Rebuttal (1/2)
Thanks for your insightful feedback! We hope the following responses address your concerns.
ULTRALPHA makes execution records the authority for continued research and gives new development observations a separate budget. The agent uses the resulting state to choose further analysis, revision, or evaluation across research rounds and parallel lanes. The comparison below distinguishes shared mechanisms from their roles in ULTRALPHA.
The table shows that ULTRALPHA builds on established memory and exploration mechanisms. AlphaAgent's experience library, RD-Agent(Q)'s knowledge forest, QuantaAlpha's trajectories, and XAlpha's layered memory provide precedents for inheriting research history. AlphaAgent and QuantaAlpha constrain structural redundancy, RD-Agent(Q) filters correlated factors, and XAlpha plans from coverage and verified hypotheses.
ULTRALPHA's distinction is to connect this history with explicit rules for continuation and new evaluations. Model notes carry explanations and revision plans, while the host reconstructs candidate status from execution records. Notes cannot establish a pass or override recorded failures. The atlas combines this status with coverage and overlap to guide further work. New development observations consume a shared budget, while results are replayed when code and evaluation settings match. Parallel lanes retain separate contexts and merge their execution records at epoch boundaries.
ULTRALPHA jointly governs verified research continuation and access to new development feedback. To our knowledge, the compared systems do not connect execution records, autonomous planning, and shared evaluation budgets through such an explicit protocol.
We have clarified the protocol in Sections 3.1–3.2 and expanded the discussion of its component precedents and technical distinction in Section 6.
Following your suggestion, we added four interventions to test which parts of the protocol contribute to the observed quantity and quality of factors. The interventions alter research guidance, inherited information, or the development allowance while keeping final evaluation common. Each setting uses one Kimi-K3 campaign with four epochs, four lanes, and a 64M token cap.
Positive % = Kept/Dedup. Top-10 is selected and oriented using development data before deduplication. All-dedup averages Test IC across all representatives, including negative values. Both use daily 24h Rank IC under the evaluation protocol in Appendix E.1.
With current verification and the final evaluator unchanged, removing atlas guidance lowers Top-10 Test IC to 0.0238, while identity checks remain. Hiding historical verification outcomes and derived atlas recommendations lowers mean Test IC to 0.0297. An alternative candidate scope and evaluation preserve Full's mean-IC advantage but reverse the Top-10 ordering. Removing inherited notes and handoffs lowers mean Test IC to 0.0293, while verified state, the atlas, and current epoch notes remain. Increasing the quota from 3 to 12 queries per lane per epoch raises actual queries from 44 to 93. Dev IC rises while Test IC falls, nearly doubling their gap to 0.0431. Full has the highest Top-10 and mean Test IC in the main evaluation, while several interventions produce more positive representatives.
These results support the contribution of research information and feedback allocation to candidate quality under common final evaluation. The effects are conditional on the retained components, and the memory intervention does not compare continuation with independent restarts.
We have added the component results in Appendix E.2 and the quota comparison in Appendix E.3, with a reference in Section 4.1.