acceptodds
Under review as a conference paper at ICLR 2027

Ultralpha: Continual Alpha Mining as Research Compute Scales

19 views

Abstract

Mining alpha factors is an iterative research process. In practice, a quant researcher writes a hypothesis as factor code and backtests it, letting each outcome decide the next attempt. Large language model (LLM) agents now conduct autonomous research for days, and a growing number of works have them run this loop, writing and testing factor code to mine new factors. While these methods have proven effective at producing candidates, our systematic analysis finds that their output repeats known factors and decays quickly in future tests. To mitigate these problems, we propose Ultralpha, a closed-loop harness for long-horizon alpha research. It keeps the agent researching in one protected loop with metered feedback and recorded actions, and carries its progress in a persistent research state that deduplicates output and keeps verified evidence. We evaluate Ultralpha on the A-share CSI300 universe with three base models. Ultralpha keeps mining new alpha. Under the same frozen evaluation, after deduplication 32 to 46 of its delivered factors stay positive on the test window, a retention of 95.7% to 97.0% of all delivered factors. Its Top-10 average 24h development IC is 0.0536 to 0.0576, and that average stays positive on the test window in all three blocks. Traded factors keep compounding while baseline curves rise and fade. Further analysis shows strong generalization of the framework to minute-level crypto perpetual markets and alpha mining continuing as research compute scales.

Primary area:
foundation or frontier models, including LLMs
Keywords:
ai for quant, long-horizon agent, autoresearch

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WrMeOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual LLM-based alpha mining, with a particular focus on the failure modes that arise when an autonomous research agent repeatedly searches over factor hypotheses for extended periods. The authors argue that existing alpha-mining systems may repeatedly fit the development window, rediscover near-duplicate factors, and report production counts that overstate genuine out-of-sample discoveries.

To address these issues, the paper proposes ULTRALPHA, a host-controlled long-horizon research harness. The framework combines metered access to development feedback, a persistent verified research state reconstructed from host-side execution records, an opportunity atlas for organizing and deduplicating previously explored factor directions, structured research memory across epochs, and parallel research lanes.

The main experiments are conducted on the CSI300 universe using three LLM backbones and several recent LLM-based and symbolic alpha-mining baselines. The reported results show high retention of deduplicated factors on an unseen test window, positive out-of-sample Rank IC, and continued growth in the number of surviving factors as research width and depth increase. The paper additionally reports a transfer experiment on minute-level cryptocurrency perpetual markets and an analysis of differences in research behavior across LLM backbones.

Critical strengths
  1. The paper identifies an important and under-emphasized evaluation problem in long-horizon alpha mining. The frozen-panel diagnostic is useful because it shows that raw production count can substantially overstate genuine discovery: development performance may not transfer to unseen data, many generated factors can be near-duplicates, and a non-trivial fraction of produced candidates do not retain positive out-of-sample signal. I find this problem formulation important for evaluating autonomous research agents that repeatedly interact with historical financial data.

  2. The empirical evaluation is comparatively extensive for this line of work. The authors evaluate multiple LLM backbones, compare against several recent agentic and symbolic alpha-mining baselines, analyze both research width and depth, study model behavior through host-side logs, and additionally test transfer to minute-level cryptocurrency perpetual markets. The consistency of the reported results across several backbones is a meaningful strength of the submission.

Critical weaknesses
  1. My main concern is that the methodological novelty is not yet sufficiently established relative to an increasingly crowded literature on LLM-based alpha mining and autonomous quantitative research. The paper itself cites closely related systems that already explore iterative research loops, regularized exploration, evolutionary search over research trajectories, multi-agent factor construction, and research memory across mining cycles. ULTRALPHA introduces a host-controlled ledger, metered development feedback, persistent verified state, an opportunity atlas, and multi-epoch research continuation, but I am not yet convinced that the paper clearly identifies which of these mechanisms constitutes a genuinely new technical contribution rather than a careful integration of several existing ideas into a stronger research harness. Given the density of recent work in this area, I believe the paper needs a substantially sharper comparison with the closest prior systems and stronger evidence that the proposed framework enables capabilities that those systems do not.

  2. The current experiments show that the complete ULTRALPHA system performs well, but they do not sufficiently isolate the source of these gains. Several important mechanisms are introduced simultaneously, including development-query budgeting, host-side verification, persistent memory, deduplication, opportunity-atlas routing, and multi-epoch continuation. Without component-level ablations, it is difficult to determine whether the improvement comes from a new search methodology, a stricter evaluation protocol, better deduplication, or simply the combined effect of multiple sensible safeguards. In addition, the width/depth scaling study increases total research compute and is explicitly not compute-matched, so it demonstrates that additional compute produces additional surviving factors under ULTRALPHA, but does not establish that ULTRALPHA uses research compute more effectively than competing systems.

Questions for authors
  1. Could the authors provide a more explicit component-by-component comparison between ULTRALPHA and the closest existing LLM-based alpha-mining systems, particularly AlphaAgent, RD-Agent(Q), QuantaAlpha, and recent memory-based alpha-research systems? For each major component of ULTRALPHA, such as metered development feedback, verified research state, the opportunity atlas, persistent cross-epoch memory, and parallel research lanes, I would like to understand whether a comparable mechanism already exists in prior work and what the authors consider to be the main genuinely new technical contribution. A convincing answer to this question would materially affect my assessment of novelty.

  2. Can the authors provide component-level ablations that independently remove or modify the opportunity atlas, verified research state, metered development feedback, and persistent research memory? In particular, I would like to know how the number of surviving factors, retention, and unseen-test IC change when each mechanism is removed. Such evidence would help determine whether ULTRALPHA's gains are driven by a specific methodological innovation or by the joint effect of several engineering and evaluation safeguards.

  3. Can the authors provide a compute-matched comparison for the research-scaling claim? The current width and depth experiments intentionally increase the total research budget. It would be informative to compare ULTRALPHA with the strongest baselines under approximately matched model calls, tokens, wall-clock time, or development-query budgets, and report quantities such as surviving novel factors per unit of research compute. This would help distinguish "more compute produces more factors" from the stronger claim that ULTRALPHA provides a more scalable research process.

4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thanks for your insightful feedback! We hope the following responses address your concerns.

Concern 1: The novelty of each major component must be distinguished from its closest precedents.

ULTRALPHA makes execution records the authority for continued research and gives new development observations a separate budget. The agent uses the resulting state to choose further analysis, revision, or evaluation across research rounds and parallel lanes. The comparison below distinguishes shared mechanisms from their roles in ULTRALPHA.

FeatureAlphaAgent [1]RD-Agent(Q) [2]QuantaAlpha [3]XAlpha [4]ULTRALPHA
Research memory✓✓✓✓✓
Redundancy control✓✓✓✓✓
Parallel exploration-✓✓-✓
Iterative continuation✓✓✓✓✓
Feedback budgeting----✓
State verification----✓
Coverage planning---✓✓

The table shows that ULTRALPHA builds on established memory and exploration mechanisms. AlphaAgent's experience library, RD-Agent(Q)'s knowledge forest, QuantaAlpha's trajectories, and XAlpha's layered memory provide precedents for inheriting research history. AlphaAgent and QuantaAlpha constrain structural redundancy, RD-Agent(Q) filters correlated factors, and XAlpha plans from coverage and verified hypotheses.

ULTRALPHA's distinction is to connect this history with explicit rules for continuation and new evaluations. Model notes carry explanations and revision plans, while the host reconstructs candidate status from execution records. Notes cannot establish a pass or override recorded failures. The atlas combines this status with coverage and overlap to guide further work. New development observations consume a shared budget, while results are replayed when code and evaluation settings match. Parallel lanes retain separate contexts and merge their execution records at epoch boundaries.

ULTRALPHA jointly governs verified research continuation and access to new development feedback. To our knowledge, the compared systems do not connect execution records, autonomous planning, and shared evaluation budgets through such an explicit protocol.

We have clarified the protocol in Sections 3.1–3.2 and expanded the discussion of its component precedents and technical distinction in Section 6.

Concern 2: Component ablations are needed to identify the source of gains in factor survival, retention, and test IC.

Following your suggestion, we added four interventions to test which parts of the protocol contribute to the observed quantity and quality of factors. The interventions alter research guidance, inherited information, or the development allowance while keeping final evaluation common. Each setting uses one Kimi-K3 campaign with four epochs, four lanes, and a 64M token cap.

SettingTokens / MKept / DedupPositive %Top-10 Dev ICTop-10 Test ICAll-dedup Test IC
Full62.5821 / 21100.0%0.05380.03210.0359
No atlas guidance62.7419 / 19100.0%0.05320.02380.0257
No verified-state view62.7427 / 27100.0%0.05620.02880.0297
No persistent memory62.6629 / 29100.0%0.05710.02850.0293
Higher dev quota62.6654 / 5598.2%0.06710.02400.0200

Positive % = Kept/Dedup. Top-10 is selected and oriented using development data before deduplication. All-dedup averages Test IC across all representatives, including negative values. Both use daily 24h Rank IC under the evaluation protocol in Appendix E.1.

With current verification and the final evaluator unchanged, removing atlas guidance lowers Top-10 Test IC to 0.0238, while identity checks remain. Hiding historical verification outcomes and derived atlas recommendations lowers mean Test IC to 0.0297. An alternative candidate scope and evaluation preserve Full's mean-IC advantage but reverse the Top-10 ordering. Removing inherited notes and handoffs lowers mean Test IC to 0.0293, while verified state, the atlas, and current epoch notes remain. Increasing the quota from 3 to 12 queries per lane per epoch raises actual queries from 44 to 93. Dev IC rises while Test IC falls, nearly doubling their gap to 0.0431. Full has the highest Top-10 and mean Test IC in the main evaluation, while several interventions produce more positive representatives.

These results support the contribution of research information and feedback allocation to candidate quality under common final evaluation. The effects are conditional on the retained components, and the memory intervention does not compare continuation with independent restarts.

We have added the component results in Appendix E.2 and the quota comparison in Appendix E.3, with a reference in Section 4.1.

AuthorsOpenReview ↗

Rebuttal (2/2)

Concern 3: The scaling claim requires comparisons at matched research budgets and surviving factors per unit of compute.

To address your concern, we compared ULTRALPHA with the baselines at approximately $30 per method and reanalyzed historical widths at common token caps. These comparisons examine test quality and factor yield under comparable model budgets. We report surviving factors per dollar alongside test quality. All methods in the cost comparison use Kimi-K3 and four lanes. The ULTRALPHA row is the additional Full campaign. A common token price schedule determines complete request prefixes from the recorded runs. Baselines retain their native workflows, and candidates delivered before the cutoff receive the same offline evaluation.

MethodModel cost / USDElapsed time / hKeptTop-10 Test ICAll-dedup Test IC
ULTRALPHA29.983.56210.03210.0359
RD-Agent(Q)30.004.02170.02250.0176
AlphaAgent29.924.67350.01660.0148
QuantaAlpha29.8719.21950.02410.0126

ULTRALPHA achieves Top-10 Test IC of 0.0321, compared with 0.0225 for RD-Agent(Q) and 0.0166 for AlphaAgent, and also has higher mean Test IC. Kept/USD is 0.70 for ULTRALPHA, 0.57 for RD-Agent(Q), 1.17 for AlphaAgent, and 3.18 for QuantaAlpha. The comparison supports stronger average test signals at similar model expenditure, while AlphaAgent and QuantaAlpha produce more positive representatives per dollar.

ULTRALPHA's recorded research time is 3.56 h, compared with 4.02–19.21 h for the baselines. Time runs from the first accepted model request to the cost cutoff, including API waiting and recovery but excluding common offline evaluation. Model cost is the matched resource. These durations describe the separate observed runs.

To examine resource allocation within ULTRALPHA, we also compare historical DeepSeek widths at common total token caps across all lanes. The table below counts representatives from candidates whose development evaluations finished before each cutoff.

Total token cap / MW1 KeptW2 KeptW4 KeptW8 Kept
1000294
20002106
40035207
582.33689287

At the largest cap, W4 retains 28 factors and W8 retains seven. Eighteen of W4's factors come from later epochs, whereas W8's seven come from its first epoch. Greater width does not monotonically increase output under a common budget. W8 has higher mean Test IC than W4, 0.0322 versus 0.0254. W4 uses a different model endpoint, and development usage differs across runs. These differences limit attribution to parallelism alone.

The added evidence separates predictive quality at a common model cost from the quantity obtained by different research allocations. It supports the protocol's test quality at the reported expenditure and characterizes width choices within the recorded runs. A full comparison of scaling efficiency requires results across multiple budgets.

We have added the cost comparison in Appendix E.4, referenced from Section 4.1, and the width reanalysis in Appendix E.5, discussed in Section 5.1.

References

Thanks for the additional experiments. The new ablations and cost-matched comparison address several of my earlier concerns, particularly regarding component contributions and evaluation fairness. The development-quota experiment is also interesting, as more feedback improves dev IC but hurts test IC.

My main remaining concerns are the lack of repeated runs for the ablations and the limited evidence for scaling efficiency under a fixed compute budget. I also think the paper could be more careful in distinguishing positive test IC from statistically significant discoveries.

Overall, the rebuttal has improved my assessment, although I still have some reservations about the novelty and scaling claims.

Dear Reviewer WrMe,

Thanks for your follow-up comment and positive feedback on the new experiments.

Rather than adding another memory module, Ultralpha specifies a joint protocol in which host-verified execution records govern research continuation while new development feedback is separately budgeted. To our knowledge, no prior alpha mining system connects verified state, autonomous planning, and a shared evaluation allowance in this way.

In the next version, we will add repeated ablation runs with variance, and extend the fixed-budget scaling analysis.

Best regards,

The Authors of Ultralpha

Reviewer_GpJdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of long-term alpha mining using LLM agents and identifies three major issues with existing methods: performance on the development set fails to transfer reliably to the test set; production counts do not reliably reflect genuine discoveries; and production can be inflated by repeated or highly similar factor rewrites. To address these issues, the authors propose ULTRALPHA, a host-controlled research framework that incorporates metered development feedback, persistent verified research state, a deduplication mechanism, and parallel research lanes. Experiments on the A-share CSI300 universe with three base LLMs show high factor retention and positive out-of-sample IC, with further validation through research-compute scaling experiments and transfer to minute-level cryptocurrency perpetual markets.

Critical strengths
  1. This paper addresses a practical issue: when using LLMs for long-term alpha mining, the goal isn't simply to "generate more factors"; one must also account for overfitting on the development set, redundant searches, and whether the results actually hold up on the test set.

  2. The overall design of ULTRALPHA is sound—particularly the separation of the model's own research logs from the results verified by the host system; I believe this distinction is crucial for long-term agent-based research.

  3. The experimental work is extensive; in addition to testing three different LLM backbones, the authors included experiments on scaling and the cryptocurrency market, providing solid support for the paper's main conclusions.

Critical weaknesses
  1. In Section 4.1, the experimental budgets for ULTRALPHA and the baseline were not fully aligned, raising questions about the fairness of the main results. Given that the paper emphasizes "research compute scaling," a more rigorous compute-matched comparison would have been more compelling.

  2. Sections 3.2 and 3.3 introduce several key modules, yet the subsequent Sections 4 and 5 lack systematic ablation studies. Consequently, it is difficult to determine the specific performance gains contributed by individual components such as persistent state, opportunity atlas, and parallel lanes.

  3. The validation of long-term scaling and cross-market generalization in Sections 5.1 and 5.2 remains somewhat limited. While the current scaling scope and crypto transfer experiments offer some support, they are insufficient to conclusively demonstrate the framework's sustained effectiveness over longer timeframes and across a broader range of markets.

Questions for authors

Could the authors provide a compute-matched comparison between ULTRALPHA and the main baselines in Section 4.1? This would help clarify whether the improvements mainly come from the proposed framework or from a larger research budget.

5 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

Thanks for the helpful comments! Hope the following clarifications can resolve your concerns.

Concern 1: The main experiment does not align budgets sufficiently to rule out an advantage from additional resources.

Following your suggestion, we compared ULTRALPHA with the agent baselines at approximately $30 in model cost per method to assess predictive quality under similar model expenditure. At this cost point, ULTRALPHA achieves higher Top-10 and mean Test IC than RD-Agent(Q) and AlphaAgent.

All methods use Kimi-K3 and four lanes. ULTRALPHA uses the additional Full campaign, while RD-Agent(Q) [2], AlphaAgent [1], and QuantaAlpha [3] retain their native workflows. We apply common prices to complete request prefixes from the recorded runs and evaluate candidates delivered before the cutoff using the same offline rules.

MethodModel cost / USDElapsed time / hKeptTop-10 Test ICAll-dedup Test IC
ULTRALPHA29.983.56210.03210.0359
RD-Agent(Q)30.004.02170.02250.0176
AlphaAgent29.924.67350.01660.0148
QuantaAlpha29.8719.21950.02410.0126

The CSI300 test window runs from January 2025 to September 11, 2026 and was withheld from the agents. Top-10 selects and orients candidates using 2023–2024 development IC. All-dedup averages all representatives after orientation and deduplication on 2022–2024 data using an absolute correlation threshold of 0.9. Kept counts representatives with positive Test IC. The metric is daily 24h Rank IC.

ULTRALPHA achieves 0.0321 Top-10 Test IC and 0.0359 mean Test IC, compared with 0.0225 and 0.0176 for RD-Agent(Q), and 0.0166 and 0.0148 for AlphaAgent. The quality advantage is present both among candidates selected by development performance and across the full representative set. Some baselines produce more positive representatives, so factor count and predictive quality describe different outcomes.

ULTRALPHA's recorded research time is 3.56 h. Elapsed time runs from the first accepted model request to the cost cutoff and includes API waiting, native computation, and recovery intervals. It excludes common offline evaluation. These are observed durations from separate runs, with model cost as the matched resource.

At comparable model expenditure, ULTRALPHA achieves higher Top-10 and mean Test IC than the compared baselines. This supports the proposed research process at the reported cost point and addresses the concern that its stronger signals merely reflect a larger model budget.

We have added this comparison in Appendix E.4 and referenced it in Section 4.1.

Concern 2: The performance contribution of persistent research state has not been isolated.

To address your concern, we separately removed the historical verification view and inherited notes to test how each form of persistent state contributes to later research. Verified state establishes candidate status, while model notes retain failed hypotheses, interpretations, and plans for revision.

The state and atlas ablations use Kimi-K3, four epochs, four lanes, a 64M token cap, and a development allowance of three queries per lane per epoch.

No verified-state view hides historical verification results and atlas recommendations derived from them while preserving host records, model notes, and underlying verification. No persistent memory removes inherited notes and narrative handoffs while retaining verified state, the atlas, and notes written within the current epoch. Both interventions preserve current execution feedback and the same final evaluation.

SettingKept / DedupPositive %Top-10 Test ICAll-dedup Test IC
Full21 / 21100.0%0.03210.0359
No verified-state view27 / 27100.0%0.02880.0297
No persistent memory29 / 29100.0%0.02850.0293

Removing the verified view or inherited notes lowers mean Test IC from 0.0359 to 0.0297 and 0.0293. Top-10 Test IC also falls in the main evaluation. An alternative evaluation of these campaigns changes both candidate scope and evaluation rules. It retains Full's mean-IC advantage over No verified-state view but reverses the Top-10 ordering.

Positive % means Kept/Dedup. Each setting is one campaign, and the memory intervention includes an interrupted lane in epoch 3.

Full produces fewer positive representatives but stronger average test signals. These comparisons support using both factual outcomes and research interpretations to guide continuation. Current verification remains available in every setting, so the observed quality differences concern how historical information informs subsequent search. This is the contribution of persistent state to the research protocol.

We have clarified the two forms of persistent state in Section 3.2 and added these interventions in Appendix E.2.

AuthorsOpenReview ↗

Rebuttal (2/3)

Concern 3: The performance contribution of the opportunity atlas has not been isolated.

Following your suggestion, we compared Full with No atlas guidance to test whether coverage and overlap information help the agent generate stronger candidates. The atlas informs the choice of the next research direction before a candidate is generated. It combines explored coverage, candidate overlap, and unresolved directions. The intervention hides this guidance while retaining candidate identity checks, host records, verified state, model notes, and the same final evaluation.

SettingTokens / MKept / DedupTop-10 Test ICAll-dedup Test IC
Full62.5821 / 210.03210.0359
No atlas guidance62.7419 / 190.02380.0257

With atlas guidance, Top-10 Test IC is 0.0321 versus 0.0238, and mean Test IC across all representatives is 0.0359 versus 0.0257. Top-10 selection occurs before final deduplication. Its quality difference therefore cannot be explained by a change in that final filter.

The comparison supports using search coverage to guide candidate construction. Both IC measures are higher with atlas guidance at similar token usage, while Kept changes from 21 to 19 without it. This provides evidence for the atlas's role in directing research toward stronger candidates. The intervention includes an interrupted lane in epoch 3, so the result describes this campaign.

We have added the atlas intervention and its results in Appendix E.2.

Concern 4: The benefit of parallel lanes must be separated from the additional resources they consume.

Following your suggestion, we reanalyzed W1, W2, W4, and W8 at common total token caps to compare their output under the same model budget limit. Each cap covers all lanes combined. We take complete request prefixes from the first four epochs of the historical DeepSeek runs and evaluate candidates whose development evaluations finished before each cutoff.

All widths use the same selection, deduplication, and test rules. W4 uses a different model endpoint, and development usage differs across runs. These are retrospective budget comparisons, which characterize the recorded width allocations without isolating parallelism as the cause of their differences.

Total token cap / MW1 KeptW2 KeptW4 KeptW8 Kept
1000294
20002106
40035207
582.33689287

W4 produces the most positive representatives at all four cutoffs. The table below pairs their number with mean Test IC at the largest common cap of 582.336M tokens, using candidates with completed development evaluations.

WidthKeptAll-dedup Test IC
W180.0251
W290.0284
W4280.0254
W870.0322

At this cutoff, 18 of W4's 28 retained representatives come from later epochs, whereas W8's seven all come from its first epoch. W8 has higher mean Test IC, 0.0322 versus 0.0254 for W4. Greater width therefore does not monotonically improve either output count or average quality under these common budgets.

The results show why research width must be evaluated as a resource allocation choice. ULTRALPHA distributes work across separate contexts and merges execution history for later epochs. The reanalysis reports the quantity and quality obtained from those allocations at common token limits.

We have added the budget reanalysis in Appendix E.5 and revised Section 5.1 to distinguish it from the original study with increasing total resources.

AuthorsOpenReview ↗

Rebuttal (3/3)

Concern 5: The tested horizon is insufficient to establish sustained gains over much longer research runs.

To address your concern about sustained gains, we examined the additional and cumulative factor counts across the six epochs of the original depth study. This analysis tests whether later rounds continue to add factors within the observed horizon. It uses the DeepSeek-V4.1-Flash and GLM-5.3-Flash runs, each with four lanes. The table below reproduces the submitted paper's additional and cumulative Kept counts.

EpochDeepSeek additional KeptDeepSeek cumulative KeptGLM additional KeptGLM cumulative Kept
112121212
212241123
312361134
410461145
510561156
611671268

Under this submitted readout, the final two epochs add 21 and 23 reported Kept factors, raising the cumulative totals from 46 to 67 for DeepSeek and from 45 to 68 for GLM. Each epoch contributes 10–12 reported Kept factors over the six rounds.

The trajectories document continued research and candidate production through the six epochs studied. ULTRALPHA carries recorded outcomes and unfinished directions into fresh sessions, allowing further computation to extend the recorded candidate collection. This study follows one campaign per model as total research resources grow.

We have added the breakdown by epoch in Appendix E.6 and revised Section 5.1 to limit the conclusion to the six epochs observed.

Concern 6: One crypto experiment is insufficient to establish broad generalization across markets.

To address your concern about transferability, we added a supplementary BTC comparison of a mined ULTRALPHA factor with migrated RD-Agent(Q) and QuantaAlpha baselines in a single-contract setting. Our transfer claim concerns applying the research protocol from daily equities to minute data for perpetual contracts. The added comparison provides evidence within this evaluated market setting.

The table below summarizes the displayed portfolio curves from 2020 onward, using 500,000 USDT initial capital, non-overlapping 24h holding, and 6 bps turnover costs, excluding funding payments. This history includes the training and development years and is separate from the paper's test-period evaluation.

MethodEstimated final value / USDTEstimated cumulative return
ULTRALPHA, mined factor992,000+98.4%
RD-Agent(Q)357,000−28.6%
QuantaAlpha698,000+39.6%

The mined factor finishes above both displayed baselines. This supplements the eight-contract study with a comparison against agent baselines on a single traded asset. This is an illustration within the crypto transfer study, with the generalization claim limited to the markets and frequencies evaluated.

We have added the BTC illustration in Appendix E.7 and referenced it in Section 5.2.

References

  • [1] Tang et al. AlphaAgent: LLM-Driven Alpha Mining with Regularized Exploration to Counteract Alpha Decay. KDD, 2025.
  • [2] Li et al. R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization. NeurIPs, 2025.
  • [3] Han et al. QuantaAlpha: An Evolutionary Framework for LLM-Driven Alpha Mining. arXiv:2602.07085, 2026.
  • [4] Liu et al. XAlpha: A Memory-Driven AI Quant Researcher for Hypothesis-to-Code Alpha Discovery. arXiv:2607.08332, 2026.
Reviewer_fYygOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ULTRALPHA, a long-horizon LLM-based alpha mining harness designed to address development overfitting and redundant factor discovery. It combines persistent research memory, host-verified research state, an opportunity atlas, metered development evaluation, and parallel research lanes. Experiments on CSI300 with three LLM backbones report improved factor retention and predictive performance over several baselines. Additional analyses examine width/depth scaling, model behavior, and transfer to cryptocurrency markets.

Critical strengths

The paper addresses an important problem in continual alpha mining, particularly development overfitting and redundant factor discovery.

Critical weaknesses

Insufficiently distinguished methodological novelty. Persistent research memory and redundancy-aware exploration have substantial precedents in existing alpha mining systems. Although ULTRALPHA adds verified state management and metered evaluation, the paper does not clearly establish what is methodologically new beyond these prior approaches, making the distinct contribution difficult to assess.

Questions for authors

Could the authors clarify the methodological distinctions between ULTRALPHA's memory/novelty mechanisms and related continual alpha mining approaches, especially the unique contribution of verified state management and metered evaluation? Could the authors provide controlled component-level ablations to identify which proposed mechanisms drive the improvements, ideally under comparable evaluation budgets?

4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thanks for your feedback and insightful suggestions! Hope the following responses address your concerns.

Concern 1: The methodological novelty relative to existing memory and redundancy control is unclear.

ULTRALPHA specifies which past results to rely on, where to continue research, and when to obtain another development evaluation. It allows additional model computation to support analysis and revision while development access remains explicitly budgeted.

The table below identifies shared capabilities and the research controls relevant to this distinction.

FeatureAlphaAgent [1]RD-Agent(Q) [2]QuantaAlpha [3]XAlpha [4]ULTRALPHA
Research memory✓✓✓✓✓
Redundancy control✓✓✓✓✓
Parallel exploration-✓✓-✓
Iterative continuation✓✓✓✓✓
Feedback budgeting----✓
State verification----✓
Coverage planning---✓✓

AlphaAgent [1] retains success and failure experience, RD-Agent(Q) [2] uses a knowledge forest and execution feedback, and QuantaAlpha [3] evolves evaluated trajectories. XAlpha [4] already combines verified hypotheses, feedback summaries, and coverage in planning. These provide precedents for learning from research history. ULTRALPHA specifies three linked rules governing how that history affects the next action.

  • Establish the research state. The host reconstructs candidate status from execution records across lanes. Model notes preserve explanations and revision plans, but cannot establish a pass or override recorded failures. Later rounds therefore inherit both research interpretations and a separately maintained account of what the experiments established.

  • Use that state to choose further work. Verified outcomes inform the atlas alongside coverage and overlap. A direction can have many attempts and still remain unsupported. The agent uses this distinction, together with its notes, to choose further analysis, training screens, revisions, or exploration of another direction.

  • Allocate the next evaluation. Stored results are replayed when candidate code and evaluation settings match. New development observations consume an allowance enforced by the host across lanes. The agent decides when an experiment warrants this opportunity, and the resulting execution record updates the state available to subsequent research.

Together, these rules define how autonomous research advances from recorded evidence under a separate development budget. To our knowledge, the closest systems reviewed here do not specify this joint protocol.

We have clarified these rules in Sections 3.1–3.2 and expanded the comparison with prior work in Section 6.

Concern 2: The value of verified state beyond existing research memory needs clarification.

Following your suggestion, we compared Full with No verified-state view to test whether historical verification information improves subsequent research beyond model notes and current checks. Verified state records which conclusions are supported by execution, while notes retain hypotheses, explanations, and revision plans. Completing an evaluation and passing verification have different implications for the next decision, as the case below illustrates.

Recorded itemObservation
Development execution statusAll three candidates are marked dev_completed.
Verification outcomeAll three are diagnostic-falsified. None passed development verification.
Lead for the next roundRevision suggestions are retained, while new candidates still require screening and verification.

We test the value of this distinction with No verified-state view. It hides historical verification results and atlas recommendations derived from them, while retaining model notes, current execution feedback, and underlying verification. Both settings can check new candidates. The comparison tests access to historical verification information and its derived guidance.

SettingTop-10 Test ICAll-dedup Test IC
Full0.03210.0359
No verified-state view0.02880.0297

Full has higher mean Test IC across all deduplicated representatives, 0.0359 versus 0.0297, and higher Top-10 Test IC in the main evaluation. An alternative readout changes both candidate scope and evaluation rules. It preserves Full's mean-IC advantage but reverses the Top-10 ordering.

The mean-IC results support using historical verification evidence when deciding what to investigate next. Current checks and model notes remain available in both settings, so the comparison concerns the information used to continue research. This is the role of verified state in our protocol, beyond validating each newly generated candidate.

We have clarified the roles of notes and verified state in Section 3.2 and added the ablation in Appendix E.2.

AuthorsOpenReview ↗

Rebuttal (2/2)

Concern 3: The methodological distinction and role of metered evaluation need clarification.

To address your concern, we compared the default allowance with a higher allowance for development queries at similar model token usage.

We test whether relaxing this constraint improves the resulting factors by increasing the allowance from 3 to 12 queries per lane per epoch. Feedback precision and all other components remain unchanged. Model usage is similar at 62.58M and 62.66M tokens.

SettingQuota/lane/epochTotal allowanceFresh feedbackTop-10 Dev ICTop-10 Test ICDev − Test
Full348440.05380.03210.0217
Higher dev quota12192930.06710.02400.0431

The larger allowance raises actual queries from 44 to 93 and Top-10 Dev IC from 0.0538 to 0.0671, but Test IC falls from 0.0321 to 0.0240. The gap between development and test IC nearly doubles. Mean Test IC across all deduplicated representatives also falls, from 0.0359 to 0.0200.

Full obtains stronger test signals with fewer development evaluations at similar model usage. This supports treating development access as a research decision with its own budget. ULTRALPHA allows the agent to continue working on a hypothesis before spending another evaluation opportunity.

We have clarified feedback allocation in Section 3.1 and added the quota comparison in Appendix E.3.

Concern 4: Controlled component ablations under comparable evaluation budgets are needed.

Following your suggestion, we added four component interventions to test how historical information, search guidance, and feedback allowance affect output under common final evaluation. All five settings use Kimi-K3, four epochs, four lanes, and a 64M token cap. The three information interventions retain the same development allowance. Higher dev quota changes that allowance while keeping model computation similar.

No atlas guidance hides coverage, overlap, and opportunity guidance while retaining candidate identity checks, host records, verified state, model notes, and final deduplication. No persistent memory removes inherited notes and narrative handoffs across epochs while retaining verified state, the atlas, and notes written within the current epoch.

SettingTokens / MKept / DedupPositive %Top-10 Dev ICTop-10 Test ICAll-dedup Test IC
Full62.5821 / 21100.0%0.05380.03210.0359
No atlas guidance62.7419 / 19100.0%0.05320.02380.0257
No verified-state view62.7427 / 27100.0%0.05620.02880.0297
No persistent memory62.6629 / 29100.0%0.05710.02850.0293
Higher dev quota62.6654 / 5598.2%0.06710.02400.0200

Top-10 selects and orients candidates using 2023–2024 development IC before deduplication. All-dedup uses 2022–2024 development data for direction and deduplication using an absolute correlation threshold of 0.9, then averages every representative, including negative IC. Kept counts positive representatives, and Positive % is Kept/Dedup. These CSI300 results use daily 24h Rank IC on January 2025–September 11, 2026, withheld from the agents.

Each setting is one campaign. No atlas guidance and No persistent memory each include an interrupted lane in epoch 3.

Full achieves the highest Top-10 Test IC of 0.0321 and mean Test IC across all representatives of 0.0359 in the main evaluation. Removing atlas guidance, the verified view, or inherited notes lowers both. Both the Top-10 selected on development data and the full representative set show a quality difference.

Several interventions produce more positive representatives but lower mean Test IC. Full's quality advantage persists under common final scoring rules at similar model usage. The interventions therefore provide evidence that historical information and feedback allocation contribute to the quality of candidates produced during search.

We have added the evaluation protocol and component results in Appendices E.1–E.2, with a reference in Section 4.1.

References

  • [1] Tang et al. AlphaAgent: LLM-Driven Alpha Mining with Regularized Exploration to Counteract Alpha Decay. KDD, 2025.
  • [2] Li et al. R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization. NeurIPs, 2025.
  • [3] Han et al. QuantaAlpha: An Evolutionary Framework for LLM-Driven Alpha Mining. arXiv:2602.07085, 2026.
  • [4] Liu et al. XAlpha: A Memory-Driven AI Quant Researcher for Hypothesis-to-Code Alpha Discovery. arXiv:2607.08332, 2026.

Dear Reviewer fYyg,

Thanks for your constructive review.

We hope our responses have addressed your concerns on methodological novelty and component contributions. In particular, we clarified that Ultralpha's distinction lies in the joint protocol of host-verified research state and metered development feedback, and we added controlled ablations isolating the atlas, verified state, persistent memory, and the dev quota.

With the rebuttal deadline approaching, we would be glad to clarify any remaining questions or provide additional experiments.

Best regards,

The Authors of Ultralpha

Dear Reviewer fYyg,

Thanks again for your constructive review. Since the rebuttal will close in about two hours, please let us know if you have any remaining questions or concerns. We would be happy to address them.

Best,

The Authors of Ultralpha

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to Area Chair and All Reviewers

We thank the reviewers for their thoughtful comments. Their shared concerns center on methodological novelty, component contributions, and research budget comparability. Below, we summarize the additional evidence and identify where the corresponding changes appear in the revised manuscript.

Concern 1: The methodological contribution beyond existing memory and redundancy control needs clarification. (Reviewers fYyg and WrMe)

ULTRALPHA couples research state reconstructed from execution records with a separate budget for new development evaluations across rounds and parallel lanes. Existing systems already provide research memory, redundancy control, and iterative exploration. Our contribution specifies how accumulated evidence and feedback access jointly govern subsequent research decisions.

The host establishes which candidates passed, failed, or remain untested, while model notes preserve explanations and revision plans. Recorded outcomes inform the opportunity atlas alongside coverage and overlap. The agent uses this history to choose further analysis, revision, another direction, or a development evaluation. Stored results can be reused when candidate code and evaluation settings match. New evaluations consume the shared allowance, so additional model computation need not automatically increase exposure to development feedback.

The novelty lies in connecting verified continuation with the allocation of new evaluations. The individual responses compare these operations with AlphaAgent, RD-Agent(Q), QuantaAlpha, and XAlpha.

We have clarified feedback allocation in the “Confirmation under budget” paragraph of Section 3.1, distinguished model notes from verified state in Section 3.2, and expanded the comparison with prior work in Section 6.

Concern 2: Controlled ablations are needed to establish the contribution of each component. (Reviewers fYyg, GpJd, and WrMe)

Following your suggestion, we added four interventions to test how research guidance, historical information, and feedback allowance affect the resulting factors. All settings use Kimi-K3, four epochs, four lanes, a 64M token cap, and common final evaluation. The three information interventions retain the same development allowance. Higher dev quota increases it from 3 to 12 queries per lane per epoch while preserving feedback precision.

SettingTokens / MKept / DedupPositive %Top-10 Dev ICTop-10 Test ICAll-dedup Test IC
Full62.5821 / 21100.0%0.05380.03210.0359
No atlas guidance62.7419 / 19100.0%0.05320.02380.0257
No verified-state view62.7427 / 27100.0%0.05620.02880.0297
No persistent memory62.6629 / 29100.0%0.05710.02850.0293
Higher dev quota62.6654 / 5598.2%0.06710.02400.0200

No atlas guidance hides coverage and overlap guidance. No verified-state view hides historical verification outcomes and the atlas recommendations derived from them. No persistent memory removes inherited notes and narrative handoffs while retaining verified state and the atlas. Current execution feedback and underlying verification remain available in every setting.

These CSI300 comparisons use daily 24h Rank IC on January 2025–September 11, 2026, withheld from the agents. Top-10 candidates are selected and oriented on 2023–2024 development data before deduplication. All-dedup averages every representative, including those with negative Test IC, after orientation and deduplication on 2022–2024 data using an absolute correlation threshold of 0.9. Positive % is Kept divided by Dedup.

Full has the highest Top-10 and All-dedup Test IC in the main evaluation. Removing atlas guidance, the verified view, or inherited notes lowers both. Since final evaluation is common, the comparisons support the contribution of historical information and search guidance to the quality of generated candidates. Each intervention tests a component with the others retained.

At nearly identical token usage, the higher development allowance increases actual new evaluations from 44 to 93. Top-10 Dev IC rises from 0.0538 to 0.0671, while Test IC falls from 0.0321 to 0.0240. This supports managing development access separately from model computation. Several interventions produce more positive representatives but lower average IC, so we report factor quantity alongside predictive quality.

We have added the evaluation protocol and component results in Appendices E.1–E.2, and the detailed quota comparison in Appendix E.3. Section 4.1 now points to the component experiments, and Section 3.1 clarifies the separate control of model computation and development feedback.

AuthorsOpenReview ↗

General Response to Area Chair and All Reviewers

Concern 3: Research quality and scaling should be evaluated under comparable budgets. (Reviewers GpJd and WrMe)

To address your concern, we compared methods at approximately $30 in model expenditure using the same Kimi-K3 backbone and four lanes. A common price schedule determines complete request prefixes from recorded runs. Baselines retain their native workflows, and candidates delivered before the cutoff receive the same offline evaluation as above. ULTRALPHA uses the additional Full campaign.

MethodModel cost / USDElapsed time / hKeptTop-10 Test ICAll-dedup Test IC
ULTRALPHA29.983.56210.03210.0359
RD-Agent(Q)30.004.02170.02250.0176
AlphaAgent29.924.67350.01660.0148
QuantaAlpha29.8719.21950.02410.0126

ULTRALPHA achieves 0.0321 Top-10 Test IC and 0.0359 All-dedup Test IC at this expenditure. Kept per dollar is 0.70 for ULTRALPHA, 0.57 for RD-Agent(Q), 1.17 for AlphaAgent, and 3.18 for QuantaAlpha. The comparison shows stronger average test signals for ULTRALPHA, while AlphaAgent and QuantaAlpha produce more positive representatives per dollar.

Elapsed time measures recorded research from the first accepted model request to the cost cutoff and excludes common offline evaluation. These are separate observed runs, with model cost as the matched resource.

We also reanalyzed the historical DeepSeek widths at common total token caps across all lanes. The table below counts representatives from candidates whose development evaluations finished before each cutoff.

Total token cap / MW1 KeptW2 KeptW4 KeptW8 Kept
1000294
20002106
40035207
582.33689287

Greater width does not monotonically increase factor yield under a common token budget. At the largest cutoff, W4 retains 28 factors and W8 retains seven, while W8 has higher mean Test IC, 0.0322 versus 0.0254. W4 uses a different model endpoint, and development usage differs across runs. The analysis therefore describes the recorded resource allocations without isolating a causal effect of parallelism.

These comparisons separate predictive quality at a common model cost from factor yield under different allocations of a fixed token budget. They support the reported quality results and make the scope of the scaling evidence explicit.

We have added the cost comparison in Appendix E.4, referenced from Section 4.1. The width reanalysis appears in Appendix E.5, and Section 5.1 now distinguishes it from the original experiments in which total research resources grow with width and depth.

Area_Chair_NAxfOpenReview ↗

Engaging with the authors

Dear reviewers,

The authors have posted several replies to your review. Please have a look and interact with the authors as needed. Reviewer WrMe has left a comment and how about Reviewer fYyg and GpJd?

Thank you,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.