acceptodds
Withdrawn from ICLR 2027

ParaFusionBench: A Unified Benchmark for Cross-Paradigm Fusion in Factor Mining and Trading Decision-Making

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Quantitative investment research has evolved from conventional models to large language models (LLMs) and multi-agent collaboration, forming two primary research trajectories: factor mining and trading decision-making. Their intersection yields four paradigms: Conventional Factor Mining (CFM), LLM-driven Factor Mining (LFM), Conventional Trading Decision (CTD), and LLM-driven Trading Decision (LTD). Existing works mostly develop individual paradigms in isolation, lacking unified formal characterization and cross-paradigm fusion evaluation. Current benchmarks similarly tend to focus on a single paradigm or task, limiting themselves to model ranking while offering little insight into paradigm fusion. To address this gap, we propose ParaFusionBench, a cross-paradigm fusion benchmark, and construct a quantitative paradigm fusion framework, including formal fusion definitions, a seven-dimensional feasibility criterion, fusion injection methods, candidate fusion screening methods, and fusion effect evaluation metrics. It designs the EDD three-layer evaluation system to guide multi-dimensional evaluation of different paradigm instances. Experimental validation shows that the complementary strengths and weaknesses of individual paradigms provide feasible conditions for paradigm fusion. 83% of fusion combinations help reduce drawdown, and 66.7% and 58.3% of combinations achieve Sharpe and ARR gains, respectively. ParaFusionBench also identifies effective combinations such as AlphaGen + TradingAgents and AlphaProbe + FinMem, demonstrating its ability to uncover beneficial cross-paradigm compositions. These results further provide practical insights into when and how heterogeneous quantitative paradigms can be effectively integrated. Our code is available at https://anonymous.4open.science/r/ParaFusionBench-301F.

Primary area:
datasets and benchmarks
Keywords:
AI-driven Quantitative Investment, LLM Agents, Factor Mining, Trading Decision-Making, Benchmark, Cross-Paradigm Fusion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LbqDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ParaFusionBench organizes quantitative investment methods along two dimensions: factor mining versus trading decisions, and conventional methods versus LLM methods. It proposes a fusion framework, the FUSE-7 feasibility criteria, the EDD evaluation system, and TriSift candidate screening.

The experiments evaluate 11 systems and 12 factor-source/decision-system combinations. Factors enter conventional traders through state augmentation and LLM traders through prompts. The reported results show substantial variation across combinations: some improve performance considerably, while others cause severe degradation.

Critical strengths

The integration problem is consequential, and the benchmark provides useful infrastructure. Evaluating factor discovery and downstream decision-making together can reveal limitations that isolated factor-quality or trading benchmarks miss. Appendix G documents shared adapters, budgets, validation-based factor selection, and point-in-time constituent filtering. These are useful foundations for reproducible comparisons.

The paper reports failures and provides an informative injection ablation. TradingAgents benefits from some factor sources but deteriorates sharply with others, while FinMem improves across the tested sources. The explanation/constraint ablation also helps distinguish alternative integration choices within one setup. Reporting this heterogeneity is more informative than presenting only successful combinations.

Critical weaknesses
  1. The experiments do not sufficiently isolate the value of fusion or validate the discovery procedure.

    The “values-only” injection already includes a frozen LightGBM directional forecast. All main fusion results additionally use GOV, which constrains forecast-inconsistent decisions, scales positions, and introduces a weak-signal holding rule. Comparing this package against the native downstream trader therefore mixes several changes: additional predictive information, trading constraints, and interaction with the downstream system.

    The most important missing baseline is direct execution of the identical forecast under the identical governor, without the LLM trader. Equivalent risk-control baselines and matched forecast horizons are also needed. Table 6’s VAL-versus-GOV comparison is useful, but does not establish whether the downstream agent adds value beyond the predictor and rules.

    TriSift’s effectiveness is similarly unproven. Its pair score averages independent source and consumer rankings, without a pair-specific compatibility term. Table 15 ranks AlphaProbe + TradingAgents fifth, although it produces ΔSharpe −3.193; AlphaProbe + FinMem ranks twenty-third, yet produces the largest positive ΔSharpe, +1.264. These examples do not alone invalidate screening, but a comparison against random selection or simpler selection rules is needed to support its claimed discovery value.

  2. Protocol ambiguities and limited validation weaken confidence in the benchmark’s conclusions.

    There are material inconsistencies in the execution description. Appendix G, Table 17 specifies same-day closing settlement and score-proportional, dollar-neutral long–short weights. Appendix H, Figures 6–10 instead describe next-day settlement, sign-only positions, and a long-only FinRL environment. These differences affect exposure, turnover, and returns. The paper needs to identify precisely which executor generated each main table.

    Candidate-selection timing also needs clarification. Section 4.2.2 describes selecting candidates using preceding experimental results; RQ1 evaluates full-year 2025, while fusion evaluates H1 2025. Appendix G separately states that 2024 validation supports model selection. The authors should establish whether every TriSift ranking was frozen using validation data, independently of the overlapping test period. The current text leaves this unresolved.

    Finally, fusion is evaluated on five stocks over six months, and the ablation uses one stock and one specified seed. This supports observations about particular configurations, but provides limited evidence for general claims such as “compatibility is more important than factor quality.” Uncertainty estimates and broader held-out evaluation are missing.

Questions for authors

Which execution rules generated Tables 1, 5, and 6? Please reconcile the settlement, sizing, and portfolio descriptions, including when inputs become available relative to execution.

How do the fusion systems compare against the same h=1 LightGBM forecast executed directly with the same GOV rules, calendar, costs, and exposure?

Were all TriSift rankings computed exclusively from 2024 validation data? Does TriSift outperform random or simpler selection under the same evaluation budget?

Do the principal findings persist across additional held-out periods and repeated runs, with uncertainty estimates?

Reviewer_JWjPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper groups quantitative research into four paradigms by crossing factor mining versus trading decisions with conventional models versus LLMs. It proposes ParaFusionBench to evaluate instances of these paradigms and to test fusions in which a mined factor source is injected into a downstream trading agent. The framework adds a seven-item feasibility checklist, three injection styles, a three-stage screen called TriSift, and a three-layer metric system. Experiments cover eleven open-source systems. Twelve fusions are run, mostly by placing factor output into FinRL-PPO, TradingAgents, or FinMem. The abstract reports that 83% of fusions reduce drawdown and that about two thirds improve Sharpe.

Critical strengths

1.The organizing question is reasonable. Factor-mining papers and trading-agent papers are usually evaluated apart, so a shared harness that asks whether a factor source helps a decision agent is a useful benchmark goal. The anonymous code release matches that goal.

2.The paper does not hide failed fusions. AlphaProbe with TradingAgents and AlphaAgent with TradingAgents are large negative transfers, and the text states that complementary strengths do not guarantee a gain.

3.The injection ablation asks the right mechanistic question. It separates raw factor values, added economic explanations, and a rule-based governor, and it varies factor count and factor type. That is more informative than a single fused headline number.

Critical weaknesses

1.The reported fusion gains are not gains over a serious alternative. In the metric appendix, Sharpe and ARR deltas are the fused system minus the downstream agent alone. AlphaProbe alone is far stronger in Table 5 than AlphaProbe combined with FinMem, yet that combination is still counted as a success because FinMem alone is near zero. The abstract’s 83%, 67%, and 58% are counts over twelve such contrasts, with no uncertainty. Those twelve runs use five stocks and only the first half of 2025. The ablation that reaches a Sharpe above 4 is narrower still: GOOGL alone, for about 120 days.

2.The returns are not shown to come from the factors. In the ablation, the injected signals have Rank IC near zero, and one fusion block in Table 5 assigns AlphaGen a negative Rank IC on every downstream despite a positive standalone Rank IC. Values-only injection already lifts the FinMem baseline from a negative Sharpe to about 3. The governor adds a dead zone, position scaling, and a penalty on unjustified reversals. Without a shuffled-factor or random-score control under the same rules, the experiment does not separate factor information from prompt and position constraints.

3.The sample cannot support the economic claims. Table 1 and Table 5 give sharply different ARR and Sharpe figures for AlphaProbe, TradingAgents, and FinMem. The appendix places the fusion on the first half of 2025 and the broader comparison on a longer window, but the main text discusses both as evidence about the same systems. TradingAgents is credited with a bear-market Sharpe above 6 on 37 days. Five correlated mega-cap names, daily rebalancing, and a one-basis-point cost make annualized returns above 100% too fragile to establish a fusion principle.

4.The screening procedure does not identify successful fusions. AlphaProbe with TradingAgents sits near the top of the TriSift ranking and produces the worst transfer in the study. AlphaProbe with FinMem sits near the bottom and is presented as a discovery. Standalone ranks therefore neither select compatible pairs nor avoid negative transfer. The seven feasibility scores are author-assigned ratings, and the joint objective written into the formal definition is not the objective optimized in the experiments.

Questions for authors

1.Under the same governor, what happens if the injected values are shuffled across days, or replaced by constant scores?

2.Were the 2025 decisions produced by a model whose pretraining cutoff precedes 2025? If not, which parts of the input were restricted to point-in-time records?

3.Why does TriSift rank the worst transfer near the top and a reported success near the bottom? Was the ranking fixed before the twelve fusion runs?

1 comment

As author do not have a reasonable response, so I will maintain my score.

Reviewer_nuiGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ParaFusionBench, a benchmark and framework for cross paradigm fusion in quantitative trading, where factors produced by one paradigm, conventional or LLM driven factor mining, are injected into trading decision methods from another paradigm, conventional or LLM driven. It defines four paradigms, a seven dimensional feasibility checklist for fusion, a three layer evaluation system, and a screening procedure, then evaluates eleven open source paradigm instances on US market data and reports fusion effects across twelve factor strategy combinations, claiming that most combinations reduce drawdown and that more than half improve Sharpe ratio and annualized return.

Critical strengths

The motivating observation is reasonable: factor mining and trading decision research have largely evolved separately, and a systematic comparison of which factor sources help which decision methods has practical value. The paper also assembles a broad set of recent open source systems and reports their economic and efficiency metrics side by side, which saves readers some integration effort, and the three injection forms it compares, values only, values with explanations, and values with rule based decision constraints, are a sensible design axis for LLM decision stages.

Critical weaknesses

The empirical base cannot support any of the headline claims. The entire study runs on five US mega cap stocks in a single backtest window favorable enough that a buy and hold baseline already earns about 34 percent annualized, regime level Sharpe ratios are computed on samples as small as 37 bear market days, and there is no significance testing, no confidence interval, no out of sample evaluation, and no transaction cost accounting anywhere in the paper. Statements such as 83 percent of fusion combinations reducing drawdown are counts over twelve hand picked pairs, and Table 5 reports fused pipelines achieving annualized returns above 100 percent while their rank correlations are negative, which in any serious empirical finance venue would trigger a robustness demand rather than a celebration. The conceptual framework adds little beyond relabeling the act of feeding factor values into a strategy input as paradigm fusion through a bridge mapping, and the seven dimensional feasibility criterion is an author scored checklist with no validation. There are also clear signs of careless generation in the text, including duplicated sentences and a misspelled figure label, which sits awkwardly next to an AI use statement claiming generative tools only assisted with interpretation and polishing. Practical advice such as preferring rule guided injection and a moderate number of factors comes from ablations on a single fusion pair under the same five stock setup, so none of the conclusions about when and how to fuse survive contact with the data scale. I recommend clear rejection.

Questions for authors

None. No robustness claim in the current setup would change my assessment; the study needs a broader asset universe, multiple market periods, and proper statistical testing before its empirical questions become meaningful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.