- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
ParaFusionBench organizes quantitative investment methods along two dimensions: factor mining versus trading decisions, and conventional methods versus LLM methods. It proposes a fusion framework, the FUSE-7 feasibility criteria, the EDD evaluation system, and TriSift candidate screening.
The experiments evaluate 11 systems and 12 factor-source/decision-system combinations. Factors enter conventional traders through state augmentation and LLM traders through prompts. The reported results show substantial variation across combinations: some improve performance considerably, while others cause severe degradation.
Critical strengths
The integration problem is consequential, and the benchmark provides useful infrastructure. Evaluating factor discovery and downstream decision-making together can reveal limitations that isolated factor-quality or trading benchmarks miss. Appendix G documents shared adapters, budgets, validation-based factor selection, and point-in-time constituent filtering. These are useful foundations for reproducible comparisons.
The paper reports failures and provides an informative injection ablation. TradingAgents benefits from some factor sources but deteriorates sharply with others, while FinMem improves across the tested sources. The explanation/constraint ablation also helps distinguish alternative integration choices within one setup. Reporting this heterogeneity is more informative than presenting only successful combinations.
Critical weaknesses
-
The experiments do not sufficiently isolate the value of fusion or validate the discovery procedure.
The “values-only” injection already includes a frozen LightGBM directional forecast. All main fusion results additionally use GOV, which constrains forecast-inconsistent decisions, scales positions, and introduces a weak-signal holding rule. Comparing this package against the native downstream trader therefore mixes several changes: additional predictive information, trading constraints, and interaction with the downstream system.
The most important missing baseline is direct execution of the identical forecast under the identical governor, without the LLM trader. Equivalent risk-control baselines and matched forecast horizons are also needed. Table 6’s VAL-versus-GOV comparison is useful, but does not establish whether the downstream agent adds value beyond the predictor and rules.
TriSift’s effectiveness is similarly unproven. Its pair score averages independent source and consumer rankings, without a pair-specific compatibility term. Table 15 ranks AlphaProbe + TradingAgents fifth, although it produces ΔSharpe −3.193; AlphaProbe + FinMem ranks twenty-third, yet produces the largest positive ΔSharpe, +1.264. These examples do not alone invalidate screening, but a comparison against random selection or simpler selection rules is needed to support its claimed discovery value.
-
Protocol ambiguities and limited validation weaken confidence in the benchmark’s conclusions.
There are material inconsistencies in the execution description. Appendix G, Table 17 specifies same-day closing settlement and score-proportional, dollar-neutral long–short weights. Appendix H, Figures 6–10 instead describe next-day settlement, sign-only positions, and a long-only FinRL environment. These differences affect exposure, turnover, and returns. The paper needs to identify precisely which executor generated each main table.
Candidate-selection timing also needs clarification. Section 4.2.2 describes selecting candidates using preceding experimental results; RQ1 evaluates full-year 2025, while fusion evaluates H1 2025. Appendix G separately states that 2024 validation supports model selection. The authors should establish whether every TriSift ranking was frozen using validation data, independently of the overlapping test period. The current text leaves this unresolved.
Finally, fusion is evaluated on five stocks over six months, and the ablation uses one stock and one specified seed. This supports observations about particular configurations, but provides limited evidence for general claims such as “compatibility is more important than factor quality.” Uncertainty estimates and broader held-out evaluation are missing.
Questions for authors
Which execution rules generated Tables 1, 5, and 6? Please reconcile the settlement, sizing, and portfolio descriptions, including when inputs become available relative to execution.
How do the fusion systems compare against the same h=1 LightGBM forecast executed directly with the same GOV rules, calendar, costs, and exposure?
Were all TriSift rankings computed exclusively from 2024 validation data? Does TriSift outperform random or simpler selection under the same evaluation budget?
Do the principal findings persist across additional held-out periods and repeated runs, with uncertainty estimates?
As author do not have a reasonable response, so I will maintain my score.