- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper replaces the API-based Planner and Realizer of an LLM alpha-mining loop with locally post-trained Qwen3.8-27B policies. The Planner proposes plans from a fixed 38-mechanism catalogue, the Realizer writes two structurally different factor specifications per plan, and a LightGBM predictor decides which factors are retained. The agents are trained by SFT on selected traces from 72 API research loops, then by joint GRPO with a shared reward U = 0.75Q + 0.25D, where Q is the predictive score gain and D rewards new directions of prediction change relative to earlier research; Planner and Realizer receive separate advantages. Research uses May 2022–Dec 2025 and the frozen model is evaluated once on Jan–Aug 2026 across four Chinese stock universes, where the method reports the best predictive metrics on CSI300 and the highest number of useful mechanisms in all four markets.
Critical strengths
-
A well-chosen diversity objective. Rewarding new directions of prediction change, net of what earlier research already covers and calibrated against noise factors, targets repeated exploration of the same mechanism more directly than formula-level deduplication. Removing it lowers useful coverage (Table 4: U 23 → 18 on CSI300).
-
A sensible evaluation protocol. Research feedback is confined to the inner period, the outer period is evaluated once, and agentic baselines share data and a matched 160-slot budget. The authors also report results that do not favor them (e.g., GLM-5.3 in Table 19).
Critical weaknesses
W1. The performance gains are not statistically established.
Each configuration is run once on an outer window of about eight months (~160 trading days). From the paper's own CSI300 numbers, the standard error of the mean daily IC is about 0.014 (0.0378 / 0.2128 / √160), while the IC gap to QuantaAlpha is 0.0038; the standard error of an annualized IR over this window is roughly 1.3, while the IR gap is 0.23, and QuantaAlpha has the higher ARR. In addition, the three intermediate variants in Table 23 improve by nearly equal steps in every market and on many metrics at once (e.g., CSI300 ARR 32.57 → 35.17 → 37.77; CSI1000 IR 0.492 → 0.677 → 0.862; A-share MDD 7.38 → 6.67 → 5.96), which is unusual for separately trained policies evaluated by independent backtests. (Q1, Q2)
W2. The comparisons do not isolate the value of the mining process or its practical value.
The ML/DL baselines receive only the 21 base factors (C.3), while the agents can use V, which includes Alpha158 and minute-level features (C.1). The gap to these baselines therefore largely reflects access to more information; notably, on A-share, Ridge and Transformer on the 21 factors already exceed the proposed method in RIC (0.0409 and 0.0405 vs. 0.0392). Separately, all returns are gross of costs for a daily-rebalanced portfolio with no turnover reported (C.2), so tradability cannot be judged, and the confidentiality motivation is weakened by the fact that the SFT traces were generated by external APIs on the same environments. (Q3, Q4, Q5)
Questions for authors
Q1. [Clarification] Table 23.
Is each of the "Planner only", "Realizer only" and "w/o diversity reward" rows an independent GRPO run followed by its own 20-round loop and outer evaluation? Please explain the near-constant increments and provide the daily outer prediction and return series for these variants.
Q2. [Requested analysis] Run-to-run variance.
Table 6 shows a full run costs $2–7 in API fees or about 1.4 H200-hours. Could you report 3 seeds of the proposed method and QuantaAlpha on CSI300 and A-share (mean ± std), with a paired test on the daily IC difference (e.g., Newey–West t-statistic)?
Q3. [Requested analysis] Zero-agent baseline.
What does the same LightGBM pipeline achieve when given B ∪ V directly, without any mining? This needs one fit per market and shows how much the 20 research rounds add beyond the features already available to the agents.
Q4. [Requested analysis] Tradability.
Please report average daily turnover of the top-quintile portfolio and the break-even one-way cost at which ARR falls to zero, for the proposed method and the strongest baselines. Both can be computed from existing backtests.
Q5. [Clarification] Confidentiality claim.
Since the SFT traces come from GPT-5.5, Grok-4.6 and GLM-5.3 running on the same data, can the local agents be trained without external APIs? If not, should the claim be limited to inference-time confidentiality?