- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FinPILOT adapts a pretrained trading actor at each decision using imagined returns from an independently trained forecaster. Under the price-taker assumption, forecasts supply market trajectories and portfolio accounting supplies rewards. The paper evaluates five RL algorithms on equities and FX over three rolling test years, and studies sensitivity to forecast quality.
Critical strengths
- The separation of market forecasting from portfolio accounting gives a clear way to use forecasts with an existing policy. The method does not require an action-conditioned market model or changes to the original RL training procedure.
- The paired evaluation is substantial: five algorithms, two markets, three test years, and five seeds per setting. Returns improve in 28 of 30 settings and Sharpe in 27 of 30. The paper also reports adverse drawdown results and clearly separates the oracle forecast diagnostic from deployable results.
Critical weaknesses
- The comparison with direct portfolio optimization does not isolate the benefit of optimizing policy parameters. The MPO baseline in Equation (13) uses a different risk objective and ignores within-horizon weight drift, whereas FinPILOT uses scenario-based downside risk, compounded wealth, and a terminal critic. Receiving the same forecasts does not control these differences. The results establish gains over frozen policies, but leave the explanation for the policy-versus-action optimization gap unresolved.
- The implemented update differs from the gradient of the stated objective. Section 3.1 stops gradients through the particle mean inside the downside term. That mean depends on the actor, and its derivative does not generally cancel for downside variance. This may be a useful approximation, but the paper neither justifies it nor measures its effect. It is therefore unclear how much of the reported behavior comes from optimizing Equation (2) versus this modified update.
Questions for authors
- How does FinPILOT compare with direct action-sequence optimization using the same scenarios, accounting, risk term, and terminal value, initialized from the pretrained policy's actions?
- Why is the mean detached inside the downside penalty? Do the gains persist when differentiating through that mean, with the other adaptation settings unchanged?
2 comments
Dear Reviewer EmBi,
Thank you again for your thoughtful review. We wanted to briefly follow up before the discussion period closes on November 18, particularly regarding the two experiments you suggested.
First, we added a direct action-sequence optimization baseline using FinPILOT's forecast scenarios, portfolio accounting, downside-risk objective, and optimization budget, initialized from either the pre-trained policy or equal weights. At its best tested step size, the action optimizer achieves −0.03 mean Sharpe from the pre-trained start, compared with +0.30 for FinPILOT (+0.02 versus +0.34 from equal weights). We also clarified the remaining distinction between open-loop action optimization and state-conditioned policy optimization (Section 4.3, Appendix D.5).
Second, we derived the effect of detaching the particle mean and reran FinPILOT with the full gradient. The full-gradient variant retains 88% of the Sharpe improvement and improves performance in 26 of 30 settings (Appendices D.8 and E).
We hope these additional results help address your concerns. If you have a chance to take another look, we would really appreciate hearing whether anything remains unclear or insufficiently addressed.
Thank you again for your time!
Response to Reviewer EmBi (1/1)
Thank you for the constructive questions. We agree that the original comparison with Boyd SPO and MPO did not fully isolate the effect of adapting policy parameters. We have added a more controlled action-sequence baseline and an ablation of the detached-mean gradient.
Q1. Direct action-sequence optimization
We implemented an action-sequence optimizer using FinPILOT's forecast scenarios, portfolio accounting (including weight drift, compounding, and transaction costs), downside-risk objective, and optimization budget (Section 4.3, Table 4; Appendix D.5). Instead of adapting the actor, it optimizes an H-step sequence of portfolio actions shared across the K forecast scenarios. Both methods use E=2 Adam updates per decision.
Following your suggestion, we initialize the sequence from the pre-trained policy's actions. We also test equal-weight initialization. The sequence is shifted forward to warm-start the next decision, analogous to carrying FinPILOT's updated parameters forward. We use the terminal critic where applicable. For SAC, TD3, and DDPG, the action-sequence baseline evaluates the critic using the pre-trained actor's terminal action, since the sequence does not specify an action beyond its horizon.
These are mean changes in Sharpe relative to the frozen policy over 30 settings (± pooled seed SD of paired differences); the counts are market–years with improved mean Sharpe.
We swept the action optimizer's step size and report its best tested value, α=0.001 (Table 19). Larger step sizes performed worse. Even at this setting, policy optimization has an average Sharpe advantage of 0.32 from either initialization, with a positive average gap for all five algorithms (Table 20). Unlike Boyd SPO and MPO on FX, these matched action optimizers remain invested (mean cash below 8%). Removing FinPILOT's terminal critic also preserves 91% of its Sharpe improvement (Appendix D.6).
These results support an advantage over the matched-objective open-loop action-sequence optimizer we evaluated. An important remaining difference is the planning representation: FinPILOT produces state-conditioned allocations across imagined scenarios, whereas the direct optimizer uses a shared open-loop action sequence. Section 4.3 now notes this as one possible explanation for the gap.
Q2. Detaching the particle mean
We agree that detaching the mean changes the gradient of the stated objective. We added a derivation (Appendix E) and reran FinPILOT with the full gradient (Appendix D.8, Table 27).
Let dk=Jt(k)−Jˉt and
ρt=D^t+ϵK−1∑k:dk<0∣dk∣.
The gradients satisfy
gdet(λ)=(1+ρtλ)gfull(1+ρtλλ).
Thus, at a given update, the detached gradient is proportional to the exact gradient evaluated at a slightly smaller effective risk weight. The observed median ρt is 0.58 (5th–95th percentile: 0.54–0.61), corresponding to an effective risk weight 3–8% below nominal λ for our grid {0.05,0.10,0.15}. This is a local gradient relationship, not a claim that the full Adam trajectories are identical.
With all other settings unchanged, the full-gradient variant achieves +0.26 ± 0.25 mean ΔSharpe versus +0.30 ± 0.28 with detachment, retaining 88% of the gain. It improves in 26/30 settings and all 6 market–years. The difference from the detached variant ranges from 0.00 for SAC to −0.13 for DDPG (Table 27). We now report and discuss this approximation in Sections 3.1 and 4.5.