acceptodds
Under review as a conference paper at ICLR 2027

Plan Before You Trade: Inference-Time Optimization for RL Trading Agents

Abstract

Reinforcement learning agents for portfolio management are typically trained and deployed as static policies, with no mechanism for policy adjustments using price forecasts at inference time. We propose FinPILOT (Financial Plugin Inference-time Learning for Optimal Trading), a plugin inference-time policy optimization framework inspired by Model Predictive Control (MPC). Under the price-taker assumption, portfolio actions do not affect future market prices, allowing a multi-step price forecaster to provide imagined market trajectories without iterative action-conditioned rollouts. At each decision step, FinPILOT uses the forecaster's predicted price trajectory to construct an allocation-based imagined return objective, and optimizes the pre-trained policy at inference-time before executing one step of the trade. Our framework is compatible with any pre-trained policy and adapts the policy using the forecaster's predictions without modifying the original RL training pipeline. Evaluated with five pre-trained policies (PPO, SAC, A2C, TD3, DDPG) across DJ30 equities and foreign exchange (FX) over three rolling held-out test years (30 settings in total), FinPILOT improves Total Return in 28 of 30 settings and Sharpe ratio in 27 of 30, with gains spanning both stochastic and deterministic policies. Direct single- and multi-period portfolio optimizers supplied with the same forecasts do not reproduce these aggregate gains under our evaluation protocol. Further, using controlled forecasts at calibrated quality levels, we show that both the magnitude and consistency of FinPILOT's performance increase as forecast error is reduced.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
portfolio management, stock trading, policy optimization, model predictive control, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EmBiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FinPILOT adapts a pretrained trading actor at each decision using imagined returns from an independently trained forecaster. Under the price-taker assumption, forecasts supply market trajectories and portfolio accounting supplies rewards. The paper evaluates five RL algorithms on equities and FX over three rolling test years, and studies sensitivity to forecast quality.

Critical strengths
  1. The separation of market forecasting from portfolio accounting gives a clear way to use forecasts with an existing policy. The method does not require an action-conditioned market model or changes to the original RL training procedure.
  2. The paired evaluation is substantial: five algorithms, two markets, three test years, and five seeds per setting. Returns improve in 28 of 30 settings and Sharpe in 27 of 30. The paper also reports adverse drawdown results and clearly separates the oracle forecast diagnostic from deployable results.
Critical weaknesses
  1. The comparison with direct portfolio optimization does not isolate the benefit of optimizing policy parameters. The MPO baseline in Equation (13) uses a different risk objective and ignores within-horizon weight drift, whereas FinPILOT uses scenario-based downside risk, compounded wealth, and a terminal critic. Receiving the same forecasts does not control these differences. The results establish gains over frozen policies, but leave the explanation for the policy-versus-action optimization gap unresolved.
  2. The implemented update differs from the gradient of the stated objective. Section 3.1 stops gradients through the particle mean inside the downside term. That mean depends on the actor, and its derivative does not generally cancel for downside variance. This may be a useful approximation, but the paper neither justifies it nor measures its effect. It is therefore unclear how much of the reported behavior comes from optimizing Equation (2) versus this modified update.
Questions for authors
  1. How does FinPILOT compare with direct action-sequence optimization using the same scenarios, accounting, risk term, and terminal value, initialized from the pretrained policy's actions?
  2. Why is the mean detached inside the downside penalty? Do the gains persist when differentiating through that mean, with the other adaptation settings unchanged?
2 comments
AuthorsOpenReview ↗

Response to Reviewer EmBi (1/1)

Thank you for the constructive questions. We agree that the original comparison with Boyd SPO and MPO did not fully isolate the effect of adapting policy parameters. We have added a more controlled action-sequence baseline and an ablation of the detached-mean gradient.

Q1. Direct action-sequence optimization

We implemented an action-sequence optimizer using FinPILOT's forecast scenarios, portfolio accounting (including weight drift, compounding, and transaction costs), downside-risk objective, and optimization budget (Section 4.3, Table 4; Appendix D.5). Instead of adapting the actor, it optimizes an HH-step sequence of portfolio actions shared across the KK forecast scenarios. Both methods use E=2E=2 Adam updates per decision.

Following your suggestion, we initialize the sequence from the pre-trained policy's actions. We also test equal-weight initialization. The sequence is shifted forward to warm-start the next decision, analogous to carrying FinPILOT's updated parameters forward. We use the terminal critic where applicable. For SAC, TD3, and DDPG, the action-sequence baseline evaluates the critic using the pre-trained actor's terminal action, since the sequence does not specify an action beyond its horizon.

InitializationPolicy optimizationAction-sequence optimization
Pre-trained policy+0.30 ± 0.28 (6/6)−0.03 ± 0.06 (0/6)
Equal weights, no critic+0.34 ± 0.30 (5/6)+0.02 (3/6)

These are mean changes in Sharpe relative to the frozen policy over 30 settings (± pooled seed SD of paired differences); the counts are market–years with improved mean Sharpe.

We swept the action optimizer's step size and report its best tested value, α=0.001\alpha=0.001 (Table 19). Larger step sizes performed worse. Even at this setting, policy optimization has an average Sharpe advantage of 0.32 from either initialization, with a positive average gap for all five algorithms (Table 20). Unlike Boyd SPO and MPO on FX, these matched action optimizers remain invested (mean cash below 8%). Removing FinPILOT's terminal critic also preserves 91% of its Sharpe improvement (Appendix D.6).

These results support an advantage over the matched-objective open-loop action-sequence optimizer we evaluated. An important remaining difference is the planning representation: FinPILOT produces state-conditioned allocations across imagined scenarios, whereas the direct optimizer uses a shared open-loop action sequence. Section 4.3 now notes this as one possible explanation for the gap.


Q2. Detaching the particle mean

We agree that detaching the mean changes the gradient of the stated objective. We added a derivation (Appendix E) and reran FinPILOT with the full gradient (Appendix D.8, Table 27).

Let dk=Jt(k)−Jˉtd_k=J_t^{(k)}-\bar J_t and

ρt=K−1∑k:dk<0∣dk∣D^t+ϵ.\rho_t=\frac{K^{-1}\sum_{k:d_k<0}|d_k|}{\sqrt{\hat D_t+\epsilon}}.

The gradients satisfy

gdet(λ)=(1+ρtλ) gfull(λ1+ρtλ).g_{\mathrm{det}}(\lambda)=(1+\rho_t\lambda)\,g_{\mathrm{full}}\left(\frac{\lambda}{1+\rho_t\lambda}\right).

Thus, at a given update, the detached gradient is proportional to the exact gradient evaluated at a slightly smaller effective risk weight. The observed median ρt\rho_t is 0.58 (5th–95th percentile: 0.54–0.61), corresponding to an effective risk weight 3–8% below nominal λ\lambda for our grid {0.05,0.10,0.15}\{0.05,0.10,0.15\}. This is a local gradient relationship, not a claim that the full Adam trajectories are identical.

With all other settings unchanged, the full-gradient variant achieves +0.26 ± 0.25 mean Δ\DeltaSharpe versus +0.30 ± 0.28 with detachment, retaining 88% of the gain. It improves in 26/30 settings and all 6 market–years. The difference from the detached variant ranges from 0.00 for SAC to −0.13 for DDPG (Table 27). We now report and discuss this approximation in Sections 3.1 and 4.5.

Dear Reviewer EmBi,

Thank you again for your thoughtful review. We wanted to briefly follow up before the discussion period closes on November 18, particularly regarding the two experiments you suggested.

First, we added a direct action-sequence optimization baseline using FinPILOT's forecast scenarios, portfolio accounting, downside-risk objective, and optimization budget, initialized from either the pre-trained policy or equal weights. At its best tested step size, the action optimizer achieves −0.03 mean Δ\DeltaSharpe from the pre-trained start, compared with +0.30 for FinPILOT (+0.02 versus +0.34 from equal weights). We also clarified the remaining distinction between open-loop action optimization and state-conditioned policy optimization (Section 4.3, Appendix D.5).

Second, we derived the effect of detaching the particle mean and reran FinPILOT with the full gradient. The full-gradient variant retains 88% of the Sharpe improvement and improves performance in 26 of 30 settings (Appendices D.8 and E).

We hope these additional results help address your concerns. If you have a chance to take another look, we would really appreciate hearing whether anything remains unclear or insufficiently addressed.

Thank you again for your time!

Reviewer_kXLYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper proposes FinPILOT, an inference-time policy optimization framework that integrates independently trained price forecasters with pre-trained reinforcement learning trading agents. Under the price-taker assumption, the method constructs imagined market trajectories without requiring an action-conditioned world model.

  • At each trading decision, FinPILOT updates the pre-trained actor through gradient-based optimization of a risk-adjusted portfolio-return objective over forecasted trajectories. The adapted policy executes one action, and its parameters are carried forward to subsequent decisions.

  • Experiments with five RL algorithms (PPO, SAC, A2C, TD3, and DDPG), two financial markets (DJ30 and FX), and three rolling test years (2023–2025) report improvements in total return in 28 of 30 settings and Sharpe ratio in 27 of 30. The authors also compare FinPILOT against direct portfolio optimization methods and investigate how performance varies with forecast accuracy.

Critical strengths
  • Intriguing empirical benefits of inference-time policy adaptation. The paper demonstrates that forecast-driven inference-time optimization improves the deployed policies in most evaluated settings across algorithms, markets, and test years. The magnitude and consistency of these improvements make the phenomenon particularly worthy of investigation.

  • Practical framework with extensive empirical validation. FinPILOT provides a modular way to incorporate independent price forecasts into existing RL policies without modifying their original training procedures. The evaluation across five RL algorithms, two financial markets, and three rolling test years provides encouraging evidence of its practical potential and general applicability.

Critical weaknesses
  • Unclear contribution of RL pre-training and accumulated policy adaptation. Several frozen RL policies appear remarkably close to cash-inclusive equal-weight portfolios. For example, in DJ30 2023, scaling the equal-weight return by 28/29 gives approximately 14.28%, compared with 14.25% for SAC Base. Similar patterns appear across markets and years (Tables 7–10). Although this does not establish identical allocations, it raises an important question about what RL pre-training actually contributes. Furthermore, the actor parameters are carried forward across decisions, resulting in hundreds of cumulative gradient updates rather than merely a few isolated local refinements. The paper does not adequately distinguish the contributions of pre-training, local adaptation, and accumulated online optimization.

  • Insufficiently controlled comparisons with alternative optimization methods. FinPILOT and the Boyd-based baselines differ in their objectives, initialization, and optimization variables. Moreover, the direct optimizers remain almost entirely in cash in the FX experiments. These differences make it difficult to attribute FinPILOT's superior performance specifically to policy-parameter optimization. Although the controlled forecast-skill experiments are informative, they do not isolate the contribution of the original predictive signals from additional optimization and the newly introduced downside-risk objective.

Questions for authors

The empirical improvements are intriguing, but two important questions remain insufficiently addressed.

1. What does RL pre-training actually contribute to FinPILOT?

Several frozen policies appear remarkably close to cash-inclusive equal-weight portfolios (Tables 7–10). Could the authors report the actual portfolio allocations and compare FinPILOT against the same architecture initialized to produce uniform allocations, but without RL pre-training?

Furthermore, since actor parameters are carried forward across decisions, the method effectively performs cumulative online optimization. A comparison that resets the actor to its pre-trained parameters at every decision would help distinguish genuine local adaptation from accumulated learning.

These experiments would clarify whether the improvements arise from useful structure acquired during RL pre-training or primarily from subsequent optimization.

2. Are the comparisons with direct portfolio optimization sufficiently fair?

FinPILOT and the Boyd-based baselines differ in their objectives and initialization, while the latter remain almost entirely in cash in the FX experiments.

Could the authors compare policy-parameter optimization with direct portfolio optimization under matched objectives and forecast scenarios, with comparable initialization and information constraints?

Additionally, while Figure 4 shows that stronger forecasts improve performance, it does not establish the incremental value of the original forecasts relative to uninformative alternatives. A no-information or shuffled-forecast ablation would help isolate this contribution.

These comparisons would help establish which aspects of FinPILOT are responsible for its reported advantages.

4 comments
AuthorsOpenReview ↗

Response to Reviewer kXLY (1/1)

Thank you for pointing out that the original experiments did not clearly separate the contributions of RL pre-training, accumulated adaptation, and forecast information. We added the requested comparisons in Section 4.5 and Appendices D.5–D.7.

Q1. RL pre-training and accumulated adaptation

We first examined the actual allocations (Table 21). Your observation was correct: frozen PPO, A2C, and SAC policies differ from cash-inclusive equal weights by only 0.2–6.6% of portfolio weight (half the ℓ1\ell_1 distance). TD3 and DDPG differ by 51–58%. FinPILOT also outperforms the daily-rebalanced 1/N1/N portfolio in 26/30 settings (mean Δ\DeltaSharpe +0.22 ± 0.22; Table 24).

We then ran the uniform-start ablation you suggested: the same actor architecture, initialized to output equal weights across the assets and cash, without RL pre-training or a terminal critic. The main results are:

VariantΔ\DeltaSharpe DJ30Δ\DeltaSharpe FXMarket–years improved
FinPILOT+0.38 ± 0.30+0.22 ± 0.256/6
Uniform start (no pre-training, no critic)+0.31 ± 0.38+0.37 ± 0.185/6
Reset every decision−0.44 ± 0.19−0.63 ± 0.320/6
Zero-information forecasts−0.30 ± 0.32−0.28 ± 0.302/6
Date-permuted forecasts−0.33 ± 0.45+0.26 ± 0.204/6

Values are changes in Sharpe relative to the frozen RL policy, averaged over five algorithms and three test years per market (± pooled seed SD of paired differences). The frozen policies average Sharpe ratios of 1.04 on DJ30 and 0.09 on FX.

The uniform-start actor achieves comparable aggregate gains (+0.05 ± 0.29 relative to FinPILOT across all settings), although it can learn different portfolios. For TD3 and DDPG, the final allocations differ by 29–54% between the two initializations; their uniform-start Sharpe gains relative to FinPILOT are +0.13 and +0.19, respectively, both within one seed SD. Removing only the terminal critic retains 91% of FinPILOT's gain (Appendix D.6).

We also tested resetting the actor to its pre-trained parameters, with a fresh optimizer, before each decision. This removes the gains: average Δ\DeltaSharpe is −0.54 ± 0.26 across both markets, below the frozen policy in 29/30 settings. Mean daily turnover rises from 2.7% under FinPILOT to 10.8% with reset; using a tenfold larger reset step size makes performance worse still (−1.76 ± 0.50 versus the frozen policy).

Finally, Table 23 measures the allocation distance from the original pre-trained policy on the same state. By the end of the year, it reaches 74–84% for PPO and A2C, 29–33% for SAC, and 90–92% for TD3 and DDPG. These results suggest that gains depend primarily on cumulative forecast-driven policy adaptation under our tested settings, rather than isolated local updates or RL pre-training alone. We have revised the interpretation in Section 4.5 and the conclusion accordingly.


Q2. Direct optimization and forecast information

We agree that the original comparisons with Boyd SPO and MPO differed in objective and initialization. We therefore added a matched-objective open-loop action-sequence optimizer, using the same scenarios, accounting, risk objective, number of updates, and warm start (Section 4.3, Appendix D.5). From the pre-trained policy's actions, it achieves −0.03 ± 0.06 mean Δ\DeltaSharpe at its best tested step size, compared with +0.30 ± 0.28 for FinPILOT. From equal weights, the corresponding results are +0.02 and +0.34 ± 0.30. The average gap is 0.32 from either start and is positive for every algorithm (Table 20). Boyd SPO and MPO stay in cash on FX because the trading-cost multipliers selected on the preceding years are all γtrade≥20\gamma_{\rm trade}\ge 20, and every grid value ≥7\ge 7 keeps them in cash (Appendix C.4). The matched action optimizers remain invested (mean cash below 8%), so this does not explain the gap.

To isolate the role of the forecasts, we kept the adaptation setup fixed and replaced only the forecast input (Appendix D.7). A trailing historical-mean predictor (the zero-information control in Table 4, with measured R2≈0R^2\approx0) removes the average gains in both markets: −0.29 mean Δ\DeltaSharpe, with improvements in only 6/30 settings. This suggests that the additional optimization and downside-risk objective do not, by themselves, explain the main results.

Date-permuted forecasts remove the DJ30 gains but preserve the FX gains (+0.26 mean Δ\DeltaSharpe on FX, versus +0.22 with the original forecasts). Permutation reduces forecast rank IC in all three DJ30 years but raises it in all three FX years (Table 26), consistent with persistent cross-sectional currency information. This interpretation is suggestive rather than conclusive. We now report both controls in Section 4.5.

Dear Reviewer kXLY,

Thank you again for your detailed review. We appreciate your positive comments on the empirical results and the modular design of FinPILOT. We wanted to briefly follow up before the discussion period closes on November 18.

We added several experiments in response to your questions and updated the manuscript accordingly.


First, we examined the role of RL pre-training and cumulative adaptation. As shown in our ablation table, a uniformly initialized actor achieves comparable improvements to FinPILOT (+0.31 vs. +0.38 Δ\DeltaSharpe on DJ30, +0.37 vs. +0.22 on FX). In contrast, resetting the actor before every decision removes the gains (−0.44 on DJ30 and −0.63 on FX). These results suggest that cumulative forecast-driven adaptation is more important than RL pre-training alone (Section 4.5, Appendix D.6).

Second, we added a matched-objective action-sequence optimization baseline using the same forecast scenarios, portfolio accounting, and optimization budget. FinPILOT outperforms this baseline on average across all five algorithms. Unlike Boyd SPO and MPO, the matched action optimizer remains invested on FX, so its weaker performance cannot be explained by cash-only allocations (Section 4.3, Appendix D.5).

Finally, we ran the forecast ablations. Replacing the learned forecasts with a historical-mean control removes the average gains (-0.30 on DJ30 and -0.28 on FX). Date permutation also removes the DJ30 gains, although the improvements persist on FX. Interestingly, permutation raises forecast rank IC in all three FX years (Table 26), suggesting that persistent cross-sectional currency information may explain why the FX gains remain (Appendix D.7).


We would appreciate hearing whether these additional experiments address your main concerns, or if there is anything else we could clarify before the discussion closes.

Thank you again for your time!

I thank the authors for the detailed responses and the substantial additional experiments run within a short rebuttal period. Several of my concerns are resolved. The matched-objective action-sequence baseline, the reported allocations, and the zero-information forecast control are the comparisons I asked for. The zero-information result in particular shows convincingly that the gains are not produced by extra optimization or the downside-risk objective alone.

However, the new results reveal a mechanism that differs substantially from how the paper frames its contribution. The submission presents FinPILOT as a plugin that refines a pre-trained RL policy at inference time, in the spirit of MPC. The rebuttal shows instead that (i) an equal-weight actor with no RL pre-training and no critic performs comparably, and better on FX (+0.37 vs. +0.22); (ii) resetting to the pre-trained actor before each decision is worse than the frozen policy in 29 of 30 settings; and (iii) by year-end, the adapted allocations differ from the original policy’s by 74–92% for four of the five algorithms. The reset variant most directly isolates per-decision refinement of the pre-trained policy, which is the mechanism the paper emphasizes, yet it is the variant that fails; what works is cumulative learning across the test year. The method is therefore better described as online training of a policy on forecast-generated scenarios than as inference-time adaptation of a pre-trained agent. The title, the abstract, the “compatible with any pre-trained policy” claim, and the five-algorithm evaluation design all rest on the latter framing.

This raises questions the paper does not yet address. If the reported gains do not require RL pre-training, the relevant baselines should also include other ways of learning a closed-loop policy from forecast scenarios. Examples include training the actor offline on forecaster scenarios, or a scenario-based formulation with recourse. The added baseline is a shared open-loop action sequence, and the authors acknowledge in Section 4.3 that this difference in representation may explain the gap. As it stands, the comparison does not establish that optimizing policy parameters is what drives the advantage. The FX results also leave the source of the gains unclear. Date-permuted forecasts retain the original gains while raising rank IC, so it is not yet clear whether time-specific forecast information or persistent cross-sectional information drives performance.

Finally, the evaluation still covers three test years in two markets. The 30 configurations share the same six market-years and, within each, the same forecaster, so they are not 30 independent pieces of evidence. The reported ± values reflect seed variation, not variation across market periods.

The direction is interesting, and the revised analysis is considerably clearer than the original submission. But if the method works without RL pre-training, it is not yet clear why its gains should be attributed to adapting pre-trained RL policies. Since the paper’s framing rests on that attribution, I maintain my score.

Thank you for your follow-up. We appreciate your acknowledgment of the additional experiments and would like to clarify a few remaining points.

1.⁠ ⁠As discussed in the revised Section 4.5 and conclusion, the gains appear to come primarily from cumulative forecast-driven policy optimization rather than RL pre-training alone. However, carrying the adapted parameters forward was already part of the original algorithm. The method itself has not changed, but the ablations clarify which components contribute most to its performance. The actor is optimized using scenarios from a forecaster trained before the test year, without gradient updates based on realized test-period returns. RL pre-training still influences the resulting allocations. For TD3 and DDPG, the final portfolios differ by 29–54% between pre-trained and uniform initialization.

2.⁠ ⁠We agree that the matched open-loop action-sequence baseline does not completely isolate the effect of optimizing policy parameters from that of closed-loop planning. Offline training on forecast scenarios and scenario-based optimization with recourse are reasonable alternatives that remain to be evaluated. Our results establish an advantage over the tested open-loop optimizer, not over all forms of direct optimization.

3.⁠ ⁠The FX date-permutation results suggest that persistent cross-sectional information may contribute to the gains, although they do not fully isolate its contribution from time-specific predictive signals. We acknowledge this limitation.

4.⁠ ⁠The ±\pm values in Table 4 represent seed standard deviations, not uncertainty in the aggregate mean. We also report consistency across the six market–years, although the 30 settings share market histories and forecasters and should not be treated as independent observations.

Thank you again for your time and feedback.

Reviewer_s1tDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to introduce an algorithm for reconciling financial forecasters and RL algorithms for learning portfolio optimisation. This is meant to provide a way to augment deep RL learners with the ability to adapt to residual information captured by a prediction model, or (presumably also) to adapt to non-stationarity.

The paper is essentially an experimental exploration of the problem and a proof-of-concept of how to construct an algorithm. The algorithm is of a bare-bones design: at each test step, "sample" trajectories from the forecaster and take some policy gradient steps of a risk-adjusted return functional to adjust the learner weights.

Critical strengths

The paper's writing is serviceable, though large parts are clearly AI-generated.

Critical weaknesses

To be honest, there just isn't a lot of meat to eat on this bone. The text itself is very low in information density, with lots of repetition of the same basic facts about the setup, and the result is just not up to the significance bar for the conference.

  • The situation of the paper in the finance literature is non-existent, and limited to a handful of papers in this so-called "RLFT" niche. Stochastic control and mathematical finance methods are not discussed as base forecasters, despite their relevance; neither is the literature of "learning-augmented algorithms/algorithms with predictions", nor the literature on low-regret learners like universal portfolios.
  • The algorithm is very basic and isn't contextualised; what design choices matter is unclear. The historical data study essentially suggests that you can indeed improve performance if you get some information about the future! This conclusion is foregone, what would have been interesting is to design an algorithm designed to make the policy updates cheap, robust, or something. More generally, the only reason I can find that this work is limited to portfolio optimisation is that this is the dataset the authors decided to study. This could equally have been done with optimal execution, hedging, robot control, navigation, etc.
Questions for authors
  • In the experiments, you (essentially) allow retraining at each step of the roll-out. So it is possible for the RL policy to learn new patterns (non-stationarity) in the data. How can you perform an ablation of this effect? That this could be an issue is suggested to me by the fact that the high-performance years in Fig. 3. are the same across models but differ between the two asset classes, suggesting that there is a confounder here in the data itself.
  • How can we evaluate that what is happening is minor adjustments of the policy (a kind of transfer learning) and not complete re-learning of the optimal portfolio conditional on the prediction at each step?

Future Work:

  • Aside from the highly unusual COVID period, the test data is entirely a bull market; it would be better to have a longer period.
5 comments
AuthorsOpenReview ↗

Response to Reviewer s1tD (1/2)

Thank you for the review. We have expanded the related work and added experiments to clarify which parts of the method account for the results. This comment covers positioning and significance, including EG and ONS on our test years, and the next covers your two questions and the evaluation period.

Positioning and significance

We have expanded Appendix A to discuss online portfolio selection, including universal portfolios (Cover, 1991), exponentiated gradient (Helmbold et al., 1998), online Newton step (Agarwal et al., 2006), and algorithms with predictions (Mitzenmacher & Vassilvitskii, 2022). The key distinction is that classical online portfolio selection learns from realized returns, often with regret guarantees, whereas FinPILOT updates the policy using forecasted returns before the next decision. We do not claim a regret guarantee. We report EG and ONS on our test years at the end of this comment.

We also clarified the connection to financial MPC. Like Boyd et al. (2017), we plan over a horizon, execute one decision, and replan. However, FinPILOT adapts a state-conditioned policy rather than directly optimizing a portfolio action sequence. Under the price-taker assumption, actions do not affect future market prices, and portfolio rewards can be computed from forecasted prices and known accounting rules; an action-conditioned world model or learned reward model is therefore unnecessary for this setting. We do not claim a new forecasting method. Any model that produces HH-step price scenarios can serve as the forecaster, including stochastic-control and mathematical-finance models of returns; the two models tested here have out-of-sample R2R^2 between 0.005 and 0.039 across market–years (Table 7).

With forecasts this weak, it is not obvious that they help at all, and our new experiments show that it depends on how they are used. Under matched forecast scenarios, accounting, risk objective, and optimization budget, the open-loop action-sequence optimizer stays close to the frozen policy at its best tested step size, whereas adapting the policy yields an average Sharpe advantage of 0.32 (Section 4.3, Table 4). Resetting the actor before every decision also removes the gains; so do zero-information forecasts, which only predict the trailing mean of recent price changes (mean Δ\DeltaSharpe −0.29). In contrast, an equal-weight initialized actor without RL pre-training performs comparably to FinPILOT. These results point to cumulative, forecast-driven policy adaptation as the main observed mechanism, not simply access to future predictions or structure acquired during RL pre-training.

Appendix D.4 also examines the importance of update magnitude. Step size and number of epochs affect the gains, and larger updates can be harmful in DJ30 2024, the year with the weakest forecaster (R2=0.005R^2=0.005). Two other choices matter little: removing the terminal critic keeps 91% of the gain, and differentiating through the particle mean keeps 88% (Appendices D.6 and D.8). Regarding scope, our decomposition applies most directly when actions do not affect the forecasted state and rewards can be computed from known dynamics; hedging a small position satisfies both. Market-impact-aware optimal execution, robotic control and navigation do not generally satisfy the first condition and would need an action-conditioned model.


EG and ONS on our test years

We also ran exponentiated gradient (EG) and the online Newton step (ONS) in our environment, with step sizes chosen on the validation year and the same 10 bp transaction costs. Both start from equal weights over the risky assets; EG's multiplicative update then keeps it fully invested, as FinPILOT nearly is on DJ30 (under 2% cash for every algorithm). Each entry is the mean of its annual metric over the three test years; frozen policies and FinPILOT are also averaged over five algorithms (Table 1). Best per market in bold:

Return (%)SharpeSortinoCalmarMDD (%)
DJ30
EG18.760.741.230.9817.52
ONS18.070.861.310.9616.62
Frozen policies12.931.041.531.3710.46
FinPILOT27.841.422.232.2612.04
FX
EG−1.35−0.31−0.350.5510.20
ONS−4.14−0.64−0.96−0.349.24
Frozen policies0.760.090.300.654.97
FinPILOT1.780.310.750.824.37

On the three-year averages, FinPILOT outperforms both EG and ONS across all five reported metrics in DJ30 and FX. Across the six market–years, FinPILOT achieves the highest Sharpe ratio among the four methods in all six and the highest total return in four. EG achieves higher returns in DJ30 2023 and FX 2025, but with larger maximum drawdowns. EG and ONS also trail the frozen policies in mean Sharpe, Sortino, and Calmar ratios.

AuthorsOpenReview ↗

Response to Reviewer s1tD (2/2)

Q1. Non-stationarity and possible market–year effects

FinPILOT does not update using realized test-period rewards. At each decision, the actor is optimized only against imagined returns under the current forecast scenarios; the forecaster was trained before the test year and stays frozen. That the good years are shared across algorithms is what one would expect if the forecasts drive the gains: all algorithms in a market–year use the same forecaster, whose skill varies by year (Table 7). We test this directly.

We added two controls with the policy, market–year, and optimization procedure held fixed (Appendix D.7). Zero-information forecasts remove the average gains in both markets (−0.29 mean Δ\DeltaSharpe; improvements in 6/30 settings). Date permutation removes the DJ30 gains but not the FX gains. The latter is an important qualification: permutation increases rank IC on FX, so persistent cross-sectional forecast information may still be useful after the dates are shuffled. These findings do not rule out every regime effect, but they show that additional optimization on the same market years is not sufficient to reproduce the main results. On DJ30, the weakest forecast year is 2024 (R2=0.005R^2=0.005), where larger adaptation steps hurt performance (Appendix D.4).


Q2. Local refinement or substantial re-learning

We examined both daily trading behavior and the policy's cumulative movement from its initial state. FinPILOT has mean daily turnover of 2.7%, compared with 1.4% for the frozen policy (Appendix D.6). But its allocation difference from the original policy, evaluated on the same states, grows over the year: by the final period, this reaches 74–92% for PPO, A2C, TD3, and DDPG, and 29–33% for SAC (Table 23). Thus, small daily portfolio changes do not imply that the deployed policy remains close to its initial behavior.

The reset ablation helps distinguish these mechanisms. Restoring the pre-trained actor before every decision makes performance worse than the frozen policy in 29/30 settings and increases mean turnover to 10.8%. Direct open-loop action-sequence optimization also does not reproduce the gains (Table 4). Meanwhile, a uniform-start actor with no RL pre-training achieves comparable aggregate improvement. FinPILOT is therefore best described as small forecast-driven updates that accumulate across decisions, and none of them uses realized rewards (Q1). Section 4.5 and the conclusion now say this.


Evaluation period

Section 5 of the original submission already lists the three test years (2023–2025) as a limitation, and the revision keeps it. The FX 2024 evaluation provides a partial counterpoint to favorable-market concerns: all five frozen policies lost 5.7–7.4%, while FinPILOT reduced both losses and maximum drawdown for every algorithm (Table 9). This is not a substitute for longer historical evaluation, but it is evidence that the reported improvements are not confined to positive-return periods.

Dear Reviewer s1tD,

Thank you again for your review. We wanted to briefly follow up before the discussion period closes on November 18. We made several changes to the manuscript and ran additional experiments in response to your concerns.


First, we expanded the related work to cover online portfolio selection and algorithms with predictions (Appendix A). We also ran EG and ONS under the same transaction costs (results in our response above). FinPILOT outperforms both methods across all five metrics on the three-year averages in DJ30 and FX. Across the six individual market-years, it achieves the highest Sharpe ratio in all six and the highest total return in four.

Second, we added a matched-objective action-sequence optimizer using the same forecast scenarios, portfolio accounting, and optimization budget. FinPILOT outperforms this baseline by 0.32 Sharpe points on average (Section 4.3, Appendix D.5). We also examined which design choices matter: removing the terminal critic retains 91% of the gain, and differentiating through the particle mean retains 88% (Appendices D.6 and D.8).

Third, we investigated whether the gains come from market-year effects or accumulated policy adaptation. FinPILOT does not update on realized rewards, and replacing the learned forecasts with a historical-mean control removes the average gains in both markets. We also found that while daily portfolio turnover is relatively small (2.7%), the changes accumulate over time, and resetting the actor before every decision removes the gains (Section 4.5, Appendices D.6-D.7).

Finally, Section 5 of the original submission already lists the three test years as a limitation, and the revision keeps it. FX 2024 provides a useful negative-return setting: all five frozen policies lost money, while FinPILOT reduced both losses and maximum drawdown for every algorithm (Table 9).


We would appreciate hearing whether these additional experiments address your main concerns, or if there is anything else we could clarify before the discussion closes. Thank you again for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.