acceptodds
Under review as a conference paper at ICLR 2027

Bad Outcomes Need Not Mean Bad Actions: Counterfactually Verified Pain Memory for Policy Optimization

Abstract

Bad outcomes do not always mean bad actions. Policy-gradient methods update policies using advantage estimates and TD errors, while failure-memory approaches often retain experiences based on observed failures or their severity. When exogenous randomness dominates the effects of individual actions, these signals can mistake bad luck for poor decisions, while later policy updates can erase useful lessons from rare but consequential events. We propose , a two-channel framework built on Proximal Policy Optimization (PPO). The Advantage channel learns from current rollouts, while the Pain channel retains only corrections verified through counterfactual replay, without modifying rewards or advantage estimates. For a candidate failure, we intervene on the executed action and replay the episode in closed loop under the same recorded conditions. A correction enters memory only if the alternative action improves the outcome. Its influence grows when the same corrective direction recurs across distinct realizations. Each stored lesson defines a local, revisable, one-sided retention constraint whose gradient is combined with the PPO gradient only when needed to resist forgetting. We evaluate PAIN-PPO in both controlled simulations and real-world financial markets. In simulation, PAIN-PPO distinguishes actionable policy errors from adverse outcomes driven by exogenous randomness, and its gains largely disappear when counterfactual verification is removed, while selective admission further filters corrections that do not recur consistently. On held-out financial market data, PAIN-PPO demonstrates promising lower-tail risk performance under the prescribed mean-cost budget. Together, these results show that counterfactually verified memory can improve tail-sensitive policy learning by retaining rare but actionable corrections without conflating unfavorable outcomes with poor decisions.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Policy Optimization, Counterfactual Credit Assignment, Risk-Sensitive Reinforcement Learning, Conditional Value-at-Risk

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_y99TOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper gives an interesting take on giving rewards in PPO-style RLHF, but instead of punishing bad outcomes, it does a different execution under identical randomness and checks whether you have improvements across different scenarios. Then it starts a connection as a one-sided constraint for future PPOs

I like that they combine causal reasoning with some elve protection against "forgetting". None of the ingredients is new, but the combination is the contribution. The best results are in financial institutions.

To produce the counterfactual verification and one-sided memory constraint. The hope is to retain a correction that we can action on without misstating bad luck for a bad decision.

Critical strengths

The strong point is the causal attribution before memory. I like this idea a lot. Also, the paper gives control ablations that separate verification from retention.

Critical weaknesses

The theory part is considerably weaker than the empirical result suggest.

They don't establish improvement in policy quality or satisfaction of the strict constraints, so for me, this is a red flag. The actual implementation is different from what they analyze.

I like the financial results and what they achieved. At the same do depend on the strong assumption that the market will not react, and also we cannot say that they statistically establish that there is a very tight mean budget :/

Questions for authors

Q1 How are the financial improvements? What would happen if there was some endogenous market response? Are the results robust to this? Can the authors provide any mean budget feasibility beyond the point estimates?

Q2: Wah I would like really is the constraint satisfaction or convergence for the actual algorithm, including when the gradient with PPO vanishes - cna authors provide that?

5 comments
AuthorsOpenReview ↗

Response to Reviewer y99T (1/5): Summary, W1

Summary

Thank you for your positive view of attributing an outcome before storing it in memory. Your concerns fall into two groups. The first is theoretical. It asks whether the algorithm we actually run satisfies its constraints and converges, including when the PPO gradient vanishes. The second concerns the financial results. It asks about the assumption that the market does not react to our orders, and whether the 0.2 bp mean budget can be supported beyond point estimates.

For the theory, the revision adds Section 4.3, which addresses the memory directly. Proposition 3 gives the criterion for whether the constraints hold, and Theorem 2 proves, in a tail-risk model and starting from empty memory, the formation and retention of the correction and the resulting tail gain (W1, W2, and Q2). For the financial results, we estimated the market impact on our own data, and the conclusions do not change once the estimated impact is added. On the mean budget, the interval pooled over the five test months and the intervals of the two added months each lie entirely above the budget line (W3 and Q1).

W1. The revision strengthens the theory. The new Section 4.3 addresses the memory directly. Starting from empty memory, it proves the formation and the retention of the correction and the resulting tail gain, and its prediction is tested in actual training (revised page 6, line 308; proofs in Appendices C.3 and C.4).

A one-step law (Proposition 3, for any differentiable policy, revised page 6, line 316). Let gP,tg_{\mathrm P,t} be the gradient of the memory constraint loss, g^P,t\hat g_{\mathrm P,t} its unit vector, and gA,tg_{\mathrm A,t} the PPO gradient. When the cap binds, the expected first-order change of the constraint loss in one update is

Et⟨gP,t,,θt+1−θt⟩=−ηt,∣gP,t∣,Et∣gA,t∣,(ρP−ρt⋆),\displaystyle \mathbb E_t\langle g_{\mathrm P,t},\\,\theta_{t+1}-\theta_t\rangle=-\eta_t\\,\\|g_{\mathrm P,t}\\|\\,\mathbb E_t\\|g_{\mathrm A,t}\\|\\,(\rho_{\mathrm P}-\rho_t^\star), ρt⋆=−Et⟨g^P,t,,gA,t⟩Et∣gA,t∣.\displaystyle \rho_t^\star=\frac{-\mathbb E_t\langle\hat g_{\mathrm P,t},\\,g_{\mathrm A,t}\rangle}{\mathbb E_t\\|g_{\mathrm A,t}\\|}.

We call ρt⋆\rho_t^\star the cost-to-budget ratio. It is the pull of the base gradient against the stored corrections relative to the size of the base gradient, and it can be measured from the training logs. When the cap ρP\rho_{\mathrm P} is above it, the constraint loss falls in expectation and the stored corrections are kept, and when the cap is below it they are eroded.

An end-to-end theorem (Theorem 2, revised page 7, line 337). In a tail-risk model and starting from empty memory, both the formation and the retention of the correction are proved, and neither is assumed. When the cap is above ρcrit\rho_{\mathrm{crit}}, the largest cost-to-budget ratio over the teaching interval, there are a deterministic T0T_0 and an explicit Δtail>0\Delta_{\mathrm{tail}} > 0 such that

\displaystyle \Pr\Big\\{\mathrm{CVaR}_\alpha\big(G_{\theta_t^{\mathrm{PAIN}}}\big)-\mathrm{CVaR}_\alpha\big(G_{\theta_t^{\mathrm{PPO}}}\big)\ge\Delta_{\mathrm{tail}}\\ \text{ for all } T_0\le t\le T\Big\\}\ge 1-\delta.

When the cap is below a second explicit threshold ρ1\rho_1, the probability that the safe action ever becomes more likely is exponentially small for any memory content, so the correction cannot be installed.

A test of the law in actual training (revised Appendix H.4, page 63, line 3371). The one-step law says that corrections are kept only when the cap is above the cost-to-budget ratio. We measured this ratio from the training logs of each arena, and in the first, middle, and last third of training it lies below the calibrated cap (revised Table 23, page 63, line 3386).

ArenaCalibrated cap ρP\rho_{\mathrm P}Measured ratio ρ⋆\rho^\star
Corridor, exact verifier0.25−0.011 to +0.004
Corridor, noisy verifier1−0.10 to +0.014
Sepsis0.05−0.069 to −0.034
Guarded Maze1+0.22 to +0.24
Execution0.90.58 to 0.65

The law also makes a testable prediction: once the budget is removed, corrections that have already been learned are eroded by the base objective. In the maze, after the corrections are learned and training continues with the Pain channel off, 62 of 68 seeds lose them, against 27 with the channel kept. In the corridor the same test gives 11 of 39 seeds against 2 with the budget kept.

AuthorsOpenReview ↗

Response to Reviewer y99T (2/5): W2

W2.

On improvement in policy quality, policy quality in this paper is the return in the worst 5% tail under a given mean budget. Theorem 2 of the revision proves its improvement. In a tail-risk model and starting from empty memory, the CVaR of PAIN-PPO exceeds that of PPO by an explicit amount Δtail\Delta_{\mathrm{tail}} with probability at least 1−δ1-\delta, at a controlled cost in the mean (W1, revised page 7, line 337). Empirically the tail improves in all four arenas, as the table below shows.

On satisfaction of the constraints, each stored correction is a one-sided constraint that asks the policy to retain at least part of the verified change. The constraints enter the update through a capped gradient, and the paper does not claim that they are enforced at every step (Section 3.3, revised page 5, line 239). What the theory proves is that they are kept. Proposition 3 shows that when the cap is above the cost-to-budget ratio, the constraint loss falls in expectation to first order. Theorem 2 shows that in its model the correction is installed and held with high probability.

We also measured directly how far the constraints are satisfied in actual training. A policy that has learned its corrections is copied twice, one copy keeps the memory and the other drops it, and both continue training. In the sepsis simulator, after 1,600 further updates on a different family of patients, the probability of the verified actions stays at 0.999 with the memory, against 0.75 to 0.80 without it (revised page 44, line 2375). Entry by entry, only 87 of 1,148 constraint records fall short of their target, by at most 0.032 (revised Appendix H.4.1, page 64, line 3448).

On the implementation differing from the analysis, the revision unifies PAIN into one complete algorithm, three gates and one capped update. The same specification runs in all four arenas, and all main results were rerun under it. The update that the theory analyzes is this update, Eq. (4). Only the cap ρP\rho_{\mathrm P} is set per arena, by one fixed rule (Section 3.3, revised page 5, line 232). We also tested the prediction of the theory directly in actual training. Once the budget is removed, the learned corrections are eroded, and this holds in the maze and in the corridor (W1). The results of this one algorithm in the four arenas are below (revised Table 22, page 63, line 3349).

ArenaρP\rho_{\mathrm P}Main resultRetention after continued training, or noisy verifier
Corridor0.25median tail gain +42.6, 18 of 20 seeds feasibleunder noise the same rule selects ρP=1\rho_{\mathrm P}=1, tail gain +47.6 and +45.2
Execution0.9tail increment of the Pain channel +8.60 bp, positive and feasible in all 15 seed-months+4.7 and +7.4 bp in September and October, all 10 seed-months feasible
Sepsis0.05+0.0449 over PPO without Pain, and +0.0329 on 20 fresh seedsprobability of the verified actions 0.999, against 0.75 to 0.80 with the memory off
Guarded Maze1all 20 seeds converge, against 16 for Return Capping27 of 68 seeds lose the learned route, against 62 with the Pain channel off
AuthorsOpenReview ↗

Response to Reviewer y99T (3/5): W3

W3.

On the assumption that the market does not react, replay on the historical order book is the standard evaluation for low-participation execution strategies (Nevmyvaka et al., 2006; Fang et al., 2021; Zhang et al., 2023), and it does not model how other participants react to the agent. We added the check that historical data does allow. We estimated the impact on these data. With the estimated impact added, the ordering and the feasibility of the three systems are unchanged (numbers in Q1). We also redid the 16,051 paired verifications behind the admitted corrections under the estimated impact. After this, 92.2% of the corrections still have a majority of positive supporting evidence, and 87.4% at twice the impact (new, revised Appendix F.4.3, page 52, line 2759). The case in which other participants adapt their behavior to the algorithm has to be measured in live trading with paired randomized deviations, and we state it as a condition for applying the method.

On the mean budget, we added the data of September and October after the submission, which extends the test set from three months and 26 days to five months and 46 days. The mean budget is a line at −0.2 bp. A single month has only 8 to 10 trading days and the daily standard deviation is about 13 bp, so an interval of ±0.2 bp would need about 16,000 independent trading days. The mean budget is therefore judged on point estimates, as pre-registered, which is also how execution studies report costs (Nevmyvaka et al., 2006; Fang et al., 2021; Zhang et al., 2023), and all 25 seed-months are feasible (Appendix E, revised page 34, line 1821). Pooled over all 46 days, the mean improvement over POV is +6.19 bp with a 95% interval of [+2.94, +9.75] (revised page 50, line 2680). September and October alone give +4.71 [+1.24, +9.94] and +12.33 [+2.91, +21.73] (revised page 49, line 2639). All three lower ends are above zero, not only above the budget line.

Reviewer_nXorOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PAIN-PPO, which adds a second "Pain" channel to PPO. Severe episodes are inspected; the executed action is replaced by local probe actions and the episode is replayed in closed loop under the same recorded randomness; a correction is stored only when an alternative improves the outcome, and it gains authority only when the same corrective direction recurs across distinct realizations. Stored corrections become one-sided retention constraints whose gradient is added to the PPO gradient with a relative norm cap, and which can attenuate conflicting PPO components (a "veto"). The authors show that the update equals PPO with an adaptive regularizer, prove a finite-time bound on the update residual, and give a two-parameter example in which PAIN-PPO reaches the optimum while PPO does not. Experiments cover a synthetic corridor with labelled failure classes, the Guarded Maze, a sepsis Gumbel-max SCM simulator, and A-share order execution on replayed limit-order-book data.

Critical strengths

Good question and clear positioning: separating actionable from exogenous failures before storing them is a real gap between risk-sensitive RL and failure-memory methods. Careful experimental practice: pre-registered criteria, per-seed paired comparisons, ablations that isolate admission from retention (Table 2), a negative-results register, and released code with scripts that regenerate each table. The corridor arena with ground-truth failure classes is a clean test of the admission rule and shows exactly which gate removes which class (Appendix F.1).

Critical weaknesses

Exact same-realization replay is assumed everywhere. Every arena provides exact counterfactuals by construction: the corridor is built around them, the sepsis simulator uses noise the authors draw themselves, the maze replays recorded noise, and execution replays tapes under a small-agent assumption. The only test of an imperfect verifier is additive Gaussian noise on the corridor (Figure 3a). A learned or model-based verifier, which the authors themselves call "the regime a learned or model-based verifier lives in", is never tested. Since counterfactuals are not identified from observational data (Oberst & Sontag, 2019), a replay model can confidently return the wrong counterfactual, and it is unknown how the admission gates behave under such structured error.

PAIN-PPO is not one algorithm. The table in Appendix H.2 shows that each arena uses different inspection rules, probe sets, admission thresholds (at least 2, 3, or 40 tapes), teaching mechanisms (one-sided latent constraint, squared-error loss, probability margin), update rules (uncapped reliability-weighted learning rate in execution; Eq. 4 with αP = 1000 and cap 0.05 in sepsis and the maze; no arbitration and no veto in the corridor), persistence, and budget mechanics. The canonical update analysed in Section 4 is therefore not what runs in the corridor or in the execution experiments. One fixed configuration should be run on all arenas with a sensitivity study.

The theory adds little. Proposition 1 is a rewriting. Theorem 1 bounds the residual of PAIN's own update, not the return or the quality of the stored constraints, and the authors note that it "does not imply global optimality". Proposition 2 is a hand-built two-parameter instance with designed interference, a specific initial point, and a correction that happens to align with the optimum. None of the results say when a stored correction helps or harms.

The execution results are claimed beyond what the data can resolve. There are three test months with 8–10 independent trading days each; the trading-day-clustered interval of the monthly mean is 13–21 bp wide against a 0.2 bp budget, and the authors show that removing the two best days puts every month below the budget. Most of the tail gain comes from the admission and regime layers: the same system with the Pain channel off already achieves 54/18/29 bp of the 66/28/37 bp, so the abstract's "promising lower-tail risk performance" belongs to the three-layer system rather than to PAIN. Market impact is handled only by an open-loop penalty that the policy does not react to.

Cost and side effects. In the Guarded Maze PAIN-PPO adds about 1M replay steps (roughly 50% more interactions), is costlier to converge on 52 of 76 seeds when replays are counted, and lowers mean return (0.96–1.18 vs 1.59–2.07). In the sepsis simulator 3 of 48 arms collapse through spillover and the remedy (a return-guarded shadow anchor) is introduced post hoc. The discrete-maze pilot failed its pre-registered criterion and was dropped.

Presentation. The paper is 41 pages and the main text relies on appendix tables; a notation appendix is needed to resolve symbol collisions; Figures 2 and 3 pack nine small panels; many sentences are long chains of numbers that are hard to parse.

Overall, the central claim is that counterfactual admission, not outcome severity, separates actionable from exogenous failures. That claim is tested in four arenas with ablations that isolate it, and the weaknesses above concern scope and reproducibility more than correctness. I lean toward acceptance and would raise my score if one fixed configuration worked across arenas and a learned verifier were tested.

Questions for authors

Can the authors run PAIN-PPO with a learned replay model as the verifier (for example a fitted SCM for the sepsis simulator, or a learned dynamics model in the corridor) and report admission precision and tail gain?

Can one fixed configuration (admission floor, teaching mechanism, cap, veto) be run on all four arenas? Which conclusions change?

For execution, what is the marginal contribution of the Pain channel at the trading-day level over a longer test period, and how sensitive is the budget conclusion to the handful of sharp-drop days?

How should n_min, n_scale, the state-matching bandwidth, and the retention fraction η be chosen in a new environment without ground-truth failure labels?

8 comments
AuthorsOpenReview ↗

Response to Reviewer nXor (1/8): Summary, W1

Summary

Thank you for reading our central claim accurately and for the two concrete requests. We carried out both.

The first request was one configuration for all four arenas. The revision unifies PAIN-PPO into one algorithm, and the four arenas use the same gates, the same memory rule, and the same update. All main results were rerun under it, and the conclusions hold (W2 and Q2). The second request was a test of a learned verifier. We replaced the verifier with a model fitted from data. As long as the logged data contain the counterfactual actions, training gains +0.0415, against +0.0449 with exact replay (W1 and Q1).

Your other concerns are mainly about the theory and the financial results. On the theory, the revision adds Section 4.3. It gives the criterion for when stored corrections are kept and proves, in a tail-risk model, that PAIN-PPO starting from empty memory forms a correction, keeps it, and obtains a tail gain (W3). On the financial results, we extended the test set from three months and 26 days to five months and 46 days and estimated the market impact on these data. The interval of the tail increment of the Pain channel excludes zero in all five months, and the ordering of the systems is unchanged once the estimated impact is added (W4 and Q3).

W1.

We have added learned-model verifier experiments and directly examined the quality of corrections after passing the admission gates. The results show that in the sepsis experiments where the logged data covers counterfactual actions, the learned verifier still retains gains. Specific details are as follows:

  1. Learned verifier: replacement experiments have been added, and gains remain.

    We fit a transition model using previously logged simulated treatment trajectories and replace the exact replay verifier with this model. Policy training and final evaluation still take place in the original sepsis simulator. (revised Appendix F.3.2, page 46, line 2434)

    Across 16 seeds, the results are as follows. All gains represent the mean return improvement relative to same-seed PPO without the Pain channel. (revised page 46, line 2438)

    VerifierAccuracy of Admitted CorrectionsPost-Training Gain [95% Interval]
    Exact replay94.6%+0.0449 [+0.0250, +0.0671]
    Learned model: randomly selecting treatments during logging88.2%+0.0415 [+0.0207, +0.0632]
    Learned model: applying paired randomized deviations to policy actions during logging85.2%+0.0451 [+0.0229, +0.0689]
  2. Admission performance under structural errors: most corrections remain accurate when coverage is present.

    The admission accuracies of the two learned models are 88.2% and 85.2%, lower than the 94.6% of exact replay, yet the majority of admitted corrections are still accurate. This directly quantifies the performance of the admission gates under learned-model errors. (revised page 46, line 2438)

  3. Random errors: examined separately from the systematic errors of the learned model.

    The original corridor experiment tested errors that vary randomly across each replay. A learned model, by contrast, may repeatedly give erroneous judgments in the same state. The newly added experiments directly inspect what proportion of corrections passing the admission gates remain accurate under such circumstances, so the examination is no longer limited to Gaussian noise. (revised page 38, line 2029; revised page 46, line 2438)

AuthorsOpenReview ↗

Response to Reviewer nXor (2/8): W2

W2.

We have unified the algorithmic procedure of PAIN-PPO, rerun all four arenas with the same update equation, and added sensitivity analysis experiments. The unified method still maintains its advantage over the baselines across all four arenas; in the sensitivity experiments, the method's advantage also holds after varying the tested gate thresholds and retention parameters. Specific evidence is as follows:

  1. Unifying the algorithm, with the corridor and the execution task also adopting the update equation analyzed in Section 4.

    All four arenas share the pipeline of "three-gate correction filtering → one-sided memory constraint → norm-capped update," all using Eq. (4) to synthesize the update gradient and removing the conflict veto term in the update. Main training uniformly uses αP=1000\alpha_{\mathrm P}=1000 and η=0.5\eta=0.5. (revised page 5, line 232; the two constants, revised Table 22, page 63, line 3349; Algorithm 1, revised page 57, line 3024)

    The cap ρP\rho_{\mathrm P} is calibrated by arena: corridor, sepsis, and maze use the same selection rule on independent seeds, while the execution task is set to 0.9 based on training logs without reading the test months. Task-specific settings—including inspection rules, probes, admission thresholds, state matching, and memory refresh—and how they are determined are itemized in Appendix H.2 and Table 20. (revised page 5, line 232; revised Appendix H.2.5, page 60, line 3201; revised Table 20, page 62, line 3295)

    The results after the unified update are as follows. (revised Table 22, page 63, line 3349)

    ArenaρP\rho_{\mathrm P}Results
    Corridor0.25Median tail gain on new seeds is +42.6, with 18/20 feasible
    Execution0.9Average tail increment of the Pain channel is +8.60 bp, with 15/15 seed-months having positive increments and being feasible
    Sepsis0.05Relative to PPO without Pain, the return gain is +0.0449; on another 20 new seeds it is +0.0329
    Maze120/20 converge on new seeds, compared to 16/20 for Return Capping
  2. Adding sensitivity experiments, where the advantage holds after varying the tested parameters.

    We vary one parameter at a time, keep the remaining settings unchanged, and retrain. Variations in the retention fraction and gate threshold in the corridor do not change the number of feasible seeds, and the tail changes are also very small; the number of converged seeds in the maze remains close to the default setting; and the tail changes in the execution task are less than 2 bp. (revised page 61, line 3293; revised Table 20, page 62, line 3295; revised Table 21, page 62, line 3330)

    ParameterArenaSwept ValuesResults
    Retention fraction η\etaCorridor0.3, 0.7, 0.9, default 0.5All are 39/40 feasible; paired changes in tail gain do not exceed 0.17
    Direction consistency thresholdCorridor0.4, 1.0, default 0.7All are 39/40 feasible; maximum tail change is 0.08
    Direction purity thresholdMaze0.5, 0.7, default 0.6103/120 and 102/120 converge, compared to 105/120 for default
    Minimum supporting realizations nmin⁡n_{\min}Maze1–418–19/20 converge
    Minimum supporting realizations nmin⁡n_{\min}Execution1, 4, default 2All are 15/15 feasible, with tail changes less than 2 bp
    Verification tolerance ϵnoise\epsilon_{\mathrm{noise}}Maze0–318–20/20 converge

    (revised page 62, line 3323)

AuthorsOpenReview ↗

Response to Reviewer nXor (3/8): W3

W3.

We have added theory directly targeting the value of corrections and memory retention. The newly added results provide sufficient conditions for corrections to improve tail returns as well as their mean cost, and under an explicit risk model, prove that corrections are formed from an empty memory, learned, and retained, ultimately yielding positive tail gains.

Proposition 1, Theorem 1, and Proposition 2 characterize the update form, the PAIN update residual bound, and existence separation in specific instances, respectively. The newly added gain conclusions are given by Proposition 5 and Theorem 2, specifically as follows:

  1. When corrections are beneficial and when they incur a cost: Proposition 5 provides a direct return criterion.

    Keeping the subsequent policy in replay fixed and applying the correction at a specified decision point with probability u>0u > 0, denote its return change under the same realization as Δ\Delta; then

    CVaRα(Gu)−CVaRα(G)≥uα(Δ+−Δ−),\displaystyle \mathrm{CVaR}_{\alpha}(G_u)-\mathrm{CVaR}_{\alpha}(G) \ge \frac{u}{\alpha}(\Delta_+-\Delta_-), E[Gu]−E[G]=u,E[Δ].\displaystyle \mathbb E[G_u]-\mathbb E[G]=u\\,\mathbb E[\Delta].

    Here, Δ+\Delta_+ measures the truncated gain on originally tail samples, and Δ−\Delta_- measures the loss from pulling originally non-tail samples into the tail. Therefore, when Δ+>Δ−\Delta_+ > \Delta_-, the correction is guaranteed to improve tail returns; when E[Δ]<0\mathbb E[\Delta] < 0, it reduces mean returns. This conclusion applies to a single-step action replacement that keeps the subsequent policy fixed. (revised Proposition 5, page 22, line 1185)

  2. Whether corrective updates can resist forgetting: Proposition 3 provides a local criterion.

    For the analyzed Euclidean update, when the corrective term reaches its norm cap in all conditional batches, if ρP>ρt⋆\rho_{\mathrm P} > \rho_t^\star, the aggregate memory loss decreases to first order in conditional expectation; if ρP<ρt⋆\rho_{\mathrm P} < \rho_t^\star, it increases to first order. ρt⋆\rho_t^\star measures the force with which the PPO gradient pushes back against the correction, relative to the magnitude of the PPO gradient. This result applies to any differentiable policy class and clarifies the relationship between the correction budget and forgetting pressure. (revised Proposition 3, page 6, line 316)

  3. From forming corrections to final gains: Theorem 2 provides an end-to-end guarantee.

    Under the specified tail-risk model and explicit parameter conditions, when ρcrit<ρP≤1\rho_{\mathrm{crit}} < \rho_{\mathrm P}\le1, starting from an empty memory, PAIN-PPO achieves a positive tail gain over mean-return PPO across the entire specified time window with probability at least 1−δ1-\delta:

    Pr⁡!(CVaRα(GθtPAIN)−CVaRα(GθtPPO)≥Δtailfor all T0≤t≤T)≥1−δ,\displaystyle \Pr\\!\left( \mathrm{CVaR}_{\alpha}(G_{\theta_t^{\mathrm{PAIN}}}) -\mathrm{CVaR}_{\alpha}(G_{\theta_t^{\mathrm{PPO}}}) \ge \Delta_{\mathrm{tail}} \\ \text{for all }T_0\le t\le T \right)\ge1-\delta, Δtail>0.\displaystyle \Delta_{\mathrm{tail}} > 0.

    The formation and retention of corrections are results of the proof rather than pre-assumed. The theorem also provides an explicit upper bound on the mean loss, as well as an upper bound on the probability of reaching a specified correction level when the budget is insufficient; Corollary 2 further extends the tail gain and mean cost guarantees to two outputs sharing parameters. (revised page 7, line 333; mean cost, revised page 24, line 1294; explicit conditions, revised page 25, line 1308; Corollary 2, revised page 29, line 1512)

Retention experiments in actual training also provide corresponding evidence: when turning off versus retaining the Pain channel, the seeds that lose learned corrections are 62/68 vs. 27/68 in the maze, respectively, and 11/39 vs. 2/39 in the corridor, respectively. These results support the retention role of continuous corrective updates. (revised page 63, line 3370)

Reviewer_vSftOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PAIN-PPO, a policy optimization framework that combines PPO with counterfactually verified correction memory. The method addresses two difficulties: distinguishing poor decisions from adverse outcomes caused by exogenous randomness, and preventing useful corrections from being forgotten during subsequent policy updates.

For selected failures, alternative actions are evaluated through counterfactual rollouts under the same realization of environmental randomness. Corrections supported across distinct realizations are retained as one-sided policy constraints, whose gradients selectively modify PPO updates. The paper provides theoretical results on adaptive regularization and update behavior, together with experiments in controlled environments, a sepsis simulator, and historical financial order execution.

Critical strengths
  • An interesting approach to counterfactual policy correction. Comparing alternative actions under identical environmental realizations can reduce exogenous noise in action evaluation. The framework offers a practical mechanism for identifying potentially avoidable failures without directly modifying the underlying reward or advantage estimator.

  • Informative experiments across different environments. The experiments distinguish avoidable from unavoidable failures and examine robustness under noisy verification. The Guarded Maze and sepsis experiments are encouraging, while the financial results suggest potential improvements in lower-tail execution performance. The paper also provides useful ablation studies.

Critical weaknesses
  • Insufficient justification for the reliability of counterfactual action comparisons. A positive counterfactual gain under a particular realization does not necessarily imply a positive difference in conditional expected returns or improvement in the relevant risk-sensitive objective. Cross-realization directional agreement partially addresses this issue but does not fully resolve selection bias arising from the choice of adverse episodes and candidate actions. Appendix H.2 acknowledges limitations of the idealized statistical assumptions. The theoretical results characterize the update mechanism but do not establish general policy improvement guarantees.

  • Limited evidence of practical advantages and generalizability. The approach requires additional counterfactual rollouts and an environment capable of replaying alternative actions under matched exogenous conditions. Its computational advantages over simpler rollout-based action evaluation or model-based actor-critic methods remain unclear. The financial evaluation is also limited to 26 independent out-of-sample trading days. The authors acknowledge that the resulting uncertainty is too large to establish the tight mean-performance constraint at the confidence-interval level. Furthermore, historical order-book replay cannot fully account for endogenous market responses.

Questions for authors

The proposed approach is interesting, but its theoretical and practical advantages could be established more convincingly.

  1. Reliability of counterfactual comparisons. Could the authors evaluate whether the identified corrections agree with independently estimated action-value differences or improvements in the relevant risk-sensitive objective? How sensitive is the false-admission rate to the number of independent realizations and the selection of candidate actions?
  2. Computational efficiency. How does PAIN-PPO compare with simpler rollout-based action evaluation or model-based actor-critic methods under matched total simulation budgets? In particular, when does paired counterfactual verification provide a meaningful computational or statistical advantage?
  3. Robustness of financial results. Given the limited number of independent trading days and the uncertainty surrounding the mean-performance constraint, how reliably can the reported tail improvements generalize to additional market regimes?
6 comments
AuthorsOpenReview ↗

Response to Reviewer vSft (1/4): Summary, W1

Summary

Thank you for directing your concerns at the core of the method. The two main ones are whether a single counterfactual comparison can count as evidence, and what a paired comparison gains over plain rollout evaluation. On the first, a single comparison is only a candidate in our method, and a correction enters memory only when the same direction recurs on different realizations of the randomness. Checked against the exact conditional expectation, about nine in ten admitted corrections are right. On the second, pairing removes exactly the luck that does not depend on the action, so at the same number of simulator calls its variance is about 850 times smaller. On the financial results, the test set has grown from three months and 26 days to five months and 46 days, and the tail improvement holds in the two added months.

W1.

On the reliability of counterfactual comparisons, in our method a correction enters memory only after the same direction recurs on at least two distinct realizations, and a gain on a single realization does not count as evidence (revised Section 3.1, page 4, line 194). In the sepsis simulator we computed the conditional expected gain DD of each correction exactly by dynamic programming, and about nine in ten corrections admitted this way are better in expectation, not only on the realizations that were recorded. Of the 287 admitted corrections, 90.6% have D>0D > 0, with a 95% interval of [86.1%, 94.2%] (new, revised Appendix F.3.3, page 46, line 2450).

On selection bias, a candidate is picked from failed episodes, which makes its gain look larger than it is. In this experiment each candidate is therefore verified again on a separate set of episodes that is not selected by outcome. The more episodes are used for verification, the smaller the bias. The table shows the same candidates under different numbers of verification episodes (revised Table 9, page 46, line 2471).

Verification episodes per candidateAdmitted corrections that are better in expectation (D>0D > 0)Overestimate of the gain
4050.7%+0.165
10069.7%+0.081
30084.2%+0.033
1,500 (used in the paper)90.6%+0.013

With only 40 verification episodes, half of the admitted corrections are better in expectation. With the 1,500 used in the paper, nine in ten are, and the overestimate of the gain falls from 0.165 to 0.013.

On Appendix H.2, it says that the admission thresholds are values we chose by design and not significance levels of a statistical test. The reliability of the method therefore does not rest on the statistical assumption discussed there. It is measured directly against the environment's ground truth, as above (revised page 59, line 3185).

On the theory, the revision adds two results in Section 4.3. Proposition 3 holds for any differentiable policy. It defines the cost-to-budget ratio ρt⋆\rho_t^\star, the pull of the base gradient against the corrections relative to the size of the base gradient. When the cap ρP\rho_{\mathrm P} of the update is above this ratio, the stored corrections are kept in expectation, and when it is below they are eroded (revised page 6, line 316). Theorem 2 proves, in a tail-risk model and starting from empty memory, that the correction forms and is retained, and it gives the tail gain (revised page 7, line 337),

\displaystyle \Pr\Big\\{\mathrm{CVaR}_\alpha\big(G_{\theta_t^{\mathrm{PAIN}}}\big)-\mathrm{CVaR}_\alpha\big(G_{\theta_t^{\mathrm{PPO}}}\big)\ge\Delta_{\mathrm{tail}}\\ \text{ for all } T_0\le t\le T\Big\\}\ge 1-\delta,

with explicit T0T_0 and Δtail>0\Delta_{\mathrm{tail}} > 0. This is a guarantee in that model. In general environments the reliability rests on the measurements of the conditional expectation above.

AuthorsOpenReview ↗

Response to Reviewer vSft (2/4): W2

W2.

On the need for a replayable environment, replay is used only during training to verify corrections. Training yields an ordinary policy network, and deployment needs no replay (Section 2.2, revised page 3, line 161). Replay also need not be exact. In the sepsis simulator, with the verifier replaced by a model fitted to data, the gain after training is +0.0415, against +0.0449 with exact replay, provided the logged data contain the counterfactual actions (new, revised Appendix F.3.2, page 46, line 2434).

On the extra cost, we measured the verification volume and the training time in all four arenas (revised Table 2, page 35, line 1847). Replays are counted in the interaction budget of each arena (Q2).

ArenaVerification volumeTraining time, relative to training without Pain
Corridorclosed form, no simulator call1.4 times
Guarded Mazeabout 0.96 million replay steps per run, equal to 48% of the training steps1.12 times
Sepsisone evidence store built before training, about 380,000 replays1.2 times in the teaching phase
Executionone refresh every 10 updates, about 6,200 episodes each1.14 to 1.20 times

On the advantage over simpler rollout evaluation, what pairing buys is sample efficiency. In the corridor, with the same two simulator calls per candidate, verification by independent rollouts gives a tail gain of only +1.9, and paired replay gives +32.6. With 4,096 samples per action, independent rollouts also reach +35.1. Independent rollouts can therefore replace pairing, but they need about four thousand times as many simulator calls, because pairing removes the luck that does not depend on the action and the variance of its estimate is about 850 times smaller (revised Appendix F.1.5, page 41, line 2186). The full comparison at matched budgets is in Q2.

On model-based actor-critic methods, we tested their most favorable case, in which the model has no error and the extra simulated episodes come from the true simulator. We let PPO, which is itself an actor-critic, run twice as many episodes in every update. Its median tail gain is only +0.84, against +47.4 for PAIN-PPO, and PAIN-PPO leads on 36 of 40 seeds (corridor, new). Even with a perfect model, more simulated data alone does not give the tail gain of PAIN-PPO. The difference is in how simulation is used, since PAIN uses it to verify specific actions by paired comparison.

On the 26 trading days and the mean constraint, we added the data of September and October after the submission, which extends the test set from three months and 26 days to five months and 46 days. The table gives the tail increment of the Pain channel, that is PAIN-PPO minus the same system retrained with the Pain channel off, with intervals that resample trading days. The interval excludes zero in all five months, so the tail result holds at the level of trading days (revised Tables 14 and 15, page 49, line 2610, and page 50, line 2647).

Test monthTrading daysTail increment of the Pain channel (bp)95% interval
June8+7.0[+2.9, +11.3]
July10+10.8[+3.3, +19.9]
August8+8.0[+5.1, +10.8]
September (new)10+4.7[+0.8, +9.4]
October (new)10+7.4[+3.7, +11.3]

The mean budget is a line at −0.2 bp. A single month has only 8 to 10 trading days and the daily standard deviation is about 13 bp, so an interval of ±0.2 bp would need about 16,000 independent trading days. The mean budget is therefore judged on point estimates, as pre-registered, which is also how execution studies report costs (Nevmyvaka et al., 2006; Fang et al., 2021; Zhang et al., 2023), and all 25 seed-months are feasible. Pooled over the five months an interval can be given. The mean is +6.19 bp with an interval of [+2.94, +9.75], which lies entirely above the budget line (revised page 50, line 2680).

On endogenous market response, replay on the historical order book is the standard evaluation for low-participation execution strategies (Nevmyvaka et al., 2006; Fang et al., 2021; Zhang et al., 2023), and it does not model how other participants react to the agent. We added the check that historical data does allow. We estimated the impact on these data and redid the 16,051 paired verifications behind the admitted corrections under it. After this, 92.2% of the corrections still have a majority of positive supporting evidence, and 87.4% at twice the estimated impact (new, revised Appendix F.4.3, page 52, line 2759). The case in which other participants adapt their behavior to the algorithm has to be measured in live trading with paired randomized deviations, and we state it as a condition for applying the method.

AuthorsOpenReview ↗

Response to Reviewer vSft (3/4): Q1, Q2

Q1. The identified corrections agree with independently estimated action-value differences or improvements in the risk-sensitive objective. In the sepsis simulator we computed the action-value difference of each correction independently and exactly by dynamic programming, that is, how much the expected return rises when the action is changed to the corrected one. In direction, 90.6% of the 287 admitted corrections have a positive difference, so the actions we judged better are indeed better by the independent computation. In size, corrections with a larger gain under paired replay also have a larger independent difference, with a Spearman correlation of 0.68. Conversely, a direction whose difference is not positive is admitted with probability only 0.5% (new, revised Appendix F.3.3, page 46, line 2458).

We tested the sensitivity to the number of independent realizations in the corridor. A parameter of the method sets on how many distinct realizations a correction must point in the same direction before it is admitted, and we varied it from 1 to 6. Failures in the corridor are of two kinds, those the action could have avoided and those it could not, and the kind of each failure is known, so a correction admitted from a failure of the second kind is a false admission (new, revised Appendix F.1.3, page 39, line 2086). With ample data the parameter makes no difference, and the false-admission rate is below 0.1%. At training scale each state collects only a few realizations, and a failure of the second kind occasionally shows the same direction on two of them by chance. Only then does the parameter matter, with the results below (revised page 40, line 2132). Requiring more realizations lowers false admissions and also loses more true corrections, and the paper uses 2.

Realizations required to agreeFalse-admission rateTrue corrections retained
19.3%100%
2 (used in the paper)6.6%98.7%
33.5%97.4%
60.2%86.0%

The false-admission rate is not sensitive to the choice of candidate actions. Changing the probe amplitude from 0.2 to 0.1 or 0.3, or probing only the one or three steps of each failed episode with the largest gain, leaves it below 0.1% (revised page 39, line 2099).

Q2. At the same simulation budget PAIN-PPO does better. The advantage of pairing is statistical, and it turns directly into a saving in computation, since simple evaluation needs about four thousand times as many simulator calls for the same tail gain. PAIN-PPO itself takes 1.12 to 1.4 times the training time of the same system without Pain (W2).

Take simpler rollout evaluation first, which runs each of the two actions on fresh randomness and compares the outcomes. We replaced paired verification with it in the corridor and changed nothing else. The tail gains are below (revised Appendix F.1.5, page 41, line 2186). With the same two calls, pairing leads by 32.5, and the interval excludes zero. Simple evaluation needs about four thousand times as many calls to catch up, and one training run then takes about 20 minutes, against about 6 seconds with pairing.

Simulator calls per candidateSimple rollout evaluationPairingPairing minus simple [95% interval]
2+1.9+32.6+32.5 [+17.9, +53.4]
2,048+28.0+32.6+0.87 [+0.06, +2.28]
8,192+35.1+32.6−1.22 [−2.31, −0.96]

Now model-based actor-critic. We tested its most favorable case, in which the model has no error, by letting PPO run twice as many episodes of the true simulator in every update. In the other arenas the comparison gets the same total interaction or more, with replays counted. The results are below (corridor: new; execution: revised Appendix F.4.5, page 54, line 2912; maze: revised Appendix F.2, page 42, line 2219).

ArenaSimulation given to the comparisonResult
CorridorPPO with twice the episodes in every updatetail gain +0.84, against +47.4 for PAIN-PPO, which leads by 46.1 [+19.7, +61.6]
Executionthe system without Pain trained for 160 updates, at least the interaction of PAIN-PPO at 120 updatesPAIN-PPO tail still higher by 11.3 [+5.3, +17.1], 7.9 [+5.1, +10.1], and 8.8 [+5.3, +12.0] bp in the three months
Guarded Mazereplay steps counted, both at about 1.5 million interactionsseeds meeting the criterion: 102 for PAIN-PPO, 76 for Return Capping (p = 2 × 10⁻⁴)

When pairing helps depends on how much of the return is luck. Pairing lets the two actions face the same luck, which cancels in the difference, and in the corridor the variance of the estimate is about 850 times smaller. In the sepsis simulator the return is driven mainly by disease progression and luck plays a smaller part, so simple evaluation at the same sample size as pairing already gains +0.049 [+0.031, +0.067] (revised page 41, line 2198).

Reviewer_cy3FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors claim that a bad outcome doesn’t always mean the agent acted badly, so an RL agent shouldn’t learn the same lesson from every failure. It proposes PAIN-PPO, which adds Pain memory on top of Proximal Policy Optimization (PPO). For a severe episode, it replays the same recorded randomness with a different action at a chosen step. A correction is kept only if the alternative does better. It only gains influence after the same correction direction shows up on at least n_min different tapes (the authors call one recording a tape). Each stored correction becomes a one-sided constraint at a stored state, and its gradient is added to PPO’s with a norm cap, plus a small veto that removes conflicting PPO components. The theory rewrites the update as PPO plus an adaptive one-sided regularizer (Proposition 1), bounds the residual of this update in finite time (Theorem 1), and builds a two-context bandit with shared parameters in which PPO converges to return 1/2 and PAIN-PPO to 1 (Proposition 2). Experiments use a synthetic Clean Corridor with labeled failure classes, the continuous Guarded Maze on top of Return Capping, the Oberst & Sontag sepsis simulator treated as a Gumbel-max SCM with a dial on the share of avoidable deaths, and optimal execution on replayed A-share order-book data with three test months, where PAIN-PPO is one layer of a system that also contains an admission classifier and a regime layer.

Critical strengths
  • The proposed method decides whether an action was at fault before remembering the failure and tests it in settings with known ground truth. It tunes the dial until no death is avoidable, resulting in exactly zero gain. Reading corrections off observed associations at the same failure points gets about a tenth of the gain, with an interval that includes zero.
  • The admission principle is well designed, and the controls that isolate it are the strongest part of the paper.
  • The experiments are well designed, and criteria were pre-registered. The maze uses the baseline authors’ own code over 120 seeds with Fisher tests, and the comparison counting replay steps holds correctly (109 vs. 76 of 120).
  • Comparisons are paired by seed, tape, or trading day; uncertainty is reported at the level of independent units (seeds, trading-day clusters). This makes the claims checkable.
Critical weaknesses
  • The method relies on a replay oracle, but the results don’t show how much they depend on it. PAIN needs to replay an episode under the same noise with a different action. This condition can be achieved in simulators, but in the real-world setting it’s only an approximation. The market replay assumes the agent’s own orders don’t change the participants’ behavior.
  • The impact check in Table 12 is an open-loop penalty that the authors themselves describe as assumed, but it should be estimated. Therefore, it doesn’t test whether the counterfactual gains that drive admission would survive a closed-loop market.
  • On the forgetting side, gradient-memory and regularization methods (A-GEM, EWC) are cited, but they are not run as baselines.
  • The system has three layers: an admission classifier trained on counterfactual labels, the policy, and a regime layer. The Pain-off retrain already keeps two-thirds to four-fifths of the tail gain, and without the admission classifier, the budget is not sufficient. Therefore, most of the gain comes from the counterfactual-label classifier, not the memory.
  • The method is complicated, and the theory does not strongly support it. Theorem 1 is the standard nonconvex SGD bound, and it assumes V_T = o(S_T), which is exactly the property of the memory that would need proving.
Questions for authors
  • What exactly is the tail condition?
  • How many replay queries does verification take per update in each arena, and what’s the wall-clock overhead compared with PPO?
  • Does PAIN still win when the baselines get the same number of extra environment steps everywhere, not only in the maze?
  • In the corridor, which arms had states without a lesson anchored to the shadow backbone? Please report supervised regression and failure memory with the same anchor, and PAIN-PPO without it.
  • Can a single specification of PAIN-PPO be run in all four arenas, or at least in the corridor and execution?
  • How sensitive are the results to n_min and ε_noise in the maze and the execution task? Fig. 3a only shows the corridor.
  • How were the 67 trading days chosen, given that January to August 2026 contains far more trading days?
  • How many admitted corrections come from the few sharp-drop days? Is the tail gain still there if you drop the 2–3 worst market days in each test month?
  • Were the June to August results of the development arms (b ≤ 0.3, 60 updates) seen before b ≤ 0.2 and 120 updates were fixed?
  • Can you add later months that played no role in development?
  • How do you reconcile the per-regime constraint, which binds on ordinary days, with an out-of-sample ordinary-day ΔMean of −0.9 to −1.6 bp?
  • Have you tried any closed-loop check of the no-impact assumption? A correction admitted under the open-loop replay could be wrong once impact is closed-loop.
  • How does PAIN compare with A-GEM/EWC-style retention?
9 comments
AuthorsOpenReview ↗

Response to Reviewer cy3F (1/9): Summary, W1

Summary

We thank you for the careful reading and for recognizing the admission principle. We respond to your three main concerns as follows. First, in execution, the Pain channel adds a tail gain of its own. The increment remains when the admission classifier is switched off and when the regime layer is removed, and it holds in September and October, which we added after submission. Second, exact replay is not required, and the ranking of the systems holds under estimated market impact. Replay is used only in training. The method still works with independent sampling or with a learned model whose data cover the counterfactual actions. The impact is estimated on our own data. Third, PAIN-PPO is one algorithm. One fixed specification was rerun in all four arenas, and every main result holds. We also added the EWC and A-GEM baselines. We answer each point below in your order.

W1.

An exact replay oracle is not a necessary condition for obtaining gains, though PAIN-PPO requires a counterfactual verification interface during the training phase. The revision examines the dependence on same-noise pairing and an exact replay model through two replacement experiments—independent sampling and a learned-model verifier, respectively: the former primarily affects sample efficiency, while the latter still retains gains when the data covers the actions being compared. Counterfactual queries are used only during training, and deployment does not require the real environment to re-experience past random events (revised page 3, line 161). The exogeneity assumption of market replay has precedents in execution studies, and we explicitly state it as an applicability condition. Specific evidence is as follows:

  1. Independent sampling experiments quantify the dependence on same-noise pairing. In the corridor, when each action is sampled once, the tail gain of independent verification is +1.9, compared to +32.6 for paired verification; after increasing independent sampling to 4,096 times per action, the gain reaches +35.1. In sepsis, under the same sample budget, independent verification achieves a gain of +0.049 [+0.031, +0.067] over the backbone, with 15 out of 16 seeds being positive. (Appendix F.1.5, revised page 41, line 2186)

  2. The learned-model verifier still retains gains. In sepsis, after fitting the verifier using 100,000 random treatment trajectories, the precision of accepted corrections is 88.2%, compared to 94.6% for the exact verifier. Evaluated in the original simulator, the gain of the model verifier over the backbone is +0.0415 [+0.0207, +0.0632], compared to +0.0449 [+0.0250, +0.0671] for the exact verifier. (revised Appendix F.3.2, page 46, line 2434)

  3. Market exogeneity is a retained applicability boundary.

    Historical replay assumes that this policy's orders do not alter the behavior of other participants. This assumption has precedents in execution studies. Nevmyvaka et al. (2006) make the same assumption explicitly, Fang et al. (2021) train and evaluate on historical data and assume that the market returns to equilibrium at the next timestep, and Zhang et al. (2023, §3.2) treat the market variables as a context that the policy's actions do not affect. We have explicitly stated this assumption in the revision (revised page 61, line 3279). We further examine the stability of results under order impact using a data-estimated impact model; see W2.

AuthorsOpenReview ↗

Response to Reviewer cy3F (2/9): W2, W3

W2.

We have estimated market impact on this dataset and re-examined policy performance as well as the counterfactual gains driving admission accordingly. The results show that even when the estimated impact is doubled, PAIN-PPO still maintains its tail performance advantage. The newly added results are shown in Table 18 of the revised manuscript, page 52, line 2780. Specific evidence is as follows:

  1. Impact intensity is estimated from the data.

    We adopt the data calibration method (Busseti & Lillo, 2012, §5.1) of the propagator model (Bouchaud et al., 2004) to estimate market impact across 71,179 trading windows from June to August. Converted according to the definition in Table 18, the per-stock median of the equivalent permanent impact coefficient is 11.9 bp, which is higher than the original sweep upper bound of 10 bp. (revised page 52, line 2759; revised page 52, line 2770)

  2. After re-examination, the majority of admitted corrections remain supported by positive evidence.

    We recalculated the 16,051 paired verifications behind the 4,350 admitted corrections. After adding the estimated impact, 92.2% of the corrections are still supported by majority positive evidence; after doubling the impact intensity, this proportion is 87.4%. (revised page 52, line 2771; revised page 53, line 2824)

  3. Policy performance is also maintained under the estimated impact.

    Across the 15 seed–months from June to August, tail performance consistently maintains the ranking of PAIN-PPO > Pain-off retrain > supervised selector, with all satisfying the mean budget. After doubling the estimated impact, both the ranking and feasibility are preserved; under both intensities, the minimum seed–month tail advantage of PAIN-PPO over the Pain-off retrain is 3.7 bp. (revised page 52, line 2767 and Table 18)

These newly added tests directly cover the local action return differences driving admission, supporting that the direction of evidence for most corrections remains stable under the estimated price impact. The tests are conducted on fixed execution trajectories; the impact estimates are derived from market-wide order flow and incorporate the information carried by trades. (revised page 51, line 2750; revised page 52, line 2776) Indirect impacts caused by other participants adjusting their own behavior in response to the algorithm cannot be modeled from historical data (Cheridito & Weiss, 2025) and can only be measured in live trading using paired randomized deviations; we do not generalize our results there.

W3.

We have added A-GEM and EWC baselines and compared them under otherwise identical experimental conditions. The results show that PAIN-PPO outperforms both baselines in retaining learned corrections. The newly added results are shown in Table 8 of the revised manuscript, page 45, line 2407. Specific evidence is as follows:

  1. Comparison under identical conditions. The experiments use 24 seeds, with each method using the same initial policy, subsequent training steps, optimizer, and evaluation pipeline. (revised page 45, line 2385)

  2. PAIN-PPO achieves better retention of learned corrections.

    MethodProbability of Verified Action After Continued Training
    No memory0.754–0.802
    EWC, λ = 0.40.823–0.838
    A-GEM0.927–0.958
    EWC, λ = 4000.963–0.972
    PAIN-PPO0.999

    The table shows the range of the 24-seed medians across four levels of old-state occurrence frequency. The paired difference intervals in retention probability of PAIN-PPO relative to EWC and A-GEM are all above zero. (revised page 45, line 2400)

  3. Under the setting where old states occur normally, the return on the new patient cohort also outperforms strong EWC. Under the setting where old states occur at their natural frequency (f=1), PAIN-PPO achieves a return on the new patient cohort that is 0.024 [0.001, 0.057] higher than strong EWC (λ=400\lambda=400). Therefore, under this setting, PAIN-PPO simultaneously achieves better correction retention and new patient cohort return. (revised page 45, line 2395)

AuthorsOpenReview ↗

Response to Reviewer cy3F (3/9): W4, W5

W4.

We have added an ablation study of "admission classifier on/off × Pain channel on/off". The results show that when the classifier is turned off, the Pain channel still brings significant tail gains, and PAIN-PPO also still satisfies the mean budget. Retraining without Pain indeed retains a substantial portion of the total tail gain; we define the contribution of Pain as the increment over this retrained baseline. (revised page 10, line 511) Specific evidence is as follows:

  1. When the classifier is turned off, the tail increment of Pain is still maintained.

    The table below shows the tail gain of PAIN-PPO relative to the Pain-off retrain, in bp:

    Admission ClassifierJuneJulyAugustSeptemberOctober
    On+7.0+10.8+8.0+4.7+7.4
    Off+7.7+11.3+9.4+6.3+9.1

    Under both settings, the 95% intervals of the increments across all five months are above zero. Therefore, the increment of the Pain channel does not depend on the classifier being turned on. (Table 17 of the revised manuscript, page 51, line 2719)

  2. When the classifier is turned off, PAIN-PPO still satisfies the mean budget.

    In the revised manuscript, regardless of whether the classifier is turned on or off, PAIN-PPO satisfies the mean budget across all 25 seed–months over the five months. When the classifier is turned off and the Pain channel is further removed, all 5 seeds in August violate the budget. Therefore, the claim that "the budget cannot be satisfied without the classifier" is inconsistent with the results in the revised manuscript. (page 51, line 2708)

  3. The supervised selector on the same batch of counterfactual evidence is also inferior to the execution policy.

    For example, in July, the tail gains of the Supervised Selector, the Pain-off retrain, and PAIN-PPO relative to POV are 23.6, 54.2, and 65.0 bp, respectively, and the ranking is the same in June and August. This indicates that the gain retained by the Pain-off retrain cannot be directly equated to the contribution of the classifier; it also includes the gain of the base execution policy. (revised page 2, line 93; Table 12 of the revised manuscript, page 48, line 2546)

W5.

The revision has uniformly presented PAIN as "three gates + one norm-bounded update," and the four arenas share this algorithmic procedure; see Q5 for specific settings. (revised page 5, line 232; Algorithm 1, revised page 57, line 3024)

We have added theory directly targeting the memory mechanism. The newly added results do not rely on V_T=o(S_T), and under the specified model and explicit parameter conditions, prove that corrections are formed from an empty memory, learned, and retained, ultimately yielding positive tail gains. Specific additions are as follows:

  1. Theorem 2 directly proves the connection between memory formation, retention, and tail gains. Under the tail-risk model and explicit parameter conditions, when ρcrit<ρP≤1\rho_{\mathrm{crit}} < \rho_{\mathrm P}\le1, starting from an empty memory, PAIN-PPO achieves a positive CVaR gain over mean-return PPO across the entire specified time window with probability at least 1−δ1-\delta. The successful formation and retention of corrections are results of the proof rather than pre-assumed. (revised page 7, line 333; see revised page 25, line 1308 for explicit conditions)
  2. Numerical verification and retention experiments provide corresponding evidence. Across 20,000 simulated trajectories in the model formation stage, 97.8% form the correct memory within 53 steps, with a theoretical lower bound of 97.49%. (revised page 33, line 1752) In retention forks during actual training, when turning off versus retaining the Pain channel, the regressing seeds in the maze are 62/68 vs. 27/68, respectively, and in the corridor are 11/39 vs. 2/39, respectively, with a paired test in the corridor of p=0.004. These experiments support the retention role of continuous corrective updates. (revised page 63, line 3370)

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the four reviewers for their careful comments. The revised paper is uploaded, with the main additions and changes marked in blue. Following the reviewers' suggestions, we made two major changes: PAIN-PPO is now one algorithm, and we added an end-to-end theory. We also added evidence on verification reliability and on the financial results. Point-by-point answers are in the individual responses.

1. What is unified: PAIN-PPO is now one algorithm. All four arenas now share the same verification gates, memory rule, and gradient-capped update, with only the cap calibrated per arena (revised Section 3, page 4, line 166). All main results have been rerun, and the conclusions remain unchanged.

2. What is proved: starting from empty memory, a correction forms, is kept, and yields a tail gain. This is the question that the new Section 4.3 answers (revised page 6, line 308).

Proposition 3 gives, for any differentiable policy, the criterion for keeping a stored correction (revised page 6, line 316). The PPO gradient sometimes pulls against the stored corrections. We call the size of this pull, relative to the size of the PPO gradient, the cost-to-budget ratio. A cap above this ratio keeps the corrections, and a cap below it lets PPO erode them.

Theorem 2 proves the whole chain in a tail-risk model (revised page 7, line 337). When the cap is above an explicit threshold, PAIN-PPO starting from empty memory forms the correction by a fixed time with a prescribed high probability and keeps it for the rest of training, and its tail return exceeds that of PPO by an explicit positive amount. The formation of the correction, its retention, and the tail gain are all proved, and none is assumed. Conversely, when the cap is below a second explicit threshold, the probability that the correction is ever installed is exponentially small, whatever the memory contains.

Actual training agrees with this criterion. The ratio measured from the training logs lies below the calibrated cap in all four arenas (revised Table 23, page 63, line 3386).

3. What the new evidence answers.

  • Verification reliability. Exact dynamic programming confirms positive expected gains for 90.6% of 287 admitted corrections (Appendix F.3.3, page 46).
  • Inexact replay. A learned verifier achieves nearly the same gain as exact replay (+0.0415 vs. +0.0449) when counterfactual actions are covered by the logged data (Appendix F.3.2, page 46).
  • Extended execution tests. Five months (46 trading days) yield a pooled improvement of +6.19 bp over POV (95% CI [+2.94, +9.75]), with all 25 seed-months feasible (pages 49–50).
  • Market impact. Accounting for estimated market impact preserves both system rankings and feasibility across all 15 evaluated seed-months (Table 18, page 52).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.