- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper gives an interesting take on giving rewards in PPO-style RLHF, but instead of punishing bad outcomes, it does a different execution under identical randomness and checks whether you have improvements across different scenarios. Then it starts a connection as a one-sided constraint for future PPOs
I like that they combine causal reasoning with some elve protection against "forgetting". None of the ingredients is new, but the combination is the contribution. The best results are in financial institutions.
To produce the counterfactual verification and one-sided memory constraint. The hope is to retain a correction that we can action on without misstating bad luck for a bad decision.
Critical strengths
The strong point is the causal attribution before memory. I like this idea a lot. Also, the paper gives control ablations that separate verification from retention.
Critical weaknesses
The theory part is considerably weaker than the empirical result suggest.
They don't establish improvement in policy quality or satisfaction of the strict constraints, so for me, this is a red flag. The actual implementation is different from what they analyze.
I like the financial results and what they achieved. At the same do depend on the strong assumption that the market will not react, and also we cannot say that they statistically establish that there is a very tight mean budget :/
Questions for authors
Q1 How are the financial improvements? What would happen if there was some endogenous market response? Are the results robust to this? Can the authors provide any mean budget feasibility beyond the point estimates?
Q2: Wah I would like really is the constraint satisfaction or convergence for the actual algorithm, including when the gradient with PPO vanishes - cna authors provide that?
5 comments
Response to Reviewer y99T (2/5): W2
W2.
On improvement in policy quality, policy quality in this paper is the return in the worst 5% tail under a given mean budget. Theorem 2 of the revision proves its improvement. In a tail-risk model and starting from empty memory, the CVaR of PAIN-PPO exceeds that of PPO by an explicit amount with probability at least , at a controlled cost in the mean (W1, revised page 7, line 337). Empirically the tail improves in all four arenas, as the table below shows.
On satisfaction of the constraints, each stored correction is a one-sided constraint that asks the policy to retain at least part of the verified change. The constraints enter the update through a capped gradient, and the paper does not claim that they are enforced at every step (Section 3.3, revised page 5, line 239). What the theory proves is that they are kept. Proposition 3 shows that when the cap is above the cost-to-budget ratio, the constraint loss falls in expectation to first order. Theorem 2 shows that in its model the correction is installed and held with high probability.
We also measured directly how far the constraints are satisfied in actual training. A policy that has learned its corrections is copied twice, one copy keeps the memory and the other drops it, and both continue training. In the sepsis simulator, after 1,600 further updates on a different family of patients, the probability of the verified actions stays at 0.999 with the memory, against 0.75 to 0.80 without it (revised page 44, line 2375). Entry by entry, only 87 of 1,148 constraint records fall short of their target, by at most 0.032 (revised Appendix H.4.1, page 64, line 3448).
On the implementation differing from the analysis, the revision unifies PAIN into one complete algorithm, three gates and one capped update. The same specification runs in all four arenas, and all main results were rerun under it. The update that the theory analyzes is this update, Eq. (4). Only the cap is set per arena, by one fixed rule (Section 3.3, revised page 5, line 232). We also tested the prediction of the theory directly in actual training. Once the budget is removed, the learned corrections are eroded, and this holds in the maze and in the corridor (W1). The results of this one algorithm in the four arenas are below (revised Table 22, page 63, line 3349).
| Arena | Main result | Retention after continued training, or noisy verifier | |
|---|---|---|---|
| Corridor | 0.25 | median tail gain +42.6, 18 of 20 seeds feasible | under noise the same rule selects , tail gain +47.6 and +45.2 |
| Execution | 0.9 | tail increment of the Pain channel +8.60 bp, positive and feasible in all 15 seed-months | +4.7 and +7.4 bp in September and October, all 10 seed-months feasible |
| Sepsis | 0.05 | +0.0449 over PPO without Pain, and +0.0329 on 20 fresh seeds | probability of the verified actions 0.999, against 0.75 to 0.80 with the memory off |
| Guarded Maze | 1 | all 20 seeds converge, against 16 for Return Capping | 27 of 68 seeds lose the learned route, against 62 with the Pain channel off |
Response to Reviewer y99T (3/5): W3
W3.
On the assumption that the market does not react, replay on the historical order book is the standard evaluation for low-participation execution strategies (Nevmyvaka et al., 2006; Fang et al., 2021; Zhang et al., 2023), and it does not model how other participants react to the agent. We added the check that historical data does allow. We estimated the impact on these data. With the estimated impact added, the ordering and the feasibility of the three systems are unchanged (numbers in Q1). We also redid the 16,051 paired verifications behind the admitted corrections under the estimated impact. After this, 92.2% of the corrections still have a majority of positive supporting evidence, and 87.4% at twice the impact (new, revised Appendix F.4.3, page 52, line 2759). The case in which other participants adapt their behavior to the algorithm has to be measured in live trading with paired randomized deviations, and we state it as a condition for applying the method.
On the mean budget, we added the data of September and October after the submission, which extends the test set from three months and 26 days to five months and 46 days. The mean budget is a line at −0.2 bp. A single month has only 8 to 10 trading days and the daily standard deviation is about 13 bp, so an interval of ±0.2 bp would need about 16,000 independent trading days. The mean budget is therefore judged on point estimates, as pre-registered, which is also how execution studies report costs (Nevmyvaka et al., 2006; Fang et al., 2021; Zhang et al., 2023), and all 25 seed-months are feasible (Appendix E, revised page 34, line 1821). Pooled over all 46 days, the mean improvement over POV is +6.19 bp with a 95% interval of [+2.94, +9.75] (revised page 50, line 2680). September and October alone give +4.71 [+1.24, +9.94] and +12.33 [+2.91, +21.73] (revised page 49, line 2639). All three lower ends are above zero, not only above the budget line.
Response to Reviewer y99T (1/5): Summary, W1
Summary
Thank you for your positive view of attributing an outcome before storing it in memory. Your concerns fall into two groups. The first is theoretical. It asks whether the algorithm we actually run satisfies its constraints and converges, including when the PPO gradient vanishes. The second concerns the financial results. It asks about the assumption that the market does not react to our orders, and whether the 0.2 bp mean budget can be supported beyond point estimates.
For the theory, the revision adds Section 4.3, which addresses the memory directly. Proposition 3 gives the criterion for whether the constraints hold, and Theorem 2 proves, in a tail-risk model and starting from empty memory, the formation and retention of the correction and the resulting tail gain (W1, W2, and Q2). For the financial results, we estimated the market impact on our own data, and the conclusions do not change once the estimated impact is added. On the mean budget, the interval pooled over the five test months and the intervals of the two added months each lie entirely above the budget line (W3 and Q1).
W1. The revision strengthens the theory. The new Section 4.3 addresses the memory directly. Starting from empty memory, it proves the formation and the retention of the correction and the resulting tail gain, and its prediction is tested in actual training (revised page 6, line 308; proofs in Appendices C.3 and C.4).
A one-step law (Proposition 3, for any differentiable policy, revised page 6, line 316). Let gP,t be the gradient of the memory constraint loss, g^P,t its unit vector, and gA,t the PPO gradient. When the cap binds, the expected first-order change of the constraint loss in one update is
Et⟨gP,t,,θt+1−θt⟩=−ηt,∣gP,t∣,Et∣gA,t∣,(ρP−ρt⋆), ρt⋆=Et∣gA,t∣−Et⟨g^P,t,,gA,t⟩.
We call ρt⋆ the cost-to-budget ratio. It is the pull of the base gradient against the stored corrections relative to the size of the base gradient, and it can be measured from the training logs. When the cap ρP is above it, the constraint loss falls in expectation and the stored corrections are kept, and when the cap is below it they are eroded.
An end-to-end theorem (Theorem 2, revised page 7, line 337). In a tail-risk model and starting from empty memory, both the formation and the retention of the correction are proved, and neither is assumed. When the cap is above ρcrit, the largest cost-to-budget ratio over the teaching interval, there are a deterministic T0 and an explicit Δtail>0 such that
\displaystyle \Pr\Big\\{\mathrm{CVaR}_\alpha\big(G_{\theta_t^{\mathrm{PAIN}}}\big)-\mathrm{CVaR}_\alpha\big(G_{\theta_t^{\mathrm{PPO}}}\big)\ge\Delta_{\mathrm{tail}}\\ \text{ for all } T_0\le t\le T\Big\\}\ge 1-\delta.
When the cap is below a second explicit threshold ρ1, the probability that the safe action ever becomes more likely is exponentially small for any memory content, so the correction cannot be installed.
A test of the law in actual training (revised Appendix H.4, page 63, line 3371). The one-step law says that corrections are kept only when the cap is above the cost-to-budget ratio. We measured this ratio from the training logs of each arena, and in the first, middle, and last third of training it lies below the calibrated cap (revised Table 23, page 63, line 3386).
The law also makes a testable prediction: once the budget is removed, corrections that have already been learned are eroded by the base objective. In the maze, after the corrections are learned and training continues with the Pain channel off, 62 of 68 seeds lose them, against 27 with the channel kept. In the corridor the same test gives 11 of 39 seeds against 2 with the budget kept.