acceptodds
Under review as a conference paper at ICLR 2027

DIHAS: Estimating Conditional Action Contributions for Reinforcement Learning in Trade Execution

Abstract

Executing a fixed parent order requires balancing immediate trading costs against exposure to future prices. Because final cost also depends on subsequent decisions and market innovations, it provides limited evidence about the contribution of one current action. We introduce Data-Induced High-Dimensional Action-Conditional Ambiguity Suppression (DIHAS), an action-conditional evidence layer that compares a candidate option with frozen-reference continuation under shared market innovations. A local financial surrogate and bounded residual estimate the resulting paired saving; a paired inventory control variate improves conditional-mean estimation, and ambiguity aggregation evaluates actions under alternative conditional price means. This evidence supplies bounded action biases to DDQN, PPO, and GRPO, while a held-out calibration diagnostic audits overstatement. Full-model training uses a verified terminal paired-saving reward, and final evaluation uses transaction cost. On held-out test dates within each year of the fixed 52-stock Shenzhen replay panel (2019–2021), all nine DIHAS configurations achieve lower three-year mean test-set transaction cost than locally retrained DQN and PPO baselines and TWAP. Diagnostics on Dev, an internal training diagnostic split, show lower mean trace variance with PICV and lower mean pinball loss with the residual across three algorithms. These retraining comparisons use each variant’s own probes; ambiguity and policy-cost effects remain mixed.

Primary area:
reinforcement learning
Keywords:
trade execution, reinforcement learning, conditional action contribution estimation, paired inventory control variate, variance reduction, conditional mean ambiguity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BLHNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DIHAS estimates the cost saving from replacing one execution option while keeping the continuation policy fixed. It combines shared-innovation replay, a financial surrogate, a bounded residual, an inventory control variate, and alternative price-mean scenarios. The estimates bias RL action selection. Experiments report execution costs and estimator diagnostics on a 52-stock Shenzhen panel.

Critical strengths
  1. The paired-saving target is clearly defined. Holding the reference continuation and market innovations fixed gives the action comparison a precise interpretation within the simulator. The PICV derivation states its assumptions, and the separate variance guard provides an empirical check before using fitted coefficients.
  2. The full systems show consistent cost improvements over the local references across years. The date-block analysis in Table A1 is useful because repeated windows and training seeds are not independent market observations. The paper also distinguishes these system-level results from its narrower mechanism diagnostics.
Critical weaknesses
  1. The main comparison does not establish the contribution of the evidence layer. DIHAS uses 12 options and a paired terminal reward, while the local DQN/PPO baselines use 99 primitive actions; additional replay work also differs. The closest evidence-off comparison is limited to three seeds in 2020 and has a schedule mismatch. Its mean result favors evidence-off for DDQN and PPO (Table A6). A fully matched comparison is essential to determine whether the proposed evidence feedback improves execution.
  2. The benefit of the structured estimator is not directly established on a common prediction task. In RQ2, independently retrained variants are evaluated on their own states and reference policies, so lower pinball loss need not reflect a better estimator of the same target. The financial surrogate has no separately reported held-out saving-prediction comparison. The PICV guard supports variance reduction, but does not resolve whether the surrogate and residual improve conditional-mean estimation or action ranking.
Questions for authors
  1. What happens when evidence is disabled across all three years while retaining the same 12 options, terminal reward, training schedule, and seeds? Please report paired test-cost differences and the additional replay cost.
  2. On fixed held-out states with a common frozen reference and replay draws, how do the full estimator, surrogate-only model, and an equally sized unstructured predictor compare in mean-saving error and action-ranking regret?
3 comments
AuthorsOpenReview ↗

Response to Reviewer BLHN

Thank you for asking us to isolate the effect of feedback and evaluate the predictors on the same task. We apologize that the original system comparison varied the options, terminal reward, and replay work together, leaving the feedback's contribution unclear. The original pinball losses also came from different states and continuations, leaving the structured estimator's performance on a shared target unclear. We address two questions. How much does turning feedback off change paired test costs and replay work when the 12 options, terminal reward, training schedule, and seeds match? How do the structured estimator, standalone surrogate, and capacity-matched unstructured predictor compare in mean-saving error and option regret on the same held-out states, frozen reference, and replay draws? We report the matched retraining in RQ4 (Table 3) and the common-state prediction test in Appendix H.

W1 & Q1 | What does matched evidence feedback change?

W1 | What the main comparison measures

The main table compares complete execution systems. DIHAS uses 12 options to express decisions that can adjust price and size for several minutes. Its paired terminal reward uses the cost difference between two complete executions of the same parent order on the same replay path. This ties the training signal to the cost of completing the order. We retain the local DQN and PPO implementations' 99 primitive actions to evaluate those baselines in their configured form. The complete-system comparison therefore includes differences in the action interface, reward, and additional replay. To measure the contribution of evidence feedback, Full and Off share the 12 options and the paired terminal reward. We enable feedback only in Full and report the incremental replay work separately.

In the revised paper. Figure 1 and Section 3 for the paired terminal reward (pp. 2–3), Section 5.1 and Table 1 for the system comparison and compute accounting (pp. 5–6), Appendix F.2–F.3 for the action interfaces (pp. 28–29), and Appendix G for the matched design (p. 31).

Q1 | Matched training and the cost of replay

The 2020 comparison used three seed pairs per backend and mixed planned training horizons. Off minus Full was −0.268 bp for DDQN and −0.185 bp for PPO, favoring Off in both cases (original Table A6). We retrained both arms with the same 12 options, terminal reward, planned 120K-step horizon, and five matched seeds per backend and year. The training-window draws and optimizer random streams follow the same seed rules. Both arms use identical test windows and weights (Appendix G).

Full has lower mean TC in 5 of 9 cells in the original three years (45 pairs) and 9 of 15 through 2023 (75 pairs). Five of the 15 pointwise intervals favor Full, five favor Off, and five include zero. The largest estimated saving is 0.671 bp for 2021 PPO (Table 3).

Table R2. Off minus Full Test TC (bp). Positive values favor Full. Brackets give pointwise 95% date-block intervals for five fixed seed pairs. Data from Table 3.

YearTest datesDDQNPPOGRPO
201943+0.118 [0.079, 0.156]−0.115 [−0.196, −0.023]+0.093 [−0.080, 0.264]
202044+0.067 [0.028, 0.113]+0.011 [−0.216, 0.228]−0.144 [−0.214, −0.074]
202145−0.095 [−0.166, −0.026]+0.671 [0.545, 0.787]−0.135 [−0.193, −0.078]
202244−0.015 [−0.103, 0.073]−0.200 [−0.316, −0.079]+0.278 [0.232, 0.325]
202341+0.076 [0.022, 0.140]+0.023 [−0.041, 0.088]+0.023 [−0.018, 0.063]

By 120K steps, each Full run uses 658,944 additional evidence queries, covering 10.46 to 11.09 million simulated minutes. Annual mean refresh times range from 66.0 to 77.8 seconds. Both arms retain the terminal verifier. Tables S9 and S10 report costs for both arms, paired seed differences, and replay work. Section 5.1 describes the input timing limitation.

In the revised paper. Appendix B.5 for the archived schedule mismatch (p. 18), RQ4 in Section 5.2 and Table 3 for the new comparison (p. 10), and Appendix G, Tables S9 and S10, for seed-paired costs and replay work (p. 31).

AuthorsOpenReview ↗

Response to Reviewer BLHN

W2 & Q2 | How do the predictors compare on common states?

W2 | Quantile loss and a common prediction target

Figure 5 measures pinball loss after independent retraining, with each variant using its own states and continuation. We agree that a lower loss there leaves open which model predicts a common target better. In Appendix H, we fit all three predictors on the same 512 states per year. Each training label averages 16 PICV-adjusted paired savings for one state. The structured and unstructured refits have 100,398 and 100,395 trainable parameters, respectively.

On each year's 256 held-out states, the predictors share the legal options, a frozen DDQN continuation for that year, and 4,096 replay draws per state. We evaluate mean saving in the center scenario. Regret measures the gap in mean saving between the best legal option and the one the predictor selects.

In the revised paper. RQ2 in Section 5.2 (pp. 7–8), Figure 5 (p. 8), and Appendix C.1 (p. 20) for the original diagnostic. Appendix H describes the common task, fitting and test states, and reference policy (pp. 32–33).

Q2 | Standalone surrogate, residual, and the PICV guard

The financial surrogate alone has the lowest mean absolute error (MAE) among the three refits in every year. Adding the bounded residual raises the MAE in this comparison. For the surrogate and unstructured refits, all paired regret intervals against the structured refit include zero (Table S13). These comparisons leave the ranking benefit of the bounded residual unresolved. The PICV guard in Table 2 tests variance reduction in the saving estimate. The common-state results below address prediction error and option ranking.

Table R3. Mean absolute error (MAE) of center-mean savings and option regret (bp). S is the surrogate plus residual, F the financial surrogate alone, and U the unstructured refit. The asterisk marks the original 2019 probe. Data from Table S12.

YearSaving MAE SSaving MAE FSaving MAE UOption regret SOption regret FOption regret U
2019*0.1580.1460.1550.1440.1300.128
20200.1680.1330.1730.1110.0960.110
20210.1240.1110.1260.0960.1060.098
20220.1100.0950.1140.0810.0830.085
20230.0950.0880.0990.0650.0660.063

The deployed model also uses quantiles, positive-saving probabilities, and scenario aggregation. Table S15 evaluates its frozen center-scenario mean output on the same states. This output has higher mean regret than each refit in all five years. In 2023, its MAE and regret are 0.084 and 0.076 bp, compared with 0.088 and 0.066 bp for the financial surrogate alone.

In the revised paper. Appendix H, Tables S12 and S13, for common-task results and paired intervals (p. 33), Table S15 for the frozen deployed model (p. 34), and Table 2 (p. 8) and Appendix C.2 (p. 20) for the PICV variance guard.

AuthorsOpenReview ↗

Response to Reviewer BLHN

Thank you for asking us to isolate what evidence feedback contributes. Appendix G (Table S11, p. 33) separates the effect of turning feedback on at execution time from differences between the policies learned with and without it. On a fixed sample of 2,048 windows for 2020 DDQN, turning feedback on for the same Full checkpoint saved 0.311 bp. With feedback disabled at execution, the policy trained with feedback cost 0.299 bp more than the policy trained without it. This left a net saving of 0.012 bp on that sample. The full test set gave 0.067 bp (Table 3).

The new frozen-policy replays also evaluate feedback using inputs available at decision time. On 225 common trading dates in 2024, the same Shenzhen-trained DDQN checkpoint pairs saved 0.0376 bp in Shenzhen and 0.0533 bp in Shanghai. Both paired 95% date-block intervals were above zero. For PPO and GRPO, the intervals included zero at both venues (Tables 4 and S39). These results concern execution cost. The common-state predictor comparison in Appendix H found the lowest mean absolute error for the standalone surrogate in all five years, with no consistent ranking advantage from adding the residual.

Reviewer_yxQdOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work focuses on order execution tasks with reinforcement learning. Specifically, it introduces Data-Induced High-Dimensional Action-Conditional Ambiguity Suppression, an action-conditional evidence layer that compares a candidate option with frozen-reference continuation under shared market innovations. With 9 different configuration on Shenzhen stock market, the proposed method can lower the execution cost.

Critical strengths
  • RL-based trading execution is an interesting topic with potential for real-world deployment.
  • The proposed component is a general layer useful for various types of RL algorithms.
  • The writting of this paper is easy to follow.
Critical weaknesses
  • In general, the contribution of DIHAS is a bit incremental to me. Adding an extra data-driven layer on existing RL framework can not meet the bar of top AI conferences such as ICLR.
  • For experiments, 52 stocks on Shenzhen stock markets are not enough. Experiments on US stocks can improve this work.
  • The authors use data from 2019-2021. However, the market 5 years ago is usually less efficient than it is now. As the performance of RL-based algorithms is highly relevant to market status, I suggest the authors conducing experiments on different market status (e.g., bull/bear) using recent market data (e.g., 2024-2026).
  • Some theoritical analysis on the effectiveness of proposed method will be very uesful. Otherwise, it will require a large amounts of experiment to make the SOTA performance of proposed methods promising.
  • For order execution, market impact is important. I am curious how the authors modeling market impact as the experiments are runned on some simulations.
  • More details on execution system implementation and how realistic the system is are required.
  • The baseline performance of DQN, PPO is quite bad. What kind of hyperparameters are used?
  • What is the deployment potential of this work? Is it possible to offer some results on live market? Even small scale live results can significantly make the methods more convincing.
Questions for authors

See weakness above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer yxQd

Thank you for asking us to clarify the contribution and broaden the evaluation. We apologize for leaving key controls and implementation details unclear. We added 2022 and 2023 Shenzhen data, extended the evaluation to five years, and added matched evidence tests. The cost of licensing more recent Shenzhen Stock Exchange feeds has limited access. We are exploring ways to obtain those data and will extend the evaluation if we secure access. We also obtained small US tick-by-tick samples to test replay compatibility. Appendix L reports the results.

W1 | Contribution

DIHAS estimates how much the final cost of a parent order changes when we replace only the current execution option. The two branches start from the same state and share the same market shocks. After taking the candidate or reference option, both use the same frozen reference policy. Different fills can leave the branches with different inventories, so that policy can choose different actions later. We subtract the candidate cost from the reference cost to obtain paired saving, the target for training the evidence estimator. The paired inventory control variate (PICV) reduces the variance of this estimate. Full uses predicted savings to adjust action scores within fixed bounds. Off disables the adjustments.

In Appendix G, we freeze the policy and match how often it switches actions and how often a switch returns it to the reference action. For example, in 2020 DDQN, evidence saves 0.168 bp versus random fallback and 0.112 bp versus shuffled scores.

In the revised paper. Sections 3 and 4.1–4.4 define paired saving and feedback (pp. 3–5). The frozen-policy comparisons are in the paragraph after Table S11 in Appendix G (p. 32).

W2 | Stock coverage and US replay

We used the liquidity-screened A-share universe released with Zhang et al. (IJCAI 2023) as our starting point. We then checked whether domestic tick-by-tick order and trade feeds were available and complete, yielding a fixed panel of 52 Shenzhen stocks (Appendix F.1). As a precedent for using a liquid panel, DeepLOB evaluates order-book prediction on five liquid London Stock Exchange stocks with one year of data. We plan to expand coverage and test additional stocks as complete feeds become available. Zhang et al., IJCAI 2023 · DeepLOB, IEEE Transactions on Signal Processing

On US feeds, the matcher completed 1,062 diagnostic orders for AAPL, AMZN, and MSFT using one day of Nasdaq ITCH data. The two-day IEX check used the same 63 parent orders across checkpoints. These tests establish replay compatibility (Section 5.3 and Appendix L).

In the revised paper. Section 5.1 and Appendix F.1 for the panel (pp. 5 and 28). Section 5.3 (p. 10) and Appendix L (pp. 43–44) for the US replay checks.

W3 | Later years and market conditions

All nine DIHAS configurations have lower five-year mean test TC than the five local comparators (Table 1). We fit models separately for each year from 2019 to 2023 and evaluate them on held-out dates within that year (Section 5.1). Table R1 reports paired comparisons of the complete systems under the recorded replay protocol.

Table R1. Base system comparison on the Shenzhen panel. Positive values favor DIHAS. Intervals use trading-date blocks and hold the five fitted seeds fixed. Data from Table A1.

Test periodSystemsBaseline − DIHAS (bp)95% interval
2022Local DQN vs. DIHAS DDQN+4.524[4.030, 5.007]
2022Local PPO vs. DIHAS PPO+4.370[3.666, 5.062]
2023Local DQN vs. DIHAS DDQN+4.256[3.911, 4.606]
2023Local PPO vs. DIHAS PPO+5.415[4.955, 5.867]
Five-year meanLocal DQN vs. DIHAS DDQN+4.436[4.257, 4.614]
Five-year meanLocal PPO vs. DIHAS PPO+5.813[5.533, 6.093]

We group test dates by the previous available trading day's return, realized volatility, and quoted spread, with thresholds set on training dates. In all three 2022 groups based on prior return, Full GRPO has lower mean TC than Off (Table S18). In 2023, the estimated mean TC difference (Off minus Full) for GRPO was −0.019 bp in the low prior-return group and +0.080 bp in the high group, where the PPO estimate was −0.065 bp (Table S19). Evaluation for 2024 to 2026 and tests across longer bull and bear periods remain open.

In the revised paper. Section 5.1 for the annual split (p. 5), RQ1 in Section 5.2 and Table 1 for test costs (pp. 6–7), and Table A1 for the date-block intervals (p. 14). Appendix I.2, Tables S18 and S19, reports results by prior-day market conditions (pp. 36–37).

AuthorsOpenReview ↗

Response to Reviewer yxQd

W4 | Theory

Uncertainty about the conditional price mean. Sun, Wang, Zhang, and Huang (2023) represent ambiguity about the mean asset payoff with an interval of plausible means. Even with the same interval, ambiguity-averse investors evaluate the least favorable mean and ambiguity-loving investors the most favorable mean. This difference changes asset demand and market participation in their model. Horst et al. apply worst-case evaluation to uncertain drivers of liquidation costs. Sun et al., 2023 · Gilboa and Schmeidler, 1989 · Horst et al., 2022

Section 3 adapts the interval to conditional price means in fixed-order execution. At each candidate mean, we average paired savings over shared market shocks, then use the lowest mean saving across the tested scenarios as conservative evidence. A change in the assumed price mean can reverse the ranking of two options when their inventory exposures differ (Appendix D.3). Using the same checkpoints and development states, AA50 improves mean worst-scenario saving over the center-only rule in all 15 year-by-backend combinations. Mean regret falls in some combinations and rises in others (RQ3 and Table S36).

A second-order cost model for execution schedules. Building on the fixed-quantity execution setting of Almgren and Chriss, the financial surrogate uses a local quadratic cost for reallocating volume across time slots while preserving the total quantity. A positive diagonal models per-slot curvature, and a low-rank term captures how adjustments at different times interact as liquidity, inventory, and time remaining change. The bounded residual corrects the approximation. The feedback discounts the evidence score for downside risk and disagreement among ensemble members, and bounds the resulting adjustment to the action score (Sections 4.2 to 4.4). Almgren and Chriss, 2001

Variance reduction and decision error.

Sharing shocks preserves the mean of the paired cost difference. It lowers variance relative to independent shocks when the branch costs have positive covariance. PICV also preserves the mean if its control variate has zero conditional mean and we fix its coefficient before evaluation (Appendix D).

On development dates, Full reduces the variance measure in Figure 5 by 30.5%, 17.1%, and 38.7% for DDQN, PPO, and GRPO, respectively, relative to variants retrained without PICV. All 75 separate held-out checks pass. The largest 95% upper bound for adjusted variance divided by raw variance is 0.979 (Table 2). Appendix K shows that, at a fixed state and scenario, a greedy choice incurs at most 2ε regret if the absolute error in mean saving is at most ε for every legal option.

In the revised paper. Sections 3 and 4.2–4.4 (pp. 3–5) and Appendix D (p. 24) for the theory, Figure 5 and Table 2 (p. 8) for variance, Appendix J.2 and Table S36 (pp. 42–43) for the scenario comparison, and Appendix K (p. 43) for the regret bound.

W5 | Market impact

Aggressive orders consume displayed depth, and the simulator uses the trades recorded in that minute to determine passive fills. The replay uses the same recorded path for subsequent prices and liquidity replenishment regardless of how the strategy's simulated orders fill (Appendix F.2). The stress tests measure how sensitive execution costs are to fill assumptions. For 2023 PPO, doubling the parent order and capping passive fills at 25% of best-ask depth raises Full TC from −2.815 to +1.148 bp. The mean share of the parent order still open when liquidation begins rises from 0.67% to 14.70% (Table S16).

In the revised paper. Appendix F.2 for the fill rules and fixed market path (pp. 28–29), Section 5.3 for replay scope (p. 10), and Appendix I, Table S16, for stress tests (pp. 34–35).

AuthorsOpenReview ↗

Response to Reviewer yxQd

W6 | Execution implementation

Our policy uses the order book, remaining inventory, time to the deadline, and fill history. We organize the actions available in these execution situations into 12 options in five families covering neutral execution, patient liquidity provision, patient size increases, urgent catch-up, and impact control. The shifts below are relative to the neutral quote and size indices. The simulator recalculates the command each minute from the current state (Appendix F.2).

OptionQuote-index shiftSize-index shiftDuration (minutes)
0001
1+101
2+102
3+202
4+1+12
5+2+13
60+11
70+12
8−101
9−1+12
10−2+12
110−11

Option 0 keeps the neutral indices for one minute. Option 5 shifts both indices for up to three minutes. The simulator uses the neutral command if either shifted index is invalid and ends the option at the execution deadline.

The simulator sizes each parent order using prior-day volume and matches orders on a minute grid. At the start of the final minute, it sweeps five bid levels, fills any remainder at the fifth bid, and applies a 2 bp penalty relative to the arrival midpoint (Appendix F.2).

The immediate market-order cost input uses final same-session volume in the 2022 and 2023 test data we inspected (Section 5.1). We reproduced this look-ahead dependency in an archived 2021 data pack. Evaluating the policies with inputs available at decision time remains open.

In the revised paper. Section 3 for policy inputs and minute-by-minute option remapping (p. 3), Appendix F.1–F.2 for parent orders and matching (pp. 28–29), and Section 5.1 for input timing (p. 5).

W7 | DQN and PPO settings

We evaluate local implementations of DQN and PPO on the same replay panel as DIHAS and keep each baseline's configuration fixed across years (Appendix F.3, Table S7). Both use 99 primitive actions and a discount of 0.998. DQN uses a learning rate of 1e−5, a batch size of 128, and a target update every 2,000 steps. PPO uses a learning rate of 5e−5, a minibatch size of 256, and clipping of 0.2. Its final rollout ends at 120,960 steps.

In the revised paper. Appendix F.3, Table S7, for settings (p. 29) and Appendix F.4 for checkpoint alignment (pp. 29–30).

W8 | Deployment

With Full evidence feedback, a single Xeon Platinum 8358 CPU thread produces a single-order decision with a p95 latency of 3.67 to 3.78 ms after warm-up in our 2022 and 2023 tests. This is at most 0.0063% of the one-minute decision interval. The NVIDIA A800 takes 9.13 to 9.54 ms at p95 under the same batch-one protocol. Across matched years and backends, the CPU's p95 is 59.4% to 60.5% lower. These timings cover controller computation. We have no broker-confirmed live execution results.

Fang et al. also use a one-minute decision interval. Their Qlib.RL toolkit shows its capacity for concurrent execution by returning decisions for 1,000 simultaneous orders within 50 ms on a machine with an NVIDIA P40 GPU and a Xeon 8171M CPU. The DIHAS measurement shows the single-order response time with evidence feedback on one CPU thread. Fang et al., KDD 2023, Section 6

In the revised paper. Appendix I.1, Table S17 and the text following it, for the timing protocol, results, and published comparison (pp. 35–36).

Reviewer_4yJNOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

In this paper, the authors propose the DIHAS framework to address the problem that RL-based trade execution algorithms often cannot precisely estimate the value of each decision due to the combined effects of market price volatility and subsequent decisions. The authors introduce the concept of Paired Saving by comparing a candidate action with a reference action under the same market state. They then propose PICV to reduce variance in Monte Carlo estimation. Experimental results show that DIHAS can help RL-based trade execution algorithms execute parent orders with lower trading costs, including negative trading costs.

Critical strengths

None

Critical weaknesses

None

Questions for authors

The technical parts of this paper are solid, and the experimental results appear promising. The major issue is that the main body of the paper is not self-contained, making the current version is written for experts in RL and trade execution, which makes it difficult to follow. The authors are encouraged to revise the main text to improve readability.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4yJN

Thank you for pointing out that the main text was difficult to follow. We apologize for overlooking how much background the original draft required. To make the paper easier to read, we added a numerical example of paired saving in Section 1, explained the transaction-cost benchmark in Section 3, and revised the caption of Figure 1 to explain how we train the evidence estimator and the execution policy.

Q1 | Make the main text self-contained

Consider an order to sell a fixed quantity within 30 minutes. The benchmark is the average bid-ask midpoint over that window. Transaction cost (TC) is negative when the average sale price exceeds the benchmark (Section 3).

Section 1 then follows two simulated branches from the same state. One takes the reference option, and the other replaces that option once. Both share the market shocks and follow the same frozen reference policy thereafter. If their costs are 4.0 and 3.4 basis points (bp), the replacement saves 0.6 bp on that draw. Figure 1 shows how this local comparison trains the evidence estimator and how the cost difference between two complete executions supplies the terminal reward. We evaluate final performance using the completed order's TC.

In the revised paper. Section 1 for the worked example (p. 1), Figure 1 for the two training paths (p. 2), and Section 3 for the cost benchmark (p. 3).

I will maintain my rating, as it was given on the condition that the authors improve the readability of the paper.

AuthorsOpenReview ↗

Response to Reviewer 4yJN

Thank you for emphasizing the importance of a clear, self-contained main text. In the next revision, we will explain DIHAS through a simple question. Given the current market state, how would choosing a different action now change the cost of completing the order? DIHAS estimates that cost difference before acting and uses it to guide the current choice. We will organize the main text around this question to help readers new to reinforcement learning or trade execution understand the method and its purpose.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Reviewer yxQd

Thank you for suggesting that we test whether a policy trained in Shenzhen transfers to other assets and venues. We have now completed four historical replay tests, including Shanghai and the US exchange IEX. We froze the policy weights, normalization, and calibration before each test. Within each panel, DIHAS with evidence feedback (Full), its matched policy trained without feedback (Off), and TWAP executed the same 30-minute sell orders under the same fill rules.

Transfer results. The last three columns give paired mean savings in basis points (Off minus Full). Positive values favor Full. The two 2024 rows use the same Shenzhen checkpoints on the same 225 trading dates (Table 4 and Table S37).

Test panelShenzhen training yearStocksDatesDDQNPPOGRPO
Shenzhen 2024202352225+0.0376+0.0141+0.0067
Shanghai 2023202245234−0.0041−0.2694+0.2456
Shanghai 2024202345225+0.0533+0.0227+0.0152
IEX 202020194442+0.7845+6.2764+4.5697

Across all eligible dates on the three Chinese panels, Full saved 8.0–9.2 bp against TWAP, with positive paired 95% intervals in all nine comparisons (Table S38). The matched Full/Off comparison isolates the smaller contribution of feedback on these panels. On the common 2024 dates, DDQN saved 0.0376 bp in Shenzhen and 0.0533 bp in Shanghai. Both intervals were above zero, while the PPO and GRPO intervals included zero at both venues (Table S39). In Shanghai in 2023, feedback lowered GRPO cost but raised PPO cost for all five seeds (Table S41).

On IEX, 44 of 100 candidate stocks contributed 206,863 eligible order windows across 42 trading dates in November–December 2020. Compared with TWAP, Full saved 3.59 bp for PPO and 6.71 bp for GRPO. DDQN cost 2.60 bp more on average, and its interval included zero. Compared with Off, all three mean savings favored Full, although only three of five seed pairs improved for each algorithm. The paired intervals use five-trading-day blocks and condition on the five frozen seeds (Appendix N.2 and Tables S37, S38, and S41).

Implementation checks. Each order was sized at 0.5% of the preceding trading day's volume at that venue. The immediate market-order cost input also used preceding-day volume. In China we checked sampled five-level books against recorded quotes. On IEX we applied grouped price-level updates together and checked the session clock against exchange messages, including both early closes. On sampled windows, truncating the feed at decision time preserved the inputs and selected actions (Appendix N.1). The share of parent quantity left for forced execution at the deadline was 0.45–0.81% for Full on the Chinese panels and 1.92–2.22% on IEX, versus 3.33% for TWAP (Table S40). These results measure simulated fills on recorded market paths.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.