acceptodds
Under review as a conference paper at ICLR 2027

Almost Exogenous: Regret Guarantees for Reinforcement Learning under Bounded Action Influence

Abstract

Exogenous Markov Decision Processes (Exo-MDPs) model settings where uncertainty is driven by external factors independent of the learner's actions. In many real systems, this separation is only approximate: actions weakly influence variables otherwise treated as exogenous. We study reinforcement learning under bounded violations of exogeneity, where each action perturbs the exogenous transition law. The perturbation is controlled by an impact parameter . We introduce two optimistic algorithms for this perturbation: a tabular method and Exogenous Value-Targeted Regression (Exo-VTR), which exploits linear-mixture structure in large exogenous spaces. We prove high-probability regret of the form , where is the number of episodes, retaining the statistical benefit of exogenous structure while paying an explicit perturbation penalty. In the tabular setting, no-bonus predict-then-optimize (PTO) attains the same regret order. We also provide lower bounds on first-order loss for specified exogenous planner classes. A separate two-stage construction establishes first-order regret for Exo-UCBVI-, PTO, and Exo-VTR- with a one-dimensional reference representation. Experiments validate the execution-family separation and probe the roles of clipping and optimism.

Primary area:
reinforcement learning
Keywords:
Exogenous Markov Decision Processes, Reinforcement Learning, Approximate Exogeneity, Model Misspecification, Regret Analysis, Value-Targeted Regression, Predict-Then-Optimize

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CoRrOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies almost exogenous MDP in which some variables which are treated as exogenous are in fact affected by the learner action up to a strength controlled by some known parameter ε\varepsilon. The reward and the dynamics of the endogenous system are known to the learner. The paper contributions read as follows 1) upper bound for tabular almost exogenous MDPs in which the term in K\sqrt{K} scales only with the number of exogenous states and not with the cardinality of the full state space and a linear term in KK which is controlled by ε\varepsilon (which should be interpreted as the price to pay for treating as exogenous a variable which is in fact endogenous), 2) Lower bounds against a class of algorithms that does not see the reward of the environment and one for algorithms which can (notice that Exo-UCBVI-ε\varepsilon belongs to the latter class) and 3) finally the upper bound is extended to linear mixture almost exogenous MDP in which the almost exogenous kernel can be written as a linear combinations of dd dimensional features. This results allows the number of exogenous states to be potentially infinite while retaining efficiency because only dd shows up in the bound.

Critical strengths

The results are very interesting.

Moreover, I think that the problem is important. In practice many times we might consider as exogenous variables which are in fact endogenous.

Critical weaknesses

Presentation can clearly be improved, there are many pointers to the appendix: I think presentation is problematic because many important definitions and the algorithms are deferred to the Appendix. I would for example put your algorithms in the main and leave in the Appendix the pointers to some refined results such as the one that avoids assuming a maximum bound on the action (i.e. Theorem 2).

Simple setting Some assumptions are not very well justified. Why are the actions supposed to be scalars? And why the endogenous dynamics are assumed to be known? Please make sure to mention these limitations and draft some possible solutions.

Questions for authors

ε\varepsilon needs to be known the bonuses used by Exo-UCBVI-ε\varepsilon and Exo-VTR-ε\varepsilon need to know the action influence parameter ε\varepsilon. PTO has the benefit of not requiring this knowledge. What happens if you drop the terms depending on ε\varepsilon from the exploration bonuses? Can you still get guarantees?

Lower Bounds I do not think that I understand why the lower bound in Theorem 3 is meaningful. Indeed, I do not think that algorithms which are not allowed to observe the reward function can be expected to have sublinear regret. Theorem 4, looks at a more interesting class of algorithms but I do not understand why the lower bounds holds only on the UP instance. I suggest to the author to state the lower bound in an information theoretic min max manner. That is, for any algorithm in the class, there exists an instance in {UP,DOWN}\{ UP, DOWN \} such that the regret is at least εK\varepsilon K. I think this is already what the authors prove but it is not stated in this way which I find easier to read.

Avoiding Exploration Why does the PTO approach does not work for Linear Mixture MDPs?

Please see also my questions in the weakness session.

4 comments
AuthorsOpenReview ↗

Critical Weaknesses 1 and 2

Thank you for the feedback. The revised version includes both algorithms and makes the model assumptions and lower-bound classes explicit. The additional analysis answers the two bonus-removal questions separately.

1-Critical Weakness 1: Algorithms 1 and 2 now appear in Sections 4 and 6. Section 3 states the known reward and successor map, the admissible policy class, the budget convention, and regret against the actual-model optimum. The cap-free refinement is in Appendix D.6, and the reward-blind result is in Appendix E.3. Section 5 focuses on the reward-aware lower bound.

2-Critical Weakness 2: Scalar actions express traded volume in the execution examples. The upper-bound arguments apply to arbitrary action spaces equipped with a known measurable nonnegative cost ch(x,a,ξ)c_h(x,a,\xi), provided the remaining model assumptions, including the measurability and planning-oracle requirements, hold. Appendix C.5 replaces the influence and budget conditions by

\le \varepsilon c_h(x,a,\xi),$$ and $$\sum_{h=1}^H c_h(x_h,a_h,\xi_h)\le\bar X.$$ The budget condition holds pathwise. The capped guarantees additionally use $c_h\le c_{\max}$, with $c_{\max}$ replacing $a_{\max}$. The budget augmentation preserves the external state size and reference feature dimension. The execution lower bounds retain their stated scalar constructions. The known endogenous map supplies information needed for sample reuse: given an observed external successor, the learner can compute the corresponding endogenous successor for each candidate action. Inventory and resource-balance equations fit this assumption. Appendix C.2 also gives a random-delivery example whose observed external state determines the delivered fraction. Additional randomness can be incorporated into the observed external state, provided the augmented model remains Markov, the endogenous update is known, and the influence and complexity assumptions hold. The analysis assumes this update is known; learning it from data requires additional analysis.
AuthorsOpenReview ↗

Questions 1, 2 and 3

3- Question 1: For tabular Exo-UCBVI, the answer is affirmative. Appendix D.7 analyzes the variant with only the statistical bonus (H−h)βnk(ξ)/2(H-h)\beta_{n_k(\xi)}/2. On the row-confidence event, which has probability at least 1−δ1-\delta, the deficit relative to the reference-optimal values is bounded by

Dh=εamax⁡4(H−h)(H−h+1).D_h=\frac{\varepsilon a_{\max}}{4}(H-h)(H-h+1).

Accounting for that deficit yields Theorem 1's capped guarantee, with the cap-dependent term sharpened to εamax⁡H(H−1)K/2\varepsilon a_{\max}H(H-1)K/2. Theorem PTO gives the same regret order after removing the entire bonus. Appendix D.7 also gives cap-free guarantees for both variants. Both tabular variants operate without ε\varepsilon as an input.

For Exo-VTR, removing only the influence term leaves a valid but weaker guarantee. Appendix F.6 obtains an additional deficit term

(H−1)∑k=1Kek,(H-1)\sum_{k=1}^K e_k,

ek=εH2amax⁡Xˉ(k−1).e_k=\frac{\varepsilon H}{2}\sqrt{a_{\max}\bar X(k-1)}.

At fixed positive influence, this expression can grow as K3/2K^{3/2}, and the regret bound can be capped by HKHK. The additional term quantifies the value-estimation deficit introduced by removing the influence-dependent bonus. Its K3/2K^{3/2} growth is an upper-bound contribution, not a lower bound on the modified algorithm's regret.

4- Question 2: The reward-aware result is now the main lower-bound result. Theorem 4 covers planners that optimize an action-independent model selected from the stated pooled point-estimate or confidence-set classes, while using the known reward function freely. The reward-blind theorem is retained as a secondary result in the appendix.

UP is a common witness because pooling attributes an action-induced upward transition to the external process. After pinning, the pooled point-estimate planner expects the price to increase even while waiting and selects concentrated sales. Confidence-set planners can still choose a temporary partial hedge after pinning, and that schedule also incurs a first-order loss. Under the actual transition law, spreading sales has higher first-order value. In DOWN, concentrated sales are first-order optimal, so the same construction gives no corresponding persistent loss.

For sufficiently small ε\varepsilon, Theorem 4 states the expected lower bound for pooled point-estimate planners, including their random pinning time, and the probability-1−2δ1-2\delta lower bound for confidence-set planners, including deterministic burn-in. Section 5 also states the requested minimax formulation for the confidence-set class:

\max_{v\in\\{\mathrm{UP},\mathrm{DOWN}\\}} \mathbb E[\operatorname{Regret}_{v}^{\mathcal A}(K)] \ge (1-2\delta)_{+} \frac{g\varepsilon}{96} (K-T_{\mathrm{cs}})_{+} (1-C_{g}\varepsilon).$$ With $g\in(0,1)$ is the price gap, $T_{\mathrm{cs}}$ is the deterministic burn-in threshold in Theorem 4, and $C_g$ is independent of $K$ and the planner. The bound holds for sufficiently small $\varepsilon$, and $(z)_+=\max\{z,0\}$. UP supplies the same witness for every planner in this class. Appendix E.8 gives separate algorithm-specific results. **5- Question 3:** Tabular PTO estimates complete external transition rows. The present VTR estimator instead updates from the value-targeted features of selected actions. Appendix F.6 gives a two-stage exact-exogeneity example where waiting has continuation feature $1/2$ and selling has feature zero. With the zero-initialized ridge estimate, the greedy action values are zero for waiting and $1/4$ for selling. Selling is the unique greedy choice and supplies only the regression pair $(0,0)$, so the estimate stays zero in every episode. The optimal policy waits and earns $1/2$, giving the modified learner exactly $K/4$ regret. The feature map in this example determines the reference kernel through normalization, so a planner enforcing that information avoids the failure. The counterexample therefore establishes a failure of bonus deletion for the specified ridge-VTR estimator, without implying that optimism is necessary for every algorithm under A1L.

Thank you for the clarifications.

I am happy to keep my positive score

Thank you. We appreciate your constructive feedback and the time you took to review our revisions. Your comments have helped us improve the presentation and clarify the theoretical contributions of the paper.

Reviewer_k6T8OpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates reinforcement learning in approximately exogenous Markov decision processes, where agents can weakly influence environmental variables that would otherwise evolve independently of their actions. The authors introduce a bounded action-influence model, develop tabular and linear-mixture algorithms, and establish high-probability regret bounds of the form O~K+O~εK\tilde O \sqrt K +\tilde O \sqrt {\varepsilon K}. They also show that pure exploitation retains the same regret order in the tabular setting and establish first-order lower bounds for specific classes of exogenous planners. Synthetic experiments and a historical market replay investigate the theoretical mechanisms and their practical consequences.

Critical strengths

1- Exact action independence is a restrictive assumption in many sequential decision problems. Introducing an action-dependent perturbation controlled by $\varepsilon$ and a trajectory-level action budget provides a meaningful intermediate model between exogenous MDPs and unrestricted action-dependent dynamics.

2- The tabular and linear-mixture analyses establish that the statistical component of regret can remain independent of the endogenous state and action-space sizes while explicitly accounting for model misspecification. The proof carefully distinguishes estimation noise from systematic action-induced bias, including when the observations are collected adaptively under the perturbed dynamics.

3- Establishing that no-bonus predict-then-optimize achieves the same tabular regret order is particularly interesting. The lower-bound constructions additionally illustrate how action-independent planning can incur persistent first-order losses when small perturbations change which actions are optimal.

Critical weaknesses

1- The regret structure is not, by itself, novel. For example, Li and Yang (2024) already establish regret bounds combining statistical estimation error and a term linear in model misspecification. The manuscript acknowledges that its Exo-VTR analysis specializes existing robust value-targeted regression techniques and that its principal refinement is the explicit dependence on action caps and trajectory budgets. Although this specialization is useful, the authors should more clearly establish which genuinely new theoretical insights extend beyond existing robust RL analysis and why the improvement warrants a standalone ICLR contribution.

Li, Y., & Yang, L. (2024, April). On the model-misspecification in reinforcement learning. In International Conference on Artificial Intelligence and Statistics (pp. 2764-2772). PMLR.

2- The paper establishes first-order regret lower bounds only for restricted classes of action-independent planners and the particular proposed algorithms, rather than for arbitrary learning algorithms. An action-aware algorithm can potentially learn the perturbed transition dynamics and avoid persistent linear regret. Indeed, the manuscript itself constructs action-aware alternatives with substantially smaller regret on its hard instances. Consequently, the lower bounds characterize the cost of continuing to use an action-independent model, rather than an unavoidable statistical difficulty of learning in approximately exogenous MDPs. This considerably limits the scope of the claimed upper–lower-bound correspondence. A stronger contribution would characterize when exploiting exogenous structure is preferable to learning action-dependent dynamics, ideally through a general regret trade-off or adaptive algorithm.

3- The controlled experiments primarily evaluate specially designed environments, including instances constructed to exhibit the behavior predicted by the lower bounds. More importantly, the historical market replay does not test positive action influence. The authors explicitly set ε=0\varepsilon = 0, since historical price observations do not identify the causal effect of the learner's actions. The learned policy also fails to outperform the simple time-weighted average price (TWAP) benchmark on the tested parameter grid.

4- The paper establishes that bonus-free PTO attains the same tabular regret order as Exo-UCBVI-ε\varepsilon, raising questions about the practical necessity of the more complicated optimistic method. In the tabular algorithm, the exploration bonus is action-independent at a given state and stage and can affect the action ranking only through truncation effects. The linear-mixture algorithm has action-dependent bonuses, but the experiments show that removing optimism consistently improves performance in the tested configurations.

Questions for authors

NA

5 comments
AuthorsOpenReview ↗

Critical Weakness 1

Thank you for the feedback. The revised paper clarifies three main contributions: robustness of tabular pure exploitation to action-dependent observations, budget-sensitive control of transition bias, and persistent decision errors caused by pooled planning. Section 5 specifies the planner classes and the corresponding lower-bound guarantees.

1- Critical Weakness 1: Theorem PTO establishes that bonus-free greedy planning retains the tabular regret order under action-dependent observations, without knowing ε\varepsilon. The technical obstacle is that the learner now affects the external trace: reference-kernel increments that are centered under exact exogeneity acquire conditional drift. Appendix D.5 controls a bounded model-error envelope uniformly over policies and endogenous states, then follows it along the actual interaction trajectory. This preserves Theorem 1's statistical term and frozen-count remainder, with the coefficient of εHXˉK\varepsilon H\bar X K increasing from one to two, without a comparator-coverage assumption.

The Exo-VTR analysis distinguishes two sources of influence cost: total action volume controls policy-value error, while squared volume controls regression bias. For the comparison with Li and Yang (2024), define

ρ4=min⁡{amax⁡,(amax⁡3Xˉ/H)1/4}.\rho_4 = \min\{a_{\max}, (a_{\max}^3 \bar X/H)^{1/4}\}.

The trajectory budget also benefits Li and Yang's occupancy condition. On the common bounded linear model class, with the same feasible policies and actual-model comparator, the required oracles, and εHρ4/2≤1\varepsilon H\rho_4/2\le1, their Theorem 5.6 applied with value-prediction error εHρ4/2\varepsilon H\rho_4/2 gives

dH^{3/2}\sqrt K+ \varepsilon K\sqrt d\,H^2\rho_4 \right).$$ Our leading regression-bias term is $$\widetilde O\!\left( \varepsilon K\sqrt{dH^3a_{\max}\bar X} \right),$$ so the ratio of our leading regression-bias coefficient to the influence coefficient from the direct value-error substitution into Li and Yang's Theorem 5.6 is $(\bar X/(Ha_{\max}))^{1/4}$, up to constants and logarithmic factors, when $0<a_{\max}\le\bar X<Ha_{\max}$. The improvement also holds after accounting for the remaining terms in a concrete regime. For $H\ge2$, $a_{\max}=\bar X>0$, $B_\theta=O(\sqrt d)$, and $K\ge dH$, Theorem 5 reduces, up to logarithmic factors, to $$\widetilde O\!\left( dH^{3/2}\sqrt K+ \varepsilon\bar X\sqrt d\,H^{3/2}K \right).$$ In the same regime, Li and Yang's Theorem 5.6 yields $$\widetilde O\!\left( dH^{3/2}\sqrt K+ \varepsilon\bar X\sqrt d\,H^{7/4}K \right).$$ For this direct value-error comparison, the statistical order is preserved and the influence coefficient in our bound is smaller by a factor of $H^{1/4}$, up to constants and logarithmic factors. Appendix B gives the comparison assumptions and separately discusses the kernel-error corollary in Remark 5.8 of Li and Yang (2024).
AuthorsOpenReview ↗

Critical Weakness 2

2- Critical Weakness 2: The lower bounds characterize persistent decision error from the specified action-independent planning rules. Theorem 4 allows the planner to use the known rewards freely. In UP, pooling makes an action-induced price increase appear possible even while waiting, favoring concentrated sales. Under the actual dynamics, sales induce the increase and spreading has higher first-order value. Accumulating observations can therefore stabilize an incorrect decision.

For sufficiently small ε\varepsilon, UP is a common witness for every planner in each stated class. The point-estimate result is in expectation and accounts for a random pinning time; the confidence-set result holds with probability at least 1−2δ1-2\delta after deterministic burn-in. Section 5 also gives the expected-regret minimax statement over the specified confidence-set class. The correspondence with the upper bounds concerns first-order dependence on εK\varepsilon K for these planning rules. Appendix E.8 gives separate algorithm-specific constructions, including exact εK/16\varepsilon K/16 regret for PTO and Exo-VTR.

Appendix D.8 quantifies the tradeoff with action-aware learning under identical rewards, update map, feasible actions, observations, and comparator. Pooling estimates S∙S_\bullet external rows; the action-aware learner estimates NrowN_{\mathrm{row}} actual (x,ξ,a)(x,\xi,a) rows, including the budget state. With a common confidence allocation, let Uexo,0U_{\mathrm{exo},0} denote the right-hand side of Theorem 1 evaluated at ε=0\varepsilon=0, and let UawareU_{\mathrm{aware}} denote the action-aware regret bound. The pooled bound is no larger whenever

\le U_{\mathrm{aware}}-U_{\mathrm{exo},0}.$$ This gives an explicit finite-sample comparison between the statistical saving from estimating fewer rows and the influence penalty. For fixed finite row count and horizon, the action-aware guarantee is sublinear in $K$, while the pooled influence term grows linearly at fixed positive $\varepsilon$. This comparison identifies when the pooled regret bound is no larger than the action-aware bound under the same information assumptions.
AuthorsOpenReview ↗

Critical Weakness 3

3- Critical Weakness 3: The controlled experiments isolate the mechanisms studied theoretically. UP tests persistent loss from pooling, while DOWN tests the case where the same planning bias selects an appropriate concentrated schedule. The impact-aware reference uses the parametric form of the hard family. Across the other four controlled families, regret grows approximately linearly with influence in one family and is non-monotone in the others. This distinguishes the worst-case influence penalty from the behavior of individual instances.

The BTC replay evaluates the chosen state and reward model at ε=0\varepsilon=0. The policy matches TWAP at all tested η≥0.0016\eta\ge0.0016; at the three smaller tested values it changes its trades and incurs higher mean cost. The reference fitted on held-out dynamics also fails to beat TWAP at the two reported values. The replay documents when the fitted model departs from TWAP and whether those departures improve execution. Positive transition influence is evaluated in the controlled environments.

Reviewer_kfDWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies RL in environments where actions weakly affect a process otherwise treated as exogenous. It proposes tabular and linear-mixture algorithms with regret bounds consisting of a statistical term and an action-influence penalty. It also gives lower bounds for specific planner classes.

Critical strengths
  1. The analysis offers some useful insights beyond the main bound. In particular, the tabular PTO result shows that optimism is unnecessary for the same regret order, while the lower-bound examples explain how pooling action-dependent observations can lead to persistently wrong decisions.
Critical weaknesses
  1. Bounds combining a statistical term with a penalty linear in the misspecification level and number of episodes are already established. Here, such a penalty seems largely expected from the assumed ε\varepsilon-bounded transition deviation and the assumption 3 on cumulative action budget. While a rigorous proof is still valuable, I am unsure whether this contribution is enough to support acceptance and I currently see the paper as borderline. The key question is what additional insight the analysis provides beyond this expected dependence.
  2. I feel that the practical insight remains limited. It is unclear when one should retain the exogenous approximation rather than learn the action-dependent dynamics. The market replay uses zero transition impact and does not outperform TWAP, so it is unclear what additional insight this experiment provides.
Questions for authors
  1. What bounds follow from directly applying existing misspecification results, and which parameter dependencies does the proposed analysis improve? A direct theoretical comparison would help clarify the contribution.
  2. The tabular result that optimism is unnecessary is interesting. How much does it rely on the model structure and assumptions? For example, you assume that all transition randomness comes from the exogenous variable, does this assumption matter? You find that optimism is needed in the linear-mixture setting. Is it fundamentally necessary or only for the current analysis? What explains this difference?
5 comments
AuthorsOpenReview ↗

Critical Weakness 1 and Question 1

Thank you for the review. We clarify the contribution of tabular PTO under action-dependent observations and quantify the improvement in budget dependence relative to the value-error bound obtained from Li and Yang (2024). We have revised the paper accordingly.

1- Critical Weakness 1 and Question 1: The main insight is that tabular pure exploitation remains valid under action-dependent external observations: greedy pooled planning retains the regret order of the optimistic method, without a bonus or knowledge of ε\varepsilon. This extends the algorithmic benefit of exact exogeneity to a setting where the learner changes the law of its own observations. The budget-sensitive VTR analysis and the lower bounds then quantify two consequences of retaining the exogenous model: estimation bias and persistent decision error.

For the requested comparison, let ρ4=min⁡{amax⁡,(amax⁡3Xˉ/H)1/4}\rho_4 = \min\{a_{\max}, (a_{\max}^3 \bar{X}/H)^{1/4}\}. The trajectory budget also benefits Li and Yang's occupancy condition: Edπ∣a∣m≤ρ4m\mathbb E_{d^\pi}|a|^m\le\rho_4^m for every admissible policy and m=1,…,4m=1,\ldots,4. On the common bounded linear model class, with the same feasible policies and actual-model comparator, the required oracles, and εHρ4/2≤1\varepsilon H\rho_4/2\le1, applying their Theorem 5.6 with value-prediction error εHρ4/2\varepsilon H\rho_4/2 gives O~ ⁣(dH3/2K+εKd H2ρ4).\widetilde O\!\left(dH^{3/2}\sqrt K+\varepsilon K\sqrt d\,H^2\rho_4\right).

Our analysis uses total action volume for policy-value error and squared volume for regression bias, giving the leading influence terms O(εHXˉK)+O~ ⁣(εKdH3amax⁡Xˉ).O(\varepsilon H\bar X K)+\widetilde O\!\left(\varepsilon K\sqrt{dH^3a_{\max}\bar X}\right).

The ratio of our leading regression-bias coefficient to the influence coefficient from the Theorem 5.6 substitution is (Xˉ/(Hamax⁡))1/4(\bar X/(Ha_{\max}))^{1/4} for 0<amax⁡≤Xˉ<Hamax⁡0<a_{\max}\le\bar X<Ha_{\max}, up to constants and logarithmic factors.

The improvement also holds after accounting for the remaining terms. Consider amax⁡=Xˉ>0a_{\max}=\bar X>0, Bθ=O(d)B_\theta=O(\sqrt d), and K≥dHK\ge dH. Expanding Theorem 5, its regularization and frozen-update terms are absorbed, up to logarithmic factors, into O~ ⁣(dH3/2K+εXˉd H3/2K).\widetilde O\!\left(dH^{3/2}\sqrt K+\varepsilon\bar X\sqrt d\,H^{3/2}K\right).

In the same regime, the Theorem 5.6 substitution above gives O~ ⁣(dH3/2K+εXˉd H7/4K).\widetilde O\!\left(dH^{3/2}\sqrt K+\varepsilon\bar X\sqrt d\,H^{7/4}K\right). Thus the statistical order is preserved, and the influence coefficient is smaller by a factor of H1/4H^{1/4} in this regime, up to constants and logarithmic factors. This comparison is specifically with the value-error substitution; Appendix B separately discusses the kernel-error corollary in Remark 5.8 of Li and Yang (2024) and the conditions required for its application.

The lower bounds add a decision-level explanation: pooled observations can make an action-induced transition appear possible while waiting, so accumulating data stabilizes an incorrect action ranking. The stated planner classes can consequently incur first-order loss after burn-in. This identifies the cost of the planning abstraction even when statistical uncertainty shrinks.

AuthorsOpenReview ↗

Question 2

2- Question 2: The obstacle is that the shared external trace now depends on the learner's actions. Reference-kernel increments that are centered under exact exogeneity acquire conditional drift. Appendix D.5 controls a bounded error envelope uniformly over policies and endogenous states, then follows it along the actual interaction trajectory. The budget bounds the accumulated drift, and the statistical calculation uses actual visits, without a comparator-coverage assumption. PTO preserves Theorem 1's statistical term and frozen-count remainder; the coefficient of εHXˉK\varepsilon H\bar X K increases from one to two.

The known rewards, known successor map, and observed external state enable this reuse of samples across candidate actions. Extra randomness is covered when it admits an observable Markov external augmentation with a known update and the stated influence and complexity bounds. Unknown endogenous dynamics require a separate learning analysis.

VTR uses a different information mechanism: its regression features depend on the selected actions and continuation values. Appendix F.6 gives an exact-exogeneity example where deleting the entire bonus makes the ridge estimator repeatedly select a zero-feature action, leaving its estimate unchanged and producing exactly K/4K/4 regret. Kernel normalization identifies the reference kernel in this example, so a planner enforcing that information avoids the failure. The example establishes a limitation of this estimator modification; a general bonus-free guarantee under A1L remains open.

AuthorsOpenReview ↗

Critical Weakness 2

3- Critical Weakness 2: Appendix D.8 gives an explicit comparison under identical rewards, update map, feasible actions, observations, and comparator. The action-aware learner estimates NrowN_{\mathrm{row}} actual (x,ξ,a)(x,\xi,a) rows, including the budget state; the pooled learner estimates S∙S_\bullet external rows. With a common confidence allocation, let Uexo,0U_{\mathrm{exo},0} denote the right-hand side of Theorem 1 evaluated at ε=0\varepsilon=0, and let UawareU_{\mathrm{aware}} denote the action-aware regret bound. The pooled bound is no larger when

\frac{U_{\mathrm{aware}}-U_{\mathrm{exo},0}} {K(a_{\max}H^2/2+\bar XH)},$$ provided the numerator is nonnegative and the denominator is positive. This quantifies the regime where the statistical saving from pooling exceeds its influence penalty. At fixed positive influence and fixed finite row count and horizon, the action-aware bound is sublinear in $K$, while the pooled influence term grows linearly. The comparison therefore depends on both sample budget and influence strength. It orders guarantees; selecting the better learner on a particular instance remains open. The replay probes a separate question: when the chosen state and execution-cost model produces departures from TWAP. With transition influence fixed at $\varepsilon=0$, the learned policy matches TWAP at all tested $\eta\ge0.0016$ and departs at the three smaller tested values, with higher mean execution cost. At $\eta=0.0002$, its paired cost is $+1.0972$ basis points, compared with $+0.8577$ for the reference fitted on held-out dynamics. The latter is a diagnostic reference fitted on evaluation data. These results document how the fitted model changes trading behavior and whether that behavior improves execution. Positive transition influence is tested in the controlled environments.

Comments on the paper

1 comment

As the discussion period is approaching its end, please take a moment to review the authors’ responses and share any remaining thoughts or updates to your assessment. Thanks for your thoughtful reviews and discussion!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.