acceptodds
Under review as a conference paper at ICLR 2027

LEARNING HOW TO BE RISK-SENSITIVE: STATE-DEPENDENT RISK FUNCTIONALS FOR REINFORCEMENT LEARNING

Abstract

Adapting risk preferences to the state expands the decisions a risk-sensitive agent can implement. We prove a constructive separation within a specified conditional value at risk (CVaR) family: a stationary state-dependent risk map strictly im- plements an outer-optimal policy beyond the implementation capacity of every time-dependent, state-independent map. We develop a learning procedure that selects risk maps through empirical planning and separate outer-utility evaluation before fitting a neural critic. This separates the value of a risk preference from the fidelity of its learned execution. Controlled experiments recover optimal util- ity and verify policy fidelity separately. In a complementary sequential model, adaptive risk exceeds the specified shared-risk baseline under both design and shifted transition laws, matching direct optimization with equal total data at the largest evaluated budget. Exact certificates verify that learned state-only maps implement the target policy under both laws. In financial risk management, a simu- lated currency-liquidation benchmark with hold-or-sell-all policies shows higher mean revenue than selected shared and time-dependent risk controls. Historical Bit- coin quote replay adds evidence of temporal transfer: a frozen risk map improves lower-tail proceeds over those controls in two later periods on Coinbase’s exchange (GDAX in the dataset). Together, these results connect the expressive power of state-dependent risk to a practical procedure for learning explicit valuation rules and verifying their execution.

Primary area:
reinforcement learning
Keywords:
Risk-sensitive reinforcement learning, adaptive risk preferences, distributional reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5mNsOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This manuscript proposes a state-dependent mapping over risk functionals, applied in a nested manner, to optimize an outer static risk-sensitive objective. They construct an example where their mapping can express the optimal policy for the outer objective whereas a time-only (undefined?) map is unable to. They propose a selection rule, over a finite bank of risk maps, learned from observed trajectories to optimize the outer objective. Experiments show their method outperforms time-only and global (?) baselines.

Critical strengths

None.

Critical weaknesses

The paper is very telegraphically written with dense, undefined jargon making it very difficult to understand and review.

The prose, starting from the intro, contains too much jargon and terms without definition. The authors use terms like "inner risk functional", and "outer optimization" before defining the problem. Here are a few clarity concerns from the first few pages:

  • "We distinguish the outer objective, which measures the value of the resulting behavior, from the inner risk functional, which ranks available actions." Value of the resulting behavior with respect to what? Ranking with respect to what, the outer behavior or something else?

  • "Learn maps with verifiable execution", what does verifiable execution mean in this context?

  • The entire background section summarizes prior work in less than 10 words, and does not adequately link prior work to this work. The section on coherent risk measures states axioms, without explaining their relevance, and its not clear how these definitions tie to this work.

  • L085: The objective and Bellman operator must agree - meaning what? Agree in what way?

  • L086: Static CVaR decompositions require particular care in policy optimization - what kind of care?

  • L087: We learn a nested risk map and test the decisions it implements - in what context, to reveal what? How does this relate to the above?

  • L100: "We supply a constructive obstruction and a strict state-dependent implementation" obstruction of what, implementation of what?

  • L106: we examine sharing of the valuation rule; sharing in what sense? What is being shared? Valuation of what? How does this differ from prior works' uncertainty in learned dynamics?

  • L111: outer objective is not defined. Same with inner functional.

  • L119: maximized by what, with respect to what (policy? risk sensitive functional? Static or dynamic?)

  • Eq.1: Quantile function should be defined. Why is the specific mixture in (1) constructed this way?

  • Eq 2,3,4: More development is needed here, it is not clear what lambda is doing. These equations are simply a recursion with respect to rho, and should be stated as such. It would help to define V in expanded form (in terms of discounted return) to establish that it's a random variable, so that taking rho of V and Q makes sense.

  • Definition 1: It is not clear to me what this definition means, since pi is in the definition and I am still not sure how lambda plays a role in anything.

  • L139: "Our impossibility statements allow favorable resolution of ties" Which statements? Not clear what this means.

  • L140: More definitions, about maps finally but it's not clear what any of this means. We are defining lambda in terms of other lambdas, but I am not sure what lambda is.

  • Eq. 5 should be defined much earlier.

I stopped reviewing here because the clarity concerns above are too numerous, and I am not confident in my ability to review the paper with sufficient understanding beyond this point. I will mention however that the theoretical contribution in this work is specific to a very narrow family of MDPs (3 states, 2 steps) and the author's proposed method lacks any guarantees, motivation, and is completely heuristic in relying on a finite bank of risk functionals to use.

The work is difficult to understand, and the contribution is narrow so I am of the opinion that the work is not ready for publication.

Questions for authors

See weaknesses.

7 comments
AuthorsOpenReview ↗

Response to 5mNs (1/3): Objectives, valuation and execution — points 1–5

Thank you for identifying the places where the formulation was too compressed. We have reorganized it around four objects: the policy objective, the local valuation rule, its sequential recursion, and the condition that a policy follows that rule.

The central question is now stated before the formalism: can the decisions preferred by the policy objective be expressed through a shared local valuation, or does their implementation require state-dependent valuation? Below we address each clarification point and then the concerns about theoretical scope and the learning procedure.

1. “Outer objective” versus “inner risk functional”: value and ranking with respect to what?

The outer objective evaluates a complete policy:

\qquad u(r)=r-5\,\mathbf{1}(r<0).$$ Here $R(t)$ denotes the reward at time $t$, and $\mathbf{1}(r<0)$ is the indicator of a negative reward. The local action score is $$Q_t^{\pi,\lambda}(s,a)=\rho_{\lambda_{t,s}}\left(R+\gamma V_{t+1}^{\pi,\lambda}(S')\mid s,a\right).$$ Thus $J$ evaluates achieved behavior according to the designer’s objective. $Q$ ranks a local action according to the declared risk valuation and continuation policy. Section 2 introduces the objective first, followed by the valuation and recursion. ## 2. What does “verifiable execution” mean? For the executed policy $\widehat{\pi}$, define $$m(\widehat{\pi},\lambda)=\min_{t,s,a\ne\widehat{\pi}_t(s)}\left[Q_t^{\widehat{\pi},\lambda}(s,\widehat{\pi}_t(s))-Q_t^{\widehat{\pi},\lambda}(s,a)\right].$$ **Execution is faithful when $m\geq 0$**: every executed action maximizes the selected valuation under the policy’s own continuation. This differs from achieving high outer utility. The controlled confirmation contains 80/80 outer-optimal state-only policies but 79/80 faithful executions. In the remaining case, exact selected-map scores prefer a different action despite the deployed policy’s optimal outer utility. Exact checks use known controlled dynamics. The empirical audit uses simultaneous probability bounds to account for uncertainty in the scores. ## 3. How do the background and coherent-risk axioms relate to this method? The revised background connects the ingredients explicitly: - Spectral risk defines the distribution-to-score rule. - Markov risk evaluation defines its sequential composition. - Quantile critics approximate the outcome distributions used by that composition. Monotonicity and translation equivariance provide the stability property used in the perturbation proof: $$\lVert X-Y\rVert_\infty\leq\epsilon \quad\Longrightarrow\quad |\rho(X)-\rho(Y)|\leq\epsilon.$$ Indeed, $$Y-\epsilon\leq X\leq Y+\epsilon,$$ and applying the two properties gives the bound. Our lower-tail CVaR mixtures are concave reward valuations. Their negatives correspond to the loss-risk convention. The revision distinguishes this nested local valuation from applying a static spectral functional to total return. The theoretical contribution concerns the sequential restriction imposed by sharing the valuation, rather than the introduction of spectral risk itself. ## 4. L085: What does agreement between an objective and its Bellman operator mean? **The recursion must evaluate the criterion attributed to it.** Direct planning for the additive outer objective uses $$U_t(s)=\max_a\mathbb{E}\left[u(R)+\gamma U_{t+1}(S')\mid s,a\right].$$ The risk-map controller uses $$V_t^{\star,\lambda}(s)=\max_a\rho_{\lambda_{t,s}}\left(R+\gamma V_{t+1}^{\star,\lambda}(S')\mid s,a\right).$$ These are different recursions with different roles. The first defines the policy-design objective. The second determines the controller induced by a map. Selection evaluates that controller against the outer objective. ## 5. L086: What care is required with static CVaR decompositions? In general, evaluating total return once differs from composing conditional CVaRs: $$C_\alpha(R_0+\gamma R_1)\ne C_\alpha\left(R_0+\gamma C_\alpha(R_1\mid S_1)\right).$$ **Our controller explicitly uses nested valuation.** We then evaluate the resulting policy against the separately specified outer objective. The revision replaces the compressed sentence with this distinction and an expanded two-stage recursion.
AuthorsOpenReview ↗

Response to 5mNs (2/3): Definitions and recursion — points 6–12

6. L087: What does learning and testing a nested map reveal?

The procedure is:

  1. Construct a policy for each candidate map using inner data.
  2. Evaluate candidate policies on separate meta data.
  3. Select the map by outer utility.
  4. Fit the selected map’s critic.
  5. Assess utility and execution fidelity separately.

The final checks are J(π^)J(\widehat{\pi}) and m(π^,λ^)≥0m(\widehat{\pi},\widehat{\lambda})\geq 0. Together they determine whether useful outcomes are accompanied by faithful execution of the selected valuation. Section 3 and the architecture figure now present this order explicitly.

7. L100: Obstruction of what, and implementation of what?

The target is the outer-optimal policy π⋆\pi^\star. The theorem establishes

∀λt,s=λt,m(π⋆,λ)<−ε,\forall\lambda_{t,s}=\lambda_t,\qquad m(\pi^\star,\lambda)<-\varepsilon,

while

∃λˉt,s=λˉs,m(π⋆,λˉ)≥δ>0.\exists\bar{\lambda}_{t,s}=\bar{\lambda}_s,\qquad m(\pi^\star,\bar{\lambda})\geq\delta>0.

Every time-only valuation ranks an alternative above a target action somewhere. One stationary state-only valuation strictly supports every target action. The implementation margin is now defined before the theorem.

8. L106: What is shared, and how does this differ from uncertainty in the dynamics?

Sharing constrains which decision cells use the same valuation weights:

ClassValuation at decision cell (t,s)(t,s)
Globalλ\lambda
Time-onlyλt\lambda_t
Stationary state-onlyλs\lambda_s
State-timeλt,s\lambda_{t,s}

A time-only rule uses the same quantile weights across states at a given stage. A stationary state-only rule uses the same weights whenever a physical state recurs.

The theorem assumes exact dynamics and exact scores. Model-estimation uncertainty is a separate empirical issue. Section 2.2 now defines all four classes and distinguishes these questions.

9. L111: Where are the outer objective and inner functional defined?

The outer objective now appears as Equation (1). We then define

ρλ(Y)=∑jλjCαj(Y)\rho_\lambda(Y)=\sum_j\lambda_j C_{\alpha_j}(Y)

and the assignment

(t,s)↦λt,s(t,s)\mapsto\lambda_{t,s}

before introducing the recursion and implementation criterion.

10. L119: What is maximized, over which policies, and is the criterion static or dynamic?

The theoretical target is

π⋆∈arg⁡max⁡π∈ΠDMJ(π),\pi^\star\in\arg\max_{\pi\in\Pi_{\mathrm{DM}}}J(\pi),

where ΠDM\Pi_{\mathrm{DM}} is the class of finite-horizon deterministic Markov policies.

The theoretical JJ is additive expected transformed reward. The local risk valuation is nested. In the liquidation applications, the separately declared outer criterion is episodic revenue CVaR. The revised formulation distinguishes these criteria rather than transferring a guarantee from one to another.

11. Equation 1: What is the quantile function, and why use this mixture?

For 0<v<10<v<1,

FY−1(v)=inf⁡{y:Pr⁡(Y≤y)≥v},F_Y^{-1}(v)=\inf\{y:\Pr(Y\leq y)\geq v\},

and lower-tail CVaR is

Cα(Y)=1α∫0αFY−1(v) dv.C_\alpha(Y)=\frac{1}{\alpha}\int_0^\alpha F_Y^{-1}(v)\,dv.

Nonnegative mixture weights sum to one and determine emphasis on lower-tail outcomes. The level α=1\alpha=1 includes expectation. The integral handles fractional probability mass at atoms.

The five-level grid is an experimental parameterization. The revised theorem permits probability mixtures over all positive tail levels.

12. Equations 2–4: What does λ\lambda do, and which quantities are random?

At a fixed successor state, Vt+1π,λ(s′)V_{t+1}^{\pi,\lambda}(s') is a scalar. Conditional on (s,a)(s,a), the pair (R,S′)(R,S') is random, so

R+γVt+1π,λ(S′)R+\gamma V_{t+1}^{\pi,\lambda}(S')

is random. The parameter λt,s\lambda_{t,s} specifies how its distribution is valued. Section 2.2 now defines RR, QQ and VV immediately before Equations (3)–(5).

For two stages,

V1π,λ(s′)=ρλ1,s′(R1∣s′,π1(s′)),V_1^{\pi,\lambda}(s')=\rho_{\lambda_{1,s'}}(R_1\mid s',\pi_1(s')),

and

Q0π,λ(s,a)=ρλ0,s(R0+γρλ1,S′(R1∣S′,π1(S′))∣s,a).Q_0^{\pi,\lambda}(s,a)=\rho_{\lambda_{0,s}}\left(R_0+\gamma\rho_{\lambda_{1,S'}}(R_1\mid S',\pi_1(S'))\mid s,a\right).

The inner conditional valuation is a scalar at each realized successor. The terminal condition is VH=0V_H=0. Section 2.2 now explains this expansion.

AuthorsOpenReview ↗

Response to 5mNs (3/3): Implementation, scope and guarantees — points 13–16

13. Definition 1: Why does π\pi appear, and how does λ\lambda affect implementation?

We evaluate the proposed policy’s continuation under the map, then check

\qquad\text{for every }t,s,a.$$ Changing $\lambda$ changes action rankings. If every comparison holds, backward induction makes $\pi$ optimal for its map. **Evaluating its own continuation tests implementation without assuming optimality.** ## 14. L139: Which statements allow favorable resolution of ties? Implementation permits tied actions. The separation theorem proves that every time-only map has a margin strictly below $-\varepsilon$ somewhere. Consequently, favorable tie resolution cannot rescue implementation. The definition and theorem now state this convention. ## 15. L140: What do the different lambdas mean? $\lambda$ is a vector of CVaR mixture weights. A map assigns such a vector to each decision cell. For example, using expectation at one state and $C_{1/2}$ at another defines a stationary state-dependent map. A time-only map may change the mixture with the stage, but must use the same mixture across states at that stage. In the generalized theorem, a probability measure over tail levels replaces the finite weight vector. ## 16. Equation 5 should appear earlier The objective now precedes the controller as Equation (1). The example gives $$u(-0.1)=-5.1,\qquad u(0.9)=0.9,$$ and expected utility four for an equiprobable zero/eight reward. ## Broader concerns: narrow theory, motivation and learning guarantees **Scope and novelty.** **The three-state, two-stage restriction belongs to Theorem 1's separation construction.** Section 2.2 defines the framework for general finite-horizon MDPs; we now say this explicitly after the theorem. The construction proves that every admissible time-only mixture fails to support the target policy, while a stationary state-dependent map strictly supports it. It establishes an existence result, not separation in every MDP. The theorem now covers arbitrary probability mixtures of positive-level CVaRs. Appendix O adds symbolic parameter conditions and a finite-support feasibility characterization with dual certificates. The contribution is the sequential sharing obstruction and separate selection/execution assessment; state-dependent preferences, spectral valuations and quantile critics are established ingredients. Sections 1–2 and Appendix H explain the differences from the closest formulations. **What is learned and why use a finite bank?** We select a valuation within an explicit admissible family: $$\widehat{\lambda}\in\underset{\lambda\in\Lambda}{\operatorname{arg\,max}}\;\widehat{J}\left(\widehat{\pi}^{\lambda};D^{\mathrm{meta}}\right).$$ Here $\widehat{\pi}^{\lambda}$ is constructed using inner data, and $D^{\mathrm{meta}}$ denotes separate selection data. Critic fitting follows selection. Neural approximation errors therefore do not determine candidate selection in the corrected method. The finite bank makes this search explicit and reproducible. We do not claim unrestricted learning optimality. Appendix E includes every global candidate in each adaptive bank: state-only optimality remains 40/40 in both settings, while time-only control optimality rises from 22/40 to 40/40, identifying a coverage effect. **What is guaranteed?** For exact optimal nested action scores with unique-action gap $\Delta>0$, $$\lVert\widehat{Q}-Q\rVert_\infty<\Delta/2 \quad\Longrightarrow\quad \widehat{Q}_t(s,a^\star)-\widehat{Q}_t(s,a) \geq\Delta-2\lVert\widehat{Q}-Q\rVert_\infty>0.$$ This standard sufficient condition guarantees faithful greedy decisions **conditional on score accuracy**; it does not prove that training attains the bound. We evaluate execution separately. **Empirical validation.** The independent audit certifies 149 faithful executions and one violation among 240 frozen pairs, leaving 90 unresolved; certificates agree with exact truth, while a plug-in audit makes four errors. Appendix P supplies simultaneous bounds over permitted sampling prefixes, under known support and independent generative observations. Adaptive stopping selects within that covered family. On the original cohort, guided acquisition saves 42.6% against uniform. On 24 new parameter settings of the same graph and 48 independent fitted controller sets, savings are 21.1% against uniform and 13.2% against widest-interval allocation, with positive adjusted intervals. Stronger adaptive allocators have similar costs; unresolved cases remain included. Direct control given the same valuation can use the same audit and repairs. Section 4.5 adds NASA maintenance applications, with positive crossed intervals over pooled planning on N-CMAPSS and batteries. These test empirical valuation selection outside the theoretical graph, separately from neural fidelity. The gains concern the specified pooled planner, not direct optimization generally.
Reviewer_YFtcOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the learning of state-dependent risk measures in risk-sensitive RL, a crucial problem in many applications such as finance and autonomous driving. A major insight of this work is distinguishing between utility and fidelity; in many practical training procedures, risk-sensitive decisions are not directly aligned with the chosen risk measure, leading some researchers to treat risk parameters merely as hyperparameters rather than true risk measures.

This poses a critical problem in risk-sensitive RL applications, as what we intend to align with differs from what we actually choose. Especially in financial trading RL, this is closely related to maximal drawdown, which is a key feature of the risk-return profile that cannot be directly represented by standard risk-sensitive RL. State-dependent risk is essential for handling such cases.To this end, the authors learn λ^k\hat{\lambda}_k via data partitioning in a meta-learning manner. While a limitation of this meta-approach is its inherent policy dependence, the problem setup is well-formulated as an offline RL task.

Critical strengths

State-dependent risk is an important topic in risk-sensitive RL from an alignment perspective. Furthermore, distinguishing between utility and fidelity is a novel approach, as the evaluation of risk preferences does not rely solely on the fitted critic. Constructing the risk measure using Range Value-at-Risk (or spectral CVaR mixtures) is also well-aligned with established work in actuarial science.

Critical weaknesses

Some theorems and lemmas (e.g., Proposition 1, which assumes nominal rewards in [−1,8][-1, 8]) rely on very specific scenarios that could be further generalized. I strongly recommend that the authors polish and generalize these results as much as possible.

The reason why the reward is not aligned with the outer utility is not well explained, which may make it difficult for readers unfamiliar with the topic to understand the problematic situation. Providing a more specific example would be helpful for readers. I aware that this is not so critical problem, though, some revision in the introduction would be helpful to further polish the paper.

Questions for authors

Could the proposed framework be extended to align with path-dependent financial objectives such as maximal drawdown (MDD)?

3 comments
AuthorsOpenReview ↗

Response to YFtc (1/2): Generalized theory and reward–utility distinction

Thank you for identifying the importance of separating achieved utility from fidelity to the chosen risk measure. Your suggestions led us to broaden the theoretical results, introduce a worked reward–utility example, and implement and evaluate the drawdown extension.

W1. Generalize the scenario-specific theoretical results, including Proposition 1’s nominal-reward assumptions.

The revision broadens both the admissible valuation class and the perturbation result.

Arbitrary CVaR mixtures. The separation now covers

ρμ(Y)=∫(0,1]Cα(Y) μ(dα)\rho_\mu(Y)=\int_{(0,1]}C_\alpha(Y)\,\mu(d\alpha)

for any probability measure μ\mu. The key identity is

Cα(c+vBp)=c+v(p+α−1)+α,v≥0.C_\alpha(c+vB_p)=c+v\frac{(p+\alpha-1)_+}{\alpha},\qquad v\geq0.

The Bernoulli factor is nondecreasing in pp at every positive tail level. The ordering used in the proof therefore survives integration against any mixture. Appendix A establishes the strict separation and stationary state-dependent witness for the constructed family, including favorable action ties.

Appendix O additionally gives symbolic sufficient conditions varying rewards, probabilities, discount and the outer penalty on the same graph. For general finite-support laws at fixed continuation, it characterizes shared implementation through

Dw≥0,w∈Δ,Dw\geq0,\qquad w\in\Delta,

where DD contains target-minus-alternative CVaR scores at cumulative-probability breakpoints. A dual certificate identifies incompatible sharing requirements. The sequential characterization retains complete feasible continuation vectors.

General reward interval and horizon. The perturbation proposition now permits bounded nominal rewards in [a,b][a,b] and any finite horizon HH. Fix the policy, map and successor identities on a common finite outcome-label space, with 0≤γ≤10\leq\gamma\leq1.

For reward perturbation at most dd, joint-law total-variation perturbation at most η\eta, and mixture support in [αmin⁡,1][\alpha_{\min},1], define

Lt=(b−a)∑k=0H−t−1γk,L_t=(b-a)\sum_{k=0}^{H-t-1}\gamma^k,

EH=0,Et=d+γEt+1+Ltηαmin⁡.E_H=0,\qquad E_t=d+\gamma E_{t+1}+\frac{L_t\eta}{\alpha_{\min}}.

Every target-policy action score changes by at most EtE_t, and each action margin changes by at most 2Et2E_t.

Monotonicity and translation equivariance bound outcome perturbations. The CVaR variational representation bounds the probability-law contribution. Appendix A provides the induction and common-support assumptions. The positive tail floor is needed for this uniform perturbation bound; the separation itself permits every positive tail level.

W2. Explain the reward–outer-utility distinction with a concrete example accessible to unfamiliar readers.

We now introduce the distinction before the nested recursion.

Reward records the monetary increment; utility specifies the designer’s preference over that increment. In the construction,

u(r)=r−51{r<0}.u(r)=r-5\mathbf1\{r<0\}.

Thus

u(−0.1)=−5.1,u(0.9)=0.9.u(-0.1)=-5.1,\qquad u(0.9)=0.9.

An equiprobable lottery paying zero or eight has expected utility four, exceeding a certain three. The outer objective can therefore favor Growth in one state while rejecting a loss-bearing route toward it in another.

Direct planning on u(R)u(R) optimizes this outer objective. Our implementation question asks whether local CVaR valuations of monetary rewards and continuation values can support those same decisions. The revised Section 2.1 states the objective, example and roles of the two valuations before introducing the controller.

AuthorsOpenReview ↗

Response to YFtc (2/2): Maximum drawdown and execution verification

Q1. Can the framework accommodate path-dependent financial objectives such as maximal drawdown (MDD)?

Yes, through state augmentation. We have implemented and evaluated this extension.

For positive wealth, define the running peak and maximum relative drawdown by

Pt+1=max⁡(Pt,Wt+1),P_{t+1}=\max(P_t,W_{t+1}),

Dt+1=max⁡(Dt,  1−Wt+1Pt+1),D_{t+1}=\max\left(D_t,\;1-\frac{W_{t+1}}{P_{t+1}}\right),

with P0=W0P_0=W_0 and D0=0D_0=0. Adding (Wt,Pt,Dt)(W_t,P_t,D_t) to the market and holdings state makes the path-dependent quantity available in a Markov formulation.

We select policies against

Jκ(π)=Eπ[WH−κDH].J_\kappa(\pi)=\mathbb{E}_\pi[W_H-\kappa D_H].

The inner recursion values monetary increments and continuation. The augmented state and outer criterion supply the drawdown dependence.

The bounded three-decision study combines two payoff pairs, two regime-persistence settings and three penalties. The adaptive banks include the global candidates; the drawdown-aware bank includes market-only rules. All planners receive the augmented state.

ControllerExact-model optimal /12Fitted optimal /60Fitted faithful /60
Global risk125458
Time-only risk125458
Market-only risk125760
Drawdown-aware risk125760
Direct, inner model—54—
Direct, pooled model—59—

Each fitting split supplies 100 generative outcomes per regime. Pooled direct control uses both splits. Training draws are reused across penalties: the three additional adaptive optimum recoveries over global risk arise from one training draw evaluated at three penalties.

Market-only and drawdown-aware maps agree throughout this study, and shared risk is optimal under exact dynamics in every condition. The result establishes a working drawdown-objective extension and quantifies fitted-model recovery; it does not isolate a benefit from explicit drawdown dependence in the map. Appendix F reports every condition and independent trajectory-based arithmetic checks.

A complementary local-protection experiment. We also imposed

C0.25(R∣x,a)≥0C_{0.25}(R\mid x,a)\geq 0

in five prespecified protected regions after fitting.

Under drawdown-triggered protection, adaptive valuation attains utility 108.296, compared with 103.000 for the finite shared bank, and matches constrained direct planning. A continuous shared mixture with optimized state-dependent ties also reaches 108.296. All 100 adapted comparisons satisfy the requirement without new observations; direct planning is faster.

Five frozen-policy transition shifts produce 500 comparisons. Protection remains satisfied where the feasible action sets are unchanged. The persistence shifts and moderate success-probability shifts preserve adaptive and direct optimality. Regime reversal changes the utility ranking: the adaptive-minus-finite-global difference becomes −3.741-3.741 in the drawdown-triggered cases.

Appendix G reports the complete intervention and shift results. These experiments distinguish two different uses of drawdown information: optimizing a drawdown-aware objective and triggering a local protection requirement. They evaluate empirical planning; neural execution is tested separately.

Execution verification. The controlled confirmation separately checks utility and fidelity (80/80 versus 79/80 for state-only control). Accounting for shared uncertainty raises faithful certificates from 126 to 149 among 240 frozen pairs, with one certified violation and 90 unresolved cases; a plug-in audit makes four errors. On newly fitted controllers within the same graph, guided acquisition saves 21.1% against uniform and 13.2% against widest-interval allocation; stronger adaptive allocators have similar costs. Appendix P states the known-support, generative-sampling and simultaneous-coverage assumptions. A direct controller with the same rule can use the same audit. This supplements the drawdown extension without attributing its results to neural execution.

I appreciate the authors for their rebuttal. I will stay on the positive side.

Reviewer_kcjJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies risk preferences that vary across states. Within a specified family of CVaR mixtures, it constructs sequential problems where a state-dependent risk map implements the outer-optimal policy, while every time-dependent, state-independent map fails. It also proposes selecting risk maps through empirical planning and separate utility evaluation before fitting a neural critic.

Critical strengths
  1. The results are somehow interesting and the separation result gives a concrete reason to adapt risk preferences across states. It goes beyond comparing a few learned policies by showing a limitation of the entire specified shared-risk family.
Critical weaknesses
  1. I think the current evidence is not sufficient. It looks like direct utility planners also reach the optimum in the controlled experiments, and the larger-budget comparison shows matching performance rather than an advantage for adaptive risk. The explicit risk rule may be useful, but the paper needs stronger evidence of when this extra layer is worth learning.
  2. The paper only considers a particular CVaR family, while some empirical candidate banks are not nested, making the effect of state dependence harder to isolate. The application results are mixed: Currency Exchange tail improvements are inconclusive, and the historical replay benefit appears on one venue but does not persist across the reported refits.
Questions for authors
  1. The choice of the five CVaR levels needs more justification. Are these levels essential to the separation, or mainly an implementation choice? The proof in Appendix A seems to rely on properties shared by CVaR mixtures more generally, so a broader statement may be possible.
  2. When should a practitioner learn a risk map instead of directly optimizing the outer objective?
  3. What causes the sensitivity across historical-data refits?
5 comments
AuthorsOpenReview ↗

Response to kcjJ (1/3): Practical value

Thank you for recognizing the separation result and asking for stronger evidence of the valuation layer's practical value. We answer the weaknesses first, followed by Questions 1–3 in their original order.

W1. Evidence that learning the valuation layer is worthwhile when direct planners attain the controlled optimum and match adaptive risk at larger budgets.

We added two complementary tests of practical value: maintenance decisions under estimated dynamics, and the observation cost of checking adherence to a declared valuation. Matching direct planning in a controlled model establishes useful implementation, but does not establish an additional performance benefit. The new studies test decision performance and verification cost separately.

Maintenance decisions on NASA datasets (Section 4.5; Appendix Q). C-MAPSS FD004 and nine N-CMAPSS subsets contain simulated aircraft-engine degradation; NASA Randomized Battery Usage contains measured battery trajectories. Controllers continue operation or replace a component; utility rewards operation and penalizes replacement and failure. All methods share the same remaining-life predictor and fitting data. Thirty paired training-unit bootstrap refits per dataset give:

MethodC-MAPSSN-CMAPSSBatteries
State-dependent risk219.7554.425.78
Shared risk215.0453.755.76
Time-only risk217.0553.445.90
Pooled direct planner217.4451.014.50
Adaptive − pooled [crossed 95% CI]2.30 [−0.16, 5.54]3.41 [1.20, 5.84]1.28 [0.54, 2.03]

Adaptive risk has higher mean utility than pooled planning on all three tasks; crossed intervals support gains on N-CMAPSS and batteries. It also leads shared and time-only risk on both engine datasets. Time-only risk leads on batteries.

The expectation candidate reproduces pooled planning on the fitted model. Separate validation data can retain that policy or select another valuation-induced policy. The gains support the usefulness of valuation selection under estimated dynamics. Shared and time-only comparisons further assess state-conditioned valuation, but candidate coverage and selection also matter: the entire gain cannot be attributed exclusively to state dependence. The comparison concerns the specified pooled planner.

The crossed intervals resample training refits and assessment units. The fixed cohorts contain 50 C-MAPSS engines, 39 N-CMAPSS engines and six battery trajectories: these repetitions measure resampling robustness, not independent dataset collection.

For the three adaptive-minus-pooled comparisons, Holm-adjusted two-sided paired-seed p-values are 5.10×10−45.10\times10^{-4}, 2.40×10−42.40\times10^{-4} and 3.00×10−53.00\times10^{-5}, respectively, calculated from the raw p-values. These tests condition on the assessment cohorts. The crossed intervals also include assessment-unit uncertainty, explaining why the C-MAPSS interval includes zero despite its significant paired-seed test.

Verification cost (Appendix P). An explicit valuation allows us to check whether deployment follows its declared action ranking. In the controlled confirmation, state-only policies attain optimal utility in 80/80 cases and faithful execution in 79/80. In the remaining case, the exact selected-map margin is −0.0683-0.0683; the discrepancy already appears in empirical planning. Utility alone would not reveal it.

On 24 new parameter settings of the same two-stage graph, 48 independently fitted controller sets and three acquisition streams per fit give 432 repeated assessments per allocator. Guided acquisition uses 400,444 capped observations per fitted unit versus 507,771 for uniform and 461,576 for widest-interval acquisition. It certifies 400/432 assessments versus 398/432 for each comparator. Savings are 21.1% [13.4%, 30.4%] and 13.2% [5.3%, 22.9%], using multiplicity-adjusted intervals. Gap-sensitive and lookahead allocators have similar costs; unresolved cases remain included.

On this fresh audit cohort, pooled direct planning attains mean utility 2.923 versus 2.857 for state-only risk and reaches the true optimum in all 48 fits. The observation savings compare audit procedures, not policy utility. A direct controller given the same valuation can use the same audit. Thus NASA tests the performance of valuation selection; Appendix P tests the cost of verifying a declared rule. We explain when to learn that rule in Question 2.

Finally, Table 2 retains the full controlled comparison: at the largest tested budget, state-only, state-time and pooled direct control match under both evaluated laws. Additional time dependence provides no utility gain in those conditions. This three-stage experiment is separate from Theorem 1's two-stage construction.

AuthorsOpenReview ↗

Response to kcjJ (2/3): Scope, candidate coverage and applications

W2. Restricted CVaR scope and nonnested banks complicate attribution to state dependence; FX tail gains are inconclusive, and historical gains occur on one venue and do not persist across refits.

We addressed the risk-family restriction, candidate coverage and application robustness separately.

Risk-family scope. The revised Theorem 1 covers arbitrary probability mixtures of positive-level CVaRs, rather than five selected levels. The five-level grid remains an experimental parameterization. The result establishes strict separation for the constructed sequential family, even allowing favorable action ties. Question 1 below explains the proof extension. Appendix O adds symbolic parameter conditions and finite-support feasibility certificates.

Candidate coverage. Appendix E re-evaluates saved selection scores after including every global candidate in each adaptive bank:

BankOuter-optimal: necessary cases /40Outer-optimal: controls /40
Global040
Original state-only4040
State-only including global4040
Original time-only022
Time-only including global040

State-only optimality is unchanged. The time-only control deficit disappears, identifying a coverage effect rather than an inherent expressive limitation on those controls. The unions contain 218 candidates versus 109 global candidates; this is a retrospective inclusion check with expanded search, not a new equal-budget experiment. The theorem's shared-class obstruction is independent of finite-bank coverage.

Currency Exchange. Mean revenue improves over global and time-only risk by 0.468 [0.342, 0.597] and 3.106 [2.820, 3.406], respectively. Tail point estimates are positive, but their intervals include zero. We distinguish the resolved mean improvement from the uncertain tail improvement.

Historical Bitcoin. On 7,183 later Coinbase/GDAX episodes, one frozen controller improves CVaR10 over shared risk by 0.875 bp [0.232, 1.730] and over time-only risk by 1.231 bp [0.621, 2.037]. The shared-relative difference is positive in both fixed reporting periods. Bitstamp and Bitfinex select shared maps and tie; immediate sale retains stronger GDAX downside protection.

This establishes temporal transfer of one selected fit. Repeated selection need not rediscover that benefit. The complete twenty-refit record remains in Appendix D: repeated-selection mean/CVaR differences average −0.024/−0.032-0.024/-0.032 bp; fixed-form reconstructions average −0.007/−0.227-0.007/-0.227 bp. We explain the observed selection ties and changes in fitted action rankings under Question 3.

The new NASA comparisons in our first comment provide additional decision-performance evidence. We keep their conclusion separate from historical fitting stability: positive maintenance results do not resolve the Bitcoin refitting sensitivity. The manuscript distinguishes structural expressivity, application performance and the stability of selection under refitting.

AuthorsOpenReview ↗

Response to kcjJ (3/3): CVaR levels, use cases and refitting sensitivity

Q1. Why these five CVaR levels: are they essential or an implementation choice, and can the proof extend to broader mixtures?

They are an implementation choice. Following your observation, the theorem now allows

ρμ(Y)=∫(0,1]Cα(Y) μ(dα)\rho_\mu(Y)=\int_{(0,1]}C_\alpha(Y)\,\mu(d\alpha)

for any probability measure μ\mu. For Bp∼Bernoulli(p)B_p\sim\mathrm{Bernoulli}(p) and v≥0v\geq0,

h_p(\alpha)=\frac{(p+\alpha-1)_+}{\alpha}.$$ For $q\geq p$, $h_q(\alpha)\geq h_p(\alpha)$ at every positive tail level. Integration preserves the inequality for every admissible mixture. Supporting Growth therefore forces a shared valuation to value its favorable continuation sufficiently highly; that same valuation then makes the risky route toward Growth too attractive to support Safe elsewhere. State dependence separates these requirements. Appendix A proves the strict margins. Appendix O extends the parameter conditions and characterizes finite-support shared implementation at fixed continuation by $Dw\geq0$, $w\in\Delta$, with a dual certificate for incompatible comparisons. These results broaden the risk class and parameter analysis while retaining the separation theorem's existence scope. **Q2. When should a practitioner learn a risk map rather than directly optimize the outer objective?** When they want outcome data to select a useful local valuation and want that valuation to be an explicit, testable deployment specification. **Learning and specifying the rule are different choices.** If the desired valuation is already known, specify it directly. Learning is useful when the admissible family is known but its particular state-specific rule must be chosen from data. Separate evaluation data select among the resulting policies. Adopting the selected rule as a deployment requirement is then a distinct design decision. For a policy and map, adherence means $$m(\pi,\lambda)=\min_{t,s,a\ne\pi_t(s)} [Q_t^{\pi,\lambda}(s,\pi_t(s))-Q_t^{\pi,\lambda}(s,a)]\geq0.$$ Outer utility measures outcomes; this margin checks whether decisions follow the declared valuation. The NASA comparisons test the usefulness of validation-based valuation selection, with state dependence assessed through the shared and time-only controls; the independent audits test execution and verification cost. A direct controller supplied with the same rule can use the same audit and repair procedure. When outer utility alone defines the task, direct optimization remains the appropriate reference. **Q3. What causes sensitivity across historical-data refits?** The saved-case diagnosis identifies **selection ties and changes in fitted action rankings**. In 14 of the original 15 venue refits, the best varying map ties the best shared map on selection CVaR. The fixed first-index rule then selects shared risk. For ordered returns $y_{(i)}$, $$\widehat C_\alpha= \frac{\sum_{i=1}^{m}y_{(i)}+(\alpha n-m)y_{(m+1)}}{\alpha n}, \qquad m=\lfloor\alpha n\rfloor.$$ The fractional term is omitted when its coefficient is zero. Changes outside the contributing tail can change mean proceeds without changing CVaR. In the remaining GDAX refit, adaptation improves selection CVaR by 0.598 bp and assessment mean by 0.044 bp. Its two changed assessment outcomes lie outside the contributing tail, explaining the zero assessment CVaR difference. Changing the tie rule is not thereby shown to improve performance. Holding the adaptive recipe fixed across the other four reconstructed GDAX models changes action rankings on 688–847 mutually supported counterfactual queries. Preprocessing, projection neighbors and fitted outcome laws change together. This identifies sensitivity in the fitted pipeline without isolating one causal component. Appendix E reports the diagnosis; Appendix D retains the complete refit results. Successful frozen-policy transfer and reliable rediscovery through refitting are therefore distinct findings.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Contributions of the Updated Paper

We thank the reviewers for their time and substantive feedback. The discussion has strengthened the paper through a broader theoretical statement, explicit candidate-coverage checks, additional maintenance and audit experiments, and a clearer presentation of the motivation and learning procedure.

The revised manuscript brings together three contributions:

  • Sequential expressivity (Section 2; Appendix A). Within the constructed family, a stationary state-dependent valuation strictly implements the target policy, while every time-only probability mixture of positive-level CVaRs fails. The result extends beyond the experimental grid and finite candidate banks.

  • Learning useful valuations (Sections 3–4; Appendix Q). Empirical planning and separate validation select a valuation before critic fitting. On the three NASA maintenance tasks, selection improves mean utility over the specified pooled direct planner, with crossed confidence intervals supporting gains on N-CMAPSS and batteries.

  • Independent execution verification (Section 3; Appendix P). Execution checks distinguish faithful adherence from achieved utility. Guided auditing reduces capped observations by 21.1% relative to uniform sampling on newly fitted controllers at new parameter settings of the controlled graph.

Sections 1–3 now explain the representation-theoretic motivation, distinguish the objective from the local valuation, and introduce the learning procedure in its operational order. These changes bring the intuition developed during the discussion into the manuscript itself.

Together, the results connect the expressive capacity of risk valuations, their selection from data, and the verification of their learned execution. We believe this provides a substantive contribution to risk-sensitive reinforcement learning.

We appreciate the opportunity to strengthen the work through this discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.