- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
This manuscript proposes a state-dependent mapping over risk functionals, applied in a nested manner, to optimize an outer static risk-sensitive objective. They construct an example where their mapping can express the optimal policy for the outer objective whereas a time-only (undefined?) map is unable to. They propose a selection rule, over a finite bank of risk maps, learned from observed trajectories to optimize the outer objective. Experiments show their method outperforms time-only and global (?) baselines.
Critical strengths
None.
Critical weaknesses
The paper is very telegraphically written with dense, undefined jargon making it very difficult to understand and review.
The prose, starting from the intro, contains too much jargon and terms without definition. The authors use terms like "inner risk functional", and "outer optimization" before defining the problem. Here are a few clarity concerns from the first few pages:
-
"We distinguish the outer objective, which measures the value of the resulting behavior, from the inner risk functional, which ranks available actions." Value of the resulting behavior with respect to what? Ranking with respect to what, the outer behavior or something else?
-
"Learn maps with verifiable execution", what does verifiable execution mean in this context?
-
The entire background section summarizes prior work in less than 10 words, and does not adequately link prior work to this work. The section on coherent risk measures states axioms, without explaining their relevance, and its not clear how these definitions tie to this work.
-
L085: The objective and Bellman operator must agree - meaning what? Agree in what way?
-
L086: Static CVaR decompositions require particular care in policy optimization - what kind of care?
-
L087: We learn a nested risk map and test the decisions it implements - in what context, to reveal what? How does this relate to the above?
-
L100: "We supply a constructive obstruction and a strict state-dependent implementation" obstruction of what, implementation of what?
-
L106: we examine sharing of the valuation rule; sharing in what sense? What is being shared? Valuation of what? How does this differ from prior works' uncertainty in learned dynamics?
-
L111: outer objective is not defined. Same with inner functional.
-
L119: maximized by what, with respect to what (policy? risk sensitive functional? Static or dynamic?)
-
Eq.1: Quantile function should be defined. Why is the specific mixture in (1) constructed this way?
-
Eq 2,3,4: More development is needed here, it is not clear what lambda is doing. These equations are simply a recursion with respect to rho, and should be stated as such. It would help to define V in expanded form (in terms of discounted return) to establish that it's a random variable, so that taking rho of V and Q makes sense.
-
Definition 1: It is not clear to me what this definition means, since pi is in the definition and I am still not sure how lambda plays a role in anything.
-
L139: "Our impossibility statements allow favorable resolution of ties" Which statements? Not clear what this means.
-
L140: More definitions, about maps finally but it's not clear what any of this means. We are defining lambda in terms of other lambdas, but I am not sure what lambda is.
-
Eq. 5 should be defined much earlier.
I stopped reviewing here because the clarity concerns above are too numerous, and I am not confident in my ability to review the paper with sufficient understanding beyond this point. I will mention however that the theoretical contribution in this work is specific to a very narrow family of MDPs (3 states, 2 steps) and the author's proposed method lacks any guarantees, motivation, and is completely heuristic in relying on a finite bank of risk functionals to use.
The work is difficult to understand, and the contribution is narrow so I am of the opinion that the work is not ready for publication.
Questions for authors
See weaknesses.
7 comments
Response to 5mNs (2/3): Definitions and recursion — points 6–12
6. L087: What does learning and testing a nested map reveal?
The procedure is:
- Construct a policy for each candidate map using inner data.
- Evaluate candidate policies on separate meta data.
- Select the map by outer utility.
- Fit the selected map’s critic.
- Assess utility and execution fidelity separately.
The final checks are and . Together they determine whether useful outcomes are accompanied by faithful execution of the selected valuation. Section 3 and the architecture figure now present this order explicitly.
7. L100: Obstruction of what, and implementation of what?
The target is the outer-optimal policy . The theorem establishes
while
Every time-only valuation ranks an alternative above a target action somewhere. One stationary state-only valuation strictly supports every target action. The implementation margin is now defined before the theorem.
8. L106: What is shared, and how does this differ from uncertainty in the dynamics?
Sharing constrains which decision cells use the same valuation weights:
| Class | Valuation at decision cell |
|---|---|
| Global | |
| Time-only | |
| Stationary state-only | |
| State-time |
A time-only rule uses the same quantile weights across states at a given stage. A stationary state-only rule uses the same weights whenever a physical state recurs.
The theorem assumes exact dynamics and exact scores. Model-estimation uncertainty is a separate empirical issue. Section 2.2 now defines all four classes and distinguishes these questions.
9. L111: Where are the outer objective and inner functional defined?
The outer objective now appears as Equation (1). We then define
and the assignment
before introducing the recursion and implementation criterion.
10. L119: What is maximized, over which policies, and is the criterion static or dynamic?
The theoretical target is
where is the class of finite-horizon deterministic Markov policies.
The theoretical is additive expected transformed reward. The local risk valuation is nested. In the liquidation applications, the separately declared outer criterion is episodic revenue CVaR. The revised formulation distinguishes these criteria rather than transferring a guarantee from one to another.
11. Equation 1: What is the quantile function, and why use this mixture?
For ,
and lower-tail CVaR is
Nonnegative mixture weights sum to one and determine emphasis on lower-tail outcomes. The level includes expectation. The integral handles fractional probability mass at atoms.
The five-level grid is an experimental parameterization. The revised theorem permits probability mixtures over all positive tail levels.
12. Equations 2–4: What does do, and which quantities are random?
At a fixed successor state, is a scalar. Conditional on , the pair is random, so
is random. The parameter specifies how its distribution is valued. Section 2.2 now defines , and immediately before Equations (3)–(5).
For two stages,
and
The inner conditional valuation is a scalar at each realized successor. The terminal condition is . Section 2.2 now explains this expansion.
Response to 5mNs (3/3): Implementation, scope and guarantees — points 13–16
13. Definition 1: Why does appear, and how does affect implementation?
We evaluate the proposed policy’s continuation under the map, then check
\qquad\text{for every }t,s,a.$$ Changing $\lambda$ changes action rankings. If every comparison holds, backward induction makes $\pi$ optimal for its map. **Evaluating its own continuation tests implementation without assuming optimality.** ## 14. L139: Which statements allow favorable resolution of ties? Implementation permits tied actions. The separation theorem proves that every time-only map has a margin strictly below $-\varepsilon$ somewhere. Consequently, favorable tie resolution cannot rescue implementation. The definition and theorem now state this convention. ## 15. L140: What do the different lambdas mean? $\lambda$ is a vector of CVaR mixture weights. A map assigns such a vector to each decision cell. For example, using expectation at one state and $C_{1/2}$ at another defines a stationary state-dependent map. A time-only map may change the mixture with the stage, but must use the same mixture across states at that stage. In the generalized theorem, a probability measure over tail levels replaces the finite weight vector. ## 16. Equation 5 should appear earlier The objective now precedes the controller as Equation (1). The example gives $$u(-0.1)=-5.1,\qquad u(0.9)=0.9,$$ and expected utility four for an equiprobable zero/eight reward. ## Broader concerns: narrow theory, motivation and learning guarantees **Scope and novelty.** **The three-state, two-stage restriction belongs to Theorem 1's separation construction.** Section 2.2 defines the framework for general finite-horizon MDPs; we now say this explicitly after the theorem. The construction proves that every admissible time-only mixture fails to support the target policy, while a stationary state-dependent map strictly supports it. It establishes an existence result, not separation in every MDP. The theorem now covers arbitrary probability mixtures of positive-level CVaRs. Appendix O adds symbolic parameter conditions and a finite-support feasibility characterization with dual certificates. The contribution is the sequential sharing obstruction and separate selection/execution assessment; state-dependent preferences, spectral valuations and quantile critics are established ingredients. Sections 1–2 and Appendix H explain the differences from the closest formulations. **What is learned and why use a finite bank?** We select a valuation within an explicit admissible family: $$\widehat{\lambda}\in\underset{\lambda\in\Lambda}{\operatorname{arg\,max}}\;\widehat{J}\left(\widehat{\pi}^{\lambda};D^{\mathrm{meta}}\right).$$ Here $\widehat{\pi}^{\lambda}$ is constructed using inner data, and $D^{\mathrm{meta}}$ denotes separate selection data. Critic fitting follows selection. Neural approximation errors therefore do not determine candidate selection in the corrected method. The finite bank makes this search explicit and reproducible. We do not claim unrestricted learning optimality. Appendix E includes every global candidate in each adaptive bank: state-only optimality remains 40/40 in both settings, while time-only control optimality rises from 22/40 to 40/40, identifying a coverage effect. **What is guaranteed?** For exact optimal nested action scores with unique-action gap $\Delta>0$, $$\lVert\widehat{Q}-Q\rVert_\infty<\Delta/2 \quad\Longrightarrow\quad \widehat{Q}_t(s,a^\star)-\widehat{Q}_t(s,a) \geq\Delta-2\lVert\widehat{Q}-Q\rVert_\infty>0.$$ This standard sufficient condition guarantees faithful greedy decisions **conditional on score accuracy**; it does not prove that training attains the bound. We evaluate execution separately. **Empirical validation.** The independent audit certifies 149 faithful executions and one violation among 240 frozen pairs, leaving 90 unresolved; certificates agree with exact truth, while a plug-in audit makes four errors. Appendix P supplies simultaneous bounds over permitted sampling prefixes, under known support and independent generative observations. Adaptive stopping selects within that covered family. On the original cohort, guided acquisition saves 42.6% against uniform. On 24 new parameter settings of the same graph and 48 independent fitted controller sets, savings are 21.1% against uniform and 13.2% against widest-interval allocation, with positive adjusted intervals. Stronger adaptive allocators have similar costs; unresolved cases remain included. Direct control given the same valuation can use the same audit and repairs. Section 4.5 adds NASA maintenance applications, with positive crossed intervals over pooled planning on N-CMAPSS and batteries. These test empirical valuation selection outside the theoretical graph, separately from neural fidelity. The gains concern the specified pooled planner, not direct optimization generally.
Response to 5mNs (1/3): Objectives, valuation and execution — points 1–5
Thank you for identifying the places where the formulation was too compressed. We have reorganized it around four objects: the policy objective, the local valuation rule, its sequential recursion, and the condition that a policy follows that rule.
The central question is now stated before the formalism: can the decisions preferred by the policy objective be expressed through a shared local valuation, or does their implementation require state-dependent valuation? Below we address each clarification point and then the concerns about theoretical scope and the learning procedure.
1. “Outer objective” versus “inner risk functional”: value and ranking with respect to what?
The outer objective evaluates a complete policy:
\qquad u(r)=r-5\,\mathbf{1}(r<0).$$ Here $R(t)$ denotes the reward at time $t$, and $\mathbf{1}(r<0)$ is the indicator of a negative reward. The local action score is $$Q_t^{\pi,\lambda}(s,a)=\rho_{\lambda_{t,s}}\left(R+\gamma V_{t+1}^{\pi,\lambda}(S')\mid s,a\right).$$ Thus $J$ evaluates achieved behavior according to the designer’s objective. $Q$ ranks a local action according to the declared risk valuation and continuation policy. Section 2 introduces the objective first, followed by the valuation and recursion. ## 2. What does “verifiable execution” mean? For the executed policy $\widehat{\pi}$, define $$m(\widehat{\pi},\lambda)=\min_{t,s,a\ne\widehat{\pi}_t(s)}\left[Q_t^{\widehat{\pi},\lambda}(s,\widehat{\pi}_t(s))-Q_t^{\widehat{\pi},\lambda}(s,a)\right].$$ **Execution is faithful when $m\geq 0$**: every executed action maximizes the selected valuation under the policy’s own continuation. This differs from achieving high outer utility. The controlled confirmation contains 80/80 outer-optimal state-only policies but 79/80 faithful executions. In the remaining case, exact selected-map scores prefer a different action despite the deployed policy’s optimal outer utility. Exact checks use known controlled dynamics. The empirical audit uses simultaneous probability bounds to account for uncertainty in the scores. ## 3. How do the background and coherent-risk axioms relate to this method? The revised background connects the ingredients explicitly: - Spectral risk defines the distribution-to-score rule. - Markov risk evaluation defines its sequential composition. - Quantile critics approximate the outcome distributions used by that composition. Monotonicity and translation equivariance provide the stability property used in the perturbation proof: $$\lVert X-Y\rVert_\infty\leq\epsilon \quad\Longrightarrow\quad |\rho(X)-\rho(Y)|\leq\epsilon.$$ Indeed, $$Y-\epsilon\leq X\leq Y+\epsilon,$$ and applying the two properties gives the bound. Our lower-tail CVaR mixtures are concave reward valuations. Their negatives correspond to the loss-risk convention. The revision distinguishes this nested local valuation from applying a static spectral functional to total return. The theoretical contribution concerns the sequential restriction imposed by sharing the valuation, rather than the introduction of spectral risk itself. ## 4. L085: What does agreement between an objective and its Bellman operator mean? **The recursion must evaluate the criterion attributed to it.** Direct planning for the additive outer objective uses $$U_t(s)=\max_a\mathbb{E}\left[u(R)+\gamma U_{t+1}(S')\mid s,a\right].$$ The risk-map controller uses $$V_t^{\star,\lambda}(s)=\max_a\rho_{\lambda_{t,s}}\left(R+\gamma V_{t+1}^{\star,\lambda}(S')\mid s,a\right).$$ These are different recursions with different roles. The first defines the policy-design objective. The second determines the controller induced by a map. Selection evaluates that controller against the outer objective. ## 5. L086: What care is required with static CVaR decompositions? In general, evaluating total return once differs from composing conditional CVaRs: $$C_\alpha(R_0+\gamma R_1)\ne C_\alpha\left(R_0+\gamma C_\alpha(R_1\mid S_1)\right).$$ **Our controller explicitly uses nested valuation.** We then evaluate the resulting policy against the separately specified outer objective. The revision replaces the compressed sentence with this distinction and an expanded two-stage recursion.