- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper addresses the question of vector Bellman theory for recurrent-class rewards and transition uncertainty. The authors consider the situation where a controller evolves in a Markov decision process whose dynamics are controlled by a second agent, called Nature, that selects the transitions in an adversarial way. They first extend previous results of Grand-Clément et al. (2023) on the averaged payoff to an all-state result with finite-horizon convergence. Then the authors extend the multichain gain-bias separation to the controller-Nature setting. The remainder of the paper consists of characterizing solvability for the fixed-value equation and for the full optimal-control system. Lastly, they present an algorithm to estimate the optimal robust gain and the optimal gain vector for a given policy.
Critical strengths
S1: The paper extends the “gain first, bias second” separation to the robust setting, imposing it on both the controller and Nature. Any finite solution of the resulting system yields the optimal robust gain and the optimal stationary strategies for both the controller and Nature.
S2: The authors propose an algorithm that works directly with the average reward and, when the system is solvable, approximates the optimal robust gain.
Critical weaknesses
W1: The paper can not be reviewed by a human. It relies on an extensively long appendix, full of equations and mathematical points. To understand the results, a reader has to study dozens of pages of the appendix. The contribution does not suit ICLR format. I recommend the authors to submit to an other venue.
W2: Section 3 does not appear particularly deep, and the proposed generalization of Grand-Clément et al. (2023) seems relatively straightforward.
W3: The exact criteria in Theorems 5.1 and 5.2 cannot be checked in practice.
W4: The iteration after which the extracted policy is optimal is only shown to exist; the paper provides no stopping rule or computable error bound.
W5: The numerical experiments only illustrate the theory on a very simple toy model. They neither test its practical value nor explore the results beyond the stated hypotheses.
Questions for authors
Q1. The sufficient conditions of Proposition 5.3 cover polytopic sets, for which planning methods already exist. Could the authors provide a summary of the uncertainty sets commonly used in the robust MDP and robust RL literature, indicating for each whether finite solvability is guaranteed?
Q2. Apart from polytopic sets and sets where the worst case never removes a transition, which commonly used uncertainty sets are covered by your results?
Q3. Would it be possible to obtain similar results in the case where Nature chooses its action based only on the controller’s state, before the controller acts?
Q4. Would it be possible to provide a stopping rule (see W3)?
Q5. Could you perform numerical experiments on larger environments, and study the performance of the algorithm when the assumptions are violated?