- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper considers reinforcement-learning settings in which an agent acts only at intermittent decision times. At each decision time, it observes the current state, commits to an action or action chunk, and cannot intervene until an adversarially selected holding time elapses; the next observation arrives fresh. The paper models this setting as a decision-time-robust SMDP (DTR-SMDP). Under -rectangular timing uncertainty, it derives Bellman and policy-iteration results, proposes a PPO-based adversarial-clock method, and reports strong timing-robustness gains on MuJoCo tasks.
Critical strengths
-
The paper formulates robustness to state-dependent uncertainty in decision times, extending fixed-delay and i.i.d. delay models studied in the literature.
-
It distinguishes general globally coupled timing sets from tractable -rectangular timing sets and provides exact dynamic-programming results for the latter.
-
DTR substantially outperforms the evaluated baselines under the paper's fixed timing schedules and under separately trained state-dependent adversaries.
Critical weaknesses
-
Limited novelty of Theorem 3. For each fixed timing rule, the theorem applies the standard SMDP reduction to obtain a decision-epoch MDP. The robust-MDP uncertainty set is then defined as the image of the admissible timing set under this reduction. Thus, the result is a structured robust-MDP interpretation of temporal uncertainty, rather than a general equivalence to arbitrary physical dynamics perturbations.
-
The physical-dynamics claim exceeds the scope of Theorem 3. Theorem 3 guarantees robustness only to transition-reward models induced by admissible timing rules. The paper does not show that gravity scaling or random action replacement induces such a model. Thus, the dynamics experiments show empirical transfer evidence rather than a robustness guarantee.
-
The discounted duality does not match the main experiments. The experiments impose . The discounted equivalence in Theorems 7-9 requires . Hence, the experiments do not directly instantiate the discounted equivalence result.
-
DTR-PPO is not covered by the policy-iteration theorem. Theorem 2 assumes finite spaces and exact Bellman evaluation and greedy improvement. DTR-PPO uses neural function approximation and approximate minimax optimization.
-
Rectangularity restricts the timing model. -rectangularity excludes timing laws coupled across states or time, such as persistent outages and shared-resource constraints.
-
The fixed-mean constraint restricts state dependence. Since for every , adverse states cannot have larger expected delays. State dependence changes only the shape of the local delay distribution.
-
Theorem 6 is policy dependent. The uncertainty set is constructed from the optimal chunk policy's action-tail distribution. It is therefore a post hoc characterization, rather than a policy-independent uncertainty set specified before optimization.
Questions for authors
-
What is conceptually new in Theorem 3 beyond the standard embedded-MDP reduction of an SMDP and the image of a rectangular timing set?
-
How does the fixed state-wise mean-delay constraint model the motivating examples in the introduction, where adverse states are associated with longer observation or intervention delays?
-
What is the practical interpretation of Theorem 6 when its uncertainty set depends on the optimal chunking policy's tail?