acceptodds
Under review as a conference paper at ICLR 2027

Temporally Robust Reinforcement Learning

Abstract

Reinforcement learning (RL) agents face two kinds of distribution shift at deployment: changing *physics* (mass, friction, dynamics) and changing *clock* (delays, missed observations, asynchronous control). Although robust RL addresses the first, the second remains much less addressed, with existing work mostly confined to restrictive i.i.d. delay models. In this work, we introduce decision-time-robust semi-Markov decision processes (DTR-SMDPs), where the interaction time between the agent and the environment is state-dependent and chosen adversarially from an uncertainty set. We prove that the general problem is NP-hard, and propose a rectangularity assumption under which a policy iteration scheme converges. Our central insight is an equivalence between rectangular DTR-SMDPs and a transition-robust MDP class, with a physically plausible uncertainty set, free of the impossible transitions that standard norm-ball sets admit. Within this class, resilience to bad timing *is* resilience to bad physics. This equivalence has a practical consequence: an agent can achieve dynamics generalization by training against an adversarial clock, with no simulator-side physics randomization. We develop a practical algorithm and empirically confirm both its temporal robustness and its zero-shot transfer to physical perturbations. Finally, we characterize which robust-RL baselines do and do not inherit temporal robustness, a split our theory predicts.

Primary area:
reinforcement learning
Keywords:
Robust Reinforcement Learning, Semi-Markov Decision Processes, Delayed Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mV63OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper considers reinforcement-learning settings in which an agent acts only at intermittent decision times. At each decision time, it observes the current state, commits to an action or action chunk, and cannot intervene until an adversarially selected holding time elapses; the next observation arrives fresh. The paper models this setting as a decision-time-robust SMDP (DTR-SMDP). Under ss-rectangular timing uncertainty, it derives Bellman and policy-iteration results, proposes a PPO-based adversarial-clock method, and reports strong timing-robustness gains on MuJoCo tasks.

Critical strengths
  1. The paper formulates robustness to state-dependent uncertainty in decision times, extending fixed-delay and i.i.d. delay models studied in the literature.

  2. It distinguishes general globally coupled timing sets from tractable ss-rectangular timing sets and provides exact dynamic-programming results for the latter.

  3. DTR substantially outperforms the evaluated baselines under the paper's fixed timing schedules and under separately trained state-dependent adversaries.

Critical weaknesses
  1. Limited novelty of Theorem 3. For each fixed timing rule, the theorem applies the standard SMDP reduction to obtain a decision-epoch MDP. The robust-MDP uncertainty set is then defined as the image of the admissible timing set under this reduction. Thus, the result is a structured robust-MDP interpretation of temporal uncertainty, rather than a general equivalence to arbitrary physical dynamics perturbations.

  2. The physical-dynamics claim exceeds the scope of Theorem 3. Theorem 3 guarantees robustness only to transition-reward models induced by admissible timing rules. The paper does not show that gravity scaling or random action replacement induces such a model. Thus, the dynamics experiments show empirical transfer evidence rather than a robustness guarantee.

  3. The discounted duality does not match the main experiments. The experiments impose E[d∣s]=c\mathbb E[d\mid s]=c. The discounted equivalence in Theorems 7-9 requires E[γd∣s]=γc\mathbb E[\gamma^d\mid s]=\gamma^c. Hence, the experiments do not directly instantiate the discounted equivalence result.

  4. DTR-PPO is not covered by the policy-iteration theorem. Theorem 2 assumes finite spaces and exact Bellman evaluation and greedy improvement. DTR-PPO uses neural function approximation and approximate minimax optimization.

  5. Rectangularity restricts the timing model. ss-rectangularity excludes timing laws coupled across states or time, such as persistent outages and shared-resource constraints.

  6. The fixed-mean constraint restricts state dependence. Since E[d∣s]=c\mathbb E[d\mid s]=c for every ss, adverse states cannot have larger expected delays. State dependence changes only the shape of the local delay distribution.

  7. Theorem 6 is policy dependent. The uncertainty set is constructed from the optimal chunk policy's action-tail distribution. It is therefore a post hoc characterization, rather than a policy-independent uncertainty set specified before optimization.

Questions for authors
  1. What is conceptually new in Theorem 3 beyond the standard embedded-MDP reduction of an SMDP and the image of a rectangular timing set?

  2. How does the fixed state-wise mean-delay constraint model the motivating examples in the introduction, where adverse states are associated with longer observation or intervention delays?

  3. What is the practical interpretation of Theorem 6 when its uncertainty set depends on the optimal chunking policy's tail?

Reviewer_GMANOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces decision-time-robust SMDPs (DTR-SMDPs), where a state-depend adversary controls the interval between agent decisions. It formulates the discounted dynamic programming framework, illustrate the worst0-case timing distributions, and connects temporal uncertainty to an induced robust MDP. Empirical studies suggest that adversarial-clock training improves temporal robustness and transfer to physical perturbations on a few basic MuJoCo simulation tasks.

Critical strengths

1, The formulation is detailed, which captures state-dependent timing disruptions that standard i.i.d. delay models miss./ 2, The theoretical contribution is sound under the stated assumptions including policy-iteration contraction, fixed -point, value convergence, optimality, etc.

Critical weaknesses

1, This work misses out the line of works in delayed RL[1-4] for both literature summary and empirical study.
2, In the Intro, authors state about the worst case delay scenario, but in the empirical study authors only test on fixed constant delay and uniform delay setting, which is not sufficient to support authors' claim.
3, Given the paper focus on solving the latency/delay robust RL problem, the task suites here are overly simple with only HalfCheetah, Hopper and Walker2d and the compared baselines are largely robust RL methods.
4, It would be better break down the evaluation of temporal robustness and physical robustness into two separate illustration storyline, given current illustration makes the empirical study section quite misleading and hard to understand.
5, The temporal robustness testing setting is fairly easy with constant delay set to 2 and uniform delay range from 1-3. It would be great to check some literature on conventional delay/latency RL eval setting [1-3].

[1]: Wu, Qingyuan, et al. "Boosting reinforcement learning with strongly delayed feedback through auxiliary short delays." arXiv preprint arXiv:2402.03141 (2024).
[2]: Wikman, John, Alexandre Proutiere, and David Broman. "Adaptive Reinforcement Learning for Unobservable Random Delays." arXiv preprint arXiv:2506.14411 (2025).
[3]: Wu, Qingyuan, et al. "Variational delayed policy optimization." Advances in neural information processing systems 37 (2024): 54330-54356.
[4]: Karamzade, Armin, et al. "Reinforcement learning from delayed observations via world models." arXiv preprint arXiv:2403.12309 (2024).

Questions for authors

1, What is the advantage of formulating the RL latency robust problem here as DTR-SMDP not Delayed MDP or more general SMDP?
2, What is the motivation of proving DTR-SMDP is NP hard here? How does it correlate with the fix-pointed contraction, equivalence, and optimality thm?
3, what is precisely

Reviewer_pZA4OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes decision-time-robust semi-Markov decision processes (DTR-SMDPs), a framework for robust reinforcement learning under temporal uncertainty. The authors formulate state-dependent interaction intervals as adversarial choices and establish theoretical results on NP-hardness, convergence of policy iteration, and equivalence to a class of transition-robust MDPs. Experiments on MuJoCo tasks demonstrate improved robustness to temporal perturbations and zero-shot generalization to unseen dynamics perturbations.

Critical strengths
  • The paper provides a principled formulation of temporal robustness, along with theoretical guarantees on convergence and an equivalence between DTR-SMDPs and transition-robust MDPs. In particular, this theoretical connection is an interesting contribution, as it provides a novel perspective on achieving dynamics robustness through temporal perturbations without explicit physics randomization.

  • Experimental results across multiple MuJoCo tasks and comparisons with robust RL baselines demonstrate substantial improvements under adversarial timing, along with promising zero-shot dynamics generalization.

Critical weaknesses
  • The framework assumes that fresh observations are always available at each decision time step. This limits its direct applicability to more general delayed RL settings, such as environments with observation delays, where the agent receives st−Δs_{t-\Delta} instead of sts_t at decision time tt (Δ\Delta denotes the observation delay). This limitation is also acknowledged by the authors in the main text.
Questions for authors
  • The framework assumes that fresh observations are always available at each decision time step. Could the authors clarify which aspects of the proposed formulation and theoretical guarantees would remain applicable under delayed observations? If such an extension is feasible, could the framework also accommodate action execution delays?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.