acceptodds
Under review as a conference paper at ICLR 2027

When to Commit and When to Defer: Maturing Markov Decision Processes under Refining Information and Expiring Opportunities

Abstract

Sequential decisions often become better informed as action opportunities expire. We introduce Maturing Markov Decision Processes (MMDPs), a structured subclass of augmented finite-horizon Markov decision processes, for organizing when to commit and when to defer. The Expiring-Actions-First Principle provides a sufficient condition for deferral based on information gain and delay loss. This structure motivates stage-local policies, action abstraction, and policy-guided planning. Replenishment experiments show improved learning and test performance under the MMDP interface and, in the primary setting, a larger test-performance gain from forecast refinement. In synthetic cash management, MMDP improves PPO test returns across both network sizes; five-account equal-size mask controls show that action count alone does not explain its learned gain, and the advantage persists with action abstraction and policy-guided search. A production-scale cash-management case study adds complementary evidence. Together, these results support commitment timing as a useful inductive bias for learning and planning.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, Markov decision processes, information refinement, action abstraction, policy-guided planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6RFHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Maturing Markov Decision Processes (MMDPs), where information becomes progressively refined while action opportunities expire. It establishes a sufficient condition for deferring persistent actions and proposes a stage-local action restriction combined with standard RL algorithms. Experiments cover replenishment and cash management.

Critical strengths
  1. I think the MMDP formulation itself captures an interesting and practically relevant decision structure.
  2. The paper includes several empirical comparisons and a production-scale case study.
Critical weaknesses
  1. Despite the interesting and general MMDP formulation, the proposed method simply removes persistent actions at each stage and applies standard RL algorithms. It hard-codes deferral rather than learning when to commit or defer, without checking the condition in Theorem 3.1. The additional action abstraction and search components are also standard. Overall, the method is essentially a predefined action mask rather than a new RL algorithm.
  2. The main theorem is a straightforward comparison of information gain and delay cost, both involving unknown optimal value functions. These quantities are neither estimated nor used by the algorithm. It seems that no meaningful learning guarantees or approximation bounds are provided.
  3. The experiment appears to be overly simplified. The main controlled experiments involve two-stage decision cycles, with externally provided forecast updates. In replenishment, deferral is lossless by construction. These experiments do not adequately capture the challenges of the general MMDP formulation.
  4. It appears that the experiments mainly show that a predefined action restriction can improve standard RL performance. They do not systematically examine the information-gain–delay-cost trade-off or the consequences of violating the theorem's condition.
Questions for authors
  1. What is the main algorithmic contribution beyond applying standard RL with a predefined action mask?
  2. How can the condition in Theorem 3.1 be verified or used in practice? It seems that none of these terms are estimated or evaluated in experiments or algorithm design.
  3. Can the authors evaluate more general multi-stage settings with nontrivial delay costs and quantify the optimality loss from action restriction?
Reviewer_K8C4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Maturing Markov Decision Processes (MMDPs), a specific type of finite-horizon MDP where information gradually improves as some action options become unavailable over time. It explains a key condition that allows focusing on decisions related to actions about to expire, all while still making the best possible choice. Based on this idea, the authors introduce a framework for stage-local policy interfaces that includes options for simplifying actions and conducting policy-guided searches.

Critical strengths

The paper explores a commitment–deferral structure in sequential decision-making and establishes a sufficient condition to eliminate persistent actions from the decision set. A replenishment experiment effectively illustrates this approach, as delaying flexible orders reveals their significant impacts.

The authors assess various learning and planning methods, including PPO, Random, MILP, and several control strategies. The paper notes key limitations, such as the theoretical guarantees not extending to cash management and the reliance on observational evidence for production deployment.

Critical weaknesses

1- The findings presented in the paper don't completely pinpoint the benefits of the suggested temporal action restriction. While the improvements observed in the cash-management experiments are promising, the equal-size mask controls discussed in Section 5 do help in making the comparisons stronger. Still, the authors admit that these controls can't fully separate the effects of expiration semantics from variations in route selection and decoding. Additionally, Section 4 points out that the Flat and MMDP implementations rely on action encodings that depend on the interface. Because of this, it's hard to see just how much of the improvement can be attributed to the proposed commitment–deferral structure compared to other differences in policy interfaces.

2- There’s limited theoretical and practical advantage beyond the evaluated interfaces. The sufficient condition in Theorem 3.1 relies on optimal continuation values, which the implemented policies don't verify. Although it's mathematically sound, it doesn't offer a practical way to determine when restrictions are safe. Additionally, the rolling-horizon MILP baseline yields higher returns than the learned controllers in synthetic cash-management situations and significantly less controller time than Search in the ten-account experiment. This reduces the practical benefits of the proposed planning approach and leaves its overall scientific significance in question.

Questions for authors

1- Can the authors better distinguish the contribution of stage-local action eligibility from differences in policy action representations and decoding in the cash-management experiments? Would the improvements persist under a common action representation where only the eligibility restrictions differ?

2- Beyond cases of lossless deferral, such as the replenishment example, are there nontrivial problem classes where the condition in Theorem 3.1 can be verified using readily available model quantities, without computing the optimal action-value functions?

Reviewer_bgZpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Maturing Markov Decision Processes (MMDPs), which are a structured subclass of augmented finite-horizon MDPs in which the agent learns more as time progresses, however actions disappear as more is learned. Inspired by real-world sequential decision problems which often evolve in a structurally asymmetric manner, MMDPs capture dynamics commonly evolving from dynamic pricing, inventory control, resource allocation, and financial liquidity management. The authors use a collection of experiments to demonstrate improved learning and test performance under the MMDP interface.

Critical strengths
  1. Relevant problem

The paper begins with “Many real-world sequential decision problems evolve in a structurally asymmetric way”, which I very much agree with. This is a very common problem structure, and MMDPs capture it with a precise formalization of how information is gained as potential actions expire. This research direction is undoubtedly important.

I very much appreciate the inclusion of Section 5.3 with the production-scale cash management system rather than just sticking to synthetic benchmarks. This really helps justify the real-world applicability of the MMDP framework.

Critical weaknesses
  1. Experimental results (supporting claims)

In Section 5.2.1 (five-account setting), PPO, MILP, and Random are compared on two interfaces (flat and MMDP). MILP (flat and MMDP) scores -330.3, whereas the best learned policy (MMDP PPO) scores -434.3. Similarly, in Section 5.2.2 (ten account setting), the authors state that “[…] MILP obtains the highest mean returns of −206.29 under MMDP” while the best learned method (MMDP-AA with Search) scores -293.7. So, in both the five and ten account settings, the standard optimization method of MILP beats all of the included RL methods (and is not too much slower comparatively). It is unclear what the RL methods offer over MILP here. I don’t think it’s fatal that MILP outperforms competition, however Section 4 (structure-aware RL for MMDPs) presents structure-aware RL as a main methodological contribution.

Looking at the Random policy in both Section 5.2.1 and 5.2.2, it improves significantly under MMDP versus flat (+69.5 in five-account and +128.6 in ten-account) with no learning. The authors also note this directly, saying “[…] this non-learning gain shows that the MMDP interface can improve outcomes without policy training in this benchmark”; however, for Random in Table 1, the difference between masking all persistent actions and masking an equal number of other actions is only +1.76 (95% CI [-1.11, 4.69]). This seems to suggest that part of MMDP’s benefit comes from the restriction acting as a filter that removes bad actions (basically just action masking [1]), which is not discussed.

  1. Applicability

Following Theorem 3.1, the authors note that if cn(s)≤δn(s)c_n(s) \leq \delta_n(s), then “[…] persistent decisions can remain open without excluding an optimal current choice. It does not require executing every expiring action or always waiting. When cn(s)=0c_n(s) = 0, the condition holds automatically.” However, many (even most) real-world scenarios that can be formulated as an MMDP would deal with situations in which cn(s)>n(s)c_n(s) > n(s) (i.e., the cost of delaying a persistent action outweighs the value of information gained by waiting for a subsequent stage). Despite this, no experiment in Section 5 seems to clearly test a case in which cn(s)>n(s)c_n(s) > n(s). An experiment varying cn(s)c_n(s) and n(s)n(s) such that there are cases in which cn(s)>n(s)c_n(s) > n(s) is confirmed and analyzed directly could demonstrate whether the MMDP restriction underperforms flat. This is very important to its real-world adoption.

Additional notes:

  1. It would have been ideal to include the repository for clear replication of these experiments to pair with provided environmental and simulation details (C.1) and MMDP implementation.

  2. The manuscript begins with a presentation of an interesting maze environment (Figure 1); I would have really appreciated to see it included in the paper itself (formally represented as an MMDP and included in the experiments). This would have provided something a little more tangible compared to the inventory control and cash management problems.

References: [1] Shengyi Huang, Santiago Ontañón: A Closer Look at Invalid Action Masking in Policy Gradient Algorithms. CoRR abs/2006.14171 (2020)

Questions for authors

Q1) What are the RL methods offering over MILP in the five-account setting (Section 5.2.1)?

Q2) Given the performance of MILP in the five-account setting, why is this method not included in Section 5.3 for production-scale cash management?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.