- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces Maturing Markov Decision Processes (MMDPs), where information becomes progressively refined while action opportunities expire. It establishes a sufficient condition for deferring persistent actions and proposes a stage-local action restriction combined with standard RL algorithms. Experiments cover replenishment and cash management.
Critical strengths
- I think the MMDP formulation itself captures an interesting and practically relevant decision structure.
- The paper includes several empirical comparisons and a production-scale case study.
Critical weaknesses
- Despite the interesting and general MMDP formulation, the proposed method simply removes persistent actions at each stage and applies standard RL algorithms. It hard-codes deferral rather than learning when to commit or defer, without checking the condition in Theorem 3.1. The additional action abstraction and search components are also standard. Overall, the method is essentially a predefined action mask rather than a new RL algorithm.
- The main theorem is a straightforward comparison of information gain and delay cost, both involving unknown optimal value functions. These quantities are neither estimated nor used by the algorithm. It seems that no meaningful learning guarantees or approximation bounds are provided.
- The experiment appears to be overly simplified. The main controlled experiments involve two-stage decision cycles, with externally provided forecast updates. In replenishment, deferral is lossless by construction. These experiments do not adequately capture the challenges of the general MMDP formulation.
- It appears that the experiments mainly show that a predefined action restriction can improve standard RL performance. They do not systematically examine the information-gain–delay-cost trade-off or the consequences of violating the theorem's condition.
Questions for authors
- What is the main algorithmic contribution beyond applying standard RL with a predefined action mask?
- How can the condition in Theorem 3.1 be verified or used in practice? It seems that none of these terms are estimated or evaluated in experiments or algorithm design.
- Can the authors evaluate more general multi-stage settings with nontrivial delay costs and quantify the optimality loss from action restriction?