- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies generative-model learning in countable-state, average-reward MDPs. This work assumes a known Lyapunov function and stabilizing policy. They develop a model-based algorithm LyCore that learns inside a finite core and uses the stabilizing policy. Its analysis separates performance learning from stability certification, with respective accuracy exponents and , where nu describes core growth. Matching minimax lower bounds hold in the stated parameter regime. Experiments include synthetic diagnostics, birth-death processes, and voltage control.
Critical strengths
-
The bounds quantify how drift strength and core growth affect learning complexity, separating reward optimization from stability certification. The transition at and the dependence on stability slack are informative. The statistical characterization is the main contribution.
-
The analysis addresses substantial technical issues. Independent drift samples retain the stable comparator while leaving the one-out arguments and shell-specific variance estimates to handle model-dependent bias functions without a next-state support factor. The lower bounds allow randomized output policies and adaptive sampling.
Critical weaknesses
-
The second part of Lemma C.4 assumes drift only in the projected MDP (p. 17). This is insufficient since an exit from the core can appear contracting after projection, while the actual switching policy cycles between an in-core and an outside state.
-
The rate diagnostics supply exact drift or rewards and pair independent trials. The complete implementations use uniform per-row sampling. Consequently, the experiments do not directly establish the full algorithm's shell-allocation advantage when both drift and dynamics are learned.
Questions for authors
-
Can the Lemma C.4 be corrected with original-kernel drift and verify that every use satisfies this hypothesis without changing the rates?
-
What explains the aggregate-versus-replay violation gap? Can the author report the replay result and clarify Proposition I.1's scope in the main text.
-
Can the author compare full LyCore with uniform sampling at equal budgets and provide the allocation grids and sampling procedure for Figure 1?