- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the performative reinforcement learning where deploying the policy can change the environment. The traditional repeated-retraining algorithms can converge to a stable policy while the visited state-action pairs are constrained to the frequent-visited space and the remaining directions form a blind subspace. If the blind space can change the optimal policy, then the converging policy derived from the traditional repeated-retraining algorithm is suboptimal. This paper propose three methods to deal with the sub-optimality: optimism, posterior sampling, and structure-directed probes. The generated experiments on synthetic data show that the proposed methods reduce the regret.
Critical strengths
The paper studies an important problem in performative reinforcement learning, where deploying a policy changes the environment. his policy-environment feedback can cause standard repeated-retraining or certainty-equivalent methods to converge to a stable but sub-optimal policy. Under a structured trigger model, this paper identify the blind space caused by limited observability. This paper also proposes different methods such as optimism, posterior sampling and structured probing to deal with the sub-optimality trap with theoretical guarantees.
Critical weaknesses
- The performative MDP model in eq. 1 is kind of restrictive. It assumes the transition dynamis is composed by fixed parameters B, W, and \phi, while only \epsilon is unknown. I'm not sure if this holds in general settings. It would strengthen the paper to provide further justification for this model in realistic applications and to study the robustness of the proposed methods under misspecified or estimated response structures. 2.The main empirical comparisons are conducted on relatively small tabular MDPs (6 states and 3 actions). Can the exploration methdos scale up to larger state/action space and beyond the tabular setting? Also, current settings are synthetic dataset, and more realistic benchmarks are missed.
Questions for authors
Please refer to weakness part.
Dear Reviewer JqfK,
Thank you again for your review. As the discussion period nears its end, we wanted to check whether our response and revision address your concerns. In brief:
Response design: misspecified, incomplete and learned designs (Apps. P and S), and a new response outside the exponential tilt (App. T). Probing keeps its advantage over certainty equivalence in all of them. Scale: learners at up to 64 states, the size of the standard benchmark of Mandal et al. (2023) (App. Q). Certainty equivalence never recovers the decision-relevant action, while probing always does. Realism: the battery instance is now performative (§5.3, App. C).
All changes are marked in blue in the revised PDF. If any point remains unclear, we would be glad to clarify it before the discussion closes. If our response resolves your concerns, we would be grateful if you would consider reflecting this in your assessment.
Best regards, The authors