- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes STRAT, a PPO-based RL method that augments policy learning with an auxiliary task of predicting a templated textual description of the agent's state, including its position, goal, subgoal, previous action, progress, and inventory. The aim is to improve learned representations and sample efficiency in sparse-reward environments. Experiments report improvements over PPO, alongside ablations and analyses of representation capacity and linear-probe predictability.
Critical strengths
The paper proposes a potentially useful approach to sparse-reward RL. The method is conceptually simple and appears straightforward to implement, making it an interesting approach to incorporating additional task-relevant supervision into policy learning.
Critical weaknesses
-
Inadequate baselines. Improved performance over vanilla PPO is not particularly surprising given the additional privileged symbolic information available to STRAT. The evaluation should include baselines that exploit the same information, such as an asymmetric actor–critic, to establish whether STRAT offers benefits beyond conventional uses of privileged information.
-
Insufficient justification for textual representations. It is unclear why predicting a textual description of the state abstraction should be more effective than directly predicting its individual structured components. This distinction should be evaluated empirically using comparable auxiliary objectives.
-
Insufficient methodological detail. The method lacks a sufficiently precise mathematical description of how abstract MDP features are used and how the environment's production rules contribute to constructing the supervision. The main paper should also provide substantially more implementation detail, particularly regarding the interaction between the textual representation and the network architectures. The description in Appendix A is insufficient to assess these design choices or reproduce the method reliably.
Questions for authors
-
Could the authors include baselines that exploit the same privileged information, particularly an asymmetric actor–critic, and directly compare textual trace prediction against auxiliary prediction of the corresponding structured state variables?
-
What approaches do the authors envision for overcoming the performance collapse on the Align family of tasks, where the proposed method also fails? What does this failure reveal about the limitations of the method?