- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper addresses the limitations of reward machines, focusing on cross-task transfer and learning reward-machine components (the transition operator and the labelling/grounding function that maps an observation to an event in the reward machine) from task specifications. The paper proposes "language operators", a learned operator that maps a task specification to a "grounded reward interface"; the operator is trained on a family of task specifications and their corresponding reward interfaces, and the paper shows that the operator can be applied to map a new task specification to a reward interface.
Critical strengths
- The paper's claims and contributions are clear to some extent.
- The experimental results on the benchmarks support the claim.
Critical weaknesses
- The paper is not clear; the preliminaries and the method sections contain many undefined terms, such as: prefix-validity, shaping potential, formal monitor, mission identity, and scene-dependent grounding.
- Even the mathematically defined terms contain undefined terms: What are the target candidates , and what do they represent?
- The paper should be self-contained; it does not contain any reinforcement learning background in the preliminaries section.
- The method section lacks sufficient detail to provide me with an understanding of the method, and it also contains undefined terms such as valid next events, task completion, grounded targets, and Executable formal monitors. The appendix is not useful either and provides little detail on how the method is actually implemented.
- The experiments focus on the superiority of the method in comparison to the baselines, but offer little analysis or an ablation study on what components are important for the method to work effectively. The presentation of the results seems somewhat redundant by writing the exact numbers presented in the learning curves.
- No explanation or definition of what semantic correctness is (line 241) and semantic reference (lines 80 and 243).
- Baselines are not clear: what is the context in context-only PPO, and how is it being represented? How is the task represented in task-conditioned PPO?
Questions for authors
- I wrote my questions in the Weaknesses section.
- Could the authors provide the paper's code? For further inspection.
- How can I know the implementation details of the method? If I want to code up the method, will the paper provide sufficient detail for this?