- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to minimize the information needed to specify a task while still obtaining an optimal policy. It defines Minimal Complete Specification Languages (MCSLs) using sets of tasks that share an optimal policy. The paper proposes SEB, an exact algorithm based on Bellman-style backups, and ALSES, a deep RL approach using discrete codes to approximate the specification language. Both methods are evaluated in gridworld environments.
Critical strengths
The problem of identifying which task details are necessary for optimal behavior is interesting and potentially useful. The idea of grouping tasks that share an optimal policy provides a simple and intuitive way to formalize this problem. It also offers an interesting perspective on task specification by focusing on the information needed for optimal behavior rather than fully describing the task.
Critical weaknesses
-
The overall contribution is not sufficiently clear. SEB seems closely related to the related works, while ALSES takes a rather different approach without clearly building on SEB. The connection between the theoretical and empirical parts feels weak, making it difficult to identify the main technical contribution. Also, minimizing the number of specifications does not necessarily mean reducing human effort, which is the main motivation of the paper.
-
I have concerns about the theoretical assumptions and the empirical evidence. Some statements and examples seem inconsistent, and the exact scope of SEB's correctness is unclear. ALSES also shows unstable performance and often fails to recover the optimal partition. The experiments are limited to gridworlds and do not provide sufficient comparisons to establish the benefits of the proposed approach.
Questions for authors
-
Could the authors clarify the main technical differences between SEB and the related works? In particular, what is new in the backup procedure beyond applying existing equivalence-based compression ideas to task specification? Also, ALSES seems to use a different approach based on discrete latent codes. Is there a specific property of SEB that ALSES is designed to approximate or preserve?
-
The UMDP definition allows transitions to depend on the level, but Algorithm 1 uses a single successor state (T(s,a)). Does SEB assume shared transitions across levels? For example, if the same action leads to different states in different levels, a single state-dependent policy could still be optimal in both. How would the backup handle this case, and does the correctness proof still apply?
-
Theorem 1 seems to treat partitions and covers as equivalent, although maximal SESs can overlap. Would it be more precise to state that their minimum cardinalities are equal, rather than that the objects themselves are equivalent? Also, in Section 5.1, the final minimum cover seems inconsistent with the previously listed maximal SESs and their policy labels. Could the authors clarify these points?
-
ALSES minimizes a combination of expected return and code-use entropy, which seems different from minimizing the number of specifications while preserving optimality for every task. What justifies this surrogate objective? In Figure 7, even the best run achieves optimal behavior in only 1,512 out of 2,048 worlds. Could the authors report per-world optimality gaps alongside code counts to better show this trade-off?
-
ALSES recovers the correct partition in only 4/8 seeds in the noisy maze and 2/8 seeds in the colored key maze. What is the main source of this instability? It would also be useful to compare against simpler policy-based clustering or discrete representation learning baselines to better understand whether the proposed training method offers a clear advantage.
2 comments
Thank you to the authors for the detailed response, which addresses some of my concerns. However, I still have reservations about the technical novelty and theoretical soundness of the proposed approach.
We firstly wanted to say thank you for the thorough review and helping to strengthen this paper through your feedback. To address your review:
Answering your questions about the work:
We hope that with these new results and clarifications we have addressed your main concerns. We would be grateful if you would consider updating your scores, and if anything remains unclear or any concerns remain, please let us know before the discussion period closes.