- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The high-level idea is to use planning failures to motivate new skill-learning problems, in order to learn to connect a set of achievable fluents up to fluents that are crucial for achievement of the goal. This idea of spawning RL subproblems based on detected gaps in the domain is a good one.
The actual use case is kind of unclear. We're in the middle of planning for some goal, we find we can't achieve it, and then we stop to do some RL? That might be hard to motivate in any kind of real situation.
Critical strengths
The most high-level motivation is good: we really do need a way to assemble a skill library that "covers" a subdomain, and a good way to do it is by noticing and filling gaps.
Critical weaknesses
At the beginning of section 2, the paper says that when an agent fails on a task, either the existing skills may have been composed incorrectly or a capability is missing from the library. First, it's important to distinguish planning failure from execution failure. For planning failure, it might also be that the goal is infeasible, or that there has been an error in perception (that makes the goal seem infeasible). I understand most of the paper is focused on acquiring missing operators; but do you have a way of thinking about when the LLM producing the plan might have been wrong?
It would be important to also incentivize the RL problem to not make any other fluents (or at least any other fluents needed for this goal) go false.
N is in general not a set: when there are multiple ways of making a particular fluent true, it will be a general Boolean formula (without negation).
It seems like it might take very very careful curriculum design (that almost knows what the operator set needs to be) in order to be sure each new bridge problem is small and it's completely clear what needs to be learned.
Minor
- In typical work on logic and symbolic planning, "predicate" means the name of the relation (like "On") and we use "fluent" or "atom" or "literal" to mean that relation applied to arguments (like "On(X,Y)" or "On(b1, b3)")
- "Aeronautiques et al" is not a correct citation: the first author is Malik Ghallab, whose institution is Malik Ghallab Ecole Nationale Superieure D'ingenieur des Constructions Aeronautiques
- "can only fail when Gis nonempty" is ambiguous; do you mean "has no other option but to fail..."?
- What is an "environment-level checker"?
- \vee means that both (not several) predicates must hold
Questions for authors
Exactly how do you identify "reachable support conditions"? It seems to me that there could be a huge reachable set, most of which is irrelevant for this new operator, making it end up being much too narrow.
During learning, how do you pick the set of bindings of the variables? How do you set up a reasonable set of initial states?
Do you tell the LLM what skills you do have, so that it tries to find a plan that works with those skills? There's a risk that it finds a plan with a gap in a case that you actually have the skills necessary for a different plan to accomplish the same goal.
3 comments
Unfortunately, the rebuttal is very densely written and I couldn't decode from it an answer to my most pressing question, which is what would an actual use case in the actual world be?
Given that, the paper doesn't seem acceptable to me.
Thank you for the follow-up.
In the real-world use case, a deployed agent can use PBD to identify which skill to learn when a new task fails. The agent may successfully carry out much of the task with its existing skills, then get stuck at a point where further progress depends on a capability it has yet to learn. With PBD, the agent can identify the requirement its current skills cannot satisfy and use that bridge gap to define what the new skill should achieve.
During routine operation, the agent can record the gap when the task fails. It can pause a training task later, during its next scheduled update, to learn the missing skill. Once it validates that the new skill reliably achieves the required condition, the agent can add it to its library. It can then combine the new skill with existing ones to retry the original task or tackle other tasks that require the same capability. Over successive updates, the agent can use failures encountered in practice to expand its skill library.
We thank the reviewer for the comments and questions.
1. Precondition construction (Q1)
For the selected bridge, we take the nearest preceding subformula under
SEQ, retain its atomic predicates belonging to R, and form their conjunction as the support conditions. After removing duplicates, we prune an atom only if removing it from a copy of R leaves the unchanged predecessor formula satisfied. Concrete objects are generalized using consistent typed variables across supports and target.2. Learning setup(Q2&W2)
Initial-state preparation. Bindings match the template's types and binding constraints. The preceding sequence of existing skills can bring the agent to a state where the candidate's preconditions hold (Appendix D.5). For each binding, rollout initialization prepares and checks the starting state. The bridge policy starts only when all preconditions hold and the target is false; failed preparations are resampled.
Reward and preservation. The reward targets bridge achievement with a step penalty. Initiation conditions may be consumed after use. In our formal guarantee, preparation/restoration calls must preserve conditions already established for the pending call. Execution updates the actual state; available executable skills can restore conditions. Target-predicate validation measures the selected effect's success rate.
3. LLM input (Q3)
The LLM receives the instruction, predicate vocabulary, domain guidance, and grounded environment state, but not the current skill library. PBD subsequently uses the executable library to check whether the program’s required predicates can be achieved by composing existing skills.
Alternative routes. Gap diagnosis is relative to the supplied program. A different route can solve one task while leaving a required capability absent. We acquire that capability to support subsequent programs that require it.
4. Program validation and refinement (W1)
To reduce predicate-program generation errors, our implementation applies two stages of program checking and refinement:
Gap diagnosis assumes a feasible program and correctly grounded state. Execution failures of existing policies are recorded separately. Infeasible goals or incorrect grounding require correcting the task or state description.
5. Curriculum ablation (W4)
Table 3 (Sec. 4.4) compares acquisition with and without curriculum.
Setup.
PBD-directperforms discovery and training inside the target compositional environments, whilePBD-currtrains bridge skills in simpler acquisition environments before reuse.PBD-direct performs worse than PBD-curr on BabyAI but matches it on both MiniHack tasks. Thus, curriculum can improve performance, but is not required for PBD to discover and learn missing skills in these tasks.
6. Why N Is a Set (W3)
Backward reasoning selects an operator-support explanation for the current program requirements. N collects the ground atoms in this selected explanation; conjunctions and temporal dependencies remain in the program and support structure. G=N∖R identifies unsupported requirements of that explanation. Other explanations may use other operators and induce different requirements; N does not represent all alternatives simultaneously.
7. Minor Comments
(1) Terminology. We use “grounded predicate” to mean a ground atom: a predicate applied to concrete object arguments. Accordingly, the elements of R and N are ground atoms.
(2) Author name correction. The incorrect author attribution resulted from a metadata parsing error in Google Scholar. We have corrected the bibliography entry.
(3) Failure wording. The intended statement is: for the supplied program, a nonempty gap identifies a required ground atom that is unreachable under the current symbolic skill model. This statement concerns unsupported program requirements, rather than all possible causes of execution failure.
(4) Environment-level checker. This is the domain-specific function reading predicate truth values from envirnoment state, as described in the symbolic grounding in Appendix D.2.
(5) Logical connective. Our program grammar uses ∧ for conjunction. In ψ1∧ψ2, both subformulas must hold.