- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a system that autonomously learns symbolic representations of manipulation skills without predefined predicates, operators, or tasks, using foundation models to achieve this in a human interpretable manner. They also provide formal definitions and proofs for the soundness and completeness of the resulting model.
Critical strengths
Scalable. Human interpretable.
Critical weaknesses
The presentation of this paper is very good. The algorithm is explained well, grounded in formal definitions, with both simulated and real-world experiments for evaluation. However, the soundness is lacking. The proofs only hold given best case assumptions that cannot be implemented, such as infinite data and no noise. Without these, the contribution is mostly the heuristics that are given to the foundation model as criteria for evaluating the predicates it comes up with. The paper highlights how the system does not need any predefined predicates, operators, or tasks as input, but gives it a very detailed hand-designed type hierarchy, which is a common substitute for predicates. No FM ablation even though the system is heavily dependent on it for performance. Experimental results are based on very few runs. The difference between system performance and baselines could be one run. Predicate acceptance threshold is different per domain in the experiments, suggesting domain tuning.
Questions for authors
- Success labels: How is success or failure determined for each real-robot execution? If a human or VLM labels it, what is the label error rate, and how does it propagate through Conditions 1 and 2?
- Lemma 1: How does the proof handle a failed transition with s' in Xi\setminus beta but Lambda(Delta(s, s')) != Eff, e.g., when Delta = empty? Does Condition 2 need strengthening?
- Proposition 1: Correctness constrains only predicates in Pre union Eff. How do you rule out unmodeled side effects on other predicates, which would break the exact abstract-state equality in Definition 3?
- Theory-algorithm mismatch: With threshold h approximately 0.5 and pruning of small clusters, under what conditions are "all contrastive pairs resolved"? How often does the invention loop end with unresolved inconsistencies?
- Theorem 2: Can you report the numerical value of Eq. 17 for your domains? Is the convergence statement non-vacuous for any realistic P_max?
- Effect ambiguity: In your published Robotouille model, Place_1 and Place_2 overlap with different effects (stove vs. any station), and Cut_1 has no cutting-board requirement. How often do generated plans exploit such ambiguities, and how many plan-budget attempts does this account for?
- Impossible-task metric: Can you report a joint feasibility-detection metric (precision/recall of "infeasible" across all problems), given that Random reaches 100% on impossible tasks?
- Expert real-world performance: What is VLM classification accuracy for the expert predicates? Were their semantic descriptions written with the same level of visual detail as the invented ones?
- Baselines: Why not run Pix2Pred or VisualPredicator with their required supervision, or SkillWrapper with that same supervision? Does FM Invent include any predicate selection or scoring?
- Ablations: Please report Condition-1-only, sensitivity to h and r*_suc, and the full loop with every proposal accepted. Why does h differ across domains?
- Type hierarchy: How much of performance comes from the provided types (e.g., sponge, cookable)? What happens with a flat type system?
- Resets and autonomy: How many human resets were needed per run, especially in the Kuka dead-end domain? How were failed plan attempts handled at evaluation time?
- Goal specification: When the invented vocabulary cannot express part of the goal, how is success judged? Is it against the goal image or against abstract-goal satisfaction?
- Vocabulary variance: How different are the invented vocabularies across the 5 (or 3) runs? Please include all learned models.
- FM dependence: Given Qwen3-VL's failure at invention, how does the full system perform with a weaker or open model? Is the contribution separable from GPT-5's capability?
- Problem construction: Who designed the evaluation problems, and were they fixed before running the method?
5 comments
W1. Theoretical contribution. This concern addresses the applicability of our theory rather than the soundness of its results. Under their stated assumptions, Theorem 1 establishes consistency on a finite observed dataset, and Theorem 2 establishes asymptotic convergence. Together, they define a principled objective for abstraction learning and conditions that guide predicate invention. Without a correct learning objective, a system can still learn an incorrect abstraction even with oracle labels and infinite data. Our theory addresses this fundamental issue, which better FM predictions and more data alone cannot resolve.
W2. Experimental evaluation and FM dependence. Our number of runs and use of a single FM backbone for full-system evaluation are consistent with related work [1-5]. Our controlled comparisons hold the FM fixed, isolating our method’s contribution from differences in FM capability. To our knowledge, we provide the most extensive real-robot evaluation among related predicate-invention methods [1-5], only two of which include real-robot experiments. We additionally evaluate FM capabilities in predicate classification and invention, including an alternative open-source model, providing analyses absent from these prior works.
W3. Role of the type hierarchy. The review does not explain or provide references for how a type hierarchy substitutes for predicates, so we respond based on our understanding. All cited generative predicate-invention methods [1-3] require predefined predicates, partial operators, or training tasks; none can operate without its required inputs simply by receiving a type hierarchy. Our hierarchy is derived entirely from the existing skills’ parameter types. For example, the valid object bindings for Cook are a subset of those for PickUp, so cookable is a subtype of pickupable. This introduces no information beyond the existing skill interfaces.
W4. Domain tuning. We use h=0.6 in Robotouille and h=0.5 in both real-robot domains, as documented in Appendix I.3. Within each domain, Random Exploration, FM Invent, and No Heuristic use the same threshold as SkillWrapper. Thus, the comparisons control for this parameter, and the observed performance differences cannot be attributed to SkillWrapper receiving a different threshold from these baselines.
Q1: Success labeling and label errors
Humans observe each skill execution and provide success/failure labels. Incorrect labels can introduce spurious contrastive pairs. We mitigate it by predicate scoring over the full dataset and pruning small effect clusters during operator learning (L239–243).
Q2: Lemma 1 and the coverage of Condition 2
We agree that Condition 2 misses this case as written. We have strengthened it with an alternative trigger whenever a failed execution terminates in the predicted success set, , regardless of its observed effect,, including empty effects. Under the lemma’s remaining assumptions, resolving all violations detected by the strengthened condition restores Lemma 1’s conclusion. We have revised the condition and proof accordingly.
Although the original Condition 2 misses this case, we checked the learned models and collected transitions at the end of each learning episode and found no failed executions requiring the new term. This suggests that the original effect-invention condition already covers most practically relevant cases in our evaluated settings, while the added term completes its theoretical coverage.
Q3: Unmodeled effects and planning soundness
SkillWrapper does not omit side effects when constructing operators: it computes each effect from the full abstract-state difference over the predicate vocabulary. Thus, every changed predicate in the supporting transitions is included in the effects. Distinct conditional outcomes are represented by separate effect clusters and operators. When multiple operators for the same skill are simultaneously applicable with different effects, we restrict the soundness guarantee to effect-unambiguous plans, as stated at L273–278 and in Appendix B.
Q4: Theoretical assumptions and unresolved contrastive pairs
As clarified above, complete resolution of contrastive pairs is an assumption of the theoretical guarantee, not a claimed property of every SkillWrapper run. Neither the acceptance threshold nor cluster pruning ensures complete resolution.
Across five Robotouille runs, the unresolved contrastive-pair rates after successive learning episodes were , , , , and . Thus, intermediate invention loops ended with unresolved pairs, but all pairs were resolved by the final episode in these runs. This outcome is not guaranteed under a fixed FM-call budget, which may be exhausted without producing an acceptable predicate. The theory provides the objective and predicate-invention criteria; it does not guarantee that finite-budget execution always satisfies its assumptions.
Q5: Numerical interpretation of the convergence bound
Our theory defines the objective for predicate invention and derives the invention conditions, while the derived bound establishes the stated convergence guarantee under its assumptions. We have not numerically evaluated this bound for our domains, but this does not invalidate the theoretical result.
Q6: Operator overlap and missing preconditions
Overlap between Place_1 and Place_2 does not by itself imply incorrect planning. Our objective is to learn models that support planning, not to enforce mutually exclusive operators. The reported results demonstrate that these models remain useful for planning despite such overlaps; overlap alone does not establish an additional planning-budget cost. Learning the cutting-board requirement requires relevant contrastive evidence, such as a failed attempt to cut lettuce on the stove. This requirement is captured in some of the five learned models. More generally, discovering such requirements depends on observing the relevant contrastive evidence, which a fixed real-world interaction budget does not guarantee.
Q7: Joint feasibility-detection metrics
We now report precision, recall, and F1 for infeasibility detection across all problems. Recall corresponds to the “Impossible” rate already reported, while precision accounts for feasible tasks incorrectly classified as infeasible. SkillWrapper achieves the highest F1 score in Franka and the second-highest in Robotouille, behind Expert Operators.
Precision, recall and F1 (%) of identifying impossible tasks.
| Method | RoboTouille Precision | RoboTouille Recall | RoboTouille F1 | Franka Precision | Franka Recall | Franka F1 |
|---|---|---|---|---|---|---|
| Expert Ops. | 83.6 ± 4.4 | 100.0 ± 0.0 | 91.0 ± 2.6 | 100.0 ± 0.0 | 53.3 ± 9.4 | 69.0 ± 8.4 |
| Sys Preds. | 72.3 ± 8.5 | 42.0 ± 7.5 | 52.9 ± 7.8 | -- | -- | -- |
| ViLa | 66.0 ± 23.1 | 20.0 ± 11.0 | 29.9 ± 13.8 | 100.0* | 6.7 ± 9.4 | 11.1 ± 15.7 |
| Random | 22.7 ± 0.0 | 100.0 ± 0.0 | 37.0 ± 0.0 | 51.2 ± 19.0 | 60.0 ± 16.3 | 53.8 ± 14.7 |
| FM Invent | 43.1 ± 24.0 | 78.0 ± 27.1 | 45.1 ± 9.0 | 36.1 ± 0.3 | 86.7 ± 9.4 | 50.9 ± 1.2 |
| No Heuristic | 71.1 ± 3.4 | 80.0 ± 21.0 | 74.1 ± 11.1 | 39.7 ± 4.5 | 53.3 ± 9.4 | 45.5 ± 6.4 |
| Ours | 75.5 ± 8.4 | 100.0 ± 0.0 | 85.8 ± 5.2 | 78.3 ± 2.4 | 73.3 ± 9.4 | 75.6 ± 6.3 |
*ViLa predicts no impossible task in Franka runs 1–2; their undefined precision is excluded, so this value is from run 3 only.
Author questions
Hi Reviewer NRT4,
Can you kindly address the author questions raised below? Thanks!