acceptodds
Under review as a conference paper at ICLR 2027

SkillWrapper: Generative Predicate Invention for Task-level Robot Planning

Abstract

Generalizing from individual skill executions to long-horizon tasks is a core challenge in building autonomous robots. A promising direction is learning high-level, symbolic representations of low-level robot skills, enabling abstract reasoning independent of the low-level state space. Recent advances in foundation models have made it possible to generate symbolic predicates that operate on raw sensory inputs—a process we call generative predicate invention—to facilitate downstream representation learning. However, prior work learns these abstractions using heuristic or ad-hoc procedures, leaving unclear which formal properties they ought to satisfy, and how these properties can guide representation learning. We address these questions by characterizing conditions under which learned representations support sound and complete task-level planning, and using them to guide the design of SkillWrapper, a system that autonomously learns symbolic representations of black-box skills without predefined tasks, predicates, or operators. Our approach leverages foundation models to actively collect robot data and learn human-interpretable, plannable representations directly from RGB observations. Our extensive empirical evaluation in simulation and on real robots shows that SkillWrapper learns abstract representations that enable robots to compose black-box skills to solve unseen, long-horizon tasks in the real world.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotic Planning, Concept Learning, Abstraction Learning for Planning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NRT4OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a system that autonomously learns symbolic representations of manipulation skills without predefined predicates, operators, or tasks, using foundation models to achieve this in a human interpretable manner. They also provide formal definitions and proofs for the soundness and completeness of the resulting model.

Critical strengths

Scalable. Human interpretable.

Critical weaknesses

The presentation of this paper is very good. The algorithm is explained well, grounded in formal definitions, with both simulated and real-world experiments for evaluation. However, the soundness is lacking. The proofs only hold given best case assumptions that cannot be implemented, such as infinite data and no noise. Without these, the contribution is mostly the heuristics that are given to the foundation model as criteria for evaluating the predicates it comes up with. The paper highlights how the system does not need any predefined predicates, operators, or tasks as input, but gives it a very detailed hand-designed type hierarchy, which is a common substitute for predicates. No FM ablation even though the system is heavily dependent on it for performance. Experimental results are based on very few runs. The difference between system performance and baselines could be one run. Predicate acceptance threshold is different per domain in the experiments, suggesting domain tuning.

Questions for authors
  1. Success labels: How is success or failure determined for each real-robot execution? If a human or VLM labels it, what is the label error rate, and how does it propagate through Conditions 1 and 2?
  2. Lemma 1: How does the proof handle a failed transition with s' in Xi\setminus beta but Lambda(Delta(s, s')) != Eff, e.g., when Delta = empty? Does Condition 2 need strengthening?
  3. Proposition 1: Correctness constrains only predicates in Pre union Eff. How do you rule out unmodeled side effects on other predicates, which would break the exact abstract-state equality in Definition 3?
  4. Theory-algorithm mismatch: With threshold h approximately 0.5 and pruning of small clusters, under what conditions are "all contrastive pairs resolved"? How often does the invention loop end with unresolved inconsistencies?
  5. Theorem 2: Can you report the numerical value of Eq. 17 for your domains? Is the convergence statement non-vacuous for any realistic P_max?
  6. Effect ambiguity: In your published Robotouille model, Place_1 and Place_2 overlap with different effects (stove vs. any station), and Cut_1 has no cutting-board requirement. How often do generated plans exploit such ambiguities, and how many plan-budget attempts does this account for?
  7. Impossible-task metric: Can you report a joint feasibility-detection metric (precision/recall of "infeasible" across all problems), given that Random reaches 100% on impossible tasks?
  8. Expert real-world performance: What is VLM classification accuracy for the expert predicates? Were their semantic descriptions written with the same level of visual detail as the invented ones?
  9. Baselines: Why not run Pix2Pred or VisualPredicator with their required supervision, or SkillWrapper with that same supervision? Does FM Invent include any predicate selection or scoring?
  10. Ablations: Please report Condition-1-only, sensitivity to h and r*_suc, and the full loop with every proposal accepted. Why does h differ across domains?
  11. Type hierarchy: How much of performance comes from the provided types (e.g., sponge, cookable)? What happens with a flat type system?
  12. Resets and autonomy: How many human resets were needed per run, especially in the Kuka dead-end domain? How were failed plan attempts handled at evaluation time?
  13. Goal specification: When the invented vocabulary cannot express part of the goal, how is success judged? Is it against the goal image or against abstract-goal satisfaction?
  14. Vocabulary variance: How different are the invented vocabularies across the 5 (or 3) runs? Please include all learned models.
  15. FM dependence: Given Qwen3-VL's failure at invention, how does the full system perform with a weaker or open model? Is the contribution separable from GPT-5's capability?
  16. Problem construction: Who designed the evaluation problems, and were they fixed before running the method?
5 comments
Area_Chair_4BRYOpenReview ↗

Author questions

Hi Reviewer NRT4,

Can you kindly address the author questions raised below? Thanks!

W1. Theoretical contribution. This concern addresses the applicability of our theory rather than the soundness of its results. Under their stated assumptions, Theorem 1 establishes consistency on a finite observed dataset, and Theorem 2 establishes asymptotic convergence. Together, they define a principled objective for abstraction learning and conditions that guide predicate invention. Without a correct learning objective, a system can still learn an incorrect abstraction even with oracle labels and infinite data. Our theory addresses this fundamental issue, which better FM predictions and more data alone cannot resolve.

W2. Experimental evaluation and FM dependence. Our number of runs and use of a single FM backbone for full-system evaluation are consistent with related work [1-5]. Our controlled comparisons hold the FM fixed, isolating our method’s contribution from differences in FM capability. To our knowledge, we provide the most extensive real-robot evaluation among related predicate-invention methods [1-5], only two of which include real-robot experiments. We additionally evaluate FM capabilities in predicate classification and invention, including an alternative open-source model, providing analyses absent from these prior works.

W3. Role of the type hierarchy. The review does not explain or provide references for how a type hierarchy substitutes for predicates, so we respond based on our understanding. All cited generative predicate-invention methods [1-3] require predefined predicates, partial operators, or training tasks; none can operate without its required inputs simply by receiving a type hierarchy. Our hierarchy is derived entirely from the existing skills’ parameter types. For example, the valid object bindings for Cook are a subset of those for PickUp, so cookable is a subtype of pickupable. This introduces no information beyond the existing skill interfaces.

W4. Domain tuning. We use h=0.6 in Robotouille and h=0.5 in both real-robot domains, as documented in Appendix I.3. Within each domain, Random Exploration, FM Invent, and No Heuristic use the same threshold as SkillWrapper. Thus, the comparisons control for this parameter, and the observed performance differences cannot be attributed to SkillWrapper receiving a different threshold from these baselines.

Q1: Success labeling and label errors

Humans observe each skill execution and provide success/failure labels. Incorrect labels can introduce spurious contrastive pairs. We mitigate it by predicate scoring over the full dataset and pruning small effect clusters during operator learning (L239–243).

Q2: Lemma 1 and the coverage of Condition 2

We agree that Condition 2 misses this case as written. We have strengthened it with an alternative trigger whenever a failed execution terminates in the predicted success set, s′∈Ξω∖βωs'\in\Xi_\omega\setminus\beta_\omega, regardless of its observed effect,, including empty effects. Under the lemma’s remaining assumptions, resolving all violations detected by the strengthened condition restores Lemma 1’s conclusion. We have revised the condition and proof accordingly.

Although the original Condition 2 misses this case, we checked the learned models and collected transitions at the end of each learning episode and found no failed executions requiring the new term. This suggests that the original effect-invention condition already covers most practically relevant cases in our evaluated settings, while the added term completes its theoretical coverage.

Q3: Unmodeled effects and planning soundness

SkillWrapper does not omit side effects when constructing operators: it computes each effect from the full abstract-state difference over the predicate vocabulary. Thus, every changed predicate in the supporting transitions is included in the effects. Distinct conditional outcomes are represented by separate effect clusters and operators. When multiple operators for the same skill are simultaneously applicable with different effects, we restrict the soundness guarantee to effect-unambiguous plans, as stated at L273–278 and in Appendix B.

Q4: Theoretical assumptions and unresolved contrastive pairs

As clarified above, complete resolution of contrastive pairs is an assumption of the theoretical guarantee, not a claimed property of every SkillWrapper run. Neither the acceptance threshold nor cluster pruning ensures complete resolution.

Across five Robotouille runs, the unresolved contrastive-pair rates after successive learning episodes were 14.7±20.2%14.7 \pm 20.2\%, 10.0±13.7%10.0 \pm 13.7\%, 2.5±0.7%2.5 \pm 0.7\%, 2.4±0.9%2.4 \pm 0.9\%, and 0±0%0 \pm 0\%. Thus, intermediate invention loops ended with unresolved pairs, but all pairs were resolved by the final episode in these runs. This outcome is not guaranteed under a fixed FM-call budget, which may be exhausted without producing an acceptable predicate. The theory provides the objective and predicate-invention criteria; it does not guarantee that finite-budget execution always satisfies its assumptions.

Q5: Numerical interpretation of the convergence bound

Our theory defines the objective for predicate invention and derives the invention conditions, while the derived bound establishes the stated convergence guarantee under its assumptions. We have not numerically evaluated this bound for our domains, but this does not invalidate the theoretical result.

Q6: Operator overlap and missing preconditions

Overlap between Place_1 and Place_2 does not by itself imply incorrect planning. Our objective is to learn models that support planning, not to enforce mutually exclusive operators. The reported results demonstrate that these models remain useful for planning despite such overlaps; overlap alone does not establish an additional planning-budget cost. Learning the cutting-board requirement requires relevant contrastive evidence, such as a failed attempt to cut lettuce on the stove. This requirement is captured in some of the five learned models. More generally, discovering such requirements depends on observing the relevant contrastive evidence, which a fixed real-world interaction budget does not guarantee.

Q7: Joint feasibility-detection metrics

We now report precision, recall, and F1 for infeasibility detection across all problems. Recall corresponds to the “Impossible” rate already reported, while precision accounts for feasible tasks incorrectly classified as infeasible. SkillWrapper achieves the highest F1 score in Franka and the second-highest in Robotouille, behind Expert Operators.

Precision, recall and F1 (%) of identifying impossible tasks.

MethodRoboTouille PrecisionRoboTouille RecallRoboTouille F1Franka PrecisionFranka RecallFranka F1
Expert Ops.83.6 ± 4.4100.0 ± 0.091.0 ± 2.6100.0 ± 0.053.3 ± 9.469.0 ± 8.4
Sys Preds.72.3 ± 8.542.0 ± 7.552.9 ± 7.8------
ViLa66.0 ± 23.120.0 ± 11.029.9 ± 13.8100.0*6.7 ± 9.411.1 ± 15.7
Random22.7 ± 0.0100.0 ± 0.037.0 ± 0.051.2 ± 19.060.0 ± 16.353.8 ± 14.7
FM Invent43.1 ± 24.078.0 ± 27.145.1 ± 9.036.1 ± 0.386.7 ± 9.450.9 ± 1.2
No Heuristic71.1 ± 3.480.0 ± 21.074.1 ± 11.139.7 ± 4.553.3 ± 9.445.5 ± 6.4
Ours75.5 ± 8.4100.0 ± 0.085.8 ± 5.278.3 ± 2.473.3 ± 9.475.6 ± 6.3

*ViLa predicts no impossible task in Franka runs 1–2; their undefined precision is excluded, so this value is from run 3 only.

Reviewer_hjRaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper learns a symbolic model of a robot's black-box skills, so that an off-the-shelf planner can compose them to solve unseen long-horizon tasks. SkillWrapper starts with no predicates or operators. A foundation model proposes exploration sequences, invents a predicate when two executions look the same under the current predicates but have different outcomes, and labels predicates in RGB images. Operators are then learned by clustering successful transitions by their effects. The paper characterizes when such a model supports sound and complete planning, derives its invention criteria from these conditions, and proves consistency and convergence results. Experiments in a simulated kitchen, on a Franka arm and on a bimanual Kuka measure planning success, including on impossible problems that a sound model should identify as unsolvable. SkillWrapper is compared with foundation-model baselines, ablations and hand-written expert operators, and is reported to perform competitively with the latter (L432-433).

Critical strengths
  1. The problem is interesting, and the work is substantial. The closest generative predicate-invention methods assume predefined predicates, partial operators or training tasks (L355-358), whereas SkillWrapper starts from none and runs as a closed loop on two real robots. This is the core of the contribution.
  2. The trigger for inventing predicates is clean and easy to follow, and it helps in practice. In simulation, the full system beats all three ablations on Hard problems (40.0 against at most 24.0, Table 1). This supports the core mechanism apart from the robot results.
Critical weaknesses
  1. The guarantees describe an idealized learner rather than the implemented system, yet they are the basis for calling the criteria principled (L077-080, L089-091).
    • The proof of Theorem 1 places every successful transition in a cluster used for operator learning (L306-307, L1164-1166), but the implementation prunes small clusters as outliers (L242-243). The paper flags the label assumption of Theorem 2 (L343-347), but not this gap, and it states that Theorem 1 holds without correct labels (L343-345).
    • Theorem 2 assumes correct FM labels (L345-346). On the Franka, gripper_empty and holding are labeled with only 60% accuracy, and these two predicates caused all planning failures (Table 4, L2730-2731).
    • The theory is what the paper offers beyond the system, so the gap between the two should be stated explicitly.
  2. The evidence that the learned models are sound is weak, and this point is the more serious one.
    • The Impossible problems are meant to probe soundness (L396-398). On the Franka, SkillWrapper returns plans for a third of them (66.7 in Table 2). Since all Franka planning failures trace to two mislabeled predicates (L2730-2731), this probe cannot separate unsound operators from labeling errors.
    • The metric also rewards incomplete models. In Table 1, Random scores 100 on Impossible, which App. G.1 calls "a byproduct of model incompleteness" (L1932).
    • Some reported numbers do not fit the stated protocol. These are set out in Question 2.
    • Sound planning is what the invention criteria are designed to deliver, so these are the paper's only tests of it. Questions 2 and 3 ask for the numbers and for a test that separates the two error sources.

I understand that such a system is hard to tune on real robots. Clear answers to these questions would settle both weaknesses.

Questions for authors
  1. [Clarification] At the end of a typical run, how many successful transitions are pruned, and how many contrastive pairs remain unresolved? If both are near zero, I would read Theorem 1 as describing the system, which would change my assessment. If not, please state briefly where the gap lies and what it implies.
  2. [Clarification] Some reported means do not fit the stated protocol, and I would like to check the experimental setup against the results. Each Franka column has five problems over three runs (L360-361, L416-418), so a mean over three runs of five problems is a multiple of 20/3, about 6.67, yet SkillWrapper's In-domain score is 76.7. Expert Operators' 46.7 ± 0 on Impossible would need three identical runs, which give multiples of 20. In Table 1, FM Invent's 23.3, 1.7 and 63.3 match three runs rather than the five stated at L360-361, and the Hard scores 58.1 and 13.9 fit neither three nor five runs. Could you give per-run solved counts for both tables, define ±, confirm the number of runs per baseline, and state the number of Kuka problems? The gaps to Expert Operators and No Heuristic in Table 2 are also at most about one problem per run, so these counts matter for reading the comparison.
  3. [Requested analysis] On the Franka, how often do SkillWrapper's model and the expert model return a plan for an Impossible problem, and solve the other problems, when both plan from hand-labeled states? What is the FM's accuracy on the expert predicates, measured as in Table 4? App. G.1 calls Expert Operators "the theoretical upper bound" in simulation and notes that they degrade on the real robots (L1902-1905). On the Kuka they solve about 33% of problems against about 67% for ViLa (Fig. 5). This test would show whether the learned operators are sound and match the expert ones once labeling errors are removed.
4 comments

W1: The theoretical basis of the invention criteria

We describe the criteria as principled because they are derived from our theoretical analysis and proofs. The term refers to their theoretical justification, rather than asserting that the implemented system unconditionally satisfies all theoretical guarantees.

W1.1: Cluster pruning and label correctness in Theorem 1

We confirm that Theorem 1 does not require correct labels. It establishes consistency with all observed abstract transitions, regardless of whether their labels are correct or incorrect. Without pruning, the learned model would therefore also fit incorrect labels. We prune small clusters to avoid fitting suspected labeling errors. This does not mean Theorem 1 relies on correct labels: the theorem concerns consistency with the observed abstract transitions, rather than whether the learned model correctly describes the underlying environment.

W1.2: Theorem 2 and performance with noisy FM labels

The observation is correct, and SkillWrapper still performs well despite these labeling errors. We reiterate that our theory guides the derivation of the predicate-invention conditions and justifies the learning objective. We do not claim that all theoretical assumptions are satisfied in practice.

W1.3: Clarifying the gap between theory and implementation

We will add a paragraph explicitly stating which theoretical assumptions are not satisfied in the real-world implementation.

W2 / W2.1 / W2.4 / Q3: Separating planning-model errors from FM labeling errors

We conducted the requested experiments to separate planning-model errors from FM labeling errors. We manually annotated all problems in Robotouille (50), Franka (15), and Kuka (10), using the predicate vocabularies of every learned model (5 for Robotouille and 3 each for Franka and Kuka) and the expert operators. With ground-truth labels, expert operators achieve 100% planning success across all three domains. SkillWrapper also achieves 100% on Franka and Kuka, while remaining errors occur on hard Robotouille tasks. Thus, removing labeling errors eliminates all observed Franka planning failures.

We additionally evaluated FM labeling accuracy on the expert predicates. Although the average accuracy is high, frequent errors on a few predicates substantially reduce the proportion of states with all predicates labeled correctly. Examples include is_clean and hand_empty in Franka, and contains_ingredient in Kuka. We report the results below.

Planning with ground-truth state labels.

DomainSetExpertSkillWrapper
RoboTouilleEasy (/20)2020.0 ± 0.0
RoboTouilleHard (/20)2014.8 ± 2.6
RoboTouilleImpossible (/10)1010 ± 0.0
FrankaIn-domain (/5)55.0 ± 0.0
FrankaGeneralization (/5)55.0 ± 0.0
FrankaImpossible (/5)55 ± 0.0
KukaSuccess (/10)1010.0 ± 0.0

Predicate classification of the VLM with expert predicates, against ground-truth labels (%).

DomainAccuracyPrecisionRecallF1States fully correct
RoboTouille98.2 ± 0.195.1 ± 0.497.0 ± 0.196.1 ± 0.278.0 ± 0.7
Franka85.9 ± 0.882.3 ± 2.963.9 ± 1.571.9 ± 1.920.6 ± 3.4
Kuka92.6 ± 0.990.6 ± 2.192.8 ± 0.691.7 ± 1.043.3 ± 8.5

Per-predicate classification accuracy (%) of the VLM with expert predicates.

RoboTouille predicateAccuracyFranka predicateAccuracyKuka predicateAccuracy
is_cooked88.8 ± 0.8is_clean38.3 ± 2.4contains_ingredient75.0 ± 7.1
is_cut94.8 ± 0.8hand_empty66.7 ± 2.7is_clean83.3 ± 4.7
is_on_top97.6 ± 0.1is_holding88.0 ± 0.9is_upright83.3 ± 2.4
obj_free97.8 ± 0.2plate_empty94.4 ± 4.2spread_on_top93.3 ± 2.4
hand_empty99.0 ± 1.2is_on_top98.5 ± 1.0righthand_empty98.3 ± 2.4
is_on_station99.3 ± 0.4container_filled100 ± 0lefthand_empty100 ± 0
is_holding99.9 ± 0.1in_lefthand100 ± 0
station_free100 ± 0in_righthand100 ± 0
is_opened100 ± 0

W2.2: Interpreting the Impossible-task metric

We agree that soundness alone does not establish overall model quality, which is why we evaluate both rejection of Impossible problems and success on feasible problems. Random Exploration baseline illustrates this: an incomplete model can reject Impossible problems while also failing to solve feasible ones. We additionally report precision and recall for identifying Impossible problems, which more clearly demonstrate SkillWrapper’s overall advantage.

Precision, recall and F1 (%) of identifying impossible tasks.

MethodRoboTouille PrecisionRoboTouille RecallRoboTouille F1Franka PrecisionFranka RecallFranka F1
Expert Ops.83.6 ± 4.4100.0 ± 0.091.0 ± 2.6100.0 ± 0.053.3 ± 9.469.0 ± 8.4
Sys Preds.72.3 ± 8.542.0 ± 7.552.9 ± 7.8------
ViLa66.0 ± 23.120.0 ± 11.029.9 ± 13.8100.0*6.7 ± 9.411.1 ± 15.7
Random22.7 ± 0.0100.0 ± 0.037.0 ± 0.051.2 ± 19.060.0 ± 16.353.8 ± 14.7
FM Invent43.1 ± 24.078.0 ± 27.145.1 ± 9.036.1 ± 0.386.7 ± 9.450.9 ± 1.2
No Heuristic71.1 ± 3.480.0 ± 21.074.1 ± 11.139.7 ± 4.553.3 ± 9.445.5 ± 6.4
Ours75.5 ± 8.4100.0 ± 0.085.8 ± 5.278.3 ± 2.473.3 ± 9.475.6 ± 6.3

*ViLa predicts no impossible task in Franka runs 1–2; their undefined precision is excluded, so this value is from run 3 only.

W2.3 / Q2: Corrected results, run counts, and experimental protocol

We thank the reviewer for checking these numbers carefully. We confirm that each baseline uses five runs in Robotouille and three runs each in Franka and Kuka. The ± values denote standard deviations across runs, and Kuka contains 10 evaluation problems. We compiled the results manually and increased the Robotouille experiments from three to five runs; arithmetic and transcription errors occurred in preparing the tables. We have corrected these errors in the paper and provide the per-run solved counts for all baselines across all three domains in a separate comment below. RoboTouille: problems solved in each of the five runs.

MethodRun 1Run 2Run 3Run 4Run 5
Easy (/20)
SkillWrapper1515141416
Expert Ops1615171617
System Pred.1516171516
ViLa6861314
No Heuristic1614161416
FM Invent70714
Random11010
Hard (/20)
SkillWrapper88789
Expert Ops1112111311
System Pred.36706
ViLa00154
No Heuristic88800
FM Invent10100
Random00000
Impossible (/10)
SkillWrapper1010101010
Expert Ops1010101010
System Pred.35544
ViLa22114
No Heuristic9101065
FM Invent51041010
Random1010101010

Franka: problems solved (/5) in each of the three runs.

MethodIn-domain R1In-domain R2In-domain R3Generalization R1Generalization R2Generalization R3Impossible R1Impossible R2Impossible R3
SkillWrapper344333344
Expert Ops334332323
ViLa223010001
No Heuristic323312323
FM Invent022011544
Random000000234

Bimanual Kuka: problems solved (/10) in each of the three runs, final iteration.

MethodRun 1Run 2Run 3
SkillWrapper977
ViLa677
No Heuristic774
Expert Ops343
FM Invent424
Random000

Q1: Unresolved contrastive pairs and pruned transitions

At the end of training, the unresolved contrastive-pair rate was 0±0%0 \pm 0\% across five Robotouille runs. Across successive learning episodes, the rates were 14.7±20.2%14.7 \pm 20.2\%, 10.0±13.7%10.0 \pm 13.7\%, 2.5±0.7%2.5 \pm 0.7\%, 2.4±0.9%2.4 \pm 0.9\%, and 0±0%0 \pm 0\%. We do not have the logs needed to recover the pruning counts. We clarify the relationship between cluster pruning and Theorem 1 in our response to W1.1. Although all contrastive pairs were resolved by the final episode in these runs, this is not guaranteed under a fixed FM-call budget: the budget may be exhausted without producing an acceptable predicate. Theorem 1 provides a theoretical learning objective and the basis for deriving the invention conditions; we do not claim that its conditions are always satisfied in practice.

Thank you for the response, the corrected counts, and the new evaluations on hand-labeled states. Those experiments address my main empirical concern by showing that the real-robot failures come from VLM perception errors rather than broken operators, even though pruning still creates a gap with Theorem 1 that should be stated more carefully. Note that the revised PDF omits the promised paragraph on unmet theoretical assumptions as well as the new rebuttal tables and protocol details, so please make sure they are included in the final version. With my primary empirical question resolved, I am raising my score to 3 while keeping Soundness at 2.

We thank the reviewer for their reassessment and further comments. We propose adding the paragraph below to the end of the theoretical analysis section. We will incorporate it together with the rebuttal tables and protocol details once all reviewers have confirmed that their concerns have been addressed or, at the latest, in the final version of the paper.

Limited exploration, a finite budget for predicate invention, and labeling errors affect how our theoretical guarantees apply on real robots. Theorem 1 establishes consistency on the observed transitions once all identified contrastive pairs have been resolved. In practice, an incorrect success or failure label can create a spurious pair, and the FM may exhaust its call budget before finding a predicate that distinguishes a genuine pair. Theorem 2 additionally assumes correct predicate evaluations and shows that the model’s error under the exploration distribution converges to zero in probability as the number of episodes grows. In a finite robot experiment, rare failure modes might be unobserved, and the FM can misclassify predicates in images. SkillWrapper mitigates the impact of labeling errors by scoring candidate predicates across all observed transitions and pruning small effect clusters during operator learning.

Reviewer_8yCqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SkillWrapper, a framework that uses foundation models (FMs) to automatically learn symbolic planning models for robots from visual observations and interactions.

Critical strengths
  • The paper has a strong theoretical foundation.
  • The paper aims to reduce the manual engineering common in FM-based planning.
Critical weaknesses
  • The approach depends heavily on the FM's perception and reasoning abilities.
  • The experiments cover a relatively small set of objects and actions.
  • The method relies on given engineered action sets, limiting its usability to simple tasks. The method's practical usability and scalability are questionable.
Questions for authors
  • The model relies heavily on LLM prompting. This makes me wonder whether explicitly learning a symbolic planning model is necessary, or whether a more direct approach like Code as Policies could achieve comparable performance given the same information as your method (environment observation, permissible actions and goal). Also, for Vila, did you use the same LLM? In their paper, they use GPT-4V; you are using GPT-5.
  • The method requires exploratory executions to test the FM-proposed plans. How will the exploration time scale with a given larger object and skill set on more complex tasks?
  • What happens if a failure occurs while executing a plan during inference time?
  • The skills included in the paper are rather simple. How will the method perform in more dexterous settings?
  • While the authors acknowledge this, it is unclear how well autonomous data collection will scale in the real world. Even in the single task included in the paper (the bimanual one), there are many irreversible states, and I think the situation would only worsen in more challenging ones. How are environment resets and recovery from failed or irreversible skill executions handled during exploration? How much human intervention is required?
2 comments

We thank the reviewer for the feedback. We address the concerns below.

W1 / Q1: FM dependence and the value of symbolic planning

Like other generative predicate invention methods [1–3], SkillWrapper relies on FMs for predicate invention and visual grounding. These capabilities enable learning semantically meaningful predicates directly from images, and the framework can benefit from improvements in the underlying FM. The FM is guided by an objective that targets correct planning abstractions. Comparisons with baselines using the same FM demonstrate the benefit of this guidance beyond the FM’s underlying perception and reasoning capabilities.

Explicitly learning a planning model allows the robot to capture constraints specific to its skills and environment that commonsense reasoning may miss. Direct planning approaches such as Code as Policies and ViLa can perform well when their prior knowledge matches the available skills. However, a robot’s actual capabilities may differ from these expectations. For example, in our bimanual Kuka environment, the robot cannot pick up the knife again once it has been dropped. SkillWrapper learns these constraints from execution data and incorporates them into a reusable symbolic model for planning.

We evaluate direct FM planning through ViLa, which receives exactly the inputs specified in your question (environment observation, permissible actions, and goal), plus the action history. We used GPT-5 for both SkillWrapper and ViLa, as well as all other FM-based baselines. Under this shared backbone, SkillWrapper substantially outperforms ViLa, e.g., achieving 60.0% versus 6.7% success on the Franka generalization tasks (Table 2). These results support the value of learning an explicit planning model beyond directly prompting the same FM.

W2: Experimental scope

To our knowledge, our robot experiments are the most extensive among related prior work in predicate invention [1-5]. Only two of them have real-world robot experiments.

W3: Assumption of predefined skills

As in prior predicate invention work [1–5], we assume access to a set of object-oriented skills. Engineered action sets are also commonly used to represent prior knowledge when tasks involve executing heterogeneous skills. Having a predefined skill set does not restrict tasks to simple ones: composing these skills can require long-horizon planning with complex dependencies.

Q2: Exploration scalability

Larger object and skill sets are precisely where FM-guided exploration can offer a greater advantage by prioritizing informative executions within a rapidly growing search space. Traditional methods rely on demonstrations or templated sweeps of the state space [4, 5], while SkillWrapper uses FM reasoning to propose targeted sequences that explore relevant skill execution conditions. Exploration time still depends on the complexity of skill preconditions and the difficulty of reaching informative states.

Q3: Handling failures during inference

If a plan fails to reach the goal state, we try the next plan within the planning budget. Once the budget is used up, we count the task as a failure.

Q4: Applicability to dexterous settings

SkillWrapper treats skills as black boxes, so the key requirement is that their preconditions and effects can be represented and grounded. More dexterous settings may require predicates over precise object poses, contact states, or tactile feedback that cannot be reliably inferred from RGB observations alone. However, with FMs supporting more modalities, our framework could learn these planning representations, guided by the same objective of obtaining correct abstractions.

Q5: Resets, recovery, and human intervention

We agree that autonomous data collection remains challenging at scale, which motivates our use of FM reasoning to guide exploration. Our bimanual environment deliberately includes multiple dead ends to test data collection under these constraints. Despite these challenges, SkillWrapper collected informative data and achieved the reported performance.

During exploration, if a skill execution fails, we continue executing applicable skills in the remaining sequence. If the robot becomes stuck in an irreversible state, it remains there for the rest of the episode. A human resets the scene at the beginning of each learning episode.

References

[1] VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning.

[2] From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Models.

[3] ExoPredicator: Learning Abstract Models of Dynamic Worlds for Robot Planning.

[4] From Skills to Symbols: Learning Symbolic Representations for Abstract High-Level Planning.

[5] Predicate Invention for Bilevel Planning.

Thank you again for your time and thoughtful feedback. With the discussion period ending today, we wanted to follow up and ask whether you have any remaining questions or concerns we can address.

Comments on the paper

3 comments
Reviewer_NRT4OpenReview ↗

Response to author comments

In PDDL, a type is equivalent to a static unary predicate. The :typing requirement was introduced as a convenience: any typed domain can be compiled into an untyped one. Each type t becomes a unary predicate (t ?x), true for objects of that type in the initial state. Each typed parameter ?x - t becomes an extra precondition (t ?x). See McDermott et al. (1998), PDDL — The Planning Domain Definition Language, the original specification of :typing; and Haslum, Lipovetzky, Magazzeni & Muise (2019), An Introduction to the Planning Domain Definition Language, which presents types as shorthand for static unary predicates. Planners routinely perform this compilation internally. So in the formal sense, a type assignment is a set of predicates whose truth values are given rather than learned.

Example: the Robotouille expert model expresses the Cut constraint through typing: ?board - cuttingboard. Appendix G reports that SkillWrapper sometimes invented on_cutting_board(item) to capture the same distinction. That is one fact represented once as a type and once as an invented predicate. The reverse also happens: the learned Place_2 and Cut_1 operators lack any stove or cutting-board distinction, because stations are not subtyped. As a result, Place_2 predicts on_stove_surface after placing on any empty station, and Cut_1 permits cutting on any station.

On W2: matching the run counts of related work is a reasonable norm, holding the FM fixed is the right control for internal ablations, and the real-robot breadth is a genuine strength. However, the specific issues raised are not about convention: real-robot differences such as Ours 60.0 ± 0 vs. Expert 53.3 ± 9.4 amount to roughly one problem, so the claims comparing against expert operators should be softened; the impossible-task metric rewards incompleteness (Random scores 100%), so a joint feasibility-detection metric is still needed; expert-predicate classification accuracy would resolve the perception confound in the expert comparison; and does FM Invent include any predicate selection? I am satisfied with the author response on all the other questions and am comfortable with a weak accept. I do strongly recommend that the additional information provided in the rebuttal be worked into the main draft.

We thank the reviewer for the clarification and reassessment.

Type assignments and the role of the hierarchy. We had understood your concern to be about the hierarchy itself as a substitute for predicates. We agree that each PDDL type corresponds to a static unary predicate, and that the hierarchy specifies subset relations among these predicates. For example, cuttable is a subtype of pickupable, so is_cuttable(x) implies is_pickupable(x). This information makes operator learning easier because SkillWrapper does not need to invent predicates to recover those memberships, though relations such as on_top_of(x, y) and changing properties such as is_cooked(x) still need to be represented.

In our experiments, all provided types come from the existing skill parameters. This explains why our hierarchy has different levels of detail for items and stations. The skill parameters distinguish cuttable and cookable items within pickupable, but give SkillWrapper only a generic station type. In the expert Cut operator, ?board - cuttingboard together with is_on_station(?item, ?board) requires the item to be on a cutting board. SkillWrapper must learn that requirement from executions. Some runs captured it by inventing on_cutting_board(item). As you observed, the illustrated Cut_1 and Place_2 operators missed the relevant station distinction and therefore generalized too broadly. We view this uneven specificity as a natural feature of practical skill interfaces in the real world, and our results show that SkillWrapper still performs competitively under this uneven type hierarchy.

Comparison with expert operators. Our Discussion describes SkillWrapper as performing competitively with Expert Operators (L432). We will use that wording consistently and revise the stronger statement at L423–424, where we say that SkillWrapper is “surpassing all baselines within five iterations”.

Joint feasibility detection. We evaluated infeasibility detection across both feasible and impossible problems in Robotouille and Franka. Taking “impossible” as the positive class, recall is the fraction of impossible problems correctly identified, which is the original Impossible column. Precision is the fraction of problems predicted impossible that actually are impossible, so it penalizes a model for rejecting feasible tasks. This distinction is visible in Robotouille, where Random and SkillWrapper both achieve 100% recall, but Random has 22.7% precision and 37.0% F1, compared with 75.5% precision and 85.8% F1 for SkillWrapper.

Precision, recall and F1 (%) of identifying impossible tasks.

MethodRoboTouille PrecisionRoboTouille RecallRoboTouille F1Franka PrecisionFranka RecallFranka F1
Expert Ops.83.6 ± 4.4100.0 ± 0.091.0 ± 2.6100.0 ± 0.053.3 ± 9.469.0 ± 8.4
Sys Preds.72.3 ± 8.542.0 ± 7.552.9 ± 7.8------
ViLa66.0 ± 23.120.0 ± 11.029.9 ± 13.8100.0*6.7 ± 9.411.1 ± 15.7
Random22.7 ± 0.0100.0 ± 0.037.0 ± 0.051.2 ± 19.060.0 ± 16.353.8 ± 14.7
FM Invent43.1 ± 24.078.0 ± 27.145.1 ± 9.036.1 ± 0.386.7 ± 9.450.9 ± 1.2
No Heuristic71.1 ± 3.480.0 ± 21.074.1 ± 11.139.7 ± 4.553.3 ± 9.445.5 ± 6.4
Ours75.5 ± 8.4100.0 ± 0.085.8 ± 5.278.3 ± 2.473.3 ± 9.475.6 ± 6.3

*ViLa predicts no impossible task in Franka runs 1–2; their undefined precision is excluded, so this value is from run 3 only.

Expert-predicate classification and planning with manual labels. We measured the VLM’s accuracy on expert predicates against manual labels. Accuracy was 98.2 ± 0.1% in Robotouille, 85.9 ± 0.8% in Franka, and 92.6 ± 0.9% in Kuka. The corresponding fractions of states in which every predicate was classified correctly were 78.0%, 20.6%, and 43.3%. In Franka, for example, the VLM classified the expert predicate is_clean correctly only 38.3% of the time.

We also evaluated every learned model and the expert operators using manually labeled states. With these labels, Expert Operators answered every evaluation problem correctly across all three domains. SkillWrapper also answered every Franka and Kuka problem correctly. Its remaining errors were confined to the Robotouille Hard set, where it solved 14.8 of 20 problems on average. The missing station distinctions you identified are one example of a remaining operator error.

Planning with ground-truth state labels.

DomainSetExpertSkillWrapper
RoboTouilleEasy (/20)2020.0 ± 0.0
RoboTouilleHard (/20)2014.8 ± 2.6
RoboTouilleImpossible (/10)1010 ± 0.0
FrankaIn-domain (/5)55.0 ± 0.0
FrankaGeneralization (/5)55.0 ± 0.0
FrankaImpossible (/5)55 ± 0.0
KukaSuccess (/10)1010.0 ± 0.0

Predicate selection in FM Invent. Yes. FM Invent proposes predicates in batches from the observed images, then applies the same predicate scoring and selection procedure as SkillWrapper.

We will incorporate these clarifications and the main findings into the revised manuscript and include the detailed results in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.