- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SMOOTH is a framework for repairing a robot’s symbolic plan in place when a skill fails even though its modeled preconditions are satisfied. Instead of globally replanning, it diagnoses the missing condition, invents a new predicate, synthesizes an operator to establish it, verifies the repair physically on a checkpointable simulator/digital twin, splices the repair before the failed step, and persists the new knowledge for future tasks.
Core contributions Online in-place plan repair. SMOOTH splices a compensatory repair segment directly before the failed action and keeps the remaining plan suffix intact. It verifies symbolic consistency and physically rehearses the repair, avoiding global replanning churn.
Failure-driven predicate discovery. When an action fails despite satisfied preconditions, a VLM proposes rival open-vocabulary predicates. SMOOTH tests them via directed physical interventions: a predicate is accepted only if satisfying it makes the failed skill succeed.
Operator synthesis via skill chaining. To establish the new predicate, SMOOTH reuses an existing operator, binds a named skill, or composes primitive skills into a sequence. It derives the PDDL operator schema and validates it physically, avoiding sample-inefficient RL.
Persistent abstraction memory. Discovered predicates, grounding classifiers, and composed operators are stored long-term and reused zero-shot across later tasks.
Validation The paper evaluates SMOOTH on:OPENREPAIRBENCH: 77 long-horizon manipulation tasks across Kitchen, Study Desk, and Clinic domains in a cuRobo-backed digital twin.
Critical strengths
-
The system invents both predicates and executable operators, and fixes the plan in place instead of globally replanning. The key mechanism is “VLM proposes + physical hypothesis testing”—the VLM only suggests candidate predicates, and a candidate is accepted only if a physical intervention makes the failed skill succeed. Ablation shows this isn’t just for show: removing physical validation drops turn-1 success from 100% to 26.9%.
-
New predicates become executable classifiers, new operators become PDDL schemas, and skill bindings plus composition recipes are written into JSON memory. Later tasks can plan and reuse them directly, and after convergence, it can run with zero VLM calls.
Critical weaknesses
-
Lack of closed-loop validation of the core repair pipeline. First, there is no real-robot closed-loop validation of the physical hypothesis testing pipeline, and the symbolic-to-execution loop is also incomplete. The physical test depends on a rollback-able digital twin; real-world rollback is often impossible, and rehearsal is skipped when the gripper holds an object—precisely when grasping, moving, and placing are most failure-prone. The paper reports only a preliminary real-hardware transfer test on a Pantry pick-and-place task, not a closed-loop repair, and no detailed quantitative real-robot results; Appendix D is offline analysis, so sim-to-real validity remains unverified. Moreover, operator synthesis only binds to existing named skills or compositions of primitives; target poses and affordances are assumed to be provided by the skill library, not learned or validated. If a newly invented predicate requires a novel affordance or motion strategy, SMOOTH cannot guarantee it will be converted into a real executable action, leaving the execution layer open.
-
Optimistic bias and comparison fairness concerns in the evaluation. Headline numbers use best-of selection over 3 seeds and 56–57 configurations (best 95.8% vs. pooled 54.5%). The physical ablation uses only 26 tasks, 1 seed, and some subgroups have as few as 2–4 tasks. In Table 4, SMOOTH uses a best GPT-4 configuration while global replanning uses Qwen3-VL-8B; LASP/CLIMB are re-implemented, and task sets differ for Lu et al. These issues make the claim of significant superiority not fully supported.
Questions for authors
When a newly invented predicate cannot be established by any existing operator, named skill, or composition of primitives, what does SMOOTH do? Does it learn a new skill, or does the repair fail? How often does this happen in the experiments? Does operator synthesis include geometric parameters such as grasp points, placement poses, and approach directions, or does it assume the skill library already provides them?
In many real-world settings, arbitrary rollback is impractical, and rehearsal is skipped when the gripper holds an object. How does physical hypothesis testing work on a real robot in practice? If it is not performed on the real robot, how do you ensure the repair is valid?
In Table 5, the best configuration reaches 95.8%, but pooled success is only 54.5%—a gap of over 40 points. Can you report per-scenario average success under a single fixed configuration (e.g., the default), without best-of selection? Does best-configuration selection significantly exaggerate the method’s advantage?
In Table 4, SMOOTH uses a best GPT-4 configuration while global replanning uses Qwen3-VL-8B, so the model capabilities are not comparable. Can you re-run all baselines with the same VLM backbone and the same task set? How were the LASP/CLIMB re-implementations validated against the original papers’ performance?
How do you manage growth, conflicts, redundancy, and forgetting in the knowledge library over long-term runs?
3 comments
Response to Reviewer ves2: Fallbacks, Real-Robot Testing and Fixed-Config Results (2/3)
Q2. How does physical hypothesis testing work on a real robot? If it is not performed there, how is the repair known to be valid?
-
It is not performed on the real robot. All tests run in the twin (App. H), so an incorrect hypothesis cannot alter the state of the real robot or its environment. The simulator thus acts as a validator, filtering out invalid repairs proposed by the VLM.
-
The validity of the final accepted repair is therefore guaranteed only to the extent that the twin matches the real world. We make no stronger claim; as noted in §6, the pipeline currently assumes perfect state transfer.
-
Partial evidence off the perfect twin, offline: over 340 evaluations of 45 learned predicates, crisp code keeps .932 accuracy under pose and reading noise, and falls to .621 when joint and latent readings are withheld. Routing unobservable readings to a calibrated judge restores .944 (App. D, Table 14).
-
Why test in the twin at all: with no physical check, 71.0% of accepted repairs are invalid (Table 6). Those would otherwise be discovered on hardware.
Q3. Per-group success under a single fixed configuration. Did best-of selection exaggerate the advantage?
Yes, it inflated SMOOTH's absolute numbers. Fixed-configuration results, 62 tasks × 5 seeds, Qwen3-VL-8B-Thinking, success by failure group:
| Setting | Method | Group 1 Direct (22 tasks) | Group 2 Masked (20) | Group 3 Multi-Gate (20) | All |
|---|---|---|---|---|---|
| Predicate invention (Table 2) | SMOOTH | 100% | 80% | 100% | 93.5% |
| LASP | 32% | 63% | 18% | 37.5% | |
| CLIMB | 32% | 20% | 20% | 24.3% | |
| Operator synthesis (Table 3) | SMOOTH | 97% | 80% | 100% | 92.5% |
| Lu et al. | 71% | 55% | 55% | 61.0% | |
| Compound (Table 4) | SMOOTH | 88% | 80% | 100% | 89.3% |
| VLM + Lu et al. | 52% | 52% | 35% | 46.5% |
-
Size of the inflation: the submission reported 100%, 100% and 95.8% for the three settings. The fixed-configuration numbers are 93.5%, 92.5% and 89.3%.
-
The advantage over baselines holds without selection: +31.5 to +69.2 points of success on identical tasks and backbone.
-
The same configuration on other backbones, turn 1, 186 runs each: Qwen3-VL-8B 88.7%, GPT-4.1-mini 84.9%, GPT-4o 75.3% (Table 11).
-
Where SMOOTH still fails: Group 2, on clinic-relational tasks, where the 8B model misses the probe-in-cradle state (§5.2).
Q4. Re-run baselines with the same backbone and tasks; how were the re-implementations validated?
Re-run: done (table above). Validation: each re-implementation was run on its original tasks with its original models (App. E).
| Baseline | Published | Our re-implementation, original models | Same code, Qwen3-VL-8B | Table |
|---|---|---|---|---|
| LASP (9 cases × 5 runs) | 42/45 (93%) | 40/45 (89%; 90% range 80–96%) | 34/45 (76%) | 15 |
| CLIMB zero-shot, mean of 3 domains | 0.37 | 0.34 | 0.23 | 16 |
| CLIMB few-shot, mean of 3 domains | 0.80 | 0.74 | 0.59 | 16 |
| Lu et al., operator discovery (4 domains × 10 runs) | 37/40 | 35/40 (90% CI 0.76–0.95) | 27/40 | 17 |
-
All six CLIMB cells fall inside the published 95% intervals. LASP's published scores fall inside our 90% range. The one exception is Lu et al. on Nut Assembly (9/10 vs 10/10 published).
-
Every baseline loses accuracy when moved to Qwen3-VL-8B. Part of their gap on OpenRepairBench is therefore a backbone effect, which is why SMOOTH is evaluated on that same backbone.
-
Lu et al. with its original RL skill learner solves 9/62 of our tasks (14.5%). We replaced the learner with a pre-formed skill, which is an upper bound for that method (App. E).
Response to Reviewer ves2: Baseline Validation and Library Growth (3/3)
Q5. Growth, conflicts, redundancy and forgetting in the knowledge library.
The submission did not study this. The revision adds a library-growth experiment (App. C.4): the kitchen domain is extended to 4–14 appliances, giving libraries of 38 to 102 predicates and up to 64 operators, with plans of about 30 steps. 5 seeds × 5 library sizes × 2 conditions.
| Condition (25 runs per row) | Retrieval | Success | Duplicates added | Reuse | VLM calls | Twin trials |
|---|---|---|---|---|---|---|
| Known condition (already in the library) | Off | 92.0% | 100.0% | 96.0% | 6.36 | 3,383 |
| On | 100.0% | 0.0% | 100.0% | 2.00 | 372 | |
| Novel condition | Off | 84.0% | 0.0% | 88.0% | 6.44 | 4,223 |
| On | 88.0% | 0.0% | 96.0% | 5.12 | 485 |
| Aspect | Mechanism | Evidence or limit |
|---|---|---|
| Growth | Predicate retrieval: rank library predicates by relevance to the failed step (reads the step's objects, false at failure, lexical overlap); list the top 24 in the prompt; screen only predicates over the step's objects | With retrieval, success and calls do not grow from 38 to 102 predicates: 5/5 at every size, 2.0 calls (Table 12) |
| Redundancy | Library predicates are screened physically before any new one is proposed (App. A.4); one operator per skill; known operators are searched before the VLM is asked (App. A.5) | An honest negative result: without retrieval, all 25 known-condition runs re-invented an existing predicate under a new name. With retrieval: 0 of 25 (Table 13) |
| Conflicts | Nothing enters the library without passing the physical tests; a rejected repair removes every predicate, attachment and operator it added (App. A.6); existing operators gain effects only on newly invented predicates (App. A.5); attachments whose operator no longer exists are quarantined at load (App. A.7) | No dedicated conflict-resolution experiment |
| Forgetting | None. The library is append-only; nothing is pruned or evicted | Limitation |
Two caveats we want to be explicit about:
-
Retrieval is off by default, so the main results (Tables 2–6) were obtained without it (Table 7). Their libraries are kept per task (App. C.4).
-
The growth study is a separate experiment on GPT-4.1-mini with 5 runs per cell. It shows the trend; it is not a long-horizon lifelong deployment.
Response to Reviewer ves2: Comparison Fairness and Closed-Loop Validation (1/3)
Thank you for the most detailed review we received. Your second weakness was correct on every count and led us to re-run the entire evaluation. Your first weakness is partly fixed and partly a limitation we now state in the main text. References are to the revised PDF.
W2. Optimistic bias and comparison fairness.
One change to flag: the compound baseline in Table 4 is now VLM + Lu et al. (a VLM proposes the missing precondition, Lu et al. synthesizes the operator; App. E). Replanning with SMOOTH's own learned model is compared separately in Table 5.
W1. No closed-loop validation of the core repair pipeline. Three sub-points:
put_down,open_door, pick, with a tail repairclose_doorbefore activate (App. G, Listing 7). A skill blocked only by a full hand is re-screened with the hand emptied (App. A.5)Q1a. When no operator, named skill or primitive composition can establish a new predicate, what does SMOOTH do?
The repair fails for that predicate. No new skill is learned. The fallback order (App. A.5, Algorithms 2–3):
Q1b. How often does this happen?
Every benchmark task is solvable with the provided skills and primitives, so the benchmark does not measure the case of an inexpressible skill. We do not claim coverage of it.
What we do observe: 14.1% of proposed repair segments are rolled back at the local gate (Qwen, App. C.3, Table 11), and one primitive task (desk dial) fails at the gate on all three backbones we tried (App. C.3).
A repair that needs a new continuous motion (e.g. tossing, compliant insertion) fails. §6 names a human-in-the-loop fallback and VLA policies as future work.
Q1c. Does operator synthesis include grasp points, placement poses, approach directions?
Grasp points, placement poses, and approach directions are not passed to the operators directly; instead, they are bound to the objects the operators take as arguments. Operator synthesis selects skills and binds them to objects. Geometry is resolved inside the skill layer (App. H.3):
Primitives are parameterized by a target, e.g.
reach(target="door[?appliance]")(App. G, Listing 5).