- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Imagine you want to heat up some milk for your coffee in a microwave. A sequence of skills (openFridge, pickUpMilk, placeOnCounter, ..., pickUpMug, placeOnMicrowave) is a task plan you find by having a set of operators defined over symbolic predicates that summarize what each skill does, and specific instantiations of these actions (how you pick up the milk, etc.) correspond to generating motion plans, which is also known as the task and motion planning (TAMP) formulation. Usually, the task plan is generated first, and then you try to find skill parameters that'll solve the task. Many TAMP planners search for these parameters in a DFS style, and backtrack if one value prevents the execution of the future skills. If you pick up a mug from the top, you can't put it in the microwave, because your hand collides---you have to pick it up from the side. You can imagine these types of failures might happen arbitrarily long in the horizon, and the worst case of DFS is just exponential search.
The paper proposes a new approach where we give the task plan and the current poses of objects to a transformer, and predict these parameters in an autoregressive fashion. The motivation is that since the transformer is conditioned on the whole plan, it would generate values that are more coherent with the overall plan (e.g., seeing that the next plan step is putting the mug into the microwave, it would output a distribution of picks that are from the side).
The approach is tested, I guess, on a pick-and-place domain, though the exact details are not in the main text. It performs better than some other baselines in terms of task completion, but it should be noted that no classical TAMP solvers are part of these baselines.
Critical strengths
I think the key strength of this paper is the problem that it's attacking. It is indeed true that backtracking can be a problem, especially when the horizon and the complexity of the task increase. And the approach taken by the paper, using transformers to one-shot refine this plan, is quite interesting. If formulated in some other manner, still using the same motivation, the paper could be very impactful. One such formulation could be maybe shaping the samplers' outputs in a way that we reduce backtracking significantly in practice (while it'll still be exponential in theory).
Critical weaknesses
Following the motivation of the paper, removing backtracking, samplers and everything, one highly expects a comparison with one of those TAMP solvers with samplers that backtrack, like Srivastava et al. (2014). Otherwise, the claim of backtracking being an important problem has not been addressed at all.
Also, this would require training a transformer with every such new predicate. How do we ensure that the system will stay sound? The appeal of TAMP approaches is that they deliberately isolate the problem of task planning from motion planning, and they gain generalization out of this. Here, whenever we have a new set of predicates, we’ll need to train the system from scratch, for new object poses and everything. On the other hand, with usual TAMP approaches, you don’t need to do that. And you also verify that you don’t collide. What do we do here, if one of the poses is actually in collision?
Isn’t the set of demonstrations too much? I’m not sure if it would be a fair comparison, but using a generic grasp detector (which is itself also trained on bunch of objects though) would just allow using a TAMP solver that can backtrack and find solutions 100%, provided that the task predicates are sound and complete, which is another problem in either way. The paper doesn’t show the benefit of using a transformer here, which just requires many samples per task, whereas a TAMP solver with backtracking does it for free. And the last sentence of the paper mentions checking the candidate poses with a motion planner, which contradicts with the motivation of the paper in the first place: doing no backtracking and predicting every pose in one autoregressive run.
Questions for authors
How does this compare to a good TAMP implementation that utilizes parallel motion planners and everything?
Isn't the set of demonstrations too much?
In Section 5, what is the task?
4 comments
Additional Data Efficiency Results
We trained the model on random subsets of both data pools used for training the main model (about 360 to 6,100 episodes). Each variant was trained for the same number of training steps, and evaluated like Table 1 in our main paper.
| Method | TP 9–12 | TP 13–16 | TP 17–20 | FS 9–12 | FS 13–16 | FS 17–20 | Real |
|---|---|---|---|---|---|---|---|
| Identity (no movement) | 0.630 | 0.610 | 0.575 | 0.005 | 0.000 | 0.005 | 0.331 |
| Diffusion-CCSP | 0.839 | 0.838 | 0.827 | 0.100 | 0.115 | 0.105 | 0.760 |
| StructDiffusion | 0.817 | 0.806 | 0.804 | 0.075 | 0.097 | 0.075 | 0.794 |
| CDGS (own recipe) | 0.841 | - | - | 0.083 | - | - | - |
| Plan2Pose, 3% | 0.939 | 0.930 | 0.925 | 0.270 | 0.290 | 0.275 | 0.814 |
| Plan2Pose, 10% | 0.964 | 0.960 | 0.958 | 0.480 | 0.473 | 0.473 | 0.876 |
| Plan2Pose, 25% | 0.976 | 0.971 | 0.967 | 0.585 | 0.585 | 0.540 | 0.909 |
| Plan2Pose, 50% | 0.972 | 0.969 | 0.965 | 0.560 | 0.565 | 0.555 | 0.908 |
| Plan2Pose, 100% | 0.981 | 0.981 | 0.979 | 0.657 | 0.678 | 0.647 | 0.931 |
Real: per-atom satisfaction on the 51 real captures.
These experiments use a slightly modified variant of our model that uses more of the structure given by the symbolic plan. In the submitted model, the contribution of static and dynamic objects in the relation is purely determined by the symbolic domain. This modified variant instead learns to combine the probability maps dynamically. Our current training pipeline was set up for this variant. Due to the time constraints of the rebuttal, we used it for the data-efficiency experiments.
With 10% of the data, task progress drops by about 0.02 and the share of fully correct plans by 0.17–0.21. Even with 3%, the model exceeds all baselines trained on the full data in task progress and fully correct plans. More demonstrations therefore mainly prescient full-plan correctness, but not general placement correctness and transfer to real captures.
I never assumed that Plan2Pose replaces a TAMP planner. What I implied was the lack of comparison with a system that does backtracking, and potentially can run into an exponential runtime. And thank you for your new experiments regarding this concern. The table you provided shows success rates, but we don't know sufficient details about these tasks. Regarding the use of geometric information, I think the occlusion comes for free from the depth camera, without needing any privileged information. And therefore, a true TAMP system would use that to check for collision before stepping any further. Some do more informed stuff such as simulating what happens after the action, but that would require a simulator + a privileged version of the object, which I'm not asking for.
Regarding your results, you said the baseline matches Plan2Pose with geometric information, but the table reports some other number. Which one is it?
If the baseline matches, which I was expecting to since it would eventually find the correct parameterization by backtracking, the selling point would be comparing on how many samples your system finds the solution vs. the other, together with wall-clock time, which you have provided. I think this is the right metric to compare this with an uninformed myopic sampler. But I bet there are others that I don't know as well. Just the top of my head, you could do Bayesian updates to the sampler distribution as you try, and so on.
Not that I'm against transformers, but I just want to make sure that it is the transformer itself that makes this happen.
I'm updating my score to 'weak reject' for the effort of new experiments, but I don't think we have sufficient information about the experimentation procedure, and the conclusions to draw from that.
Thank you for taking the time to follow-up and for raising your score. We apologise for misreading your original concern. The follow-up questions are very helpful for improving the way we present results in the paper. We hope these further clarifications help.
Implementation of the TAMP baseline: We re-implemented the plan-refinement of Srivastava et al., 2014 (Alg. 1 and 2) with the task plan fixed for both the baseline and Plan2Pose. The sampler follows the released placement generator: shuffled uniform positions on the support surface, adapted to our tasks (e.g., objects resting on two supports for the bridge). Plan2Pose runs in the same search-with-backtracking loop and draws each candidate from its predicted placement distributions.In place of a motion planner, every candidate produced by every method is checked geometrically (relations, collision, support), and each check counts as one motion-planning call. The sampler can apply perception-level checks (collision, resting, etc.) from point clouds which do not count as checks. In further variants, the baseline filters candidates with hand-coded geometric definitions of the predicates and of support, or adapts its proposal distribution after failed checks.
Reported numbers: The table in our general response reports the comparison in which the baseline's geometric tests count against the budget of 2,000 checks (0.72 / 0.06 / 0.53). If the tests are free, it matches Plan2Pose (1.00 / 0.93 / 1.00) at the same budget, which is what our text referred to. Following your suggestions, we report the median number of checks and wall-clock time each method needs per problem. We did not capture all of these metrics, therefore we needed to re-run all the experiments leading to slightly different results. We also report median instead of average numbers here, to better handle outliers from unsolved problems.
| Median checks / time per problem | 4-block | 6-block | Bridge |
|---|---|---|---|
| Uniform (Srivastava et al.) | 980 / 0.84 s | >cap | >cap |
| + perception checks | 129 / 0.30 s | 2,614 / 9.3 s | 6,426 / 6.6 s |
| + adaptive proposals | 132 / 0.36 s | 1,824 / 7.1 s | 3,026 / 4.6 s |
| + hand-coded tests | 23 / 0.24 s | 448 / 5.8 s | 106 / 3.2 s |
| Plan2Pose without future plan | 316 / 1.35 s | >cap | 255 / 0.89 s |
| Plan2Pose | 12 / 0.27 s | >cap | 3 / 0.14 s |
| Plan2Pose + perception checks | 4 / 0.25 s | >cap | 3 / 0.14 s |
Medians over all 100 problems, with unsolved problems counted as exceeding the limits. Reported times include all tests and model calls. ‘>cap’: the majority of problems are not solved within the limits (20,000 checks, or 60 s).
With the same perception checks, Plan2Pose needs 4 instead of 129 checks on the 4-block platform and 3 instead of 6,426 on the bridge. Without the future plan, they rise to 316 and 255. The uniform sampler's cost grows from 86 to 226 to 980 checks for 2, 3 and 4 blocks, Plan2Pose's from 5 to 8 to 12. With additional, task-specific hand-coded tests, the baseline matches Plan2Pose in time on the platform and solves more problems on tasks outside of our training distribution.
Transformer: We encode the whole plan and initial point cloud, and the placement is a product of the relevant relations per object. Without the future plan, the same model also keeps the plan serializable in 0.33 instead of 0.89 of the 4-block problems, and 0.46 instead of 0.98 on the bridge (Table 2). Conditioning on the future steps of the plan is therefore key to our results, which we implemented through the transformer’s cross-attention. We do not claim that transformers are inherently better than well-designed samplers, but that non-myopic placements can be learned from automatically annotated demonstrations, without predicate-specific samplers or tests at deployment, as shown by the additional measurements we now supplied. Our implementation is also relatively small (24M parameters, 12–19 h of training on one GPU) and comparable in size to our baseline LINGO-Space (27M trainable parameters).
Tasks: We want to give a better intuition about the tasks. In the bridge task, the plan first places two pillars next to each other and, in the final step, a short or a long lintel on them. A sampler that only looks at the current step places the pillars anywhere valid and finds out only at the last step that the lintel the planner has chosen might not fit, so it must backtrack to the pillars. The symbolic plan does not carry information on the lintel geometry. Plan2Pose sees the lintel in the plan from the start (due to its point cloud from the initial state being encoded) and spaces the pillars accordingly. The platform tasks work the same way: whether later blocks go to the left or the right of the first block is decided only by later steps, so a first block placed without looking ahead can leave no room for them to be properly placed
Role of Plan2Pose: Several concerns seem to assume that Plan2Pose replaces a TAMP solver, and we will make its role clearer in the paper. Plan2Pose replaces only the sampler of a TAMP system (Sec. 1): given a symbolic plan, it proposes target poses for the objects that each action moves. Grasps and motions are still planned, and every pose is still checked, by the downstream motion planner or policy. In the mug example of the summary, choosing a side grasp remains the motion planner's task. What Plan2Pose changes is which poses are proposed: since it sees the rest of the plan, it largely avoids myopic placements that make later steps infeasible and would require backtracking over earlier steps.
Comparison with backtracking TAMP: As suggested, we compared Plan2Pose against Srivastava et al. (2014); see our general response “Comparison with backtracking TAMP”. Regarding a TAMP implementation with parallel motion planners: Plan2Pose only replaces the sampler and can be combined with any such implementation. Since every proposed pose must be checked by the motion planner, fewer proposals mean fewer motion-planning calls: on the 4-block platform, Plan2Pose needs about 185 samples per problem, compared to 799 for the baseline, and about 50 compared to 1,042 on the bridge. This also shows that backtracking is not free: a backtracking solver with sound and complete predicates eventually finds a solution, but at a cost that grows with the plan length. Within 2,000 checks, the baseline solves 0.73 of the 4-block problems but only 0.06 of the bridge problems, which Plan2Pose solves completely.
Too many demonstrations: Plan2Pose is trained once for all tasks, not per task, on about 12k episodes generated in simulation (App. B). Adapting it to new tasks needs far fewer. The foresight experiments fine-tune on 1,000 episodes per family (Sec. 5.3). To test whether this many demonstrations are needed, we are training Plan2Pose on random subsets of 3%, 10%, 25% and 50% of all training episodes, and will post the results here as soon as they are complete. With 10%, the share of satisfied relations on 13–16-step plans remains almost identical (0.964 vs. 0.978), while the share of fully correct plans drops from 0.635 to 0.450. With 25%, it is 0.585.
New predicates and objects: New objects and scenes do not require retraining: Plan2Pose transfers to scenes with 9–10 objects, new scenes of YCB objects, and real point clouds without retraining (Table 3). New predicates require labelled demonstrations, but only fine-tuning, not training from scratch: after fine-tuning on 4,000 episodes with near and right_of, both are satisfied in 0.955–0.970 of the cases, while the existing predicates keep their accuracy (Table 12). We can use VLMs to generate labelled data from demonstrations automatically, which can be derived from existing datasets. With a Qwen 3.8 27B FP8 model, we can label each scene in 2.7s. The model learns to replicate the VLM ground truth predicates with similar satisfaction as our geometrically annotated data. In classical TAMP, a new predicate requires a hand-written sampler and geometric test instead. The separation between task and motion planning is preserved, since Plan2Pose sits between the two.
Soundness and collisions: Since every proposed pose is checked by the TAMP system (see above), soundness does not depend on Plan2Pose. A pose in collision is rejected, and a further candidate is drawn from Plan2Pose's predicted placement distribution (App. E.3). Checking candidates for the current step, as proposed in our conclusion, is local. Backtracking over earlier steps is only needed if an earlier placement leaves no valid placement for the current step, which Plan2Pose's foresight makes much less likely but does not rule out.
Task in Section 5: The task is refining symbolic plans into object poses for tabletop rearrangement, with on, in, left and south. The training data consists of a primary pool with 1,965 bin-packing episodes and a replay pool of 10,400 tabletop episodes with 2–9 states and 3–7 objects, built from geometric primitives, YCB objects, GraspNet scans and cubes (Sec. 5.1, App. B, Table 5) for data diversity. The main evaluation uses general tabletop plans with 9–20 states and 4–6 objects, i.e., longer than any plan seen in training (Table 1). Generalization was evaluated on scenes with 9–10 objects, on new scenes of YCB objects, and on 51 real point-cloud captures from a setup in our lab (Table 3, App. B.2), all without retraining. The foresight evaluation consists of the platform and bridge task families, after fine-tuning all learned models on 1,000 episodes of each family (Sec. 5.3, App. A.1 and F.1). We will expand this description in the main text of the paper (Sec. 5.1).