acceptodds
Under review as a conference paper at ICLR 2027

Plan2Pose: Prescient Downward Refinement of Symbolic Plans over Long Horizons

Abstract

Task and motion planning uses symbolic abstractions to determine a high-level, symbolic action plan, and then refines the actions into continuous robot configurations. This downward refinement typically makes use of hand-designed samplers, constraint solvers, or predicate-specific learned models, which require hand-crafted predicate-specific supervision or losses. Learned models are myopic and only consider the current configuration’s requirements, producing refinements that may cause subsequent actions to be unrealizable. This is particularly problematic in long-horizon tasks that require many sequential refinements. We propose a refinement learning method that is prescient where needed, i.e., it learns the relevant aspects of the remaining plan instead of myopically focusing on the current action. In contrast to previous approaches, we use a predicate-agnostic approach that avoids hand-crafted predicate-specific losses. Given a PDDL plan and an object-segmented point cloud of the initial scene, a transformer embeds the whole action sequence. For a given state, we learn placement fields that are composed to produce an *SE(3)* pose that satisfies all object relations described in the state. These placement fields are learned without predicate-specific losses, evaluators, or hand-coded geometric semantics, but rather purely from learning to replicate the distributions seen in demonstrations. Our approach, **Plan2Pose**, is trained on synthetic rearrangement plans of at most nine states and outperforms existing baselines on target-atom satisfaction, especially on OOD plan lengths. Under autoregressive rollout on 200 plans of length 17–20, Plan2Pose satisfies 95.5% of state atoms in the final state, compared to 58.4-76.8% achieved by learned baselines, demonstrating substantially more robust long-horizon refinement. Plan2Pose transfers zero-shot to real point clouds, and fine-tuning adds new predicates without forgetting.

Primary area:
applications to robotics, autonomy, planning
Keywords:
TAMP, robotics, symbolic planning, downward refinement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fN1XOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Imagine you want to heat up some milk for your coffee in a microwave. A sequence of skills (openFridge, pickUpMilk, placeOnCounter, ..., pickUpMug, placeOnMicrowave) is a task plan you find by having a set of operators defined over symbolic predicates that summarize what each skill does, and specific instantiations of these actions (how you pick up the milk, etc.) correspond to generating motion plans, which is also known as the task and motion planning (TAMP) formulation. Usually, the task plan is generated first, and then you try to find skill parameters that'll solve the task. Many TAMP planners search for these parameters in a DFS style, and backtrack if one value prevents the execution of the future skills. If you pick up a mug from the top, you can't put it in the microwave, because your hand collides---you have to pick it up from the side. You can imagine these types of failures might happen arbitrarily long in the horizon, and the worst case of DFS is just exponential search.

The paper proposes a new approach where we give the task plan and the current poses of objects to a transformer, and predict these parameters in an autoregressive fashion. The motivation is that since the transformer is conditioned on the whole plan, it would generate values that are more coherent with the overall plan (e.g., seeing that the next plan step is putting the mug into the microwave, it would output a distribution of picks that are from the side).

The approach is tested, I guess, on a pick-and-place domain, though the exact details are not in the main text. It performs better than some other baselines in terms of task completion, but it should be noted that no classical TAMP solvers are part of these baselines.

Critical strengths

I think the key strength of this paper is the problem that it's attacking. It is indeed true that backtracking can be a problem, especially when the horizon and the complexity of the task increase. And the approach taken by the paper, using transformers to one-shot refine this plan, is quite interesting. If formulated in some other manner, still using the same motivation, the paper could be very impactful. One such formulation could be maybe shaping the samplers' outputs in a way that we reduce backtracking significantly in practice (while it'll still be exponential in theory).

Critical weaknesses

Following the motivation of the paper, removing backtracking, samplers and everything, one highly expects a comparison with one of those TAMP solvers with samplers that backtrack, like Srivastava et al. (2014). Otherwise, the claim of backtracking being an important problem has not been addressed at all.

Also, this would require training a transformer with every such new predicate. How do we ensure that the system will stay sound? The appeal of TAMP approaches is that they deliberately isolate the problem of task planning from motion planning, and they gain generalization out of this. Here, whenever we have a new set of predicates, we’ll need to train the system from scratch, for new object poses and everything. On the other hand, with usual TAMP approaches, you don’t need to do that. And you also verify that you don’t collide. What do we do here, if one of the poses is actually in collision?

Isn’t the set of demonstrations too much? I’m not sure if it would be a fair comparison, but using a generic grasp detector (which is itself also trained on bunch of objects though) would just allow using a TAMP solver that can backtrack and find solutions 100%, provided that the task predicates are sound and complete, which is another problem in either way. The paper doesn’t show the benefit of using a transformer here, which just requires many samples per task, whereas a TAMP solver with backtracking does it for free. And the last sentence of the paper mentions checking the candidate poses with a motion planner, which contradicts with the motivation of the paper in the first place: doing no backtracking and predicting every pose in one autoregressive run.

Questions for authors

How does this compare to a good TAMP implementation that utilizes parallel motion planners and everything?

Isn't the set of demonstrations too much?

In Section 5, what is the task?

4 comments

Role of Plan2Pose: Several concerns seem to assume that Plan2Pose replaces a TAMP solver, and we will make its role clearer in the paper. Plan2Pose replaces only the sampler of a TAMP system (Sec. 1): given a symbolic plan, it proposes target poses for the objects that each action moves. Grasps and motions are still planned, and every pose is still checked, by the downstream motion planner or policy. In the mug example of the summary, choosing a side grasp remains the motion planner's task. What Plan2Pose changes is which poses are proposed: since it sees the rest of the plan, it largely avoids myopic placements that make later steps infeasible and would require backtracking over earlier steps.

Comparison with backtracking TAMP: As suggested, we compared Plan2Pose against Srivastava et al. (2014); see our general response “Comparison with backtracking TAMP”. Regarding a TAMP implementation with parallel motion planners: Plan2Pose only replaces the sampler and can be combined with any such implementation. Since every proposed pose must be checked by the motion planner, fewer proposals mean fewer motion-planning calls: on the 4-block platform, Plan2Pose needs about 185 samples per problem, compared to 799 for the baseline, and about 50 compared to 1,042 on the bridge. This also shows that backtracking is not free: a backtracking solver with sound and complete predicates eventually finds a solution, but at a cost that grows with the plan length. Within 2,000 checks, the baseline solves 0.73 of the 4-block problems but only 0.06 of the bridge problems, which Plan2Pose solves completely.

Too many demonstrations: Plan2Pose is trained once for all tasks, not per task, on about 12k episodes generated in simulation (App. B). Adapting it to new tasks needs far fewer. The foresight experiments fine-tune on 1,000 episodes per family (Sec. 5.3). To test whether this many demonstrations are needed, we are training Plan2Pose on random subsets of 3%, 10%, 25% and 50% of all training episodes, and will post the results here as soon as they are complete. With 10%, the share of satisfied relations on 13–16-step plans remains almost identical (0.964 vs. 0.978), while the share of fully correct plans drops from 0.635 to 0.450. With 25%, it is 0.585.

New predicates and objects: New objects and scenes do not require retraining: Plan2Pose transfers to scenes with 9–10 objects, new scenes of YCB objects, and real point clouds without retraining (Table 3). New predicates require labelled demonstrations, but only fine-tuning, not training from scratch: after fine-tuning on 4,000 episodes with near and right_of, both are satisfied in 0.955–0.970 of the cases, while the existing predicates keep their accuracy (Table 12). We can use VLMs to generate labelled data from demonstrations automatically, which can be derived from existing datasets. With a Qwen 3.8 27B FP8 model, we can label each scene in 2.7s. The model learns to replicate the VLM ground truth predicates with similar satisfaction as our geometrically annotated data. In classical TAMP, a new predicate requires a hand-written sampler and geometric test instead. The separation between task and motion planning is preserved, since Plan2Pose sits between the two.

Soundness and collisions: Since every proposed pose is checked by the TAMP system (see above), soundness does not depend on Plan2Pose. A pose in collision is rejected, and a further candidate is drawn from Plan2Pose's predicted placement distribution (App. E.3). Checking candidates for the current step, as proposed in our conclusion, is local. Backtracking over earlier steps is only needed if an earlier placement leaves no valid placement for the current step, which Plan2Pose's foresight makes much less likely but does not rule out.

Task in Section 5: The task is refining symbolic plans into object poses for tabletop rearrangement, with on, in, left and south. The training data consists of a primary pool with 1,965 bin-packing episodes and a replay pool of 10,400 tabletop episodes with 2–9 states and 3–7 objects, built from geometric primitives, YCB objects, GraspNet scans and cubes (Sec. 5.1, App. B, Table 5) for data diversity. The main evaluation uses general tabletop plans with 9–20 states and 4–6 objects, i.e., longer than any plan seen in training (Table 1). Generalization was evaluated on scenes with 9–10 objects, on new scenes of YCB objects, and on 51 real point-cloud captures from a setup in our lab (Table 3, App. B.2), all without retraining. The foresight evaluation consists of the platform and bridge task families, after fine-tuning all learned models on 1,000 episodes of each family (Sec. 5.3, App. A.1 and F.1). We will expand this description in the main text of the paper (Sec. 5.1).

AuthorsOpenReview ↗

Additional Data Efficiency Results

We trained the model on random subsets of both data pools used for training the main model (about 360 to 6,100 episodes). Each variant was trained for the same number of training steps, and evaluated like Table 1 in our main paper.

MethodTP 9–12TP 13–16TP 17–20FS 9–12FS 13–16FS 17–20Real
Identity (no movement)0.6300.6100.5750.0050.0000.0050.331
Diffusion-CCSP0.8390.8380.8270.1000.1150.1050.760
StructDiffusion0.8170.8060.8040.0750.0970.0750.794
CDGS (own recipe)0.841--0.083---
Plan2Pose, 3%0.9390.9300.9250.2700.2900.2750.814
Plan2Pose, 10%0.9640.9600.9580.4800.4730.4730.876
Plan2Pose, 25%0.9760.9710.9670.5850.5850.5400.909
Plan2Pose, 50%0.9720.9690.9650.5600.5650.5550.908
Plan2Pose, 100%0.9810.9810.9790.6570.6780.6470.931

Real: per-atom satisfaction on the 51 real captures.

These experiments use a slightly modified variant of our model that uses more of the structure given by the symbolic plan. In the submitted model, the contribution of static and dynamic objects in the relation is purely determined by the symbolic domain. This modified variant instead learns to combine the probability maps dynamically. Our current training pipeline was set up for this variant. Due to the time constraints of the rebuttal, we used it for the data-efficiency experiments.

With 10% of the data, task progress drops by about 0.02 and the share of fully correct plans by 0.17–0.21. Even with 3%, the model exceeds all baselines trained on the full data in task progress and fully correct plans. More demonstrations therefore mainly prescient full-plan correctness, but not general placement correctness and transfer to real captures.

I never assumed that Plan2Pose replaces a TAMP planner. What I implied was the lack of comparison with a system that does backtracking, and potentially can run into an exponential runtime. And thank you for your new experiments regarding this concern. The table you provided shows success rates, but we don't know sufficient details about these tasks. Regarding the use of geometric information, I think the occlusion comes for free from the depth camera, without needing any privileged information. And therefore, a true TAMP system would use that to check for collision before stepping any further. Some do more informed stuff such as simulating what happens after the action, but that would require a simulator + a privileged version of the object, which I'm not asking for.

Regarding your results, you said the baseline matches Plan2Pose with geometric information, but the table reports some other number. Which one is it?

If the baseline matches, which I was expecting to since it would eventually find the correct parameterization by backtracking, the selling point would be comparing on how many samples your system finds the solution vs. the other, together with wall-clock time, which you have provided. I think this is the right metric to compare this with an uninformed myopic sampler. But I bet there are others that I don't know as well. Just the top of my head, you could do Bayesian updates to the sampler distribution as you try, and so on.

Not that I'm against transformers, but I just want to make sure that it is the transformer itself that makes this happen.

I'm updating my score to 'weak reject' for the effort of new experiments, but I don't think we have sufficient information about the experimentation procedure, and the conclusions to draw from that.

Thank you for taking the time to follow-up and for raising your score. We apologise for misreading your original concern. The follow-up questions are very helpful for improving the way we present results in the paper. We hope these further clarifications help.

Implementation of the TAMP baseline: We re-implemented the plan-refinement of Srivastava et al., 2014 (Alg. 1 and 2) with the task plan fixed for both the baseline and Plan2Pose. The sampler follows the released placement generator: shuffled uniform positions on the support surface, adapted to our tasks (e.g., objects resting on two supports for the bridge). Plan2Pose runs in the same search-with-backtracking loop and draws each candidate from its predicted placement distributions.In place of a motion planner, every candidate produced by every method is checked geometrically (relations, collision, support), and each check counts as one motion-planning call. The sampler can apply perception-level checks (collision, resting, etc.) from point clouds which do not count as checks. In further variants, the baseline filters candidates with hand-coded geometric definitions of the predicates and of support, or adapts its proposal distribution after failed checks.

Reported numbers: The table in our general response reports the comparison in which the baseline's geometric tests count against the budget of 2,000 checks (0.72 / 0.06 / 0.53). If the tests are free, it matches Plan2Pose (1.00 / 0.93 / 1.00) at the same budget, which is what our text referred to. Following your suggestions, we report the median number of checks and wall-clock time each method needs per problem. We did not capture all of these metrics, therefore we needed to re-run all the experiments leading to slightly different results. We also report median instead of average numbers here, to better handle outliers from unsolved problems.

Median checks / time per problem4-block6-blockBridge
Uniform (Srivastava et al.)980 / 0.84 s>cap>cap
+ perception checks129 / 0.30 s2,614 / 9.3 s6,426 / 6.6 s
+ adaptive proposals132 / 0.36 s1,824 / 7.1 s3,026 / 4.6 s
+ hand-coded tests23 / 0.24 s448 / 5.8 s106 / 3.2 s
Plan2Pose without future plan316 / 1.35 s>cap255 / 0.89 s
Plan2Pose12 / 0.27 s>cap3 / 0.14 s
Plan2Pose + perception checks4 / 0.25 s>cap3 / 0.14 s

Medians over all 100 problems, with unsolved problems counted as exceeding the limits. Reported times include all tests and model calls. ‘>cap’: the majority of problems are not solved within the limits (20,000 checks, or 60 s).

With the same perception checks, Plan2Pose needs 4 instead of 129 checks on the 4-block platform and 3 instead of 6,426 on the bridge. Without the future plan, they rise to 316 and 255. The uniform sampler's cost grows from 86 to 226 to 980 checks for 2, 3 and 4 blocks, Plan2Pose's from 5 to 8 to 12. With additional, task-specific hand-coded tests, the baseline matches Plan2Pose in time on the platform and solves more problems on tasks outside of our training distribution.

Transformer: We encode the whole plan and initial point cloud, and the placement is a product of the relevant relations per object. Without the future plan, the same model also keeps the plan serializable in 0.33 instead of 0.89 of the 4-block problems, and 0.46 instead of 0.98 on the bridge (Table 2). Conditioning on the future steps of the plan is therefore key to our results, which we implemented through the transformer’s cross-attention. We do not claim that transformers are inherently better than well-designed samplers, but that non-myopic placements can be learned from automatically annotated demonstrations, without predicate-specific samplers or tests at deployment, as shown by the additional measurements we now supplied. Our implementation is also relatively small (24M parameters, 12–19 h of training on one GPU) and comparable in size to our baseline LINGO-Space (27M trainable parameters).

Tasks: We want to give a better intuition about the tasks. In the bridge task, the plan first places two pillars next to each other and, in the final step, a short or a long lintel on them. A sampler that only looks at the current step places the pillars anywhere valid and finds out only at the last step that the lintel the planner has chosen might not fit, so it must backtrack to the pillars. The symbolic plan does not carry information on the lintel geometry. Plan2Pose sees the lintel in the plan from the start (due to its point cloud from the initial state being encoded) and spaces the pillars accordingly. The platform tasks work the same way: whether later blocks go to the left or the right of the first block is decided only by later steps, so a first block placed without looking ahead can leave no room for them to be properly placed

Reviewer_jZg9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The goal of this paper is to improve the speed and correctness of bi-level Task and Motion Planning methods by learning to predict, conditioned on a complete task skeleton, a feasible set of placement poses for the objects. It is an improvement on previous work that has learned separate models for individual constraints in the constraint graph. Its performance is documented to be a significant improvement over some previous approaches, but the domain of application is somewhat limited.

Critical strengths

The paper is clear and the goal is good: we do need better ways of generating sets of satisfying continuous parameters for the constraint sets that arise in TAMP.

The design of the transformer, taking the entire plan skeleton as input, is good.

Critical weaknesses

I am concerned that perhaps the approach is limited effectively to 2D problems (by treating them as a projection onto the plane). If that is correct, it's a substantial limitation.

In addition, to make this method practically useful, it would need to be able to take into account kinematic and grasp constraints.

Ultimately this paper seems competent, but not deep or highly novel. It's a sensible small step beyond previous work.

Questions for authors

Do you do any kind of shape completion on the point clouds? How do you handle needing to place one object in an area that is occluded by another? (If this is all top-down that is a serious limitation in practice). I was originally confused about the "planar pose and height" description at the top of section 4 but that makes me think this is all not really happening in 3d.

It's not clear to me that staying close to the demonstrations is an unalloyed virtue. What are the strengths and weaknesses of doing that?

Although you are using the language of probability distributions to talk about what you're learning, it's really a set, right? That is, the constraints are either satisfied or not. It's fine to model this way, but may be worth mentioning this.

4 comments

3D predictions: Plan2Pose is not limited to the plane. It predicts the height of each placement as a learned distribution, separately from the planar position because most predicates only constrain the plane; this is what "planar position and height" in Sec. 4 refers to, and we will rephrase it. Stacking, containment and the bridge task, where a lintel must rest on two pillars, are 3D placements: on the bridge, 0.978 of our first placements keep the plan completable, compared to at most 0.260 for the baselines (Table 2). On two and three-level stacks, the median height error is 2.6 and 4.3mm respectively. Outside of the training distribution (e.g., four and five level stacks), we see this error rise to 55mm. Similarly, the model outputs a full rotation. Since our demonstrations only contain vertical axis rotations, the evaluated models only learned yaw. We revised the rotation head and yield a median error of 1.3 deg, with not change to the Table 1 metrics.

Kinematic and grasp constraints: Plan2Pose replaces the sampler of a TAMP system, not the motion planner: it predicts object poses in given states, and grasps and motions are found and checked downstream. Because it stays close to demonstrations that a motion planner executed, its placements tend to be feasible: every placement is executable in 0.300 of 13–16-step plans, compared to 0.075 for the diffusion baselines (Table 13). We are currently revising the executor that was used to check the demonstrations, and preliminary results indicate a significant gain for Plan2Pose. As a sampler in a backtracking search loop, it outperforms Srivastava et al. (2014) (general response "Comparison with backtracking TAMP").

Novelty: Our contribution is the formulation of downward refinement as a distribution over poses conditioned on the whole plan, learned with a single loss and with predicates as opaque symbols. It makes long-horizon refinement robust: 0.54–0.59 of plans with 9–20 states end in a correct final state, compared to at most 0.12 for the baselines (Table 1).

Occlusion: We do not use shape completion; the model works with the visible points. On real captures from a single camera, which sees objects only partially, it transfers zero-shot and satisfies 0.929 of the atoms, compared to at most 0.847 for the baselines (Table 3). In simulation, dropping any one of three cameras has no measurable effect. Using a shape-completion model (e.g., GeoFormer) preserves the performance under partial occlusion. We also train with merged three-view point clouds and will describe this in App. B.

Learning from demonstrations: The strength is that errors compound less: our placements stay closer to the demonstrations, so along 17–20-step rollouts we keep 0.946 of the atoms, while the baselines drop to 0.76–0.79 (Table 11). The weakness is that the model stays within the demonstrated distribution; for example, its foresight does not transfer to the out-of-distribution six-block platform (Sec. 5.3).

Probabilities vs sets: We agree that the valid placements form a set. We learn a density because we only observe positive examples, and multiplying the per-atom densities corresponds to intersecting their sets. We will add this to Sec. 3.

Please take a moment to respond the authors. Thank you!

Thanks! I would disagree that in more realistic TAMP problems, there are all kinds of constraints that are not planar: things are in the oven, or glued together, or the door is latched, etc. But your lintel example is good. I guess you are limiting your domain rougly to "generalized rearrangement".

But if you don't use shape completion and you don't assume full top-down observability, then there are many problems you just can't solve. If you can do your test set with a single camera and only the point clouds, then you have a very simple test set.

In the end: I think this work makes a good improvement over a set of methods that is highly restrictive in their domain assumptions, and unfortunately retains many of the assumptions. However, the general architectural idea could be more broadly useful. But it is not, in itself a huge intellectual step to condition on the whole plan skeleton.

So, I guess, I'd move up from "clear reject" to "weak reject" but encourage the authors to reach farther.

Thank you for raising your score and for the constructive assessment. We agree that our shown settings are best described as generalized rearrangement within TAMP.

More realistic examples: Constraints such as a latched door enter as symbolic preconditions handled by the task planner. Plan2Pose refines the geometric relations of a plan, including containment in boxes or other objects. Given a motion planner/policy that can open doors, there's no technical reason why we wouldn't be able to predict new poses for objects inside and outside of drawers. Since this point was raised at the end of the discussion period, we cannot provide results on articulated objects in time, but we would be glad to add them in a revision. Learning to predict the actual geometric change of deformable objects/articulated objects is beyond the scope of non-generative models (we see the trade-off between speed and this specific set of capabilities), but an integration would be interesting to investigate.

View-dependency: with only a side or front camera, final-state correctness drops by at most about 0.05 in our experiments. Of course camera position matters, but we strictly do not rely on a top down view. Reasoning about viewpoints is a complementary problem. As in a standard TAMP system, if a placement proposed by Plan2Pose cannot be verified or realized because of occlusion, active perception or a change of viewpoint can supplement it, and Plan2Pose can then propose new candidates from the updated observation. Our approach is compatible with integrated TAMP approaches, as our backtracking-search experiments show for the search itself.

Tasks and test set: the scenes are simple to perceive by design, but the difficulty lies in the horizon. On single steps, the diffusion baselines are close to Plan2Pose (0.955 vs. 0.964 of the atoms), but over plans of 9-20 states, only Plan2Pose keeps most plans correct (0.54-0.59 vs. at most 0.12). We agree that conditioning on the whole plan is a simple idea on the high level. However, making this work reliably in a single model, especially without predicate-specific samplers or losses, is what the prior work in this space has not done. Its effect shows over long horizons, in the reduced search, and in the ability to use VLMs to annotate data.

Reviewer_w8MzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Strength:

  • The compositional placement architecture is promising.
  • The evaluation is broad and unusually transparent about limitations

Weakness:

  • Limited comparison: the paper does not compared to more recent SOTA methods such as CDGS [1]

[1] Mishra, Utkarsh, et al. "Compositional diffusion with guided search for long-horizon planning." International Conference on Learning Representations. Vol. 2026. 2026.

Critical strengths

See summary

Critical weaknesses

See summary

Questions for authors

See summary

2 comments
AuthorsOpenReview ↗

Comparison with CDGS

CDGS and Plan2Pose address different problems: CDGS plans a whole long-horizon task from its goal, whereas Plan2Pose refines a given symbolic plan into object poses, between a task planner and a motion planner. We therefore compare them in both settings: in the setting of CDGS, where we combine Plan2Pose with a PDDL planner, and in ours, where we give CDGS the plan.

CDGS is trained on the same data as Plan2Pose, with its own training recipe. The best validation checkpoint is reached after approximately 3.4 times as many training steps as Plan2Pose (including task-specific fine-tuning).

Setting of CDGS (goal only): In the setting of CDGS, both methods only receive the goal, and Plan2Pose realizes a plan that a PDDL planner computes for it. CDGS receives the goal relations through its goal inpainting mechanism. We evaluate four goal types and also report Plan2Pose fine-tuned for planner-generated plans, and CDGS without search, the variant from the paper without pruning and resampling. The following table shows the share of problems in which every goal relation holds in the final state:

GoalPlan2Pose + plannerPlan2Pose + planner, fine-tunedCDGSCDGS without search
Stacking only.968.984.075.016
Stacking + 2 placement relations.540.863.453.085
Stacking + 4 placement relations.425.734.395.048
Four-level towers.547.573.009.001

Plan2Pose outperforms CDGS on all four goal types. CDGS mostly fails by placing an object on the wrong support or not moving it at all, a choice that the task planner makes in our pipeline. Placing the objects takes Plan2Pose 0.024–0.032 s per problem, compared to 1.23–1.25 s for CDGS. The symbolic planner adds 0.19–0.22 s.

Our setting (plan given): We give CDGS information it does not receive in its original formulation; the plan structure (which object moved in each step and the relationships it establishes), exact poses and geometric information. We evaluate this variant on our main plan-length evaluation. Even then, on plans of length 9–12, CDGS satisfies 0.79 of the relations, compared to 0.97 for Plan2Pose, and realizes 0.08 of the plans fully correctly, compared to 0.56. Its diffusion-based sampling takes 1–4.8 s per placement, compared to 30 ms for Plan2Pose.

Relation to our baselines: CDGS belongs to the same family as Diffusion-CCSP, against which we compare in the paper: both compose local diffusion models and sample iteratively, Diffusion-CCSP over the constraints of a problem and CDGS over overlapping segments of a long-horizon plan, combined with guided search. Plan2Pose instead trains one model with a single loss and places each object in one forward pass, 18–30 times faster than the diffusion-based baselines in the paper (App. H). We will add CDGS to the related work and these results to the appendix.

Please take a moment to respond the authors. Thank you!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Comparison with backtracking TAMP

We want to thank the reviewers for their suggestions and the questions they raised regarding role of Plan2Pose relative to TAMP, how it holds up with recent SOTA methods like CDGS, and its ability to perform predictions beyond the plane. To address the concerns and questions, we ran several additional experiments, including integrating Plan2Pose into a backtracking TAMP framework. We provide detailed answers to the specific issues raised by each reviewer below their reviewers.

Like other TAMP planners, we assume a three-layered architecture: a high-level planner operates purely on symbolic states, a sampler predicts end poses, and a motion planner realizes them. Plan2Pose replaces the sampler with a transformer that takes the long-term context of the plan into account. We therefore provide additional results comparing Plan2Pose against the sampling-based approach of Srivastava et al. (2014) and evaluate only the sampling part; neither implementation includes a motion planner. Their backtracking search was reimplemented with the task plan fixed, and both approaches use the same geometric check: a solution must satisfy the demanded relations, be free of collisions, and rest stably on its supports. Plan2Pose was included as a learned sampler in the same backtracking framework. We use the 4-block platform, 6-block platform and bridge tasks, which require foresight, with 600 problems per task and the same budget of at most 2,000 checks for both approaches. We also evaluate a variant of the baseline whose sampler is given additional geometric information and constraints, to which Plan2Pose has no access.

The following table shows the success rates:

TaskPlan2PoseSrivastava et al.Srivastava et al. + geometric sampler
4-block platform0.990.730.72
6-block platform0.040.030.06
Bridge1.000.060.53

Given the same budget, Plan2Pose outperforms the standard version of the baseline on all three tasks. On the 4-block platform, Plan2Pose needs on average 185 samples per problem, the baseline 799 samples, on the bridge 50 and 1042. With the geometric sampler, the baseline can match Plan2Pose (excluding the additional tests from the budget yields 1.00 / 0.93 / 1.00). However, it does outperform our method on the 6-block platform task which was out of distribution of the evaluated model. In 19 failure cases we inspected, the baseline samples myopically: it commits to a first placement in the initial state that breaks downstream placements, and then spends its budget trying to refine the other parts of the solution. It resamples so often that it exceeds the budget, hence its lower success rate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.