acceptodds
Under review as a conference paper at ICLR 2027

Hybrid Reasoning through Compositional Structural Energy Minimization

Abstract

Many real-world reasoning problems require solving for high-level discrete structure with continuous values. For example, a robot solving a manipulation problem must determine both a high-level task structure and continuous actions; a model for scene understanding infers both the objects' pairwise relations and their poses from a partial camera view. Existing machine learning approaches face important limitations: end-to-end monolithic models often struggle to solve problems that are harder than those seen during training, while compositional methods typically require the correct problem structure to be specified in advance. We introduce Compositional Structural Energy Minimization (CSEM), a general framework that learns from small training instances and jointly solves for the discrete structure and continuous solutions of harder problems. We evaluate CSEM on a wide range of hybrid reasoning problems, including task-and-motion-planning, 3D scene understanding, and reconstructing physical trajectories and contact modes from sparse observations. We demonstrate that CSEM generalizes without additional training to larger structures and longer horizons, outperforming strong learning-based baselines.

Primary area:
generative models
Keywords:
hybrid reasoning, energy-based models, compositional generalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NKeSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CSEM combines learned local energies with structural edits and continuous refinement to solve larger planning, scene, and trajectory problems than those used for training.

Critical strengths
  1. Scale generalization is strong: five-bin packing reaches 52% versus 8% without edits, and length-100 trajectory validity reaches 72% versus 49%.

  2. Search controls using the same energies and matched evaluation budgets help isolate the method’s contribution.

Critical weaknesses
  1. The calibration claims need more evidence. BCE training and AUC do not establish calibrated probabilities on larger problems, and ranking differs from the stated energy objective.

  2. Evaluation counts do not directly show runtime cost. Trajectory validity also measures self-consistency rather than recovery of the true trajectory.

Questions for authors
  1. How reliable are the scores and stopping rule on larger problems? Please clarify how optimization and candidate ranking differ.

  2. Can you report comparisons at the same wall-clock budget and position errors against the true trajectories?

2 comments

Thank you for your constructive feedback and positive comments. Following your suggestion, we have added calibration and stopping diagnostics, clarified optimization versus ranking, and reported matched-time comparisons with ground-truth trajectory errors. We address these points below.

Question 1. Score calibration, stopping reliability, and candidate ranking

Thanks for raising this question. We have now checked the learned energies and stopping rule on larger problems and added the results in Appendix B.6, Tables 17–19.

We find that reliability depends on the domain. On five-bin packing, 29 of 30 stopped runs return valid solutions (97%). On rearrangement with 12 objects and three boxes, this is 18 of 30 (60%); on the harder 20-object, five-box setting, it is 10 of 44 stopped runs (23%) at 1M evaluations. For eight-object stacking, we find a valid plan during search in 26 of 30 instances, but return one in 7 of 30, showing that ranking the candidates remains difficult at larger sizes.

We also clarify the distinction between optimization and ranking: we use time-conditioned energy gradients to refine continuous variables under a fixed graph, then use the sum of softplus-transformed clean factor energies to compare candidates across graphs. We stop when every factor’s predicted violation probability for the best candidate falls below the threshold.

Question 2. Matched wall-clock budgets and true-trajectory position errors

Thanks for this suggestion. We have now compared the methods under the same wall-clock budgets and measured position errors against the ground-truth trajectories. We report these results in Appendix D.3, Table 23:

SettingTime budgetCSEM error (cm)ALiBi error (cm)
T = 201,000 s2.51.9
T = 501,500 s4.38.8
T = 1003,000 s6.616
T = 50, sparse observations1,500 s1223

We find that CSEM recovers positions more accurately at longer horizons and with sparse observations, while ALiBi has lower error at T = 20. For each trajectory, we average position errors over timesteps, then report the median across 100 fresh trajectories. We run all methods on four CPU cores and exclude model loading, warm-up, and grading from the timing.

Thank you for the author's reply. Most of my concerns have been well resolved. I will maintain my score.

Reviewer_c5pNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Compositional Structural Energy Minimization (CSEM) for problems that couple discrete structural decisions with continuous variables. The method learns reusable local energy factors from small sub problems using denoising score matching and calibrates their energy values to indicate whether constraints are satisfied. At inference, CSEM composes energy factors under a proposed structure, minimizes the resulting energy, and uses local energy values to guide structural revisions. After each revision, it applies local backtracking and continuous refinement to adapt the solution to the updated structure.

The main contribution is the integration of local energy based refinement, validity guided structural search, and local backtracking. Validation covers stacking, multi-bin packing, rearrangement, 3D scene inference, and physical trajectory reconstruction, including generalization to larger instances and sparser observations. This paper also provides ablations of structural editing, verification timing, and edit targeting, together with a matched-budget search comparison.

Critical strengths
  1. A useful approach to solving problems with both discrete and continuous decisions. The paper presents an interesting approach that allows the model to revise the problem structure when optimizing continuous variables alone is not sufficient. By combining local energy models with structural search, CSEM can adjust both the relationships between objects and their actual positions. I think this is a useful direction for solving more complex planning problems.

  2. Strong experimental results on larger problems. The paper shows promising results across several tasks, especially in multi-bin packing. For example, CSEM achieves a 52% success rate on the largest packing setting, compared to only 8% without structural edits. The ablation studies also provide evidence that structural revision and energy-guided search contribute to the performance improvements.

Critical weaknesses

1.The comparison with existing approaches could be more convincing. Some of the main ideas, such as combining local constraint models with planning and search, have already been explored in previous work. For example, the CCSP baseline uses a random structure or a simple heuristic, while CSEM can repeatedly revise its structure during inference. This makes it difficult to tell how much of the improvement comes from the proposed energy-guided search strategy, rather than from introducing structural search itself. A stronger comparison with existing planning methods that also support structural search would help clarify the contribution.

In addition, some of the tasks studied in this paper, such as object arrangement and scene understanding, could potentially be addressed by LLM or VLM agents combined with planning and optimization tools. However, the experiments mainly compare CSEM with specialized learning-based methods. It remains unclear what advantages CSEM offers over LLM/VLM-based approaches in terms of performance, generalization, or computational cost. A discussion or comparison with such approaches would help better show the value of this method.

2.The practical advantages and reliability of the learned energy models are not fully demonstrated.

Most experiments are conducted in simulated environments where object sizes, possible relationships, and constraint types are already defined. Many constraints, such as collision avoidance and containment, can also be checked directly using geometric methods. It is therefore unclear when learning energy models provides a clear advantage over traditional planning and optimization approaches. Although the results are promising, more evidence would help establish the practical value of the proposed framework.

Another concern is the reliability of the learned energy values on more complex problems. CSEM relies on these values to identify which constraints are not satisfied and guide structural revisions. However, it is unclear how reliable these predictions remain as the problem size increases. For example, in the harder rearrangement setting with 20 objects and 5 boxes, CSEM fails on all 50 test scenes, and the authors mention errors in the learned verification process. A more detailed analysis of these failures would help explain the limitations of the method and its ability to generalize to more complex problems.

Questions for authors

1.Comparison with existing structural search methods: Why does the CCSP baseline use a random structure or a simple heuristic instead of combining CCSP with a structural search procedure? How would this affect the comparison with CSEM?

2.Contribution of energy-guided search: How much of CSEM's improvement comes specifically from using energy values to guide structural revisions, rather than simply allowing the structure to be revised? Could the authors further clarify this based on the existing ablation results?

3.Comparison with LLM/VLM-based approaches: Could the authors discuss how CSEM compares with LLM/VLM agents that use external planning and optimization tools? In what scenarios would CSEM be expected to offer advantages over such approaches?

4.Failure analysis on larger problems: In the rearrangement experiment with 20 objects and 5 boxes, none of the 50 scenes are solved. What are the main reasons for these failures? Are they primarily caused by incorrect structural decisions, inaccurate energy predictions, or difficulties in continuous optimization?

5.Computational cost: Since CSEM performs multiple rounds of structural revisions and continuous refinement, how does its inference time compare with alternative planning and optimization methods? Would its performance advantages remain under similar computational budgets?

2 comments

Thank you for your constructive feedback and for recognizing the value of structural revision and the results on larger problems. We have strengthened the structural-search comparisons and targeting ablations, added failure and computational-cost analyses, and clarified the role of learned constraints and their connection to tool-using agents. We address each point below.

Question 1. Comparison with structural-search methods

Thanks for your suggestion! We have now augmented Diffusion-CCSP with structural search using CSEM’s learned verifiers to score candidates and determine stopping. The following results report solve rates (%) under a common 120k evaluation cap on 100 original test instances per setting: 3D stacking

Method4 objects5678
CCSP + search604223113
CSEM8571533730
3D multi-bin packing
Method2 bins345
-------------------------------------------
CCSP + search7749165
CSEM95989998

We found that CSEM achieves higher solve rates across all evaluated sizes. We have now included these comparisons in Section 4, Table 1, with implementation details and additional evaluation budgets in Appendix B.3, Table 9.

Question 2. Energy guidance versus simply allowing edits

To address your question, we now isolate energy-guided targeting by replacing it with uniform selection while retaining structural revisions, repair, and the 120k evaluation cap. On fresh sets, the current method gives:

SettingEnergy-guidedUniform targetsPaired p
Scene, 8 objects86%76%< 0.001
Scene, 10 objects79%67%< 0.001
Packing, 5 bins99%94%0.062
Rearrangement, (12,3)63%55%0.12

Question 3. LLM/VLM agents with planning tools

Thanks for this observation. A tool-using LLM/VLM agent could call CSEM for constraint solving while handling task interpretation and decomposition, as we now discuss in Appendix B.3. CSEM would supply a reusable solver for the coupled discrete and continuous constraints within those subtasks. We do not claim a performance or cost advantage over complete agent systems; a controlled comparison would require specifying their tools, feedback, and computational budgets.

Practical value of learned constraints (Weakness 2). The key benefit of using learned constraints is that they can capture constraints from data when accurate analytical formulas are difficult to specify analytically—for example, physical stability or consistency with partially observed scenes.

Question 4 . Failures on 20-object, five-box rearrangement

Our added diagnostics identify both continuous optimization difficulty and learned stopping errors. On the same 50 hard rearrangement instances, the current method solves 6/50 (12%) at 120k evaluations and 12/50 (24%) at 1M evaluations, compared with the submitted solver’s 0/50. Appendix B.6, Table 19 reports the breakdown:

Outcome (number of instances)120k evaluations1M evaluations
Solved612
Stopping rule accepted an invalid state2034
Valid candidate found but not returned02
No valid candidate found232
Therefore, one common failure mode is that the stopping rule accepted an invalid solution, which means that the learned energies can become less reliable as object size increases. An immediate solution could be to train on datasets with larger sizes and compose the models. Other methods could also be interesting for future studies.

Question 5. Comparable computational budgets

Thanks for your question. We now report common-evaluation comparisons, matched-time temporal comparisons, and packing compute scaling:

  • Appendix B.3, Table 9: Diffusion-CCSP with ULA and the structural-search wrapper at 30k, 60k, and 120k caps. At 120k, fresh five-bin packing success is 99% for CSEM versus 11% for CCSP; ULA’s 11k evaluations per sample limit it to ten initial samples at this cap.
  • Appendix B.5, Table 13: independent restarts using the same factors, minimizer, and 120k cap. CSEM reaches 98% versus 51% in five-bin packing and 78% versus 19% in ten-object scene inference.
  • Appendix D.3, Table 23: temporal inference at matched wall-clock budgets. At T = 100 and 3,000 s, CSEM achieves 47% transition validity versus 7.4% for the best baseline, with median position error 6.6 cm versus 16 cm for ALiBi.

Figure 4 now shows the gains from additional computation: with P = 16, CSEM improves from 80% at 100 s to 96% at 300 s and 98% at 600 s. These compute-scaling results complement the evaluation-budget controls.

I thank the authors for their detailed rebuttal. Their responses have adequately addressed most of my concerns. Overall, I remain positive about this work and will maintain my original rating.

Reviewer_KKZYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose CSEM, a method designed to solve hybrid reasoning problems with discrete-continuous states. The key idea is to exploit the compositional power of EBMs where the composition of learned local energy factors is inferred at inference time, allowing for the optimization of both discrete and continuous variables without assuming a fixed discrete structure. Local energy factors are learned via denoising score matching plus a calibration term at the clean timestep intended to make the energy scores of different discrete graphs comparable for verification. These learned factors are composed at test time and used for verification, such that likely invalid configurations are identified and addressed through structural edits and continuous refinement, until a criterion is met or a max budget is exhausted. Results assess the performance of CSEM in three different domains, each including its unique set of baselines, demonstrating how CSEM generalizes to harder configurations of the training tasks by exploiting test-time compute.

Critical strengths

Overall, this method shows promise for solving problems that require joint reasoning over discrete structures and continuous variables. However, there are specific items that require further clarification in the method and evaluation sections. Once those are addressed, I would be willing to increase my score.

Critical weaknesses

The method section can be improved with by clarifying the following:

  1. After reading Section 3.3, it is hard to understand how exactly the graph edits work or are applied. The authors wrote "...while the domain provides its syntax-valid alternatives." and provided valid edits for each domain in Table 5. However, there needs to be further discussion of this particular step in the main manuscript as this is one of the main ingredients of the method. For instance, once a high-energy factor is sampled, how is a candidate edit proposed? Is the edit sampled uniformly at random from the list of available edits? How do you determine if a variable outside your local set (S_i) should be considered in the new edit (as shown in Fig. 1, where the revised graph (\hat{G}) connects node 3 to node 4, which was not originally contained in the local set (S_i) between nodes 3 and 5)? Clarifying this aspect of your method will improve its clarity and help readers better understand it.

  2. From reading the method section, it is not clear when a sample is being denoised or when it is being improved via gradient-based optimization as in equation 2. Please consider clarifying this in the main paper instead of leaving most of the discussion in A.4. Moreover, how was the S_r = 3S steps heuristic determined? Consider adding an ablation study on the effect of the number of clean steps S.

  3. Please provide more preliminaries or an extended presentation of denoising score matching, particularly equation (6). Why the noise-scaled gradient of the energy function works as the noise predictor in DDPM may not be evident.

  4. As optimizing the loss function in equation (6) involves mixed second order derivatives during training. Please provide more details about the scalability of your approach, particularly when training factors involving more variables (larger |S_i|), including the associated time and memory costs. Does increasing the number of variables within a factor make training less stable or more difficult? If the method relies on constraints being decomposable into small factors for training purposes, please consider listing this as an assumption of your method.

  5. Is the calibration term in equation (7) applied at every training step? Please clarify this point.

  6. Lines 264–266 state that local backtracking helps variables escape basins associated with previous graphs. However, it would be nice to have an ablation quantifying the effect of re-noising variables after an edit versus not re-noising them in terms of solve rate.

In terms of evaluation, I have the following questions/suggestions:

  1. Diffusion-CCSP (Yang et al., 2023, Appendix E) describes integration with a task planner that searches over discrete plans, whereas your ccsp-rand baseline uses random or first-fit structures. Please include a comparison with Diffusion-CCSP coupled to task-level search, or justify its omission, to better establish the benefits of CSEM’s energy-guided structural search.

  2. Figure 4 illustrates the benefits of more compute at test time but there is no discussion in terms of the wall-clock time needed as particles/edit budget increases. To better understand this accuracy/speed tradeoff, consider adding a version of Figure 4 that plots solve rate against end-to-end inference time.

  3. In the visual evaluation of section 5, are all objects in the scene provided a priori, including fully occluded ones, with the system inferring their poses and relations from the image? Appendix C.1 appears to imply this through its no-observation embedding. If so, please state this assumption clearly in the main manuscript.

  4. It is not clear to me what the ablation of CSEM w/o edits is measuring. Is it just refining the continuous variables of a fixed (initial) graph structure, with the comparison to full CSEM intended to quantify the effect of local edits? If this is the case, why does it outperform several baselines that also do not perform structural edits? Please clarify these details in the main text.

  5. How does the approach compare to no training alternatives such as cuTAMP that leverages parallelism to speed up the search during inference (cuTAMP: Differentiable GPU-Parallelized Task and Motion Planning, RSS2025)

Questions for authors

These were listed above.

2 comments

Thank you for your constructive feedback and detailed suggestions! Following your suggestions, we have added more explanations in our method section and ran more experiments. We present our updates here:

Method 1. Graph editing details

Thanks for pointing out that we need further clarification. We have now added these clarifications in Section 3.3 of the updated draft. Once an edit target is sampled based on its incident factor energies, we enumerate all its valid alternatives. These edits are valid as long as they satisfy the domain-specific graph syntax, and replacement variables need not belong to the original local set (SiS_i). For example, for stacking, a support graph must be acyclic. As illustrated in our Figure 1, block 4 can replace block 5 as block 3’s supporter if the resulting support graph remains acyclic. For bin packing, valid edits assign the chosen block to different bins.

Method 2. Denoising versus optimization, and the 3S heuristic

Since we train time-conditioned EBMs, denoising and energy-based refinement are closely connected. It would be helpful to check out our explanation in M3. We have also added the clarification in Sections 3.2–3.3.

In the submitted minimizer, we used (Sr=3SS_r=3S) as a heuristic for additional clean refinement. The current minimizer uses stochastic DDIM, so it no longer uses (Sr=3SS_r=3S).

Method 3. Why can energy gradients predict noise?

Thanks for raising this concern and it is related to M2. In response to your suggestion, we have added the preliminary in Section 3.2. For your convenience, we add the explanation here as well:

Given a training dataset (X,Y)(X,Y), we want to train an EBM such that e−Eθ(x,y)∝pdata(y∣x)e^{-E_\theta(\mathbf{x},\mathbf{y})} \propto p_{\mathrm{data}}(\mathbf{y}\mid\mathbf{x}). For the ease of training and sampling, rather than learning a single energy landscape Eθ(x,y)E_\theta(\mathbf{x},\mathbf{y}), we learn a family of time-conditioned energy landscapes Eθ(x,y,t)E_\theta(\mathbf{x},\mathbf{y},t), each representing the distribution of corrupted samples y~=aty++stϵ\tilde{\mathbf y}=a_t\mathbf y^+ +s_t\boldsymbol\epsilon, where at=αˉta_t=\sqrt{\bar\alpha_t}, st=1−αˉts_t=\sqrt{1-\bar\alpha_t} and ϵ∼N(0,I)\boldsymbol\epsilon\sim\mathcal N(\mathbf 0,\mathbf I).

Let pt(y~∣x)p_t(\tilde{\mathbf y}\mid\mathbf x) denote the density of y~\tilde{\mathbf y}, then we want to learn e−Eθ(x,y~,t)/st∝pt(y~∣x)e^{-E_\theta(\mathbf x,\tilde{\mathbf y},t)/s_t} \propto p_t(\tilde{\mathbf y}\mid\mathbf x) for t>0t>0. For Gaussian corruption, the noise prediction and score are related by −st∇y~log⁡pt(y~∣x)=E[ϵ∣y~,x,t]-s_t\nabla_{\tilde{\mathbf y}}\log p_t(\tilde{\mathbf y}\mid\mathbf x)=\mathbb E[\boldsymbol\epsilon\mid\tilde{\mathbf y},\mathbf x,t] (Vincent, 2011). Therefore, we can associate the time-conditioned energy gradient with diffusion scores ∇y~Eθ(x,y~,t)=E[ϵ∣y~,x,t]\nabla_{\tilde{\mathbf y}}E_\theta(\mathbf x,\tilde{\mathbf y},t)=\mathbb E[\boldsymbol\epsilon\mid\tilde{\mathbf y},\mathbf x,t] and then train energy gradients with the loss

\mathbb E_{\substack{(\mathbf x,\mathbf y^+)\sim p_{\mathrm{data}},\\\\ t>0,\\;\boldsymbol\epsilon\sim\mathcal N(\mathbf 0,\mathbf I)}}\left[\left\\|\nabla_{\tilde{\mathbf y}}E_\theta(\mathbf x,\tilde{\mathbf y},t)-\boldsymbol\epsilon\right\\|_2^2\right].

Method 4. Mixed derivatives, factor size, training cost, and stability

Thanks for raising this question. We keep the same 2.7M-parameter architecture and batch size of 256, and measure complete training updates, including mixed derivatives, on one A100:

Objects per factorUpdate time (ms)Peak GPU memory (GB)Validation denoising MSE
2270.18—
4270.290.20
8270.520.22
16331.00.26
32652.0—

The study uses synthetic packing constraints with fixed bin fill. Increasing factor size from ∣Si∣=4|S_i| = 4 to 32 objects increases update time by 2.4×. At 4, 8, and 16 objects, all three-seed training runs complete 30k updates without nonfinite losses or gradients, with mean verifier AUC above 0.999.

Method 5. Is clean-energy supervision applied at every update?

Yes, the clean-energy supervision is applied at every training update alongside the denoising loss. We have now stated this explicitly in Section 3.2, including the combined loss and its weighting.

Method 6. Does re-noising help?

Following your suggestion, we added an ablation that does not re-noise from post-edit repair while keeping the same number of optimization steps per repair. Re-noising improves solve rate from 84% to 99% on five-bin packing and 52% to 63% on rearrangement ((12,3)), and hybrid accuracy from 63% to 79% on ten-object scene inference. These comparisons use 100 fresh instances per setting and the same 120k evaluation cap. We have now included these results in main-text Tables 2, with the full noise-level ablation in Appendix B.5, Table 16.

Evaluation 1. Diffusion-CCSP with structural search

Thanks for your suggestion. We have now augmented Diffusion-CCSP with structural search using CSEM’s learned verifiers to score candidates and determine stopping. The following results report solve rates (%) under a common 120k evaluation cap on 100 original test instances per setting:

3D stacking

Method4 objects5678
CCSP + search (ULA)604223113
CSEM8571533730

3D multi-bin packing

Method2 bins345
CCSP + search (ULA)7749165
CSEM95989998

CSEM achieves higher solve rates across all evaluated sizes. We have now included these comparisons in Section 4, Table 1, with implementation details and additional evaluation budgets in Appendix B.3, Table 9. Under this cap, the ULA baseline exhausts its budget during initialization, before structural expansion.

Evaluation 2. Accuracy versus wall-clock time

Following your suggestion, we have updated Figure 4 to plot solve rate against wall-clock search time, including initialization and refinement. Using four CPU cores and 16 initial candidates, CSEM reaches 80% success at 100 s and 98% at 600 s on 100 fresh five-bin packing instances. We also evaluated the same search on an A100 GPU: median search time decreases from 36 s on CPU to 14 s on A100, and to 3.4 s with CUDA graphs, reaching 99% success by 100 s in the latter configuration.

Evaluation 3. Are fully occluded objects provided?

Yes, all objects and their dimensions are provided a priori, including fully occluded objects. Here we assume we know the geometry of all the objects, while we need to infer their poses and spatial relations. Prior work such as 3DP3 (Gothoskar et al., 2021) also uses known object-shape models to infer scene structure and object poses. Extending our method to unknown object geometries is something we hope to address in future work. We have now stated our assumption explicitly in Section 5.1.

Evaluation 4. What does the no-edit control measure?

Thanks for your questions. The control experiments CSEM w/o edits refine 16 randomly chosen initial graphs, minimize each of them, and return the lowest-scoring candidate according to our learned energies. This is why it outperforms many baselines. It retains our learned factors, continuous refinement, and energy-based selection; we have now clarified this in Section 4.

Evaluation 5. Comparison with training-free cuTAMP

Thank you for raising this question. We have now evaluated both methods on the same A100 GPU using 100 fresh five-bin packing instances. With checker-exact acceptance, cuTAMP achieves 100% success with a median runtime of 0.057 s. CSEM achieves 99% with a median runtime of 3.4 s using CUDA graphs, which accelerate execution without changing our search algorithm. Both timings exclude model loading, warm-up, and grading.

Notice that cuTAMP uses analytical verifiers with exact geometry, whereas CSEM uses learned verifiers. Although our method CSEM is slower, the key benefit of using learned verifiers is that they can capture constraints from data when accurate analytical formulas are difficult to specify analytically, while cuTAMP relies on known geometry and hand-coded differentiable constraints. Moreover, our central theme is to develop a framework that supports hybrid reasoning beyond task-and-motion planning, and we explore this broader applicability through scene inference and temporal physical inference.

Comments on the paper

2 comments

We thank R1 (KKZY), R2 (NKeS), and R3 (c5pN) for their thorough and constructive feedback. Their comments helped us clarify the presentation and strengthen the evaluation. We summarize the main updates and provide the updated main-paper results below; the individual responses address each reviewer’s questions.

Method updates. We retain CSEM’s overall framework and have added compositional fine-tuning, improved the continuous minimizer, and modified the graph search algorithm to improve success rates and inference efficiency (Sections 3.2–3.3; Appendices A.3–A.5).

Experimental updates. We have updated the results across all five domains and added controlled ablations of structural revision, energy-guided targeting, and stochastic repair (Tables 1–5). Table 1 now also includes Diffusion-CCSP with ULA sampling and structural search. Compute scaling, training measurements, and reliability analyses appear in Appendices A.3, B.3–B.6, and D.3.

Compared with the September submission, our method updates substantially improve performance in most domains, with same or fewer computation budget --- five-bin packing success increases from 52% to 98%, rearrangement success at (12 objects, 3 boxes) from 25% to 60%, and ten-object scene inference accuracy from 32% to 78%. The tables below include the submitted CSEM results alongside the current results.

In our updated draft, revisions to the method, experimental results, and explanations added in response to reviewers are highlighted in blue.

Updated results

Each current setting contains 100 test instances. Planning tables report solve rate (%); scene tables report hybrid accuracy (%). Original and fresh test sets are identified separately. “CSEM (September submission)” reproduces the reported values from submitted Tables 1, 3, and 4; “CSEM (current)” gives the updated results. These compare the reported versions, whose inference configurations and budgets differ, rather than a matched-budget ablation.

1. Task-and-motion planning (Table 1)

Models train on small instances and generalize zero-shot to larger problems. Results use the original test sets. HD, CHD, and HDP denote Hierarchical Diffuser, Coupled Hierarchical Diffuser, and Hybrid Diffusion Planner; ccsp-rand uses a fixed random/first-fit structure, and CDGS denotes Compositional Diffusion with Guided Search. CCSP + search uses CSEM’s learned verifiers; the ULA baseline and CSEM use a 120k evaluation cap.

3D stacking. Training uses at most four blocks; success is checked through simulated build-up.

Method4 blocks5678
Diffuser7133957
HD61171371
CHD47161254
HDP6638382714
CDGS453120109
ccsp-rand311815117
CCSP + search (ULA)604223113
CSEM (Sept submission)7668515349
CSEM (current)8571533730
CSEM w/o edits6453433030

Labeled-box rearrangement. Training uses four objects and two boxes; success requires valid move-by-move execution. Columns are (objects, boxes).

Method(4,2)(6,2)(8,2)(10,2)(6,3)(12,3)
Diffuser2311050
HD2310020
CHD2000000
HDP2100000
CDGS29210150
CSEM (September submission)656642395825
CSEM (current)929477698360
CSEM w/o edits929269608343

Multi-bin packing. Training uses two bins; five-bin tests contain up to 19 objects. Success requires all objects to fit without overlap.

Method2 bins345
Diffuser1000
HD1000
CHD1000
HDP33000
CDGS5100
ccsp-rand13210
CCSP + search (ULA)7749165
CSEM (September submission)92776352
CSEM (current)95989998
CSEM w/o edits733494

2. 3D scene inference (Table 3)

All methods receive segmented RGB-D observations, camera geometry, and the object inventory and dimensions, including fully hidden objects. Hybrid accuracy requires the correct support parent and pose within 2 cm per axis and 10 degrees, accounting for symmetry. Models train on 2–4 objects; results below use the original test sets.

Increasing object count.

Method4 objects6810
Feed-forward transformer61564440
Joint diffusion54301811
CSEM (September submission)77684532
CSEM (current)94928478
CSEM w/o edits42241511

Occlusion settings.

Method4 objects68
Feed-forward transformer302218
Joint diffusion381915
CSEM (September submission)574835
CSEM (current)847363
CSEM w/o edits291613

3. Physical mode inference (Table 5)

Models are trained at horizon T = 20. Transition validity checks local dynamics and collisions; mode accuracy checks the physical mode and contact surface. Both metrics are averaged over trajectories on the original test sets. The submitted temporal results used 100/40/20/40 trajectories at T = 20/50/100/50-sparse, whereas the current results use 100 per setting and a 120k evaluation cap. The submission used larger inference budgets; its rows are included as historical results, not a comparison at equal compute.

Transition validity (%).

MethodT = 20T = 50T = 100T = 50, sparse
Joint diffusion33203
Joint diffusion + ALiBi3911113
AR bridge4016818
CSEM (September submission)83747279
CSEM (current)77584159
CSEM w/o edits54312542

Mode accuracy (%).

MethodT = 20T = 50T = 100T = 50, sparse
Joint diffusion92573253
Joint diffusion + ALiBi92613054
AR bridge91583248
CSEM (September submission)85673657
CSEM (current)78573254
CSEM w/o edits81593357

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.