acceptodds
Under review as a conference paper at ICLR 2027

Goal-Conditioned Multi-Object Manipulation via Compositional Actions

Abstract

Goal-conditioned reinforcement learning (GCRL) for multi-object manipulation from offline images promises robust, scalable, and generalizable control, but existing generative subgoal methods struggle in test and out-of-distribution settings where combinations of object state-to-goal trajectories lie far from the training distribution. We present COMPositional ACTions (COMPACT), a compositional GCRL framework that learns generalizable manipulation policies end-to-end. COMPACT combines an object-centric encoder that discovers objects via feature connectivity and factorizes them into appearance and pose representations, with a compositional policy that aligns state and goal objects by appearance and selects the least displaced object for action prediction. The selected state-goal object pair, together with a learned pose difference embedding, conditions a goal-conditioned policy trained with implicit Q-learning. In multi-object manipulation environments based on OGBench and Push-T, COMPACT improves task success and sample efficiency over state-of-the-art generative subgoal and flat baselines. Importantly, COMPACT is the only method that achieves non-trivial zero-shot performance in settings with an increasing number of objects. Our results demonstrate that object discovery and representation, appearance-based alignment, and compositional action selection provide an effective foundation for data-efficient, generalizable multi-object manipulation from offline visual data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
object-centric learning, goal-conditioned multi-object manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZWqoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

COMPACT learns goal-conditioned manipulation policies from offline images. It represents each object by appearance and pose, matches objects between the current and goal images, and selects one object at a time for action prediction. The policy is trained using implicit Q-learning. Experiments on four simulated environments show improvements over the tested baselines, including when the number of objects increases at test time.

Critical strengths
  1. The approach is simple and addresses a useful problem: reusing object-level manipulation skills across different scene configurations. Matching state and goal objects before predicting actions provides a clear structure for the policy.

  2. The generalization results are promising. When PPP-Cube increases from three to six objects, COMPACT retains 64.86% success compared with 9.24% for HECRL. The experiments cover four environments and report results across four seeds. The ablations also show that both object representations and the compositional policy contribute to performance.

Critical weaknesses
  1. Theorem 3.1 does not appear to hold as stated. It claims that any object-selection policy has a multi-object value gap bounded by Kε, provided the single-object policy has error at most ε. However, even perfect single-object policies can give different discounted returns depending on which object is handled first. In Appendix A, Equations 16 and 17 use the same scheduling coefficients for the compositional policy and the optimal policy without justification. This removes the cost of choosing a poor object order from the bound. The theorem therefore does not establish the claimed guarantee for the proposed selection rule.

  2. Important details of the claimed end-to-end training are missing. Object discovery uses hard pixel grouping, matching uses argmax, and object selection uses argmin. The paper does not explain how gradients pass through these decisions or how the selection threshold is learned. It also introduces environment-specific ordering weights for Stack-Cube and Scene without describing their implementation. These details matter because joint representation learning and object selection are central contributions, and the current ablations do not separate their effects.

Questions for authors
  1. Why can Equations 16 and 17 use the same scheduling coefficients when the two policies may choose different object orders? Can the theorem be corrected to account for scheduling error?

  2. How are the hard grouping, matching, and selection operations handled during training? Which parts receive gradients, and how are the threshold and environment-specific ordering weights learned?

  3. How does the policy avoid repeatedly selecting an object that has already reached its goal? Algorithm 1 always selects the smallest pose distance but does not show a completed-object exclusion rule.

  4. Can you compare the proposed selection rule with random selection and a learned selector using the same encoder? A frozen-encoder comparison would also help separate the benefit of joint training from the policy design.

  5. Which success definition is used in the reported results? Appendix E defines success by a pixel-coverage threshold, while the main text and some environment descriptions use object-position thresholds. Please clarify the thresholds and whether evaluation uses 100 or 200 episodes for Stack-Cube.

5 comments

Thank you for your thorough and constructive review. We provide responses to the questions below:

Q1: Why can Equations 16 and 17 use the same scheduling coefficients when the two policies may choose different object orders? Can the theorem be corrected to account for scheduling error?

Yes, the theorem can be revised accordingly. We agree that Equations 16 and 17 can share scheduling coefficients only under a common fixed schedule. We are finalizing and ensuring correctness of the revised theorem to explicitly account for the object-selection gap, preserving its significant theoretical contribution of distinguishing compositional control error from scheduling loss, and will share the full updated theorem and proof within this discussion period.

Q2: How are the hard grouping, matching, and selection operations handled during training?

Object discovery uses soft grouping based on shortest-path distances, allowing gradients to flow through the object representations and selected-path edge costs, while matching and selection operations are not directly differentiated. The learnable threshold and ranking network receive gradients through straight-through surrogates, as detailed in our response to Q2a below. We will clarify the details of the soft-grouping formulation and its gradient flow in Section 3.1 of the revised paper.

Q2a: How are the threshold and environment-specific ordering weights learned?

The threshold and environment-specific ordering weights both are learned from the policy loss using a straight-through estimator, with hard selection in the forward pass and soft thresholding and ranking surrogates in the backward pass. We include the full details of this training mechanism and its gradient flow in the revised paper.

Q3: How does the policy avoid repeatedly selecting an object that has already reached its goal? Algorithm 1 always selects the smallest pose distance but does not show a completed-object exclusion rule.

The policy excludes object pairs whose pose distance does not exceed the learned completion threshold described in Section 3.2, and selects the closest remaining unfinished object. As clarified in Q2a, this threshold is learned from the policy loss using a straight-through estimator. We will clarify the completed-object exclusion rule in Line 13 of Algorithm 1.

Q4: Can you compare the proposed selection rule with random selection and a learned selector using the same encoder? A frozen-encoder comparison would also help separate the benefit of joint training from the policy design.

We run the suggested additional experiments on Push-Tetris, comparing random selection, a learned selector (via straight-through estimator), and the proposed COMPACT selection rule using the same encoder architecture and training setup, and present the results in the table below.

Selection ruleCoverage ↑Success ↑Chamfer ↓
Random selection44.74 ± 1.5313.84 ± 3.670.57 ± 0.12
Learned selector50.47 ± 1.6527.64 ± 4.570.27 ± 0.18
COMPACT62.12 ± 1.6955.87 ± 4.250.10 ± 0.02

We confirm that COMPACT significantly outperforms both random selection and the learned selector, demonstrating the benefit of the proposed selection rule under the same encoder and compositional policy setup. Qualitative analysis shows that random selection significantly increases the number of steps required to reach the goal due to frequent object switching, while the learned selector suffers from many more object collisions and thus reduces the optimality of the manipulation policy. In contrast, COMPACT reaches the goal in the smoothest manner. We will include the full ablation results across all datasets and add qualitative visualizations in the final version of the paper.

Q5: Which success definition is used in the reported results? Appendix E defines success by a pixel-coverage threshold, while the main text and some environment descriptions use object-position thresholds. Please clarify the thresholds and whether evaluation uses 100 or 200 episodes for Stack-Cube.

Success is defined using a pixel-coverage threshold for Push-Tetris; all other environments use object-position thresholds. Stack-Cube is evaluated over 200 episodes, while the remaining environments use 100 episodes. We will clarify these environment-specific success criteria and evaluation episode counts in the main text and relevant appendix sections.

W1: Theorem 3.1 does not appear to hold as stated. It claims that any object-selection policy has a multi-object value gap bounded by Kε, provided the single-object policy has error at most ε. However, even perfect single-object policies can give different discounted returns depending on which object is handled first. In Appendix A, Equations 16 and 17 use the same scheduling coefficients for the compositional policy and the optimal policy without justification. This removes the cost of choosing a poor object order from the bound. The theorem therefore does not establish the claimed guarantee for the proposed selection rule.

We have now revised Theorem 3.1 and its proof in Appendix A to replace the original KεK\varepsilon bound with

\leq \Delta_{\mathrm{sel}} +\frac{1-\gamma^{H+1}}{1-\gamma}\varepsilon_{\mathrm{ctrl}}.$$ Here, $\Delta_{\mathrm{sel}}=V^\star-V_\sigma^\star$ captures the loss from fixing the object-selection rule $\sigma$, while $\varepsilon_{\mathrm{ctrl}}$ uniformly bounds the per-step control Bellman error under that rule. The revised proof uses the finite-horizon version of the performance-difference identity [1], removing the assumption of shared scheduling coefficients. This conditional bound reinforces the theoretical rationale for COMPACT’s two complementary design choices: an explicit object-selection mechanism to address selection loss, and a shared single-object action head to address control error. **W2: Important details of the claimed end-to-end training are missing. Object discovery uses hard pixel grouping, matching uses argmax, and object selection uses argmin. The paper does not explain how gradients pass through these decisions or how the selection threshold is learned. It also introduces environment-specific ordering weights for Stack-Cube and Scene without describing their implementation. These details matter because joint representation learning and object selection are central contributions, and the current ablations do not separate their effects.** Building on our response to Q2a above, the completion threshold and ordering weights are learned jointly from the policy loss using straight-through estimators, with hard gating and selection in the forward pass and soft surrogates in the backward pass. Specifically, a ranking network scores each state–goal pair using its pair features and a pooled summary of all matched pairs, enabling configuration-dependent ordering. The ranking mechanism and completion-threshold rule are now specified in Algorithm 1 of the revised paper, with straight-through training described in the accompanying text. **Q4 (follow-up): Can you compare the proposed selection rule with random selection and a learned selector using the same encoder?** Following the Push-Tetris comparison above, we further evaluate selection strategies on Stack-Cube, an order-dependent task, using the same encoder architecture and training setup, and present the results below. | Selection rule | Coverage ↑ | Success ↑ | Chamfer ↓ | |---|---|---|---| | Random selection | 2.19 ± 0.22 | 0.00 ± 0.00 | 2.67 ± 0.39 | | Learned selector | 11.90 ± 2.88 | 24.11 ± 1.26 | 1.71 ± 0.50 | | COMPACT | **17.57 ± 2.89** | **51.56 ± 5.64** | **1.14 ± 0.43** | COMPACT outperforms on Stack-Cube, supporting the benefit of its structured selection rule on an order-dependent task. We will include the full ablation results on all environments in the final paper. [1] Kakade, Sham, and John Langford. "Approximately optimal approximate reinforcement learning." ICML 2002.

Dear reviewer, as the time for the end of discussion is approaching, may we ask if you have managed to review our responses, and whether you have any remaining concerns? Thank you.

Reviewer_UWJZOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents COMPACT, a method for learning goal-conditioned multi-object manipulation from offline image data. The method identifies and matches objects between the current and goal scenes, selects one object, and uses a shared policy to predict an action. The paper evaluates the method on simulated tasks, including pushing and stacking, and tests its ability to handle more objects than seen during training.

Critical strengths
  1. The approach is intuitive. Decomposing multi-object tasks into object-by-object manipulation provides a clear design for handling varying numbers of objects.
  2. The reported generalization results are promising. After training on three objects, COMPACT achieves a success rate of 64.86% on six-object PPP-Cube, compared with the reported 9.24% for HECRL. The experiments also cover several types of manipulation tasks.
Critical weaknesses
  1. The ablations do not isolate the contributions of individual components. The “without compositional policy” variant in Table 4 removes both appearance matching and pose-based ranking, making it difficult to determine how much each contributes. Evaluating these components separately would provide stronger support for the design choices.
  2. The reporting of experimental settings is inconsistent. Table 1 lists 2.5M and 1.5M “Transitions” for PPP-Cube and Scene, respectively, whereas Appendix D.1 gives dataset sizes of 3M and 1M. The former figures correspond to the numbers of gradient updates reported in the appendix. In addition, the main text states that evaluation uses 100 episodes, while the appendix specifies 200 for Stack-Cube. These inconsistencies make the experiments harder to interpret and reproduce and should be corrected.
Questions for authors
  1. The method prioritizes the object closest to its goal. Once an object reaches its goal, how does the method avoid repeatedly selecting it? Is the learnable threshold mentioned in the main text used for this purpose?
  2. In the generalization experiments with increasing numbers of objects, is the maximum evaluation episode length kept fixed or increased with the object count? Do all methods use the same settings?
5 comments

Thank you for your detailed and constructive review. We provide responses to the questions below:

Q1: The method prioritizes the object closest to its goal. Once an object reaches its goal, how does the method avoid repeatedly selecting it? Is the learnable threshold mentioned in the main text used for this purpose?

Yes, the policy uses the learned completion threshold described in Section 3.2 to exclude object pairs whose pose distance does not exceed the threshold, unless all pairs are considered complete. This threshold is learned from the policy loss using a straight-through estimator. We will clarify the completed-object exclusion rule in the revised Algorithm 1.

Q2: In the generalization experiments with increasing numbers of objects, is the maximum evaluation episode length kept fixed or increased with the object count? Do all methods use the same settings?

The maximum evaluation episode length is increased with the object count, up to 2000 steps for all methods and all environments. We will clarify this detail in the revised paper.

W1: The ablations do not isolate the contributions of individual components. The “without compositional policy” variant in Table 4 removes both appearance matching and pose-based ranking, making it difficult to determine how much each contributes. Evaluating these components separately would provide stronger support for the design choices.

We carry out additional ablation experiments to separate the contribution of the selection policy design choice. We run the suggested additional experiments on Push-Tetris, comparing random selection, a learned selector, and the proposed COMPACT selection rule using the same encoder architecture and training setup, and present the results in the table below.

Selection ruleCoverage ↑Success ↑Chamfer ↓
Random selection44.74 ± 1.5313.84 ± 3.670.57 ± 0.12
Learned selector50.47 ± 1.6527.64 ± 4.570.27 ± 0.18
COMPACT62.12 ± 1.6955.87 ± 4.250.10 ± 0.02

We see that COMPACT significantly outperforms both random selection and the learned selector, demonstrating the benefit of the proposed selection rule under the same encoder setup. Qualitative analysis shows that random selection significantly increases the number of steps required to reach the goal due to frequent object switching, while the learned selector suffers from many more object collisions and thus reduces the optimality of the manipulation policy. In contrast, COMPACT reaches the goal in the smoothest manner. We will include the full ablation results across all datasets and add qualitative visualizations in the final version of the paper.

W2: The reporting of experimental settings is inconsistent. Table 1 lists 2.5M and 1.5M “Transitions” for PPP-Cube and Scene, respectively, whereas Appendix D.1 gives dataset sizes of 3M and 1M. The former figures correspond to the numbers of gradient updates reported in the appendix. In addition, the main text states that evaluation uses 100 episodes, while the appendix specifies 200 for Stack-Cube.

“Transitions” in the main text refers to training updates; we will clarify this terminology and distinguish the two quantities in Table 1 and the experimental descriptions. Stack-Cube uses 200 evaluation episodes, while the remaining environments use 100; we will make these counts consistent throughout the revised paper.

Dear reviewer, as the time for the end of discussion is approaching, may we ask if you have managed to review our responses, and whether any concerns remain? Thank you.

Please take a moment to respond the authors. Thank you!

Reviewer_ZXNQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackled goal conditioned object centric RL. There are two central ideas in the paper:

  1. An object centric representation based on using feature connectivity to discover objects, and hand designed geometric functions based on the features, such as shape and pose.
  2. Separate the multi-object goal conditioned tasks into single object manipulation tasks, performed in sequence according to the distance between the object and the goal.

The policy therefore only gets as input the object-goal pair features selected for manipulation in the sequence.

Training is performed using off-the-shelf IQL.

Evaluations on standard GCRL benchmarks show more efficient learning and compositional generalization to several baselines.

Critical strengths
  1. The experimental results show clear improvement over some baselines.
  2. The compositional generalization results are impressive.
Critical weaknesses
  1. The idea of separating a multi-object GCRL task into object-goal pairs was previously implemented in SMORL [1]. A paper that is not cited, but already suggested the main idea (though implemented very differently).

  2. A main limitation of SMORL, which carries over to the approach in this paper too, is when tasks need to be performed in a particular order. See for example the "ordered push" experiment in ECRL [2].

  3. While the experimental results show improved performance with recent methods like ECRL, it is not clear to me exactly why the method is better than ECRL. Is it because of separating the task into object-goal pairs? (In the ECRL paper, it was already shown that SMORL outperforms ECRL for tasks that can be decomposed as such).

[1] Zadaianchuk, Andrii, Maximilian Seitzer, and Georg Martius. "Self-supervised visual reinforcement learning with object-centric representations." arXiv preprint arXiv:2011.14381 (2020).

[2] Haramati, Dan, Tal Daniel, and Aviv Tamar. "Entity-centric reinforcement learning for object manipulation from pixels." International Conference on Learning Representations. Vol. 2024. 2024.

Questions for authors

See weaknesses above.

  1. Can the authors provide more insight into why their method outperforms ECRL?

  2. What is the novelty w.r.t. SMORL?

2 comments

Thank you for your comprehensive and constructive review. We provide responses to the questions below:

W1-3: Technical differences between SMORL, ECRL and COMPACT.

COMPACT’s key contribution is a more explicit compositional structure, with a key difference in ranking state–goal pairs: SMORL selects one goal at a time without ranking state–goal pairs, while ECRL learns pairwise state–goal associations through attention without explicit pair ranking. COMPACT explicitly matches objects and ranks the matched pairs, with learned ranking weights for task-specific ordering, focusing policy learning on generalizable object-level manipulation.

Q1: Can the authors provide more insight into why their method outperforms ECRL?

ECRL learns state–goal associations and ranking implicitly through attention, whereas COMPACT explicitly matches state-goal pairs by appearance and ranks the pairs using pose displacement and learnable weights for task-specific ordering. This separates matching, ordering-aware selection, and generalizable manipulation, reducing what the actor must infer implicitly and encouraging generalizable object-level manipulation. This stronger compositional structure helps explain the improved manipulation performance, sample efficiency, and compositional generalization observed over the ECRL baseline.

Q2: What is the novelty w.r.t. SMORL?

SMORL selects one goal at a time and cycles through them without ranking state–goal pairs, limiting its ability to handle ordering dependencies. In contrast, COMPACT explicitly matches state–goal objects and ranks the matched pairs with learnable weights for task-specific execution ordering. Following the reviewer’s suggestion, we integrated SMORL into our pipeline similarly to ECRL and compare below on Stack-Cube, which requires ordering-sensitive manipulation, as does ECRL’s Ordered-Push task.

MethodCoverage ↑Success ↑Chamfer ↓
SMORL5.41 ± 1.709.99 ± 1.392.07 ± 0.68
ECRL8.97 ± 2.4417.66 ± 1.431.81 ± 0.38
COMPACT17.57 ± 2.8951.56 ± 5.641.14 ± 0.43

These results support the effectiveness of COMPACT’s explicit matching and ordering-aware pair selection, extending single-object decomposition toward more generalizable manipulation. We will include SMORL as a new baseline in all experiments in the final version.

Dear reviewer, as the time for the end of discussion is approaching, may we ask if you have managed to review our responses, and whether you have any remaining questions or concerns? Thank you.

Comments on the paper

2 comments

Summary of revisions and additional results

We thank all reviewers for their thorough and constructive feedback. We have revised the paper and provided additional experiments in our responses to address the concerns.

1. Revised theoretical guarantee

We thank the reviewer for highlighting the need to make the effect of object ordering explicit in the theoretical analysis. We have revised the theorem and proof to account for differences in object-selection rules, rather than assuming shared scheduling coefficients.

We have revised Theorem 3.1 and its proof in Appendix A, replacing the original KεK\varepsilon bound with

\leq \Delta_{\mathrm{sel}} +\frac{1-\gamma^{H+1}}{1-\gamma}\varepsilon_{\mathrm{ctrl}}.$$ Here, $\Delta_{\mathrm{sel}}=V^\star-V_\sigma^\star$ captures the loss from fixing the object-selection rule $\sigma$, while $\varepsilon_{\mathrm{ctrl}}$ uniformly bounds the per-step control Bellman error under that rule. The revised proof uses the finite-horizon version of the performance-difference identity [1], removing the assumption of shared scheduling coefficients. This conditional bound reinforces the theoretical rationale for COMPACT’s two complementary design choices: an explicit object-selection mechanism to address selection loss, and a shared single-object action head to address control error. **2. Clarified ranking, completion thresholding, and gradient flow** We have expanded Algorithm 1 and its description to clarify the implemented selector. COMPACT ranks matched pairs using pose displacement and a learned correction based on pair features and pooled pair context, enabling configuration-dependent ordering. A learned completion threshold restricts selection to unfinished pairs, with a fallback to all manipulable pairs if none remain. We also clarify gradient flow: object discovery uses soft grouping, and the completion threshold and ranking network are trained through straight-through estimators for completion thresholding and pair selection. **3. Additional selection-rule comparisons** Following the reviewers’ suggestions, we compare random selection, a learned selector trained via a straight-through estimator, and COMPACT using the same encoder architecture and training setup. #### Push-Tetris | Selection rule | Coverage ↑ | Success ↑ | Chamfer ↓ | |---|---|---|---| | Random selection | 44.74 ± 1.53 | 13.84 ± 3.67 | 0.57 ± 0.12 | | Learned selector | 50.47 ± 1.65 | 27.64 ± 4.57 | 0.27 ± 0.18 | | COMPACT | **62.12 ± 1.69** | **55.87 ± 4.25** | **0.10 ± 0.02** | COMPACT outperforms both alternatives across all metrics. Qualitative analysis shows that random selection causes frequent object switching and longer trajectories, while the learned selector produces more collisions. In contrast, COMPACT reaches the goal more smoothly. #### Stack-Cube | Selection rule | Coverage ↑ | Success ↑ | Chamfer ↓ | |---|---|---|---| | Random selection | 2.19 ± 0.22 | 0.00 ± 0.00 | 2.67 ± 0.39 | | Learned selector | 11.90 ± 2.88 | 24.11 ± 1.26 | 1.71 ± 0.50 | | COMPACT | **17.57 ± 2.89** | **51.56 ± 5.64** | **1.14 ± 0.43** | COMPACT also outperforms both alternatives on Stack-Cube, supporting the benefit of its structured selection rule on an order-dependent task. We will include the full ablation results across all environments and add qualitative visualizations in the final version of the paper. **4. Differences from SMORL and ECRL and an additional comparison** COMPACT’s key contribution is a more explicit compositional structure. While SMORL composes single-object skills sequentially and ECRL learns state–goal associations and inter-object dependencies through attention, COMPACT explicitly matches objects by appearance and ranks matched pairs using pose displacement, pair features, and pooled context. This separates matching, ordering-aware selection, and pair-conditioned control, reducing what the actor must infer implicitly. Following the reviewer’s suggestion, we integrated SMORL into our pipeline similarly to ECRL and compare below on Stack-Cube, which requires ordering-sensitive manipulation. | Method | Coverage ↑ | Success ↑ | Chamfer ↓ | |---|---|---|---| | SMORL | 5.41 ± 1.70 | 9.99 ± 1.39 | 2.07 ± 0.68 | | ECRL | 8.97 ± 2.44 | 17.66 ± 1.43 | 1.81 ± 0.38 | | COMPACT | **17.57 ± 2.89** | **51.56 ± 5.64** | **1.14 ± 0.43** | COMPACT outperforms both baselines across all metrics, supporting its combined design of explicit matching and ordering-aware pair selection rather than single-object decomposition alone. We will include SMORL as a new baseline in all experiments in the final version. **5. Clarified experimental settings** We distinguish dataset sizes from training updates and clarify the evaluation episode counts, environment-specific success criteria, and episode lengths used across methods. [1] Kakade, Sham, and John Langford. “Approximately Optimal Approximate Reinforcement Learning.” ICML, 2002.

Additional results and clarifications

We provide further ablations and comparisons below.

6. Additional matching and selection ablations

We replace COMPACT’s explicit appearance matching with learned attention while retaining the same downstream selection and control components and training setup. We report this comparison alongside the previous selection ablations.

Push-Tetris

AblationCoverage ↑Success ↑Chamfer ↓
Random selection44.74 ± 1.5313.84 ± 3.670.57 ± 0.12
Learned selector50.47 ± 1.6527.64 ± 4.570.27 ± 0.18
Learned attention48.48 ± 1.5517.73 ± 3.260.29 ± 0.17
COMPACT62.12 ± 1.6955.87 ± 4.250.10 ± 0.02

Stack-Cube

AblationCoverage ↑Success ↑Chamfer ↓
Random selection2.19 ± 0.220.00 ± 0.002.67 ± 0.39
Learned selector11.90 ± 2.8824.11 ± 1.261.71 ± 0.50
Learned attention8.99 ± 2.1012.78 ± 1.181.81 ± 0.68
COMPACT17.57 ± 2.8951.56 ± 5.641.14 ± 0.43

COMPACT outperforms all three alternatives on both environments, supporting the contributions of explicit matching and ordering-aware selection. We will include the full ablation results across all environments in the final paper.

7. Additional SMORL comparison and goal-persistence ablation

We clarify that our SMORL-style baseline follows ECRL’s released implementation, where one goal is selected at a time.

We additionally compare on Push-Tetris, which does not require ordering-sensitive manipulation. We also evaluate COMPACT + goal persistence, which adopts SMORL-style persistent goal selection while retaining COMPACT’s matching and controller.

MethodCoverage ↑Success ↑Chamfer ↓
SMORL26.19 ± 1.990.00 ± 0.000.47 ± 0.44
ECRL28.85 ± 1.430.00 ± 0.000.57 ± 0.17
COMPACT + goal persistence52.06 ± 2.6048.71 ± 5.230.16 ± 0.07
COMPACT62.12 ± 1.6955.87 ± 4.250.10 ± 0.02

SMORL performs comparably to ECRL, but both underperform COMPACT. COMPACT + goal persistence achieves closer performance, although COMPACT remains the top performer. We will include these comparisons and implementation details in the final version of the paper.

8. Clarified selection behavior

We clarify that COMPACT’s ranking combines pose distance with learned weights and does not require a fixed spatial ordering. For the reviewer’s example of arranging blocks in a row, the closest unfinished state–goal pair provides a default preference when the row can be assembled in any execution order. Push-Tetris illustrates this setting, whereas Stack-Cube requires ordering-sensitive placement.

We additionally measure how often COMPACT selects the closest unfinished pair and the mean distance rank of its selected pair, where rank 1 denotes the closest.

TaskClosest-pair agreement (%) ↑Mean distance rank ↓
Push-Tetris96.91.1
Stack-Cube37.42.2

COMPACT selects the closest unfinished pair in nearly all Push-Tetris decisions, but only about a third on Stack-Cube, consistent with a distance-based preference when no execution order is required.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.