acceptodds
Under review as a conference paper at ICLR 2027

TopoBox: towards Physically Feasible 3D Object Re-Arrangement via Topology-Aware Box Reasoning

Abstract

3D object re-arrangement aims to discover vacant and well‑suited spaces for object movement and placement, which is an essential task in embodied interaction scenarios.While existing methods have effectively maintained **semantic consistency** with instructive conditions, they still show limitations on arranging spaces with **physical feasibility** (e.g., 3D dimension and orientation). Meanwhile, previous approaches ignore the **topological rationality** of the interaction between source objects and surrounding environments (e.g., structural support and collision-free connectivity), which also constrains the physical representation. To address these limitations, we propose **TopoBox**, a two-stage framework that leverages **Topo**logy-aware **Box** reasoning for physically feasible 3D object re-arrangement. At the first stage, we design a **Space Arrangement Predictor (Predictor)**, which explicitly localizes the 3D bounding box and concurrently forecasts an **interactive BoxSlot field**. At the second stage, we design a **Topological Box Reasoner (Reasoner)**, which utilizes the BoxSlot field as the core guidance, and realizes iterative topology refinement via multi-granularity and multi-surface feature sampling (i.e., box bottom, side, and upper surfaces). Furthermore, to address the lack of data for jointly modeling instruction-guided object rearrangement and its physical feasibility, we introduce ReArrange3D, a dataset comprising 78K+ instruction-guided re-arrangement instances, with comprehensive annotations to support modeling and evaluation. Extensive experiments conducted on **ReArrange3D** demonstrate that TopoBox surpasses State-Of-The-Art (SOTA) counterparts on placement accuracy and physical feasibility.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Object Re-arrangement, Embodeid Interaction, Spatial Reasoning, 3D Representation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mSwBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

TopoBox addresses the instruction-guided 3D object rearrangement problem by predicting the placement box rather than just the position. The first stage predicts the 3D bounding box of the source object and a BoxSlot field, which identifies the placement area associated with the instruction. The second stage utilizes scene features sampled at different resolutions around the box surface to optimize the position and yaw angle. The authors also introduce ReArrange3D, which extracts valid placement positions and spatial instructions from the RGB-D scene. Experiments compare this method with a finely tuned point-based baseline method, showing that it achieves higher placement feasibility while maintaining similar semantic correctness.

Critical strengths

1.This paper defines the placement target as a well-defined 3D cube with dimensions and yaw angle, thus providing a clear geometric object for prediction and evaluation. I believe this addresses a gap in real-sim tabletop fild research to some extent.

2.The paper contributes a tabletop-level dataset, details the dataset construction and annotation process, and separates the evaluation metrics from semantic consistency, support, and collision detection.

3.The paper is clearly structured and easy to understand.

Critical weaknesses

The conversion from point-based baseline predictions to placement boxes does not state which yaw angle is used when the ground-truth source box is placed at the predicted location. Because support and collision depend on orientation, part of the reported feasibility gap may come from this difference rather than only from the proposed Reasoner.

Questions for authors
  1. For point-based baseline methods, what yaw angle is used when the ground truth source box is placed at the predicted location? How does this yaw angle compare to the yaw angle predicted by TopoBox? Explicitly specifying this convention would be helpful, as it directly impacts support and collision assessments.

  2. Based on experimental results, the final location seems to focus more on the available space for placement, without considering the solvability of the inverse kinematics equations during motion—that is, whether the robotic arm can reach that location when grasping the object.

  3. I want to know if there are objects in the dataset that are difficult to represent with 3D bounding boxes, such as thin sheets or small objects, and how well they would be represented.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors of W1

We thank Reviewer mSwB for the careful reading, helpful comments, and positive assessment of our work. We appreciate your questions about the evaluation settings and the scope and limitations of our approach, which we address below.

W1: Yaw conventions and their influence on placement feasibility.

R1: Thank you for your comment. We clarify the baseline yaw convention and provide additional yaw** search experiments. These experiments show that orientation adjustment improves baseline feasibility, but does not fully explain the gap to TopoBox.

1) Yaw convention.

For point-based baselines, we place the ground-truth source box at the predicted 3D location while retaining the observed source yaw from the input scene. TopoBox predicts both the placement position and yaw. All methods use ground-truth source dimensions for SR and CFR evaluation. These evaluation conventions are specified in Appendix A.2.2 and A.2.4.

2) Effect of yaw search.

We keep each baseline position fixed and test yaw angles at 15° intervals over [0°, 180°), including the observed source yaw as an additional candidate. A placement passes PF if any tested orientation simultaneously satisfies SRC, SR, and CFR. TopoBox uses its predicted position and yaw without additional search.

MethodYaw evaluationSRC ↑SR ↑CFR ↑PF ↑
RoboPoint-7BOriginal56.4977.0082.5439.47
RoboPoint-7BSearch53.1780.9184.9543.59
RoboRefer-2BOriginal60.9073.6372.4040.13
RoboRefer-2BSearch60.9181.7580.0048.10
RoboBrain2.5-4BOriginal69.8476.0379.6546.44
RoboBrain2.5-4BSearch66.8981.7183.9253.57
TopoBox (Ours)—70.1295.1495.7865.16

Yaw search improves all three baselines. The strongest baseline, RoboBrain2.5, increases from 46.44% to 53.57% PF, but remains 11.59 percentage points below TopoBox. This confirms that orientation affects feasibility while showing that the original yaw convention alone does not account for the gap. We have added these results to Section 5.2 (Table 2) and the search protocol to Appendix A.2.3.

3) Role of the Reasoner.

A predicted position may leave insufficient space for the full object, so adjusting yaw alone cannot resolve support or collision failures. The Reasoner samples scene features around the current box and iteratively adjusts both position and yaw to reassess these constraints. The internal ablation provides complementary evidence: removing the TopoBox Reasoning Module reduces PF@1 from 65.16% to 60.49%. Position-related failure examples are provided in Section 5.2 (Figure 4) and Appendix A.2.6 (Figure 13).

AuthorsOpenReview ↗

Rebuttal by Authors of Q1 to Q3

Q1: Yaw convention for point-based baselines.

A1: As clarified in our response to W1, point-based baselines use the observed source yaw from the input scene when constructing the placement box, whereas TopoBox predicts the placement yaw.

Q2: Robotic arm reachability and inverse kinematics.

A2: Our study focuses on visual perception and spatial reasoning for placement prediction under the defined geometric criteria. Robotic arm reachability, inverse kinematics, motion planning, and execution remain outside the current scope. We have clarified this focus in Section 7 (Limitation) and identified integration with real-world robotic interaction as future work.

Q3: Representation of small and thin objects.

A3: ReArrange3D includes small objects, but does not cover thin sheets such as paper. Our bounding boxes represent object dimensions and spatial extent for placement, with different capabilities and limitations for these cases.

1) Small objects.

TopoBox predicts the source dimensions and uses them to construct placement boxes and determine surface sampling locations. Small rigid objects can therefore be represented according to their dimensions. Qualitative examples involving toy cars are provided in Appendix A.2.6 (Figure 9). These illustrate individual cases.

2) Thin and deformable objects.

The finest voxel resolution of 1 cm cannot reliably capture thin structures such as sheets of paper. Rigid bounding boxes also cannot describe shape changes in flexible materials.

We have explicitly discussed these limitations in Section 7 (Limitation) and identified more adaptive, detailed 3D representations for thin and deformable objects as future work.

AuthorsOpenReview ↗

Additional response to Q2: Robotic arm reachability and inverse kinematics.

A2: Thank you for raising the question of whether the robot can reach the predicted placement while holding the source object. Since our previous response,We have additionally conducted robot execution experiments in simulation to directly examine this issue.

We used a Panda robot arm in ManiSkill 3 across three scene configurations: open tables, smaller tables, and cluttered tables with multiple obstacles. For both TopoBox and RoboBrain, model predictions determine the placement targets. Both methods use the same execution procedure, with top-down grasping based on known object bounding boxes and MPLib motion planning. The placement model specifies the target, while the planner checks whether the robot can reach it and generates the motion required to transport the object.

A trial is successful only if the robot successfully grasps, transports, and releases the source object, and the resulting placement satisfies the instructed spatial relation, remains supported and stable, and involves no unintended collisions during execution. Planning failures, execution collisions, and unstable placements all count as failures. Reachability and motion feasibility therefore directly affect the reported execution success.

Results on 55 paired cases are as follows:

MethodSuccessful executionsSuccess rate ↑
TopoBox37/5567.27%
RoboBrain28/5550.91%

TopoBox completes nine more tasks overall, improving execution success by 16.36 percentage points under the shared grasping and planning procedure. These results provide direct evidence that the predicted placements can support successful robot execution in the evaluated simulation settings.

We have also added Figure 14 in the revised appendix, which follows predicted placement targets through grasping and transport to stable states after release. Together, the execution results and qualitative examples extend the evaluation to whether the robot can physically realize the predicted placements.

Reviewer_uZ7uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TopoBox addresses language-guided 3D object placement from RGB-D observations using explicit object bounding boxes. Its first stage predicts the source box and an instruction-conditioned “BoxSlot” field identifying candidate placement regions. Its second stage iteratively refines placement position and yaw by sampling scene features across box surfaces and spatial resolutions. The paper also introduces ReArrange3D, containing 78,506 instruction-guided instances with multiple valid placement positions and orientations, derived from five RGB-D datasets. Validation includes baseline comparisons, source-grounding evaluation, component ablations, and qualitative examples. TopoBox achieves 65.16% placement feasibility at rank one, versus 46.44% for the strongest evaluated baseline, with similar semantic correctness. Improvements are reported across all five data sources.

Critical strengths
  • Substantial, consistently observed feasibility improvements. TopoBox improves PF@1 by 18.72 percentage points over RoboBrain2.5 while maintaining comparable semantic correctness (Table 1). Removing the reasoning module lowers PF@1 from 65.16% to 60.49%, with almost unchanged semantic correctness (Table 3). This supports the intended contribution of geometric interaction features.

  • Useful supervision for multiple valid placements. ReArrange3D preserves multiple valid centers and yaw angles instead of imposing a single target pose. Its combination of real RGB-D observations, object dimensions, spatial instructions, and geometric feasibility annotations provides a useful resource for studying placement under ambiguity.

Critical weaknesses
  • The comparisons do not establish the advantage over strong geometry-aware placement methods. All three placement baselines predict points and are evaluated using the source object’s observed yaw, whereas TopoBox predicts yaw and explicitly models object extent (Appendix A.2.3). Consequently, the large external-baseline improvement combines several differences beyond the proposed reasoning mechanism. The internal ablations establish usefulness within TopoBox, but not superiority over an alternative geometric approach. Relevant work such as PlaceIt3D is discussed but not evaluated. A suitable geometry-aware comparator or point baseline augmented with geometric refinement and yaw search would materially strengthen the contribution.

  • Physical feasibility is validated only through geometric proxies. Support is determined from a projected height band, followed by morphological closing and hole filling; collision is checked against annotated object boxes (Appendix A.2.2). These are reproducible criteria, but they do not directly verify actual contact or stability, and filling gaps can affect support judgments. Without an independent audit of these checks, the results establish benchmark constraint satisfaction more firmly than physical feasibility. Robot execution is not essential, but representative contact checks or metric-sensitivity analysis would strengthen this central claim.

Questions for authors
  • Additional validation: Does the advantage persist against a geometry-aware placement method or the strongest point baseline augmented with yaw search and geometric refinement using the same available inputs?

  • Clarification: Are the splits disjoint by scene or recording sequence, beyond grouping instructions from the same frame? The current description leaves possible overlap between related frames unresolved.

  • Additional validation: How closely does the support metric agree with independently checked contact geometry, and how sensitive are the conclusions to its height tolerance and hole-filling operations?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors of W1

We thank Reviewer uZ7u for the careful evaluation and constructive suggestions. We address your concerns about the baseline comparisons, evaluation reliability, and data split through the additional results and clarifications below.

W1: Comparison with geometry-aware placement methods.

R1: Thank you for your suggestion. Following your recommendation, we have added a comparison with PlaceWizard, the geometry-aware placement method introduced in PlaceIt3D [A], and further evaluated point-based methods with yaw search.

1) Comparison with PlaceWizard. We fine-tune PlaceWizard on the same ReArrange3D training split and provide it with the source object grounded by Stage I of TopoBox to predict the placement position and orientation. This allows us to compare PlaceWizard with Stage II of TopoBox using the same source grounding.

We also explored comparisons with M2T2 [B] and AnyPlace [C]. However, substantial differences in input construction, prediction targets, and training supervision prevented meaningful comparisons with TopoBox under our current dataset setting. We therefore report PlaceWizard as the additional geometry-aware comparator.

Results on ReArrange3D (%, higher is better):

MethodSRC ↑SR ↑CFR ↑PF@1 ↑
PlaceWizard61.9977.5984.5347.88
TopoBox (Ours)70.1295.1495.7865.16

TopoBox achieves higher scores on all four metrics, with a 17.28 percentage point improvement in PF@1 over PlaceWizard under this adaptation.

We have also added qualitative comparisons with PlaceWizard in Section 5.3 (Figure 5) and Appendix A.2.6 (Figure 9), with the latter covering all eight spatial relations.

2) Point-based methods with yaw search. To address your concern about evaluating point-based methods using the observed source yaw, we additionally evaluate RoboPoint, RoboRefer, and RoboBrain2.5 with yaw search. At each predicted position, we use the ground-truth source dimensions and test yaw angles at 15° intervals over [0°, 180°), including the observed source yaw as an additional candidate. A placement passes PF if any tested orientation simultaneously satisfies SRC, SR, and CFR. TopoBox is evaluated using its predicted position and yaw.

MethodYaw evaluationSRC ↑SR ↑CFR ↑PF ↑
RoboPoint-7BOriginal56.4977.0082.5439.47
RoboPoint-7BSearch53.1780.9184.9543.59
RoboRefer-2BOriginal60.9073.6372.4040.13
RoboRefer-2BSearch60.9181.7580.0048.10
RoboBrain2.5-4BOriginal69.8476.0379.6546.44
RoboBrain2.5-4BSearch66.8981.7183.9253.57
TopoBox (Ours)—70.1295.1495.7865.16

Yaw search improves PF for all three baselines. However, the strongest baseline, RoboBrain2.5, reaches 53.57%, compared with 65.16% for TopoBox, leaving a gap of 11.59 percentage points. Its support and collision-free rates also remain lower. These results show that even under the optimal source yaw, the point-based method still cannot surpass TopoBox.

We attribute this result to insufficient consideration of the full object extent when selecting placement positions. A predicted point may satisfy the instructed spatial relation, yet the placed object can extend beyond the supporting region or intersect nearby obstacles even when its yaw is adjusted. This highlights the importance of accounting for object extent and reasoning about structural support and collision constraints when selecting and refining placement positions, as done in TopoBox.

We have added the comparisons and analysis to Sections 5.2–5.3 (Tables 1–2; Figures 4–5), with protocol and adaptation details in Appendix A.2.3–A.2.4 and additional visualizations in Appendix A.2.6 (Figures 9 and 13).

Together with the PlaceWizard comparison above, these experiments show that TopoBox maintains higher placement feasibility against both an alternative geometry-aware method and point-based methods augmented with yaw search.

[A] PlaceIt3D: Language-Guided Object Placement in Real 3D Scenes, in ICCV, 2025.

[B] M2T2: Multi-Task Masked Transformer for Object-Centric Pick and Place.

[C] AnyPlace: Learning Generalizable Object Placement for Robot Manipulation, in CoRL 2025.

AuthorsOpenReview ↗

Rebuttal by Authors of W2 to Q2

W2: Independent validation of physical feasibility.

R2: Thank you for your suggestion. We have added two independent checks of SR, based on human judgments and a static stability criterion[D], together with visualizations to make its decisions easier to inspect.

1) Human validation.

To assess whether SR agrees with human judgments of contact and support, we manually inspected 100 TopoBox Top-1 predictions. Each of the five data sources contributes 20 predictions, equally divided between SR passes and failures.

SR judgmentHuman: PassHuman: Fail
Pass473
Fail1436

Agreement reaches 83%, with Cohen κ of 0.66, supporting consistency between SR and human judgments. SR rejects 14 placements accepted by humans but accepts only three rejected manually, indicating more conservative acceptance in this sample. We have added these results and the corresponding analysis to Appendix A.2.3 (Table 5).

2) Independent validation with SSM.

To independently check the stability of placements accepted by SR, we use the static stability margin (SSM). Under a uniform rigid-box assumption, this check requires the projected center of mass to lie strictly inside the convex hull of observed support points. We retain the benchmark height tolerance but omit closing and hole filling.

MethodSR–SSM Agreement*TopoBox SuccessRoboBrain2.5 Success
SSM99.42%67.52%57.09%

* SR–SSM Agreement denotes the proportion of TopoBox predictions accepted by SR that also pass SSM. Success denotes PF@1 with SSM replacing SR, retaining SRC and CFR, evaluated on instructions with predictions from both methods.

The 99.42% rate shows that nearly all TopoBox placements accepted by SR also satisfy the independent stability criterion, supporting the reliability of SR acceptance. TopoBox remains 10.43 percentage points ahead of RoboBrain2.5, showing that its advantage persists when the support criterion is replaced. We have added these SSM validation results and the corresponding analysis to Appendix A.2.3.

3) Visualizations of SR judgments.

To help readers visually assess how SR judges contact and support, we have added examples in Appendix A.2.6 (Figure 12). These show accepted and rejected placements across all five data sources, illustrating how SR distinguishes supported placements from failures involving floating, penetration of the supporting surface, or insufficient support.

[D] McGhee, Robert B., and Andrew A. Frank. "On the stability properties of quadruped creeping gaits." Mathematical Biosciences 3 (1968): 331-351.

Q1: Advantage over stronger placement baselines.

A1: Yes. As discussed in our response to W1, the advantage persists in the additional comparisons with a geometry-aware placement method and the strongest point-based baseline augmented with yaw search.

Q2: Are the splits disjoint by scene or recording sequence?

A2: Thank you for requesting this clarification. The splits are disjoint by RGB-D frame, but not by scene or recording sequence. Our evaluation focuses on spatial perception and placement reasoning under different observations and instructions within shared scenes. Shared scene identity therefore does not necessarily imply an identical placement task.

1) Frame sampling and grouping. We select frames at intervals from each video to reduce redundancy between consecutive observations. We then assign the selected frames to the training, validation, or test set, keeping each frame and all its associated instruction instances together. No frame or its associated instruction instances is shared across splits.

2) Differences across splits. The following statistics characterize differences in frame indices, camera viewpoints, and annotated target direction sets:

ComparisonMean original frame index gapMean camera rotation differenceTasks with different target direction sets
Validation → Training27.3412.60°60.2%
Test → Training26.9410.68°59.9%
Test → Validation145.1825.82°76.3%

These results show variation in observations and placement requirements within shared scenes. Each task requires interpreting the current observation and requested spatial relation to identify placements that satisfy the geometric constraints.

We have clarified the data split and evaluation scope in Section 5.1 of the main paper and Appendix A.1.4.

AuthorsOpenReview ↗

Rebuttal by Authors of Q3

Q3: Reliability and sensitivity of the support metric.

A3: TopoBox maintains higher PF@1 than RoboBrain2.5 under all tested settings, including different height tolerances and removal of hole filling. Independent validation through human judgments and SSM is discussed in our response to W2.

Height bands are in centimeters relative to the box bottom. ΔSR denotes the change from the default setting; the PF@1 gap is TopoBox minus RoboBrain2.5. Changes and gaps are in percentage points.

Evaluation settingTopoBox ΔSRRoboBrain2.5 ΔSRPF@1 gap
Height [−1, +1]−23.30−3.784.96
Height [−3, 0]−26.41−33.1820.26
Height [−2, +1]−1.19−1.0118.30
Height [−3, +2]+0.55+3.0117.46
Without hole filling−1.51−9.3820.57
Default: [−3, +1], hole filling——18.72

1) Height tolerance.

The height band determines which scene observations contribute to the support region. Narrowing it can exclude observations needed to cover the entire object footprint, reducing SR and consequently PF@1 while SRC and CFR remain unchanged. Although the scores change with the support criterion, the comparative advantage of TopoBox persists across all tested height bands.

2) Hole filling.

Removing hole filling under the default height tolerance reduces TopoBox SR and PF@1 by only 1.51 and 0.75 points, respectively. Its PF@1 advantage remains 20.57 points, showing limited dependence on hole filling and confirming that the advantage persists without this operation.

We have added these results and the corresponding analysis to Appendix A.2.3 (Table 6).

Dear Reviewer,

This is a gentle reminder that the discussion phase will close soon. We would appreciate it if you could go over the authors’ rebuttal and share with us whether your original concerns have been resolved, or if you have additional points to raise with the authors.

Thank you for your effort.

Reviewer_rfbeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TopoBox predicts object placement positions and orientations from RGB-D observations and language instructions. It represents objects as boxes and iteratively refines candidate placements using features sampled around their surfaces. The paper also introduces ReArrange3D, with automatically generated placement annotations.

Critical strengths

Accounting for the full object extent is a sensible improvement over predicting a placement point alone. TopoBox substantially improves placement feasibility on ReArrange3D, and the ablations show that local geometric features contribute to this improvement.

Critical weaknesses
  1. The connection between the evaluated task and the motivating applications is insufficiently established. The benchmark covers eight horizontal spatial relations and upright box placements, with feasibility defined by footprint support and box overlap. It does not evaluate reachability, motion planning, or execution. These are reasonable simplifications, but the paper does not show whether improvements under these geometric criteria translate into more successful object rearrangement. The results therefore support a narrower claim than the embodied applications used to motivate the work.

  2. The need for the proposed learned refinement method remains unclear. The main comparisons are against point-based placement methods, which do not explicitly optimize the geometric constraints central to this benchmark. A more informative baseline would combine language grounding and estimated object geometry with explicit collision checking and placement search, using the same observations available to TopoBox. Without this comparison, it is difficult to tell how much of the gain requires the proposed method rather than simply accounting for object extent and support.

Questions for authors
  1. Is there evidence from simulation or real execution that higher benchmark feasibility leads to more successful placements?

  2. How does TopoBox compare with a geometry-based placement search using predicted object dimensions and the observed scene, without access to privileged annotations?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors of W1 to Q2

We thank Reviewer rfbe for for the valuable comments and suggestions. We have clarified the study scope and strengthened the experimental comparisons in response to your comments, as detailed below.

W1: Connection between the evaluated task and the motivating applications.

R1: We apologize for not making the scope of our study sufficiently clear in the original presentation.

We have revised the task definition in the Abstract to focus on identifying suitable object placement locations through instruction-guided spatial perception and reasoning. In Section 1 (Introduction), we clarify that this capability concerns object dimensions and orientation, structural support, and collision constraints, and describe the motivating applications as potential uses of this capability. This capability could support embodied interaction by providing candidate placement locations for subsequent planning and execution.

Our results demonstrate improvements in placement prediction under the defined geometric criteria. Reachability, motion planning, and execution remain outside the scope of this study, and the reported improvements are reflected in visual perception and spatial reasoning. We have also discussed this limitation in Section 7 (Limitation) and identified extending TopoBox to real-world robotic interaction as a direction for future work.

W2: Necessity of the learned refinement method.

R2: Thank you for your suggestion. Following your recommendation, we added a comparison with the geometry-aware method PlaceWizard[A] on ReArrange3D.

Results on ReArrange3D (%, higher is better; —: not reported):

MethodSRC ↑SR ↑CFR ↑PF@1 ↑PF@5 ↑
PlaceWizard61.9977.5984.5347.88—
TopoBox (Ours)70.1295.1495.7865.1679.50

TopoBox outperforms PlaceWizard on all four shared metrics, with a 17.28 percentage point improvement in PF@1. We also explored comparisons with M2T2[B] and AnyPlace[C]. However, substantial differences in input construction, prediction targets, and training supervision prevented meaningful comparisons with TopoBox under our current dataset setting. The PlaceWizard comparison and adaptation settings are reported in Section 5.2 (Table 1) and Appendix A.2.4, respectively.

Furthermore, we examined the refinement design through internal ablations, as reported in the paper:

MethodSRC ↑SR ↑CFR ↑PF@1 ↑
w/o Explicit Pose Update68.3093.2091.7562.12
w/o TopoBox Reasoning70.0189.5091.2360.49
TopoBox (Full)70.1295.1495.7865.16

The nearly unchanged SRC after removing topology reasoning, together with lower SR and CFR, further supports its contribution to structural support and collision avoidance. The ablation results and further analysis are provided in Section 5.2 (Table 4) and Appendix A.2.5, respectively.

[A] Placeit3d: Language-guided object placement in real 3d scenes, in ICCV 2025

[B] M2t2: Multi-task masked transformer for object-centric pick and place.

[C]AnyPlace: Learning generalizable object placement for robot manipulation, in CoRL 2025.

Q1: Evidence from simulation or real execution.

A1: As clarified in our response to W1, our study evaluates visual perception and spatial reasoning under the defined geometric criteria and does not include simulation or real execution validation. We discuss this limitation in Section 7 (Limitation) and leave execution validation to future work.

Q2: Comparison with geometric placement search.

A2: We apologize for not fully understanding the intended setup for geometry-based placement search. As complementary evidence, we evaluated point-based methods with yaw search at fixed predicted positions, using ground-truth source dimensions and benchmark annotations for evaluation.

MethodYaw evaluationSRC ↑SR ↑CFR ↑PF ↑
RoboPoint-7BOriginal56.4977.0082.5439.47
RoboPoint-7BSearch53.1780.9184.9543.59
RoboRefer-2BOriginal60.9073.6372.4040.13
RoboRefer-2BSearch60.9181.7580.0048.10
RoboBrain2.5-4BOriginal69.8476.0379.6546.44
RoboBrain2.5-4BSearch66.8981.7183.9253.57
TopoBox (Ours)—70.1295.1495.7865.16

RoboBrain2.5 improves from 46.44% to 53.57% PF, but remains 11.59 percentage points below TopoBox (65.16%), showing that yaw search alone does not close the placement feasibility gap. We have added the yaw search results to Section 5.2 (Table 2), the protocol to Appendix A.2.3, and corresponding visualizations to Section 5.2 (Figure 4) and Appendix A.2.6 (Figure 13).

Reviewer_rfbeOpenReview ↗

Response to the authors' rebuttal

We thank the authors for the additional experiments and the candid clarifications regarding the study scope. Having considered the rebuttal, our concerns remain, for two reasons.

  1. The revision narrows the claims rather than supporting them.

Our original weakness was that the evaluated task supports a narrower claim than the embodied applications motivating the work. The authors' response is to rewrite the abstract and introduction to match the narrower claim, and to defer reachability, motion planning, and execution to future work. We appreciate the honesty, but this resolves the inconsistency by shrinking the contribution: the paper now studies instruction-guided placement prediction evaluated by geometric criteria on a self-constructed benchmark, with its relevance to any downstream use asserted but not demonstrated. In our judgment, a contribution of this narrowed scope is not sufficient for this venue, regardless of how clearly it is presented.

  1. The paper does not establish that the studied problem requires — or benefits from — a learned solution.

Two observations underlie this concern. First, the paper provides no concrete account of who would use this capability and to what end: the motivating applications appear as a single sentence in the introduction and are never revisited, and no experiment, simulation, or case study connects higher benchmark feasibility to any downstream outcome. Second, the benchmark's feasible placements are themselves produced by an explicit geometric pipeline (support-surface detection and collision-free enumeration), and the evaluation metrics are exactly those support and overlap criteria. Feasibility on ReArrange3D is therefore, by construction, what a rule-based geometric search over the observed scene would produce. The rebuttal's own yaw-search experiment illustrates how much such non-learned geometric reasoning can recover (RoboRefer CFR 72.40 → 80.00; RoboBrain2.5 PF 46.44 → 53.57), yet the most direct control we asked for — language grounding combined with estimated object geometry, explicit collision checking, and placement position search — was still not run. PlaceWizard is another learned method and does not substitute for it. Without that comparison, the paper cannot exclude the possibility that its learned reasoner is, in effect, re-deriving the geometric rules that defined the benchmark, at substantially greater cost.

In summary, the rebuttal clarifies what the paper does not claim, but does not supply what our review found missing: demonstrated relevance beyond the benchmark itself, and evidence that learning is needed over explicit geometric reasoning. Our rating is unchanged.

AuthorsOpenReview ↗

Reply for further discussion w1

Thank you for your further comments. In response to the issue you raised, we conducted a direct examination through robot execution experiments in simulation and evaluated the direct geometric baseline you requested.

W1: Connection between placement prediction and downstream robot execution.

R1: Following your suggestion, we conducted additional experiments to directly evaluate whether the predicted placements support successful robot execution.

We used a Panda robot arm and objects in ManiSkill 3, with three scene configurations: open tables, smaller tables, and cluttered tables with multiple obstacles. For both methods, the model predictions determine the placement targets. The same execution procedure uses top-down grasping based on known object bounding boxes and MPLib motion planning to grasp, transport, and release the source object.

A trial is successful only if the robot completes these actions and the released object satisfies the instructed spatial relation, remains supported and stable, and causes no unintended collisions during execution. Planning failures, execution collisions, and unstable placements all count as failures. Reachability, motion planning, and physical execution therefore directly affect the reported success rate.

Results on 55 paired cases are as follows:

MethodSuccessful executionsSuccess rate ↑
TopoBox37/5567.27%
RoboBrain28/5550.91%

TopoBox completes nine more tasks overall, improving execution success by 16.36 percentage points. These results provide direct simulation evidence that the predicted placements can support downstream robot tasks, with higher task completion under the shared grasping and planning procedure.

We have also added Figure 14 in the revised appendix to illustrate how predicted placements translate into robot execution. Each sequence connects the predicted target to grasping, transport, and a stable placement after release. The caption specifies the experimental conditions, including frozen model weights, seven forward passes for candidate checking, and the use of known source poses and scene geometry by the executor.

The core contribution remains instruction-guided placement prediction that accounts for semantic relations, structural support, and collision constraints. The added experiments connect this capability to observable task outcomes: the predictions of TopoBox can be integrated into an existing planning and control pipeline to support successful pick-and-place execution.

AuthorsOpenReview ↗

Reply for further discussion w2

W2: Comparison with explicit geometric placement search.

R2: Thank you for pointing out the missing direct geometric comparison. We have now evaluated Stage I + rule-based geometric search, which shares the frozen first stage of TopoBox to obtain predicted object dimensions and instruction-conditioned BoxSlot scores. It then generates placement candidates through support-surface detection, explicit support and collision checks, and a joint search over position and yaw. This comparison evaluates replacing the learned second stage with explicit geometric search while retaining the same first-stage predictions.

Results on the 7,626 test instructions are shown below (%, higher is better):

MethodSRC ↑SR ↑CFR ↑PF@1 ↑
Stage I + rule-based geometric search67.6296.0996.0463.01
TopoBox70.1295.1495.7865.16

The geometric baseline achieves slightly higher SR and CFR, confirming that explicit geometric search is effective at satisfying support and collision constraints. However, geometrically feasible placements do not necessarily satisfy the instruction. Among the 2,821 failed cases for this baseline, 2,244 pass both support and collision checks but fail the spatial relation criterion. TopoBox improves SRC and PF@1 by 2.50 and 2.15 percentage points, respectively. These results support the benefit of the learned second stage, which uses language and scene features to refine and rank placement candidates jointly.

We also measured inference latency for both complete pipelines on the same machine, using the same stratified sample of 200 validation instructions. Timing includes input loading, model inference, candidate output, and scene processing for each request in the geometric baseline.

Latency is reported in milliseconds:

MethodScene preprocessingSearch and rankingComplete second stageFull pipeline, median ↓Full pipeline, P95 ↓
Stage I + rule-based geometric search81.0837.83120.58148.17334.78
TopoBox——17.4847.3881.35

All values except the final column are medians. Medians of individual stages do not necessarily sum to the median of the complete pipeline.

The geometric baseline incurs additional cost from RANSAC support-surface detection, construction of connected support regions, and subsequent position and yaw search. TopoBox instead applies four refinement layers to at most 48 candidates, with a median latency of 17.48 ms for the complete second stage. Its median latency for the full pipeline is 47.38 ms, compared with 148.17 ms for the geometric baseline, a 68.0% reduction.

Under the evaluated implementation and timing conditions, TopoBox achieves higher overall placement success, measured by PF@1, at lower inference latency. This provides direct evidence of a practical benefit from learned refinement relative to the explicit geometric alternative.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.