- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TopoBox addresses the instruction-guided 3D object rearrangement problem by predicting the placement box rather than just the position. The first stage predicts the 3D bounding box of the source object and a BoxSlot field, which identifies the placement area associated with the instruction. The second stage utilizes scene features sampled at different resolutions around the box surface to optimize the position and yaw angle. The authors also introduce ReArrange3D, which extracts valid placement positions and spatial instructions from the RGB-D scene. Experiments compare this method with a finely tuned point-based baseline method, showing that it achieves higher placement feasibility while maintaining similar semantic correctness.
Critical strengths
1.This paper defines the placement target as a well-defined 3D cube with dimensions and yaw angle, thus providing a clear geometric object for prediction and evaluation. I believe this addresses a gap in real-sim tabletop fild research to some extent.
2.The paper contributes a tabletop-level dataset, details the dataset construction and annotation process, and separates the evaluation metrics from semantic consistency, support, and collision detection.
3.The paper is clearly structured and easy to understand.
Critical weaknesses
The conversion from point-based baseline predictions to placement boxes does not state which yaw angle is used when the ground-truth source box is placed at the predicted location. Because support and collision depend on orientation, part of the reported feasibility gap may come from this difference rather than only from the proposed Reasoner.
Questions for authors
-
For point-based baseline methods, what yaw angle is used when the ground truth source box is placed at the predicted location? How does this yaw angle compare to the yaw angle predicted by TopoBox? Explicitly specifying this convention would be helpful, as it directly impacts support and collision assessments.
-
Based on experimental results, the final location seems to focus more on the available space for placement, without considering the solvability of the inverse kinematics equations during motion—that is, whether the robotic arm can reach that location when grasping the object.
-
I want to know if there are objects in the dataset that are difficult to represent with 3D bounding boxes, such as thin sheets or small objects, and how well they would be represented.
5 comments
Rebuttal by Authors of Q1 to Q3
Q1: Yaw convention for point-based baselines.
A1: As clarified in our response to W1, point-based baselines use the observed source yaw from the input scene when constructing the placement box, whereas TopoBox predicts the placement yaw.
Q2: Robotic arm reachability and inverse kinematics.
A2: Our study focuses on visual perception and spatial reasoning for placement prediction under the defined geometric criteria. Robotic arm reachability, inverse kinematics, motion planning, and execution remain outside the current scope. We have clarified this focus in Section 7 (Limitation) and identified integration with real-world robotic interaction as future work.
Q3: Representation of small and thin objects.
A3: ReArrange3D includes small objects, but does not cover thin sheets such as paper. Our bounding boxes represent object dimensions and spatial extent for placement, with different capabilities and limitations for these cases.
1) Small objects.
TopoBox predicts the source dimensions and uses them to construct placement boxes and determine surface sampling locations. Small rigid objects can therefore be represented according to their dimensions. Qualitative examples involving toy cars are provided in Appendix A.2.6 (Figure 9). These illustrate individual cases.
2) Thin and deformable objects.
The finest voxel resolution of 1 cm cannot reliably capture thin structures such as sheets of paper. Rigid bounding boxes also cannot describe shape changes in flexible materials.
We have explicitly discussed these limitations in Section 7 (Limitation) and identified more adaptive, detailed 3D representations for thin and deformable objects as future work.
Additional response to Q2: Robotic arm reachability and inverse kinematics.
A2: Thank you for raising the question of whether the robot can reach the predicted placement while holding the source object. Since our previous response,We have additionally conducted robot execution experiments in simulation to directly examine this issue.
We used a Panda robot arm in ManiSkill 3 across three scene configurations: open tables, smaller tables, and cluttered tables with multiple obstacles. For both TopoBox and RoboBrain, model predictions determine the placement targets. Both methods use the same execution procedure, with top-down grasping based on known object bounding boxes and MPLib motion planning. The placement model specifies the target, while the planner checks whether the robot can reach it and generates the motion required to transport the object.
A trial is successful only if the robot successfully grasps, transports, and releases the source object, and the resulting placement satisfies the instructed spatial relation, remains supported and stable, and involves no unintended collisions during execution. Planning failures, execution collisions, and unstable placements all count as failures. Reachability and motion feasibility therefore directly affect the reported execution success.
Results on 55 paired cases are as follows:
| Method | Successful executions | Success rate ↑ |
|---|---|---|
| TopoBox | 37/55 | 67.27% |
| RoboBrain | 28/55 | 50.91% |
TopoBox completes nine more tasks overall, improving execution success by 16.36 percentage points under the shared grasping and planning procedure. These results provide direct evidence that the predicted placements can support successful robot execution in the evaluated simulation settings.
We have also added Figure 14 in the revised appendix, which follows predicted placement targets through grasping and transport to stable states after release. Together, the execution results and qualitative examples extend the evaluation to whether the robot can physically realize the predicted placements.
Rebuttal by Authors of W1
We thank Reviewer mSwB for the careful reading, helpful comments, and positive assessment of our work. We appreciate your questions about the evaluation settings and the scope and limitations of our approach, which we address below.
W1: Yaw conventions and their influence on placement feasibility.
R1: Thank you for your comment. We clarify the baseline yaw convention and provide additional yaw** search experiments. These experiments show that orientation adjustment improves baseline feasibility, but does not fully explain the gap to TopoBox.
1) Yaw convention.
For point-based baselines, we place the ground-truth source box at the predicted 3D location while retaining the observed source yaw from the input scene. TopoBox predicts both the placement position and yaw. All methods use ground-truth source dimensions for SR and CFR evaluation. These evaluation conventions are specified in Appendix A.2.2 and A.2.4.
2) Effect of yaw search.
We keep each baseline position fixed and test yaw angles at 15° intervals over [0°, 180°), including the observed source yaw as an additional candidate. A placement passes PF if any tested orientation simultaneously satisfies SRC, SR, and CFR. TopoBox uses its predicted position and yaw without additional search.
Yaw search improves all three baselines. The strongest baseline, RoboBrain2.5, increases from 46.44% to 53.57% PF, but remains 11.59 percentage points below TopoBox. This confirms that orientation affects feasibility while showing that the original yaw convention alone does not account for the gap. We have added these results to Section 5.2 (Table 2) and the search protocol to Appendix A.2.3.
3) Role of the Reasoner.
A predicted position may leave insufficient space for the full object, so adjusting yaw alone cannot resolve support or collision failures. The Reasoner samples scene features around the current box and iteratively adjusts both position and yaw to reassess these constraints. The internal ablation provides complementary evidence: removing the TopoBox Reasoning Module reduces PF@1 from 65.16% to 60.49%. Position-related failure examples are provided in Section 5.2 (Figure 4) and Appendix A.2.6 (Figure 13).