- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
CoDimRecon is a training-free agentic framework for reconstructing editable, simulation-ready indoor scenes from multi-view RGB images. It combines geometric and generative references with agent-based scene authoring, and represents deformable objects as curves, shells, or volumetric solids. Simulated interactions guide revisions to geometry, physical models, and execution settings. Experiments evaluate scene reconstruction on Replica and ScanNet++, alongside deformable interaction demonstrations.
Critical strengths
-
The framework addresses an important extension of scene reconstruction by supporting rigid, articulated, and deformable objects with representations appropriate for rod, shell, and solid simulation.
-
The behavioral-feedback workflow goes beyond visual reconstruction, with detailed numerical checks and a controlled paper-folding experiment illustrating the distinction between simulation stability and intended physical behavior.
Critical weaknesses
-
The contribution of agent refinement is insufficiently isolated. The direct GPT-6 Astra baseline and reference ablations do not establish how much improvement comes from the initialized scene versus subsequent agent editing. Comparisons with the unedited initialization, a generic agent using identical references, and alternative agent backbones are missing.
-
Downstream robotics utility remains unverified. The interaction examples and numerical checks demonstrate simulation feasibility, but do not establish benefits for robot policy training or evaluation. Without controlled downstream experiments or comparisons against real interactions, the practical consequences of using uncalibrated physical parameters remain unclear.
-
Behavioral verification lacks independent testing. Table 6 reports revisions to trajectories, tools, contact locations, and boundary conditions following failures. Although these adjustments may be reasonable, passing the revised tests does not establish robustness to other interactions. Evaluation under held-out manipulations with the final asset configuration fixed is needed.
-
Evaluation scale and operational reliability are limited. Experiments cover six scenes, geometry ablations use one scene, and the category-specific session comparison is qualitative. Repeated end-to-end agent runs, failure rates, and reconstruction time and cost are not systematically reported, limiting assessment of robustness and practicality.
Questions for authors
-
Could you evaluate the initialized scene before agent editing, and compare generic agent refinement against the proposed workflow using identical inputs, tools, and budgets? How do the results change with another agent backbone?
-
Could you provide a controlled robotics experiment demonstrating that the reconstructed assets improve policy training or evaluation over simpler assets or default physical configurations? Quantitative comparisons with real interactions would be especially valuable for assessing the sim-to-real gap.
-
After fixing the reconstructed geometry, material parameters, and boundary conditions, how reliably do the assets support previously untested contact locations, loads, and trajectories? Could you compare this performance with and without behavioral-feedback refinement?
-
Could you extend the key ablations to more scenes and deformable objects, report variation across independent agent runs, and provide end-to-end runtime, model-call cost, and failure/retry statistics?
Convincing quantitative evidence of downstream robotics benefits, particularly through real-world validation, would substantially increase my confidence and could change my recommendation.
8 comments
Response to reviewer t6Hv - Part 2
Q2: Could you provide a controlled robotics experiment demonstrating that reconstructed assets improve policy training or evaluation over simpler assets or default physical configurations? Quantitative comparisons with real interactions would be especially valuable for assessing the sim-to-real gap.
We agree that downstream robotics utility is important. However, our paper does not propose a policy-learning method, and we do not believe that training a policy is necessary to isolate the contribution of scene reconstruction. Policy-training performance would additionally depend on the policy architecture, learning algorithm, reward design, training budget, and domain randomization, making it difficult to determine whether differences arise from the reconstructed asset itself.
Instead, we added a more direct controlled real-to-sim experiment Sec\. 4\.2, Fig\. 6 that tests a necessary condition for using the reconstructed scene for policy evaluation: whether an interaction trajectory that succeeds in the real world produces the corresponding outcome when replayed in simulation.
Setup. A robot arm places a USB cable between two fixed blocks. We record a video of the scene using an iPhone, reconstruct it with CoDimRecon, and collect 50 successful episodes in which a human teleoperates the arm to complete the task. Before each replay, we initialize the simulated cable to the shape observed in the first frame of the corresponding real episode and replay the recorded joint trajectory open loop. We compare the default physical configuration with the configuration after behavioral verification (E = 9.3 MPa). BV (behavioral verification) uses only its own diagnostic tasks and never observes the 50 teleoperated trajectories, so these constitute held-out real interactions for both configurations.
| Real | W/o BV E | W/ BV $$E<br> | |
|---|---|---|---|
| Grasp success | 100% | 2% | 76% |
| Placement success | 100% | 0% | 10% |
Results. With the default configuration, the cable's shape after rest-shape settling departs markedly from the real cable, causing the gripper to miss it in all but one replay. After BV, the cable better maintains its real resting shape under gravity, and grasp success rises from 2% to 76%. Placement success increases from 0% to 10%, but remains far from the real outcome because BV improves the cable's resting behavior without fully matching its dynamics while it is being carried.
We believe this experiment provides a cleaner test of the reconstructed asset itself than end-to-end policy training: the robot trajectory is fixed, the real interaction provides the reference outcome, and the only comparison is between the default and behavior-verified physical configurations. The results demonstrate a substantial downstream benefit while also clearly exposing the remaining sim-to-real gap.
Response to reviewer t6Hv - Part 3
Q3: After fixing the reconstructed geometry, material parameters, and boundary conditions, how reliably do the assets support previously untested contact locations, loads, and trajectories? Could you compare this performance with and without behavioral-feedback refinement?
We thank the reviewer for this question. We ran a held-out evaluation on the trash bag new Appendix L of the revised paper: Tab\. 11 and Figs\. 19–20. For this evaluation, we re-ran behavioral refinement on the trash bag, which refined two of its parameters: friction, μ from 0.1 (simulator documentation) to 0.4, and film stiffness, E from 2.25 GPa to 22.5 MPa. Both refinements came from a single development grasp . We then froze geometry, material parameters and boundary conditions, and ran configurations the agent never observed:
- New contact locations: rim grasps at 0°, 120° and 180°.
- New loads: a 0.2, 0.5 or 1.0 kg block inside the bag, grasped at 0°.
Every held-out run uses a 20 N grip and a 45 cm lift in 7.0 s, and each configuration was run once. "W/o refinement" reverts one parameter to its initial value and keeps the other refined, so the two refinements are measured separately.
| Held-out condition | w/o refinement | w/ refinement |
|---|---|---|
| Bag held, new contact points (μ 0.1 vs. 0.4) | 1/3 | 3/3 |
| Bag held, payloads (μ 0.1 vs. 0.4) | 0/3 | 1/3 |
| Bag drapes instead of keeping the bin shape, new contact points (E 2.25 GPa vs. 22.5 MPa) | 0/3 | 3/3 |
Contact locations. With μ = 0.1 the bag slips out of the gripper at 120° and 180°. This is the same failure the agent saw on the development grasp, and it prompted the friction refinement. With μ = 0.4 the bag is held at every new contact point. With the initial stiffness the lifted bag keeps the cylindrical shape of the bin, like a rigid shell, at all three points. After refinement it collapses and drapes like a plastic bag.
Loads. With μ = 0.1 the bag slips out under every payload. The refined bag carries 0.2 kg but slips at 0.5 and 1.0 kg. Refinement therefore widens the range of supported loads. However, μ was calibrated on an empty bag and does not cover heavier payloads; we now state this as a limitation Sec\. 5 and Appendix L.
Trajectories. Each new contact point changes the trajectory. The arm approaches from a different side in a different joint configuration, and it pulls the film up from a different side of the bin. The held-out lift is also 2× slower than the development lift.
Summary. Across the six distinct held-out configurations, the refined asset holds the bag in 4/6 runs, versus 1/6 without friction refinement. It also deforms plausibly at all three new contact points, versus 0/3 without stiffness refinement.
Response to reviewer t6Hv - Part 1
Thank you for recognizing that our framework handles rigid, articulated, and deformable objects, including codimensional ones such as rods and shells, and that our behavioral feedback workflow goes beyond visual reconstruction to check physical behavior. We address your concerns below and have incorporated all corresponding changes into the revised paper.
Thank you for the suggestion. We refer to everything in CoDimRecon other than the agent backbone as the harness: scene initialization from geometric and generative references Sec\. 3\.1, the refinement environment Sec\. 3\.2, and the deformable workflow with its verification skills Sec\. 3\.3. The backbone runs inside the harness unmodified. Since the initialized scene is itself an output of the harness, we compare three settings:
Init only: the initialized scene before any agent editing.
Generic agent: the same backbone, references, and tools as ours, but without the harness and hence without our initialized scene. Unlike the RGB-only GPT-6 Astra baseline in Tab. 3, it receives all of our references.
Ours: the full method.
Settings 2 and 3 have a $200 API-equivalent budget per run (agents were instructed to prepare to stop at USD 195), and both are run with Codex GPT-6 Astra and Claude Opus 5.5 (
xhigh).Initialization vs. editing. With both backbones, agent editing improves on the initialized scene in every metric, so the final quality does not come from the initialization alone.
Harness vs. generic agent. At a matched budget, the harness yields better geometry than the generic agent with both backbones.
Effect of the backbone. The harness also makes the results less sensitive to the backbone. With Claude, the generic agent's rendering and latent-similarity scores drop sharply, whereas ours remain similar across the two backbones.