acceptodds
Under review as a conference paper at ICLR 2027

CoDimRecon: Agentic Reconstruction of Sim-Ready 3D Scenes with Deformable Curves, Surfaces, and Volumes

Abstract

Reconstructing simulation-ready 3D scenes from real-world observations enables robotics, gaming, and immersive applications, yet existing methods largely assume rigid objects. This leaves an important gap for deformables, whose simulation-ready geometry depends on dimensionality—curves, surfaces, or volumes—and whose behavior may require models beyond elasticity. We present CoDimRecon, an agentic framework that reconstructs editable scenes containing rigid, articulated, and deformable objects from multi-view RGB observations. Scene-level geometric priors ground scale and layout, while object-level generated meshes guide the agent toward detailed, compact geometry; articulated rigid objects are decomposed into movable parts with explicit joints. For deformables, category-wise agent sessions reconstruct curves as centerlines with radii, surfaces as manifold shells with thickness, and volumes as watertight solids for volumetric meshing. Reusable simulator skills initialize compatible physical models and parameters, while agent-guided behavioral tests expose mismatches and trigger targeted revisions of motion, geometry, numerics, or material modeling. On evaluated Replica and ScanNet++ scenes, CoDimRecon achieves competitive compositional reconstruction accuracy while additionally producing deformable assets for rod, shell, and solid simulation. We further demonstrate robot interactions across all three representations, including a controlled paper-folding case in which behavioral testing motivates plastic bending.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Simulation-Ready Reconstruction, Physics-based Simulation, Agentic 3D Reconstruction, Interactive Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_t6HvOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

CoDimRecon is a training-free agentic framework for reconstructing editable, simulation-ready indoor scenes from multi-view RGB images. It combines geometric and generative references with agent-based scene authoring, and represents deformable objects as curves, shells, or volumetric solids. Simulated interactions guide revisions to geometry, physical models, and execution settings. Experiments evaluate scene reconstruction on Replica and ScanNet++, alongside deformable interaction demonstrations.

Critical strengths
  1. The framework addresses an important extension of scene reconstruction by supporting rigid, articulated, and deformable objects with representations appropriate for rod, shell, and solid simulation.

  2. The behavioral-feedback workflow goes beyond visual reconstruction, with detailed numerical checks and a controlled paper-folding experiment illustrating the distinction between simulation stability and intended physical behavior.

Critical weaknesses
  1. The contribution of agent refinement is insufficiently isolated. The direct GPT-6 Astra baseline and reference ablations do not establish how much improvement comes from the initialized scene versus subsequent agent editing. Comparisons with the unedited initialization, a generic agent using identical references, and alternative agent backbones are missing.

  2. Downstream robotics utility remains unverified. The interaction examples and numerical checks demonstrate simulation feasibility, but do not establish benefits for robot policy training or evaluation. Without controlled downstream experiments or comparisons against real interactions, the practical consequences of using uncalibrated physical parameters remain unclear.

  3. Behavioral verification lacks independent testing. Table 6 reports revisions to trajectories, tools, contact locations, and boundary conditions following failures. Although these adjustments may be reasonable, passing the revised tests does not establish robustness to other interactions. Evaluation under held-out manipulations with the final asset configuration fixed is needed.

  4. Evaluation scale and operational reliability are limited. Experiments cover six scenes, geometry ablations use one scene, and the category-specific session comparison is qualitative. Repeated end-to-end agent runs, failure rates, and reconstruction time and cost are not systematically reported, limiting assessment of robustness and practicality.

Questions for authors
  1. Could you evaluate the initialized scene before agent editing, and compare generic agent refinement against the proposed workflow using identical inputs, tools, and budgets? How do the results change with another agent backbone?

  2. Could you provide a controlled robotics experiment demonstrating that the reconstructed assets improve policy training or evaluation over simpler assets or default physical configurations? Quantitative comparisons with real interactions would be especially valuable for assessing the sim-to-real gap.

  3. After fixing the reconstructed geometry, material parameters, and boundary conditions, how reliably do the assets support previously untested contact locations, loads, and trajectories? Could you compare this performance with and without behavioral-feedback refinement?

  4. Could you extend the key ablations to more scenes and deformable objects, report variation across independent agent runs, and provide end-to-end runtime, model-call cost, and failure/retry statistics?

Convincing quantitative evidence of downstream robotics benefits, particularly through real-world validation, would substantially increase my confidence and could change my recommendation.

8 comments
AuthorsOpenReview ↗

Response to reviewer t6Hv - Part 1

Thank you for recognizing that our framework handles rigid, articulated, and deformable objects, including codimensional ones such as rods and shells, and that our behavioral feedback workflow goes beyond visual reconstruction to check physical behavior. We address your concerns below and have incorporated all corresponding changes into the revised paper.

Q1: Could you evaluate the initialized scene before agent editing, and compare generic agent refinement against the proposed workflow using identical inputs, tools, and budgets? How do the results change with another agent backbone?

Thank you for the suggestion. We refer to everything in CoDimRecon other than the agent backbone as the harness: scene initialization from geometric and generative references Sec\. 3\.1, the refinement environment Sec\. 3\.2, and the deformable workflow with its verification skills Sec\. 3\.3. The backbone runs inside the harness unmodified. Since the initialized scene is itself an output of the harness, we compare three settings:

  1. Init only: the initialized scene before any agent editing.

  2. Generic agent: the same backbone, references, and tools as ours, but without the harness and hence without our initialized scene. Unlike the RGB-only GPT-6 Astra baseline in Tab. 3, it receives all of our references.

  3. Ours: the full method.

Settings 2 and 3 have a $200 API-equivalent budget per run (agents were instructed to prepare to stop at USD 195), and both are run with Codex GPT-6 Astra and Claude Opus 5.5 (xhigh).

BackboneSettingCD ↓F1 ↑NC ↑PSNR ↑SSIM ↑LPIPS ↓Layout ↑Motion ↑Stable (Ground) ↑Stable (All) ↑Cost (USD)
—Init only37.6037.9566.008.080.5540.68583.3585.6986.9079.072.03
GPT-6 AstraGeneric agent30.5738.0175.9317.000.6790.33496.6898.0696.8085.10200.64
GPT-6 AstraOurs20.8349.8977.5117.390.6940.37596.1798.3197.3394.08200.75
Claude Opus 5.5Generic agent28.7345.9775.3010.270.5370.40378.1180.9396.7091.90201.19
Claude Opus 5.5Ours24.7547.2777.2516.890.6670.38396.0597.1895.5692.55200.08

Initialization vs. editing. With both backbones, agent editing improves on the initialized scene in every metric, so the final quality does not come from the initialization alone.

Harness vs. generic agent. At a matched budget, the harness yields better geometry than the generic agent with both backbones.

Effect of the backbone. The harness also makes the results less sensitive to the backbone. With Claude, the generic agent's rendering and latent-similarity scores drop sharply, whereas ours remain similar across the two backbones.

AuthorsOpenReview ↗

Response to reviewer t6Hv - Part 2

Q2: Could you provide a controlled robotics experiment demonstrating that reconstructed assets improve policy training or evaluation over simpler assets or default physical configurations? Quantitative comparisons with real interactions would be especially valuable for assessing the sim-to-real gap.

We agree that downstream robotics utility is important. However, our paper does not propose a policy-learning method, and we do not believe that training a policy is necessary to isolate the contribution of scene reconstruction. Policy-training performance would additionally depend on the policy architecture, learning algorithm, reward design, training budget, and domain randomization, making it difficult to determine whether differences arise from the reconstructed asset itself.

Instead, we added a more direct controlled real-to-sim experiment Sec\. 4\.2, Fig\. 6 that tests a necessary condition for using the reconstructed scene for policy evaluation: whether an interaction trajectory that succeeds in the real world produces the corresponding outcome when replayed in simulation.

Setup. A robot arm places a USB cable between two fixed blocks. We record a video of the scene using an iPhone, reconstruct it with CoDimRecon, and collect 50 successful episodes in which a human teleoperates the arm to complete the task. Before each replay, we initialize the simulated cable to the shape observed in the first frame of the corresponding real episode and replay the recorded joint trajectory open loop. We compare the default physical configuration E=1MPaE = 1 MPa with the configuration after behavioral verification (E = 9.3 MPa). BV (behavioral verification) uses only its own diagnostic tasks and never observes the 50 teleoperated trajectories, so these constitute held-out real interactions for both configurations.

RealW/o BV default,default, E=1MPa = 1 MPaW/ BV $$E=9.3MPa = 9.3 MPa<br>
Grasp success100% 50/5050/502% 1/501/5076% 38/5038/50
Placement success100% 50/5050/500% 0/500/5010% 5/505/50

Results. With the default configuration, the cable's shape after rest-shape settling departs markedly from the real cable, causing the gripper to miss it in all but one replay. After BV, the cable better maintains its real resting shape under gravity, and grasp success rises from 2% to 76%. Placement success increases from 0% to 10%, but remains far from the real outcome because BV improves the cable's resting behavior without fully matching its dynamics while it is being carried.

We believe this experiment provides a cleaner test of the reconstructed asset itself than end-to-end policy training: the robot trajectory is fixed, the real interaction provides the reference outcome, and the only comparison is between the default and behavior-verified physical configurations. The results demonstrate a substantial downstream benefit while also clearly exposing the remaining sim-to-real gap.

AuthorsOpenReview ↗

Response to reviewer t6Hv - Part 3

Q3: After fixing the reconstructed geometry, material parameters, and boundary conditions, how reliably do the assets support previously untested contact locations, loads, and trajectories? Could you compare this performance with and without behavioral-feedback refinement?

We thank the reviewer for this question. We ran a held-out evaluation on the trash bag new Appendix L of the revised paper: Tab\. 11 and Figs\. 19–20. For this evaluation, we re-ran behavioral refinement on the trash bag, which refined two of its parameters: friction, μ from 0.1 (simulator documentation) to 0.4, and film stiffness, E from 2.25 GPa to 22.5 MPa. Both refinements came from a single development grasp rim60°,45cmliftin35˙srim 60°, 45 cm lift in 3\.5 s. We then froze geometry, material parameters and boundary conditions, and ran configurations the agent never observed:

  • New contact locations: rim grasps at 0°, 120° and 180°.
  • New loads: a 0.2, 0.5 or 1.0 kg block inside the bag, grasped at 0°.

Every held-out run uses a 20 N grip and a 45 cm lift in 7.0 s, and each configuration was run once. "W/o refinement" reverts one parameter to its initial value and keeps the other refined, so the two refinements are measured separately.

Held-out conditionw/o refinementw/ refinement
Bag held, new contact points (μ 0.1 vs. 0.4)1/33/3
Bag held, payloads (μ 0.1 vs. 0.4)0/31/3
Bag drapes instead of keeping the bin shape, new contact points (E 2.25 GPa vs. 22.5 MPa)0/33/3

Contact locations. With μ = 0.1 the bag slips out of the gripper at 120° and 180°. This is the same failure the agent saw on the development grasp, and it prompted the friction refinement. With μ = 0.4 the bag is held at every new contact point. With the initial stiffness the lifted bag keeps the cylindrical shape of the bin, like a rigid shell, at all three points. After refinement it collapses and drapes like a plastic bag.

Loads. With μ = 0.1 the bag slips out under every payload. The refined bag carries 0.2 kg but slips at 0.5 and 1.0 kg. Refinement therefore widens the range of supported loads. However, μ was calibrated on an empty bag and does not cover heavier payloads; we now state this as a limitation Sec\. 5 and Appendix L.

Trajectories. Each new contact point changes the trajectory. The arm approaches from a different side in a different joint configuration, and it pulls the film up from a different side of the bin. The held-out lift is also 2× slower than the development lift.

Summary. Across the six distinct held-out configurations, the refined asset holds the bag in 4/6 runs, versus 1/6 without friction refinement. It also deforms plausibly at all three new contact points, versus 0/3 without stiffness refinement.

Reviewer_E1fROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoDimRecon turns multi-view RGB images of indoor scenes into editable assets for physics simulation. An agent combines geometric estimates with generated meshes, chooses representations for deformable curves, surfaces, and solids, and revises the assets after simulated robot interactions. The paper evaluates scene reconstruction on three Replica and three ScanNet++ scenes, with additional deformable-object examples and a controlled paper-folding test.

Critical strengths
  1. The main appeal for me is the support for rods, shells, and solids within the same scene. The paper-folding test shows how a behavior check can lead to a specific material-model revision. This makes the pipeline useful for creating assets that can actually participate in simulated interactions.
  2. The scene-level results are encouraging. Table 3 shows competitive geometry and strong rigid-body stability on both Replica and ScanNet++. These results give the scene-authoring contribution practical value, even though the deformable-behavior claim needs stronger validation.
Critical weaknesses
  1. My main concern is the lack of validation against real deformation. The examples show that the assets run in simulation, but I cannot tell how closely they reproduce the actual objects under matched actions. The folding test compares two simulator models without a measured real response. This leaves the paper's distinctive claim about reconstructing deformable behavior only partly supported.
  2. I also find it hard to isolate the benefit of the agentic design. Table 5 uses one scene, and the separate-session cable comparison is qualitative. ReplicateAnyScene was reimplemented and tuned per scene, with different mask sources used for quantitative and some qualitative results. Together, these limitations make the reported gains less convincing as evidence for the proposed components.

Missing references:

  • AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation. (SIGGRAPH 2026)
  • Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model. (ICLR 2025)
Questions for authors
  1. Could you clarify whether any real deformation measurements were used to validate the assets under matched actions? If none are available, should the main claim be limited to creating simulation-compatible assets?
  2. Is there evidence beyond the single-scene ablation and qualitative cable example that isolates the reference prior and agent-session design? Results across several objects or scenes would materially strengthen my assessment of these components.
  3. What was tuned for ReplicateAnyScene on each scene, and which masks were used in each quantitative comparison? Could you clarify whether the conclusions depend on these choices?
6 comments
AuthorsOpenReview ↗

Response to reviewer E1fR - Part 1

Thank you for recognizing our unified support for codimensional objects (rods, shells, and solids) within a single scene, and for finding our pipeline useful for creating assets that can actually participate in simulated interactions. We also appreciate the positive assessment of our scene-level results, including the competitive geometry and strong rigid-body stability in Table 3. We address your concerns below and have incorporated all corresponding changes into the revised paper.

Q1: Could you clarify whether any real deformation measurements were used to validate the assets under matched actions? If none are available, should the main claim be limited to creating simulation-compatible assets?

We agree that our method does not quantitatively calibrate material parameters against measured real deformations. However, we respectfully disagree that this means the contribution should be limited to merely producing simulation-compatible assets. Quantitative material system identification is a different problem and is generally underdetermined from our multi-view RGB inputs: visually similar objects can have substantially different stiffness, friction, damping, and other physical properties. Our goal is therefore not to reconstruct a quantitatively calibrated digital twin, but to construct simulation-ready assets whose physical behavior can be executed, diagnosed, and improved through simulation.

To evaluate this intended capability against the real world, we added a new real-to-sim experiment Sec\. 4\.2, Fig\. 6 using 50 held-out robot interactions. We record 50 successful real teleoperated cable-placement episodes and replay the same joint trajectories open loop in the reconstructed scene. Importantly, behavioral verification BVBV never observes these trajectories. With the default physical configuration, the simulated robot grasps the cable in only 1/50 episodes 22% and completes placement in 0/50. After BV, grasp success increases to 38/50 7676% and placement success to 5/50 1010%.

Thus, although BV does not recover physically calibrated material parameters, it substantially improves agreement with previously unseen real interactions. At the same time, the low placement success shows that a substantial sim-to-real gap remains, and we now state this limitation explicitly Sec\. 5, Limitations.

We therefore use simulation-ready to mean that the reconstructed assets can be directly loaded, simulated, interacted with, and behaviorally refined, rather than claiming that they quantitatively reproduce the real object's material response. This usage is also consistent with prior sim-ready reconstruction work [1,2], which does not calibrate all physical parameters from real measurements. We have clarified this scope in the revised paper Sec\. 5, Limitations.

[1] Xia, Hongchi, et al. "HoloScene: Simulation‑Ready Interactive 3D Worlds from a Single Video." NeurIPS 2025.

[2] Xia, Chong, et al. "SimRecon: SimReady Compositional Scene Reconstruction from Real Videos." CVPR 2026.

AuthorsOpenReview ↗

Response to reviewer E1fR - Part 2

Q2: Is there evidence beyond the single-scene ablation and qualitative cable example that isolates the reference prior and agent-session design?

In addition to the single-scene ablation in Table 5, we ran the same ablation on two more scenes, ScanNet++ acd69a1746 and Replica room_0 Appendix K, Tab\. 10. Besides removing individual components, we also remove the last two pipeline stages cumulatively: w/o deformables means disable deformable stage (deformable reconstruction and behavior verification), and w/o articulation means disable articulation.

SceneVariantCD ↓F1 ↑NC ↑
ScanNet++ acd69a1746Full10.2249.3677.27
w/o geometric refs (VGGT)10.9141.3673.70
w/o generative refs (SAM 3D)10.7941.5975.76
w/o RR10.2844.0076.93
w/o deformables10.2444.3377.18
w/o articulation10.1650.7379.04
Replica room_0Full6.0164.2584.09
w/o geometric refs (VGGT)6.2850.2679.06
w/o generative refs (SAM 3D)6.5554.7983.64
w/o RR6.0264.1583.17
w/o deformables6.0363.8783.71
w/o articulation5.8265.1386.92

Removing either reference prior (VGGT or SAM 3D) causes the largest CD increases and F1 drops on both scenes.

Adding the deformable stage changes each metric on room_0 by at most 0.38; on acd69a1746, it raises F1 by 5.0 points while CD and NC change by at most 0.09. Deformable objects occupy a small fraction of the scene surface, so their effect on surface metrics is limited. We also provide more qualitative comparisons of category-wise deformable object reconstruction in Fig. 12 of the revised paper.

Adding articulation worsens all three metrics on both scenes, and this is expected. Articulated reconstruction adds geometry that is not visible in the captured surface, such as drawer interiors and the occluded parts of doors Fig\. 18. The ground-truth meshes contain only the observed surfaces eg˙,˙FAROlaserscansforScanNet+\+e\.g\., FARO laser scans for ScanNet+\+. Points sampled from this hidden geometry have no ground-truth counterpart, which increases CD and lowers F1 and NC. Surface metrics therefore cannot capture the benefit of articulation, which is to enable interaction in simulation.

More importantly, we now directly isolate the contribution of the agent workflow through the controlled comparison requested by Reviewer t6Hv. We compare Init only, a Generic agent, and Ours Sec\. 4\.3 and Appendix I, Tab\. 7. The generic agent receives the same references and tools as ours and the same $200 API-equivalent budget, but does not use our structured harness. We repeat this experiment with both GPT-6 Astra and Claude Opus 5.5. Agent editing improves over initialization alone, showing that the final result is not explained by the initialized scene. At the same time, the full harness achieves better geometry than the generic agent with both backbones and is less sensitive to the choice of backbone. These experiments separate the contributions of reference-based initialization and subsequent structured agent refinement more directly than the original single-scene ablation.

BackboneSettingCD ↓F1 ↑NC ↑PSNR ↑SSIM ↑LPIPS ↓Layout ↑Motion ↑Stable (Ground) ↑Stable (All) ↑Cost (USD)
—Init only37.6037.9566.008.080.5540.68583.3585.6986.9079.072.03
GPT-6 AstraGeneric agent30.5738.0175.9317.000.6790.33496.6898.0696.8085.10200.64
GPT-6 AstraOurs20.8349.8977.5117.390.6940.37596.1798.3197.3394.08200.75
Claude Opus 5.5Generic agent28.7345.9775.3010.270.5370.40378.1180.9396.7091.90201.19
Claude Opus 5.5Ours24.7547.2777.2516.890.6670.38396.0597.1895.5692.55200.08
AuthorsOpenReview ↗

Response to reviewer E1fR - Part 3

Q3: What was tuned for ReplicateAnyScene on each scene, and which masks were used in each quantitative comparison? Could you clarify whether the conclusions depend on these choices?

Tuning. The official ReplicateAnyScene release omits the pose-alignment and VLM relation-reasoning modules described in its paper, so we reimplemented both AppendixD1˙Appendix D\.1. We first validated our reimplementation on the hallway example scene released with ReplicateAnyScene, where it reproduces the results shown in the original paper Fig\. 11. We did not tune it per scene. We adjusted only two settings on this hallway scene and kept them fixed for all evaluated scenes:

  • MASt3R registration: each frame is matched with 3 neighboring frames on each side (7 frames in total) instead of the default 2 (5 frames in total).
  • VGGT camera estimation: we use all input frames of the scene instead of the default 300.

We have stated these settings in Appendix D.1.

Masks. As stated in the paper, the two comparisons use different masks for ReplicateAnyScene:

  • Quantitative comparison (Table 3): ReplicateAnyScene uses its default segmentation pipeline, in which a VLM lists the objects in sampled frames, SAM 3 segments them from these names, and SAM 3-Video propagates the masks. Our method uses its own view-consistent masks (CropFormer segmentation followed by mask clustering). HoloScene receives the same masks as our method.

  • Qualitative comparisons Fig\. 3 and Appendix H: To avoid underestimating the baseline, ReplicateAnyScene receives ground-truth instance masks, which are more accurate than both its default masks and our predicted masks. Our method keeps its own predicted CropFormer masks.

Dependence of the conclusions. On our scenes, ReplicateAnyScene's default VLM + SAM 3 segmentation misses many objects, so part of its gap to our method in Table 3 may come from segmentation. Two comparisons show that our advantage does not come from segmentation alone:

  1. In Table 3, HoloScene receives exactly the same masks as our method, together with the same cameras and depth. Our method still achieves lower CD and higher NC on both datasets.

  2. In the qualitative comparisons, ReplicateAnyScene receives ground-truth masks, which are more accurate than ours, and our method still produces more complete and simulation-ready scenes. For example, ReplicateAnyScene omits the doors and the poster in Fig. 3.

Reviewer_j4BUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the problem of reconstructing scenes with multiple objects from multiple images. This involves not only estimating the geometry of each object, but also estimating parts and functions where needed (such as drawers), as well as the material behavior of deformable objects. To solve this problem, the paper proposes an agentic approach with a pipeline that defines what the agents should do to accurately recover functionality, material behavior, and geometry from images. This pipeline covers how to reconstruct and refine scenes, how to articulate objects, and how to reconstruct and simulate deformable objects while remaining faithful to the input information. Through experiments, the paper shows that the proposed method is competitive in reconstruction, while additionally making scenes simulation-ready by including articulated and deformable objects.

Critical strengths

The paper takes a step toward reconstructing scenes that also account for the parts of each object, physical stability, and materials. This brings scene representations much closer to being accurate and simulation-ready. In addition, the experiments clearly show the benefit of covering these aspects in scene representations.

Critical weaknesses

Since this is an agentic system, it is important to show stronger evidence that this relatively complex pipeline is needed, rather than letting the agents figure out how to perform the task themselves. However, this part is weak in the paper.

For example, compared with Astra, the proposed method has access to additional tools, such as multiple pretrained models, while Astra is given only Blender and the images. Considering that Astra's performance is also competitive with the proposed method, the need for the proposed pipeline should be shown more clearly. For example, Astra could be given access to the same pretrained models as the proposed method, and then the two could be compared.

In addition, a single run with a fixed seed may not be enough to show that the proposed method works robustly. Also, it is unclear whether the problem is not solvable by improving the prompts.

Questions for authors

Would it be possible to provide stronger results or analysis showing why the proposed pipeline is needed for agentic systems?

4 comments
AuthorsOpenReview ↗

Response to reviewer j4BU - Part 1

Thank you for recognizing that our method produces accurate, simulation-ready scenes that account for object parts, physical stability, and materials, and that our experiments validate these benefits. We address your concerns below and have incorporated all corresponding changes into the revised paper.

Q1: Is the CoDimRecon workflow necessary? Could an agent given the same tools and pretrained models organize the pipeline itself and achieve comparable results?

Yes, the workflow is necessary. To verify this, we focus on the reconstruction part of CoDimRecon, where CoDimRecon builds a structured harness to provide an initialized scene for the agent. Here, we remove CoDimRecon's workflow that builds this initial scene, but keep the same tools and pretrained models and ask the agent to organize them itself Appendix I, Tab\. 8 of the revised paper.

SettingsCD ↓F1 ↑NC ↑Cost (USD)
Ours w/o Workflow29.2741.9777.6354.01
Ours25.6254.8075.2913.01

On ScanNet++, CoDimRecon improves CD and F1, and achieves comparable NC. The key difference is that CoDimRecon organizes the outputs of pretrained models into a coordinate-consistent, editable scene state. Subsequent agents can then fix specific geometric errors based on multi-view observations, with far fewer downstream model calls and less context overhead.

We further isolate the contribution of the workflow in the controlled comparison requested by Reviewer t6Hv. We compare Init only, a Generic agent, and Ours under a matched $200 API-equivalent budget Sec\. 4\.3 and Appendix I, Tab\. 7. The generic agent receives the same references and tools as ours but does not use our structured harness, and we repeat the comparison with both GPT-6 Astra and Claude Opus 5.5. With both backbones, our full harness achieves better geometry than the generic agent and is less sensitive to the choice of backbone. Together, these results show that access to the tools and pretrained models alone is not sufficient; the structured workflow itself is important for accuracy, efficiency, and robustness.

BackboneSettingCD ↓F1 ↑NC ↑PSNR ↑SSIM ↑LPIPS ↓Layout ↑Motion ↑Stable (Ground) ↑Stable (All) ↑Cost (USD)
—Init only37.6037.9566.008.080.5540.68583.3585.6986.9079.072.03
GPT-6 AstraGeneric agent30.5738.0175.9317.000.6790.33496.6898.0696.8085.10200.64
GPT-6 AstraOurs20.8349.8977.5117.390.6940.37596.1798.3197.3394.08200.75
Claude Opus 5.5Generic agent28.7345.9775.3010.270.5370.40378.1180.9396.7091.90201.19
Claude Opus 5.5Ours24.7547.2777.2516.890.6670.38396.0597.1895.5692.55200.08

Q2: Is a single run with a fixed seed sufficient to demonstrate robustness?

We run CoDimRecon 3 times on the ScanNet++ dataset Appendix J, Tab\. 9. Results of these runs are quite similar, which validates the robustness of our method.

CD ↓F1↑NC ↑PSNR ↑(dB)SSIM ↑LPIPS ↓Layout ↑Motion ↑Stable(Ground) ↑Stable(All) ↑
Run 121.3541.9475.1116.680.8570.37395.5297.0997.0596.31
Run 221.8540.2174.4916.980.6830.32394.7696.27100.0095.01
Run 321.0843.9373.9217.950.7160.28695.4897.2298.6397.66
mean21.4342.0374.5117.200.7520.32795.2596.8698.5696.33
std0.391.860.600.660.0920.0440.430.521.481.33
AuthorsOpenReview ↗

Response to reviewer j4BU - Part 2

Q3: It is unclear whether the problem is not solvable by improving the prompts.

We agree that prompt design matters, and prompts are indeed one component of our harness Sec\. 3. However, the core difficulty we address cannot be resolved by prompting alone, because it is a problem of missing information rather than missing instructions. Whether an asset penetrates, becomes numerically unstable, or responds with incorrect stiffness under contact can only be observed after the asset is executed in simulation. A prompt can describe what a physically valid asset should satisfy, but it cannot tell the model whether the asset it has just produced actually behaves correctly. Simulation-based behavioral feedback in our harness provides exactly this signal; The prompts only instruct the agent on how to interpret it.

The new matched-budget generic-agent experiment in Q1 Appendix I, Tab\. 7 provides additional evidence. The generic agent receives the same references, tools, and computational budget as ours and is free to decide how to use them, but without our structured harness. Our method achieves better geometry with both GPT-6 Astra and Claude Opus 5.5 and is less sensitive to the backbone. Thus, the improvement cannot be explained simply by access to the same tools or by prompting an otherwise equivalent agent; the structured scene state, workflow, and simulation-based feedback provided by the harness are important components of the method.

Thank you for the detailed rebuttal and addition experiments, which resolved some of my concerns.

My remaining concern, which Reviewer t6Hv also raised, is about the clarity of the findings rather than the results themselves. The performance improvement is clear, but many factors are still entangled, so it is difficult to tell which design choices lead to performance improvement and how they interact. Clarifying this would make the findings much easier for follow-up work to adopt and build on.

I therefore would like to keep my initial score.

Thank you for the follow-up. We are glad that the additional experiments resolved part of your concerns and that the performance improvement itself is now clear. As we understand it, your remaining concern—which Reviewer t6Hv also raised—is less about the overall improvement than about the insight our work offers follow-up research: which design choices contribute to the improvement, and how they interact.

To address this, we ran the control that Reviewer t6Hv requested to isolate the contribution of agent refinement: a generic agent that starts from our initialized scene (Init + Generic agent). All agent settings below run on ScanNet++ with the same backbone (GPT-6 Astra), references, tools, and pretrained models as ours, and with the same USD 200 budget.

SettingInitialized sceneRefinement stages (Secs. 3.2–3.3)CD ↓F1 ↑NC ↑PSNR ↑SSIM ↑LPIPS ↓Layout ↑Motion ↑Stable (Ground) ↑Stable (All) ↑Cost (USD)
Init only (no agent)✓—37.6037.9566.008.080.5540.68583.3585.6986.9079.072.03
Generic agent——30.5738.0175.9317.000.6790.33496.6898.0696.8085.10200.64
Init + Generic agent✓—23.7744.1270.6616.440.6530.38196.1497.6999.4287.02200.95
Ours✓✓20.8349.8977.5117.390.6940.37596.1798.3197.3394.08200.75

Our method achieves the best geometry (CD, F1, and NC) and the highest overall physical stability (Stable (All)) across all settings. Together with our broader ablation suite (which separately removes VGGT, SAM 3D, articulation, and deformable reconstruction across additional scenes), these controlled comparisons isolate the key design choices of our method and show how they complement each other:

  1. Contribution of the initialized scene (Generic agent vs. Init + Generic agent): Constructing a structured scene from geometric and generative references is itself a core component of our harness. Starting from our initialized scene, the same generic agent improves CD by 22.2% and boosts F1 by 16.1%, demonstrating that the initialization provides an indispensable geometric reference that an unconstrained agent struggles to organize on its own.

  2. Contribution of structured refinement over an unconstrained generic agent (Init only & Init + Generic agent vs. Ours): While initialization provides a strong geometric starting point, unconstrained agent edits can degrade surface normals (NC drops from 75.93 to 70.66) and leave non-grounded objects physically unstable (Stable (All) at 87.02%). By checking and correcting edits against multi-view observations and physics simulation (Secs. 3.2–3.3), our refinement stages substantially improve geometry, rendering quality, and overall physical stability over both Init only and Init + Generic agent.

We hope these comparisons clarify the individual contributions and interaction of our initialization and refinement stages. If you feel any specific aspect is still not isolated, or if an additional ablation over a particular combination of initialization, refinement, references, and harness components would resolve any remaining ambiguity, we would greatly appreciate knowing which specific combination you have in mind.

Comments on the paper

1 comment

Dear AC and Reviewers,

Thank you for your time, thoughtful feedback, and active engagement during the discussion period.

We are deeply encouraged that all three reviewers recognized the core contributions of our work—specifically, bridging scene reconstruction and physics simulation (j4BU, t6Hv), unifying rods, shells, and solids within a single scene (E1fR, t6Hv), going beyond visual appearance via behavioral verification (E1fR, t6Hv), and achieving strong geometry and rigid-body stability (E1fR).

We have carefully addressed all comments and updated the manuscript accordingly (new additions in blue, corrections in dark red). Key updates include:

  • Ablation Studies: Isolated scene initialization, refinement, and deformable reconstruction, validating each module's distinct contribution.
  • Generalization to Unseen Interactions: Tested behavioral verification (BV) on novel scenarios:
    • Real-to-sim: Replaying 50 real teleoperated cable-placement episodes open-loop, BV raises grasp success from 2% to 76%.
    • Held-out contacts: With geometry and physics frozen, friction refinement improves trash bag retention from 1/6 to 4/6 unseen configurations.
  • Stability & Cost: Added 3-run statistics (mean ± std) on ScanNet++, end-to-end runtime, and API cost.
  • Discussion: Expanded related work, limitations, and failure cases.

Detailed responses are provided in each individual thread.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.