acceptodds
Under review as a conference paper at ICLR 2027

Aster: Architecting Sim-Ready Worlds from Real Scenes for Embodied AI

Abstract

Embodied AI increasingly relies on simulation for training and evaluation, creating a growing demand for scalable Real2Sim pipelines that transform real-world observations into simulation-ready environments. However, existing 3D reconstruction and digital twin approaches primarily optimize for visual fidelity, often producing unstructured meshes or cluttered scenes that are difficult to deploy in embodied simulators. We argue that Real2Sim should be formulated as a simulation-ready world generation problem rather than a geometry reconstruction problem. To this end, we present ASTER, a framework for architecting sim-ready worlds from real scenes for embodied AI. ASTER integrates simulation-oriented world understanding, world construction, and physics-based refinement to recover structured scene layouts and object assets, predict and assign physically plausible properties, and use simulation feedback to improve scene stability and executability. This design enables efficient world generation without extensive per-scene rendering. Evaluation of reconstructed scenes shows fewer collisions and higher stability, while policies trained in ASTER worlds achieve similar performance in matched ground-truth simulation scenes. In a challenging real-robot setting, adding ASTER-generated data increased observed real-robot success from 0/6 to 4/6 trials, providing a cheap way to scale up data for real-world robot deployment.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Real2Sim, Embodied AI, Synthetic Data

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sWUVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ASTER turns indoor point clouds into simulation-ready worlds. It combines SpatialLM-based layout and physical-attribute prediction, progressive object generation with ShapeR, and simulator-based refinement. Evaluation includes reconstruction and stability metrics, policy transfer across three matched simulation scenes, and a small real-robot experiment using additional synthetic data.

Critical strengths
  1. The pipeline recovers room structure and separate object assets for simulation. This is useful for embodied data generation, and the reported 5.5-minute reconstruction example is encouraging.

  2. Table 4 shows clear improvements over SpatialLM + fine-tuned ShapeR: collision decreases from 25.8 to 18.0, and stability increases from 69.9 to 78.6. Figure 5 also checks policy performance, giving useful evidence beyond reconstruction quality.

Critical weaknesses
  1. The benefit of predicted physical attributes is unclear. Appendix B measures agreement with VLM labels, while refinement can make unstable objects static. A comparison with simple parameter defaults and separate results for movable objects would help establish whether the method improves useful interactions.

  2. The downstream evidence is promising but limited. Figure 5 uses three synthetic scenes, and Figure 6 reports one workspace with six hard trials. In the real study, task objects are added separately, and there is no matched synthetic-data baseline. This makes it difficult to identify the contribution of ASTER’s proposed stages.

Questions for authors
  1. Clarification: What fraction of objects become static during refinement, and what is the stability score for objects that remain movable?

  2. Requested evidence: With the same refinement, how do category-default mass and friction compare with the predicted attributes?

  3. Requested evidence: With equal amounts of synthetic training data, how does ASTER compare with SpatialLM + fine-tuned ShapeR using the same task objects? Does the real-robot improvement hold across repeated training runs?

7 comments
AuthorsOpenReview ↗

Critical Weaknesses

Thank you for the thoughtful review. We add controlled downstream comparisons, physical-prediction results, and five-task real-world evaluation to address the concerns below.

W1. Physical attributes and useful interactions

“The benefit of predicted physical attributes is unclear.”

The main measured benefit of joint physical prediction is improved mobility prediction and scene configuration. We compare category-default mass/friction with mobility-only prediction against joint mass/friction/mobility prediction.

Metric (%)Defaults + mobility-onlyJoint prediction
Mobility agreement ↑93.598.9
Collision ↓22.118.0
Stability ↑75.978.6
Pick & Place success54.154.3
Dish Stacking success49.349.3
Object Pushing success76.076.0

Joint prediction improves mobility agreement by 5.4 percentage points, evaluated against the same VLM annotations on the same held-out set and matched objects. Mass/friction prediction provides additional physical supervision for identifying fixed and movable objects.

Task success is nearly unchanged between these configurations. The principal observed difference in the manipulation scenes is a painting falling far from the robot in Scene 3, affecting scene stability much more than task execution in the robot workspace. The downstream comparison in W2 evaluates the complete pipeline's effect on useful robot interactions.

“Refinement can make unstable objects static” and “separate results for movable objects.”

We distinguish two stages. Initial model assignments classify 15,425/65,003 objects (23.7%) as static across 10,514 valid rooms, mainly ceiling lights and other fixed elements. Refinement subsequently converts 7.2% of evaluated objects, mainly generated large furniture with uneven bases, especially bookshelves. These corrections include normally movable furniture whose generated geometry prevents stable placement.

After excluding objects that should be fixed, rooms assembled from the original outputs achieve 98.7% stability and 3.2% collision for the remaining objects. This group-specific result is reported separately from overall scene stability. Most remaining overlaps arise from small bounding-box errors.

W2. Downstream evidence and matched-data baselines

“There is no matched synthetic-data baseline,” making the contribution of ASTER's stages difficult to identify.

We add SpatialLM + fine-tuned ShapeR using the same scenes, task objects, demonstration budgets, and training conditions. Policies trained in the reconstructed environments are evaluated in the same GT scenes.

SceneComponent baselineASTER
158 / 47 / 7256 / 49 / 71
20 / 0 / 0†47 / 39 / 77
325 / 7 / 060 / 60 / 80
Overall mean23.2%59.9%

Entries report Pick & Place / Dish Stacking / Object Pushing success (%).

† A ceiling light falls onto the table in Scene 2, preventing baseline training; this counts as zero end-to-end success. In Scene 3, table–chair intersections displace the table and reduce success.

ASTER improves overall success by 36.7 percentage points. Across Scenes 1 and 3, where baseline training is possible, success remains 62.7% versus 34.8%. The advantage therefore persists among trainable scenes and demonstrates the value of constructing reliable robot-training environments at the same data budget.

“Figure 6 reports one workspace with six hard trials.”

We expand real-world evaluation to five tasks and 50 trajectory executions. A robot VLM agent generates ten successful simulated trajectories per task in the reconstructed laboratory. We instantiate the corresponding real tasks and strictly replay the trajectories.

TaskReal successes
Bowl stacking8/10
Block grasping10/10
Drawer opening7/10
Color matching of cups and plates8/10
Package grasping8/10
Total41/50 (82%)

The matched-budget experiment evaluates the reconstructed training environments; the real-world experiment evaluates direct transfer of successful simulated trajectories. Together, they demonstrate environment usability and reuse across task types. Physical experiments use the laboratory where our robot arm is available.

“Task objects are added separately.”

This enables varied object identities, positions, and task combinations within each reconstructed room. Robot and task-object placement belong to data synthesis, and both reconstruction methods use identical task objects in the matched-budget comparison.

Q1. Static conversions and remaining-object stability

“What fraction of objects become static during refinement, and what is the stability score for objects that remain movable?”

Refinement converts 7.2% of evaluated objects from dynamic to static, mainly generated large furniture with uneven bases, especially bookshelves. Initial model assignments separately classify 23.7% as static, mainly fixed lighting and other scene elements.

After excluding objects that should be fixed, original-output room assembly achieves 98.7% stability and 3.2% collision for the remaining objects. W1 distinguishes these group-specific results from overall scene stability and reports the categories affected by refinement.

Q2. Defaults versus predicted physical attributes

“With the same refinement, how do category-default mass and friction compare with the predicted attributes?”

The added comparison in W1 evaluates category-default mass/friction with mobility-only prediction against joint mass/friction/mobility prediction. The mobility predictor changes between these configurations.

Collision/stability improves from 22.1%/75.9% to 18.0%/78.6%, while mean task success is nearly unchanged: 54.1/49.3/76.0% versus 54.3/49.3/76.0% for Pick & Place, Dish Stacking, and Object Pushing.

The main measured benefit is the improvement in mobility agreement from 93.5% to 98.9%. In the manipulation scenes, the principal observed difference is a distant falling painting outside the robot workspace. These results support joint physical supervision for mobility prediction and scene configuration.

Q3. Equal-data comparison and repeated training

“With equal amounts of synthetic training data, how does ASTER compare with SpatialLM + fine-tuned ShapeR using the same task objects?”

The comparison in W2 matches the scenes, task objects, demonstration budgets, and training conditions, with policies evaluated in the same GT environments. ASTER achieves 59.9% versus 23.2% end-to-end success.

The component baseline is comparable in Scene 1. A falling ceiling light prevents training in Scene 2, while table–chair intersections displace the table in Scene 3. Across the two scenes where both methods support training, ASTER still achieves 62.7% versus 34.8%. This provides evidence for the value of ASTER's environment construction independently of increasing the prescribed training-data budget.

“Does the real-robot improvement hold across repeated training runs?”

The original real-robot learning experiment evaluates one trained policy per data condition. The expanded evaluation tests a complementary form of transfer: ten successful simulated trajectories per task are strictly replayed on the physical robot across five tasks, achieving 41/50 successes (82%).

These 50 executions measure trajectory transfer across task types. The matched-budget comparison separately controls synthetic-data volume and training conditions when evaluating the reconstructed training environments.

The added results strengthen the evidence that ASTER supports reliable environment construction, robot learning, and automated data generation across tasks. We would appreciate your reconsideration in light of this additional evidence.

Reviewer_sWUVOpenReview ↗

Response to the rebuttal

Thanks for the quick rebuttal from the authors. I think those added experiments of matched budget comparison are helpful, and the result from scene 3 can show that collisions can affect the training outcome.

I still have some quick follow-up questions:

  1. In Q2, I asked

With the same refinement, how do category-default mass and friction compare with the predicted attributes?

now you have the comparison and show that stability slightly increase but the successful rate is similar with different mobility prediction results. so I'm not sure whether those new predictions really help the downstream tasks. if not, maybe the claim should be revised. Also, since the mobility predictor changes together with mass and friction in this comparison, it doesn't really isolate the effect of the predicted mass and friction. Could you keep the jointly predicted mobility fixed and only replace mass and friction with category defaults?

  1. in the matched-budget baseline comparison experiment, you show results from three scenes. the first scene comes out similar result, and the second scene you didnt really train a policy for the baseline due to the failure of the ceiling light prediction, but that should be easily fixable.. and only scene 3 has clear comparison. I think the conclusion can be much stronger if there are some more experiment results showing positive signals in more scenes. Also, what mobility setting does the baseline use? If it simply has no static assignment, fixing scene 2 with a category rule, e.g., setting ceiling lights as static, and rerunning the comparison would make the result looks more convincing to me.

Thanks!

Reviewer_7hPQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ASTER converts indoor point clouds into executable environments that include room layouts, individual objects, and simulation attributes. The method combines SpatialLM for scene understanding, ShapeR for object-level generation, and simulation feedback to refine positions and mobility. Experiments cover geometric reconstruction, collision and stability, policy transfer between corresponding simulated environments, and preliminary real-robot data augmentation.

Critical strengths
  1. The problem has practical value. Automatically constructing interactive simulation environments helps reduce the cost of robot data construction and policy development.
  2. The system shows evidence of usability. Table 4 indicates that the complete pipeline reduces the collision rate from 25.8 to 18.0 and increases the stability rate from 69.9 to 78.6. In experiments across three scenarios and three tasks, the average success rate gap between the reconstructed environment and the corresponding ground-truth (GT) simulation environment is 3.7 percentage points.
  3. Real-world experiments yield preliminary positive results. Adding 200 simulated demonstrations to the same set of 50 real demonstrations increases the success count in difficult settings from 0/6 to 4/6.
Critical weaknesses
  1. The runtime comparison is limited to one example and does not fully specify comparable measurement conditions. Figure 4 reports runtime for one scene. ASTER omits some small objects and adds task objects separately in the real-robot study, without stating whether their preparation, placement, and configuration are timed. It remains unclear whether the speed advantage holds for equivalent environment-construction work.
  2. The added mass and friction predictions are not shown to improve task performance. The paper does not compare ASTER's predictions with category-wise preset parameters. The relevant Table 4 ablation changes both object boxes and physical parameters, so it does not isolate parameter prediction. Appendix B evaluates agreement with automatic annotations, rather than whether the predictions improve robot tasks.
  3. Counts of objects made static during refinement are not reported. The authors disclose that objects remaining unstable after placement correction are made static and attached to the world. Experiments do not report how often this occurs, how many objects are affected, or their categories.
  4. Robot experiments lack other scene-reconstruction baselines. Both the simulated and real-robot experiments in Section 4.3 lack comparisons with other scene-reconstruction methods.
Questions for authors
  1. Timing scope: Where does Figure 4's 5.5-minute measurement start and end? Does it include task-object preparation, placement, and configuration?
  2. Appendix A.1 uses ten rooms for HoloScene and the full test split for ASTER. Can both methods be reported on the same ten rooms to remove differences in scene difficulty?
  3. With meshes, placements, and mobility held fixed, can the existing robot tasks compare ASTER's predicted mass/friction against category-wise preset values?
  4. In the quantitative scenes used for Tables 3 and 4, how many objects remain unstable after placement correction and are then made static? What proportion of evaluated objects do they represent, and which categories are affected?
  5. Can Figure 5 include an existing reconstruction method using the same scenes, data volume, and training conditions as ASTER, with both policies evaluated in the same GT simulation environment?
  6. In Figure 3, some reconstructed walls differ visibly from GT, especially in the fourth column. Is this reconstruction error or visualization, and does it affect collision handling or robot tasks?
7 comments
AuthorsOpenReview ↗

Critical Weaknesses 1 2 3

Thank you for the thoughtful review. We provide matched-room and matched-budget comparisons, expanded real-robot evaluation, and the requested implementation details below.

W1. Runtime scope and equivalent reconstruction work

“Figure 4 reports runtime for one scene” and does not specify whether task-object preparation and placement are timed.

The 5.5-minute measurement starts from the point cloud and ends with the room and major-object scene. Robot insertion, task-object preparation, placement, and demonstration generation are subsequent data-synthesis operations, excluded for every reconstruction method.

Adding task objects separately enables scalable robot data generation: a reconstructed room can support different object identities, positions, and task combinations. The downstream baseline comparison in W4 uses identical task objects and prescribed demonstration/training budgets.

“It remains unclear whether the speed advantage holds for equivalent environment-construction work.”

We add MIDI and SceneGen, evaluated using the same scene-level metrics. The following times cover scene construction from each listed input.

MethodInputTime ↓Prec.@5 cm ↑Surface error ↓F@5 / @10 cm ↑Stability ↑Median penetration ↓
ASTERPoint cloud5.6 min0.705.5 cm0.46 / 0.5189%2.6 cm
MIDIOne video frame6.2 min0.507.6 cm0.09 / 0.1820%21.6 cm
SceneGenOne video frame9.9 min0.1919.4 cm0.12 / 0.220%43.7 cm

For the laboratory, processing three minutes of video into a point cloud takes approximately 9.5 minutes. We report this preprocessing separately from point-cloud-to-scene construction. The new ASTER comparison takes 5.6 minutes; the original Fig. 4 example takes 5.5 minutes.

W2. Mass/friction prediction and task utility

“The added mass and friction predictions are not shown to improve task performance.”

Task success is nearly unchanged in the evaluated configurations; the substantial benefit appears in mobility prediction and scene stability.

Metric (%)Category defaults + mobility-onlyJoint mass/friction/mobility
Mobility agreement ↑93.598.9
Collision ↓22.118.0
Stability ↑75.978.6
Pick & Place success54.154.3
Dish Stacking success49.349.3
Object Pushing success76.076.0

Mobility agreement increases by 5.4 percentage points on the same held-out set and matched objects, using the same VLM annotations. Mass/friction prediction thus supplies useful physical supervision for identifying fixed and movable scene elements.

The principal observed difference in the manipulation scenes is a painting falling far from the robot in Scene 3. It affects scene stability much more than task success in the robot workspace.

“Table 4 ... does not isolate parameter prediction.”

The added table explicitly identifies both configurations and their mobility predictors. It evaluates the joint physical-prediction design, with the main gain appearing in mobility agreement and scene stability. The matched-budget experiment in W4 separately tests the complete pipeline's value for robot learning.

W3. Objects made static during refinement

“Counts of objects made static during refinement are not reported” and their categories are unclear.

Refinement converts 7.2% of evaluated objects from dynamic to static, mainly generated large furniture with uneven bases, especially bookshelves. These conversions include normally movable furniture whose generated geometry prevents stable placement.

We distinguish these additional conversions from the model's initial identification of fixed scene elements:

Initial assignments: 10,514 valid roomsObjectsShare
All objects65,003100%
Static lighting11,89118.3%
Other static objects3,5345.4%
All initially static objects15,42523.7%

Lighting is mainly ceiling-mounted; other categories include TVs, paintings, and carpets. Enabling initial static assignments gives 78.6% stability, compared with 48.8% when disabled. Observed failures include lights and paintings falling and TVs toppling.

We also evaluate rooms assembled from the original outputs after excluding objects that should be fixed. The remaining objects achieve 98.7% stability and 3.2% collision. This result is reported separately from overall scene stability; small bounding-box errors account for most remaining overlaps.

AuthorsOpenReview ↗

Critical Weakness 4; Q1-Q4

W4. Reconstruction baselines in robot experiments

“Robot experiments lack other scene-reconstruction baselines.”

We add SpatialLM + fine-tuned ShapeR under matched demonstration and training budgets. Both methods use identical scenes and task objects; policies are tested in the same GT environments.

SceneComponent baselineASTER
158 / 47 / 7256 / 49 / 71
20 / 0 / 0†47 / 39 / 77
325 / 7 / 060 / 60 / 80
Overall mean23.2%59.9%

Entries report Pick & Place / Dish Stacking / Object Pushing success (%).

† A ceiling light falls onto the table in Scene 2, preventing baseline training; this is counted as zero end-to-end success. In Scene 3, table–chair intersections displace the table and reduce success.

ASTER improves overall end-to-end success by 36.7 percentage points. Restricting the calculation to Scenes 1 and 3, where baseline training is possible, still gives 62.7% versus 34.8%. The downstream advantage therefore persists among the trainable scenes.

The expanded real-world evaluation tests direct trajectory transfer across five tasks. A robot VLM agent generates ten successful simulated trajectories per task in the reconstructed laboratory. We instantiate the corresponding real tasks and strictly replay the trajectories.

TaskReal successes
Bowl stacking8/10
Block grasping10/10
Drawer opening7/10
Color matching of cups and plates8/10
Package grasping8/10
Total41/50 (82%)

The matched-budget comparison measures the effect of the reconstructed training environment. The real-world test measures execution of successful simulated trajectories, demonstrating reuse of one ASTER environment across five task types. Physical experiments use the laboratory where our robot arm is available.

Q1. Timing boundaries

“Where does Figure 4's 5.5-minute measurement start and end? Does it include task-object preparation, placement, and configuration?”

It starts with the input point cloud and ends with the reconstructed room and major objects. Task-object preparation, robot insertion, placement/configuration, and demonstration generation are subsequent data-synthesis operations, excluded for every method.

For the laboratory, video preprocessing takes approximately 9.5 minutes for three minutes of video. It uses SLAM initialization, multiframe depth consistency and sparse bundle adjustment, followed by multiview stereo. W1 reports construction times with their respective input interfaces.

Q2. Matched-room comparison

“Can both methods be reported on the same ten rooms to remove differences in scene difficulty?”

We evaluate ASTER on the same ten rooms used for HoloScene, using the same collision and stability metrics.

MethodRoomsCollision ↓Stability ↑
HoloScene1025.2%70.6%
ASTER1014.9%82.7%

ASTER reduces collision by 10.3 percentage points and improves stability by 12.1 percentage points. The advantage holds with room selection matched. The revised paper distinguishes this result from ASTER's full-test-split scores of 18.0% collision and 78.6% stability.

Q3. Category-default physical parameters

“With meshes, placements, and mobility held fixed, can the existing robot tasks compare ... predicted mass/friction against category-wise preset values?”

Our reported comparison uses category-default mass/friction with mobility-only prediction versus joint mass/friction/mobility prediction. It changes the mobility predictor and evaluates the physical-prediction configuration.

Task success is 54.1/49.3/76.0% versus 54.3/49.3/76.0%. The main measured benefit is the mobility agreement increase from 93.5% to 98.9%, accompanied by improved scene stability. W2 gives the complete configurations and results. This supports joint physical supervision as a way to improve mobility prediction and scene configuration.

Q4. Refinement conversion rate and categories

“How many objects remain unstable after placement correction ... what proportion ... and which categories?”

Additional conversions affect 7.2% of evaluated objects, mainly generated large furniture with uneven bases, especially bookshelves. These are refinement corrections, distinct from the initial static assignments of 23.7%, mainly fixed lighting.

W3 separates the initial assignment counts, refinement conversion rate, and remaining-object evaluation. The latter gives 98.7% stability and 3.2% collision after excluding objects that should be fixed.

Q5. Matched-budget reconstruction baseline

“Can Figure 5 include an existing reconstruction method using the same scenes, data volume, and training conditions ... evaluated in the same GT simulation environment?”

The added comparison in W4 follows this design. SpatialLM + fine-tuned ShapeR and ASTER use matched scenes, task objects, demonstration budgets, and training conditions, with policies tested in the same GT environments.

ASTER achieves 59.9% versus 23.2% end-to-end success, and 62.7% versus 34.8% across the two scenes where both methods support training. The original Fig. 5 tests cross-environment consistency of an ASTER-trained policy; the added experiment directly compares the reconstructed training environments.

Q6. Wall differences in Fig. 3

“Is this reconstruction error or visualization, and does it affect collision handling or robot tasks?”

The irregular wall openings arise during ceiling removal for visualization. ASTER's separate, simple meshes allow clean removal. In the complex GT meshes, removing ceiling elements also removes some wall elements. Collision handling and robot tasks use the underlying simulation geometry.

Windows are omitted from the ASTER rendering to avoid similar mesh-removal artifacts. The corrected figure places GT and ASTER in the proper rows, and the caption explains the display preparation.

The matched-room results, matched-budget policy comparison, and five-task real-world executions provide complementary evidence that ASTER produces reusable environments for robot learning and data generation. We would appreciate your reconsideration in light of these additional results.

Reviewer_XDR2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ASTER converts indoor point clouds into structured, simulation-ready scenes. It extends SpatialLM to predict room layouts, object boxes, and physical attributes, uses a fine-tuned ShapeR model to generate individual object meshes, and applies progressive instance processing and simulation-guided placement and mobility correction. The evaluation covers geometric reconstruction, collision and stability metrics, policy transfer between reconstructed and ground-truth simulation scenes, and a small real-robot study using additional synthetic demonstrations.

Critical strengths
  1. The system addresses a useful practical problem. Recovering room structure, independently usable object meshes, and simulator attributes within one pipeline is relevant to embodied data generation. The emphasis on executable scenes rather than visual reconstruction alone is well motivated.
  2. Tab. 4 includes a fine-tuned SpatialLM + ShapeR baseline and reports measurable improvements from the added stages. Collision decreases from 25.8 to 18.0, while stability increases from 69.9 to 78.6 under the stated metrics. The matched-scene policy experiments also provide useful preliminary evidence of downstream usability.
Critical weaknesses
  1. The choice of a point cloud as an intermediate representation is insufficiently justified. The real-world pipeline converts RGB video into a point cloud before generating object meshes. Why is this route preferable to image/video-conditioned instance-level mesh reconstruction followed by comparable scene assembly? Existing image-conditioned methods already generate separate object meshes, including MIDI [1], PartCrafter [2], and SceneGen [3]. These provide relevant alternatives for object recovery, even though they do not all address complete simulation-ready rooms. The paper neither clearly accounts for the video-to-point-cloud preprocessing cost nor evaluates how its reconstruction errors affect downstream results. Fig. 4 compares against a monolithic mesh and two substantially slower pipelines, while Tab. 4 only tests components within the chosen route. This does not establish the quality–efficiency benefit of the intermediate representation under matched observations and output requirements.
  2. The incremental contribution beyond the fine-tuned component baseline is not convincingly established. In Tab. 4, object CD improves from 1.75 to 1.67 and F-Score from 73.25 to 73.70; the larger gains concern collision and stability. However, the refinement can make remaining unstable objects static, while stability is measured by displacement below 0.2 m. The evaluation does not isolate gains from correcting geometry and placement versus making objects immovable, or verify that objects intended to move retain that ability. Tab. 6 measures agreement with VLM annotations, as the authors acknowledge, but does not establish the usefulness of predicted mass and friction over simple category-based defaults. The real-robot study adds task objects separately and compares real-only training with additional ASTER demonstrations. It therefore does not isolate the benefit of the proposed physical components from the benefit of more training data.
Questions for authors
  1. What advantage does the reconstructed point-cloud interface provide over image/video-conditioned instance-level mesh reconstruction? Please specify the video-to-point-cloud method, whether its runtime is included in the reported 5.5 minutes, and how errors in this intermediate reconstruction affect the final scene.
  2. Why are room backgrounds absent from the SimRecon + ShapeR and HoloScene results in Fig. 4? SimRecon [4] explicitly assembles floor and wall structures, and HoloScene [5] includes a background scene representation. Please clarify the baseline adaptations, input observations, hardware, and timed processing stages.
  3. What criteria determine “Scalable Scene,” “Automated Pipeline,” and “Physical Properties” in Tab. 1? For example, PolaRiS [6] is marked as lacking physical properties, although it uses default contact parameters and user-estimated object masses. The table should distinguish assigning physical parameters from automatically predicting them.
  4. How many objects are changed from dynamic to static during refinement, and how often does this affect objects that should remain movable? With geometry and mobility labels held fixed, do the predicted mass and friction improve physical consistency or downstream task success over category-based defaults? A downstream comparison against the fine-tuned component baseline with equal demonstration budgets would also help isolate the added value.

References

[1] Zehuan Huang et al. MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation. CVPR, 2025.

[2] Yuchen Lin et al. PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers. NeurIPS, 2025.

[3] Yanxu Meng et al. SceneGen: Single-Image 3D Scene Generation in One Feedforward Pass. 3DV, 2026.

[4] Chong Xia et al. SimRecon: SimReady Compositional Scene Reconstruction from Real Videos. arXiv:2603.02133, 2026.

[5] Hongchi Xia et al. HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video. arXiv:2510.05560, 2025.

[6] Arhan Jain et al. PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policies. arXiv:2512.16881, 2025.

6 comments
AuthorsOpenReview ↗

Critical Weaknesses 1 and 2

Thank you for the thoughtful review. We address your concerns below and provide additional experiments to clarify ASTER’s design choices and practical value for robot data generation.

W1. Point-cloud representation

“The choice of a point cloud as an intermediate representation is insufficiently justified.”

ASTER is designed for automated robot data generation. The point cloud provides a shared 3D reference for scene understanding, object generation, and assembly, reducing scale and pose alignment between stages. Our 3D VLM also links object descriptions to spatial boxes and simulator assets. An agent generating “place X on Y” demonstrations can therefore identify the relevant objects and their locations. Both the cropped point cloud and its caption condition mesh generation (Eq. 3).

“Why is this route preferable to image/video-conditioned instance-level mesh reconstruction?”

We add MIDI and SceneGen, evaluated using the same scene-level protocol. ASTER achieves higher scene accuracy and stability with shorter scene-construction time.

MethodTime ↓Prec.@5 cm ↑Surface error ↓F@5 cm ↑F@10 cm ↑Stability ↑Median floor penetration ↓
ASTER5.6 min0.705.5 cm0.460.5189%2.6 cm
MIDI6.2 min0.507.6 cm0.090.1820%21.6 cm
SceneGen9.9 min0.1919.4 cm0.120.220%43.7 cm

“Video-to-point-cloud preprocessing cost” and “how its reconstruction errors affect downstream results.”

For a three-minute laboratory video, preprocessing takes approximately 9.5 minutes: SLAM initializes odometry, multiframe depth consistency supports sparse bundle adjustment, and multiview stereo extracts consistent points. The original 5.5-minute measurement starts from the point cloud.

Controlled perturbations give the following object-reconstruction results:

InputCD ↓F-Score ↑
Original1.6773.70
1 cm noise1.6772.99
3 cm noise1.9270.15
50% points retained1.6175.25
25% points retained2.8259.24

Modest noise and point loss preserve quality. Severe sparsity reduces the reconstruction quality.

W2. Added value beyond the component baseline

“The incremental contribution beyond the fine-tuned component baseline is not convincingly established.”

The added comparison tests whether reconstructed environments support robot learning at equal data volume. ASTER and SpatialLM + fine-tuned ShapeR use identical settings.

SceneComponent baselineASTER
158 / 47 / 7256 / 49 / 71
20 / 0 / 0†47 / 39 / 77
325 / 7 / 060 / 60 / 80
Overall mean23.2%59.9%

Entries report Pick & Place / Dish Stacking / Object Pushing success (%).

† A ceiling light falls onto the table, preventing baseline training; this counts as zero end-to-end success. In Scene 3, table–chair intersections displace the table. ASTER resolves these environment-construction failures, improving overall success by 36.7 percentage points.

“Refinement can make remaining unstable objects static” and “verify that objects intended to move retain that ability.”

Initial model assignments classify 23.7% of objects as static, mainly ceiling lights and other fixed elements. Refinement converts an additional 7.2%, mainly generated large furniture with uneven bases, especially bookshelves. These corrections include normally movable furniture whose generated geometry prevents stable placement.

After excluding objects that should be fixed, original-output room assembly achieves 98.7% stability and 3.2% collision for the remaining objects.

“The usefulness of predicted mass and friction over simple category-based defaults.”

Joint physical prediction improves mobility agreement by 5.4 percentage points, from 93.5% to 98.9%, on the same held-out set and matched objects.

MetricDefaults + mobility-onlyJoint prediction
Mobility accuracy ↑93.5%98.9%
Collision ↓22.1%18.0%
Stability ↑75.9%78.6%
Pick & Place54.1%54.3%
Dish Stacking49.3%49.3%
Object Pushing76.0%76.0%

Task success is nearly unchanged: the principal observed difference is a distant falling painting outside the robot workspace. The main benefit is better mobility prediction and scene configuration.

“The real-robot study … does not isolate … the benefit of more training data.”

The matched-budget comparison above controls training-data volume. We additionally evaluate agent-generated trajectory transfer: ten successful simulated trajectories per task are strictly replayed on the real robot across five tasks. Successes are 8/10, 10/10, 7/10, 8/10, and 8/10, totaling 41/50 (82%). A single ASTER environment thus supports automated data generation and successful real-world execution.

AuthorsOpenReview ↗

Questions 1 2

Q1. Point-cloud interface, preprocessing, and input errors

“What advantage does the reconstructed point-cloud interface provide?”

The point cloud provides a shared 3D reference linking scene understanding, object semantics, mesh generation, and assembly. Object names are associated with specific assets and locations, allowing an agent to construct robot tasks automatically. The MIDI and SceneGen comparisons in W1 evaluate the resulting scene quality under the same evaluation protocol.

“Please specify the video-to-point-cloud method.”

We first run SLAM to initialize camera odometry and prepare per-frame depth maps. Correspondences and projective consistency across multiple depth maps support bundle adjustment over sparse points. Multiview stereo then extracts mutually consistent points from the depth maps to construct the scene point cloud.

“Whether its runtime is included in the reported 5.5 minutes?”

Processing a three-minute laboratory video into a point cloud takes approximately 9.5 minutes. The reported 5.5 minutes starts from the point cloud and ends with the reconstructed room and major objects.

Robot insertion, task-object placement, and demonstration generation are subsequent data-synthesis operations, excluded from reconstruction timing for all methods. Separating task setup allows the recovered environment to support varied object identities, positions, and task combinations.

“How errors in this intermediate reconstruction affect the final scene?”

The controlled perturbation results in W1 show that modest noise and point loss preserve object quality. CD/F-Score is 1.67/72.99 with 1 cm noise, 1.92/70.15 with 3 cm noise, and 1.61/75.25 with 50% point retention. Severe sparsity at 25% retention reduces SpatialLM object detection, degrading CD/F-Score to 2.82/59.24.

Q2. Baseline backgrounds and processing scope

“Why are room backgrounds absent from the SimRecon + ShapeR … results?”

SimRecon’s released workflow does not provide an automatic module to reconstruct floor and wall meshes. In our implementation, background observations remain in 2DGS. ASTER predicts the room layout and constructs explicit floor and wall assets for simulation.

“Why is the background absent from HoloScene?”

HoloScene’s reconstructed background contains substantial artifacts and extraneous surfaces that obscure the objects. Fig. 4 therefore displays its object assets. The revised caption explains this visualization choice.

“Please clarify the baseline adaptations, input observations, hardware, and timed processing stages.”

Both baselines use 811 RGB frames at 512 × 512, with camera poses from Replica room_0, on one L40S GPU. Instance masks come from HoloScene’s released preprocessing package. HoloScene additionally uses the provided initial mesh, support graph, and Marigold depth/normal priors. ASTER’s colored input point cloud is sampled from the same provided mesh.

MethodAdaptation and timing scope
SimRecon + ShapeRApproximately 44 min for 2DGS, semantic processing, AVO, coverage sampling, SGS, and ShapeR. ShapeR uses instance point clouds and SGS category names; bounding-box centers determine placement. AVO is included in the timed run; this adaptation uses point-cloud conditioning and category labels rather than its selected images and support relations.
HoloScene7 h 26 min for stage-1 reconstruction, including its background. Depth/normal preparation adds approximately 12 min. The run ends at stage 1 because the Isaac Sim GPU backend did not initialize.
ASTER5.5 min from point cloud to reconstructed scene in the original example. The appendix documents inference using one A100.

Robot insertion and task-object setup are excluded for every method.

AuthorsOpenReview ↗

Questions 3-4

Q3. Capability-table criteria

“What criteria determine ‘Scalable Scene,’ ‘Automated Pipeline,’ and ‘Physical Properties’?”

  • Scalable Scene: the recovered environment supports reuse across varied tasks and object configurations.
  • Automated Pipeline: understanding, asset generation, and assembly proceed from the declared inputs without manual per-scene editing or placement.
  • Physics Setup: the method explicitly configures objects for physical simulation, with the parameter source identified in the caption.

“The table should distinguish assigning physical parameters from automatically predicting them.”

The revised table marks PolaRiS as supporting physical configuration and specifies its use of Isaac Sim contact defaults and user-estimated masses. ASTER automatically predicts object-specific mass, friction, and mobility together with geometry and semantics. The revised column and caption make the parameter sources explicit.

Q4. Static conversions, physical prediction, and matched-budget comparison

“How many objects are changed from dynamic to static during refinement, and how often does this affect objects that should remain movable?”

Refinement converts 7.2% of evaluated objects, mainly generated large furniture with uneven bases, especially bookshelves. These conversions include normally movable furniture whose generated geometry prevents stable placement. Initial model assignments separately classify 23.7% as static, mainly fixed lighting and other scene elements.

After excluding objects that should be fixed, original-output room assembly achieves 98.7% stability and 3.2% collision for the remaining objects. W2 separates initial assignments, refinement conversions, and remaining-object evaluation.

“With geometry and mobility labels held fixed, do the predicted mass and friction improve … results over category-based defaults?”

The comparison reported in W2 evaluates category-default mass/friction with mobility-only prediction against joint mass/friction/mobility prediction. Task success is nearly unchanged: 54.1/49.3/76.0% versus 54.3/49.3/76.0%.

Joint prediction improves mobility agreement from 93.5% to 98.9% and collision/stability from 22.1%/75.9% to 18.0%/78.6%. These results identify improved mobility prediction and scene configuration as the main measured benefits of joint physical supervision.

“A downstream comparison … with equal demonstration budgets would … isolate the added value.”

W2 provides this comparison using identical task objects, demonstration budgets, training conditions, and GT test environments. ASTER achieves 59.9% versus 23.2% end-to-end task success. The scene breakdown shows comparable performance in Scene 1, successful environment construction in Scene 2, and reliable furniture placement in Scene 3.

We hope the added comparisons and clarifications demonstrate ASTER’s practical value for robot data generation, particularly the matched-budget downstream gains and successful real-world execution across five tasks. We would appreciate your reconsideration of the assessment in light of this additional evidence.

Comments on the paper

1 comment

Dear Program Chairs and Area Chair,

With the rebuttal period ending soon, we would greatly appreciate your help in reminding reviewer XDR2 who have not yet responded to our rebuttal to revisit the discussion.

We have carefully addressed the concerns and would welcome the opportunity to clarify any remaining questions before the deadline. The feedback at this stage would help us understand whether our responses have resolved the key issues or whether further clarification is needed.

We recognize the considerable workload involved and sincerely appreciate your help in facilitating the discussion.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.