acceptodds
Under review as a conference paper at ICLR 2027

PhySGA: A Physics-Space Grounded Agentic Framework for Generalizable and Interactive 3D Composition

Abstract

3D object composition is pivotal to digital content creation and robot simulation, while also serving as a critical testbed for physical intelligence in foundation models. Previous methods either leverage large language models to directly predict object coordinates, or utilize generative diffusion priors to synthesize composite images for visual alignment, both of which rely on static priors in digital spaces. However, these methods consistently struggle with complex compositions such as precise hanging, gravity-driven settling and hidden containment, as these tasks require interactive construction processes involving global planning, fine-grained alignment, and physical simulation. To address these challenges, we propose **PhySGA**, a **Phy**sics-**S**pace **G**rounded **A**gentic Framework for 3D composition via agent exploration and physical simulation, enabling the universal interactive assembly of rigid, articulated, and deformable objects. Directly applying vanilla LLMs or VLMs to this task proves insufficient, as current models inherently lack grounded physical understanding and interactive construction experience. To overcome this limitation, we design a collaborative feedforward-feedback architecture: an Assembly Agent focuses on geometric and kinematic reasoning for global step planning and coarse adjustment, while a Critic Agent reasons over contact dynamics for simulator state diagnosis and fine-grained closed-loop correction. Furthermore, Action Routers map semantic planning and analysis onto executable physical primitives within the simulator, while a task-specific Self-Evolving Skill Library distills historical trial-and-error feedback into reusable experiences, boosting success rates and minimizing exploration attempts. Extensive experiments within Isaac Sim demonstrate that PhySGA significantly surpasses existing methods in relation fidelity, physical plausibility, and visual quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Object Composition, Agentic Framework, Physical Intelligence

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RuaoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PhySGA, a agentic framework to address the task of 3D composition, which takes a text condition and a set of 3D assets as input and outputs physically plausible object poses (e.g., an object hanging on a hook.) The method has an outer loop with an Assembly Agent that focuses on geometric reasoning and coarse adjustment, and an inner loop with a Critic Agent that reasons over contact dynamics, which can be further connected with a physical simulator. Experiments on Isaac Sim show improvements over baselines on the proposed setup.

Critical strengths

The qualitative and quantitative results in this paper show that the system is capable of accomplishing the proposed task pretty well, and the generated layout makes sense to humans.

Critical weaknesses
  1. The task of generating 3D composition needs justification. The experiments in this paper are limited to the interaction of two given 3D objects, and I'm not sure why this task is useful. This also makes the comparison with baselines unfair: among the 3 baselines, SAGE and SceneSmith generate scenes for robot learning, and Interact3D can reconstruct a 3D scene from an image. The task in this paper is strictly weaker in terms of scale (2 objects vs. a complete scene) and input (pre-generated mesh vs. image). Please clarify who would be the target customer of the proposed system.
  2. The system of using two agents and the iteration between a critic agent and a physical validation step is carefully designed. The design can make sense, but the question is: is the manually designed system necessary for solving such a simple task (it has as few as 7 dof for the object-hanging case, where only the 6D pose and the scale of the mug can be adjusted). A simple baseline would be telling the agent the target goal and having it come up with the internal loop itself, and providing the simulator as a tool. While section 4.3 provides some comparisons on this, details are lacking (e.g., the exact prompt used for these baselines), and it is hard to judge if the tab 2 presents are the best-effort results for the vanilla coding agents.
  3. In Figure 3, baselines use different assets than the proposed method. It is possible that tasks would be simpler for certain shapes (e.g., a mug with a larger handle). I think it makes more sense to force different methods to use the same set of 3D assets for the same task.
  4. The paper writing can be improved, and overall it is not easy to follow. The intro seems heavily AI-generated (e.g., the 3rd paragraph), and it is hard to understand. Also, I don't really like how the methods section is written. It uses many terms and math formulas but I find the underlying things are simpler than it appears to be. Also details are not so clear. Examples:
  • Eqn2, "if physical validation is required, sn→sn,0s_n\rightarrow s_{n, 0}", it is unclear in what condition physical validation is required, and what happens when it is not required.
  • "Geometric Router converts translation, rotation, and uniform scale predictions along 3D orthogonal axes into exact transformations.": isn't this just a few line of numpy code?
Questions for authors

N/A

11 comments

Thanks for your thoughtful comments, questions and insightful suggestions. We provide detailed responses below.

W1: Motivation, Task Complexity, and Target Customer

We respectfully disagree that focusing on pairwise 3D composition is a "weaker" task; rather, it addresses a critical bottleneck in embodied AI. While existing macro-scene generators (e.g., SAGE, SceneSmith) excel at collision-free room layouts, they fundamentally fail to synthesize microscopic, contact-rich, and physically constrained functional relations (e.g., hanging a ladle on a specific hook, draping trousers, or inserting a tray into an oven). In the real world, objects rarely just rest flat on tables; they exist in complex, intertwined precursor states that robots must learn to engage with and manipulate. Automatically constructing these highly constrained spatial-physical setups is extremely challenging. Thus, our framework acts as a foundational task provider, supplying the embodied AI community with the rich, interactive precursor environments required to train and evaluate advanced robotic manipulation.

Within this demanding context, Interact3D shares our objective of building complex compositions, but it frequently struggles with severe physical interpenetration and remains strictly tethered to its specific image-to-3D generative pipeline. In contrast, PhySGA offers significantly higher physical fidelity and versatility, seamlessly accommodating both generative assets and diverse preexisting 3D asset libraries. Furthermore, while our main evaluation focuses on foundational pairwise relations to ensure extreme precision, this formulation is highly extensible; as demonstrated in Appendix D.1, PhySGA readily scales to multi-object sequential compositions. Ultimately, our target "customers" are researchers and industry practitioners in the embodied AI community, whom we empower with a scalable infrastructure to automatically generate complex, interaction-ready scenes.

W2: Necessity of the System Design and Vanilla Agent Baselines

We appreciate your feedback. Our framework is not over-designed. Instead, it provides essential scaffolding that adapts to task difficulty. We demonstrate this across three aspects:

1. Adaptive Efficiency: Our dual-agent system does not impose unnecessary overhead on simpler problems. As shown in Table 1, PhySGA solves simple rigid tasks efficiently with low token consumption. The computational effort naturally scales up only for complex tasks involving articulated or deformable objects.

2. Vanilla Agent Baseline: The vanilla coding agents in our experiments were exactly the baseline you suggested. They received the same target prompt, rendering tools, physical simulation APIs, and 128-model-call budget as PhySGA. Crucially, detailed execution logs reveal that these single agents utilized the simulation-feedback APIs very infrequently. Without our decoupled architectural guidance, they struggled to determine when to trigger feedback loops, preferring instead to write deterministic programs to calculate static coordinates step-by-step. This exposes a severe deficit in grounded physical intelligence among current VLMs when confronting interactive visual-physical tasks, proving the absolute necessity of our collaborative Assembly-Critic design. For completeness, we restate the results from the main paper below (Table 2).

3. Evaluation with GPT-6-Astra: We also evaluated GPT-6-Astra (released post-submission) as a vanilla agent to see if raw model capability could solve the task. As shown in Table 1, it improves upon GPT-5.5 on simple tasks but collapses on complex ones compared to ours. It also consumes significantly more tokens than PhySGA. This confirms our structured scaffolding is strictly necessary to unlock the physical reasoning capabilities of even the most advanced VLMs.

Table 1. Comparison with vanilla coding agents. Tokens are reported as per-scene means.

MethodBackboneSimple tasks (Rigid)Complex tasks (Articulated/Deformable)
RFS ↑COS ↑Tokens (K) ↓RFS ↑COS ↑Tokens (K) ↓
PhySGA (Ours)GPT-5.59.237.71323.327.646.73406.84
Vanilla Coding AgentGPT-5.53.772.92328.241.181.55219.61
Vanilla Coding AgentGPT-6-Astra6.235.23481.033.183.09547.42

Table 2. Comparison between single-agent baselines (Vanilla Coding Agents) and PhySGA.

MethodPhysical MetricsVisual Metrics
PR ↓CR ↑RFS ↑COS ↑
Qwen3.8-Max0.2580.5421.422.46
DeepSeek-V40.2080.5421.162.35
Gemini-3.1-Pro0.1580.3921.793.08
GPT-5.50.1420.3501.292.62
PhySGA + Qwen3.8-Max0.0920.8586.366.39
PhySGA + DeepSeek-V40.0750.4674.114.72
PhySGA + Gemini-3.1-Pro0.0580.7925.906.06
PhySGA + GPT-5.50.0330.9588.757.41

W3: Asset Differences and the Shared-Asset Comparison

We agree that object geometry significantly impacts task difficulty. We address this concern from two perspectives:

1. Asset Suitability: In practice, acquiring geometrically suitable assets is a necessary step for functional scene construction. An intelligent agent must understand the physical requirements of a task (e.g., hanging a mug on a peg) and proactively generate or retrieve objects with the correct physical affordances (e.g., a mug with a sufficiently large handle). Therefore, evaluating the entire pipeline along with its asset acquisition strategy reflects the practical performance of these methods.

2. Shared-Asset Evaluation: At the same time, we fully agree that a controlled comparison is valuable to isolate pure spatial composition capabilities. We conducted an additional shared-asset experiment where all baselines used the exact same initial 3D assets generated by our pipeline. As shown in Table 2, PhySGA maintains a substantial performance lead across all metrics. This confirms that our framework's advantage fundamentally stems from its physical reasoning and closed-loop assembly mechanism instead of relying on easier asset shapes.

Table 2. Comparison with existing methods using shared assets.

MethodPR ↓CR ↑RFS ↑COS ↑
SAGE0.1250.5081.342.60
SceneSmith0.0420.6672.122.63
Interact3D0.3330.7923.003.21
PhySGA (Ours)0.0420.9588.447.22
Reviewer_wLWvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies pairwise 3D object composition, e.g., hanging a mug on a peg, placing a tray inside a half-open oven, or draping trousers over a bar. The authors argue that existing LLM-layout and diffusion-guided methods rely on static priors and fail on relations that depend on contact, gravity or occlusion. They propose PhySGA, which constructs the composition step by step inside Isaac Sim. An Assembly Agent plans the steps and makes coarse metric adjustments to reach a release state. The object is then released and simulated, and a Critic Agent diagnoses the outcome and refines the release state. For position correction, the Critic marks pixel correspondences in multiple views, which are triangulated into a 3D translation. Action Routers convert agent outputs into geometric, joint and velocity commands, and a skill library stores distilled experience per task type. On a self-built benchmark of 40 tasks (120 scenes) covering rigid, articulated and deformable objects, PhySGA outperforms SAGE, SceneSmith, Interact3D and vanilla VLM coding agents with the same backbones. The comparison uses contact/penetration rates and GPT-5.5-judged relation and overall scores, and ablations remove each component.

Critical strengths
  • Building the composition as a closed loop inside a physics simulator is well motivated and fits the failure cases the paper targets. Separating the release state from the settled state lets the critic adjust the release pose or velocity after seeing the rollout. With this design, hanging, drawer containment and cloth draping can all be handled in one framework, which is hard to express with static layout methods. This is the main reason for my positive leaning.

  • The comparison with vanilla coding agents (Table 2) is the most convincing evidence in the paper, since assets and backbones are shared. The gains are large and consistent across four VLMs (e.g., GPT-5.5: CR 0.350 →\rightarrow 0.958, RFS 1.29 →\rightarrow 8.75). With the asset pipeline and base model held fixed, I believe the improvement mainly comes from the framework itself. The coarse-metric / fine-pixel split is also a sensible design given that current VLMs are weak at estimating small metric distances, and the pixel-alignment ablation supports it.

Critical weaknesses
  • The evaluation leans heavily on a VLM judge. RFS is presented as the primary indicator, but it is scored only by GPT-5.5, which is also the backbone in the main setting, with no human or rule-based check of whether the relation is actually achieved. CR only requires any contact between the two objects, so it is a fairly loose proxy for success. In Table 1 each method also uses its own asset pipeline, so the large AFS gap (and COS, which builds on AFS) partly reflects differences in asset quality. Given Table 2, I think the main conclusion likely holds. Still, a small human check or task-specific success criteria on a subset would make the numbers more trustworthy.

  • The cost and interaction budget are not reported. PhySGA uses many VLM calls, preview renderings and simulation rollouts per scene, while the vanilla coding agents do not seem to receive any simulation feedback, and the budgets of the other baselines are not stated. Without per-scene numbers on calls, rollouts and wall-clock time, it is hard to tell how much of the gain is due to the additional trials with feedback. This also matters for the claimed use case of scalable task construction.

  • The notion of ''physical validity'' could be stated more carefully. The agents teleport, rotate and also rescale objects (e.g., ''scale up the mug to fit the peg'' in Fig. 2). What the framework ensures is a stable final state after release. Whether the composition is physically reachable, or whether the assets keep their intended real-world size, is left open. Rescaling an object to satisfy a relation is somewhat in tension with asset fidelity and is not penalized by the current metrics. I'd suggest clarifying this scope and toning down statements such as ''guarantees strict physical validity'' and ''unlocking the latent physical intelligence''.

Questions for authors
  • Could you report the average number of VLM calls, simulation rollouts and wall-clock time per scene for PhySGA, the baselines and the vanilla agents? Do the vanilla agents get any simulation feedback or retries?

  • How often, and by how much, do the agents change object scale across the benchmark? Is scaling constrained? How does PhySGA perform if scale is fixed to the asset’s metric size?

  • My understanding is that the skill library keeps updating during evaluation, including across the three prompt variants of the same task. What are the results with a library frozen before evaluation?

6 comments

Thanks for your thoughtful comments, questions and insightful suggestions. We provide detailed responses below.

W1: Evaluation Reliability and Asset Suitability

Rule-Based Checks and Human Evaluation: We completely agree that Contact Rate (CR) is a loose proxy and relying solely on a VLM judge invites bias. To provide rigorous evidence of task completion, we introduced a strict, rule-based Task Completion Rate (TCR) designed specifically for each scenario (e.g., verifying exact collision pairs between the scissors' handle and the specific hook). Furthermore, we conducted a large-scale user study with 100 human raters evaluating the compositions blindly. As shown in Table 1, the human-rated scores align tightly with our physical metrics (TCR) and VLM-based visual scores, confirming the reliability of our evaluation.

Asset Suitability: While we consider our unified asset pipeline to be one of the framework's practical contributions, we fully agree that a controlled comparison is necessary to isolate pure compositional capabilities. We conducted a shared-asset experiment where SAGE, SceneSmith, and Interact3D use the exact same assets generated by our pipeline. Table 2 demonstrates that PhySGA maintains a dominant lead across all metrics when the asset variable is controlled.

Table 1. Updated results for Section 4.2 with rule-based task completion rates and human ratings.

MethodPhysical MetricsVisual Metrics
PR ↓CR ↑TCR ↑AFS (VLM / Human) ↑RFS (VLM / Human) ↑COS (VLM / Human) ↑
SAGE0.0000.0000.0003.96 / 2.110.74 / 1.673.03 / 2.83
SceneSmith0.0420.5750.3423.29 / 4.682.59 / 2.674.55 / 3.13
Interact3D0.1330.7170.4336.87 / 6.544.45 / 5.545.12 / 4.33
PhySGA0.0330.9580.9089.02 / 9.148.75 / 9.257.41 / 8.56

Table 2. Comparison with existing methods using shared assets.

MethodPhysical MetricsVisual Metrics
PR ↓CR ↑RFS ↑COS ↑
SAGE0.1250.5081.342.60
SceneSmith0.0420.6672.122.63
Interact3D0.3330.7923.003.21
PhySGA0.0420.9588.447.22

W2 & Q1: Simulation Feedback, Budgets, and Execution Costs

Vanilla Agent Baseline & Simulation Feedback: The vanilla coding agents were explicitly equipped with the exact same Action Routers, unrestricted multi-view rendering, iterative physical simulation feedback, and an identical interaction budget (128 model calls) as PhySGA. Crucially, detailed execution logs reveal that these single agents utilized the simulation-feedback APIs very infrequently. Without our decoupled architectural guidance, they struggled to determine when to trigger feedback loops, preferring instead to write deterministic programs to calculate static coordinates step-by-step. This exposes a severe deficit in grounded physical intelligence among current VLMs when confronting interactive visual-physical tasks, proving the absolute necessity of our collaborative Assembly-Critic design.

Execution Costs & Efficiency: We report the detailed cost metrics in Table 3. While our closed-loop framework inherently introduces computational overhead (e.g., token usage to process multi-view images), PhySGA achieves exceptional physical sample efficiency. It requires only 1.58 simulation rollouts per scene compared to SceneSmith's 10.54, proving that our Critic Agent resolves geometric errors precisely rather than relying on blind physical trial-and-error. While SAGE and the Vanilla Agent consume fewer tokens, their composition quality (RFS < 2.5) is too poor to be usable. Given the nearly triple improvement in RFS over the best baseline, a ~20-minute execution time per scene is highly acceptable for offline, scalable task construction.

Table 3. Composition quality, execution budgets, and costs under shared assets.

MethodBackboneModel-call limitRFS ↑COS ↑Wall-clock time (s) ↓VLM calls ↓Tokens (K) ↓Sim. rollouts ↓
SAGEGPT-5.51281.342.60420.007.38137.072.08
SceneSmithGPT-5.51282.122.631571.1472.83877.5110.54
Interact3DGPT-5.51283.003.211075.111.0011.710.00
Vanilla Coding AgentGPT-5.51282.482.24393.1618.65273.930.00
PhySGA (Ours)GPT-5.51288.447.221240.6456.45365.231.58

W3 & Q2: Scope of Physical Validity and Object Scaling

We fully agree that the phrase "strict physical validity" was an overclaim. We will revise the manuscript to clarify that PhySGA ensures "kinematic and dynamic plausibility" at the final settling state (i.e., obeying gravity and collisions without interpenetration).

Regarding object scaling, we wish to clarify its scope and rationale:

1. Preserving Common-Sense Scale: Assets are initially scaled to real-world dimensions to establish realistic physical states, preventing absurd relations (e.g., forcing a massive object into a tiny container) before interaction begins.

2. Resolving Geometric Mismatches: Because 3D assets are sourced independently, minor structural mismatches (e.g., a handle opening slightly too small for a peg) often render target relations physically impossible. Allowing the agent to autonomously make minor scale adjustments resolves these incompatibilities, demonstrating active spatial reasoning akin to a human 3D modeler fitting digital assets.

3. Is scaling constrained? While there are no hard-coded limits, our framework's design inherently prioritizes global spatial planning and precise translational alignment rather than arbitrary scaling. Empirically, the agent treats scaling strictly as a last resort, invoking it only when severe geometric mismatches fundamentally block the physical assembly.

4. Empirical Evidence: Scaling was invoked in only 7 out of 120 scenes. As shown in Table 4, disabling it caused 5 of these scenes to fail entirely due to a lack of geometric clearance, while the remaining 2 succeeded via finer translation. This confirms that scaling serves as a targeted resolution for inherent asset flaws, not a general shortcut.

Table 4. Task completion with and without object scaling. (✓ indicates success and ✗ indicates failure).

Scene IDTask descriptionScaling allowedScaling disabledFailure reason
A00-01Place a ceramic piggy bank in the top drawer of a two-drawer cabinet.✓✗Asset size mismatch
A04-01Place a baking tray on the interior rack of an oven.✓✗Asset size mismatch
R17-01Hang a soup ladle on a countertop ladle stand hook.✓✓—
R17-02Hang a soup ladle on a countertop ladle stand hook.✓✗Asset size mismatch
R18-01Place a thornless rose bouquet inside a ceramic vase.✓✗Asset size mismatch
R19-01Hang a bunch of keys on a hook.✓✓—
R19-02Hang a bunch of keys on a hook.✗✗Asset size mismatch
Reviewer_NQ7pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PhySGA, an agentic framework for constructing physically plausible 3D object compositions from language instructions. The framework treats composition as an iterative process involving planning, geometric adjustment, physical simulation, and feedback-driven refinement. An Assembly Agent performs global planning and coarse metric adjustments, while a Critic Agent diagnoses simulated failures and proposes fine-grained corrections, including pixel-grounded alignment. Specialized action routers translate these decisions into geometric transformations, articulated joint commands, and release velocities. A self-evolving textual skill library supports experience reuse. Experiments cover 120 scenes spanning rigid, articulated, and deformable objects, with comparisons against existing composition systems, vanilla VLM-based coding agents, and component ablations. The reported results show substantial improvements in contact establishment and relation fidelity.

Critical strengths
  1. An interesting and relatively underexplored problem setting. Constructing precise functional relations between objects presents challenges beyond conventional scene layout generation, particularly when success depends on contact, intermediate articulation, or deformation. The paper makes a compelling case for studying VLM agents through executable construction tasks.
  2. A coherent architecture tailored to the task. The separation between global assembly planning and local physical refinement is well motivated.
  3. Substantial empirical improvements, with useful supporting ablations. PhySGA improves relation fidelity from 4.45 to 8.75 over Interact3D and achieves a contact rate of 0.958 while maintaining a relatively low penetration rate. The same-asset comparisons against vanilla coding agents provide additional evidence that the framework contributes beyond asset acquisition. The ablations further support the importance of action routing, global planning, and pixel alignment.
Critical weaknesses
  1. The comparisons do not fully isolate the contribution of the proposed agent organization. Existing systems use their native asset acquisition pipelines, so the main comparison combines differences in asset suitability and composition capability. The same-asset coding-agent experiments partly address this issue, but it remains unclear whether these agents receive comparable observations, iterative simulation access, and interaction budgets. A single-agent baseline using the same action interfaces and feedback would help distinguish the benefits of the dual-agent architecture from those of the surrounding execution infrastructure.

  2. The evaluation provides incomplete evidence of task completion. Contact rate measures whether the target objects contact each other, rather than whether the intended functional relation is correctly established. Relation fidelity therefore carries substantial weight, but it is assessed by a single VLM evaluator from final-state images. This is particularly limiting for occluded containment and relations whose validity depends on dynamic stability. The reported metrics are useful, but should not be interpreted as a direct task-success rate.

  3. The skill-library evaluation leaves generalization difficult to assess. The appendix states that the library is initialized from preliminary experiments and continues to update after successful benchmark runs. This is a reasonable online-learning protocol, but performance may consequently depend on scene order and experience accumulated from related benchmark instances. The current results do not clearly separate transfer from prior experience, within-benchmark adaptation, and performance on genuinely unseen task types.

  4. Some claims are stronger than the demonstrated scope. The claim of “strict physical validity” is difficult to reconcile with the nonzero penetration rate and final-state evaluation protocol. Moreover, the framework permits direct pose editing and object scaling; the drawer example resolves an obstruction by shrinking the contained object. These operations are appropriate for digital scene construction, but the distinction between physically plausible scene generation and physically executable manipulation should be clearer.

  5. Efficiency and reproducibility are insufficiently characterized. The framework involves multiple reasoning stages, local adjustments, and simulation rollouts, yet the paper does not provide a systematic accounting of runtime, model calls, or simulation attempts. The execution budgets are described qualitatively without enough numerical detail to assess whether the gains are achieved under comparable computational resources.

Questions for authors
  1. Could you specify the observation access, simulation-feedback access, and interaction budgets of the vanilla coding-agent baselines? How does a single agent equipped with the same routers and feedback compare with the proposed dual-agent design?

  2. How does performance change when the skill library is frozen during evaluation or removed at initialization? Have you examined sensitivity to benchmark ordering or evaluated transfer to held-out task types?

  3. Could you report a task-level success assessment on a representative subset, using explicit relation checks or human judgments, and examine its agreement with CR and the VLM-based RFS?

  4. How frequently does the system resize objects, and how much does performance change when their original dimensions are fixed? This would help clarify how much success comes from spatial reasoning versus adapting the assets to make the relation feasible.

  5. What are the average and variability of execution time, VLM calls, and simulation rollouts per scene? Does the skill library measurably reduce these costs, in addition to improving final composition quality?

6 comments

Thanks for your thoughtful comments, questions and insightful suggestions. We provide detailed responses below.

W1 & Q1: Isolating the Dual-Agent Architecture and Asset Pipelines

Asset Suitability (Shared-Asset Comparison): To strictly isolate the contribution of our composition framework from native asset acquisition pipelines, we evaluated SAGE, SceneSmith, and Interact3D using identical 3D assets generated by our pipeline. Table 1 demonstrates that PhySGA maintains a dominant lead across all metrics when the asset variable is controlled.

Table 1. Comparison with existing methods using shared assets.

MethodPhysical MetricsVisual Metrics
PR ↓CR ↑RFS ↑COS ↑
SAGE0.1250.5081.342.60
SceneSmith0.0420.6672.122.63
Interact3D0.3330.7923.003.21
PhySGA (Ours)0.0420.9588.447.22

The Single-Agent Baseline (Vanilla Coding Agents): The vanilla coding agents evaluated in Section 4.3 of our original manuscript inherently serve as the single-agent baselines you suggested. We explicitly equipped them with the exact same Action Routers, unrestricted multi-view observation rendering, iterative physical simulation feedback, and an identical interaction budget (128 model calls) as PhySGA.

Crucially, detailed execution logs reveal that these single agents utilized the simulation-feedback APIs very infrequently. Without our decoupled architectural guidance, they struggled to determine when to trigger feedback loops, preferring instead to write deterministic programs to calculate static coordinates step-by-step. This exposes a severe deficit in grounded physical intelligence among current VLMs when confronting interactive visual-physical tasks, proving the absolute necessity of our collaborative Assembly-Critic design. For completeness, we restate the results from the main paper below.

Table 2. Comparison between single-agent baselines (Vanilla Coding Agents) and PhySGA.

MethodPhysical MetricsVisual Metrics
PR ↓CR ↑RFS ↑COS ↑
Qwen3.8-Max0.2580.5421.422.46
DeepSeek-V40.2080.5421.162.35
Gemini-3.1-Pro0.1580.3921.793.08
GPT-5.50.1420.3501.292.62
PhySGA + Qwen3.8-Max0.0920.8586.366.39
PhySGA + DeepSeek-V40.0750.4674.114.72
PhySGA + Gemini-3.1-Pro0.0580.7925.906.06
PhySGA + GPT-5.50.0330.9588.757.41

W2 & Q3: Rigorous Task Validation and Evaluator Reliability

We sincerely thank the reviewer for the constructive suggestion to provide more rigorous evidence of task success. In response, we have comprehensively upgraded our evaluation protocol.

Rule-Based Task Completion Rate (TCR): To address the limitation of the Contact Rate (CR), we introduced a strict, rule-based TCR. We designed programmatic relational checks specifically tailored for each individual scenario (e.g., verifying exact collision pairs between the scissors' handle opening and the specific hook mesh) and implemented these evaluation programs using Codex under human supervision.

Occluded Containment & Dynamic Stability: You correctly pointed out two distinct challenges for final-state VLM evaluators: visual occlusion and dynamic stability.

  1. For occluded containment: We explicitly mitigated this in our benchmark design by utilizing partially open configurations (e.g., Place a tray inside a half-open oven). This intentional design ensures the internal relation remains visually accessible for reliable VLM evaluation.
  2. For dynamic stability: Our current tasks primarily focus on reaching a final static equilibrium, making final-state evaluation largely effective. However, we completely agree that for relations heavily dependent on dynamic stability, a single frame is insufficient.
  3. To address these challenges, we will upgrade the VLM evaluator in the future by providing it with multiple frames sampled across the dynamic simulation trajectory.

Large-Scale Human Evaluation: Furthermore, we conducted a comprehensive user study covering all scene outputs in Section 4.2. We recruited 100 human raters to blindly evaluate the compositions. As shown in Table 3, while there are slight numerical variations, the performance trends across physical metrics (CR, TCR), VLM-based visual scores, and human ratings are highly consistent. This multi-faceted evaluation makes our metric system significantly more robust, and we will incorporate the new TCR and human evaluation metrics into the final manuscript.

Table 3. Updated results for Section 4.2 with rule-based task completion rates and human ratings. Each visual metric includes both VLM and human scores.

MethodPhysical MetricsVisual Metrics
PR ↓CR ↑TCR ↑AFS (VLM / Human) ↑RFS (VLM / Human) ↑COS (VLM / Human) ↑
SAGE0.0000.0000.0003.96 / 2.110.74 / 1.673.03 / 2.83
SceneSmith0.0420.5750.3423.29 / 4.682.59 / 2.674.55 / 3.13
Interact3D0.1330.7170.4336.87 / 6.544.45 / 5.545.12 / 4.33
PhySGA (Ours)0.0330.9580.9089.02 / 9.148.75 / 9.257.41 / 8.56

W3 & Q2: Skill Library Generalization, Adaptation, and Sensitivity

We appreciate the reviewer's insightful request to decouple the effects of prior experience, online learning, and zero-shot transfer.

Nature of Preliminary Skills: First, we clarify that the initial skills are distilled from small-scale, independent preliminary experiments that focus strictly on basic physical operations (e.g., simple moving, scaling, throwing). They contain no benchmark-specific task knowledge. Thus, they serve purely as a "warm start" for foundational physical interactions.

Separating Generalization and Adaptation: To explicitly separate the capabilities you mentioned, we evaluated four combinations of skill initialization and online adaptation. As shown in Table 4:

  1. Performance on genuinely unseen task types (Transfer from prior experience): By initializing with preliminary skills but freezing the library during the benchmark (Row 3), every benchmark scene becomes a "genuinely unseen task." Compared to the absolute zero-shot baseline (Row 1), this frozen prior successfully transfers to unseen tasks, boosting RFS from 7.58 to 7.94.
  2. Within-benchmark adaptation: Comparing the frozen setting (Row 3) to the fully evolved setting (Row 4) isolates the impact of within-benchmark adaptation. Continuously updating the library yields further improvements (RFS reaches 8.44). Notably, even starting with a completely empty library (Row 2), the system can effectively learn and adapt entirely from scratch, though the preliminary "warm start" yields the best overall performance.
  3. Execution cost reduction: Beyond improving composition quality, combining these components significantly optimizes token efficiency. Continuously updating skills helps the agent avoid repeated trial-and-error (reducing tokens from 473.7K in Row 1 to 416.6K in Row 2). Providing the initial skills as a "warm start" alongside online updates further lowers the initial reasoning burden, achieving the most efficient state at 365.2K tokens (Row 4).

Sensitivity to Scene Order: Regarding sequence dependency, we evaluated the benchmark under three conditions: Order A (our original randomly seeded evaluation sequence), Reversed A (the exact inverse of Order A), and Order B (a completely newly shuffled sequence using a different random seed). As shown in the last three rows of Table 4, while the sequence order inevitably causes slight fluctuations in the learning trajectory (which primarily impacts execution costs, as detailed in Q5), the final composition quality metrics (CR, RFS, COS) remain robust and broadly consistent regardless of the curriculum order.

Table 4. Skill-library ablations and sensitivity to scene order.

Preliminary SkillsBenchmark AdaptationScene OrderPhysical MetricsVisual MetricsExecution Cost
PR ↓CR ↑RFS ↑COS ↑Tokens (K) ↓
✗✗Order A (Random)0.0170.8837.587.17473.71
✗✓Order A (Random)0.0420.9428.137.36416.56
✓✗Order A (Random)0.0170.8927.947.05493.69
✓✓Order A (Random)0.0420.9588.447.22365.23
✓✓Reversed A0.0330.9428.347.81382.03
✓✓Order B (New Seed)0.0250.9678.857.58393.31

Comments on the paper

1 comment

Dear PCs, SACs, ACs, and all reviewers,

We first want to express our gratitude for your great efforts on our paper. Your feedback greatly improved the clarity and completeness of our submission. We appreciate the reviewers for highlighting the following strengths of our work:

  • An interesting and well-motivated closed-loop framework for constructing physically plausible, contact-rich 3D compositions across rigid, articulated, and deformable objects.
  • A coherent dual-agent architecture that effectively decouples global assembly planning from simulation-grounded local contact refinement and pixel-level alignment.
  • Substantial and consistent empirical gains over existing composition systems and same-asset vanilla VLM coding agents across multiple backbones, backed by thorough ablations.

We have also addressed the main concerns raised by reviewers and uploaded an updated PDF with the revisions highlighted in purple:

  • We added shared-asset comparisons against SAGE, SceneSmith, and Interact3D, clarified that vanilla coding agents share identical tools and budgets, and evaluated frontier models (GPT-6-Astra, Claude Opus 5.5, Claude Fable 5.2) with detailed failure analyses and supplementary visualizations.
  • We upgraded our evaluation with a strict, rule-based Task Completion Rate (TCR) and a 100-rater blind user study, confirming strong alignment across physical metrics, VLM scores, and human judgments.
  • We decoupled prior transfer from online adaptation across four skill-library configurations and three scene orders, demonstrating zero-shot transfer from frozen preliminary skills, further gains from online updates, and robustness to scene ordering.
  • We refined our claims to kinematic and dynamic plausibility at the settling state, and showed via ablation that object scaling is invoked in only 7 of 120 scenes to resolve inherent asset size mismatches.
  • We reported full execution costs (wall-clock time, VLM calls, token usage, and simulation rollouts), showing that online skill adaptation cuts time and tokens by ~26% while requiring only 1.58 simulation rollouts per scene.
  • We clarified the motivation for physically plausible contacts and embodied AI applications, refined the manuscript's writing and baseline details, and will further incorporate additional multi-object results in the final version.

Finally, we would like to thank all the reviewers for recognizing our contribution to physics-grounded 3D object composition and for their constructive engagement during the discussion period. We believe our collaborative feedforward-feedback architecture, grounded Action Routers, and Self-Evolving Skill Library provide a practical and extensible foundation for building complex, interaction-ready 3D environments.

Thanks again to all of you for your time and efforts.

Best regards, Authors of Submission 8029

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.