- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes PhySGA, a agentic framework to address the task of 3D composition, which takes a text condition and a set of 3D assets as input and outputs physically plausible object poses (e.g., an object hanging on a hook.) The method has an outer loop with an Assembly Agent that focuses on geometric reasoning and coarse adjustment, and an inner loop with a Critic Agent that reasons over contact dynamics, which can be further connected with a physical simulator. Experiments on Isaac Sim show improvements over baselines on the proposed setup.
Critical strengths
The qualitative and quantitative results in this paper show that the system is capable of accomplishing the proposed task pretty well, and the generated layout makes sense to humans.
Critical weaknesses
- The task of generating 3D composition needs justification. The experiments in this paper are limited to the interaction of two given 3D objects, and I'm not sure why this task is useful. This also makes the comparison with baselines unfair: among the 3 baselines, SAGE and SceneSmith generate scenes for robot learning, and Interact3D can reconstruct a 3D scene from an image. The task in this paper is strictly weaker in terms of scale (2 objects vs. a complete scene) and input (pre-generated mesh vs. image). Please clarify who would be the target customer of the proposed system.
- The system of using two agents and the iteration between a critic agent and a physical validation step is carefully designed. The design can make sense, but the question is: is the manually designed system necessary for solving such a simple task (it has as few as 7 dof for the object-hanging case, where only the 6D pose and the scale of the mug can be adjusted). A simple baseline would be telling the agent the target goal and having it come up with the internal loop itself, and providing the simulator as a tool. While section 4.3 provides some comparisons on this, details are lacking (e.g., the exact prompt used for these baselines), and it is hard to judge if the tab 2 presents are the best-effort results for the vanilla coding agents.
- In Figure 3, baselines use different assets than the proposed method. It is possible that tasks would be simpler for certain shapes (e.g., a mug with a larger handle). I think it makes more sense to force different methods to use the same set of 3D assets for the same task.
- The paper writing can be improved, and overall it is not easy to follow. The intro seems heavily AI-generated (e.g., the 3rd paragraph), and it is hard to understand. Also, I don't really like how the methods section is written. It uses many terms and math formulas but I find the underlying things are simpler than it appears to be. Also details are not so clear. Examples:
- Eqn2, "if physical validation is required, ", it is unclear in what condition physical validation is required, and what happens when it is not required.
- "Geometric Router converts translation, rotation, and uniform scale predictions along 3D orthogonal axes into exact transformations.": isn't this just a few line of numpy code?
Questions for authors
N/A
11 comments
W2: Necessity of the System Design and Vanilla Agent Baselines
We appreciate your feedback. Our framework is not over-designed. Instead, it provides essential scaffolding that adapts to task difficulty. We demonstrate this across three aspects:
1. Adaptive Efficiency: Our dual-agent system does not impose unnecessary overhead on simpler problems. As shown in Table 1, PhySGA solves simple rigid tasks efficiently with low token consumption. The computational effort naturally scales up only for complex tasks involving articulated or deformable objects.
2. Vanilla Agent Baseline: The vanilla coding agents in our experiments were exactly the baseline you suggested. They received the same target prompt, rendering tools, physical simulation APIs, and 128-model-call budget as PhySGA. Crucially, detailed execution logs reveal that these single agents utilized the simulation-feedback APIs very infrequently. Without our decoupled architectural guidance, they struggled to determine when to trigger feedback loops, preferring instead to write deterministic programs to calculate static coordinates step-by-step. This exposes a severe deficit in grounded physical intelligence among current VLMs when confronting interactive visual-physical tasks, proving the absolute necessity of our collaborative Assembly-Critic design. For completeness, we restate the results from the main paper below (Table 2).
3. Evaluation with GPT-6-Astra: We also evaluated GPT-6-Astra (released post-submission) as a vanilla agent to see if raw model capability could solve the task. As shown in Table 1, it improves upon GPT-5.5 on simple tasks but collapses on complex ones compared to ours. It also consumes significantly more tokens than PhySGA. This confirms our structured scaffolding is strictly necessary to unlock the physical reasoning capabilities of even the most advanced VLMs.
Table 1. Comparison with vanilla coding agents. Tokens are reported as per-scene means.
| Method | Backbone | Simple tasks (Rigid) | Complex tasks (Articulated/Deformable) | ||||
|---|---|---|---|---|---|---|---|
| RFS ↑ | COS ↑ | Tokens (K) ↓ | RFS ↑ | COS ↑ | Tokens (K) ↓ | ||
| PhySGA (Ours) | GPT-5.5 | 9.23 | 7.71 | 323.32 | 7.64 | 6.73 | 406.84 |
| Vanilla Coding Agent | GPT-5.5 | 3.77 | 2.92 | 328.24 | 1.18 | 1.55 | 219.61 |
| Vanilla Coding Agent | GPT-6-Astra | 6.23 | 5.23 | 481.03 | 3.18 | 3.09 | 547.42 |
Table 2. Comparison between single-agent baselines (Vanilla Coding Agents) and PhySGA.
| Method | Physical Metrics | Visual Metrics | ||
|---|---|---|---|---|
| PR ↓ | CR ↑ | RFS ↑ | COS ↑ | |
| Qwen3.8-Max | 0.258 | 0.542 | 1.42 | 2.46 |
| DeepSeek-V4 | 0.208 | 0.542 | 1.16 | 2.35 |
| Gemini-3.1-Pro | 0.158 | 0.392 | 1.79 | 3.08 |
| GPT-5.5 | 0.142 | 0.350 | 1.29 | 2.62 |
| PhySGA + Qwen3.8-Max | 0.092 | 0.858 | 6.36 | 6.39 |
| PhySGA + DeepSeek-V4 | 0.075 | 0.467 | 4.11 | 4.72 |
| PhySGA + Gemini-3.1-Pro | 0.058 | 0.792 | 5.90 | 6.06 |
| PhySGA + GPT-5.5 | 0.033 | 0.958 | 8.75 | 7.41 |
W3: Asset Differences and the Shared-Asset Comparison
We agree that object geometry significantly impacts task difficulty. We address this concern from two perspectives:
1. Asset Suitability: In practice, acquiring geometrically suitable assets is a necessary step for functional scene construction. An intelligent agent must understand the physical requirements of a task (e.g., hanging a mug on a peg) and proactively generate or retrieve objects with the correct physical affordances (e.g., a mug with a sufficiently large handle). Therefore, evaluating the entire pipeline along with its asset acquisition strategy reflects the practical performance of these methods.
2. Shared-Asset Evaluation: At the same time, we fully agree that a controlled comparison is valuable to isolate pure spatial composition capabilities. We conducted an additional shared-asset experiment where all baselines used the exact same initial 3D assets generated by our pipeline. As shown in Table 2, PhySGA maintains a substantial performance lead across all metrics. This confirms that our framework's advantage fundamentally stems from its physical reasoning and closed-loop assembly mechanism instead of relying on easier asset shapes.
Table 2. Comparison with existing methods using shared assets.
| Method | PR ↓ | CR ↑ | RFS ↑ | COS ↑ |
|---|---|---|---|---|
| SAGE | 0.125 | 0.508 | 1.34 | 2.60 |
| SceneSmith | 0.042 | 0.667 | 2.12 | 2.63 |
| Interact3D | 0.333 | 0.792 | 3.00 | 3.21 |
| PhySGA (Ours) | 0.042 | 0.958 | 8.44 | 7.22 |
Thanks for your thoughtful comments, questions and insightful suggestions. We provide detailed responses below.
W1: Motivation, Task Complexity, and Target Customer
We respectfully disagree that focusing on pairwise 3D composition is a "weaker" task; rather, it addresses a critical bottleneck in embodied AI. While existing macro-scene generators (e.g., SAGE, SceneSmith) excel at collision-free room layouts, they fundamentally fail to synthesize microscopic, contact-rich, and physically constrained functional relations (e.g., hanging a ladle on a specific hook, draping trousers, or inserting a tray into an oven). In the real world, objects rarely just rest flat on tables; they exist in complex, intertwined precursor states that robots must learn to engage with and manipulate. Automatically constructing these highly constrained spatial-physical setups is extremely challenging. Thus, our framework acts as a foundational task provider, supplying the embodied AI community with the rich, interactive precursor environments required to train and evaluate advanced robotic manipulation.
Within this demanding context, Interact3D shares our objective of building complex compositions, but it frequently struggles with severe physical interpenetration and remains strictly tethered to its specific image-to-3D generative pipeline. In contrast, PhySGA offers significantly higher physical fidelity and versatility, seamlessly accommodating both generative assets and diverse preexisting 3D asset libraries. Furthermore, while our main evaluation focuses on foundational pairwise relations to ensure extreme precision, this formulation is highly extensible; as demonstrated in Appendix D.1, PhySGA readily scales to multi-object sequential compositions. Ultimately, our target "customers" are researchers and industry practitioners in the embodied AI community, whom we empower with a scalable infrastructure to automatically generate complex, interaction-ready scenes.