acceptodds
Under review as a conference paper at ICLR 2027

AgentDreamer : Agentic 3D Motion Reasoning for Text-Controllable Physics Simulation

Abstract

We propose AgentDreamer, an agent-guided optimization framework that aligns text instructions with simulated motion through explicit 3D motion planning. Given a text instruction and scene geometry, an off-the-shelf agent translates the instruction into a structured semantic motion prior, specifying what should move, in which direction, and in what temporal order. The agent represents this prior using parametric functions over a series of temporal chunks, yielding dense 3D trajectories. We then optimize the parameters of a differentiable MPM simulator such that its rollout best matches the prior while remaining constrained by the scene dynamics. This allows the agent to provide semantic guidance from language, while the simulator grounds it in physically feasible motion. We further introduce Motion-IG, an information-theoretic metric for text-motion alignment based on masked text recovery. By comparing the likelihood of recovering masked motion-related content with and without the simulated motion, Motion-IG measures the information contributed by the motion beyond that available from the remaining text and scene context. Experiments show that AgentDreamer improves text alignment over video-based guidance, producing simulations that more closely follow the requested physical behavior.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Gaussian splatting, physics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_j4NpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a novel framework for the text-controllable physics simulation thourgh agentic generation. Specifically, the proposed method will first ask the agent to infer mathematical functions and their parameters for each temporal chunk. The authors then apply a differentiable Material Point Method (MPM) simulator and fit the parameters θ\theta to minimize the trajectory error. In addition, the authors also provide a new metric, Motion-IG to better evaluate the text-motion alignment.

Critical strengths
  1. The topic is interesting, which explores agents' understanding capability to formulate the accurate motion behaviors.
  2. The quantitative results on the proposed metrics are satisfying.
  3. The proposed metric is reasonable, although it needs to be strictly verfied by human perferencce.
Critical weaknesses
  1. The authors should report the proposed method on more evaluation metrics. For example, in MotionPhysics, they use overall consistency, cliip similiarity and energy-constrained motion score for quantative results.
  2. The authors provide a new metric to measure the motion-text alignment, the reliability should be evaluated. For example, the authors should use user study to evaluate the correlation between human preference and the proposed metric within different ablation studies.
  3. There should be more ablation study to justify the technical designs. For example, the authors select different function family for different motion cues. What if the function is not such accurate since the trajectory in the real world is usually not perfect. In addition, how these formulation and hyper-parameters are chosen? Are they fitted on the test set? If so, the evaluation is not fair for other methods.
Questions for authors

Please refer to the weakness part.

4 comments
AuthorsOpenReview ↗

Response to reviewer j4Np

[W1] Results with additional evaluation metrics

Thank you for the valuable suggestion. We agree, and below we first verify each metric and then compare the methods.

  1. Verification: Text alignment (CLIP similarity, Overall Consistency). We compared their method rankings with human evaluation of text–motion alignment. Top-1 asks whether a metric ranks the human-preferred method first; pairwise asks this per method pair (e.g., ours vs. PhysFlow or MotionPhysics). Neither matches human preferences.

    MetricTop-1 agreement ↑Pairwise agreement ↑
    Random0.330.50
    CLIP similarity0.350.55
    Overall Consistency0.300.53
    Motion-IG0.680.74
  2. Verification: Motion smoothness (Energy-Constrained Motion Score, ECMS). It largely tracks flow magnitude across frames. Thus, it favors nearly static results and penalizes larger motions even when requested.

  3. Quantitative results. CLIP similarity and Overall Consistency yield similar scores across methods, whereas ECMS scores AgentDreamer substantially worse. Unfortunately, these metrics are unreliable here, as MotionPhysics also notes, and AgentDreamer’s high ECMS reflects more dynamic motion, not worse quality.

    MethodCLIP similarity ↑Overall Consistency ↑ECMS ↓
    PhysFlow0.2440.1811402.3
    MotionPhysics0.2460.178618.7
    AgentDreamer0.2460.18029577.0

[W2] Further validation of Motion-IG

Thank you for the valuable suggestion. We examined agreement with human judgments and sensitivity to description wording.

  1. Agreement with human judgments: We compared both Motion-IG implementations and a VQA-style evaluator (MLLM) against the same human preferences. Motion-IG agrees more closely with human judgments, supporting their ranking accuracy.

    EvaluatorTop-1 agreement ↑Pairwise agreement ↑
    VQA-style evaluator (MLLM)0.3920.512
    Motion-IG (white box)0.5000.738
    Motion-IG (black box)0.6750.738
  2. Robustness to synonymous descriptions: We replaced each description with three synonymous variants while preserving the requested motion and examined consistency with human agreement. Although absolute scores vary, 95–100% of pairwise rankings are preserved, and human agreement remains comparable, supporting robustness to the tested synonymous descriptions.

    DescriptionMotion-IG ↑Ranking preserved ↑Agreement with human preferences ↑
    Original0.714–0.738
    Synonymous variant 10.6250.9500.762
    Synonymous variant 20.6641.0000.738
    Synonymous variant 30.5990.9500.762

[W3] Motion function selection, inaccurate plans, and clarification

Thank you for raising this point.

  • Function selection: We apologize for the unclear presentation. AgentDreamer generates motion functions and coefficients from the instruction and initial scene, without a predefined set. The motion-function table is a post-hoc analysis relating these outputs to familiar physical behaviors, such as rolling rotation and parabolic free flight.
  • Inaccurate plans: Generated functions serve as approximate targets for optimizing physics parameters, while the simulator determines the resulting motion. Physically inconsistent plans yield higher reconstruction losses, suggesting that conflicts with simulator dynamics limit target reproduction (Appendix C.7).
  • Clarification: Our motion-generation task does not use a conventional train/test split. All methods receive the same inputs and optimize their simulators separately for each input. AgentDreamer optimizes physical parameters, including initial velocities and material properties, against agent-generated trajectories for 50 iterations at a constant learning rate of 10⁻³. We use 256 control points, 2-second target trajectories sampled every 0.05 seconds, and an MPM timestep of 10⁻⁴ seconds.

Dear Reviewer j4Np,

We appreciate your careful feedback and valuable comments. Following your feedback, we conducted additional evaluations using diverse metrics and further validation of Motion-IG, along with further analyses of our method. These additions helped us provide stronger evidence for the validity and robustness of our proposed metric and clarified how inaccurate motion plans affect simulation outcomes.

With the discussion ending soon, we would be grateful if you could share your thoughts on our responses. Please let us know if any concerns remain or if there is anything we could clarify further.

Best regards,

The Authors

Thanks for authors' detailed response, which solves most of my concerns. After read all the reviews and authors' rebuttal, I lean to acceptance now.

We sincerely appreciate your careful review and your decision to raise your score; your suggestions have greatly improved our paper.

Reviewer_PyyTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes using an agent to translate text instructions and scene geometry into explicit 3D motion trajectories, which then guide optimization of a differentiable physics simulator. It also introduces Motion-IG, a masked-text-recovery metric for text-motion alignment, and reports gains over PhysFlow and MotionPhysics on 20 instructions across eight scenes, with a 25-participant user study. I find the problem interesting, and the reported improvements over video-guided methods are encouraging. However, I am not yet convinced about the advantage of the central design choice, and the quantitative evidence rests mostly on a metric introduced in the same paper.

Critical strengths
  1. The paper identifies a meaningful limitation of video-based motion guidance: visually plausible generated motion may fail to capture the specific physical behavior requested by the text.
  2. Using explicit 3D motion planning to connect language instructions with physical simulation is a reasonable approach. The use of parametric functions and temporal decomposition also makes the motion planning process more structured and interpretable.
  3. The experimental results are encouraging. AgentDreamer achieves better Motion-IG scores than the selected baselines, and the human evaluation further supports its effectiveness in generating motions that follow the given instructions.
Critical weaknesses
  1. My major concern is that this paper does not sufficiently justify the benefits of its intermediate 3D motion planning stage. While separating motion planning from physics optimization is reasonable, it remains unclear whether this design offers advantages over directly generating simulator controls or optimization objectives. The current ablations also lack stronger alternatives, such as direct trajectory-code generation or predefined motion primitives, making it difficult to isolate the contribution of agentic planning.
  2. The physical correctness of the generated motion is not adequately evaluated. Both Motion-IG and the human study focus on text alignment without directly assessing collision behavior, energy consistency, or object interpenetration. Moreover, the motion plans remain fixed during optimization, and the use of per-particle initial velocities raises questions about how strongly the resulting motion is constrained by physical dynamics.
  3. The reliability of Motion-IG needs further validation. Although the paper shows similar trends between Motion-IG and human evaluations across motion categories, a more direct analysis of their ranking agreement is missing. The comparison with an MLLM judge mainly demonstrates finer score resolution rather than better evaluation accuracy. Additionaly, the black-box evaluation relies on model-reported probabilities instead of actual token likelihoods, making the results less directly comparable.
  4. The experimental evaluation is relatively limited, covering only 20 instructions across eight scenes. There is insufficient analysis of robustness across repeated agent runs, complex interactions and failure cases.
Questions for authors
  1. What are the specific benefits of generating intermediate 3D trajectories instead of directly optimizing simulator controls? Could the authors compare against an agent that proposes and refines simulator parameters using simulation feedback, under a comparable computational budget?
  2. Could the authors provide stronger ablations, such as directly generating executable trajectory code or selecting functions from a predefined library of motion primitives? These comparisons would help clarify whether the improvements come from agentic reasoning or the structured motion representation.
  3. How does the framework handle physically infeasible motion plans? Since the target trajectories remain fixed during optimization, what happens when they conflict with physical constraints? Reporting the motion reconstruction loss and quantitative physical plausibility measurements, particularly for collision and bouncing motions, would be helpful.
  4. Could the authors further validate Motion-IG by reporting its overall ranking agreement with human judgments and comparing it with VQAScore-style metrics? Video-only and trajectory-only evaluations would also help clarify the contribution of each input modality.
  5. Could the authors clarify whether the compared methods share the same simulator and optimization settings, and provide the exact agent model used? It would also be helpful to explain the inconsistent category counts and the slight difference between the full-model scores in Tables 2 and 4.
  6. Could the authors provide additional details on the robustness and efficiency of the framework, including but not limited to failure rates and computational costs? Even a brief analysis would help assess its practical applicability.
7 comments
AuthorsOpenReview ↗

Response to reviewer PyyT [1/3]

[W1 & Q1] Comparison with direct simulator control

Thank you for the valuable comment. The agent specifies the desired motion through intermediate trajectories, which guide the optimization of physics parameters to reproduce the intended behavior and improve physical consistency. To test the benefit of this separation, we compared AgentDreamer with an agent that directly proposes simulator controls and refines them using simulation feedback, as follows:

  1. At similar token usage, direct simulator control with two calls achieves lower Motion-IG than AgentDreamer.

  2. With unrestricted agent calls, direct control improves from 0.573 to 0.615 but remains below AgentDreamer, despite consuming more tokens.

MethodMotion-IGAvg. tokens
AgentDreamer0.714453k
Direct simulator control (two calls)0.573395k
Direct simulator control (unrestricted)0.615711k

The unrestricted setting also leaves some instructions unfulfilled. For example, the agent terminates with nearly stationary motion for the fox head-twisting instruction. We conjecture that inferring suitable parameter updates from simulation feedback is more difficult for the agent than specifying the desired motion. Intermediate trajectories allow gradient-based optimization to handle these parameter updates, reducing the need for repeated agent decisions. The observed performance and token usage support the benefit of this separation.

[W1 & Q2] Comparison with direct code generation and predefined motion functions

We thank the reviewer for this insightful question.

  1. Direct code generation: We would like to clarify that our method already follows this approach: the agent generates executable trajectory code, including motion functions and their coefficients. Our prompts explicitly guide motion reasoning during this process.

  2. Predefined motion functions: We evaluated an alternative that selects motion functions from a predefined set and generates their coefficients. We then excluded spin or ballistic functions to test performance when the relevant functions are unavailable.

    MethodPredefined function set requiredMotion function setMotion-IG ↑
    AgentDreamerNoGenerated0.71
    Predefined motion functionsYesComplete set0.75
    Predefined motion functionsYesExcluding spin0.62
    Predefined motion functionsYesExcluding ballistic0.66

    The full set performs well, but removing spin or ballistic functions lowers the score. Without spin, the smoothstep target initially accelerates from rest instead of capturing the requested gradual deceleration; the optimized simulation barely rotates. Without ballistic functions, sinusoidal targets introduce time-varying acceleration instead of constant gravitational acceleration. These mismatches may hinder simulator optimization, highlighting the dependence on the predefined function set.

AuthorsOpenReview ↗

Response to reviewer PyyT [2/3]

[W2 & Q3] Physical correctness and handling infeasible motion plans

Thank you for the valuable comment.

  • Quantitative evaluation: We agree that physical correctness should be evaluated directly, so we measured ground-contact and energy conservation errors and checked inter-object overlap. Ground-contact error measures the object–ground distance at intended contact events in falling and bouncing scenes; energy error is evaluated where total mechanical energy should be conserved. The physics simulator compensates for the plan’s physical errors, reducing contact and energy errors by about 24× and 4×, respectively. Neither motion source shows inter-object overlap.

    Motion sourceGround-contact error ↓Energy conservation error ↓Inter-object overlap ↓
    Motion plan0.1940.1200
    AgentDreamer0.0080.0280
  • Handling infeasible plans: Physics parameters, including per-particle initial velocities, are optimized only as initial conditions, while subsequent motion follows the simulator’s dynamics. The fixed motion plan guides optimization through the reconstruction loss. Occasionally, the motion plan is physically inaccurate. For example, the soccer-ball plan specifies gravitational accelerations of −8, −6, −4, and −3.2 m/s² across chunks, whereas actual gravity is −9.8 m/s². In such cases, the simulator does not reproduce the plan exactly; it follows the plan as closely as possible while preserving physical correctness. Such plans therefore result in higher reconstruction losses. We provide examples and the corresponding losses in Appendix C.7.

[W3 & Q4] Further validation of Motion-IG

Thank you for suggesting these additional analyses.

  • Probability sources: We agree that model-reported probabilities differ from actual token likelihoods, but we would like to clarify that we compare methods only within each evaluation, where all share the same evaluator, probability source, and protocol, so the comparison remains fair. Indeed, white-box and black-box evaluations show similar trends across methods.

  • Agreement with human judgments: To assess ranking accuracy beyond score resolution, we compared our Motion-IG and a VQA-style evaluator (MLLM) with human evaluation. Top-1 agreement measures agreement on the best method; pairwise agreement measures agreement on each AgentDreamer–baseline comparison. Motion-IG (both of white-box and blackbox) agree more closely with human judgments than MLLM. These results directly support the accuracy of their method rankings.

    EvaluatorTop-1 agreement ↑Pairwise agreement ↑
    VQA-style evaluator (MLLM)0.3920.512
    Motion-IG (white-box)0.5000.738
    Motion-IG (black-box)0.6750.738
  • Information gain by input modality To test the contribution of each input, we compared video-only, trajectory-only, and combined inputs while keeping the evaluator fixed. AgentDreamer ranks highest under all three input settings. With video alone, scores are close across methods. Adding 3D trajectories raises every method’s score and widens AgentDreamer’s lead over the best baseline, indicating that explicit 3D trajectories provide key information that the evaluator does not fully capture from video alone.

    MethodVideo only3D trajectory onlyVideo + 3D trajectory
    PhysFlow0.3940.4410.519
    MotionPhysics0.4370.3510.520
    AgentDreamer0.4560.6050.714

[Q5] Clarification

Thank you for the valuable comment; we update the paper with more clear clarification.

  1. Experimental settings: We matched common simulator and optimization settings, including grid size and initialization, while retaining each paper’s values for method-specific loss weights. Our agent uses GPT Sol 5.6.

  2. Addtional clarification: Both Tables 4 and Figure 6 evaluate the same 20 instructions. Table 4 assigns each instruction to its closest category, while Figure 6 includes it in all applicable categories. Consequently, Figure 6 may count an instruction more than once. We apologize for the unclear presentation and will explicitly state the category assignments. We also corrected rounding errors in Tables 2 and 6.

AuthorsOpenReview ↗

Response to reviewer PyyT [3/3]

[W4] Extension to complex scenarios

Thank you for the helpful suggestion. Following your comment, we added scenarios involving multi-object collision, topology change, explosion, and coupling to the main text of the revised PDF - Section 4.3, Table 7.

In the collision example, AgentDreamer generates motion in which the ball strikes the can and knocks it over. In the topology-change example, the bread separates into two pieces that move apart, while the baselines only tilt or shake it. We also provide visual comparisons in our revised PDF - Section 4.3, Figure 10 and Appendix C.8 - Figure 15.

MethodMulti-object collisionTopology changeExplosionCoupling
PhysFlow−0.73−1.32−1.550.79
MotionPhysics−0.53−1.34−2.580.61
AgentDreamer+0.65+0.520.481.33

AgentDreamer achieves higher Motion-IG than both baselines in all four scenarios. These results extend the demonstrated scope beyond isolated single-object behaviors to object interactions and topology changes.

[Q6] Efficiency and robustness

Thank you for the valuable comment. We provide the computational costs and observed retry behavior below.

  1. Computational costs

    AgentDreamer generates its motion targets before simulator optimization. Its trajectory loss does not require rendering during optimization, whereas both video-guided baselines require rendered outputs for guidance. Thus, our method uses less GPU memory.

    MethodRendering during trainingTraining time (min)GPU memory (GiB)
    MotionPhysicsYes34.8325.05
    PhysFlowYes33.6413.87
    AgentDreamerNo34.6811.69
  2. Retry behavior

    When motion planning failed, we revised the prompt and retried, using up to two or three attempts and taking approximately two to three minutes. Once a suitable motion plan was obtained, simulator optimization generally reproduced the requested motion. The complete pipeline took approximately 37 minutes per instruction.

We sincerely look forward to engaging in further discussions.

Reviewer_obfBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method to construct the 3D scene from text.  Given the initial 3D scene and the text prompt, the LLM model produces the series of motion chunks of the 3D object.  This is used as the reference motion for the differentiable simulation to predict the 3D movements of the 3D model. The initial velocity of the 3D model and its material parameters such as the Young's modulus are computed by optimization such that it follows the reference motion predicted by the LLM.   For the evaluation, Motion-Conditioned Information Gain is proposed, which masks part of the prompt that corresponds to the motion.  The rendered video together with the 3D motion are given to a VLM to predict the masked words that correspond to the 3D motion.  Compared to the baselines based on video generation, the proposed method shows better performance.   As the proposed method is producing a video based on elastic models where only the initial pose is given, the synthesized results will preserve the energy, which can result in better movements that look physically plausible.  The current results only show simple phenomena, and more complex results that involve collision, breakage, external energy injection are not presented.

Critical strengths

-A method based on 3D physical simulation that can produce physically plausible results from prompt.

  • The generated results show better performance compared to video-based methods.
  • The results are simulation results, which can be further used for different physical conditions.
Critical weaknesses
  • Only very simple simulation results can be generated, where a single object's initial condition is optimized.
  • MPM may be overkill: the demonstrated behaviors (rigid rolling, simple elastic bounce, damped oscillation) might be reproduced by a lighter mass-spring system, and the paper does not ablate the simulator choice.
  • The video-based method could produce more interesting effects with complex deformation, such as explosion, coupling, etc with similar costs.
Questions for authors

The scenes produced are rather simple - can more complex scenes with collisions, topology change and coupling be shown?

5 comments
AuthorsOpenReview ↗

Response to reviewer obfB

Thank you for your thoughtful and thorough review of our work.

[W1 & Q1] Extension to complex scenarios

We are truly grateful for this suggestion, which has greatly improved our work. Following your comment, we added scenarios involving multi-object collision, topology change, explosion, and coupling to the main text of the revised PDF - Section 4.3.

  • Qualitative: In the collision example, AgentDreamer generates motion in which the ball strikes the can and knocks it over. In the topology-change example, the bread separates into two pieces that move apart, while the baselines only tilt or shake it. We also provide visual comparisons in revised PDF - Section 4.3, Figure 10.

  • Quantitative: AgentDreamer achieves higher Motion-IG than both baselines in all four scenarios. These results extend the demonstrated scope beyond simple single-object behaviors to more complex scenarios.

    MethodMulti-object collisionTopology changeExplosionCoupling
    PhysFlow−0.73−1.32−1.55+0.79
    MotionPhysics−0.53−1.34−2.58+0.61
    AgentDreamer+0.65+0.52+0.48+1.33

[W2] Choice of MPM and comparison with Spring-Gaus

Thank you for the valuable comment. We agree that a lighter spring–mass simulator can reproduce the demonstrated rolling, bouncing, and oscillation behaviors.

  1. Simulator choice

    We used MPM, as in PhysFlow and MotionPhysics, to evaluate our motion guidance within the same type of simulation framework. Our approach does not inherently require MPM, since the generated trajectories can also guide other differentiable simulators, as shown with Spring-Gaus below.

  2. Comparison with Spring-Gaus

    Following your suggestion, we examined whether a lighter spring–mass simulator is sufficient. Spring-Gaus achieved faster optimization and a higher average Motion-IG on the original instructions, supporting the reviewer’s observation. However, its scores were lower in all four additional scenarios.

    ScenarioMPM (AgentDreamer) ↑Spring-Gaus (AgentDreamer) ↑
    Original scenes (20 instructions)+0.71+0.79
    Multi-object collision+0.65+0.37
    Topology change+0.52−0.29
    Explosion+0.48−0.41
    Coupling+1.33−0.78

In summary, a lighter simulator suffices for simple motions, whereas MPM is better suited to the complex behaviors in [W1 & Q1], which supports our choice of MPM. We included this simulator ablation in the revised paper. We thank you for this suggestion, which allowed us to validate our choice of MPM and further strengthen the paper.

[W3] Comparison with video-based guidance

Thank you for the valuable comment. We agree that video-generation models can produce complex effects such as explosions and coupling. However, we would like to clarify that generating a plausible video does not ensure that its motion can be transferred accurately into a 3D simulation.

  1. Video-based methods on complex effects

    MotionPhysics and PhysFlow, our baselines in [W1 & Q1], already use video guidance. Yet in the explosion, coupling, collision, and topology-change scenarios, both scored lower than AgentDreamer.

  2. Limitation of 2D video guidance

    Video-guided methods supervise the simulation only through 2D signals: MotionPhysics scores rendered frames with a video diffusion model, and PhysFlow matches optical flow extracted from a generated video. Since 2D motion does not uniquely determine 3D trajectories, and flow can be unreliable under similar colors, weak textures, or occlusion, an effect visible in a video may not be reproduced in the simulation. AgentDreamer instead specifies explicit 3D targets.

  3. Costs

    Training time is similar across methods (about 34 min), but the video-based methods require more GPU memory because they must render frames during optimization (13.87 GiB for PhysFlow and 25.05 GiB for MotionPhysics vs. 11.69 GiB for AgentDreamer). AgentDreamer therefore achieves its higher scores on complex effects more efficiently than the video-based methods.

Thus, while video generation can depict such effects, guiding the physics simulator with explicit 3D trajectories is more reliable than 2D guidance. As physically realistic video models are beginning to emerge, combining their expressiveness with explicit 3D motion planning is a promising direction for future work.

We sincerely look forward to engaging in further discussions.

Dear Reviewer obfB,

Thank you again for your thoughtful comments and concerns. Following your suggestions, we conducted additional experiments on more complex scenarios and evaluated our approach with a different simulator. Your feedback helped us provide stronger evidence of our method’s advantages over video-based baselines in complex scenarios and demonstrate its applicability across different simulators.

As the discussion phase is coming to a close, could you kindly let us know whether our responses have adequately addressed your concerns, or whether you have any further questions? We would be happy to provide any additional clarification.

Best regards,

The Authors

Reviewer_obfBOpenReview ↗

concerns addressed

Thank you for the additional results. The additional results that include multiple object collisions and topology change will greatly strengthen the paper. The spring-mass results are also helpful for understanding in which scenes the MPM could be more helpful while simple scenes can be managed by the spring-mass system. If the additional section and tables are added, I am happy to raise my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal summary

We sincerely appreciate the reviewers for their constructive comments and suggestions. We provide a summary of our work’s contribution and the key outcomes of the discussion. Revisions are colored blue in the paper.


Strengths

Relevant technical contributions

  • Interesting and meaningful problem of translating text instructions into the requested physical behavior [PyyT, j4Np].
  • Effective contribution through agent-guided motion planning and physical simulation for text-controllable motion generation [obfB, PyyT, j4Np].
  • Identifies a meaningful limitation of video-based motion guidance: visually plausible motion may miss the physical behavior requested in the text [PyyT].
  • Proposes Motion-IG, a reasonable metric for evaluating text-motion alignment [j4Np].
  • Simulation results can be further used for different physical conditions [obfB].

Discussion

We have strengthened our work based on the discussion as follows:

DiscussionsR-obfBR-PyyTR-j4NpOur Response
Generalization to complex scenarios✔✔Main; Appendix C.8
Generalization to different physics simulators✔Main
Comparison with video-priors✔Main; Appendix C.8
Comparison with auto-agent loops✔Appendix C.1
Additional ablation studies on motion planning✔Appendix C.2
Additional analysis on Motion-IG✔✔Main; Appendix C.3, C.5
Evaluation with other metrics & handling motion plans✔✔Main; Appendix C.4; Appendix C.7
Computational efficiency & robustness✔Appendix C.6; Rebuttal - PyyT [Q6]
Clarification✔✔Rebuttal - PyyT [Q5], j4Np [W3-b]
Review scores333As of Nov 18th (end of reviewer–author discussion)
Score Acknowledged✔✔As of Nov 18th (end of reviewer–author discussion)

[Key outcomes of the discussion]

  • Complex scenarios [obfB, PyyT]: AgentDreamer outperforms both video-guided baselines on multi-object collision, topology change, explosion, and coupling.
  • Simulator choice [obfB]: The trajectories also guide a lighter spring–mass simulator (Spring-Gaus), which suffices for simple motions, while MPM better handles complex behaviors.
  • Video priors [obfB]: Video-guided methods are limited by 2D supervision and require more GPU memory at similar training time (13.87 and 25.05 vs. 11.69 GiB).
  • Intermediate 3D trajectories [PyyT]: AgentDreamer outperforms an agent that directly refines simulator controls, even when the latter uses more tokens (0.714 vs. 0.615 Motion-IG), and does not depend on a predefined motion library.
  • Physical correctness [PyyT]: The simulator reduces the plan’s ground-contact and energy errors by about 24× and 4×, with no inter-object overlap.
  • Inaccurate motion plans [PyyT, j4Np]: Infeasible plans raise the reconstruction loss instead of producing unphysical motion.
  • Motion-IG reliability [PyyT, j4Np]: Motion-IG agrees with human rankings more closely than an MLLM judge and CLIP-based metrics (pairwise 0.738 vs. 0.512 and ≤ 0.55), and preserves 95–100% of rankings under synonymous descriptions.
  • Efficiency [PyyT]: The full pipeline takes about 37 minutes per instruction, including occasional planning retries.
  • Clarifications [PyyT, j4Np]: All methods share common simulator settings and inputs, and AgentDreamer generates motion functions without a predefined set.

Contribution

Our work makes the following contributions:

  • AgentDreamer, a novel framework for text-controllable physics simulation in which an agent translates language instructions into explicit 3D motion plans and a differentiable simulator grounds them in physically feasible motion.
  • Motion-IG, a novel information-theoretic metric for evaluating simulated motion, which measures how well motion matches the physical behavior described in text by quantifying how much it helps recover masked motion-related words.
  • Empirical evidence that explicit 3D motion planning outperforms video-based guidance and direct simulator control, generalizes to complex scenarios and different simulators, and yields physically consistent motion.

Best regards,

Submission35650 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.