acceptodds
Under review as a conference paper at ICLR 2027

CarPhy: A Carrier Physics Aware Pose Correction Agent for Interactive World Models

Abstract

Interactive world models (IWMs) generate virtual environments that evolve in response to user actions, enabling users to explore and interact with generated worlds. However, they can overlook physical interactions between the camera carrier and the surrounding environment, producing implausible motion such as passing through solid walls. To address this issue, we propose CarPhy, a pose correction agent that explicitly models Carrier Physics to calibrate IWMs to generate plausible videos. Since different scenes require different physical interpretations and modeling steps, CarPhy equips the agent with multimodal scene and carrier context, along with tools for semantic analysis, world editing, and motion rollout. Guided by this context, the agent adaptively selects and combines tools to determine how the environment affects the carrier and simulate the resulting motion. We further construct a test set spanning diverse terrain and obstacle configurations to evaluate physical event recognition and pitch-response timing. Experiments demonstrate improved physical event recognition and more accurate pitch-response timing across multiple VLM and LLM backbones.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Interactive World Models, Camera Pose Correction, Physical Simulation, Tool-Using Agents, Physically Consistent Video Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1HebOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors notice that current interactive world models are likely to overlook physical interactions between carriers and the surrounding environments, which leads to a implausible motion though with high quality. To address this problem, they proposed CarPhy, an agentic pose-correction framework that improves the physical plausibility of camera motion in interactive world models. Given recent visual observations, camera poses, carrier specifications, and the user’s intended motion, CarPhy constructs a physical scene and applys verification loop checks the trajectory for contact plausibility, pose consistency, and alignment with the motion intent, prompting further refinement or abstention when necessary. Experiments on a curated benchmark of 187 samples demonstrate improved physical-event recognition and pitch-response timing across multiple model backbones.

Critical strengths
  1. The topic is well-motivated; as generative world models often struggle to account for physical constraints during video generation, leading to implausible motion. The authors take advantage of an agent to interpret visual observations while the physics engine computes the carrier’s response to the intended motion. The combination allows semantic understanding to inform simulation, offering a practical approach to improving physical plausibility

  2. The paper is clearly written and well organized, with figures that effectively illustrate the proposed agentic framework and tables that clearly present its performance gains over the baselines.

Critical weaknesses
  1. The contribution of this work is limited; CarPhy primarily uses an agent’s semantic and commonsense reasoning to interpret and refine scene assets, while established physics tools compute the carrier’s trajectory. As the framework builds on an existing coding-agent harness, its contribution appears to lie mainly in task-specific integration and workflow design.

  2. The concerns about verification. Passing verification does not guarantee that the scene has been correctly interpreted. For example, misclassifying a rigid obstacle as traversable vegetation and disabling its collision could allow a trajectory based on incorrect physical assumptions to pass verification. Although scene edits are constrained, the verifier cannot independently establish the correctness of the underlying semantic interpretation

  3. Constraining the carrier trajectory alone does not guarantee physical consistency in the generated video. The video generator may alter obstacle geometry, height, or contact relationships, so a trajectory that is feasible in the reconstructed scene may not remain consistent with the generated environment.

  4. Current experiments is mainly done through a curated dataset containing 187 samples and only 8 samples are blocked motion. The primary fine-grained metric measures the time at which the pitch angle first reaches specified thresholds. This provides a limited assessment of full-trajectory accuracy, contact dynamics, and physical responses during turning or continuous interaction.

Questions for authors

The proposed method relies on an iterative agent loop for scene refinement, trajectory simulation, and verification, which may introduce substantial computational overhead and latency. Reporting end-to-end latency and computational cost, alongside the average number of agent iterations and simulation calls, would help assess its practicality for interactive world modeling.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1Heb (1/3)

We thank you for recognizing the motivation and clarity of the work! We clarify the task-specific contribution and address verification, downstream consistency, and practical cost.

Q1. Is the contribution mainly integration on an existing coding-agent harness?

The contribution of this work is limited; CarPhy primarily uses an agent’s semantic and commonsense reasoning to interpret and refine scene assets, while established physics tools compute the carrier’s trajectory. As the framework builds on an existing coding-agent harness, its contribution appears to lie mainly in task-specific integration and workflow design.

A1. Thank you for the opportunity to clarify our contribution. CarPhy’s contribution goes beyond integrating tools into an existing coding-agent harness. Even with access to the same tools, a general coding-agent harness struggles to solve this task. Therefore, our contribution is to identify camera-carrier physics as a missing component in interactive world models and design an agent and its orchestration to address it.

First, a general coding-agent harness cannot effectively solve this task through tool access alone. Pi Full Tools achieves 43.8% Macro-F1, compared with 67.5% for CarPhy (see Table 1 below). This gap shows the importance of designing how the agent interprets the scene, models carrier–environment interactions, and uses simulation feedback.

Second, our host already provides a substantial part of the solution. It implements the physical modeling procedure we developed for this task and achieves 66.0% Macro-F1. Its performance demonstrates the contribution of CarPhy’s modeling design, which is absent from the general coding-agent harness.

Third, the agent adds flexibility through scene-dependent decisions and iterative refinement. It can revise physical interpretations that the default host procedure leaves unresolved, such as distinguishing traversable vegetation from rigid obstacles. Full CarPhy improves Macro-F1 to 67.5%, while a single repair achieves 63.2%, supporting the value of feedback-guided iteration over an isolated edit.

We report these comparisons in Section 4.7 (Pages 9–10) and Table 6 (Page 10) of the revised manuscript, with further details in Appendix I (Page 26).

Table 1. Overall comparison and component ablations of CarPhy (Kimi-K3, PU/PD/NA).

MethodmAcc (%)Macro-F1 (%)Answer rate (%)
Direct inference42.443.8100.0
Pi Basic48.349.7100.0
Pi Full Tools$\dagger$44.643.8100.0
Host only56.266.084.9
One-shot repair52.763.280.4
Full CarPhy59.167.587.7

Note. BL has only eight cases, making its class-level estimates unstable. For a more reliable comparison, event metrics use the same 179 PU/PD/NA cases for all methods. †\dagger: Pi harness with access to the same tools.


Q2. The concerns about verification. How reliable is CarPhy’s verification when an incorrect scene interpretation can still produce a trajectory that passes physical checks?

The concerns about verification. Passing verification does not guarantee that the scene has been correctly interpreted. For example, misclassifying a rigid obstacle as traversable vegetation and disabling its collision could allow a trajectory based on incorrect physical assumptions to pass verification. Although scene edits are constrained, the verifier cannot independently establish the correctness of the underlying semantic interpretation

A2. Thank you for raising this concern about semantic verification. CarPhy’s verification extends beyond the deterministic pose verifier. It combines semantic checks during scene repair with physical checks during simulation. And we also find that this combined verification process is reliable in practice, with 94.7% of accepted semantic repairs judged correct in our human audit.

For the human audit, we manually assessed all 19 accepted agent repairs against the original visual observations and checked whether incorrect repairs passed physical verification. 18 repairs were correct, and the failure raised by the reviewer occurred in one case, where a shadow on flat ground was misinterpreted as a slope (Table 2).

We have included the audit procedure, results, and failure case in Appendix N (Page 30 of the revised manuscript).

Table 2. Reliability of semantic repairs.

MeasureCount (n/N)Rate (%)
Correct semantic repairs18/1994.7
Incorrect repairs passing verification1/1—

Note. The audit covers the 19 accepted agent repairs. Denominators are all audited repairs and the incorrect repairs.

AuthorsOpenReview ↗

Response to Reviewer 1Heb (2/3)

Q3. Does constraining the carrier trajectory guarantee physical consistency in the generated video?

Constraining the carrier trajectory alone does not guarantee physical consistency in the generated video. The video generator may alter obstacle geometry, height, or contact relationships, so a trajectory that is feasible in the reconstructed scene may not remain consistent with the generated environment.

A3. Thank you for highlighting the connection between trajectory correction and video consistency. Within our scope of short- to medium-duration interactions, CarPhy's trajectory correction improves the physical consistency of generated videos. Figure 8 demonstrates consecutive corrections over 8–10 seconds, and our additional benchmark experiments confirm the benefits of trajectory correction on both video generators.

We identified two concurrent benchmarks addressing related physical interactions, WorldExam [2] and WorldRoamBench [1]. At the time of evaluation, WorldRoamBench had released its test cases but not its evaluation code, while WorldExam’s test suite and evaluation code were unavailable. We therefore select 20 wall-interaction and 9 uphill/downhill cases from WorldRoamBench and reimplement its Clipping and Terrain Following evaluation protocols.

We report the generated-video results in Section 4.8 and Table 7 (Page 10 of the revised manuscript). The evaluation setup and protocol reimplementation are detailed in Appendix M (Page 30).

Table 3. Generated-video physical evaluation on selected WorldRoamBench cases.

GeneratorPose conditionClipping score ↑Terrain Following score ↑
Lyra 2.0Without correction35.022.2
Lyra 2.0With CarPhy75.066.7
LingBot World 2.0Without correction15.011.1
LingBot World 2.0With CarPhy75.055.6

Note. Terrain Following evaluates uphill/downhill behavior, corresponding to the PU/PD events studied in our work [1]. Both scores use our reimplementation of WorldRoamBench's evaluation protocols [1].

With paired inputs, seeds, and generation budgets, CarPhy increases Clipping scores from 35.0 to 75.0 on Lyra 2.0 and from 15.0 to 75.0 on LingBot World 2.0. Terrain Following scores rise from 22.2 to 66.7 and 11.1 to 55.6, respectively (Table 3). These results show that correcting the carrier trajectory reduces wall pass-through and improves terrain following in the final generated videos.

Code. Our WorldRoamBench evaluation code is available through the anonymous link provided in Appendix M (Page 30 of the revised manuscript).


Q4. Can the authors expand the evaluation beyond pitch-event timing to assess full-trajectory accuracy, contact dynamics, and physical responses during turning or continuous interaction, with more blocked-motion cases?

Current experiments is mainly done through a curated dataset containing 187 samples and only 8 samples are blocked motion. The primary fine-grained metric measures the time at which the pitch angle first reaches specified thresholds. This provides a limited assessment of full-trajectory accuracy, contact dynamics, and physical responses during turning or continuous interaction.

A4. We appreciate your suggestions for a more comprehensive evaluation. We have expanded the evaluation to more complex scenarios from an interactive world model benchmark and reported full-trajectory position and rotation errors. The results show that CarPhy improves physical responses in generated videos, reducing wall pass-through and improving terrain following.

We evaluate 20 wall-interaction cases, including turning motions, and 9 uphill/downhill cases from WorldRoamBench [1], using our reimplementation of its evaluation protocols. CarPhy increases Clipping scores from 35.0 to 75.0 on Lyra 2.0 and from 15.0 to 75.0 on LingBot World 2.0. Terrain Following scores improve from 22.2 to 66.7 and 11.1 to 55.6, respectively (Table 4). For continuous interaction, Figure 8 demonstrates 8–10 seconds of generation through 4–5 consecutive correction windows.

AuthorsOpenReview ↗

Response to Reviewer 1Heb (3/3)

Table 4. Generated-video physical evaluation on WorldRoamBench cases, including turning motions (20 wall-interaction and 9 uphill/downhill cases, 0–100 scale).

GeneratorPose conditionClipping score ↑Terrain Following score ↑
Lyra 2.0Without correction35.022.2
Lyra 2.0With CarPhy75.066.7
LingBot World 2.0Without correction15.011.1
LingBot World 2.0With CarPhy75.055.6

Note. Both scores use our reimplementation of WorldRoamBench’s evaluation protocols [1]. Terrain Following evaluates uphill/downhill behavior, corresponding to the PU/PD events studied in our work [1].

The trajectory results also show why pointwise pose errors alone do not reliably measure the effectiveness of physical correction. Raw intent achieves the lowest position and rotation errors (Table 5), yet produces worse physical behavior in the generated videos. Figure 14 in Appendix K (Page 28) shows how differences in response timing and magnitude can produce large pointwise errors despite a physically meaningful correction.

We have included the trajectory results and supporting visualizations in Appendix K (Pages 27–28 of the revised manuscript).

Table 5. Full-trajectory position and rotation errors.

MethodPosition RMSE (m) ↓Mean rotation error (deg) ↓
Raw intent0.4635.42
Host only0.93911.03
One-shot repair0.8018.35
Full CarPhy0.8699.19

Note. Errors are averaged over the trajectories each method returns on the PU/PD/NA cases. Rotation error is the mean angular difference from ground truth at matched timestamps.


Q5. What are the latency, computational cost, average iterations, and simulation calls?

The proposed method relies on an iterative agent loop for scene refinement, trajectory simulation, and verification, which may introduce substantial computational overhead and latency. Reporting end-to-end latency and computational cost, alongside the average number of agent iterations and simulation calls, would help assess its practicality for interactive world modeling.

A5. Thank you for asking about the computational cost and latency. Full CarPhy takes 250.2 s per run, averaging 3.67 agent iterations, 3.67 model calls, and 2.75 simulator calls, with 74.3k recorded input and output tokens per run (Table 6). We also evaluate a variant in which the agent explicitly orchestrates the recurring modeling and execution steps consolidated into the host.

Notably, this latency does not preclude CarPhy’s use as a data simulator in downstream workflows that interleave pose correction with IWM generation, where high-quality video generation is prioritized over low latency.. Compared with this variant, Full CarPhy reduces model calls from 8.67 to 3.67 and runtime from 532.4 s to 250.2 s, illustrating the efficiency gained by consolidating recurring operations. These times start from cached reconstructed observations. The initial reconstruction and segmentation add about 6 s per run. At this cost, CarPhy does not yet support real-time updates for two-second interaction windows.

We have included the cost measurements and experimental setup in Appendix O (Page 31 of the revised manuscript).

Table 6. Computation cost.

MethodTime per run (s)Agent iterationsModel callsSimulator callsRecorded tokens per run (k)
Host only41.20.000.001.670.0
One-shot repair146.21.461.382.4616.9
CarPhy with agent-orchestrated execution532.48.678.673.2196.8
Full CarPhy250.23.673.672.7574.3

Note. Means cover all 24 attempts per method, starting from cached reconstructed observations. Tokens sum recorded input and output usage.


References

[1] Xu et al. WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models. arXiv:2606.31672, 2026.

[2] Yang et al. WorldExam: Benchmarking World Models from Apparent Appearance to Inherent Reactivity. arXiv:2608.02603, 2026.

Reviewer_KrWJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CarPhy, an agentic framework for correcting camera trajectories in interactive world models by explicitly modeling the physical response of the camera carrier. Given recent observations, 3D geometry, carrier specifications, and the user's intended motion, the agent constructs and revises scene assets, runs physics simulations in PyBullet, and checks the resulting trajectories against physical and motion constraints.

The authors introduce a benchmark with 187 samples covering pitch-up, pitch-down, blocked, and unaffected events. CarPhy improves event recognition and pitch-response timing over direct inference and general coding-agent baselines across several backbone models. The paper also shows qualitative examples of using corrected camera trajectories for video generation.

Critical strengths

1. The harness separates semantic reasoning from physical simulation.

I like how the authors distinguish between understanding what an object represents and determining its physical effect on the carrier. For example, vegetation and rigid obstacles may have similar geometry but should produce different collision responses.

The agent uses visual evidence to select semantic repairs, while the actual carrier motion is computed through simulation. The structured scene-editing interface, numerical verification, and abstention mechanism also prevent the agent from directly producing arbitrary trajectories without physical checks.

2. The evaluation tests both event recognition and response timing.

The paper evaluates whether the model recognizes the correct physical event and whether the predicted pitch response occurs at the correct time. I find this distinction useful because identifying an uphill transition does not necessarily mean predicting when the carrier starts climbing.

The comparisons across multiple backbones also show that the harness improves physical event recognition beyond direct VLM inference. For example, DeepSeek-V4-Flash improves from 47.5 to 76.4 Macro-F1 with CarPhy.

Critical weaknesses

1. The contribution of the agentic reasoning is not sufficiently isolated from the physics simulation and toolkits.

The paper compares CarPhy with direct inference, Pi Basic, and Pi with full tool access. However, these experiments do not fully explain which components of CarPhy are responsible for the improvements.

In particular, the host already constructs an initial scene and performs a default physics rollout before the agent is invoked (Appendix B). When the default candidate passes verification and no visual conflict is identified, the agent is instructed to accept it directly.

It is therefore unclear how often the agent actually performs semantic analysis, scene repair, or additional physics reasoning, rather than accepting the precomputed result.

I would like to see a comparison against the host's default physics rollout without agent intervention, along with separate ablations for semantic repair, scene editing, and iterative verification.

This is particularly important given related 2026 work such as Physical Simulator-in-the-Loop Video Generation (Foo et al., 2026) and Code World Model (Chen et al., 2026), which also incorporate executable simulation or coding agents into world modeling.

2. The physical verification criteria do not fully validate the predicted carrier dynamics.

The deterministic verifier is a useful part of the system, but its checks are limited in several ways.

Appendix H states that independent rollout reproduction checks vertical position with a tolerance of 0.20 m, but does not require agreement in pitch or roll. The verifier also allows a penetration increase of up to 0.10 m relative to the reference placement and does not enforce a separate absolute rollover-angle threshold.

This is a concern because predicting pitch response is one of the paper's primary evaluation targets. A trajectory could satisfy the implemented verification criteria while still containing an incorrect pitch response.

I would like to see a more direct evaluation of whether trajectories that pass the verifier actually match the reference carrier dynamics. Reporting verifier pass rates alongside pitch and pose errors would help establish how reliable these checks are.

3. The benchmark is relatively small and covers a limited range of physical interactions.

The benchmark contains 187 samples, including 42 pitch-up, 37 pitch-down, 8 blocked, and 100 unaffected cases. The blocked category is particularly small, making it difficult to assess how well the method handles collisions and impassable obstacles.

The evaluation also fixes the user action to forward motion over a two-second prediction horizon. It does not test more complex action sequences, turning while traversing uneven terrain, or longer interactions where small pose errors may accumulate.

I would like to see additional collision scenarios and longer rollouts. Since the paper targets interactive world models, evaluating consecutive corrections over multiple interaction steps would also help establish whether the method remains consistent beyond a single short prediction window.

4. The paper does not sufficiently evaluate the final generated videos.

The motivation is to improve physically plausible video generation in interactive world models. However, most quantitative results evaluate event classification and pitch timing, rather than the final generated video.

Figure 8 provides examples using Lyra 2.0 and LingBot World 2.0, but there is no systematic quantitative comparison of the generated videos before and after pose correction.

It would be useful to measure camera trajectory consistency, collision violations, and physical event correctness in the generated videos, rather than assuming that better input camera trajectories always produce better physical behavior in the video model.

Questions for authors
  1. How frequently does the agent actually modify the default scene or trajectory? Appendix B states that the host generates and verifies a default physics candidate before the agent acts. Could the authors report the percentage of samples that accept the baseline directly, require semantic repair, perform additional rollouts, or abstain?

  2. How much does the agent contribute beyond the default physics pipeline? Could the authors provide ablations for the host's default rollout, semantic repair, and iterative agentic refinement separately?

  3. Why does the verifier only check height agreement during independent rollout reproduction? Since pitch response is a primary target, could the authors explain why pitch and roll are not checked against the reproduced simulator trajectory?

  4. How well does the method generalize to longer and more complex interactions? The current benchmark uses two-second predictions with forward motion only. Could the authors evaluate longer rollouts, turning actions, or sequential pose corrections?

  5. Can the authors quantitatively evaluate the generated videos? Figure 8 shows qualitative improvements, but it would help to measure whether the corrected poses reduce physical violations in the final IWM output.

  6. How reliable are the semantic repairs? The verifier checks consistency with the edited scene, but cannot establish whether disabling a collision or removing a support surface is semantically correct. Could the authors report semantic-edit accuracy or failure cases where an incorrect repair still passes physical verification?

6 comments
AuthorsOpenReview ↗

Response to Reviewer KrWJ (1/3)

We thank you for the constructive review and for recognizing our problem formulation and method design! We address your suggestions below.


Q1. How often does the agent accept the baseline, repair the scene, perform additional rollouts, or abstain?

A1. Thank you for asking about the agent’s behavior in practice. The agent directly accepts the host candidate in 71.9% of runs, returns a repaired candidate in 9.9%, abstains in 11.5%, and exhausts its step budget without a final answer in 6.8% (Table 1). Across all runs, the agent attempts a scene repair in 26.6% and triggers additional physics rollouts in 18.8%. Accepting a suitable candidate is intentional: the host already incorporates CarPhy's modeling procedure. Adaptive reasoning addresses scene-dependent issues left unresolved by that candidate, such as distinguishing traversable foliage from rigid obstacles, and uses simulation feedback to revise the scene model.

We have included these statistics in Appendix J (Page 26 of the revised manuscript).

Table 1. Acceptance, repair, additional rollouts, and abstention.

Outcome or actionRuns (n)Share (%)
Direct acceptance of the host candidate13871.9
Accepted repaired candidate199.9
Explicit abstention2211.5
Step budget exhausted without an answer136.8
Attempted scene repair5126.6
Agent-triggered additional rollout3618.8

Note. The first four rows are mutually exclusive outcomes. The last two count runs with each agent action and can overlap.


Q2. How much does the agent contribute beyond the default physics pipeline? Could the authors provide ablations for the host's default rollout, semantic repair, and iterative agentic refinement separately?

A2. We appreciate your suggestion to separate these contributions. Semantic repair combined with iterative refinement improves Macro-F1 from 66.0% to 67.5% over the host’s default physics rollout (see Table 2 below).

The ablation results are summarized in Table 2. We compare the host’s default rollout, one additional semantic-repair step (i.e., One-shot repair), and the full iterative agent. All three use the same reconstruction and simulator and retain CarPhy’s physical modeling procedure. A single repair reduces Macro-F1 to 63.2%, whereas the full iterative agent achieves 67.5%, outperforming both Host-only and the single-repair variant. This comparison shows the importance of simulation-guided iteration for effective semantic correction.

We report these comparisons in Section 4.7 (Pages 9–10) and Table 6 (Page 10) of the revised manuscript, with further details in Appendix I (Page 26).

Table 2. Overall comparison and component ablations of CarPhy (Kimi-K3, PU/PD/NA).

MethodmAcc (%)Macro-F1 (%)Answer rate (%)
Pi Full Tools$\dagger$44.643.8100.0
Host only56.266.084.9
One-shot repair52.763.280.4
Full CarPhy59.167.587.7

Note. BL has only eight cases, making its class-level estimates unstable. For a more reliable comparison, event metrics use the same 179 PU/PD/NA cases for all methods. †\dagger: Pi harness with access to the same tools.


Q3. Why does replay verification check height but not pitch or roll?

A3. Thank you for pointing out the need to explain this verification choice. Pitch and roll are constrained by the simulated carrier dynamics and contact geometry. Semantic checks of the reconstructed scene and height-replay verification provide additional indirect constraints, although replay does not explicitly enforce pitch/roll agreement. The verifier also has two related limits. Penetration is checked relative to the initial placement, allowing up to 0.10 m of additional penetration to accommodate existing placement mismatch. Rollover is not checked by a separate roll threshold, and only the pitch rate is bounded.

Full CarPhy returns a trajectory that passes verification in 83.8% of runs, compared with 73.2% for Host-only. Over each method's returned trajectories, the pitch MAE against ground truth is 4.05° for CarPhy and 4.38° for Host-only. Table 3 also reports position and overall rotation errors.

We have included these verification and pose-error results in Appendix K (Page 27 of the revised manuscript).

AuthorsOpenReview ↗

Response to Reviewer KrWJ (2/3)

Table 3. Verification pass rates and pose accuracy of returned trajectories.

MethodVerification pass rate (%)Position RMSE (m) ↓Mean rotation error (deg) ↓Pitch MAE (deg) ↓
Host only73.20.93911.034.38
Full CarPhy83.80.8699.194.05

Note. Pass rates use all 179 PU/PD/NA runs. Errors are measured on all trajectories each method returns (Host 152, CarPhy 157), with ground-truth poses at matched timestamps. Rotation error is the mean angular difference, and pitch MAE is the mean absolute pitch error.


Q4. Can the authors evaluate more collision scenarios and longer, more complex interactions?

A4. Thank you for encouraging us to broaden the evaluation. We expand the evaluation with 20 additional wall-interaction scenarios from WorldRoamBench [1], a concurrent interactive world model benchmark covering longer interactions and more complex actions. CarPhy improves generated-video physical consistency on these scenarios under the benchmark’s evaluation protocol.

Our evaluation setup is as follows. We identified two concurrent benchmarks addressing related physical interactions, WorldExam [4] and WorldRoamBench [1]. At the time of our evaluation, WorldRoamBench had released its test cases but not its evaluation code, while WorldExam’s test suite and evaluation code were unavailable. We therefore use WorldRoamBench’s official cases and our reimplementation of the corresponding evaluation protocol, and conduct experiments on two IWMs. CarPhy increases the Clipping score from 35.0 to 75.0 on Lyra 2.0 and from 15.0 to 75.0 on LingBot World 2.0 (Table 4). Also, Figure 8 in the main paper demonstrates 8–10 seconds of generation with 4–5 consecutive two-second correction windows.

We report the additional wall-interaction experiments in Section 4.8 and Table 7 (Page 10 of the revised manuscript), with evaluation details in Appendix M (Page 30).

Table 4. Generated-video clipping scores on 20 additional wall-interaction cases (0–100 scale).

GeneratorWithout correction ↑With CarPhy ↑
Lyra 2.035.075.0
LingBot World 2.015.075.0

Code. Our WorldRoamBench evaluation code is available through the anonymous link provided in Appendix M (Page 30 of the revised manuscript).


Q5. Can the authors quantitatively evaluate the generated videos? Figure 8 shows qualitative improvements, but it would help to measure whether the corrected poses reduce physical violations in the final IWM output.

A5. We appreciate your suggestion to evaluate the generated videos quantitatively. Yes, we have added a quantitative comparison of generated videos with and without CarPhy. CarPhy improves physical consistency in the final videos on both Lyra 2.0 and LingBot World 2.0. We select 20 wall-interaction and 9 uphill/downhill cases from WorldRoamBench [1] and evaluate them using our reimplementation of its Clipping and Terrain Following protocols.

As shown in Table 5 below, under paired inputs, seeds, and generation budgets, Clipping scores improve by 40.0 points on Lyra 2.0 and 60.0 points on LingBot World 2.0. Terrain Following scores increase from 22.2 to 66.7 and 11.1 to 55.6. Thus, the benefit of pose correction is also observed in the final generated videos.

The uphill/downhill experiments are reported alongside the wall-interaction results in Section 4.8 and Table 7 (Page 10 of the revised manuscript), with evaluation details in Appendix M (Page 30).

Table 5. Generated-video physical evaluation on selected WorldRoamBench cases.

GeneratorPose conditionClipping score ↑Terrain Following score ↑
Lyra 2.0Without correction35.022.2
Lyra 2.0With CarPhy75.066.7
LingBot World 2.0Without correction15.011.1
LingBot World 2.0With CarPhy75.055.6

Note. Terrain Following evaluates uphill/downhill behavior, corresponding to the PU/PD events studied in our work [1].

AuthorsOpenReview ↗

Response to Reviewer KrWJ (3/3)

Table 5. Generated-video physical evaluation on selected WorldRoamBench cases.

GeneratorPose conditionClipping score ↑Terrain Following score ↑
Lyra 2.0Without correction35.022.2
Lyra 2.0With CarPhy75.066.7
LingBot World 2.0Without correction15.011.1
LingBot World 2.0With CarPhy75.055.6

Note. Terrain Following evaluates uphill/downhill behavior, corresponding to the PU/PD events studied in our work [1].


Q6. How reliable are semantic repairs? Can the authors report semantic-edit accuracy or failure cases where incorrect repairs pass physical verification?

A6. Thank you for raising this question about semantic-repair reliability. We conducted a human audit of the accepted semantic repairs and found that 94.7% were correct. We have included the audit procedure, results, and identified failure case in Appendix N (Page 30 of the revised manuscript).

We assessed all 19 accepted repairs against the original visual observations. Of these, 18 were correct, and one incorrect repair passed physical verification (Table 6). In that case, the agent interpreted a shadow on flat ground as a slope.

Semantic repair with iterative refinement also improves task performance. Compared with Host-only, Full CarPhy increases mAcc from 56.2% to 59.1% and Macro-F1 from 66.0% to 67.5% (Table 7).

Table 6. Reliability of semantic repairs.

MeasureCount (n/N)Rate (%)
Correct semantic repairs18/1994.7
Incorrect repairs passing verification1/1—

Note. The audit covers the 19 accepted agent repairs. Denominators are all audited repairs and the incorrect repairs within that set, respectively.

Table 7. Task-level benefit of adaptive repair and iteration (Kimi-K3, PU/PD/NA).

MethodmAcc (%) ↑Macro-F1 (%) ↑
Host only56.266.0
Full CarPhy59.167.5

Q7. How does CarPhy differ from other simulator- or coding-agent-based methods?

A7. Thank you for suggesting a clearer comparison with these methods. Unlike PSIVG [2] and Code World Model [3], CarPhy explicitly models the camera carrier’s own physical interaction with the environment. This allows us to determine how the carrier should move, tilt, or stop under the user’s motion command. Detailed differences are as follows.

Table 8. Comparison with simulator- and coding-agent-based methods.

PropertyPSIVG [2]Code World Model [3]CarPhy
Modeling focusForeground-object dynamicsObjective world stateCamera-carrier dynamics
Camera-carrier physicsNoNoYes
Requires trainingNoYesNo

PSIVG [2] corrects foreground-object dynamics while using camera motion recovered from a template video. Code World Model [3] evolves executable world state and renders proxy conditions through a fine-tuned video model. It includes camera control, but does not evaluate the observation-conditioned carrier-pose correction studied here.


References

[1] Xu et al. WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models. arXiv:2606.31672, 2026.

[2] Foo et al. Physical Simulator-in-the-Loop Video Generation. arXiv:2603.06408, 2026.

[3] Chen et al. Code World Model: Coding Agent as World Brain. arXiv:2608.25927, 2026.

[4] Yang et al. WorldExam: Benchmarking World Models from Apparent Appearance to Inherent Reactivity. arXiv:2608.02603, 2026.

Reviewer_wKrHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CarPhy, an agent-based camera pose correction framework for interactive world models. It reconstructs scene geometry from visual observations and employs semantic analysis, scene editing, and physics simulation to predict physically plausible carrier motion. The agent iteratively refines scene hypotheses based on simulation feedback. The authors also construct a benchmark for physical event recognition and pitch-response timing, reporting improvements over direct inference and general coding-agent baselines across multiple language-model backbones.

Critical strengths
  • Meaningful problem formulation: The paper addresses an important limitation of interactive world models where camera motion may violate physical constraints imposed by the carrier and environment. Explicitly incorporating carrier physics offers a useful approach to improving the physical plausibility of generated worlds.

  • Flexible integration of semantic reasoning and physics simulation: CarPhy combines visual semantics, reconstructed geometry, and simulator feedback to refine scene hypotheses. The reported improvements in physical event recognition and pitch-response timing across several backbones demonstrate the potential of this framework.

Critical weaknesses
  • Insufficient isolation of the agent's contribution: CarPhy combines scene reconstruction, semantic editing, physics simulation, and an agentic refinement loop. However, the experiments primarily compare against direct inference and general coding agents, without a sufficiently controlled comparison against a non-agentic physics pipeline using the same reconstruction and simulator. Since the system already computes a default physics rollout before invoking the agent, it remains unclear how much improvement comes specifically from adaptive agent reasoning rather than the underlying physics tools.

  • Limited evaluation of physical correctness and generalizability: The benchmark contains only 187 samples, including eight blocked cases, and focuses exclusively on forward-motion commands. Evaluation primarily measures physical event classification and pitch-response timing, without comprehensive quantitative assessment of corrected 6DoF trajectories, collision avoidance, or downstream video consistency. Consequently, the experiments do not fully establish the claimed effectiveness across diverse carrier–environment interactions.

Questions for authors
  • Since the host already constructs a scene and performs a default physics rollout, how much improvement does the agent's iterative reasoning and semantic scene editing provide over this non-agentic physics baseline? Can the authors report controlled results isolating these contributions?

  • Can the authors quantitatively evaluate corrected trajectories using full-pose errors, penetration or collision rates, and physical validity, rather than primarily event classification and pitch timing?

  • How does CarPhy perform under more complex motion commands, including turns, lateral movement, and obstacle avoidance? How robust is the correction to inaccurate scene reconstruction and carrier parameters?

7 comments
AuthorsOpenReview ↗

Response to Reviewer wKrH (1/3)

We thank you for the thoughtful review. We clarify the host's role and provide additional evaluations of adaptive reasoning, physical accuracy, and generalization.


Q1. Since the host already constructs a scene and performs a default physics rollout, how much improvement does the agent's iterative reasoning and semantic scene editing provide over this non-agentic physics baseline? Can the authors report controlled results isolating these contributions?

A1. Thanks for asking us to clarify the agent’s contribution. The agent interprets scene-dependent physical roles and revises the scene model using simulation feedback. Our controlled ablation shows that these adaptive corrections improve Macro-F1 from 66.0% (Host-only) to 67.5% (Full CarPhy).

We compare three configurations using the same reconstruction and simulator: the host's default rollout, one additional semantic repair ("One-shot repair"), and the full iterative agent. All three retain CarPhy's task-specific physical modeling implemented in the host. The host is part of our method rather than an external baseline: it implements the physical modeling procedure we developed for this task (Appendix B). A single repair achieves 63.2% Macro-F1, while the full agent reaches 67.5%, showing that an isolated edit can reduce performance and that feedback-guided iteration improves the final result. Table 1 also includes a general Pi harness with access to the same tools, which achieves 43.8% Macro-F1. The gap from 43.8% to 66.0% therefore reflects CarPhy's modeling design rather than the underlying physics tools, while the gain from 66.0% to 67.5% reflects adaptive refinement.

We report these comparisons in Section 4.7 (Pages 9–10) and Table 6 (Page 10) of the revised manuscript, with further details in Appendix I (Page 26).

Table 1. Overall comparison and component ablations of CarPhy (Kimi-K3, PU/PD/NA).

MethodmAcc (%)Macro-F1 (%)Answer rate (%)
Pi Full Tools$\dagger$44.643.8100.0
Host only56.266.084.9
One-shot repair52.763.280.4
Full CarPhy59.167.587.7

Note. BL has only eight cases, making its class-level estimates unstable. For a more reliable comparison, event metrics use the same 179 PU/PD/NA cases for all methods. (Wall interactions are evaluated separately on WorldRoamBench.) †\dagger: Pi harness with access to the same tools.


Q2. Can the authors quantify full-pose errors, collision rates, and physical validity beyond event classification?

A2. Thank you for suggesting these additional evaluations. Yes, we have evaluated position and rotation errors and assessed collision-related behavior and physical validity in generated videos on WorldRoamBench [2], which is a concurrent Interactive World Model benchmark. The results show that CarPhy’s corrected trajectories reduce wall pass-through and improve terrain following in videos generated by IWMs.

The RMSE results also reveal why pointwise pose errors alone do not reliably measure the effectiveness of physical correction. Without correction, raw intent achieves the lowest position and rotation errors (Table 2), yet produces worse physical behavior in the generated videos (Table 3). Figure 14 in Appendix K (Page 28) shows how differences in response timing and magnitude can produce large pointwise errors despite a physically meaningful correction.

We further evaluate downstream video generation using our reimplementation of WorldRoamBench’s physical evaluation protocols. With CarPhy, Clipping scores increase from 35.0 to 75.0 on Lyra 2.0 and from 15.0 to 75.0 on LingBot World 2.0. Terrain Following scores rise from 22.2 to 66.7 and 11.1 to 55.6, respectively (Table 3). These improvements demonstrate the benefit of trajectory correction in downstream video generation.

We report the trajectory results and supporting visualizations in Appendix K (Pages 27–28 of the revised manuscript). The video experiments are presented in Section 4.8 and Table 7 (Page 10), with evaluation details in Appendix M (Page 30).

Table 2. Full-trajectory position and rotation errors.

MethodPosition RMSE (m) ↓Mean rotation error (deg) ↓
Raw intent0.4635.42
Host only0.93911.03
One-shot repair0.8018.35
Full CarPhy0.8699.19

Note. Errors are averaged over all trajectories each method returns on the 179 PU/PD/NA cases. Rotation error measures the mean angular difference from ground truth at matched timestamps.

AuthorsOpenReview ↗

Response to Reviewer wKrH (2/3)

Table 3. Generated-video physical evaluation on selected WorldRoamBench cases.

GeneratorPose conditionClipping score ↑Terrain Following score ↑
Lyra 2.0Without correction35.022.2
Lyra 2.0With CarPhy75.066.7
LingBot World 2.0Without correction15.011.1
LingBot World 2.0With CarPhy75.055.6

Note. Terrain Following evaluates uphill/downhill behavior, corresponding to the PU/PD events studied in our work [2]. Both scores use our reimplementation of WorldRoamBench's evaluation protocols [2].

Code. Our WorldRoamBench evaluation code is available through the anonymous link provided in Appendix M (Page 30 of the revised manuscript).


Q3. How does CarPhy perform under more complex motion commands, including turns, lateral movement, and obstacle avoidance?

A3. Thanks for your question. CarPhy can handle turning and obstacle interactions, and retains its benefits under more complex motion commands, including large turns, in our IWM experiments.

CarPhy supports both turning and lateral movement, and we have added evaluations covering these commands. The selected WorldRoamBench cases include turning left before moving forward (rotate+W cases in WorldRoamBench) and moving diagonally forward-left with simultaneous forward and lateral inputs (W+A cases in WorldRoamBench). Table 4 shows that CarPhy still improves Clipping scores on these scenarios.

CarPhy’s separation of semantic reasoning and physical simulation also makes it adaptable to additional motion commands. Such extensions can be supported by adapting the carrier model and the simulator’s motion-control interface while retaining the agent’s scene interpretation and refinement procedure.

Table 4. Generated-video physical consistency on WorldRoamBench wall-interaction cases, including turning motions (0–100 scale).

GeneratorWithout correctionWith CarPhy
Lyra 2.035.075.0
LingBot World 2.015.075.0

Note. Clipping score on 20 wall-interaction cases of WorldRoamBench, including large turning motions.

Q4. How robust is the correction to reconstruction and carrier-parameter errors?

A4. Thank you for raising this concern about robustness. CarPhy remains robust under the tested reconstruction and carrier-parameter perturbations.

We perturb scene scale, surface height, carrier dimensions, and camera offset on 24 cases, comparing Full CarPhy with Host-only under the same eight perturbation settings. CarPhy’s largest position RMSE increase is 0.095 m, and its verification pass rate remains between 66.7% and 75.0%, compared with 75.0% without perturbation. In particular, reducing scene scale by 10% increases Host-only’s RMSE by 0.761 m, while CarPhy’s RMSE remains nearly unchanged (−0.002 m), showing that adaptive correction can mitigate sensitivity to reconstruction errors (Table 5).

We have included the perturbation settings and results in Appendix L (Page 29 of the revised manuscript).

Table 5. Sensitivity to input uncertainty on 24 cases.

Perturbation typeMagnitudeHost ΔRMSE (m) ↓CarPhy ΔRMSE (m) ↓Host pass rate (%) ↑CarPhy pass rate (%) ↑
None—0.0000.00070.875.0
Scene scale−10%+0.761-0.00270.870.8
Scene scale+10%+0.084+0.09575.075.0
Scene height−0.05 m+0.031+0.02970.870.8
Scene height+0.05 m-0.021-0.07270.870.8
Carrier dimensions−10%+0.061-0.06266.766.7
Carrier dimensions+10%+0.1450.00066.766.7
Camera offset−0.05 m+0.064+0.02370.870.8
Camera offset+0.05 m+0.030-0.09270.870.8

Note. ΔRMSE is relative to each method’s unperturbed error. Pass rates measure verification over all 24 cases.

AuthorsOpenReview ↗

Response to Reviewer wKrH (3/3)

Q5. Can the authors quantitatively evaluate downstream video consistency?

A5. We appreciate your suggestion to quantify downstream video consistency. Yes, we have quantitatively evaluated the generated videos. The results show that CarPhy improves downstream physical consistency on both generators. We use wall-interaction and uphill/downhill test suites from WorldRoamBench [2].

On Lyra 2.0, the Clipping score increases from 35.0 to 75.0 and Terrain Following from 22.2 to 66.7. LingBot World 2.0 also improves, from 15.0 to 75.0 and 11.1 to 55.6, respectively (Table 6).

Table 6. Generated-video physical evaluation on selected WorldRoamBench cases (20 wall-interaction and 9 uphill/downhill cases, 0–100 scale).

GeneratorPose conditionClipping score ↑Terrain Following score ↑
Lyra 2.0Without correction35.022.2
Lyra 2.0With CarPhy75.066.7
LingBot World 2.0Without correction15.011.1
LingBot World 2.0With CarPhy75.055.6

Note. Terrain Following evaluates uphill/downhill behavior, corresponding to the PU/PD events studied in our work [2]. Both scores use our reimplementation of WorldRoamBench's evaluation protocols [2]. WorldExam [1] is another concurrent benchmark, but its test suite was not publicly available when we conducted this evaluation.


References

[1] Yang et al. WorldExam: Benchmarking World Models from Apparent Appearance to Inherent Reactivity. arXiv:2608.02603, 2026.

[2] Xu et al. WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models. arXiv:2606.31672, 2026.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Author General Response

We sincerely thank all reviewers for their time and valuable comments!


We have revised the manuscript and supplementary material based on your valuable comments. The revised parts are highlighted in blue for easy reference.


We are pleased that reviewers recognize the importance of our problem, the effectiveness of our method, and the clarity of our writing.

1. Reviewers recognize that we address an important problem:

  • “The paper addresses an important limitation of interactive world models where camera motion may violate physical constraints imposed by the carrier and environment.” (Reviewer wKrH)
  • “Explicitly incorporating carrier physics offers a useful approach to improving the physical plausibility of generated worlds.” (Reviewer wKrH)
  • “I like how the authors distinguish between understanding what an object represents and determining its physical effect on the carrier.” (Reviewer KrWJ)
  • “The topic is well-motivated” and addresses physically implausible motion in generative world models. (Reviewer 1Heb)

2. Reviewers appreciate our method and experimental results:

  • “CarPhy combines visual semantics, reconstructed geometry, and simulator feedback to refine scene hypotheses.” (Reviewer wKrH)
  • “The reported improvements in physical event recognition and pitch-response timing across several backbones demonstrate the potential of this framework.” (Reviewer wKrH)
  • “The structured scene-editing interface, numerical verification, and abstention mechanism also prevent the agent from directly producing arbitrary trajectories without physical checks.” (Reviewer KrWJ)
  • “The comparisons across multiple backbones also show that the harness improves physical event recognition beyond direct VLM inference.” (Reviewer KrWJ)
  • “The combination allows semantic understanding to inform simulation, offering a practical approach to improving physical plausibility.” (Reviewer 1Heb)
  • “The paper evaluates whether the model recognizes the correct physical event and whether the predicted pitch response occurs at the correct time. I find this distinction useful.” (Reviewer KrWJ)

3. Reviewers appreciate our clear writing:

  • The presentation is “clear and easy to follow.” (Reviewer KrWJ)
  • “The paper is clearly written and well organized, with figures that effectively illustrate the proposed agentic framework and tables that clearly present its performance gains over the baselines.” (Reviewer 1Heb)

Our answers to the main concerns (details and tables are provided in our responses to each reviewer):

  1. Agent Contribution (wKrH, KrWJ, 1Heb): CarPhy identifies and addresses camera-carrier physics through our modeling design and agent orchestration. A general Pi harness with the same tools achieves 43.8% Macro-F1, compared with 66.0% for our Host-only configuration and 67.5% for Full CarPhy. These comparisons demonstrate the value of our modeling design and the additional benefit of adaptive refinement.

  2. Semantic edit reliability (KrWJ, 1Heb): CarPhy combines semantic checks during scene repair with physical checks during simulation. Our human audit finds that 18 of 19 accepted repairs are correct (94.7%). We also report the incorrect repair that passed physical verification and analyze its failure mechanism.

  3. Expanded evaluation on World Models (wKrH, KrWJ, 1Heb): CarPhy improves physical responses in generated videos on more complex interaction cases. We evaluate 20 wall-interaction cases, including turning and lateral movement, and 9 uphill/downhill cases from WorldRoamBench. Clipping scores improve from 35.0 to 75.0 on Lyra 2.0 and 15.0 to 75.0 on LingBot World 2.0, with improvements in Terrain Following on both generators. We additionally report full-trajectory errors and demonstrate consecutive corrections over 8–10 seconds.

  4. Robustness (KrWJ): CarPhy remains robust under the tested reconstruction and carrier-parameter perturbations. Across eight perturbation settings, its largest position RMSE increase is 0.095 m, and verification pass rates range from 66.7% to 75.0%, compared with 75.0% without perturbation.

AuthorsOpenReview ↗

Summary for the Area Chair

Dear Area Chair,

Thank you we thank the reviewers for the constructive discussion! We briefly summarize our CarPhy’s contributions, the new experiments, and the status of the reviews of the rebuttal phase.

CarPhy's Contributions.

  • An overlooked problem. Interactive world models can render a convincing wall or slope while allowing the camera to pass through the wall or move uphill without tilting. We are the first to formulate this as a pose correction problem: given the user’s intended motion, how should the camera carrier respond to the surrounding scene?
  • A framework of pose correction to address the problem. We designed CarPhy’s host to model how terrain and obstacles affect carrier motion. We then extended the host with an agent that uses visual evidence and simulation feedback to refine the host’s scene model in ambiguous cases, such as whether vegetation blocks the carrier.

Reviewers asked how much the agent adds over Host-only. During rebuttal phase, we clarified that this comparison measures the gain from adaptive refinement and emphasized our broader contribution: identifying and addressing the overlooked problem of camera-carrier physics in IWMs. The host models how the carrier responds to its surroundings, and the agent refines that model in ambiguous cases.

Main additions from the rebuttal.

  • Component contributions (wKrH, KrWJ, 1Heb). On the same 179 cases, a general Pi agent with access to the same tools achieves 43.8% Macro-F1, compared with 66.0% for Host-only and 67.5% for Full CarPhy. A single repair scores 63.2%. These comparisons show how much the host’s modeling design contributes and what feedback-guided refinement adds.
  • Physical consistency in generated videos (wKrH, KrWJ, 1Heb). On 29 selected WorldRoamBench cases, correcting the input poses raises Clipping scores from 35.0 to 75.0 on Lyra 2.0 and 15.0 to 75.0 on LingBot World 2.0. Terrain Following scores rise from 22.2 to 66.7 and 11.1 to 55.6, respectively, under matched generation conditions.
  • Reliability, robustness, and cost. A human audit found 18 of 19 accepted repairs correct. Full CarPhy raises the verification pass rate from 73.2% to 83.8% over Host-only. We also report full-trajectory errors, sensitivity to eight input perturbations, and runtime and model and simulator call counts.

Revisions (in blue). Sections 4.7–4.8 present the component and video comparisons. Appendices I–O provide the supporting analyses.

Status of the reviews.

  • Reviewer KrWJ confirmed that the additional experiments addressed their concerns and maintained weak acceptance.
  • Reviewer wKrH acknowledged that the new comparisons clarify the agent’s contribution and that the video experiments strengthen the evaluation. They maintained their rating, citing the cost of refinement relative to its incremental gain and the scope of the evaluation. Our follow-up clarified which results support the host’s modeling contribution and which support the additional refinement step.
  • Reviewer 1Heb has not posted a follow-up to our rebuttal as of this summary.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.