acceptodds
Under review as a conference paper at ICLR 2027

DeXplicit: Making Physical State Explicit in Dexterous Manipulation World Models

Abstract

Before acting on an object, a robot needs to anticipate how its hand motion will change the physical world and what the resulting scene will look like. Existing approaches typically entangle action-conditioned dynamics, camera motion, and visual synthesis within a video generation model, or represent dynamics as point trajectories without explicit object structure. We introduce DeXplicit, a dexterous manipulation world model that instead predicts how the world evolves explicitly in metric 3D before rendering its visual consequences. Given an initial scene and a future hand trajectory, a lightweight 3D dynamics network predicts object-centric motion and deformation, providing structured and geometrically grounded future world states. These predictions then serve as a strong scaffold for video generation: appearance is transported through the predicted 3D motion and projected into future views, allowing the renderer to focus on completing uncertain and newly visible content rather than implicitly learning world dynamics from pixels. This explicit-3D-first decomposition also enables efficient causal generation, requiring only two denoising steps per block while conditioning on generated history. Across HOT3D, OakInk2, and PhysTwin, DeXplicit reduces moved-point error by 27–40 relative to PointWorld and improves object fidelity IoU by 47 on HOT3D under a shared renderer. Our design with two-step causal generation achieves a 57 speedup in video synthesis. Anonymous project page: .

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dexterous Simulation, World Models, Video Diffusion, 3D, Simulation, Diffusion Distillation, Hand Object Interaction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T1kWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DeXplicit predicts metric 3D interaction states before generating their visual appearance. Given an initial scene and a future hand trajectory, it combines joint-aware hand encoding with shared object transport, local deformation and bounded pose refinement. Predicted geometry then guides appearance transport and confidence-gated video generation. The method reduces long-horizon moved-point error over a retrained PointWorld on HOT3D, OakInk2 and PhysTwin. Under the same video renderer, HOT3D object IoU improves from 0.373 to 0.549. The paper also adapts the conditioning interface to a smaller causal renderer that generates 32 future frames in 4.90 seconds using two denoising steps per block.

Critical strengths
  1. The paper measures the predicted dynamics directly across rigid, articulated and deformable interactions. Long-horizon moved-point error falls from 90.47 to 54.34 mm on HOT3D, 83.21 to 59.95 mm on OakInk2, and 60.41 to 44.04 mm on PhysTwin (Table 1). These are substantial improvements over a retrained PointWorld using the same initial metric scene and hand trajectories.

  2. The shared-renderer experiment connects better geometry to better generated observations. Replacing PointWorld trajectories with DeXplicit predictions increases object IoU from 0.373 to 0.549 on HOT3D and improves perceptual metrics as well. Holding the renderer fixed makes this comparison especially useful for judging the dynamics contribution.

  3. The causal model offers a useful speed-quality tradeoff, producing 32 frames in 4.90 seconds with object IoU 0.490. The matched post-training study also shows a concrete benefit from the video critic: IoU rises from 0.459 to 0.490 over distillation alone, with improvements in LPIPS, object position and coverage (Table 6).

Critical weaknesses
  1. The improvement on moving points comes with a substantial all-point regression on OakInk2. Long-horizon all-point error rises from 11.71 to 16.88 mm, and early-horizon error from 6.70 to 8.01 mm (Table 1). The results need a breakdown showing whether stationary regions, particular articulated objects or a small number of sequences account for this tradeoff. Whole-scene stability is important for a model intended to represent physical state.

  2. The confidence-gated readout is not evaluated separately from the other conditioning components. The paper learns pointwise log variance and converts it into confidence, but the conversion and calibration procedure are not specified precisely. The geometry, hand and appearance ablations do not show whether the learned confidence improves rendering over a visibility-based or constant-confidence mask, especially when the forecast geometry is wrong.

  3. The reported 57-fold speedup compares a 14B bidirectional renderer with a smaller causal model using two-step sampling. Backbone size, sampler and causal adaptation all change. The causal ablations study the critic, but do not isolate how much predicted object geometry contributes under that same fast renderer. This leaves the claim that explicit dynamics enables the efficient visual readout only partly tested.

  4. Some details in Figure 2 could be presented more carefully. For example, the headings of several modules are not centered within their boxes. More consistent alignment would make the diagram clearer and easier to follow.

Questions for authors
  1. What causes the OakInk2 all-point regression? Can you separate moving and stationary regions and provide per-object or per-sequence results? How does changing the moved-point supervision affect this tradeoff?

  2. What exact mapping turns log variance into confidence, and how is it calibrated? With geometry and training budget fixed, how does learned gating compare with visibility-only and constant-confidence masks?

  3. With the same small two-step causal backbone, how much do predicted object geometry and appearance transport improve over hand conditioning alone? Please also separate dynamics, projection and rendering time when reporting the full pipeline latency.

4 comments

Response to Reviewer T1kW

We thank the reviewer for the constructive assessment and for recognizing the improvements in 3D dynamics, the shared-renderer comparison, and causal generation. We conducted additional experiments to directly address the three concerns.

W1/Q1. OakInk2 all-point regression

The regression primarily reflects a tradeoff between moved-point accuracy and stationary-region preservation. OakInk2 includes articulated interactions where particular components move while other portions of the same object remain stationary. Our training objective emphasizes moved-point supervision, which favors accurate interaction-induced motion but can introduce drift in stationary components.

We separately evaluated moving and stationary points:

OakInk2HorizonMoved ↓Stationary ↓All ↓
DeXplicitLong59.9513.4116.88
PointWorldLong65.069.7010.89
DeXplicitEarly20.616.508.01
PointWorldEarly30.065.706.89

PointWorld results use our additionally retrained input-matched baseline; errors are in mm.

DeXplicit improves moved-point accuracy at both horizons, while PointWorld better preserves stationary points. This identifies stationary-region drift as the principal observed tradeoff, rather than inferior prediction of articulated motion. The decomposition does not yet attribute the regression to individual objects or sequences.

To test whether this tradeoff can be controlled, we doubled the static-point supervision weight. This reduced OakInk2 all-point error from 16.88 to 14.28 mm (−15.4%), while moved-point error increased only from 59.95 to 62.35 mm (+4.0%). The reweighted model still retains better moved-point accuracy than PointWorld (62.35 vs. 65.06 mm).

These results demonstrate that the tradeoff can be substantially mitigated through supervision weighting. We will clarify this behavior and discuss dataset-adaptive static–dynamic weighting.

W2/Q2. Confidence conversion and isolated ablation

Confidence mapping. For predicted pointwise log variance s, we first compute the mean variance v by exponentiating s after clipping to [ln(10⁻⁶), ln(100)] and averaging across C channels, where C = 1 for the evaluated checkpoint. We then map variance to confidence using:

q = 1 − (v − 10⁻⁶)/(100 − 10⁻⁶).

These are fixed implementation bounds. No held-out calibration or temperature fitting is performed; therefore, q represents a bounded variance-derived confidence score, not a calibrated probability of geometric correctness.

Controlled ablation. With visibility fixed, we compared predicted-confidence weighting against visibility-only weighting with constant confidence.

Appearance weightingPSNR ↑SSIM ↑LPIPS ↓
Visibility only (constant confidence)16.7440.5310.390
Predicted-confidence weighting16.8100.5430.376

Learned confidence improves PSNR by 0.066 dB, SSIM by 0.012, and reduces LPIPS by 3.6%. These results demonstrate a consistent, although modest, benefit beyond visibility-only weighting. We do not claim that this establishes calibration or guaranteed robustness to inaccurate geometry.

W3/Q3. Explicit geometry under the fast causal renderer and pipeline latency

(a) Contribution of predicted geometry and appearance transport.

We conducted additional conditioning ablations using the same small two-step causal renderer:

ConfigurationPSNR ↑SSIM ↑LPIPS ↓Object IoU+ ↑PCK20 ↑
Full DeXplicit17.4100.5830.3480.49044.70
Without 3D object conditioning14.5820.4530.5060.24842.63
Without appearance transport14.4410.4510.5280.23422.71

Removing 3D object conditioning reduces object IoU+ from 0.490 to 0.248 (−49.4%), while removing appearance transport reduces it to 0.234 (−52.2%). Both ablations substantially worsen perceptual quality.

These results directly demonstrate that explicit geometry and appearance transport contribute substantially to object fidelity within the same efficient causal renderer, rather than improvements arising solely from the smaller backbone.

(b) End-to-end latency and amortized 3D computation.

Importantly, 3D forecasting and geometric projection are performed only once for the entire 32-frame rollout, not once per causal latent block. The renderer generates eight latent blocks, each representing four frames.

Measured latency on one NVIDIA A800:

ComponentLatencyFrequency
3D dynamics forecast0.216 sOnce per 32 frames
Geometric projection1.092 sOnce per 32 frames
Causal video rendering4.900 sAcross 8 blocks
Complete pipeline6.208 s32 frames
Subsequent 4-frame block0.547 sNo repeated 3D computation

The 1.308-second geometric preparation cost is incurred once and amortized across all eight blocks. After this preparation, each subsequent four-frame latent block requires only 0.547 seconds, without repeating 3D forecasting or projection.

We agree that the reported 57× video-rendering acceleration is not attributable to the 3D dynamics predictor alone, as it also reflects the smaller causal backbone and two-step sampling. However, our fast renderer is not simply an off-the-shelf model substitution. We develop a two-stage geometry-conditioned causal adaptation procedure (Sec. 4.3, Appendix D): first transferring and aligning our explicit 3D, hand, and appearance-transport conditioning with the causal backbone, then performing on-policy two-step autoregressive distillation with conditional video-critic post-training. As recognized by the reviewer, our matched post-training ablation (Table 6) demonstrates the effectiveness of this procedure, improving object IoU+ from 0.459 to 0.490 over distillation alone.

Our additional ablations further demonstrate that explicit geometry and appearance transport are essential to preserving interaction fidelity under this efficient causal configuration. Importantly, geometric forecasting and projection are computed only once per rollout, without recurring overhead during subsequent latent-block generation. Thus, our contribution extends beyond adopting a faster backbone: we develop and validate a geometry-conditioned causal generation framework that combines efficient two-step autoregressive synthesis with accurate, amortized physical-state conditioning.

We will distinguish the 4.90-second rendering latency, 6.208-second full-pipeline latency, and 0.547-second subsequent-block latency in the revision.

W4. Figure 2

Thank you for noting the alignment issues. We will correct the module-title centering and box alignment in Figure 2.

Thanks for running the additional experiments. The breakdown by moving and stationary points and the reweighting experiment explain the trade-off on OakInk2. The visibility-only baseline clarifies what the learned confidence contributes, and the ablations with the same causal renderer address my questions about 3D object conditioning and appearance transport. The timing breakdown also makes the cost of the full pipeline clearer.

I appreciate your plan to revise Figure 2. Centering the module titles and aligning the boxes would make the pipeline easier to follow. Your response addresses most of my questions, and I will consider raising my rating from 2 after taking the other reviewers' feedback into account.

We thank the reviewer for the thoughtful follow-up and are glad that our additional experiments have addressed the major concerns.

We also wanted to highlight that, since your comment, we have provided further technical clarifications in our response to Reviewer REAM. In particular, we clarified that the shared 14B renderer was trained independently using GT geometry and held fixed when evaluating DeXplicit and PointWorld predictions; specified the empirical uncertainty-to-confidence distribution; and further substantiated how efficient 3D dynamics prediction, appearance transport, and our geometry-conditioned causal distillation jointly enable high-quality two-step generation.

Together with our matched-input PointWorld experiments, OakInk2 static-point analysis, and controlled causal-renderer ablations, we believe these results provide a comprehensive response to the technical concerns raised during review.

We sincerely appreciate your constructive engagement and willingness to reconsider your assessment in light of the additional experimental evidence.

Reviewer_36sBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets the problem of action-conditioned video prediction for human videos containing dexterous manipulation behaviors. Specifically, it proposes to use a lightweight explicit 3D dynamics prediction network to first predict object motion and deformation, which then gets projected to video frames, which then get completed into full RGB prediction into the future. The method is trained and evaluated on several dexterous manipulation video datasets and is shown to outperform baselines on dynamics prediction metrics and video prediction metrics. No downstream application on robot manipulation is directly shown.

Critical strengths
  • Accurate prediction of action-conditioned dynamics, especially for challenging dexterous behaviors, is an important topic due to severe occlusions and rich contact events in these interactions
  • Modeling the explicit 3D state of the object is a compelling approach for ensuring consistency, with the technique also being able to model both rigid and deformable objects.
Critical weaknesses
  • Strong reliance on structured geometric inputs and supervision. The method assumes initial metric 3D points, object identities, object-local coordinates, calibrated cameras, and future hand trajectories, with additional MANO annotations where available. These are nontrivial requirements for in-the-wild manipulation. The paper does not sufficiently evaluate robustness to imperfect reconstruction, object segmentation, or hand tracking which would arise with real-world capture.
  • Ability to tackle multi-object scenarios and cases where objectness is hard to define. The paper proposes structured object representation and shows its benefit over PointWorld that does not assume objectness. While structured object representation can effectively regularizes prediction, it remains unclear how it may generalize to multi-object scenarios possibly with varying number of objects and to cases where it’s hard to define what an object is (e.g., typing on a keyboard vs transporting a keyboard entails different object definitions)
  • No demonstration of downstream robot experiments. The paper motivates physically grounded world models for dexterous manipulation, with also dexterous manipulation in the title, which implies that the paper is meant for targeting applications in a robotic system that performs dexterous manipulation. However, evaluations focus on only 3D prediction accuracy and generated video quality. As a result, the title (and in corresponding places in abstract and intro) do not accurately reflect the true contributions of the work.
Questions for authors

See weaknesses section above.

1 comment

We thank the reviewer for the constructive assessment and for recognizing the importance of accurate 3D dynamics prediction for dexterous interactions, particularly our ability to model both rigid and deformable objects. We appreciate the suggestions and clarify the three concerns below.

W1. Structured geometric inputs and robustness

We agree that obtaining reliable metric geometry, object correspondences, and hand trajectories presents challenges for in-the-wild applications.

Our work focuses on action-conditioned dynamics forecasting given an initially reconstructed scene and a future hand trajectory, rather than end-to-end reconstruction from raw RGB observations. The model predicts future object motion and deformation, which subsequently guide video generation. Importantly, the renderer operates on predicted rather than ground-truth future geometry.

Complete MANO annotations are also not universally required; for example, PhysTwin uses sparse tracked hand-surface controls without reliable MANO joint identities.

We additionally incorporate confidence-weighted appearance transport to mitigate imperfect geometric forecasts. On HOT3D, predicted-confidence weighting improves LPIPS from 0.390 to 0.376 and SSIM from 0.531 to 0.543 relative to constant confidence.

We acknowledge that systematic robustness evaluation under noisy initial reconstruction, object segmentation, and hand tracking remains an important direction. We will clarify the input assumptions and limitations in the revised manuscript.

W2. Multi-object scenarios and object decomposition

We appreciate this question and would like to clarify that DeXplicit already supports multi-object scenes, which are included in our existing training and evaluation settings.

In particular, HOT3D contains multi-object interaction scenarios, and our original submission already illustrates interactions involving multiple objects, including the object-transfer and toy-to-container placement examples in Figures 4 and 5. OakInk2 also contains complex interactions involving multiple objects and articulated components.

Architecturally, DeXplicit explicitly supports multiple object instances. Scene points retain their object-instance identities, and masked pooling constructs per-object tokens. The object-time Transformer processes these tokens jointly with hand-action features, predicting separate SE(3) transformations and local deformation fields for the represented objects. The number of valid object tokens is determined by the object instances present in each scene.

Regarding the keyboard example, our separation of global object transport and local deformation is designed to capture precisely these different motion granularities. Transporting a keyboard primarily involves coherent global SE(3) motion, whereas typing involves localized motion while much of the assembly remains stationary. The local deformation graph allows different regions of an object to move without requiring the entire object to follow a single rigid transformation.

We agree that automatically discovering the optimal task-dependent object/part hierarchy is an interesting further direction. However, this is distinct from multi-object dynamics prediction, which our current architecture and evaluation already support.

W3. Downstream robotic manipulation and contribution framing

We agree that demonstrating downstream closed-loop robotic manipulation would be a valuable extension.

We would like to clarify that DeXplicit is an action-conditioned world model for dexterous manipulation, rather than a learned robotic manipulation policy. Its central contribution is predicting the physical and visual consequences of articulated hand actions, including object transport, articulation, and deformation.

Our evaluations directly examine this predictive capability through metric 3D forecasting and action-conditioned video generation. In particular, our controlled same-renderer comparison shows that improving the 3D dynamics predictions increases HOT3D object IoU+ from 0.373 to 0.549 (+47.2%), isolating the contribution of the explicit physical-state forecast.

We agree that integrating these predictions into downstream planning and evaluating robot task success would be valuable future work. We will clarify this distinction in the introduction to avoid suggesting that closed-loop robotic performance has already been demonstrated.

We thank the reviewer again for the helpful feedback and hope these clarifications address the concerns.

Reviewer_REAMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DeXplicit splits a hand-object world model into two stages. A lightweight 3D network first takes the initial point cloud and a future hand trajectory, and predicts each object’s SE(3) motion plus local deformation. The initial appearance is then carried along that 3D motion into future frames, and a video diffusion model renders the result. The authors also build a two-step causal version.

Critical strengths
  1. The same-renderer comparison is the strongest result in the paper. Only the 3D prediction changes, and IoU+ goes from 0.373 to 0.549 while OPE+ drops from 0.488 to 0.352. This shows that better 3D accuracy actually carries over to object fidelity in the generated video.
  2. The evaluation is careful: splits are by source sequence, and missing predictions are penalized. Table 2 also makes a useful point. The hand-only baselines get high PCK but IoU+ below 0.14, so hand alignment alone says little about interaction quality.
Critical weaknesses
  1. DeXplicit gets more input than PointWorld in the 3D comparison. It uses object labels, mesh-based canonical coordinates, MANO parameters, and hand-derived pose candidates. Table 5 lists none of these for PointWorld. With object membership known, the rigid case is just per-object 6-DoF regression. The largest gain in Table 3 (+structured transport) is where this information comes in. Static and hand-following baselines are also missing. Separately, all-point error on OakInk2 gets 44% worse, and the paper doesn’t explain why.
  2. The speedup doesn’t come from the 3D design. The 57× comes from a smaller model, 2 steps instead of 30, and causal generation. Hand2World-AR gets similar latency without any 3D input. In Table 2, only the PointWorld row is controlled. The other baselines mostly lack object information, so their low IoU is expected.
Questions for authors
  1. (W1) Could you rerun Table 1 with PointWorld given the same object labels and canonical coordinates, and add the static and hand-following baselines? If the advantage holds with matched inputs, I would raise my score.
  2. (W1) What is the error on static points in OakInk2, and what causes the worse all-point error?
  3. (W2) In the same-renderer comparison, where do PointWorld’s confidence and mask come from? Was the renderer trained on GT geometry or on DeXplicit’s predictions?
  4. (W2) Could you show how much quality the same causal pipeline loses with and without the 3D conditioning? Does the 4.90 s include the dynamics network?
9 comments

We thank the reviewer for the careful and constructive assessment. We agree that the original PointWorld comparison did not fully match access to structured object information as we wanted to maintain the original architecture of PointWorld, and that the video-generation speedup should be distinguished from the contribution of the 3D dynamics model.

We conducted additional experiments addressing these concerns directly.

W1/Q1. Matched-input PointWorld and simple dynamics baselines

We retrained PointWorld with the same object-instance labels and canonical coordinates supplied to DeXplicit. DeXplicit retains lower moved-point prediction error across all three datasets.

Specifically, we augmented each PointWorld scene-point feature with a 32-dimensional one-hot object-instance encoding and three metric canonical coordinates, adding 35 channels to its input projection while preserving its 256-dimensional hidden representation. We retained the adapted PointWorld backbone, point-action and hand conditioning, joint-transformer encoding, and MANO-relative action interface.

The augmented baseline was trained from scratch using the same frozen three-dataset training pool and sampling quotas as the retained DeXplicit model, with AdamW (learning rate 1e-4, weight decay 0.01), gradient clipping at 5, effective batch size 96, and 11,600 optimizer updates.

New results — Long-horizon 3D forecasting (states 1–32, L2 error in mm; lower is better):

MethodHOT3D Moved / AllOakInk2 Moved / AllPhysTwin Moved / All
DeXplicit54.34 / 14.0159.95 / 16.8844.04 / 25.37
PointWorld + matched inputs84.86 / 19.2665.06 / 10.8954.48 / 31.00
Closest-hand following119.86 / 314.14122.74 / 174.7174.29 / 45.20
Static prediction228.12 / 48.84119.17 / 22.6264.15 / 35.42

Relative to matched-input PointWorld, DeXplicit reduces long-horizon moved-point error by 36.0% on HOT3D, 7.9% on OakInk2, and 19.2% on PhysTwin. At the early horizon (states 1–14), the corresponding reductions are 39.1%, 31.4%, and 38.5%.

The static and nearest-hand-following baselines further show that accurate object motion cannot be recovered by simply retaining the initial scene or propagating nearby hand translations.

These results clarify the contribution of the structured dynamics model. Object membership and canonical coordinates provide correspondence and geometric structure, but not the future object motion itself. DeXplicit learns action-conditioned object-level SE(3) transport together with residual local deformation, rather than independently regressing unrestricted point trajectories. The distinction is relevant not only for rigid transport but also for the articulated and deformable interactions represented by OakInk2 and PhysTwin.

We agree that access to structured annotations is an assumption of our method, and will make the distinction between matched input information and different architectural inductive biases explicit in the revision.

W1/Q2. OakInk2 static-point and all-point errors

OakInk2 contains more partially articulated object interactions, where motion is concentrated in specific articulated components while substantial portions of each object's geometry remain stationary. Consequently, accurate preservation of static geometry has a particularly strong influence on the all-point metric.

Our training objective emphasizes moved-point accuracy to prioritize action-induced object dynamics. This explains the observed tradeoff: DeXplicit improves OakInk2 moved-point error over matched-input PointWorld (59.95 vs. 65.06 mm), but has higher static-point error (13.41 vs. 9.70 mm), resulting in a worse all-point error (16.88 vs. 10.89 mm).

To investigate this mechanism, we doubled the static-point loss weight. This improved OakInk2 all-point error from 16.88 to 14.28 mm, while moved-point error increased from 59.95 to 62.35 mm. The experiment demonstrates that static preservation and dynamic-motion accuracy can be explicitly traded off through loss weighting.

Importantly, DeXplicit retains its advantage in predicting articulated motion over matched-input PointWorld. These results identify the static–dynamic loss balance as an opportunity for dataset-adaptive optimization, rather than indicating that the structured dynamics representation is ineffective.

W2/Q3. Same-renderer comparison: confidence, masks, and geometry training

PointWorld does not predict pointwise uncertainty. For PointWorld, we assign constant confidence of 1 to predicted scene points. The spatial edit mask is constructed by the geometry projection pipeline from projected confidence and geometric coverage, rather than being predicted by a separate PointWorld mask head.

For DeXplicit, pointwise confidence is derived from its predicted log variance. Variance is clipped to the fixed range [1e-6, 100] and linearly mapped to a confidence score between 0 and 1. This is a bounded uncertainty-derived weighting, not a calibrated probability of geometric correctness.

In an additional HOT3D confidence-weighting experiment, replacing constant confidence with predicted-confidence soft weighting improved PSNR from 16.744 to 16.810, SSIM from 0.531 to 0.543, and LPIPS from 0.390 to 0.376.

We also examined the renderer's geometry-training distribution. In both evaluated settings, inference uses predicted 3D geometry. Training the renderer with ground-truth geometry achieved PSNR 17.41, SSIM 0.583, LPIPS 0.348, object IoU 0.490, and PCK20 44.7. Using predicted geometry during renderer training instead reduced these metrics to 16.744, 0.531, 0.390, 0.4435, and 41.96, respectively.

Cleaner geometric conditioning during training benefits the present renderer as it has clearner conditioning signal, we acknowledge that training with imperfect forecasts and inducing the renderer to fix the imperfect conditions as a worthwhile direction for future works.

We will clarify the geometry-training protocol and confidence-map construction. We will also distinguish the fixed-renderer comparison, which evaluates each dynamics model's complete forecast-conditioning output, from ablations that isolate individual conditioning components.

W2/Q4. Causal adaptation, post-training contribution, and amortized 3D conditioning overhead

We thank the reviewer for requesting clarification of the speedup and end-to-end latency. Importantly, our fast causal renderer is not simply an off-the-shelf few-step video model. We explicitly develop a two-stage causal adaptation and post-training procedure to transfer our geometry-conditioned generation framework into an efficient autoregressive model, as detailed in Sec. 4.3, Fig. 7, and Appendix D (Eqs. 15–19).

First, our causal generation pipeline involves substantial methodological adaptation and post-training:

  1. Stage I — Geometry-conditioned causal adaptation. We transfer our explicit 3D geometry, hand, and appearance-transport conditioning into the pretrained causal backbone. With the backbone frozen, we optimize the transferred conditioning parameters using a clean-endpoint reconstruction objective and a conditional sequence-level video discriminator. This adapts the causal generator to our structured interaction controls.

  2. Stage II — On-policy distillation and post-training. We further adapt the backbone through rank-128 LoRA. The two-step student generates autoregressively using its own predicted history, while an EMA teacher provides velocity supervision at the same student-visited noisy states using a cleaner temporal cache. Our conditional video critic additionally supervises interaction fidelity across generated clips. We also implement bounded-memory slot-wise gradient replay (Appendix D, Eq. 19), allowing training across eight causal latent blocks while preserving gradients through the appearance-transport branch.

The effectiveness of this post-training is directly validated by our controlled ablation in the original submission (Table 6):

Causal configurationObject IoU+ ↑LPIPS ↓PCK20 ↑
Fast Adapter0.4600.37041.88
On-policy distillation0.4590.37740.72
On-policy distillation + video critic0.4900.34844.72

With the same two-step causal sampling configuration, our combined post-training improves object IoU+ by 6.8%, reduces LPIPS by 7.7%, and improves PCK20 by 4.00 percentage points relative to distillation alone. These results demonstrate the contribution of our geometry-conditioned adaptation and post-training procedure beyond simply adopting a smaller causal architecture.

Second, 3D forecasting and geometric projection are performed only once at the beginning of the 32-frame rollout, rather than being repeated for each causal latent block.

Our causal renderer generates 32 frames through eight sequential latent-block inferences, with each block representing four video frames. The measured component latencies on a single NVIDIA A800 are:

ComponentLatencyFrequency
3D dynamics forecasting0.216 sOnce per 32-frame rollout
Geometric projection1.092 sOnce per 32-frame rollout
Causal video rendering4.900 sTotal across 8 latent blocks
Full 32-frame pipeline6.208 sIncluding initial geometric computation
Subsequent 4-frame latent block0.547 sNo additional 3D forecasting or projection

The key distinction is that the 1.308-second geometric computation is a one-time cost amortized across all eight latent blocks. Once the initial geometric forecast and projected conditioning have been computed, subsequent four-frame generation requires only 0.547 seconds per latent block, with no repeated 3D forecasting or geometric projection.

Thus, explicit 3D conditioning does not introduce recurrent geometric-computation overhead during causal latent-block generation. Our geometry-guided representation can condition successive video-generation steps while retaining low incremental inference latency.

Third, the 3D conditioning and appearance-transport components contribute substantially to generation quality within this efficient causal pipeline. Our additional controlled ablations demonstrate:

Causal configurationPSNR ↑LPIPS ↓Object IoU ↑
Full DeXplicit17.4100.3480.490
Without 3D object conditioning14.5820.5060.248
Without appearance transport14.4410.5280.234

Removing 3D object conditioning decreases object IoU by 49.4% and PSNR by 2.83 dB. Removing appearance transport decreases object IoU by 52.2%. These results establish that our explicit geometry and appearance-conditioning mechanisms materially improve the quality of the fast causal generator, rather than its performance arising solely from the pretrained backbone.

We agree that the reported 57× video-rendering acceleration relative to our 14B bidirectional configuration is enabled by the smaller causal backbone, two-step denoising, and autoregressive execution. However, this should not be interpreted as simply substituting an existing faster generator. Our contribution includes the explicit geometry-conditioned causal transfer, conditional video adversarial supervision, on-policy post-training, and efficient amortization of geometric computation across autoregressive blocks.

Taken together, our results demonstrate that DeXplicit achieves substantially improved interaction fidelity within a fast causal generation framework through an explicitly developed and experimentally validated adaptation procedure, while requiring only 0.547 seconds for subsequent four-frame latent-block generation after the one-time geometric computation.

We will clarify this methodological contribution, the controlled post-training ablations, and the distinction between total pipeline latency and subsequent-block latency in the revised manuscript.

Summary

The new experiments directly address the reviewer's requested matched-input comparison and demonstrate that DeXplicit's improvements in action-induced 3D motion persist after PointWorld receives the same object labels and canonical coordinates. They also quantify the contribution of explicit 3D conditioning to causal video quality, clarify the confidence and geometry-training protocols, and distinguish video-rendering acceleration from end-to-end pipeline latency.

We appreciate the reviewer's suggestions, which have helped us strengthen both the experimental controls and the precision of our claims. We hope these additional results address the concerns underlying the assessment and welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.