- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DeXplicit predicts metric 3D interaction states before generating their visual appearance. Given an initial scene and a future hand trajectory, it combines joint-aware hand encoding with shared object transport, local deformation and bounded pose refinement. Predicted geometry then guides appearance transport and confidence-gated video generation. The method reduces long-horizon moved-point error over a retrained PointWorld on HOT3D, OakInk2 and PhysTwin. Under the same video renderer, HOT3D object IoU improves from 0.373 to 0.549. The paper also adapts the conditioning interface to a smaller causal renderer that generates 32 future frames in 4.90 seconds using two denoising steps per block.
Critical strengths
-
The paper measures the predicted dynamics directly across rigid, articulated and deformable interactions. Long-horizon moved-point error falls from 90.47 to 54.34 mm on HOT3D, 83.21 to 59.95 mm on OakInk2, and 60.41 to 44.04 mm on PhysTwin (Table 1). These are substantial improvements over a retrained PointWorld using the same initial metric scene and hand trajectories.
-
The shared-renderer experiment connects better geometry to better generated observations. Replacing PointWorld trajectories with DeXplicit predictions increases object IoU from 0.373 to 0.549 on HOT3D and improves perceptual metrics as well. Holding the renderer fixed makes this comparison especially useful for judging the dynamics contribution.
-
The causal model offers a useful speed-quality tradeoff, producing 32 frames in 4.90 seconds with object IoU 0.490. The matched post-training study also shows a concrete benefit from the video critic: IoU rises from 0.459 to 0.490 over distillation alone, with improvements in LPIPS, object position and coverage (Table 6).
Critical weaknesses
-
The improvement on moving points comes with a substantial all-point regression on OakInk2. Long-horizon all-point error rises from 11.71 to 16.88 mm, and early-horizon error from 6.70 to 8.01 mm (Table 1). The results need a breakdown showing whether stationary regions, particular articulated objects or a small number of sequences account for this tradeoff. Whole-scene stability is important for a model intended to represent physical state.
-
The confidence-gated readout is not evaluated separately from the other conditioning components. The paper learns pointwise log variance and converts it into confidence, but the conversion and calibration procedure are not specified precisely. The geometry, hand and appearance ablations do not show whether the learned confidence improves rendering over a visibility-based or constant-confidence mask, especially when the forecast geometry is wrong.
-
The reported 57-fold speedup compares a 14B bidirectional renderer with a smaller causal model using two-step sampling. Backbone size, sampler and causal adaptation all change. The causal ablations study the critic, but do not isolate how much predicted object geometry contributes under that same fast renderer. This leaves the claim that explicit dynamics enables the efficient visual readout only partly tested.
-
Some details in Figure 2 could be presented more carefully. For example, the headings of several modules are not centered within their boxes. More consistent alignment would make the diagram clearer and easier to follow.
Questions for authors
-
What causes the OakInk2 all-point regression? Can you separate moving and stationary regions and provide per-object or per-sequence results? How does changing the moved-point supervision affect this tradeoff?
-
What exact mapping turns log variance into confidence, and how is it calibrated? With geometry and training budget fixed, how does learned gating compare with visibility-only and constant-confidence masks?
-
With the same small two-step causal backbone, how much do predicted object geometry and appearance transport improve over hand conditioning alone? Please also separate dynamics, projection and rendering time when reporting the full pipeline latency.
4 comments
W3/Q3. Explicit geometry under the fast causal renderer and pipeline latency
(a) Contribution of predicted geometry and appearance transport.
We conducted additional conditioning ablations using the same small two-step causal renderer:
| Configuration | PSNR ↑ | SSIM ↑ | LPIPS ↓ | Object IoU+ ↑ | PCK20 ↑ |
|---|---|---|---|---|---|
| Full DeXplicit | 17.410 | 0.583 | 0.348 | 0.490 | 44.70 |
| Without 3D object conditioning | 14.582 | 0.453 | 0.506 | 0.248 | 42.63 |
| Without appearance transport | 14.441 | 0.451 | 0.528 | 0.234 | 22.71 |
Removing 3D object conditioning reduces object IoU+ from 0.490 to 0.248 (−49.4%), while removing appearance transport reduces it to 0.234 (−52.2%). Both ablations substantially worsen perceptual quality.
These results directly demonstrate that explicit geometry and appearance transport contribute substantially to object fidelity within the same efficient causal renderer, rather than improvements arising solely from the smaller backbone.
(b) End-to-end latency and amortized 3D computation.
Importantly, 3D forecasting and geometric projection are performed only once for the entire 32-frame rollout, not once per causal latent block. The renderer generates eight latent blocks, each representing four frames.
Measured latency on one NVIDIA A800:
| Component | Latency | Frequency |
|---|---|---|
| 3D dynamics forecast | 0.216 s | Once per 32 frames |
| Geometric projection | 1.092 s | Once per 32 frames |
| Causal video rendering | 4.900 s | Across 8 blocks |
| Complete pipeline | 6.208 s | 32 frames |
| Subsequent 4-frame block | 0.547 s | No repeated 3D computation |
The 1.308-second geometric preparation cost is incurred once and amortized across all eight blocks. After this preparation, each subsequent four-frame latent block requires only 0.547 seconds, without repeating 3D forecasting or projection.
We agree that the reported 57× video-rendering acceleration is not attributable to the 3D dynamics predictor alone, as it also reflects the smaller causal backbone and two-step sampling. However, our fast renderer is not simply an off-the-shelf model substitution. We develop a two-stage geometry-conditioned causal adaptation procedure (Sec. 4.3, Appendix D): first transferring and aligning our explicit 3D, hand, and appearance-transport conditioning with the causal backbone, then performing on-policy two-step autoregressive distillation with conditional video-critic post-training. As recognized by the reviewer, our matched post-training ablation (Table 6) demonstrates the effectiveness of this procedure, improving object IoU+ from 0.459 to 0.490 over distillation alone.
Our additional ablations further demonstrate that explicit geometry and appearance transport are essential to preserving interaction fidelity under this efficient causal configuration. Importantly, geometric forecasting and projection are computed only once per rollout, without recurring overhead during subsequent latent-block generation. Thus, our contribution extends beyond adopting a faster backbone: we develop and validate a geometry-conditioned causal generation framework that combines efficient two-step autoregressive synthesis with accurate, amortized physical-state conditioning.
We will distinguish the 4.90-second rendering latency, 6.208-second full-pipeline latency, and 0.547-second subsequent-block latency in the revision.
W4. Figure 2
Thank you for noting the alignment issues. We will correct the module-title centering and box alignment in Figure 2.
Thanks for running the additional experiments. The breakdown by moving and stationary points and the reweighting experiment explain the trade-off on OakInk2. The visibility-only baseline clarifies what the learned confidence contributes, and the ablations with the same causal renderer address my questions about 3D object conditioning and appearance transport. The timing breakdown also makes the cost of the full pipeline clearer.
I appreciate your plan to revise Figure 2. Centering the module titles and aligning the boxes would make the pipeline easier to follow. Your response addresses most of my questions, and I will consider raising my rating from 2 after taking the other reviewers' feedback into account.
We thank the reviewer for the thoughtful follow-up and are glad that our additional experiments have addressed the major concerns.
We also wanted to highlight that, since your comment, we have provided further technical clarifications in our response to Reviewer REAM. In particular, we clarified that the shared 14B renderer was trained independently using GT geometry and held fixed when evaluating DeXplicit and PointWorld predictions; specified the empirical uncertainty-to-confidence distribution; and further substantiated how efficient 3D dynamics prediction, appearance transport, and our geometry-conditioned causal distillation jointly enable high-quality two-step generation.
Together with our matched-input PointWorld experiments, OakInk2 static-point analysis, and controlled causal-renderer ablations, we believe these results provide a comprehensive response to the technical concerns raised during review.
We sincerely appreciate your constructive engagement and willingness to reconsider your assessment in light of the additional experimental evidence.
Response to Reviewer T1kW
We thank the reviewer for the constructive assessment and for recognizing the improvements in 3D dynamics, the shared-renderer comparison, and causal generation. We conducted additional experiments to directly address the three concerns.
W1/Q1. OakInk2 all-point regression
The regression primarily reflects a tradeoff between moved-point accuracy and stationary-region preservation. OakInk2 includes articulated interactions where particular components move while other portions of the same object remain stationary. Our training objective emphasizes moved-point supervision, which favors accurate interaction-induced motion but can introduce drift in stationary components.
We separately evaluated moving and stationary points:
PointWorld results use our additionally retrained input-matched baseline; errors are in mm.
DeXplicit improves moved-point accuracy at both horizons, while PointWorld better preserves stationary points. This identifies stationary-region drift as the principal observed tradeoff, rather than inferior prediction of articulated motion. The decomposition does not yet attribute the regression to individual objects or sequences.
To test whether this tradeoff can be controlled, we doubled the static-point supervision weight. This reduced OakInk2 all-point error from 16.88 to 14.28 mm (−15.4%), while moved-point error increased only from 59.95 to 62.35 mm (+4.0%). The reweighted model still retains better moved-point accuracy than PointWorld (62.35 vs. 65.06 mm).
These results demonstrate that the tradeoff can be substantially mitigated through supervision weighting. We will clarify this behavior and discuss dataset-adaptive static–dynamic weighting.
W2/Q2. Confidence conversion and isolated ablation
Confidence mapping. For predicted pointwise log variance s, we first compute the mean variance v by exponentiating s after clipping to [ln(10⁻⁶), ln(100)] and averaging across C channels, where C = 1 for the evaluated checkpoint. We then map variance to confidence using:
q = 1 − (v − 10⁻⁶)/(100 − 10⁻⁶).
These are fixed implementation bounds. No held-out calibration or temperature fitting is performed; therefore, q represents a bounded variance-derived confidence score, not a calibrated probability of geometric correctness.
Controlled ablation. With visibility fixed, we compared predicted-confidence weighting against visibility-only weighting with constant confidence.
Learned confidence improves PSNR by 0.066 dB, SSIM by 0.012, and reduces LPIPS by 3.6%. These results demonstrate a consistent, although modest, benefit beyond visibility-only weighting. We do not claim that this establishes calibration or guaranteed robustness to inaccurate geometry.