acceptodds
Withdrawn from ICLR 2027

Streaming 4D Hand Interaction Forecasting with an Autoregressive Video Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present a streaming 4D interaction forecaster that turns an autoregressive video model into a structured predictor of future hand motion and contact. Given recent egocentric visual and state observations, the model predicts future interaction as a joint state of bimanual articulated hands, egocentric camera motion, and joint-wise hand-object proximity. This structured representation captures both motion and interaction cues in explicit geometry, providing a forecasting space that is more directly usable than generated video alone. We realize this representation with a 4D state decoder that extracts structured interaction trajectories from intermediate denoising features of an autoregressive video model. The decoder uses current hand and camera states to ground its predictions in the present scene, while incoming observations continually update the forecast. Trained on egocentric clips spanning lab-collected 3D data and diverse in-the-wild videos, our method improves articulated hand forecasting over prior baselines under matched observation and state access, while additionally predicting future hand-object proximity cues. We further demonstrate contact-guided interaction in physics simulation using the predicted hand motion and proximity cues as control references.

Primary area:
generative models
Keywords:
egocentric forecasting, articulated hand motion, autoregressive video models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sBVROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a streaming framework that forecasts hand motion, camera motion, and hand contacts from egocentric observations. A state-conditioned decoder extracts trajectories from intermediate features of an autoregressive video model (Wan2.2-TI2V-5B), while incoming observations update the forecasts. Experiments across four datasets evaluate forecasting accuracy and key design choices, with qualitative demonstrations of continuous prediction and contact-guided interaction in physics simulation.

Critical strengths
  1. Useful structured forecasting formulation. The method jointly predicts articulated hand motion, camera motion, and hand–object proximity, providing explicit interaction cues that are more directly usable by downstream systems than generated video alone.
  2. Broad evaluation with informative ablations. Experiments across four egocentric datasets demonstrate competitive forecasting performance, and ablations clarify the contributions of visual history, state conditioning, wrist anchoring, and denoising readout choices.
Critical weaknesses
  1. It is unclear whether the improvement comes from the video generation model’s ability. The main experiments use features from the first denoising call, before the future latent has been updated. Adapting the video generator to egocentric videos barely changes forecasting accuracy (Table 8). The key issue is that the results do not show whether these priors drive the gains, or whether a strong feature extractor and a decoder trained with extensive geometric supervision are sufficient. A comparison with a non-generative video encoder under the same training setup would help answer this question.
  2. A direct generate-then-reconstruct baseline is missing. The paper does not compare intermediate-feature decoding against generating future RGB videos with the same backbone and then applying hand reconstruction and contact estimation, similar to work in [1]. Such a comparison is needed to show the proposed design’s advantages in accuracy, efficiency, and temporal consistency.
  3. Robustness to estimated current states remains untested. Quantitative experiments use annotated current states for forecasting; even comparisons without hand-state input align predictions to the GT current wrist during evaluation. Wrist rebasing alone reduces average error from 9.33 to 7.82 cm (Table 3), showing heavy reliance on accurate localization. Evaluation with estimated hand and camera states is needed to evaluate robustness under realistic input noises.
  4. Lacking comparison with SOTA method and incomplete metrics: How Do I Do That [2] also works on forecasting of 3D hand motion and contacts, which is not compared against. Also, the metrics on contact state are hand-level, lacking finger-level precision & recall, or F1 score.

[1] Large Video Planner, CoRL 2025 [2] How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions, CVPR 2025

Questions for authors
  1. Does the improvement come from the video generation model’s ability or extensive geometric supervision?
  2. Can you compare it with a baseline that generates first and then performs reconstruction? Generally, using internal features to learn will be better than a two-stage approach, but a comparison is needed to show the effectiveness of network designs. Also, comparison with related SOTA How Do I Do That is needed.
  3. Robustness to estimated current states?
Reviewer_zCa5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a forecaster that turns an autoregressive video model (Wan2.2-TI2V-5B with LongLive-2.0) into a predictor of future hand interaction. From 29 observed egocentric frames, a language instruction, and the current hand and camera states, a trained decoder reads intermediate denoising features of the frozen backbone and outputs, for the next 32 frames (1.07 s), bimanual MANO hands, egocentric camera motion, and per-joint probabilities of lying within 2 cm of an object. The decoder conditions its queries on observed states and shifts the forecast so that it starts at the observed wrist. The forecast is updated as new frames arrive. On four motion-capture datasets, the method outperforms VITRA and Being-H0 in single-forecast accuracy under matched inference inputs and predicts future contact better than persistence. Streaming forecasting on recorded video and closed-loop execution in simulation are shown qualitatively.

Critical strengths
  1. Unified structured forecast. Predicting articulated hands, camera motion, and per-joint proximity together, rather than future pixels or a single action space, is an interesting step toward forecasts that downstream systems can use directly. Conditioning on observed states also lets the same model run with full, partial, or no state information.
  2. Careful analysis and transparency. The paper compares with recent baselines under matched inference inputs, and its ablations separate the roles of state conditioning and wrist anchoring (Table 3) and of the denoising readout (Table 4). The appendix candidly reports latency, limitations, results by motion magnitude, and the effect of the backbone adaptation.
Critical weaknesses
  1. Practical value is not demonstrated. The paper motivates structured forecasts by their use in downstream control (L037–038), but provides no downstream evaluation, such as using the forecasts for VLA or policy learning. The simulation is only a qualitative demo with a floating MANO hand and privileged geometry (L456–457, L1119–1120). The horizon is short (1.07 s), while one update takes 0.66–1.03 s (Tables 11–12), so much of the horizon has passed by the time the forecast is available. The gain over zero-motion persistence is modest (8.61 → 7.48 cm, about 13%; Table 1), and the endpoint error is still 12.38 cm. In the only qualitative example in the main text (Fig. 3), the forecast appears to miss the main left-hand motion.
  2. Missing comparison with the closest work. SFHand (Liu et al., 2026a), cited at L084–085, also performs language-conditioned, streaming 3D egocentric hand forecasting with current-state input. It runs in real time and has been shown to improve manipulation policies. The experiments include neither SFHand nor other recent articulated hand forecasters (e.g., Hatano et al., 2025; Choi et al., 2026; Shi et al., 2026). It is therefore unclear what a 5B video model adds over a lightweight forecaster, and the “streaming” claim in the title is weak in comparison.
  3. Weak support for the central claim. The paper claims to expose the video model’s predicted future, but the quantitative results use the first readout, where the future latent is nearly pure noise (σ ≈ 0.999; Eq. 13), and later readouts are worse in point accuracy (Table 4). The egocentric LoRA, mentioned only in Appendix A.7, has no effect at this readout (7.49 vs. 7.48; Table 8). Replacing forecast-block features with those of another clip degrades accuracy (Table 10), but these features are computed while attending to that clip’s observed context, so this does not show that the predicted future is used. No baseline that forecasts from hand-state history alone is reported, so the contribution of the video model is not isolated.
  4. Insufficient method description, limiting reproducibility. Key parts of Section 3 are undefined or explained only in the appendix. There is no equation for constructing X^(k) from Z_t^obs and Z_t^fut,(k) (Eq. 5), and the number of tokens per latent frame and the alignment of output and latent frames in Eq. 7 are not given. The terms a_{f,m}, A_m, and v_{f,m} in Eq. 6 are underspecified, and the frames that receive states appear to differ between the text and Appendix A.5. The encoding of z^obs and the source of the instruction ℓ are not described in the main text. Figure 2 omits the backbone and most symbols in Eqs. 3–7, and its joint notation (P̂) differs from the text (Ĵ).
  5. Incomplete evaluation. Future camera motion, a claimed output, is not evaluated; camera accuracy is reported only for observed frames (Table 6). Proximity is compared only with persistence, and the 2 cm threshold, measured from mean-shape MANO joint centers (L878), is not justified. The abstract’s “matched observation and state access” (L022–023) covers inference inputs only, while the proposed model is trained on the same four corpora used for evaluation (L357–358), and its best results use 29 frames versus one for the baselines. Finally, Figure 4 shows little difference from prior methods, and ACE-Ego-Hand is better on most hand metrics (e.g., MPJPE-p 13.0 vs. 15.0 mm; Table 6).
Questions for authors
  1. Can the authors show a downstream benefit, e.g., by using the forecasts or decoder features for VLA or policy learning, or by reporting quantitative closed-loop results (such as success rates) with and without the forecast? What applications benefit from a 1 s forecast given the 0.66–1.03 s latency, and how does accuracy change for longer horizons using multiple autoregressive blocks? (could change my assessment)
  2. How does the method compare with SFHand and other articulated hand forecasters in forecasting accuracy and runtime? (could change my assessment)
  3. Can the authors report a baseline that forecasts from the observed hand-state history alone, without video features? This would isolate the contribution of the video model beyond extrapolating past motion. (could change my assessment)
  4. What exactly is the “current hand state” in Table 1? Does it include only the annotated MANO pose and translation at frame t, or also earlier states, hand shape, or camera states? Does VITRA receive the same information? Please also clarify in the abstract that training data are not matched, and consider reporting baselines fine-tuned on the same data. (clarification)
  5. Please give an explicit equation for constructing X^(k). How many tokens per latent frame are used, and at what spatial resolution? How are output frames aligned with latent frames in Eq. 7? Why is no denoising-timestep embedding given to the decoder, and could this explain the accuracy drop at later readouts? (clarification)
  6. Please specify the representation and coordinate frame of a_{f,m}, the structure of A_m, and whether v_{f,m} is binary (Eq. 6). Which frames receive states at evaluation, given the difference from Appendix A.5? How is the instruction ℓ obtained for the evaluation datasets? (clarification)
  7. Please describe the encoding of z^obs (encoder, resolution, latent shape) in the main text. What is the forecasting accuracy with the TAE used for streaming, and what is the latency with the full autoencoder used for the quantitative results? (clarification)
  8. How accurate is the forecast of future camera motion? (clarification)
  9. What is the rationale for the 2 cm threshold measured from mean-shape MANO joint centers (L176, L878), given that observed labels use annotated hand geometry (L871)? How sensitive are the results in Table 2 to the threshold? (clarification)
  10. Please revise Figure 2 to include the backbone and the notation of Section 3.2, and clarify what Figure 4 is intended to show given that ACE-Ego-Hand performs better in Table 6. (suggestion)
Reviewer_oa1gOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a streaming 4D interaction forecaster that turns an autoregressive video model into a structured predictor of future hand motion and contact. Contributions: 1) using predictive video features. 2) a state-grounded 4D decoder. Experiments show that this paper outperforms VITRA and achieves comparable results with Being-H0.

Critical strengths

This paper demonstrate that using a video forecasting model to generate predictive video features (like world action model) might outperforms VLA models for hand/camera pose prediction task.

Critical weaknesses
  1. Figure 3 suggests that future predictions are more accurate for the relatively stationary right hand than for the moving left hand. Could the authors provide an analysis of how the motion range of observed hand affects forecasting performance? What if compare the proposed method with VITRA and Being-H0 under small / medium / large motion range ?
  2. Table 1 : when the current state is given, the proposed method outperforms VITRA, whereas without state input, it achieves performance only comparable to Being-H0. The authors should provide a more insightful analysis of this observation, particularly regarding how strongly the method depends on the input state. What happens when the input state is noisy? For example, when it is estimated from the input image using an existing hand or camera pose estimation model? How does noise in the hand, joint, and camera states individually affect the final forecasting performance?
Questions for authors
  1. How would a simple pipeline combining a future video prediction model with ACE-Ego-Hand / HaWoR perform?
  2. Supplementary Table 6 suggests that, for observed-video reconstruction, the proposed method still struggles to outperform ACE-Ego-Hand even input with state. Why can't the model fully recover the known input?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.