acceptodds
Under review as a conference paper at ICLR 2027

Learning World Dynamics from Deblurring Discrepancy for Hand Trajectory Prediction

Abstract

Predicting human hand trajectories in egocentric views is crucial for mixed reality, visual imitation learning, and human-robot collaboration. It enables autonomous systems to anticipate human intent and support timely interaction. However, this task remains challenging in dynamic interaction scenes, where moving objects and flexible hand adjustments introduce complex interaction dynamics. World models offer a promising way to address this challenge by modeling and predicting future world states beyond hand positions. Nonetheless, pronounced motion blur from fast hand-object motion and rapid viewpoint changes disturbs the observed interaction patterns and makes future visual states highly uncertain. In this paper, we propose to learn world dynamics from ***deblurring discrepancy***, a feature-level residual between blurred and deblurred visual representations. Our key insight is that motion blur manifests as visual ambiguity in image space, but originates from structured temporal integration during image formation. The feature discrepancy induced by deblurring thus preserves motion-related cues of hands, objects, and the camera. Based on this, we propose a deblurring-discrepancy-guided framework, dubbed **DynamicHand**, for egocentric hand trajectory prediction (EHTP). **DynamicHand** extracts blur-induced motion cues from deblurring discrepancy features and predicts their future evolution to facilitate world-state modeling and hand forecasting. We further design a discrepancy-aware distillation strategy, where discrepancy strength and variation adaptively weight the transfer of teacher latents to a lightweight student model for faster inference. Extensive experiments on public datasets and our new benchmark show that **DynamicHand** consistently improves EHTP performance over existing baselines, demonstrating that deblurring discrepancy can serve as an effective bridge between visual restoration and world-dynamics learning in dynamic interaction scenes.

Primary area:
learning on time series and dynamical systems
Keywords:
Hand Motion Forecasting, Hand Trajectory Prediction, Egocentric Vision, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VPgXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DynamicHand uses the feature residual between raw and deblurred observations to condition a dual-branch predictor of future latent states and hand trajectories, and distills this predictor into a raw-RGB, one-step student (Secs. 3.1-3.3). Evaluation covers Ego4D, EgoPAT3D-DT, and the proposed six-scenario EgoDyn benchmark, with additional representation controls and subject-disjoint testing (Tables 1-3, 9-12).

Critical strengths
  1. Treating restoration-induced feature changes as an auxiliary predictive signal is an interesting contribution, and the matched auxiliary-branch controls and raw/deblurred-input comparisons help distinguish its benefit from simply adding capacity or another visual input (Appendix A.7, Table 9; Appendix A.11, Table 12). The synthetic-blur experiment further provides direct evidence that discrepancy strength responds to controlled within-exposure displacement (Appendix A.8, Fig. 7).
  2. The full model improves forecasting across all six EgoDyn scenarios and both public datasets, while testing on two unseen subjects in One-Hand Tossing provides a focused check of subject transfer (Tables 2-3 and 10). The raw-RGB student also retains competitive accuracy with a reported 17.0 ms inference time, making the approach more practically useful (Tables 2-3 and 6).
Critical weaknesses
  1. The strongest-blur results on EgoDyn are difficult to interpret because annotation reliability, sample filtering, and scenario composition are not quantified. Appendix A.3 (lines 742-755) generates labels by deblurring videos and applying HaMeR, manually corrects noticeable errors, and discards clips that cannot be confidently corrected. This is a reasonable annotation procedure, but no independent localization-error audit or correction/rejection rates establish its accuracy and coverage under severe blur. All methods share the evaluation labels, so this does not establish a bias favoring DynamicHand; it does leave unclear how much the reported gains reflect more accurate actual image-plane hand locations and how well they extend to the hardest captured interactions. Moreover, Table 7 pools scenarios with different observation/prediction horizons (Table 1) and reports only relative improvements, without per-bin absolute errors or scenario composition. Its monotonic trend could therefore partly reflect which scenarios and retained clips populate each blur bin, rather than a stronger benefit within comparable interactions. Training once on the full dataset appropriately removes subset-specific training as a confound, and the public-dataset results and subject-disjoint One-Hand Tossing test provide useful corroboration (Table 3; Appendix A.9), but they do not resolve these particular uncertainties. Existing annotation audits and scenario-stratified error summaries would materially strengthen the claimed benefit under stronger blur; without them, that conclusion should be limited to the retained, detector-assisted benchmark subsets.
  2. The experiments establish a useful restoration-sensitive representation more clearly than they establish the proposed within-exposure explanation for its predictive benefit. The matched raw-feature branch and raw/deblurred input controls address simple capacity and input-access explanations (Appendices A.7 and A.11, Tables 9 and 12), while Appendix A.8 provides meaningful evidence that even the uncorrected residual responds to controlled blur. However, that experiment measures a scalar discrepancy before the learned encoder under whole-image translation, and its strongest correlation follows subtraction of an image-specific zero-motion restoration response (Eqs. (17)-(18)). The actual predictor encodes the uncorrected residual, and DACM weights samples using the norm and variation of those learned states (Eqs. (2), (4), and (13)-(14)). Thus, the controlled analysis does not establish whether the motion-sensitive component, other restoration responses, or their combination drives the learned forecasting gains and sample weights. Table 4 shows that predicting future discrepancy helps, but not which component is being propagated; the heatmaps in Fig. 5 do not close that gap. This limits the mechanistic claim that the two branches capture complementary exposure-time and inter-frame dynamics (Sec. 3.2, lines 255-260), rather than the reported forecasting improvement itself. The explanation should acknowledge this distinction. A targeted probe connecting learned-state responses to forecast changes under controlled motion would test this explanation within the task-oriented forecasting scope in Appendix A.12.
Questions for authors
  1. [Clarification; Weakness 1] What existing annotation-quality records are available for EgoDyn: the fraction of joints/clips corrected or discarded, the criteria used, and any localization-error audit independent of the initial HaMeR predictions? Please distinguish blur levels where records permit, and clarify whether severe-blur clips disproportionately failed annotation. If no independent audit exists, an explicit statement would help establish the evidence boundary; a new annotation campaign is not requested for the rebuttal.
  2. [Clarification; Weakness 1] From the existing predictions and subset assignments underlying Table 7, can you report absolute ADE/FDE and clip/video counts for each blur bin, its scenario composition, and whether the increasing benefit holds within individual scenarios? Please also specify the temporal sampling offsets and clip-generation strides for EgoDyn and the modified public-dataset protocols, so the interaction horizons and dependence among test clips can be interpreted (Table 1; Appendix A.3). This asks for the current protocol and a breakdown of existing results, not retraining.
  3. [Clarification; Weakness 2] Does any existing analysis connect the learned discrepancy states or their predicted evolution to forecast improvements while separating controlled motion effects from the zero-motion restoration response identified in Appendix A.8? In particular, is the baseline correction used anywhere in the predictor or DACM weighting, or only in the synthetic analysis? Please distinguish what is established for the pre-encoder scalar residual from what is established for the learned states; the controlled validation suggested above can remain future work.
  4. [Clarification; Weakness 2] What are the exact definitions and numerical coefficients of Ldisc and Lmot, the weights in Eqs. (15)-(16), and alpha and lambda in Eq. (14)? Please give the existing configurations used for the relevant ablations and clarify how the auxiliary residual losses differ from the branch reconstruction/distillation losses. These details determine what trains the claimed dynamics representation and how strongly discrepancy-based weighting influences it; no new hyperparameter sweep is requested.
7 comments
AuthorsOpenReview ↗

Response to Reviewer VPgX (Part 1 / 4)

We sincerely thank the reviewer for the thoughtful and constructive feedback, and for recognizing the value of deblurring discrepancy as an auxiliary predictive representation, as well as the effectiveness of our full model and discrepancy-aware distillation strategy. We greatly appreciate the reviewer’s insightful suggestions, which have helped us further improve the quality of our work.

We have conducted additional experiments, provided further quantitative analyses, and clarified the corresponding discussions in response to the reviewer’s comments. Our point-by-point responses are provided below.


Q1: What existing annotation-quality records are available for EgoDyn: the fraction of joints/clips corrected or discarded, the criteria used, and any localization-error audit independent of the initial HaMeR predictions? Please distinguish blur levels where records permit, and clarify whether severe-blur clips disproportionately failed annotation. If no independent audit exists, an explicit statement would help establish the evidence boundary; a new annotation campaign is not requested for the rebuttal.

A1: We thank the reviewer for the valuable comment on the annotation quality records of EgoDyn. As described in Appendix A. 3, EgoDyn was collected from eight participants using a head-mounted Intel RealSense D435i at 30 FPS. We first apply NAFNet to deblur the recorded videos and then use HaMeR to obtain frame-wise hand-joint annotations. We manually inspect the resulting trajectories and correct noticeable localization errors and hand identity inconsistencies. Clips with severe ambiguities that cannot be confidently resolved through manual correction are discarded.

Following the reviewer's suggestion, we provide additional annotation quality statistics across the low-, medium-, and high-blur subsets defined in Appendix A.5. Specifically, we report the number of reviewed clips, the percentages of clips and joint annotations requiring manual correction, the clip rejection rates, and the mean correction magnitudes of manually corrected joints. The latter is defined as the average normalized Euclidean displacement between the initial HaMeR predictions and the manually corrected coordinates.

As shown in Tab. 7 of the revised appendix (also provided in R3-Tab. 1), the correction rates increase with blur severity. The percentages of corrected clips and corrected joints increase from 8.7% and 2.8% under low blur to 15.6% and 5.8% under high blur, respectively. The clip rejection rate also increases from 1.0% under low blur to 4.8% under high blur, indicating that severely blurred clips are more likely to contain ambiguities that cannot be reliably resolved through manual correction. Meanwhile, the mean correction magnitude increases from 0.0108 to 0.0203, suggesting that the localization adjustments required for more severely blurred observations are generally larger. We attribute these trends to the partial loss of fine-grained hand appearance and joint boundary information under stronger motion blur, which makes accurate joint localization more challenging for HaMeR. Moreover, rapid hand movements can cause larger inter-frame positional variations and partial finger overlap. This increases the difficulty of maintaining temporally consistent hand identities and joint correspondences, thus requiring more manual intervention.

We acknowledge that no independent localization-error audit against separately annotated ground truth was conducted as part of the original annotation pipeline. Such an audit would require an additional round of independent frame-wise and fully manual annotation, involving substantial human effort beyond the original annotation and quality-control process. Therefore, the reported correction magnitudes quantify manual adjustments to the initial HaMeR predictions.

R3-Table 1: Annotation quality statistics of EgoDyn across low-, medium-, and high-blur subsets.

Blur LevelReviewed ClipsCorrected ClipsCorrected JointsDiscarded ClipsMean Correction Magnitude
Low19,3478.7%2.8%1.0%0.0108
Medium19,42612.4%3.4%2.4%0.0126
High19,20415.6%5.8%4.8%0.0203

Reviewed clips refer to all clips manually inspected during the annotation quality-control process, including those subsequently discarded.

AuthorsOpenReview ↗

Response to Reviewer VPgX (Part 2 / 4)

Q2: From the existing predictions and subset assignments underlying Table 7, can you report absolute ADE/FDE and clip/video counts for each blur bin, its scenario composition, and whether the increasing benefit holds within individual scenarios? Please also specify the temporal sampling offsets and clip-generation strides for EgoDyn and the modified public-dataset protocols, so the interaction horizons and dependence among test clips can be interpreted (Table 1; Appendix A.3). This asks for the current protocol and a breakdown of existing results, not retraining.

A2: We thank the reviewer for this helpful suggestion. To better characterize the effect of motion blur and the benefits of our dynamics-aware conditioning, we have conducted a more detailed analysis using the existing predictions and blur-level assignments, without retraining any models.

(1) Absolute forecasting errors and scenario composition across blur levels.

We first report the absolute ADE/FDE of both the World-State-Branch-only variant and the full DynamicHand model, aggregated over all clips within each blur bin.

R3-Table 2: Absolute forecasting errors and relative error reductions of DynamicHand over the World-State-Branch-only (WS-only) variant across different motion blur levels on EgoDyn.

Blur LevelTest ClipsWS-only ADE/FDEADE/FDEADE ReductionFDE Reduction
Low1,6620.053 / 0.0730.051 / 0.0694.7%5.2%
Medium1,5870.061 / 0.0850.053 / 0.07312.1%14.4%
High1,2410.072 / 0.1020.058 / 0.07820.0%23.3%

The experimental results in R3-Tab. 2 show that the absolute forecasting errors increase with blur severity for both models, while the relative improvement of DynamicHand also becomes progressively larger. In particular, the overall ADE/FDE reductions increase from 4.7%/5.2% under low blur to 20.0%/23.3% under high blur. These results suggest that deblurring discrepancy provides increasingly valuable dynamics-aware cues as motion blur becomes more severe, enabling more effective future hand trajectory prediction.

(2) Scenario-stratified analysis.

To determine whether this trend is driven by changes in scenario composition, we further examine the relative improvements within each individual scenario. We first present the scenario composition of EgoDyn (i.e., number of clips of each task) in R3-Tab. 3.

R3-Table 3: Scenario composition of the low-, medium-, and high-blur subsets in the EgoDyn test set, reported by the number of evaluation clips.

ScenarioLowMediumHighTotal
Moving-Object Grasping7135873341,634
One-Hand Tossing142183198523
Table-Tennis Bouncing139218300657
Handshaking292308214814
High-Five3978129246
Human-Robot Handover33721366616
Total1,6621,5871,2414,490

Then, we report the relative improvements within each individual scenario at different blur levels. As shown in R3-Tab. 4 , all six scenarios exhibit increasing trends in both ADE and FDE reductions from low to medium and high blur levels. This consistent within-scenario trend indicates that the increasing benefit is not merely an artifact of different scenario compositions across blur bins.

R3-Table 4: Scenario-stratified forecasting performance across different motion blur levels on EgoDyn. Relative ADE/FDE reductions are computed against the corresponding World-State-Branch-only variant within each scenario and blur subset.

ScenarioBlurClipsFull ADE/FDEADE/FDE Reduction
Moving-Object GraspingLow7130.069 / 0.0944.3% / 4.7%
Medium5870.073 / 0.10110.8% / 13.1%
High3340.083 / 0.11416.7% / 17.6%
One-Hand TossingLow1420.042 / 0.0565.7% / 7.7%
Medium1830.045 / 0.06214.9% / 15.5%
High1980.050 / 0.06720.6% / 26.6%
Table-Tennis BouncingLow1390.033 / 0.0533.8% / 5.2%
Medium2180.035 / 0.05712.5% / 14.8%
High3000.038 / 0.06423.3% / 25.8%
HandshakingLow2920.042 / 0.0586.6% / 6.1%
Medium3080.045 / 0.06313.1% / 17.2%
High2140.048 / 0.06721.6% / 27.6%
High-FiveLow390.062 / 0.0527.8% / 9.0%
Medium780.068 / 0.06015.8% / 16.1%
High1290.077 / 0.07122.1% / 27.4%
Human-Robot HandoverLow3370.031 / 0.0423.7% / 4.9%
Medium2130.032 / 0.04411.4% / 13.7%
High660.034 / 0.04721.3% / 23.3%

(3) Temporal sampling and clip-generation protocols.

We clarify that the temporal sampling interval (frame offset) and sliding-window clip-generation stride are both set to 1 frame throughout EgoDyn, the Ego4D hand forecasting benchmark, and EgoPAT3D-DT protocols. No additional temporal subsampling is applied. Thus, the test windows are not statistically independent at the clip level. However, the train/validation/test splits are video-disjoint, ensuring that no source video is shared across different splits.

AuthorsOpenReview ↗

Response to Reviewer VPgX (Part 3 / 4)

Q3: Does any existing analysis connect the learned discrepancy states or their predicted evolution to forecast improvements while separating controlled motion effects from the zero-motion restoration response identified in Appendix A.8? In particular, is the baseline correction used anywhere in the predictor or DACM weighting, or only in the synthetic analysis? Please distinguish what is established for the pre-encoder scalar residual from what is established for the learned states; the controlled validation suggested above can remain future work.

A3: We thank the reviewer for this insightful comment! Firstly, we clarify that the zero-motion baseline correction in Appendix A.8 is used exclusively for the controlled synthetic analysis. Specifically, it accounts for image-specific restoration responses in the absence of motion, allowing us to isolate the discrepancy increment attributable to controlled within-exposure motion. Neither the actual predictor nor DACM weighting applies this correction, since a corresponding zero-motion counterpart for the same dynamic interaction process is unavailable in real-world scenarios. In contrast, in the synthetic analysis, we use the zero-motion restoration response of each reference image as its baseline to examine the relationship between motion blur and deblurring discrepancy more directly. Importantly, the positive correlation between motion strength and deblurring discrepancy also persists without baseline correction, and the observed motion sensitivity is not an artifact of baseline subtraction.

To further investigate the relationship between foreground motion and deblurring discrepancy, we conduct additional independent motion measurements with a stationary camera and fixed exposure duration. As shown in Fig. 8 and Tab. 11 of the revised appendix (also provided in R3-Tab. 5), we separately record moving objects and moving hands at three clearly distinguishable speed levels (low, medium, and high), collecting 50 videos per level and 300 videos in total. We manually annotate the image-plane coordinates of object and hand centers across consecutive frames to independently measure their motion speeds. Fig. 9 of the revised appendix shows that the mean deblurring discrepancy increases monotonically with motion speed for both conditions. The Pearson and Spearman correlations with ordinal speed ranks are 0.674 and 0.693 for object motion, and 0.911 and 0.882 for hand motion, respectively. This controlled experiment provides evidence that the pre-encoder discrepancy responds systematically to real foreground motion in the absence of camera movement. Since the pre-encoder discrepancy serves as the input to the discrepancy-state encoder, its motion-related responses can potentially be encoded into the learned discrepancy states, providing a possible pathway for dynamics-aware information to contribute to subsequent temporal prediction.

R3-Table 5: Mean image-plane speeds of moving objects and hands across the low-, medium-, and high-speed groups, measured through manual annotations under a stationary camera.

Speed GroupMoving Objects (pixels/frame)Moving Hands (pixels/frame)
Low8.8210.50
Medium21.5915.05
High41.3638.33

Regarding the learned discrepancy states, Tab. 4 of the main text and Tab. 13 of the revised appendix demonstrate that predicting their future evolution provides additional benefits for hand trajectory forecasting. However, we acknowledge that the existing experiments do not directly establish how much of the motion-sensitive information identified in the pre-encoder discrepancy is retained by the learned states or contributes to the forecasting improvements. In particular, the actual predictor and DACM may also encode and exploit other restoration-induced responses. We have clarified this distinction in Appendix A. 13 and moderated our mechanistic interpretation accordingly. Directly connecting controlled motion variations to learned-state responses and their subsequent effects on forecasting mentioned by the reviewer is an interesting and important direction and we will explore in future work.

Reviewer_ygSHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses motion blur for egocentric hand trajectory prediction by forecasting the feature residual between raw and deblurred frames. The idea is well grounded, but its gain across datasets and the cost of the full pipeline need further experimental support.

Critical strengths

a. The deblurring discrepancy is simple to obtain, requiring only an off-the-shelf deblurring model and a frozen visual encoder. The dual-branch design forecasts this discrepancy to condition world-state prediction, which avoids reconstructing future images.

b. The discrepancy-aware distillation transfers the dynamics learned by the full model into a lightweight student that uses only raw RGB. The student predicts in one step, which makes the framework practical for real-time applications.

Critical weaknesses

a. Eq. (2) captures only motion within the exposure interval, yet Eq. (7) forecasts it across frame intervals. With a large sampling interval, this cue may poorly reflect hand motion between frames. The sampling interval of each dataset should be reported, together with the gain under different intervals.

b. Table 4 ablates the discrepancy conditioning only on two EgoDyn scenarios. The margins in Table 3 are smaller, e.g., 0.164 versus 0.172 ADE on Ego4D. I would like to see the World-State Branch only variant on Ego4D and EgoPAT3D-DT.

c. Table 5 reports no model size, input resolution, or latency for the four encoders. This cost matters because the full model encodes both raw and deblurred frames. Adding these complexity details would better support the choice of DINOv2.

d. Table 8 reports no latency or model size, although Sec. 3.3 states that the full model requires deblurring at inference. Both compared models are from 2022. Complexity details and a recent method, such as Kong et al., (2025), would make the comparison more complete.

Questions for authors

see above

5 comments
AuthorsOpenReview ↗

Response to Reviewer ygSH (Part 1 / 3)

We sincerely thank the reviewer for the thoughtful and constructive feedback, and for recognizing the simplicity and effectiveness of the proposed deblurring discrepancy representation and dual-branch predictive framework, as well as the efficiency and practicality of our discrepancy-aware distillation strategy. We greatly appreciate the reviewer’s valuable suggestions, which have helped us further strengthen the experimental validation and improve the quality of our paper.

We have incorporated the additional experimental results, quantitative analyses, and corresponding discussions into the revised manuscript and appendix, which are highlighted in blue for better readability. Our point-by-point responses are provided below.


Q1: Eq. (2) captures only motion within the exposure interval, yet Eq. (7) forecasts it across frame intervals. With a large sampling interval, this cue may poorly reflect hand motion between frames. The sampling interval of each dataset should be reported, together with the gain under different intervals.

A1: We sincerely thank the reviewer for raising this important concern regarding the relationship between within-exposure motion cues and inter-frame temporal prediction. We agree that the temporal sampling interval is an important factor when evaluating whether deblurring discrepancy remains informative for forecasting future interaction dynamics. Here, we first clarify that the sampling intervals of EgoDyn, EgoPAT3D-DT, and Ego4D hand forecasting benchmark are 33.3, 33.3, and 166.7 ms. Then, we conduct an additional ablation study on the sampling interval with EgoDyn to better present the effectiveness of deblurring discrepancy. Specifically, we report the forecasting performance of the full DynamicHand model and the World-State-Branch-only variant under three sampling strides (1, 2, and 4 frames) on the Handshaking task of EgoDyn. These strides correspond to 33.3, 66.7, and 133.3 ms between consecutive sampled frames at 30 FPS. The numbers of observed and future frames are kept fixed at 3 and 6 for each prediction, which is consistent with Tab. 1 of the main text.

As shown in Tab. 15 of the revised appendix (also provided in R2-Tab. 1), increasing the temporal sampling interval from 1 to 4 frames leads to higher ADE/FDE for both methods, reflecting the increased difficulty of forecasting under sparser temporal observations and longer prediction horizons. Nevertheless, DynamicHand (Full) consistently outperforms the World-State-Branch-only variant across all three sampling intervals. Specifically, dynamics-aware conditioning yields ADE/FDE reductions of 13.2%/18.9%, 11.9%/15.9%, and 6.0%/6.6% at sampling intervals of 1, 2, and 4 frames, respectively. The consistent performance improvements demonstrate that within-exposure deblurring discrepancy cues remain complementary to inter-frame world-state evolution across different temporal sampling regimes. The reduced gains at larger intervals suggest that the predictive utility of deblurring discrepancy is affected by the temporal correspondence between exposure-level dynamics and inter-frame motion. Thanks to the reviewer's constructive comments, we identify the attenuation of dynamics-aware conditioning under larger sampling intervals, providing valuable guidance for practical deployment and future improvements.

R2-Table 1: Ablation study on temporal sampling intervals on EgoDyn.

Sampling IntervalModelADE ↓FDE ↓ADE Reduction ↑FDE Reduction ↑
1 frameWorld-State Branch only0.0530.07413.2%18.9%
DynamicHand (Full)0.0460.060
2 framesWorld-State Branch only0.0590.08211.9%15.9%
DynamicHand (Full)0.0520.069
4 framesWorld-State Branch only0.0670.0916.0%6.6%
DynamicHand (Full)0.0630.085
AuthorsOpenReview ↗

Response to Reviewer ygSH (Part 2 / 3)

Q2. Table 4 ablates the discrepancy conditioning only on two EgoDyn scenarios. The margins in Table 3 are smaller, e.g., 0.164 versus 0.172 ADE on Ego4D. I would like to see the World-State Branch only variant on Ego4D and EgoPAT3D-DT.

A2: We thank the reviewer for this valuable suggestion! To further validate the effectiveness of dynamics-aware conditioning beyond EgoDyn, we extend the World-State-Branch-only ablation to Ego4D and both the seen and unseen splits of EgoPAT3D-DT, following the same experimental protocols as in Tab. 3 of the main text.

As shown in R2-Tab. 2, DynamicHand (Full) consistently outperforms the World-State-Branch-only variant across all three evaluation settings. Specifically, incorporating the Discrepancy-State Branch reduces ADE/FDE by 2.96%/4.74% on Ego4D, 11.25%/6.40% on EgoPAT3D-DT (Seen), and 9.84%/9.35% on EgoPAT3D-DT (Unseen). Although the gains on Ego4D are relatively modest, consistent improvements across all datasets demonstrate that deblurring discrepancy provides complementary motion-related information beyond conventional world-state representations. Interestingly, this observation is consistent with our findings in the response to Q1, where we show that larger temporal sampling intervals attenuate the benefits of dynamics-aware conditioning. Specifically, the smaller gains on Ego4D (166.7 ms sampling interval) compared with EgoPAT3D-DT (33.3 ms) provide further evidence of this temporal dependency.

R2-Table 2: Ablation of discrepancy-guided dual-branch modeling on Ego4D and EgoPAT3D-DT.

ApproachEgo4D ADE/FDEEgoPAT3D-DT (Seen) ADE/FDEEgoPAT3D-DT (Unseen) ADE/FDE
World-State Branch Only0.169 / 0.1900.080 / 0.1250.061 / 0.107
DynamicHand (Full)0.164 / 0.1810.071 / 0.1170.055 / 0.097

Q3: Table 5 reports no model size, input resolution, or latency for the four encoders. This cost matters because the full model encodes both raw and deblurred frames. Adding these complexity details would better support the choice of DINOv2.

A3: We thank the reviewer for this constructive comment. The suggested complexity is important for evaluating visual encoder choices. Accordingly, we extend Tab. 5 by reporting the parameter count, input resolution, and average single-image encoding latency of all four visual encoders, measured on one NVIDIA A100 GPU. All encoders follow their respective official preprocessing pipelines and model-specific input configurations.

As shown in Tab. 5 of the revised manuscript (also provided in R2-Tab. 3), DINOv2 achieves the best overall trade-off between prediction accuracy and computational efficiency. Specifically, it requires only 86.6M parameters and 13.7 ms per image for competitive forecasting performance, with substantially fewer parameters and lower latency than the larger encoders. These additional complexity measurements further support our choice of DINOv2 as the default visual encoder in DynamicHand.

R2-Table 3: Ablation study of different visual feature encoders on EgoDyn.

Visual EncoderHandshakingHuman-Robot HandoverParametersInput ResolutionLatency (ms/image)
ADE ↓FDE ↓ADE ↓FDE ↓
V-JEPA 2 (Assran et al.)0.0510.0680.0370.0441.0B384×384107.0
LingBot-Vision (Fu et al.)0.0460.0640.0310.0391.1B512×51238.7
DINOv2 (Oquab et al.)0.0460.0600.0310.03986.6M224×22413.7
DINOv3 (Siméoni et al.)0.0450.0620.0320.03987.6M1024×102458.4
AuthorsOpenReview ↗

Response to Reviewer ygSH (Part 3 / 3)

Q4: Table 8 reports no latency or model size, although Sec. 3.3 states that the full model requires deblurring at inference. Both compared models are from 2022. Complexity details and a recent method, such as Kong et al., (2025), would make the comparison more complete.

A4: We thank the reviewer for this valuable comment! We extend the experiments in Appendix A. 6 by including the suggested deblurring method, EVSSM (Kong et al., 2025), and reporting the parameter counts and average per-image deblurring latency of all three models. For a fair comparison, all three deblurring models are evaluated with EgoPAT3D-DT on a single NVIDIA A100 GPU, using the same input resolution of 960x540. The reported latency measures the end-to-end processing time for a single image, including image preprocessing and model forward computation.

As shown in Tab. 9 of the revised appendix (also provided in R2-Tab. 4), although NAFNet has more parameters (67.9M) than the other two models, it provides substantially better practical inference efficiency, i.e., the lowest deblurring latency of 77.8 ms/image. In terms of forecasting performance, NAFNet achieves the best results on Ego4D and the unseen split of EgoPAT3D-DT, while remaining competitive on the seen split. EVSSM achieves better performance on the seen split but incurs substantially higher computational cost. These results indicate that NAFNet provides better trade-off between forecasting accuracy and computational efficiency, supporting our choice of NAFNet as the default deblurring model in DynamicHand.

R2-Table 4: Ablation study of deblurring models on the Ego4D and EgoPAT3D-DT datasets.

ApproachEgo4DEgoPAT3D-DT (Seen)EgoPAT3D-DT (Unseen)Parameters (M)Latency (ms/image)
ADE ↓FDE ↓ADE ↓FDE ↓ADE ↓FDE ↓
Restormer (Zamir et al., 2022)0.1670.1850.0690.1140.0580.10126.1621.6
NAFNet (Chen et al., 2022)0.1640.1810.0710.1170.0550.09767.977.8
EVSSM (Kong et al., 2025)0.1640.1820.0660.1130.0560.10017.12519.4
Reviewer_gUqbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DynamicHand uses feature differences between raw and deblurred images to predict egocentric hand trajectories. A dual-branch predictor is distilled into a single-step RGB student.

Critical strengths

Preserving the information altered during the deblurring process for prediction is a valuable design in this paper. Branch ablations and optical-flow controls support using deblurring discrepancy as a complementary cue.

Critical weaknesses
  1. The synthetic test supports blur sensitivity, but does not separate interaction motion from camera motion or exposure effects. Controlled exposure and independent motion measurements would strengthen the interpretation.

  2. EgoDyn uses HaMeR labels from deblurred video, but manual correction and exclusion rates are unreported. The additional subject-disjoint test covers only two new participants in one scenario, limiting evidence for broader generalization.

  3. The direct-student comparison removes several teacher losses together, so its gain does not isolate discrepancy-specific knowledge. Compare teachers with and without the discrepancy branch under matched student objectives.

Questions for authors

See weaknesses. I may raise my score if authors solve my concerns.

5 comments
AuthorsOpenReview ↗

Response to Reviewer gUqb (Part 1 / 4)

We sincerely thank the reviewer for the thoughtful and constructive feedback, and for recognizing the value of preserving information altered during deblurring for hand trajectory prediction, as well as the effectiveness of our dual-branch ablations and optical-flow comparisons. We greatly appreciate the reviewer’s valuable suggestions, which have helped us improve the quality of our paper.

We have incorporated the new experimental results, quantitative analyses, and corresponding discussions into the revised manuscript and appendix, which are highlighted in blue for better readability. Our point-by-point responses are provided below.

AuthorsOpenReview ↗

Response to Reviewer gUqb (Part 2 / 4)

Q1: The synthetic test supports blur sensitivity, but does not separate interaction motion from camera motion or exposure effects. Controlled exposure and independent motion measurements would strengthen the interpretation.

A1: We sincerely thank the reviewer for the constructive suggestion on strengthening our analysis through controlled exposure experiments and independent motion measurements.

Controlled exposure experiments. Firstly, we conduct an additional controlled synthetic exposure experiment to investigate the effect of exposure duration. Specifically, we fix the reference image and image-plane velocity at 1.2 pixels/ms, while varying the simulated exposure duration from 0 to 10 ms. Following the operation in Appendix A.8, each synthetic observation is generated by averaging 16 uniformly sampled translated images over the exposure interval. We average the discrepancy responses across eight motion directions and subtract each image's zero-exposure discrepancy to account for its image-specific restoration response. As shown in Fig. 7 of the revised appendix, the mean baseline-corrected discrepancy increment increases monotonically from approximately 0.03 at 2 ms to 0.27 at 10 ms, with shaded regions indicating 95% confidence intervals. The discrepancy increments exhibit positive correlations with exposure duration, achieving a Pearson correlation coefficient of 0.794 and a Spearman rank correlation coefficient of 0.810. Since the image-plane velocity is fixed, longer exposure durations accumulate larger within-exposure displacements, resulting in stronger motion blur and larger discrepancy responses. These results provide additional controlled evidence that deblurring discrepancy responds to motion accumulated over the exposure interval, consistent with the temporal integration process described in Eq. (3) of the main text. This supports its interpretation as a proxy for motion-related visual changes induced during image exposure.

Independent motion measurements. Moreover, we implement independent motion measurements with a stationary camera and the same exposure duration, separately recording moving objects and moving hands at low, medium, and high speeds, as shown in Fig. 8 of the revised appendix. We collect 50 videos for each combination of motion type and speed level, instructing participants to deliberately adjust their actions for the three clearly distinguishable speed levels. This setup results in 300 videos in total and allows us to examine foreground-motion responses without camera movement. We manually annotate the image-plane coordinates of the centers of moving objects and hands between consecutive frames, and their mean speeds across the three levels are presented in Tab. 11 of the revised appendix (also provided in R1-Tab. 1).

R1-Table 1: Mean image-plane speeds of moving objects and hands across the low-, medium-, and high-speed groups, measured through manual annotations under a stationary camera.

Speed GroupMoving Objects (pixels/frame)Moving Hands (pixels/frame)
Low8.8210.50
Medium21.5915.05
High41.3638.33

As shown in Fig. 9 of the revised appendix, the mean L1 deblurring discrepancy increases monotonically across the three speed groups in both conditions, with error bars indicating 95% confidence intervals. The Pearson and Spearman correlations with ordinal speed ranks (low/medium/high) are 0.674 and 0.693 for object motion, and 0.911 and 0.882 for hand motion, respectively. These results provide evidence that deblurring discrepancy responds to real object and hand motion even when camera motion is absent. Besides, greater foreground motion intensity is associated with larger deblurring discrepancy, which is also consistent with the temporal integration process described in Eq. (3) of the main text. Thanks to the reviewer’s helpful comments, we hope that the independent motion measurements along with the controlled exposure experiment can strengthen our interpretation of deblurring discrepancy in our work.

AuthorsOpenReview ↗

Response to Reviewer gUqb (Part 3 / 4)

Q2: EgoDyn uses HaMeR labels from deblurred video, but manual correction and exclusion rates are unreported. The additional subject-disjoint test covers only two new participants in one scenario, limiting evidence for broader generalization.

A2: We thank the reviewer for the valuable comment on reporting manual correction and exclusion rates! As described in Appendix A.3, EgoDyn was collected from eight participants using a head-mounted Intel RealSense D435i at 30 FPS. We first apply NAFNet to deblur the recorded videos and then use HaMeR to obtain frame-wise hand-joint annotations. We manually inspect the resulting trajectories, and correct noticeable localization errors and identity inconsistencies. Clips with severe ambiguity that cannot be confidently corrected are discarded. As suggested by the reviewer, we additionally report the number of reviewed clips, the percentages of clips and joint annotations requiring manual correction, the clip rejection rates, and the mean correction magnitudes of corrected joints, i.e., the average displacement between initial HaMeR labels and manually corrected coordinates. As shown in Tab. 7 of the revised appendix (also provided in R1-Tab. 2), the statistics across the low-, medium-, and high-blur subsets defined in Appendix A.5 indicate that clips with more severe motion blur are more likely to exhibit annotation ambiguity and require greater manual intervention. We attribute this trend to the partial loss of fine-grained hand appearance and joint boundary information under stronger motion blur, which makes annotation more challenging for HaMeR. Besides, rapid hand movements can cause larger inter-frame positional variations and partial finger overlap, increasing the difficulty of maintaining temporally consistent hand identities and joint correspondences, thus requiring more manual corrections.

R1-Table 2: Annotation quality statistics of EgoDyn across low-, medium-, and high-blur subsets.

Blur LevelReviewed ClipsCorrected ClipsCorrected JointsDiscarded ClipsMean Correction Magnitude
Low19,3478.7%2.8%1.0%0.0108
Medium19,42612.4%3.4%2.4%0.0126
High19,20415.6%5.8%4.8%0.0203

Reviewed clips refer to all clips manually inspected during the annotation quality-control process, including those subsequently discarded.

We also thank the reviewer for pointing out the limited subject and scenario diversity in our original subject-disjoint evaluation. To further strengthen this evaluation, we extend the test to five new participants who are entirely unseen during the original training, validation, and testing stages, covering three representative interaction scenarios of EgoDyn, i.e., Moving-Object Grasping, One-Hand Tossing, and Human-Robot Handover. Specifically, we collect 1,000 video clips per participant for each scenario, resulting in 5,000 clips per scenario and 15,000 clips in total. All compared methods retain their checkpoints trained on the original EgoDyn training set and are evaluated directly on these new subjects without fine-tuning or test-time adaptation. As shown in Tab. 12 of the revised appendix (also provided in R1-Tab. 3), DynamicHand (Full) and DynamicHand (DACM) consistently outperform the baseline methods across all three scenarios. Moreover, compared with the original EgoDyn test results in Tab. 2 of the main text, DynamicHand (Full) exhibits slightly lower ADE and FDE on the newly collected subject-disjoint test set across all three scenarios. These results demonstrate that the benefits of the full discrepancy-aware predictive framework are preserved across previously unseen subjects and diverse interaction patterns. Discrepancy-informed future dynamics modeling provides transferable predictive information beyond subject-specific motion patterns. Thanks to the reviewer’s valuable comment, this extended evaluation substantially broadens the evidence for subject-disjoint generalization of our proposed DynamicHand. We have added the new experiments along with the corresponding analysis to Appendix A.9.

R1-Table 3: Subject-disjoint evaluation on five unseen participants across three dynamic interaction scenarios of EgoDyn.

ApproachMoving-Object GraspingOne-Hand TossingHuman-Robot Handover
ADE ↓FDE ↓ADE ↓FDE ↓ADE ↓FDE ↓
CVM (Schöller et al.)0.1820.3130.2050.3590.1170.221
HandsOnVLM (Bao et al.)0.3490.3970.2310.2540.2080.237
EMAG (Hatano et al.)0.1400.2260.1040.1800.0680.099
USST (Bao et al.)0.1160.1610.0690.1010.0470.069
S-Mamba (Wang et al.)0.0920.1210.0580.0860.0530.074
EgoMAN (Chen et al.)0.0820.1130.0590.1020.0440.067
Uni-Hand (Ma et al.)0.0850.1120.0600.0780.0450.050
DynamicHand (DACM)0.0790.1050.0530.0700.0370.045
DynamicHand (Full)0.0730.1000.0460.0630.0290.038

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Gentle Follow-up on Our Rebuttal Responses

Dear Reviewers,

We sincerely appreciate your thoughtful and constructive feedback, which has helped us further improve our manuscript. As the rebuttal deadline is approaching, we would be very grateful if you could kindly share any further thoughts or feedback on our responses.

Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.