- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DynamicHand uses the feature residual between raw and deblurred observations to condition a dual-branch predictor of future latent states and hand trajectories, and distills this predictor into a raw-RGB, one-step student (Secs. 3.1-3.3). Evaluation covers Ego4D, EgoPAT3D-DT, and the proposed six-scenario EgoDyn benchmark, with additional representation controls and subject-disjoint testing (Tables 1-3, 9-12).
Critical strengths
- Treating restoration-induced feature changes as an auxiliary predictive signal is an interesting contribution, and the matched auxiliary-branch controls and raw/deblurred-input comparisons help distinguish its benefit from simply adding capacity or another visual input (Appendix A.7, Table 9; Appendix A.11, Table 12). The synthetic-blur experiment further provides direct evidence that discrepancy strength responds to controlled within-exposure displacement (Appendix A.8, Fig. 7).
- The full model improves forecasting across all six EgoDyn scenarios and both public datasets, while testing on two unseen subjects in One-Hand Tossing provides a focused check of subject transfer (Tables 2-3 and 10). The raw-RGB student also retains competitive accuracy with a reported 17.0 ms inference time, making the approach more practically useful (Tables 2-3 and 6).
Critical weaknesses
- The strongest-blur results on EgoDyn are difficult to interpret because annotation reliability, sample filtering, and scenario composition are not quantified. Appendix A.3 (lines 742-755) generates labels by deblurring videos and applying HaMeR, manually corrects noticeable errors, and discards clips that cannot be confidently corrected. This is a reasonable annotation procedure, but no independent localization-error audit or correction/rejection rates establish its accuracy and coverage under severe blur. All methods share the evaluation labels, so this does not establish a bias favoring DynamicHand; it does leave unclear how much the reported gains reflect more accurate actual image-plane hand locations and how well they extend to the hardest captured interactions. Moreover, Table 7 pools scenarios with different observation/prediction horizons (Table 1) and reports only relative improvements, without per-bin absolute errors or scenario composition. Its monotonic trend could therefore partly reflect which scenarios and retained clips populate each blur bin, rather than a stronger benefit within comparable interactions. Training once on the full dataset appropriately removes subset-specific training as a confound, and the public-dataset results and subject-disjoint One-Hand Tossing test provide useful corroboration (Table 3; Appendix A.9), but they do not resolve these particular uncertainties. Existing annotation audits and scenario-stratified error summaries would materially strengthen the claimed benefit under stronger blur; without them, that conclusion should be limited to the retained, detector-assisted benchmark subsets.
- The experiments establish a useful restoration-sensitive representation more clearly than they establish the proposed within-exposure explanation for its predictive benefit. The matched raw-feature branch and raw/deblurred input controls address simple capacity and input-access explanations (Appendices A.7 and A.11, Tables 9 and 12), while Appendix A.8 provides meaningful evidence that even the uncorrected residual responds to controlled blur. However, that experiment measures a scalar discrepancy before the learned encoder under whole-image translation, and its strongest correlation follows subtraction of an image-specific zero-motion restoration response (Eqs. (17)-(18)). The actual predictor encodes the uncorrected residual, and DACM weights samples using the norm and variation of those learned states (Eqs. (2), (4), and (13)-(14)). Thus, the controlled analysis does not establish whether the motion-sensitive component, other restoration responses, or their combination drives the learned forecasting gains and sample weights. Table 4 shows that predicting future discrepancy helps, but not which component is being propagated; the heatmaps in Fig. 5 do not close that gap. This limits the mechanistic claim that the two branches capture complementary exposure-time and inter-frame dynamics (Sec. 3.2, lines 255-260), rather than the reported forecasting improvement itself. The explanation should acknowledge this distinction. A targeted probe connecting learned-state responses to forecast changes under controlled motion would test this explanation within the task-oriented forecasting scope in Appendix A.12.
Questions for authors
- [Clarification; Weakness 1] What existing annotation-quality records are available for EgoDyn: the fraction of joints/clips corrected or discarded, the criteria used, and any localization-error audit independent of the initial HaMeR predictions? Please distinguish blur levels where records permit, and clarify whether severe-blur clips disproportionately failed annotation. If no independent audit exists, an explicit statement would help establish the evidence boundary; a new annotation campaign is not requested for the rebuttal.
- [Clarification; Weakness 1] From the existing predictions and subset assignments underlying Table 7, can you report absolute ADE/FDE and clip/video counts for each blur bin, its scenario composition, and whether the increasing benefit holds within individual scenarios? Please also specify the temporal sampling offsets and clip-generation strides for EgoDyn and the modified public-dataset protocols, so the interaction horizons and dependence among test clips can be interpreted (Table 1; Appendix A.3). This asks for the current protocol and a breakdown of existing results, not retraining.
- [Clarification; Weakness 2] Does any existing analysis connect the learned discrepancy states or their predicted evolution to forecast improvements while separating controlled motion effects from the zero-motion restoration response identified in Appendix A.8? In particular, is the baseline correction used anywhere in the predictor or DACM weighting, or only in the synthetic analysis? Please distinguish what is established for the pre-encoder scalar residual from what is established for the learned states; the controlled validation suggested above can remain future work.
- [Clarification; Weakness 2] What are the exact definitions and numerical coefficients of Ldisc and Lmot, the weights in Eqs. (15)-(16), and alpha and lambda in Eq. (14)? Please give the existing configurations used for the relevant ablations and clarify how the auxiliary residual losses differ from the branch reconstruction/distillation losses. These details determine what trains the claimed dynamics representation and how strongly discrepancy-based weighting influences it; no new hyperparameter sweep is requested.
7 comments
Response to Reviewer VPgX (Part 2 / 4)
Q2: From the existing predictions and subset assignments underlying Table 7, can you report absolute ADE/FDE and clip/video counts for each blur bin, its scenario composition, and whether the increasing benefit holds within individual scenarios? Please also specify the temporal sampling offsets and clip-generation strides for EgoDyn and the modified public-dataset protocols, so the interaction horizons and dependence among test clips can be interpreted (Table 1; Appendix A.3). This asks for the current protocol and a breakdown of existing results, not retraining.
A2: We thank the reviewer for this helpful suggestion. To better characterize the effect of motion blur and the benefits of our dynamics-aware conditioning, we have conducted a more detailed analysis using the existing predictions and blur-level assignments, without retraining any models.
(1) Absolute forecasting errors and scenario composition across blur levels.
We first report the absolute ADE/FDE of both the World-State-Branch-only variant and the full DynamicHand model, aggregated over all clips within each blur bin.
R3-Table 2: Absolute forecasting errors and relative error reductions of DynamicHand over the World-State-Branch-only (WS-only) variant across different motion blur levels on EgoDyn.
| Blur Level | Test Clips | WS-only ADE/FDE | ADE/FDE | ADE Reduction | FDE Reduction |
|---|---|---|---|---|---|
| Low | 1,662 | 0.053 / 0.073 | 0.051 / 0.069 | 4.7% | 5.2% |
| Medium | 1,587 | 0.061 / 0.085 | 0.053 / 0.073 | 12.1% | 14.4% |
| High | 1,241 | 0.072 / 0.102 | 0.058 / 0.078 | 20.0% | 23.3% |
The experimental results in R3-Tab. 2 show that the absolute forecasting errors increase with blur severity for both models, while the relative improvement of DynamicHand also becomes progressively larger. In particular, the overall ADE/FDE reductions increase from 4.7%/5.2% under low blur to 20.0%/23.3% under high blur. These results suggest that deblurring discrepancy provides increasingly valuable dynamics-aware cues as motion blur becomes more severe, enabling more effective future hand trajectory prediction.
(2) Scenario-stratified analysis.
To determine whether this trend is driven by changes in scenario composition, we further examine the relative improvements within each individual scenario. We first present the scenario composition of EgoDyn (i.e., number of clips of each task) in R3-Tab. 3.
R3-Table 3: Scenario composition of the low-, medium-, and high-blur subsets in the EgoDyn test set, reported by the number of evaluation clips.
| Scenario | Low | Medium | High | Total |
|---|---|---|---|---|
| Moving-Object Grasping | 713 | 587 | 334 | 1,634 |
| One-Hand Tossing | 142 | 183 | 198 | 523 |
| Table-Tennis Bouncing | 139 | 218 | 300 | 657 |
| Handshaking | 292 | 308 | 214 | 814 |
| High-Five | 39 | 78 | 129 | 246 |
| Human-Robot Handover | 337 | 213 | 66 | 616 |
| Total | 1,662 | 1,587 | 1,241 | 4,490 |
Then, we report the relative improvements within each individual scenario at different blur levels. As shown in R3-Tab. 4 , all six scenarios exhibit increasing trends in both ADE and FDE reductions from low to medium and high blur levels. This consistent within-scenario trend indicates that the increasing benefit is not merely an artifact of different scenario compositions across blur bins.
R3-Table 4: Scenario-stratified forecasting performance across different motion blur levels on EgoDyn. Relative ADE/FDE reductions are computed against the corresponding World-State-Branch-only variant within each scenario and blur subset.
| Scenario | Blur | Clips | Full ADE/FDE | ADE/FDE Reduction |
|---|---|---|---|---|
| Moving-Object Grasping | Low | 713 | 0.069 / 0.094 | 4.3% / 4.7% |
| Medium | 587 | 0.073 / 0.101 | 10.8% / 13.1% | |
| High | 334 | 0.083 / 0.114 | 16.7% / 17.6% | |
| One-Hand Tossing | Low | 142 | 0.042 / 0.056 | 5.7% / 7.7% |
| Medium | 183 | 0.045 / 0.062 | 14.9% / 15.5% | |
| High | 198 | 0.050 / 0.067 | 20.6% / 26.6% | |
| Table-Tennis Bouncing | Low | 139 | 0.033 / 0.053 | 3.8% / 5.2% |
| Medium | 218 | 0.035 / 0.057 | 12.5% / 14.8% | |
| High | 300 | 0.038 / 0.064 | 23.3% / 25.8% | |
| Handshaking | Low | 292 | 0.042 / 0.058 | 6.6% / 6.1% |
| Medium | 308 | 0.045 / 0.063 | 13.1% / 17.2% | |
| High | 214 | 0.048 / 0.067 | 21.6% / 27.6% | |
| High-Five | Low | 39 | 0.062 / 0.052 | 7.8% / 9.0% |
| Medium | 78 | 0.068 / 0.060 | 15.8% / 16.1% | |
| High | 129 | 0.077 / 0.071 | 22.1% / 27.4% | |
| Human-Robot Handover | Low | 337 | 0.031 / 0.042 | 3.7% / 4.9% |
| Medium | 213 | 0.032 / 0.044 | 11.4% / 13.7% | |
| High | 66 | 0.034 / 0.047 | 21.3% / 23.3% |
(3) Temporal sampling and clip-generation protocols.
We clarify that the temporal sampling interval (frame offset) and sliding-window clip-generation stride are both set to 1 frame throughout EgoDyn, the Ego4D hand forecasting benchmark, and EgoPAT3D-DT protocols. No additional temporal subsampling is applied. Thus, the test windows are not statistically independent at the clip level. However, the train/validation/test splits are video-disjoint, ensuring that no source video is shared across different splits.
Response to Reviewer VPgX (Part 3 / 4)
Q3: Does any existing analysis connect the learned discrepancy states or their predicted evolution to forecast improvements while separating controlled motion effects from the zero-motion restoration response identified in Appendix A.8? In particular, is the baseline correction used anywhere in the predictor or DACM weighting, or only in the synthetic analysis? Please distinguish what is established for the pre-encoder scalar residual from what is established for the learned states; the controlled validation suggested above can remain future work.
A3: We thank the reviewer for this insightful comment! Firstly, we clarify that the zero-motion baseline correction in Appendix A.8 is used exclusively for the controlled synthetic analysis. Specifically, it accounts for image-specific restoration responses in the absence of motion, allowing us to isolate the discrepancy increment attributable to controlled within-exposure motion. Neither the actual predictor nor DACM weighting applies this correction, since a corresponding zero-motion counterpart for the same dynamic interaction process is unavailable in real-world scenarios. In contrast, in the synthetic analysis, we use the zero-motion restoration response of each reference image as its baseline to examine the relationship between motion blur and deblurring discrepancy more directly. Importantly, the positive correlation between motion strength and deblurring discrepancy also persists without baseline correction, and the observed motion sensitivity is not an artifact of baseline subtraction.
To further investigate the relationship between foreground motion and deblurring discrepancy, we conduct additional independent motion measurements with a stationary camera and fixed exposure duration. As shown in Fig. 8 and Tab. 11 of the revised appendix (also provided in R3-Tab. 5), we separately record moving objects and moving hands at three clearly distinguishable speed levels (low, medium, and high), collecting 50 videos per level and 300 videos in total. We manually annotate the image-plane coordinates of object and hand centers across consecutive frames to independently measure their motion speeds. Fig. 9 of the revised appendix shows that the mean deblurring discrepancy increases monotonically with motion speed for both conditions. The Pearson and Spearman correlations with ordinal speed ranks are 0.674 and 0.693 for object motion, and 0.911 and 0.882 for hand motion, respectively. This controlled experiment provides evidence that the pre-encoder discrepancy responds systematically to real foreground motion in the absence of camera movement. Since the pre-encoder discrepancy serves as the input to the discrepancy-state encoder, its motion-related responses can potentially be encoded into the learned discrepancy states, providing a possible pathway for dynamics-aware information to contribute to subsequent temporal prediction.
R3-Table 5: Mean image-plane speeds of moving objects and hands across the low-, medium-, and high-speed groups, measured through manual annotations under a stationary camera.
| Speed Group | Moving Objects (pixels/frame) | Moving Hands (pixels/frame) |
|---|---|---|
| Low | 8.82 | 10.50 |
| Medium | 21.59 | 15.05 |
| High | 41.36 | 38.33 |
Regarding the learned discrepancy states, Tab. 4 of the main text and Tab. 13 of the revised appendix demonstrate that predicting their future evolution provides additional benefits for hand trajectory forecasting. However, we acknowledge that the existing experiments do not directly establish how much of the motion-sensitive information identified in the pre-encoder discrepancy is retained by the learned states or contributes to the forecasting improvements. In particular, the actual predictor and DACM may also encode and exploit other restoration-induced responses. We have clarified this distinction in Appendix A. 13 and moderated our mechanistic interpretation accordingly. Directly connecting controlled motion variations to learned-state responses and their subsequent effects on forecasting mentioned by the reviewer is an interesting and important direction and we will explore in future work.
Response to Reviewer VPgX (Part 1 / 4)
We sincerely thank the reviewer for the thoughtful and constructive feedback, and for recognizing the value of deblurring discrepancy as an auxiliary predictive representation, as well as the effectiveness of our full model and discrepancy-aware distillation strategy. We greatly appreciate the reviewer’s insightful suggestions, which have helped us further improve the quality of our work.
We have conducted additional experiments, provided further quantitative analyses, and clarified the corresponding discussions in response to the reviewer’s comments. Our point-by-point responses are provided below.
Q1: What existing annotation-quality records are available for EgoDyn: the fraction of joints/clips corrected or discarded, the criteria used, and any localization-error audit independent of the initial HaMeR predictions? Please distinguish blur levels where records permit, and clarify whether severe-blur clips disproportionately failed annotation. If no independent audit exists, an explicit statement would help establish the evidence boundary; a new annotation campaign is not requested for the rebuttal.
A1: We thank the reviewer for the valuable comment on the annotation quality records of EgoDyn. As described in Appendix A. 3, EgoDyn was collected from eight participants using a head-mounted Intel RealSense D435i at 30 FPS. We first apply NAFNet to deblur the recorded videos and then use HaMeR to obtain frame-wise hand-joint annotations. We manually inspect the resulting trajectories and correct noticeable localization errors and hand identity inconsistencies. Clips with severe ambiguities that cannot be confidently resolved through manual correction are discarded.
Following the reviewer's suggestion, we provide additional annotation quality statistics across the low-, medium-, and high-blur subsets defined in Appendix A.5. Specifically, we report the number of reviewed clips, the percentages of clips and joint annotations requiring manual correction, the clip rejection rates, and the mean correction magnitudes of manually corrected joints. The latter is defined as the average normalized Euclidean displacement between the initial HaMeR predictions and the manually corrected coordinates.
As shown in Tab. 7 of the revised appendix (also provided in R3-Tab. 1), the correction rates increase with blur severity. The percentages of corrected clips and corrected joints increase from 8.7% and 2.8% under low blur to 15.6% and 5.8% under high blur, respectively. The clip rejection rate also increases from 1.0% under low blur to 4.8% under high blur, indicating that severely blurred clips are more likely to contain ambiguities that cannot be reliably resolved through manual correction. Meanwhile, the mean correction magnitude increases from 0.0108 to 0.0203, suggesting that the localization adjustments required for more severely blurred observations are generally larger. We attribute these trends to the partial loss of fine-grained hand appearance and joint boundary information under stronger motion blur, which makes accurate joint localization more challenging for HaMeR. Moreover, rapid hand movements can cause larger inter-frame positional variations and partial finger overlap. This increases the difficulty of maintaining temporally consistent hand identities and joint correspondences, thus requiring more manual intervention.
We acknowledge that no independent localization-error audit against separately annotated ground truth was conducted as part of the original annotation pipeline. Such an audit would require an additional round of independent frame-wise and fully manual annotation, involving substantial human effort beyond the original annotation and quality-control process. Therefore, the reported correction magnitudes quantify manual adjustments to the initial HaMeR predictions.
Reviewed clips refer to all clips manually inspected during the annotation quality-control process, including those subsequently discarded.