- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents StreamHaR, which it describes as the first method to reconstruct hands in a fixed metric world frame from egocentric video in a fully online setting. Existing world-space methods are offline, relying on global bundle adjustment or clip-level optimization, which does not suit AR, teleoperation, or robot imitation learning. StreamHaR decomposes the problem into three estimators: (1) wrist translation from per-frame closed-form least squares on 2D/3D keypoints from an anchor-loss-trained keypoint encoder; (2) finger articulation from a separate pose encoder followed by a GRU refiner; and (3) camera pose from DPVO-based causal visual odometry, re-anchored to metric scale every 8 frames using MoGe-2 depth.
Evaluation covers HOT3D-Aria, Ego-Exo4D, HOI4D, HO-Cap (HoloLens, zero-shot), and DexYCB. On HOT3D-Aria, StreamHaR gets World error comparable to HaWoR at far lower latency and higher throughput, and the best PA-MPJPE among all compared methods. It ranks first on world-space metrics on Ego-Exo4D and substantially lowers MPJPE/PA-MPJPE vs. HaWoR zero-shot on HO-Cap. HaWoR remains better on segment-level smoothness and alignment (W-/WA-MPJPE, RTE, Accel).
Critical strengths
- Practically relevant problem formulation. Fully online world-space hand reconstruction with explicit requirements tied to AR, teleoperation, and robot learning.
- Efficiency advantage. Latency drops from 16.39 s (HaWoR) and 245.40 s (Dyn-HaMR) to 0.06 s, with comparable World error.
- Consistently best articulation accuracy, beating both offline HaWoR and per-frame WiLoR (HOT3D PA-MPJPE 4.81 vs. 5.76 and 4.99 mm), with a larger margin on HO-Cap (0.95→2.45 mm).
- Zero-shot device transfer to HoloLens (HO-Cap MPJPE 17.57 vs. 24.31 mm) and more stable qualitative results on factory video with gloves and blur.
Critical weaknesses
- Clear cost vs. offline methods on trajectory quality. On HOT3D-Aria (Table 1), W-MPJPE is 56.92 vs. 44.50 mm, RTE 9.64% vs. 5.94%, and Accel 12.07 vs. 10.55.
- HOI4D results undercut the "long sequences" framing. G-/GA-MPJPE of 89.04/33.09 is worse than HaWoR (76.08/29.12), Dyn-HaMR (84.90/28.54), and WiLoR, suggesting scale anchoring does not fully stop drift.
- No long-horizon drift evaluation. The camera is incremental odometry with scale re-anchoring only, yet clip-length characterization and drift-versus-time curves are absent.
- Unclear train/test separation. Only HO-Cap is called zero-shot, implying training on the other benchmarks' splits, while HaWoR/Dyn-HaMR use released models whose data overlap with the test sequences is not stated.
- Limited deployment evidence. Speed is measured only on an H200, and 15.83 FPS is below typical real-time video rates; no headset or embedded hardware results.
Questions for authors
- What exact train/test splits were used for HOT3D-Aria, Ego-Exo4D, and HOI4D, and do the training data of HaWoR/Dyn-HaMR overlap with these test sequences? Are all methods zero-shot on benchmarks other than HO-Cap?
- Can you provide error-versus-clip-length (drift) curves, especially for minute-scale sequences?
- Why is StreamHaR worse than offline baselines on HOI4D G-/GA-MPJPE, and what does this imply for causal odometry on long sequences?
- What throughput and latency are achievable on headset-class or embedded hardware?
- What is the total number of parameters in the proposed model?
- How does the sequence length of HOI4D compare with that of other datasets? Since longer videos may lead to error accumulation, would additional metrics for long-term drift or temporal consistency be necessary?
- Can the method be evaluated on lower-end hardware than the NVIDIA H200? Reporting runtime and memory usage would help assess its practical applicability.
- How does the method handle frames where hands are not visible? Does it skip predictions or infer hand poses despite missing visual evidence? This choice could significantly affect the quantitative results.
4 comments
Official Comment by Authors (2/2)
W4/Q1: Train/test splits and overlap.
W4. "Unclear train/test separation. Only HO-Cap is called zero-shot, implying training on the other benchmarks' splits, while HaWoR/Dyn-HaMR use released models whose data overlap with the test sequences is not stated."
Q1. "What exact train/test splits were used for HOT3D-Aria, Ego-Exo4D, and HOI4D, and do the training data of HaWoR/Dyn-HaMR overlap with these test sequences? Are all methods zero-shot on benchmarks other than HO-Cap?"
Ego-Exo4D, HOI4D and HO-Cap are not in the training set of StreamHaR, HaWoR or WiLoR. Dyn-HaMR is an optimisation on top of HaMeR, whose training data contains none of the four benchmarks. We use the Ego-Exo4D validation set (65 takes) and the 100-sequence HOI4D protocol of Dyn-HaMR. All methods are zero-shot on these three benchmarks. We called only HO-Cap zero-shot because it uses a different recording device than the datasets in the training data. This is indeed misleading and we will correct this in the revision.
On HOT3D-Aria, we use participants P0003 and P0014 (313 test clips) for evaluation and the other seven participants (1,203 clips) for training, with no participant shared. This setting is used for StreamHaR, HaWoR and WiLoR. HaWoR's HOT3D partition is not released, so an overlap with our test participants cannot be excluded. Such an overlap, however, would favour HaWoR.
W5/Q4/Q7: Deployment and lower-end hardware.
W5. "Limited deployment evidence. Speed is measured only on an H200, and 15.83 FPS is below typical real-time video rates; no headset or embedded hardware results."
Q4. "What throughput and latency are achievable on headset-class or embedded hardware?"
Q7. "Can the method be evaluated on lower-end hardware than the NVIDIA H200? Reporting runtime and memory usage would help assess its practical applicability."
Thank you for raising this. We profiled the runtime on an RTX 4090 GPU with i9-12900KF CPU.
| RTX 4090 | |
|---|---|
| Hand stages per frame (ms) | 45.3 |
| Camera odometer per frame (ms) | 15.3 |
| End-to-end latency per frame (ms) | 60.6 |
| Frames per second | 16.5 |
| Peak GPU memory (GB) | 8.5 |
On an RTX 4090, the approach runs at 16.5 FPS with 8.5 GB memory. The runtime is stable and was measured over 3,000 frames. The memory is constant over time as well. Time of individual components: ViT-H encoders (32.5 ms), camera odometer (15.3 ms), depth model (7.2 ms), hand detector (4.3 ms), wrist solver and refiner (2.1 ms), where some components run in parallel. The GPU is only 65$\%$ utilised and optimizing the implementation to reduce end-to-end latency is possible.
For datasets like HOI4D, this already corresponds to the frame-rate of the stream (15 FPS). The stream is thus processed in real-time with a mean latency of 72 ms on an RTX 4090. For 30 FPS streams, the camera odometer processes every frame and the hand pose and wrist estimator process every 4th frame. We measured runtime and accuracy under this setting on HOT3D-Aria:
| Hand stages run on | FPS (RTX 4090) | World (updated frames) | World increase (all frames) |
|---|---|---|---|
| every frame | 16.5 | 30.69 | 0 |
| every 2nd frame | 24.7 | 30.95 | |
| every 4th frame | 31.8 | 31.39 |
With the hand stages processing only every fourth frame, our approach exceeds 30 FPS on an RTX 4090 with a latency of 54 ms on average and 77 ms at the 95th percentile. The latency does not grow over time. Compared to processing very frame, this increases the World error only by 0.7 mm on the frames where the hand pose has been updated. When keeping the last estimate of the hand for 4 frames until the next update, the World error increases by 4.3 mm on all frames. We did not run it on edge devices, but optimizing it for edge devices is an interesting future direction.
Q5. Total number of parameters.
Q5. "What is the total number of parameters in the proposed model?"
The model has 1.65 B parameters: two ViT-H encoders with 0.64 B each, the depth model for the scale anchor with 0.33 B, and 0.04 B for the hand detector, pose head, refiner and camera odometer.
Q8: Frames where hands are not visible.
Q8. "How does the method handle frames where hands are not visible? Does it skip predictions or infer hand poses despite missing visual evidence? This choice could significantly affect the quantitative results."
The supplementary video shows this case: the hand leaves the view at 0:05 and StreamHaR reconstructs it again when it returns at 0:08. When a hand is not detected, we keep it 45 frames (1.5 s) at the last detected position. We evaluate on all frames with an annotated hand. If we would only consider frames where hands are detected, our World error would decrease from 30.69 to 22.35 mm on HOT3D-Aria. If use the annotated bounding boxes of the benchmark, the World error would be 26.08.
Thank you to the authors for the detailed and constructive rebuttal. I appreciate the additional experiments and clarifications, which address several of my concerns.
The rebuttal has addressed or clarified most of my concerns, and I appreciate the authors' willingness to revise potentially misleading wording. My main remaining concern is that trajectory drift and long-term world-frame stability remain limitations of the fully causal setting, particularly under larger camera motion, and deployment on true headset-class or edge hardware is still left for future work. Nevertheless, I view these as limitations that can be appropriately acknowledged rather than fundamental soundness issues.
Official Comment by Authors
We thank the reviewer for confirming that our response clarified several points. We will acknowledge that avoiding camera drift in an online setting remains challenging and that the deployment on edge hardware is future work.
Official Comment by Authors (1/2)
W1: Cost against offline methods on trajectory quality.
Please note that offline methods post-process the entire video and are not applicable for many applications. Since they have access to the entire video in contrast to online methods, they are expected to estimate the trajectory more accurately. For some metrics (World, PA), our online approach even outperforms offline methods. Please also note that in Table 9 of the suppl. material, we evaluate a variant with a 1s window. This variant outperforms offline methods even for WA-MPJPE (21.88) and Accel (6.93).
W2/Q3/Q6: HOI4D G-/GA-MPJPE, causal odometry, sequence lengths.
On this benchmark, every method, including ours, are evaluated with the camera parameters provided by the benchmark. Our odometry and scale anchoring are not used on this benchmark, so the gap is not due to camera drift. We will clarify this. A short look-ahead recovers about half of the gap: with the one-second window (Table 9 in suppl. material), G-MPJPE drops from 89.04 to 82.28 and GA-MPJPE from 33.09 to 30.71. The sequence lengths are for HOT3D-Aria 5s, HOI4D 20s, HO-Cap 10 to 25s, and Ego-Exo4D minutes. We agree that 20s are not long and remove that wording. Drift over time is reported in W3 / Q2.
W3/Q2: Long-horizon drift.
Hand error against the number of frames processed (Ego-Exo4D). Median G-MPJPE per 128-frame segment grouped by the position of the segment in the video:
The error does not grow with the frame number. The longest clip has 2,128 frames, about 70s at 30 FPS.
Camera error against the frame number (HOT3D-Aria). We sorted the 313 test clips of HOT3D-Aria based on the accumulated camera motion and report the median position error in mm for various frame intervals:
Our approach does not prevent drift and the amount of drift is larger for larger head motion: 20.8 mm after 150 frames for the clips with leat head motion and 55.5 mm for the clips with most head motion. Re-anchoring bounds the scale, but not the trajectory shape. We will discuss this limitation and add the tables as plots.