acceptodds
Under review as a conference paper at ICLR 2027

StreamHaR: Streaming World-Grounded Hand Reconstruction from Egocentric Videos

14 views

Abstract

Augmented reality, teleoperation and robot learning from demonstration require the wearer's hands reconstructed in a metric world frame as the egocentric video arrives. Existing world-space methods are offline, relying on future frames and global optimization, limiting their interactive applicability. To address this, we present StreamHaR, a fully online method that emits a metric, world-space hand at every frame with a state of constant size, built to be causal, faithful, general and efficient. StreamHaR splits the world-space hand into three problems with three estimators: the wrist is measured by a per-frame Perspective-n-Point solve on projection-consistent keypoints, the articulation is recognized by an encoder, and the camera comes from a causal odometer that anchors its metric scale from monocular depth, so the trajectory is metric the moment it is emitted, with no bundle adjustment and no calibration. With zero look-ahead and its own detector, StreamHaR reaches the world-space accuracy of the offline state of the art at a lower articulation error and transfers zero-shot to unseen headsets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D hand reconstruction, streaming

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mMucOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents StreamHaR, which it describes as the first method to reconstruct hands in a fixed metric world frame from egocentric video in a fully online setting. Existing world-space methods are offline, relying on global bundle adjustment or clip-level optimization, which does not suit AR, teleoperation, or robot imitation learning. StreamHaR decomposes the problem into three estimators: (1) wrist translation from per-frame closed-form least squares on 2D/3D keypoints from an anchor-loss-trained keypoint encoder; (2) finger articulation from a separate pose encoder followed by a GRU refiner; and (3) camera pose from DPVO-based causal visual odometry, re-anchored to metric scale every 8 frames using MoGe-2 depth.

Evaluation covers HOT3D-Aria, Ego-Exo4D, HOI4D, HO-Cap (HoloLens, zero-shot), and DexYCB. On HOT3D-Aria, StreamHaR gets World error comparable to HaWoR at far lower latency and higher throughput, and the best PA-MPJPE among all compared methods. It ranks first on world-space metrics on Ego-Exo4D and substantially lowers MPJPE/PA-MPJPE vs. HaWoR zero-shot on HO-Cap. HaWoR remains better on segment-level smoothness and alignment (W-/WA-MPJPE, RTE, Accel).

Critical strengths
  • Practically relevant problem formulation. Fully online world-space hand reconstruction with explicit requirements tied to AR, teleoperation, and robot learning.
  • Efficiency advantage. Latency drops from 16.39 s (HaWoR) and 245.40 s (Dyn-HaMR) to 0.06 s, with comparable World error.
  • Consistently best articulation accuracy, beating both offline HaWoR and per-frame WiLoR (HOT3D PA-MPJPE 4.81 vs. 5.76 and 4.99 mm), with a larger margin on HO-Cap (0.95→2.45 mm).
  • Zero-shot device transfer to HoloLens (HO-Cap MPJPE 17.57 vs. 24.31 mm) and more stable qualitative results on factory video with gloves and blur.
Critical weaknesses
  • Clear cost vs. offline methods on trajectory quality. On HOT3D-Aria (Table 1), W-MPJPE is 56.92 vs. 44.50 mm, RTE 9.64% vs. 5.94%, and Accel 12.07 vs. 10.55.
  • HOI4D results undercut the "long sequences" framing. G-/GA-MPJPE of 89.04/33.09 is worse than HaWoR (76.08/29.12), Dyn-HaMR (84.90/28.54), and WiLoR, suggesting scale anchoring does not fully stop drift.
  • No long-horizon drift evaluation. The camera is incremental odometry with scale re-anchoring only, yet clip-length characterization and drift-versus-time curves are absent.
  • Unclear train/test separation. Only HO-Cap is called zero-shot, implying training on the other benchmarks' splits, while HaWoR/Dyn-HaMR use released models whose data overlap with the test sequences is not stated.
  • Limited deployment evidence. Speed is measured only on an H200, and 15.83 FPS is below typical real-time video rates; no headset or embedded hardware results.
Questions for authors
  1. What exact train/test splits were used for HOT3D-Aria, Ego-Exo4D, and HOI4D, and do the training data of HaWoR/Dyn-HaMR overlap with these test sequences? Are all methods zero-shot on benchmarks other than HO-Cap?
  2. Can you provide error-versus-clip-length (drift) curves, especially for minute-scale sequences?
  3. Why is StreamHaR worse than offline baselines on HOI4D G-/GA-MPJPE, and what does this imply for causal odometry on long sequences?
  4. What throughput and latency are achievable on headset-class or embedded hardware?
  5. What is the total number of parameters in the proposed model?
  6. How does the sequence length of HOI4D compare with that of other datasets? Since longer videos may lead to error accumulation, would additional metrics for long-term drift or temporal consistency be necessary?
  7. Can the method be evaluated on lower-end hardware than the NVIDIA H200? Reporting runtime and memory usage would help assess its practical applicability.
  8. How does the method handle frames where hands are not visible? Does it skip predictions or infer hand poses despite missing visual evidence? This choice could significantly affect the quantitative results.
4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

W1: Cost against offline methods on trajectory quality.

W1. "Clear cost vs. offline methods on trajectory quality. On HOT3D-Aria (Table 1), W-MPJPE is 56.92 vs. 44.50 mm, RTE 9.64$\%$ vs. 5.94$\%$, and Accel 12.07 vs. 10.55."

Please note that offline methods post-process the entire video and are not applicable for many applications. Since they have access to the entire video in contrast to online methods, they are expected to estimate the trajectory more accurately. For some metrics (World, PA), our online approach even outperforms offline methods. Please also note that in Table 9 of the suppl. material, we evaluate a variant with a 1s window. This variant outperforms offline methods even for WA-MPJPE (21.88) and Accel (6.93).

W2/Q3/Q6: HOI4D G-/GA-MPJPE, causal odometry, sequence lengths.

W2. "HOI4D results undercut the 'long sequences' framing. G-/GA-MPJPE of 89.04/33.09 is worse than HaWoR (76.08/29.12), Dyn-HaMR (84.90/28.54), and WiLoR, suggesting scale anchoring does not fully stop drift."

Q3. "Why is StreamHaR worse than offline baselines on HOI4D G-/GA-MPJPE, and what does this imply for causal odometry on long sequences?"

Q6. "How does the sequence length of HOI4D compare with that of other datasets? Since longer videos may lead to error accumulation, would additional metrics for long-term drift or temporal consistency be necessary?"

On this benchmark, every method, including ours, are evaluated with the camera parameters provided by the benchmark. Our odometry and scale anchoring are not used on this benchmark, so the gap is not due to camera drift. We will clarify this. A short look-ahead recovers about half of the gap: with the one-second window (Table 9 in suppl. material), G-MPJPE drops from 89.04 to 82.28 and GA-MPJPE from 33.09 to 30.71. The sequence lengths are for HOT3D-Aria 5s, HOI4D 20s, HO-Cap 10 to 25s, and Ego-Exo4D minutes. We agree that 20s are not long and remove that wording. Drift over time is reported in W3 / Q2.

W3/Q2: Long-horizon drift.

W3. "No long-horizon drift evaluation. The camera is incremental odometry with scale re-anchoring only, yet clip-length characterization and drift-versus-time curves are absent."

Q2. "Can you provide error-versus-clip-length (drift) curves, especially for minute-scale sequences?"

Hand error against the number of frames processed (Ego-Exo4D). Median G-MPJPE per 128-frame segment grouped by the position of the segment in the video:

Frames since start0-150150-300300-450450-600600-900900-18001800-2128
StreamHaR30.529.922.526.128.631.032.1

The error does not grow with the frame number. The longest clip has 2,128 frames, about 70s at 30 FPS.

Camera error against the frame number (HOT3D-Aria). We sorted the 313 test clips of HOT3D-Aria based on the accumulated camera motion and report the median position error in mm for various frame intervals:

ClipsCamera travelFrames 0-1430-4460-7490-104135-149
Least head motion (104)0.23 m4.66.611.713.520.8
Medium (104)0.39 m6.611.325.040.543.8
Most head motion (105)0.68 m12.016.835.239.855.5
All (313)6.79.420.628.334.3

Our approach does not prevent drift and the amount of drift is larger for larger head motion: 20.8 mm after 150 frames for the clips with leat head motion and 55.5 mm for the clips with most head motion. Re-anchoring bounds the scale, but not the trajectory shape. We will discuss this limitation and add the tables as plots.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W4/Q1: Train/test splits and overlap.

W4. "Unclear train/test separation. Only HO-Cap is called zero-shot, implying training on the other benchmarks' splits, while HaWoR/Dyn-HaMR use released models whose data overlap with the test sequences is not stated."

Q1. "What exact train/test splits were used for HOT3D-Aria, Ego-Exo4D, and HOI4D, and do the training data of HaWoR/Dyn-HaMR overlap with these test sequences? Are all methods zero-shot on benchmarks other than HO-Cap?"

Ego-Exo4D, HOI4D and HO-Cap are not in the training set of StreamHaR, HaWoR or WiLoR. Dyn-HaMR is an optimisation on top of HaMeR, whose training data contains none of the four benchmarks. We use the Ego-Exo4D validation set (65 takes) and the 100-sequence HOI4D protocol of Dyn-HaMR. All methods are zero-shot on these three benchmarks. We called only HO-Cap zero-shot because it uses a different recording device than the datasets in the training data. This is indeed misleading and we will correct this in the revision.

On HOT3D-Aria, we use participants P0003 and P0014 (313 test clips) for evaluation and the other seven participants (1,203 clips) for training, with no participant shared. This setting is used for StreamHaR, HaWoR and WiLoR. HaWoR's HOT3D partition is not released, so an overlap with our test participants cannot be excluded. Such an overlap, however, would favour HaWoR.

W5/Q4/Q7: Deployment and lower-end hardware.

W5. "Limited deployment evidence. Speed is measured only on an H200, and 15.83 FPS is below typical real-time video rates; no headset or embedded hardware results."

Q4. "What throughput and latency are achievable on headset-class or embedded hardware?"

Q7. "Can the method be evaluated on lower-end hardware than the NVIDIA H200? Reporting runtime and memory usage would help assess its practical applicability."

Thank you for raising this. We profiled the runtime on an RTX 4090 GPU with i9-12900KF CPU.

RTX 4090
Hand stages per frame (ms)45.3
Camera odometer per frame (ms)15.3
End-to-end latency per frame (ms)60.6
Frames per second16.5
Peak GPU memory (GB)8.5

On an RTX 4090, the approach runs at 16.5 FPS with 8.5 GB memory. The runtime is stable and was measured over 3,000 frames. The memory is constant over time as well. Time of individual components: ViT-H encoders (32.5 ms), camera odometer (15.3 ms), depth model (7.2 ms), hand detector (4.3 ms), wrist solver and refiner (2.1 ms), where some components run in parallel. The GPU is only 65$\%$ utilised and optimizing the implementation to reduce end-to-end latency is possible.

For datasets like HOI4D, this already corresponds to the frame-rate of the stream (15 FPS). The stream is thus processed in real-time with a mean latency of 72 ms on an RTX 4090. For 30 FPS streams, the camera odometer processes every frame and the hand pose and wrist estimator process every 4th frame. We measured runtime and accuracy under this setting on HOT3D-Aria:

Hand stages run onFPS (RTX 4090)World ↓\downarrow (updated frames)World increase ↓\downarrow (all frames)
every frame16.530.690
every 2nd frame24.730.95+0.7+0.7
every 4th frame31.831.39+4.3+4.3

With the hand stages processing only every fourth frame, our approach exceeds 30 FPS on an RTX 4090 with a latency of 54 ms on average and 77 ms at the 95th percentile. The latency does not grow over time. Compared to processing very frame, this increases the World error only by 0.7 mm on the frames where the hand pose has been updated. When keeping the last estimate of the hand for 4 frames until the next update, the World error increases by 4.3 mm on all frames. We did not run it on edge devices, but optimizing it for edge devices is an interesting future direction.

Q5. Total number of parameters.

Q5. "What is the total number of parameters in the proposed model?"

The model has 1.65 B parameters: two ViT-H encoders with 0.64 B each, the depth model for the scale anchor with 0.33 B, and 0.04 B for the hand detector, pose head, refiner and camera odometer.

Q8: Frames where hands are not visible.

Q8. "How does the method handle frames where hands are not visible? Does it skip predictions or infer hand poses despite missing visual evidence? This choice could significantly affect the quantitative results."

The supplementary video shows this case: the hand leaves the view at 0:05 and StreamHaR reconstructs it again when it returns at 0:08. When a hand is not detected, we keep it 45 frames (1.5 s) at the last detected position. We evaluate on all frames with an annotated hand. If we would only consider frames where hands are detected, our World error would decrease from 30.69 to 22.35 mm on HOT3D-Aria. If use the annotated bounding boxes of the benchmark, the World error would be 26.08.

Thank you to the authors for the detailed and constructive rebuttal. I appreciate the additional experiments and clarifications, which address several of my concerns.

The rebuttal has addressed or clarified most of my concerns, and I appreciate the authors' willingness to revise potentially misleading wording. My main remaining concern is that trajectory drift and long-term world-frame stability remain limitations of the fully causal setting, particularly under larger camera motion, and deployment on true headset-class or edge hardware is still left for future work. Nevertheless, I view these as limitations that can be appropriately acknowledged rather than fundamental soundness issues.

AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for confirming that our response clarified several points. We will acknowledge that avoiding camera drift in an online setting remains challenging and that the deployment on edge hardware is future work.

Reviewer_5NMXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an online world-space hand reconstruction method from egocentric video. For hand articulation, it uses a pose encoder followed by a GRU-based refiner. For wrist translation, a separate keypoint encoder predicts 2D and wrist-relative 3D keypoints, followed by a PnP solver. Together, these produce the hand reconstruction in camera coordinates. For camera pose, the method uses off-the-shelf DPVO and corrects its scale using monocular metric-depth predictions every eight frames. The camera trajectory then lifts the hands into world coordinates.

Critical strengths

The motivation is clear, and the paper is generally easy to follow. The figures help illustrate the pipeline and qualitative results. The online setting is useful for interactive applications.

Critical weaknesses
  1. The contribution beyond a simpler online pipeline is not sufficiently established. A comparison with WiLoR combined with the same camera estimator and basic causal filtering is missing. The articulation-refiner ablation shows only marginal accuracy improvements, and the advantage of PnP over direct translation prediction is not experimentally demonstrated. Without these comparisons, I am not yet convinced that the proposed hand-reconstruction components provide sufficient benefit.

  2. The suitability for the intended interactive applications remains unclear. The world-space reconstruction appears quite jittery, particularly at 0-5 s of the supplementary video. The method also reaches only 15.83 FPS on an H200, while sustained streaming latency and consumer-GPU performance remain unclear. These concerns weaken the evidence supporting practical use in AR and teleoperation.

Questions for authors
  1. The jitter is particularly noticeable in the world-space visualization, rather than the image overlays. I wonder whether the scale correction every eight frames contributes to this behavior. Although Table 1 reports acceleration error, the placement ablations do not show how scale anchoring affects temporal stability. Could the authors report camera and wrist stability separately, particularly around the correction frames, and evaluate sensitivity to the correction interval and damping? Have other online camera estimators incorporating metric-depth priors been considered?

  2. I would like to see a baseline combining WiLoR with the same metric-anchored DPVO and basic causal filtering, under matched training and evaluation conditions. My major concern is that this simpler pipeline might already achieve competitive results. This comparison would help clarify how much the proposed hand-reconstruction components contribute. In particular, removing the articulation refiner barely changes the reported HOT3D accuracy, so its benefit is not yet clear to me.

  3. Why does the keypoint encoder predict both 2D and 3D keypoints and then use PnP, rather than also predicting wrist translation directly? I understand the geometric motivation, but a per-frame PnP solve could propagate keypoint noise into the translation estimate. An alternative would be a translation head conditioned on camera intrinsics and crop information, followed by temporal refinement similar to the articulation branch. Could the authors discuss the trade-offs between these designs and clarify how keypoint noise and temporal stability are handled in the current wrist-translation estimation?

  4. At the reported throughput of 15.83 FPS, the method would not keep up with a 30-FPS input stream without skipping frames or accumulating delay. Could the authors clarify how this is handled and report sustained end-to-end latency? Performance on a more affordable consumer GPU(e.g. RTX 3090, 4090), together with GPU-memory usage and a runtime breakdown, would make the efficiency claims more informative. Edge-device results would also be useful if feasible.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

W1/Q2: Comparison with WiLoR, the same camera and causal filtering, the articulation refiner and PnP against direct translation.

W1. "A comparison with WiLoR combined with the same camera estimator and basic causal filtering is missing. The articulation-refiner ablation shows only marginal accuracy improvements, and the advantage of PnP over direct translation prediction is not experimentally demonstrated."

Q2. "I would like to see a baseline combining WiLoR with the same metric-anchored DPVO and basic causal filtering, under matched training and evaluation conditions. My major concern is that this simpler pipeline might already achieve competitive results."

As suggested, we run both comparisons under the same conditions on HOT3D-Aria. We will the tables to the paper or appendix.

A. WiLoR with the same camera estimator and basic causal filtering.

Method (HOT3D-Aria)World ↓\downarrowW ↓\downarrowAccel ↓\downarrow
WiLoR + our camera31.6157.1512.27
StreamHaR30.6956.9212.07
WiLoR + our camera + causal filter30.6658.107.47
StreamHaR + causal filter30.9667.676.84

WiLoR with our camera estimator has a higher World error and higher W-MPJPE. As causal filter, we used an exponential moving average on the world-frame wrist position (with zero look-ahead) as in OnlineHMR. The causal filter reduces, as expected, the acceleration error for both methods, while the error is still lower for our approach. The filter, however, increases W-MPJPE.

We also include a comparison on HO-Cap and will add it to Table 3 of the paper:

HO-CapMPJPE ↓\downarrowPA ↓\downarrow
WiLoR, per frame20.357.19
StreamHaR17.576.23

StreamHaR also outperforms WiLoR on this dataset.

B. The articulation refiner and PnP against direct translation.

The articulation refiner lowers PA-MPJPE on each benchmark, but we agree that the improvements are marginal. We will revise this sentence. PnP is more accurate compared to direct translation:

Wrist placement (HOT3D-Aria)World ↓\downarrow
Direct translation34.98
PnP30.69
AuthorsOpenReview ↗

Official Comment by Authors (2/3)

W2/Q1: Jitter in the world-space view, scale correction every eight frames, camera and wrist stability, interval and damping, other online camera estimators.

W2. "The world-space reconstruction appears quite jittery, particularly at 0-5 s of the supplementary video."

Q1. "The jitter is particularly noticeable in the world-space visualization, rather than the image overlays. I wonder whether the scale correction every eight frames contributes to this behavior. ... Could the authors report camera and wrist stability separately, particularly around the correction frames, and evaluate sensitivity to the correction interval and damping? Have other online camera estimators incorporating metric-depth priors been considered?"

Thank you for this question. We measured the camera and the wrist separately around the correction frames and for different intervals on HOT3D-Aria. The table will be included in the appendix.

Correction intervalWorld ↓\downarrowAccel ↓\downarrowCamera acceleration correction frames / other frames
4 frames30.4512.160.80 / 1.11
8 frames (paper)30.6912.070.83 / 0.91
16 frames32.7311.650.78 / 0.81
32 frames38.1510.980.74 / 0.70

The correction is applied inside the odometer's state. The current window is rescaled and the optimization continues from there, so the trajectory is not displaced. While the camera trajectory is smooth, the wrist position shows some jitter since it is solved independently on every frame. This is visible in the world-space view. We additionally measure the median wrist error change, from one frame to the next. On predictions of main paper Table 1, the error is 5.3 mm of which the camera accounts for 1.1 mm. About 80$\%$ of it lies along the viewing direction, where it moves the projection by only about 3 pixels in the 1024-pixel image. This is why the overlays look stable.

The interval controls scale accuracy, but has less impact on smoothness. A correction interval of 32 frames lowers Acceleration but raises World error substantially. Damping has little effect. Without damping, the World error (using 8 frames) is 30.80.

Other online camera estimators with metric priors. A single metric-depth scale from the first frame, as in HaWoR, leaves the scale uncorrected for the rest of the clip (row "one anchor at start" of Table 5 in main paper: 34.05 against 30.69). We tried the headset's own visual-inertial odometry, which is not always available. On HOT3D-Aria, it reaches 27.54 mm World error and is the better choice on such a device.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Q3: Why 2D and 3D keypoints with PnP rather than a direct translation head; keypoint noise and temporal stability.

Q3. "Why does the keypoint encoder predict both 2D and 3D keypoints and then use PnP, rather than also predicting wrist translation directly? ... a per-frame PnP solve could propagate keypoint noise into the translation estimate. An alternative would be a translation head conditioned on camera intrinsics and crop information, followed by temporal refinement similar to the articulation branch. Could the authors discuss the trade-offs between these designs and clarify how keypoint noise and temporal stability are handled in the current wrist-translation estimation?"

Thank you for this question. The measured comparison is in our reply to W1 / Q2: PnP is more accurate than predicting wrist translation directly. Here we discuss the design.

Why both 2D and 3D keypoints? The main paper ablates this in Table 5. PnP needs 3D points and their 2D projections to agree under projection. Our anchor loss trains the keypoint encoder for exactly that, by tying both sets to the same ground-truth joints. The row "no anchor loss" removes it and runs PnP on the keypoints of the pose encoder instead. In this case, the World error rises from 30.69 to 37.99 mm and W-MPJPE from 56.92 to 73.76 mm. We will improve the discussion and caption of Table 5 to make this more clear.

Keypoint noise and temporal stability The zero-delay variant does not smooth it, which results in some jitter due to noisy estimates. Using the causal filter (see reply to W1 / Q2) reduces jitter but increases W-MPJPE. In the supplementary material (Section D), we evaluate a variant that trade-offs accuracy and latency. Using a one-second window for the temporal refinement, reduces acceleration error by about 42$\%$ and lowers World Error by 4.7 mm.

OptionDelayWorld ↓\downarrowW ↓\downarrowAccel ↓\downarrow
StreamHaRnone30.6956.9212.07
+ causal filternone30.9667.676.84
+ 1 s window (Appendix Sec. D)31 frames25.9655.316.93

W2/Q4: FPS, sustained latency, consumer GPUs and memory, edge devices.

W2. "The method also reaches only 15.83 FPS on an H200, while sustained streaming latency and consumer-GPU performance remain unclear."

Q4. "the method would not keep up with a 30-FPS input stream without skipping frames or accumulating delay. Could the authors clarify how this is handled and report sustained end-to-end latency? Performance on a more affordable consumer GPU(e.g. RTX 3090, 4090), together with GPU-memory usage and a runtime breakdown, would make the efficiency claims more informative. Edge-device results would also be useful if feasible."

Thank you for raising this. We profiled the runtime on an RTX 4090 GPU with i9-12900KF CPU.

RTX 4090
Hand stages per frame (ms)45.3
Camera odometer per frame (ms)15.3
End-to-end latency per frame (ms)60.6
Frames per second16.5
Peak GPU memory (GB)8.5

On an RTX 4090, the approach runs at 16.5 FPS with 8.5 GB memory. The runtime is stable and was measured over 3,000 frames. The memory is constant over time as well. Time of individual components: ViT-H encoders (32.5 ms), camera odometer (15.3 ms), depth model (7.2 ms), hand detector (4.3 ms), wrist solver and refiner (2.1 ms), where some components run in parallel. The GPU is only 65$\%$ utilised and optimizing the implementation to reduce end-to-end latency is possible.

For datasets like HOI4D, this already corresponds to the frame-rate of the stream (15 FPS). The stream is thus processed in real-time with a mean latency of 72 ms on an RTX 4090. For 30 FPS streams, the camera odometer processes every frame and the hand pose and wrist estimator process every 4th frame. We measured runtime and accuracy under this setting on HOT3D-Aria:

Hand stages run onFPS (RTX 4090)World ↓\downarrow (updated frames)World increase ↓\downarrow (all frames)
every frame16.530.690
every 2nd frame24.730.95+0.7+0.7
every 4th frame31.831.39+4.3+4.3

With the hand stages processing only every fourth frame, our approach exceeds 30 FPS on an RTX 4090 with a latency of 54 ms on average and 77 ms at the 95th percentile. The latency does not grow over time. Compared to processing very frame, this increases the World error only by 0.7 mm on the frames where the hand pose has been updated. When keeping the last estimate of the hand for 4 frames until the next update, the World error increases by 4.3 mm on all frames. We did not run it on edge devices, but optimizing it for edge devices is an interesting future direction.

Reviewer_XRG7OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents StreamHaR, an online framework for reconstructing 3D hand poses and trajectories in metric world coordinates from egocentric videos. The method includes three components: wrist localization, hand articulation estimation, and camera motion estimation. Specifically, it employs projection-consistent 2D–3D keypoints with a closed-form PnP solver for wrist localization, a recurrent refinement module for MANO-based hand articulation, and causal visual odometry with metric depth anchoring for camera tracking. Experiments across multiple benchmarks demonstrate competitive reconstruction accuracy.

Critical strengths
  1. The paper addresses world-space hand pose estimation from egocentric videos, which is an important and challenging problem.

  2. The paper provides many experimental results across multiple benchmarks, demonstrating competitive performance compared to existing methods. It also provides both quantitative results and qualitative comparisons.

Critical weaknesses
  1. Although the paper includes ablation studies for different components, it is still unclear which part has the largest impact on the final world-space hand reconstruction accuracy. It would be helpful to evaluate the method using ground-truth MANO parameters, wrist positions, or camera poses separately to better understand where the main errors come from. In addition, I would like to see how each branch performs compared to existing state-of-the-art methods, especially in terms of hand pose estimation, wrist localization, and camera pose estimation accuracy.

  2. About more cross-dataset experiments. I suggest providing some cross-dataset experimental results to further evaluate its generalization. For example, training on HOT3D-Aria and testing on Ego-Exo4D or HO-Cap could evaluate how well the method generalizes across different datasets and capture settings.

Questions for authors

see the weakness

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

W1: Largest Impact

W1. "it is still unclear which part has the largest impact on the final world-space hand reconstruction accuracy. It would be helpful to evaluate the method using ground-truth MANO parameters, wrist positions, or camera poses separately to better understand where the main errors come from. In addition, I would like to see how each branch performs compared to existing state-of-the-art methods, especially in terms of hand pose estimation, wrist localization, and camera pose estimation accuracy."

Thank you for this suggestion. We evaluated the method using various ground-truth input and each branch. Both tables will be added to the appendix.

1. Which part has the largest impact: the wrist.

Each row replaces one estimate with its ground truth and keeps everything else predicted (HOT3D-Aria, as in Tab. 1).

SettingWorld ↓\downarrowPA ↓\downarrowW ↓\downarrowRTE \\% ↓\downarrow
StreamHaR30.694.8156.929.64
GT MANO parameters29.431.5352.589.64
GT camera poses27.464.8147.127.40
GT wrist positions14.864.8129.542.78
GT wrist + GT camera9.894.8119.050.00

Ground-truth wrist positions have the biggest impact and halve the World error. Using the ground-truth camera trajectory reduces the error further. This supports the importance of estimating the wrist position accurately.

2. Each branch against the state of the art. We compare each branch on its own: hand pose (PA-MPJPE), wrist position relative to the camera (Ego-Exo4D), and camera trajectory (ATE in mm, the absolute error of the camera positions) as in standard odometry. We report ATE instead of the RTE of Table 1 since RTE depends on both the wrist and the camera.

MethodHand pose, PA ↓\downarrowWrist, camera frame ↓\downarrow (Ego-Exo4D)Camera ATE ↓\downarrow
StreamHaR (online)4.8189.78.31
HaWoR (offline)5.7692.61.27
WiLoR (per frame)4.99170.9n/a
Dyn-HaMR (offline)11.75830.522.6

StreamHaR achieves the best hand pose and wrist localization error. As expected, the camera pose error is higher than for the offline method HaWoR.

W2: Cross-dataset experiments

W2. "training on HOT3D-Aria and testing on Ego-Exo4D or HO-Cap could evaluate how well the method generalizes across different datasets and capture settings"

Thank you for the suggestion. In the main paper, we trained the method on 4 datasets: HOT3D, ARCTIC, DexYCB and HO3D. As suggested, we additionally trained StreamHaR only on HOT3D-Aria and evaluated it on Ego-Exo4D and HO-Cap. The table will be added.

Training DataEgo-Exo4D MPJPE ↓\downarrowEgo-Exo4D PA ↓\downarrowEgo-Exo4D G ↓\downarrowHO-Cap MPJPE ↓\downarrowHO-Cap PA ↓\downarrow
HOT3D-Aria89.9111.8940.9815.836.13
Four datasets (paper)83.8911.9240.3517.576.23

The results show that the approach generalizes very well. Please note that the results in the paper are already cross-dataset evaluations on three benchmarks since Ego-Exo4D, HOI4D and HO-Cap are not in the training set of StreamHaR. We will clarify this.

Thank you for the detailed rebuttal and additional experiments. My concerns have been addressed, and I maintain my original score.

AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the response and for the positive feedback on our work. We are glad that our rebuttal clarified the open points.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank all reviewers for their time and their constructive feedback. We are glad that the reviewers find the problem important and practically relevant (XRG7, mMuc), the paper clear and easy to follow (5NMX), the online setting useful for interactive applications (5NMX), and the evaluation across multiple benchmarks thorough (XRG7), with the best articulation accuracy among the compared methods and zero-shot transfer to an unseen headset (mMuc).

We ran the experiments the reviewers asked for and answer each point in the individual responses below. All new results will be added to the revised paper.

AuthorsOpenReview ↗

Official Comment by Authors

We thank all reviewers for the constructive feedback, and we are grateful for the suggestions that helped us improve the clarity and scope of the paper. We uploaded a revised paper and supplementary material that contain the changes announced in our responses, marked in blue. The method, the main results and the numbering of the existing tables remain unchanged.


  • Table 3: additional WiLoR row (5NMX).
  • Section 4.2: clarified that all methods on HOI4D use the camera poses of the dataset, revised the sentence on the articulation refiner, and clarified the "no anchor loss" row of Table 5 (5NMX, mMuc).
  • Section 5: acknowledges camera drift as a remaining challenge and names edge deployment as future work (mMuc).
  • Appendix A: training and test splits of every benchmark (XRG7, mMuc).
  • Appendix E, Tables 10 to 12: error decomposition with ground-truth inputs, each branch against prior work, and training on HOT3D-Aria only (XRG7).
  • Appendix E, Tables 13 and 14: WiLoR with the camera of StreamHaR and a causal filter, and the interval of the scale correction (5NMX).
  • Appendix E, Figure 9: error against the number of frames processed (mMuc).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.