- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces EgoExplorer, an egocentric dataset of human loco-manipulation with 2,715 recordings (100 h) across ten task templates and five environments, with an emphasis on professional activities such as laboratory and industrial work. The recordings pair head-mounted video with metric whole-body motion and glove-measured finger articulation; recordings made with the Wuji gloves also include touch, and expert-guided step-level text annotations describe task progress. The rig consists of five VIVE Ultimate Trackers (head, both hands, both ankles), a head-mounted GoPro, and one of two glove options: MANUS Metagloves for finger articulation, or Wuji gloves that also record tactile signals. Whole-body motion is recovered from the five tracker poses with the C³ (Calibrate-Complete-Constrain) pipeline. A protocol-guided calibration first fits SMPL-X body shape, metric scale, and the sensor-toanchor transforms once per mounting, so that mounting offsets are not absorbed into the estimated articulation. An AvatarPoser-based transformer prior trained on AMASS then completes the unobserved joints, and layered constraints refine the result for tracker agreement, temporal smoothness, body surface clearance, and ground non-penetration. On a purpose-built Evaluation Subset, C³ reaches 18.0 mm bodypoint error against 64.1 mm for the strongest baseline, a five-anchor fine-tuned AvatarPoser (Tab. 2), and the glove setting outperforms vision-based hand and contact estimation (Tabs. 3 and 4). Preliminary studies use the data for multimodal generation with Cosmos3-Edge and for retargeting recordings to a Unitree G1 tracked by a ZeroWBC-style whole-body controller in simulation.
Critical strengths
- Fills a real gap for humanoid learning. From a robotics perspective, the main obstacle to using existing egocentric datasets for humanoids is that they record what the camera sees, mostly the hands, but not the arm, torso, and leg configuration that a humanoid has to control. Tab. 1 makes this concrete: among the eleven prior datasets listed, only Nymeria provides sensor-grounded full-body motion, and it has neither finger articulation nor touch. Adding five trackers to an egocentric rig is a simple and practical way to obtain metric body motion while keeping the first-person view. Because the recordings are continuous, they also preserve the transitions between approaching targets, adjusting posture, manipulating objects, and relocating (L080–082), which is exactly what locomanipulation policies need to learn. The body motion also rests on a well-motivated calibration-first reconstruction. Separating the fixed body–sensor geometry from motion addresses a real ambiguity of sparse-tracker fitting, where mounting offsets can be absorbed into the estimated articulation. The ablations support this design (L473–477): replacing the calibrated shape and scale with the mean body raises floor violations on the Natural Recordings from 6.0% to 50.9%, a measure that does not depend on reference labels, and omitting mount calibration raises body-point error from 18.0 mm to 95.0–98.1 mm. This is the main reason I lean towards acceptance: I expect the data to be useful for humanoid loco-manipulation research, and this outweighs my concerns below.
- Accurate finger articulation. Measuring the fingers directly with gloves gives far more reliable hand data than estimating them from video. In the matched hand tests (Tab. 3), the glove setting reaches a fingertip error of 9.5 mm against 27.1 mm (HaWoR) and 30.7 mm (HandFlow), a holding RMS of 0.4 mm against 4.5 and 11.3 mm, no missing frames, and 100% kinematic target recovery against 47% and 67%. Regional contact estimation is similarly strong (IoU 0.915 against 0.621 for HACO, Tab. 4). Finger articulation is what egocentric video recovers worst under self-occlusion, and it is what dexterous humanoid hands need, so I see this as one of the most valuable parts of the dataset. This is the second reason I lean towards acceptance: accurate finger data makes the dataset useful for dexterous manipulation research as well.
Critical weaknesses
- Limited accessibility for extending the dataset. The design that makes the data accurate is also what makes it hard to collect more of it. Body and finger data exist only where someone wore the full rig (five trackers, instrumented gloves, and a head camera) and went through the calibration protocol, whose profile stays valid only while the mounts remain fixed (L203–206). The dataset therefore cannot be grown from ordinary egocentric video, and any group that wants to add data needs the same hardware and procedure. The body-tracking part of the rig is far cheaper than studio motion capture (USD 1,184 for the trackers vs. at least USD 8,792 for an eight-camera OptiTrack setup, Tab. 17) and it is portable, so my concern is not cost relative to motion capture but the dependence on dedicated wearable hardware: App. G puts the hardware alone at about USD 5.4k (Wuji) to 8.0k (MANUS) per rig, excluding computers, licenses, and setup labor, and the inside-out trackers depend on suitable visual conditions (L2136–2137). This sits uneasily with the paper’s own motivation that scaling collection across environments “favors lightweight, portable capture systems” (L039– 040). Video-only egocentric collections reach thousands of hours precisely because they need only a camera (3,670 h for Ego4D and 10,000 h for Egocentric-10K in Tab. 1), whereas EgoExplorer has 100 h. This is why I recommend weak rather than clear acceptance: the same rig that makes the data accurate is required to extend it, and the paper does not weigh this trade-off against its own motivation of scalable collection (see Question 1). Minor issues
- Independence of the body accuracy reference. The 71.9% reduction in body-point error (18.0 vs. 64.1 mm, Tab. 2) is measured against tracker-fitted labels that, as App. F.2 states, “share our fitting model, calibration, and constraints, so agreement with them is not independent groundtruth accuracy and can favor related methods” (L1712–1713). The full-body evaluation also comes from a single participant, whose first two repetitions train the participant-specific proposal network (L1714–1716). Please state this limitation in the main text next to the 71.9% claim; an optical motioncapture reference on even a small subset would make the body accuracy claim much more convincing.
- Touch coverage. Measured touch comes only with the Wuji gloves and is available in 41% of regular recordings, while the MANUS gloves cover 58% (App. E, L1388–1391). By the definition in the caption of Tab. 1 (“✓/◦ are full/partial support”), the Touch entry for EgoExplorer should arguably be ◦. Please report per-modality coverage in the main text.
- Number of participants. I could not find how many people were recorded or their demographics. This matters for the diversity of body shapes and motion styles, and for judging how well models trained on the data would generalize.
- Typos. A space is missing in the caption of Fig. 1 (“EgoExplorer.It”), and item (ii) of the contribution list ends with a period instead of a semicolon (L128).
Questions for authors
For the rebuttal (responses to these could change my rating)
- Use beyond the rig (discussion and additional analysis). A discussion of how the value of the recordings can reach beyond the rig, for example (a) by using them as supervision for estimating body and hand state from egocentric video alone, which would carry the benefit over to video-only collections, and (b) by identifying the minimum sensor set. If feasible, C³ results with only the head and wrist trackers would quantify what the ankle trackers add.
- Setup time and participants (clarification). The time needed for donning and calibration per participant, and the number of participants in the dataset (Minor issue 3).
- Reference labels and coverage (requested change). A main-text statement of the reference-label limitation next to the 71.9% claim, and per-modality coverage (touch in 41% of regular recordings) reflected in Tab. 1 (Minor issues 1 and 2). Suggestions for future work (not needed for the rebuttal) • An independent optical motion-capture reference on a small subset, which would make the body accuracy claim much more convincing (Minor issue 1)
2 comments
Response to Reviewer MHcs 2/2
Q1(a). Use for video-only estimation. Video alone is not sufficient for fine manipulation data, and the paper and our new experiments quantify why:
- On the same trials, the glove and touch sensors outperform video-based estimation on finger articulation and contact (Tabs. 3–4).
- Under camera motion and occlusion, video-based hand estimates lose frames (23% missing under occlusion) and stay 24–33 mm from the instructed hand pose, versus 6.8 mm for the glove.
- In the Natural Recordings, at least one wrist is outside the head camera's view in 8–16% of frames, and in 10–21% of frames with measured contact, depending on the camera's field of view. Video-only labels would have to be invented for these frames.
This is exactly what makes the recordings useful beyond the rig: every egocentric frame is paired with sensor-measured body, finger and contact labels, including the lower body and occluded or out-of-view hands, so they can supervise and benchmark video-only estimators where those are weakest.
Q1(b). Minimum sensor set. We ran C³ with the head and wrist trackers only, under the same calibration and constraints, removing the ankle trackers from every term. Ten test recordings, mm:
| Inputs | Body pts↓ | Wrists↓ | Upper untracked↓ | Knees↓ | Lower body↓ |
|---|---|---|---|---|---|
| C³, head + wrists (3 trackers) | 113.6 | 11.1 | 64.5 | 191.6 | 212.0 |
| C³, all five trackers | 18.0 | 8.4 | 20.5 | 26.3 | 17.2 |
With head and wrists only, refinement repairs the arms but leaves the legs unconstrained (212 mm lower-body error), and heel slide during support on the Natural Recordings rises from 4.5 to 31.5 cm/s. Since the legs are mostly outside the egocentric view, five trackers are the minimum for whole-body loco-manipulation data.
We thank the reviewer again. If these changes address the concerns, we would be grateful if the reviewer would consider raising the score.
Response to Reviewer MHcs 1/2
We thank Reviewer MHcs for the review. All discussions and new experiments below will be incorporated into the revised paper.
W1. Dedicated hardware and scalability. Our aim is the lightest, lowest-cost rig that still records whole-body motion, finger articulation and touch, which video-only capture does not provide at the same fidelity (Tabs. 2–4). In short, the rig is lightweight enough for parallel, accompany-the-work collection; its cost keeps falling at volume; and companies are already scaling this kind of wearable collection.
Lightweight and low-cost. All components are off-the-shelf consumer or industrial products. At USD 5.4k–8.0k, a rig costs a fraction of a studio motion-capture volume (Tab. 17) or a humanoid teleoperation station, and the cost falls further at volume.
Throughput. Donning takes a few minutes, the 2.5-min calibration protocol is reused while mounts stay fixed (L203–206), and a collector then records continuously through an 8-hour working day, accompanying the work as in egocentric video collection. No instrumented space is needed.
Scale through parallel deployment. Like video-only corpora, which scale by equipping many wearers (Ego4D: 931 wearers at 74 sites), the rig can be deployed in parallel by enterprises and data-collection companies. Industry is already moving in this direction, beyond video and handheld grippers to sensing worn by the workers themselves:
With consumer body trackers at about USD 200 each, a rig like ours is cheap to replicate and suits this kind of parallel deployment; C³ calibration keeps data from many collectors and mountings consistent.
W2/Q3. Optical motion-capture reference. We recorded a new participant with optical motion capture (Fig. 20 of the revised PDF) and the rig at the same time, in a dedicated session of about one hour, and scored every method against motion capture (held-out repetitions, mm):
C³ is the most accurate method against the independent reference, and we will state next to the 71.9% figure that it is measured against reference labels, together with this result.
W3/Q3. Touch coverage. We will revise this as suggested: Tab. 1 marks the touch entry as ◦ ("measured on the Wuji subset"), and Sec. 4 reports per-modality coverage.
W4/Q2. Participants and setup time. Two collectors recorded the dataset, both university students trained in laboratory operations; a third participant was recorded only for the independent motion-capture evaluation. Share of each collector's recording time per environment:
Donning the rig takes a few minutes; the calibration protocol takes about 2.5 min and is reused while mounts stay fixed. The release will attach a participant ID to every recording, and further recordings will balance the collectors across tasks.
W5. Typos. Thank you; both are fixed: "EgoExplorer. It" in the caption of Fig. 1, and the semicolon at the end of item (ii) in L128.