acceptodds
Under review as a conference paper at ICLR 2027

EgoExplorer: A Vision-Motion-Touch Observatory for Human Loco-Manipulation In Situ

Abstract

Learning humanoid loco-manipulation from human demonstrations rests on observing how locomotion, posture, and object interaction evolve together as tasks unfold. Scaling collection across diverse environments, in turn, favors lightweight, portable capture systems. Yet, existing datasets provide only partial views of this coordination. Thus, we introduce EgoExplorer, a 100-hour vision-motion-touch observatory of human loco-manipulation built with a wearable system. With an emphasis on professional activities, its continuous recordings align egocentric video with reconstructed whole-body motion, fine-grained finger articulation, and tactile observations, accompanied by step-level descriptions. To achieve this coverage with sparse body sensing, we develop the Calibrate-Complete-Constrain pipeline. Concretely, reusable calibration fixes participant geometry and sensor attachments before motion inference to disentangle mounting offsets from articulation. A temporal prior then completes unobserved joint motion, while layered constraints promote measurement agreement, temporal continuity, and geometric consistency, properties that motion plausibility alone does not guarantee. Against tracker-fitted references, the method reduces body-point error by 71.9% relative to the strongest baseline; matched evaluations also show gains in hand stability and contact estimation. Preliminary studies explore multimodal generation and conditioning alongside humanoid retargeting with whole-body control in simulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, Dataset, Loco-manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MHcsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EgoExplorer, an egocentric dataset of human loco-manipulation with 2,715 recordings (100 h) across ten task templates and five environments, with an emphasis on professional activities such as laboratory and industrial work. The recordings pair head-mounted video with metric whole-body motion and glove-measured finger articulation; recordings made with the Wuji gloves also include touch, and expert-guided step-level text annotations describe task progress. The rig consists of five VIVE Ultimate Trackers (head, both hands, both ankles), a head-mounted GoPro, and one of two glove options: MANUS Metagloves for finger articulation, or Wuji gloves that also record tactile signals. Whole-body motion is recovered from the five tracker poses with the C³ (Calibrate-Complete-Constrain) pipeline. A protocol-guided calibration first fits SMPL-X body shape, metric scale, and the sensor-toanchor transforms once per mounting, so that mounting offsets are not absorbed into the estimated articulation. An AvatarPoser-based transformer prior trained on AMASS then completes the unobserved joints, and layered constraints refine the result for tracker agreement, temporal smoothness, body surface clearance, and ground non-penetration. On a purpose-built Evaluation Subset, C³ reaches 18.0 mm bodypoint error against 64.1 mm for the strongest baseline, a five-anchor fine-tuned AvatarPoser (Tab. 2), and the glove setting outperforms vision-based hand and contact estimation (Tabs. 3 and 4). Preliminary studies use the data for multimodal generation with Cosmos3-Edge and for retargeting recordings to a Unitree G1 tracked by a ZeroWBC-style whole-body controller in simulation.

Critical strengths
  1. Fills a real gap for humanoid learning. From a robotics perspective, the main obstacle to using existing egocentric datasets for humanoids is that they record what the camera sees, mostly the hands, but not the arm, torso, and leg configuration that a humanoid has to control. Tab. 1 makes this concrete: among the eleven prior datasets listed, only Nymeria provides sensor-grounded full-body motion, and it has neither finger articulation nor touch. Adding five trackers to an egocentric rig is a simple and practical way to obtain metric body motion while keeping the first-person view. Because the recordings are continuous, they also preserve the transitions between approaching targets, adjusting posture, manipulating objects, and relocating (L080–082), which is exactly what locomanipulation policies need to learn. The body motion also rests on a well-motivated calibration-first reconstruction. Separating the fixed body–sensor geometry from motion addresses a real ambiguity of sparse-tracker fitting, where mounting offsets can be absorbed into the estimated articulation. The ablations support this design (L473–477): replacing the calibrated shape and scale with the mean body raises floor violations on the Natural Recordings from 6.0% to 50.9%, a measure that does not depend on reference labels, and omitting mount calibration raises body-point error from 18.0 mm to 95.0–98.1 mm. This is the main reason I lean towards acceptance: I expect the data to be useful for humanoid loco-manipulation research, and this outweighs my concerns below.
  2. Accurate finger articulation. Measuring the fingers directly with gloves gives far more reliable hand data than estimating them from video. In the matched hand tests (Tab. 3), the glove setting reaches a fingertip error of 9.5 mm against 27.1 mm (HaWoR) and 30.7 mm (HandFlow), a holding RMS of 0.4 mm against 4.5 and 11.3 mm, no missing frames, and 100% kinematic target recovery against 47% and 67%. Regional contact estimation is similarly strong (IoU 0.915 against 0.621 for HACO, Tab. 4). Finger articulation is what egocentric video recovers worst under self-occlusion, and it is what dexterous humanoid hands need, so I see this as one of the most valuable parts of the dataset. This is the second reason I lean towards acceptance: accurate finger data makes the dataset useful for dexterous manipulation research as well.
Critical weaknesses
  1. Limited accessibility for extending the dataset. The design that makes the data accurate is also what makes it hard to collect more of it. Body and finger data exist only where someone wore the full rig (five trackers, instrumented gloves, and a head camera) and went through the calibration protocol, whose profile stays valid only while the mounts remain fixed (L203–206). The dataset therefore cannot be grown from ordinary egocentric video, and any group that wants to add data needs the same hardware and procedure. The body-tracking part of the rig is far cheaper than studio motion capture (USD 1,184 for the trackers vs. at least USD 8,792 for an eight-camera OptiTrack setup, Tab. 17) and it is portable, so my concern is not cost relative to motion capture but the dependence on dedicated wearable hardware: App. G puts the hardware alone at about USD 5.4k (Wuji) to 8.0k (MANUS) per rig, excluding computers, licenses, and setup labor, and the inside-out trackers depend on suitable visual conditions (L2136–2137). This sits uneasily with the paper’s own motivation that scaling collection across environments “favors lightweight, portable capture systems” (L039– 040). Video-only egocentric collections reach thousands of hours precisely because they need only a camera (3,670 h for Ego4D and 10,000 h for Egocentric-10K in Tab. 1), whereas EgoExplorer has 100 h. This is why I recommend weak rather than clear acceptance: the same rig that makes the data accurate is required to extend it, and the paper does not weigh this trade-off against its own motivation of scalable collection (see Question 1). Minor issues
  2. Independence of the body accuracy reference. The 71.9% reduction in body-point error (18.0 vs. 64.1 mm, Tab. 2) is measured against tracker-fitted labels that, as App. F.2 states, “share our fitting model, calibration, and constraints, so agreement with them is not independent groundtruth accuracy and can favor related methods” (L1712–1713). The full-body evaluation also comes from a single participant, whose first two repetitions train the participant-specific proposal network (L1714–1716). Please state this limitation in the main text next to the 71.9% claim; an optical motioncapture reference on even a small subset would make the body accuracy claim much more convincing.
  3. Touch coverage. Measured touch comes only with the Wuji gloves and is available in 41% of regular recordings, while the MANUS gloves cover 58% (App. E, L1388–1391). By the definition in the caption of Tab. 1 (“✓/◦ are full/partial support”), the Touch entry for EgoExplorer should arguably be ◦. Please report per-modality coverage in the main text.
  4. Number of participants. I could not find how many people were recorded or their demographics. This matters for the diversity of body shapes and motion styles, and for judging how well models trained on the data would generalize.
  5. Typos. A space is missing in the caption of Fig. 1 (“EgoExplorer.It”), and item (ii) of the contribution list ends with a period instead of a semicolon (L128).
Questions for authors

For the rebuttal (responses to these could change my rating)

  1. Use beyond the rig (discussion and additional analysis). A discussion of how the value of the recordings can reach beyond the rig, for example (a) by using them as supervision for estimating body and hand state from egocentric video alone, which would carry the benefit over to video-only collections, and (b) by identifying the minimum sensor set. If feasible, C³ results with only the head and wrist trackers would quantify what the ankle trackers add.
  2. Setup time and participants (clarification). The time needed for donning and calibration per participant, and the number of participants in the dataset (Minor issue 3).
  3. Reference labels and coverage (requested change). A main-text statement of the reference-label limitation next to the 71.9% claim, and per-modality coverage (touch in 41% of regular recordings) reflected in Tab. 1 (Minor issues 1 and 2). Suggestions for future work (not needed for the rebuttal) • An independent optical motion-capture reference on a small subset, which would make the body accuracy claim much more convincing (Minor issue 1)
2 comments
AuthorsOpenReview ↗

Response to Reviewer MHcs 1/2

We thank Reviewer MHcs for the review. All discussions and new experiments below will be incorporated into the revised paper.

W1. Dedicated hardware and scalability. Our aim is the lightest, lowest-cost rig that still records whole-body motion, finger articulation and touch, which video-only capture does not provide at the same fidelity (Tabs. 2–4). In short, the rig is lightweight enough for parallel, accompany-the-work collection; its cost keeps falling at volume; and companies are already scaling this kind of wearable collection.

  • Lightweight and low-cost. All components are off-the-shelf consumer or industrial products. At USD 5.4k–8.0k, a rig costs a fraction of a studio motion-capture volume (Tab. 17) or a humanoid teleoperation station, and the cost falls further at volume.

  • Throughput. Donning takes a few minutes, the 2.5-min calibration protocol is reused while mounts stay fixed (L203–206), and a collector then records continuously through an 8-hour working day, accompanying the work as in egocentric video collection. No instrumented space is needed.

  • Scale through parallel deployment. Like video-only corpora, which scale by equipping many wearers (Ego4D: 931 wearers at 74 sites), the rig can be deployed in parallel by enterprises and data-collection companies. Industry is already moving in this direction, beyond video and handheld grippers to sensing worn by the workers themselves:

    • PsiBot (2026) and TARS (2026) equip logistics, retail and cleaning workers with finger-tracking gloves, fingertip tactile sensing and egocentric cameras during their normal shifts. Each reports on the order of 100k hours, at about one-tenth the cost of real-robot collection (PsiBot).
    • Sunday Robotics (2026) trained a home robot without robot data, from a ~USD 200 skill-capture glove used in more than 500 homes.
    • Daimon (2026) releases visuotactile glove data recorded with wrist trackers and an egocentric headset, and PaXini (2026) runs a 150-station tactile data-collection factory.

    With consumer body trackers at about USD 200 each, a rig like ours is cheap to replicate and suits this kind of parallel deployment; C³ calibration keeps data from many collectors and mountings consistent.

W2/Q3. Optical motion-capture reference. We recorded a new participant with optical motion capture (Fig. 20 of the revised PDF) and the rig at the same time, in a dedicated session of about one hour, and scored every method against motion capture (held-out repetitions, mm):

MethodAnchorsBody pts↓Non-tracked↓PA↓Floor viol.↓
AvatarPoser379.074.266.580.7%
AvatarJLM371.066.858.176.5%
HMD-Poser*5131.4128.469.919.0%
IK with VPoser pose prior585.692.053.02.2%
AGRoL*559.558.238.769.9%
AvatarJLM*551.751.636.798.8%
C³ (ours)545.549.324.60.4%

C³ is the most accurate method against the independent reference, and we will state next to the 71.9% figure that it is measured against reference labels, together with this result.

W3/Q3. Touch coverage. We will revise this as suggested: Tab. 1 marks the touch entry as ◦ ("measured on the Wuji subset"), and Sec. 4 reports per-modality coverage.

W4/Q2. Participants and setup time. Two collectors recorded the dataset, both university students trained in laboratory operations; a third participant was recorded only for the independent motion-capture evaluation. Share of each collector's recording time per environment:

LaboratoryFood preparationStudyLiving roomIndustrial
Collector 189%65%44%34%53%
Collector 211%35%56%66%47%

Donning the rig takes a few minutes; the calibration protocol takes about 2.5 min and is reused while mounts stay fixed. The release will attach a participant ID to every recording, and further recordings will balance the collectors across tasks.

W5. Typos. Thank you; both are fixed: "EgoExplorer. It" in the caption of Fig. 1, and the semicolon at the end of item (ii) in L128.

AuthorsOpenReview ↗

Response to Reviewer MHcs 2/2

Q1(a). Use for video-only estimation. Video alone is not sufficient for fine manipulation data, and the paper and our new experiments quantify why:

  • On the same trials, the glove and touch sensors outperform video-based estimation on finger articulation and contact (Tabs. 3–4).
  • Under camera motion and occlusion, video-based hand estimates lose frames (23% missing under occlusion) and stay 24–33 mm from the instructed hand pose, versus 6.8 mm for the glove.
  • In the Natural Recordings, at least one wrist is outside the head camera's view in 8–16% of frames, and in 10–21% of frames with measured contact, depending on the camera's field of view. Video-only labels would have to be invented for these frames.

This is exactly what makes the recordings useful beyond the rig: every egocentric frame is paired with sensor-measured body, finger and contact labels, including the lower body and occluded or out-of-view hands, so they can supervise and benchmark video-only estimators where those are weakest.

Q1(b). Minimum sensor set. We ran C³ with the head and wrist trackers only, under the same calibration and constraints, removing the ankle trackers from every term. Ten test recordings, mm:

InputsBody pts↓Wrists↓Upper untracked↓Knees↓Lower body↓
C³, head + wrists (3 trackers)113.611.164.5191.6212.0
C³, all five trackers18.08.420.526.317.2

With head and wrists only, refinement repairs the arms but leaves the legs unconstrained (212 mm lower-body error), and heel slide during support on the Natural Recordings rises from 4.5 to 31.5 cm/s. Since the legs are mostly outside the egocentric view, five trackers are the minimum for whole-body loco-manipulation data.

We thank the reviewer again. If these changes address the concerns, we would be grateful if the reviewer would consider raising the score.

Reviewer_MjigOpenReview ↗
Rating
3
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces EgoExplorer, a 100-hour dataset combining egocentric video, reconstructed full-body motion, hand articulation, touch, and step-level descriptions. Its pipeline combines body–sensor calibration, learned motion completion, and constrained optimization to reconstruct motion from five wearable trackers. Experiments assess annotation quality and explore multimodal generation and human-to-robot motion transfer in simulation.

Critical strengths
  1. Rich multimodal coverage. EgoExplorer combines egocentric video, full-body motion, finger articulation, touch where available, and step-level descriptions across continuous activities. This provides a valuable resource for studying how locomotion and manipulation interact.

  2. Practical capture and reconstruction pipeline. The wearable setup supports collection across environments without external camera arrays. The C3 pipeline combines participant and sensor calibration, motion completion, and geometric constraints, with ablations showing their contributions to reconstruction quality.

Critical weaknesses

1. Motion accuracy needs independent and dynamic validation

The full-body reference labels share the proposed method's tracker observations, body model, calibration, and fitting constraints. Agreement with these labels may fit related methods but does not establish absolute motion accuracy. Cosmos uses reconstructed action targets, so its action MSE accumulates over label errors, although its head, wrist, and fingertip features have direct sensor support. The hand evaluation focuses on held configurations recorded separately for the glove and video methods, leaving dynamic accuracy and latency unclear.

2. Generalization to new participants and scenes remains untested

The body evaluation adapts the network using the first two repetitions of ten actions from the evaluation participant and tests on the third repetition. Cosmos holds out sessions within each task, without a reported participant-disjoint or scene-disjoint split. The robot controller is evaluated on its training clips. These settings provide limited evidence of transfer to new users and environments.

Questions for authors

1. Comparison with a simple IK baseline

The constraint-layer ablations share the same learned motion proposal, leaving its contribution relative to direct inverse kinematics unclear. A sequential IK baseline using the same tracker inputs, body geometry, and mount calibration would be useful.

2. Reconsider the aggregate score

The aggregate score averages nine quantities with different units and scales. Changing units can change their relative contributions and potentially the ranking. Please emphasize individual metrics or define and justify normalization and weights. The full method's lowest aggregate score should be discussed alongside its higher body-point error and jerk than Layer 1 alone.

3. Evaluation settings do not fully match the dataset processing pipeline

The contact benchmark uses annotated rest windows, excludes frames within 100 ms of button transitions, and scores only task-specified regions. It also averages tactile readings into 30 Hz bins, while dataset export preserves any valid contact detection within each bin. Its scores leave the accuracy of exported contact labels during natural activity unclear. The body evaluation also changes the torso-lean limit from the production setting of 15° to 90° and adds participant-specific constraints.

4. Cross-modal synchronization needs quantitative validation.

The paper describes timestamp alignment and resampling to a common timeline, but does not quantify residual timing errors between video, trackers, and gloves. Please clarify how device timing offsets and latency are handled, and report alignment error using events observable across modalities.

5. A minor clarification about dataset composition.

Please report the number of participants, recordings per participant, and participant overlap across splits.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Mjig 1/3

We thank Reviewer Mjig for the review. All discussions and new experiments below will be incorporated into the revised paper.

W1. Motion accuracy: independent and dynamic validation.

Body reference. Joint centers lie inside the body, so every body benchmark scores against a fitted reference; we will call our targets reference labels. They are not C³ run longer: they are corrected frame by frame toward the protocol-defined guide motion of the instructed action, which fixes the articulation where the trackers admit several and which the model never sees; they also use an initialization from an earlier five-anchor AvatarPoser checkpoint never trained on this participant, and a five-times larger optimization budget, none of which C³ receives at test time.

Independent reference. We recorded a new participant with optical motion capture (photos of the setup: Fig. 20 of the revised PDF) and the rig at the same time, in a dedicated session of about one hour, and scored every method against motion capture (held-out repetitions, mm):

MethodAnchorsBody pts↓Non-tracked↓PA↓Floor viol.↓
AvatarPoser379.074.266.580.7%
AvatarJLM371.066.858.176.5%
HMD-Poser*5131.4128.469.919.0%
IK with VPoser pose prior585.692.053.02.2%
AGRoL*559.558.238.769.9%
AvatarJLM*551.751.636.798.8%
C³ (ours)545.549.324.60.4%

C³ is the most accurate method against the independent reference.

Cosmos action targets. We measured the label error over Natural Recordings. It is stable over the length of a recording: the median is 10.1 mm in the first quarter of a recording and 11.1 mm in the last, and for recordings longer than 5 min the median slope is −0.004 mm/min. The trackers localize against a map rather than integrating inertial signals, so the error behaves as a small, constant offset and does not accumulate over time.

Hands under motion and occlusion, and latency. We added hand experiments under camera motion and under occlusion, where an occluder passes in front of the hand as a held object would. The hand holds the instructed configuration, so the per-frame distance to the instructed target is the error.

MethodConditionSuccess (%)↑Missing (%)↓Per-frame target error (mm)↓Spurious finger speed (mm/s)↓
HaWoRmoving / occluded50 / 400 / 23.23325 / 26
HandFlowmoving / occluded70 / 400 / 23.02437 / 38
Glove (ours)any10006.80.4

The glove stream lags the wrist tracker by a median of 33 ms (p95 50 ms), measured by cross-correlating wrist angular speed.

AuthorsOpenReview ↗

Response to Reviewer Mjig 2/3

Response to Reviewer Mjig (2/3)

W2. Generalization.

Body: cross-participant and cross-action. We adapt each method on the same data and test on data that none of them saw, scored against synchronous optical motion capture on the new participant: (A) adapted on the Tab. 2 participant only, tested on the new participant; (B1, B2) adapted on five of the ten actions of the new participant, tested on the other five (two splits). Body pts / Non-tracked / PA in mm, and floor violations:

MethodA: cross-participantB1: cross-actionB2: cross-action
AvatarPoser*108.1 / 101.1 / 64.4 / 77%85.3 / 78.0 / 56.9 / 82%79.4 / 70.7 / 50.5 / 82%
HMD-Poser*99.7 / 92.6 / 64.6 / 46%77.3 / 70.1 / 37.3 / 92%72.7 / 69.3 / 38.7 / 66%
AGRoL*58.7 / 57.8 / 38.1 / 76%50.1 / 47.7 / 28.2 / 77%51.5 / 49.8 / 30.1 / 90%
C³ (ours)46.6 / 46.5 / 33.4 / 0%44.6 / 49.3 / 23.6 / 0%42.1 / 45.1 / 27.6 / 0%

C³ generalizes best across participants and across actions, with no floor violations.

Cosmos. We evaluated the joint model (5K steps) separately on five representative tasks from three environments:

Task (environment)Video PSNR↑ (dB)Action MSE↓
Weighing (laboratory)17.860.082
Oven sampling (laboratory)15.240.090
Rotary evaporation (laboratory)19.130.065
Sandwich making (food preparation)15.170.122
Shelf organizing (study)15.180.114

We also held out two tasks entirely from training (rotary evaporation, extraction). On these unseen tasks, action generation reaches an MSE of 0.045 and 0.031, in line with the 0.044 obtained on the in-distribution validation split.

Robot. The policy trained on two laboratory clips was applied without retraining to eight new recordings not used in training:

ClipsDurationRelative MPJPE↓Global wrist error↓
2 training clips5.5 min4.6 / 3.7 cm5.4 / 4.2 cm
8 new recordings24.7 min5.3 cm9.0 cm
AuthorsOpenReview ↗

Response to Reviewer Mjig 3/3

Response to Reviewer Mjig (3/3)

Q1. IK with the same inputs. Both IK baselines use the same five tracker streams, SMPL-X shape and scale, mount calibration, joint limits, and surface and floor terms as C³, without a learned proposal; plain IK starts every frame from a standing pose, and the second variant adds the VPoser pose prior. Ten test recordings of Tab. 2, mm:

MethodBody pts↓PA↓Elbows↓Knees↓
Plain IK, 300 it.48.741.253.782.9
Plain IK, 1,000 it.47.041.254.380.5
IK with VPoser pose prior, 300 it.52.039.654.181.4
C³, 60 it.18.013.319.326.3

With identical inputs, geometry and calibration, C³ is clearly more accurate than IK, which shows the contribution of the learned proposal.

Q2. Aggregate score. Most of the quantities in Tab. 7 are millimeter-based (pose errors in mm, velocity and slide in mm/s) and take values on a similar scale, so no single quantity dominates the plain average. Unit-free summaries give the same ranking:

ConfigurationAvg↓Ratio to proposal↓Geometric mean ratio↓Mean z-score↓
Full28.40.500.35−0.71
Layer 1+331.20.530.36−0.58
Layer 1+234.10.690.45−0.50
Layer 136.50.700.45−0.39
Proposal only74.21.001.00+1.00

Trade-off. The layers trade off slightly across metrics. Layer 1 and Layer 1+3 have about 2 mm lower body-point error and 4 m/s³ lower jerk, whereas Full nearly halves limb crossing (38.1 vs. 72.8–73.0 pairs per frame) and keeps floor violations at 6.0% versus 57.3% for Layer 1 alone. Interpenetration and sub-floor feet are the errors that matter most for data that drives a physical humanoid, so we consider the small cost on the other metrics acceptable.

Q3. Evaluation versus export settings.

  • Contact. The benchmark follows a controlled protocol with rest baselines, transition exclusions and task masks, so that every method is scored under identical conditions. The export keeps every valid detection, unprocessed, so users can apply their own rule.
  • Torso lean. The release configuration uses the 90° torso-lean limit together with the knee-hyperextension and pelvis-roll terms, the same as the evaluation. The 15° in Eq. 11 is a typo.

Q4. Synchronization. Video, gloves and trackers are aligned by their host timestamps and resampled to a common timeline; on the Wuji rig, video (30 Hz), gloves (120 Hz) and trackers (about 65 Hz) are brought to 30 Hz. The camera has a constant capture-path delay, which we measure per session by cross-correlating head-video motion with head-tracker motion and remove. Measured on events visible in both modalities, the remaining misalignment between video and trackers is 20 ms (median; p95 60 ms), below one 30 Hz frame, and the glove lags the trackers by 33 ms (median).

Q5. Composition. Two collectors recorded the dataset; a third participant was recorded only for the independent motion-capture evaluation. Share of each collector's recording time per environment:

LaboratoryFood preparationStudyLiving roomIndustrial
Collector 189%65%44%34%53%
Collector 211%35%56%66%47%

The release will attach a participant ID to every recording, and further recordings will balance the two collectors across tasks. The third participant appears in no training data.

We thank the reviewer again.

Reviewer_ZdbROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper starts by describes a portable rig for capturing an egocentric dataset for human loco-manipulation focused on professional tasks. The rig included a GoPro head camera, a VIVE Ultimate tracker, trackers on both writs and ankles and MANUS gloves or Wuji tactile gloves. The authors use that rig to capture 100h of loco-manipulation tasks in five environments. The majority is from chemistry-lab procedures, the rest are home, study, food-preparation and industrial tasks. Additionally, the paper presents a pipeline for extracting body motion from this sparse measurement data. The pipeline does: Calibrate, Complete, and Constrain (C^3), including a guided fit of SMPL-X shape and scale models, and some tracker mount offset calibration. Then AvatarPoser is used to for the completion step (fine tuned on AMASS, synthetic data and some real data). The constraint step takes care of self-penetration and floor penetration. Glove data and touch are fused afterwards. Text describing the steps taken in the data comes from an VLM guided by expert “skill documents” and reviewed by humans. The experiment section evaluation body error vs sparse tracker baselines (where the 71.9% improvement comes from) as well as glove vs. video hand methods and tactile vs RGB contact estimation. Finally a Cosmos-based multimodal generation pilot is shown. There are multiple related other datasets out there, as shown in Table 1. The novelty as shown by the table lies in the combination of whole-body motion, glove measured fingers and tactile in real workspaces. No prior dataset has this combination, but each component exists separately. The novelty of the “method” is modest (according to my point of view). Sensor to segment calibration with a prescribed pose protocol with anthropometrics is standard in VR full-body tracking (see e.g. Fitted avatars: automatic skeleton adjustment for self-avatars in virtual reality, Ponton et al 2023 – uncited by the authors). Furthmore, the completion step is AvatarPoser with two additional inputs. Refinement with tracker, prior, self-penetration, floor terms, smoothness are very similar to SMPLify-X and EgoAllo fitting, optimization of learned priors is similar to DragPoser. MPLify-X: G. Pavlakos et al., "Expressive Body Capture: 3D Hands, Face, and Body from a Single Image," CVPR 2019. EgoAllo: B. Yi et al., "Estimating Body and Hand Motion in an Ego-sensed World," CVPR 2025. DragPoser: J. L. Ponton et al., "DragPoser: Motion Reconstruction from Variable Sparse Tracking Signals via Latent Space Optimization," Computer Graphics Forum, 2025. Fitted avatars: J. L. Ponton et al., "Fitted avatars: automatic skeleton adjustment for self-avatars in virtual reality," Virtual Reality, 2023.

Critical strengths
  • The dataset features a unique combination of inputs/offer a combination of modalities that did not exist before. The professional lab procedures include "recovery" episodes, i.e. failures and corrections which are typically not present in egocentric data sets. Those may offer opportunities for robotics which are not present anywhere else.
  • The presented capture rig is lightweight, base-station-free and has a cost analysis which is still considered reasonable. It is plausible for other labs to replicate the rig.
  • The paper offers a concreate release plan with exact formats etc. (LeRobot v3 etc).
  • The appendix goes into a lot of detail that can be valuable, including limitations such as the fact that the body labels are not independent.
  • The authors ran a the text annotation audit and found 11% factual errors before correction, indicating that the authors care about quality and are honest about the data set.
Critical weaknesses
  • If I understand it correctly, the body accuracy evaluation is circular. The appendix notes that there is "no independent optical reference". So the "ground truth" is the same five-tracker refinement (300 vs. 60 iterations) with the same calibration and constraints. Thus the 71.9% result also from the abstract and all other measurements mostly measure agreement with itself and there is no way to judge how well the pipeline works or what can actually be drawn form the data set.
  • The evaluation scale is way too small. The full body evaluation is a single participant with 10 scripted actions and 3 repetitions each. The network is fine-tune on repetition 1 and 2 of the single participant and the test uses repetition 3. There is no accuracy evaluation on longer, locomotion-heavy recordings. The only metrics are self-consistency (tracker RMS against the tracker inputs, penetration and jerk).
  • The baseline comparisons fall under the category of apples to oraganes. Most baselines receive only 3 trackers (head and wrists), whereas the method additionally receives the ankles, so better leg and foot accuracy is to be expected. The only 5-anchor baselines are versions the authors retrained themselves (AvatarPoser and HMD-Poser). There is no comparison to a simple IK baseline using the same 5 tackers, which would be the natural baseline.
  • The hand and contact comparisons do not give any load bearing meaning. They show that sensors beat vision from RGB only. This is of course true. As there is no independent finger ground truth available there is no way to validate the datasets accuracy.
  • There is no basic data set documentation. I could not find information on number of participants, demogrpahics, whether the lab tasks were done by actual professionals. Also there is no information on consent/IRB or license reported as far as I cantell.
  • The dataset coverage compared to the loco-manipulation framing seems rather thin. Head path lengths are 5-50m per session and the data seems to be dominated by stationary bench work. The touch analysis is dominated by lab work (about 146-165 lab vs 3 -10 otherwise).
  • The cosmos pilot appears weak. If I understand correctly, the authors used only 5k steps with one seed and the conditional study gave the model the future video as input. Furthermore, retargeting trains and evaluates on the same two clips. There is no evidence that the date improves robot or humanoid policies or beats anything trained on existing data sets.
Questions for authors
  1. Can you validate body and hand accuracy against an independent reference, possibly even briefly and on a few participants?
  2. How many participants are there overall? Can you provide information about their expertise and background? Please provide information on consent/IRB process and license.
  3. How does C3 compare with a plain 5 tracker IK baseline using the same calibration?
  4. Is there any downstream result where training on EgoExplorer beats existing data?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ZdbR 1/3

We thank Reviewer ZdbR for the review. All discussions and new experiments below will be incorporated into the revised paper.

W1/Q1. Body evaluation and an independent reference.

Joint centers cannot be measured directly. Joint centers lie inside the body, so no sensor observes them on a moving person. Every body benchmark therefore scores against a reference obtained by fitting a body model to measurements; optical motion capture, for example, fits the body model to surface markers (MoSh++ in AMASS). The targets of Tab. 2 are such a reference, built from the rig's own sensors, and we will call them reference labels throughout.

What the labels contain that C³ does not get. Each label is built from the five tracker streams and the calibration profile, plus three things C³ does not receive at test time:

  • initialization from an earlier five-anchor AvatarPoser checkpoint that was never trained on this participant, a different checkpoint from the one C³ uses;
  • the instructed action (B01–B10): its guide motion, which we define in the protocol (Fig. 17), corrects the hip, spine, knee, ankle and foot rotations frame by frame toward the instructed pose, so the label follows the action that was actually performed where the trackers admit several articulations (e.g., a forward bend versus a squat). C³ never has this information;
  • 300 refinement iterations, five times the C³ budget.

The labels are therefore not "C³ run longer". They share SMPL-X, the calibration and the constraint terms with C³, but AvatarPoser* receives the same five calibrated anchors and was adapted on the same participant, yet reaches 64.1 mm versus 18.0 mm.

An independent reference. To remove the shared parts entirely, we recorded a new participant, never seen in training, with optical motion capture (NOKOV) and the rig at the same time, in a dedicated session of about one hour of synchronized recording, performing the ten actions including walking, sit-to-stand and kicks (photos of the setup: Fig. 20 of the revised PDF). Test repetitions, mm:

MethodAnchorsBody pts↓Non-tracked↓PA↓Floor viol.↓
AvatarPoser379.074.266.580.7%
AvatarJLM371.066.858.176.5%
HMD-Poser*5131.4128.469.919.0%
IK with VPoser pose prior585.692.053.02.2%
AGRoL*559.558.238.769.9%
AvatarJLM*551.751.636.798.8%
C³ (ours)545.549.324.60.4%

Against an independent motion-capture reference, C³ is the most accurate method.

AuthorsOpenReview ↗

Response to Reviewer ZdbR 2/3

W2. Evaluation scale.

  • A second participant, independent reference. To address the single-participant concern, we evaluated on a new participant against synchronous optical motion capture, covering locomotion (a rectangular walk, sit-to-stand and kicks) as well as reaching and bending (table above). C³ again has the lowest error: 45.5 mm body-point error, versus 51.7 mm for the best learned five-tracker baseline.
  • Large-scale in-situ recordings. For in-situ data, no full-body joint labels can exist, so we use measures that need none. On the Natural Recordings, the five measured device poses are reproduced through the full kinematic chain and fixed mounts with a median RMS of 13.8 mm for C³ versus 146 mm for three-anchor AvatarPoser; self-intersecting frames are 3.2% versus 62.7%, and heel slide during support is 4.5 versus 22.2 cm/s.
  • Visual comparison. Fig. 22 of the revised PDF compares reconstructions of all methods side by side on the evaluation subset, where the differences are directly visible.

W3/Q3. Input-matched baselines and IK. Tab. 2 already includes three baselines that receive at least five inputs. We now add two more learned five-tracker baselines and two IK baselines that use the same five trackers and calibration as C³. Ten test recordings of Tab. 2, mm:

MethodInputsBody pts↓Non-tracked↓PA↓
SparsePoser6219.2189.9134.4
HMD-Poser*5136.7131.571.3
AvatarPoser*564.162.121.0
AGRoL* (new)596.393.241.4
AvatarJLM* (new)562.664.927.2
C³ (ours)518.020.313.3
IK with the same five trackers and calibration (new)Body pts↓PA↓Elbows↓Knees↓
Plain IK, 300 it.48.741.253.782.9
Plain IK, 1,000 it.47.041.254.380.5
IK with VPoser pose prior, 300 it.52.039.654.181.4
C³, 60 it.18.013.319.326.3

With the same inputs, C³ remains the most accurate method, both against the learned five-tracker baselines and against IK.

W4. What the hand and contact tests establish.

They confirm the premise of EgoExplorer. The largest egocentric datasets (EgoDex, EgoVerse, Open-AoE, EgoLive; Tab. 1) label hands from video. EgoExplorer is built on the premise that such estimates are not accurate enough for the finger articulation and contact that manipulation learning needs, so dedicated finger and touch sensing is required (Sec. 1, paragraph 2; Sec. 3.1). Tabs. 3–4 confirm this premise on the same trials: video-based estimation falls well short of the sensors. The gap also matters for use: retargeted to the 20-DoF hand while the human hand is held still, video drives the fastest robot joint at 0.22–0.35 rad/s with jerk 158–370 rad/s³, against 0.01 rad/s and 16 rad/s³ for the glove, so a robot imitating video labels would twitch.

The evaluation does include references independent of the glove.

  • Contact (Tab. 4). Contact is scored against a hand-held button pressed during contact, a device separate from the tactile glove, over the task-specified regions (Fig. 19). Against this reference, the touch channel reaches F1 0.956 (IoU 0.915), versus 0.766 for RGB (HACO) and 0.490 for hand pose alone.
  • Hand stability and availability (Tab. 3). Hold RMS, Missing and Gap p95 need no finger reference: during a held configuration, any motion of the reconstructed fingers is error, and missing frames are counted directly. The glove gives 0.4 mm Hold RMS and no missing frames, versus 4.5–11.3 mm and 7.8–7.9% missing for video.
  • A sensor-free check (new). In the five opposition gestures, the instruction names the fingertips that must touch the thumb. With a fixed 20 mm contact rule on the retargeted hand, the glove is correct in 15/15 trials and HaWoR and HandFlow in 10/15 and 12/15. The median thumb–target gap is 12 mm for the glove versus 41 and 72 mm for video, and the input frames show the pinches closed, so the gap comes from reconstruction, not from execution.
AuthorsOpenReview ↗

Response to Reviewer ZdbR 3/3

W5/Q2. Participants, consent and license.

  • Participants. Two collectors recorded the dataset. Both are university students trained in laboratory operations, who followed the expert-written skill documents (App. D.2) for the laboratory procedures. A third participant was recorded only for the independent motion-capture evaluation. Share of each collector's recording time per environment:
LaboratoryFood preparationStudyLiving roomIndustrial
Collector 189%65%44%34%53%
Collector 211%35%56%66%47%

We will balance the two collectors across tasks in further recordings, and the release will attach a participant ID to every recording.

  • Consent and privacy. All collectors gave informed consent to recording and public release, and all recordings were made at our own facilities. The Ethics Statement following the conclusion (L501–506) describes the privacy processing applied before release: recordings, metadata and textual annotations are reviewed, and identifying or sensitive information is removed or masked.
  • License. CC BY-NC 4.0.

W6. Coverage.

  • Loco-manipulation is well covered. More than half of the recorded hours (54%) come from outside the laboratory: the living room, study, food-preparation area and industrial workspace. In the household and industrial recordings alone, the collectors travel 33.9 km in total, and the share of session time spent moving reaches 60% in the study and 36% in food preparation (Fig. 13(c)). The laboratory procedures are not stationary bench work either: they chain several workstations (balance, oven, rotary evaporator, column), alternating manipulation with walks between them. The recordings therefore cover the indoor cycle of relocating, adjusting posture and manipulating that loco-manipulation requires.
  • Touch. Laboratory tasks contribute many short recordings, so recording counts understate the other environments. Measured by duration, 45% of the touch data comes from non-laboratory environments, and we continue to record non-laboratory touch data toward a balanced task distribution.

W7/Q4. Pilot studies. The pilot studies are preliminary trials whose purpose is to verify that the released data can be used for learning: that the modalities can be generated jointly, and that the exported motion can drive a humanoid in physics. They are not designed as a dataset-versus-dataset policy comparison.

Same-interval video. Supplying video of the same interval is part of the study's design: it asks whether touch adds information about the action beyond the video of that interval, the situation of labeling actions from video. The unconditioned row of Tab. 6 (None) is the history-only reference: it receives the same video/action/touch history but no observation of the generated interval.

Training budget. We re-ran the conditioning study with 15K training steps and two seeds.

Condition (action generation)5K, paper15K, seed 0 / seed 1 (new)
None0.04780.0436 / 0.0436
+ video0.02770.0286 / 0.0284
+ video and touch0.02640.0270 / 0.0267
Separate / Joint5K, paper15K, seed 0 (new)15K, seed 1 (new)
Video PSNR↑ (dB)15.94 / 16.6215.99 / 16.4315.83 / 16.49
Action MSE↓0.059 / 0.0440.058 / 0.0430.057 / 0.046
Contact F1↑0.610 / 0.6940.617 / 0.6430.617 / 0.655

The ordering is unchanged with longer training and across seeds: video almost halves the action error, and touch reduces it further.

Robot tracking on new recordings. The controller was trained on two clips to verify that the exported motion can be retargeted to a humanoid and tracked in physics. We applied the same policy, without retraining, to eight new recordings that were not used in training:

ClipsDurationRelative MPJPE↓Global wrist error↓
2 training clips5.5 min4.6 / 3.7 cm5.4 / 4.2 cm
8 new recordings24.7 min5.3 cm9.0 cm

The controller tracks the new recordings with about 1 cm higher relative error than its training clips.

Please let us know which concerns remain unresolved.

Comments on the paper

2 comments

Dear Reviewers,

We sincerely thank you for your careful reviews and insightful suggestions. We have updated the manuscript in response to your feedback and marked the revised text in blue to make the changes easy to identify.

We welcome any further questions or concerns and would be happy to discuss them and provide additional clarifications.

Dear Reviewers,

As the discussion period draws to a close, we would greatly appreciate it if you could take a final look at our responses and revised manuscript. We would be happy to address any remaining questions or concerns.

Thank you again for your time and thoughtful feedback.

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.