acceptodds
Under review as a conference paper at ICLR 2027

EvUBody: Resolving Kinematic Ambiguity in Event-Assisted Joint Upper-Body and Hand Reconstruction

Abstract

Extracting precise human manipulation trajectories from video is critical for scaling embodied robot learning. While sparse RGB frames frequently miss high-frequency kinematics and suffer from motion blur, event streams have emerged as a powerful modality to capture asynchronous dynamic changes. However, existing event-based paradigms are overwhelmingly restricted to isolated hand motion, failing to account for the macro upper-body movements that induce complex, ambiguous event patterns across the hand during real-world manipulation. Transitioning to a joint upper-body and hand framework is strictly necessary, yet it has been severely bottlenecked by a fundamental data collection conflict: obtaining precise, continuous 3D joint annotations typically requires invasive wearable sensors or dense optical markers, which destroy the natural visual appearance of the human and suffer from severe occlusion during hand-object interaction. Furthermore, merging separated body and hand tracking systems introduces persistent spatial discontinuities at the wrist. To solve this, we introduce EvUBody, the first comprehensive dataset and benchmark that successfully balances natural RGB-Event visual observation with high-fidelity, spatially aligned 3D kinematic references for the coupled arm-and-hand system. Comprising both simulated and real-world subsets, EvUBody enables our proposed causal reconstruction framework, which fuses sparse RGB spatial anchors with continuous event streams via joint spatio-temporal modeling. By evaluating across varying frame rates and lighting conditions, we demonstrate that jointly modeling the upper body resolves event ambiguities, allowing our framework to recover true manipulation trajectories that RGB-only priors and hand-isolated approaches fail to capture.

Primary area:
datasets and benchmarks
Keywords:
Event-Based Vision, Upper-Body and Two-Hand Motion Reconstruction, RGB–Event Fusion, Human Manipulation, Datasets and Benchmarks.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bLV2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EvUBody introduces a dataset and a causal framework for joint upper-body and two-hand reconstruction from sparse RGB frames and event streams. The model combines RGB anchors, global and local event features, quality-conditioned fusion, and joint spatiotemporal attention. The evaluation covers simulated and recorded events, with an RGB-only ablation examining the contribution of event information.

Critical strengths
  1. The dataset addresses a useful gap in capturing coordinated arm and finger motion during natural manipulation. The real subset uses independent multiview RGB references while preserving an uninstrumented appearance (Section 3).
  2. The structured decoder is well motivated: expressing hand joints relative to the predicted body wrist guarantees consistent wrist positions by construction (Eq. 3).
  3. Evaluating timestamps without current RGB observations directly addresses inter-frame reconstruction. Under this protocol, the full model reduces MPJPE from 23.40 to 20.97 mm relative to the independently trained RGB-only baseline (Table 2).
Critical weaknesses
  1. Evaluation scope and baseline coverage need clarification. Table 1 labels the proposed method's OakInk2 result of 20.97 mm as “Body,” whereas Table 2 reports the same value for the full 57-node representation and Table 3 reports a body error of 22.66 mm. It is unclear whether these results use the same evaluation conditions. Moreover, AiOS [Sun et al., CVPR 2024] and SMPLest-X [Yin et al., TPAMI 2025] support hand reconstruction but are evaluated only in the Body group of Table 1. Consequently, the hand comparison excludes strong RGB alternatives. Please define the joint sets and evaluation conditions consistently and report hand results for all applicable baselines before drawing broad superiority claims.

  2. The experiments do not clearly isolate the central mechanism or component contributions. The title and abstract attribute ambiguity resolution to joint upper-body–hand modeling, but removing global events does not remove body context. A matched joint-versus-hand-only comparison using the same hand metrics is needed. The “Ours Hands” row in Table 1 does not establish this comparison because its relationship to the joint model is unclear. Furthermore, Appendix E.1 states that the full model uses fixed RGB-anchored regions while the four component variants retain an adaptive region policy. This confounds the component comparisons in Section 5.4. These variants should be compared under the same region policy.

  3. The reference-accuracy claim requires a reproducible validation protocol. Section 3 reports 0.8 mm MPJPE against optical motion capture, but the manuscript does not adequately specify the validation sample, evaluated joints, alignment procedure, or hand-specific accuracy. These details are important because reference quality is central to the dataset contribution, particularly during fast, occluded manipulation. Please provide an independent validation with separate body and hand errors, error distributions, and a description of temporal alignment.

Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Author rebuttal for Reviewer bLV2

Response 1/3: W1 - evaluation scope and hand-baseline coverage

Thank you for recognizing the dataset, wrist-consistent decoder, and evaluation without current RGB. We address the three weaknesses in three responses. Manuscript locations refer to the revised version; M/PA denote MPJPE/PA-MPJPE in mm.

W1.1 Corrected labels and evaluation conditions. We apologize for the labeling and reporting errors in Table 1, and thank the reviewer for identifying the inconsistency. 20.97 mm on OakInk2 is Full MPJPE over 57 nodes. Revised Table 1 labels our joint entry and the RGB body-hand entries as Full (p. 7): 15 body nodes plus 21 per hand, including duplicated wrists and derived hand centers with their scoring weights (Appendix D, pp. 18-19).

For our model, Table 1's 20.97 is the Full result detailed in Table 2; Table 3 gives its regional breakdown. These measurements use the same checkpoint and conditions: OakInk2 validation, withheld RGB, EV0, k=3, phase zero, seed 42, and the final epoch-40 checkpoint (Sec. 5.4, p. 8; Appendix E.1, p. 19).

QuantityEvaluated nodesM
Full, Table 2All 5720.97
Body, Table 30-1422.66
Left hand, Table 315-3520.33
Right hand, Table 336-5620.39

Thus, 20.97/22.66 describe the complete skeleton/body region under the same ablation protocol. MPJPE sums Euclidean errors and valid-node counts across sequences before division. Full PA fits one per-frame similarity transform over valid 57-node sets, including translation, scale, and reflection-free rotation; hand PA independently fits each 21-node hand (Appendix D).

Temporal subsets. Table A4 (p. 20) reports OakInk2 Full M of 20.81/20.50/20.97 over All/Retained/Withheld timestamps. Interact Table 1 reports Full M/PA of 37.81/22.22 at EV0/All (Table A17, p. 28); Table 2 reports 38.68/22.72 at Withheld timestamps. Our Interact Full PA is corrected from 19.88 to 22.22, consistent with the appendix. These scopes distinguish anatomy from RGB availability.

W1.2 Expanded hand comparison. Appendix K/Table A28 (pp. 33-34) includes AiOS, SMPLest-X, and other RGB methods with articulated hand outputs. We reproduce the Interact results below and additionally summarize the adapted LME hand output from Table A15 (p. 27). RGB-containing methods use EV0/All; event-only methods use Event/All. Each hand has 21 evaluated nodes and independent PA alignment.

MethodInputHand MHand PA
EvRGBHandRGB + events67.0210.73
Ev2HandsEvents130.8312.73
EventHandsEvents105.3910.91
AiOSRGB37.546.52
SMPLest-XRGB41.387.25
SAM-Body4DRGB video154.9513.24
Hand4Whole++RGB53.9813.07
PEARRGB82.368.63
GEM-XRGB video48.3410.09
LME, adaptedEvents105.1912.19
Ours, joint model scored on handsRGB + events36.367.89

Hand summaries equally average the reported left/right errors; Full uses valid-node aggregation. LME is calculated from Table A15: (110.840+99.545)/2 for M and (11.831+12.546)/2 for PA. Its Interact adaptation predicts 57 nodes; original OakInk2 LME predicts body joints only (Appendix G.2.9). SAM-Body4D retains training-free evaluation; methods retain their own architectures/training recipes.

Our joint model has the lowest hand M among these Interact methods, 36.36 versus AiOS's 37.54, while supplying upper-body motion. AiOS has the lowest independent-hand PA (6.52), followed by SMPLest-X (7.25); ours is 7.89. This expanded comparison includes strong RGB alternatives and identifies the advantage by metric. Response 2, W2.1 separately compares joint prediction with matched hand-only training.

AuthorsOpenReview ↗

Author rebuttal for Reviewer bLV2

Response 2/3: W2 - joint modeling, component controls, and application

W2.1 Ours Hands and matched hand-only training. Table 1's Ours Hands evaluates hand outputs of the full 57-node joint model; body errors are excluded from that score. Appendix K/Table A28 (pp. 33-34) distinguishes Ours (Table 1) from the separately trained Ours (Hand-Only Training).

We trained/evaluated the new variant on real Interact with seed 42, using RGB and events and the same training, cropping, and evaluation conditions as the joint model. It predicts 21 joints per hand including wrists (42 total). Training/evaluation remove downstream body labels, the body head, body supervision, and shoulder/elbow coordinates. We retain the full official pretrained visual encoder and initialize downstream components from scratch. Crops use causal hand predictions without ground-truth or body-joint inputs (Appendix K).

We additionally report hand-only capacity here. The shared encoder has 631,014,400 parameters; hand-only downstream modules have approximately 4.323M. Full/encoder counts appear in Appendix C, p. 18, lines 926-927. Errors below use the same hand metrics in mm.

ConfigurationOutputParametersHand MHand PA
Joint57 joints; hands scored635,345,70936.367.89
Hand-only42 hand joints635,337,23436.937.33

Counts differ by 8,475 (0.00133%). Joint prediction lowers hand M by 0.57 mm (1.54%) while supplying upper-body motion; hand-only has 0.56 mm lower independent-hand PA. Both placement and aligned articulation are explicitly evaluated.

Without-global-events tests global event features while retaining the body-hand framework; the hand-only experiment separately tests joint prediction. The revised abstract describes joint representation as important for coordinated manipulation (p. 1, lines 018-020) and reports event-input gains versus RGB-only on both subsets (032-034).

Potential robotic application. Our anonymous robot motion-retargeting demonstration illustrates how coordinated upper-body and hand motion can be retargeted for robot control. It provides a qualitative example of the manipulation-oriented use of complete body-hand trajectories, complementing the reconstruction comparisons above.

W2.2 Common crops and component contributions. Full and all component variants use regions from the latest available RGB-predicted joints, fixed until the next RGB update. Auxiliary localization predictions do not determine actual crop centers/scales (Appendix E.1, Configurations, p. 19). This holds region selection constant across variants.

All use seed 42 and k=3: 40 epochs/OakInk2 validation and 30 epochs/Interact test (Sec. 5.4, p. 8). Table 2 (p. 9) reports withheld-RGB EV0 errors over 57 nodes:

ConfigurationOakInk2 MOakInk2 PAInteract MInteract PA
RGB-only23.4014.5341.7925.54
Without global events22.1613.5939.6823.38
Without hand events22.2413.7339.3924.20
Concatenation fusion23.0913.9439.1024.53
Without hand refinement22.4613.7638.5723.11
Full20.9713.5738.6822.72

Against independently trained RGB-only, Full reduces M/PA by 10.38%/6.61% on OakInk2 and 7.44%/11.04% on Interact. Full leads both OakInk2 metrics and Interact PA; without refinement has slightly lower Interact M (38.57 vs. 38.68).

Table 3 gives regional effects. OakInk2 body/left/right M decrease from 25.07/22.59/23.02 to 22.66/20.33/20.39; refinement lowers independent left/right PA from 6.19/6.33 to 5.91/6.11. Interact body/right M decrease from 49.30/43.20 to 42.19/39.06, while left M is 35.79 vs. 35.01. Interact refinement changes left/right PA from 7.90/7.85 to 7.68/8.15.

W2.3 Real lighting and reproducibility. Interact includes physically recorded normal/low-light RGB and events (Appendix J, Tables A22-A25, pp. 31-32). At low-light withheld timestamps, M/PA improve from 40.73/25.02 to 37.16/22.12. Additional exposure perturbations change only RGB and retain native events; low-light EV0 already contains real low-light RGB and events (Sec. 5.3; Appendix J.4).

Independent full-model seeds 42/123/2026 yield Interact EV0/All M of 37.81/37.33/37.09 and PA of 22.22/22.08/21.98 (Appendix M, Tables A30-A31, p. 35). These document full-model reproducibility separately from seed-42 component and hand-only comparisons.

AuthorsOpenReview ↗

Author rebuttal for Reviewer bLV2

Response 3/3: W3 - reference validation and temporal alignment

W3.1 Correcting the published figure. We apologize for the original mischaracterization of 0.8 mm. MAMMA's published held-out-marker errors are 22.481 mm for MAMMA and 21.619 mm for Vicon-based MoSh++ fits; 0.862 mm is their difference. Revised Sec. 3 (p. 4) and Appendix A (p. 15) distinguish this comparison from our independent joint-position validation: 3.4 mm overall root-aligned MPJPE over 57 joints.

W3.2 Sample, hands, hardware, and spatial alignment. Validation includes three participants, each contributing one sequence from each of 30 base action categories, for 90 sequences. Participants wore reflective-marker suits with the manufacturer's 50-marker body layout and additional markers on fingers, fingertips, hand dorsum, and wrists. MoSh++ fits a parametric model to captured 3D marker trajectories; we compare its reference skeleton with corresponding MAMMA joints frame by frame, covering all 57 upper-body/two-hand nodes, including fingers (Appendix A, p. 15, lines 803-809). Setup and marker photographs show the equipment and placement.

Eight LUSTER SWIFT120A cameras operate at 4096 x 3072 and 120 fps, connecting to Motar over Ethernet. Reconstructed results reach the acquisition program at 60 Hz via ZeroMQ/TCP with Protocol Buffers (p. 16, lines 810-813).

After a fixed coordinate transformation and root alignment, Euclidean errors are averaged over corresponding joints and retained timestamps, yielding 3.4 mm root-aligned MPJPE. Root alignment removes global translation and measures relative skeletal agreement (p. 16, lines 829-834).

W3.3 Temporal matching and production alignment. The optical-validation setup uses the Hikvision group's 30 Hz PTP/scheduled-trigger timeline. Device-clock/QPC mapping places optical-frame host reception times on this timeline. Matching uses half the median reception interval, approximately 8.33 ms at 60 Hz; unmatched samples are excluded. This criterion governs reception-time matching; optical acquisition/processing/transmission delays are outside this timing calibration (p. 16, lines 816-824). A camera-frame drop excludes the entire timestamp. Approximately 90% of frames remain usable, and only retained timestamps enter evaluation (lines 824-826).

APS images and events are internally synchronized by the DVSync sensor. Production uses eight RealSense D435i RGB cameras calibrated with a shared board into a common world coordinate system (Appendix A/Fig. 5, p. 14). Host-time alignment is followed by offline frame-by-frame comparison and correction. We observed a maximum 19 ms raw inter-view RGB timestamp discrepancy and an initial approximately 30 ms APS-to-RGB delay. Comparing APS/RGB images estimates and corrects the offset (pp. 14-15); internal APS/event synchronization carries the corrected mapping to the event stream.

We additionally report here a measured residual discrepancy of approximately 10 ms between corrected event-camera data (APS and events) and the multi-view reference, providing near frame-by-frame alignment on the 30 Hz reference timeline. The 19/30 ms values describe initial differences; approximately 10 ms characterizes the achieved correspondence after correction.

W3.4 Annotation rationale and reported statistics. Validation and production share the MAMMA reconstruction algorithm. Independent optical capture provides the validation reference; calibrated, temporally corrected multi-view RGB provides production labels while preserving natural manipulation appearance without suits/gloves. DVSync APS/DVS inputs are excluded from reference fitting (Sec. 3; Appendix A, pp. 14-15). This shared procedure and independent validation support MAMMA's use for complete body-hand references.

The optical report provides the 57-joint aggregate, 90-sequence coverage, and usable-frame proportion. Separate body/left/right, motion-speed and occlusion-stratified error distributions, and exact valid-frame/joint counts are not tabulated. Regional model-prediction errors in Tables 3/A28 are reported separately from optical-reference errors.

Questions for Authors ("See weaknesses"). Please see Response 1, W1.1-W1.2 for joint sets, evaluation conditions, and baseline hand coverage; Response 2, W2.1-W2.3 for matched hand-only training, common crops, component results, real-event evidence, and the robotic application demonstration; W3.1-W3.4 above for reference accuracy, validation protocol, and temporal alignment.

Reviewer_HtJJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EvUBody, a benchmark for reconstructing 3D upper-body and two-hand motion from sparse RGB observations and asynchronous event streams. It combines 627 OakInk2 sequences with events synthesized by v2e and video interpolation, and 1,809 newly recorded EvUBody-Interact sequences from 15 participants with real RGB/event sensing and markerless multi-view 3D references. The proposed reconstruction system uses RGB-derived skeleton anchors and fixed hand regions between retained frames, short- and long-window event features, quality-conditioned RGB–event fusion, and joint-wise spatial/causal temporal attention with wrist-consistent decoding. The authors report lower MPJPE than selected hand-only, body-only, and RGB body–hand baselines, alongside simulated-event ablations at withheld-RGB timestamps (23.40 to 20.97 mm overall MPJPE compared with a separately trained RGB-only model). The main question is whether the evidence actually establishes that real events resolve kinematic ambiguity through joint body–hand modeling, as opposed to offering gains under a favorable synthetic-data protocol and an incompletely controlled comparison.

Critical strengths
  1. The paper addresses a concrete gap between hand-only event reconstruction and full manipulation trajectories. Pairing detailed two-hand and upper-body annotations with both simulated and genuine event observations is valuable. In particular, the real subset contains 1,809 sequences from 15 participants spanning 120 tasks and 12 task categories, with a participant-disjoint 10/2/3 train/validation/test split. The separate multi-view annotation system also avoids using the target DVSync RGB/event input directly to fit the reference poses. If the claimed annotation quality and alignment are documented convincingly, this resource could support a meaningful new benchmark.
  2. The model has an explicit causal observation-access definition and evaluates timestamps at which no current RGB frame is available. Details of event-window boundaries, masks, hand regions, output anatomy, and error aggregation are unusually concrete (Secs. 4–5; Appendices B–E). On the OakInk2 validation split, a separately trained RGB-only baseline scores 23.40 mm MPJPE, versus 20.97 mm for the full RGB–event system at withheld timestamps (Table 2), and interval displacement error decreases from 2.93 to 2.20 mm (Table A5). These are meaningful empirical signals, even though they do not by themselves establish the paper’s broader causal explanation.
Critical weaknesses
  1. The central claim is not convincingly isolated, particularly on real events. The title and abstract claim that jointly modeling the upper body resolves ambiguous event evidence. Yet there is no matched-capacity hand-isolated versus joint body–hand ablation that directly tests this explanation. More importantly, the RGB-only and architecture ablations are reported only on EvUBody-OakInk2 with simulated events (Sec. 5.4; Appendix E.1); a controlled RGB-only comparison on the real-sensor EvUBody-Interact subset is missing. OakInk2 events are generated from the full 30 Hz RGB sequence using SuperSloMo, including interpolation that may access future source frames, before RGB frames are withheld (Sec. 3; Appendix B). This is an acceptable synthetic benchmark design when disclosed, but it cannot alone demonstrate recovery of independently observed physical inter-frame dynamics. Appendix E.1 further states that the four component variants retain an adaptive region policy whereas the full model uses fixed RGB-anchored regions; these comparisons therefore change the region policy as well as the named component. The observed ablation gains cannot be attributed cleanly to the proposed event-fusion and refinement modules. Likewise, the EV−3 to EV+3 exposure tests alter only RGB while keeping events fixed (Appendix E.2), so they are not a physically coupled low-light sensor test. Together these omissions undermine the paper’s principal mechanistic and real-world claims.
  2. Evaluation semantics and ground-truth uncertainty remain insufficiently established. Table 1 describes results as separate “Body” and “Hands” evaluation regions and highlights “Body” MPJPE of 37.80 mm on Interact. However, Appendix Table A17 (EV0, All) reports 37.809 mm as overall 57-node MPJPE and 41.870 mm for its explicit “Body” column. The corresponding SMPLest-X entry of 42.79 mm in Table 1 likewise matches the overall value 42.791 mm in Table A7, not the body-only error 46.733 mm. Thus the Table 1 “Body” label appears to denote a joint/full-output aggregate rather than the 15-node body region advertised in the caption and Sec. 5.2. This must be corrected or reconciled, since it affects the headline anatomical comparison. The real reference annotations are said to achieve 0.8 mm MPJPE against optical motion capture (Sec. 3), but no validation sample size, alignment convention, per-region statistics, occlusion breakdown, or independent protocol is given. Appendix A reports a sampled software grouping skew of approximately 20.45 ms while explicitly stating that it is not a measurement of cross-sensor/exposure synchronization accuracy. These uncertainties matter because the reported overall “Body” margins over strong RGB baselines can be small (20.97 vs. 21.22 mm on OakInk2; 37.80 vs. 39.18 mm on Interact, Table 1). The OakInk2 component results also use a single seed and the validation split, with no multi-seed intervals or independent test split (Appendix E.1). Accordingly, the accuracy and significance of the purported improvements are not yet established to the standard required for the claims.
Questions for authors
  1. Does the benefit of real event sensing survive a matched RGB-only experiment? Please train/evaluate the same architecture and training budget with and without event input on EvUBody-Interact, reporting retained/withheld timestamp performance, normal versus captured low-light conditions, and regional (body/left/right hand) metrics. A hand-isolated model with matched capacity and inputs would directly test the paper’s “joint modeling resolves ambiguity” hypothesis.
  2. What exactly is the “Body” evaluation region in Table 1? Please reconcile Table 1 with Tables A7/A17. For example, is the reported Interact 37.80 mm the 57-node aggregate (37.809 mm in A17), rather than the 15-node body error (41.870 mm in A17)? Provide consistent side-by-side 15-body-node, two-hand, and full-57-node evaluations for all body–hand methods, including a clear definition of PA alignment.
  3. How were the 0.8 mm reference accuracy and true RGB–event temporal alignment established? Please specify the number of recordings/subjects and tested joints, optical mocap configuration, alignment procedure, and per-hand/body plus occlusion-dependent error distributions for the claimed 0.8 mm MPJPE. In addition, report the measured synchronization error between DVSync APS/DVS observations and the multi-view reference timeline, not only the software grouping skew among RGB streams.
6 comments
AuthorsOpenReview ↗

Author rebuttal for Reviewer Htjj

Response 1/4: W1 - joint modeling and real-event controls

Thank you for the detailed feedback. We address W1 in Responses 1-2 and W2 in Responses 3-4; Q1-Q3 refer to these sections in Response 4. Manuscript locations refer to the revised PDF. M/PA denote MPJPE/PA-MPJPE in mm.

W1.1 Matched hand-only training. Recovering coordinated manipulation requires both the arm motion that positions the hands and the finger motion that manipulates the object. Hand-only output leaves torso, shoulder, and elbow motion unspecified, even with accurate hand positions. Our retargeting demonstration illustrates the use of complete body-hand trajectories for robot control.

We extend Table A28's matched comparison to seeds 42, 123, and 2026 on real Interact, using the same RGB/event inputs, training, cropping, and evaluation settings for both models. The hand-only variant predicts 21 joints per hand including wrists (42 total). Its training/evaluation remove downstream body labels, the body prediction head, body supervision, and shoulder/elbow coordinates. Both retain the full official pretrained visual encoder with downstream components initialized from scratch. Hand-only crops use causal hand predictions without ground-truth or body-joint inputs (Appendix K, pp. 33-34).

The shared encoder has 631,014,400 parameters; hand-only downstream modules have approximately 4.323M. Full-model/encoder counts are in Appendix C, p. 18, lines 926-927. We report the additional paired-seed results here on the same 42 hand joints at Interact EV0/All timestamps (mean +/- sample SD); PA independently aligns each hand.

ConfigurationOutputParametersHand MHand PA
Joint57 joints; hands evaluated635,345,70936.175 +/- 0.2217.734 +/- 0.199
Hand-only42 hand joints635,337,23436.782 +/- 0.1497.449 +/- 0.116

Counts differ by 8,475 (0.00133%). Across all three paired seeds, joint prediction lowers hand M, by 0.607 mm on average, while hand PA increases by 0.285 mm. Joint prediction therefore provides upper-body trajectories while consistently lowering unaligned hand error across seeds. Table A28's seed-42 differences were 0.57 mm lower M and 0.56 mm higher PA.

The revised abstract describes joint representation as important for coordinated manipulation (p. 1, lines 018-020), and reports lower overall error with events versus RGB-only on both subsets (lines 032-034; Table 2). The hand-only comparison tests joint prediction on shared hand joints; the RGB-only comparison below tests event assistance between retained RGB observations. Together they evaluate hand accuracy and recovery of complete manipulation trajectories.

W1.2 Matched RGB-only experiment on recorded events. We independently trained RGB-only and component variants on Interact using 30 epochs, seed 42, and the same k=3 sparse-RGB protocol, and evaluated the participant-disjoint test split (Sec. 5.4, p. 8; Tables 2-3, p. 9). RGB-only uses the common framework with the event pathway removed. RGB-only/Full have 634.382M/635.346M parameters, a 0.964M (0.152%) difference (Table A29, p. 34). All configurations use the common crop policy detailed in Response 2, W1.4.

The following EV0 results use original captured observations. Full M/PA cover 57 nodes. Retained/withheld indicate whether the current RGB observation is available (Table A19, p. 29).

ConfigurationAll timestampsRetained RGBWithheld RGB
RGB-only41.13 / 25.1939.80 / 24.5041.79 / 25.54
Full37.81 / 22.2236.06 / 21.2138.68 / 22.72

Events reduce withheld M by 7.44% and PA by 11.04%, with gains also at retained timestamps. These results directly demonstrate event assistance on real sensor recordings when current RGB is unavailable.

Regional withheld results (Table 3, p. 9) use 15 body and 21 nodes per hand; hand PA independently aligns each hand:

ConfigurationBody MLeft MRight MLeft PARight PA
RGB-only49.3035.0143.208.098.17
Full42.1935.7939.067.688.15

Body and right-hand M and both hand PA values improve. Left-hand M is 35.79 versus 35.01 mm. The table identifies the regional contributions to the overall gain. Response 2, W1.5 gives retained/withheld and regional results separately for captured normal and low light.

AuthorsOpenReview ↗

Author rebuttal for Reviewer Htjj

Response 2/4: W1 - event provenance, crop controls, and lighting

W1.3 Simulated and physically recorded events. OakInk2 provides controlled video-derived event experiments; Interact supplies independent sensor observations. OakInk2 events are generated from the complete 30 Hz source video using v2e/SuperSloMo before RGB restriction; interpolation may access subsequent source frames (Sec. 3, p. 4; Appendix B, p. 16). The model's causal access rule governs available RGB and preceding event windows at each target time. Response 1, W1.2 provides the real-event RGB-only comparison, and W1.5 below reports captured-lighting results, including timestamps without current RGB.

W1.4 Common cropping and component comparisons. Full and all ablation configurations use the same RGB-anchored hand regions, derived from the latest available RGB-predicted joints and fixed until the next RGB update. Auxiliary localization predictions do not set actual crop centers/scales (Appendix E.1, Configurations, p. 19). This common policy controls cropping while evaluating the named components. Real Interact withheld results are in Table 2 (p. 9); M/PA are 57-node errors in mm.

ConfigurationMPA
RGB-only41.7925.54
Without global events39.6823.38
Without hand events39.3924.20
Concatenation fusion39.1024.53
Without hand refinement38.5723.11
Full38.6822.72

Full achieves the lowest overall PA. Without refinement has slightly lower M (38.57 versus 38.68). Refinement changes left/right hand PA from 7.90/7.85 to 7.68/8.15 mm (Table 3), documenting its regional effects alongside the overall results.

W1.5 Actual capture lighting and additional RGB perturbations. Interact includes 1,060 low-light and 749 normal-light sequences. Train/validation/test counts are 664/60/336 and 535/178/36, respectively (Appendix J.1-J.2, Table A22, p. 31). Normal light corresponds to the paper's non-low-light label. Both RGB and events were physically recorded under each sequence's illumination.

The following EV0 results use original captured RGB and events. Entries are full 57-node M/PA in mm (Table A25, p. 32).

Capture groupModelAllRetainedWithheld
Low lightRGB-only40.12 / 24.6938.87 / 24.0440.73 / 25.02
Low lightFull36.33 / 21.6434.65 / 20.6637.16 / 22.12
Normal lightRGB-only51.84 / 30.4749.67 / 29.3652.91 / 31.03
Normal lightFull53.45 / 28.3650.97 / 26.9454.68 / 29.07

In captured low light, both metrics improve across all three temporal subsets; withheld M/PA decrease by 8.77%/11.59%. In normal light, PA improves across all subsets, while withheld M is 54.68 versus 52.91 mm. These results directly characterize event assistance under the two recorded conditions.

Withheld regional results (Table A24, p. 32) use 15 body and 21 nodes per hand, with independent hand PA:

GroupModelBody MLeft MRight MLeft PARight PA
LowRGB-only48.2133.3842.757.947.98
LowFull40.9433.9137.727.548.01
NormalRGB-only60.7352.2448.009.7110.16
NormalFull55.4055.6253.229.149.68

Table A23 (p. 31) additionally reports regional errors over all timestamps in each lighting group.

The additional EV-3 to EV+3 test perturbs the RGB input of the full RGB-event model while retaining each sequence's native events and reference labels. EV0 means no additional RGB perturbation; low-light EV0 contains physically captured low-light RGB and events. This evaluates robustness to RGB degradation with the corresponding event observations preserved (Sec. 5.3, p. 8; Appendix E.2, pp. 19-20; Tables A20/A26, pp. 30/33).

The joint RGB/event-count numerical perturbations are reported separately (Appendices H, I.3, J.5; Tables A18/A21/A27).

AuthorsOpenReview ↗

Author rebuttal for Reviewer Htjj

Response 3/4: W2 - evaluation definitions and reproducibility

W2.1 Full versus Body. We apologize for the labeling and reporting errors in Table 1, and thank the reviewer for identifying the inconsistency. We incorrectly labeled the full 57-node evaluation as "Body" and misreported our Full PA-MPJPE.

Revised Table 1 labels our joint entry and the RGB body-hand methods as Full (p. 7). Our Interact 37.809 mm is 57-node MPJPE; the 15-node Body error is 41.870 mm. SMPLest-X has 42.791 mm Full and 46.733 mm Body errors. Table 1 reports rounded Full values 37.81/42.79. Our Full PA is corrected from 19.88 to 22.22 mm, matching 22.218 in Table A17; Sec. 5.2 is updated accordingly.

Below we consolidate Interact EV0/All results from Tables A7-A11 and A15-A17 (pp. 23-28). LME uses its event-only Interact adaptation with 57-node heads (Table 1 caption; Appendix G.2.9, p. 27). All errors are in mm.

MethodBody M (15)Left M (21)Right M (21)Full M (57)
AiOS43.77837.42737.64939.180
SMPLest-X46.73339.37743.38842.791
SAM-Body4D188.394151.452158.444163.750
Hand4Whole++50.86752.66955.29153.161
PEAR79.13280.73383.98681.510
GEM-X68.36145.97650.70153.607
LME (adapted)81.695110.84099.54599.009
Ours41.87034.74837.96937.809

Full PA uses one body-hand alignment; Left/Right PA independently align each hand:

MethodFull PALeft PARight PA
AiOS20.5276.3616.676
SMPLest-X20.6977.0737.432
SAM-Body4D48.24812.91313.572
Hand4Whole++27.78912.61013.529
PEAR30.2968.4558.799
GEM-X32.3759.80510.369
LME (adapted)51.03311.83112.546
Ours22.2187.6578.123

Full comprises 15 body and 21 nodes per hand. Duplicated wrists and derived hand centers retain their scoring weights. MPJPE averages Euclidean position errors over valid nodes. Full PA fits one per-frame similarity transform over valid 57-node sets, with translation, scale, and reflection-free rotation; hand PA independently fits each 21-node hand. Errors and valid-node counts are summed across sequences before division, preserving the specified weighting (Appendix D, pp. 18-19). Appendix K/Table A28 (pp. 33-34) provides the complementary hand-output comparison. These definitions distinguish complete body-hand placement from independently aligned hand articulation.

W2.4 Reproducibility and effect sizes. We added independent full-model training runs with seeds 42, 123, and 2026, keeping all other settings fixed. All are evaluated on the participant-disjoint Interact test split. EV0 results cover all original timestamps and 57 nodes (Appendix M, Table A30, p. 35):

SeedFull MPJPEFull PA-MPJPE
4237.8122.22
12337.3322.08
202637.0921.98
Mean +/- sample SD37.41 +/- 0.3722.09 +/- 0.12

We calculate mean/sample SD from the three displayed results to summarize training-seed variability. All three M values are below AiOS's 39.18 mm Full result under the same EV0/All scope. Table A31 (p. 35) also reports all three seeds across seven exposure settings, documenting reproducibility on real events.

The matched RGB-only comparison in Response 1, W1.2 directly evaluates event assistance within our framework: withheld M/PA decrease by 7.44%/11.04%. Response 2, W1.5 supplies the captured-lighting and regional evidence. These controlled results complement the comparisons against external baselines.

Component ablations use seed 42: 40 epochs with OakInk2 validation evaluation, and 30 epochs with Interact test evaluation (Sec. 5.4, p. 8; Appendix E.1, p. 19). The full-model multi-seed study documents training variability, while fixed-seed ablations examine component effects under common conditions. Response 1, W1.1 additionally reports the paired three-seed joint/hand-only comparison on the same 42 hand joints. W2.2-W2.3 in Response 4 address reference accuracy and temporal alignment.

Reviewer_uWgnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EvUBody for reconstructing upper-body and two-hand joint trajectories from sparse RGB images and events. It combines existing OakInk2 recordings with simulated events and newly captured Interact data with MAMMA-derived annotations. A reconstruction model fuses RGB and event features and jointly predicts body and hand joints. Evaluation focuses on reconstruction errors and recovery between retained RGB frames.

Critical strengths

The real RGB–event recordings paired with coordinated upper-body and detailed hand annotations could be a useful resource for a setting less covered by hand-only datasets. Separately evaluating timestamps without an RGB observation is also relevant to the intended use of events.

Critical weaknesses

Major

  1. Limited methodological novelty. I do not find a sufficiently novel methodological contribution. The model mainly combines familiar components: RGB-guided hand localization, RGB–event feature fusion, and spatial/temporal attention. Quality-conditioned fusion and wrist-consistent body–hand decoding are reasonable design choices, but the paper does not establish a clear algorithmic advance over existing RGB–event and body–hand reconstruction approaches. Separately, the claim that joint modeling resolves motion ambiguity lacks a matched hand-only comparison; component ablations are limited to simulated data and also change cropping policies.

  2. The dataset and evaluation do not yet substantiate the claimed reconstruction fidelity. Interact labels are MAMMA/SMPL-X estimates, so agreement with them does not establish true 3D accuracy. The claimed 0.8 mm validation against optical motion capture lacks sufficient detail, particularly about hand joints, alignment, synchronization, and sample size. Table 1 and Table A17 also need reconciliation: 37.80 mm is labeled as Body in the former, whereas 37.809 mm is the overall error in the latter. These issues weaken the dataset contribution, which is central given the limited methodological novelty. No downstream experiment demonstrates the proposed utility for robot learning.

Minor

The training relationship between the two subsets and the runtime/resource comparison with baselines should be stated more clearly.

Questions for authors
  1. Clarification: What exactly was evaluated in the claimed 0.8 mm optical-motion-capture validation: how many subjects/frames, which joints (including fingers?), and what spatial alignment and temporal synchronization?
  2. Clarification: How do the settings and evaluated joints in Table 1 correspond to Table A17? Were models trained separately on the two subsets, jointly, or through pretraining and fine-tuning?
  3. Evidence needed for the central claim: Is there a real-data comparison of hand-only versus joint body–hand prediction with matched inputs, capacity, training data, and cropping? Such evidence could materially change my assessment.
8 comments
AuthorsOpenReview ↗

Author rebuttal for Reviewer uWgn

Response 1/4: Major weakness 1 - contribution and controlled comparisons

Thank you for the constructive feedback. We address your concerns in four parts; manuscript locations refer to the revised version. This part also answers Q3.

Contribution and task scope. Our primary contribution is a dataset, task, and evaluation framework for reconstructing the complete upper body and both hands from RGB and events, together with an effective, reproducible reference baseline (Introduction/contributions, pp. 2-3; Sec. 3, p. 4). To our knowledge, EvUBody is the first dataset and benchmark dedicated to this setting. For manipulation, arms determine the workspace and transport the hands, while fingers perform fine interactions; a joint representation captures this coordinated motion across the upper body and hands (Introduction, p. 1, lines 038-047). Our model predicts all 57 joints (Sec. 4.4, p. 6; Appendix D, p. 18).

Matched hand-only comparison. We trained a hand-only variant on real EvUBody-Interact with seed 42, using RGB and events and the same training, cropping, and evaluation conditions as the joint model. It predicts 21 joints per hand, including wrists (42 total). Throughout training/evaluation, we remove downstream body labels, the body prediction head, body supervision, and shoulder/elbow coordinates. We retain the full official pretrained visual encoder and initialize downstream components from scratch. Crops use causal hand predictions without ground-truth or body-joint inputs. The variant and results are in Appendix K, pp. 33-34, Table A28.

We additionally report the hand-only parameter count here. The shared encoder has 631,014,400 parameters; hand-only downstream modules have approximately 4.323M. Full-model/encoder counts appear in Appendix C, p. 18, lines 926-927.

ConfigurationOutputTotal parametersHand MPJPEHand PA-MPJPE
Joint57 joints; scored on hands635,345,70936.367.89
Hand-only42 hand joints635,337,23436.937.33

Errors are in mm. Parameter counts differ by only 8,475 (0.00133%). Joint prediction lowers hand MPJPE by 0.57 mm (1.54%), while hand-only lowers PA-MPJPE by 0.56 mm. PA independently aligns each hand. Thus, joint prediction maintains competitive hand accuracy while also providing upper-body motion, with modestly lower unaligned error in this run. This comparison supports joint prediction as a reference model for the complete task and makes the trade-off between unaligned and independently aligned hand accuracy explicit.

Abstract clarification. The revised abstract describes joint representation as important for capturing coordinated manipulation (p. 1, lines 018-020). Its empirical claim is that incorporating events reduces overall error versus RGB-only on both subsets under the sparse-RGB protocol (lines 032-034; Table 2, p. 9). The RGB-only/full-model comparison supports this statement; the matched hand-only experiment separately evaluates the role of joint prediction.

Relation to existing methods. To our knowledge, no prior method jointly reconstructs both regions using RGB and events. We adapt representative RGB-event hand, event-only, and RGB body-hand methods using official implementations/training recipes, distinguishing input modalities, output coverage, and evaluated regions (Secs. 5.1-5.2, pp. 6-8; Table 1, p. 7).

The following dedicated-hand comparison evaluates the same hand joints only; body joints do not enter these errors. Values are MPJPE / PA-MPJPE in mm (Table 1; expanded Interact comparison: Appendix K, Table A28).

MethodOutputOakInk2Interact
EvRGBHandHands46.92 / 14.4267.02 / 10.73
Ev2HandsHands37.87 / 9.78130.83 / 12.73
EventHandsHands28.28 / 7.88105.39 / 10.91
OursBody + hands; scored on hands20.21 / 12.8436.36 / 7.89

Our model lowers hand MPJPE by 28.5%/45.7% versus the strongest dedicated hand baseline on OakInk2/Interact, respectively, and has lower Interact hand PA-MPJPE than these three baselines, while also reconstructing the upper body.

Real-data ablations with fixed cropping. We added Interact ablations using the same RGB-guided hand-region selection and cropping policy across configurations (Sec. 5.4, p. 8; Tables 2-3, p. 9; cropping policy: Appendix E.1, p. 19). At withheld-RGB timestamps, full-model MPJPE decreases from 41.79 to 38.68 mm and PA-MPJPE from 25.54 to 22.72 mm versus RGB-only (7.44% and 11.04%). These results support event assistance on recorded data under a common cropping policy.

AuthorsOpenReview ↗

Author rebuttal for Reviewer uWgn

Response 2/4: Major weakness 2 - independent optical validation

This part answers Q1 and the annotation-validation concern. Production capture, table consistency, and robotic relevance are addressed in Response 3/4.

Independent validation and metric clarification. We assess the reliability of MAMMA reference skeletons through an independent optical motion-capture experiment, which yields 3.4 mm overall root-aligned MPJPE. We have also clarified the original 0.862 mm value: it is the difference between MAMMA's and Vicon-based MoSh++ fits' mean held-out-marker reconstruction errors in the original MAMMA evaluation. The revised manuscript reports this published error difference and our skeleton-to-skeleton joint-position metric separately (Sec. 3, "Multi-view reference annotation," p. 4; Appendix A, "Published MAMMA accuracy and metric interpretation," p. 15).

Participants, actions, and reference construction. We set up optical motion capture in a larger space and evaluated three participants wearing motion-capture suits with reflective markers. Each participant contributes one sequence from each of 30 base action categories: 90 sequences in total. Following the marker-based reference construction used in MAMMA's evaluation, we fit a parametric body model to the captured 3D marker trajectories with MoSh++ (AMASS, 2019). We compare the resulting reference poses with MAMMA reconstructions frame by frame over all 57 upper-body and hand skeletal joints. These details are in Appendix A, "Independent optical motion-capture validation" (p. 15, lines 803-809).

Body and finger coverage. Body placement follows the manufacturer's FZMotion body motion-capture suit marker-placement manual, v1.0: 50 markers covering the head, trunk, upper limbs, pelvis, and lower limbs. Additional hand markers cover the dorsal finger surfaces, fingertips, hand dorsum, and wrist region. We use the physical markers to fit reference poses and evaluate 57 corresponding skeletal joints from the MAMMA and marker-based reconstructions. The layout/coverage and fitting procedure are described on p. 15, lines 806-809. Four individually captioned photographs show the equipment, capture space, and placement: Supplementary setup and marker-placement photos.

Hardware and acquisition. Eight LUSTER SWIFT120A cameras operate at 4096 x 3072 and 120 fps, connected over Ethernet to Motar for 3D marker reconstruction and pose estimation. Results reach our acquisition program at 60 Hz through ZeroMQ/TCP with Protocol Buffers encoding; camera and transmitted-output rates are distinct (Appendix A, p. 16, lines 810-813).

Temporal matching and valid frames. The reference timeline is the Hikvision camera group's 30 Hz acquisition timeline, established with IEEE 1588 PTP and scheduled triggering. Mapping between device clocks and QPC (the host's high-resolution monotonic clock) converts motion-capture host reception times to this timeline. The base tolerance is half the median reception interval, approximately 8.33 ms at 60 Hz; samples failing the matching criteria are excluded. This is a reception-time matching tolerance. Acquisition, processing, and transmission delays have not been independently calibrated, so it is not measured cross-device physical synchronization accuracy.

For any timestamp affected by a dropped camera frame, all associated data are excluded. Approximately 90% of frames remain usable; only retained valid timestamps enter the error calculation. Timing and filtering are specified in Appendix A, "Validation timing and sample matching" (p. 16, lines 816-826).

Spatial alignment, error, and scope. After a fixed coordinate transformation and root alignment, we average Euclidean distances between corresponding MAMMA and MoSh++-fitted skeletal joints over all retained valid timestamps and all 57 joints, obtaining 3.4 mm overall root-aligned MPJPE. Root alignment removes global translation, so this metric evaluates relative skeletal joint-position agreement under the stated validation/matching protocol. We report it separately from the published held-out-marker errors, which evaluate marker positions. The result appears in Sec. 3 (p. 4) and Appendix A, "Validation error and scope" (p. 16, lines 829-834).

AuthorsOpenReview ↗

Author rebuttal for Reviewer uWgn

Response 3/4: Major weakness 2 continued; subset training

Why use visual reference annotation? Our choice of MAMMA considers complete body/hand labels and natural capture appearance. In the marker layouts and estimation pipeline we tested, simultaneously maintaining stable body tracking and fine-grained hand labels was difficult. Combining motion-capture gloves with body tracking also produced wrist-connection biases, including inward bending. Gloves obscure bare-hand appearance and introduce a visual domain difference from everyday activities. Production annotation therefore uses multi-view RGB without suits or gloves. MAMMA runs offline at approximately 1.33 s/frame on one NVIDIA RTX PRO 6000 GPU (Appendix A, "Annotation choice and offline cost," p. 15).

Production capture and spatial calibration. Eight Intel RealSense D435i cameras provide production RGB video. A shared calibration board establishes a common world coordinate system: it is simultaneously observed by multiple cameras, with calibration observations collectively covering all cameras to establish cross-view relationships (Appendix A, "Sensors, spatial calibration, and production synchronization," and Fig. 5, p. 14).

Production temporal alignment. RGB uses host-time-based software synchronization. The maximum timestamp difference among the eight raw RGB streams in our tests was 19 ms. Offline frame-by-frame comparison and temporal correction establish multi-view correspondences and reduce misalignment effects on reconstruction. The event camera also uses host-time-based software synchronization. We observed an initial APS-to-RGB delay of approximately 30 ms and estimated/corrected this offset by comparing APS and RGB images frame by frame, establishing event-to-reference-label correspondence. The 19 ms and approximately 30 ms figures are pre-correction RGB timestamp differences and cross-modal delay, respectively; neither is a post-correction residual synchronization error (Sec. 3, p. 4; Appendix A, pp. 14-15).

How validation supports production annotation. Validation and production use the same MAMMA algorithm to reconstruct upper-body and hand skeletons from multi-view RGB. The independent optical experiment evaluates reconstruction against a marker-based reference, yielding 3.4 mm root-aligned MPJPE over all 57 joints. For production, common spatial calibration and offline temporal correction establish cross-view and cross-modal correspondences for the same reconstruction algorithm, while visual capture preserves natural appearance without suits or gloves. This design combines independent validation with a common reconstruction procedure, providing an empirical basis for using MAMMA to generate Interact's body-hand reference annotations (Appendix A, pp. 14-16).

Table 1/Table A17 correspondence. We corrected our complete body-hand evaluation label from "Body" to "Full" in Table 1 and defined its scope in the caption. Our Interact Full entry corresponds to Table A17's EV0 / All row (Appendix G.2.11, p. 28), covering all 57 joints. Table 1 now reports 37.81 mm MPJPE and 22.22 mm PA-MPJPE, matching 37.809 and 22.218 mm after rounding. The erroneous 19.88 mm PA-MPJPE is corrected, and Sec. 5.2's discussion is updated (p. 7).

Robotic relevance. Our robot motion-retargeting demonstration illustrates how coordinated upper-body and hand motion can be retargeted for robot control, providing a qualitative example of the manipulation-oriented use of a joint body-hand representation.

Subset training (minor weakness). Models are trained separately on EvUBody-OakInk2 and EvUBody-Interact and evaluated on their respective test sets, without pooled training or pretraining on one subset followed by fine-tuning on the other. Separate training/evaluation is explicit in Sec. 5.2 (p. 7). Component ablations use 40 training epochs with OakInk2 validation evaluation and 30 epochs with Interact test evaluation (Sec. 5.4, p. 8). Together with the table-correspondence paragraph above, this answers Q2. Runtime/resources are addressed in Response 4/4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.