- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EvUBody introduces a dataset and a causal framework for joint upper-body and two-hand reconstruction from sparse RGB frames and event streams. The model combines RGB anchors, global and local event features, quality-conditioned fusion, and joint spatiotemporal attention. The evaluation covers simulated and recorded events, with an RGB-only ablation examining the contribution of event information.
Critical strengths
- The dataset addresses a useful gap in capturing coordinated arm and finger motion during natural manipulation. The real subset uses independent multiview RGB references while preserving an uninstrumented appearance (Section 3).
- The structured decoder is well motivated: expressing hand joints relative to the predicted body wrist guarantees consistent wrist positions by construction (Eq. 3).
- Evaluating timestamps without current RGB observations directly addresses inter-frame reconstruction. Under this protocol, the full model reduces MPJPE from 23.40 to 20.97 mm relative to the independently trained RGB-only baseline (Table 2).
Critical weaknesses
-
Evaluation scope and baseline coverage need clarification. Table 1 labels the proposed method's OakInk2 result of 20.97 mm as “Body,” whereas Table 2 reports the same value for the full 57-node representation and Table 3 reports a body error of 22.66 mm. It is unclear whether these results use the same evaluation conditions. Moreover, AiOS [Sun et al., CVPR 2024] and SMPLest-X [Yin et al., TPAMI 2025] support hand reconstruction but are evaluated only in the Body group of Table 1. Consequently, the hand comparison excludes strong RGB alternatives. Please define the joint sets and evaluation conditions consistently and report hand results for all applicable baselines before drawing broad superiority claims.
-
The experiments do not clearly isolate the central mechanism or component contributions. The title and abstract attribute ambiguity resolution to joint upper-body–hand modeling, but removing global events does not remove body context. A matched joint-versus-hand-only comparison using the same hand metrics is needed. The “Ours Hands” row in Table 1 does not establish this comparison because its relationship to the joint model is unclear. Furthermore, Appendix E.1 states that the full model uses fixed RGB-anchored regions while the four component variants retain an adaptive region policy. This confounds the component comparisons in Section 5.4. These variants should be compared under the same region policy.
-
The reference-accuracy claim requires a reproducible validation protocol. Section 3 reports 0.8 mm MPJPE against optical motion capture, but the manuscript does not adequately specify the validation sample, evaluated joints, alignment procedure, or hand-specific accuracy. These details are important because reference quality is central to the dataset contribution, particularly during fast, occluded manipulation. Please provide an independent validation with separate body and hand errors, error distributions, and a description of temporal alignment.
Questions for authors
See weaknesses.
5 comments
Author rebuttal for Reviewer bLV2
Response 2/3: W2 - joint modeling, component controls, and application
W2.1 Ours Hands and matched hand-only training. Table 1's Ours Hands evaluates hand outputs of the full 57-node joint model; body errors are excluded from that score. Appendix K/Table A28 (pp. 33-34) distinguishes Ours (Table 1) from the separately trained Ours (Hand-Only Training).
We trained/evaluated the new variant on real Interact with seed 42, using RGB and events and the same training, cropping, and evaluation conditions as the joint model. It predicts 21 joints per hand including wrists (42 total). Training/evaluation remove downstream body labels, the body head, body supervision, and shoulder/elbow coordinates. We retain the full official pretrained visual encoder and initialize downstream components from scratch. Crops use causal hand predictions without ground-truth or body-joint inputs (Appendix K).
We additionally report hand-only capacity here. The shared encoder has 631,014,400 parameters; hand-only downstream modules have approximately 4.323M. Full/encoder counts appear in Appendix C, p. 18, lines 926-927. Errors below use the same hand metrics in mm.
| Configuration | Output | Parameters | Hand M | Hand PA |
|---|---|---|---|---|
| Joint | 57 joints; hands scored | 635,345,709 | 36.36 | 7.89 |
| Hand-only | 42 hand joints | 635,337,234 | 36.93 | 7.33 |
Counts differ by 8,475 (0.00133%). Joint prediction lowers hand M by 0.57 mm (1.54%) while supplying upper-body motion; hand-only has 0.56 mm lower independent-hand PA. Both placement and aligned articulation are explicitly evaluated.
Without-global-events tests global event features while retaining the body-hand framework; the hand-only experiment separately tests joint prediction. The revised abstract describes joint representation as important for coordinated manipulation (p. 1, lines 018-020) and reports event-input gains versus RGB-only on both subsets (032-034).
Potential robotic application. Our anonymous robot motion-retargeting demonstration illustrates how coordinated upper-body and hand motion can be retargeted for robot control. It provides a qualitative example of the manipulation-oriented use of complete body-hand trajectories, complementing the reconstruction comparisons above.
W2.2 Common crops and component contributions. Full and all component variants use regions from the latest available RGB-predicted joints, fixed until the next RGB update. Auxiliary localization predictions do not determine actual crop centers/scales (Appendix E.1, Configurations, p. 19). This holds region selection constant across variants.
All use seed 42 and k=3: 40 epochs/OakInk2 validation and 30 epochs/Interact test (Sec. 5.4, p. 8). Table 2 (p. 9) reports withheld-RGB EV0 errors over 57 nodes:
| Configuration | OakInk2 M | OakInk2 PA | Interact M | Interact PA |
|---|---|---|---|---|
| RGB-only | 23.40 | 14.53 | 41.79 | 25.54 |
| Without global events | 22.16 | 13.59 | 39.68 | 23.38 |
| Without hand events | 22.24 | 13.73 | 39.39 | 24.20 |
| Concatenation fusion | 23.09 | 13.94 | 39.10 | 24.53 |
| Without hand refinement | 22.46 | 13.76 | 38.57 | 23.11 |
| Full | 20.97 | 13.57 | 38.68 | 22.72 |
Against independently trained RGB-only, Full reduces M/PA by 10.38%/6.61% on OakInk2 and 7.44%/11.04% on Interact. Full leads both OakInk2 metrics and Interact PA; without refinement has slightly lower Interact M (38.57 vs. 38.68).
Table 3 gives regional effects. OakInk2 body/left/right M decrease from 25.07/22.59/23.02 to 22.66/20.33/20.39; refinement lowers independent left/right PA from 6.19/6.33 to 5.91/6.11. Interact body/right M decrease from 49.30/43.20 to 42.19/39.06, while left M is 35.79 vs. 35.01. Interact refinement changes left/right PA from 7.90/7.85 to 7.68/8.15.
W2.3 Real lighting and reproducibility. Interact includes physically recorded normal/low-light RGB and events (Appendix J, Tables A22-A25, pp. 31-32). At low-light withheld timestamps, M/PA improve from 40.73/25.02 to 37.16/22.12. Additional exposure perturbations change only RGB and retain native events; low-light EV0 already contains real low-light RGB and events (Sec. 5.3; Appendix J.4).
Independent full-model seeds 42/123/2026 yield Interact EV0/All M of 37.81/37.33/37.09 and PA of 22.22/22.08/21.98 (Appendix M, Tables A30-A31, p. 35). These document full-model reproducibility separately from seed-42 component and hand-only comparisons.
Author rebuttal for Reviewer bLV2
Response 3/3: W3 - reference validation and temporal alignment
W3.1 Correcting the published figure. We apologize for the original mischaracterization of 0.8 mm. MAMMA's published held-out-marker errors are 22.481 mm for MAMMA and 21.619 mm for Vicon-based MoSh++ fits; 0.862 mm is their difference. Revised Sec. 3 (p. 4) and Appendix A (p. 15) distinguish this comparison from our independent joint-position validation: 3.4 mm overall root-aligned MPJPE over 57 joints.
W3.2 Sample, hands, hardware, and spatial alignment. Validation includes three participants, each contributing one sequence from each of 30 base action categories, for 90 sequences. Participants wore reflective-marker suits with the manufacturer's 50-marker body layout and additional markers on fingers, fingertips, hand dorsum, and wrists. MoSh++ fits a parametric model to captured 3D marker trajectories; we compare its reference skeleton with corresponding MAMMA joints frame by frame, covering all 57 upper-body/two-hand nodes, including fingers (Appendix A, p. 15, lines 803-809). Setup and marker photographs show the equipment and placement.
Eight LUSTER SWIFT120A cameras operate at 4096 x 3072 and 120 fps, connecting to Motar over Ethernet. Reconstructed results reach the acquisition program at 60 Hz via ZeroMQ/TCP with Protocol Buffers (p. 16, lines 810-813).
After a fixed coordinate transformation and root alignment, Euclidean errors are averaged over corresponding joints and retained timestamps, yielding 3.4 mm root-aligned MPJPE. Root alignment removes global translation and measures relative skeletal agreement (p. 16, lines 829-834).
W3.3 Temporal matching and production alignment. The optical-validation setup uses the Hikvision group's 30 Hz PTP/scheduled-trigger timeline. Device-clock/QPC mapping places optical-frame host reception times on this timeline. Matching uses half the median reception interval, approximately 8.33 ms at 60 Hz; unmatched samples are excluded. This criterion governs reception-time matching; optical acquisition/processing/transmission delays are outside this timing calibration (p. 16, lines 816-824). A camera-frame drop excludes the entire timestamp. Approximately 90% of frames remain usable, and only retained timestamps enter evaluation (lines 824-826).
APS images and events are internally synchronized by the DVSync sensor. Production uses eight RealSense D435i RGB cameras calibrated with a shared board into a common world coordinate system (Appendix A/Fig. 5, p. 14). Host-time alignment is followed by offline frame-by-frame comparison and correction. We observed a maximum 19 ms raw inter-view RGB timestamp discrepancy and an initial approximately 30 ms APS-to-RGB delay. Comparing APS/RGB images estimates and corrects the offset (pp. 14-15); internal APS/event synchronization carries the corrected mapping to the event stream.
We additionally report here a measured residual discrepancy of approximately 10 ms between corrected event-camera data (APS and events) and the multi-view reference, providing near frame-by-frame alignment on the 30 Hz reference timeline. The 19/30 ms values describe initial differences; approximately 10 ms characterizes the achieved correspondence after correction.
W3.4 Annotation rationale and reported statistics. Validation and production share the MAMMA reconstruction algorithm. Independent optical capture provides the validation reference; calibrated, temporally corrected multi-view RGB provides production labels while preserving natural manipulation appearance without suits/gloves. DVSync APS/DVS inputs are excluded from reference fitting (Sec. 3; Appendix A, pp. 14-15). This shared procedure and independent validation support MAMMA's use for complete body-hand references.
The optical report provides the 57-joint aggregate, 90-sequence coverage, and usable-frame proportion. Separate body/left/right, motion-speed and occlusion-stratified error distributions, and exact valid-frame/joint counts are not tabulated. Regional model-prediction errors in Tables 3/A28 are reported separately from optical-reference errors.
Questions for Authors ("See weaknesses"). Please see Response 1, W1.1-W1.2 for joint sets, evaluation conditions, and baseline hand coverage; Response 2, W2.1-W2.3 for matched hand-only training, common crops, component results, real-event evidence, and the robotic application demonstration; W3.1-W3.4 above for reference accuracy, validation protocol, and temporal alignment.
Author rebuttal for Reviewer bLV2
Response 1/3: W1 - evaluation scope and hand-baseline coverage
Thank you for recognizing the dataset, wrist-consistent decoder, and evaluation without current RGB. We address the three weaknesses in three responses. Manuscript locations refer to the revised version; M/PA denote MPJPE/PA-MPJPE in mm.
W1.1 Corrected labels and evaluation conditions. We apologize for the labeling and reporting errors in Table 1, and thank the reviewer for identifying the inconsistency. 20.97 mm on OakInk2 is Full MPJPE over 57 nodes. Revised Table 1 labels our joint entry and the RGB body-hand entries as Full (p. 7): 15 body nodes plus 21 per hand, including duplicated wrists and derived hand centers with their scoring weights (Appendix D, pp. 18-19).
For our model, Table 1's 20.97 is the Full result detailed in Table 2; Table 3 gives its regional breakdown. These measurements use the same checkpoint and conditions: OakInk2 validation, withheld RGB, EV0, k=3, phase zero, seed 42, and the final epoch-40 checkpoint (Sec. 5.4, p. 8; Appendix E.1, p. 19).
Thus, 20.97/22.66 describe the complete skeleton/body region under the same ablation protocol. MPJPE sums Euclidean errors and valid-node counts across sequences before division. Full PA fits one per-frame similarity transform over valid 57-node sets, including translation, scale, and reflection-free rotation; hand PA independently fits each 21-node hand (Appendix D).
Temporal subsets. Table A4 (p. 20) reports OakInk2 Full M of 20.81/20.50/20.97 over All/Retained/Withheld timestamps. Interact Table 1 reports Full M/PA of 37.81/22.22 at EV0/All (Table A17, p. 28); Table 2 reports 38.68/22.72 at Withheld timestamps. Our Interact Full PA is corrected from 19.88 to 22.22, consistent with the appendix. These scopes distinguish anatomy from RGB availability.
W1.2 Expanded hand comparison. Appendix K/Table A28 (pp. 33-34) includes AiOS, SMPLest-X, and other RGB methods with articulated hand outputs. We reproduce the Interact results below and additionally summarize the adapted LME hand output from Table A15 (p. 27). RGB-containing methods use EV0/All; event-only methods use Event/All. Each hand has 21 evaluated nodes and independent PA alignment.
Hand summaries equally average the reported left/right errors; Full uses valid-node aggregation. LME is calculated from Table A15: (110.840+99.545)/2 for M and (11.831+12.546)/2 for PA. Its Interact adaptation predicts 57 nodes; original OakInk2 LME predicts body joints only (Appendix G.2.9). SAM-Body4D retains training-free evaluation; methods retain their own architectures/training recipes.
Our joint model has the lowest hand M among these Interact methods, 36.36 versus AiOS's 37.54, while supplying upper-body motion. AiOS has the lowest independent-hand PA (6.52), followed by SMPLest-X (7.25); ours is 7.89. This expanded comparison includes strong RGB alternatives and identifies the advantage by metric. Response 2, W2.1 separately compares joint prediction with matched hand-only training.