- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
IMU4D predicts human motion, activity descriptions, and interacting object categories and poses from sparse wearable IMU signals. It uses a pretrained language-model backbone with modality-specific encoders and decoders. Root motion is represented through locally normalized and quantized increments, while body poses are regressed continuously. The model is trained on mixed datasets and then fine-tuned for individual tasks. Experiments evaluate motion reconstruction, captioning, and static and dynamic object prediction, including motion reconstruction from real IMU recordings.
Critical strengths
-
I find the extension from IMU-based motion capture to human–object understanding interesting. Predicting activity descriptions and interacting objects adds useful context to the recovered motion. The proposed model connects these tasks through shared representations and can use datasets that provide only some of the required annotations.
-
The motion results on real IMU data are convincing. Under the three-sensor setting, MPJPE decreases from 62.91 to 47.31 mm on DIP-IMU and from 61.96 to 42.45 mm on IMUPoser compared with MobilePoser. I also appreciate the comparison in Table 8: using the same architecture and pretraining for joint, single-task, and cascaded models helps isolate the benefit of joint training. The sensor-count and real-data fine-tuning experiments provide useful information about how performance changes with the available inputs and training data.
Critical weaknesses
-
Appendix E.1 states that some HumanML3D test clips have mirrored counterparts in the training set. These are closely related sequences, so the HumanML results may overestimate generalization to unseen motions. Training all methods on the same split does not resolve this concern, since different models may benefit differently from the overlap. Several main results and ablations rely on HumanML. Please report the number of affected test clips and repeat the key comparisons with each original motion and its mirrored variants assigned to the same split.
-
The comparison with real-time baselines needs clearer control of temporal context and inference cost. IMU4D processes the input sequence with bidirectional motion queries and averages predictions from two shifted inference passes, while MobilePoser is designed for real-time operation. It is unclear whether the evaluated methods have the same access to future frames. A comparison with matched temporal context and an ablation of shifted-pass averaging would help identify how much of the gain comes from these choices. Inference time and memory usage should also be reported to assess the feasibility of the proposed wearable applications.
Questions for authors
-
The main object results use five-point synthetic IMU inputs, while the real-device experiments use three sensors and report motion and text metrics. Do you have quantitative object-category or object-pose results for real three-sensor inputs? Please also provide the number and duration of sequences in IMU4D-Real, explain whether training and test sets share participants or recording sessions, and describe how the accuracy of the GVHMR-derived reference trajectories was checked.
-
For the object benchmarks, are test participants, object instances, or motion–object combinations unseen during training, or are the splits only at the sequence level? Similar body motions can correspond to different objects. Results on held-out combinations, or a comparison with a simple activity-conditioned object predictor, would help show what the model learns beyond common motion–object associations.
3 comments
W2 — Access to future frames, two-pass averaging, and cost.
All methods see exactly the same past and future frames, because every method runs offline on the same input window. IMU4D's accuracy advantage does not come from averaging two shifted passes. On long windows IMU4D runs faster than most baselines, but it needs more GPU memory.
Every method receives the same input window (60 or 480 frames). The pose networks of IMUPoser and MobilePoser are bidirectional LSTMs and IMU4D uses bidirectional attention, so all of them can use both earlier and later frames. As suggested, we test IMU4D with and without two-pass averaging (Table R10) and measure speed and memory (Table R11). With a single pass, IMU4D's error is still 35% lower than the XL baselines; averaging reduces the error by only 0.7 mm (38.95 → 38.23 mm; on LINGO: 38.61 → 38.18 mm). We have added both tables to the revised paper (Tables 15 and 16). Thank you for the suggestion.
Table R10. Effect of averaging two shifted passes (clean test set, 3 sensors, 60 frames). (Revised paper: Table 15.)
| Method | MPJPE (mm) ↓ | PA-MPJPE (mm) ↓ |
|---|---|---|
| IMUPoser-XL | 60.31 | 47.64 |
| MobilePoser-XL | 60.66 | 49.23 |
| IMU4D, one pass | 38.95 | 33.35 |
| IMU4D, average of two shifted passes (as in Table R7) | 38.23 | 32.81 |
Table R11. Speed and memory (3 sensors, one sequence at a time, one RTX PRO 6000 GPU). (Revised paper: Table 16.) Baselines run in 32-bit floating point; IMU4D's language-model backbone runs in bfloat16. We report the median time per input window and the peak GPU memory.
| Method | Parameters (millions) | Time, 60 frames (ms) | Time, 480 frames (ms) | GPU memory (GiB) |
|---|---|---|---|---|
| IMUPoser | 10.67 | 1.58 | 10.25 | 0.17 |
| IMUPoser-XL | 164.44 | 11.57 | 88.16 | 1.90 |
| MobilePoser | 6.67 | 9.90 | 64.98 | 0.40 |
| MobilePoser-XL | 162.89 | 17.85 | 128.01 | 1.26 |
| IMU4D | 1511.35 | 17.75 | 35.67 | 13.72 |
Times cover motion reconstruction (including two-pass averaging), not caption generation. IMU4D processes the whole window at once, so an 8× longer input only doubles its running time, while the LSTM baselines process frames one after another and become 6–8× slower; on 480-frame windows IMU4D is faster than IMUPoser-XL, MobilePoser and MobilePoser-XL. We state in the revised paper that the wearable sensors stream their data to a GPU; these numbers do not show that IMU4D can run on the device itself or in real time.
Q1 — Real-data objects, dataset size, splits, and reference quality.
We do not have quantitative results for object category or object pose on real IMU input, because the real IMU recordings have no verified object labels. Our object metrics therefore come from datasets with object annotations, where the IMU signals are synthesized. On a new test with an unseen participant, IMU4D still outperforms both baselines.
IMU4D-Real data details. We provide the number of recordings, the total duration, the split, and the reference-motion checks of IMU4D-Real below (see also our response to Reviewer QyV8 Q6).
- Number of recordings and duration. 48 recordings (about 1 hour) from 2 participants:
- 15 meeting-room sessions with multi-view reference motion (about 34 minutes): 10 with three sensors (20.4 min train / 2.8 min test) and 5 with two sensors (9.5 min train / 1.1 min test).
- 33 single-camera recordings (about 30 minutes), used only for training and supervised on local joint angles only.
- Data splits. The original test set is a held-out time segment of each session, separated from the training segments by a 2-second gap. Training and test therefore share participants and sessions, so we add a test on an unseen participant below.
- Reference motion. We combine GVHMR estimates from three calibrated static cameras, visually check that the result projects correctly into all three views, and remove time intervals where it fails. We also check the alignment in the initial T-pose and that body height stays consistent while seated. These checks are not an independent measurement of accuracy during motion.
Test on an unseen participant. As suggested, we train on one participant and test on the other, so that no participant or session is shared. All three methods are fine-tuned for the same fixed 2,000 steps on the same 80 clips from participant 1, and tested on 22 clips from participant 2. Both sides use only the phone and the watch (2 sensors), since participant 2 has no usable earbud data. IMU4D starts from a checkpoint that train only on synthetic data; the baselines start from their pretrained checkpoints and keep their original training losses. Participant 2 is never used to choose checkpoints.
Table R12. Unseen-participant test (2 sensors, 22 test clips cut into 320 windows of 60 frames). (Revised paper: Table 17.)
| Method | MPJPE (mm) ↓ | PA-MPJPE (mm) ↓ | MPJRE (°) ↓ | MTE (mm) ↓ |
|---|---|---|---|---|
| IMU4D | 83.82 | 76.45 | 15.61 | 223.89 |
| IMUPoser | 92.37 | 84.93 | 16.13 | — |
| MobilePoser | 91.80 | 84.31 | 17.11 | 337.55 |
IMU4D has the lowest error on every metric for the unseen participant, with a clear margin on position and translation and a comparable joint rotation error (15.61° vs. 16.13°); 480-frame windows give the same ranking (MPJPE 82.58 vs. 92.66 and 92.20 mm). IMUPoser does not predict body translation. We have added the dataset statistics and this experiment to the revised paper (Appendix E.4 and A.5, Tables 24 and 17). Thank you for the question.
Q2 — Object benchmark splits and a retrieval baseline.
We summarize below which test participants and object instances are seen during training.
- Participant splits: whether test participants are seen/unseen.
- HiPHI: unseen. 48 for training, and 10 for testing.
- OMOMO: unseen. 15 for training, and 2 for testing.
- HUMOTO: seen, but test sequences are unseen. All motions were recorded by a single actor, thus the participant split is not feasible. We split by sequence instead (685 for training, and 50 for testing).
- Object splits: all categories are seen during training. IMU4D predicts objects from a closed vocabulary; unseen categories are outside the scope of this paper.
To test whether IMU4D does more than recall common motion–object pairs, we compare it with three baselines on HiPHI (500 test clips; object-category F1):
- Always predicting the most common training object gives 35.80%.
- Nearest-neighbor retrieval, which uses the signals from the same five IMU sensors to retrieve the most similar training clip and copies its objects, gives 36.20%.
- A motion-conditioned object predictor based on a bidirectional LSTM gives 43.4%.
- IMU4D, using the same five sensors, reaches 45.85%, higher than all three.
We have added the split details and this comparison to the revised paper (Appendix E.1 and B.5). Thank you for the suggestion.
W1 — HumanML mirrored overlap.
Yes. 2,000 of the HumanML test clips (which is a subset of MotionMillions) we evaluated are affected: each has a mirrored copy in the training data of the official MotionMillion split (Fan et al., 2025, Go to Zero, ICCV). We re-evaluate our model and all baselines on a new clean test set (2,046 clips from MotionUnion and MotionLLAMA, also a subset of MotionMillions) without this overlap, and IMU4D retains the best motion and text results and a clear advantage over the baselines and cascaded variants (Tables R7, R8). Also, the new evaluation shows no sign that the mirrored overlap inflates IMU4D's results.
We followed the standard split as in prior work, but we agree that the overlap is a problem. We therefore built a clean test set of 2,046 MotionMillion clips: neither these motions nor their mirrored versions appear anywhere in training. Since we could not retrain on a re-split HumanML within the rebuttal period, we built the clean test set from MotionUnion and MotionLLAMA by removing mirrored overlap with training and deduplicating the data (Appendix E.1). Thank you for pointing this out.
Table R7. Results on the clean test set (2,046 clips). (Revised paper: Table 11.) (a) Motion error: MPJPE in mm (lower is better).
(b) Text quality: CIDEr (higher is better). As in Tables 3 and 10 of the revised paper, each motion baseline is paired with MotionGPT3, which writes a caption from the baseline's predicted motion.
IMU4D's motion error is 37–48% lower than the enlarged (XL) baselines in every setting. Its text is 9–15 CIDEr points better than the best two-step baseline in every setting.
Table R8. Ablation: One joint model vs. separate models (clean test set, 5 sensors, 480 frames; counterpart of Table 8a,b). (Revised paper: Table 12.)
Training one joint model improves both motion (28.07 → 24.95 mm) and text (CIDEr 117.13 vs. 98.44–99.45). The backbone and initialization ablations show the same trend: Phi-1.5 (1.3B) gives the lowest motion error (35.57 vs. 36.28 mm for Qwen3-0.6B and 47.94 mm for GPT-2 medium), pretrained initialization beats random initialization (35.57 vs. 39.02 mm), and text quality stays on par (CIDEr 109–114).
Table R9. Clean vs. leaked test clips (MotionLLAMA and MotionUnion, 60 frames). MPJPE in mm, lower is better. (Revised paper: reported in Appendix A.3.)
From the same two sources (MotionLLAMA and MotionUnion), we also compare the 2,046 clean test clips with a different group of 2,046 test clips whose mirrored copies appear in training. Every method's MPJPE differs by at most 2.6% between these groups (Table R9). While IMU4D shows slightly lower errors on the overlapping group, the baselines exhibit the opposite trend. Given IMU4D's larger model capacity, we cannot rule out a potential benefit from training overlap. Nevertheless, IMU4D consistently outperforms all baselines by a substantial margin on both clean and overlapping test clips, demonstrating that its strong performance is maintained even in the absence of mirrored train–test overlap.