acceptodds
Under review as a conference paper at ICLR 2027

IMU4D: 4D Human-Object Understanding from Wearable IMUs

Abstract

Understanding human activities and the objects they interact with typically relies on visual perception, yet cameras pose persistent challenges in privacy, safety, energy efficiency, and scalability. We explore an alternative: 4D perception without vision. Its goal is to recover human motion and the objects the person interacts with purely from everyday wearable sensors. For this we introduce IMU4D, a framework that repurposes large language models for non-visual spatiotemporal understanding of human-object dynamics. IMU4D uses data from a few inertial sensors from earbuds, watches, or smartphones and predicts detailed 4D human motion together with surrounding objects the person interacts with. Experiments across diverse human motion datasets and human-object datasets show that IMU4D yields more coherent and temporally stable results than SoTA cascaded pipelines, suggesting wearable motion sensors alone can support 4D understanding of human–object interactions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Inertial measurement units, Multimodal learning, Human-scene understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_or1UOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

IMU4D predicts human motion, activity descriptions, and interacting object categories and poses from sparse wearable IMU signals. It uses a pretrained language-model backbone with modality-specific encoders and decoders. Root motion is represented through locally normalized and quantized increments, while body poses are regressed continuously. The model is trained on mixed datasets and then fine-tuned for individual tasks. Experiments evaluate motion reconstruction, captioning, and static and dynamic object prediction, including motion reconstruction from real IMU recordings.

Critical strengths
  1. I find the extension from IMU-based motion capture to human–object understanding interesting. Predicting activity descriptions and interacting objects adds useful context to the recovered motion. The proposed model connects these tasks through shared representations and can use datasets that provide only some of the required annotations.

  2. The motion results on real IMU data are convincing. Under the three-sensor setting, MPJPE decreases from 62.91 to 47.31 mm on DIP-IMU and from 61.96 to 42.45 mm on IMUPoser compared with MobilePoser. I also appreciate the comparison in Table 8: using the same architecture and pretraining for joint, single-task, and cascaded models helps isolate the benefit of joint training. The sensor-count and real-data fine-tuning experiments provide useful information about how performance changes with the available inputs and training data.

Critical weaknesses
  1. Appendix E.1 states that some HumanML3D test clips have mirrored counterparts in the training set. These are closely related sequences, so the HumanML results may overestimate generalization to unseen motions. Training all methods on the same split does not resolve this concern, since different models may benefit differently from the overlap. Several main results and ablations rely on HumanML. Please report the number of affected test clips and repeat the key comparisons with each original motion and its mirrored variants assigned to the same split.

  2. The comparison with real-time baselines needs clearer control of temporal context and inference cost. IMU4D processes the input sequence with bidirectional motion queries and averages predictions from two shifted inference passes, while MobilePoser is designed for real-time operation. It is unclear whether the evaluated methods have the same access to future frames. A comparison with matched temporal context and an ablation of shifted-pass averaging would help identify how much of the gain comes from these choices. Inference time and memory usage should also be reported to assess the feasibility of the proposed wearable applications.

Questions for authors
  1. The main object results use five-point synthetic IMU inputs, while the real-device experiments use three sensors and report motion and text metrics. Do you have quantitative object-category or object-pose results for real three-sensor inputs? Please also provide the number and duration of sequences in IMU4D-Real, explain whether training and test sets share participants or recording sessions, and describe how the accuracy of the GVHMR-derived reference trajectories was checked.

  2. For the object benchmarks, are test participants, object instances, or motion–object combinations unseen during training, or are the splits only at the sequence level? Similar body motions can correspond to different objects. Results on held-out combinations, or a comparison with a simple activity-conditioned object predictor, would help show what the model learns beyond common motion–object associations.

3 comments

W1 — HumanML mirrored overlap.

Yes. 2,000 of the HumanML test clips (which is a subset of MotionMillions) we evaluated are affected: each has a mirrored copy in the training data of the official MotionMillion split (Fan et al., 2025, Go to Zero, ICCV). We re-evaluate our model and all baselines on a new clean test set (2,046 clips from MotionUnion and MotionLLAMA, also a subset of MotionMillions) without this overlap, and IMU4D retains the best motion and text results and a clear advantage over the baselines and cascaded variants (Tables R7, R8). Also, the new evaluation shows no sign that the mirrored overlap inflates IMU4D's results.

We followed the standard split as in prior work, but we agree that the overlap is a problem. We therefore built a clean test set of 2,046 MotionMillion clips: neither these motions nor their mirrored versions appear anywhere in training. Since we could not retrain on a re-split HumanML within the rebuttal period, we built the clean test set from MotionUnion and MotionLLAMA by removing mirrored overlap with training and deduplicating the data (Appendix E.1). Thank you for pointing this out.

Table R7. Results on the clean test set (2,046 clips). (Revised paper: Table 11.) (a) Motion error: MPJPE in mm (lower is better).

Method3 sensors, 60 frames5 sensors, 60 frames3 sensors, 480 frames5 sensors, 480 frames
IMU4D38.2326.2037.4424.95
IMUPoser67.0655.2068.5257.16
IMUPoser-XL60.3146.6660.3047.67
MobilePoser71.2458.3972.2159.63
MobilePoser-XL60.6646.7260.4047.14

(b) Text quality: CIDEr (higher is better). As in Tables 3 and 10 of the revised paper, each motion baseline is paired with MotionGPT3, which writes a caption from the baseline's predicted motion.

Method3 sensors, 60 frames5 sensors, 60 frames3 sensors, 480 frames5 sensors, 480 frames
IMU4D (from sensors)103.23107.87114.46117.13
IMUPoser → MotionGPT386.4592.78104.23106.70
IMUPoser-XL → MotionGPT388.7693.34105.00106.85
MobilePoser → MotionGPT371.1273.0388.6791.30
MobilePoser-XL → MotionGPT379.4480.2299.00101.55

IMU4D's motion error is 37–48% lower than the enlarged (XL) baselines in every setting. Its text is 9–15 CIDEr points better than the best two-step baseline in every setting.

Table R8. Ablation: One joint model vs. separate models (clean test set, 5 sensors, 480 frames; counterpart of Table 8a,b). (Revised paper: Table 12.)

MethodMotion error, MPJPE (mm) ↓Text quality, CIDEr ↑
Joint model (IMU4D): sensors → motion and text24.95117.13
Separate model: sensors → motion only28.07—
Separate model: sensors → text only—99.45
Two steps: sensors → motion, then motion → text28.0798.44

Training one joint model improves both motion (28.07 → 24.95 mm) and text (CIDEr 117.13 vs. 98.44–99.45). The backbone and initialization ablations show the same trend: Phi-1.5 (1.3B) gives the lowest motion error (35.57 vs. 36.28 mm for Qwen3-0.6B and 47.94 mm for GPT-2 medium), pretrained initialization beats random initialization (35.57 vs. 39.02 mm), and text quality stays on par (CIDEr 109–114).

Table R9. Clean vs. leaked test clips (MotionLLAMA and MotionUnion, 60 frames). MPJPE in mm, lower is better. (Revised paper: reported in Appendix A.3.)

Method3 sensors: clean3 sensors: leakedChange5 sensors: clean5 sensors: leakedChange
IMU4D38.2338.08−0.4%26.2025.53−2.6%
IMUPoser67.0668.35+1.9%55.2056.22+1.8%
IMUPoser-XL60.3161.01+1.2%46.6647.18+1.1%
MobilePoser71.2472.48+1.7%58.3959.37+1.7%
MobilePoser-XL60.6661.71+1.7%46.7247.48+1.6%

From the same two sources (MotionLLAMA and MotionUnion), we also compare the 2,046 clean test clips with a different group of 2,046 test clips whose mirrored copies appear in training. Every method's MPJPE differs by at most 2.6% between these groups (Table R9). While IMU4D shows slightly lower errors on the overlapping group, the baselines exhibit the opposite trend. Given IMU4D's larger model capacity, we cannot rule out a potential benefit from training overlap. Nevertheless, IMU4D consistently outperforms all baselines by a substantial margin on both clean and overlapping test clips, demonstrating that its strong performance is maintained even in the absence of mirrored train–test overlap.

W2 — Access to future frames, two-pass averaging, and cost.

All methods see exactly the same past and future frames, because every method runs offline on the same input window. IMU4D's accuracy advantage does not come from averaging two shifted passes. On long windows IMU4D runs faster than most baselines, but it needs more GPU memory.

Every method receives the same input window (60 or 480 frames). The pose networks of IMUPoser and MobilePoser are bidirectional LSTMs and IMU4D uses bidirectional attention, so all of them can use both earlier and later frames. As suggested, we test IMU4D with and without two-pass averaging (Table R10) and measure speed and memory (Table R11). With a single pass, IMU4D's error is still 35% lower than the XL baselines; averaging reduces the error by only 0.7 mm (38.95 → 38.23 mm; on LINGO: 38.61 → 38.18 mm). We have added both tables to the revised paper (Tables 15 and 16). Thank you for the suggestion.

Table R10. Effect of averaging two shifted passes (clean test set, 3 sensors, 60 frames). (Revised paper: Table 15.)

MethodMPJPE (mm) ↓PA-MPJPE (mm) ↓
IMUPoser-XL60.3147.64
MobilePoser-XL60.6649.23
IMU4D, one pass38.9533.35
IMU4D, average of two shifted passes (as in Table R7)38.2332.81

Table R11. Speed and memory (3 sensors, one sequence at a time, one RTX PRO 6000 GPU). (Revised paper: Table 16.) Baselines run in 32-bit floating point; IMU4D's language-model backbone runs in bfloat16. We report the median time per input window and the peak GPU memory.

MethodParameters (millions)Time, 60 frames (ms)Time, 480 frames (ms)GPU memory (GiB)
IMUPoser10.671.5810.250.17
IMUPoser-XL164.4411.5788.161.90
MobilePoser6.679.9064.980.40
MobilePoser-XL162.8917.85128.011.26
IMU4D1511.3517.7535.6713.72

Times cover motion reconstruction (including two-pass averaging), not caption generation. IMU4D processes the whole window at once, so an 8× longer input only doubles its running time, while the LSTM baselines process frames one after another and become 6–8× slower; on 480-frame windows IMU4D is faster than IMUPoser-XL, MobilePoser and MobilePoser-XL. We state in the revised paper that the wearable sensors stream their data to a GPU; these numbers do not show that IMU4D can run on the device itself or in real time.

Q1 — Real-data objects, dataset size, splits, and reference quality.

We do not have quantitative results for object category or object pose on real IMU input, because the real IMU recordings have no verified object labels. Our object metrics therefore come from datasets with object annotations, where the IMU signals are synthesized. On a new test with an unseen participant, IMU4D still outperforms both baselines.

IMU4D-Real data details. We provide the number of recordings, the total duration, the split, and the reference-motion checks of IMU4D-Real below (see also our response to Reviewer QyV8 Q6).

  • Number of recordings and duration. 48 recordings (about 1 hour) from 2 participants:
    • 15 meeting-room sessions with multi-view reference motion (about 34 minutes): 10 with three sensors (20.4 min train / 2.8 min test) and 5 with two sensors (9.5 min train / 1.1 min test).
    • 33 single-camera recordings (about 30 minutes), used only for training and supervised on local joint angles only.
  • Data splits. The original test set is a held-out time segment of each session, separated from the training segments by a 2-second gap. Training and test therefore share participants and sessions, so we add a test on an unseen participant below.
  • Reference motion. We combine GVHMR estimates from three calibrated static cameras, visually check that the result projects correctly into all three views, and remove time intervals where it fails. We also check the alignment in the initial T-pose and that body height stays consistent while seated. These checks are not an independent measurement of accuracy during motion.

Test on an unseen participant. As suggested, we train on one participant and test on the other, so that no participant or session is shared. All three methods are fine-tuned for the same fixed 2,000 steps on the same 80 clips from participant 1, and tested on 22 clips from participant 2. Both sides use only the phone and the watch (2 sensors), since participant 2 has no usable earbud data. IMU4D starts from a checkpoint that train only on synthetic data; the baselines start from their pretrained checkpoints and keep their original training losses. Participant 2 is never used to choose checkpoints.

Table R12. Unseen-participant test (2 sensors, 22 test clips cut into 320 windows of 60 frames). (Revised paper: Table 17.)

MethodMPJPE (mm) ↓PA-MPJPE (mm) ↓MPJRE (°) ↓MTE (mm) ↓
IMU4D83.8276.4515.61223.89
IMUPoser92.3784.9316.13—
MobilePoser91.8084.3117.11337.55

IMU4D has the lowest error on every metric for the unseen participant, with a clear margin on position and translation and a comparable joint rotation error (15.61° vs. 16.13°); 480-frame windows give the same ranking (MPJPE 82.58 vs. 92.66 and 92.20 mm). IMUPoser does not predict body translation. We have added the dataset statistics and this experiment to the revised paper (Appendix E.4 and A.5, Tables 24 and 17). Thank you for the question.

Q2 — Object benchmark splits and a retrieval baseline.

We summarize below which test participants and object instances are seen during training.

  • Participant splits: whether test participants are seen/unseen.
    • HiPHI: unseen. 48 for training, and 10 for testing.
    • OMOMO: unseen. 15 for training, and 2 for testing.
    • HUMOTO: seen, but test sequences are unseen. All motions were recorded by a single actor, thus the participant split is not feasible. We split by sequence instead (685 for training, and 50 for testing).
  • Object splits: all categories are seen during training. IMU4D predicts objects from a closed vocabulary; unseen categories are outside the scope of this paper.

To test whether IMU4D does more than recall common motion–object pairs, we compare it with three baselines on HiPHI (500 test clips; object-category F1):

  • Always predicting the most common training object gives 35.80%.
  • Nearest-neighbor retrieval, which uses the signals from the same five IMU sensors to retrieve the most similar training clip and copies its objects, gives 36.20%.
  • A motion-conditioned object predictor based on a bidirectional LSTM gives 43.4%.
  • IMU4D, using the same five sensors, reaches 45.85%, higher than all three.

We have added the split details and this comparison to the revised paper (Appendix E.1 and B.5). Thank you for the suggestion.

Reviewer_QyV8OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper predicts human motion, a short text description of the activity, and the objects being interacted with (category and 6-DoF pose), using only 1–5 wearable IMUs (earbuds, watch, phone) and no camera. The model is a fine-tuned Show-o/Phi-1.5 transformer: it predicts the motion first, using a new window-normalized motion tokenizer, then generates the text and the objects. It is pretrained on large motion datasets with synthesized IMU signals, then fine-tuned on each dataset. It beats IMUPoser and MobilePoser on motion on real IMU data, and on text and objects it beats both cascaded pipelines and single-task versions of its own model.

Critical strengths
  1. An interesting, practical setting. Inferring motion, activity, and interacted objects from commodity wearables, with no camera, is useful and privacy-friendly. Putting all three outputs in one model is a natural and new combination.

  2. Strong IMU-to-motion results. With 3 sensors it clearly beats MobilePoser on real IMU data (DIP-IMU MPJPE 47.3 vs 62.9; IMUPoser 42.5 vs 62.0), and Table 8 shows joint training helps compared with single-task training using the same architecture.

Critical weaknesses
  1. The model picks one of a few dozen known objects per dataset with categories and meshes known in advance and is fine-tuned separately on each dataset. This is far from the “4D human–object understanding” the paper claims.

  2. Thin real-world evaluation and missing calibration details. IMU4D-Real has only 2 participants, with ground truth from GVHMR and test captions from Qwen3-VL. The only calibration described is a 2–3 s T-pose. The paper never explains how sensor-to-body alignment is done, how drift is handled without a magnetometer, how the AirPods, Watch, and iPhone are synchronized and resampled, or how IMU signals are synthesized for training. Fig. 3 overlays predictions on RGB frames, but no IMU–camera calibration or synchronization is described. These details decide whether the real-world results can be reproduced.

  3. Inconsistent main results. In the same setting (HumanML, 5 sensors, 480 frames), MPJPE is 14.69 in Tables 3 and 8 but 23.60 in Tables 7 and 14. Some HumanML test clips also have mirrored copies in training (Appendix E.1).

  4. Minor: editing errors. F.1 cites a BERTScore of −6.07 that isn’t in Table 11, A.1 describes Table 2 wrongly, there’s a broken “Sec. ??” reference, and OMOMO is cited twice.

Questions for authors
  1. Which checkpoint produces the main results, and why is HumanML MPJPE 14.69 in Tables 3 and 8 but 23.60 in Tables 7 and 14?

  2. Can you evaluate object prediction on real IMU recordings?

  3. Can the model handle objects or meshes it has never seen in training?

  4. How are the predictions in Fig. 3 projected into the RGB frames: aligned to the GVHMR ground truth, or through an IMU–camera calibration? Was MobilePoser aligned the same way?

  5. Which SMPL-X body shape is used? Is MPJVE computed with ground-truth or neutral shape, and is it the same for the baselines?

  6. How is calibration done beyond the T-pose? Specifically: (a) how sensor-to-segment orientation is estimated for each device; (b) how heading and gyro drift are handled without a magnetometer; (c) how the three devices are time-synchronized and resampled to a common rate; and (d) how IMU signals are synthesized for training (virtual sensor placement, noise and drift models). How sensitive are the real-world results to calibration errors?
    
  7. Was IRB approval obtained for IMU4D-Real, and how large is it (sequences and hours)?

4 comments

W1 / Q3 — Unseen objects.

Unseen object categories? No. IMU4D targets closed-vocabulary object understanding, following most human–object interaction work (e.g., Object Pop-up, TriDi); this scope is stated in Sec. 3.1, and we will make it more prominent. Since our backbone is a language model, emitting the object as text tokens instead of a fixed class label is a natural route to open-set prediction, which we leave to future work.

Known meshes? The model predicts each object's category and pose, not its geometry; meshes are neither input nor output, and are only attached afterward for visualization and evaluation. A new instance of a seen category can therefore be substituted, provided it follows that category's canonical frame.

Claims. We respectfully maintain that "4D human–object understanding" describes the task: jointly inferring the human motion, the object involved, and its placement over time. A closed vocabulary is the standard setting in many human–object interaction works (e.g., Object Pop-up, TriDi) and, in our case, the object must additionally be inferred from body-worn IMUs that never observe it. We will state the closed-set scope more prominently in the abstract and introduction so the claim is not read as open-world.

Q1 / W3 — Main results vs. ablation results.

Checkpoints. The 14.69 mm (Tables 3 and 8) comes from our fully trained model (500K steps). The 23.60 mm (Tables 7 and 14) came from the same model trained to 157K steps, which we used for ablations to limit compute and explained in the subcaption in the original submission.

Mirrored clips. The reviewer is correct: some HumanML test clips have mirrored copies in training (as described in Appendix E.1). We use this standard HumanML split so that results are comparable to prior work, and all baselines are trained on the same data, so the overlap affects every method equally.

To remove the overlap, we re-evaluate on the clean, non-mirrored clips from MotionMillion (2,046 in total). On this set, the fully trained model achieves 24.95 mm MPJPE with 5 IMUs and 480 frames and remains well ahead of the baselines (47.67 / 47.14 mm for IMUPoser-XL / MobilePoser-XL); full results are in Table 11. The error is higher than on HumanML partly because these clips are less represented in training (16 source datasets averaging 2.4% of training clips each, vs. 13.75% for HumanML), and over half are two-person interactions (Inter-X, InterHuman). We also report joint vs. single-task and cascaded formulations with fully trained models on this set in Table 12. As for the 157K-trained models, we repeat both ablations on this set in Table 13 and Table 14 (shown below as Tables R4 and R5).

Table R4. LLM backbone comparison — MotionMillion-clean, 2,046 clips, 5 IMUs, 480 frames, at 157K steps.

Base modelParamsMPJPE (mm) ↓PA-MPJPE (mm) ↓MPJRE (°) ↓MPJVE (mm) ↓MTE (mm) ↓BLEU@1 (%) ↑BLEU@4 (%) ↑ROUGE-L (%) ↑CIDEr (×100) ↑BERTScore (%) ↑
GPT-2 medium355M47.9435.968.7358.91141.5330.06%4.57%33.88%113.5221.23%
Qwen3-0.6B0.6B36.2826.736.9844.7089.2829.58%4.19%33.18%109.3820.29%
Phi-1.51.3B35.5726.156.8443.5587.5830.24%4.38%33.62%110.6222.03%

Table R5. Ablation on initialization from an LLM — MotionMillion-clean, 2,046 clips, 5 IMUs, 480 frames, at 157K steps.

Base modelInit from LLM?MPJPE (mm) ↓PA-MPJPE (mm) ↓MPJRE (°) ↓MPJVE (mm) ↓MTE (mm) ↓BLEU@1 (%) ↑BLEU@4 (%) ↑ROUGE-L (%) ↑CIDEr (×100) ↑BERTScore (%) ↑
Phi-1.5:x:39.0229.217.4647.9797.8929.94%4.30%33.50%110.8521.33%
Phi-1.5:white_check_mark:35.5726.156.8443.5587.5830.24%4.38%33.62%110.6222.03%

On the clean set, the conclusions of both ablations hold. In Table R4, the two recent backbones are substantially better than GPT-2 medium on all five motion metrics (MPJPE 47.94 → 36.28 / 35.57 mm for Qwen3-0.6B / Phi-1.5), with Phi-1.5 slightly ahead of Qwen3-0.6B on every motion metric. Text metrics are similar across the three backbones. In Table R5, initializing from the pretrained LLM improves all five motion metrics (MPJPE 39.02 → 35.57 mm, MTE 97.89 → 87.58 mm), while text metrics change only slightly. Both tables are updated in the paper.

Q2 — Objects on real IMU recordings. Not quantitatively at present. All the real-IMU datasets we use (DIP-IMU, IMUPoser, and our own IMU4D-Real) have no object annotations, so there is no ground truth to score object predictions against. We therefore evaluate objects on human–object interaction datasets that do provide them (HUMOTO, HiPHI, OMOMO), with IMUs synthesized from their motion capture, and use the real recordings to evaluate motion and text; this distinction is stated in the paper. Qualitatively, IMU4D produces plausible object predictions, as shown in Figure 4 and the supplementary videos. Since IMU4D infers objects through the predicted motion and text, and both are validated on real IMUs, we expect object prediction to transfer.

Q4 — Fig. 3 projection for real-world capture.

The predictions are aligned to the GVHMR reference shape using their first frame, and MobilePoser is aligned in the exact same way. This alignment is used for visualization only. The model takes only IMU signals as input and predicts motion in a frame normalized by the first-frame position and yaw. To overlay the result on the RGB frames, we undo this normalization with the GVHMR reference's first-frame position and yaw, then project the motion with the calibrated camera. This is one fixed rigid transform, so there is no data leakage, and no IMU–camera calibration is needed. The GVHMR reference never enters the model, and we do not fit the predicted trajectory to it or correct it frame by frame. The subsequent motion and trajectory drift in Figure 3 are therefore each method's own prediction.

Q5 — Body shape and MPJVE.

We use the neutral SMPL-X body (β = 0) for both predictions and ground truth, including all baselines. MPJPE and MPJVE are computed from local body pose with root translation zeroed and root orientation set to identity. MPJPE uses the 22 body joints, and MPJVE computes the mean per-vertex error. All methods, including the baselines in all tables of the paper, are scored by the same evaluation code. We add these details to the paper (see Appendix F.1).

Q6 — Quantity of real IMU data, IMU calibration, synchronization, and synthesizing details.

Quantity of real IMU data. Our real-world evaluation is not limited to IMU4D-Real. It covers three datasets of physically recorded IMU signals: DIP-IMU and IMUPoser (manuscript Table 2) and IMU4D-Real (manuscript Table 4), all evaluated under the practical 3-sensor setup (head, wrist, thigh). Table R6 summarizes their details.

Table R6. Real-IMU datasets (3 sensors). MPJPE in mm, lower is better.

DatasetParticipantsRecordingsHoursMobilePoser MPJPE ↓IMU4D MPJPE ↓
DIP-IMU10601.4462.9147.31
IMUPoser101670.9661.9642.45
IMU4D-Real2481.0687.4637.12

IMU processing details. The pipeline uses a T-pose calibration for initial device alignment, offline synchronization for evaluation, and noise augmentation during training. We document these steps in Appendix E.4:

  • (a) Sensor-to-segment. We place the phone in the right pocket, the watch on the left wrist, and AirPods in both ears, of which only the right-ear sensor is used for calibration. A T-pose of about 2 seconds at the start of each session provides a constant mounting rotation per device that aligns its orientation to the SMPL-X zero pose, together with a static acceleration-bias estimate.
  • (b) Drifting. IMU4D has no explicit drift-correction step, because it does not recover pose by integrating individual IMU signals, which is where drift accumulates. No magnetometer is used: gravity fixes each device's pitch and roll, and heading is referenced to the initial T-pose. The model learns a direct mapping from all sensors jointly to full-body motion, so it implicitly uses the structure of the human body to resolve readings that would be ambiguous or drifting for a single sensor. Two training choices make this robust on real devices: heading-mismatch and yaw-drift augmentation during pre-training, and fine-tuning on real IMU recordings. This is the same data-driven principle as prior sparse-IMU methods such as IMUPoser and MobilePoser, and it gives accurate poses on all three real-IMU datasets (Table R6).
  • (c) Sync/resampling. All streams carry UTC timestamps from SensorLogger (the watch clock is synchronized to the paired iPhone by watchOS). Native rates are 100 Hz (phone, watch) and 50 Hz (AirPods). Streams are resampled to 30 Hz after a 12 Hz zero-phase anti-alias filter, with SLERP for orientations. Offline synchronization applies a fixed per-session IMU–video offset; this offset is estimated by cross-correlating measured gyro/specific-force norms with IMU signals synthesized from the video-reconstructed motions.
  • (d) Synthetic IMUs. We synthesize IMU signals from SMPL-X motion in three steps.
    • Placement. We attach one virtual sensor to each of six SMPL-X vertices (left ear, right ear, left wrist, right wrist, left thigh, right thigh).
    • Clean signals. Let p\mathbf{p} be a sensor's world-frame position and R\mathbf{R} its sensor-to-world rotation, with time derivatives taken by finite differences. The accelerometer reading (specific force) is a=R⊤(p¨−g)\mathbf{a} = \mathbf{R}^\top(\ddot{\mathbf{p}} - \mathbf{g}), where g\mathbf{g} is gravity. The gyroscope reading (angular velocity) ω\boldsymbol{\omega} is taken from the skew-symmetric part of R⊤R˙\mathbf{R}^\top\dot{\mathbf{R}}.
    • Noise and drift. A second pre-training stage perturbs the clean signals to mimic real devices: random 3–10 Hz low-pass filtering, 5° mounting error, yaw drift, attitude jitter, accelerometer and gyroscope noise and bias, ±1-frame time shifts, and per-sensor heading mismatch.

Sensitivity to calibration error. It is reported in Table R3 (XAYL Q3). To quantify the effect, we add a random rotation of 5°/10°/20° per sensor axis to the calibrated inputs, without fine-tuning on such noise. Pose accuracy is stable up to 20° (MPJPE +3.4 mm). Translation is stable up to 10° and degrades moderately at 20°. IMU4D also tolerates an arbitrary heading: with a random yaw offset in [−180°, 180°], MPJPE is 49.12 ± 1.76 mm. As with any IMU-to-motion method, including IMUPoser and MobilePoser, some calibration is still required. We add this analysis to Appendix B.4.

IMU4D-Real camera calibration and synchronization. We estimate the intrinsics and extrinsics of the three GoPro cameras with an AprilTag target moved slowly through their shared field of view. Camera timestamps give a coarse alignment. We then refine and verify it with a clapperboard sound cue: the clap is recorded in the audio track of every camera, and aligning the claps synchronizes the three videos. GVHMR predictions from the three views are fused into one reference motion in camera-calibration coordinates. For the meeting-room sessions, we map this motion to the calibrated IMU frame with a single fixed rotation, averaged over the T-pose.

Q7 — IRB and dataset size.

IRB. Yes, the IRB review is in progress. We are familiar with human data collection protocols and are following institutional guidelines. Both participants are healthy adults who provided informed consent.

Size. IMU4D-Real contains 48 sessions (about 1.1 hours at 30 Hz) from 2 participants, grouped by how the ground truth was captured:

  • 3-camera subset: 10 sessions (20.4 min train / 2.8 min test).
  • 2-camera subset: 5 sessions (9.5 min train / 1.1 min test).
  • 1-camera subset: 33 sessions (30 min, train only; supervised on local joint angles, since the global trajectory from a single view is unreliable).

All 48 sessions are used in the participant-disjoint experiment (43 from participant 1, 5 from participant 2).

Minor — Editing errors. Thank you for pointing out. We have corrected these issues in the revised PDF: the BERTScore in F.1, the description of Table 2 in A.1, the broken "Sec. ??" reference, and the duplicate OMOMO citation.

Reviewer_XAYLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper focuses on the problem of estimating human mesh parameters, text description of the scene and object class and pose from only IMU signals.
  • The input IMU signals are of three kinds: earbuds, smartwatch and phone in pocket. Importantly, no camera, no text and no object information assumed. The IMU signals are 6 DoF (acceleration, angular velocity and gyroscope).
  • The key idea, similar to many current works, is to repurpose an LLM (Phi-1.5B in this context, initialized from Show-o) for multimodal sequence modeling. The decoding order of the LLM being: motion -> text -> object ids and poses
  • Datasets: DIP-IMU, IMUPose, HumanML3D, LINGO, etc. The paper also proposes IMU4D-Real (2 participants).
  • Baselines: IMUPose, MobilePose for motion. MobilePose -> MotionGPT3 -> SUMMON/Pose2Room and others for text and objects.
  • The results show that IMU4D consistently outperforms the baseliens across benchmarks and metrics.
Critical strengths
  • The focus problem: I would like to appreciate the focus problem picked by this work. The problem is technically very challenging and has tremendous applications if executed well. This work opens up this potential research direction and can act as an anchor for future works. An important distinction I would like to highlight is that the work is focused on full-body pose along with objects, this is ambitious but general formulations like these are exactly what helps motivate more future works.

  • Real-world motion results: IMU4D shows good results on real IMU data (3pt consumer setup - ear, wrist, pocket). Appreciate the qualitative results and the videos here. The improvements compared to the baselines are substantial: MobilePose: MPJPE 62.9 -> 47.3 mm ; 62.0 -> 42.5 mm; 87.5 -> 37.1 mm. There is tremendous room to improve ofcourse, compared to vision methods, which would be interesting to close the gap. An interesting direction would be to use distillation techniques (vision -> imu) to scale up the data and supervision.

  • Simple, unified formulation. Informative ablations: The single model design (imu -> motion -> text -> objects), Fig 2 is simple and clean. One model that handles any subject of 1-5 devices through masking. From a technical perspective, these are early signs of building multimodal always-on llms. I found the ablations to be thorough and informative - joint vs single task vs cascaded; backbone scaling, sensor counts etc.

  • The qualitative results were plenty, really appreciate it. This is fundamentally a temporal problem and video showcase of the results helps understand the performance of the method intuitively.

Critical weaknesses
  • Performance on object recovery: I feel the object part of the output space performs a supporting role compared to the human pose. The sensors dont exactly capture the object interaction, the motivation for an accurate recovery for the objects is weak. Furthermore, the predicted objects seem to come from generated captions and not from the motion. I would consider repositioning the work to dial down the claims on the object pose estimation part.

  • Synthetic vs Real data. Comparison to baselines: Please correct me if I am wrong, it appears that the majority of the training data is synthetic IMU generated from mocap - no gyro drift or sensor noise. Finetuning on real imu data seems critical to obtain good performance, according to Table 2. Were the baselines, IMUPose and MobilePose also finetuned with the same real data splits? Additionally, IMU4D is a 1.5B parameter model, whereas the baselines are lightweight models which are designed to run realtime on edge devices. Given the paper motivates everyday wearables and energy efficiency, please consider reporting the number of parameters, FLOPs and inference latency for all methods.

    Minor:

  • L64: framework that infer -> framework that infers

  • L364: IMU-to-4D's -> IMU4D's

  • L372: Table 6 caption, consistently outperform -> consistently outperforms

  • L983: IMU-to-MotionTextScene -> IMU-to-MotionTextObject

  • L1128: broken reference, Sec. ??

  • L1287: As shown in 6 -> As shown in Fig. 6

  • L1289: SenorLogger -> SensorLogger

  • Table 6 appears before Table 5 in the main paper, consider reordering.

Questions for authors
  • Object recovery: Is it even possible to accurate recover the object pose and interaction details form the current inputs? Additionally, can we understand the effect of the caption accuracy on the object recovery quantitatively? For instance, please report the object accuracy when the generated caption is wrong.

  • Baseline comparisons: Are the baselines finetuned on the exact same train sets? Additionally, how can we isolate the effect of compute (number of parameters) across the models? The current primary baselines are an order of magnitude lighter than the proposed method.

  • Robustness to real world imu conditions: The real data captures require an initial calibration - this seems like a crucial piece which would be difficult to replicate in the real world. In practice, when such a calibration would be not available or noisy, how does this affect IMU4Ds performance?

2 comments

W1 / Q1 — Object recovery and dependence on captions.

The reviewer's reading is correct, and it is by design. The IMUs never observe the object, so inferring it directly from the sensors is hard. IMU4D therefore decodes objects after motion and text and cross-attends to both: text supplies the semantics needed to identify the object category, and body motion supplies where the object is and how it moves.

Body motion as an intermediate step. Object pose estimation benefits from body motion reasoning. Predicting objects directly from IMUs gives 1.76 3D-IoU and 15.15 ID-R on HUMOTO; predicting them jointly with motion gives 8.01 and 32.32 (manuscript Table 8c).

Text caption as an intermediate step. Caption reasoning is what makes object-category prediction feasible. With an empty caption, category F1 is only 3.31/18.18/17.93% on HUMOTO/HiPHI/OMOMO; with the predicted caption it rises to 28.44/45.85/49.94%. With the GT caption it reaches 71.50/95.20/92.84% (Table R1), so object reasoning improves directly as caption reasoning improves.

Table R1. Ablation on input caption for object-category F1 ↑ (%).

Dataset# eval clipsPredicted captionGT captionEmpty caption
HUMOTO5028.44%71.50%3.31%
HiPHI50045.85%95.20%18.18%
OMOMO43349.94%92.84%17.93%

We can see caption accuracy does affect which object is predicted: once the category is correct, mesh-velocity error is essentially the same with predicted and GT captions (410.23 vs. 414.42 mm/s on HiPHI; 631.67 vs. 651.39 mm/s on OMOMO).

We will follow your suggestion and make this division explicit in the revision and describe the object output as category and approximate placement from a closed-set library, rather than precise 6-DoF recovery.

W2 / Q2 — Baseline fine-tuning and model capacity.

(A) Same training data? Yes. IMUPoser and MobilePoser were trained on exactly the same synthetic data and real-data fine-tuning splits as IMU4D; IMU4D's only extra pretraining is the text-only pretraining of its language backbone. Our simulation injects noise and bias (drift) into both the gyroscope and accelerometer.

(B) Isolating the effect of compute. Separately from (A), we trained larger baselines on MotionMillion, our largest dataset: IMUPoser-XL (164M, 15×) and MobilePoser-XL (163M, 24×), with latency matching our method (17.85 for MobilePoser-XL and 17.75 for IMU4D with 60 frames). Scaling baseline helps, but with diminishing returns: MPJPE drops from 68.52 to 60.30 mm for IMUPoser and from 72.21 to 60.40 mm for MobilePoser, while IMU4D reaches 37.44 mm (3 sensors, 480 frames). With 5 sensors and 480 frames: 47.67 / 47.14 mm (IMUPoser-XL / MobilePoser-XL) vs. 24.95 mm (IMU4D). An exact parameter match is difficult, since recurrent baselines are not designed to scale to 1.5B and most of IMU4D's parameters come from its pretrained language backbone. Showing that IMU perception benefits from such a backbone, which also enables joint motion, text, and object prediction, is one of the goals of this paper.

(C) Parameters, FLOPs, latency, and efficiency. See our response to or1U W2. In brief, IMU4D has 1.5B parameters, 292 GLOPs, and a latency of 17.75 ms on 60 frames, compared with the IMU-to-motion baseline MobilePoser (6.7M parameters, 0.8 GFLOPs, 9.90 ms) and its scaled-up variant MobilePoser-XL (163M parameters, 19.5 GLOPs, 17.85 ms) (Table R11 in our response to or1U). Although IMU4D is larger, its latency grows slowly with window length: from 60 to 480 frames, its time only doubles (17.75 → 35.67 ms), whereas the baselines’ latency grows by about 6.5–7×. As a result, on a 480-frame window, IMU4D (35.67 ms) is faster than MobilePoser (64.98 ms), IMUPoser-XL (88.16 ms), and MobilePoser-XL (128.01 ms). Only the much smaller IMUPoser is faster at both lengths.

(D) Effect of real-data fine-tuning. To show the effect of fine-tuning, we evaluate IMUPoser, MobilePoser, and IMU4D on IMU4D-Real before fine-tuning, using models trained only on synthetic IMU data (Table R2). We find that fine-tuning on real data consistently lowers the error for all methods. IMU4D leads both before and after fine-tuning: its error is about half that of the baselines in both settings, showing that its advantage comes from the model itself rather than the real data alone.

Table R2. IMU-to-Motion before and after fine-tuning on IMU4D-Real (3 sensors, 60 frames, 85 windows). MPJPE in mm, lower is better.

MethodBefore fine-tuningAfter fine-tuning
IMUPoser242.8385.96
MobilePoser254.5381.88
IMU4D118.5539.58

Q3 — Robustness to calibration on real IMUs.

Our real captures already contain imperfect calibration, since the T-pose is performed by users and is never exact. To quantify the effect further, we add calibration error to the calibrated inputs: a random rotation of 5°/10°/20° independent per sensor axis, without finetuning with such noises. As Table R3 shows, pose accuracy is stable up to 20° (MPJPE +3.4 mm), and translation is stable up to 10° and degrades moderately at 20°. IMU4D also tolerates an arbitrary heading reference: with a random yaw offset in [−180°, 180°], MPJPE is 49.12 ± 1.76 mm. Without any calibration, i.e., with no knowledge of how the subject wears each device or of the IMU's coordinate convention, failure is expected, as for any IMU-to-motion method; some calibration step is therefore required and has been done in prior IMU-to-Motion work (e.g., IMUPoser, MobilePoser). We will add this analysis to the appendix.

Table R3. Calibration robustness (3 IMUs; 15 clips of 60 frames from the IMU4D-Real test set; 3 seeds)..

Input conditionMPJPE ↓ (mm)MTE ↓ (mm)
Original calibrated input39.5889.06
Added device-alignment error: 5°39.76 ± 0.5490.45 ± 1.40
Added device-alignment error: 10°40.45 ± 1.0595.98 ± 2.83
Added device-alignment error: 20°42.98 ± 2.11112.24 ± 3.81
Added arbitrary heading references: yaw [−180°, 180°]49.12 ± 1.76135.14 ± 8.51

Minor — Editing errors. Thank you for pointing them out. We have corrected all of these errors (L64, L364, L372, L983, L1128, L1287, L1289) and reordered Tables 5 and 6 in the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.