acceptodds
Under review as a conference paper at ICLR 2027

KiJacoMMP: Kinematics-Aware Jacobian Iterative Solver for Multi-Person Motion Prediction

Abstract

Multi-person motion prediction aims to infer the future 3D poses of all individuals in a scene from historical observations. Existing methods typically predict future joint coordinates directly by learning spatiotemporal correlations. However, small coordinate errors do not guarantee that a predicted pose can be continuously reached from the current motion state. To address these issues, we propose KiJacoMMP, a kinematics-aware jacobian iterative solver for multi-person motion prediction. KiJacoMMP firstly exploits acceleration field encoder characterizing the evolution of individual velocities and interaction relation encoder modeling spatial dependencies among different individuals. Subsequently, the model takes the last observed pose as the initial condition, integrates the preliminary velocities frame by frame into joint positions, and constructs constraint residuals from predefined skeletal length. We further derive a sparse skeletal Jacobian matrix and an edge coupling matrix, transforming each nonlinear skeletal correction into a damped linearized system. Through multiple iterations, the system jointly updates shared joints, thereby producing kinematically feasible future trajectories while preserving the preliminary motion trend as much as possible. The proposed method requires neither body mass nor an external physics simulator. Experiments on current multi-person motion prediction datasets demonstrate the effectiveness of our proposed method.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Multi-person motion prediction, Jacobian solver

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1PCMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses multi-person 3D motion prediction. The authors argue that regression-based methods predict joint coordinates directly and so may produce skeletons that are not kinematically consistent. They propose KiJacoMMP, which predicts a future velocity field with a GCN+attention encoder-decoder. Acceleration-field and inter-person interaction encoders feed this decoder. The predicted velocities are then integrated frame by frame from the last observed pose. A Jacobian Iterative Solver (JIS) corrects the integrated joint positions to satisfy bone-length constraints, using an analytic sparse Jacobian and a damped linearized system. The method needs no mass, forces, or physics simulator.

The paper reports lower JPE and APE than recent baselines (including IAFormer and ST-MoE) on CMU-Mocap, Mix1 (6 people), and Mix2 (10 people), with larger gains in crowded scenes. Validation consists of a module ablation on CMU-Mocap, an analysis of solver iteration count (converging by about 10 iterations), 20 repeated runs showing small variance, and one qualitative example.

Critical strengths
  1. A simple, practical idea that is easy to adopt. Enforcing bone-length consistency with an analytic sparse Jacobian and a damped solver requires no physical parameters, so it can be used on any skeleton-only mocap data. The solver reaches saturation within about 10 iterations, which makes it plausible as a drop-in component. This is the main reason I find the paper worth encouraging.

  2. Consistent gains over strong, recent baselines. The method improves JPE and APE at every horizon on all three settings, including against AAAI'26 ST-MoE, and the improvement grows in the more crowded Mix2 setting (e.g., 15 mm at 1.0 s). The ablations show every module helping, and repeated runs indicate that the margins are well above run-to-run noise.

Critical weaknesses
  1. The central claim, "kinematically feasible predictions," is never directly measured, and the source of the gain is not isolated.

The motivation is that baselines violate skeletal constraints, but the paper reports only JPE/APE. There is no bone-length error, no measurement of how much baselines actually violate constraints, and no before/after view of what JIS corrects. In addition, the model integrates predicted velocities, which naturally drifts, and the "w/o JIS" variant (162 at 1.0 s) is worse than IAFormer (161) and ST-MoE (158). So part of the JIS gain may come from compensating for drift introduced by the velocity-integration design itself, rather than from constraints improving over coordinate regression. It is also unclear whether "w/o JIS" is retrained or only evaluated with the solver removed. Without feasibility metrics and a cleaner control, the paper's main message is not yet supported by its evidence. I think this is fixable.

  1. Limited novelty and missing comparisons to the natural alternatives.

The acceleration and interaction encoders are standard (finite differences, GCN, attention), and the main new element is applying a classical constraint solver as a layer. The paper does not compare against simple bone-length projection or classical IK/PBD-style corrections, and does not show JIS as a plug-in on other methods' outputs (e.g., IAFormer, ST-MoE). These would show what the proposed solver adds and whether it generalizes. Evaluation is also narrow (CMU-Mocap plus two mixed sets built partly from pseudo-labeled data, ablations only on CMU-Mocap), and efficiency and key hyperparameters are not reported. Closely related recent work (KinemaDiff, Lu et al.) is cited but not discussed or compared.

Questions for authors

Clarification on these could change my assessments.

  1. Is the "w/o JIS" model retrained without the solver, or is JIS only removed at inference? Is the solver differentiated through during training, and how many iterations are used at train versus test time?

  2. Can you report bone-length error (mean and variance over time) for KiJacoMMP, the "w/o JIS" variant, and the baselines? This is the direct evidence for the feasibility claim.

  3. How does JIS compare with a simple bone-length rescaling/projection or classical IK/PBD correction applied to the same preliminary predictions?

  4. Does JIS also improve the outputs of IAFormer and ST-MoE when applied as post-processing?

1 comment
AuthorsOpenReview ↗

Response to The Reviewer

We thank the reviewers for suggesting additional experiments to further clarify the independent contribution, transferability, and computational overhead of JIS. We have added the following experiments and analyses to the revised manuscript.

(1) Cross-backbone plug-in experiment (Section 4.6). To examine whether the benefits of JIS are limited to the velocity prediction architecture of KiJacoMMP, we apply it to the predictions of IAFormer and ST-MoE without changing their backbone architectures. After incorporating JIS, both methods exhibit lower JPE at all reported horizons. At 1.0 s, their errors decrease from 159.31 to 158.09 mm and from 158.95 to 157.54 mm, respectively. These modest but consistent improvements support the transferability of JIS. Meanwhile, the complete KiJacoMMP achieves a lower error of 153.01 mm, indicating that simply adding JIS cannot reproduce the performance of the complete framework.

(2) Bone length error comparison experiment (Section 4.7). To directly evaluate the bone segment length consistency targeted by JIS, rather than relying solely on joint coordinate errors, we introduce the bone length error metric, BLE, together with its calculation formula. BLE measures the mean absolute difference between predicted bone lengths and their reference lengths from the last observed frame. At 1.0 s, the BLEs of IAFormer and ST-MoE reach 21.701 and 22.994 mm, respectively. KiJacoMMP also exhibits increasing bone length deviations with longer prediction horizons when JIS is removed, whereas incorporating JIS reduces BLE to the order of 10⁻⁵ mm at all reported horizons. This provides direct evidence of consistency for the predefined fixed-length bone segments, but does not imply that complete kinematic or dynamic feasibility is guaranteed.

(3) Comparison with simpler solvers (Section 5.3). We compare JIS with edge-wise forward and reverse correction applied to the same preliminary predictions. Without correction, the JPE at 1.0 s is 159.11 mm. Edge-wise forward and reverse correction reduce it to 155.54 and 158.00 mm, respectively, indicating that simple corrections are also effective, but their processing order affects the results. JIS further reduces the error to 153.01 mm and achieves the lowest JPE at all three reported horizons. These results support the effectiveness of jointly updating shared joints through a coupled system relative to the edge-wise correction procedures tested in this experiment.

(4) Runtime efficiency experiment (added after the existing content of Section 5.2). We report mean inference latency and the P50 and P95 statistics for batches of 96 three-person scenes, measured over 30 runs after 10 warm-up runs. Without JIS, the mean latency is 132.325 ms. With 1, 3, and 10 iterations, it increases to 165.942, 203.848, and 340.159 ms, respectively, reaching 1140.3 ms with 50 iterations. For 10 iterations, P50 and P95 are 340.012 and 342.011 ms, respectively. We explicitly acknowledge the additional computational overhead introduced by JIS and explain that the iteration budget should be selected by jointly considering constraint satisfaction, prediction accuracy, and inference efficiency in practical applications.

Reviewer_7A9JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper solves the problem of multi-person motion prediction from historical observations. The paper proposes a method to explicitly incorporate kinematic-awareness and physical-skeletal constraints into the pipeline. Different from existed methods, which estimate the future joint 3D poses directly, the method instead estimates velocities. The final 3D poses are achieved by integration from previous poses. The estimated poses are further updated with skeleton correction, which favors the skeletal length constraint. Experiments are conducted with multiple numbers of individuals from multiple datasets. Ablation study validates the contributions of each proposed GCN components in the pipeline, the impact of different number of iterations, and the accuracy stability.

Critical strengths
  • The main contributions are from the kinematic awareness and the incorporation of physical constraint; these terms are well motivated. The kinematic awareness is obtained in the sense that the future 3D poses are estimated by integral of estimated velocity over time steps. This differs from baseline methods that estimates 3D joints directly.
  • The physical constraint is obtained in the sense that the estimated joints should preserve the previous skeletal length. This constraint is imposed during the Jacobian Iterative Solver. In which, the final joints predictions are corrected via the iterative loop, which update the predictions towards the skeleton solution that has same length with the last observed poses.
  • The experiments are conducted carefully and comprehensively and the proposed method achieves SOTA performance.
  • The experiments have examined multiple cases involving different numbers of people, with stable results across 20 runs.
  • Ablation study investigated the impact of each components, including the impact of kinematic awareness (base model), and the impact of physical constraint (w/o Jacobian Iterative Solver).
Critical weaknesses
  • The paper is generally understandable, but requires substantial effort of reading with multiple times to recognize the value of paper's contributions. The methodology writing is hard to relate to the motivation of kinematic-awareness and skeletal constraint construction.
  • The proposed acceleration field encoder is not well motivated, i.e. the estimated acceleration field is not being used explicitly in any kinematic-based or physical constraint-based formulations. Instead, it appears to be a latent representation, which is then being fused to estimate velocity field.
  • The interaction relation encoder block is not new. The baseline methods, e.g. JRFormer has handled this, but in a implicit way by concatenating the velocity information, relative distance between joints, skeleton structure of joints connecting to bones, and connectivity between joints. Whereas the paper claims that they explicitly encode the cross-interaction between individuals with a GCN together with self-attention layers. Therefore, methodology sections of 3.1, 3.2, 3.3 are not presented as the core contribution of this work and are unnecessarily lengthy and detailed.
  • The Jacobian Iterative Solver (JIS), which reflects the explicit kinematic awareness and physical injection, is expected to be detailed and further motivated, but written in a compact way.
Questions for authors

Q1: Please further motivate why the AFE block is necessary? Even the ablation study shows that AFE improves the accuracy, however AFE is not explicitly used in any kinematic part, and not every being supervised on during training, then why the present of AFE helps the method.

Q2: Why the interactions modeled by GCN is better than Transformer-based architecture? Transformer architecture in e.g. JRFormer implicitly models the interactions across individuals' joints, why proposed GCN only looks into interactions from a person's joints to the other person's chest. Is the any motivation for this construction? If this is the most straightforward to model the interaction relation, I do not consider it a substantial contribution.

Q3: In #252, author states that the last observed pose is X^50m\hat{X}^{m}_{50}, does this mean that the method only works when 50 historical observations are given? The number 5050 here is arbitrary, which causes confusing for first time of reading.

Q4: The notions of pe,cep_e, c_e in #257 and Eq. 11 are not defined? How the pep_e and cec_e are pre-selected from the given observation XgtmX^{m}_{gt}?

Q5: In Sec. 3.4, please specify which calculation part appears in training, and which appears in inference only.

Q6: The "signed incidence matrix B" in #266 is never defined any where, how to obtain it? This is crucial, without this clarification, the Jacobian matrix cannot be constructed, and therefore the iterative correction in Eq. 15 can not run.

Q7: The skeletal correction interactions happen in Eq. 15 using the edge multiplier update factor to direct the estimated pose towards the predefined skeletal length, defined in Eq. 11. However, this length constraint is always fixed at observation 5050, is it more natural to also re-calculated after new pose is estimated? By this, the estimation at future time will favor the physical length of the right previous state. By detail, I mean, given observation at t=50t=50, after the pose at t=51t=51 is estimated, the lel_e in Eq. 11 can be re-computed based on state t=51t=51, then the estimation at t=52t=52 in Eq. 15 will be consistent with state of t=51t=51.

Q8: The iteration process increases the accuracy, but it comes with extra computation cost. It is worthy to investigate how much computation is raised after 50 iterations.

1 comment
AuthorsOpenReview ↗

Response to The Reviewer

We thank the reviewers for suggesting additional experiments to further clarify the independent contribution, transferability, and computational overhead of JIS. We have added the following experiments and analyses to the revised manuscript.

(1) Cross-backbone plug-in experiment (Section 4.6). To examine whether the benefits of JIS are limited to the velocity prediction architecture of KiJacoMMP, we apply it to the predictions of IAFormer and ST-MoE without changing their backbone architectures. After incorporating JIS, both methods exhibit lower JPE at all reported horizons. At 1.0 s, their errors decrease from 159.31 to 158.09 mm and from 158.95 to 157.54 mm, respectively. These modest but consistent improvements support the transferability of JIS. Meanwhile, the complete KiJacoMMP achieves a lower error of 153.01 mm, indicating that simply adding JIS cannot reproduce the performance of the complete framework.

(2) Bone length error comparison experiment (Section 4.7). To directly evaluate the bone segment length consistency targeted by JIS, rather than relying solely on joint coordinate errors, we introduce the bone length error metric, BLE, together with its calculation formula. BLE measures the mean absolute difference between predicted bone lengths and their reference lengths from the last observed frame. At 1.0 s, the BLEs of IAFormer and ST-MoE reach 21.701 and 22.994 mm, respectively. KiJacoMMP also exhibits increasing bone length deviations with longer prediction horizons when JIS is removed, whereas incorporating JIS reduces BLE to the order of 10⁻⁵ mm at all reported horizons. This provides direct evidence of consistency for the predefined fixed-length bone segments, but does not imply that complete kinematic or dynamic feasibility is guaranteed.

(3) Comparison with simpler solvers (Section 5.3). We compare JIS with edge-wise forward and reverse correction applied to the same preliminary predictions. Without correction, the JPE at 1.0 s is 159.11 mm. Edge-wise forward and reverse correction reduce it to 155.54 and 158.00 mm, respectively, indicating that simple corrections are also effective, but their processing order affects the results. JIS further reduces the error to 153.01 mm and achieves the lowest JPE at all three reported horizons. These results support the effectiveness of jointly updating shared joints through a coupled system relative to the edge-wise correction procedures tested in this experiment.

(4) Runtime efficiency experiment (added after the existing content of Section 5.2). We report mean inference latency and the P50 and P95 statistics for batches of 96 three-person scenes, measured over 30 runs after 10 warm-up runs. Without JIS, the mean latency is 132.325 ms. With 1, 3, and 10 iterations, it increases to 165.942, 203.848, and 340.159 ms, respectively, reaching 1140.3 ms with 50 iterations. For 10 iterations, P50 and P95 are 340.012 and 342.011 ms, respectively. We explicitly acknowledge the additional computational overhead introduced by JIS and explain that the iteration budget should be selected by jointly considering constraint satisfaction, prediction accuracy, and inference efficiency in practical applications.

Reviewer_dKp5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

KiJacoMMP addresses multi-person 3D motion prediction. It is comprised of acceleration-field encoder (GCN plus self-attention), an interaction encoder based on offsets from each joint to other people's chest joints, and a cross-attention velocity decoder predict future joint velocities. These are integrated frame by frame from the last observed pose. Then, a Jacobian iterative solver (JIS) corrects the positions so that bone lengths match those of the last observed frame.

Critical strengths
  1. The paper shows consistent improvements on different datasets. Against baselines (such ST-MoE), average JPE drops from 95 to 90 mm on CMU-Mocap, 110 to 101 on Mix1 and 107 to 96 on Mix2.

  2. JIS needs no mass, inertia or simulator, converges in about 10 iterations (Fig. 5), and could in principle be attached to any position-based predictor.

Critical weaknesses
  1. JIS is a standard damped Gauss–Newton projection onto bone-length constraints and not that new. Similar approaches have been widely explored in human motion estimation, where keypoint positions are refined to satisfy bone-length constraints. (e.g. IKOL [1], SMPL-IK [2], HybrIK [3]).

  2. The authors should discuss using other simpleer post-processing optimizations methods, such as Adam or L-BFGS on a bone-length penalty.

  3. The method only enforces bone-length constraints without guaranteeing kinematic feasibility. The authors should clearly acknowledge and discuss this limitation in the paper.

[1]. Ikol: Inverse kinematics optimization layer for 3d human pose and shape estimation via gauss-newton differentiation [2]. SMPL-IK: Learned Morphology-Aware Inverse Kinematics for AI Driven Artistic Workflows [3]. HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation

Questions for authors
  1. What happens when JIS is applied as post-processing to IAFormer and ST-MoE? How does it compare with a bone-length loss or a joint-rotation output head with forward kinematics?

  2. Please compare JIS with simpler and generic alternatives: closed-form bone rescaling along the kinematic tree, and Adam or L-BFGS on a bone-length penalty.

  3. Why take bone lengths from a single observed frame rather than averaging over the history? How sensitive is JIS to noisy inputs, such as estimated rather than motion-capture poses?

1 comment
AuthorsOpenReview ↗

Response to The Reviewer

We thank the reviewers for suggesting additional experiments to further clarify the independent contribution, transferability, and computational overhead of JIS. We have added the following experiments and analyses to the revised manuscript.

(1) Cross-backbone plug-in experiment (Section 4.6). To examine whether the benefits of JIS are limited to the velocity prediction architecture of KiJacoMMP, we apply it to the predictions of IAFormer and ST-MoE without changing their backbone architectures. After incorporating JIS, both methods exhibit lower JPE at all reported horizons. At 1.0 s, their errors decrease from 159.31 to 158.09 mm and from 158.95 to 157.54 mm, respectively. These modest but consistent improvements support the transferability of JIS. Meanwhile, the complete KiJacoMMP achieves a lower error of 153.01 mm, indicating that simply adding JIS cannot reproduce the performance of the complete framework.

(2) Bone length error comparison experiment (Section 4.7). To directly evaluate the bone segment length consistency targeted by JIS, rather than relying solely on joint coordinate errors, we introduce the bone length error metric, BLE, together with its calculation formula. BLE measures the mean absolute difference between predicted bone lengths and their reference lengths from the last observed frame. At 1.0 s, the BLEs of IAFormer and ST-MoE reach 21.701 and 22.994 mm, respectively. KiJacoMMP also exhibits increasing bone length deviations with longer prediction horizons when JIS is removed, whereas incorporating JIS reduces BLE to the order of 10⁻⁵ mm at all reported horizons. This provides direct evidence of consistency for the predefined fixed-length bone segments, but does not imply that complete kinematic or dynamic feasibility is guaranteed.

(3) Comparison with simpler solvers (Section 5.3). We compare JIS with edge-wise forward and reverse correction applied to the same preliminary predictions. Without correction, the JPE at 1.0 s is 159.11 mm. Edge-wise forward and reverse correction reduce it to 155.54 and 158.00 mm, respectively, indicating that simple corrections are also effective, but their processing order affects the results. JIS further reduces the error to 153.01 mm and achieves the lowest JPE at all three reported horizons. These results support the effectiveness of jointly updating shared joints through a coupled system relative to the edge-wise correction procedures tested in this experiment.

(4) Runtime efficiency experiment (added after the existing content of Section 5.2). We report mean inference latency and the P50 and P95 statistics for batches of 96 three-person scenes, measured over 30 runs after 10 warm-up runs. Without JIS, the mean latency is 132.325 ms. With 1, 3, and 10 iterations, it increases to 165.942, 203.848, and 340.159 ms, respectively, reaching 1140.3 ms with 50 iterations. For 10 iterations, P50 and P95 are 340.012 and 342.011 ms, respectively. We explicitly acknowledge the additional computational overhead introduced by JIS and explain that the iteration budget should be selected by jointly considering constraint satisfaction, prediction accuracy, and inference efficiency in practical applications.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to The Reviewers

We thank the reviewers for suggesting additional experiments to further clarify the independent contribution, transferability, and computational overhead of JIS. We have added the following experiments and analyses to the revised manuscript.

(1) Cross-backbone plug-in experiment (Section 4.6). To examine whether the benefits of JIS are limited to the velocity prediction architecture of KiJacoMMP, we apply it to the predictions of IAFormer and ST-MoE without changing their backbone architectures. After incorporating JIS, both methods exhibit lower JPE at all reported horizons. At 1.0 s, their errors decrease from 159.31 to 158.09 mm and from 158.95 to 157.54 mm, respectively. These modest but consistent improvements support the transferability of JIS. Meanwhile, the complete KiJacoMMP achieves a lower error of 153.01 mm, indicating that simply adding JIS cannot reproduce the performance of the complete framework.

(2) Bone length error comparison experiment (Section 4.7). To directly evaluate the bone segment length consistency targeted by JIS, rather than relying solely on joint coordinate errors, we introduce the bone length error metric, BLE, together with its calculation formula. BLE measures the mean absolute difference between predicted bone lengths and their reference lengths from the last observed frame. At 1.0 s, the BLEs of IAFormer and ST-MoE reach 21.701 and 22.994 mm, respectively. KiJacoMMP also exhibits increasing bone length deviations with longer prediction horizons when JIS is removed, whereas incorporating JIS reduces BLE to the order of 10⁻⁵ mm at all reported horizons. This provides direct evidence of consistency for the predefined fixed-length bone segments, but does not imply that complete kinematic or dynamic feasibility is guaranteed.

(3) Comparison with simpler solvers (Section 5.3). We compare JIS with edge-wise forward and reverse correction applied to the same preliminary predictions. Without correction, the JPE at 1.0 s is 159.11 mm. Edge-wise forward and reverse correction reduce it to 155.54 and 158.00 mm, respectively, indicating that simple corrections are also effective, but their processing order affects the results. JIS further reduces the error to 153.01 mm and achieves the lowest JPE at all three reported horizons. These results support the effectiveness of jointly updating shared joints through a coupled system relative to the edge-wise correction procedures tested in this experiment.

(4) Runtime efficiency experiment (added after the existing content of Section 5.2). We report mean inference latency and the P50 and P95 statistics for batches of 96 three-person scenes, measured over 30 runs after 10 warm-up runs. Without JIS, the mean latency is 132.325 ms. With 1, 3, and 10 iterations, it increases to 165.942, 203.848, and 340.159 ms, respectively, reaching 1140.3 ms with 50 iterations. For 10 iterations, P50 and P95 are 340.012 and 342.011 ms, respectively. We explicitly acknowledge the additional computational overhead introduced by JIS and explain that the iteration budget should be selected by jointly considering constraint satisfaction, prediction accuracy, and inference efficiency in practical applications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.