acceptodds
Under review as a conference paper at ICLR 2027

MoPRU: Learning Motion Prediction and Recurrent Motion-State Update for 3D Single Object Tracking

Abstract

Motion-centric methods for LiDAR-based 3D single-object tracking (3D SOT) estimate relative target motion (RTM) from consecutive point-cloud frames for current target localization. Historical tracking information is further exploited for temporal motion modeling. However, learning the motion update only for current localization overlooks its effect on the recurrent motion state used for subsequent motion prediction. To address this issue, we propose MoPRU, a 3D SOT framework that unifies the learning of motion prediction and recurrent motion-state update. MoPRU learns how current motion information should be integrated into the recurrent motion state, extending motion-update learning beyond current-frame localization. Specifically, MoPRU aggregates historical tracking features into a recurrent motion state and predicts a motion prior for the current frame. The motion prior and current RTM determine a shared motion update, which refines the current RTM for target localization and updates the recurrent motion state for subsequent motion prediction. This motion update is learned by comparing short-horizon tracking results under alternative current updates. MoPRU achieves Success/Precision scores of 73.8/91.0 on KITTI and 62.93/75.66 on NuScenes, and improves zero-shot KITTI-to-Waymo transfer from 47.5/63.6 to 50.3/68.3.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D single-object tracking, LiDAR point cloud tracking, motion-centric tracking, relative target motion, motion prediction, recurrent motion-state update

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Zv2pOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies motion-centric single object tracking task on LiDAR pointclouds. The porposed method, MOPRU, combines the standard relative motion prediction and recurrent long-term motion modeling via a learnable gating mechanism. Experiments demonstrate MOPRU achieve state-of-the-arts performance on KITTI, NuScene and Waymo.

Critical strengths
  1. The distinction between a correction that gives a good current box and one that leaves a useful recurrent state is meaningful and novel.
  2. The short closed-loop horizon used in training can help the model learn from its consequences, alleviating the accumulated errors.
Critical weaknesses
  1. The mean improvements seem to be minor, especially on NuScenes and Waymo
  2. Figure 2 and Figure 3 are hard to follow. It is better to add more detailed description in the captions.
  3. The effect of the motion-state update in Equation(4) is not studied in the ablation experiments.
Questions for authors
  1. Is the model’s performance sensitive to the category-specific bounds cpc_p? How to set these bounds?
  2. How does the proposed supervision compare with a matched multi-frame unrolled training objective, and what happens when training history is longer than five frames?
1 comment
AuthorsOpenReview ↗

Clarifications on Recurrent Motion-State Updates, Supervision, and Experimental Analysis

We thank the reviewer for the constructive feedback. We have added fixed-parameter interventions (Table 6) and a comparison of motion update supervision objectives (Table 7) to the main text (p. 10), revised the captions of Figures 2 and 3 (pp. 3–4), and added sensitivity analyses of training clip length and motion clipping bounds (Tables A24–A25, Appendix H). We address the concerns below.

W1: NuScenes and Waymo improvements.

On NuScenes (Table 2), MoPRU improves SiamMo by +2.62/+2.98 Success/Precision points, reaching 62.93/75.66 and surpassing TrajTrack (62.25/75.15). On zero-shot KITTI-to-Waymo transfer (Table 3), it improves SiamMo by +2.8/+4.7 points over 427,483 frames, with +6.0/+7.3 on hard Pedestrian. MoPRU achieves the highest reported Vehicle/Pedestrian mean scores on Waymo.

W2: Figures 2 and 3.

We have revised both captions (pp. 3–4), with the changes highlighted in blue in the manuscript. Figure 2 presents MoPRU's unified learning of motion prediction and recurrent motion-state update. Shared update weights are learned from differences in short-horizon tracking outcomes to guide current RTM refinement and recurrent motion-state update for subsequent motion prediction. Figure 3 clarifies that four branches differ only in the current update, then evolve separately to construct training supervision. These branches are not used at inference.

W3: Recurrent motion state update (Eq. 4).

Equation (4) combines the pairwise motion feature and motion cues into the state-update input using the same weights as RTM refinement. With model parameters and current target boxes fixed, Table 6 reports next motion prior displacement/heading errors of 0.260 m/3.22° (learned), 0.273 m/3.38° (zero input), and 0.281 m/3.49° (no update). Under refined RTM, the corresponding H=2H=2 tracking errors are 0.780/0.802/0.816; under pairwise RTM, learned and zero input updates both yield 0.835. Recurrent-state updates affect subsequent localization through motion-prior-guided RTM refinement (Eqs. 3–5).

Q1: Motion clipping bounds.

Equation (3) bounds the correction toward the motion prior by (1−αtj)cj(1-\alpha_t^j)c_j, with category-specific bounds reported in Table A4. Table A25 uniformly scales these bounds while keeping other settings, including H=2H=2, unchanged.

Sensitivity to motion clipping bounds on KITTI (Table A25, Appendix H). Success/Precision (%).

Clipping bounds scaleSuccess ↑Precision ↑
0.50×73.290.6
0.75×73.690.8
1.00× (default)73.891.0
1.25×73.790.9
1.50×73.490.7

The default clipping bounds achieve the highest Success/Precision, while both tighter and looser bounds reduce performance, indicating the importance of an appropriate correction range for motion-prior-guided RTM refinement.

Q2: Motion update supervision.

Table 7 compares objectives under the same shared motion update structure.

Comparison of motion update supervision on KITTI (Table 7, p. 10). Success/Precision (%).

Supervision objectiveHorizonSuccess ↑Precision ↑
Current frame073.090.5
Direct future error272.890.4
Multi-frame unrolling273.490.7
Outcome differences (MoPRU)273.891.0

At H=2H=2, outcome differences outperform multi-frame unrolling and direct future error supervision. The four updates share the initial box and recurrent state, but evolve separate tracking rollouts after the current update (Sec. 3.3). Future observations are used only to construct training targets.

Training history and supervision horizon.

Training uses five-frame clips, resetting the recurrent state per clip; inference propagates it over the full tracklet (Sec. 4.1; Appendix A). The supervision horizon H=2H=2 covers two subsequent tracking frames. Table A24 varies clip length with the same supervision horizon and motion-update formulation.

Sensitivity to training clip length on KITTI (Table A24, Appendix H). Success/Precision (%).

Training clip lengthSuccess ↑Precision ↑
5 (default)73.891.0
773.690.8
973.490.7

Increasing clip length beyond five frames does not improve tracking performance.

Reviewer_PAJHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a new framework for single 3D object tracking. The key idea is to combine motion prediction with the recurrent motion update, which can not only help with the current localization but also predict how the motion will evolve for future frames given the motion histories. Normally, the prediction takes care of where the target should be before the sensor observations, and the recurrent motion-state update takes care of what the tracker should believe and remember for future motion; yes, their targets are different but correlated, so it does make a lot of sense to unify them. The results are fruitful and strong.

Critical strengths
  1. The results are strong, achieving the SOTA on KITTI and Zero-shot KITTI-to-Waymo transfer benchmarks, and second-best on NuScenes. The baseline approach, "SiamMo," is already strong, and the improvement is not trivial, though not particularly significant ( 72.3/90.1 VS 73.8/91.0). The ablation studies are rich, indicating that the shared weights do bring some small extra gains compared with separate weights. Overall, sections 4.3 and 4.4 are very informative and colorful to help understand the various design choices and behaviors.
  2. Overall, I like the motivation of this paper, and agree that aiming to unify motion prediction and recurrent motion update using shared model weights is a decent idea (which is simpler than how it is presented, though )
Critical weaknesses
  1. The paper's readability is poor, with many notations, making it very hard to follow. Consider simplifying the presentation and using fewer MathML notations, but more insight and intuition discussion. My overall impression is that the core idea is simpler than the way it is currently presented in the submission, over-complicated than it should be.
  2. This paper seems to mainly focus on improving the 3D SOT tracking accuracy, instead of the inference speed and cost, which is fine. But given such a heavy system, I am expecting that on the inference side it can be cheaper beyond just a moderate accuracy boost.
  3. It would be great to discuss the failure cases more, when this approach tends to fail, and its limitations and future work.
Questions for authors
  1. Aside from the 3 datasets, have you tried Argoverse 2?

  2. Any comments on how the approach might inspire multi-object tracking?

1 comment
AuthorsOpenReview ↗

Clarifications on Method Presentation, Inference Efficiency, Limitations, and 3D MOT Extension

We thank the reviewer for the careful review and constructive suggestions. We have added fixed-parameter interventions (Table 6) and a comparison of motion update supervision objectives (Table 7) to the main text (p. 10), revised the captions of Figures 2 and 3 (pp. 3–4), with changes highlighted in blue, and added sensitivity analyses of training clip length and motion clipping bounds (Tables A24–A25, Appendix H). We address the concerns below.

W1: Presentation and readability.

We have revised the captions of Figures 2 and 3 (pp. 3–4), with changes highlighted in blue. Figure 2 presents MoPRU's unified learning of motion prediction and recurrent motion-state update. Differences in short-horizon tracking outcomes supervise shared weights for current RTM refinement and recurrent motion-state update for subsequent motion prediction. Figure 3 clarifies that four branches share the previous box and recurrent state, differ only in the current update, and subsequently track independently to construct training targets. We also streamlined the overview (Sec. 3.1) and clarified the shared weights' two roles in Sec. 3.2 without changing the update formulation.

W2: Inference efficiency.

Four-branch supervision is confined to training; inference uses a single update. Table A21 reports online tracking efficiency using the original hardware measurements of each tracker.

Online tracking efficiency (Table A21, Appendix H). Reported measurements from the corresponding trackers.

TrackerComputation (GFLOPs per tracking step)Tracking throughput (FPS)GPU
SiamMo0.82108RTX 4090
PointRePar6.6637RTX 3090
MoPRU1.39101RTX 4090

MoPRU achieves 101 FPS versus SiamMo's 108 FPS, with an additional 0.57 GFLOPs per tracking step. This runtime overhead accompanies +1.5/+0.9, +2.62/+2.98, and +2.8/+4.7 Success/Precision gains on KITTI, NuScenes, and zero-shot KITTI-to-Waymo transfer, respectively (Tables 1–3).

W3: Failure Analysis and Limitations.

Appendix G (Fig. A5) shows a target-loss case: when the target leaves the search region, the current RTM becomes unreliable, and the recurrent motion prior may not prevent tracking drift. On KITTI Pedestrian, Success decreases from 68.6 to 67.7, despite a Precision gain from 93.9 to 94.2 (Table 1). Table A20 in Appendix F further shows that Pedestrian Success is sensitive to the heading update in both RTM refinement and recurrent motion-state update. These limitations motivate future work on target reacquisition and more reliable motion updates under incomplete observations.

Q1. Evaluation on Argoverse 2.

We evaluate MoPRU within AB3DMOT on the REGULAR_VEHICLE category of the Argoverse 2 Sensor validation split (Appendix I, Table A26).

3D MOT extension on Argoverse 2 (Table A26, Appendix I). REGULAR_VEHICLE HOTA (%) on the Sensor validation split.

MethodHOTA ↑Δ HOTA
AB3DMOT63.8–
AB3DMOT + MoPRU64.9+1.1

Under identical detection inputs and association settings, MoPRU improves HOTA by 1.1 points, supporting the applicability of the learned motion update to 3D multi-object tracking.

Q2. Extension to multi-object tracking.

We extend MoPRU to a tracking-by-detection framework by maintaining an independent recurrent motion state for each track. The predicted motion prior guides data association. For matched tracks, shared update weights control current RTM refinement and recurrent motion-state update, which informs subsequent motion prediction. This retains the original detection inputs, association algorithm, and track management procedures. The results in Table A26 further support this extension.

Reviewer_TsEmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MoPRU combines pairwise motion estimation with a recurrent motion prior for LiDAR-based 3D single-object tracking. Learned translation and yaw weights control both current-frame refinement and recurrent-state writing. These weights are supervised by comparing alternative updates over the current frame and two subsequent frames. Future observations are used only during training. With SiamMo, the method achieves 73.8/91.0 Success/Precision on KITTI and 62.93/75.66 on nuScenes, with additional gains in KITTI-to-Waymo transfer and on MBPTrack.

Critical strengths
  1. The method addresses how a current update affects both immediate localization and future predictions. Sharing component-wise weights across localization and state writing is a concrete design, supported by several relevant ablations.

  2. The evaluation covers two datasets, cross-dataset transfer, and a second backbone. Five-seed results and bootstrap intervals strengthen the empirical evidence. Matched-hardware inference remains practical at 100 FPS, compared with 106 FPS for SiamMo; the increased training cost is also reported.

Critical weaknesses
  1. The inference-time role of the state in State-only is unclear. Section 4.3 states that this configuration uses pairwise motion directly for localization. Figure 2 shows the recurrent state affecting localization through the prior and refinement. If that path is disabled at every frame, with no other state-conditioned connection, state writes should not affect predicted boxes for a fixed checkpoint. Improvements after retraining could instead reflect joint training. This distinction matters because Appendix F attributes the State-only gains to recurrent-state evolution. Table A20 provides a useful intervention in the full model, but does not clarify the State-only configuration.

  2. Beyond motion-prior fusion in TrajTrack and recurrent memory in ChronoTrack, the contribution is the shared update rule and its outcome-difference targets. Tables A13 and A14 provide relevant supervision controls, while Appendix C examines continuous weights and normalization. However, the KITTI target-quality and dense-grid diagnostics use 4,800 training frames (Table A1). It remains unclear whether the reported relationship between binary-derived targets and continuous update quality holds on unseen sequences. This limits the generality of the mechanism analysis. Table A10 also needs to distinguish the splits used for its diagnostic score gaps and tracking metrics.

Questions for authors
  1. Does the recurrent state condition the pairwise predictor or another localization input in State-only? Is refinement disabled throughout inference? Please provide pseudocode and distinguish retrained ablations from inference-only interventions. With a frozen checkpoint and the current box fixed, can changing the state write affect subsequent predictions in this configuration?

  2. Can the target-versus-grid diagnostics be repeated on held-out sequences? Please specify the split used for each column of Table A10 and clarify the influence of the (0,0) branch, which supplies a zero input to the state update, on the target-quality analysis.

  3. Does MoPRU train separate models for individual categories, or share one model across categories? Please state the training regime explicitly for each dataset and distinguish category-specific from category-unified results in the benchmark comparisons, including PointRePar.

1 comment
AuthorsOpenReview ↗

Clarifications on Motion-State Updates, Evaluation Protocols, and Benchmark Comparisons

We thank the reviewer for the careful assessment and constructive questions. We have revised the main text (p. 10) by adding Table 6 to examine recurrent motion-state updates under fixed model parameters, Table 7 to compare motion update supervision objectives, and Figure 7 to evaluate update quality on training and held-out validation sequences. We address the concerns below.

W1 / Q1: State-only configuration and recurrent feedback.

In State-only, learned weights update the recurrent motion state, while localization directly uses pairwise RTM. RTM refinement is disabled throughout inference, and FpairF_{\mathrm{pair}} does not receive the recurrent state or motion prior. The execution is:

(m_cur, f) = F_pair(P_prev, P_cur, B_prev)  # Pairwise RTM and features

(m_prior, h) = F_state(M_prev)  # Motion prior and context

z_p = m_cur.p - m_prior.p  # Translation residual

z_psi = wrap(m_cur.psi - m_prior.psi)  # Yaw residual

(alpha_p, alpha_psi) = sigmoid(g_theta(h, z_p, z_psi, s))  # Shared update weights

B_cur = Transform(B_prev, m_cur)  # Current target localization

q = RMSNorm(0.5*(alpha_p + alpha_psi)*E_f(f)
            + alpha_p*E_p(r_p) + alpha_psi*E_psi(r_psi))  # State-update input

M_cur = Write(M_prev, q)  # Recurrent state update

With a frozen checkpoint, changing the state write cannot affect subsequent predicted boxes in State-only. The 73.6/90.8 Success/Precision in Table 4 comes from a separately trained configuration, not an inference-time effect of recurrent-state writing.

Effect of recurrent motion state updates on subsequent tracking (Table 6, p. 10). Model parameters and current target boxes are fixed. Displacement and heading denote next-frame motion-prior errors; tracking error is −R-R at H=2H=2.

RTM for localizationMotion state updateDisplacement (m) ↓Heading (°) ↓Tracking error ↓
Refined RTMLearned0.2603.220.780
Refined RTMZero input0.2733.380.802
Refined RTMNo update0.2813.490.816
Pairwise RTMLearned0.2603.220.835
Pairwise RTMZero input0.2733.380.835

Under pairwise RTM, state updates change the next motion prior without affecting subsequent tracking error. Under refined RTM, the learned update improves the next motion prior and reduces subsequent tracking error through motion-prior-guided RTM refinement (Eqs. 3–5).

W2 / Q2: Held-out update quality, evaluation splits, and the (0,0) branch.

For Table A10, Success/Precision are evaluated on KITTI test sequences 19–20 (Sec. 4.1), while normalized score gaps are computed on 4,800 training anchors from sequences 0–16 (Table A1). These metrics assess tracking performance and short-horizon motion update quality, respectively.

Continuous motion update quality on KITTI (Figure 7, p. 10). Mean normalized gaps under H=2H=2 closed-loop tracking (lower is better).

Motion updateTraining sequences (0–16)Held-out validation (17–18)
Best binary update0.0460.052
Short-horizon target weights0.0190.024
MoPRU predicted weights0.0280.034

MoPRU predictions achieve lower normalized gaps than the best binary update on both splits, preserving the relative quality of continuous motion updates on unseen sequences.

In the (0,0)(0,0) branch, both RTM components receive the maximum bounded correction, with qt=0q_t=0 and Mt=Write(Mt−1,0)M_t=\mathrm{Write}(M_{t-1},0) rather than skipping the state update (Sec. 3.3). Its tracking score R00R_{00} contributes to both component-wise supervision targets through Eqs. (8–9). Table A15 reports 73.4/90.7 (00 reference), 73.6/90.8 (11 reference), and 73.8/91.0 (averaging both references).

Q3: Category-specific training and benchmark comparisons.

MoPRU uses category-specific training on KITTI and NuScenes, with independently trained checkpoints for four and five categories, respectively. On KITTI, loss weights, target temperatures, and motion clipping bounds are category-dependent (Table A4).

Table A16 compares shared and category-dependent update parameters under the same category-specific training protocol. On KITTI, Success/Precision are 73.4/90.8 and 73.8/91.0, respectively. Both use independently trained category checkpoints.

PointRePar adopts category-unified training, whereas PointReParCS uses category-specific training (Liu et al., 2026; see our Tables 1–2). On NuScenes, MoPRU and PointReParCS both use category-specific training, achieving 62.93/75.66 and 64.15/76.81 Success/Precision, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.