acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Numerical Forecasting Models for Language-Based Trajectory Prediction

Abstract

Language-based trajectory predictors represent coordinates as discrete tokens and learn auxiliary tasks such as destination and group reasoning. This formulation enables the model to capture behavioral intent and social context beyond coordinate dynamics alone. However, token-level objectives provide only indirect guidance for continuous coordinate-space dynamics. To address this limitation, we introduce MoRE (Mixture Of Reward Experts), a refinement framework that transfers numerical forecasting priors into a pretrained language-based predictor through reinforcement learning. Five frozen numerical predictors provide complementary coordinate-level knowledge of motion and interactions. Their predictions are converted into expert rewards and combined through an uncertainty-weighted consensus that penalizes disagreement. A ground-truth reward anchors the prediction to the target trajectory. To focus refinement on difficult cases, MoRE refines the policy using the top 1% of training samples ranked by predictive entropy. Expert predictions are computed once and cached before PPO training, so the experts are not run during policy updates or inference. In this way, MoRE combines the contextual modeling of the language-based predictor with coordinate-level feedback from numerical experts. On ETH-UCY, MoRE reduces ADE from 0.22 to 0.20 m and FDE from 0.32 to 0.29 m. Relative to the base policy, ADE decreases by 17.9% on SDD and 12.7% on NBA. On ETH-UCY, MoRE also reduces collision rates and better matches ground-truth pedestrian spacing, without increasing measured inference memory or latency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Trajectory Prediction, Language-Based Prediction, Multi-Expert Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tG9QOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MoRE, a refinement framework for a pretrained language-based trajectory predictor. Five frozen numerical forecasting models provide cached predictions, which are converted into coordinate-space rewards. These rewards are aggregated using a mean-minus-standard-deviation objective and combined with a ground-truth reward. The policy is refined through reward-guided optimization interleaved with supervised learning, using LoRA and the top 1% of training samples ranked by base-policy entropy. Experiments on ETH–UCY, SDD, GCS, and NBA report improved best-of-20 forecasting accuracy, supplemented by collision and spacing measurements, behavioral-subset analyses, and ETH–UCY-to-SDD transfer evaluation.

Critical strengths
  • The reported improvements extend across several datasets and evaluation settings.
  • The framework provides a practical way to introduce coordinate-space feedback without deploying the numerical experts.
Critical weaknesses
  • The necessity and complementarity of the five-expert combination are insufficiently established. The paper assigns different forecasting roles to the five models, but these roles largely follow their architectural descriptions rather than a direct assessment of complementary capabilities and errors. Table 9 provides useful single-expert, leave-one-out, replacement, and addition experiments. However, the best listed single-expert configuration, MoFlow, achieves 0.206/0.307 compared with 0.204/0.285 for the five-expert configuration, and the robustness of this difference across training seeds is unknown. The study also does not establish whether a smaller expert subset would provide comparable benefits. Thus, the results favor the selected combination at the reported point estimates, but do not sufficiently explain why these five experts are warranted or whether the gains specifically arise from complementary forecasting knowledge. A clearly specified ground-truth-only reward baseline, followed by matched-budget comparisons with a strong single expert and smaller expert subsets, is needed to substantiate the central contribution.
  • The interpretation of expert consensus as transferring complementary forecasting knowledge needs stronger distribution-level evidence. Appendix A.6 correctly shows that the mean expert reward is, up to a candidate-independent constant, negative MSE to the average cached expert trajectory. The UWO penalty discourages unequal expert assessments, but disagreement can represent valid alternative futures rather than unreliable guidance. Consequently, the reward is plausible as a regularizer, while preservation of complementary multimodal knowledge is less clearly demonstrated. The mean-aggregation and distillation controls are useful, but best-of-20 errors and collision statistics do not determine whether improvements retain plausible mode coverage or suppress legitimate alternatives. An analysis relating expert disagreement to prediction correctness and mode coverage would materially strengthen the motivation. The absence of variability across training seeds also limits confidence in the reported differences between expert configurations.
  • The statistical reliability of the reported improvements is unclear. The paper does not report means and variances across independent training runs. The authors should repeat the main experiments and key ablations with different random seeds, keeping the data splits and evaluation protocols fixed, and report the mean and variance of ADE/FDE.
Questions for authors

See above weaknesses

Reviewer_QULcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MoRE (Mixture of Reward Experts), a refinement framework for language-based trajectory prediction. Starting from a pretrained LMTraj model, MoRE uses predictions from five frozen numerical trajectory predictors as coordinate-level reward signals. These rewards are aggregated using an uncertainty-weighted consensus objective and combined with a ground-truth reward to refine the language-based predictor using PPO. To reduce refinement cost, the method applies expert-guided refinement only to the top 1% of training samples ranked by predictive entropy. The numerical experts are used only during training and therefore introduce no additional inference-time cost. The method is evaluated on ETH-UCY, SDD, GCS, and NBA, with additional analyses of social interactions, nonlinear behaviors, cross-dataset transfer, expert configurations, reward aggregation, sample selection, and computational cost.

Critical strengths
  • The idea of using numerical trajectory predictors to provide coordinate-level feedback to a language-based predictor is intuitive and well motivated.
  • The method shows consistent improvements over LMTraj-SUP across several standard benchmarks.
  • The experiments are reasonably comprehensive, including ablations, nonlinear behavioral subsets, social-interaction statistics, cross-dataset transfer, and computational analysis.
Critical weaknesses
  1. Generalization across domains is not sufficiently demonstrated.

Although MoRE is evaluated on ETH-UCY, SDD, GCS, and NBA, the main evaluations appear to train/refine the model separately under each benchmark's protocol. Thus, these results demonstrate that MoRE can be applied to multiple datasets, rather than that the learned model or transferred numerical priors generalize across domains. The ETH-UCY --> SDD experiment without fine-tuning is encouraging, but it represents only one source-target pair. This is particularly important for MoRE because its refinement relies on pretrained numerical experts whose priors may themselves be domain-dependent or share systematic errors under distribution shift. Consequently, the current evidence is insufficient to determine how robust the proposed expert-guided refinement is beyond the domains represented during training.

  1. The improvement over the closely related W2W method is limited and insufficiently analyzed.

W2W is particularly relevant because it also optimizes a language-based trajectory predictor using reinforcement learning, while MoRE's main distinction is the introduction of numerical-expert reward feedback. However, the paper provides limited discussion of their technical differences and little analysis of when or why the proposed numerical-expert rewards provide an advantage. On ETH-UCY (Table 1), the results are mixed: MoRE improves average ADE only slightly (0.20 vs. 0.21), has the same average FDE (0.29), and is slightly worse than W2W on some individual scenes. The improvement is clearer on SDD (6.4/9.5 vs. 7.4/10.1), while W2W results are unavailable for GCS and NBA. Given the close methodological relationship and relatively small gains on the primary ETH-UCY benchmark, stronger differentiation and analysis relative to W2W are needed to establish the significance of the proposed contribution.

  1. The effectiveness of entropy-based sample selection is not directly validated.

The paper shows that the top-entropy 1% contains harder, more nonlinear, and more collision-involved samples, but the random and bottom-entropy subsets are used only to characterize sample difficulty. This establishes that entropy identifies difficult cases, but not that prioritizing these cases leads to more effective refinement. In particular, high-entropy samples could be informative hard cases, but they could also represent rare, ambiguous, or long-tail cases that contribute less to overall generalization. Since entropy-based selection is a key component of the proposed efficient refinement strategy, a matched comparison between top-entropy 1% and random 1% under the same MoRE refinement procedure is important for substantiating its contribution.

Questions for authors
  1. Are separate models trained/refined for the main SDD, GCS, and NBA evaluations? If so, please clarify this distinction from the ETH-UCY --> SDD cross-dataset experiment. Could the authors provide additional evidence or discussion regarding larger domain shifts, particularly when the numerical experts may share domain-specific biases?

  2. Can the authors compare top-entropy 1% selection against random 1% selection using the same MoRE refinement procedure and training budget?

  3. Given the close results with W2W on ETH-UCY, what are the key technical differences between W2W and MoRE, and what might explain why MoRE provides a clearer improvement on SDD but only marginal improvement on ETH-UCY?

  4. How does the formulation extend to longer-term operation across successive prediction windows, where agents may enter or leave the scene and social groups may form, split, merge, or change membership?

  5. Although best-of-20 is standard for these benchmarks, could the authors additionally report performance with fewer samples (e.g., best-of-3/5/10)? This would help determine whether the improvement persists when a downstream application cannot rely on oracle selection among 20 candidate trajectories.

Reviewer_mF42OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper improve token-base trajectory model LMTraj, with five numerical prediction experts supplying trajectories. The method combines the mean expert reward with a penalty on score disagreement and an explicit ground-truth reward, then refines a policy using LoRA and interleaved supervised and PPO updates. The reported evaluation covers ETH-UCY, SDD, GCS, NBA, and ETH-UCY-to-SDD transfer scenarios.

Critical strengths

The paper reports improved displacement errors on four datasets and in cross-dataset transfer, as well as lower ETH-UCY collision rates and spacing statistics closer to the ground truth. The gains over the base model are particularly substantial on SDD and NBA. If the evaluation protocol is confirmed, this is useful evidence that numerical feedback can improve a language-based trajectory predictor without retaining the experts at deployment.

Critical weaknesses
  1. The reason why the authors pick 5 numerical experts looks mysterious. The paper would benefit from a more rigorous explanation of the selection criteria and the specific contributions of each expert.
  2. In the paper, the writers do not specify clearly why PPO algorithm is selected for this type of problem. Given the problem formulation, it is unclear whether a critic-based method is necessary or whether simpler alternatives like GRPO, or other policy-gradient methods without a learned value function, could achieve comparable performance with reduced computational complexity. Also, in NBA dataset, the paper does not compare MoRE with several competitive trajectory forecasting methods, like LED, MoFlow.
Questions for authors

What makes PPO particularly suitable for this problem compared to other policy optimization algorithms?

Can the authors provide comparisons with stronger trajectory forecasting baselines, particularly LED and MoFlow, on the NBA dataset? If there are constant improvements over these datasets, I will consider raising my scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.