acceptodds
Under review as a conference paper at ICLR 2027

JFER: Joint Future Exploration and Reasoning for Interactive Motion Forecasting

Abstract

Interactive motion forecasting provides downstream planning with a small set of joint futures, although the number of possible agent-intention combinations can far exceed the prediction budget. Similar hypotheses may occupy several output slots, leaving other plausible interactions unrepresented. We introduce JFER, a Joint Future Exploration and Reasoning framework that addresses this limitation through structured hypothesis construction and complementary exploration. JFER organizes agent-level intentions into a joint-intention bank and assigns six prediction slots to selected pairs before full trajectory decoding. It then identifies additional intentions using their predicted support and distance from the decoded joint endpoints. Each selected intention guides the adaptation of an existing trajectory pair, producing a complementary hypothesis without another full decoding pass. Temporal and cross-candidate attention refine the expanded set, and reliability-constrained selection retains six joint predictions. On the Waymo Open Motion Dataset (WOMD) Interaction Prediction benchmark, JFER achieves a test Soft-mAP of 0.2868. Candidate-set analyses show that complementary exploration increases joint-future coverage and that part of this gain is retained under the original six-mode output budget, with larger gains at longer prediction horizons.

Primary area:
applications to robotics, autonomy, planning
Keywords:
interactive motion forecasting, joint motion forecasting, multimodal prediction, autonomous driving, trajectory prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ANe1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

JFER constructs a bank of joint endpoint intentions, decodes six base trajectories, and selects six additional intentions based on predicted support and endpoint coverage. It creates trajectories for the additional intentions by adapting base trajectories, then refines twelve candidates and returns six predictions. The paper reports a test Soft-mAP of 0.2868 on WOMD Interaction Prediction.

Critical strengths

Interactive motion forecasting needs to represent plausible joint futures while avoiding repeated predictions of similar outcomes. JFER separates these concerns by first selecting joint intentions supported by the scene and then checking the decoded trajectories for endpoint regions that remain underrepresented. This distinction between plausibility and coverage provides a useful way to think about candidate selection.

Critical weaknesses

The paper’s main weakness is that its many design choices are difficult to understand, insufficiently motivated, and incompletely specified.

Presentation

  • The method is difficult to follow. Key steps are introduced without a clear account of how they connect; for example, the anchor selection mentioned in Line 171 is explained only in Appendix A.3 (Lines 688–708).
  • Core representations are unclear. Eq. (3) does not adequately define how the intention features are constructed or how scene context differs from scene memory (Lines 164–181).

Design motivation

  • Same-index reference pairs lack a rationale. Eq. (4) favors pairs formed from the two agents’ anchors at the same slot index, without explaining why those indices correspond across agents (Lines 182–187).
  • Donor-based construction is difficult to justify. A new intention may involve different interaction timing or crossing order from the nearest base trajectory; endpoint proximity alone does not explain why that trajectory is a suitable motion donor (Lines 214–240).

Implementation and evidence

  • The bank-prior logit is underspecified. The paper states that ℓpbank\ell_p^{\mathrm{bank}} combines marginal intention evidence with learned pair compatibility, but does not specify how these terms are computed and combined. This matters because the logit affects both initial pair assignment in Eq. (4) and complementary selection in Eq. (5) (Lines 180–187, 196–211).
  • Assignment supervision is unclear. The paper mentions straight-through assignment and coverage and uniqueness objectives, but does not clearly connect these objectives to the energy in Eq. (4) (Appendix A.6, Lines 750–754).
  • Trajectory generation and refinement are underspecified. The inputs used to predict the corrections in Eq. (8), and the temporal tokens and prediction features used in Sec. 3.4, are not sufficiently defined (Lines 227–253).
  • Final selection is underspecified. Selection and admission logits determine which candidates are output, yet their input features and prediction heads are unclear; the supervision of selection logits is also unclear.
  • The source of the main performance gain is unclear. In Table 3, Direct uses the same scene encoder and joint decoder as the Structured Base but bypasses the entire intention bank and pair-assignment pipeline. The large gap therefore does not isolate the benefit of structured candidate selection from the effects of additional features, conditioning, or model capacity. Meanwhile, the gains from the individual design choices in Tables 4 and 5 are relatively small. It remains unclear which part of the proposed method accounts for the substantial improvement in Table 3.
Questions for authors
  1. How are the intention features, scene context, and bank-prior logit computed (Eqs. (3)–(4))? Why are the two agents’ anchors paired by slot index, and how is the assignment energy supervised?

  2. What donor and new-intention features are used to predict the gate and corrections in Eq. (8)? How can donor adaptation represent a different interaction timing or crossing order?

  3. How are the temporal tokens and residuals in Sec. 3.4 constructed? What produces the selection and admission logits, and how is each score supervised?

  4. Which components account for the large gain over Direct in Table 3? Please separate the effects of the bank, pair selection, role-aware assignment, and query adaptation, and quantify how much complementary expansion contributes to the final six predictions.

Reviewer_gundOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses joint motion forecasting for two interacting agents on the Waymo Open Motion Dataset (WOMD) Interaction Prediction benchmark. The benchmark allows only six joint predictions. The authors argue that several of these six often describe the same outcome, so other plausible interactions are left out.

JFER builds on the MTR encoder. For each agent it keeps eight endpoint intentions, and pairs them into an 8 x 8 "joint-intention bank" of 64 endpoint pairs. Six prediction slots each pick one distinct pair, and only these six are decoded into full trajectories (the "structured base"). Next, six more pairs are chosen that have support but lie far from the decoded endpoints. Instead of being decoded, each one is made by warping the nearest decoded trajectory pair toward the new endpoint. The resulting pool of twelve candidates is refined with temporal and cross-candidate attention. A rule-based selection then keeps at least four base candidates and admits at most two complementary ones.

The method reports a test Soft-mAP of 0.2868 on WOMD, second to Parallel ModeSeq among the listed methods. Validation ablations cover structured versus direct decoding, allocation and bank size, refinement and consolidation, and selection rules. Candidate-set diagnostics (coverage, endpoint duplication, oracle Soft-mAP, rescue/harm counts) show where the gains come from, and a cost table reports about 51% more time per scene.

Critical strengths

S1. A real problem in joint forecasting. With only six outputs, spending several slots on near-duplicate outcomes is a known weakness. The paper states this problem clearly. It also separates two things that are often mixed: how many intention combinations are considered, and how many trajectories are actually decoded.

S2. Transparent analysis of where gains come from. The paper does not stop at leaderboard numbers. It tracks coverage and duplication at each stage (base, expanded pool, refined pool, final set). It reports rescue and harm counts for the final selection (302 recovered, 9 lost), and measures the extra cost of the added modules. It also shows failure cases and states that endpoint-only intentions can miss differences in crossing order. Code is released.

Critical weaknesses

W1. Limited novelty, and the method does not model the case that motivates it. This is my biggest concern. Most parts of JFER are known in motion forecasting. Endpoint anchors come from TNT/DenseTNT and MTR. Distance-based suppression to pick six outputs is used in MTR. Attention-based trajectory refinement is used in MTR, QCNet, and SmartRefine. Pairing two agents' marginal intentions into a joint set is also standard. What is new is mainly the glue between these parts: a slot-to-pair assignment, a smoothstep "donor warp" (Eq. 8), and a rule-based selection with several hand-set thresholds.

The introduction motivates the work with crossing order and yielding at an intersection. But a "joint intention" in JFER is only a pair of endpoints. Two futures where one car yields or the other goes first can end at the same endpoints. The conclusion admits this: endpoint-based selection "may overlook interaction outcomes that reach similar destinations but differ in crossing order or timing." So the method does not address the exact interaction case the paper uses to motivate it. The overall gain over the structured base is +0.014 Soft-mAP on validation (0.2727 to 0.2868). It costs about 51% more time per scene (Table 10).

W2. Narrow evaluation, without error bars. All results are on one dataset, WOMD Interaction Prediction, and only for two agents. Other interactive benchmarks, such as the INTERACTION dataset and the Argoverse 2 multi-agent track, are not used, so it is unclear whether the hand-set rules transfer. No result has a standard deviation or more than one seed. Many ablation gains are very small. Table 4 ranges from 0.2698 to 0.2727. Table 8 goes from 0.2833 to 0.2868. Table 5 goes from 0.2818 to 0.2868. Without variance across seeds, these differences may be noise.

Two comparisons also look weaker than they should be. In Table 3, the "Direct" baseline reaches only 0.1918 val mAP. MTR++ reaches 0.2398 val mAP with the same kind of encoder. So the large gain from structured construction is partly due to a weak baseline. In Table 1, the validation block leaves out Parallel ModeSeq and IMPACT. Parallel ModeSeq is better than JFER on every test metric. The claim of the highest validation mAP holds only because the strongest method is missing from that block.

Questions for authors

Q1. Why six prediction slots? Six outputs are fixed by WOMD (Eq. 1). But six is also used inside the method: six world-context tokens, six base slots, and six complementary candidates (E = 6). Table 7 stops at E = 6 while Soft-mAP is still rising (0.2819, 0.2847, 0.2858, 0.2868). What happens with E = 8 or 12? In the same way, Table 4 stops at M = 8 intentions per agent. Why not more? How were R = 2, the 0.60 admission threshold, the 0.05 margin, and the 2.0 m suppression distance chosen? Were they tuned on the validation set that is also reported in Table 1?

Q2. Why only one dataset? Can you report results on another interactive benchmark (for example INTERACTION or the Argoverse 2 multi-agent track)? This would show that the bank, the donor warp, and the selection thresholds are not tuned to WOMD.

Q3. Doesn't the number of joint predictions grow combinatorially? The bank is a Cartesian product, so it has M x M = 64 entries for two agents (Eq. 2). For N agents it becomes M^N. How would JFER work for scenes with more than two interacting agents? What happens to the cost of greedy assignment and complementary selection?

Q4. Crossing order. Since joint intentions are endpoint pairs, how can JFER represent two futures with the same endpoints but a different crossing order? Can you report how often the final six predictions contain both orders in scenes where the order is ambiguous?

Q5. Direct baseline (Table 3). Why does Direct perform far below MTR++ on validation (0.1918 vs 0.2398 mAP) with the same encoder? Was it trained with the same epochs and tuning as the structured base?

Q6. World-context module (App. A.2). This module is trained with a prior and a posterior and then frozen. It is never ablated. What is the result without it?

Minor:

Q7. Table 1 reports Soft-mAP 0.2868 on both validation and test. Please confirm that both numbers are correct.

Q8. Please add mean and standard deviation over several seeds for Tables 3 to 8.

Reviewer_a4QMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies interactive motion forecasting, focusing on improving joint-future coverage under a limited prediction budget. For two interacting agents, the number of possible intention combinations can greatly exceed the number of prediction outputs. Existing methods may allocate multiple prediction slots to similar future patterns and miss other plausible interaction outcomes. However, increasing the number of predictions usually requires additional trajectory decoding, resulting in higher computational costs. To address this problem, the paper proposes JFER, a framework that explicitly models joint intentions and explores diverse joint futures while maintaining a fixed six-mode prediction budget.

Specifically, JFER first constructs an agent-level intention bank and selects joint intention pairs before trajectory decoding, assigning limited prediction slots to different joint intentions. It then performs complementary exploration to identify additional intentions that are insufficiently covered by the initial predictions and generates complementary trajectories by adapting existing trajectories guided by selected intentions. The expanded candidate set is further refined through temporal attention and cross-candidate attention, followed by reliability-constrained consolidation to obtain the final six predictions.

Experiments are conducted on the Waymo Open Motion Dataset (WOMD) Interaction Prediction benchmark, where JFER achieves a Soft-mAP of 0.2868 and demonstrates competitive performance compared with existing state-of-the-art approaches. The paper further provides extensive analyses to investigate the effectiveness of each component.

Critical strengths

The paper addresses an important and practical problem in interactive motion forecasting: how to improve joint-future coverage under a limited prediction budget. By explicitly considering the space of agent-intention combinations rather than directly generating a fixed number of trajectory hypotheses, the paper introduces a reasonable perspective for allocating prediction capacity. The proposed JFER framework provides a comprehensive pipeline including joint-intention construction, complementary exploration, trajectory adaptation, and candidate consolidation. In particular, generating complementary hypotheses by adapting existing trajectories instead of performing additional full decoding provides an alternative strategy for exploring more joint futures without simply increasing the number of decoded trajectories.

Critical weaknesses

Limited Contributions of the Core Modules Despite Substantial Computational Overhead. Although JFER introduces a conceptually promising exploration mechanism, Table 5 shows that its two key modules, trajectory refinement and candidate consolidation, yield negligible performance gains across all evaluation metrics, casting serious doubt on their effectiveness and practical necessity. Meanwhile, Table 10 indicates that the complete framework increases the average processing time per scene by 50%. This substantial computational overhead, coupled with limited performance gains, calls into question the necessity and practical value of the proposed mechanism. Overall, while the underlying idea is promising, the empirical results fail to demonstrate performance benefits commensurate with the increased computational complexity.

Questions for authors

Q1. According to Table 5, trajectory refinement and candidate consolidation yield only marginal improvements across all evaluation metrics, while Table 10 reports a 50.8% increase in processing time compared with the Structured Base. Could the authors provide a module-wise breakdown of the computational overhead and demonstrate whether these designed modules offer a meaningful advantage over simpler alternatives under comparable computational budgets?

Q2. Section 3.3 identifies complementary intentions primarily through distances in joint-endpoint space. However, distinct interaction outcomes may share similar endpoints while differing in yielding behavior or crossing order. How does JFER recover such interaction modes when their endpoint distances are small? Could the authors provide targeted evaluations of interaction-order diversity beyond endpoint-based coverage metrics?

Q3. Table 6 shows that Oracle Soft-mAP improves only from 0.4012 to 0.4054 (0.0042) between the Structured Base and Final Set, whereas the Soft-mAP increases from 0.2727 to 0.2868 (0.0141). Could the authors disentangle the contributions of improved trajectory coverage, geometric refinement, and confidence ranking to the final performance gains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.