acceptodds
Under review as a conference paper at ICLR 2027

Interaction-Aware Neighbor Representation for Vehicle Trajectory Prediction

Abstract

Accurate trajectory prediction in autonomous driving requires not only expressive model architectures but also informative representations of surrounding vehicles. However, raw kinematic features do not explicitly encode the structure or relative importance of inter-vehicle interactions. We propose a model-agnostic neighbor representation framework that abstracts raw kinematics into discrete longitudinal and lateral interaction states and integrates them into a unified importance score. Slot-semantic weighting and importance-based filtering further refine the representation according to driving context. Evaluated on real-world highway driving datasets across diverse single- and multi-agent prediction models, the proposed representation improves prediction accuracy across most evaluated backbones and metrics. It achieves up to a 28.47% reduction in ADE with negligible forward-pass runtime overhead on an embedded platform.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Trajectory Prediction, Interaction-Aware Representation, Neighbor Influence Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jaqdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an interaction-aware neighbor representation for vehicle trajectory prediction. The method converts neighboring vehicles’ raw kinematic information into longitudinal and lateral interaction states, computes neighbor importance using a fixed exponential function, and refines these scores through driving-context-dependent slot weighting and top-n filtering. The resulting scores are incorporated as additional input features into several single-agent and multi-agent prediction models. The authors evaluate the method on highD and exiD through prediction benchmarks, ablation studies, and inference latency measurements on an embedded platform.

Critical strengths

The paper addresses a practically relevant problem. The representation of neighboring agents can affect trajectory prediction, and explicitly incorporating lane relationships and interaction priors has potential practical value.

The method is relatively simple and modular. The proposed importance features can be integrated into different prediction architectures through input adapters, and the authors evaluate multiple backbones.

The experiments include component analysis and efficiency evaluation. Beyond prediction accuracy, the paper reports ablations, results averaged over multiple random seeds, and forward-pass latency on an NVIDIA Jetson Orin Nano. Some model–dataset combinations show noticeable improvements.

Critical weaknesses
  1. The methodological novelty is limited, and comparisons with the closest prior work are insufficient.

The method primarily combines TTC-like measures, discrete interaction states, lane-slot semantics, and neighbor importance filtering. These ideas have been extensively studied in vehicle interaction modeling and trajectory prediction. Incorporating them as additional inputs into different backbones does not, by itself, establish substantial methodological novelty.

In particular, ASPILin (IROS 2025), which the authors already cite, also combines interacting-agent selection with physical correlation coefficients and evaluates its approach on datasets including highD. However, the paper provides neither a direct experimental comparison nor a sufficiently detailed analysis of the technical differences. Furthermore, SmartPretrain (ICLR 2025), also cited by the authors, explicitly proposes a model-agnostic representation learning framework. The characterization of existing representation methods as dependent on specific backbones therefore requires greater precision.

  1. The baselines are substantially outdated and do not adequately reflect the current state of the field.

CS-LSTM and SGAN were published in 2018, GRIP++ is cited as a 2020 method, and QCNet and MTP-GO were published in 2023. For an ICLR 2027 submission, even the most recent established baselines are four years old. The custom NeighFormer backbone cannot substitute for recognized recent strong methods.

These experiments demonstrate some architectural compatibility, but they do not establish whether the proposed handcrafted representation still provides consistent additional value when modern prediction models already have stronger interaction modeling capabilities. The authors should include recent strong models publicly available before submission, such as ModeSeq (CVPR 2025) and NDPNet (CVPR 2026), and compare their original versions against versions augmented with the proposed representation.

  1. The dataset coverage is insufficient to support the generalization claims.

The main experiments are restricted to highD and exiD, both highway datasets that closely match the method’s assumptions about relative lane relationships, surrounding slots, and lane-change contexts. The paper does not adequately explain why it omits evaluation on Argoverse, Argoverse 2, or the Waymo Open Motion Dataset.

AV2 and WOMD contain richer urban traffic and interaction scenarios, making them suitable for assessing applicability to complex road topologies and different types of traffic participants. WOMD also provides annotations and evaluation protocols specifically for interaction prediction. A single intersection example from inD in Figure 5 cannot replace comprehensive quantitative evaluation. The authors should evaluate on established urban driving benchmarks or explicitly restrict the scope of the method and its conclusions to highway vehicle prediction.

  1. The acquisition of driving context is unclear, raising a potential risk of future information leakage.

Appendix A.4 defines driving contexts using lane-following, lane-change direction, and pre-LC/post-LC categories, and states that contexts are assigned using the ego maneuver label and lane information. However, the paper does not explain how these labels are obtained using only observed history at test time, particularly how the future lane-change direction is determined before the maneuver occurs.

If test-time contexts are assigned from complete ground-truth trajectories, the augmented models may receive advance information about future maneuvers, compromising the validity of the comparisons. The current description is insufficient to conclude that leakage actually occurs, but this issue must be clarified. The authors should provide a context estimation procedure that uses only information available at prediction time, separately report results with ground-truth and predicted contexts, and ensure that the baselines receive the same legitimately available context information.

  1. Critical control experiments are missing, preventing clear attribution of the performance gains.

The full method introduces not only discrete interaction states but also lane semantics, maneuver context, and slot weights learned through an auxiliary trajectory prediction objective. Consequently, comparisons against a raw-kinematics baseline cannot determine whether the improvements arise from the proposed representation or from additional context and auxiliary supervision.

Under matched information and training budgets, the authors should compare against directly adding lane/slot features, continuous TTC or time-headway features, learned importance functions, and alternative neighbor ranking strategies. The finding that directly concatenating the two interaction states degrades performance is also insufficient to establish that the fixed exponential fusion is superior to a reasonably designed learned fusion mechanism.

  1. The results lack statistical uncertainty estimates, and the claims of overall effectiveness are stronger than the evidence supports.

Although the main experiments report averages over five runs, they provide neither standard deviations nor confidence intervals. Improvements are below 1% in several settings. For example, CS-LSTM improves ADE and FDE on exiD by only 0.11% and 0.04%, respectively, leaving their reliability unclear. MTP-GO instead shows worse minADE and minFDE on exiD.

The largest reported ADE reduction of 28.47% comes from a relatively weak GRIP++ setting, whose improved performance remains substantially worse than some other unmodified baselines in the same table. This maximum relative gain therefore does not adequately represent the method’s overall value for strong models. The authors should also report separate results for lane changes, merging, and dense interactions to verify that the gains actually arise from improved interaction modeling.

  1. The mathematical definitions are inconsistent with the claimed interaction semantics.

The authors state that longitudinal discretization preserves the sign distinguishing approaching from receding interactions. However, the zero-state interval in Table 5 is ([-3.1883,,2.2974)), which contains both negative and positive values and therefore cannot distinguish approaching from receding within this interval. Equation (3) subsequently uses (|\sigma_x|), assigning identical importance scores to corresponding positive and negative longitudinal states when the lateral state is unchanged.

The original kinematic inputs still contain directional information, but the added importance feature itself does not fully preserve the directional semantics emphasized by the authors. In addition, lateral states are determined using a fixed position threshold, without sufficient evaluation of early lane-change recognition, threshold sensitivity, or robustness to observation noise. These issues weaken the interpretability claims.

  1. The ablation conclusions and reproducibility details are insufficient.

In Table 3, using the importance score alone yields an ADE of 0.2394. Adding Slot or NF individually worsens ADE to 0.2496 and 0.2766, respectively, while adding both improves it to 0.2138. This suggests a possible interaction between the components, but does not establish that each component provides a stable contribution. Ablations across additional backbones, parameter sensitivity analysis, and statistical uncertainty estimates are needed.

Table 3 also does not explicitly identify the dataset, and its NeighFormer baseline differs from the results for both datasets in Table 1. The differences in experimental settings should be explained. The descriptions of data splits, trajectory window sampling, the custom NeighFormer architecture, and auxiliary network training are insufficient. Whether overlapping trajectory windows occur across splits remains unclear, and these omissions limit reproducibility. The authors should also explain how scene-level metrics are computed from consistent joint prediction samples and whether latency measurements include context acquisition and feature computation.

Questions for authors

see weaknesses.

1 comment

Thank you for your detailed review. We address the main concerns below.

(W1) Our contribution lies in integrating longitudinal and lateral interaction information into an explicit neighbor-importance representation that can be added to different prediction backbones. The original kinematic features are retained, and the internal backbone architectures remain unchanged. Our objective is to evaluate the usefulness of this input-level representation across different prediction models.

(W2, W3) The experiments are intended to evaluate the representation across different backbone architectures rather than compare state-of-the-art prediction performance. We therefore include six backbones covering recurrent, generative, graph-based, transformer-based, and multi-agent prediction approaches. Our primary evaluation focuses on lane-structured driving environments. In addition to conventional highway scenarios, exiD includes merging, on-ramp, and off-ramp interactions.

(W4) During auxiliary slot-weight training, pre-/post-LC samples are classified using recorded lane-change events and directions. Context-specific slot weights are learned and fixed using their validation-set means. During trajectory prediction, the context is assigned using lane-ID transitions within the observed history. Pre-LC and post-LC weights are applied before and after an observed transition, respectively. If no transition is observed, lane-position-dependent weights are used. This inference-time procedure does not require future ground-truth maneuver labels or trajectories. We will clarify the distinction in Section 3.3 and Appendix A.4.

(W5, W8) The ablation study evaluates the proposed components under the same training configuration. Directly combining the longitudinal and lateral states performs worse than using the unified importance score. Although Slot weighting and neighbor filtering individually increase prediction errors, their combination achieves the lowest mean errors across all three metrics.

To further assess the consistency of these results, we have added mean and standard deviation values across three random seeds in Appendix C. The full configuration achieves an ADE of 0.2138 ± 0.0216, compared to 0.2394 ± 0.0231 for importance alone. While these results suggest complementary effects between Slot weighting and neighbor filtering, they do not establish the optimality of the individual design choices or statistical significance of the improvements.

(W6) The improvement varies across models and datasets. Some gains are small, and MTP-GO shows slightly worse minADE and minFDE on exiD. The reported 28.47% ADE reduction refers specifically to GRIP++ on exiD and is not intended to represent the improvement across all models.

(W7) The importance score is designed to represent neighbor interaction relevance rather than collision risk. Although Eq. (3) assigns equal importance to positive and negative longitudinal states with the same magnitude, the original kinematic features retain their directional information. We also acknowledge that the central quantization interval contains both positive and negative interaction times, so the sign is not strictly preserved in every interval. We will correct this description.

(W8) Table 3 reports the ablation study using Neighformer on exiD with 100 training epochs, whereas the main experiments in Table 1 use 200 epochs. All ablation configurations were trained under the same 100-epoch budget due to computational constraints. Therefore, Table 3 is intended for within-table component comparisons rather than direct comparison with Table 1. We will specify these settings in the revised manuscript.

Reviewer_xZYjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an interaction-aware neighbor representation framework for vehicle trajectory prediction. The method abstracts neighboring vehicles' raw kinematic features into discrete longitudinal and lateral interaction states, which are combined into a unified importance score. Slot-semantic weighting and importance-based filtering further refine the representation according to the ego vehicle's driving context. The resulting features are integrated into six trajectory prediction backbones and evaluated on the highD and exiD datasets. The authors report improvements across several architectures, including a maximum ADE reduction of 28.47%, with limited forward-pass runtime overhead.

The problem is practically relevant, and exploring backbone-independent interaction representations is worthwhile. However, I have substantial concerns regarding the methodological novelty, the availability of driving-context information at inference time, and whether the experiments adequately support the claimed generality and effectiveness.

Critical strengths
  1. The paper addresses an important aspect of trajectory prediction: how to represent the influence of surrounding vehicles beyond raw kinematic observations.

  2. The separation of longitudinal and lateral interactions provides an intuitive interpretation of neighboring vehicles' relevance. The representation can be incorporated into existing prediction models without substantially modifying their architectures.

  3. The experiments cover six backbones spanning single-agent and multi-agent prediction, including graph-based and transformer-based approaches.

Critical weaknesses
  1. The core framework largely combines established concepts, including TTC-like longitudinal indicators, lane-relative motion states, heuristic importance decay, context-dependent weighting, and top-k selection. Highly similar to KnowMTP published in CVPR 2026.

Bai, Rufan, et al. "KnowMTP: A Knowledge-Guided Framework for Multi-Agent Trajectory Prediction in Autonomous Driving." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

  1. Potential future-information leakage in driving-context assignment. The slot-semantic weighting mechanism relies on ego driving contexts, including pre-/post-lane-change states and target-lane information. Appendix A.4 indicates that these contexts are assigned using ego maneuver labels and lane information. However, the paper does not clarify how the maneuver labels, particularly the intended lane-change direction before the maneuver occurs, are obtained at inference time. If they are derived from future trajectories or complete maneuver annotations, the framework may implicitly access information unavailable to a real-world prediction system.

  2. The proposed importance formulation does not convincingly capture interaction relevance. Equation (3) uses the absolute longitudinal state, treating approaching and receding interactions with equal state magnitudes identically, despite their substantially different implications for collision risk. Furthermore, the empirical discretization in Table 5 places both negative and positive interaction times within the central state, potentially conflating qualitatively different behaviors. These choices appear inconsistent with the paper's motivation of distinguishing hazardous interactions through driver-oriented semantics. In addition, the neighbor filtering mechanism only suppresses importance scores while retaining all neighboring agents' kinematic features.

  3. Insufficient evidence for generalization beyond the evaluated highway settings. Both primary datasets, highD and exiD, focus on highway environments, where lane structures and vehicle interactions are relatively constrained. The representation itself relies heavily on predefined lane-relative slots and fixed interaction thresholds. Its applicability to complex intersections, irregular road geometries, and dense multi-agent interactions is therefore uncertain. The single qualitative example from inD is insufficient to establish cross-scenario generalization. Furthermore, the runtime experiments measure forward-pass latency rather than the full online pipeline, leaving the computational cost of context estimation and interaction preprocessing unexamined.

Questions for authors
  1. What specific benefit comes from the proposed discretization and exponential importance function? How does this work differ from KnowMTP?

  2. How are the ego maneuver labels, target-lane information, and pre-lane-change contexts obtained during inference? Are they determined exclusively from the observed two-second history? If an online context classifier is required, how does its prediction uncertainty affect the reported results?

  3. Which dataset and experimental configuration are used in Table 3, and why do its Neighformer baseline results differ from Table 1?

  4. Can the authors provide quantitative results on more diverse scenarios, particularly intersections and complex merging situations, and analyze performance separately for lane-following, lane-changing, and highly interactive cases?

1 comment

Thank you for your review and comments. We address your concerns below.

(Q1) Our main contribution is not the individual use of TTC-like indicators or neighbor selection, but their integration into an explicit interaction-aware neighbor representation. The proposed importance score is added to the original kinematic features without modifying the internal structure of the prediction backbones. This allows us to evaluate the effect of the same input-level representation across different architectures.

(Q2) During auxiliary slot-weight training, pre-/post-LC samples are classified using recorded lane-change events and directions. The learned weights are then fixed using their validation-set means. During trajectory prediction, weight assignment relies on lane-ID transitions within the observed history. Frames before and after an observed transition receive pre-LC and post-LC weights, respectively. When no transition is observed, lane-position-dependent weights are used. No future ground-truth trajectories or maneuver labels are required for this inference-time assignment. We will clarify this procedure in Section 3.3 and Appendix A.4.

(Q3) The importance score in Eq. (3) represents neighbor interaction relevance rather than a direct estimate of collision risk. The absolute longitudinal state assigns equal importance to positive and negative states of the same magnitude, while their directional information remains available in the original kinematic features. We also acknowledge that the central quantization interval contains both positive and negative interaction times, so their signs are not strictly preserved in every interval. We will correct this description. Neighbor filtering suppresses low-ranked importance scores without removing the original neighbor kinematics.

(Q4) Our evaluation focuses on lane-structured driving environments. While highD covers conventional highway traffic, exiD also includes merging, on-ramp, and off-ramp scenarios. The inD example provides a qualitative result rather than quantitative evidence of generalization to urban intersections. As stated in Section 4.4, Table 4 reports prediction-backbone forward-pass latency, not the runtime of the complete online pipeline.

(Table 3) The baseline discrepancy arises from different training budgets. Table 1 uses 200 epochs, whereas Table 3 uses 100 epochs due to computational constraints. The ablation study was conducted using Neighformer on exiD, with all configurations trained under the same 100-epoch budget. We will clarify these settings in Table 3.

Reviewer_Z7yxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an interaction-aware neighbor representation for vehicle trajectory prediction. Longitudinal interaction time and lateral motion relations are converted into importance scores, which are adjusted using context-dependent slot weights and sparsified through top-n selection. The original neighbor kinematics are retained. Experiments on highD and exiD across six prediction backbones show improvements in many settings. The main contribution is an explicit interaction representation that can be incorporated into different prediction architectures.

Critical strengths
  1. Studying interaction representation separately from the prediction backbone is a useful direction. The method provides an explicit interaction prior while retaining the original kinematic information, making it relatively straightforward to incorporate into different architectures.
  2. The experimental evaluation covers six backbones and two datasets, with multiple random seeds, component ablations, and embedded-platform timing. Several improvements are substantial, particularly for GRIP++ on exiD. These results provide useful evidence that explicit interaction features can benefit predictors with different architectures.
Critical weaknesses
  1. The source of the test-time maneuver context is unclear. Section 3.3 and Appendix A.4 use maneuver labels, lane-change direction, and pre-/post-LC context, but do not explain how these are obtained from observation history alone. Identifying a pre-LC sample is particularly unclear without knowing the subsequent maneuver. This does not establish information leakage, but it leaves the forecasting inputs insufficiently defined. It is also unclear whether the baselines receive equivalent lane and maneuver information. Consequently, the reported gains cannot yet be attributed specifically to the proposed representation rather than additional semantic inputs.
  2. The experiments establish the usefulness of the complete pipeline more clearly than the benefits of its particular design choices. Same-information controls using continuous interaction features or simple TTC-based scores are missing. The robustness motivation also needs evidence: distance and relative-velocity errors propagate through the interaction-time ratio, while bin boundaries and top-n selection can introduce discrete changes. Epsilon stabilization and quantization alone do not establish that the final representation or predictions are robust to noisy inputs. Recent comparisons are also limited. The established external backbones mainly date from 2018–2023. A compatible recent highway predictor, such as C2F-TP (AAAI 2025), would help assess whether the representation remains useful alongside more recent methods. ASPILin (IROS 2025), already cited in the manuscript, is directly relevant to the physical interaction design and merits a comparison or an equivalent controlled study under matched inputs and evaluation protocols. Finally, Table 3 does not identify its dataset or explain why its baseline ADE is 0.2848, compared with 0.2195/0.2329 for Neighformer in Table 1. Multiple seeds are reported, but variability is not provided. Slot and NF individually perform worse than I alone, whereas their combination improves performance. The reason for this interaction and its consistency across
Questions for authors
  1. How are maneuver labels, lane-change direction, and pre-/post-LC context obtained at inference time? Do the reported experiments use history-derived estimates or ground-truth maneuver labels, and do the baselines receive equivalent semantic information?
  2. Do the gains remain when the baseline receives the same semantic inputs and the proposed scores are compared with continuous interaction features or simple TTC-based scores? Can a controlled perturbation experiment, using identical noisy historical inputs and unchanged future ground truth, substantiate the claimed stabilization benefit?
  3. Which dataset and configuration are used in Table 3, and why does its baseline differ from Tab
1 comment

Thank you for your review and comments. We address your questions below.

(Q1) During slot-weight training, pre-/post-LC samples are classified using recorded lane-change events and directions. Context-specific weights are learned through an auxiliary prediction network and fixed using their validation-set means. During trajectory prediction, contexts are determined using lane-ID transitions within the observed history. If a lane change is observed, pre-LC weights are assigned to frames before the transition and post-LC weights to frames after it. Otherwise, lane-position-dependent weights are used. This procedure does not require future ground-truth trajectories or maneuver labels. We will clarify these procedures in Section 3.3 and Appendix A.4.

(Q2) Our experiments aim to evaluate the effect of the proposed interaction representation across different prediction backbones, rather than establish the optimality of individual design choices. We agree that comparisons with continuous interaction features, TTC-based scores, and recent predictors would help isolate their effects. Discretization also does not guarantee robustness to observation noise, particularly near bin boundaries and top-n selection thresholds.

(Q3) The baseline discrepancy between Tables 1 and 3 results from different training budgets. Table 1 uses 200 epochs, while Table 3 reports an ablation study using the Neighformer backbone on exiD with 100 epochs due to computational constraints. All configurations in Table 3 were trained under the same 100-epoch budget. We will clarify these experimental settings in the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.