acceptodds
Under review as a conference paper at ICLR 2027

SKIP: Efficient Trajectory Prediction via Sparse Keyframe Interaction Propagation

Abstract

Trajectory predictors often recompute agent and map interactions at every historical frame, even though nearby frames can share similar spatial context. We propose Sparse Keyframe Interaction Propagation (SKIP), which reduces these updates while retaining the full motion history. SKIP executes the native spatial block at selected keyframes and forms an interaction residual by subtracting the block input from its output. At each skipped frame, it adds the residual of the nearest keyframe to the same agent’s own feature. The receiving frame keeps its motion state; only the interaction increment is reused. SKIP introduces no extra parameters and leaves temporal processing, the decoder, and the training objective unchanged. We analyze its approximation error and computational cost, and evaluate it on QCNet and HiVT. On Argoverse 2, reducing QCNet’s spatial updates from 50 to 17 per layer increases forward throughput by 24.5% and reduces peak GPU memory by 43.0% at batch size 4, while validation minFDE changes from 1.5103 to 1.5045 m. On Argoverse 1, reducing HiVT’s local interaction updates from 20 to 7 increases forward throughput by 21.8% at batch size 32, with minFDE changing from 1.0304 to 1.0264 m. On these two backbones, spatial updates can be executed less frequently without discarding motion history or increasing validation minFDE.

Primary area:
applications to robotics, autonomy, planning
Keywords:
trajectory prediction, efficiency, sparse interaction, keyframe

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3958OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a minor modification of QCNet (Query-centric trajectory prediction, 2023) and HiVT (HiVT: Hierarchical vector transformer for multi-agent motion prediction, 2022) for trajectory prediction. It subsamples and aggregates the previous frames using a fixed stride. As a results, the approach is, as expected, faster than QCNet and HiVT. The approach is evaluated on Argoverse 1 for HiVT and Argoverse 2 for QCNet.

Critical strengths

Although it is not a critical strength, it needs to be mentioned that the approach is evaluated on Argoverse 1 for HiVT and Argoverse 2 for QCNet. The Appendix contains several ablation studies like impact of stride.

Critical weaknesses

Lack of novelty:

The paper presents a minor modification of QCNet (Query-centric trajectory prediction, 2023) and HiVT (HiVT: Hierarchical vector transformer for multi-agent motion prediction, 2022) for trajectory prediction. Subsampling observational data is an implementation detail and a common approach to reduce inference time. The approach also does it differently for QCNet and HiVT, but the difference is only regarding the interactions that are considered. Overall, the paper does not contain a major contribution..

Insufficient experimental evaluation:

The approach is compared only to its baselines from 2022 and 2023. The paper does not include any comparison to state-of-the-art methods. For each baseline, even a different dataset has been used. The results do not demonstrate the generalization across datasets and methods.

Poor presentation:

The paper is poorly structured and written. The introduction summarizes the results instead of clearly discussing the contribution. Figure 1(b) is not understandable since normalized change variance is not explained. The term is also only used in the figure and caption but not in the text. Figure 2 does not add anything to the paper. The approach is very simple but not well described. Section 3 starts with unimportant details. The rest provides a description what has been done, but without providing any intuition or motivation. The theoretical discussion does not add anything to the paper. The connection of Proposition 2 to the results are not well described.

Questions for authors

The AI use statement declares that generative AI tools assisted with language polishing. The poorly structured paper, emphasis on unimportant details, and meaningless figures, however, indicate that the paper and figures have have been generated in large part by AI tools.

Provide alpha, beta, lambda (Proposition 2) for the networks HiVT and QCNet.

Reviewer_A32LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SKIP, a method for reducing redundant spatial interaction computation in trajectory prediction. Instead of running spatial interaction modules at every historical frame, SKIP computes interaction residuals at selected keyframes and propagates them to nearby frames while preserving each frame's motion features. The method introduces no additional parameters and is evaluated on QCNet with Argoverse 2 and HiVT with Argoverse 1. The reported results show throughput improvements of 24.5% and 21.8%, respectively, with comparable minFDE. QCNet also achieves a 43.0% reduction in peak GPU memory.

Critical strengths
  1. Practical efficiency gains with a simple design. SKIP reduces spatial computation without introducing additional parameters or redesigning the original forecasting architecture. The improvements in throughput and memory usage are meaningful, especially given the limited changes in prediction accuracy. Results on both QCNet and HiVT suggest that the approach is not restricted to a single backbone.
  2. Clear analysis of computational savings. The paper goes beyond reporting inference speed. It includes runtime profiling, memory analysis, and ablations on propagation and execution stride. In particular, the comparison with distance pruning helps explain why skipping complete spatial updates can be more effective than simply reducing the number of graph edges.
Critical weaknesses
  1. The paper demonstrates efficiency gains over dense computation, but does not sufficiently establish the advantage of SKIP over other temporal reuse strategies. The main contribution of SKIP is to compute spatial interaction residuals at selected keyframes and propagate them to nearby frames. The experiments show that this approach is faster than the original dense computation and distance pruning. However, these comparisons mainly demonstrate the benefit of reducing spatial interaction updates, rather than the advantage of the proposed residual propagation mechanism itself. Previous work has explored temporal feature reuse, selective computation, and residual propagation. The paper does not provide direct comparisons with simpler alternatives, such as reusing complete spatial features or interpolating features between keyframes. As a result, it remains unclear whether the proposed residual formulation offers a meaningful advantage over existing or simpler reuse strategies under comparable computational budgets.
  2. The ablation studies do not fully explain when residual propagation is necessary or when it may fail. The paper includes useful ablations on execution stride and removing residual propagation. However, the results suggest that the contribution of residual propagation varies across backbones. On QCNet, removing propagation and retraining increases minFDE from 1.5045 to 1.5375, while on HiVT the corresponding difference is much smaller (1.0264 versus 1.0307). This raises the question of whether the performance gains primarily come from skipping spatial computation or from the specific residual propagation design.
  3. Moreover, SKIP uses a fixed keyframe interval, although interactions between agents can change rapidly in complex traffic scenarios. The experiments do not separately evaluate prediction performance under different interaction dynamics, such as lane changes, intersections, or dense traffic. Without such analysis, it is difficult to assess when reusing interaction residuals is reliable and how broadly the reported accuracy–efficiency tradeoff generalizes.
Questions for authors
  1. The paper compares SKIP with dense computation, distance pruning, and removing residual propagation. However, it remains unclear whether the proposed residual propagation is more effective than simpler alternatives, such as directly reusing spatial features or interpolating residuals between keyframes. Could the authors clarify the advantage of SKIP over these alternatives under the same computational budget?
  2. The dense QCNet baseline achieves a validation minFDE of 1.5103, while the released QCNet checkpoint reports 1.25 on Argoverse 2. The paper uses 32 training epochs, whereas the original implementation defaults to 64, and the effective training batch sizes may also differ. Could the authors explain whether these configuration differences account for the performance gap, and whether SKIP is expected to retain its efficiency gains with the stronger baseline configuration?
  3. Appendix A.6 analyzes interaction residual differences between stable and changing neighbor sets. However, it is unclear how these differences affect the final prediction accuracy. Could the authors report or clarify whether SKIP produces larger prediction errors in scenarios with rapidly changing interactions, particularly when using a fixed keyframe interval?
Reviewer_U4CsOpenReview ↗
Rating
1(−1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Sparse Keyframe Interaction Propagation (SKIP), an efficient trajectory prediction framework designed to reduce redundant spatial interaction computations across historical frames. Based on the observation that neighboring frames often share similar interaction contexts, SKIP performs spatial interaction modeling only at uniformly sampled keyframes and propagates the resulting interaction residuals to neighboring non-keyframes while preserving their original motion features. The method introduces no additional learnable parameters and retains the backbone's temporal processing, decoder, and training objective. The authors further provide theoretical analyses of keyframe coverage, approximation errors, and computational efficiency. Experiments on QCNet and HiVT demonstrate throughput improvements of 24.5% and 21.8%, respectively, without degrading validation minFDE, while reducing peak GPU memory by 43.0% on QCNet.

Critical strengths
  1. Computational Efficiency Improvements SKIP reduces redundant spatial interaction computations, achieving throughput improvements of 24.5% on QCNet and 21.8% on HiVT without degrading validation minFDE.

  2. Simple Design and Model Compatibility SKIP introduces no additional learnable parameters and requires no modifications to the original temporal processing modules, decoders, or training objectives. Its applicability is demonstrated on two trajectory prediction backbones.

Critical weaknesses
  1. Limited applicability and insufficient robustness to abrupt interaction changes: SKIP is based on the assumption that interaction residuals exhibit local temporal continuity, allowing non-keyframes to reuse interaction information from nearby keyframes. However, this assumption may not hold in safe-critical scenarios. When vehicles brake abruptly or pedestrians suddenly cross the road, interaction features may vary significantly even over short time intervals. Fixed-interval keyframe selection may miss such critical events, while residual reuse may fail to capture these changes, leading to substantial approximation errors in interaction representations. Although Proposition 2 provides a prediction error bound under Lipschitz assumptions, its effectiveness depends on the magnitude of interaction residual variations and network sensitivity, making it difficult to guarantee a meaningful error bound under abrupt interaction changes. Therefore, the method lacks an adaptive mechanism to respond to abrupt interaction changes and corresponding theoretical robustness guarantees, limiting its applicability to long-tail safe-critical scenarios.

  2. Limited methodological novelty in keyframe selection: SKIP essentially reduces spatial interaction computations through fixed-interval keyframe sampling and interaction residual reuse, making it more of an engineering-oriented acceleration strategy. However, the method does not adequately investigate the underlying nature of redundant interaction computations, particularly which historical interaction information contributes meaningfully to future prediction and which computations can be omitted without losing critical predictive information. The so-called keyframes are selected solely based on predefined temporal intervals rather than interaction importance or predictive relevance. This effectively assumes that temporal proximity implies the reusability of interaction information, although the two are not necessarily equivalent. The fundamental objective of keyframe selection should be to identify and eliminate redundant interaction computations while preserving critical interaction changes, rather than simply covering the entire historical sequence with fewer keyframes. Therefore, the method lacks substantive modeling of interaction redundancy and information importance, thereby limiting its methodological novelty.

Questions for authors
  1. Why does the proposed keyframe selection rely solely on fixed temporal intervals rather than interaction importance or predictive relevance? Can the authors demonstrate the advantages of this strategy over adaptive keyframe selection under the same computational budget?

  2. How does SKIP perform in safety-critical scenarios involving abrupt vehicle braking or sudden pedestrian crossings? Can the authors provide experimental evaluations to validate its robustness to abrupt interaction changes?

  3. Can the authors demonstrate that the theoretical error bound in Proposition 2 remains practically meaningful under substantial variations in interaction residuals, particularly in complex dynamic scenarios?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Core Mechanism, Strong Baselines, and Multi-Benchmark Evaluations

We thank all reviewers for their constructive feedback. Below, we clarify our core motivation, present new empirical results from the revised manuscript, and address concerns regarding strong baselines, reuse strategies, multi-backbone generality, and domain robustness.


1. Core Motivation and Mechanism

  • Dense Motion vs. Sparse Interaction: SKIP is distinct from temporal subsampling. Subsampling observations discards frames, severely hurting prediction (dropping 1/2 of frames degrades minFDE by +6.2%, 2/3 by +9.9%). While kinematics change rapidly, spatial interaction fields evolve much more slowly (<1% variance within 0.1 s). SKIP keeps kinematic states 100% dense across all frames, updating spatial interaction at only 17 keyframes. This retains full sensitivity with minimal accuracy delta (-0.0058 m on QCNet, -0.0040 m on HiVT, +0.0047 m / +0.37% on converged QCNet).
  • Skipping Complete Updates vs. Edge Pruning: Distance pruning removes 72% of edges with only +4.3% throughput and zero memory reduction on GPUs, as graph building and projections still run every frame. Skipping complete spatial blocks eliminates redundant stages, achieving +33.1% throughput and 43.6% memory reduction on converged models.

2. Fine-Tuning on Official Converged Checkpoint (Argoverse 2 Full 24,988 Scenes, RTX 4090 D)

See Appendix A.13 (Table 11, Page 23) in the revised manuscript:

MethodScheduleminADE (↓)minFDE (↓)MR (↓)Batch 4 (sc/s) (↑)GainPeak Mem (MiB) (↓)
Dense QCNet (Official FT)50 (Dense)0.63341.252515.86%84.59 ± 0.14—1,664.0
SKIP (Residual Prop., Ours)17 (Fixed)0.63631.257215.89%112.63 ± 0.37+33.1%938.3 (-43.6%)
Full Feature Copy17 (Fixed)0.64271.261916.03%112.53 ± 0.09+33.0%938.3
Linear Interpolation17 (Fixed)0.63631.259616.24%111.79 ± 0.14+32.2%938.3
Adaptive Residual17 (Adaptive)0.63581.260716.13%44.53 ± 0.21-47.4%441.2
  • Strong Baseline: On official SOTA level, SKIP reaches 1.2572 m minFDE (+0.37% delta) with +33.1% throughput and 43.6% memory savings.
  • Reuse Comparison: Full feature copy erases intermediate kinematics (1.2619 m); interpolation introduces non-causal latency (1.2596 m). Residual propagation (zt+Δkz_t + \Delta_k) achieves best accuracy.
  • Overhead of Adaptive Selection: Dynamic thresholding breaks GPU parallelism. Throughput collapses by 60.5% (44.53 vs. 112.63 sc/s) with no accuracy gain (1.2607 vs. 1.2572 m). Fixed interval is optimal for deployment.

3. Generalization Across Backbones and Datasets

  • HPNet (2024 SOTA Backbone, Appendix A.15, Table 13): Dense HPNet achieves 0.9873 m minFDE; SKIP reaches 1.0018 m (gap 0.0145 m / +1.46%) with matched MR (8.49% vs. 8.50%). Agent-Attention latency drops from 7.77 to 4.19 ms (1.86× speedup), raising throughput to 90.63 sc/s (+8.25%). Disabling propagation degrades minFDE to 1.0088 m (MR 8.74%).
  • NeurIPS Shifts Benchmark (Weather & Domain Shift, Appendix A.16, Table 14):
    • QCNet on Shifts: Matches minADE/minFDE (0.7712/1.5584 vs. 0.7691/1.5453 m) with better MR (20.26% vs. 20.90%). Spatial attention is 2.09× faster (1.88 vs. 3.92 ms), throughput rises +13.2% (132.58 vs. 117.11 sc/s), and memory falls -47.3% (561.5 vs. 1,065.3 MiB).
    • HPNet on Shifts: Achieves 1.7708 vs. 1.7326 m minFDE with better MR (21.52% vs. 22.01%), 2.13× spatial attention speedup (7.14 vs. 15.22 ms), and +13.3% throughput (64.59 vs. 57.02 sc/s).
  • Across three backbones and three independent benchmarks, SKIP proves robust and architecture-agnostic.

4. Safety-Critical Scenarios (Official Checkpoint, Appendix A.14, Table 12)

  • Abrupt Braking (1,330 scenes): SKIP minFDE improves to 1.5092 m vs. Dense 1.5235 m (MR: 19.55% vs. 20.15%). In last-1s severe braking (186 scenes), SKIP reaches 1.6769 m vs. Dense 1.7326 m (-0.0557 m; MR: 23.12% vs. 24.73%). Anchoring at t=T−1t=T-1 while keeping raw kinematics ztz_t avoids temporal over-smoothing.
  • Pedestrian Crossing (75 scenes): SKIP minFDE is 1.7146 m vs. Dense 1.6869 m (gap < 2.8 cm; MR: 26.67% vs. 28.00%).
  • High Churn (160 scenes, rate ≥ 0.1): SKIP minFDE is 1.2051 m vs. Dense 1.1952 m (gap < 1 cm; MR: 13.75% vs. 16.25%).

5. Computational Scope and Camera-Ready Plan

Due to tight rebuttal time and compute constraints, we prioritized representative multi-backbone and OOD evaluations. We will provide full-split extended curves in the camera-ready version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.