acceptodds
Under review as a conference paper at ICLR 2027

TCMA: Training-Free Causal Online Memory Augmentation for Black-Box Pedestrian Trajectory Prediction

Abstract

A deployed trajectory predictor accumulates information unavailable during training, namely the realized futures of its earlier predictions in the same environment. Existing approaches for exploiting such delayed feedback typically update the base model or train auxiliary memory modules, which may require internal access, gradients, or additional training and can be difficult in black-box or resource-constrained deployment. We therefore study whether completed trajectories can improve the quality of a frozen predictor's candidate set under a fixed output budget. Our key insight is that only trajectories whose futures have fully elapsed are causally admissible as memory. We introduce Training-free Causal Memory Augmentation (TCMA), an output-level adapter that retrieves such trajectories and replaces candidate slots with low historical win rates. To capture cross-scene motion patterns and within-recording recurrence, TCMA combines a scene-invariant egocentric channel with a scene-aware world-frame channel, both tuned online from elapsed labels. We analyze candidate replacement by decomposing its effect into retrieval benefit and displacement harm, and provide finite-sample risk bounds under explicit sampling and selection assumptions. On ETH-UCY, our prospective evaluation across seven predictors reports mean relative reductions of 6.2% in minADE@20 and 8.5% in minFDE@20. These results support output-level adaptation as a promising direction for reusing deployment experience, turning delayed outcomes into causally safe inference-time memory for deployed black-box predictors.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Pedestrian Trajectory Prediction, Causal Memory Augmentation, Training-free Adaptation, Black-Box Prediction, Online Memory Augmentation, Trajectory Retrieval

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2sUSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TCMA, a training-free output-level adapter for frozen K-candidate pedestrian trajectory predictors. TCMA stores trajectories only after their full future horizon has elapsed, retrieves causally eligible candidates through an ego-centric channel and a world-frame channel, and replaces candidate slots with low historical winning frequencies under a fixed output budget. It further introduces a causal self-tuning procedure that selects retrieval and gating configurations from elapsed labels split into fitting and checking sets.

Critical strengths

1.Practical black-box deployment setting. Improving a frozen K-candidate predictor without retraining, internal access, or increasing the number of guesses is a meaningful and timely problem. The causal eligibility rule is simple and appropriate for deployment time memory.

2.Broad predictor evaluation with low measured overhead. The paper evaluates seven heterogeneous predictors on ETH-UCY and reports sub-millisecond retrieval/injection latency.

Critical weaknesses

1.The empirical comparison is insufficient to support the claimed adapter-level contribution. The paper does not directly compare TCMA with existing online adaptation methods such as TrajICL, T4P, OAT-Mem, or AOL under a comparable black-box and fixed-K budget.

2.The theoretical safety guarantee is not operational at inference time and does not support the claimed non-degradation property. While the causal rule is sound, Lemma 1 conditions on the current query's unobserved winning slot W(P,x), which is unknowable at inference time. Additionally, the finite-sample bound is marginal, assumes IID sampling, and ignores slot-selection bias and temporal dependence. Thus, the theory offers no operational per-query non-degradation guarantee.

Questions for authors

1.Please compare TCMA directly against at least one existing online adaptation method, such as TrajICL, T4P, OAT-Mem, or AOL, under the same frozen-predictor and fixed-K setting. If a direct comparison is impossible, please provide a controlled surrogate comparison and explain the mismatch.

2.Lemma 1 conditions on W(P,x), the winning slot under the current query's unobserved future. Since only past queries' futures have elapsed, this condition cannot be checked at inference time. Can the authors provide an operational deployment time condition, or explicitly clarify that the guarantee is post-hoc and marginal rather than per-query? Please also clarify whether the finite-sample bound requires multiple-comparison correction over K slots and whether it holds under temporal dependence.

2 comments
AuthorsOpenReview ↗

Response 1

We sincerely thank Reviewer for the careful assessment of the experimental comparisons and theoretical analysis. We particularly appreciate the comments concerning the compatibility of existing online adaptation methods with TCMA's deployment setting and the statistical interpretation of the proposed guarantees.

W1/Q1. Comparability with Existing Online Adaptation Methods

We agree that comparisons under a shared deployment setting are valuable for understanding the contribution of TCMA. We therefore added two controlled experiments that examine complementary alternatives to our approach.

A faithful direct comparison with the original methods under TCMA's frozen black-box, no-training constraint is not possible because their adaptation mechanisms require different forms of model access or separately trained components.

Specifically:

  • T4P updates predictor representations and memory using delayed ground truth, requiring access to the predictor and test-time optimization.
  • OAT-Mem uses separately trained memory and decoding components.
  • AOL relies on a separately trained adaptation module.
  • TrajICL performs in-context adaptation using a separately trained predictor rather than operating as an output-level adapter for arbitrary frozen candidate sets.

OAT-Mem and AOL therefore require training auxiliary components outside TCMA's training-free constraint. We have clarified these differences in the revised Related Work and experimental discussion.

Instead of presenting unmatched implementations as direct comparisons, we added two controlled surrogates.

R2: Causal online fine-tuning surrogate

We compare TCMA against causal online fine-tuning of TUTR using the same completed labels, chronological fitting/checking procedure, evaluation windows, and K=20 candidate budget.

The fine-tuning surrogate additionally has access to TUTR's parameters and gradients, whereas TCMA operates using only the exported candidate predictions.

The fine-tuning configuration searches learning rates from 10⁻⁵ to 3 × 10⁻⁴ and epoch counts from {1, 3, 10}, using the elapsed fitting and checking sets. If no configuration is accepted, the original predictor is retained.

The results are:

MethodMean minADE@20 gainMean minFDE@20 gain
Causal online fine-tuning−1.75%−3.82%
TCMA+4.15%+6.17%

Fine-tuning improves TUTR on UNIV, achieving +6.75% minADE improvement compared with +4.13% for TCMA. However, it reduces performance on the other four scenes, resulting in negative overall averages.

This experiment provides a controlled comparison between parameter-level adaptation and black-box output-level augmentation using the same causally available supervision.

R3: Similarity-based and recency-based output-level proxies

We additionally compare TCMA with two output-level proxies inspired by example-based adaptation.

Both proxies use causally completed same-scene examples and replace the last q=2 candidate slots under the same frozen-predictor and fixed-K setting.

The two alternatives are:

  1. Similarity-based selection, which retrieves examples using its similarity criterion.
  2. Recency-based selection, which retrieves the most recently completed trajectories.

Unlike TCMA, these proxies do not use its joint position/heading/speed matching, self-tuned gates, or abstention mechanism.

The results are:

MethodMean minADE@20 gainMean minFDE@20 gain
TCMA6.52%9.10%
Similarity proxy3.19%4.57%
Recency proxy0.71%0.71%

The per-query degradation rates are 5.9%, 9.0%, and 9.3%, respectively. The corresponding numbers of negative predictor-scene outcomes are one for TCMA, 11 for the similarity proxy, and 17 for the recency proxy.

These controlled comparisons help isolate the contribution of TCMA's retrieval, gating, and replacement decisions under a common output-level evaluation setting.

We emphasize that these experiments are controlled surrogates rather than faithful reproductions of the original T4P, OAT-Mem, AOL, or TrajICL systems. We have made this distinction explicit in the manuscript.

Changes in manuscript:

  • Section 2, Related Work, page 3, Online Adaptation.
  • Section 4.3, page 9, Additional Controlled Experiments.
  • Appendix N.2, page 25, Experiment R2 and Table 13.
  • Appendix N.3, page 26, Experiment R3 and Table 14.
AuthorsOpenReview ↗

Response 2

W2/Q2. Operational Interpretation and Statistical Assumptions of the Theoretical Bounds

We appreciate this observation and agree that the original presentation required a clearer distinction between the theoretical condition and the operational activation procedure.

1. Interpretation of Lemma 1

Lemma 1 provides a conditional non-degradation result: replacing candidate slots cannot increase minADE@K if the original winning candidate remains in the candidate set.

The reviewer correctly points out that the current query's winning slot depends on its unobserved future. Consequently, this condition cannot be verified at inference time.

We have clarified this explicitly in the revised manuscript. Lemma 1 characterizes the condition under which replacement is harmless, rather than providing an operational per-query guarantee.

2. Operational activation conditions

Although TCMA cannot identify the current query's winning candidate before its future is observed, it controls replacement through observable deployment-time conditions:

  • A memory entry becomes eligible only after its complete future horizon has elapsed.
  • The query must satisfy the retrieval-density and admission requirements.
  • The selected retrieval and gating configuration must improve performance on both chronological fitting and checking subsets of elapsed labels.

The configuration is applied only to subsequent queries. If sufficient completed trajectories are unavailable or no configuration satisfies the acceptance conditions, TCMA retains the base prediction.

These conditions determine when TCMA is permitted to use retrieved candidates. They do not imply that every activated replacement is guaranteed to improve the current prediction.

3. Multiple-comparison correction

We have clarified the statistical corrections used in Proposition 1.

For the main last-q policy, q is selected from {1, 2, 3} using reference data. We therefore apply Bonferroni correction at α/3 to account for selecting among the three candidate sets under the independent-sample assumptions.

For data-dependent historical win-frequency selection, simultaneous bounds over the K candidate slots are required. The revised analysis explicitly includes this correction.

4. Temporal dependence

We agree that the independent-sample assumption is important. The Clopper–Pearson bound does not automatically establish coverage under arbitrary temporal dependence.

We have therefore separated the theoretical independent-sample result from the empirical temporal-dependence assessment.

In Experiment R5, the corrected last-q bound holds in 23 of 26 active predictor-scene cases. The mean upper bound is 0.102, compared with an observed displacement frequency of 0.077. For historical win-frequency selection, the Bonferroni-corrected bound holds in all 26 cases.

We additionally evaluate temporally contiguous block-bootstrap confidence intervals. The 95% gain intervals have positive lower endpoints in 24 of 26 cases using 8-second blocks and 23 of 26 cases using 16-second blocks.

Experiment R6 further shows that 5.9% of activated predictions experience individual degradation, with every degraded case involving displacement of the original winning candidate. Abstention reduces negative predictor-scene outcomes from four under forced activation to one under the standard activation policy.

These results provide empirical evidence regarding the practical behaviour of TCMA under prospective evaluation and temporal dependence. We do not interpret them as establishing a formal dependence-aware guarantee or universal per-query non-degradation.

We have revised the theoretical discussion to make these distinctions explicit.

Changes in manuscript:

  • Section 3.3, page 6, including Lemma 1 and Proposition 1.
  • Appendix C, theoretical derivation and finite-sample analysis.
  • Appendix N.5, page 27, Experiment R5 and Table 16.
  • Appendix N.6, pages 27–28, Experiment R6 and Tables 17–18.
Reviewer_3gnaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to improve black-box pedestrian trajectory prediction models using historical trajectories collected during deployment, without retraining the models. To address this problem, the authors propose TCMA, which employs a causal memory mechanism to store completed trajectories, retrieve similar motion patterns, and replace candidate trajectories that have historically contributed less to prediction accuracy. The method also introduces an online self-tuning strategy to adapt retrieval configurations using observed data. Experiments on seven predictors using the ETH-UCY dataset demonstrate promising results, achieving average improvements of 6.2% in minADE and 8.5% in minFDE.

Critical strengths
  1. The paper addresses a practically relevant problem, particularly for deployment scenarios where retraining or modifying existing models is not feasible.

  2. The proposed method is simple and flexible, allowing it to be applied to different prediction models without additional training.

  3. The experiments cover various types of trajectory predictors and demonstrate relatively consistent performance improvements.

Critical weaknesses
  1. The methodological novelty is relatively limited. The ideas of historical trajectory retrieval and memory augmentation have already been explored in MANTRA [1] and MemoNet [2], while the use of historical winning frequencies for candidate trajectories has also been considered in TrajEvo [3]. Therefore, the proposed method appears to be more of a combination and extension of existing techniques rather than a substantially novel approach.

  2. The effectiveness of the core design choices is not sufficiently validated. The paper lacks comparisons with simpler alternatives, such as random candidate replacement or other candidate selection strategies. Consequently, it remains unclear whether the proposed replacement mechanism provides meaningful advantages over simpler approaches.

  3. The theoretical analysis offers limited practical insights. The proposed non-degradation guarantee relies on conditions that cannot be verified at inference time. Therefore, the theoretical results do not sufficiently establish the robustness of the method in practical applications.

  4. The experimental evaluation is relatively limited. The experiments are conducted only on the ETH-UCY dataset, and the improvements on stronger prediction models are relatively modest. Therefore, the effectiveness and generalization ability of the proposed method in more complex scenarios remain unclear.

References

[1] Marchetti et al. MANTRA: Memory Augmented Networks for Multiple Trajectory Prediction. CVPR, 2020.

[2] Xu et al. Remember Intentions: Retrospective-Memory-Based Trajectory Prediction. CVPR, 2022.

[3] Zhao et al. TrajEvo: Trajectory Prediction Heuristics Design via LLM-Driven Evolution. AAAI, 2026.

Questions for authors

See weaknesses

5 comments
AuthorsOpenReview ↗

Response 1

We sincerely thank Reviewer for the detailed evaluation and for identifying several important aspects requiring further clarification and experimental validation. We particularly appreciate the comments regarding methodological novelty, candidate-selection alternatives, the practical interpretation of the theoretical analysis, and generalization beyond ETH-UCY.

W1. Methodological Novelty and Distinction from Existing Memory-Based Methods

We appreciate this observation and agree that trajectory retrieval, memory augmentation, and historical candidate statistics have precedents in the literature. We have therefore revised the Related Work section to make the distinction between these approaches and TCMA more explicit.

The contribution of TCMA is not the introduction of trajectory retrieval or memory augmentation as isolated components. Instead, we investigate how deployment-completed trajectories can be used to adapt arbitrary frozen trajectory predictors under three simultaneous constraints:

  1. Frozen black-box predictor: TCMA operates using exported candidate predictions without requiring access to model parameters, internal features, or gradients.
  2. Fixed candidate budget: TCMA replaces existing candidate slots rather than increasing the number of output trajectories.
  3. Causal memory eligibility: Only trajectories whose complete future horizons have elapsed can be stored and retrieved.

These requirements distinguish TCMA from the related approaches.

MANTRA and MemoNet: Both employ learned memory-based prediction mechanisms. MANTRA uses learned controllers for associative trajectory memory, while MemoNet retrieves trajectories through a learned addresser. Their memory mechanisms are integrated into separately trained prediction architectures. In contrast, TCMA uses explicit descriptors and output-level replacement without requiring a memory-aware predictor or an auxiliary retrieval network.

TrajEvo: We agree that TrajEvo also uses per-slot winning statistics. However, the purpose and use of these statistics differ. TrajEvo uses per-slot win statistics as feedback to an LLM during offline heuristic design. TCMA uses win statistics for output-level slot replacement under a fixed K, with deterministic last-q replacement as its primary policy. We additionally evaluate historical win-frequency selection as an alternative output-level strategy.

Online adaptation methods: Existing approaches such as T4P, OAT-Mem, and AOL employ predictor updates or separately trained adaptation components. TrajICL uses a separately trained predictor to perform in-context adaptation. TCMA instead operates directly on candidate outputs without requiring gradients or trainable auxiliary components.

To examine whether these design choices provide benefits beyond straightforward combinations of existing retrieval and memory mechanisms, we added several controlled experiments.

First, Experiment R3 compares TCMA with simpler similarity-based and recency-based output-level proxies under the same frozen-predictor and fixed-K setting.

MethodMean minADE@20 gainNegative predictor-scene outcomes
TCMA+6.52%1
Similarity proxy+3.19%11
Recency proxy+0.71%17

TCMA therefore achieves a higher mean improvement and fewer negative predictor-scene outcomes than these simpler alternatives.

Second, Experiment R4 examines the importance of causal memory eligibility. Allowing unfinished future trajectories into memory increases the measured mean minADE@20 improvement from 6.52% to 17.47%, illustrating how non-causal evaluation can substantially inflate apparent performance gains.

Third, Experiment R1 shows that candidate-slot selection also affects displacement risk. Replacing the most frequently winning slots reduces the mean minADE@20 improvement from 6.52% to 3.71%, while increasing the best-candidate displacement rate from 7.4% under the primary last-q policy to 27.2%.

Together, these experiments demonstrate that TCMA's contribution is not simply the combination of historical retrieval and candidate replacement. The interaction between causal eligibility, retrieval and gating, and fixed-budget replacement materially affects the observed performance and displacement behaviour.

We have revised the Related Work and experimental sections to make these distinctions explicit and to position TCMA as a deployment-constrained output-level adaptation framework rather than claiming novelty for individual memory or retrieval components.

Changes in manuscript:

  • Section 1, Introduction, pages 1–2.
  • Section 2, Related Work, page 3.
  • Section 3, Methodology, pages 3–6.
  • Section 4.3, Additional Controlled Experiments, page 9.
  • Appendix N.1, N.3, and N.4.
AuthorsOpenReview ↗

Response 2

W2. Effectiveness of Candidate-Slot Replacement Strategies

We agree that evaluating alternative candidate-selection strategies is important for understanding the contribution of the replacement policy.

We therefore added Experiment R1 in Appendix N.1, comparing five candidate-slot replacement strategies across all seven predictors.

To isolate the effect of candidate selection, we keep the following components fixed:

  • Frozen predictor configurations.
  • Retrieved candidate trajectories.
  • Admission gates.
  • Replacement count (q).
  • Evaluation windows.

Only the original candidate slots selected for replacement are changed.

The five strategies are:

  1. Deterministic last-q replacement, used in the main TCMA results.
  2. Historical win-frequency selection.
  3. Random candidate replacement, evaluated over five repetitions.
  4. Ensemble-disagreement selection.
  5. Reverse selection, which replaces the most frequently winning slots.

The results are summarized below.

Replacement strategyMean minADE@20 gainMean minFDE@20 gain
Last-q (main method)6.52%9.10%
Win-frequency6.86%9.47%
Random6.33%8.74%
Disagreement6.87%9.58%
Reverse (most-winning slots)3.71%5.32%

We additionally examine how frequently each strategy displaces the original best candidate:

Replacement strategyBest candidate displaced
Last-q7.4%
Win-frequency4.8%
Random8.5%
Disagreement4.4%
Reverse27.2%

The results provide several useful observations.

First, random replacement retains much of the improvement achieved by the main last-q policy, with 6.33% versus 6.52% mean minADE improvement. This indicates that retrieving useful causally completed trajectories is a major source of the overall improvement.

Second, historical win-frequency and disagreement-based selection achieve slightly higher mean gains, while displacing the original best candidate less frequently.

Third, reverse selection substantially reduces performance. Its mean minADE improvement decreases to 3.71%, and its best-candidate displacement rate increases to 27.2%.

These findings demonstrate that the selection policy influences the risk of removing useful original predictions, while retrieval provides additional candidate coverage.

Importantly, we also corrected the methodological description in Section 3.2. The revised section now explicitly states that the main TCMA results use deterministic last-q replacement, while historical win-frequency selection is evaluated separately as an alternative strategy.

Changes in manuscript:

  • Section 3.2, page 5.
  • Figures 1 and 2, pages 4–5.
  • Section 4.3, page 9.
  • Appendix N.1, pages 24–25.
  • Table 12, page 25.
AuthorsOpenReview ↗

Response 3

W3. Practical Interpretation of the Theoretical Guarantees

We appreciate this important observation and agree that the distinction between conditional non-degradation and operational deployment-time guarantees must be explicit.

We have revised the theoretical presentation to distinguish three aspects: the conditional result in Lemma 1, the marginal displacement-risk bound in Proposition 1, and the operational activation procedure used by TCMA.

1. Conditional non-degradation result

Lemma 1 states that replacing a set of candidate slots cannot increase minADE@K if the original winning candidate remains outside the replaced set.

This is a deterministic conditional result. However, the condition depends on the current query's ground-truth future, which is unavailable at inference time.

We therefore explicitly clarify that Lemma 1 does not provide a per-query deployment-time safety certificate. Instead, it characterizes the condition under which candidate replacement is guaranteed not to increase the minimum prediction error.

2. Marginal displacement-risk bound

Proposition 1 provides a marginal upper confidence bound on the probability that the replacement set contains the original winning candidate.

We have generalized the formulation to an arbitrary replacement set S, rather than restricting the analysis to a particular candidate-slot policy.

We have also clarified the multiple-comparison requirements:

  • For the reported last-q policy, q is selected from {1, 2, 3}. We therefore apply Bonferroni correction at α/3 to account for the selection among these three candidate sets under the independent-sample assumptions.
  • For data-dependent historical win-frequency selection, simultaneous bounds over the K candidate slots are required.

The manuscript also explicitly states that the Clopper–Pearson bound assumes independent reference samples and does not automatically provide coverage under arbitrary temporal dependence.

3. Prospective empirical validation and temporal dependence

To examine the practical behaviour of these bounds, we added Experiment R5.

Across 26 active predictor-scene combinations:

  • The corrected last-q bound holds in 23 of 26 cases, compared with 22 of 26 without the q-selection correction.
  • The mean upper bound is 0.102, compared with an observed displacement frequency of 0.077.
  • The naive win-frequency bound holds in 12 of 26 cases, increasing to 26 of 26 after Bonferroni correction over the K candidate slots.
  • Temporal block-bootstrap gain intervals have positive lower endpoints in 24 of 26 cases using 8-second blocks and 23 of 26 cases using 16-second blocks.

4. Per-query degradation and abstention

We additionally added Experiment R6 to examine how frequently TCMA worsens individual predictions.

Across activated predictions, the pooled per-query degradation rate is 5.9%. Every degraded prediction involves displacement of the original winning candidate, with zero exceptions to the deterministic implication of Lemma 1.

We also compare the standard activation policy against forced activation:

EvaluationTCMA with abstentionForced activation
Mean all-scene minADE gain6.52%7.38%
Per-query degradation rate5.9%5.9%
Negative predictor-scene results14

Although forced activation produces a higher aggregate mean improvement, it increases the number of negative predictor-scene outcomes from one to four. This demonstrates the practical role of abstention in controlling scene-level performance losses.

Together, these analyses provide empirical evidence regarding displacement risk, temporal robustness, and activation behaviour. We do not interpret them as establishing universal or per-query non-degradation.

Changes in manuscript:

  • Section 3.3, page 6, including Lemma 1 and Proposition 1.
  • Appendix C, theoretical derivation and finite-sample analysis.
  • Appendix N.5, page 27, Experiment R5 and Table 16.
  • Appendix N.6, pages 27–28, Experiment R6 and Tables 17–18.
Reviewer_Q936OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes to improve trajectory prediction by leveraging real-world trajectories observed during deployment through a training-free causal online memory augmentation (TCMA) strategy. TCMA consists of a scene-invariant egocentric channel and a scene-aware world-frame channel, where the former retrieves similar motion patterns across different locations by transforming trajectories into agent-centric coordinates, while the latter exploits spatial regularities within the same scene by considering the positions, headings, and speeds of trajectories. The retrieved trajectories are then used to replace less useful candidate predictions, improving prediction performance without retraining the base predictor. Experimental results on the ETH-UCY benchmark demonstrate their performance.

Critical strengths

The paper proposes a training-free causal online memory augmentation strategy for trajectory prediction. The approach is particularly useful for real-world deployed trajectory predictors because it improves prediction performance without retraining the underlying model, reducing computational costs. Moreover, it effectively leverages trajectories routinely collected during deployment to enhance subsequent predictions. This approach offers a practical, promising way to reuse deployment data, improving prediction performance while avoiding the additional costs of model retraining. Furthermore, the proposed approach's model design appears novel. I recommend acceptance.

Critical weaknesses

I believe the proposed method should also be evaluated on autonomous driving datasets. Autonomous driving scenarios typically involve larger data volumes and a more diverse set of agent types, whose motion patterns can vary substantially across categories. Demonstrating the proposed method's effectiveness on autonomous driving benchmarks would provide stronger evidence of its generalizability and practical applicability in more complex real-world scenarios.

Questions for authors
  1. Will you release your project code?
  2. My understanding is that all trajectories are rotated to align their headings to 0°. Why, then, is the angular difference between the query trajectory and the retrieved memory trajectory still calculated?
3 comments
AuthorsOpenReview ↗

Response 1

We sincerely thank the reviewer for the positive assessment of our work and for recognizing the practical value of training-free causal memory augmentation for deployed trajectory predictors. We particularly appreciate the suggestions regarding cross-dataset evaluation, code availability, and the distinction between the two retrieval coordinate frames.

W1. Generalization to Pedestrian Prediction in Mixed-Agent Autonomous-Driving Scenes

We agree that evaluating TCMA beyond ETH-UCY is important for assessing its applicability across different trajectory distributions. We initially selected ETH-UCY as the primary benchmark because it is an established and widely used dataset in pedestrian trajectory prediction, providing a standard setting for comparison with previous methods. Nevertheless, we agree that additional datasets are valuable for assessing broader generalization. To address this concern, we extended our evaluation to include the Stanford Drone Dataset (SDD) and nuScenes.

The nuScenes dataset contains mixed agents, including vehicles and pedestrians. We evaluate pedestrian prediction to test whether TCMA remains effective in a more complex mixed-agent environment.

We use CVM-S as the frozen base predictor because it requires no training and exports a fixed K=20-candidate set. This provides a controlled setting to isolate the transfer effect of TCMA without introducing additional predictor training or modifications.

Results on nuScenes. On the official nuScenes validation split, TCMA achieves the following pooled improvements for pedestrian prediction across activated recording-agent streams:

MetricRelative Improvement95% Confidence Interval
minADE@20+5.92%[2.88%, 9.51%]
minFDE@20+8.49%[4.26%, 13.31%]

Three recording-agent streams (Holland Village vehicles and pedestrians, and Queenstown pedestrians) do not satisfy the causal activation requirements and therefore retain their base predictions unchanged.

Vehicle results are also reported as auxiliary cross-category context, with pooled improvements of 9.71% in minADE@20 and 12.33% in minFDE@20. However, the evaluation focuses on pedestrian prediction.

Results on SDD. We additionally evaluate TCMA on SDD, which provides a complementary pedestrian trajectory prediction benchmark with different recording environments and motion patterns from ETH-UCY. The evaluation covers 17 test videos and 1,741 prospective evaluation windows using three predictors:

  1. CVM-S: Does not require learned weights.
  2. TrajEvo: Transferred zero-shot from ETH-UCY.
  3. TUTR: Trained using the official SDD configuration because no released SDD checkpoint was available for this evaluation.

The resulting minADE@20 improvements are:

PredictorminADE@20 Improvement
CVM-S+18.48%
TrajEvo+7.36%
TUTR+1.79%

These SDD results are reported as point estimates without confidence intervals, and we do not claim statistical significance for TUTR's relatively small improvement.

Scope and limitations. The use of CVM-S on nuScenes is a deliberate first-step scope choice. Its training-free operation and fixed K-candidate output allow us to isolate the transfer effect of TCMA in a mixed-agent autonomous-driving environment. We are extending this evaluation to additional training-free predictors on nuScenes. If additional results become available, we will incorporate them into the manuscript and report them here. Evaluation with heterogeneous driving predictors remains an important direction for future work.

Changes in the Manuscript

  • Section 4.4: Cross-Dataset Generalization, page 9.
  • Section 5: Discussion & Limitations, page 10.
  • Appendix N.7: Stanford Drone Dataset Evaluation (R8), page 28.
  • Appendix N.8: nuScenes Pedestrian Prediction in a Mixed-Agent Autonomous-Driving Environment (R7), pages 28–29, including Table 20.
AuthorsOpenReview ↗

Response 2

Q1. Code and Configuration Release

Yes. We appreciate the reviewer raising this question. We plan to release the complete implementation and experimental configurations upon publication.

Changes in the Manuscript

  • Section 6: Conclusion, page 10. Added: "Code and configurations will be released upon publication."

Q2. Heading Difference in the World-Frame Retrieval Channel

Thank you for pointing out the need to clarify the distinction between the two retrieval channels.

TCMA employs two complementary retrieval protocols, and heading normalization is applied only in the ego-centric channel.

Ego-centric retrieval channel. In the ego-centric channel, Eq. (8) translates and rotates each observed trajectory into the agent's local coordinate system. This transformation removes absolute position and orientation differences, allowing trajectories with similar motion characteristics to be retrieved across different scenes. Accordingly, the descriptor in Eq. (9) contains motion-related features without an absolute heading component.

World-frame retrieval channel. The world-frame channel operates differently. It retains the original scene coordinates rather than applying the ego-centric rotation. Its retrieval distance in Eq. (10) includes an explicit angular term to capture directional differences between agents.

This distinction is important because two agents may occupy similar spatial locations and have similar speeds while moving in substantially different directions. The angular term allows the world-frame channel to distinguish these cases and capture scene-specific directional regularities.

Therefore, the angular difference is not calculated after applying the ego-centric heading normalization. It is independently used in the world-frame retrieval channel, which preserves the original orientation information.

We have clarified this distinction in the revised manuscript.

Changes in the Manuscript

  • Section 3.1: Pages 4–5.
  • Eqs. (8)–(9): Ego-centric retrieval.
  • Eq. (10): Scene-aware world-frame retrieval.
  • Figure 2: Page 5.

Thank you for your detailed response. All my concerns have been addressed. I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.