- Rating
- 2
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper studies whether self-supervised pretraining helps multi-agent motion forecasting, since past reports disagree. It proposes Actor-JEPA, a JEPA-style pretraining for the agent encoder only. The encoder first mixes agents with attention at each timestep (the "cross-agent stage"). It then runs a six-layer transformer along time for each agent (the "per-agent core"). Timesteps of this fused sequence are masked, and a predictor must predict the latents of the masked timesteps from an EMA target encoder. The loss is smooth-L1 in latent space plus a VICReg variance term, with per-token LayerNorm on the targets. The pretrained encoder then replaces the context encoder of an MTR decoder, which is fine-tuned on WOMD. The map is used only by the decoder.
The main result is an internal comparison against the same model trained from scratch, over four label fractions with 14 independent pretraining runs. Peak mAP improves by 5% to 7% relative (for example 0.2089 to 0.2245 at 20% labels). The paper also reports three analyses. Transferring only the cross-agent stage keeps the gain, while the per-agent core alone gives nothing. Pretraining can collapse silently late in training, which effective rank detects but the loss does not. Pretraining on WOMD and fine-tuning on Argoverse 2 gives no gain.
Critical strengths
S1. Careful statistics and honest reporting. The paper treats the pretraining run, not the fine-tuning seed, as the unit of replication, and shows why this matters (0.016 spread across runs vs 0.0024 across seeds). It reports means, standard deviations, and two significance tests for every regime. It shows every run, reports a conservative late-window metric beside peak mAP, and reports results that go against it: the null cross-dataset result, four of 14 runs below the from-scratch mean, and the weaker same-subset result in the limitations. This level of rigor is rare in motion forecasting papers.
S2. A useful practical warning. The silent collapse result is clear and easy to act on. The pretext loss keeps falling while effective rank drops from 56 to 2 in one epoch, and the collapsed encoder is worse than random initialization. The analysis of why the VICReg variance term cannot see this collapse is also useful to others who pretrain trajectory encoders.
Critical weaknesses
W1. The gain seems to come from extra data, not from the pretraining objective. This is my biggest concern. The "100% labels" setting does not use the full training split. It keeps only a subset of the scenarios (Sec. 4). Pretraining, on the other hand, uses every scenario in the split. It also uses the whole recording, including the future part of each trajectory (Sec. 3.2). In motion forecasting, those future trajectories are the labels. They come from the same tracking pipeline as the inputs and cost nothing extra. So the pretrained model sees the future motion of many scenes that the from-scratch model never sees. The paper's own control in the limitations section supports this concern. When pretraining is limited to the same subset used for fine-tuning, the gain becomes small and is no longer significant. The missing baseline is simple: supervised training on all scenarios, or supervised pretraining on all scenarios followed by fine-tuning. Without it, the paper shows that more data helps, which is expected. It does not show that the JEPA objective itself helps. For the same reason, the "label scarcity" framing does not fit this task well, because the labels are not hand annotations.
W2. All main results come from one non-standard model. The encoder never sees the map, and the absolute mAP is well below published results on WOMD. A gain on a weak model may not hold for a strong one. The paper does not compare against any other pretraining method for motion forecasting (for example Forecast-MAE, DECAMP, Traj-MAE, or SEPT) in the same pipeline. It also does not test the design it argues against, a joint agent-and-map token set (Sec. 3.1 says "We do not test that alternative here"). So the claim that the factorized target is the right choice is not supported. Three more issues weaken the main claim:
- The primary metric is peak mAP, the best epoch on the validation set, and the validation set is also the only evaluation set. The paper itself says this metric is biased upward. On the more conservative late-window metric, the main result is weaker and not significant under one of the two tests.
- Table 2 does not agree with Table 1. In Table 2, transferring the complete encoder gives a small, non-significant gain, and the cross-agent stage alone does better. Yet the method transfers both stages (Sec. 3.4).
- Pretraining on WOMD and fine-tuning on Argoverse 2 gives no gain. The benefit is tied to the pretraining dataset, which limits the practical value of the approach.
Questions for authors
Q1. Can you add a supervised baseline trained on all WOMD training scenarios, without subsampling? And a supervised pretraining baseline on all scenarios, followed by fine-tuning on the subset? If JEPA pretraining still beats these, my concern in W1 would be resolved.
Q2. Pretraining sees the future part of every training trajectory. What happens if pretraining sees only the past steps, so that it never sees the forecasting targets?
Q3. How does Actor-JEPA compare with other self-supervised methods for motion forecasting, such as Forecast-MAE, DECAMP, Traj-MAE, or SEPT, in the same pipeline and with the same data?
Q4. Section 3.1 argues that masking after cross-agent fusion is what makes the pretext task well posed. This is not tested. Can you compare against masking before fusion, or against a joint agent-and-map token set?
Q5. Does the gain hold when the pretrained encoder is used in a standard MTR-style model with map input, at published-level performance?
Q6. Table 1 vs Table 2. Why does the same complete-transfer setting give a clearly lower result in Table 2 than in Table 1? If the cross-agent stage alone is at least as good, why transfer both stages?
Q7. Please report results with the checkpoint chosen on a held-out split that is separate from the evaluation split, or at a fixed final epoch.
Minor:
Q8. The collapse in Section 6.3 happens far later in training than the budget used for all main runs. Does it ever occur under the main recipe? Also, effective rank as a collapse signal is close to RankMe (Garrido et al., ICML 2023). Please discuss how your use differs.
Q9. Table 6 compares the variants against a single run that is one of the best of the main runs, not against the average. Please compare against the average over runs instead.
Q10. Appendix C mentions "our frozen-encoder experiments" with adapters. These results do not appear in the paper. Please add them or remove the text.
Q11. The abstract says code "will be made publicly available", while the reproducibility statement says "we additionally publish our code". No anonymous link is given. Please clarify.
Q12. Typos: "proposesd" (Sec. 3.2). The second sentence of Sec. 3.1 is hard to parse.
5 comments
Response to Reviewer V1sK (2/3)
Q3 (Forecast-MAE, Traj-MAE, DECAMP, SEPT). We ran the closest controlled comparison first: the same encoder, masking, predictor, data and schedule, with only the target changed to raw-state statistics of the masked steps. At the best checkpoint it transfers at least as well as the latent target: 0.2385 +- 0.0038 vs 0.2245 +- 0.0159 (n = 6 vs 14, Welch p = 0.007, rank test p = 0.070).
That control was run two months earlier, so a pre-registered concurrent replication followed (n = 10 vs 10): +5.8%, CI [+0.7%, +10.9%], one-sided Welch p = 0.015, rank test p = 0.061. Its direction was set by the first six raw-target runs, which are also in its test group.
The difference is reliability at the best checkpoint. 11 of 24 latent runs land in a lower mode, and none of the 10 raw-target runs do. In the late window and at epoch 30 the two targets are not resolved at 4% labels.
A pre-registered follow-up repeats the comparison at other label fractions with the same raw-target encoders, all ten at 1% and 2% and the first six at 20%. Its interim rule was fixed before any of its runs started, and these three fractions are complete (n = 10 vs 12 at 1% and 2%, 6 vs 6 at 20%). The raw target is ahead under every view at 1% and 2%. At 20% the difference has the same sign and is not resolved (Appendix B, Table 8). Each cell gives the difference, its 95% CI and the Welch p:
| raw vs latent | 1% labels | 2% labels | 20% labels |
|---|---|---|---|
| best checkpoint | +10.3% [+6.4%, +14.2%], p = 1e-4 | +10.0% [+5.7%, +14.4%], p = 2e-4 | +2.7% [-0.7%, +6.2%], p = 0.10 |
| late window | +11.8% [+8.2%, +15.4%], p = 1e-5 | +10.3% [+5.5%, +15.1%], p = 4e-4 | +3.1% [-0.4%, +6.5%], p = 0.07 |
| epoch 30 | +9.5% [+5.6%, +13.5%], p = 2e-4 | +10.2% [+5.8%, +14.6%], p = 2e-4 | +2.7% [-1.1%, +6.4%], p = 0.13 |
The rank test gives p = 9e-5 for each row at 1%, p <= 0.002 at 2% and p >= 0.09 at 20%. Over scratch the raw target gives +18.2% at 1% (0.1967 +- 0.0017 vs 0.1664 +- 0.0061, one-sided Welch p = 8e-8), +16.9% at 2% (0.2212 +- 0.0046 vs 0.1891 +- 0.0058, p = 4e-10) and +7.8% at 20% (0.3019 +- 0.0025 vs 0.2799 +- 0.0086, p = 5e-4), against +7.2%, +6.3% and +5.0% for the latent target. At 1% no fine-tune has an evaluation dip, and every raw-target run is above every latent run. At 2% the raw target does not remove the dips: 4 of its 10 fine-tunes have one, in epochs 20 to 24, against 4 of the 12 latent ones. At 20% it is 2 of 6 for each target. The latent runs are those of Table 1, pretrained two months before the raw-target runs. At 4% a concurrent latent control reproduced that group (p = 0.97).
We therefore no longer attribute the gain to the latent space, and the final version would present the raw target as the recommended variant (point 2 of the general response).
For the named methods: to our knowledge only Forecast-MAE has released code, and only for Argoverse 2. All four use map tokens and per-agent tokens that our map-free, scene-fused encoder does not have. We therefore ported their trajectory objectives onto our encoder, data and schedule:
- Forecast-MAE: predict the future from the history, or the history from the future
- Traj-MAE temporal masking and SEPT's masked trajectory modeling: reconstruct half the observed steps from the other half
- SEPT's tail prediction: forecast the last observed steps
These are pre-registered with n = 6 each, and we will post the results that arrive during the discussion. They are ports, not reproductions, and the paper will say so. DECAMP's combined objective needs both of our losses in one pass and is not included.
Q4 (masking before fusion, joint agent-and-map tokens). We test neither alternative. Sec. 3.1 motivates masking after fusion and now states that the ordering is not ablated, and the paper claims no advantage over either alternative. The new controls vary the target and the window, not the ordering or the token set.
Q5 (standard MTR with map input). Our forecaster at 100% labels reaches 0.3508 mAP, against MTR's published 0.4164 on the validation split (Shi et al., 2022). Our encoder never sees the map, by design: that keeps per-agent and cross-agent computation separable (Sec. 3.1), which the analyses need. Whether the gain survives inside MTR's joint agent-map encoder is open, and the limitations say so. With the map removed from the forecaster as well, pretraining still adds +6.5% (one-sided p = 0.004).
Response to Reviewer V1sK (3/3)
Q6 (Table 1 vs Table 2, now Table 4). Table 2's complete-transfer row (0.2147) uses five of Table 1's fourteen encoders, fine-tuned a second time. In Table 1 the same five average 0.2273 (all fourteen: 0.2245), so the gap of 0.0126 lies between two fine-tunes of the same encoders. One encoder accounts for 0.0093 of it. Its two fine-tunes share configuration, checkpoint, seed and evaluation. Their logs are identical through the first training iteration and diverge only through GPU nondeterminism. The Table 2 run hit a transient evaluation collapse that had not recovered by epoch 30, so its peak is the epoch-10 value (0.1822 against 0.2288). Such collapses occur in 12 of 14 runs of both arms at this regime. The revision reports both draws for the five encoders and corrects the within-encoder spread to its pooled value (SD 0.0100, heavy-tailed).
On transferring both stages: the data do not separate them. Paired by encoder, cross-agent only minus complete is +0.0090 with Table 2's draws (p = 0.46) and -0.0037 with Table 1's (p = 0.52). We transfer the complete encoder because that is what Table 1 measures, and we claim nothing for the core beyond it.
Q7 (checkpoint selection). Every Table 1 comparison is now also reported at the fixed final epoch 30, with no selection:
| labels | epoch-30 gain | p |
|---|---|---|
| 4% | +11.6% | 0.004 |
| 20% | +4.4% | 0.0495 |
| 2% | +5.5% | 0.035 |
| 1% | +7.2% | 0.005 |
| 100% | 0.0% | - |
For Table 1 the conclusions are unchanged. The cleanest row is 1%: no fine-tune there has an evaluation dip, the best epoch is the final one in 18 of 21 runs, and the gain is +7.2% at the peak and at epoch 30 and +7.5% over the late window, with minADE, minFDE and MissRate each significant under all three. The raw-target control repeated at 1%, 2% and 20% agrees across the three views in the same way (Q3).
For the new controls of Sec. 6.1 the conclusions are not unchanged: at epoch 30 none of their differences is resolved:
- raw vs concurrent latent target: -5.7%
- raw vs scratch: +0.3%, p = 0.49 (first six runs)
- past-only raw vs scratch: +2.5%, p = 0.42
- latent vs supervised on degraded data: +10.6%, p = 0.42
The cause is the fine-tune with the map at 4% labels. In 12 of 14 runs of each arm, an evaluation between epochs 20 and 30 falls below 60% of the run's peak, so any single epoch is noisy. The map-free forecaster has no such dips (0 of 28 runs), and there the peak, epoch 30 and late window agree (+6.5%, +7.1%, +7.9%). Appendix B reports every control at all three summaries.
Two corrections to Table 1:
- The 1% and 2% rows compare 12 pretrained runs against 9 scratch runs, not 12.
- The late window covers all six epochs 25-30. The submitted text said that two of them were evaluated. The numbers are unchanged.
Q8 (collapse under the main recipe, RankMe).
- Within the 25 epochs of the main recipe, no: all 14 Table 1 encoders measure effective rank 21.7-29.4, with no degenerate dimension. Trained far beyond that, yes: a 200-epoch run of the final recipe collapsed at epoch 141 (rank 42.5 to 11.3 in one epoch, 1.3-4.6 from epoch 145 on, new in Sec. 6.3). The submission's run collapsed at about epoch 165.
- Among healthy encoders, rank does not track transfer (Spearman +0.11, p = 0.71).
- RankMe (Garrido et al., 2023) uses effective rank to select models. We use it only to detect collapse. It is now cited and discussed.
- The collapse mechanism needs the latent target: the raw-target variant has no teacher whose output can degenerate. Its encoders keep their rank (22.2-23.8), but that was measured after 25 epochs, before any latent run collapsed, so it does not test the difference.
Q9 (Table 6, now Table 11, vs a single run). Against the mean of the 14 base-recipe runs, every variant lies within +0.7 to +1.0 SD and inside the 95% prediction interval of a single run. With one run per variant, they cannot be ranked against the base recipe in either direction, and the caption now says so.
Q10 (frozen-encoder experiments). Removed. They used the earlier, incomplete transfer.
Q11 (code). The code is at https://anonymous.4open.science/r/actor-jepa-CFF7: pretraining, transfer, fine-tuning, the pre-registered protocols, and the per-run results with the scripts that recompute Tables 1-5, 7 and 8. Figure scripts and the remaining appendix analyses are not included. The word "Code" in the abstract links to it, as it did in the submission, and the reproducibility statement now prints the URL.
Q12. Both are fixed. The typo in Sec. 3.2 is corrected, and the opening of Sec. 3.1 is rewritten.
Response to Reviewer V1sK (1/3)
W1/Q1 (data vs objective). We agree, and the revision says so.
The first baseline Q1 asks for is Table 1's 100% row. Trained from scratch on every scenario, the model reaches 0.3508, and pretraining adds nothing (-0.7%). Confining pretraining to the labeled 4% at equal gradient steps leaves +1.6% (p = 0.62, n = 3). The gain needs samples beyond the labeled set. A sample is a WOMD scene centered on one agent to predict (Sec. 4 now defines it). The labeled 4% of samples come from 18% of the scenes (Appendix I).
We ran the second baseline on the degraded, map-free pool of Sec. 5.1 (the 96% of training samples outside the labeled set), where both arms see the same trajectories. Supervising the same encoder on its 8 s futures, then fine-tuning identically, matches the pretext task and is numerically ahead:
The degraded futures serve as noisy labels. We did not run it on clean WOMD, and we claim no advantage over labels that exist.
What supervision in the labeled format cannot use is a scene without its future. There, masked prediction on the 11 observed steps gives +14.7% over scratch (Q2).
On the framing: we agree. Futures cost nothing, and the result above shows that even tracker-quality futures work as labels. What is scarce in practice is curated scenarios (offboard-quality tracks, selected target agents, a map), not futures. The revised claim is that pretraining on uncurated logs, which we simulate by degrading WOMD, helps a forecaster trained on few curated scenarios, by masked prediction or by supervision on the logs' own futures.
W2. Its points are answered under Q5 (non-standard model), Q3 (other pretraining methods), Q4 (joint token set), Q7 (peak mAP) and Q6 (Table 2). On Argoverse 2 we agree. At matched label fractions, WOMD pretraining gives no significant gain on AV2: +0.5% at 20% (p = 0.68, power 0.99 against the in-domain +5.0%), and +1% to +2% at 5% (two cohorts, neither significant, and pooled post hoc +1.6%, p = 0.033). The benefit is therefore tied to the pretraining distribution. Sec. 5.2 and the conclusion say so. The practical case is pretraining on unlabeled logs from the deployment distribution.
Q2 (past-only). Six encoders pretrained on the 11 observed steps with the raw target reach 0.2397 +- 0.0059, against 0.2089 +- 0.0111 from scratch (n = 6 vs 14):
That equals full-window pretraining with the same target (-1.1%, p = 0.39). With the latent target, the same test was inconclusive (+6.5%, p = 0.080) and its fine-tunes were unstable.
Caveats, stated in the paper: