acceptodds
Under review as a conference paper at ICLR 2027

ACTOR-JEPA: MULTI-AGENT MOTION FORECASTING USING LATENT-PREDICTIVE PRETRAINING

Abstract

Motion forecasting models must capture how agents influence one another, and are trained on large labeled trajectory datasets. Self-supervised pretraining promises to reduce that dependence on labels, yet reported results conflict, ranging from substantial gains to null results. We introduce Actor-JEPA, which applies the Joint-Embedding Predictive Architecture to multi-agent scene encoding. Its encoder factorizes the scene into a cross-agent interaction stage followed by per-agent temporal encoding, and that factorization defines the prediction target: agents are mixed before masking, so the model predicts masked agent-timestep positions in a representation that already carries interaction context, rather than reconstructing coordinates. Across four label regimes and independent pretraining runs on the Waymo Open Motion Dataset, Actor-JEPA raises peak mAP from to at labels ( relative, , ) and from to at full labels (, , ), in each case against the identical architecture trained from scratch. Within a dataset, two conditions govern whether any benefit appears: the cross-agent stage must be transferred, since the per-agent core alone performs at from-scratch level, and pretraining must stop before a representation collapse that neither the pretext loss nor the variance regularizer reveals, which we detect by monitoring the effective rank of the encoder's features. Code will be made publicly available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, joint-embedding predictive architecture, motion forecasting, autonomous driving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_V1sKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies whether self-supervised pretraining helps multi-agent motion forecasting, since past reports disagree. It proposes Actor-JEPA, a JEPA-style pretraining for the agent encoder only. The encoder first mixes agents with attention at each timestep (the "cross-agent stage"). It then runs a six-layer transformer along time for each agent (the "per-agent core"). Timesteps of this fused sequence are masked, and a predictor must predict the latents of the masked timesteps from an EMA target encoder. The loss is smooth-L1 in latent space plus a VICReg variance term, with per-token LayerNorm on the targets. The pretrained encoder then replaces the context encoder of an MTR decoder, which is fine-tuned on WOMD. The map is used only by the decoder.

The main result is an internal comparison against the same model trained from scratch, over four label fractions with 14 independent pretraining runs. Peak mAP improves by 5% to 7% relative (for example 0.2089 to 0.2245 at 20% labels). The paper also reports three analyses. Transferring only the cross-agent stage keeps the gain, while the per-agent core alone gives nothing. Pretraining can collapse silently late in training, which effective rank detects but the loss does not. Pretraining on WOMD and fine-tuning on Argoverse 2 gives no gain.

Critical strengths

S1. Careful statistics and honest reporting. The paper treats the pretraining run, not the fine-tuning seed, as the unit of replication, and shows why this matters (0.016 spread across runs vs 0.0024 across seeds). It reports means, standard deviations, and two significance tests for every regime. It shows every run, reports a conservative late-window metric beside peak mAP, and reports results that go against it: the null cross-dataset result, four of 14 runs below the from-scratch mean, and the weaker same-subset result in the limitations. This level of rigor is rare in motion forecasting papers.

S2. A useful practical warning. The silent collapse result is clear and easy to act on. The pretext loss keeps falling while effective rank drops from 56 to 2 in one epoch, and the collapsed encoder is worse than random initialization. The analysis of why the VICReg variance term cannot see this collapse is also useful to others who pretrain trajectory encoders.

Critical weaknesses

W1. The gain seems to come from extra data, not from the pretraining objective. This is my biggest concern. The "100% labels" setting does not use the full training split. It keeps only a subset of the scenarios (Sec. 4). Pretraining, on the other hand, uses every scenario in the split. It also uses the whole recording, including the future part of each trajectory (Sec. 3.2). In motion forecasting, those future trajectories are the labels. They come from the same tracking pipeline as the inputs and cost nothing extra. So the pretrained model sees the future motion of many scenes that the from-scratch model never sees. The paper's own control in the limitations section supports this concern. When pretraining is limited to the same subset used for fine-tuning, the gain becomes small and is no longer significant. The missing baseline is simple: supervised training on all scenarios, or supervised pretraining on all scenarios followed by fine-tuning. Without it, the paper shows that more data helps, which is expected. It does not show that the JEPA objective itself helps. For the same reason, the "label scarcity" framing does not fit this task well, because the labels are not hand annotations.

W2. All main results come from one non-standard model. The encoder never sees the map, and the absolute mAP is well below published results on WOMD. A gain on a weak model may not hold for a strong one. The paper does not compare against any other pretraining method for motion forecasting (for example Forecast-MAE, DECAMP, Traj-MAE, or SEPT) in the same pipeline. It also does not test the design it argues against, a joint agent-and-map token set (Sec. 3.1 says "We do not test that alternative here"). So the claim that the factorized target is the right choice is not supported. Three more issues weaken the main claim:

  • The primary metric is peak mAP, the best epoch on the validation set, and the validation set is also the only evaluation set. The paper itself says this metric is biased upward. On the more conservative late-window metric, the main result is weaker and not significant under one of the two tests.
  • Table 2 does not agree with Table 1. In Table 2, transferring the complete encoder gives a small, non-significant gain, and the cross-agent stage alone does better. Yet the method transfers both stages (Sec. 3.4).
  • Pretraining on WOMD and fine-tuning on Argoverse 2 gives no gain. The benefit is tied to the pretraining dataset, which limits the practical value of the approach.
Questions for authors

Q1. Can you add a supervised baseline trained on all WOMD training scenarios, without subsampling? And a supervised pretraining baseline on all scenarios, followed by fine-tuning on the subset? If JEPA pretraining still beats these, my concern in W1 would be resolved.

Q2. Pretraining sees the future part of every training trajectory. What happens if pretraining sees only the past steps, so that it never sees the forecasting targets?

Q3. How does Actor-JEPA compare with other self-supervised methods for motion forecasting, such as Forecast-MAE, DECAMP, Traj-MAE, or SEPT, in the same pipeline and with the same data?

Q4. Section 3.1 argues that masking after cross-agent fusion is what makes the pretext task well posed. This is not tested. Can you compare against masking before fusion, or against a joint agent-and-map token set?

Q5. Does the gain hold when the pretrained encoder is used in a standard MTR-style model with map input, at published-level performance?

Q6. Table 1 vs Table 2. Why does the same complete-transfer setting give a clearly lower result in Table 2 than in Table 1? If the cross-agent stage alone is at least as good, why transfer both stages?

Q7. Please report results with the checkpoint chosen on a held-out split that is separate from the evaluation split, or at a fixed final epoch.

Minor:

Q8. The collapse in Section 6.3 happens far later in training than the budget used for all main runs. Does it ever occur under the main recipe? Also, effective rank as a collapse signal is close to RankMe (Garrido et al., ICML 2023). Please discuss how your use differs.

Q9. Table 6 compares the variants against a single run that is one of the best of the main runs, not against the average. Please compare against the average over runs instead.

Q10. Appendix C mentions "our frozen-encoder experiments" with adapters. These results do not appear in the paper. Please add them or remove the text.

Q11. The abstract says code "will be made publicly available", while the reproducibility statement says "we additionally publish our code". No anonymous link is given. Please clarify.

Q12. Typos: "proposesd" (Sec. 3.2). The second sentence of Sec. 3.1 is hard to parse.

5 comments
AuthorsOpenReview ↗

Response to Reviewer V1sK (1/3)

W1/Q1 (data vs objective). We agree, and the revision says so.

  • The first baseline Q1 asks for is Table 1's 100% row. Trained from scratch on every scenario, the model reaches 0.3508, and pretraining adds nothing (-0.7%). Confining pretraining to the labeled 4% at equal gradient steps leaves +1.6% (p = 0.62, n = 3). The gain needs samples beyond the labeled set. A sample is a WOMD scene centered on one agent to predict (Sec. 4 now defines it). The labeled 4% of samples come from 18% of the scenes (Appendix I).

  • We ran the second baseline on the degraded, map-free pool of Sec. 5.1 (the 96% of training samples outside the labeled set), where both arms see the same trajectories. Supervising the same encoder on its 8 s futures, then fine-tuning identically, matches the pretext task and is numerically ahead:

    • 0.2411 +- 0.0133 vs 0.2304 +- 0.0095 (n = 8 vs 12)
    • Welch p = 0.076, rank test p = 0.070
    • 95% CI for JEPA relative to supervision: [-9.4%, +0.5%]

    The degraded futures serve as noisy labels. We did not run it on clean WOMD, and we claim no advantage over labels that exist.

  • What supervision in the labeled format cannot use is a scene without its future. There, masked prediction on the 11 observed steps gives +14.7% over scratch (Q2).

On the framing: we agree. Futures cost nothing, and the result above shows that even tracker-quality futures work as labels. What is scarce in practice is curated scenarios (offboard-quality tracks, selected target agents, a map), not futures. The revised claim is that pretraining on uncurated logs, which we simulate by degrading WOMD, helps a forecaster trained on few curated scenarios, by masked prediction or by supervision on the logs' own futures.

W2. Its points are answered under Q5 (non-standard model), Q3 (other pretraining methods), Q4 (joint token set), Q7 (peak mAP) and Q6 (Table 2). On Argoverse 2 we agree. At matched label fractions, WOMD pretraining gives no significant gain on AV2: +0.5% at 20% (p = 0.68, power 0.99 against the in-domain +5.0%), and +1% to +2% at 5% (two cohorts, neither significant, and pooled post hoc +1.6%, p = 0.033). The benefit is therefore tied to the pretraining distribution. Sec. 5.2 and the conclusion say so. The practical case is pretraining on unlabeled logs from the deployment distribution.

Q2 (past-only). Six encoders pretrained on the 11 observed steps with the raw target reach 0.2397 +- 0.0059, against 0.2089 +- 0.0111 from scratch (n = 6 vs 14):

  • +14.7%, 95% CI [+10.9%, +18.6%]
  • one-sided Welch p = 2e-7, rank test p = 3e-5

That equals full-window pretraining with the same target (-1.1%, p = 0.39). With the latent target, the same test was inconclusive (+6.5%, p = 0.080) and its fine-tunes were unstable.

Caveats, stated in the paper:

  • The advantage holds at the best checkpoint. In the late window it is +7.3% (p = 0.13), and at epoch 30 +2.5% (p = 0.42). Q7 explains why single epochs are noisy here.
  • A masked step inside the history is close to short-horizon prediction on the history. A pre-registered tail-forecasting control is running and is not in the paper yet.
  • WOMD's 9 s windows overlap within a 20 s segment, so the control is future-free per scene, not per segment.
AuthorsOpenReview ↗

Response to Reviewer V1sK (2/3)

Q3 (Forecast-MAE, Traj-MAE, DECAMP, SEPT). We ran the closest controlled comparison first: the same encoder, masking, predictor, data and schedule, with only the target changed to raw-state statistics of the masked steps. At the best checkpoint it transfers at least as well as the latent target: 0.2385 +- 0.0038 vs 0.2245 +- 0.0159 (n = 6 vs 14, Welch p = 0.007, rank test p = 0.070).

That control was run two months earlier, so a pre-registered concurrent replication followed (n = 10 vs 10): +5.8%, CI [+0.7%, +10.9%], one-sided Welch p = 0.015, rank test p = 0.061. Its direction was set by the first six raw-target runs, which are also in its test group.

The difference is reliability at the best checkpoint. 11 of 24 latent runs land in a lower mode, and none of the 10 raw-target runs do. In the late window and at epoch 30 the two targets are not resolved at 4% labels.

A pre-registered follow-up repeats the comparison at other label fractions with the same raw-target encoders, all ten at 1% and 2% and the first six at 20%. Its interim rule was fixed before any of its runs started, and these three fractions are complete (n = 10 vs 12 at 1% and 2%, 6 vs 6 at 20%). The raw target is ahead under every view at 1% and 2%. At 20% the difference has the same sign and is not resolved (Appendix B, Table 8). Each cell gives the difference, its 95% CI and the Welch p:

raw vs latent1% labels2% labels20% labels
best checkpoint+10.3% [+6.4%, +14.2%], p = 1e-4+10.0% [+5.7%, +14.4%], p = 2e-4+2.7% [-0.7%, +6.2%], p = 0.10
late window+11.8% [+8.2%, +15.4%], p = 1e-5+10.3% [+5.5%, +15.1%], p = 4e-4+3.1% [-0.4%, +6.5%], p = 0.07
epoch 30+9.5% [+5.6%, +13.5%], p = 2e-4+10.2% [+5.8%, +14.6%], p = 2e-4+2.7% [-1.1%, +6.4%], p = 0.13

The rank test gives p = 9e-5 for each row at 1%, p <= 0.002 at 2% and p >= 0.09 at 20%. Over scratch the raw target gives +18.2% at 1% (0.1967 +- 0.0017 vs 0.1664 +- 0.0061, one-sided Welch p = 8e-8), +16.9% at 2% (0.2212 +- 0.0046 vs 0.1891 +- 0.0058, p = 4e-10) and +7.8% at 20% (0.3019 +- 0.0025 vs 0.2799 +- 0.0086, p = 5e-4), against +7.2%, +6.3% and +5.0% for the latent target. At 1% no fine-tune has an evaluation dip, and every raw-target run is above every latent run. At 2% the raw target does not remove the dips: 4 of its 10 fine-tunes have one, in epochs 20 to 24, against 4 of the 12 latent ones. At 20% it is 2 of 6 for each target. The latent runs are those of Table 1, pretrained two months before the raw-target runs. At 4% a concurrent latent control reproduced that group (p = 0.97).

We therefore no longer attribute the gain to the latent space, and the final version would present the raw target as the recommended variant (point 2 of the general response).

For the named methods: to our knowledge only Forecast-MAE has released code, and only for Argoverse 2. All four use map tokens and per-agent tokens that our map-free, scene-fused encoder does not have. We therefore ported their trajectory objectives onto our encoder, data and schedule:

  • Forecast-MAE: predict the future from the history, or the history from the future
  • Traj-MAE temporal masking and SEPT's masked trajectory modeling: reconstruct half the observed steps from the other half
  • SEPT's tail prediction: forecast the last observed steps

These are pre-registered with n = 6 each, and we will post the results that arrive during the discussion. They are ports, not reproductions, and the paper will say so. DECAMP's combined objective needs both of our losses in one pass and is not included.

Q4 (masking before fusion, joint agent-and-map tokens). We test neither alternative. Sec. 3.1 motivates masking after fusion and now states that the ordering is not ablated, and the paper claims no advantage over either alternative. The new controls vary the target and the window, not the ordering or the token set.

Q5 (standard MTR with map input). Our forecaster at 100% labels reaches 0.3508 mAP, against MTR's published 0.4164 on the validation split (Shi et al., 2022). Our encoder never sees the map, by design: that keeps per-agent and cross-agent computation separable (Sec. 3.1), which the analyses need. Whether the gain survives inside MTR's joint agent-map encoder is open, and the limitations say so. With the map removed from the forecaster as well, pretraining still adds +6.5% (one-sided p = 0.004).

AuthorsOpenReview ↗

Response to Reviewer V1sK (3/3)

Q6 (Table 1 vs Table 2, now Table 4). Table 2's complete-transfer row (0.2147) uses five of Table 1's fourteen encoders, fine-tuned a second time. In Table 1 the same five average 0.2273 (all fourteen: 0.2245), so the gap of 0.0126 lies between two fine-tunes of the same encoders. One encoder accounts for 0.0093 of it. Its two fine-tunes share configuration, checkpoint, seed and evaluation. Their logs are identical through the first training iteration and diverge only through GPU nondeterminism. The Table 2 run hit a transient evaluation collapse that had not recovered by epoch 30, so its peak is the epoch-10 value (0.1822 against 0.2288). Such collapses occur in 12 of 14 runs of both arms at this regime. The revision reports both draws for the five encoders and corrects the within-encoder spread to its pooled value (SD 0.0100, heavy-tailed).

On transferring both stages: the data do not separate them. Paired by encoder, cross-agent only minus complete is +0.0090 with Table 2's draws (p = 0.46) and -0.0037 with Table 1's (p = 0.52). We transfer the complete encoder because that is what Table 1 measures, and we claim nothing for the core beyond it.

Q7 (checkpoint selection). Every Table 1 comparison is now also reported at the fixed final epoch 30, with no selection:

labelsepoch-30 gainp
4%+11.6%0.004
20%+4.4%0.0495
2%+5.5%0.035
1%+7.2%0.005
100%0.0%-

For Table 1 the conclusions are unchanged. The cleanest row is 1%: no fine-tune there has an evaluation dip, the best epoch is the final one in 18 of 21 runs, and the gain is +7.2% at the peak and at epoch 30 and +7.5% over the late window, with minADE, minFDE and MissRate each significant under all three. The raw-target control repeated at 1%, 2% and 20% agrees across the three views in the same way (Q3).

For the new controls of Sec. 6.1 the conclusions are not unchanged: at epoch 30 none of their differences is resolved:

  • raw vs concurrent latent target: -5.7%
  • raw vs scratch: +0.3%, p = 0.49 (first six runs)
  • past-only raw vs scratch: +2.5%, p = 0.42
  • latent vs supervised on degraded data: +10.6%, p = 0.42

The cause is the fine-tune with the map at 4% labels. In 12 of 14 runs of each arm, an evaluation between epochs 20 and 30 falls below 60% of the run's peak, so any single epoch is noisy. The map-free forecaster has no such dips (0 of 28 runs), and there the peak, epoch 30 and late window agree (+6.5%, +7.1%, +7.9%). Appendix B reports every control at all three summaries.

Two corrections to Table 1:

  • The 1% and 2% rows compare 12 pretrained runs against 9 scratch runs, not 12.
  • The late window covers all six epochs 25-30. The submitted text said that two of them were evaluated. The numbers are unchanged.

Q8 (collapse under the main recipe, RankMe).

  • Within the 25 epochs of the main recipe, no: all 14 Table 1 encoders measure effective rank 21.7-29.4, with no degenerate dimension. Trained far beyond that, yes: a 200-epoch run of the final recipe collapsed at epoch 141 (rank 42.5 to 11.3 in one epoch, 1.3-4.6 from epoch 145 on, new in Sec. 6.3). The submission's run collapsed at about epoch 165.
  • Among healthy encoders, rank does not track transfer (Spearman +0.11, p = 0.71).
  • RankMe (Garrido et al., 2023) uses effective rank to select models. We use it only to detect collapse. It is now cited and discussed.
  • The collapse mechanism needs the latent target: the raw-target variant has no teacher whose output can degenerate. Its encoders keep their rank (22.2-23.8), but that was measured after 25 epochs, before any latent run collapsed, so it does not test the difference.

Q9 (Table 6, now Table 11, vs a single run). Against the mean of the 14 base-recipe runs, every variant lies within +0.7 to +1.0 SD and inside the 95% prediction interval of a single run. With one run per variant, they cannot be ranked against the base recipe in either direction, and the caption now says so.

Q10 (frozen-encoder experiments). Removed. They used the earlier, incomplete transfer.

Q11 (code). The code is at https://anonymous.4open.science/r/actor-jepa-CFF7: pretraining, transfer, fine-tuning, the pre-registered protocols, and the per-run results with the scripts that recompute Tables 1-5, 7 and 8. Figure scripts and the remaining appendix analyses are not included. The word "Code" in the abstract links to it, as it did in the submission, and the reproducibility statement now prints the URL.

Q12. Both are fixed. The typo in Sec. 3.2 is corrected, and the opening of Sec. 3.1 is rewritten.

Reviewer_bb5yOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(−1)
Read review
Paper summary

This work leverages JEPA for multi-agent scene encoding, explicitly capturing interactions among agents. It offers useful insights into the importance of transferring the cross-agent stage and the choice of pretraining strategies. In addition, the proposed method achieves performance improvements, particularly in terms of mAP, even with a limited amount of labeled data on Waymo dataset.

Critical strengths
  1. Some good insight on training JEPA for multi-agent trajectory prediction. 2)Independent pretraining runs on the Waymo Open Motion Dataset, followed by fine-tuning with smaller subsets of labeled data, would provide some insights into the effectiveness of the proposed approach.
Critical weaknesses
  1. The authors primarily rely on mAP as the performance metric. However, mAP alone may not adequately reflect trajectory prediction quality, particularly in multi-agent settings where interaction consistency and joint prediction performance are critical.
  2. Evaluating the method with 20% labeled data does not provide sufficiently strong evidence of label efficiency. Experiments under more challenging low-label regimes (e.g., 1%, 5%, and 10%) would better demonstrate the benefits of the proposed pretraining approach.
  3. The insight obtained is largely based on earlier JEPA models and would benefit from further consideration of recent advances in JEPA-based approaches.
Questions for authors
  1. Why do the authors use mAP as the primary evaluation metric for multi-agent trajectory prediction, given that it may not adequately capture interaction consistency and joint prediction quality?
  2. Why not compare with some latest methods, such as Drive-JEPA, for performance comparsion.
  3. Why do the authors only consider 20% labeled data? This setting alone does not provide sufficient insight into the label efficiency of the proposed method, and experiments with smaller proportions of labeled data would offer a more comprehensive evaluation.
3 comments
AuthorsOpenReview ↗

Response to Reviewer bb5y

W1/Q1 (mAP only). We use mAP because it is the headline metric of the WOMD motion-prediction benchmark and of MTR, and it is the pre-registered primary metric of our main comparison. We agree that it is not enough alone. The revision adds minADE, minFDE and MissRate for every Table 1 row (Appendix A).

  • At the best checkpoint, every label-scarce row has lower minADE, minFDE and MissRate when pretrained (minADE -2.0% to -4.1%).
  • The displacement metrics are individually significant at 1% labels, and at epoch 30 for minADE and MissRate at 4%.
  • In the late window at 4% and 2% they are flat or slightly worse, none significantly (minFDE +4.7% and +0.7%).
  • At 100% no metric moves.

mAP moves more than the displacement errors because it also scores the confidence ranking of the modes. Our decoder predicts each agent marginally, and the evaluation code we inherit does not compute the overlap rate (it prints 0), so Appendix A omits it. We do not measure joint consistency, and Appendix J now says so.

W2/Q3 (only 20% labels). The lower fractions are in Table 1. The submitted "20%" was 4% of the training scenarios (point 1 of the general response), and the table now covers 1%, 2%, 4%, 20% and 100% of them. Peak-mAP gain over scratch:

  • 1%: +7.2% (p = 0.005)
  • 2%: +6.3% (p = 0.013)
  • 4%: +7.4% (p = 0.006)
  • 20%: +5.0% (p = 0.026)
  • 100%: -0.7% (p = 0.15)

The gain is present at every scarce fraction and gone with every scenario labeled.

W3/Q2 (recent JEPA work, Drive-JEPA).

  • Drive-JEPA (cited) pretrains a video JEPA for end-to-end planning from camera input. It has no trajectory tokens and no WOMD forecasting head, so it cannot be compared in our pipeline.
  • LeJEPA (Balestriero & LeCun, 2025) replaces EMA, stop-gradient and similar heuristics with an isotropic-Gaussian regularizer (SIGReg) motivated by collapse. Our collapse finding is the failure SIGReg targets. We cite it as the remedy aimed at that failure but did not test it. Our raw-target variant has no teacher whose output can degenerate. Whether it avoids the collapse in long training is not yet measured.
Reviewer_bb5yOpenReview ↗

reply to authors' reply

It is fine to use mAP as a performance metric. My question is whether your main finding, which is based on mAP, would change if you evaluated it using FDE, JFDE, or miss rate.

Results at other label fractions also matter because “this pretraining improves label efficiency” is one of your main claims. I would like to see whether this finding holds across different label fractions. Please also explain how you sampled the data for each fraction, since the sampling strategy matters when using a small portion of the data.

AuthorsOpenReview ↗

Reply to Reviewer bb5y

Thank you for the clarification. We answer the three points in order.

1. FDE and miss rate. Appendix A (Table 5) repeats every row of Table 1 on minADE, minFDE and the miss rate (MissRate in Table 5). The table below gives the difference relative to training from scratch at each run's best checkpoint, which is the epoch of its peak mAP. The two-sided Welch p is in parentheses, and lower is better.

LabelsLatent target, minFDELatent target, miss rateRaw target, minFDERaw target, miss rate
1%-2.3% (0.033)-3.8% (0.005)-4.1% (2e-5)-6.5% (2e-5)
2%-1.4% (0.19)-2.5% (0.051)-2.9% (0.007)-4.4% (3e-4)
4%-0.8% (0.33)-2.1% (0.071)-1.6% (0.045)-3.4% (0.004)
20%-1.4% (0.13)-2.0% (0.14)-1.6% (0.062)-3.3% (0.016)
100%+0.4% (0.41)+0.3% (0.64)+0.8% (0.044)+1.0% (0.12)

The latent-target columns are in Table 5. The raw-target columns are new and are not in the PDF. They use the runs of Tables 3 and 8 and the definitions of Table 5. This analysis was not registered, and the p-values are not corrected for the number of tests.

The finding keeps its direction on both metrics. At the best checkpoint, pretraining lowers minFDE and the miss rate at every fraction below 100%, and at 100% it does not. The effect is smaller than on mAP, because mAP also scores the confidence ranking of the modes. In this table the latent target is individually significant at 1% labels only. With the raw target the gain is clear at 1% and 2% labels, and it is there at epoch 30 as well (minFDE -3.9% and -2.9%, miss rate -6.1% and -4.4%, p <= 8e-4). Over epochs 25-30 it holds at 1% (-4.8% and -7.2%, p <= 3e-6) and is not significant at 2% (-1.7% and -2.4%, p = 0.19 and 0.14).

At 4% labels the two targets differ away from the best checkpoint. With the latent target, epoch 30 points the same way as the best checkpoint: minFDE is at -7.7% (p = 0.066) and the miss rate at -9.5% (p = 0.038). Over epochs 25-30 they are at +4.7% and +0.8% (p = 0.33 and 0.84). With the raw target the gain holds at the best checkpoint only, on these metrics as on mAP. At epoch 30 minFDE and the miss rate are at +14.3% (p = 0.16) and +10.8% (p = 0.21), and over epochs 25-30 they are at +12.1% (p = 0.022) and +7.3% (p = 0.075). We attribute this to the evaluation dips late in the fine-tuning schedule at this fraction (Appendix A). Each of the ten raw-target fine-tunes has at least one, and seven have one within epochs 25 to 30. Of the 14 from-scratch fine-tunes, 12 have at least one, and three have one within epochs 25 to 30. The counts for the raw target and for epochs 25 to 30 are not in the PDF.

We do not report a joint FDE. The decoder predicts each agent marginally, and the evaluation scores the target agents one by one. We have not built joint predictions from the marginal modes. Appendix J names this limitation.

2. Other label fractions. Table 1 covers 1%, 2%, 4%, 20% and 100% of the training samples, which the paper calls scenarios. The gain in peak mAP over scratch is +7.2%, +6.3%, +7.4%, +5.0% and -0.7% (p = 0.005, 0.013, 0.006, 0.026 and 0.15). For the raw target, Table 8 gives +18.2%, +16.9%, +7.8% and -1.3% at 1%, 2%, 20% and 100%, and Table 3 gives +13.5% at 4%. With both targets the gain is present at every fraction below 100% and absent at 100%.

3. How the fractions are sampled. Sec. 4 and Appendix I describe it.

  • A training sample is a recorded WOMD scene centered on one agent to predict. The training split has 2,177,854 samples from 486,995 scenes.
  • A label fraction keeps every k-th sample of the training list in its fixed order, with k = 100, 50, 25, 5 and 1. There is no random draw and no stratification.
  • Every run of every arm at a fraction therefore uses the same labeled samples, and each smaller set is a subset of the larger ones. The spread across runs comes from the fine-tuning seed in the from-scratch arm and from the pretraining run in the pretrained arms. We did not draw the labeled set more than once, so our intervals do not include that source of variation.
  • The samples of one scene are adjacent in the list. The labeled samples therefore come from more scenes than their share suggests: 4.5%, 8.9%, 17.9% and 76.2% of the scenes at 1%, 2%, 4% and 20% labels (Appendix I).
  • In the runs of the table above, pretraining is not subsampled. It uses every training sample, including the other samples of the labeled scenes. Evaluation always uses the full validation split.
Reviewer_8rDqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ActorJEPA for multi-agent forecasting and asks if self-supervised pre-training can help in motion forecasting. The agent encoder training is a two-stage process. First, a cross-agent interaction lets all agents share information at each timestep, followed by a per-agent core that processes each agent's sequences across time. The encoder is then plugged into an MTR decoder and fine-tuned on WOMD. Across four label regimes and 14 independent pretraining runs, this improves peak mAP by 5 to 7% over the same model trained from scratch.

Critical strengths
  1. The paper is generally well-written and easy to follow.
  2. The experimental rigor is good, and both significant and non-significant results are reported properly. The paper is transparent about the limitations in experiments.
  3. The collapse analysis is useful, and the paper also explains why the VICReg variance term is flat. One hinge is saturated, and the other is zero under a low-rank collapse. This is a practical warning for anyone pretraining trajectory encoders with JEPA-style objectives.
Critical weaknesses
  1. The proposed design does not appear to be fully isolated, as the experiments compare training from scratch vs the complete pre-training recipe, but we are not sure if the gains come from alternative self-supervised objectives, masking before fusion, or joint agent-and-map encoding.
  2. Pretraining uses all training scenarios and their complete trajectories, whereas downstream supervision uses sampled subsets. This is mentioned in the paper, but it complicates claims about annotation savings because the pretraining inputs already contain future trajectory coordinates. A supervised baseline using all available scenarios would help establish the practical value of this training strategy.
  3. The results use one modified forecasting architecture. Evaluation with a stronger standard configuration, additional trajectory-quality metrics, and replicated pretraining baselines would strengthen the contribution.
Questions for authors
  1. Since pretraining uses complete trajectories from all training scenarios, can you include supervised training or supervised pretraining on those same scenarios? This would clarify the practical benefit beyond the sampled-label comparison.
  2. Appendix B states that targets are sampled without temporal bias, while Appendix D describes a 3× bias toward future targets. I am not sure which one is correct.
  3. (Clarification) Figure 2 (right) does not match Table 2. The figure shows interaction-only transfer below the from-scratch model (about 0.2), while Table 2 reports 0.2237 (+7.1%).
  4. (Clarification) I could not find the "frozen encoder experiments" mentioned in Appendix C.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 8rDq (1/2)

W1 (what the gain comes from). The submission did not isolate this. The revision adds pre-registered controls, each replicated over independent pretraining runs (Sec. 6.1, Table 3, Appendix B).

  • The objective. We changed only the target, from the teacher's latent to raw-state statistics of the masked steps. Encoder, masking, predictor, data and schedule stayed fixed. The raw target transfers at least as well at 4% labels (+5.8% at the best checkpoint over a latent control run in the same week, one-sided Welch p = 0.015, rank test p = 0.061) and better at 1% and 2% (+10.3% and +10.0% at the best checkpoint, +9.5% and +10.2% at the final epoch, p <= 3e-4 for each, Appendix B, Table 8). At 20% the two targets are not resolved (+2.7% at the best checkpoint, p = 0.10), and the raw target gives +7.8% over scratch (p = 5e-4). These three fractions are the complete part of a pre-registered follow-up that is still running. The gain does not come from the latent objective, and the paper no longer attributes it to one.
  • Other objectives. We ported the trajectory objectives of Forecast-MAE, Traj-MAE and SEPT onto the same encoder, data and schedule. These runs are pre-registered with n = 6 each and in progress. We will post the results that arrive during the discussion.
  • The stage of the encoder. Transferring the cross-agent stage alone gives +7.1% (p = 0.011), and the per-agent core alone -1.6% (Table 2 in the submission, Table 4 now).
  • Not tested. We do not compare against masking before fusion or against a joint agent-and-map token set. Sec. 3.1 motivates the ordering and now states that it is not ablated, and the paper claims no advantage over either alternative.

W2/Q1 (supervision on the pretraining scenarios). We agree. The revision claims a gain only from scenarios beyond the labeled set, and concedes that supervision on their futures does as well.

  • Supervised training on all scenarios is the new 100% row of Table 1. The submission's "100%" was the standard 20% subset (point 1 of the general response). Trained from scratch on every scenario, the model reaches 0.3508, far above any model fine-tuned on 4% (0.2245 with pretraining). Pretraining adds nothing there: -0.7%, 95% CI [-1.9%, +0.5%] (n = 3 vs 3).
  • Supervised pretraining on the same scenarios. We ran it on the degraded, map-free pool of Sec. 5.1 (the 96% of training samples outside the labeled set), where both arms see the same trajectories. Supervising the encoder on the 8 s futures, then fine-tuning identically at 4% labels, matches the pretext task and is numerically ahead: 0.2411 +- 0.0133 vs 0.2304 +- 0.0095 (n = 8 vs 12, Welch p = 0.076, rank test p = 0.070). We did not run it on clean WOMD, and we claim no advantage over labels that exist.
  • Future coordinates are not needed as pretraining input. Pretraining on the 11 observed steps alone, with the raw target, gives +14.7% over scratch (n = 6 vs 14, Welch p = 2e-7, rank test p = 3e-5).

The practical case that remains is narrower than the submission's. A forecaster trained on few curated scenarios gains from logs of the same distribution degraded to simulate uncurated ones, with no map and tracker-quality noise: +10.3% at 4% labels (one-sided p = 1e-5). It gains by masked prediction or by supervision on the logs' own futures. What supervision cannot use is a scene without its future.

W3.

  • Standard configuration. Our forecaster at 100% labels reaches 0.3508 mAP, against MTR's published 0.4164 on the validation split (Shi et al., 2022). Our encoder never sees the map, by design: that keeps per-agent and cross-agent computation separable, which the analyses need. Whether the gain survives inside MTR's joint agent-and-map encoder is open, and the limitations say so. With the map removed from the forecaster as well, pretraining still adds +6.5% (one-sided p = 0.004).
  • Trajectory metrics. The new Appendix A reports minADE, minFDE and MissRate for every Table 1 row, at the best checkpoint, at the fixed final epoch and over epochs 25-30. At the best checkpoint every label-scarce row improves on all three (minADE -2.0% to -4.1%), and the differences are individually significant at 1% labels.
  • Replicated pretraining baselines. Every pre-registered arm of Sec. 6.1 is replicated over independent pretraining runs (n = 6 to 12). That includes a concurrent rerun of the latent baseline, which reproduces Table 1 (n = 10, -0.1%, p = 0.97). The ported objectives above have n = 6 each.
AuthorsOpenReview ↗

Response to Reviewer 8rDq (2/2)

Q2 (temporal bias). Appendix B of the submission is correct (Appendix D now). In every reported run the two target sets are drawn uniformly over the 91 steps. The implementation has a future-bias parameter, but only a contiguous-block masking strategy reads it, and the reported recipe does not use that strategy. The sentence in the variants appendix described this unused setting. It is corrected (Appendix F now).

Q3 (Figure 2 right vs Table 2). You read the panel correctly, and the table is the correct one. We ran this experiment three times, and the submission mixed the versions:

peak mAPfrom scratchcore onlycross-agent onlyboth stages
submitted Figure 2 (first version)0.20890.19910.20190.2245
second version (one encoder)0.20890.20970.22320.2379
Table 2 (five encoders)0.20890.20560.22370.2147

In the first version the two partial arms were split from a checkpoint that had already collapsed (Sec. 6.3), so they measured a broken encoder, and the bar for both stages was the Table 1 mean. We repeated it with all arms split from one healthy encoder, and then from five. Table 2 and the text reported the five-encoder version. The figure still showed the first version, and the captions of the figure and of the table still described the second, without its numbers. The revision redraws the panel from the table (Table 4 now) and corrects both captions. The single-encoder 0.2379 for complete transfer did not survive four more encoders. A second fine-tune of the five complete-transfer encoders gives 0.2273, and Sec. 6.2 reports both.

Q4 (frozen-encoder experiments). Removed. The sentence was a remnant of experiments that used an earlier, incomplete transfer and are not in the paper.

Reviewer_8rDqOpenReview ↗

Thank you for your rebuttal

The rebuttal addresses most of my concerns, and I appreciate the new experiments testing which parts of the method are needed for the gains. However, by running these experiments, we do have some more insight (as highlighted by the authors in Sec 6.1 and Table 8) into whether the latent target itself helped. From Table 8, raw vs. latent at 1% and 2%, it seems it does not matter as much. This narrows the case for latent prediction as the main methodological contribution, and hence I will keep my score.

AuthorsOpenReview ↗

Reply to Reviewer 8rDq

Thank you for reading the rebuttal and the new experiments.

We agree with your conclusion, and Table 8 supports a stronger form of it. At 1% and 2% labels the raw target does not merely match the latent target. It is ahead by 10.3% and 10.0% at the best checkpoint and by 9.5% and 10.2% at epoch 30 (p <= 3e-4). At 4% labels it is ahead at the best checkpoint only (+5.8%, one-sided p = 0.015, Table 7). At 20% and 100% labels the difference between the two targets is not resolved.

The revised paper therefore does not claim the latent target as the contribution, although its title and framing are still built around it. The abstract and Sec. 6.1 state that the gain does not need the latent target. The contribution list says that what we propose is the factorization of the encoder and where it places the target, not the latent space. The order of fusion and masking in that factorization is not ablated, as our first response to you says. As our general response says, the title cannot change during the discussion.

The raw-target runs at 100% labels that were still in progress have finished. The registered prediction of no gain holds, and the registered two-sided test finds a deficit of 1.3% against scratch at the best checkpoint. We report the result in a separate closing comment.

Comments on the paper

4 comments

Thank you for the careful reviews. The revised PDF differs from the submission in more than the points raised. We list the changes of substance, and the PDF diff shows the rest.

  1. Label fractions now count all training scenarios, and the full-label claim is withdrawn. The submission measured fractions against the standard MTR 20% subset, as its Sec. 4 stated. Its "100%" row was therefore 20% of the training scenarios, "20%" was 4%, "10%" was 2% and "5%" was 1%, and the abstract's "+5.0% at full labels" was a 20% result. We have since run the true 100%. Pretraining gives nothing there: -0.7%, 95% CI [-1.9%, +0.5%] (n = 3 vs 3). Table 1 now reports 1%, 2%, 4%, 20% and 100%, and the paper claims label efficiency only. A scenario throughout is one training sample: a WOMD scene centered on one agent to predict.
  2. What the gain needs (new Sec. 6.1, Table 3, Appendix B). Following V1sK, we pre-registered five controls, each with independent pretraining runs. Each protocol was fixed before its runs and is in the repository (Appendix B). The controls also answer 8rDq's W1 and W2.
    • The latent target is not needed. Predicting raw-state statistics of the masked steps does at least as well at the best checkpoint at 4% labels: +5.8% over a latent control run in the same week (one-sided Welch p = 0.015, rank test p = 0.061). At 1% labels, where no fine-tune has evaluation dips, it does better at the best checkpoint, over the late window and at the final epoch: +10.3%, +11.8% and +9.5% (Welch p <= 2e-4). At 2% labels, where fine-tunes of both targets have dips, it is +10.0%, +10.3% and +10.2% (Welch p <= 4e-4). At 20% the targets are not resolved (+2.7%, p = 0.10). These fractions belong to a pre-registered follow-up that is still running (Appendix B, Table 8). Table 1 still reports the latent target, and the final version would present the raw target as the recommended variant.
    • With that target, the scene's future is not needed. Pretraining on the 11 observed steps alone gives +14.7% over scratch (Welch p = 2e-7, rank test p = 3e-5).
    • Where futures exist, supervision does as well. Supervised pretraining on the same degraded trajectories matches the pretext task and is numerically ahead (0.2411 vs 0.2304, Welch p = 0.076, rank test p = 0.070). We concede V1sK's W1 and 8rDq's W2 here.
    • At 4% labels these differences hold at the best checkpoint. At the fixed final epoch none of them is resolved there (V1sK Q7).
  3. Cheap pretraining data (new Sec. 5.1, Appendix I). Two pre-registered experiments not in the submission:
    • Pretraining on map-free data with simulated tracker noise, with no sample in common with the labeled set (the scenes overlap, Appendix I), adds +10.3% at 4% labels (one-sided p = 1e-5).
    • A forecaster with no map at all gains +6.5% (one-sided p = 0.004).
  4. More metrics (new Appendix A, for bb5y W1 and 8rDq W3). We add minADE, minFDE and MissRate for every Table 1 row, at the peak epoch, at the fixed epoch 30 and over epochs 25-30.
  5. Corrections.
    • The 1% and 2% rows compare 12 pretrained runs with 9 scratch runs, not 12.
    • The late window covers all six epochs (text corrected, numbers unchanged).
    • The within-encoder spread is now the pooled SD over all replicate sets (0.0100, heavy-tailed).
    • The factorial (Table 2 in the submission, Table 4 now): Sec. 6.2 reports both fine-tunes of the complete-transfer encoders. Figure 2 (right) is redrawn from the table. The submitted panel showed an earlier version of the experiment (8rDq Q3).
    • The variants table (Table 6 in the submission, Table 11 now) is compared with the mean of the 14 base runs. Its appendix no longer says that the base sampler favors future targets. Targets are drawn uniformly (8rDq Q2).
    • The frozen-encoder remnant is removed (V1sK Q10, 8rDq Q4).
    • RankMe, LeJEPA, Traj-MAE and SEPT are cited.
    • New in Sec. 6.3: the rank-based stopping rule, tested on a 200-epoch run.
    • Argoverse 2 (Sec. 5.2 now): the full-label test has twelve runs per arm (-0.5%, p = 0.44), where the submission had six (+0.6%, p = 0.50). The matched fractions 20% and 5% are new.
  6. Claims changed accordingly. The contribution is the encoder, which mixes agents before masking, and pretraining on scenarios beyond the labeled set, which masked prediction provides without labels. The latent target is one valid choice. The title cannot change during the discussion. We would align it in a camera-ready version.
  7. Moved to the appendix to stay within 10 pages: the representational-similarity analysis (Appendix H), the checkpoint-selection table (Appendix G) and one related-work paragraph (Appendix E).

Further pre-registered runs are in progress. We will post those that finish during the discussion. They cover:

  • the raw target at 100% labels
  • tail forecasting on the history
  • map-free ports of the Forecast-MAE, Traj-MAE and SEPT objectives
AuthorsOpenReview ↗

Follow-up result: raw target against supervision on the degraded pool

One of our pre-registered follow-up experiments has finished. It answers V1sK W1 and 8rDq W2 for the raw target.

Design. The supervised-pretraining control of Sec. 6.1 compared supervision with the latent target. This experiment repeats it with the raw target. The pretraining data are the same degraded, map-free pool (Sec. 5.1), the fine-tune at 4% labels is the same, and each of the eight runs is an independent pretraining. It belongs to the follow-up protocol named in Appendix B, which was fixed before any of its runs started.

pretraining on the degraded poolnpeak mAPepochs 25-30epoch 30
none (from scratch)140.2089 +- 0.01110.18830.1982
supervised on the 8 s futures80.2411 +- 0.01330.18060.1910
masked prediction, latent target120.2304 +- 0.00950.20220.2113
masked prediction, raw target (new)80.2402 +- 0.00720.20640.2212

Registered tests.

  • Primary, raw target vs supervision at the best checkpoint (two-sided): -0.4%, 95% CI [-5.3%, +4.5%], Welch p = 0.87, rank test p = 0.65. Over epochs 25-30 the raw target is +14.3% (p = 0.086), and at epoch 30 it is +15.8% (p = 0.25).
  • Raw vs latent target on the same pool (two-sided): +4.2% at the best checkpoint, 95% CI [+0.8%, +7.7%], Welch p = 0.019, rank test p = 0.023. The difference has the same sign at epoch 30 (+4.7%, p = 0.38).
  • Raw target vs scratch (one-sided): +14.9% at the best checkpoint (Welch p = 7e-8, rank test p = 1e-5), +9.6% over epochs 25-30 (p = 0.013) and +11.6% at epoch 30 (p = 0.024, registered as exploratory).

Reading. The primary test is not significant. The reading registered for that outcome is that masked raw-target pretraining and supervised pretraining on the same degraded logs are not distinguishable, and the interval limits a difference at the best checkpoint to about 5% in either direction. Our concession stands: where the logs carry futures, supervision on them does as well as masked prediction. Two things change. The numerical lead that supervision had over the latent target (0.2411 vs 0.2304, p = 0.076) is absent with the raw target. And with the raw target the gain from degraded logs over scratch holds over the late window and at the fixed final epoch, not only at the best checkpoint.

Limits. One label fraction (4%), one fine-tuning seed and eight runs in each arm of the primary test. The comparison groups were run earlier than the new arm. One exploratory count that was not registered: of the 15 evaluations of a fine-tune, 3.2 on average fall below 0.12 mAP with the raw target, against 6.6 with supervision (rank test p = 0.003), 2.1 with the latent target and 2.4 from scratch.

Peak mAP of the eight new runs: 0.2265, 0.2346, 0.2375, 0.2449, 0.2479, 0.2401, 0.2420, 0.2477.

The revised PDF reports this experiment in Appendix B (Table 8).

AuthorsOpenReview ↗

Follow-up results: ported objectives and tail forecasting

Two pre-registered follow-up experiments have finished: the comparison with ported pretraining objectives (V1sK Q3, 8rDq W1) and the tail-forecasting control (V1sK Q2). Both belong to the follow-up protocol named in Appendix B, which was fixed before any of its runs started. Every run is an independent pretraining, fine-tuned at 4% labels.

Design. We ported three published trajectory objectives onto our encoder, data and schedule, each predicting raw states:

  • Forecast-MAE: the future from the history or, with equal probability per scene, the history from the future
  • Traj-MAE temporal masking and SEPT's masked trajectory modeling (MTM): six of the 11 observed steps from the other five
  • SEPT's tail prediction: the last two observed steps from the nine before them

These are map-free, scene-level ports, not reproductions. The published methods use map tokens and per-agent tokens that our encoder does not have.

pretrainingtargetnpeak mAPepochs 25-30epoch 30
none (from scratch)140.2089 +- 0.01110.18830.1982
ours (Table 1 and its concurrent rerun)latent240.2244 +- 0.01550.19690.2183
ours, random masking (Sec. 6.1)raw100.2372 +- 0.00460.19540.2018
ours, observed steps only (Sec. 6.1)raw60.2397 +- 0.00590.20190.2032
Forecast-MAE portraw60.2383 +- 0.00330.16450.2327
MTM portraw60.2426 +- 0.00270.18460.2190
tail-prediction portraw60.2383 +- 0.00520.19610.2235

Registered tests.

  • Primary, our latent-target method against each port (two-sided Welch, Holm over the three). At the best checkpoint it is below every port: -5.9% [-8.8%, -2.9%] against Forecast-MAE, -7.5% [-10.4%, -4.7%] against MTM and -5.8% [-9.1%, -2.6%] against tail prediction (Holm p = 7e-4, 3e-5 and 0.001, rank test p = 0.09, 0.004 and 0.09).
  • Over epochs 25-30 our method is ahead of the Forecast-MAE port (+19.7%, p = 0.04) and not resolved against the other two (p >= 0.32). At epoch 30 no difference is resolved (p >= 0.07).
  • Same raw target, different masking, best checkpoint (two-sided): Forecast-MAE port vs our random masking +0.5% [-1.3%, +2.3%], p = 0.56. MTM port vs random masking on the observed steps +1.2% [-1.4%, +3.8%], p = 0.31. Tail prediction vs the same -0.6% [-3.6%, +2.4%], p = 0.66.
  • Each port over scratch at the best checkpoint (one-sided): +14.1%, +16.1% and +14.1% (p <= 2e-7). Over epochs 25-30 none is resolved.

Reading.

  • The reading registered for a port above our method is that its objective is the better pretext task in this pipeline. That holds for all three at the best checkpoint, and we claim no advantage of our objective over those of Forecast-MAE, Traj-MAE or SEPT.
  • What separates the ports from the submitted method is the target, not what is masked. With the raw target held fixed, none of the three masking schemes differs measurably from ours. This is the result of Sec. 6.1, that the raw target transfers at least as well as the latent one, seen through three more objectives.
  • Tail forecasting settles the caveat under V1sK Q2. The past-only gain is reachable by forecasting the last two observed steps (0.2 s at 10 Hz) and does not depend on random masking, so the pretext task on histories is short-horizon prediction. Our sentence "what supervision cannot use is a scene without its future" therefore holds for supervision in the benchmark's 8 s label format. We claim no advantage of masking over a forecaster with a 0.2 s horizon trained on the same histories.

Limits. The advantage of the ports is one of the best checkpoint. Every port fine-tune has an evaluation below 60% of its peak between epochs 20 and 30 (18 of 24 for our method), and its best epoch is one of the last two in 17 of 18 runs. These counts were not registered. The experiment has one label fraction (4%), one fine-tuning seed and six runs per port. The comparison groups were run earlier, between a few days and two months before the ports. DECAMP is not ported.

Changes in the PDF since November 13. Appendix B has two new paragraphs, Table 8 has nine new rows, and Sec. 6.1 has one new and one changed sentence. We also edited the language throughout. Long sentences are split and incomplete ones are completed, with no change to any number or claim. The per-run values and the registered result files are in the anonymous repository.

AuthorsOpenReview ↗

Closing note: raw target at 100% labels

The last experiment of the follow-up protocol has finished: the raw target at 100% labels, which our general response listed as in progress. The label-fraction experiment of Appendix B is now complete.

Result. The registered prediction was no gain. It holds, and the registered two-sided test finds a deficit. With every scenario labeled, the raw target is 1.3% below scratch at the best checkpoint (95% CI [-1.7%, -0.8%], Welch p = 0.0015). Every raw-target run is below every from-scratch run, and with three runs per arm the rank test is at its floor of p = 0.10. The difference is -1.2% over epochs 25-30 (p = 0.028) and -0.7% at epoch 30 (p = 0.22). With three runs per arm we report this deficit as measured and draw no further conclusion from it. Against the latent target the raw target is 0.6% lower at the best checkpoint (95% CI [-1.8%, +0.6%], p = 0.21), and this difference is not resolved. The estimates at 1%, 2% and 20% labels do not change in the final analysis. At the best checkpoint both targets gain at 1% to 20% labels, and neither gains at 100%. Sec. 6.1 states the result, and Appendix B and Table 8 give the per-arm values and the limits of this comparison.

A qualification of our previous comment. We wrote there that, with the raw target held fixed, none of the three masking schemes differs measurably from ours. That holds at the best checkpoint only. Away from it the Forecast-MAE port differs from our random masking in both directions: it is 15.8% lower over epochs 25-30 (p = 0.047) and 15.3% higher at epoch 30 (p = 0.021). Appendix B now says so.

Other changes in the PDF since November 16.

  • Appendix D.1 corrects the description of the masking. The initial context is always 80% of the timesteps, the two target sets are drawn independently, and the target ratio is redrawn for every batch from the same range throughout training.
  • Sec. 3.1 and the caption of Figure 1 now state that the cross-agent stage averages over the agents. The abstract and the contribution list therefore say that the pretext task masks timestep positions, where they previously said agent-timestep positions.
  • The abstract and the conclusion no longer say that the gain needs scenarios beyond the labeled set. They say that we observe it only with such scenarios. The reason is the control of Sec. 6.1 that confines pretraining to the labeled 4%. At the best checkpoint its three runs are 1.6% above scratch (p = 0.62) and 5.5% below full-split pretraining (p = 0.13), and they resolve neither difference.
  • Sec. 6.3 says that we expect no collapse without a latent target and that this is untested.
  • Appendix B names DECAMP as not ported and gives the reason.
  • The remaining changes are smaller corrections of statements and wording, and none changes a result.

The per-run values and the registered result files are in the anonymous repository. We have updated the abstract field of the forum to the revised abstract.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.