acceptodds
Under review as a conference paper at ICLR 2027

NFL-World: A Multimodal World Model for Understanding American Football Plays

Abstract

We propose NFL-World, a multimodal world model for American football that learns compact, predictive game-state representations from synchronized video and player-tracking data. NFL-World compresses each modality into latent tokens and combines them through a causal Transformer that autoregressively predicts subsequent modality tokens, capturing both player interactions and evolving play dynamics. We first learn modality-specific representations and then train predictive fusion, allowing the model to build on informative states from both streams. Experiments show that NFL-World outperforms evaluated baselines in overall play classification, next-second player-motion forecasting, and similar-play retrieval. Further analyzes show that video complements tracking to improve play understanding and tactical retrieval, while compression substantially reduces the fusion token budget and preserves or improves play understanding performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World models, Self-supervised learning, Multimodal representation learning, Latent prediction, Token compression, Sports analytics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Bnd5OpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes NFL-World, a multimodal world model that integrates synchronized American football video and player-tracking data. It combines modality-specific representation learning, token compression, and causal Transformer-based predictive fusion to learn game-state representations. Experiments demonstrate improvements in play classification, similar-play retrieval, and next-second player-motion prediction.

Critical strengths

S1: Addresses an interesting multimodal sports-understanding problem using real-world NFL data.

S2: Evaluates three downstream tasks with multiple baselines, ablations, and scaling analyses.

S3: Token compression substantially reduces fusion computation while maintaining competitive performance.

Critical weaknesses

W1: The evaluation focuses on representation-based classification, retrieval, and one-second forecasting, without demonstrating longer-horizon simulation, counterfactual reasoning, or planning capabilities.

W2: Experiments cover only the 2025 NFL season, without cross-season generalization or independent expert validation of the annotation-derived retrieval relevance.

W3: NFL-World fine-tunes its fusion components for classification, whereas several competing pretrained models are evaluated using frozen representations, making it difficult to isolate architectural improvements.

Questions for authors

See the above

2 comments
AuthorsOpenReview ↗

Response to Reviewer Bnd5

We thank the reviewer for the supportive review and for recognizing the merits of our work. We answer each point in the order raised. We have uploaded a revised paper; all changes are marked in blue.

W1. No longer-horizon simulation, counterfactual reasoning or planning

  1. NFL-World can simulate longer horizons. We added autoregressive rollouts to 5 seconds, in which the model is fed its own predicted tracking tokens and never the true future. At 3 and 4 seconds it outperforms BDB and STT, which are trained for those horizons. Results are in Table 1.
  2. We do not claim counterfactual reasoning or planning. We define a world model as a model that learns a compact state of its environment and predicts how that state evolves (Ha & Schmidhuber, 2018; LeCun, 2022; Assran et al., 2025), and football has no meaningful per-second action labels to intervene on. We agree with the reviewer that both are promising directions, and we plan to pursue them in future work.

Table 1. Forecasting error at 2 to 5 seconds (TEST RMSE in yards).

Model2 s3 s4 s5 s
NFL-World (rolled out)0.8541.9433.1434.272
BDB (trained for 4.8 s)0.8301.9553.223—
STT (trained for 4.8 s)1.1792.3713.603—
Cosmos-Predict2.5-2B (head per future second)1.3852.7584.018—
Motus (head per future second)1.3932.7734.039—
Constant velocity1.6403.3444.9546.172

W2. A single season, and no expert validation of retrieval relevance

  1. Within the season our evaluation is fully held out, and football changes little from season to season. The 14 test games share no plays with any training stage. The 2025 season is the only synchronized video-and-tracking data available to us so far, and we expect to obtain more.
  2. The retrieval rubric is built from human-labeled play charting. We agree that more human labels or expert rankings would make it stronger, and we leave this to future work.

W3. Fine-tuned NFL-World against frozen baselines

In classification, NFL-World still scores highest when it is completely frozen and read by exactly the same probe as the baselines. We froze the NFL-World model entirely and trained the same attentive probe used for BDB, with the identical recipe, over 5 random seeds. Results are in Table 2.

Table 2. Classification with the same probe (TEST 11-task weighted F1).

ModelProtocolF1
NFL-Worldfrozen + probe, 5 seeds82.24 ± 0.37
BDBfrozen + probe, 5 seeds81.72 ± 0.42
STTtrained end to end on the labels78.46
V-JEPA 2.1 (best video baseline)frozen + probe70.36

We thank the reviewer again for the encouraging assessment.

I am familiar with the research direction, and the authors have addressed my concerns. Therefore, I maintain my score.

Reviewer_8zCbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

NFL-World learns representations of American-football plays from synchronized player tracking and video. In Stage 1, each modality learns to predict its next 1-second window in latent space (JEPA-style, EMA targets). The video encoder is a frozen, football-adapted V-JEPA 2.1, and learned queries compress each modality to a few tokens. In Stage 2, a causal transformer over interleaved tracking and video tokens predicts the current video tokens from tracking and the next tracking tokens from video. The model is evaluated on play classification tasks, similar-play retrieval and next-second player motion, using 2025 NFL data (270 games), and is best on all three.

Critical strengths
  • Combining video with tracking, and using compression to keep only the useful visual information, is an interesting and well-motivated direction.
  • Joint tracking-video modelling is underexplored for multi-agent sports. The case studies (Fig. 4) show concrete situations where video resolves ambiguities in tracking (contested catches, run–pass options).
  • The appendix is very detailed (baselines, readers, retrieval rubric, compute accounting).
Critical weaknesses
  1. The main claims are not supported.
  • The model is not validated as a world model: there are no rollouts or counterfactuals, and the only future test (next-second positions) is better without video.
  • The video matters claim is weak: compared with the tracking-only model (the tracking donor), video adds +0.73 F1 in classification, with several tasks getting worse. Forecasting also gets worse (0.1685 vs 0.1612 yards).
  • Compression improves quality only for video on play-level tasks. Compressing tracking hurts forecasting.
  1. The evaluation is neither fair nor robust.
  • Classification. NFL-World is fine-tuned with labels, while the baselines are frozen probes. The tracking donor already beats the strongest baseline BDB, so the gain comes mostly from tracking and fine-tuning, not from video.
  • Retrieval. NFL-World pooled cosine (frozen features) variant alone loses to Chalkboarding. The win comes from a reader trained on the same rubric used for evaluation, which external baselines never get.
  • Forecasting. BDB and STT are trained for 4.8s and evaluated on their first second, while NFL-World reader is built for 1s. Also lack of comparison with specialized SOTA architectures on public datasets e.g., Sports-Traj (Xu & Fu, 2025).
  • Noise: there are no seeds or confidence intervals for classification. Appendix Tables B2–B3 show changes of up to about 1 point from small recipe changes, often with opposite signs on validation and test, which is the same size as the claimed gains.
  • Staged training: the from-scratch comparison changes several things at once (it is labelled in Fig. 3a but only defined in Appendix)
  • Private dataset: results can’t be reproduced.

Minor:

  • Related work misses multi-agent trajectory modelling and sports understanding models, e.g., JointDiff (Capellera et al., 2026), and SoccerMaster (Yang et al., 2026).
Questions for authors

Q1. Can you report classification results over several seeds, with confidence intervals? Q2. Can you compare under the same protocol: NFL-World frozen with the same probe as the baselines, or the baselines fine-tuned with the same reader? Q3. Can you apply the same retrieval reader to baseline features (raw tracking, BDB)? Q4. How do BDB and STT perform when trained for a 1-second horizon? Q5. Beyond next-second positions, can the model roll out longer horizons? If not, why call it a world model?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 8zCb (1/3): the main claims (Weakness 1)

We thank the reviewer for the detailed review and for recognizing the merits of our work. We quote each bullet point and answer it in the order raised. We have uploaded a revised paper; all changes are marked in blue.

Weakness 1: "The main claims are not supported"

The comparisons below are run over 5 random seeds, and confidence intervals are 95% intervals from a paired bootstrap over the 14 test games.

1.1 World model

"The model is not validated as a world model: there are no rollouts or counterfactuals, and the only future test (next-second positions) is better without video."

  1. We define a world model as a model that learns a compact state of its environment and predicts how that state evolves, following Ha & Schmidhuber (2018), LeCun (2022) and Assran et al. (2025). NFL-World fits this definition exactly.
  2. NFL-World has rollout capability, even if rollout is taken as a requirement for a world model. We added autoregressive rollouts to 5 seconds: the model's predicted tracking tokens are fed back as input, with no future video, and each step is decoded with the same 1-second reader. Results are in Table 1.
  3. We do not frame a world model as requiring action conditioning or counterfactuals, and football has no meaningful per-second action labels.

Table 1. Forecasting error at 2 to 5 seconds (TEST RMSE in yards).

Model2 s3 s4 s5 s
NFL-World (rolled out)0.8541.9433.1434.272
Tracking-only model (rolled out)0.8541.9913.2604.477
BDB (trained for 4.8 s)0.8301.9553.223—
STT (trained for 4.8 s)1.1792.3713.603—
Sports-Traj (trained on our data)2.2503.7835.051—
Cosmos-Predict2.5-2B (head per future second)1.3852.7584.018—
Motus (head per future second)1.3932.7734.039—
Constant velocity1.6403.3444.9546.172

1.2 Video

"The video matters claim is weak: compared with the tracking-only model (the tracking donor), video adds +0.73 F1 in classification, with several tasks getting worse. Forecasting also gets worse (0.1685 vs 0.1612 yards)."

(1) Video provides supplementary information that tracking alone cannot see, and NFL-World's fusion does exactly this. Specifically, classification with fusion improves on tasks such as run-pass option and contested catch, which rely on visual evidence, while the small decreases are mostly on tasks determined by player geometry. The largest per-task gains and the overall gain are in Table 2; the full per-task table is in the appendix of the revision.

Table 2. Classification gain from video (TEST F1, 5-seed mean).

TaskNFL-WorldTracking-onlyGain from video
Run-pass option81.277.0+4.2
Contested catch62.659.0+3.6
Penalty50.449.4+1.0
QB outside the pocket89.288.5+0.7
All 11 tasks82.24 ± 0.3781.86 ± 0.23+0.38

(2) Fusion improves long-horizon forecasting. Rolled out, NFL-World beats its tracking-only model at every horizon after 2 seconds, and the 5-second gap is significant, while the two are nearly identical at one second (Tables 1 and 3).

(3) Video also significantly improves defensive retrieval (Table 3).

Table 3. NFL-World against its tracking-only model (TEST, 5-seed mean).

MeasureNFL-WorldTracking-onlyDifference
Next-second error, yards0.16850.1612+0.007
5-second rollout error, yards4.2724.477−0.205 (95% CI [−0.267, −0.164])
Defensive retrieval, nDCG@100.5160.493+0.023 (95% CI [+0.009, +0.038])

1.3 Compression

"Compression improves quality only for video on play-level tasks. Compressing tracking hurts forecasting."

Compression does not hurt forecasting; what hurts is removing the per-player structure.

  1. On the video side, compression does not hurt forecasting. Compressing video from 1,152 tokens per second to one leaves next-second error unchanged, or even slightly better (0.1691 → 0.1685 yards).
  2. On the tracking side, forecasting degrades only when the per-player structure is removed. The 23 tracking tokens are already compact: each encodes one player's ten frames in that second as a single token. Merging them into one token forces the reader to recover every player from a single vector.
AuthorsOpenReview ↗

Response to Reviewer 8zCb (2/3): evaluation (Weakness 2): classification, retrieval, forecasting

Weakness 2: "The evaluation is neither fair nor robust"

The gains do not come from the protocol. We redid every comparison with NFL-World completely frozen and with the reader protocols matched exactly, and the comparison results still hold under this setting.

2.1 Classification

"NFL-World is fine-tuned with labels, while the baselines are frozen probes. The tracking donor already beats the strongest baseline BDB, so the gain comes mostly from tracking and fine-tuning, not from video."

Completely frozen and read by exactly the same probe as the baselines, NFL-World still scores highest. We froze the NFL-World model entirely and trained the same attentive probe used for BDB, with the identical recipe, over 5 random seeds. The tracking-only model the reviewer mentions is NFL-World's own first stage, trained without labels, so its strength is part of our method. Results are in Table 4.

Table 4. Classification with the same probe (TEST 11-task weighted F1).

ModelProtocolF1
NFL-Worldfrozen + probe, 5 seeds82.24 ± 0.37
Tracking-only model (our first stage)frozen + probe, 5 seeds81.86 ± 0.23
BDBfrozen + probe, 5 seeds81.72 ± 0.42
STTtrained end to end on the labels78.46
V-JEPA 2.1 (best video baseline)frozen + probe70.36

2.2 Retrieval

"NFL-World pooled cosine (frozen features) variant alone loses to Chalkboarding. The win comes from a reader trained on the same rubric used for evaluation, which external baselines never get."

The gain comes from the representation, not from our reader. We trained our reader on raw tracking, which is the input of Chalkboarding, with the same rubric pairs, training budget and validation selection. (Chalkboarding itself cannot be given a reader: it is a matching method that compares role-aligned trajectories directly, with no learned representation.) Results are in Table 5.

Table 5. Retrieval with the same reader (TEST nDCG@10).

InputReaderDefenseOffense
NFL-Worldsame reader0.5150.448
Raw trackingsame reader0.2010.277
Chalkboardingits own matching0.3090.258

2.3 Forecasting

"BDB and STT are trained for 4.8s and evaluated on their first second, while NFL-World reader is built for 1s. Also lack of comparison with specialized SOTA architectures on public datasets e.g., Sports-Traj (Xu & Fu, 2025)."

(1) Even at the horizons BDB and STT are trained for, NFL-World outperforms them. BDB and STT are trained to predict 4.8 seconds, so we rolled NFL-World out to 2, 3 and 4 seconds, where it was never trained, rather than retraining them for one second, which is not their recipe. BDB is slightly ahead only at 2 seconds.

(2) NFL-World also outperforms Sports-Traj, which we trained on our own data. We used its released code and recipe on our training plays, chose its history length on validation, and scored it on the same test windows. Its released checkpoint reproduces its published Football-U result (minADE20 3.55). On our data it is behind NFL-World at every horizon.

Results for both points are in Table 6.

Table 6. Forecasting against BDB, STT and Sports-Traj (TEST RMSE in yards).

Model2 s3 s4 s
NFL-World (rolled out)0.8541.9433.143
BDB (trained for 4.8 s)0.8301.9553.223
STT (trained for 4.8 s)1.1792.3713.603
Sports-Traj (trained on our data)2.2503.7835.051
AuthorsOpenReview ↗

Response to Reviewer 8zCb (3/3): evaluation (Weakness 2, continued), minor point, questions

Weakness 2 (continued)

2.4 Noise

"There are no seeds or confidence intervals for classification. Appendix Tables B2–B3 show changes of up to about 1 point from small recipe changes, often with opposite signs on validation and test, which is the same size as the claimed gains."

We now report classification over 5 random seeds for BDB and for our models (Table 4), and the retrieval and rollout comparisons carry confidence intervals (Table 3). Tables B2–B3 are design ablations (initialization, objective, trainable compressor), not repeated runs of one recipe, so they do not measure seed variation.

2.5 Staged training

"The from-scratch comparison changes several things at once (it is labelled in Fig. 3a but only defined in Appendix)."

The differences in the from-scratch run are largely what training from scratch requires, and they are exactly why we stage. Trained from scratch, the targets must come from EMA copies of the encoders being trained, and such moving targets make joint latent prediction prone to collapse: the run's loss falls, yet it never predicts better than copying the current state forward.

2.6 Private dataset

"Results can't be reproduced."

The NFL plays are licensed, so unfortunately we cannot release them. We release our full code and everything else we can provide without breaking the license. We will also request permission from the NFL to release a small subset of plays and a checkpoint of our model.

Minor: related work

"Related work misses multi-agent trajectory modelling and sports understanding models, e.g., JointDiff (Capellera et al., 2026), and SoccerMaster (Yang et al., 2026)."

We now discuss Sports-Traj (Xu & Fu, 2025), JointDiff (Capellera et al., 2026) and SoccerMaster (Yang et al., 2026) in the related work.

Questions

  • Q1. See 2.4.
  • Q2. See 2.1.
  • Q3. See 2.2. With the same reader, BDB reaches 0.442 nDCG@10 on defense and 0.443 on offense.
  • Q4. See 2.3.
  • Q5. See 1.1.

We hope these results address the reviewer's concerns, and we would be grateful if the reviewer would reconsider the score in light of them.

Reviewer_wdyROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NFL-World, a multimodal world model that learns compact game-state representations of American football plays from synchronized player-tracking data and sideline video. It introduces two key components: modality-specific representation and compression, in which each modality is pretrained through next-window latent prediction and learned queries compress its features into a few tokens per second; and causal multimodal predictive fusion, in which a causal Transformer over the interleaved token sequence predicts current video representations at tracking positions and next-window tracking representations at video positions. Experiments report improvements over tracking-based, video foundation model, and retrieval baselines on play classification, similar-play retrieval, and next-second player-motion forecasting.

Critical strengths

S1. Meaningful research problem. Jointly modeling player-tracking data and game video is a natural but underexplored direction for football understanding, since trajectories capture movement geometry while video can disambiguate physical interactions and outcomes.

S2. Simple and label-free fusion objective. The interleaved design, in which tracking positions predict current video and video positions predict next-window tracking, connects the two modalities through latent prediction without requiring action controls or task annotations during pretraining.

S3. Broad evaluation on real-world data. The model is evaluated on game-disjoint real NFL data across understanding, retrieval, and forecasting, with analyses of fusion, compression, and scaling that also report negative findings, such as tracking alone remaining best for forecasting.

S4. Insightful findings on token compression. The paper reports that compression substantially reduces the fusion input while the most compact configuration achieves the best play-classification score, whereas motion forecasting requires retaining entity-level tracking tokens, suggesting that different tasks may require state representations of different granularity; this is a useful empirical finding rather than a new compression algorithm.

Critical weaknesses

W1. Limited gains from multimodal fusion. The paper's core contribution is a multimodal world model that combines video and tracking, yet compared with its tracking donor, fusion improves classification by only 0.73 pp, leaves offensive retrieval nearly unchanged, and worsens next-second motion forecasting (Table 2), and these are point estimates on only 14 test games, reported without multiple seeds or confidence intervals for these differences. The benefit of video therefore appears concentrated in some classification tasks and defensive retrieval, and the weaker forecasting relative to tracking alone is in tension with the paper's motivation of learning future dynamics through multimodal prediction.

W2. Asymmetric comparison protocols. For classification, NFL-World fine-tunes its compressors, input adapters, and fusion trunk with task labels, while BDB and the video baselines are frozen probes; the retrieval reader is supervised by the same rubric used for evaluation (L1017–1018), while Chalkboarding and play2vec are unsupervised; and the forecasting baselines are trained for a 48-frame horizon and evaluated on their first ten frames (L1268–1269, L1312–1314). The reported gains may therefore partly reflect the training protocol rather than the representation.

W3. Limited novelty and insufficient validation of the world-model claim. The model largely combines existing techniques, including the V-JEPA 2.1 video encoder, Perceiver-style learned-query compression, JEPA-style latent prediction with EMA targets, causal Transformer fusion, and modality dropout, and its most novel component, interleaved cross-modal prediction, is essentially a specific design of JEPA-style latent prediction for video and tracking. Moreover, the experiments mainly validate play classification, similar-play retrieval, and next-second coordinate prediction through a separately supervised decoder, without evaluating autoregressive multi-step rollouts, long-horizon forecasting, or action-conditioned prediction, so the model is closer to a multimodal predictive representation model than to a world model with simulation capability.

W4. Potential outcome shortcut from event-dependent input intervals. Each play's input ends 0.5 s after the first end event (L734–735), and in the released code these events include incomplete passes, interceptions, and sacks but not completed passes, so completed passes typically yield longer inputs and the model may infer the Pass completion and Sack labels simply from where the input ends. This deserves verification because, based on Table 1(a), the overall gain over BDB shrinks by more than half when Sack is excluded.

Questions for authors

Q1. Which events end the play interval, and what do Pass completion and Sack achieve under a fixed, event-independent window, or for a baseline that uses only the number of valid windows?

Q2. What are the means and standard deviations of the main comparisons in Tables 1 and 2 over multiple seeds, or their game-clustered bootstrap confidence intervals, and does the 0.73 pp gain over the tracking donor exceed run-to-run variance?

Q3. How does NFL-World perform when its representation is kept frozen for classification, or when the baselines receive the same fine-tuning budget?

Q4. Can the authors add experiments on autoregressive multi-step rollouts, long-horizon forecasting, or action-conditioned prediction to support the world-model claim, or otherwise reposition the work as multimodal predictive representation learning?

6 comments
AuthorsOpenReview ↗

Response to Reviewer wdyR (1/2): the benefit of video (W1)

We thank the reviewer for the careful review and for recognizing the merits of our work. We answer each point in the order raised. We have uploaded a revised paper; all changes are marked in blue.

W1. Limited gains from multimodal fusion over the tracking-only model, reported without seeds or confidence intervals

We reran the comparison with the tracking-only model over 5 random seeds, with 95% confidence intervals from a paired bootstrap over the 14 test games.

(1) Video provides supplementary information that tracking alone cannot see, and NFL-World's fusion does exactly this. Specifically, classification with fusion improves on tasks such as run-pass option and contested catch, which rely on visual evidence. The largest per-task gains and the overall gain are in Table 1; the full per-task table is in the appendix of the revision.

Table 1. Classification gain from video (TEST F1, 5-seed mean).

TaskNFL-WorldTracking-onlyGain from video
Run-pass option81.277.0+4.2
Contested catch62.659.0+3.6
Penalty50.449.4+1.0
QB outside the pocket89.288.5+0.7
All 11 tasks82.24 ± 0.3781.86 ± 0.23+0.38

(2) Fusion improves long-horizon forecasting. We roll both models out to 5 seconds by feeding each model's predicted tracking tokens back as input. NFL-World beats its tracking-only model at every horizon after 2 seconds, and the 5-second gap is significant (Tables 2 and 3).

Table 2. Forecasting error at 2 to 5 seconds (TEST RMSE in yards).

Model2 s3 s4 s5 s
NFL-World (rolled out)0.8541.9433.1434.272
Tracking-only model (rolled out)0.8541.9913.2604.477
BDB (trained for 4.8 s)0.8301.9553.223—
STT (trained for 4.8 s)1.1792.3713.603—
Cosmos-Predict2.5-2B (head per future second)1.3852.7584.018—
Motus (head per future second)1.3932.7734.039—
Constant velocity1.6403.3444.9546.172

(3) Video also significantly improves defensive retrieval (Table 3).

Table 3. NFL-World against its tracking-only model (TEST, 5-seed mean).

MeasureNFL-WorldTracking-onlyDifference
5-second rollout error, yards4.2724.477−0.205 (95% CI [−0.267, −0.164])
Defensive retrieval, nDCG@100.5160.493+0.023 (95% CI [+0.009, +0.038])
AuthorsOpenReview ↗

Response to Reviewer wdyR (2/2): protocols (W2), novelty and world model (W3), play interval (W4), questions

W2. Asymmetric comparison protocols: the gains may reflect the protocol rather than the representation

The gains do not come from the protocol. We redid every comparison with NFL-World completely frozen and with the reader protocols matched exactly, and the comparison results still hold under this setting.

(1) In classification, NFL-World still scores highest when it is completely frozen and read by exactly the same probe as the baselines. We froze the NFL-World model entirely and trained the same attentive probe used for BDB, with the identical recipe, over 5 random seeds. Results are in Table 4.

Table 4. Classification with the same probe (TEST 11-task weighted F1).

ModelProtocolF1
NFL-Worldfrozen + probe, 5 seeds82.24 ± 0.37
Tracking-only model (our first stage)frozen + probe, 5 seeds81.86 ± 0.23
BDBfrozen + probe, 5 seeds81.72 ± 0.42
STTtrained end to end on the labels78.46
V-JEPA 2.1 (best video baseline)frozen + probe70.36

(2) In retrieval, the gain comes from the representation, not from our reader. We trained our reader on raw tracking, which is the input of Chalkboarding, with the same rubric pairs, training budget and validation selection. (Chalkboarding itself cannot be given a reader: it is a matching method that compares role-aligned trajectories directly, with no learned representation.) Results are in Table 5.

Table 5. Retrieval with the same reader (TEST nDCG@10).

InputReaderDefenseOffense
NFL-Worldsame reader0.5150.448
Raw trackingsame reader0.2010.277
Chalkboardingits own matching0.3090.258

(3) Even at the horizons BDB and STT are trained for, NFL-World outperforms them. BDB and STT are trained to predict 4.8 seconds, so we rolled NFL-World out to 2, 3 and 4 seconds, where it was never trained, rather than retraining them for one second, which is not their recipe. Results are in Table 6; BDB is slightly ahead only at 2 seconds.

Table 6. Forecasting at the horizons BDB and STT are trained for (TEST RMSE in yards).

Model2 s3 s4 s
NFL-World (rolled out)0.8541.9433.143
BDB (trained for 4.8 s)0.8301.9553.223
STT (trained for 4.8 s)1.1792.3713.603

W3. Limited novelty and insufficient validation of the world-model claim

(a) Novelty. We would argue that building on existing components does not mean a lack of novelty: the novelty lies in how they are combined. NFL-World makes interleaving two very different modalities possible and cheap. Each modality is first pretrained and compressed into a compact per-second state, and these states are then interleaved in one causal sequence in which each modality predicts the other. Compression cuts the fusion input from 1,175 tokens per second to 24. To our knowledge, NFL-World is also the first model trained on synchronized official NFL video and tracking.

(b) World model.

  1. We define a world model as a model that learns a compact state of its environment and predicts how that state evolves, following Ha & Schmidhuber (2018), LeCun (2022) and Assran et al. (2025). NFL-World fits this definition exactly.
  2. NFL-World has rollout capability, even if rollout is taken as a requirement for a world model (Table 2).
  3. We do not frame a world model as requiring action conditioning, and football has no meaningful per-second action labels.

W4. Potential outcome shortcut from event-dependent input intervals

The interval ends when the play ends, for every play and every method. It runs from line-set to 0.5 s after the first end event: tackle, out of bounds, touchdown, incomplete pass, interception, sack, kneel, spike or fumble. A completed pass therefore ends at the tackle, out of bounds or touchdown after the catch. BDB and the video baselines use exactly the same plays and intervals. The protocol is therefore completely fair and standard.

Questions

  • Q1. See W4.
  • Q2. See Tables 1 to 6. They report 5-seed means, with confidence intervals for the retrieval and rollout gains.
  • Q3. See W2 (1).
  • Q4. See W3 (b).

We hope these results address the reviewer's concerns, and we would be grateful if the reviewer would reconsider the score in light of them.

AuthorsOpenReview ↗

Brief Follow-up Before the Discussion Deadline: Additional Results and Clarifications

Follow-up to Reviewer wdyR

Dear Reviewer,

Thank you again for your time and thoughtful feedback. As the discussion deadline approaches, we would appreciate your consideration of a few updates that may be particularly relevant to your concerns, in addition to our previous response.

1. Additional evaluations support fusion gains

Classification. Across three pre-training seeds per model, matched light tuning gives fusion a higher aggregate score. The paired confidence interval excludes zero, supporting a statistically significant aggregate gain under the paper's main protocol.

Classification measureResult
NFL-World fusion: weighted macro-F1 (%)83.40 ± 0.34
Tracking-only: weighted macro-F1 (%)82.73 ± 0.14
Paired fusion gain, percentage points [95% CI]+0.68 [+0.19, +1.16]

Model scores are mean ± sample standard deviation across three pre-training seeds. The paired gain uses a Student-t confidence interval. Full per-task results: Table A9.

Rollout. The five-reader-seed results show that fusion improves rollout over its tracking donor at 3–5 seconds. An additional control keeps the fusion model and forecasting reader fixed and replaces observed video tokens with the learned null token, without refitting. Retaining video gives a small benefit at every reported horizon, with all corresponding confidence intervals excluding zero.

HorizonFused − tracking-onlyFused − video-nulled
1 s+0.0074 [+0.0069, +0.0079]−0.0015 [−0.0018, −0.0011]
2 s−0.0001 [−0.0036, +0.0022]−0.0035 [−0.0044, −0.0025]
3 s−0.0485 [−0.0642, −0.0392]−0.0049 [−0.0069, −0.0032]
4 s−0.1169 [−0.1528, −0.0935]−0.0061 [−0.0099, −0.0032]
5 s−0.2050 [−0.2668, −0.1644]−0.0062 [−0.0111, −0.0018]

Chained-rollout RMSE differences in yards: five-reader-seed means [95% paired game-bootstrap CIs]. Negative differences favor fusion. Means appear in Table 2(c); full intervals and protocol appear in Appendix A.6.3, Table A10.

Defensive retrieval. The previously reported improvement also has a confidence interval above zero (Appendix A.6.1).

MetricFusionTracking donorGain [95% CI]
Defensive nDCG@100.5160.493+0.023 [+0.009, +0.038]

Means over five reader seeds; paired game-bootstrap interval.

2. NFL-World retains an advantage under the matched frozen-probe BDB comparison

Both models use frozen representations and the same probe recipe. NFL-World retains the higher mean in this matched comparison (Appendix A.3.1).

ModelWeighted macro-F1 (%)
NFL-World, frozen82.24
BDB, frozen81.72

Mean over five probe seeds.

3. NFL-World supports autoregressive forecasting beyond its training horizon

Reusing its next-second reader autoregressively, NFL-World achieves the lowest error among the external baselines at 1, 3 and 4 seconds; BDB is better at 2 seconds. BDB is the strongest external comparator at each reported horizon.

HorizonNFL-WorldBest external baseline: BDB
1 s0.16850.2517
2 s0.8540.830
3 s1.9431.955
4 s3.1433.223

Coordinate RMSE in yards, lower is better; main-protocol results from Table 1(c), distinct from the multi-seed paired controls above.

These results demonstrate forecasting beyond the one-second training horizon. Together with the video-nulling control, they support the model's predictive capabilities and a contribution from observed visual context.

Taken together, these results support our central contribution: compressed game-state representations and interleaved cross-modal prediction for classification, retrieval and forecasting. Compression reduces fusion inputs from 1,175 to 24 tokens per second in the entity-preserving configuration, making this a practical approach to efficient multimodal predictive modeling.

We appreciate the time you have devoted to evaluating our work. We hope these additional results address your concerns and clarify the contribution. If they change your assessment, we would be grateful if you would reconsider your score. We would also welcome any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.