acceptodds
Under review as a conference paper at ICLR 2027

Narrate to Simulate: Trajectory-Language Diffusion for Multi-Agent Scenes

Abstract

A simulated multi-agent scene is useful only if it can be read: not just where the agents go, but what happens between them. Generative models of multi-agent motion give the first and leave the second to the user. We extend joint continuous–discrete diffusion with a text modality: one denoiser generates the trajectories, the possession sequence and an event narration of a football play under one shared timestep, so every sample carries its own narration, grown from an empty sequence by an insertion-only edit flow. Insertion flows under-generate: a well-calibrated model reads its own short text as evidence of a short target and stops early, and more steps narrow the gap without closing it. , a one-line training augmentation, repairs it: at ten calls it improves narration content by a fifth and CIDEr two- to threefold, surpassing the unaugmented model at fifty calls and matching it at a hundred. We evaluate fidelity to the real play, coherence between the streams and control through the narration. On a season of a top European league and public World Cup 2022 data, the co-generated narration costs no significant accuracy against a matched baseline without it, only ball tempo, which tracks narration length; each narration describes its own play about as faithfully as the reference describes the real one; and editing it steers the play: swapping a pass target for a nearby teammate moves the ball to that player in most samples, with no retraining or guidance. Upon acceptance we release the league data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Agent, Diffusion, Edit Flows, Trajectory

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_b8zgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper jointly generates football trajectories, possession, and narration, and proposes LAG to mitigate text under-generation in few-step insertion sampling.

Critical strengths

The analysis of under-generation is informative, and the controlled ablations demonstrate improvements from LAG.

Critical weaknesses
  1. The benefit of joint generation is insufficiently established. On identical generated plays, joint generation improves carrier-consistency F1 by only 0.02, while CIDEr favors post-hoc narration on League and joint generation on World Cup (Appendix E.2). Trajectory quality also does not consistently improve. These results provide limited justification for coupling the two generation processes instead of generating trajectories first and narrating them afterward.

  2. LAG leaves the motivating problem substantially unresolved. It changes training corruption but preserves the sampler’s structural growth constraint (§3.5). Narrations still reach only 66–67% of reference length at ten calls, and fifty calls do not close this gap. Shorter text alone need not indicate failure, but the residual under-generation is also associated with reduced ball motion (§4.3). LAG improves its baseline without establishing sufficiently complete narration that avoids degrading the simulated scene.

Questions for authors
  1. What clear quality or measured efficiency advantage does joint generation offer over a competitive two-stage approach?

  2. Can a growth-aware schedule or multi-token insertion address the remaining failure more directly than LAG?

6 comments

(1/2)

We thank the reviewer for a precise review. Each answer starts with the short version. Numbers cited with a table, section or appendix are from the paper; all others are new. Pairs of numbers are World Cup / League (the paper's tables list League first), and "steps" are the paper's denoiser calls.

W1/Q1 — What does joint generation offer over a competitive two-stage approach? A pipeline that simulates first and narrates afterwards cannot steer the play with text: its simulator ignores the narration. Coupling the two makes the narration a control. Against pipelines whose simulator does take text, we measure two quality advantages: at equal change to the rest of the play, local edits (resumed mid-sampling) succeed 12–14 points more often (point 1), and the play is simulated better than by a pipeline that narrates first, on every seed (point 2). Narrations also agree slightly more with their play (point 3). A post-hoc captioner narrates finished plays closer to the references (Table 1, point 4).

  1. Edit a sample and keep the rest of it. A user may like a generated play but want one change. In this experiment, the change is that the last pass goes to the receiver's nearest teammate, swapped throughout the narration. Sampling takes 10 steps from noise, and we store the state after each step. To edit, we pin the sample's own narration, edited, and re-run its last steps from a stored state: "resumed after step 3" keeps the first 3 steps and re-runs the other 7; step 0 regenerates the whole sample from its initial noise. Resuming earlier follows the edit more often but changes more of the play. A pipeline can do the same if its simulator accepts text: text-guided JointDiff (JointDiff+Guid) generates the play, a T5 captioner trained on our data narrates it, and JointDiff+Guid re-runs the same steps with the edited narration as its prompt. "Play moves" is how much the procedure itself changes the play: we re-run the same steps with the narration unchanged and take the mean distance to the original sample, over all players and the ball in the generated frames. Each cell gives the share of samples where the new receiver gets the ball (higher is better) and, in brackets, how far the play moves (lower is better):

    Resumed after step (of 10)0 (whole sample)357
    WC, ours0.69 (1.5 m)0.58 (0.8 m)0.39 (0.3 m)0.26 (0.1 m)
    WC, pipeline0.74 (1.9 m)0.67 (1.2 m)0.45 (0.7 m)0.25 (0.4 m)
    League, ours0.84 (1.5 m)0.77 (0.9 m)0.57 (0.4 m)0.33 (0.2 m)
    League, pipeline0.73 (2.0 m)0.72 (1.5 m)0.65 (0.9 m)0.45 (0.5 m)

    At every step ours changes the play less. At equal change (ours resumed two steps earlier: 3 against 5, 5 against 7), ours follows the edit 12–14 points more often, and 6–9 points more after subtracting how often the new receiver gets the ball without the edit (paired bootstrap over test windows; all 95% intervals exclude zero). Regenerating the whole sample, ours follows more and changes less on League, while the pipeline follows more on WC (0.74 vs 0.69). An edited play has no ground truth, so ES is measured with the narration unchanged: equal or lower for ours at every step. Our model does this zero-shot; JointDiff+Guid is trained for text-guided simulation.

  2. Simulate without inheriting the errors of a narration written first. Narrations from an autoregressive T5 forecaster make JointDiff+Guid worse than an empty prompt (ES 2.93 vs 2.80 on WC, 3.35 vs 3.29 on League; empty prompt: Table 9). Pinned into our model in place of its own narration, they hurt it too (2.77–2.84 vs 2.71; 3.28–3.33 vs 3.25 in Table 1; given whole or revealed along the schedule). Generating narration and play together avoids this: ours beats the forecaster pipeline on every seed (WC 2.71–2.83 over 5 seeds, League 3.24–3.27 over 3).

  3. Coherence. On identical plays, co-generation adds one to two points of cF1 (0.83 vs 0.81 on WC, 0.81 vs 0.79 on League; Appendix E.2). It holds for cF1 and order on both corpora.

  4. Where the pipeline is better or equal. Autoregressive T5 narrators reach higher CIDEr: 1.41 / 1.23 for the captioner on finished plays and 1.26 / 1.13 for the forecaster of point 2, against our 0.79 / 0.59 in Table 1. The forecaster's narrations agree less with the real play's ball carriers (cF1_gt 0.48 / 0.45, against our 0.51 / 0.47 in Table 1). We do not claim an efficiency advantage. Trajectories are on par: JointDiff alone is slightly better (seed means, ES 2.72 vs 2.77 on WC, 3.17 vs 3.25 on League; Section 4.1), and initialising our model from JointDiff matches it on WC (mean ES 2.73 over three seeds, against 2.74 for JointDiff on the same seeds and 2.71 when trained for the same total number of epochs) with the same narration quality.

(2/2)

W2 — Under-generation and reduced ball motion. The 66–67% (Table 3) is measured in words. In events the gap is smaller, and closing it further does not move the ball.

  • At ten steps our narrations write 76% / 85% as many complete events as the reference. At fifty steps complete events reach 85% / 86%, with 67% / 61% of the words (Appendix E.5). Much of the missing length is qualifiers (pressure, zones).
  • For a narration given as input, ball travel follows events, not words. The reference without its qualifiers (66–71% of the words) gives the same ball path as the full reference (0.98 / 0.94 vs 0.98 / 0.92 of real, the latter in Table 3); cutting events drops it (0.78 / 0.67).
  • For our own narrations, more events do not move the ball. From 13% to 85% as many complete events as the reference (WC; 5 to 50 steps, or a doubled insertion rate), the ball path stays at 0.89–0.91 (0.83–0.86 on League). Timing does move it on WC: pinning the same final narration from the first step raises it from 0.91 to 0.96, close to the full reference's 0.98 (0.91 and 0.98 are in Table 3).

These results refine the tempo analysis of Section 4.3: a narration given as input steers the play through its events, not its length in words. The revision says "events" where the abstract, introduction, Section 4.3, Appendix D.4 and the conclusion say "length", and reports the timing result in Section 4.3. LAG already helps the scene: without it the ball path is 0.80 / 0.65 (Table 11), against 0.91 / 0.86 (Table 3). The limitations now report the timing effect and the remaining ball-path gap.

Q2 — Growth-aware schedule or multi-token insertion? Both act on the growth limit, which is not the main cause of the short narrations: the model's length inference is, and that is what LAG fixes. We traced the sampler step by step and tested a growth-aware schedule at sampling (starting values in this answer are from Table 1):

  • Where the length is lost. The growth limit (one insertion per site, so the text at most doubles per step) leaves the text behind schedule in the first two to three of 10 steps. Without LAG, the model then infers its target length from its own short text (Eq. 3) and stays behind, ending at about a third of the reference tokens. With LAG it catches up most of the way, and the limit binds again only at the last step, where it costs 0.09 / 0.08 of the reference length.
  • Growth-aware schedule. At sampling, we spaced the 10 steps so that the length the schedule asks for never more than doubles from one step to the next (more steps early, fewer at the end). For the model without LAG, this raises CIDEr only from 0.375 to 0.446 (WC) and from 0.213 to 0.293 (League), far from LAG's 0.785 / 0.587. On our model it lowers CIDEr (0.785 → 0.567 on WC). LAG instead improves narration and trajectories together (ES 2.82 → 2.71, 3.46 → 3.25).
  • Next steps. Training with a growth-aware reveal schedule and multi-token insertion would act on the remaining deficit and are complementary to LAG; our limitations (Appendix F) already list both. On top of LAG, doubling the insertion rate at sampling already helps: CIDEr 0.785 → 0.854 / 0.587 → 0.629, at unchanged ES.

Thank you for the additional experiments and explanations. The new results strengthen the evidence that LAG reduces text under-generation. However, I still have concerns about the benefit of jointly generating the play and its narration.

If the goal is to describe a generated play, generating trajectories first and then using a dedicated captioner already achieves higher CIDEr. The joint model also does not clearly improve trajectory quality over JointDiff alone, and the authors do not claim an efficiency advantage. Editing the play through text is therefore an important reason to use the joint model.

The editing results do not yet convince me. Simple edits can succeed, but harder edits remain unreliable. The new local-editing experiments show some positive results, but they do not clearly establish how much better the joint model is at making the requested change while preserving the parts of the play that should remain unchanged.

Meanwhile, the main method, LAG, addresses short narrations. The additional experiments support its effectiveness, but they do not show that more complete narrations lead to better editing. This leaves my central question unanswered: if the editing benefits are still limited, why generate the play and narration jointly rather than generate the play first and describe it afterward?

The authors have explained more clearly how to improve the proposed model, but have not sufficiently shown why we should choose this model over the simpler alternative.

Reviewer_bmM7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper presents a joint trajectory–language diffusion framework for multi-agent scene simulation. Building on JointDiff, the proposed method jointly generates continuous trajectories, discrete possession events, and variable-length event narrations using a shared denoising process. The paper further identifies a length-collapse problem in insertion-only text flows and introduces LAG, a behind-schedule training augmentation designed to reduce the mismatch between training and inference. Experiments on two football datasets evaluate trajectory fidelity, narration quality, cross-modal coherence, and text-based control.

Critical strengths

Strengths

S1. Jointly generating multi-agent trajectories and their semantic narrations is a meaningful direction for interpretable simulation. The framework naturally supports both scene generation and language-based interaction.

S2. The distinction between the combinatorial insertion limit and the learned frozen-ratio behavior provides a useful explanation of why insertion-only flows may under-generate at low sampling budgets.

S3. LAG substantially improves narration quality over the unaugmented insertion flow, particularly under ten denoiser calls. The comparisons across sampling budgets, datasets, and insertion-rate scaling provide useful support for its effectiveness.

Critical weaknesses

Weaknesses

W1. The paper compares joint generation with a pipeline that first generates trajectories and then narrates them. However, the two-stage baseline relies on the proposed text model conditioned on a completed play, rather than an independently optimized captioning model.

On League, the two approaches achieve similar CIDEr scores, while the improvements in coherence are relatively modest. A stronger two-stage baseline with matched computational resources would help clarify whether joint generation offers a substantial advantage in narration quality, cross-modal consistency, or efficiency.

W2. LAG effectively addresses under-generation by exposing the model to behind-schedule states during training. However, the resulting improvements in narration metrics may partly arise from generating more complete event chains.

An evaluation controlling for generated narration length would help determine whether LAG also improves semantic accuracy and event consistency independently of sequence completeness.

W3. The pass-target editing experiments provide encouraging evidence that the model responds to linguistic constraints. However, the replacement targets are selected as nearby teammates, and the first-receiver success rates are substantially lower than the overall receiver success rates.

More challenging interventions involving multiple sequential events, distant targets, or coordinated multi-agent behaviors would provide stronger evidence of compositional language control.

W4. The paper reports that generated narrations are substantially shorter than their references and that the corresponding ball trajectories exhibit reduced travel distances.

Although the authors analyze this coupling and show that conditioning on reference narrations improves trajectory realism, the issue remains unresolved when the narration is generated by the model itself.

This raises concerns about whether the joint model preserves the diversity and temporal dynamics of realistic multi-agent motion. Further analysis of the trade-off between narration completeness and motion fidelity would strengthen the work.

Questions for authors

Questions for Authors

Q1. Can the authors compare against an independently trained, computation-matched two-stage narration baseline?

Q2. Does LAG improve event accuracy and cross-modal coherence when controlling for generated narration length?

Q3. How does the model perform under multiple sequential narration edits rather than a single pass-target replacement?

5 comments

(1/2)

We thank the reviewer for asking for stronger baselines and harder tests. We ran all of them; each answer starts with the result. Numbers cited with a table, section or appendix are from the paper; all others are new. Pairs of numbers are World Cup / League (the paper's tables list League first), and "steps" are the paper's denoiser calls.

W1/Q1 — An independently trained two-stage baseline. Joint generation is at least as consistent with its play's ball carriers, gives better trajectories than a pipeline that writes the narration first, and edits a sample while keeping more of it; a dedicated captioner narrates finished plays closer to the references (Table 1). We trained two autoregressive baselines on the same data, each with a pretrained T5-base decoder: a captioner that narrates JointDiff's plays (150M parameters with JointDiff, against our 124M in Appendix A), and a forecaster that writes the narration before JointDiff+Guid simulates the play (below). Both are added to Table 1 in the revision. We do not claim an efficiency advantage.

CIDEr WC / LeagueCarrier F1 vs play, WC / League
Ours (one model)0.79 / 0.590.789 / 0.776
JointDiff → T5 captioner1.41 / 1.230.783 / 0.751

Our CIDEr is from Table 1; the rest is new. Carrier F1 uses the possessor derived geometrically from the generated play, so both rows are scored the same way.

  • Narration quality: the captioner wins clearly on CIDEr. Its decoder was pretrained to write text left to right and is only fine-tuned on our data; our insertion model has no pretrained generator (only its T5 encoder is pretrained) and learns where and what to insert from our data. We do not claim better narration than a dedicated captioner. An autoregressive narrator, however, cannot be generated with the play: it writes after the play, when the text can no longer steer it, or before it (below). With the same non-autoregressive narrator (Table 1, JointDiff → Ours), co-generation narrates better than narrating afterwards on WC (CIDEr 0.79 vs. 0.71, eF1 0.51 vs. 0.47) and on par on League (0.59 vs. 0.60, eF1 0.47 vs. 0.45).

  • Ball-carrier consistency: comparable on WC, and ours is ahead on League.

  • Narrate then simulate (the forecaster pipeline; ours from Table 1):

    CIDErcF1_gt (vs real play)ES
    Ours0.79 / 0.590.51 / 0.472.71 / 3.25
    T5 forecaster → JointDiff+Guid1.26 / 1.130.48 / 0.452.93 / 3.35

    Its narrations read closer to the references but name the real play's carriers less often, and they make JointDiff+Guid worse than an empty prompt (2.80 / 3.29 in Table 9). Pinned into the same run of our model in place of its own narration, they hurt it too (2.77–2.84 / 3.28–3.33, whether given whole or revealed along the schedule). A narration written before the play passes its errors on; generating the two together avoids this, and ours beats this pipeline on every seed (WC 2.71–2.83 over 5 seeds, League 3.24–3.27 over 3).

  • Editing: joint generation edits a sample while keeping more of it. When this captioner narrates a JointDiff+Guid play and JointDiff+Guid finishes it again from the edited narration, the new receiver gets the ball 12–14 points less often than with our model in local edits (resumed mid-sampling), at equal change to the rest of the play (reply to Reviewer b8zg).

W2/Q2 — Does LAG help beyond producing longer narrations? Yes for event and player accuracy. For coherence with the play, LAG is on par at equal length and gains by narrating more of the play. For each window and sample, we cut both narrations (with and without LAG) to the shorter one's number of events and compared them:

(with LAG vs without)WCLeague
Event F1 of the prefix0.688 vs. 0.5790.639 vs. 0.515
CIDEr of the prefix2.79 vs. 1.872.23 vs. 1.53
Player precision of the prefix0.724 vs. 0.6100.800 vs. 0.751
Event precision of the prefix0.694 vs. 0.5920.662 vs. 0.534

At equal length, the LAG narration gets more events and more players right. All four rows score the cut narration against the reference's first events, so they are not comparable to Table 1.

For coherence, LAG's gain is the completeness it is designed for. Cut to equal length, the carriers a narration names hold the ball in its play about as often with or without LAG (precision 0.97 vs. 0.96 / 0.91 vs. 0.93, counting the observed frames, which the narration also covers). LAG's higher cF1 in Table 2 comes from naming more of its play's carriers (recall 0.88 vs. 0.77 / 0.87 vs. 0.81, Table 13).

(2/2)

W3/Q3 — Multiple sequential edits and distant targets. Our model follows harder edits well above the rate without the edit. A model trained only for text-guided simulation (JointDiff+Guid) follows them more often, but by leaving the distribution of real play. Each edit swaps the original receiver with the new one throughout the narration; success is scored as "the new target receives the ball":

Ours WC / LeagueJointDiff+GuidNo edit (own narration)
Nearest teammate (ours and no edit: Table 3)0.69 / 0.910.77 / 0.960.27 / 0.26
Farthest teammate, first pass (passer→receiver 44–48 m)0.22 / 0.170.59 / 0.480.04 / 0.03
Farthest teammate, last pass (the pass Table 3 edits; 40–45 m)0.33 / 0.270.53 / 0.620.07 / 0.05
Two edits (first and last pass, random teammates), both succeed0.24 / 0.320.49 / 0.700.01 / 0.02

At the last observed frame, real receivers stand more than 40 m from the passer in only 1.1% (WC) / 1.4% (League) of passes, against 62% / 71% of the farthest-teammate edits. JointDiff+Guid obeys them by distorting the play: the Wasserstein distance between its ball paths and real ones rises from 2.7 / 1.8 with the unedited reference narration to 15.5 / 6.1 with the edit. With the same edit, ours is at 3.2 / 3.2. Editing the last pass instead: 7.9 / 4.2 for JointDiff+Guid, 3.6 / 3.1 for ours. The revision reports success as a function of distance (Appendix E.7). For a simulator, an implausible instruction should not override realism, and our model behaves that way.

Coordinated behaviour: when our model follows the farthest-teammate edit, the other players adjust. Defenders close down the new receiver to the distance seen at real receptions, and the defending team shifts about 3.5 m toward him (reply to Reviewer r6kp, Q2).

W4 — Narration completeness vs motion fidelity. Motion fidelity does not trade off against how complete our own narration ends up. On WC it depends more on timing: during sampling, our narration is written step by step alongside the play.

  • Narrations given as input: ball travel follows their events, not their words. Removing the modifiers (pressure, zones) from the reference keeps its events and 66–71% of its words, and keeps the ball path at 0.98 / 0.94 of real, as high as the full reference (0.98 / 0.92, Table 3). Truncating its events drops it to 0.78 / 0.67.
  • Our own narrations: how complete they end up does not move the ball. At ten steps they reach 67% / 66% of the reference words (Table 3) and 76% / 85% of its complete events. Across 5 to 50 steps, or with a doubled insertion rate, they end with 13% to 85% as many complete events (7% to 90% on League), while the ball path stays at 0.89–0.91 / 0.83–0.86 of real.
  • Timing (WC): giving each sample its own final narration from the first step, with the same initial noise, raises the ball path from 0.91 (Table 3) to 0.96, at the same ES.
  • Diversity: compared with JointDiff, which does not narrate, our 20 samples are 6–7% closer to one another (spread, the mean distance between samples), and the average sample is as good or better (sADE_avg 4.82 vs 4.91 on WC, 5.65 vs 5.75 on League; seed means, Appendix E.1).

These results refine Section 4.3: a narration given as input steers the play through its events, not its length in words. The revision says so wherever the paper says "length", reports the timing result, and adds the remaining ball-path gap of our own plays (0.91 / 0.86 of real, Table 3) to the limitations.

Dear reviewer, the authors have responded to your comment. Could you please take a look? Let us know if you have further concerns.

Reviewer_f6x3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends JointDiff to jointly generate football trajectories, possession sequences, and event narrations. It uses an insertion-only text flow and introduces LAG, a training augmentation that exposes the model to narrations lagging behind the generation schedule. Experiments on League and World Cup data show improved narration quality. The paper also evaluates consistency between the generated outputs and shows that pinning edited event chains can steer the play.

Critical strengths
  1. Generating a play together with its narration is useful. The editing experiments show that the text also provides control without additional training for that task.
  2. LAG is a small change with clear empirical benefits. The five-seed World Cup results, experiments with different LAG settings, and comparison with inference-time rate scaling support its effectiveness.
  3. The evaluation includes thoughtful controls, especially re-narrating the same generated plays to account for scene complexity. The authors also report trade-offs, including reduced ball travel.
Critical weaknesses
  1. The frozen-ratio explanation needs a more direct check. The probe in Appendix D.5 evaluates predictions on corrupted reference states rather than tracking the length deficit during actual sampling. It therefore provides indirect support for Eq. 3.
  2. The joint model's mean ball path length is 86% of the ground-truth mean on League and 91% on World Cup, compared with 96% and 99% for JointDiff (Table 3). Pinning the reference narration changes both text length and event content, so it does not isolate the proposed effect of length.
  3. The coherence metrics mainly compare the narrated ball carriers and their order with the generated possession sequence. They do not establish whether pressure, actions, outcomes, and zones match the generated trajectories; the reference-based metrics in Tables 7-8 address a different question (Appendix C.3).
Questions for authors
  1. How do text length and predicted missing-token counts evolve at each sampling step, with and without LAG?
  2. Is the cap of 20 on omega applied only during training, or also during sampling? If it is also used during sampling, how does it affect the 50- and 200-call results?
  3. Does ball travel change when a narration is shortened or expanded without changing its meaning?
  4. What are the cF1 and Order scores when possession is derived directly from the generated trajectories? Do narrated zones and pass directions agree with the generated ball path?
3 comments

(1/2)

We thank the reviewer for four precise checks. We ran all of them; each answer starts with the result. Numbers cited with a table, section or appendix are from the paper; all others are new. Pairs of numbers are World Cup / League (the paper's tables list League first), and "steps" are the paper's denoiser calls.

W1/Q1 — Length and predicted missing tokens per sampling step, with and without LAG. The trace shows both effects of Section 3.4 during real sampling: the one-insertion-per-site cap and the frozen ratio of Eq. (3). At each of the 10 steps (first 1,024 test windows, one sample each), we logged the text length and the length implied by the model's predicted insertions (current plus predicted missing tokens), in tokens relative to the reference (Table 3 counts words).

  • Without LAG, the one-insertion-per-site cap binds in the first two to three steps and leaves the text at 0.3 / 0.2 of its schedule. From the sixth step the implied length matches n/κ, the current length divided by the schedule, within 0.03, as Eq. 3 assumes. The text then only grows by the schedule's ratio and ends at 0.35 / 0.34 of the reference, while the model inserts only 0.1–0.3 tokens per site, far below the cap.
  • With LAG, the implied length stays at 0.71–0.83 / 0.80–0.91 and the text reaches 0.73 / 0.80. The cap binds again only at the last step, where it costs 0.09 / 0.08 of the reference length.

Without LAG, the cap creates the early deficit and the model's own rates carry most of it to the end (Eq. (3)); LAG largely removes this second effect. The revision replaces the toy example of Figure 2 (left) with this trace.

Q2 — Is the cap of 20 on omega used at sampling? No: it caps only the training loss weight. The sampler uses the exact ω(s̄) at every budget, including 50 and 200 calls. Appendix A now states this.

W2/Q3 — Does ball travel change when a narration is shortened or expanded without changing its meaning? For a narration given as input, no: ball travel follows its events, not its words. We pinned three versions of the reference narration (ball path relative to real):

Pinned narrationWords keptEvents keptBall path
Full reference (Table 3)100%100%0.98 / 0.92
Modifiers removed (pressure, zones), same events66–71%100%0.98 / 0.94
Truncated to its first events, same T5 token count as row 243–51%44–51%0.78 / 0.67

Row 2 → row 1 is the expansion: adding the modifiers back (40–50% longer, same events) leaves the ball path practically unchanged.

For the model's own narrations, the ball path does not depend on the final length; on WC it depends on how early in sampling the narration is available:

  • With 5, 10 or 50 sampling steps, or a doubled insertion rate, our narrations end with 13% to 85% as many complete events as the reference (WC), and the ball path stays at 0.89–0.91 (WC) and 0.83–0.86 (League).
  • We regenerated each sample from its own initial noise, pinning its own final narration from the first step. On WC the ball path rises from 0.91 (Table 3) to 0.96, close to the full reference's 0.98, at the same ES; pinned from steps 3, 5 or 7 it helps less (0.94, 0.92, 0.92).

The revision refines the tempo analysis of Section 4.3 accordingly: "events" replaces "length" in the abstract, introduction, Section 4.3, Appendix D.4 and conclusion, and Section 4.3 reports the timing result. We add the timing effect and the remaining ball-path gap to the limitations.

(2/2)

W3/Q4 — Coherence from the trajectories themselves; zones and pass directions. Carriers and order hold when possession is derived geometrically; pass direction holds partly; zones do not beyond scene level. We took the possessor from the generated trajectories (nearest player to the ball) instead of the possession head:

cF1Order
Ours, narration vs. its own play0.789 / 0.7760.668 / 0.654
Ceiling: reference narration vs. real play (Table 2)0.83 / 0.790.73 / 0.67
Control: narration vs. another sample's play, same scene0.564 / 0.5230.435 / 0.396

The possession head already matches the geometric possessor in 94% of frames (Acc_gen, Table 2), so our scores are within 0.03–0.05 of those Table 2 reports for our model with the possession head. The ceiling uses the same geometric possessor, which defines real possession (Appendix B).

Pass direction (League; WC narrations do not state it) agrees with the generated ball path in 0.72 of passes: above the shuffled control (0.67) and the majority class (0.52), below the ceiling (0.79).

Narrated zones: each clause is matched to the next stretch of the generated play in which its player holds the ball, and its zone is compared with the ball's position there. They agree in 0.75 / 0.79 of clauses, against 0.73 / 0.73 for the control and 0.98 / 0.98 for the reference narrations. Since where a player receives the ball depends mostly on the scene, the control is already high: our zones place the play in the right area of the pitch, but not each event of their own play. Pressure, action type and outcome are not verified against the trajectories either. The limitations now say so, and the revision scopes the coherence claim of the abstract and Section 4.2 to ball carriers and their order.

Dear reviewer, the authors have responded to your comment. Could you please take a look? Let us know if you have further concerns.

Reviewer_r6kPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper jointly generates football player and ball trajectories, possession, and an accompanying event narration from a short observed sequence. Its central technical contribution is LAG, a training augmentation addressing premature stopping in insertion-based text generation. The paper also demonstrates that editing the narration can steer the generated play. Evaluation on league and World Cup data examines trajectory fidelity, agreement between the generated modalities, and control through text.

Critical strengths

I view the contribution primarily as interpretable and controllable multi-agent generation. It is a focused methodological contribution with a useful demonstration. Its novelty lies in the joint formulation, the diagnosis of text undergeneration, and the training remedy. Critical strengths

A useful connection between movement and language. Jointly generating a play and its description makes the output easier to inspect and potentially manipulate. This is valuable when examining many generated scenarios.

A focused technical contribution. The paper identifies why insertion-based narration becomes too short and proposes a simple training intervention. The improvements in narration content with relatively few generation steps support the usefulness of this intervention.

Evaluation beyond trajectory accuracy. Separating fidelity to recorded play, consistency within generated samples, and responsiveness to narration edits is appropriate. A plausible alternative need not reproduce the recorded future exactly, but its movement and narration should agree.

Promising control without edit-specific retraining. Changing the intended pass receiver influences the generated trajectories. This provides an initial demonstration of language as an interface for exploring alternative plays.

Critical weaknesses

Its relevance to tactical understanding and coaching is interesting, but these applications require a clearer distinction between describing events and understanding the decisions behind them.

The application goal is not well defined. The motivation refers to coaches, analysts, and broadcasters, while the experiments establish joint generation and limited editing control. These uses have different requirements. A convincing generated scenario may be useful for visualization, but its value for coaching depends on whether it supports sound interpretation of decisions and alternatives.

Event narration is not directly linked to explanation. Describing who passed to whom does not explain why the pass was appropriate, what the player perceived, or which alternatives were available. The paper should distinguish descriptive consistency from tactical or causal understanding, particularly when referring to narration as an explanation.

Tacit knowledge and expertise remain unresolved. Experienced players may anticipate, disguise intentions, or exploit opportunities differently from less experienced players. Similar observed configurations can therefore produce different reasonable actions. The model may capture statistical regularities associated with these behaviors, but the evaluation does not establish that it represents expertise, intent, or the quality of a decision. This limits the interpretation of its outputs for learning and coaching.

Including both teams is necessary, but does not establish responsive interaction. The model represents all players, which is a strength. However, joint trajectory generation does not by itself demonstrate a perception–action process in which players observe, act, and adapt to opponents. The editing experiments would be more informative if they examined how defenders and supporting teammates adjust, beyond whether the requested receiver eventually obtains possession.

Successful prediction and good decision-making are different. A tactically reasonable action can fail because of execution or an opponent’s response. Conversely, a poor decision can succeed. Although the paper already evaluates multiple samples and distributional fidelity, it remains unclear whether generated alternatives capture meaningful tactical diversity. Agreement with one recorded continuation cannot fully resolve this question.

Remaining generation limitations affect practical usefulness. Narrations remain substantially shorter than the references, and reduced narration length is associated with reduced ball movement. Furthermore, eventual reception by the edited target is considerably more successful than immediate reception after the passer. These findings indicate that language completeness and precise event control remain material limitations.

The paper in my opinions needs to more carefully bound its claims about explanation and tactical usefulness, and clarify the limits of interaction modeling and narration-based control.

Questions for authors
  1. what is the primary intended application? I see scenario generation, forecasting, tactics, coaching ? Each of these are different - how would you align the claims and evaluations based on the diverse goals one can have?
  2. When a pass target changes, do defenders and others make appropriate adjusts?How do experts assess these responses quantitively?
  3. How can a diversity of plausible continuations include failed passes and interceptions? What can the model anticipate is the question.
  4. Length of prediction and relation to narration accuracy.
  5. What happens when the edit is unfavorable? How does the model represent things like failure, alteration of play, or does it produce a continuation that is not possible?
  6. Related work in InfoGCN++ - representation prediction for online action recognition and M2D2M in language condition generation and coherence provide a relevant context for partial observation understanding and sequential motion generation.
5 comments

(1/2)

We thank the reviewer for reading the paper as a contribution to interpretable, controllable generation. We agree with this framing, and with the request to bound the claims. Each answer starts with the result. Numbers cited with a table are from the paper; all others are new, computed on the paper's test windows. Pairs of numbers are World Cup / League (the paper's tables list League first), and "steps" are the paper's denoiser calls.

Q1 — What is the primary intended application? Scenario generation for analysis: many plausible continuations of an observed situation, each with a description that can be read, searched and edited. We do not claim forecasting of the single recorded future, decision quality or coaching value, and the revision removes the wording that suggests them. In particular, the narration describes events (who passed to whom, under which pressure, with which outcome); it does not explain why a decision was made, and the revision no longer calls it an explanation. A limitations paragraph now states that the model learns statistical regularities of play, not intent, expertise or decision quality. The evaluation matches this aim: fidelity as a distribution, agreement between the generated modalities, and control through text.

Q2 — When the pass target changes, do defenders and teammates adjust? Yes, in the samples that follow the edit. We take the hardest edit, which swaps the receiver of the first completed pass with the teammate farthest from him, throughout the narration. At the frame where the new target receives the ball, we measure where the other players are and compare it with real receptions and with the same player at the same frame when the unedited narration is pinned (WC / League):

At the reception frameReal receptionsEdited narrationUnedited narration, same player and frame
Distance from the receiver to the nearest opponent (median)7.1 / 5.8 m7.4 / 6.2 m11.8 / 8.8 m
Same, after passes of 30 m or more8.7 / 8.5 m8.3 / 7.1 m—
Teammates within 15 m of the receiver, after passes of 30 m or more (mean)0.8 / 1.00.7 / 1.00.3 / 0.4
Shift of the defending team's centroid toward him since the last observed frame (median)—+3.6 / +3.3 m−0.6 / −0.6 m

Defenders close down the new receiver to the distance seen at real receptions, and the defending team shifts about 3.5 m toward him. Paired over windows, the edit brings the nearest opponent 4.1 / 3.5 m closer (95% CI 3.9–4.3 / 3.2–3.7). Teammates also move into support, nearly as many as at real long passes. The nearest-teammate edit of Table 3 changes the play less, and the defending team still shifts toward the new receiver (+0.4 m on both corpora). These measures cover the samples that follow the edit, 22% / 17% for this implausible request; in the others the model keeps a plausible play (Q5). We do not claim a perception–action model: joint generation learns how players move together in the data. As a quantitative reference we use what professional players do at real receptions (first column of the table); a study with human experts is future work.

Q3 — Do the alternatives include failed passes and interceptions? What can the model anticipate? Yes. In 42% (WC) / 72% (League) of test windows, the 20 samples include both plays that lose the ball (a defender holds it for at least three consecutive frames) and plays that keep it. The share of samples that lose the ball also anticipates the recorded outcome: used as a score for whether the recorded play has a turnover, it reaches an AUC of 0.72 / 0.76 (0.5 is chance), excluding windows whose first generated ball holder is a defender.

(2/2)

Q4 — Prediction length and narration accuracy. Agreement with the recorded play falls with the horizon, as expected in a stochastic setting such as a team sport, where many continuations are plausible. The model generates the fifty frames after the ten observed ones (Section 4), 8 s on League and 8.3 s on WC:

  • Counting frames where a player holds the ball, generated possession matches the recorded possessor in 69% / 72% of them in the first fifth of the horizon (ten frames), and in 9% / 11% in its last fifth.
  • The second narrated ball carrier matches the recorded one in 50% / 61% of samples, the third in 18% / 21%. The first (85% / 89%) can already be on the ball during the observed frames, which the narration also covers.

Beyond about 3 s the samples are alternatives rather than reconstructions, which is why fidelity is scored as a distribution. From the second fifth of the horizon on, the co-generated model agrees slightly more than JointDiff, which does not narrate (in the last fifth, 9.2% vs 7.5% of frames on WC and 10.9% vs 7.9% on League).

Q5 — What happens when the edit is unfavourable? The model mostly keeps a plausible play rather than producing an impossible one. Our least favourable edit gives a pass to the teammate farthest from its original receiver. At the last observed frame he stands a median 44 / 48 m from the passer, where real receivers stand more than 40 m away in only 1.1% / 1.4% of passes.

  • The model follows it in 22% / 17% of samples, against 4% / 3% without the edit (33% / 27% when the edit targets the last pass instead of the first).
  • When it does not, the original receiver gets the ball in 60% / 71% of samples.
  • The motion stays realistic: the ball-path Wasserstein distance is 3.2 / 3.2, against 15.5 / 6.1 for a text-guided simulator that follows the edit more often. So does the team shape: the Wasserstein distance of team width to real play is 0.24 / 0.10 for ours and 0.51 / 0.43 for the text-guided simulator; team compactness behaves the same way.

It does not, however, turn an unfavourable instruction into a failed pass: the turnover rate is unchanged (19.5% vs 19.0% WC, 31.7% vs 31.1% League, against the unedited reference narration on the same windows). We report both behaviours and leave consequence-aware editing (re-generating the narration after the edited event) to future work.

Narration length and first reception.

  • The shortfall is larger in words than in events: at ten steps our narrations reach 66–67% of the reference words (Table 3) and write 76% / 85% as many complete events as the reference (85% / 86% at fifty steps).
  • For a narration given as input, ball travel follows events, not words. For our own narrations on WC it depends more on when the narration is available during sampling than on its length: the same narration pinned from the first step raises the ball path from 0.91 (Table 3) to 0.96 (WC). The revision says "events" instead of "length" in the paper's tempo claims (details in our reply to Reviewer f6x3).
  • Immediate reception after the edited pass is the stricter criterion; Table 3 reports both.

Q6 — Related work. We thank the reviewer for both references; the revision cites them in Section 2. InfoGCN++ is close in structure: from a partially observed skeleton sequence it predicts a discrete output, the action class at every frame, together with continuous future poses, much as our model completes a scene from its observed frames with discrete events and continuous trajectories. M2D2M is close in sequential generation: it generates several text-specified motions with discrete diffusion and keeps their transitions coherent, as our model generates several events in order, together with their narration.

Dear reviewer, the authors have responded to your comment. Could you please take a look? Let us know if you have further concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.