acceptodds
Under review as a conference paper at ICLR 2027

Colouring motion: forward-process design for multi-agent trajectory diffusion

Abstract

Diffusion models are a popular choice for multi-agent trajectory generation. Typically, the noising process uses stochastic white noise, even though real trajectories evolve smoothly over time, with the position at one time step affecting the next. We argue that this mismatch presents an extra burden to the denoising process. In this paper, we replace white noise with a range of temporally correlated noise fields that can be tuned to suit the underlying process being modelled. Sport provides us with large standardised datasets to work with. Compared to the same model trained with white noise, correlated noise models reduced averaged scene error by 5.1% on association football and 2.3% on American football. Our best model beats the current state-of-the-art on a public sport trajectory generation benchmark, improving averaged and best-of-20 scene error on Bundesliga football by 9.6% and 5.9%. We further explore the properties that make temporal noise fields effective, showing that slower-varying noise helps performance. The training objective is equally important. Whitening the loss has a negative effect on the results compared to plain squared loss. We also demonstrate that at 25Hz, varying the noise by agent type makes the ball more realistic. These results establish the importance of tuning the forward process in multi-agent diffusion models.

Primary area:
learning on time series and dynamical systems
Keywords:
diffusion models, forward process, correlated noise, multi-agent trajectory generation, time series, sport analytics, generative modelling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_86e1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies forward-process design for multi-agent trajectory diffusion. Instead of using temporally independent white Gaussian noise, the authors introduce temporally correlated noise fields and investigate how the temporal structure of the corruption process should be matched to the dynamics of the underlying trajectories. The paper further proposes agent-type-specific forward processes, assigning different noise fields to players and the ball. Experiments on football, American football, and basketball trajectory generation show that appropriately structured noise can reduce displacement error and improve some aspects of motion realism

Critical strengths
  1. The paper studies an important but relatively under-examined design choice in trajectory diffusion.
    Rather than modifying the denoising backbone, the work asks whether the forward corruption process itself should reflect the temporal structure of the modeled data. The empirical setup is clear and the central hypothesis is easy to understand.

  2. The experimental analysis is systematic and informative.
    The paper evaluates several noise fields with different temporal frequency characteristics and analyzes their behavior using low-frequency power and participation ratio. The sampling-rate experiment is particularly useful: the benefit of low-pass noise is substantially larger at 25 Hz than at 6.25 Hz, which supports the authors’ argument that the usefulness of correlated noise depends on the temporal structure of the data.

  3. The agent-type-specific forward process is an interesting extension.
    Modeling players and the ball with different corruption processes is well motivated by their different dynamics, and the experiments show that the bespoke ball process can improve acceleration realism at high sampling rates.

Critical weaknesses
  1. There is a substantial gap between the generality of the paper’s framing and the breadth of its empirical evidence.
    The paper is framed as a general contribution to forward-process design for multi-agent trajectory diffusion, with sports presented primarily as a test bed rather than as the scope of the method. The abstract states that the noise fields can be tuned to the underlying process being modeled, and the conclusion formulates broader principles about matching corruption to temporal structure, sampling rate, and agent-specific behavior. However, all empirical validation remains within sports tracking domains. Although the paper covers multiple datasets, these settings share similar characteristics: continuous spatial motion, human players, and ball dynamics. It is therefore unclear whether the proposed design principle transfers to trajectory domains with abrupt maneuvers, contact events, different interaction structures, or substantially different temporal spectra. In my view, the current experiments support a domain-specific empirical observation more strongly than a general modeling principle.

  2. The core correlated-noise formulation has significant conceptual overlap with prior diffusion work.
    Non-isotropic Gaussian noise, temporally correlated noise for sequential data, and data-dependent or structured priors have all been explored previously. The main novelty therefore appears to lie in the systematic application to sports multi-agent trajectories, the sampling-rate analysis, and the agent-type-specific extension, rather than in the broader idea of replacing white noise with structured corruption. The paper would benefit from positioning this boundary more explicitly.

Questions for authors
  1. Can the authors validate the proposed forward-process design principle on at least one non-sports trajectory domain?
    For example, pedestrian trajectory forecasting, autonomous-driving trajectories, human motion, or robot trajectories would provide a substantially stronger test of the claim that the corruption process should generally be matched to the temporal structure of the data.
5 comments

We thank the reviewer for this comment, which we agree with and have acted on. In the revised version we have added experiments on two public datasets outside sport and adjusted the framing to match the evidence.

Response to Weakness 1 and Question 1

New experiments. We evaluated the velocity-level field against white noise on:

  1. THÖR-MAGNI (Schreiter et al., 2025): people walking in a motion-capture lab, in groups and carrying objects, sometimes alongside a teleoperated robot. It has no ball, a different interaction structure (people and a robot rather than two teams), and up to 8 agents.
  2. Juvenile zebrafish groups (Romero-Ferrero et al., 2019; Heras et al., 2019): non-human agents that swim in bursts. Their motion is much more abrupt than that of human players, which speaks directly to the reviewer's question about abrupt manoeuvres.

We used the velocity-level field with the same damping time (τ = 20 frames) as on the sports data, with no retuning per dataset. Both datasets were resampled to 25 Hz and then downsampled again to 6.25 Hz, so we could also test the sampling-rate claim outside sport. We found similar improvements on the non-sport datasets as we did on the sport datasets.

Results.

  • Zebrafish: velocity-level noise reduced averaged scene error against white noise at all three training seeds and at both rates: by 2.3–3.0% at 25 Hz and by 0.9–1.4% at 6.25 Hz. At every seed, the reduction in error at 25 Hz was more than twice the reduction in error at 6.25 Hz, and confidence intervals between the groups do not overlap, demonstrating a real difference between the two.
  • THÖR-MAGNI: velocity-level noise reduced error at all three seeds, by 2.1–5.1% at 25 Hz. At 6.25 Hz, the reduction was smaller and less certain: 0.3–5.0%, with the intervals between the groups overlapping in two of the three seeds. The reduction in error was larger at 25 Hz at two of the three seeds. As we saw with sport, the improvement is sensitive to frequency. Further, at 25 Hz the robot's error fell by about 16% at two of three seeds, compared with about 2% for the people. This is consistent with our argument that agent types with different dynamics respond differently to the forward process.

Response to Weakness 2: overlap with prior structured-noise work

We agree with the reviewer's reading of where the novelty lies, and we have revised the paper to draw this boundary explicitly. We have made our position against previous work clearer in our introduction and related work.

Reviewer_86e1OpenReview ↗

Response to Rebuttal

I appreciate the authors' substantial additional experiments, which meaningfully address my concern regarding generalization beyond sports trajectories. I also acknowledge that the paper provides valuable empirical insights into the relationship between noise structure, sampling rate, and trajectory generation performance.

However, my concern regarding methodological novelty remains. Given the existing literature on temporally correlated noise and non-isotropic Gaussian diffusion, the core approach largely builds upon established formulations. The agent-type-specific forward process, while intuitively motivated, essentially assigns manually designed noise distributions to predefined agent types, without introducing a fundamentally new diffusion mechanism or a principled method for learning or adapting these distributions.

Therefore, while I recognize the strength of the empirical analysis and the improvements made during rebuttal, I remain unconvinced that the methodological contribution is sufficiently novel. I will maintain my original rating.

Thank you for your reply, and for confirming that the non-sport experiments address the concern about generalisation.

The focus of the paper is the design and refinement of the forward process, which you acknowledge as an important but relatively under-examined design choice. It states that correlated and non-isotropic noise are established, and that Ge et al. (2023) use the same AR(1) construction as our velocity-level field. In your review, you placed the novelty in the systematic application to multi-agent trajectories, the sampling-rate analysis and the agent-type-specific extension. That is how the revised paper frames its contribution.

Three of its findings change how structured noise should be used for trajectories:

  1. The training objective decides whether correlated noise helps. Earlier work with structured noise trains with the likelihood-based (whitened) loss (Lee et al., 2022; Biloš et al., 2023). On Bundesliga, whitening the loss of the velocity-level field raised averaged scene error by 7.9% over three seeds, which removed the whole benefit. The plain squared loss kept it. Appendix D derives why.
  2. A field can be checked before training. Low-frequency power ordered the six fields that trained (Spearman's ρ = −0.94 over seed means). A new sensitivity analysis over eight damping times (Appendix F) shows that the field outperforms white noise at a participation ratio of 1.76 and is worse than it at 1.49. Both statistics are computed from the noise field alone, before training.
  3. The benefit depends on the sampling rate. On public zebrafish data, the error reduction at 25 Hz was more than twice that at 6.25 Hz across all three seeds.

On the agent-type-specific process, you are right that the fields are designed, not learned. This was deliberate. The fields have no learned parameters, so every change in error can be attributed to the field itself. The choice of field is not arbitrary. Point 2 gives criteria that can be computed before training, and the paper shows that each agent type's field should be set in frames for its dataset and sampling rate. Learning the fields is a natural next step, and point 1 bears directly on it: training through the likelihood bound removed the benefit of correlated noise, so a field learned that way would not necessarily lower displacement error.

Reviewer_wfAYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper replaces independent white noise with temporally correlated noise to better match continuous motion. It also introduces a specialized noise field that mimics impulsive ball strikes. On the JointDiff football benchmark, the best model reduces average scene error by 9.6% and best-of-20 error by 5.9%. The experiments also show that the benefits depend on sampling frequency, and lower positional error does not necessarily imply more realistic motion.

Critical strengths
  1. Distinguishing smooth player motion from impulsive ball motion improves the ball’s acceleration distribution and shows why positional error alone is insufficient to assess generation quality.
  2. The paper separately examines correlated noise, v-prediction, and loss functions. Acceleration distribution and burstiness metrics further clarify both the benefits and limitations.
Critical weaknesses
  1. The specialized ball noise improves realism at 25 Hz but increases positional error; at 6.25 Hz, both metrics worsen. The paper does not explain when this design should be preferred.
  2. Whitening the loss removes the benefits of correlated noise, while ordinary squared loss preserves them. This is an important finding, but the underlying reason remains unclear.
Questions for authors

The acceleration-level noise is smoother but produces an error of 38.8 m. You attribute this to its participation ratio of only 1.0. Is there further experimental evidence supporting this explanation? When designing new noise fields, could participation ratio or another metric help distinguish beneficial smoothness from excessive smoothness that makes denoising difficult?

4 comments

Response to Weakness 1: the accuracy–realism trade-off

We thank the reviewer for this point. We have added some practical guidance to our discussion (Section 5) on when to prioritise realistic agent behaviour over displacement error. The choice depends on how the model outputs will be used. In cases where the behaviour of a specific heterogeneous agent type is being analysed, realism is more important and a bespoke noise field is appropriate.

Response to Weakness 2: why the whitened loss fails

We agree that this needed an explanation and have added it to our discussion (Section 5, with the derivation in Appendix D). In brief, the variational bound does not fail. It optimises a different quantity from the one displacement error measures. The whitened loss weighs frame-to-frame changes in the error far more than the error itself. Correlated noise changes less frame-to-frame than white noise, so it is affected differently. This undoes the error reduction in our use case.

Response to Question: a participation-ratio threshold

We thank the reviewer for this question. We cannot give an exact threshold, but we have performed more experiments to plot out the relationship between participation ratio and displacement error. We found that a participation ratio of approximately 1.5 in this case resulted in a model that produced similar error to white noise. Any lower resulted in a larger error than white noise.

Thank you again for your review. Since our response, the participation-ratio experiment has finished, and the revised paper reports it (Appendix F, with a table and figure).

We retrained the velocity-level model at eight damping times, giving participation ratios from 4.55 to 1.07. At a ratio of 1.76, the field still outperformed white noise (−1.6% averaged scene error). At 1.49, it was 1.5% worse than white noise, and error rose steeply below that, doubling at 1.07. The ratio can therefore identify a field that is too smooth before training.

We hope this answers your question, and would be glad to answer any others.

The authors partially answer my questions and I will consider my final score.

Thank you for your engagement with our paper and your further consideration. It is appreciated.

Reviewer_vLrROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

task Given the first few seconds of a football play, predict how all 22 players and the ball will move over the next 6.4 seconds (40 positions per agent at 6.25 positions per second), using a diffusion model, i.e. a generator that is trained to turn random noise back into realistic trajectories.

problem Standard diffusion models are trained by adding independent random noise to every position at every time step, but real movement is smooth (where you are now almost fixes where you are next), so, as claimed by this work, the diffusion model wastes capacity learning to remove a kind of jitter that never occurs in real data.

key insight Instead of independent noise, add noise that is itself smooth in time (each step's noise is a filtered version of the previous step's noise, like noise in velocity rather than in position), keep everything else about the model unchanged, and optionally give the ball a "spiky" noise that mimics kicks.

Result On the public Bundesliga benchmark the smooth noise reduces the average position error by 5% compared with the same model trained with ordinary noise (2% on American football, no change on basketball) and gives the best numbers reported so far; the gain is about five times larger on a private dataset recorded at 25 positions per second; the spiky ball noise makes the ball's accelerations look statistically more like real kicks but makes its position error 3% worse.

What I can somehow learn Shaping the training noise to match how the data moves helps, and helps more at higher frame rates, but on public data the effect is a few percent on a single model, the headline comparison with prior work also includes other changes the authors made, and without code, the private data, or any animation of generated plays I cannot tell whether the improvement is visible to a human.

Critical strengths
  1. A clean, honest controlled study. The denoiser, noise schedule and evaluation code are held fixed. Therefore the effect of the noise field is isolated (Table 1), and negative results (acceleration-level field, whitened loss, NBA, ball displacement error) are reported plainly. This is why I rate soundness as having only minor concerns.

  2. One genuinely informative observation. The gain from correlated noise is about five times larger at 25 Hz than at 6.25 Hz (Table 4), which tells practitioners when such priors matter. This is interesting!

Critical weaknesses
  1. The contribution is incremental and the effect is small, narrow and partly unreproducible. Correlated noise priors for diffusion are established in previous works (Ge et al. 2023; Chang et al. 2024 etc), and the gain here is 2–5% over the authors' own white-noise model on one architecture (Bundesliga/NFL, ~0 on NBA), mostly single-seed, while the agent-specific ball field, the one new design idea, makes the primary metric worse (+3.1% avgSADE, Table 5). The strongest result (sampling rate, Table 4) and all ball-realism results rest on a commercially licensed dataset (authors have claimed they will not release that) with no code released, so the generality claimed in the title cannot be checked.

  2. The qualitative evidence is too thin to judge what the gains mean. The only qualitative material is Figure 3 (Appendix) I feel it not obvious to judge. Would the author provide some video demo as those in JointDiff?

Questions for authors
  1. Please provide animations, or at minimum multi-seed, multi-draw static plots, of generated scenes for the same context under white noise vs the velocity-level field, and of the ball trajectories with and without the ball-specific field, so the nature of the improvement can be seen rather than inferred from aggregate errors.

  2. How many training seeds underlie the ± values in Table 1, and can the Table 3 sweep be repeated with at least three seeds so the ordering claim (velocity-level vs pink) is testable? The text (p.6) says the ordering is not stable across seeds.

3 comments

Response to Weakness 1: incremental, small, narrow and partly unreproducible

We thank the reviewer for a careful review. We agree that correlated noise priors have been used in previous work and we position our novelty more carefully against this in our introduction and related work. We now also report three training seeds for Table 1 and Table 3. On the Bundesliga dataset, the velocity-level noise field improves on the white-noise model as follows (Table 1; metres, lower is better; mean over three training seeds, with the seed standard deviation in brackets):

Metric (m)White noiseVelocity-levelImprovement
minSADE2.552 (0.017)2.324 (0.033)8.9%
avgSADE3.485 (0.029)3.308 (0.016)5.1%
minSFDE5.154 (0.037)4.705 (0.067)8.7%
avgSFDE7.761 (0.070)7.471 (0.039)3.7%

Each improvement is not explained by the seed-to-seed variance.

We agree that the ball field does not improve displacement error and we do not claim that it does. Its purpose is realism and at 25 Hz the ball's own displacement error is unchanged. Its benefit is acceleration metrics that more closely match patterns in the ground truth data.

The sampling-rate result no longer rests on licensed data. We repeated our experiments on two public non-sport datasets (please see our response to Reviewer 86e1). On public zebrafish data, velocity-level noise reduced error more than twice as much at 25 Hz as at 6.25 Hz, at each of three seeds. On public THÖR-MAGNI data the error reduction was larger at 25 Hz at two of three seeds.

Regarding code, the training code cannot be released because the project is industry-funded. However:

  • the denoiser is that of JointDiff, fully specified in Appendix A;
  • the noise field is a drop-in replacement for the white noise of Equation 1;
  • every benchmark number is produced with JointDiff's released evaluation code;
  • we have now released code that generates each noise field and computes the statistics of Table 3, with a notebook that checks them (Reproducibility Statement): https://anonymous.4open.science/r/colouring-motion/.

Response to Weakness 2 and Question 1: qualitative evidence

We agree, and have added the material requested in a supplementary repository (https://anonymous.4open.science/r/colouring-motion/). This includes videos that compare the velocity-level and white-noise models of Table 1. These can be found inside the 'videos' folder in the top left. The differences in their outputs are annotated. The videos illustrate that white-noise-model agents change direction less than the ground truth, i.e. they continue straight more than they should. The velocity-level model agents better follow the ground truth. We also show how the acceleration of the ball more closely matches the ground truth when the ball is given its own field.

Response to Question 2: seeds

We have added more visibility to the number of seeds used in different parts of the paper. All four of our rows in Table 1 use three seeds. Table 3 has been repeated at three seeds. The reviewer is right that the velocity-level versus pink ordering is not stable. In the ablation of Table 3, their means are 3.302 ± 0.023 and 3.304 ± 0.016 m respectively, which are indistinguishable at this number of seeds. We have therefore withdrawn the ordering claim and describe the two as a plateau (Section 4.2). The claim we do make is that error falls as low-frequency power rises across the six fields that train. We use the velocity-level field in our best model because, in that configuration, it had 1.0% less error than pink at two of three seeds. We present this as a marginal choice, not a finding.

Thank you for your effort in rebuttal.

My concerns are partially addressed. And I see the video (it is in an outside link).

However, I still feel the contribution is a bit incremental. The novelty is partially limited. Therefore I maintain my original score.

Thank you for your reply, and for taking the time to watch the videos.

On novelty, the focus of the paper is the design and refinement of the forward process. Its contribution is an empirical account of how to design the forward process for multi-agent trajectories. Three findings from it change how such noise should be used:

  • The benefit needs the plain squared loss. Whitening the loss, as the likelihood bound implies and as earlier structured-noise work does, removed the whole benefit (+7.9% averaged scene error, three seeds; Appendix D).
  • A field that is too smooth can be identified before training. A new sensitivity analysis over eight damping times shows that the error reduction relative to white noise disappears as the participation ratio falls from 1.76 to 1.49 (Appendix F).
  • The sampling-rate effect persists on public non-sport data. On zebrafish, the error reduction at 25 Hz was more than twice that at 6.25 Hz across all three seeds.

On seeds, Table 2 is now also at multiple seeds. On NFL, velocity-level noise reduced averaged scene error against white noise by 2.3% across three seeds (Table 2).

You wrote that your concerns were partially addressed. If any besides novelty remain, we would be glad to answer them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion

We thank all three reviewers. The revised PDF is uploaded. The discussion has resolved the following:

  • Generality beyond sport (86e1, vLrR). We added two public non-sport datasets: pedestrians walking, sometimes alongside a robot (THÖR-MAGNI), and zebrafish. Reviewer 86e1 agreed that this addresses the concern.
  • Licensed data behind the sampling-rate result (vLrR). The result is now demonstrated on public zebrafish data across three seeds.
  • Seeds and qualitative evidence (vLrR). Tables 1, 2, and 3 are at multiple training seeds. Videos and code that generates each of our noise fields are in the anonymous repository.
  • The whitened loss, the accuracy-realism trade-off, and the participation-ratio limit (wfAY). We added a derivation (Appendix D), practical guidance (Discussion), and a sensitivity analysis over eight damping times (Appendix F).

Our paper focuses on the design and refinement of the forward process and is positioned accordingly. Its contribution is empirical: which temporal structure suits multi-agent trajectories, how to check a field before training, how the benefit depends on sampling rate, why it needs the plain loss rather than the likelihood bound, and that the forward process can differ by agent type for different use cases.

With velocity-level noise, our model set a new state of the art on the Bundesliga benchmark (9.6% on averaged and 5.9% on best-of-20 scene error). The field alone, a drop-in change with no learned parameters, reduced averaged scene error by 5.1% against the same model with white noise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.