acceptodds
Withdrawn from ICLR 2027

WorldTraces: A Data Attribution Framework for Predictive World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Data attribution measures how much each training sample shapes what a model predicts. However, it has not been explored in predictive world models that learn representations from large-scale sequential data such as trajectories and rollouts. We remark that there is a disconnection between existing training-data attribution methods and distinct behaviors learned by a world model such as JEPA. Specifically, the majority of existing data attribution methods are built around a differentiable loss against a ground-truth label, whereas JEPA's exponential moving average target and multi-step latent rollouts leave no single scalar output for a sample's influence to be traced back to. To fill this gap, we propose WorldTraces to measure training data's attribution in future-latent prediction, representation consistency, and temporal latent structure. It combines these responses into a query-conditioned ranking of training examples. While the architectures of existing world models varied differently from each other, we focus on JEPA because it has become a leading architecture for learning general-purpose world representations, making it a practically important test case for tracing which training data drives its learned dynamics and downstream behaviors. Experiments show that WorldTraces rankings predict the behavioral effects of grouped checkpoint-level forgetting-directed interventions across several subset sizes and maintain a consistent ordering across all three behaviors. WorldTraces aligns more closely with the effects observed under subset interventions and shows a sharper response when top-ranked examples are corrupted. It also avoids the need for subset retraining or stored training trajectories, with runtime scaling roughly linearly in the candidate pool size. Together, these results position WorldTraces as a practical foundation for training-data attribution of world models.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Training-data attribution, Predictive world models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iEztOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

WorldTraces scores candidate training examples by applying one norm-clipped gradient-ascent step on their training loss and measuring the resulting changes on reference queries. For V-JEPA 2, it keeps the target encoder fixed and measures future-latent prediction, consistency between transformed views, and changes in temporal latent transitions.

Weakness 2 is one of the main reasons for a lower score, I would increase the score if I misunderstood something there.

Critical strengths
  1. The problem is useful and well motivated. Separating prediction, representation consistency, and temporal structure gives more specific attribution targets and is desirable.

  2. The grouped-intervention results show a consistent ordering across prediction, representation consistency, and temporal structure.

  3. The IRIS results provide useful evidence that the method applies beyond JEPA.

  4. The sensitivity analysis is thorough, covering the reference queries, target-encoder coupling, and intervention parameter subspace.

Critical weaknesses
  1. Figures 3, 6, and 8 evaluate grouped versions of the same gradient-ascent intervention used to compute WorldTraces. This shows that individual scores predict grouped intervention effects, but we need to see whether they predict the effects of changing the training data. Figure 6 also uses this intervention rather than the subset-retraining target used in standard LDS [1].

  2. It is unclear whether all V-JEPA 2 candidates were used in the checkpoint’s pretraining. Section 4 includes WebVid in Mini-VM22M, but V-JEPA 2 I believe was trained on YT-Temporal-1B [2]. How was Mini-VM22M constructed, and how did you know if the videos were in pre-training if you did not use the same dataset?

  3. Minor: I think you should cite Influence-SSL [3], it also uses consistency between augmented representations to define self-influence in self-supervised models almost exactly as your equation 4.

4: Minor: The main V-JEPA 2 settings are not given. Appendix B.2 does not give the intervention step size, clipping threshold, or predictive prefix-suffix split.

  1. Grad-Sim and LESS use the JEPA training loss, while WorldTraces directly measures the behaviors used for evaluation. Comparing against a first-order approximation of the same behavioral responses, with the same rank aggregation, would help establish whether the gains come from the checkpoint intervention or simply from choosing different attribution targets.

[1] Park, Sung Min, et al. "Trak: Attributing model behavior at scale." arXiv preprint arXiv:2303.14186 (2023).

[2] Assran, Mido, et al. "V-jepa 2: Self-supervised video models enable understanding, prediction and planning." arXiv preprint arXiv:2506.09985 (2025).

[3] Harilal, Nidhin, et al. "Where Did Your Model Learn That? Label-free Influence for Self-supervised Learning." arXiv preprint arXiv:2412.17170 (2024).

Questions for authors
  1. Are the subset gradients in Figure 3 averaged as in Eq. (2), and is the intervention step size fixed across subset sizes? Because the Random effects grow approximately with subset size.

  2. How is Top–Bottom separation computed in Figure 4? The equal-weight result does not appear to match the difference between the corresponding Top and Bottom percentages in Figure 3.

  3. How are the optimizer states needed for LESS’s Adam-aware candidate updates obtained [4]? Does the IRIS implementation use Adam-adjusted updates or projected raw gradients?

  4. What are the total runtime and peak GPU memory for WorldTraces, Grad-Sim, and LESS under the candidate and query settings used in Figure 6?


[4] Xia, Mengzhou, et al. "Less: Selecting influential data for targeted instruction tuning." arXiv preprint arXiv:2402.04333 (2024).

Reviewer_h44TOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WorldTraces, a checkpoint-level training data attribution method for predictive world models. It constructs local, forgetting-oriented parameter interventions for candidate training examples and measures three types of responses: future prediction, representational consistency, and temporal structure. These responses are then aggregated into a relative ranking within the candidate pool. The authors primarily compare WorldTraces against adapted versions of Grad-Sim and LESS on V-JEPA 2, and conduct controlled cross-architecture validation using IRIS. The method requires neither retraining for each candidate nor storing training trajectories. Its ranking is an ordinal score relative to a fixed candidate pool and evaluation protocol, and its interventions measure local sensitivity rather than achieving exact unlearning.

Critical strengths
  1. The method requires neither retraining for each candidate nor storing complete optimization trajectories. The authors clearly describe the procedure and distinguish local parameter sensitivity from the causal effects of removing training data.

  2. The experiments include subset interventions, linear datamodeling scores, controlled data perturbations, and sensitivity analyses covering query sets, parameter subspaces, and behavioral readouts.

  3. Controlled continued training in the IRIS experiments ensures that the candidate samples actually participated in training, providing a useful test of the method beyond JEPA.

Critical weaknesses
  1. Both candidate scoring and subset evaluation use one-step gradient ascent and the same behavioral readouts. The results show that individual scores can predict similar subset interventions, but it remains unclear whether they reflect the samples’ actual contributions during training.

  2. Grad-Sim and LESS use training-loss gradients, while WorldTraces directly measures changes in the three behaviors used for evaluation. This difference may partly explain its advantage, especially for representation and temporal structure.

  3. Gradient clipping does not give every candidate or subset the same update norm. Some samples may produce larger behavioral changes simply because they cause larger parameter perturbations.

  4. The method computes gradients for each candidate and evaluates candidate–query pairs, which could be costly at scale. Linear scaling alone does not show an efficiency advantage; matched runtime and memory comparisons are needed.

Questions for authors
  1. Can WorldTraces rankings predict the effects of removing data and retraining in the IRIS setting?

  2. Does WorldTraces still outperform baselines that use the same behavioral targets?

  3. Do the rankings and subset results remain stable when all interventions use the same update norm?

Reviewer_44HxOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is interested in understanding how individual training samples contribute to the overall learned representation of world models. This is indeed very useful since we usually rely on scaler losses, gradients or optimization trajectories, which may not fully capture the different behaviors learned by a WM.

The authors provide and propose WorldTraces, which is a data attribution framework that can estimate the importance of training examples through their effect on the future-latent prediction, the representation consistency and the temporal latent structure.

The main backbone that the authors are considering is V-JEPA-2, but I believe that extending to other approaches shall be direct.

Critical strengths
  • I find the research direction that the authors are considering to be very interesting and relevant. I believe it is missing from the current literature on WMs. I believe that understanding which training experiences contribute to the predictive and representational capabilities of world models is an important problem.
  • The data attribution perspective is very interesting and the overall proposed framework is elegant. I think applying the gradient-scent intervention and measuring how the model’s behavior changes offer an intuitive mechanism for investigating the sensitivity of a pretrained model (even though it was previously applied in other settings).
  • The experimental results are encouraging. In particular, WorldTraces achieves a LDS Score better than the considered baselines.
Critical weaknesses
  • So the method applies the single gradient-step on a considered candidate sample and measure the resulting behavioral change. However, this measure the sensitivity, so I am not sure how relevant it is to the contribution that the candidate made during training. Note that similar question arises also in the explanaibility domain and for instance the un-learning domain. So I would be grateful to the authors if they clarify why they think there is a relationship here.
  • In the evaluation step, the proposed behavioral components measure principaly the latent prediction error, representation consistency, and changes in first-order latent transitions. It would have been also great to see if these quantities accurately reflect meaningful capabilities such as object understanding, physical reasoning or action prediction and planning which are usually why we consider World Models.
  • I also have some question about the practicality of the proposed framework in terms of complexity and the added time. Can the authors provide additional results about this ? and clarify their position?
Questions for authors
  • Could the authors clarify more explicitly what notion of data attribution WorldTraces is intended to estimate? And if they can also help understand how as a user we can interpret a high attribution score if it doesn’t not necessarily correspond to effect of removing that sample from the training dataset.
  • I am also wondering how the attribution scores relate to downstream capabilities of the world model. For instance, if we remove or corrupt highly attributed examples, do we observe measurable changes in action prediction or any other downstream task that the author can consider.
  • Could also the authors clarify how the approach could scale to much larger candidate pools?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.