acceptodds
Under review as a conference paper at ICLR 2027

RoboJEPA: Scaling Robotic Latent World Models

69 views

Abstract

Latent world models have shown a remarkable ability to predict future states and to plan in the real world. In practice, however, we lack a principled way to estimate how their capabilities scale with model size, data, and compute, an open problem that slows progress in the field. In this work we present RoboJEPA, a world model based on the Joint Embedding Predictive Architecture (JEPA) and trained on a large-scale dataset spanning 12 robotic embodiments. We show that RoboJEPA's imagination error, the error of its latent rollouts, follows a second-order power law in compute, allowing us to predict model quality well beyond the scale at which the law is fit. We further show that downstream robotic planning performance improves predictably with compute, and that imagination error is strongly correlated with it, making it a reliable proxy for real-robot evaluation. Finally, we demonstrate that latent world models can be deployed zero-shot as robotic agents, planning toward a single goal image to solve tasks requiring long-horizon planning on real hardware. We will release all model checkpoints together with our training and robot deployment code. To our knowledge, this is the first work to establish scaling laws for multi-embodiment robotic world models trained on real robot data, and RoboJEPA, at 8B parameters, is the largest JEPA predictor model trained to date.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, scaling laws, robotics, planning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_x8RrOpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates the scaling relationship between training compute and prediction error in action-conditioned latent world models. To explore this, the authors developed the RoboJEPA model family, ranging from 22M to 8B parameters, by adapting the V-JEPA 2-AC architecture and training the predictors on large-scale, multi-embodiment robotic datasets. The authors then systematically analyzed prediction errors across different model sizes and training compute budgets and proposed a second-order power law as an empirical scaling function. Among the scaling functions evaluated, the second-order power law achieves lower extrapolation error when predicting the latent prediction error of larger models. Furthermore, the authors examined the relationship between model scale and downstream control performance through both simulated and real-world robotic planning experiments.

Critical strengths

S1: Building on the V-JEPA 2-AC architecture, this paper introduces the RoboJEPA model family, ranging from 22M to 8B parameters, and trains the predictors on large-scale, multi-embodiment robotic datasets.

S2: Leveraging RoboJEPA, the authors provide an analysis of the scaling relationship between training compute and prediction error and propose a second-order power law as an empirical scaling function. Among the evaluated scaling functions, it achieves lower extrapolation error when predicting the errors of larger models.

Critical weaknesses

W1: Overclaim. In the Introduction (L45–47), the authors state a problem that previous latent world models were typically trained at a single scale and that how model quality changes with data size, model parameters, and training compute has not been sufficiently characterized. And the problem is also the main research problem addressed in this paper. However, in the Related Work section, which is placed in the Appendix, the authors have already discussed prior studies [1]–[4] that investigate different aspects of world-model scaling. In particular, [2] explicitly investigated power-law relationships involving model size, data size, and compute in world modeling. Therefore, the Introduction overclaims the gap in existing research. Since the Related Work is placed in the Appendix, despite its importance for positioning the paper against existing studies, readers cannot clearly understand the actual novelty of this work from the main text alone. After considering the related work, the positioning of this paper differs substantially from that presented in the Introduction.

W2: Insufficient Positioning Against Prior Work. In the Introduction (L47–50), the authors mainly categorize existing scaling studies into two groups:

  1. Generative video models or interactive environments outside robotic control.
  2. Policy scaling rather than world-model scaling.

However, the Related Work in the Appendix shows that previous studies [1]–[4] have already investigated different aspects of world-model scaling, some of which are directly related to the research scope of this paper. Furthermore, other recent studies, such as WestWorld [5], have investigated world-model scalability across heterogeneous robotic systems, including how model performance changes with the number of robotic environments and generalization to unseen robotic systems. Therefore, the Introduction does not sufficiently position this work against existing studies. Related work on world-model scaling and multi-embodiment world models is not adequately discussed in the main text, although it is important for distinguishing the existing research from the contributions of this paper.

W3: Ambiguous Method Definition. The paper describes Eq. (5) as a per-token l1l_{1} objective, but the equation uses summation without normalization, which is inconsistent with the description. According to L239–241, the complete loss definition, including per-frame masking and patch normalization, is provided in Appendix D. However, Appendix D (L1713–1717) states that the loss is only defined in Eq. (5), leaving the complete loss definition inconsistent between the main text and the Appendix. Furthermore, Appendix C.5 (Eq. (12), L1620–1627) defines mt,vm_{t,v} and states that it is used in Eq. (5), but this mask does not appear in Eq. (5).

W4: Unreasonable Results.

  1. In L345–346, the authors claim that they verify that the improvement translates into a better success rate. However, the corresponding results in Fig. 2 report normalized cumulative reward rather than success rate. According to the definition in Appendix H.4, the reward is based on the distance to the goal, rather than binary task success. These two metrics are not interchangeable. Even if the reward reaches 0.9, the episode may still be unsuccessful if the goal has not been reached.
  2. Fig. 4 and Table 3 report the scaling of success rates in real-world experiments. However, according to the evaluation protocol in Appendix H.2, each RoboJEPA model–task combination is evaluated over 30 episodes. If each episode is classified as either successful or unsuccessful, the success rate should change in increments of approximately 3.333.33\\%. Similarly, each VLA model–task combination is evaluated over 50 episodes, so the success rate should change in increments of 22\\%. However, why does RoboJEPA-4B report a success rate of 2121\\%, while π0.5\pi_{0.5} reports 55\\% in Table 3 and Table 12? These experimental results require clarification, as the reported success rates are inconsistent with the stated evaluation protocol.

[1] Terver, B., et al. What Drives Success in Physical Planning with Joint-Embedding Predictive World Models? (TMLR 2026)

[2] Pearce, T., et al. Scaling Laws for Pre-training Agents and World Models. (ICML 2025)

[3] Baldassarre, F., et al. Back to the Features: DINO as a Foundation for Video World Models. (arXiv 2025)

[4] Huang, W., et al. PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation. (CVPR 2026)

[5] Wang, Y., et al. WestWorld: A Knowledge-Encoded Scalable Trajectory World Model for Diverse Robotic Systems. (ICML 2026)

Questions for authors

Q1: How is the loss normalized in Eq. (5)? How is the mask mt,vm_{t,v} defined in Appendix C.5 incorporated into Eq. (5)?

Q2: Why does the last input column in Fig. 1 contain aτ+Ka_{\tau+K}, while the corresponding output is still z~τ+K\tilde{z}_{\tau+K}?

Q3: Appendix B.4 states that the maximum action input width is 64 and explains that the largest DROID joint-velocity action-key chunk is 8 × 8, indicating that each joint-velocity command is 8-dimensional. However, Appendix C.6 states that when DROID is downsampled from 60 Hz to 5 Hz, 12 controller commands are concatenated into a single world-model action. If each joint-velocity command is 8-dimensional, the resulting action dimension would be 12 × 8 > 64, which contradicts the definition in Appendix B.4. Is there an additional downsampling step?

Q4: The main experiment supporting the claimed understanding of object dynamics is the Object Push task in Fig. 3(d), which reports success rate. However, the main text defines task success as moving the object toward a target position, while Appendix H.4 specifies the initial distance between the object and its target. Does any movement toward the target count as success, regardless of the distance moved, or must the object reach a specified distance threshold? If success only requires movement in the correct direction, the results provide limited evidence for the claimed understanding of object dynamics.

Q5: The start-goal definitions for ObstacleReach are inconsistent between Appendix H.4 and Table 14. Which definition was actually used in the evaluation?

3 comments

We thank the reviewer for a detailed review. We would like to address some of the concerns:

W1&W2. We would like to clarify that the novelty we are claiming is that our work is the first one to build a scaling law for multiembodiment world models. We agree that the specific wording that we used in L45-L47 does not exclude unambiguous interpretation, therefore, we updated the paper with an unambiguous wording.

We would like to highlight that neither works [1-5], nor any prior work which we know builds extrapolatable scaling laws for multiembodiment robotics. The closest work to ours is [2] (Pearce et al. 2024) which presents an extrapolatable single embodiment scaling law in their Appendix and connects it to pixel generation quality of their robotic world model (but not to the planning performance of the WM). Please note that going from single embodiment to multiembodiment training is highly non-trivial.

Additionally, we would like to emphasize the difference between a scaling law and an empirical scaling trend, where the former fits a parametric curve on the latter and it ensures that the fitted curve extrapolates (potentially redefining the scaling recipe toward an extrapolatable one). We believe the empirical trend is a prerequisite for the scaling law and an extrapolating scaling law requires a more rigorous study that finds optimal hyperparameters for compute-optimal scaling. Works [1,3-5] build empirical scaling trends for single or multiembodiment robotics while our work builds world model scaling law for multiembodiment robotics for the first time.

On related work. We agree that having a discussion of related works in the main text rather than the appendix would help the readers better understand the positioning of our work. Since our related work section is almost 2 pages long, we updated the main text with a shorter version of the related works section that discusses critical works (such as [1-5] and more) and refers to the longer related works section in the appendix.

To summarize, our main contribution is the first scaling law for multiembodiment robotics while past works only either do single embodiment scaling laws, build single or multiembodiment scaling trends, or build world models at single scale.

W3&Q1&Q2. We would like to correct the reviewer regarding the validity of the Equation (5). Since zt(v)z_t^{(v)} is defined as a tensor of size [Hp,Wp,De][H_p, W_p, D_e] (see L161), therefore, for example ∣∣z^t(v)−zt(v)∣∣1||\hat{z}_t^{(v)} - z_t^{(v)} ||_1 becomes the sum of all absolute elementwise differences between two tensors containing the sum over patches. This means that Equation (5) is consistent with respect to its definition in L233-241. The lines L1620–1627 describe the loss masking that is used in the implementation of the loss from Equation (5). The loss masking is needed since different training sequences may have different lengths so masking each sequence is necessary because it is padded. We believe this is a full definition of the loss so we removed the reference to the expanded loss We updated the section C.5 to properly refer to Eq (5). We thank the reviewer for spotting a typo in Fig. 1 which we fixed.

W4. 1. We would like to provide clarification regarding the planning result in RoboCasa. Each of the 4 tasks we test, we use the success metric (either discounted return, final reward, or success rate) that most faithfully reflects the progress toward the goal. In the reaching task with the perfect match success criterion, the optimal agent would be unsuccessful because of physical limitations of RoboCasa’s simulated franka robot that only drives it 5mm to 1cm from the goal. We updated the paper motivating the choice of each performance metric used for each RoboCasa task.

W4. 2. Fig. 4, and Tables 3 and 12 provide real world evaluation results. However, sometimes these evaluations fail due to external reasons such as hardware errors. Due to a significant cost of real world evaluation, we ran the full benchmark (8 models, 3 tasks, 30 episodes) only once. We updated the paper with the new table with valid episode count.

Q3. Appendix C.6 contains a mistake which we corrected. For droid commanded actions we use 7.5Hz control and 5Hz control for droid state-delta action formulation (see Appendix C.5, C.6). The original description of B.4 is correct.

Q4. We use the threshold-based success criterion where success = final_reward > 0.1. We believe that our conclusion stands for this definition, since models under 1e22 FLOPS mostly keep the robot at init (for push, init and goal have the same robot pose) and only start moving the object and returning after that amount of compute.

Q5. We thank the reviewer for pointing this out. Table 14 is correct, and we have corrected the inconsistency stated in Appendix H.4.

We hope that we have resolved the reviewer’s concerns. If that is the case, we are kindly asking the reviewer x8Rr to increase the score of our work.

Thank you for the detailed response and the revised manuscript. I appreciate the improvements to the Related Work and the clarification of the implementation details. The new episode counts and confidence intervals also resolve my earlier concern about the numerical consistency of the reported success rates.

However, I remain concerned about the main contribution and its supporting evidence.

First, Section 3 states that the scaling-law candidates are fitted on compute-optimal frontier points, whereas Appendix K describes fitting BNSL and UNSL on 47 small-model checkpoints and evaluating them on 33 large-model checkpoints. It is still unclear how these checkpoints relate to the frontier points and whether all candidate functions are compared under the same protocol. Furthermore, the 4B and 8B models are used to select the best functional form, so the selected scaling law has not been independently validated beyond this selection.

Second, I appreciate that the authors have revised the claim about imagination error being a reliable proxy for planning performance. However, this also limits the practical implications of the main result. The paper establishes an empirical scaling relationship for latent prediction error within a specific predictor family, but does not show that the resulting law can reliably predict downstream robot control performance or guide broader world-model training decisions.

Finally, the comparison between short- and long-horizon training changes both the rollout horizon and input resolution. Without separating these factors, it is difficult to determine how much of the improvement comes specifically from longer-horizon training.

I recognize the scale of the experiments and the improvements made during rebuttal. Nevertheless, I remain unconvinced that the current evidence establishes a sufficiently general or practically significant scaling-law contribution. These concerns continue to affect my overall assessment.

Thanks for the revised assessment and for increasing the score. Let us elaborate the new concerns below:

1.1 We thank the reviewer for finding a reporting gap. All four candidates use the same protocol (fit on 47 checkpoints and extrapolate to larger 33 frontier checkpoints). We updated the Appendix Section K to properly reflect that.

1.2 Please note that training world models beyond 8B parameters would require a substantial amount of the training compute, on the scale of training large language models of similar size (e.g., Qwen3 [1]). We believe validating the selected scaling law at those scales is a valuable direction for future work. As outlined in the paper, our current data setting is approaching saturation at the largest scale, so this would also require a substantially larger training dataset.

  1. We would like to point the Reviewer’s attention to the fact that all scaling laws are specific to not only their domain but also to parameterization, and individual architecture choices [2]. For example, the diffusion world modeling formulation yields scaling that is specific to that formulation and, to the best of our knowledge, it is not known to transfer to other formulations. At the same time, predicting the planning performance across world model architectures, training formulations (e.g. stochastic diffusion-based or deterministic feedforward), or other broad world modeling decisions is a highly non-trivial problem that is beyond the scope of this work. In this work we only focus on bridging the planning performance to the world model quality measured by the prediction error or the training compute within the same model family.

The paper establishes an empirical scaling relationship for latent prediction error within a specific predictor family, but does not show that the resulting law can reliably predict downstream robot control performance.

We would like to clarify that the widely accepted view in the scaling laws community is that downstream performance metrics do not extrapolate reliably for large-scale models [3, 4] (as we discussed it in the response to the Reviewer 6KGA). In the revised version of the paper we showed that our domain is similar as it has several phase transitions of the downstream performance. To this reason, we decided to not fit the scaling laws for downstream performance and only qualitatively predict that the downstream performance improves (in the form of the correlation trend between downstream performance and compute or downstream performance and prediction error).

We believe this places only a minimal limitation on the practical implications of the main result, since prediction error is strongly linked to training compute, which deterministically predicts prediction error for compute-optimal models.

  1. As described in Section 3.2, long-horizon rollouts are introduced only during the cooldown phase; the main training stage always uses short-horizon training. Depending on total training compute, long-horizon training accounts for between 33% and 2% of training compute, since the number of cooldown steps is fixed while the main training stage varies in duration. Despite this variation, we observe a consistent improvement across scales (Figure 35), suggesting that the long-horizon cooldown amplifies capabilities developed during the main training stage. Although our final experiments change both input resolution and rollout horizon during cooldown, in earlier experiments we did not observe improvements in long-horizon prediction from higher resolution alone. In other words, we believe the long-horizon, multi-resolution cooldown is too short to develop long-horizon prediction as an entirely new capability.

[1] Yang, A., et al. (2025). Qwen3 Technical Report. arXiv preprint arXiv:2505.09388. https://arxiv.org/abs/2505.09388

[2] Everett, K. E., Xiao, L., Wortsman, M., Alemi, A. A., Novak, R., Liu, P. J., Gur, I., Sohl-Dickstein, J., Kaelbling, L. P., Lee, J., & Pennington, J. (2024). Scaling Exponents Across Parameterizations and Optimizers. ICML 2024.

[3] Lourie, N., Hu, M. Y., & Cho, K. (2025). Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check. Findings of the Association for Computational Linguistics: EMNLP 2025

[4] Schaeffer, R., Schoelkopf, H., Miranda, B., Mukobi, G., Madan, V., Ibrahim, A., Bradley, H., Biderman, S., & Koyejo, S. (2025). Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive? ICML 2025

Reviewer_6KGAOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains RoboJEPA, a family of action-conditioned latent world models (22M–8B predictor parameters) on top of a frozen V-JEPA 2.1 encoder, using a 23-dataset, 12-embodiment mixture (~15k video hours). It fits four candidate scaling-law forms to the compute-optimal frontier of 9-step rollout L1 error on 22M–2B models, then checks how well each extrapolates to 4B/8B; a second-order power law extrapolates best on DROID and RoboCasa holdouts. It then evaluates CEM planning with image goals on four RoboCasa tasks and three real Franka tasks, and shows that a long-horizon (K=10) cooldown recipe improves both the scaling law and long-horizon planning.

Critical strengths
  1. Scale and rigor of the offline study. Eight model sizes, compute measured per step instead of via 6ND, and the law is selected by held-out extrapolation rather than in-sample fit. This is the most systematic scaling study of robotic latent world models I am aware of, and the promised release of checkpoints and code makes it a useful resource.

  2. The recipe finding is practically useful: spending compute on long-horizon rollout during cooldown lowers the fitted irreducible error, and with this recipe long-horizon planning becomes much more strongly related to compute (Fig. 3c).

Critical weaknesses
  1. The claim that imagination error is a "reliable proxy" for planning is not established in the regime where it matters. The two greedy tasks (planning horizon H=1) are already close to their ceiling at around 10^21 FLOPs (100M–300M models), and larger models add only about 0.02–0.03 normalized return (Figs. 32–33). Under the default recipe, offline error decreases smoothly, but obstacle-reach reward is only weakly correlated with compute and smaller models often beat larger ones, e.g., 300M vs. 1B (Fig. 3b/35). Under the long-horizon recipe, the highest obstacle-reach and push results come from checkpoints smaller than 8B, the curves are noisy, no law is fit, and no correlation statistic is reported anywhere in the paper (Figs. 36–37). The proxy therefore seems to hold mainly for tasks that do not need scale, and it depends on the recipe.

  2. The real-robot evidence for scaling is weak. With 30 episodes per cell, the standard error is up to about ±9 points, and results are non-monotonic (pick-and-place: 8B 27% vs. 2B 31%; lift: 4B 30% vs. 2B 40%). The VLA comparison is confounded by different controllers (blocking EEF control vs. 15 Hz joint velocity), episode counts (30 vs. 50) and goal modalities, and the main text does not state that the wrist view has zero weight in the planning cost.

Questions for authors
  1. Please report the correlation (e.g., Spearman) between imagination L1 and planning return/success across all checkpoints, per task and per recipe. Does short-horizon-recipe loss predict long-horizon planning at all? (Affects the main claim.)

  2. Can you fit a scaling law on the downstream metrics for obstacle-reach and push, and does it extrapolate to 8B?

  3. Please give confidence intervals for Table 12, and ideally more episodes for 2B/4B/8B.

  4. The extrapolation only spans about 4x in parameters. Does a fit on 22M–1B predict 2B–8B equally well?

  5. Clarifications: rho_max is 0.05/0.10 m in App. G.4 but 0.15/0.30 in Tables 13/14; the world-model rate is 5 Hz in App. C.5 but ≈2 Hz in App. E.1. Which values were used? Why is the wrist-view weight 0 on the real robot?

2 comments

We thank the reviewer for a well-thought review.

W1&Q2. We are not claiming that we build scaling laws for downstream performance. Instead, our only claim regarding that is that this performance is correlated with the training compute.

The idea of scaling laws is to scale a quantity on the x-axis (e.g. training compute) while keeping all other hyperparameters optimal with respect to each individual value of the x-axis. However, we chose to not fit scaling curves downstream planning since the scaling trends on downstream tasks often have many phase transitions [1] and these trend curves still keep growing after saturation. This can limit what part of the scaling behavior the fitted curve can explain (for example, the log-sigmoid fit on the reaching task does not explain the post-saturation growth). And phase transitions (e.g. sudden growth in long-horizon tasks) make it harder to test extrapolation. This is a widely accepted view in the scaling laws community as evidenced by works [2] and [3] from NLP. At the same time, long-horizon tasks use sparse task progress metrics (final reward or success rate) which makes evaluations more noisy.

Due to the reasons above, our focus is to build a general and practical recipe to verify the downstream performance rather than ensure compute optimality of planning. For all tasks in RoboCasa, we use a unified set of planning hyperparameters and only vary the planning horizon. A larger planning horizon may require exponentially more samples during planning, which we hypothesize is one of the reasons behind much noisier downstream performance trend in the Obstacle and the Push tasks and a weaker form of the performance frontier.

We would like to clarify that the main recipe of our paper is the one under the long-horizon training not the short-horizon one. Among other recipe components, the long horizon training is something that substantially increases the training computation which motivated us to show its utility. Finally, as the Section 3.2 shows, the long-horizon training is more optimal with respect to the long-horizon planning tasks, which is why the long-horizon training should be used as a proxy for them.

W2. We agree that the amount of real world evaluation episodes is smaller than is traditionally done in robotics papers. However, RoboJEPA does not train a policy, instead it derives the one zero-shot at test-time via computationally expensive planning. This limited the amount of real world evaluations we could perform in a reasonable time.

We would like to clarify that we explicitly highlight in the original submission that the VLA policy evaluations should not be read as baselines to the method but rather as contextual references that provide a measure of the task complexity in our specific rig. In L438-449 and L3126-3131, we argue that due to different goal formulations, and different controllers these are not directly comparable.

Q1. We added the Appendix O that reports Spearman correlations between training compute, imagination error and planning. Since within a recipe the imagination error is perfectly correlated with compute (ρ=−0.99\rho = -0.99), these correlations largely reflect compute, and we revised the paper to claim only that planning improves with training compute.

Q3. We have updated the paper adding confidence intervals.

Q4. Since our scaling law is second order (i.e. ln⁡(L−E)\ln (L-E) is a quadratic function of ln⁡C\ln C) so the slope itself depends on the amount of compute. Because of that our scaling law would not extrapolate as well from 1B as it does from 2B because 1B does not provide enough data to fit the second-order term. (−γ⋅ln⁡C-\gamma \cdot \ln C in the exponent)

Q5. We used the values in Tables 13/14: ρ_max = 0.15 m (real robot) and 0.30 m (RoboCasa). G.4 listed code defaults and is now corrected. The world model runs at 5 Hz (7.5 Hz for DROID action-key streams, App. C.6). The "~2 Hz" in E.1 was an error, and the decoder runs at 7.5 Hz. The real-robot controller is blocking, so each step is one end-effector displacement. The wrist-view weight is 0 because the wrist-view cost caused a vertical movement bias toward the goal pose (now stated in Section 3.3)

We hope our review satisfactorily addressed the reviewer’s concerns. If that is the case, we are kindly asking the reviewer to consider raising the score or the soundness.

[1] Weng, L. (2026). Scaling Laws, Carefully. Lilian Weng's Blog. https://lilianweng.github.io/posts/2026-06-24-scaling-laws/

[2] Lourie, N., Hu, M. Y., & Cho, K. (2025). Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check. Findings of the Association for Computational Linguistics: EMNLP 2025

[3] Schaeffer, R., Schoelkopf, H., Miranda, B., Mukobi, G., Madan, V., Ibrahim, A., Bradley, H., Biderman, S., & Koyejo, S. (2025). Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive? ICML 2025

We thank the reviewer again for the review. As we come closer to the end of the discussion period, we would like to kindly ask the reviewer if they have an availability to evaluate our rebuttal.

We are happy to clarify if any concerns remain and we look forward to the reviewer's reply.

Reviewer_2U35OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how action-conditioned latent world models for robots scale with training compute. The authors train RoboJEPA, a family of predictors from 22M to 8B parameters, on top of a frozen V-JEPA 2.1 encoder. The predictor takes multi-view visual tokens, action tokens, and proprioceptive state tokens. It predicts the encoder features of the next step. The training data is a mixture of 23 public datasets with 12 robot platforms and 26 action spaces (about 6.7k action-labeled hours). The loss is the sum of a teacher-forced L1 loss and a K-step autoregressive rollout L1 loss. At test time, the model plans toward a single goal image with CEM.

The paper has three main results. First, the 9-step rollout L1 error on held-out DROID and RoboCasa scenes follows a second-order power law in compute. A fit on models up to 2B predicts the 4B and 8B models better than three other functional forms. Second, planning performance in four custom RoboCasa tasks improves with compute. For the long-horizon tasks, this is true only when the cooldown uses a longer rollout (K=10 instead of K=2). Third, on a real Franka with the DROID setup, success on grasp and lift increases with model size. The 8B model does better than π0-FAST and π0.5 on these two tasks, but not on pick-and-place.

Critical strengths
  1. A careful scaling study at a scale that is new for robot world models. The protocol is good. The authors fit on small models (22M–2B) and test the fit on held-out large models (4B, 8B). They compare four functional forms. They count FLOPs directly and do not use the C ≈ 6ND approximation. The branched WSD cooldowns give many compute points at low cost. The result is a tested way to forecast world-model quality before a large training run. The planned release of the 8B checkpoints and the multi-embodiment data pipeline will also help the community.

  2. The result that the training recipe changes downstream scaling. With K=2 rollout training, obstacle-reach performance does not scale cleanly. With K=10, it does. This shows that the training horizon must match the planning horizon, and that offline loss alone is not enough. The order in which capabilities appear (end-effector motion, then carrying an object, then avoiding obstacles, then pushing objects) is also useful for people who design world-model evaluations.

Critical weaknesses
  1. The claim that imagination error is a reliable proxy for planning performance is stronger than the evidence. In Fig. 3a–b, the error of the K=2 models decreases with compute, but obstacle-reach reward does not follow. So the relation holds inside one recipe, but it fails across recipes. A proxy is most needed exactly in that case.

  2. Also, both quantities increase with compute, so the correlation can come from this common cause. The paper gives no correlation statistic and no analysis at matched compute. The greedy tasks (Fig. 2c–d) saturate near 1e20 FLOPs, about three orders of magnitude below the largest budget, so they give little evidence at large scale. I ask the authors to make this claim weaker or to add the analysis in Q7.

  3. The real-robot evaluation is small, and some numbers are not consistent. There are 30 episodes per cell, so the confidence intervals of neighboring model sizes overlap (for example, 67% at n=30 has a 95% Wilson interval of about [49%, 81%]). Pick-and-place shows no trend with size, and both VLAs do better on it. Several success rates in Table 12 are not possible with the stated episode counts: 2%, 8%, 18% and 24% (Object Lift) and 21% and 31% (Pick and Place) with n=30, and 5% and 53% for π0.5 with n=50. For Grasp, progress equals success in every RoboJEPA row, but not in the VLA rows, so the scoring may be different. The closest prior method, V-JEPA 2-AC, uses the same planning setup but is not evaluated. The VLA comparison also has confounders: goal image vs. text instruction, and blocking end-effector control vs. 15 Hz joint-velocity control.

I think these problems can be fixed. For this reason, I still support acceptance.

Questions for authors
  1. Table 12: please give the number of episodes and the number of successes for each cell. Please confirm that RoboJEPA and the VLA baselines use the same scoring rubric.
  2. Scaling-law fit: Section 3 says the fit uses the frontier points. Appendix K says it uses all 47 small-model checkpoints, and that the held-out error is the mean over all 33 large-model checkpoints. Which one is correct? The law predicts the frontier, so please also report the extrapolation error on the frontier points only.
  3. How many passes over the data does each model size make? Appendix K.2 says "our runs are single-epoch", but the conclusion talks about multi-epoch training. My rough estimate for the 8B model is many passes. This affects how we should read the "close to saturation" result.
  4. Do all sizes use the same peak learning rate (5e-5)? Do you use µP? Appendix D.3 mentions "µP lr scale factors" only once. A learning rate that is not tuned per size can change the shape of the frontier.
  5. What does the goal image show for each real-robot task? Appendix H.6 says that the goal and the initial robot pose are the same. How did you define the DROID holdout? Are the scenes of your real-robot lab in the DROID training data?
2 comments

We sincerely thank the reviewer for a thoughtful and positive review. Below, we would like to address some of the concerns:

W1&W2. We added the Appendix O where we report the Spearman correlations: planning performance is strongly correlated with training compute (0.77 to 0.94), and imagination error with planning (-0.77 to -0.94). The latter reflects their common dependence on compute, since within a recipe the imagination error is almost a function of compute (ρ = -0.99). We therefore revised the claim: we now claim only that planning performance improves with training compute when the training horizon matches the planning horizon, and we no longer claim that the imagination error is a reliable proxy for planning.

W3. Please note that RoboJEPA does not train a policy, instead it derives the one at test-time via computationally expensive planning. This limited the amount of real world evaluations we could perform in a reasonable time. We performed the full evaluation (8 models, 3 tasks, 30 episodes per task) only once and a few episodes may have failed due to hardware errors (which we do not count as model failures). The new Appendix Table 15 shows the valid episode counts.

Regarding the pick-and-place trend, we also would like to point the reviewer to the fact that the success rate of all models below 2B is strictly worse than the success rate of models at that size or above. That being said, we believe this shows the trend exists. This highlights the difference between VLAs and planning since the former is bottlenecked by the training data coverage and the latter is by task horizon. Additionally, we discuss the differences between VLAs and the RoboJEPA-planning asking the readers to therefore view VLA evaluations as contextual references for the task complexity instead of full baselines (see L438-L460 for discussion).

We would like to correct the reviewer regarding the planning similarity to V-JEPA 2. The main difference is that it uses subgoals to solve long horizon planning tasks and within the subgoal the planning remains greedy in V-JEPA 2. RoboJEPA uses non-greedy long-horizon planning with a single image goal.

We also thank the reviewer for pointing out an interesting property of RoboJEPA. The task progress is much closer to the success rate in RoboJEPA than in VLAs. Both are evaluated under the same success and progress criteria and this is another property of planning. The planning policies always remain at the goal state when they achieve it. VLAs on the contrary, very often behave toward decreasing the task progress. We observed that due to this phenomenon, planning policies have a much smaller gap between the success rate and the planning progress.

Q3. We do not have a well-defined notion of an epoch because each multiview episode is sampled with a variable number of views, a random partial crop within each video and on a random chunk of frames. This means two realizations of sampling from the same episode can potentially produce disjoint (or partially overlapping) chunks of training data. We removed the claim that our training is single epoch. The expected number of full passes over each training dataset can be recovered from the training steps, batch size and dataset sampling weights.

Q4. Yes, we use the same peak LR for all sizes which we tuned to be the optimal across several different model sizes (at 2B and below). We did not use µP as we found it to be scaling suboptimally. We removed its mention from the paper.

Q5. The original submission presents examples of initial and goal robot poses. We corrected the claim that poses coincide between start and and goal as it only holds for one RoboCasa task. For the droid holdout, we filter one building and use it as a holdout set. We updated the paper with more details.

We hope that we resolved the reviewer’s concerns. If so, we are kindly asking the reviewer to consider increasing the score of the paper.

Thank you for the detailed response and the revisions. The new Spearman analysis in Appendix O and the revised claim about imagination error address my main concern. The valid episode counts in Table 15 and the clarifications on epochs, learning rate, and the DROID holdout are also helpful. I will take the rebuttal into account in my final assessment.

Reviewer_2dzjOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

While there has been a clear understanding of scaling laws in many other fields (e.g., NLP), the robotics world models lacks a principled way to predict how latent world model capabilities scale with model size, data, and compute.

This paper first presents RoboJEPA, a feed-forward Transformer predictor scaled from 22M to 8B parameters across 23 datasets spanning 12 robotic embodiments. The central finding is that imagination error, which I believe is the error of latent rollouts on held-out data, follows a second-order power law in compute.

Downstream planning performance improves predictably with compute, and imagination error is shown to be a reliable proxy for real-robot evaluation. The model is deployed on real hardware, planning toward a single goal image.

To the authors' knowledge, this is the first work to establish scaling laws for multi-embodiment robotic world models trained on real robot data. I also believe that, at 8B parameters, RoboJEPA is the largest JEPA predictor trained to date. Since the visual encoder is frozen throughout, the scaling laws shall not apply to the encoder.

Critical strengths
  1. Seems to be first scaling laws for robot world models. Significant.

  2. Large-scale and diverse multi-embodiment training.

Critical weaknesses

No big weaknesses. But, the only element here is that the frozen encoder technically limits what we could call scaling laws. The paper would benefit from a writing early on that emphasizes that limitation.

Questions for authors
  1. An important question is how confident the authors are that the scaling laws do hold when the scale spans further, perhaps a 10x or 100x extrapolation.

  2. I am not sure whether I am missing where it is explained. How many episodes per model size per task?

2 comments

We thank the reviewer for a positive review. Let us elaborate on the concerns of the reviewer:

W1. Our work builds scaling laws by showing improvements in imagination error that come with scaling the predictor. The encoder therefore has to remain frozen because imagination error has to be computed in a fixed latent space. Unfreezing the encoder would mean that the geometry of the latent space changes with every optimization step so two models would become incomparable and the scaling law of imagination error would be meaningless. We would like to emphasize that our paper already has a short discussion of this in Introduction, see L061-L062.

Q1. As the Table 2 and Section 3 suggest, scaling of RoboJEPA over 8B is very close to saturation. To see this, consider the irreducible error term from Table 2 which is $ E=0.218 $ for Droid. At the same time the 8B world model obtains the imagination error of around $ 0.223 $, which suggests that the model is very close to saturation. We hypothesize that the scaling law would hold if we extrapolate it further. Figures 2(a), 2(b) show that larger models remain compute optimal for almost the entire duration of their training, after some training compute. This means that all of them lie on the performance frontier which follows the scaling law. Therefore, our current results suggest that the imagination error will extrapolate further unless we observe a phase transition or a higher-order term’s effect that is only visible at an extreme scale. However, our results do not contain any evidence that either of these effects exist.

Q2. We run a total of 8 models with 3 tasks per model and 30 episodes per task per model. A few episodes failed due to external reasons (e.g. hardware error). We added Table 15 that contains valid real world episode counts for our real world evaluation.

We hope that we addressed the reviewer's concerns. If we did, we are kindly asking the reviewer to increase the score.

We again thank the reviewer for a positive review. Since the end of the discussion period is approaching, we are kindly asking the reviewer to evaluate our rebuttal.

We look forward to the reviewer's reply and are happy to answer any questions.

Comments on the paper

1 comment

We thank all reviewers for their insightful comments. We appreciate their recognition of the scale and diversity of our multi-embodiment training study (Reviewers 2dzj, 2U35, 6KGA, x8Rr), the significance of systematically studying scaling in robotic world models (Reviewers 2dzj, 2U35, 6KGA), the rigorous experimental protocol (Reviewers 2U35, 6KGA), and the predictive value of the second-order power law when extrapolating to larger models (Reviewers 2U35, 6KGA, x8Rr). We also appreciate the recognition of the practical insights from long-horizon rollout training (Reviewers 2U35, 6KGA), the clear presentation (Reviewers 2dzj, 2U35, 6KGA), and the value of the planned release for the research community (Reviewers 2U35, 6KGA).

Below we list the main updates we added

  • We added a Related Work section to the main text (Section 4), covering world models for robotics and the scaling of world models, including WestWorld and Pearce et al. (x8Rr). The full discussion is in Appendix A. We also revised the introduction to position our contribution as the first extrapolating scaling law for multi-embodiment world models (x8Rr, 2dzj).
  • Appendix O reports Spearman correlations between training compute, imagination error and planning performance for all four RoboCasa tasks. Following these results, we revised the claim. It now states that only planning performance improves with training compute (2U35, 6KGA). Section 3.2 explains why we do not fit scaling laws to downstream planning (6KGA).
  • For the real robot, we added episode counts (Table 15), 95% Wilson confidence intervals (Table 16, Figure 26), and a discussion of why task progress and success rate are close for RoboJEPA (2dzj, 2U35, 6KGA, x8Rr).
  • We now state the zero wrist-view weight (Section 3.3), motivate each RoboCasa metric (Appendix H.4), and define the DROID holdout (Appendix C.1) (6KGA, x8Rr, 2U35).
  • We corrected the Fig. 1 indices, the masking text (C.5), the DROID control rates (C.6), the Reach and ObstacleReach distances (H.4), ρ_max (G.4), the decoder rate (E.1), the deployment-control paragraph (C.5), the start/goal pose statement (H.6), and D.3 (x8Rr, 6KGA, 2U35).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.