acceptodds
Under review as a conference paper at ICLR 2027

Scaling in Game: Continued Pre-Training For Embodied Agents In Diverse Virtual Worlds

Abstract

Scaling data, diversity and model capacity has improved embodied learning in physical environments, where diverse experiences remain grounded in shared physical laws. Whether these gains extend to virtual worlds with different visual styles, dynamics, controls, and objectives remains unclear. We investigate this question through a controlled study of continued pre-training (CPT) across three axes: data volume, diversity, and model size. Our 120 training configurations span four model sizes from 0.8B to 27B, six data budgets from 50 to 1,600 hours, and five mixtures ranging from Minecraft-only data to diverse multi-world corpora. We assess performance and generalization using three-level validation losses and gameplay benchmarks from in-corpus worlds to unseen worlds. Our results show: (1) more data improves in-corpus performance, diversity improves transfer to related worlds with limited gains in distant worlds, and model size improves data efficiency but generalizes worse to distant worlds under single-world training; (2) we observe an early *interface phase* with broad cross-world benefits and a later *world phase* with gains narrowed to in-corpus and related worlds, and fit an empirical scaling law relating validation loss to data volume, diversity and model size; and (3) our models are competitive with Minecraft specialists and frontier models in 3D worlds, and perform best in real-time settings. However, the limits of imitation and persistent gaps in distant worlds suggest that building generalist embodied agents requires richer training stages beyond continued pre-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
continued pre-trained, embodied agent, scaling law, generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Uf5dOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies continued pre-training for game agents, examining how model size, gameplay-data volume, and mixture composition affect performance and cross-game transfer. It covers four Qwen3.5 model sizes, six gameplay-data budgets, and five mixtures, producing 120 evaluated conditions through 20 shared training trunks and budget-specific branches. The evaluation combines prediction losses on three held-out sets with four interactive benchmarks. The paper analyzes in-corpus gains, transfer to related games, distant-world degradation, and early versus late learning behavior. It also fits empirical loss laws and compares gameplay in paused and real-time environments.

Critical strengths
  • The experimental coverage is systematic: Studying model size, data volume, and mixture composition together helps identify where training gains come from.
  • The study measures both offline loss and interactive gameplay. Four benchmarks cover training worlds, related unseen worlds, and more distinct worlds.
  • Real-time evaluation provides useful practical evidence. The paper reports both paused and real-time performance, together with deployment details, response latency, and output length, showing how inference speed affects gameplay.

Overall, I think this paper involves a significant amount of work, and the evaluation and analysis are quite thorough.

Critical weaknesses
  1. Section 3.1 describes S1–S5 as having increasing mixture diversity. However, Appendix A.1 explains that S2–S4 use the same game inventory, with diversity changing through more even data allocation, while S5 is primarily a control excluding Minecraft rather than a further diversity increase. Table 3 also reports diversity values of 60.1 for S4 and 58.7 for S5, which do not support a monotonic increase across S1–S5. The main text should explain that the metric accounts for both game count and data allocation, and distinguish S5 from the increasing S1–S4 sequence.
  2. Section 5 interprets the proposed models' advantage over several Minecraft specialists as evidence that backbone capability matters more than specialized design. It also contrasts near-zero instruction-task scores from models trained on tens of thousands of gameplay hours with the 36% task completion of 0.8B-S3, arguing that a language foundation matters more than data volume. Appendix B.2 explains that these prior scores are quoted from other papers without aligning models, data, action interfaces, or evaluation conditions. This limitation should accompany the statements in the main text.
Questions for authors
  1. Section 3.1 describes S1–S5 as increasing in mixture diversity, but Table 3 gives S5 a lower value than S4, and Appendix A.1 treats S5 as a control excluding Minecraft. Perhaps a more precise expression is needed here?
  2. Equation (3) states that all fitted parameters are strictly positive, while Appendix C.6 permits zero values for α, β, and γ and negative values for η.
Reviewer_ZgSGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how data volume, game mixture, and model size affect continued pretraining of agents that interact with games through keyboard and mouse. 120 configurations across four model sizes, six data budget and five mixtures, have been evaluated. Their findings over this evaluation is more data improves performance in familiar games, while broader mixtures mainly help transfer to related environments. They also observe early improvements across environments followed by greater specialization, and fit an empirical scaling relationship. The findings are assessed through validation losses and gameplay evaluations on different games.

Critical strengths

1- The paper systematically compares different model sizes, data budgets and game mixtures under a consistent training setup. Evaluating these configurations on both familiar and unseen environments provides useful evidence about where additional pretraining helps and where its benefits fail to transfer. This is a useful setup for assessing generalisation beyond the training.

Critical weaknesses

1- My biggest concern for this paper is framing promises a much larger conclusion while experiments support narrower conclusion. The experiments mainly change the balance between Minecraft and fixed pool of other games, and reducing Minecraft's share gives the model more training data from those other games including games related to transfer tasks. It supports that improvement could therefore come from greater exposure to relevant data. Therefore, my claim about narrower conclusion is supported by limited diversity of this particular mixture.

2- Single training seed doesn't support any consistency of results. Even though appendix reports uncertainty in the fitted scaling curves, its not enough to show how the results would change with multiple seeds. This matters because judging smaller differences between configurations, since the paper's contribution claims on these empirical comparisons.

Questions for authors

1- Could you repeat a small set of key mixture comparisons with independent training seeds ( at least 5 seeds are expected as standard for these type of work). It's important to see mean and standard deviation for this type of experiment setting. This will also support the reported differences are whether reproducible or not.

2- I couldn't find a comparison where the baselines use reduced reasoning budgets. So, could authors test whether real time advantage remains when a baseline is configured for faster responses?

3- It's clear that reducing Minecraft data share allows more data from other games, but how can authors tell whether better transfer comes from better diversity or simply more training on games similar to evaluation tasks?

1 comment

Dear reviewer ZgSG,

Greetings from the authors!

Thank you for your constructive feedback, which helped us strengthen the paper. We have uploaded the revised manuscript, with changes highlighted in blue:

  • Framing and diversity: narrowed the conclusions and clarified the exposure on seen games (§3.1; Appendices E.2 and E.4).
  • Seed sensitivity: added 48 second-seed results for S1/S5 and a joint 168-result refit, with the scope of this partial check stated explicitly (Appendix E.1; Table 17).
  • Faster baselines: added lower-reasoning-effort comparisons under otherwise identical conditions, including latency and real-time scores (Appendix E.3; Table 19).

As the discussion period draws to a close, could you let us know whether any concerns remain? We hope our rebuttal and revisions address your concerns and further strengthen your recognition and confidence in our paper!

Best regards,

The Authors

Reviewer_xbRAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a systematic empirical study of scaling CPT for embodied agents operating across heterogeneous virtual game worlds. The authors investigate three dimensions: model capacity (0.8B–27B), training data volume (50–1,600 gameplay hours), and data mixture diversity, resulting in 120 evaluated configurations. The study evaluates performance using validation losses and four gameplay benchmarks, covering in-corpus worlds (Minecraft and ViZDoom), unseen related worlds (OmniGameArena), and unseen distant worlds (GameWorld). The main findings are that increasing training data improves in-corpus performance, more diverse training mixtures enhance transfer to related worlds but provide limited benefits in distant worlds, and larger models exhibit improved data efficiency but may suffer stronger negative transfer under single-world training. The authors also identify two empirical learning phases—an early interface phase and a later world-specific phase—and propose an empirical scaling law relating validation loss to model size, training hours, and data mixture proportions.

Critical strengths
  1. Systematic experimental investigation across multiple scaling dimensions. The paper conducts an extensive experimental study spanning four model sizes, six data budgets, and five data mixtures.

  2. Valuable empirical observations about generalization and negative transfer. The paper goes beyond conventional in-distribution scaling evaluations by examining transfer to unseen game environments. An interesting finding is that additional training can improve performance on familiar worlds while providing diminishing or even negative returns on unseen worlds.

Critical weaknesses

A primary concern is whether the explored data regime is sufficiently large to support the paper's broader conclusions about scaling in virtual embodied environments. Unlike language modeling, embodied learning must account for heterogeneous visual observations, action dynamics, environment-specific mechanics, and long-horizon interactions. Thus, the amount of experience required to observe stable generalization trends may be substantially larger. Although the study considers up to 1,600 gameplay hours, this budget is distributed across as many as 80 game-world labels, potentially leaving relatively limited exposure to individual environments and interaction skills.

Besides, the diversity-scaling experiments do not sufficiently isolate the causal contribution of world diversity. The paper identifies diversity as one of its three principal scaling dimensions. However, the construction of the S1–S5 training mixtures makes it difficult to determine whether the observed improvements are genuinely attributable to diversity itself. As explicitly described in Appendix A.1, S2, S3, and S4 contain the same 80 canonical world labels. Their primary difference is the proportion of Minecraft trajectories, which decreases from 51.4% in S2 to 27.1% in S3 and 2.8% in S4. Correspondingly, the proportion and absolute training exposure of the remaining games increase under a fixed total data budget.

Another major concern is the definition of generalization distance used in the paper. The authors categorize OGA as unseen related worlds and GameWorld as unseen distant worlds. This distinction is intuitively plausible: OGA primarily evaluates 3D navigation, exploration, combat, and shooting, which overlap with common skills in the training corpus, whereas GameWorld contains many 2D, 2.5D, arcade, and puzzle-oriented tasks. However, the paper does not provide an independent or quantitative measure of the distance between the training distribution and these evaluation environments. Instead, the related/distant distinction is largely determined by the choice of two benchmarks with substantially different task compositions.

Questions for authors

Please see the weakness part above.

3 comments

Dear reviewer xbRA,

Greetings from the authors!

Thank you for your constructive feedback, which helped us strengthen the paper! We have uploaded the revised manuscript, with changes highlighted in blue:

  • Scale and robustness: clarified the tested-range limitations and added a partial second-seed check with a 168-result refit (Appendices E.1 and E.4).
  • Mixture exposure: clarified the diversity–exposure confound and added approximately Minecraft-hour-matched comparisons (§3.1; Appendix E.2).
  • Generalization distance: clarified the descriptive basis of the related/distant benchmark groups (§3.2; Appendix E.4).

As the discussion period draws to a close, could you let us know whether any concerns remain? We hope our rebuttal and revisions address your questions and strengthen your recognition and confidence in our paper!

Best regards,

The Authors

Thank you for your response. I have checked the author response and will maintain the positive evaluation.

Thank you for taking the time to review our rebuttal and revised manuscript. We sincerely appreciate your recognition in our work. Your constructive suggestions on data regime, difference between training worlds and evaluation worlds, and the scope of diversity have helped us strengthen both the experimental evidence and the presentation.

Thank you again for your thoughtful feedback and valuable time!

Comments on the paper

1 comment
AuthorsOpenReview ↗

The Overview of Rebuttal Period

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their time and constructive feedback on our paper. Through our rebuttal and revisions, we have addressed the questions raised by all three reviewers, and all three have now confirmed positive evaluations of 3 (Weak Accept).

  • Reviewer xbRA reviewed our responses and confirmed that they would maintain their positive evaluation.
  • Reviewer ZgSG recognized the additional seed checks, lower-reasoning-effort comparisons, and clearer framing of diversity effects, raised their rating from 2 to 3, and subsequently confirmed 3 as their final rating.
  • Reviewer Uf5d confirmed that our clarifications addressed most of their concerns and maintained their positive rating.

The revision adds 48 second-seed results for S1/S5, a joint refit using 168 results, and faster-baseline comparisons. It also clarifies mixture construction, the scope of generalization claims, cross-paper comparisons, and fitting constraints. Changes are highlighted in blue in the uploaded manuscript.

We are confident that these revisions, driven by the reviewers' valuable feedback, make our paper more robust, rigorous, and clear. We are pleased that the rebuttals were well-received and all the reviewers keep a positive recommendation in light of our responses sufficiently addressing their concerns. We promise to incorporate rebuttals into the final manuscript.

Best regards,

Authors of Paper #26063

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.