- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies continued pre-training for game agents, examining how model size, gameplay-data volume, and mixture composition affect performance and cross-game transfer. It covers four Qwen3.5 model sizes, six gameplay-data budgets, and five mixtures, producing 120 evaluated conditions through 20 shared training trunks and budget-specific branches. The evaluation combines prediction losses on three held-out sets with four interactive benchmarks. The paper analyzes in-corpus gains, transfer to related games, distant-world degradation, and early versus late learning behavior. It also fits empirical loss laws and compares gameplay in paused and real-time environments.
Critical strengths
- The experimental coverage is systematic: Studying model size, data volume, and mixture composition together helps identify where training gains come from.
- The study measures both offline loss and interactive gameplay. Four benchmarks cover training worlds, related unseen worlds, and more distinct worlds.
- Real-time evaluation provides useful practical evidence. The paper reports both paused and real-time performance, together with deployment details, response latency, and output length, showing how inference speed affects gameplay.
Overall, I think this paper involves a significant amount of work, and the evaluation and analysis are quite thorough.
Critical weaknesses
- Section 3.1 describes S1–S5 as having increasing mixture diversity. However, Appendix A.1 explains that S2–S4 use the same game inventory, with diversity changing through more even data allocation, while S5 is primarily a control excluding Minecraft rather than a further diversity increase. Table 3 also reports diversity values of 60.1 for S4 and 58.7 for S5, which do not support a monotonic increase across S1–S5. The main text should explain that the metric accounts for both game count and data allocation, and distinguish S5 from the increasing S1–S4 sequence.
- Section 5 interprets the proposed models' advantage over several Minecraft specialists as evidence that backbone capability matters more than specialized design. It also contrasts near-zero instruction-task scores from models trained on tens of thousands of gameplay hours with the 36% task completion of 0.8B-S3, arguing that a language foundation matters more than data volume. Appendix B.2 explains that these prior scores are quoted from other papers without aligning models, data, action interfaces, or evaluation conditions. This limitation should accompany the statements in the main text.
Questions for authors
- Section 3.1 describes S1–S5 as increasing in mixture diversity, but Table 3 gives S5 a lower value than S4, and Appendix A.1 treats S5 as a control excluding Minecraft. Perhaps a more precise expression is needed here?
- Equation (3) states that all fitted parameters are strictly positive, while Appendix C.6 permits zero values for α, β, and γ and negative values for η.
Dear reviewer ZgSG,
Greetings from the authors!
Thank you for your constructive feedback, which helped us strengthen the paper. We have uploaded the revised manuscript, with changes highlighted in blue:
As the discussion period draws to a close, could you let us know whether any concerns remain? We hope our rebuttal and revisions address your concerns and further strengthen your recognition and confidence in our paper!
Best regards,
The Authors