- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GamePilot investigates how a fixed-weight vision-language model can continuously improve while playing games without pauses or resets. The paper presents three contributions with corresponding evaluations. First, it extends optimization beyond game knowledge to observation and action interfaces; full optimization improves progress by 10.6–11.4 percentage points over knowledge-only optimization. Second, it introduces an Evolver that performs failure localization, code revision, offline validation, and online version selection in the background. Ablation studies show that removing localization guidance, comprehensive admission checks, or advantage-based scoring reduces performance. Third, it introduces GamePilot Bench across four games, combining one-hour continuous-play evaluations with human-rated task evaluations across three backbone models. GamePilot achieves 1.85–2.00× the continuous-play progress of the strongest baseline. These results provide preliminary evidence that adapting interaction mechanisms improves gameplay, although each main continuous-play condition includes only one run, leaving robustness and reliable failure correction insufficiently established.
Critical strengths
The paper studies a meaningful online adaptation problem: improving observation and action mechanisms from gameplay experience while keeping model weights fixed and the game continuously running. The knowledge-only comparison provides initial support for this approach.
The paper provides a multi-level evaluation across four games, combining continuous-play progress with human-rated interaction and game-knowledge tasks, rather than relying solely on successful demonstrations.
Critical weaknesses
Unclear attribution of the Evolver’s benefits.The Evolver relies on GPT 6 Astra for analysis, code revision, and evaluation. The results do not sufficiently distinguish the benefits of the auxiliary model’s capabilities from those of the Evolver’s specific design.
Insufficient evidence of result stability.Each condition in the main Milestone experiments and associated ablations is evaluated through only one one-hour run, making it difficult to determine whether the reported improvements are consistent or influenced by run-to-run randomness.
Questions for authors
With the same auxiliary model, editable scope, and comparable computational budget, how much additional benefit does the full Evolver provide over a simpler trajectory-based code-revision pipeline?
Are the main performance improvements and key ablation differences consistent across independent repeated runs?
3 comments
Q2 & W2. Consistency across independent repeated runs
Are the main performance improvements and key ablation differences consistent across independent repeated runs?
Reply: GamePilot’s lead over GV-Expert and the two repeated ablations persists across all three campaigns in every game. We report per-game means and sample standard deviations.
The four conditions use Gemini 3.6 Flash and share the starting save and endpoint configuration within each game. Each campaign lasts one hour, and the table reports Milestone progress (%).
Tab. R4a
| Condition | MC | WW | HoK | GI |
|---|---|---|---|---|
| GamePilot | 51.23 ± 1.29 | 45.27 ± 1.55 | 40.80 ± 1.30 | 47.20 ± 1.40 |
| GV-Expert | 27.63 ± 1.22 | 23.63 ± 0.96 | 20.43 ± 1.35 | 25.20 ± 1.20 |
| GamePilot w/o Advantage Credit | 46.83 ± 1.46 | 41.23 ± 1.25 | 36.43 ± 1.35 | 43.47 ± 1.35 |
| GamePilot w/o Localization Guidance | 42.40 ± 1.28 | 35.60 ± 1.35 | 37.17 ± 1.35 | 41.63 ± 1.22 |
In every game, the lowest GamePilot score exceeds the highest score of GV-Expert and either ablation. The main gain and the effects of localization and advantage credit recur across these campaigns.
Close replication of main-text findings (Tab. 2 in submitted paper): The 3-campaign means (51.23%, 45.27%, 40.80%, 47.20%) closely mirror the single-campaign scores reported in Table 2 (51.6%, 45.3%, 40.8%, 47.2%). The tight sample standard deviations (≤ 1.55%) across all four games and zero-overlap separation confirm that the approximately 2× performance lead over GV-Expert is statistically robust.
We have added these results to Appendix F of the revised manuscript. Specifically, F.1 covers the same-Astra pipeline and budget comparison, and F.4 the repeated campaigns and their summaries.
Dear Reviewer jXCW,
Thank you again for your time and constructive feedback throughout the review process. As the discussion period draws to a close, we have posted our detailed responses and updated the manuscript (including Appendix F) accordingly. Please let us know if any points warrant further discussion, as we would be glad to clarify. We appreciate your consideration in your final evaluation.
Best regards,
Authors of Submission #2216
Thank you for your questions. We address both points below. Specifically, we show the additional value of the Evolver over Direct Harness Editing under the same editable scope (Q1), and the consistency of the gains across repeated campaigns (Q2).
Q1 & W1. Contribution of the Evolver beyond the auxiliary model
Reply: GamePilot adds gains beyond Direct Harness Editing, using the same auxiliary model and editable scope at similar estimated API cost. We separate the value of the harness’s optimization targets from the procedure used to revise them.
(1) The same-Astra comparison measures the contribution of the optimization procedure. Both methods use Gemini 3.6 Flash for gameplay and GPT 6 Astra for harness revision. They share the editable scope, initial harness, starting save, and one-hour budget. Direct Harness Editing sends Astra the trajectory and current harness at each boundary and adopts its revision directly. GamePilot adds localization, replay admission, and advantage-based selection. The Fixed Harness condition provides the starting reference.
Games and metric. MC, WW, HoK, and GI denote Minecraft, Wuthering Waves, Honor of Kings: World, and Genshin Impact. Progress is reported as Milestone percentages.
Tab. R1
Direct Harness Editing improves over the Fixed Harness, and GamePilot improves further in every game. The additional gain over Direct Harness Editing is 13.4–14.4% in absolute progress. Both revision pipelines have a 150-call auxiliary ceiling. Calls and tokens below refer to the auxiliary model, while estimated cost includes gameplay and background work.
Tab. R1b
GamePilot makes more progress at similar estimated API cost.
(2) Editable scope already validated in Section 6.1 (Tab. 3): The harness exposes observation/action middleware, context construction, game knowledge, and delegation for revision. As reported in Table 3 of the submitted manuscript, adapting observation and action mechanisms adds 10.6–11.4% in absolute progress over revising game knowledge alone.
The quest-marker revision illustrates a concrete optimization target. A navigation failure leads to a change in observation processing that supplies explicit target geometry to later decisions.
(3) Component ablations already established in Section 6.3 (Tab. 4): Table 4 of the submitted paper systematically ablated each Evolver component: withholding localization guidance costs 6.85 pp on average, removing regression admission checks costs 6.20 pp, and removing advantage credit costs 4.20 pp.
Tab. 4
GamePilot achieves higher progress than each component ablation in every game. Together, Tab. 3 and Tab. 4 in the submitted paper establish the necessity of the editable scope and individual Evolver components, while Tab. R1 confirms that under the exact same auxiliary model and budget, the structured Evolver pipeline outperforms naive direct code editing by +13.4–14.4 pp.