acceptodds
Under review as a conference paper at ICLR 2027

SHUGYO: On-Policy Reinforcement Learning of Game Agents in a Generative World Model

Abstract

Game agents trained by supervised imitation on video–action data drift into states that demonstrations rarely cover, where small errors compound into task failure. Reinforcement learning (RL) can provide the missing experience of acting, failing, and recovering, but closed-source commercial games lack the fast dynamics, programmatic resets, and reward signals that RL requires. We present SHUGYO, a framework that adapts a pretrained video diffusion model into a learned environment for on-policy RL in the AAA action game Black Myth: Wukong. The simulator predicts future frames from rendered observations and native keyboard-and-mouse controls, resets rollouts from specified starting frames, and generates each chunk in four denoising steps. A rubric-guided vision–language judge scores the resulting rollouts, providing rewards without access to internal game state. Starting from a supervised vision–language agent, we perform RL post-training with GRPO and a two-stage curriculum entirely within the simulator, using a load-balanced client–server architecture for scalable rollout generation. Experiments show that the simulator follows native controls, remains visually coherent over extended rollouts, and adapts to a second game with only three hours of data. Per-task success rates in the simulator correlate with those in the real game, and RL post-training in the simulator raises real-game success from 58.6% to 68.6%. These results suggest that action-conditioned video world models can serve as practical training environments for game agents when the real game is not RL-ready.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Action-Conditioned World Models, On-Policy Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zaUxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SHUGYO adapts a pretrained video diffusion model into an action-conditioned simulator, using separate buttons/mouse conditioning, persistent initial-frame context, and four-step Self-Forcing distillation. It then trains an instruction-following VLA policy inside that simulator using GRPO, a prompted vision–language reward judge, behavioral penalties, and a task curriculum. Validation includes single-chunk prediction benchmarks, simulator-component ablations, a simulator–real comparison across 14 tasks based on the video game Black Myth: Wukon, and policy evaluations in two real games Black Myth: Wukon and Honor of Kings: World.

Critical strengths
  • The paper tests whether imagined training improves execution in the real environment. The policy is evaluated through the real game’s observation/action interface after training in the learned simulator, and improvements are reported in the two games. That experiment directly addresses whether the simulator is useful for learning behavior.
  • The controlled second-game adaptation comparison provides evidence of reusable learned representations.
Critical weaknesses
  • The central outcome measure is not sufficiently specified or independently validated. If the training judge also determines evaluation success, shared judgment errors or adaptation to its preferences could affect the measured gains. This is a possibility requiring validation, not demonstrated reward hacking.
  • The experimental design does not establish a robust advantage for the proposed training framework over simpler alternatives. The SFT-versus-full-system comparison is promising, but uncertainty across training runs is unreported and the key curriculum, reward, and generated-experience alternatives are untested. Because the broad world-model-plus-GRPO recipe already exists in closely related work, convincing evidence for the game-specific execution and design choices is especially important.
Questions for authors
  • Could you provide some kind of confidence intervals and/or independent training-run variability, second-game trial counts?
4 comments
AuthorsOpenReview ↗

Response to Reviewer zaUx (1/2)

We sincerely thank Reviewer zaUx for the thoughtful and constructive feedback, which will help improve our work. Our responses are as follows.

W1: The central outcome measure lacks independent validation.

Response: All reported real-game success rates are obtained by human evaluation. The VLM judge used in training plays no part in any success decision. Human evaluation is performed by multiple evaluators blind to the model setting, following task-category-specific criteria (Appendix D.4). Shared judgment errors between the evaluator and the reward model therefore cannot enter the reported success rates.

The VLM judge is used only for rewards inside the simulator and for pre-selecting three candidates among the saved checkpoints. Pre-selection is the only coupling between the two, and we verified it. In an early stage of the project, we compared several commercial models (Gemini series and Qwen3.8) with human scores on n = 54 real-game rollouts (criteria in Appendix D.4):

JudgePearsonSpearmanMAE|Δ| ≤ 0.2Bias (model − human)
Gemini-3.1-Pro0.6250.5990.11185.2%+0.015
Gemini-3.6-Flash (used)0.6680.6650.10990.6%+0.079
Qwen3.8-27B0.6490.5840.13088.9%+0.100

Gemini-3.6-Flash agrees best with humans in ranking. Since GRPO depends only on within-group ranking, ranking agreement matters more than overall bias, which is why we chose it. On 200 human-annotated simulator rollouts sampled during RL training, Gemini-3.6-Flash reaches a Pearson correlation of 0.609 with 83% within 0.2. Moreover, across four RL seeds, the ranking by mean judge score matches the ranking by human success (see Q1 below). The full protocol and agreement analysis are in Sec. 5 and Appendix D.4 (Table 6) of the revision.

W2: Missing comparisons with simpler alternatives.

Response: We evaluated four simpler alternatives in the real game, each replacing or removing a single component of SHUGYO. All variants are initialized from the same WukongAgent-SFT checkpoint and evaluated by blinded human evaluators on the same 14 tasks (10 trials each). The results are summarized below.

Simpler alternativeOverall (%)Δ vs. SHUGYOMost affected category
No RL (WukongAgent-SFT)58.6−10.0—
Optimizer: GRPO → REINFORCE57.9−10.7Cont. nav., Traversal (−20.0 each)
Generated experience: SHUGYO → Matrix-Game 3.057.1−11.5Cont. nav. (−23.3)
Training schedule: w/o curriculum63.6−5.0Cont. nav. (−10.0), Compound (−8.4, below SFT)
Reward: w/o STOP/SHIFT penalties63.6−5.0Cont. nav., Traversal (−10.0 each)
SHUGYO (full)68.6——

Generated experience and optimizer are both necessary. Replacing either one eliminates the entire RL gain. With experience from Matrix-Game 3.0, continuous navigation even falls below SFT (76.7% vs. 83.3%), which is consistent with the policy fitting dynamics that do not transfer to the real game. Generic generated experience combined with GRPO is therefore insufficient. The gain requires a simulator aligned with the real game's control interface and dynamics. REINFORCE lacks the within-group baseline of GRPO and thus yields high-variance gradients under sparse trajectory-level rewards.

Curriculum and reward shaping act where they are designed to. Each contributes about 5 points. Without the curriculum, atomic tasks still improve but compound tasks drop below SFT (43.3% vs. 45.0%), so staged training is what carries atomic gains over to compound tasks. Without the penalties, the loss concentrates on navigation tasks in which overshooting or failing to stop directly causes failure.

The world-model-plus-GRPO recipe alone thus does not account for the improvement. The gain emerges only from the combination of an action-aligned simulator, group-relative optimization, a staged curriculum, and a stopping-aware reward. Per-category results are reported in the revised Appendix F.2 (Table 9).

AuthorsOpenReview ↗

Response to Reviewer zaUx (2/2)

Q1: Confidence intervals and training-run variability.

Response: We repeated the full RL stage with four random seeds, each using the same checkpoint selection procedure (the VLM judge selects the top-3 candidates, which are then evaluated by humans, and we report their mean). Results are shown below.

Seed 1 (paper)Seed 2Seed 3Seed 4Mean ± std
Real-game success (human)68.6%72.1%70.0%68.6%69.8 ± 1.7%
Mean VLM-judge score0.6210.6500.6280.6140.628 ± 0.016

All four seeds exceed SFT (58.6%), and the seed reported in the paper ties for the lowest. The judge-score ranking matches the human-success ranking across seeds, so VLM pre-selection and human evaluation corroborate each other. The table is in the revised Appendix F.3 (Table 10).

With Wilson intervals, SFT reaches 58.6% (95% CI [50.3, 66.4]) and a single RL seed 68.6% ([60.5, 75.7]). Pooling the 560 trials of the four seeds gives 69.8% ([65.9, 73.5]). The difference from SFT has p ≈ 0.011 under a two-proportion test.

Q2: Trial counts on the second game.

Response: On Honor of Kings: World, each task is evaluated over 10 independent trials (100 in total). SFT reaches 52% (95% CI [42.3, 61.5]) and RL 74% ([64.6, 81.6]), with p ≈ 0.001 under a two-proportion test. The trial counts are stated in the revised Appendix F.4.

We hope these responses address the reviewer's concerns. Should any questions remain, we would be grateful if the reviewer could let us know, and we will respond as promptly as possible.

Dear Reviewer zaUx,

We sincerely thank you again for the constructive comments, which led us to add the real-game component ablations, the multi-seed results with confidence intervals, and the judge–human agreement study. As the author–reviewer discussion period is closing soon, we would be grateful if you could let us know whether these responses resolve your concerns about the evaluation protocol and the comparison with simpler alternatives. If you find that the concerns you described as minor or fixable have been addressed, we would kindly ask you to consider raising your Soundness and overall Rating scores. It would be our pleasure to clarify any remaining questions within the remaining time.

Dear Reviewer zaUx,

We understand that this is a busy period, and we sincerely appreciate your time. As the discussion period closes in a few hours, we would be grateful if you could take a brief look at our responses and the revised paper. If our additional experiments have addressed your concerns, we would kindly ask you to consider updating your Soundness and Rating scores. Thank you again for your constructive review.

Reviewer_ATjTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

Shugyo adapts a pretrained video diffusion model into a world model simulator for the AAA game Black Myth: Wukong, trained on about 190 hours of gameplay paired with keyboard and mouse actions. It has 3 distinct components - an action encoder, a first frame anchor that keeps rollouts consistent and a distillation into a fast 4 step autoregressive model. The agent is a VLA model trained by behaviour cloning on demonstrations followed by more post training with GRPO inside the world model - the reward is from a Gemini judge, and training uses a 2 stage easy-to-hard curriculum. RL inside the world model raises real-game success from 58.6% to 68.6%, and per-task success in the simulator correlates with the real game (r = 0.88). They also adapt the world model to a second game, Honor of Kings: World, with only 3 hours of data, where RL raises success from 52% to 74%.

Critical strengths
  1. Doing a full loop of a real commercial AAA game is honestly great work with a world model. Building the model where a VLA agent is trained with RL inside it and then tested on an actual game and further repeated on a second game with 3 hours of data is really good.
  2. The simulator is checked properly before being used for RL. The same agent runs the same 14 tasks in the world model and the real game and the per-task success lines up well. The authors are also quite upfront that it is harsher than the real game and weaker on multi-step tasks.
Critical weaknesses
  1. It is not clear how success is actually decided and is also never checked against human judgement. Section 5 says success is decided by the same rubric-guided vision-language judge, which I guess is the same as the Gemini model used. Appendix D.4 says rules based checks were applied to all 14 with an image based check on only 5. If the evaluator is the reward model, the agent is graded by the model it was trained to please.
  2. All tasks are navigation based and Wukong is Combat heavy, so the results do support a narrower claim
Questions for authors
  1. How is success decided? Section 5 says a VLM judge, but Appendix D.4 says rule-based checks plus an image judge on 5 tasks. Is the judge the same Gemini model, and what score counts as a success?
  2. Can you hand-label the existing real-game runs, and a sample of simulator runs, and report how well they match your success labels
  3. Which RL checkpoint was tested in the real game, and how was it chosen? Can you add at least 3 RL seeds?
  4. All tasks are navigation. If you haven’t tried combat or interaction tasks, please limit the claims to navigation.
  5. What changed in the agent’s behaviour after RL - for example, how often it presses Stop or sprints - especially on Tasks 4 and 14? How well does the RL agent do inside the simulator?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ATjT (1/2)

We sincerely thank Reviewer ATjT for the kind words on our end-to-end real-game loop and simulator validation, and for the thoughtful and constructive feedback, which will help improve our work. Our responses are as follows.

W1 & Q1: How success is decided, and whether the evaluator is the reward model.

Response: To remove any ambiguity about how rollout success is determined, we state the facts below. Sec. 5 and Appendix D.4 have been clarified accordingly in the revised version.

  1. All reported real-game success rates come from human evaluation. Every trial is recorded and judged as success or failure by multiple human evaluators who are blind to the model setting, following the task-specific criteria in Appendix D.4. The VLM judge used in training plays no part in any success decision. The agent is therefore not graded by the model it was trained to please.
  2. The VLM judge has exactly two roles: providing rewards in the simulator during RL, and scoring the policy during training to pre-select candidates for human evaluation. Both use the same model and rubric (Gemini-3.6-Flash, Appendix D.1–D.2).
  3. The "rule-based checks" and "image-based judge" in Appendix D.4 describe an obsolete pipeline. Early in the project we evaluated with automated scripts on the game client. As the evaluation set grew, this was replaced by VLM pre-selection followed by human evaluation, but the appendix text was not updated. The obsolete description has been removed and the current protocol is stated.

The only coupling between the judge and evaluation is checkpoint pre-selection. Q2 and Q3 below verify the agreement between judge scores and human judgment.

Q2: Agreement between VLM scores and human labels.

Response: The real-game success labels reported in the paper are themselves human labels. In an early stage of the project, to validate the VLM judge as a training reward, we compared several commercial models (Gemini series and Qwen3.8) with human scores on n = 54 real-game rollouts (criteria in Appendix D.4):

JudgePearsonSpearmanMAE|Δ| ≤ 0.2Bias (model − human)
Gemini-3.1-Pro0.6250.5990.11185.2%+0.015
Gemini-3.6-Flash (used)0.6680.6650.10990.6%+0.079
Qwen3.8-27B0.6490.5840.13088.9%+0.100

Gemini-3.6-Flash agrees best with humans in ranking. Gemini-3.1-Pro has the smallest overall bias but a weaker ranking correlation. Since GRPO uses only within-group ranking, ranking agreement matters more than overall bias, which is why we chose Gemini-3.6-Flash.

We further had humans annotate 200 simulator rollouts sampled during RL training. Gemini-3.6-Flash reaches a Pearson correlation of 0.609 with human scores, with 83% within 0.2. Agreement is slightly lower than on real-game rollouts but remains at the same level, confirming that the judge provides a reliable reward signal on generated observations. This analysis is in the revised Appendix D.4 (Table 6).

Q3: Checkpoint selection and multiple seeds.

Response: Every 50 optimizer steps during RL, the VLM judge scores the current policy. The three highest-scoring candidates are each evaluated by humans on all 14 tasks (140 trials), and we report their mean success rate.

A single RL run can over- or under-estimate a method's effect. We therefore repeated the full RL stage with four random seeds under identical settings and the same selection procedure. Results are shown below.

Seed 1 (paper)Seed 2Seed 3Seed 4Mean ± std
Real-game success (human)68.6%72.1%70.0%68.6%69.8 ± 1.7%
Mean VLM-judge score0.6210.6500.6280.6140.628 ± 0.016

(1) All four seeds fall in 68.6%–72.1% (mean 69.8%, std 1.7 points), all well above SFT (58.6%). The seed reported in the paper ties for the lowest, so the original result does not overstate the effect. Pooling the 560 trials gives 69.8% (95% CI [65.9, 73.5]). (2) Across seeds, the ranking by mean judge score matches the ranking by human success (seed 2 is highest on both), supporting the use of the judge for checkpoint pre-selection. The table and analysis are in the revised Appendix F.3 (Table 10).

AuthorsOpenReview ↗

Response to Reviewer ATjT (2/2)

W2 & Q4: Tasks are limited to navigation, so claims should be scoped.

Response: SHUGYO is designed around native keyboard-and-mouse control and is currently applied to instruction-following navigation. We will make this scope explicit in the revision. Within this scope, the contribution is concrete: in a closed-source commercial AAA game, up to 16 simulator instances run concurrent rollouts and reset from arbitrary start frames, providing on-policy RL with training conditions that the real game engine cannot offer, and the resulting gains are verified in the real game.

Combat and other broader settings depend on game state not directly visible in pixels. A viable direction is a non-end-to-end, proxy-based world model that predicts world state, renders a geometric proxy, and lets a video model synthesize appearance, as in Code World Model [1] and Proxy2World [2]. These methods require extra state modeling and proxy rendering, with higher per-step latency and cost, and are not yet suitable as RL environments that demand massive parallel rollouts. SHUGYO has established that training in a learned simulator transfers to real-game gains. Integrating explicit state modeling into this framework and extending on-policy RL to combat and interaction tasks is the next step we are actively pursuing.

[1] Y. Chen, G. Lin, C. Zhang. Code World Model: Coding Agent as World Brain. arXiv:2608.25927, 2026.

[2] H. Xu, W. Yan, A. Wang, C. Zou, S. Hong, J. Huang. Proxy2World: Learning to Generate Worlds from Lightweight Proxies without Seeing Them. arXiv:2609.35023, 2026.

Q5: Behavioral changes after RL.

Response:

  1. Real game. RL markedly improves entity-directed and compound tasks: Task 4 ("Move forward and approach the shrine ahead") rises from 40% to 90%, and Task 14 ("Cross the muddy ground ahead, and then move toward the temple to the front-left") from 20% to 70%. Beyond the benchmark, feedback from our human evaluators on a broader set of tasks likewise indicates that after RL the agent passes the navigation landmarks more reliably, deviates from the route less often, and stops more promptly. The ability to emit STOP upon completion clearly improves: over all 140 trials, episodes stuck against a wall for more than 3 s drop from 11 (SFT) to 3 (RL), directly reflecting the reward penalty on continuing after completion and failing to stop.
  2. Simulator. We tested an early (step 50) and a late (step 450) policy on 40 simulator scenes. The fraction of rollouts reaching the horizon limit (20 decision steps × 12 frames = 240 frames) drops from 82.5% to 65.0%. Within the limited horizon, a growing fraction of rollouts therefore completes the task and stops without redundant actions.

Thanks for the responses and the revision and i'll update my score on soundness and rating. Few things i wanted to clarify :-

  1. Which task does the judge score when pre-selecting checkpoints - the 14 test tasks or the simulator training tasks ?
  2. How many human evaluators judged each trial ? And how often do they agree?

We sincerely thank Reviewer ATjT for the careful and detailed review, which has substantially improved both our work and its presentation, and for raising the score. It is our pleasure that our responses addressed the earlier concerns. We clarify the two remaining points below.

Q1: Tasks used for checkpoint pre-selection.

The judge scores simulator rollouts on the RL training tasks only. The 14 real-game test tasks are never used for selection.

Q2: Human evaluators and agreement.

Each trial is judged independently by three blinded evaluators, and the final label is decided by majority vote. Since the criteria rely on observable outcomes (e.g., reaching a target landmark and stopping), disagreements are rare and are resolved by re-examining the recording. All recordings are archived under anonymized model IDs for audit. The criteria are in Appendix D.4 (Table 5), and both points are now stated in the revised Appendix D.

Reviewer_4T9HOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SHUGYO turns an action-conditioned video diffusion model into a simulated environment for training game-playing agents. The simulator accepts native keyboard and mouse actions, generates subsequent observations, and uses a vision-language model to evaluate task completion. A VLM agent is then trained with on-policy reinforcement learning inside this learned environment.

The central result is that training in the generated environment improves performance in the actual game. In Black Myth: Wukong, success increases from 58.6% to 68.6%. The paper also reports improvement from 52% to 74% in Honor of Kings: World after adapting the framework to that game.

Critical strengths
  1. Downstream validation in actual games. The evaluation goes beyond visual quality and shows that imagined experience can improve real-environment task performance.
  2. Evidence from a second game. Adaptation to another commercial game provides some support for the framework’s broader applicability.
  3. Substantial systems integration. The framework combines long-horizon simulation, task evaluation, parallel rollout collection, and policy optimization into a working training pipeline.
Critical weaknesses
  1. The source of the improvement is not fully isolated. The main comparison is between a supervised agent and the complete RL pipeline. More controlled downstream experiments would help establish which simulator and training components drive the gains. Several world-model ablations focus on video metrics rather than resulting policy performance.
  2. The real-game evaluation remains limited. The Wukong evaluation covers 14 tasks with 10 trials each, largely involving navigation and traversal. Broader task coverage would strengthen the generality claim.
  3. VLM-based evaluation needs independent validation. Human spot checks or an additional evaluation mechanism would help establish the reliability of the task-completion judgments.
Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4T9H (1/2)

We sincerely thank Reviewer 4T9H for the thoughtful and constructive feedback, which will help improve our work. Our responses are as follows.

W1: The contributions of the RL-stage components are not isolated.

Response: To isolate the source of the gain, we ran four component ablations in the real game, changing one component at a time: (i) GRPO → REINFORCE; (ii) the SHUGYO simulator → Matrix-Game 3.0 as the rollout environment; (iii) removing the two-stage curriculum; (iv) removing the STOP and SHIFT penalties. All variants start from the same WukongAgent-SFT checkpoint and are evaluated by blinded human evaluators on the same 14 tasks with 10 trials each. Results are shown below (success rate, %).

VariantCont. nav. (3)Entity nav. (4)Traversal (1)Compound (6)Overall (14)
WukongAgent-SFT83.352.590.045.058.6
GRPO → REINFORCE80.050.080.048.357.9
SHUGYO → Matrix-Game 3.076.755.080.045.057.1
w/o curriculum90.065.0100.043.363.6
w/o STOP/SHIFT penalties90.060.090.048.363.6
SHUGYO (full)100.062.5100.051.768.6
  1. Policy optimization. With REINFORCE, overall success is 57.9%, on par with SFT (58.6%): the RL gain disappears. The reward is a sparse trajectory-level score; GRPO normalizes it against rollouts from the same task and initial state, which directly amplifies the relatively successful trajectories. REINFORCE lacks this within-group baseline, and its gradient estimate has high variance when task difficulty varies widely.
  2. Rollout environment. Training in Matrix-Game 3.0 yields 57.1%, again at the SFT level and 11.5 points below SHUGYO. This ties the simulator gap measured by video metrics in Sec. 4 (FDS, CFD) directly to downstream policy performance: how closely the simulator matches the real game's control interface and dynamics determines whether RL improves real-game behavior.
  3. Curriculum. Without the curriculum, overall success drops to 63.6%, and the drop is concentrated on compound tasks (51.7% → 43.3%, below SFT), while atomic skills still improve. This is exactly what the curriculum is designed for: consolidate atomic skills before introducing compound tasks that are more sensitive to accumulated errors.
  4. Behavioral penalties. Without them, overall success also drops to 63.6%. In the real game, the penalties markedly reduce stalling and overshooting: over all 140 trials, episodes stuck against a wall for more than 3 s fall from 11 (SFT) to 3 (RL).

Conclusion. GRPO and the action-aligned SHUGYO simulator are each necessary: removing either returns performance to the SFT level. The curriculum and the penalties each add about 5 points. The full table and analysis are in the revised Appendix F.2 (Table 9).

W2: The real-game evaluation is limited in scope and centered on navigation.

Response: SHUGYO is currently evaluated on instruction-following navigation (continuous navigation, entity-directed navigation, traversal, and their compositions). We will update the Limitations paragraph in the revision to scope our claims accordingly. Within this scope, the result stands on its own: on-policy RL is carried out entirely inside a learned simulator of a closed-source commercial game, using its native keyboard-and-mouse interface, and the resulting gains are verified in the real game.

Broader settings such as combat depend on world state that is not directly visible in pixels (health, damage, enemy state). A natural direction is to insert an explicit intermediate representation between native controls and video: predict or maintain a structured world state, render it as a geometric proxy, and let a video model synthesize the appearance, as in Code World Model [1] and Proxy2World [2]. Such cascaded designs maintain state consistency more explicitly, but each step adds state modeling, language-model reasoning, and proxy rendering, which raises per-step latency and cost and currently makes them unsuitable as on-policy RL environments that require massive parallel rollouts. SHUGYO generates observations directly from native controls and is designed precisely to serve RL at high throughput. Combining the two lines is our ongoing work.

[1] Y. Chen, G. Lin, C. Zhang. Code World Model: Coding Agent as World Brain. arXiv:2608.25927, 2026.

[2] H. Xu, W. Yan, A. Wang, C. Zou, S. Hong, J. Huang. Proxy2World: Learning to Generate Worlds from Lightweight Proxies without Seeing Them. arXiv:2609.35023, 2026.

AuthorsOpenReview ↗

Response to Reviewer 4T9H (2/2)

W3: The VLM-based evaluation lacks independent validation.

Response: All real-game success rates reported in the paper are obtained by human evaluation. The VLM judge is used only to pre-select three candidate checkpoints among those saved during training; these three candidates are evaluated by humans, and we report their mean task success rate. The wording in Sec. 5 of the original submission was misleading and has been corrected in the revision.

Real-game trials are judged by multiple human evaluators who are blind to the model setting, following task-specific criteria. To validate the judge itself, in an early stage of the project we compared several commercial models (Gemini series and Qwen3.8) against human scores on n = 54 real-game rollouts (criteria in Appendix D.4):

JudgePearsonSpearmanMAE|Δ| ≤ 0.2Bias (model − human)
Gemini-3.1-Pro0.6250.5990.11185.2%+0.015
Gemini-3.6-Flash (used)0.6680.6650.10990.6%+0.079
Qwen3.8-27B0.6490.5840.13088.9%+0.100

Gemini-3.6-Flash is the best on Pearson, Spearman, MAE, and the fraction within 0.2. Its scores are slightly higher than human scores on average (+0.079), but GRPO depends only on within-group ranking, so a uniform offset does not affect the advantages; ranking agreement with humans is what matters. We therefore use Gemini-3.6-Flash for both reward and checkpoint pre-selection. We further had humans annotate 200 simulator rollouts sampled during RL training: Gemini-3.6-Flash reaches a Pearson correlation of 0.609 with human scores, with 83% within 0.2, showing that the agreement holds on generated observations as well. The evaluation protocol and this analysis are now in Sec. 5 and Appendix D.4 (Table 6).

Dear Reviewer 4T9H,

We sincerely thank you again for the positive assessment and the helpful suggestions, which guided our component ablations of the RL stage, the clarification of the human evaluation protocol, and the explicit scoping of our claims to navigation. As the discussion period is closing soon, we hope that our responses have addressed your concerns, and we would be happy to clarify any remaining questions within the remaining time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

List of Revisions

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their thoughtful and constructive feedback, and the Area Chair for handling our submission. We attach great importance to every comment and have uploaded a revised version of the paper, with all revisions highlighted in blue. For the Area Chair's convenience, the main revisions are summarized below:

  • Sec. 5 and Appendix D.4: clarified the evaluation protocol. Reported real-game success is judged by three blinded human evaluators per trial (majority vote) under task-specific criteria (Table 5). The VLM judge is used only for rewards and checkpoint pre-selection. (4T9H, ATjT, zaUx)
  • Appendix D.4, Table 6: added a judge–human agreement study on 54 real-game and 200 simulator rollouts. (4T9H, ATjT, zaUx)
  • Appendix D.5: specified checkpoint selection. Three candidates are pre-selected on the RL training tasks only, and their mean real-game success is reported. (ATjT)
  • Appendix F.2, Table 9: added real-game component ablations of the optimizer, rollout environment, curriculum, and behavioral penalties. (4T9H, zaUx)
  • Appendix F.3, Table 10: added results over four RL seeds with mean, standard deviation, and confidence intervals. (ATjT, zaUx)
  • Appendix F.4: stated the trial counts on Honor of Kings: World (10 per task). (zaUx)
  • Appendix G: added a Scope and Limitations section that explicitly restricts our claims to instruction-following navigation. (4T9H, ATjT)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.