- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposed a self taught framework to train VLM agents to play games from raw pixels called STPI. The main idea is to use reasoning tokens instead of action tokens as the main post-training target. During training, game states are injected into the prompt so the model generate more grounded reasoning and actions, with a reflection step after failed actions. The samples are filtered by a reasoning-state match rate and action success, then distilled back into a pixel-only policy by SFT, and this rollout-distillation loop is repeated for several rounds with experience replay.
Critical strengths
- The idea of using privileged state to get grounded reasoning traces and then distill them to a pixel-only policy is simple and easy to implement
- The act. vs Full comparison in Table 1 is convincing. Using the same rollouts, only supervising action tokens almost lose all the gain on the bigfish game which support the significance of reasoning tokens.
- The improvements over the baselines are obvious on three games.
Critical weaknesses
- The evaluation is not statistically solid. Each game is only evaluated on 16 episodes with single training seed and no variance is reported, so 31.3% vs 6.3% on bigfish is actually 5/16 vs 1/16 episodes. Also in Fig. 3 the reported checkpoint is the round with highest pixel-only reward, it is not clear if this is selected on the same evaluation seeds, and whether baselines get the same selection.
- Some results seems not consisten. Bigfish reward is integer per episode, so the mean over 16 episodes should be multiple of 1/16, but SFT/GRPO/GiGPO reward (5.41, 7.96, 7.30) are not, why?
- The claim of “no human annotation” and data-efficiency is a bit overclaimed. It seems that the privileged prompt contains a lot of hand-written strategy rules which is also human knowledge. And the train data only count the filtered SFT samples, the rollouts, filtered-out samples, reflection and Q parsing calls are not counted, so comparing with GRPO/GiGPO by sample number is not fair.
- Some important baselines are missing. The method is close to a combination of privileged teacher-student distillation such as learning by cheating, STaR/rejection sampling fine-tuning and Reflexion, but there is no comparison with STaR/RFT without privileged state, or reasoning SFT from a stronger teacher. The SFT baseline also looks weak, on Crafter it is even lower than zero-shot Qwen.
- The hallucination metric is 1-Q, which is the same score used to filter the training data, and Q is parsed by the VLM itself, so is it a bit circular to use it as evaluation?
Questions for authors
See weakness questions: Is the best round in Fig. 3 selected on the evaluation seeds? Why the bigfish rewards are not multiple of the episode number? Are all methods evaluated with the same 16 episodes?
Additional questions: 1. What is the total cost of each method, including rollouts that are filtered out? What is the acceptance rate of the filter in each round? 2. In the “w/o State-Grounding” ablation, is the strategy guidance also removed? How does STPI perform with only the privileged state but no hand-written strategy rules, and how does it compare to STaR/RFT without privileged information?
4 comments
Response to Reviewer FQJ4 (Part 2/2)
Training Cost and Sample Filtering
We agree that counting only retained SFT samples is incomplete. Here, “steps” for STPI include all interactions before filtering; generated tokens are also counted before filtering; GPU hours include rollout, reflection, parsing/filtering, and training. All GPU-hour measurements use an 8$\times$ NVIDIA H20 setup. For the RL baselines, a rollout is terminated early when the agent executes the same invalid action for 10 consecutive steps, so that uninformative rollouts do not continue to the environment step cap.
| Method | Benchmark | Total Samples (Steps) | Generated Tokens (Unfiltered) | GPU Hours |
|---|---|---|---|---|
| SFT | Bigfish | 15k | 0 | 5 |
| SFT | MiniHack | 18k | 0 | 8 |
| SFT | Crafter | 100k | 0 | 30 |
| GRPO | Bigfish | 120k | 30M | 180 |
| GRPO | MiniHack | 242k | 21M | 200 |
| GRPO | Crafter | 144k | 50M | 420 |
| GiGPO | Bigfish | 127k | 32M | 190 |
| GiGPO | MiniHack | 230k | 21M | 190 |
| GiGPO | Crafter | 150k | 52M | 440 |
| STPI (full) | Bigfish | 23k unfilt / 15k filt | 3.4M | 43 |
| STPI (full) | MiniHack | 25k unfilt / 18k filt | 4.4M | 48 |
| STPI (full) | Crafter | 47k unfilt / 30k filt | 11M | 80 |
Strategy Guidance and STPI Without Strategy Text
We tested STPI in all three environments without game-strategy text while retaining privileged information, the basic game background, and basic game rules. The table reports the mean test reward of the best checkpoint selected on the validation set:
| Environment | With Strategy | Without Strategy | Difference () |
|---|---|---|---|
| Bigfish | 13.88 | 8.44 | -5.44 |
| MiniHack | 0.52 | 0.47 | -0.05 |
| Crafter | 27.9 | 19.62 | -8.28 |
Removing strategy guidance reduces STPI's performance in every environment. This is expected because STPI's training samples are generated through its own rollouts: removing strategy guidance reduces the quality of reasoning and decisions during rollout, which in turn reduces the quality of the samples used for SFT. We agree that “no human annotation” was too broad. STPI does not require per-step human-labeled expert trajectories, but it does use fixed human-written task rules and strategy guidance as task-level prior knowledge rather than per-step annotation.
Is the Hallucination Metric Circular?
We clarify that is not directly scored by the VLM. The VLM first parses the model's free-form reasoning into a predefined JSON format; rules then compare the parsed JSON fields with the ground-truth state to calculate the match score and hallucination rate . The VLM only formats and parses the reasoning text and does not perform the actual scoring.
The original manuscript was ambiguous on this point. The parser is the pre-trained VLM before STPI updates, rather than the continually updated policy. We use the fixed pre-trained model because the updated policy's instruction-following and structured-parsing ability becomes weaker during training. Therefore, the hallucination metric does not involve the circular self-evaluation described in the concern.
STaR / Rejection-Sampling Fine-Tuning Without Privileged Information
RFT generates candidate responses for a given input, filters them with a verifier, and uses only samples with correct answers for SFT. STaR additionally gives the ground-truth answer as a hint after an incorrect attempt and asks the model to reconstruct the chain of thought. In game-agent settings, however, each step has no unique correct action, so correctness cannot be verified in the same way as a mathematical answer. Samples can only be filtered by reasoning quality, i.e., the metric in the original manuscript, and STaR therefore falls back to a procedure similar to multi-round RFT.
The w/o State-Grounding ablation in Table 3 already provides this comparison. After removing state grounding, i.e., privileged information, its principle is approximately equivalent to multi-round RFT in the game setting. Its average test rewards decrease substantially in all three environments:
| Environment | STPI (ours) | Multi-round RFT (game-domain STaR) | Relative change | Relative change (%) |
|---|---|---|---|---|
| Bigfish | 13.88 | 7.97 | -5.91 | -42.6% |
| MiniHack | 0.52 | 0.29 | -0.23 | -44.2% |
| Crafter | 27.9 | 17.0 | -10.9 | -39.1% |
I appreciate the authors' effort to clarify the concerns and provide new results. But I tend to remain the score as I am unconvinced by the reliability of the reported findings, particularly given the numerous numerical inconsistencies in the original manuscript. The authors also acknowledged that they "directly used an AI agent to calculate and analyze the experimental results," in the response to me, which was not adequately disclosed in the AI usage statement. This raises further concerns about the reported results and findings.
Thank you for the follow-up. We acknowledge that our earlier wording, “directly used an AI agent to calculate and analyze the experimental results,” was insufficiently precise. We take responsibility for both this wording and the two rounding errors.
The raw trajectories and episode outcomes came from actual model–environment interactions under fixed protocols. AI assistance was used in post-processing, including numerical aggregation and statistical analysis. It did not generate or alter raw outcomes, select favorable test seeds, or select checkpoints using test results.
The two Bigfish discrepancies were decimal-reporting errors: 255/32 = 7.96875, corrected from 7.96 to 7.97, and 234/32 = 7.3125, corrected from 7.30 to 7.31. Neither correction changes the method ranking or the associated conclusions. We also clarified that checkpoints are selected using eight held-out validation seeds, validation and test seeds are disjoint, all methods share the same test seeds, and the reported Bigfish results use 32 test episodes.
Beyond these corrections, we provided additional experimental evidence. Across three Bigfish training seeds, STPI achieves mean rewards of 13.88, 15.41, and 12.81, compared with SFT’s 5.41, 4.28, and 5.94, with each model evaluated on the same 32 held-out test seeds. We also added standard deviations and their definitions, and reported unfiltered interactions, generated tokens, and GPU hours.
We agree that the original AI-use disclosure should have been more explicit. The revised AI Use Statement now expressly includes assistance with “implementing parts of the experimental code and performing statistical analysis of some experimental results,” while retaining the authors’ responsibility for the final content and findings.
We therefore respectfully disagree that AI assistance, by itself, establishes that the findings are unreliable. The original reporting errors warranted scrutiny, but the corrected results, clarified protocols, author verification, and additional experiments also warrant explicit consideration. Continuing to question reliability without identifying a specific unresolved discrepancy in the revised evidence does not provide a concrete basis for discounting that evidence. We welcome any remaining specific concern and ask that the reliability assessment address the revised manuscript and rebuttal evidence, rather than rely solely on the original errors or the use of AI assistance.
Response to Reviewer FQJ4 (Part 1/2)
We sincerely thank the reviewer for your valuable suggestions.
Training-Seed Variance and Episode-Level Variance
Because of limited training resources, the original Table 1 used one training seed for each method in each game. Evaluation results are means over multiple episode seeds, and Figure 3 reports the standard deviation of STPI (full) over eight validation seeds. During rebuttal, we additionally ran SFT and STPI (full) on Bigfish with three different training seeds. Each resulting model was evaluated on 32 different test episode seeds:
The “±” within each row is the standard deviation; the last row is the standard deviation across the three training-seed means.
We also revised the following entries in Table 1 to report standard deviations. For Bigfish and Crafter, ± is the standard deviation across test episodes. For MiniHack, each evaluation seed is first reduced to the mean reward over its 10 tasks, and ± is the standard deviation of these per-seed means across seeds:
Checkpoint Selection and Seed Alignment
Test episodes are never used to select checkpoints. SFT uses its final one-epoch checkpoint. GRPO/GiGPO are evaluated every 10 epochs on 8 held-out validation episode seeds; STPI is evaluated after each round on 8 held-out validation episode seeds. In both cases, the highest-validation-reward checkpoint is selected, and validation seeds are disjoint from test seeds. Figure 3 shows STPI reward across these validation seeds; Table 1 reports the selected checkpoint on independent test seeds. Within each benchmark, every method uses exactly the same test episode seeds.
Bigfish Episode Count and Rounding
Your observation is correct. We initially used 16 episode seeds to evaluate all three environments. Because Bigfish has high evaluation variance, we later expanded Bigfish to 32 seeds. Appendix A.3.1 now states that Bigfish uses 32 held-out episodes. All Bigfish values in Table 1 are based on these 32 episodes.
Because we directly used an AI agent to calculate and analyze the experimental results, its inconsistent rounding produced the previous 7.96/7.30 entries. They are corrected to 7.97/7.31. Appendix A.3.1 now reports 32 Bigfish episodes.