acceptodds
Under review as a conference paper at ICLR 2027

Self-Taught Policy Improvement for Vision-Language Game Agents

Abstract

Vision-language models (VLMs) have achieved strong performance on visual question answering (QA), yet they are ineffective at playing games directly from raw pixels. This gap stems from a fundamental mismatch: VLM pre-training is based on image-grounded QA, whereas game playing requires goal-directed action generation, which is far outside the VLM pre-training distribution. Compared with raw action generation, in-game reasoning about the visual scene is structurally closer to image QA. Based on this insight, we propose a self-taught policy improvement framework that uses reasoning tokens, rather than action tokens, as the primary alignment target for VLM post-training in games, and requires no human annotation. Concretely, we first generate high-quality, state-grounded traces by rolling out the initial policy with privileged environment state information injected into the prompt, then distill these traces back into a pixel-only policy; iterating this rollout-and-distillation loop progressively improves reasoning and decision-making. Our approach raises Qwen3-VL-8B-Instruct's win rate from 0.0% to 31.3% on ProcGen-bigfish and from 13.8% to 58.1% on MiniHack, while more than doubling average Crafter achievements from 3.3 to 7.5. It also outperforms Gemini-3.5-Flash, SFT, GRPO, and GiGPO in task reward across all three games. Our approach outperforms all trained baselines in task reward with substantially fewer training samples.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language-Action, Game-Agent, Vision-language Models, Self-Improvement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FQJ4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a self taught framework to train VLM agents to play games from raw pixels called STPI. The main idea is to use reasoning tokens instead of action tokens as the main post-training target. During training, game states are injected into the prompt so the model generate more grounded reasoning and actions, with a reflection step after failed actions. The samples are filtered by a reasoning-state match rate and action success, then distilled back into a pixel-only policy by SFT, and this rollout-distillation loop is repeated for several rounds with experience replay.

Critical strengths
  1. The idea of using privileged state to get grounded reasoning traces and then distill them to a pixel-only policy is simple and easy to implement
  2. The act. vs Full comparison in Table 1 is convincing. Using the same rollouts, only supervising action tokens almost lose all the gain on the bigfish game which support the significance of reasoning tokens.
  3. The improvements over the baselines are obvious on three games.
Critical weaknesses
  1. The evaluation is not statistically solid. Each game is only evaluated on 16 episodes with single training seed and no variance is reported, so 31.3% vs 6.3% on bigfish is actually 5/16 vs 1/16 episodes. Also in Fig. 3 the reported checkpoint is the round with highest pixel-only reward, it is not clear if this is selected on the same evaluation seeds, and whether baselines get the same selection.
  2. Some results seems not consisten. Bigfish reward is integer per episode, so the mean over 16 episodes should be multiple of 1/16, but SFT/GRPO/GiGPO reward (5.41, 7.96, 7.30) are not, why?
  3. The claim of “no human annotation” and data-efficiency is a bit overclaimed. It seems that the privileged prompt contains a lot of hand-written strategy rules which is also human knowledge. And the train data only count the filtered SFT samples, the rollouts, filtered-out samples, reflection and Q parsing calls are not counted, so comparing with GRPO/GiGPO by sample number is not fair.
  4. Some important baselines are missing. The method is close to a combination of privileged teacher-student distillation such as learning by cheating, STaR/rejection sampling fine-tuning and Reflexion, but there is no comparison with STaR/RFT without privileged state, or reasoning SFT from a stronger teacher. The SFT baseline also looks weak, on Crafter it is even lower than zero-shot Qwen.
  5. The hallucination metric is 1-Q, which is the same score used to filter the training data, and Q is parsed by the VLM itself, so is it a bit circular to use it as evaluation?
Questions for authors

See weakness questions: Is the best round in Fig. 3 selected on the evaluation seeds? Why the bigfish rewards are not multiple of the episode number? Are all methods evaluated with the same 16 episodes?

Additional questions: 1. What is the total cost of each method, including rollouts that are filtered out? What is the acceptance rate of the filter in each round? 2. In the “w/o State-Grounding” ablation, is the strategy guidance also removed? How does STPI perform with only the privileged state but no hand-written strategy rules, and how does it compare to STaR/RFT without privileged information?

4 comments

Response to Reviewer FQJ4 (Part 1/2)

We sincerely thank the reviewer for your valuable suggestions.


Training-Seed Variance and Episode-Level Variance

Because of limited training resources, the original Table 1 used one training seed for each method in each game. Evaluation results are means over multiple episode seeds, and Figure 3 reports the standard deviation of STPI (full) over eight validation seeds. During rebuttal, we additionally ran SFT and STPI (full) on Bigfish with three different training seeds. Each resulting model was evaluated on 32 different test episode seeds:

Training seedSFT rewardSTPI (full) reward
05.41 ± 7.2813.88 ± 18.12
14.28 ± 7.8715.41 ± 18.31
25.94 ± 5.9012.81 ± 17.82
Std. across the three run means0.851.31

The “±” within each row is the standard deviation; the last row is the standard deviation across the three training-seed means.

We also revised the following entries in Table 1 to report standard deviations. For Bigfish and Crafter, ± is the standard deviation across test episodes. For MiniHack, each evaluation seed is first reduced to the mean reward over its 10 tasks, and ± is the standard deviation of these per-seed means across seeds:

Env./metricQwen3-VLGemini-3.5-flashSFTGRPOGiGPOSTPI act.STPI full
Bigfish reward0.56±2.313.00±4.425.41±7.287.97±10.237.31±7.935.38±9.2113.88±18.12
Bigfish survival82±3980±66156±62144±108139±87174±121271±108
MiniHack reward-0.16±0.080.41±0.120.30±0.200.23±0.290.22±0.140.26±0.210.52±0.18
Crafter reward12.1±6.314.8±7.216.9±7.916.3±8.420.5±8.215.2±5.927.9±9.3
Crafter achievements3.3±1.93.9±3.12.2±1.84.2±2.94.9±4.12.0±2.57.5±4.2
Crafter survival122±77103±86158±84142±99149±70143±119171±92

Checkpoint Selection and Seed Alignment

Test episodes are never used to select checkpoints. SFT uses its final one-epoch checkpoint. GRPO/GiGPO are evaluated every 10 epochs on 8 held-out validation episode seeds; STPI is evaluated after each round on 8 held-out validation episode seeds. In both cases, the highest-validation-reward checkpoint is selected, and validation seeds are disjoint from test seeds. Figure 3 shows STPI reward across these validation seeds; Table 1 reports the selected checkpoint on independent test seeds. Within each benchmark, every method uses exactly the same test episode seeds.


Bigfish Episode Count and Rounding

Your observation is correct. We initially used 16 episode seeds to evaluate all three environments. Because Bigfish has high evaluation variance, we later expanded Bigfish to 32 seeds. Appendix A.3.1 now states that Bigfish uses 32 held-out episodes. All Bigfish values in Table 1 are based on these 32 episodes.

MethodQwen3-VLGeminiSFTGRPOGiGPOSTPI act.STPI full
Episodes32323232323232
Total reward1896173255234172444
Exact mean0.56253.00005.406257.968757.31255.37513.875
AI-agent output0.5635.417.967.35.3813.88
Original manuscript0.563.005.417.967.305.3813.88
Corrected mean0.563.005.417.977.315.3813.88

Because we directly used an AI agent to calculate and analyze the experimental results, its inconsistent rounding produced the previous 7.96/7.30 entries. They are corrected to 7.97/7.31. Appendix A.3.1 now reports 32 Bigfish episodes.

Response to Reviewer FQJ4 (Part 2/2)


Training Cost and Sample Filtering

We agree that counting only retained SFT samples is incomplete. Here, “steps” for STPI include all interactions before filtering; generated tokens are also counted before filtering; GPU hours include rollout, reflection, parsing/filtering, and training. All GPU-hour measurements use an 8$\times$ NVIDIA H20 setup. For the RL baselines, a rollout is terminated early when the agent executes the same invalid action for 10 consecutive steps, so that uninformative rollouts do not continue to the environment step cap.

MethodBenchmarkTotal Samples (Steps)Generated Tokens (Unfiltered)GPU Hours
SFTBigfish15k05
SFTMiniHack18k08
SFTCrafter100k030
GRPOBigfish120k30M180
GRPOMiniHack242k21M200
GRPOCrafter144k50M420
GiGPOBigfish127k32M190
GiGPOMiniHack230k21M190
GiGPOCrafter150k52M440
STPI (full)Bigfish23k unfilt / 15k filt3.4M43
STPI (full)MiniHack25k unfilt / 18k filt4.4M48
STPI (full)Crafter47k unfilt / 30k filt11M80

Strategy Guidance and STPI Without Strategy Text

We tested STPI in all three environments without game-strategy text while retaining privileged information, the basic game background, and basic game rules. The table reports the mean test reward of the best checkpoint selected on the validation set:

EnvironmentWith StrategyWithout StrategyDifference (Δ\Delta)
Bigfish13.888.44-5.44
MiniHack0.520.47-0.05
Crafter27.919.62-8.28

Removing strategy guidance reduces STPI's performance in every environment. This is expected because STPI's training samples are generated through its own rollouts: removing strategy guidance reduces the quality of reasoning and decisions during rollout, which in turn reduces the quality of the samples used for SFT. We agree that “no human annotation” was too broad. STPI does not require per-step human-labeled expert trajectories, but it does use fixed human-written task rules and strategy guidance as task-level prior knowledge rather than per-step annotation.


Is the 1−Q1-Q Hallucination Metric Circular?

We clarify that QQ is not directly scored by the VLM. The VLM first parses the model's free-form reasoning into a predefined JSON format; rules then compare the parsed JSON fields with the ground-truth state to calculate the match score and hallucination rate 1−Q1-Q. The VLM only formats and parses the reasoning text and does not perform the actual scoring.

The original manuscript was ambiguous on this point. The parser is the pre-trained VLM before STPI updates, rather than the continually updated policy. We use the fixed pre-trained model because the updated policy's instruction-following and structured-parsing ability becomes weaker during training. Therefore, the hallucination metric does not involve the circular self-evaluation described in the concern.


STaR / Rejection-Sampling Fine-Tuning Without Privileged Information

RFT generates candidate responses for a given input, filters them with a verifier, and uses only samples with correct answers for SFT. STaR additionally gives the ground-truth answer as a hint after an incorrect attempt and asks the model to reconstruct the chain of thought. In game-agent settings, however, each step has no unique correct action, so correctness cannot be verified in the same way as a mathematical answer. Samples can only be filtered by reasoning quality, i.e., the 1−Q1-Q metric in the original manuscript, and STaR therefore falls back to a procedure similar to multi-round RFT.

The w/o State-Grounding ablation in Table 3 already provides this comparison. After removing state grounding, i.e., privileged information, its principle is approximately equivalent to multi-round RFT in the game setting. Its average test rewards decrease substantially in all three environments:

EnvironmentSTPI (ours)Multi-round RFT (game-domain STaR)Relative changeRelative change (%)
Bigfish13.887.97-5.91-42.6%
MiniHack0.520.29-0.23-44.2%
Crafter27.917.0-10.9-39.1%

I appreciate the authors' effort to clarify the concerns and provide new results. But I tend to remain the score as I am unconvinced by the reliability of the reported findings, particularly given the numerous numerical inconsistencies in the original manuscript. The authors also acknowledged that they "directly used an AI agent to calculate and analyze the experimental results," in the response to me, which was not adequately disclosed in the AI usage statement. This raises further concerns about the reported results and findings.

Thank you for the follow-up. We acknowledge that our earlier wording, “directly used an AI agent to calculate and analyze the experimental results,” was insufficiently precise. We take responsibility for both this wording and the two rounding errors.

The raw trajectories and episode outcomes came from actual model–environment interactions under fixed protocols. AI assistance was used in post-processing, including numerical aggregation and statistical analysis. It did not generate or alter raw outcomes, select favorable test seeds, or select checkpoints using test results.

The two Bigfish discrepancies were decimal-reporting errors: 255/32 = 7.96875, corrected from 7.96 to 7.97, and 234/32 = 7.3125, corrected from 7.30 to 7.31. Neither correction changes the method ranking or the associated conclusions. We also clarified that checkpoints are selected using eight held-out validation seeds, validation and test seeds are disjoint, all methods share the same test seeds, and the reported Bigfish results use 32 test episodes.

Beyond these corrections, we provided additional experimental evidence. Across three Bigfish training seeds, STPI achieves mean rewards of 13.88, 15.41, and 12.81, compared with SFT’s 5.41, 4.28, and 5.94, with each model evaluated on the same 32 held-out test seeds. We also added standard deviations and their definitions, and reported unfiltered interactions, generated tokens, and GPU hours.

We agree that the original AI-use disclosure should have been more explicit. The revised AI Use Statement now expressly includes assistance with “implementing parts of the experimental code and performing statistical analysis of some experimental results,” while retaining the authors’ responsibility for the final content and findings.

We therefore respectfully disagree that AI assistance, by itself, establishes that the findings are unreliable. The original reporting errors warranted scrutiny, but the corrected results, clarified protocols, author verification, and additional experiments also warrant explicit consideration. Continuing to question reliability without identifying a specific unresolved discrepancy in the revised evidence does not provide a concrete basis for discounting that evidence. We welcome any remaining specific concern and ask that the reliability assessment address the revised manuscript and rebuttal evidence, rather than rely solely on the original errors or the use of AI assistance.

Reviewer_G771OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes STPI, an iterative self-training method for vision-language agents in games. During training, the model is provided with privileged environmental states to generate reasoning, reflection, and action trajectories. These trajectories are then filtered based on state consistency and the outcomes of action execution, and subsequently distilled into a policy that relies solely on visual inputs. Experiments on Bigfish, MiniHack, and Crafter demonstrate that this method outperforms the SFT, GRPO, and GiGPO baselines.

Critical strengths

1.The method is practical. It leverages simulator states to automatically generate and filter supervised data, reducing reliance on human demonstrations, while eliminating the need for privileged state inputs during the inference phase. 2.The supervision of reasoning is evaluated through targeted comparisons. By contrasting a model supervised solely on actions against one supervised on both reasoning and actions—using the same privileged trajectories—the study demonstrates the value of reasoning trajectories for policy learning. 3.The component analysis is comprehensive. It covers state injection, reflection, sample filtering, and experience replay, while tracking performance across multiple training rounds as well as changes in reasoning hallucinations.

Critical weaknesses
  1. Since the various methods differ in their training data sources, scale, and supervisory information, it is currently difficult to distinguish between the contributions of method design and data advantages when making comparisons.
  2. Given the limited number of baseline models, it is difficult to directly demonstrate the effectiveness of the proposed method. Could you include experimental results for additional models? Furthermore, do the conclusions hold true for both stronger and weaker models?
  3. Figure 3 illustrates the training iteration that yielded the highest reward when the primary model used pixel-based inputs. It should be clarified whether this selection process utilized an independent validation seed, and whether a consistent set of rules was applied across all baselines to ensure that the evaluation set did not influence model selection.
  4. On Bigfish, the win rate against multiple baselines is zero. It is recommended to provide an analysis of the reasons for these failures and to expand the scale of the evaluation in order to confirm the stability of the observed performance differences.
Questions for authors

ref weakness

2 comments

Response to Reviewer G771

We sincerely thank the reviewer for these important questions.


Differences in Training Data, Supervision, and Privileged Information

Different methods intrinsically require different forms and amounts of training data and different supervision signals. For STPI, using reasoning generated with privileged information as the supervision signal is part of the method's design and motivation. We therefore provide controlled comparisons and ablations to clarify the contribution of each component as far as possible.

To isolate the benefit of privileged information itself, Table 1 compares STPI (act.) and STPI (full) under the same data volume. During rollout, both variants receive the same privileged information; the only difference is whether the training labels contain action text alone or both reasoning and action text. This controlled comparison demonstrates the role of reasoning as the supervision bridge, which the other two reviewers also identified as a critical strength.

Regarding data volume, on all three benchmarks the total training data used by the SFT and RL baselines are no less than those used by STPI, and are substantially larger in some cases. Nevertheless, STPI (full) achieves the best game performance in every environment, supporting the training-efficiency advantage of the proposed method.


Comparison Across Backbone Sizes

Because rebuttal time and compute source were limited, we compared SFT and STPI on Bigfish using backbones of different sizes:

BackboneSFTSTPI (full)
Qwen3-VL-2B4.358.13
Qwen3-VL-8B5.4113.88
Qwen3-VL-32B5.7814.91

STPI clearly outperforms SFT at every backbone size. However, reducing the backbone from 8B to 2B causes a larger performance drop for STPI than for SFT. SFT mainly depends on the quality of its pre-annotated data, and we use the same training data for every backbone size. In contrast, STPI depends on trajectories generated by the model itself using privileged information, so the reasoning ability of the base model has a stronger influence on STPI. The 2B model produces lower-quality reasoning during privileged rollout. In Bigfish, we observe little difference in privileged-rollout reasoning quality between the 8B and 32B models, so scaling from 8B to 32B brings only a modest additional gain. Nevertheless, STPI remains better than SFT at every tested model size.


Checkpoint Selection

Our protocol is:

  • SFT: use the final checkpoint after one complete epoch.
  • GRPO/GiGPO: evaluate every 10 epochs on 8 held-out validation episode seeds, disjoint from test seeds, and select the highest-validation-reward checkpoint.
  • STPI: evaluate after every self-distillation round on 8 held-out validation episode seeds, also disjoint from test seeds, and select the highest-validation-reward checkpoint.

Figure 3 shows the reward on multiple validation seeds after each training round. Table 1 reports test performance from the best checkpoint selected on the validation set.


Why Several Bigfish Methods Have 0% Win Rate

In Bigfish, success requires the agent to eat 30 smaller fish. Touching a larger fish causes immediate death, and exceeding the maximum number of steps also counts as failure. This is therefore a strict success criterion: baseline methods can generally eat some fish and obtain nonzero reward, but completing 30 consecutive successful captures without dying is difficult.

Completing an episode requires a low hallucination rate and reliable pixel-based judgments of the relative sizes and positions of nearby fish. For zero-shot Qwen3-VL and Gemini-3.5-flash, more than 80% of test episodes end within 150 steps because the agent is eaten by a larger fish. STPI (full) has a lower reasoning hallucination rate and survives longer, giving it a meaningful probability of completing the game.

Stability under more evaluation episodes

Because Bigfish has high episode-level variance, we extended the same STPI checkpoint from 32 to 64 held-out test episodes:

Test episodesWinning rateRewardSurvival steps
3231.3%13.88 ± 18.12271 ± 108
6426.6%12.91 ± 17.25283 ± 119

The mean results over 64 episodes differ only slightly from those over 32 episodes. We therefore retain the 32-episode results in Table 1.

AuthorsOpenReview ↗

Follow-up to Reviewer G771

Dear Reviewer G771,

Thank you again for your time and constructive review of our work. As the discussion period progresses, we wanted to check whether our response and the revised manuscript have addressed your questions and concerns. Please let us know if anything remains unclear or if you have any further questions, and we will be glad to clarify.

Best regards, The Authors

Reviewer_BbVBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes STPI, a self-taught way to improve vision-language model (VLM) game agents. In each round, Qwen3-VL-8B plays with the true game state and a hand-written strategy list added to its prompt. It writes a reasoning trace and an action, and after a failed action it reflects on the outcome and writes a corrected action. A step is kept for training only if the action succeeds and the reasoning matches the true state (match score Q ≥ 0.7). Q is computed by having the VLM convert its own reasoning to JSON and comparing it with the simulator's state. The true state is then removed from the prompts, and the model is fine-tuned (SFT) on the full reasoning, reflection and action text, plus older-round data via a replay buffer. The authors describe this as a KL projection and a REINFORCE-like update. On ProcGen-bigfish, MiniHack and Crafter, STPI beats zero-shot Qwen3-VL and Gemini-3.5-Flash, SFT on expert actions, GRPO and GiGPO. It reports 31.3% vs. 6.3% win rate on bigfish, 58.1% vs. 53.8% on MiniHack, and 7.5 vs. 4.9 achievements on Crafter, using 5–13× less training data than the RL baselines. Training on actions alone loses most of the gain.

Critical strengths
  • The idea is simple and practical: training-time access to the game state helps the model produce well-grounded reasoning, which is then distilled into a policy that plays from pixels only.

  • The action-only ablation (STPI-act) is informative. It shows the gain comes mainly from training on reasoning, not on actions.

  • Results are consistent across three games, and the ablations are clear. The token-level NLL analysis gives some evidence for the proposed mechanism.

Critical weaknesses

I think the overall quality of this paper is good. I just have one concern:

W1. The RL baselines seem under-tuned. On MiniHack, GRPO (36.9% win) and GiGPO (32.5%) are worse than zero-shot Gemini (50.0%) and plain SFT (53.8%). That suggests the RL setup, rather than RL as an approach, is the bottleneck. The RL settings also differ from STPI's:

  • GRPO and GiGPO are capped at 512 output tokens, while STPI has no listed cap, which could cut off reasoning.

  • They sample at temperature 0.4 vs. 0.7 for STPI.

  • They use a constant learning rate, a group size of 8, and one fixed hyperparameter setting with no reported tuning.

No RL learning curves are shown, so it is unclear whether these runs converged. Learning curves, a small hyperparameter sweep, and matched output-length limits are needed before the claim of "outperforming RL" holds.

Questions for authors
  • For every method, what are the total environment steps, generated tokens and GPU-hours?

  • Was the checkpoint round chosen using the evaluation seeds? How were baseline checkpoints chosen?

4 comments

Response to Reviewer BbVB (Part 1/2)

We sincerely thank the reviewer for your valuable suggestions.


Were the RL Baselines Sufficiently Tuned?

Why RL is weaker on MiniHack

The weaker performance of GRPO and GiGPO than Gemini and SFT on MiniHack is expected. The main reason is the task's reward structure rather than insufficient hyper-parameter tuning. Under MiniHack's default setting, the agent receives +1 only after completing the final task and -0.01 for an invalid action, such as walking into a wall or interacting with an item before its preconditions are satisfied. This reward is highly sparse, making effective step-level credit assignment difficult during long rollouts.

Our failure-case analysis shows a corresponding action-space collapse: GRPO and GiGPO mostly learn movement actions, but rarely invoke task-critical verbs such as PICKUP or WEAR. They consequently learn to avoid collisions rather than to use key skills to complete the task. SFT instead clones expert trajectories containing more diverse key actions. Although Gemini-3.5-flash is not trained in the environment, its stronger overall capability and lower visual hallucination rate allow it to select key skill actions more often. In contrast, on environments with denser intermediate rewards—Bigfish rewards each fish eaten and Crafter rewards each achievement—RL generally matches or exceeds SFT and Gemini-3.5-flash.

Matched output length

Appendix Table 5 lists the maximum response length as 512 for GRPO and GiGPO and 256 for STPI. The MiniHack output statistics are:

MethodAvg. output tokensTruncation rate
STPI91.00.00%
GRPO92.90.87%
GiGPO94.20.29%

The methods therefore have comparable output lengths, and RL outputs are slightly longer on average. Their truncation rates are all very low, indicating that RL reasoning chains are generally not truncated and that the output limits are sufficient for both RL and STPI.

Why rollout temperatures differ

During training, we found that setting the RL temperature to 0.7 caused many action-text parsing failures early in training, making exploration difficult and producing worse models than those reported in Table 1. We therefore used 0.4, the default recommended by the official GiGPO repository. STPI uses privileged state grounding as an anchor, which reduces output-format parsing failures. After filtering, one SFT round makes the format reliably parseable, so we use 0.7 to preserve reasoning diversity.

Hyper-parameter search

We swept learning rates {5×10−5,1×10−5,1×10−6}\{5\times10^{-5},1\times10^{-5},1\times10^{-6}\} and group sizes {16,8,4}\{16,8,4\}. Learning rate has different effects across environments: we use 1×10−51\times10^{-5} for Bigfish and 1×10−61\times10^{-6} for MiniHack and Crafter. Group sizes 8 and 16 perform similarly across all three environments, while group size 4 performs slightly worse. The table reports GRPO's average reward:

EnvironmentGroup 16Group 8Group 4
Bigfish7.287.976.03
MiniHack0.240.230.18
Crafter15.816.313.9

Appendix Table 4 reports the selected per-environment learning rates, and Appendix Table 5 reports the shared settings used in the paper, including rollout group size 8.

Convergence evidence

We additionally inspected the GRPO/GiGPO learning curves. Their performance largely converges within the maximum budget of 50 epochs. In the revised appendix, each marker pools the current epoch and the preceding four epochs (320 episodes) and reports their mean episode return; the shading is one sample standard deviation.

Response to Reviewer BbVB (Part 2/2)


Environment Steps, Generated Tokens, and GPU Hours

The total training costs of all methods are shown below. To avoid over-claiming efficiency from retained samples alone, STPI's sample count reports both pre-filter interactions and retained samples; generated tokens are counted before filtering, and GPU hours include rollout, reflection, state parsing/filtering, and optimization. All GPU-hour measurements use an 8$\times$ NVIDIA H20 setup. For the RL baselines, a rollout is terminated early when the agent executes the same invalid action for 10 consecutive steps, so that uninformative rollouts do not continue to the environment step cap.

MethodEnvironmentSamples / environment stepsGenerated tokens (pre-filter)GPU hours
SFTBigfish15k05
SFTMiniHack18k08
SFTCrafter100k030
GRPOBigfish120k30M180
GRPOMiniHack242k21M200
GRPOCrafter144k50M420
GiGPOBigfish127k32M190
GiGPOMiniHack230k21M190
GiGPOCrafter150k52M440
STPI (full)Bigfish23k pre-filter / 15k retained3.4M43
STPI (full)MiniHack25k pre-filter / 18k retained4.4M48
STPI (full)Crafter47k pre-filter / 30k retained11M80

Checkpoint Selection

  • SFT: we use the final checkpoint after one complete training epoch.
  • GRPO/GiGPO: every 10 epochs, we evaluate on 8 held-out validation episode seeds, disjoint from test seeds, and select the checkpoint with the highest validation reward.
  • STPI: after every self-distillation round, we evaluate on 8 held-out validation episode seeds, also disjoint from test seeds, and select the highest-validation-reward checkpoint.

Figure 3 shows the reward on multiple validation seeds after each training round. The results in Table 1 are test results from the best checkpoint selected on the validation set.

Reviewer_BbVBOpenReview ↗

Reply

Thank you for the response and revision. I stand corrected: Table 5 does list a 256-token cap for STPI, and the new learning curves, validation-based checkpoint selection, and cost table address my concerns.

Thank you for your response. We are pleased that our additional experiments and clarifications have addressed your concerns. Should you have any further questions you would like to discuss, please feel free to let us know—we would be happy to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.