- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a two-stage framework for using a pretrained LLM (Qwen3-1.7B; Qwen2.5-3B for Sokoban/FrozenLake) as the backbone of an RL policy. In Stage-A, GPT-5.5 is prompted with the environment description, the structured-state definition and hand-written task rules. It produces discrete label fields (“state-fact” and “rule-conditioned”, including a high-level goal field) and a deterministic Python labeler. The LLM receives encoded structured-state and rule tokens and is trained with LoRA so that linear heads can predict these labels from its representation . In Stage-B, the heads are discarded, and the same LoRA and encoders, together with a new policy head and critic, are trained with PPO for ~1M environment steps. The method reports large gains over PPO, LLM-Explorer, LaRe and SR on five Atari games and Crafter, and over PPO on Sokoban/FrozenLake.
Critical strengths
- The pipeline clearly described. The full labeler-generation prompt and most hyperparameters are provided.
- Ablation study is informative. Random Label shows that correct labels matter, not just an extra supervised stage. Single-Stage compares sequential and joint training at a matched budget.
- The gains over the identical model without Stage-A are consistent across environments. On Sokoban this holds even with Stage-A data from a random policy.
Critical weaknesses
- By construction, the Stage-A labels are a deterministic function of the pre-decision structured state and the task rules, which is exactly the policy input at every step. The labeler can therefore be run at test time at negligible cost. Stage-A trains a 1.7B model to approximately reproduce this function (92–97% all-field accuracy, Table 3), and the LLM then runs at every environment step in Stage-B. In my view, the key missing comparison is a non-LLM policy with the same access to the rule-derived knowledge: an MLP-PPO policy whose input is the structured state concatenated with the exact labeler outputs. The rule-conditioned labels already incorporate the thresholds. Together with a tuned MLP-PPO on the structured state alone, this would compare LLM and MLP backbones both with and without the rule-derived knowledge, and directly isolate the contribution of the LLM. Without it, I do not think the central claim that the LLM acquires decision-relevant knowledge is supported.
- Contribution 1 is “formulating decision-relevant knowledge as an explicit learning objective before policy optimization”. It is seemed like auxiliary supervision of the representation followed by RL. This is a well-established line: auxiliary tasks for RL (e.g., UNREAL, Jaderberg et al., 2017), which the paper does not discuss, and representation fine-tuning before value or policy learning (e.g., Digi-Q, which the authors cite). The potentially novel element is that an LLM designs the auxiliary tasks and writes the labeler from rule text. However, this element is not evaluated enough. Moreover, a separate model is trained for each environment, with environment-specific rules, labeler and data. There are no experiments on unseen rules, rephrased rules or multi-environment training. These are the settings where I would expect an LLM backbone and the natural-language rule interface to pay off. The per-task setup cost makes the approach hard to scale beyond the evaluated games.
- The PPO baseline puts a 1.7B LoRA backbone under PPO from scratch, which is known to be hard to optimize. Its numbers fall well below standard references: Crafter: PPO here reaches 4.82, whereas a tuned pixel-based PPO with a ResNet encoder reaches 15.6 on the official leaderboard at the same 1M-step budget. The proposed method (21.99) is on par with Achievement Distillation (21.8) and below SPRING (27.3). Atari: the reported scores are of the same order as standard pixel-based PPO at 40M frames (Schulman et al., 2017). OCAtari (Delfosse et al., 2024) reports that PPO on object-centric states performs on par with or better than pixel-based PPO. So I consider a properly tuned non-LLM PPO on the same structured states as a strong and necessary reference.
Questions for authors
- How do the following perform: a tuned MLP-PPO on the structured state, and an MLP-PPO on the structured state concatenated with the exact labeler outputs?
- What happens if the goal field is removed from Stage-A supervision? Since this field encodes the hand-designed rule priorities, the answer would show how much of the gain comes from the rules themselves.
- What is the score of the CleanRL expert used to collect Atari Stage-A data, and how do the results change if Stage-A data come from a random policy, as for Sokoban?
4 comments
Question 3: CleanRL Expert Performance and the Impact of Random-Policy Data on Stage-B
Thank you for raising this question. First, we report the performance of the CleanRL expert policies used to collect Atari Stage-A data. The following results show the mean scores and standard deviations over 100 episodes. The expert policies were trained using the same PPO configuration as CleanRL PPO, with an online interaction budget of 20 million steps.
| Environment | Expert Policy |
|---|---|
| Assault | 8,040.85 ± 2,854.50 |
| BeamRider | 3,885.68 ± 1,440.66 |
| DemonAttack | 55,720.10 ± 28,722.96 |
| SeaQuest | 2,073.42 ± 113.26 |
| Alien | 2,664.12 ± 445.27 |
These results characterize the performance of the expert policies used for data collection. We further examine how replacing expert-policy or human-collected Stage-A data with random-policy data affects subsequent Stage-B performance.
Following your suggestion, we conducted additional experiments to investigate the impact of Stage-A data collection policies on subsequent Stage-B performance. Specifically, we replaced the original expert-policy or human-collected data with random-policy data, while keeping all other training settings unchanged. We evaluated this setting on SeaQuest, Assault, and Crafter. The results are shown below.
| Environment | Original Data | Random-Policy Data | Performance Drop |
|---|---|---|---|
| SeaQuest | 1618.6 ± 113.7 | 1446.40 ± 56.69 | 10.6% |
| Assault | 5188.8 ± 807.5 | 4217.85 ± 1996.89 | 18.7% |
| Crafter | 21.99 ± 0.96 | 13.04 ± 0.14 | 40.7% |
The results show that our method maintains effective performance across all three environments when using random-policy data. In particular, SeaQuest and Assault retain 89.4% and 81.3% of their original mean performance, respectively. This suggests that our method does not necessarily rely on expert-collected states to learn effectively.
To better understand these performance differences, we further analyzed the distributions of task-related events in the two datasets.
In SeaQuest, random-policy data covers 6 out of 7 event categories, compared with 7 out of 7 for expert-policy data. The proportion of states requiring the agent to surface for oxygen decreases from 31.29% to 1.35%, while states involving the delivery of a full load of divers are entirely absent. Meanwhile, random-policy data contains more states involving collision threats and nearby divers. These results indicate clear differences in both event coverage and the distribution of decision-making situations.
In Crafter, achievement-event coverage decreases from 22/22 to 8/22, while the number of unique achievement combinations drops from 530 to 25. This indicates that random policies have difficulty reaching states associated with more advanced task progress, which is consistent with the larger performance drop observed in this environment.
In Assault, both datasets cover 5 out of 6 decision-goal event categories, although the final performance still decreases.
Importantly, Stage-A does not use expert actions or policy outputs as supervision targets. As described in Section 3.2 and Appendix A.2, all supervision labels are generated from structured states and task rules through deterministic annotation programs. The LLM learns environmental facts and rule-conditioned decision-relevant knowledge rather than directly imitating expert actions.
Overall, these additional experiments show that expert-collected data is not a necessary condition for our method to achieve performance improvements. The differences in Stage-B performance are closely related to the event categories and state distributions covered by the two data collection policies. In complex environments such as Crafter, random policies have difficulty collecting states associated with advanced task progress. Consequently, the LLM receives fewer training examples for learning decision-relevant knowledge in these situations, leading to a larger performance drop in Stage-B. In contrast, the random and expert policies cover nearly the same event categories in the two Atari environments, where the performance drops are relatively smaller.
Question 2 Ablation Study on the Goal Field in Stage-A
Thank you for raising this question. To further investigate the impact of the Goal field on Stage-A training and final policy performance, we conducted an additional No Goal ablation experiment. We removed the Goal field from Stage-A supervision and compared the final policy performance across three Atari environments.
First, the Goal field in our paper represents a high-level decision objective. It indicates what the agent should prioritize based on the current environment state and task rules. Specifically, the model needs to combine environmental facts and task rules to determine the decision objective associated with the current state. For example, in SeaQuest, when oxygen is low, the current objective should be to return to the surface to replenish oxygen, rather than continue exploring or collecting divers. Therefore, the Goal field provides supervision that connects environmental facts and rule conditions to the current decision objective.
We conducted the No Goal ablation experiments in Assault, Alien, and SeaQuest. The results are shown below:
| Environment | Full Method | No Goal |
|---|---|---|
| Assault | 3679.68 ± 501.9 | 4200.6 ± 376.0 |
| Alien | 1520.70 ± 13.1 | 1494.0 ± 274.0 |
| SeaQuest | 1339.77 ± 47.66 | 417.93 ± 38.51 |
The results show that the contribution of the Goal field varies across environments.
In Assault, removing Goal leads to a higher mean score. However, given the variability in both results, we do not interpret this change as a performance improvement. This may be related to redundancy in the Goal labels in Assault. For example, track_attack_target accounts for 74.9% of the training data. Some of the other decision-related fields already contain information needed to infer the current goal. Therefore, explicitly supervising the Goal field may provide limited additional benefits.
In Alien, the mean score after removing Goal remains close to that of the Full Method. This suggests that, under the current experimental setting, the additional contribution of the Goal field is not pronounced.
In contrast, SeaQuest shows a much stronger dependence on Goal supervision. After removing Goal, the mean score drops from 1339.77 to 417.93, a decrease of approximately 68.8%. We believe this is related to the decision-related information contained in different high-level goals in SeaQuest. For example, return_for_air and collect_diver represent different objectives: returning to the surface to replenish oxygen and collecting divers, respectively. In addition, oxygen management is critical to survival and score accumulation in SeaQuest. If the agent fails to return to the surface in time to replenish oxygen, it may run out of oxygen and lose the opportunity to continue accumulating scores. Therefore, the model needs to determine which objective should take priority between survival and other task goals based on the current environment state and task rules. This makes the Goal field an important source of high-level decision supervision, helping the model understand which objective should be prioritized in the current state.
Overall, we conclude that the contribution of the Goal field depends on the task characteristics of each environment and the decision-related information it contains. In Assault and Alien, some Goal information may already be captured by other supervision fields, resulting in relatively small performance changes after removing Goal. In SeaQuest, however, Goal provides more important supervision for high-level decision objectives.
Response to Rebuttal
I thank the authors for the detailed rebuttal and the additional experiments. The new MLP-PPO baselines, the random-policy Stage-A data and the goal ablation address part of my concerns. However, not all of my points were answered. In particular, my concern about the limited generalization and applicability of the method remains: a separate model is still trained per environment, with environment-specific rules, labeler and data, and there is no evaluation on unseen or rephrased rules, nor any transfer across environments. I therefore keep my original score.
We thank the reviewer for the constructive feedback and valuable suggestions. In response, we conducted additional experiments to further examine the role of the policy backbone and the impact of Stage-A data collection policies. We present our responses and the corresponding experimental results below.
Question 1: Comparison with MLP-PPO Baselines and the Role of Stage-A
Thank you for raising this important question. Following your suggestion, we conducted additional experiments with two MLP-PPO baselines: MLP-PPO using only structured states as input and MLP-PPO using structured states concatenated with the exact labels generated by the deterministic annotation program.
Both MLP-PPO baselines use the same structured-state interface and approximately 1 million online environment interactions.
The MLP policy backbone has approximately 25.2 million parameters, with a backbone learning rate of 1e-4. PPO (LLM) uses Qwen3-1.7B and updates its parameters through LoRA, with a LoRA learning rate of 5e-6. The results are shown below.
As shown in the table, providing exact rule-derived labels to MLP-PPO improves performance in five of the six environments. This indicates that these labels contain useful decision-relevant information. However, even with direct access to the exact labels generated by the annotation program, MLP-PPO still performs below our full method in all six environments. These results show that simply concatenating rule-derived labels with structured states as policy inputs is not sufficient to achieve the performance of our method.
The comparison with the PPO (LLM) baseline provides further evidence. With the same LLM backbone and online training settings, Stage-A supervised learning effectively improves subsequent policy performance.