- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether multi-turn reinforcement learning with environmental feedback can enhance Large Language Models' internalization of state-transition dynamics, enabling them to solve sequential decision-making tasks during inference without relying on repeated interactions.
The authors propose WMAct, a framework based on multi-turn PPO that incorporates two mechanisms: (1) reward rescaling, which scales the outcome reward based on the proportion of actions that trigger changes in the environment state; and (2) interaction frequency annealing, which gradually reduces the number of allowed interaction turns during training to encourage offline reasoning.
The method is evaluated across three fully observable grid-world environments: Sokoban, Maze, and Taxi. Experiments compare WMAct against single-turn PPO, interactive PPO, and StarPO-S, and include challenging task configurations, component ablations, and tests with different model backbones. The authors also introduce metrics for action validity, solvability, and state-prediction probing, and assess transferability to general reasoning benchmarks.
Critical strengths
An effective exploration of world model reasoning capabilities that goes beyond final task success rates. The paper not only reports task accuracy but also introduces probing evaluations focused on action validity, solvability, and state prediction. As shown in Table 3(a), compared to PPO-Interactive, WMAct improves solvability accuracy from 54% to 78% and state prediction accuracy from 18% to 40%. These results provide more direct evidence—beyond mere task success rates—that the model’s reasoning capabilities regarding state transitions have been enhanced. (This aspect is quite important for my evaluation.)
Critical weaknesses
The reward rescaling mechanism is not fully aligned with the stated goal of interaction efficiency. Equation (4) defines action validity solely based on whether an action alters the environment state; consequently, it fails to penalize trajectories composed of valid yet redundant state-changing actions. A shortest path and a longer path involving repeated back-and-forth movements could yield the same scaled reward. This reward does not directly optimize trajectory length, prevent cycles, or distinguish between productive state transitions and unnecessary repeated visits. The paper sometimes attributes broader interaction efficiency gains to this mechanism—claims that go beyond what its mathematical definition guarantees. Furthermore, the interaction frequency annealing limits the number of rounds rather than the total number of actions, as multiple actions can be issued per round. The paper should report actual trajectory lengths, the frequency of repeated states, invalid action rates, and inference costs in terms of tokens or actions.
Evidence regarding world model internalization and transferable reasoning has not been sufficiently disentangled from general RL improvements. All primary environments are deterministic, fully observable grid worlds; direct state-transition probing was conducted on Sokoban, which was also used for training. More importantly, Table 2 primarily compares general benchmark performance between the base Qwen3-8B-SFT and WMAct-Sokoban, without providing corresponding results for matched PPO-EntirePlan or PPO-Interactive models. Without these controls, it is difficult to determine how much of the improvement in general reasoning stems from WMAct versus general RL training on verifiable tasks. Additionally, the eight evaluation runs utilized a fixed held-out problem set and repeated decoding rather than independent training seeds, limiting the evidence for robustness at the training level. While these shortcomings do not negate the improvements at the task level, they weaken the stronger claim that WMAct specifically induces the internalization of a generalizable world model.
These two points directly influenced my rating.
Questions for authors
Q1: Given that the reward mechanism in Equation (4) assigns the same scaling factor to all successful trajectories consisting entirely of "state-changing" actions, how does it suppress unnecessary loops or excessively circuitous behavior? Could the authors provide data comparing WMAct with key baselines on the following metrics: average number of actions, state repetition frequency, invalid action rate, and token consumption during inference? These metrics would help assess whether the method genuinely improves trajectory execution efficiency, beyond simply reducing invalid actions or feedback rounds.
Q2: Could the authors provide results for PPO-EntirePlan and PPO-Interactive on general benchmarks? These models should utilize the same backbone networks and training environments as WMAct, with comparable reinforcement learning (RL) resource allocation. Such comparisons are crucial for determining whether the performance gains observed on HMMT25, GPQA-Diamond, and other benchmarks stem from the proposed mechanism or merely reflect general effects associated with RL training.
Q3: Could the authors describe the size and generation process of the world model probe dataset used in Table 3(a), and confirm whether the results remain consistent across multiple independent RL training runs?
6 comments
[W2 & Q2] Isolating WMAct from Generic RLVR on General Benchmarks
We originally included Table 2 to rule out the possibility that RL training overfits to Sokoban and to assess whether the model generalizes to broader reasoning tasks. The comparison against Qwen3-8B-SFT shows that WMAct does not gain in-domain accuracy through environment overfitting and even improves on general benchmarks. Following your suggestion, we further examine where these gains come from. Table R2 additionally reports PPO-EntirePlan and PPO-Interactive, both trained on Sokoban, on the same six math, science, and code benchmarks.
Table R2. General-benchmark results of Sokoban-trained models.
\hline \begin{array}{l} \phantom{\textbf{M}} \\\\ \begin{array}{l} \textbf{Model} \\\\ \hline \text{Qwen3-8B-SFT} \\\\ \text{PPO-EntirePlan} \\\\ \text{PPO-Interactive} \\\\ \text{WMAct} \end{array} \end{array} & \begin{array}{c} \textbf{AVG@64} \\\\ \hline \begin{array}{cccc} \textbf{AIME24}&\textbf{AIME25}&\textbf{B-AIME}&\textbf{HMMT25} \\\\ \hline 85.10&77.92&52.88&63.44 \\\\ 85.89&79.27&54.72&\mathbf{68.54} \\\\ 86.15&79.01&51.95&65.89 \\\\ \mathbf{86.56}&\mathbf{79.48}&\mathbf{55.14}&68.49 \end{array} \end{array} & \begin{array}{c} \textbf{AVG@16} \\\\ \hline \begin{array}{cc} \textbf{GPQA-D}&\textbf{LCB-v5} \\\\ \hline 59.91&65.32 \\\\ 60.51&66.87 \\\\ 61.05&65.64 \\\\ \mathbf{62.15}&\mathbf{67.14} \end{array} \end{array} \\\\ \hline \end{array}$$ - **Math and code (AIME24, AIME25, B-AIME, HMMT25, LCB-v5):** WMAct and PPO-EntirePlan show nearly identical gains over SFT, whereas PPO-Interactive keeps pace on AIME24/25 but falls notably behind on the much harder B-AIME and HMMT25. Competition math and coding reward continuous single-pass reasoning with self-verification, a pattern that vanilla single-turn RLVR (PPO-EntirePlan) naturally encourages. In contrast, multi-turn feedback training teaches PPO-Interactive to seek intermediate external validation, which undermines long, self-contained derivations on difficult problems. WMAct avoids this pitfall through turn annealing. - **Science (GPQA-D):** GPQA-D shows a gap pattern distinct from math. PPO-EntirePlan barely improves over SFT, PPO-Interactive surpasses PPO-EntirePlan, and WMAct achieves the clearest margin (62.15; +2.24 over SFT and +1.64 over PPO-EntirePlan). Many GPQA-D items involve multi-step reaction chains (e.g., 4–5-step organic synthesis) or coupled physical constraints, where relaxing one intermediate condition invalidates all downstream steps. Longer CoT alone, without grounding, still lets such constraints drift, whereas penalizing invalid steps and annealing turns instills strict step-by-step tracking of preconditions and constraints within a single pass, consistent with our chemistry trace analysis in Appendix C.5.[W2] Consistency Across RL Runs and Test Splits
Unlike small-network RL trained from random initialization, LLM RL post-training starts from a fixed pretrained/SFT checkpoint and incurs substantial multi-turn rollout cost. Training one checkpoint per configuration and evaluating it over repeated runs to control evaluation variance is therefore standard practice in LLM RL and multi-turn agent RL, e.g., RAGEN/StarPO (Wang et al., 2025d), DeepSeek-R1 (Guo et al., 2025a), and DAPO (Yu et al., 2025c). We will clarify this explicitly for the ± std in Table 1 (Sec. 4.1).
Moreover, rather than relying on a single setup, we validated WMAct in 8 independent end-to-end RL training runs across base models and environments (Qwen3-8B × 3 tasks, Qwen3-8B-SFT × 3 tasks, and Qwen3-14B × Sokoban in Tables 1 & 8, plus Qwen3-8B-SFT × PO-Sokoban), and WMAct outperforms the matched PPO baseline in every run.
Finally, we agree that evaluating on a single fixed 256-board split may limit the robustness assessment. To examine this directly, we generated three additional Sokoban test sets with distinct random seeds (256 unseen boards per seed and difficulty, mutually disjoint and deduplicated against RL training boards) and report the mean ± std over all four splits:
Table R3. Sokoban success rates (%), mean ± std over four disjoint test splits (256 boards per split and difficulty).
\hline \textbf{Method}&\textbf{Standard}&\textbf{Hard-1}&\textbf{Hard-2} \\\\ \hline \text{PPO-Interactive}&62.24 \pm 4.78&43.10 \pm 3.05&41.78 \pm 1.06 \\\\ \text{WMAct } (\tau=100)&\mathbf{75.62 \pm 4.07}&\mathbf{50.60 \pm 2.19}&\mathbf{49.83 \pm 3.91} \\\\ \hline \end{array}$$ WMAct outperforms PPO-Interactive on every split and at every difficulty level.
[W1 & Q1] Coupled Role of Eq. (4) & Turn Annealing, and Trajectory Metrics
Analytical scope. Your observation is correct, and we will state this corner case explicitly in Sec. 3.3. For a successful N-action trajectory with k state-invariant moves, inserting a 2m-step legal cycle raises the state-change ratio from (N−k)/N to (N−k+2m)/(N+2m), an increase of Δ=N(N+2m)2mk. Three properties discourage exploiting this:
Trajectory and token efficiency metrics. To complement the held-out single-turn accuracy in Table 3b, we replayed 12,800 late-stage training rollouts and measured the following metrics:
Table R1. Trajectory and token efficiency on late-stage Sokoban training rollouts (lower is better).
\hline \begin{array}{l} \phantom{\textbf{M}} \\\\ \begin{array}{l} \textbf{Method} \\\\ \hline \text{PPO-Interactive} \\\\ \text{+ reward rescaling} \\\\ \text{WMAct }(\tau=100) \end{array} \end{array} & \begin{array}{c} \textbf{Solved Actions} \\\\ \hline \begin{array}{cc} \textbf{Count}&\textbf{Act./Opt} \\\\ \hline 11.02&1.17\times \\\\ 11.31&1.18\times \\\\ \mathbf{10.11}&\mathbf{1.15\times} \end{array} \end{array} & \begin{array}{c} \textbf{Redundant Moves}\\ (\\%) \\\\ \hline \begin{array}{cc} \textbf{Revisit}&\textbf{Invariant} \\\\ \hline 0.78&0.66 \\\\ 0.69&0.68 \\\\ \mathbf{0.23}&\mathbf{0.10} \end{array} \end{array} & \begin{array}{c} \textbf{Avg. Tokens} \\\\ \hline \begin{array}{cc} \textbf{Solved}&\textbf{All} \\\\ \hline 11{,}170&13{,}191 \\\\ 10{,}986&13{,}038 \\\\ \mathbf{6{,}424}&\mathbf{7{,}710} \end{array} \end{array} \\\\ \hline \end{array}$$ The three rows show how the two mechanisms work together. Reward rescaling alone improves performance but leaves path length (1.18× optimal), revisits (0.69%), and tokens (10,986) close to PPO-Interactive. Once it is coupled with turn annealing (WMAct), single-turn open-loop execution cuts state-changing revisits by 3.4×, state-invariant moves by 6.6×, and generated tokens by 42%, while keeping solved paths near BFS-optimal.