acceptodds
Under review as a conference paper at ICLR 2027

Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interaction

Abstract

A central challenge for LLM agents is modeling how environment states evolve under actions, which calls for robust world modeling to improve foresight and enhance decision-making under uncertainty. Multi-turn RL provides direct feedback for learning these transitional dynamics, yet current approaches impose fixed reasoning structures that limit models to surface-level heuristics, often at the cost of inefficient interaction or over-dependence on external signals. Whether these interactive experiences truly lead to internalized knowledge of state transitions, however, remains unclear. To this end, we explore world-model internalization through efficient interaction and active reasoning (WMAct), which uses two mechanisms to convert interaction into internalized transition knowledge without prescribing a reasoning template: (1) reward rescaling that weights the outcome reward by the proportion of state-changing actions, discouraging ineffective interaction; (2) interaction-frequency annealing that gradually tightens the turn budget, encouraging less reliance on online feedback and more on internal transition prediction. Across Sokoban, Maze, and Taxi, our results reveal that \methodabb enables effective single-turn solving of tasks that previously demanded multi-turn interaction. This performance gain, achieved without recourse to multi-turn feedback, is indicative of genuine planning from internalized state transitions rather than reactive pattern matching. Moreover, the learned capability transfers to general benchmarks (e.g., +5.05 on HMMT25, +2.24 on GPQA-Diamond).

Primary area:
reinforcement learning
Keywords:
reasoning, interactive learning, world model, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_yZH1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether multi-turn reinforcement learning with environmental feedback can enhance Large Language Models' internalization of state-transition dynamics, enabling them to solve sequential decision-making tasks during inference without relying on repeated interactions.

The authors propose WMAct, a framework based on multi-turn PPO that incorporates two mechanisms: (1) reward rescaling, which scales the outcome reward based on the proportion of actions that trigger changes in the environment state; and (2) interaction frequency annealing, which gradually reduces the number of allowed interaction turns during training to encourage offline reasoning.

The method is evaluated across three fully observable grid-world environments: Sokoban, Maze, and Taxi. Experiments compare WMAct against single-turn PPO, interactive PPO, and StarPO-S, and include challenging task configurations, component ablations, and tests with different model backbones. The authors also introduce metrics for action validity, solvability, and state-prediction probing, and assess transferability to general reasoning benchmarks.

Critical strengths

An effective exploration of world model reasoning capabilities that goes beyond final task success rates. The paper not only reports task accuracy but also introduces probing evaluations focused on action validity, solvability, and state prediction. As shown in Table 3(a), compared to PPO-Interactive, WMAct improves solvability accuracy from 54% to 78% and state prediction accuracy from 18% to 40%. These results provide more direct evidence—beyond mere task success rates—that the model’s reasoning capabilities regarding state transitions have been enhanced. (This aspect is quite important for my evaluation.)

Critical weaknesses

The reward rescaling mechanism is not fully aligned with the stated goal of interaction efficiency. Equation (4) defines action validity solely based on whether an action alters the environment state; consequently, it fails to penalize trajectories composed of valid yet redundant state-changing actions. A shortest path and a longer path involving repeated back-and-forth movements could yield the same scaled reward. This reward does not directly optimize trajectory length, prevent cycles, or distinguish between productive state transitions and unnecessary repeated visits. The paper sometimes attributes broader interaction efficiency gains to this mechanism—claims that go beyond what its mathematical definition guarantees. Furthermore, the interaction frequency annealing limits the number of rounds rather than the total number of actions, as multiple actions can be issued per round. The paper should report actual trajectory lengths, the frequency of repeated states, invalid action rates, and inference costs in terms of tokens or actions.

Evidence regarding world model internalization and transferable reasoning has not been sufficiently disentangled from general RL improvements. All primary environments are deterministic, fully observable grid worlds; direct state-transition probing was conducted on Sokoban, which was also used for training. More importantly, Table 2 primarily compares general benchmark performance between the base Qwen3-8B-SFT and WMAct-Sokoban, without providing corresponding results for matched PPO-EntirePlan or PPO-Interactive models. Without these controls, it is difficult to determine how much of the improvement in general reasoning stems from WMAct versus general RL training on verifiable tasks. Additionally, the eight evaluation runs utilized a fixed held-out problem set and repeated decoding rather than independent training seeds, limiting the evidence for robustness at the training level. While these shortcomings do not negate the improvements at the task level, they weaken the stronger claim that WMAct specifically induces the internalization of a generalizable world model.

These two points directly influenced my rating.

Questions for authors

Q1: Given that the reward mechanism in Equation (4) assigns the same scaling factor to all successful trajectories consisting entirely of "state-changing" actions, how does it suppress unnecessary loops or excessively circuitous behavior? Could the authors provide data comparing WMAct with key baselines on the following metrics: average number of actions, state repetition frequency, invalid action rate, and token consumption during inference? These metrics would help assess whether the method genuinely improves trajectory execution efficiency, beyond simply reducing invalid actions or feedback rounds.

Q2: Could the authors provide results for PPO-EntirePlan and PPO-Interactive on general benchmarks? These models should utilize the same backbone networks and training environments as WMAct, with comparable reinforcement learning (RL) resource allocation. Such comparisons are crucial for determining whether the performance gains observed on HMMT25, GPQA-Diamond, and other benchmarks stem from the proposed mechanism or merely reflect general effects associated with RL training.

Q3: Could the authors describe the size and generation process of the world model probe dataset used in Table 3(a), and confirm whether the results remain consistent across multiple independent RL training runs?

6 comments
AuthorsOpenReview ↗

[W1 & Q1] Coupled Role of Eq. (4) & Turn Annealing, and Trajectory Metrics

Analytical scope. Your observation is correct, and we will state this corner case explicitly in Sec. 3.3. For a successful NN-action trajectory with kk state-invariant moves, inserting a 2m2m-step legal cycle raises the state-change ratio from (N−k)/N(N-k)/N to (N−k+2m)/(N+2m)(N-k+2m)/(N+2m), an increase of Δ=2mkN(N+2m)\Delta = \frac{2mk}{N(N+2m)}. Three properties discourage exploiting this:

  1. No gain at the optimum. The ratio reaches its maximum 1.01.0 iff k=0k=0, where padding gives Δ=0\Delta = 0.
  2. 10× weaker than removing state-invariant moves. For N=20N=20 and k=2k=2, the ratio is 0.9000.900; a 2-step cycle yields 0.9090.909 (+0.009+0.009), whereas dropping the two state-invariant moves reaches 1.0001.000 (+0.100+0.100).
  3. Multiplicative failure risk and budget caps. Because the ratio multiplies RoutcomeR_{\text{outcome}}, extra moves risk irreversible Sokoban deadlocks that forfeit the entire reward, while consuming the 16k-token limit.

Trajectory and token efficiency metrics. To complement the held-out single-turn accuracy in Table 3b, we replayed 12,800 late-stage training rollouts and measured the following metrics:

  • Solved Actions (Count, Act./Opt): the mean number of executed actions per solved rollout (Count) and its mean ratio to the board's exact BFS shortest-path length (Act./Opt).
  • Revisit (State-Change Revisit): the percentage of executed actions in solved rollouts that change the board state (st≠st−1s_t \neq s_{t-1}) but return to an earlier state st∈{s0,…,st−2}s_t \in \lbrace s_0, \dots, s_{t-2} \rbrace (e.g., A→B→AA \to B \to A). This isolates valid back-and-forth or cyclic movement from stationary wall hits.
  • Invariant (State-Invariant): the percentage of executed actions in solved rollouts that leave the board unchanged (st=st−1s_t = s_{t-1}), covering blocked moves/pushes and format no-ops.
  • Avg. Tokens (Solved, All): the mean number of generated assistant tokens per rollout, over solved rollouts and over all rollouts.

Table R1. Trajectory and token efficiency on late-stage Sokoban training rollouts (lower is better).

\hline \begin{array}{l} \phantom{\textbf{M}} \\\\ \begin{array}{l} \textbf{Method} \\\\ \hline \text{PPO-Interactive} \\\\ \text{+ reward rescaling} \\\\ \text{WMAct }(\tau=100) \end{array} \end{array} & \begin{array}{c} \textbf{Solved Actions} \\\\ \hline \begin{array}{cc} \textbf{Count}&\textbf{Act./Opt} \\\\ \hline 11.02&1.17\times \\\\ 11.31&1.18\times \\\\ \mathbf{10.11}&\mathbf{1.15\times} \end{array} \end{array} & \begin{array}{c} \textbf{Redundant Moves}\\ (\\%) \\\\ \hline \begin{array}{cc} \textbf{Revisit}&\textbf{Invariant} \\\\ \hline 0.78&0.66 \\\\ 0.69&0.68 \\\\ \mathbf{0.23}&\mathbf{0.10} \end{array} \end{array} & \begin{array}{c} \textbf{Avg. Tokens} \\\\ \hline \begin{array}{cc} \textbf{Solved}&\textbf{All} \\\\ \hline 11{,}170&13{,}191 \\\\ 10{,}986&13{,}038 \\\\ \mathbf{6{,}424}&\mathbf{7{,}710} \end{array} \end{array} \\\\ \hline \end{array}$$ The three rows show how the two mechanisms work together. Reward rescaling alone improves performance but leaves path length (1.18× optimal), revisits (0.69%), and tokens (10,986) close to PPO-Interactive. Once it is coupled with turn annealing (WMAct), single-turn open-loop execution cuts state-changing revisits by 3.4×, state-invariant moves by 6.6×, and generated tokens by 42%, while keeping solved paths near BFS-optimal.
AuthorsOpenReview ↗

[W2 & Q2] Isolating WMAct from Generic RLVR on General Benchmarks

We originally included Table 2 to rule out the possibility that RL training overfits to Sokoban and to assess whether the model generalizes to broader reasoning tasks. The comparison against Qwen3-8B-SFT shows that WMAct does not gain in-domain accuracy through environment overfitting and even improves on general benchmarks. Following your suggestion, we further examine where these gains come from. Table R2 additionally reports PPO-EntirePlan and PPO-Interactive, both trained on Sokoban, on the same six math, science, and code benchmarks.

Table R2. General-benchmark results of Sokoban-trained models.

\hline \begin{array}{l} \phantom{\textbf{M}} \\\\ \begin{array}{l} \textbf{Model} \\\\ \hline \text{Qwen3-8B-SFT} \\\\ \text{PPO-EntirePlan} \\\\ \text{PPO-Interactive} \\\\ \text{WMAct} \end{array} \end{array} & \begin{array}{c} \textbf{AVG@64} \\\\ \hline \begin{array}{cccc} \textbf{AIME24}&\textbf{AIME25}&\textbf{B-AIME}&\textbf{HMMT25} \\\\ \hline 85.10&77.92&52.88&63.44 \\\\ 85.89&79.27&54.72&\mathbf{68.54} \\\\ 86.15&79.01&51.95&65.89 \\\\ \mathbf{86.56}&\mathbf{79.48}&\mathbf{55.14}&68.49 \end{array} \end{array} & \begin{array}{c} \textbf{AVG@16} \\\\ \hline \begin{array}{cc} \textbf{GPQA-D}&\textbf{LCB-v5} \\\\ \hline 59.91&65.32 \\\\ 60.51&66.87 \\\\ 61.05&65.64 \\\\ \mathbf{62.15}&\mathbf{67.14} \end{array} \end{array} \\\\ \hline \end{array}$$ - **Math and code (AIME24, AIME25, B-AIME, HMMT25, LCB-v5):** WMAct and PPO-EntirePlan show nearly identical gains over SFT, whereas PPO-Interactive keeps pace on AIME24/25 but falls notably behind on the much harder B-AIME and HMMT25. Competition math and coding reward continuous single-pass reasoning with self-verification, a pattern that vanilla single-turn RLVR (PPO-EntirePlan) naturally encourages. In contrast, multi-turn feedback training teaches PPO-Interactive to seek intermediate external validation, which undermines long, self-contained derivations on difficult problems. WMAct avoids this pitfall through turn annealing. - **Science (GPQA-D):** GPQA-D shows a gap pattern distinct from math. PPO-EntirePlan barely improves over SFT, PPO-Interactive surpasses PPO-EntirePlan, and WMAct achieves the clearest margin (62.15; +2.24 over SFT and +1.64 over PPO-EntirePlan). Many GPQA-D items involve multi-step reaction chains (e.g., 4–5-step organic synthesis) or coupled physical constraints, where relaxing one intermediate condition invalidates all downstream steps. Longer CoT alone, without grounding, still lets such constraints drift, whereas penalizing invalid steps and annealing turns instills strict step-by-step tracking of preconditions and constraints within a single pass, consistent with our chemistry trace analysis in Appendix C.5.
AuthorsOpenReview ↗

[W2] Consistency Across RL Runs and Test Splits

Unlike small-network RL trained from random initialization, LLM RL post-training starts from a fixed pretrained/SFT checkpoint and incurs substantial multi-turn rollout cost. Training one checkpoint per configuration and evaluating it over repeated runs to control evaluation variance is therefore standard practice in LLM RL and multi-turn agent RL, e.g., RAGEN/StarPO (Wang et al., 2025d), DeepSeek-R1 (Guo et al., 2025a), and DAPO (Yu et al., 2025c). We will clarify this explicitly for the ± std in Table 1 (Sec. 4.1).

Moreover, rather than relying on a single setup, we validated WMAct in 8 independent end-to-end RL training runs across base models and environments (Qwen3-8B × 3 tasks, Qwen3-8B-SFT × 3 tasks, and Qwen3-14B × Sokoban in Tables 1 & 8, plus Qwen3-8B-SFT × PO-Sokoban), and WMAct outperforms the matched PPO baseline in every run.

Finally, we agree that evaluating on a single fixed 256-board split may limit the robustness assessment. To examine this directly, we generated three additional Sokoban test sets with distinct random seeds (256 unseen boards per seed and difficulty, mutually disjoint and deduplicated against RL training boards) and report the mean ± std over all four splits:

Table R3. Sokoban success rates (%), mean ± std over four disjoint test splits (256 boards per split and difficulty).

\hline \textbf{Method}&\textbf{Standard}&\textbf{Hard-1}&\textbf{Hard-2} \\\\ \hline \text{PPO-Interactive}&62.24 \pm 4.78&43.10 \pm 3.05&41.78 \pm 1.06 \\\\ \text{WMAct } (\tau=100)&\mathbf{75.62 \pm 4.07}&\mathbf{50.60 \pm 2.19}&\mathbf{49.83 \pm 3.91} \\\\ \hline \end{array}$$ WMAct outperforms PPO-Interactive on every split and at every difficulty level.
Reviewer_S7jDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper propose world-model internalization through efficient interaction and activate reasoning (WMAct) by reward rescaling and interaction-frequency annealing. These two neat technique tricks shows improvement on Qwen3-8B and Qwen3-8B-SFT on three interactive environments, comparable to claude sonnet 4.5. The ablation shows that both reward rescaling and interaction-frequency annealing are useful to some extent.

Critical strengths
  1. The paper is well-written and easy to follow. The paper is well motivated with the challenge of LLM agents facing environments, and propose neat techniques to improve the performance.
  2. The paper has extensive experiments on both open source models, close sourced models, multiple benchmarks and clear ablations.
Critical weaknesses
  1. The technical and novelty contribution is limited. Reward rescaling and interaction-frequency annealing are nice tricks, but themselves as a contribution seems weak, because Table 3 (b) shows that the core improvement comes from PPO-Interactive. TTI[1] shows a way of integrating adaptive rollout length adjustment into agent curriculum training.

[1] Shen, Junhong, et al. "Thinking vs. doing: Agents that reason by scaling test-time interaction." arXiv preprint arXiv:2506.07976 (2025).

Questions for authors
  1. Other than reward rescaling and interaction-frequency annealing, what's the novelty/technical contribution of this paper?
5 comments
AuthorsOpenReview ↗

[W1a & Q1] Contributions Beyond the Two Mechanisms

Our core contribution is formulating and answering a central question for LLM agents: when trained with multi-turn interaction, does an LLM remain feedback-dependent, or can it internalize the environment dynamics and plan in a single turn without feedback? Beyond the two mechanisms, we contribute:

  1. Internalization Measurement Protocol: Tracking, in fully observable environments, how the gap between single-turn and multi-turn accuracy evolves during training, as a measure of internalized dynamics (Fig. 4).
  2. Decoupling Task Success from Dynamics Internalization: Showing empirically that unconstrained interactive RL, as in PPO-Interactive, raises task completion without improving forward state prediction, and that reward rescaling combined with turn annealing closes this gap.
  3. Zero-Shot Dynamics Probing Suite: Probing action validity, multi-step solvability, and forward state prediction directly as evidence beyond task reward (Table 3a).
  4. Necessary Model Priors: Demonstrating that internalization requires reflective reasoning priors. For example, Qwen2.5-7B-Instruct improves in multi-turn rollouts, but its single-turn accuracy stays near zero (Appendix C.4).
AuthorsOpenReview ↗

[W1b] Isolating the Contribution of WMAct from That of Interaction

Across the full task suite and probes, WMAct's two mechanisms contribute as much as, or far more than, interaction alone:

Table R1. Gains from interaction vs. from WMAct. Green: gain over the column to the left.

\hline \textbf{Task / Probe (Tables 1 \\& 3a)}&\textbf{PPO-EntirePlan}&\textbf{PPO-Interactive}&\textbf{WMAct} \\\\ \hline \text{Sokoban Standard}&49.12&64.21\\;\color{green}{(+15.09)}&\mathbf{78.56}\\;\color{green}{(+14.35)} \\\\ \text{Maze Hard}&26.51&36.52\\;\color{green}{(+10.01)}&\mathbf{50.59}\\;\color{green}{(+14.07)} \\\\ \text{Taxi Standard}&38.92&39.16\\;\color{green}{(+\phantom{0}0.24)}&\mathbf{62.16}\\;\color{green}{(+23.00)} \\\\ \text{Overall Task Avg.}&35.66&50.02\\;\color{green}{(+14.36)}&\mathbf{63.97}\\;\color{green}{(+13.95)} \\\\ \hline \text{Probe: Solvability}&34.00&54.00\\;\color{green}{(+20.00)}&\mathbf{78.00}\\;\color{green}{(+24.00)} \\\\ \text{Probe: State Prediction}&16.00&18.00\\;\color{green}{(+\phantom{0}2.00)}&\mathbf{40.00}\\;\color{green}{(+22.00)} \\\\ \hline \end{array}$$ Below we also report PPO-Interactive and WMAct, both trained on Sokoban, on the six math, science, and code benchmarks. WMAct improves over Qwen3-8B-SFT on all six, whereas PPO-Interactive is on par with WMAct on AIME24/25 but degrades on the harder B-AIME and yields only modest gains on HMMT25 and GPQA-D. **Table R2.** General-benchmark results of PPO-Interactive and WMAct. $$\begin{array}{l|c|c} \hline &\textbf{AVG@64}&\textbf{AVG@16} \\\\ \hline \begin{array}{l} \textbf{Model} \\\\ \hline \text{Qwen3-8B-SFT} \\\\ \text{PPO-Interactive} \\\\ \text{WMAct} \end{array} & \begin{array}{cccc} \textbf{AIME24}&\textbf{AIME25}&\textbf{B-AIME}&\textbf{HMMT25} \\\\ \hline 85.10&77.92&52.88&63.44 \\\\ 86.15&79.01&51.95&65.89 \\\\ \mathbf{86.56}&\mathbf{79.48}&\mathbf{55.14}&\mathbf{68.49} \end{array} & \begin{array}{cc} \textbf{GPQA-D}&\textbf{LCB-v5} \\\\ \hline 59.91&65.32 \\\\ 61.05&65.64 \\\\ \mathbf{62.15}&\mathbf{67.14} \end{array} \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

[W1c] Distinction from TTI

TTI [1] and WMAct pursue orthogonal directions with distinct motivations: TTI expands rollout horizons via a prescribed schedule to scale test-time interaction on open-ended web tasks where observations reveal previously unseen external information, whereas we study the benefit of compressing turns into internal world-model reasoning. Our main testbeds are fully observable, so single-turn accuracy cleanly isolates this internalization. Broadening the turn budget naturally helps up to a point, both to gather genuinely unseen information as in [1] and, early in learning, to let an uncertain policy probe and recover through trial and error. Once the policy has learned how actions transform states, however, relying on extra turns to check predictable consequences becomes inefficient compared with simulating transitions internally. We will cite and contrast [1] in Sec. 2. This view also motivates two design choices in WMAct: Lmax⁡L_{\max} is adapted from the policy's own recent turn usage via Lˉ\bar{L} and Lmax⁡′L'_{\max}, and the reward remains multiplicative in task success.

[1] J. Shen et al. Thinking vs. Doing: Improving Agent Reasoning by Scaling Test-Time Interaction. NeurIPS 2025.

Reviewer_k3KmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WMAct, a reinforcement learning method to help LLMs internalize environment dynamics through multi-turn interaction. Specifically, they propose two mechanisms. First, reward rescaling aims to discourage brute-force interaction, and secondly, interaction-frequency annealing periodically shrinks the maximum turn budget based on recent turn statistics, pushing the model to rely on internal prediction rather than environment feedback. Experiments on Sokoban, Maze, and Taxi show improved single-turn planning, and some gains on general reasoning benchmarks.

Critical strengths

Keeping evaluation to fully observable environments can motivate single turn accuracy as a measure of internalized dynamics, and tracking gap between the single-turn and multi-turn accuracy during training provides some signal to this. The mechanisms proposed are cheap and there are consistent improvements across the environments tested. Solvability and state prediction also provide additional evidence beyond task success.

Critical weaknesses

The proposed mechanisms and evaluation depend on full observable grid scenarios. It's unclear whether annealing interactions is beneficial in partially observable or more open-ended setting (web, tool use, code, etc) as fewer turns can leave the agent with less information to work with. Also some related works instead find benefits from increasing interaction horizons on web tasks [1]. The effectiveness criteria and state change may be hard to apply outside these grid worlds which leaves the method's applicability beyond toy tasks unclear. The general benchmark transfer gains are not compared to standard PPO training, so unable to isolate the effects of the gains as it could be due to generic RLVR effects.

[1] J. Shen et al. Thinking vs. Doing: Improving Agent Reasoning by Scaling Test-Time Interaction. NeurIPS 2025.

Questions for authors
  1. Have you evaluated WMAct in a partially observable setting, and do the proposed mechanisms still help in this case?

  2. How would you define action effectiveness beyond grid worlds, particularly for useful information-gathering actions that do not change the underlying environment state? Results on one such environment would help establish that the method applies beyond grid worlds.

5 comments
AuthorsOpenReview ↗

[W1 & Q1] Partially Observable Setting

To test whether WMAct still helps when the agent must gather information through interaction, we evaluate it on Partially Observable Sokoban (PO-Sokoban; 7×7, 2 boxes). Walls and goals are always visible, but boxes are visible only within Manhattan distance 1 of the player; cells outside this local view are shown as "?", so the agent must track box positions across turns. An inspect action reveals boxes within distance 2. We evaluate WMAct and PPO-Interactive under a 30-turn cap on four unseen evaluation sets generated with different seeds:

Table R1. WMAct vs. PPO-Interactive on PO-Sokoban at it350. Success is avg@4 over 1,024 unseen maps (4,096 episodes per model; 3,233 vs. 3,302 solved). WMAct's training budget is annealed as 30→22→14→1030 \to 22 \to 14 \to 10.

\hline \textbf{Method}&\textbf{Success (\\%)}\uparrow&\textbf{Turns (All)}\downarrow&\textbf{Turns (Solved)}\downarrow&\textbf{Training }L_{\max} \\\\ \hline \text{PPO-Interactive}&78.93&10.46&7.64&30 \\\\ \text{WMAct}&\mathbf{80.62}&\mathbf{7.70}&\mathbf{5.93}&10 \\\\ \hline \end{array}$$ With its training budget annealed to $L_{\max} = 10$, WMAct outperforms PPO-Interactive on all four unseen map sets (80.62% vs. 78.93% overall) while using 26.4% fewer turns overall (7.70 vs. 10.46) and 22.4% fewer on solved episodes (5.93 vs. 7.64). Both methods are evaluated under the same 30-turn cap.
AuthorsOpenReview ↗

[Q2] Action Effectiveness Beyond Grid Worlds

Conceptually, action effectiveness itself is not bound to grid worlds or physical state changes. We chose deterministic grid worlds not because WMAct is restricted to them, but to test our core hypothesis in a noise-free diagnostic setting. We instantiate action effectiveness in Eq. (4) as a deterministic rule to show that incentivizing action efficiency benefits the internalization of environment dynamics; the deterministic rule simply eliminates judgment noise. Beyond grid worlds, effectiveness naturally extends to whether an action advances the task state or the agent's available information, which can be scored by an LLM judge.

At the same time, we fully agree that this should be validated directly in an environment featuring useful information-gathering actions that do not alter the physical state. We therefore evaluated WMAct on the PO-Sokoban setting above (Table R1), whose radius-2 inspect action gathers hidden box-occupancy information without moving the player or pushing boxes. Here, an action is marked effective if it changes physical player/box positions or reveals previously unobserved occupancy facts, so informative inspect calls are rewarded while redundant re-inspections and wall collisions are penalized. With its training budget settling at Lmax⁡=10L_{\max} = 10, WMAct still outperforms PPO-Interactive on all four unseen map sets (80.62% vs. 78.93% overall).

AuthorsOpenReview ↗

[W2] Relation to TTI

TTI [1] and WMAct study orthogonal directions with distinct motivations: TTI expands rollout horizons via a prescribed schedule to scale test-time interaction on open-ended web tasks where observations reveal unseen external information, whereas we study the benefit of compressing turns into internal world-model reasoning. Our main testbeds are fully observable, so single-turn accuracy cleanly isolates this internalization. Broadening the turn budget naturally helps up to a point, both to gather genuinely unseen information (as in [1]) and, early in learning, to let an uncertain policy probe and recover via trial and error. Once the policy has learned how actions transform states, however, relying on extra turns to check predictable consequences becomes inefficient compared with simulating transitions internally. We will cite and contrast [1] in Sec. 2. This view also motivates two design choices in WMAct: Lmax⁡L_{\max} is adapted from the policy's own recent turn usage via Lˉ\bar{L} and Lmax⁡′L'_{\max}, and the reward remains multiplicative in task success.

[1] J. Shen et al. Thinking vs. Doing: Improving Agent Reasoning by Scaling Test-Time Interaction. NeurIPS 2025.

Reviewer_638HOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WMAct, an interactive reinforcement-learning method combining reward rescaling with an adaptive interaction budget. Experiments on Sokoban, Maze, and Taxi assess single-turn planning and generalization. Additional tests examine dynamics reasoning and transfer.

Critical strengths
  1. Consistent gains across tasks and probes. WMAct improves single-turn performance over PPO-Interactive on Sokoban, Maze, and Taxi, including harder instances. Gains are substantial on several tasks and on the solvability and state-prediction probes.

  2. A useful application of curriculum learning. The adaptive budget offers a sensible transition from feedback-assisted exploration to more independent planning.

Critical weaknesses
  1. The reward can favor redundant movement. For a fixed positive outcome reward, inserting a cycle of legal state-changing actions before completion increases Equation 4's multiplier if the trajectory contains state-invariant actions and remains within budget. This can reward redundant movement, although whether trained policies exploit it is unclear.

  2. The sensitivity-analysis configuration is unclear. Table 4's default state-change scores differ from both reward rescaling alone and full WMAct in Table 3b, but the paper does not state whether annealing or other training settings differ between these experiments.

Questions for authors
  1. How does reward rescaling affect the frequency of redundant movement cycles compared with PPO-Interactive?

  2. What training configuration produced Table 4's default state-change scores, and how does it relate to the reward-only and full WMAct settings in Table 3b?

3 comments
AuthorsOpenReview ↗

[W1 & Q1] Redundant Movement

Analytical scope. Your observation is correct, and we will state this corner case explicitly in Sec. 3.3. For a successful NN-action trajectory with kk state-invariant moves, inserting a 2m2m-step legal cycle raises the state-change ratio from (N−k)/N(N-k)/N to (N−k+2m)/(N+2m)(N-k+2m)/(N+2m), an increase of Δ=2mkN(N+2m)\Delta = \frac{2mk}{N(N+2m)}. Three properties discourage exploiting this:

  1. No gain at the optimum. The ratio reaches its maximum 1.01.0 iff k=0k=0, where padding gives Δ=0\Delta = 0.
  2. 10× weaker than removing state-invariant moves. For N=20N=20 and k=2k=2, the ratio is 0.9000.900; a 2-step cycle yields 0.9090.909 (+0.009+0.009), whereas dropping the two state-invariant moves reaches 1.0001.000 (+0.100+0.100).
  3. Multiplicative failure risk and budget caps. Because the ratio multiplies RoutcomeR_{\text{outcome}}, extra moves risk irreversible Sokoban deadlocks that forfeit the entire reward, while consuming the 16k-token limit.

Empirical cycle dynamics. We replayed all solved training rollouts, tracking the full board state st=(player,box positions)s_t = (\text{player}, \text{box positions}), and measured three metrics:

  • Cycle Ep., episode incidence: percentage of solved rollouts containing at least one contiguous legal cycle sa→⋯→sb=sas_a \to \dots \to s_b = s_a, with st≠st−1s_t \neq s_{t-1} at every step so that stationary wall hits are excluded.
  • Cyclic Act., cyclic step share: percentage of total executed actions across solved rollouts belonging to such cycles, identified via online loop erasure. Segment [a,b)[a, b) is popped upon revisiting sas_a, so nested or overlapping cycles are never double-counted.
  • Imm. Rev., immediate backtrack share: percentage of total executed actions across solved rollouts that immediately undo the prior transition, i.e., st+1=st−1s_{t+1} = s_{t-1} with st≠st−1s_t \neq s_{t-1}.

Table R1. Frequency of redundant movement during training (%, lower is better). Columns are 50-iteration training windows.

\hline \textbf{Metric}&\textbf{Method}&\textbf{it1–50}&\textbf{it101–150}&\textbf{it301–350}&\textbf{it351–400} \\\\ \hline &\text{PPO-Interactive}&17.93&12.64&4.04&2.79 \\\\ \text{Cycle Ep.}&\text{+ reward rescaling}&17.93&\mathbf{12.10}&3.24&2.23 \\\\ &\text{WMAct } (\tau=100)&\mathbf{15.97}&12.31&\mathbf{1.36}&\mathbf{1.62} \\\\ \hline &\text{PPO-Interactive}&6.20&5.05&1.42&0.97 \\\\ \text{Cyclic Act.}&\text{+ reward rescaling}&6.49&4.65&1.16&0.79 \\\\ &\text{WMAct } (\tau=100)&\mathbf{5.63}&\mathbf{4.56}&\mathbf{0.42}&\mathbf{0.49} \\\\ \hline &\text{PPO-Interactive}&2.70&1.71&0.42&0.26 \\\\ \text{Imm. Rev.}&\text{+ reward rescaling}&2.80&\mathbf{1.50}&0.30&0.20 \\\\ &\text{WMAct } (\tau=100)&\mathbf{2.37}&1.65&\mathbf{0.13}&\mathbf{0.15} \\\\ \hline \end{array}$$ If policies exploited Eq. (4), the cycle metrics of + reward rescaling would increasingly exceed those of PPO-Interactive as training proceeds. Instead, the two are on par in it1–50, after which + reward rescaling is lower on all three metrics (by 18%–29% relatively in the last two windows), and WMAct is lowest in the last two windows.
AuthorsOpenReview ↗

[W2 & Q2] Training Configuration of Table 4

Thank you for flagging this. The discrepancy comes from an omitted backbone header in Table 4. The default action-efficacy row of Table 4 matches the reward-only row of Table 3b except that it is trained from Qwen3-8B instead of Qwen3-8B-SFT.

Table R2. Training configurations behind the Sokoban rows of Table 3b and Table 4.

\hline \textbf{Experiment Row}&\textbf{Base Backbone}&\textbf{Reward Rescaling}&\textbf{Freq. Annealing} \\\\ \hline \text{Table 3b: + reward rescaling}&\text{Qwen3-8B-SFT}&\text{Binary state change}&\text{Off} \\\\ \text{Table 4: action-efficacy with binary state change}&\text{Qwen3-8B}&\text{Binary state change}&\text{Off} \\\\ \text{Table 3b and Table 4: WMAct }(\tau=100)&\text{Qwen3-8B-SFT}&\text{Binary state change}&\text{On} \\\\ \hline \end{array}$$ Specifically, Table 3b and the upper block of Table 4 use Qwen3-8B-SFT, matching Table 2, to verify that WMAct's gains are not subsumed by a strong SFT initialization, whereas the action-efficacy variants, namely recent-window, count-based soft, and binary state change, were ablated on the public Qwen3-8B model to isolate how each reward formulation shapes exploration dynamics without confounding from in-house SFT priors. We will split Table 4 with explicit backbone headers to avoid misunderstanding.
AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer 638H,

Thank you again for your review. In our responses, we analyzed the redundant-movement corner case of Eq. (4) with cycle statistics over training, and clarified the training configuration behind Table 4. As the discussion period ends soon (Nov 18, AoE), we would appreciate knowing whether these address your concerns, and we are happy to clarify anything further before then.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.