- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes WorldPO, which adds an auxiliary next-observation prediction loss to standard RL policy optimization objectives for LLM agents. Using the same on-policy rollouts, a shared model is trained with the policy optimization objective plus a a token-level NLL of the environment’s returned observation, conditioned on the interaction history and the current generated response. The authors evaluate on ALFWorld, WebShop, and search-based QA with Qwen2.5 models from 1.5B to 7B and report improvements over other baselines that only train a model with the standard policy optimization objective.
Critical strengths
The method is simple. The evaluation spans three environments with different observation types and multiple model scales.
Critical weaknesses
My main concern is limited novelty relative to prior RL work. Jointly training a policy with auxiliary predictive losses on shared parameters is well established in deep RL. Examples include UNREAL (Jaderberg et al., 2017), Shelhamer et al. (2016), Mirowski et al. (2017), DeepMDP (Gelada et al., 2019), and SPR (Schwarzer et al., 2020). The paper cites SPR and Dreamer only briefly and does not engage with this auxiliary-task literature. The paper therefore reads as an application of an existing RL technique to LLM agent environments.
Related to above, there appears to be no methodological difference from the closest LLM-agent precedents. Even within LLM agents, DyMo (Guo et al., 2025) combines on-policy RLOO with next-state cross-entropy in a shared model, and ECHO (Shrivastava et al., 2026) adds environment-token cross-entropy to GRPO on the same rollouts. The paper acknowledges these as the closest objective-level precedents but does not state how WorldPO’s objective differs, and I could not identify a difference. Neither method is included as a baseline.
Also, the authors emperical analysis shows a strange/concerning result: shuffled observation targets perform as well as or better than correct ones. In Fig. 6, within-task reassigned targets outperform correctly paired targets in all three environments. Table 11 further shows that training on correctly paired targets yields higher NLL on the true observations than training on shuffled targets. These results undercut the paper’s central interpretation that the gains come from learning a world model of action consequences.
Finally, the empirical evaluation is week. All experiments use a single seed, and ALFWorld and WebShop are evaluated on 128 episodes, so one binomial standard error is roughly 4 percentage points at typical success rates. Two runs of what appears to be the same configuration (1.5B GRPO+WM on ALFWorld) report 63.28% (Table 1) and 72.66% (Fig. 6). That gap is roughly twice the claimed GRPO→WorldPO improvement in that setting. The authors’ own paired bootstrap intervals (Table 20) include zero for several comparisons against the policy-only baseline. Without multiple training seeds, it is difficult to tell which reported improvements exceed run-to-run variance.
(minor weakness relative to the above) The theoretical rationale does not add new insight. Eqs. (4)–(5) are standard identities. The theory therefore does not explain the reported gains, and I don't quite understand why the authors included it in the main text.
Questions for authors
Please state precisely how WorldPO’s objective and training procedure differ from DyMo and ECHO. Can you include at least one of them as a baseline?
How do you position WorldPO relative to the auxiliary-task RL literature (e.g., UNREAL, DeepMDP, SPR)? What, beyond the application domain, is new?
Given that shuffled within-task and cross-task targets match or outperform correctly paired targets, what evidence supports the claim that WorldPO’s benefit comes from world modeling?
Can you report the main results (Table 1 and the Search results) over multiple training seeds, with mean and variance?
10 comments
Response of review 4S9J (2/6)
Observation 3: WorldPO's benefits extend to both likelihood-based and preference-based world modeling. We additionally use direct preference optimization (DPO) for observation prediction in WorldPO. For the same agentic history and current generated response, the actual returned observation is preferred to an observation returned after another action in the same task. The training budget and auxiliary-loss weight follow the SFT settings. We set , which scales the preference margin relative to the reference model, and use a frozen copy of the initial model as the reference. The table compares WorldPO with DPO against the reported baselines trained using only RL:
| Environment | Model | RL method | RL only success (%) | WorldPO with DPO success (%) | Improvement over RL only (%) |
|---|---|---|---|---|---|
| ALFWorld | 1.5B | GRPO | 58.59 | 72.66 | +14.06 |
| ALFWorld | 1.5B | RLOO | 71.09 | 77.34 | +6.25 |
| ALFWorld | 1.5B | GiGPO | 80.47 | 85.94 | +5.47 |
| ALFWorld | 3B | GRPO | 67.19 | 73.44 | +6.25 |
| ALFWorld | 3B | RLOO | 80.47 | 64.06 | −16.41 |
| ALFWorld | 3B | GiGPO | 75.00 | 89.06 | +14.06 |
| WebShop | 1.5B | GRPO | 59.38 | 60.94 | +1.56 |
| WebShop | 1.5B | RLOO | 65.63 | 67.19 | +1.56 |
| WebShop | 1.5B | GiGPO | 57.03 | 74.22 | +17.19 |
WorldPO with DPO achieves higher success than the reported baseline trained using only RL in eight of the nine settings (Tab. 2 and Fig. 5 in the revised manuscript). Together with the results for WorldPO with SFT, this supports the framework's applicability to both likelihood-based and preference-based observation learning.
Observation 4: World modeling complements action-level credit assignment. GiGPO already assigns credit at the action level. Adding world modeling improves its success in all four ALFWorld and WebShop comparisons in Tab. 1; for example, 3B ALFWorld success rises from 75.00% to 92.19%. Observation supervision therefore provides complementary value even with fine-grained credit assignment.
Observation 5: Auxiliary world modeling improves action validity and interaction efficiency. The behavioral analyses further show practical improvements: WorldPO reduces the ALFWorld rate of environment-invalid actions from 25.28% to 12.19% (Fig. 7). On tasks successfully completed by both methods, average interaction turns decrease from 11.72 to 10.89 on ALFWorld, 7.03 to 6.53 on WebShop, and 2.58 to 2.39 on Search (Fig. 9).
In summary, our experiments show that auxiliary world modeling can improve task performance and interaction efficiency, while our analyses show that policy improvements can persist without exact action-observation pairing. Together, these findings support WorldPO as a unified framework applicable across policy optimizers and world-modeling objectives. We have expanded the discussion of classical auxiliary learning, clarified WorldPO's relationship to DyMo/ECHO, and added the results using DPO for observation prediction to the updated manuscript. We will add the baseline comparison results once available.
Response of review 4S9J (3/6)
W3. Q3. Why WorldPO improves even without correct action-observation pairing
Reply: Thank you for raising this comment. We would like to clarify that world modeling can benefit policy optimization even when observation targets come from other transitions. Correct pairing directly supervises prediction of the actual observation given the agentic history and current generated response. The reassignment study examines how much the policy benefits depend on that exact correspondence. Specifically, we (1) compare task performance and observation prediction, (2) evaluate performance on unseen ALFWorld tasks, and (3) discuss two possible explanations.
(1) Within-task supervision achieves the highest success rates, and supervision with unrelated text also improves performance in both environments. We evaluate the fixed 1.5B checkpoints on the same 128 task instances for each of three evaluation seeds. The table reports success rates averaged across seeds, with standard deviations. We measure perplexity (PPL) on the same environment observations for all checkpoints, using the same recorded agentic histories and generated responses as inputs. For correct pairing on ALFWorld, we use the Fig. 6 checkpoint from a separate training run of the same WorldPO method reported in Tab. 1. For WebShop, we use the main WorldPO checkpoint from Tab. 1. For irrelevant-observation supervision, we use excerpts from Shakespeare's writings as prediction targets and retain the original agentic history and generated response as inputs.
| Method | ALFWorld success (%) | WebShop success (%) | ALFWorld PPL ↓ | WebShop PPL ↓ |
|---|---|---|---|---|
| GRPO | 60.68 ± 1.63 | 52.08 ± 4.58 | 10.81 | 2.84 |
| Correct-pair | 72.92 ± 1.19 | 58.33 ± 1.97 | 1.23 | 1.80 |
| Within-task | 75.52 ± 2.74 | 65.63 ± 3.41 | 1.41 | 1.86 |
| Cross-task | 66.15 ± 2.26 | 61.46 ± 1.97 | 1.61 | 1.77 |
| Irrelevant-observation supervision | 74.74 ± 0.90 | 62.50 ± 4.35 | 5.62 | 2.69 |
Both reassignment variants improve success over GRPO on ALFWorld and WebShop. Within-task supervision achieves the highest success rates in both environments and the highest average accuracy on Search among the settings in Fig. 6. Irrelevant-observation supervision also improves success over correctly paired supervision by 1.82% on ALFWorld and 4.17% on WebShop in absolute terms. All three variants trained on environment observations achieve lower PPL than GRPO and the model trained with irrelevant-observation supervision, showing that environment observations remain useful for prediction after reassignment.
Response of review 4S9J (1/6)
Thank you for reviewing our submission and for noting the simplicity of the method and the breadth of the evaluation. Below, we address your comments on prior work, observation supervision, statistical evidence, and the role of the theoretical analysis point by point. We have included all new results and clarifications in the revised manuscript and highlighted the changes in blue.
W1. W2. Q1. Q2. WorldPO's contribution beyond prior auxiliary learning
Reply: Thank you for raising this comment. We agree that the discussion of classical auxiliary learning and WorldPO's relationship to the closest language-agent methods can be improved. We would like to clarify a misunderstanding about our contribution: WorldPO builds on these foundations and contributes a unified formulation and systematic study of collaborative policy optimization and world modeling in language agents. Specifically, (1) we explain the framework's foundations and research scope, (2) we clarify its relationship to DyMo/ECHO and discuss the requested baseline comparison, and (3) we present five empirical observations about learning-objective choices, complementarity with action-level credit assignment, observation-target correspondence, and agent behavior.
(1) WorldPO builds on the foundations of classical auxiliary learning. We acknowledge the auxiliary-learning foundations established by UNREAL, Shelhamer et al., Mirowski et al., DeepMDP, and SPR. We build on their principle of training shared parameters with auxiliary objectives and systematically study how observation learning affects policy performance in language agents. WorldPO uses the same LLM for outcome-guided policy learning on generated responses and for predicting returned observations conditioned on the agentic history and current generated response. We study how observation learning depends on the policy optimizer, world-modeling objective, and target construction, and how it complements action-level credit assignment and affects agent behavior.
(2) WorldPO provides a unified formulation and systematic study of policy optimization and world modeling. DyMo [1] and ECHO [2] combine observation cross-entropy with RLOO and GRPO, respectively. These joint learning objectives can be accommodated as particular instantiations of the WorldPO framework. They share the training principle of using on-policy experience to jointly update the same LLM's parameters through policy learning and observation prediction. WorldPO systematically examines combinations of policy optimizers and world-modeling objectives, including GRPO, RLOO, and GiGPO with likelihood-based and preference-based observation learning. Our contribution is the systematic study of these learning choices and the empirical insights they yield, including their complementarity with action-level credit assignment, robustness to target reassignment, and effects on agent behavior.
Given the limited time and computational resources, we are working to complete the baseline comparison. We will share the results as they become available during the rebuttal period.
(3) Our systematic experiments and analyses yield five observations about policy optimization and world modeling.
Observation 1: WorldPO can improve agent performance across different RL methods. Tab. 1 evaluates WorldPO with SFT under GRPO, RLOO, and GiGPO on ALFWorld and WebShop at two model sizes. Success improves in most settings, supporting the applicability of observation learning across policy optimizers.
Observation 2: WorldPO's policy benefits do not require exact action-observation pairing. To test how much these benefits depend on exact pairing, we keep the GRPO objective fixed and vary the source of the observation targets in Fig. 6. Within-task targets drawn from other transitions in the same task yield better performance than correctly paired supervision on ALFWorld, WebShop, and Search. The supplementary evaluations in Tab. 3 further show that both within-task and cross-task supervision improve average success over GRPO on ALFWorld and WebShop.