acceptodds
Under review as a conference paper at ICLR 2027

WorldPO: Holistic Agentic Learning with Policy Optimization and World Modeling

Abstract

Although pretrained large language models possess broad capabilities, limited environment-specific knowledge can hinder their adaptation to unfamiliar agentic environments. Policy optimization improves behavior through task feedback, yet uses information-rich environmental observations primarily as decision context, leaving their value for environmental understanding underexploited. We propose **WorldPO** for *holistic agentic learning* that integrates behavioral improvement with environmental understanding through collaborative policy optimization and world modeling. Using the same on-policy rollouts, WorldPO combines feedback-guided policy learning on generated responses with world modeling supervised by environmental observations, optimizing both objectives in a shared LLM. We evaluate WorldPO in three environments with distinct observations: ALFWorld, WebShop, and Search. WorldPO improves task completion and complements GiGPO's action-level optimization, raising 3B ALFWorld success from 75.00% to 92.19%. On Search, 3B WorldPO outperforms GiGPO on all four non-training datasets. Our analyses suggest that WorldPO benefits from task-relevant environmental knowledge beyond exact action-observation correspondences, with greater gains from successful experience. WorldPO also reduces invalid actions in ALFWorld and uses fewer turns on jointly solved tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic Learning, Agentic RL, World Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_4S9JOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes WorldPO, which adds an auxiliary next-observation prediction loss to standard RL policy optimization objectives for LLM agents. Using the same on-policy rollouts, a shared model is trained with the policy optimization objective plus a a token-level NLL of the environment’s returned observation, conditioned on the interaction history and the current generated response. The authors evaluate on ALFWorld, WebShop, and search-based QA with Qwen2.5 models from 1.5B to 7B and report improvements over other baselines that only train a model with the standard policy optimization objective.

Critical strengths

The method is simple. The evaluation spans three environments with different observation types and multiple model scales.

Critical weaknesses

My main concern is limited novelty relative to prior RL work. Jointly training a policy with auxiliary predictive losses on shared parameters is well established in deep RL. Examples include UNREAL (Jaderberg et al., 2017), Shelhamer et al. (2016), Mirowski et al. (2017), DeepMDP (Gelada et al., 2019), and SPR (Schwarzer et al., 2020). The paper cites SPR and Dreamer only briefly and does not engage with this auxiliary-task literature. The paper therefore reads as an application of an existing RL technique to LLM agent environments.

Related to above, there appears to be no methodological difference from the closest LLM-agent precedents. Even within LLM agents, DyMo (Guo et al., 2025) combines on-policy RLOO with next-state cross-entropy in a shared model, and ECHO (Shrivastava et al., 2026) adds environment-token cross-entropy to GRPO on the same rollouts. The paper acknowledges these as the closest objective-level precedents but does not state how WorldPO’s objective differs, and I could not identify a difference. Neither method is included as a baseline.

Also, the authors emperical analysis shows a strange/concerning result: shuffled observation targets perform as well as or better than correct ones. In Fig. 6, within-task reassigned targets outperform correctly paired targets in all three environments. Table 11 further shows that training on correctly paired targets yields higher NLL on the true observations than training on shuffled targets. These results undercut the paper’s central interpretation that the gains come from learning a world model of action consequences.

Finally, the empirical evaluation is week. All experiments use a single seed, and ALFWorld and WebShop are evaluated on 128 episodes, so one binomial standard error is roughly 4 percentage points at typical success rates. Two runs of what appears to be the same configuration (1.5B GRPO+WM on ALFWorld) report 63.28% (Table 1) and 72.66% (Fig. 6). That gap is roughly twice the claimed GRPO→WorldPO improvement in that setting. The authors’ own paired bootstrap intervals (Table 20) include zero for several comparisons against the policy-only baseline. Without multiple training seeds, it is difficult to tell which reported improvements exceed run-to-run variance.

(minor weakness relative to the above) The theoretical rationale does not add new insight. Eqs. (4)–(5) are standard identities. The theory therefore does not explain the reported gains, and I don't quite understand why the authors included it in the main text.

Questions for authors

Please state precisely how WorldPO’s objective and training procedure differ from DyMo and ECHO. Can you include at least one of them as a baseline?

How do you position WorldPO relative to the auxiliary-task RL literature (e.g., UNREAL, DeepMDP, SPR)? What, beyond the application domain, is new?

Given that shuffled within-task and cross-task targets match or outperform correctly paired targets, what evidence supports the claim that WorldPO’s benefit comes from world modeling?

Can you report the main results (Table 1 and the Search results) over multiple training seeds, with mean and variance?

10 comments
AuthorsOpenReview ↗

Response of review 4S9J (1/6)

Thank you for reviewing our submission and for noting the simplicity of the method and the breadth of the evaluation. Below, we address your comments on prior work, observation supervision, statistical evidence, and the role of the theoretical analysis point by point. We have included all new results and clarifications in the revised manuscript and highlighted the changes in blue.

W1. W2. Q1. Q2. WorldPO's contribution beyond prior auxiliary learning

Reply: Thank you for raising this comment. We agree that the discussion of classical auxiliary learning and WorldPO's relationship to the closest language-agent methods can be improved. We would like to clarify a misunderstanding about our contribution: WorldPO builds on these foundations and contributes a unified formulation and systematic study of collaborative policy optimization and world modeling in language agents. Specifically, (1) we explain the framework's foundations and research scope, (2) we clarify its relationship to DyMo/ECHO and discuss the requested baseline comparison, and (3) we present five empirical observations about learning-objective choices, complementarity with action-level credit assignment, observation-target correspondence, and agent behavior.

(1) WorldPO builds on the foundations of classical auxiliary learning. We acknowledge the auxiliary-learning foundations established by UNREAL, Shelhamer et al., Mirowski et al., DeepMDP, and SPR. We build on their principle of training shared parameters with auxiliary objectives and systematically study how observation learning affects policy performance in language agents. WorldPO uses the same LLM for outcome-guided policy learning on generated responses and for predicting returned observations conditioned on the agentic history and current generated response. We study how observation learning depends on the policy optimizer, world-modeling objective, and target construction, and how it complements action-level credit assignment and affects agent behavior.

(2) WorldPO provides a unified formulation and systematic study of policy optimization and world modeling. DyMo [1] and ECHO [2] combine observation cross-entropy with RLOO and GRPO, respectively. These joint learning objectives can be accommodated as particular instantiations of the WorldPO framework. They share the training principle of using on-policy experience to jointly update the same LLM's parameters through policy learning and observation prediction. WorldPO systematically examines combinations of policy optimizers and world-modeling objectives, including GRPO, RLOO, and GiGPO with likelihood-based and preference-based observation learning. Our contribution is the systematic study of these learning choices and the empirical insights they yield, including their complementarity with action-level credit assignment, robustness to target reassignment, and effects on agent behavior.

Given the limited time and computational resources, we are working to complete the baseline comparison. We will share the results as they become available during the rebuttal period.

(3) Our systematic experiments and analyses yield five observations about policy optimization and world modeling.

Observation 1: WorldPO can improve agent performance across different RL methods. Tab. 1 evaluates WorldPO with SFT under GRPO, RLOO, and GiGPO on ALFWorld and WebShop at two model sizes. Success improves in most settings, supporting the applicability of observation learning across policy optimizers.

Observation 2: WorldPO's policy benefits do not require exact action-observation pairing. To test how much these benefits depend on exact pairing, we keep the GRPO objective fixed and vary the source of the observation targets in Fig. 6. Within-task targets drawn from other transitions in the same task yield better performance than correctly paired supervision on ALFWorld, WebShop, and Search. The supplementary evaluations in Tab. 3 further show that both within-task and cross-task supervision improve average success over GRPO on ALFWorld and WebShop.

AuthorsOpenReview ↗

Response of review 4S9J (2/6)

Observation 3: WorldPO's benefits extend to both likelihood-based and preference-based world modeling. We additionally use direct preference optimization (DPO) for observation prediction in WorldPO. For the same agentic history and current generated response, the actual returned observation is preferred to an observation returned after another action in the same task. The training budget and auxiliary-loss weight follow the SFT settings. We set β=0.1\beta=0.1, which scales the preference margin relative to the reference model, and use a frozen copy of the initial model as the reference. The table compares WorldPO with DPO against the reported baselines trained using only RL:

EnvironmentModelRL methodRL only success (%)WorldPO with DPO success (%)Improvement over RL only (%)
ALFWorld1.5BGRPO58.5972.66+14.06
ALFWorld1.5BRLOO71.0977.34+6.25
ALFWorld1.5BGiGPO80.4785.94+5.47
ALFWorld3BGRPO67.1973.44+6.25
ALFWorld3BRLOO80.4764.06−16.41
ALFWorld3BGiGPO75.0089.06+14.06
WebShop1.5BGRPO59.3860.94+1.56
WebShop1.5BRLOO65.6367.19+1.56
WebShop1.5BGiGPO57.0374.22+17.19

WorldPO with DPO achieves higher success than the reported baseline trained using only RL in eight of the nine settings (Tab. 2 and Fig. 5 in the revised manuscript). Together with the results for WorldPO with SFT, this supports the framework's applicability to both likelihood-based and preference-based observation learning.

Observation 4: World modeling complements action-level credit assignment. GiGPO already assigns credit at the action level. Adding world modeling improves its success in all four ALFWorld and WebShop comparisons in Tab. 1; for example, 3B ALFWorld success rises from 75.00% to 92.19%. Observation supervision therefore provides complementary value even with fine-grained credit assignment.

Observation 5: Auxiliary world modeling improves action validity and interaction efficiency. The behavioral analyses further show practical improvements: WorldPO reduces the ALFWorld rate of environment-invalid actions from 25.28% to 12.19% (Fig. 7). On tasks successfully completed by both methods, average interaction turns decrease from 11.72 to 10.89 on ALFWorld, 7.03 to 6.53 on WebShop, and 2.58 to 2.39 on Search (Fig. 9).

In summary, our experiments show that auxiliary world modeling can improve task performance and interaction efficiency, while our analyses show that policy improvements can persist without exact action-observation pairing. Together, these findings support WorldPO as a unified framework applicable across policy optimizers and world-modeling objectives. We have expanded the discussion of classical auxiliary learning, clarified WorldPO's relationship to DyMo/ECHO, and added the results using DPO for observation prediction to the updated manuscript. We will add the baseline comparison results once available.

AuthorsOpenReview ↗

Response of review 4S9J (3/6)

W3. Q3. Why WorldPO improves even without correct action-observation pairing

Reply: Thank you for raising this comment. We would like to clarify that world modeling can benefit policy optimization even when observation targets come from other transitions. Correct pairing directly supervises prediction of the actual observation given the agentic history and current generated response. The reassignment study examines how much the policy benefits depend on that exact correspondence. Specifically, we (1) compare task performance and observation prediction, (2) evaluate performance on unseen ALFWorld tasks, and (3) discuss two possible explanations.

(1) Within-task supervision achieves the highest success rates, and supervision with unrelated text also improves performance in both environments. We evaluate the fixed 1.5B checkpoints on the same 128 task instances for each of three evaluation seeds. The table reports success rates averaged across seeds, with standard deviations. We measure perplexity (PPL) on the same environment observations for all checkpoints, using the same recorded agentic histories and generated responses as inputs. For correct pairing on ALFWorld, we use the Fig. 6 checkpoint from a separate training run of the same WorldPO method reported in Tab. 1. For WebShop, we use the main WorldPO checkpoint from Tab. 1. For irrelevant-observation supervision, we use excerpts from Shakespeare's writings as prediction targets and retain the original agentic history and generated response as inputs.

MethodALFWorld success (%)WebShop success (%)ALFWorld PPL ↓WebShop PPL ↓
GRPO60.68 ± 1.6352.08 ± 4.5810.812.84
Correct-pair72.92 ± 1.1958.33 ± 1.971.231.80
Within-task75.52 ± 2.7465.63 ± 3.411.411.86
Cross-task66.15 ± 2.2661.46 ± 1.971.611.77
Irrelevant-observation supervision74.74 ± 0.9062.50 ± 4.355.622.69

Both reassignment variants improve success over GRPO on ALFWorld and WebShop. Within-task supervision achieves the highest success rates in both environments and the highest average accuracy on Search among the settings in Fig. 6. Irrelevant-observation supervision also improves success over correctly paired supervision by 1.82% on ALFWorld and 4.17% on WebShop in absolute terms. All three variants trained on environment observations achieve lower PPL than GRPO and the model trained with irrelevant-observation supervision, showing that environment observations remain useful for prediction after reassignment.

Reviewer_PzwpOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether environmental observations can provide useful auxiliary supervision during reinforcement learning of LLM agents. WorldPO combines a policy optimization loss with next-observation token prediction in a shared language model, using the same on-policy trajectories. The observation predictor is used only during training, without introducing an additional prediction or planning stage at inference.

Critical strengths
  • The approach is simple and practically appealing. It uses observations already collected during interaction, requires no additional target annotation, and preserves the original inference interface.
  • The evaluation covers several environments and policy optimizers. Some reported improvements are substantial, particularly the increase from 75.00% to 92.19% for 3B GiGPO on ALFWorld.
Critical weaknesses
  • The incremental contribution of observation supervision is not convincingly established. WorldPO adds observation-token cross-entropy to an existing policy objective, closely resembling the shared-model auxiliary prediction approaches already acknowledged in DyMo and ECHO. Its contribution therefore rests largely on the empirical evidence. However, the Search improvement decreases from 5.41 percentage points against the published baseline (Table 3) to only 0.50 points against local GRPO under a common evaluation protocol (Table 4). This makes it difficult to determine how much of the reported benefit is attributable to the proposed auxiliary objective.
Questions for authors
  1. What explains the substantially smaller Search improvement in Table 4 compared with Table 3?
  2. What distinct methodological or empirical contribution does WorldPO establish beyond DyMo and ECHO?
6 comments
AuthorsOpenReview ↗

Response of review Pzwp (1/3)

Thank you for reviewing our submission and for noting the simplicity of WorldPO and its use of observations already collected during interaction. Below, we address your questions about WorldPO's contribution and its results on Search point by point. We have included all new results and clarifications in the revised manuscript and highlighted the changes in blue.

W1. Q2. WorldPO's contribution beyond DyMo/ECHO

Reply: Thank you for raising this comment. We would like to clarify a misunderstanding about our contribution: WorldPO contributes a systematic formulation and study of collaborative policy optimization and world modeling, together with empirical insights into the benefits of combining them. Specifically, (1) we formulate collaborative policy optimization and world modeling as a unified agent-learning framework and systematically evaluate its design choices, (2) our experiments and analyses yield five observations about its applicability, complementarity, robustness to target reassignment, and effects on agent behavior, and (3) we discuss the additional direct experimental comparisons with DyMo/ECHO.

(1) WorldPO provides a unified formulation and systematic study of policy optimization and world modeling. Prior work such as DyMo [1] and ECHO [2] demonstrates the benefits of combining policy learning with observation prediction, with RL instantiations based on RLOO and GRPO, respectively. Building on these precedents, WorldPO makes explicit two design choices: how outcome feedback supervises policy learning on generated responses, and how returned observations supervise world modeling. The objectives collaboratively update the same LLM using on-policy experience. ALFWorld, WebShop, and Search expose the agent to household state descriptions, product information, and retrieved passages, respectively. We evaluate combinations of policy optimizers and observation-learning objectives across environments and model sizes, and analyze observation prediction, target construction, gradients, and agent behavior.

(2) Our systematic experiments and analyses yield five observations about policy optimization and world modeling.

Observation 1: WorldPO can improve agent performance across different RL methods. Tab. 1 evaluates WorldPO with SFT under GRPO, RLOO, and GiGPO on ALFWorld and WebShop at two model sizes. Success improves in most settings, supporting the applicability of observation learning across policy optimizers.

Observation 2: WorldPO's policy benefits do not require exact action-observation pairing. To test how much these benefits depend on exact pairing, we keep the GRPO objective fixed and vary the source of the observation targets in Fig. 6. Within-task targets drawn from other transitions in the same task yield better performance than correctly paired supervision on ALFWorld, WebShop, and Search. The supplementary evaluations in Tab. 3 further show that both within-task and cross-task supervision improve average success over GRPO on ALFWorld and WebShop.

AuthorsOpenReview ↗

Response of review Pzwp (2/3)

Observation 3: WorldPO's benefits extend to both likelihood-based and preference-based world modeling.** We additionally use direct preference optimization (DPO) for observation prediction in WorldPO. For the same agentic history and current generated response, the actual returned observation is preferred to an observation returned after another action in the same task. The training budget and auxiliary-loss weight follow the SFT settings. We set β=0.1\beta=0.1, which scales the preference margin relative to the reference model, and use a frozen copy of the initial model as the reference. The table compares WorldPO with DPO against the reported baselines trained using only RL:

EnvironmentModelRL methodRL only success (%)WorldPO with DPO success (%)Improvement over RL only (%)
ALFWorld1.5BGRPO58.5972.66+14.06
ALFWorld1.5BRLOO71.0977.34+6.25
ALFWorld1.5BGiGPO80.4785.94+5.47
ALFWorld3BGRPO67.1973.44+6.25
ALFWorld3BRLOO80.4764.06−16.41
ALFWorld3BGiGPO75.0089.06+14.06
WebShop1.5BGRPO59.3860.94+1.56
WebShop1.5BRLOO65.6367.19+1.56
WebShop1.5BGiGPO57.0374.22+17.19

WorldPO with DPO achieves higher success than the reported baseline trained using only RL in eight of the nine settings (Tab. 2 and Fig. 5 in the revised manuscript). Together with the results for WorldPO with SFT, this supports the framework's applicability to both likelihood-based and preference-based observation learning.

Observation 4: World modeling complements action-level credit assignment. GiGPO already assigns credit at the action level. Adding world modeling improves its success in all four ALFWorld and WebShop comparisons in Tab. 1; for example, 3B ALFWorld success rises from 75.00% to 92.19%. Observation supervision therefore provides complementary value even with fine-grained credit assignment.

Observation 5: Auxiliary world modeling improves action validity and interaction efficiency. The behavioral analyses further show practical improvements: WorldPO reduces the ALFWorld rate of environment-invalid actions from 25.28% to 12.19% (Fig. 7). On tasks successfully completed by both methods, average interaction turns decrease from 11.72 to 10.89 on ALFWorld, 7.03 to 6.53 on WebShop, and 2.58 to 2.39 on Search (Fig. 9).

(3) Direct experimental comparison with DyMo/ECHO. Given the limited time and computational resources, we are working to complete these additional experiments. We will share the results as they become available during the rebuttal period.

In summary, our experiments show that auxiliary world modeling can improve task performance and interaction efficiency, while our analyses show that policy improvements can persist without exact action-observation pairing. Together, these findings support WorldPO as a unified framework applicable across policy optimizers and world-modeling objectives. We have clarified WorldPO's contribution beyond DyMo/ECHO and added the results using DPO for observation prediction to the updated manuscript. We will add the baseline comparison results once available.

AuthorsOpenReview ↗

Response of review Pzwp (3/3)

W2. Q1. The gain on Search remains limited

Reply: Thank you for raising this comment. Tab. 7 (Tab. 3 in the original submission) uses the published baseline trained with the Search-R1 pipeline. Tab. 8 (Tab. 4 in the original submission) uses our GRPO rerun, trained with the same verl-agent pipeline as WorldPO. The stronger rerun baseline accounts for most of the reduction in the observed gap. Nevertheless, we would like to clarify that world modeling provides a statistically significant improvement in overall Search accuracy. Specifically, we quantify this improvement using correct-answer counts, distinguish average accuracy from overall accuracy, and examine a possible explanation for the smaller improvement than on ALFWorld and WebShop.

We have computed paired confidence intervals, and the results support a meaningful, statistically significant improvement in WorldPO's overall Search accuracy. To make this benefit concrete, WorldPO answers 398 more questions correctly than our local GRPO baseline on the same 51,713 questions across seven benchmarks. In Tab. 8, average accuracy is the mean of the six benchmark accuracies. Here, we also include Bamboogle and compute overall accuracy as the number of correct answers divided by all 51,713 questions. We compute paired bootstrap 95% CIs by resampling questions within each benchmark, keeping repeated questions together:

DatasetGRPO correct answers (success rate)WorldPO correct answers (success rate)Net increasePaired 95% CI of accuracy difference (%)
NQ1,594 (44.16%)1,652 (45.76%)+58[0.44, 2.77]
TriviaQA6,825 (60.33%)6,933 (61.28%)+108[0.35, 1.55]
PopQA6,306 (44.20%)6,553 (45.93%)+247[1.20, 2.27]
HotpotQA2,933 (39.61%)2,956 (39.92%)+23[−0.54, 1.16]
2WikiMultiHopQA4,794 (38.12%)4,794 (38.12%)0[−0.83, 0.83]
MuSiQue364 (15.06%)326 (13.49%)−38[−2.89, −0.25]
Bamboogle39 (31.20%)39 (31.20%)0[−6.40, 6.40]
Total22,855 (44.20%)23,253 (44.97%)+398[0.45, 1.09]

Across all 51,713 questions, the 398 additional correct answers correspond to a 0.77% absolute improvement in overall accuracy, from 44.20% to 44.97% (paired 95% CI: [0.45%, 1.09%]). This overall accuracy improvement is statistically significant.

To examine the smaller improvement on Search, we compare the observations available for learning per trajectory and per task:

EnvironmentPer trajectoryPer task
ALFWorld (3B)35.52284.12
WebShop (3B)10.1280.97
Search (3B)1.467.29

Search provides only 1.46 observations per trajectory, fewer than in either ALFWorld or WebShop. This may partly explain its smaller improvement from world modeling.

In summary, WorldPO improves average Search accuracy over local GRPO and achieves a statistically significant improvement in overall Search accuracy. We agree that the presentation of the Search results can be improved. In the updated manuscript, we have included the local GRPO comparison in the main text, clarified the two accuracy measures, and added the correct-answer counts and observation-frequency analysis.

References

[1] World Modelling Improves Language Model Agents, arXiv:2506.02918, 2025.

[2] ECHO: Terminal Agents Learn World Models for Free, arXiv:2605.24517, 2026.

Reviewer_gJmdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

WorldPO adds a second loss to agentic RL. From the same on-policy rollouts, one LLM is trained both with a standard policy surrogate (GRPO, RLOO or GiGPO) on its own responses, and with token-level likelihood on the observation the environment returns after each action. The predictor shares all parameters with the policy and is dropped at test time, so nothing changes at deployment. Section 3.3 motivates this with three identities about conditional entropy and gradient linearity. Evaluation covers ALFWorld, WebShop and search QA, with Qwen2.5 at 1.5B to 7B, and the paper reports gains in all three. Section 5 then asks where the gain comes from, mainly by reassigning the observation targets and by counting invalid actions and interaction turns.

Critical strengths

The behavioural analysis is the part of this paper I believe. The agents trained with the extra loss issue far fewer actions the environment rejects, and Appendix I shows this holds when you control for how much the two agents act at all, which is the comparison that matters and the one most papers skip.

The appendix also makes the paper auditable, which I want to credit explicitly. The authors run proper paired statistics, disclose which run each figure comes from, and say plainly where their own analysis stops. Note, that every criticism below comes from their own tables, and I found no arithmetic error anywhere in the paper.

Critical weaknesses

Wherever the authors run the controlled version of a comparison, the advertised effect mostly disappears, and the main body reports the uncontrolled version instead. Search is the clearest case. The text compares against a GRPO number taken from Search-R1, but the authors also ran their own GRPO under matched conditions, and against that control the gain is about half a point rather than five (Tab. 4). The same pattern holds on ALFWorld and WebShop, where the evaluation is small enough that re-measuring the same frozen checkpoints moves success by more than the effects being claimed. On top of this, no confidence interval anywhere in the paper covers the proposed method, and every success interval which does speak to whether the extra loss helps includes zero (Tab. 20). Appendix I suggests the authors already hold the logs needed to close this gap, so it is an odd omission rather than an impossible one.

The mechanism evidence has the same shape. The reassignment study is the right experiment, and it does not come out the way the paper needs. On ALFWorld, supervising on an observation taken from a completely different task does exactly as well as supervising on the correct one, and a mismatched observation from the same task is more favorable to it still (Fig. 6). The authors read this as evidence for task-relevant knowledge (L666-668). Although, it is not clear to me how that reading survives an exact tie with scrambled supervision. What the paper has shown is that adding observation text to the objective helps somewhat; what it claims is that predicting the actual consequence of the actual action is what helps, and its own ablation does not separate those two.

Overall, the idea of auxiliary predictions is quite established one and often found with mixed results across many settings in RL (and RL in LLM as well), I could not find any insight for myself that will make me say that yes, this is the recipe that provides enough external validity and i need this additional training machinery for the gains claimed (which is very often the case with auxillary objectives).

Questions for authors
  1. You ran your own GRPO on Search and report it in Tab. 4, but the main text compares against Search-R1's published number instead. Which is the control, and is half a point the effect of world modeling here?

  2. Appendix I says you already re-evaluated the proposed model against GRPO on paired episodes in all three environments. Please report the paired success intervals in more details. Do any of them exclude zero?

  3. Why is correctly paired supervision the proposed method, when reassigned targets match or beat it in your own Fig. 6? Relatedly, is there an arm with the auxiliary loss alone, and a sweep over its weight, which is fixed throughout?

  4. Which ALFWorld split do you evaluate on? Your baselines sit well below the published GiGPO numbers on some cells and above them on others, and without the split I cannot tell whether this is a protocol difference or a weaker implementation.

8 comments
AuthorsOpenReview ↗

Response of review gJmd (1/6)

Thank you for reviewing our submission and for highlighting the behavioral analysis and the transparency of our experimental reporting. Below, we address your comments on the performance comparisons, observation supervision, and WorldPO's contribution point by point. We have included all new results and clarifications in the revised manuscript and highlighted the changes in blue.

W1. Q1. Q2. Whether WorldPO's performance improvements are meaningful and statistically supported

Reply: Thank you for raising this comment. Fig. 4 includes both the officially reported Search-R1 results and our GRPO rerun using verl-agent; the appendix summarizes the local comparison in Tab. 8 (Tab. 4 in the original submission). We agree that these comparisons can be presented more clearly. Nevertheless, we would like to clarify that world modeling provides a statistically significant improvement in overall Search accuracy even against our rerun GRPO baseline. Specifically, (1) we quantify this improvement using correct-answer counts and distinguish average accuracy from overall accuracy, and (2) we report repeated evaluations and paired confidence intervals on ALFWorld and WebShop.

(1) We have computed paired confidence intervals, and the results support a meaningful, statistically significant improvement in WorldPO's overall Search accuracy. WorldPO answers 398 more questions correctly than our local GRPO baseline on the same 51,713 questions across seven benchmarks. Tab. 8 reports average accuracy across benchmarks; here, we additionally report overall accuracy across all 51,713 questions. We compute paired bootstrap 95% CIs by resampling questions within each benchmark, keeping repeated questions together:

DatasetGRPO correct answers (success rate)WorldPO correct answers (success rate)Net increasePaired 95% CI of accuracy difference (%)
NQ1,594 (44.16%)1,652 (45.76%)+58[0.44, 2.77]
TriviaQA6,825 (60.33%)6,933 (61.28%)+108[0.35, 1.55]
PopQA6,306 (44.20%)6,553 (45.93%)+247[1.20, 2.27]
HotpotQA2,933 (39.61%)2,956 (39.92%)+23[−0.54, 1.16]
2WikiMultiHopQA4,794 (38.12%)4,794 (38.12%)0[−0.83, 0.83]
MuSiQue364 (15.06%)326 (13.49%)−38[−2.89, −0.25]
Bamboogle39 (31.20%)39 (31.20%)0[−6.40, 6.40]
Total22,855 (44.20%)23,253 (44.97%)+398[0.45, 1.09]

Across all 51,713 questions, the 398 additional correct answers correspond to a 0.77% absolute improvement in overall accuracy, from 44.20% to 44.97% (paired 95% CI: [0.45%, 1.09%]). This overall accuracy improvement is statistically significant.

To examine the smaller improvement on Search, we compare the observations available for learning per trajectory and per task:

EnvironmentPer trajectoryPer task
ALFWorld (3B)35.52284.12
WebShop (3B)10.1280.97
Search (3B)1.467.29

Search provides only 1.46 observations per trajectory, fewer than in either ALFWorld or WebShop. This may partly explain its smaller improvement from world modeling.

AuthorsOpenReview ↗

Response of review gJmd (2/6)

(2) WorldPO improves average success on ALFWorld and WebShop in repeated evaluations. Our computational resources are limited, and these multi-turn agent experiments required considerable time and effort. Each result in Tab. 1 is based on a single training run. The original evaluations used one seed, and we did not select training or evaluation seeds for favorable results. To show the consistency of our results, we reevaluate the fixed GRPO and WorldPO checkpoints across three evaluation seeds, each selecting 128 task instances per environment shared by both methods. We agree that statistical uncertainty and evaluation variability can be presented more clearly and have therefore added paired confidence intervals for the main WorldPO-GRPO comparisons to complement the target-reassignment intervals in Tab. 27 (Tab. 20 in the original submission). We report mean ± std across seeds and paired bootstrap 95% CIs, keeping both models' results and repeated evaluations of each task together when resampling:

EnvironmentModelMethod / comparisonSeed 0 (%)Seed 1 (%)Seed 2 (%)Mean ± std (%)95% CI of mean improvement (%)
ALFWorld1.5BGRPO62.5059.3860.1660.68 ± 1.63N/A
ALFWorld1.5BWorldPO60.9464.8464.8463.54 ± 2.26N/A
ALFWorld1.5BWorldPO − GRPO-1.56+5.47+4.69+2.86 ± 3.85[-3.97, 9.66]
WebShop1.5BGRPO55.4746.8853.9152.08 ± 4.58N/A
WebShop1.5BWorldPO60.1656.2558.5958.33 ± 1.97N/A
WebShop1.5BWorldPO − GRPO+4.69+9.38+4.69+6.25 ± 2.71[0.52, 11.95]

WorldPO achieves a statistically significant improvement on WebShop, with an average absolute improvement of 6.25% and a paired 95% CI of [0.52%, 11.95%]. Although the ALFWorld interval includes zero, WorldPO achieves a 2.86% average advantage over GRPO.

In summary, WorldPO improves average Search accuracy over local GRPO and achieves a statistically significant improvement in overall Search accuracy; supplementary 1.5B evaluations demonstrate a statistically significant improvement on WebShop and higher average success on ALFWorld. In the updated manuscript, we have included the local GRPO comparison in the main text, clarified the Search accuracy measures, and added the repeated evaluations and paired confidence intervals reported above.

AuthorsOpenReview ↗

Response of review gJmd (3/6)

W2. Why WorldPO improves even without correct action-observation pairing

Reply: Thank you for raising this comment. We would like to clarify that auxiliary observation learning can improve policy performance beyond exact action-observation pairing. Correct pairing is the most direct way to construct next-observation targets in WorldPO. Specifically, we (1) compare task performance and observation prediction, (2) evaluate performance on unseen ALFWorld tasks, and (3) discuss two possible explanations.

(1) Within-task supervision achieves the highest success rates, and supervision with unrelated text also improves performance in both environments. We evaluate the fixed 1.5B checkpoints on the same 128 task instances for each of three evaluation seeds. The table reports success rates averaged across seeds, with standard deviations. We measure perplexity (PPL) on the same environment observations for all checkpoints, using the same recorded agentic histories and generated responses as inputs. For correct pairing on ALFWorld, we use the Fig. 6 checkpoint from a separate training run of the same WorldPO method reported in Tab. 1. For WebShop, we use the main WorldPO checkpoint from Tab. 1. For irrelevant-observation supervision, we use excerpts from Shakespeare's writings as prediction targets and retain the original agentic history and generated response as inputs.

MethodALFWorld success (%)WebShop success (%)ALFWorld PPL ↓WebShop PPL ↓
GRPO60.68 ± 1.6352.08 ± 4.5810.812.84
Correct-pair72.92 ± 1.1958.33 ± 1.971.231.80
Within-task75.52 ± 2.7465.63 ± 3.411.411.86
Cross-task66.15 ± 2.2661.46 ± 1.971.611.77
Irrelevant-observation supervision74.74 ± 0.9062.50 ± 4.355.622.69

Both reassignment variants improve success over GRPO on ALFWorld and WebShop. Within-task supervision achieves the highest success rates in both environments and the highest average accuracy on Search among the settings in Fig. 6. Irrelevant-observation supervision also improves success over correctly paired supervision by 1.82% on ALFWorld and 4.17% on WebShop in absolute terms. All three variants trained on environment observations achieve lower PPL than GRPO and the model trained with irrelevant-observation supervision, showing that environment observations remain useful for prediction after reassignment.

(2) Within-task and irrelevant-observation supervision improve performance on unseen ALFWorld tasks. We evaluate the five variants on the same tasks from the unseen split (valid_unseen, Tab. 4). Within-task and irrelevant-observation supervision both outperform GRPO and Cross-task.

MethodUnseen ALFWorld success (%)
GRPO52.86 ± 1.63
Correct-pair66.15 ± 2.39
Within-task73.18 ± 2.39
Cross-task61.20 ± 4.58
Irrelevant-observation supervision73.44 ± 0.00

(3) The results suggest two complementary explanations for the policy gains.

  • Task representation learning. By checking the behavior in WebShop tasks, we find that the within-task policy averages 0.53 repeat clicks on already selected options per episode, compared with 1.31 for the cross-task policy, across the 155 episodes completed by all four policies (Tab. 35). Both reassignment variants achieve lower PPL on actual environment observations than GRPO and the model trained with irrelevant-observation supervision, consistent with learning recurring patterns in environment feedback. Within-task targets also preserve information specific to the current task, which may help the policy keep track of its selections and avoid repeating clicks.
  • Auxiliary regularization. On unseen ALFWorld tasks, irrelevant-observation supervision raises success from 52.86% with GRPO to 73.44%. The gains from predicting unrelated text are consistent with auxiliary language learning helping the policy generalize to unseen tasks. InstructGPT [1] adds a language-modeling loss on pretraining text to mitigate regressions on general NLP tasks, and LaMo [2] uses auxiliary next-token prediction to stabilize policy learning on Kitchen.

In summary, task representation learning and auxiliary regularization offer complementary explanations for why auxiliary learning can improve policy performance beyond exact pairing. We have added the repeated evaluations and PPL results to Tab. 3 and the unseen-task results to Tab. 4, and clarified these interpretations in the main text.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A General Summary by the Authors

Dear Area Chair,

Thank you for overseeing the review of our submission. We summarize WorldPO’s contributions, the strengths noted by the reviewers, and how we addressed their questions.

(1) Main contributions

  • Holistic agentic learning. To systematically study how world modeling supports policy optimization, we introduce WorldPO, a unified framework that trains the same LLM to improve its policy using task feedback and to predict observations returned during those interactions.
  • Systematic evaluation. Across ALFWorld, WebShop, and Search, we study how policy optimizers, world-modeling objectives, and observation targets affect task performance.
  • Understanding the benefits of world modeling. We find that policy improvements can persist when observation targets are reassigned. We also identify practical benefits, including fewer invalid actions and fewer turns on jointly solved tasks.

(2) Strengths highlighted by the reviewers

  • Simple, practical design. Pzwp and 4S9J noted the method’s simplicity. Pzwp highlighted its reuse of observations and its unchanged inference interface.
  • Broad evaluation. Pzwp and 4S9J noted the range of environments, model sizes, and policy optimizers.
  • Behavioral evidence and clear reporting. gJmd highlighted the reduction in invalid actions and the appendix’s documentation of paired evaluations and experimental runs.

(3) Responses

Discussion areaReviewer questionOur response
Contribution and prior workHow does WorldPO extend prior work on auxiliary learning, including DyMo and ECHO? (gJmd W3, Pzwp W1/Q2, 4S9J W1/W2/Q1/Q2)We clarify the contribution of the systematic study, add an ECHO-based comparison on ALFWorld, and analyze how the loss formulations affect the auxiliary update.
Role of world modelingWhat do the observation-target comparisons reveal about how auxiliary supervision helps? (gJmd W2, 4S9J W3/Q3)We add comparisons on seen and unseen tasks and explain how the results support task representation learning and auxiliary regularization as complementary explanations.
How do the choice of prediction targets and the auxiliary objective affect performance? (gJmd Q3)We explain the default target choice and report experiments that vary the loss weight and train with observation prediction alone.
What is the predictive rationale for WorldPO? (4S9J W5)We retain the rationale for observation prediction in the main text and move the theoretical analysis to the appendix.
Evaluation and baselinesHow does WorldPO perform against GRPO on Search? (gJmd W1/Q1/Q2, Pzwp W2/Q1)We report both Search baselines in the main text and add a paired confidence interval supporting the improvement in overall Search accuracy.
What evidence supports the ALFWorld and WebShop improvements across evaluation and training runs? (gJmd W1/Q2, 4S9J W4/Q4)We report repeated evaluations and independent training runs. WorldPO achieves higher average success on both ALFWorld and WebShop, with a statistically significant task-level improvement on WebShop.
How were the ALFWorld policy baselines evaluated? (gJmd Q4)We explain why we reran the baselines in the same framework and how to ensure the fairness of the experimental comparisons.

We have incorporated these results and clarifications into the revised manuscript and highlighted the changes in blue. Thank you for considering the additional evidence alongside the reviews.

Best regards,

Authors of #32717

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.