acceptodds
Under review as a conference paper at ICLR 2027

CoWISE: Co-Evolving World Models with Policies through Semantic–Behavioral Coordination

Abstract

World models enable language agents to predict the consequences of their actions and improve their policies through simulated interaction. To train these models, existing approaches typically use trajectories collected from agent-environment interactions to reproduce real observations. However, even semantically similar simulated observations can still induce different policy actions (e.g., simulated observations closely match the real observation in overall content but miss key objects, leading the policy to take incorrect actions), which we identify as behavioral mismatch. This behavioral mismatch grows as the policy continues learning while the world model remains fixed, thereby further misleading the policy into choosing wrong actions and degrading its performance. To reduce this mismatch, we propose **CoWISE**, the first dual-level co-evolution framework that continually adapts the world model to the evolving policy through semantic-behavioral coordination. Specifically, at the policy–world level, CoWISE enables the world model and the policy to co-evolve by leveraging each other’s predictions. At the semantic–behavioral level, it adaptively balances real-world observational supervision and policy-derived behavioral supervision. To evaluate CoWISE, we conduct extensive experiments across 8 benchmarks, demonstrating that CoWISE improves policy performance and learns a world model with higher behavioral fidelity. Furthermore, using this learned world model for test-time scaling yields further gains, outperforming combinations of separately trained policy and world model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
text-based world models, co-evolution, behavior fidelity, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cs8iOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
3(+2)
Presentation
3(+2)
Soundness
2(+1)
Read review
Paper summary

This paper proposes CoWISE, a method that jointly evolves a policy and a world model while maintaining both semantic fidelity and behavioral fidelity. The method demonstrates substantial performance improvements across eight benchmarks.

Critical strengths

S1. The method is evaluated across a diverse set of benchmarks. Reporting results on eight benchmarks provides sufficient evidence that its performance gains generalize across tasks.

Critical weaknesses

W1: The method lacks novelty. As discussed in the paper’s Related Work section, an architecture in which the policy and world model share a single set of parameters has already been adopted in many prior studies. The use of behavioral fidelity for GRPO training also does not differ substantially from BehR (Huang et al., 2026). Although the fixed objective weights in Equation 5 are presented as a novel contribution, they appear to be heuristic, and no ablation study examines the contribution of each term. Moreover, the ablation results in Table 2 show no meaningful performance improvement from using fixed objective weights.

W2: The ablation results are inconclusive. On ALFWorld, the lowest score among the ablated variants is 91.2 for the variant without TTS, which is still 0.4 points higher than the strongest baseline, GiGPO, at 90.8. This raises the question of whether CoWISE’s performance gains can actually be attributed to the proposed method. A clearer explanation of these results is needed.

Questions for authors

Q1. Why was test-time scaling not applied to PaW and TAPO? If it can be applied, what performance would these methods achieve?

Q2. In Figure 1(a), behavioral fidelity decreases consistently on SearchQA but follows a roughly sinusoidal pattern on ALFWorld. What explains this difference?

14 comments
AuthorsOpenReview ↗

Response to Reviewer cs8i (1/5)

We sincerely thank the reviewer for the careful feedback and for recognizing our evaluation across eight benchmarks. Below, we explain CoWISE’s distinction from prior methods and the mathematical basis of adaptive weighting, and present additional ablations and test-time comparisons with PaW and TAPO.


W1: The method lacks novelty. As discussed in the paper’s Related Work section, an architecture in which the policy and world model share a single set of parameters has already been adopted in many prior studies. The use of behavioral fidelity for GRPO training also does not differ substantially from BehR (Huang et al., 2026). Although the fixed objective weights in Equation 5 are presented as a novel contribution, they appear to be heuristic, and no ablation study examines the contribution of each term. Moreover, the ablation results in Table 2 show no meaningful performance improvement from using fixed objective weights.

  • (1) Novelty and distinction from prior methods. We thank the reviewer for raising this concern. To clarify the novelty of CoWISE, we compare it with related methods across multiple dimensions in the table below. To our knowledge, CoWISE is the first framework to jointly optimize policy and world-model capabilities and their coordination within a single model, enabling both roles to co-evolve through mutual feedback.

    MethodCapabilities trainedOptimization focusPolicy → WM trainingWM learning objectiveWM → Policy trainingPolicy learning method
    W2WWMIndividual capabilityOffline trajectoriesSemantic fidelity——
    BehRWMIndividual capabilityFixed-policy behavioral supervisionSemantic + behavioral (staged)——
    PaW / TAPOPolicy + WMIndividual capabilitiesOnline trajectoriesSemantic fidelityAuxiliary prediction loss through shared parametersRL
    COMAPPolicy + WMIndividual capabilitiesOnline trajectoriesSemantic fidelityPredicted observations for action refinementSupervised learning (requires expert trajectories)
    CoWISEPolicy + WMCapabilities + coordinationOnline trajectories + current-policy behavioral supervisionSemantic + behavioral (joint)Simulated feedback for policy learningRL
    • (1.1) Compared with world-model training methods (W2W/BehR). W2W and BehR train standalone world models. W2W uses offline interaction trajectories to learn observation prediction, while BehR derives behavioral supervision from a fixed reference policy. Specifically, as shown in Figure 1(b), world models trained with BehR's fixed-policy supervision lose behavioral fidelity as the policy evolves. In contrast, CoWISE jointly trains policy and world-model capabilities within one model, enabling both roles to co-evolve. The evolving policy provides online interaction trajectories and updated behavioral supervision, while the world model provides simulated feedback for policy learning. Freezing the behavioral supervisor lowers ALFWorld TTS performance from 94.2 to 92.3 (Table 2).

    • (1.2) Compared with shared-parameter and co-evolution methods (PaW, TAPO, and COMAP). These methods focus on improving individual policy and world-model capabilities through auxiliary prediction or iterative interaction. However, strong performance in policy-only task execution and next-state prediction does not ensure effective joint use: simulated observations can still induce decisions inconsistent with real feedback (Sec. 2.2). In contrast, CoWISE jointly optimizes both capabilities and their coordination within a single model. Specifically, the evolving policy supervises whether simulated observations preserve its responses to real observations, while the policy learns to use simulated feedback through real task rewards. Removing behavioral supervision lowers ALFWorld TTS performance from 94.2 to 91.6 (Table 2), supporting the value of explicitly optimizing this coordination.

    Together, these comparisons highlight CoWISE's contribution: jointly improving policy and world-model capabilities and their coordination within a single model, so that both roles co-evolve through mutual feedback.

AuthorsOpenReview ↗

Response to Reviewer cs8i (2/5)

Continued response to W1: motivation and ablation evidence for objective weighting.

  • (2) Motivation and mathematical basis of adaptive weighting. We would first like to clarify that CoWISE uses adaptive objective weighting, with β\beta updated during training by Eq. (5), and we consider this adaptive mechanism one of our core contributions. The fixed-weight setting (β=0.5\beta=0.5) in Table 2 is an ablation used to evaluate this mechanism.

    Figure 1(c) shows that behavioral fidelity can continue to improve while prediction of real observations deteriorates. We therefore adapt the balance between the two objectives to improve behavioral fidelity while maintaining accurate observation prediction.

    Since semantic learning trains the model to predict real observations, we measure this quality through perplexity PkP_k on a fixed calibration set. Appendix B.4 shows that

    dk=log⁡PkPk⋆,d_k=\log\frac{P_k}{P_k^\star},

    where dkd_k is the increase in prediction loss from its historical minimum, and Pk⋆P_k^\star is the best historical perplexity. An allowed relative increase in perplexity therefore defines a prediction-loss budget. Equation (5) increases the behavioral weight while budget remains and shifts weight toward real-observation supervision when the budget is exceeded. The update uses measured prediction loss to balance adaptation to the current policy with accurate observation prediction.

  • (3) Ablation evidence. Table 2 separately removes real-observation supervision and behavioral supervision, and compares adaptive with fixed weights. On ALFWorld, removing either supervision term lowers the TTS score from 94.2 to 91.6. Replacing adaptive weighting with fixed weights lowers TTS scores by 0.3, 0.8, and 0.8 points on SearchQA, ALFWorld, and SciWorld, respectively. ALFWorld's TTS gain increases from 0 with fixed weights to +0.8 points with adaptive weighting.

    We further compare both variants using three independent training seeds per setting, reporting TTS scores (%) as mean ± standard deviation:

    EnvironmentAdaptive weightingFixed weights
    SearchQA46.2 ± 0.5245.9 ± 0.28
    ALFWorld94.2 ± 0.6393.4 ± 1.59

    The mean gains remain in both environments after independent retraining, supporting adaptive weighting for decisions made using world-model predictions.

AuthorsOpenReview ↗

Response to Reviewer cs8i (3/5)

W2: The ablation results are inconclusive. On ALFWorld, the lowest score among the ablated variants is 91.2 for the variant without TTS, which is still 0.4 points higher than the strongest baseline, GiGPO, at 90.8. This raises the question of whether CoWISE’s performance gains can actually be attributed to the proposed method. A clearer explanation of these results is needed.

We thank the reviewer for raising this question. We would like to explain why the ablated variants still outperform GiGPO without world-model-assisted TTS. CoWISE improves the policy itself through joint training with the world model, using shared parameters (internalizing world knowledge from real observations) and simulated training feedback (learning about the environment and how to act on simulated observations). These training gains therefore remain when world-model-assisted TTS is disabled. Below, we explain these two mechanisms and the corresponding ablation evidence.

  • (1) Shared parameters allow world-model learning to improve the policy itself. Real-observation supervision trains the world model to predict how the environment responds to actions. These updates modify the same parameters used by the policy, allowing the learned environment knowledge to support action selection even without test-time simulation (Sec. 3.4, Eq. 4). Consistent with this mechanism, removing real-observation supervision lowers ALFWorld policy-only success from 93.4% to 91.2%, a 2.2-point drop (Table 2).

  • (2) Simulated feedback allows the policy to learn world knowledge even without real-observation supervision. As training progresses, world-model predictions are gradually incorporated into policy trajectories. The policy learns from these predicted action consequences using real task rewards, providing another route for world knowledge to improve the trained policy (Secs. 3.1–3.2, Appendix B.2). To assess this remaining contribution, we additionally remove simulated feedback from the variant without real-observation supervision, while retaining behavioral supervision.

    Real-observation supervisionWith simulated training feedbackWithout simulated training feedback
    Retained93.493.4
    Removed91.290.5 (new)

    The 91.2 variant still benefits from CoWISE’s simulated training feedback. Further removing this feedback lowers policy-only success to 90.5 (−0.7 points), close to GiGPO’s 90.8. The 2.9-point gap between full CoWISE and this joint ablation (93.4 vs. 90.5) supports the contribution of CoWISE’s training mechanisms even without world-model-assisted TTS.

Together, these results show that CoWISE’s world model improves the policy through shared parameters and simulated feedback during training. These improvements are retained in the learned policy, explaining its strong performance even without world-model-assisted TTS.

Reviewer_c5LPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoWISE jointly trains a policy and a textual world model using a shared backbone. World-model training combines real-observation supervision with behavioral supervision derived from the evolving policy, while an adaptive controller balances the two objectives. Simulated observations are incorporated into policy learning and uncertainty-gated test-time action selection. Experiments cover six search-augmented QA datasets, ALFWorld, and SciWorld.

Critical strengths
  1. The paper identifies a meaningful limitation of behavioral alignment with a frozen policy: observations that preserve an earlier policy’s decisions may no longer preserve those of an updated policy. The motivating diagnostics and frozen-supervisor ablation support refreshing behavioral supervision during training.

  2. The evaluation separates policy-only performance, behavioral fidelity, and test-time simulation. Controlled comparisons under a common frozen policy strengthen the evidence that the co-evolved world model provides more useful decision support. Improvements are observed across both open-ended and rule-governed tasks.

Critical weaknesses
  1. The contribution of the central coordination mechanisms is concentrated in test-time scaling. In Table 2, freezing the behavioral supervisor changes policy-only performance by only 0.2–0.3 points, removing simulated training feedback does not reduce policy-only performance, and fixed objective weights remain within 0.1 points of the full model. Thus, the evidence supports improved usability of simulations more strongly than a substantial policy-learning benefit from the complete co-evolution mechanism.

  2. Several incremental improvements are modest: test-time scaling raises the overall score from 52.8 to 53.5, while behavioral diagnostics use relatively small samples. The reported results do not establish robustness across independent training runs. Additional world-model sampling, behavioral scoring, and test-time candidate evaluation also introduce computation whose trade-off against the observed gains needs clearer quantification.

Questions for authors
  1. Do the gains from adaptive weighting and current-policy behavioral supervision persist across independent training runs?

  2. What are the additional training and inference costs relative to GRPO and shared-backbone observation-supervision baselines, and how does performance compare under matched computational budgets?

5 comments
AuthorsOpenReview ↗

Response to Reviewer c5LP (1/4)

We sincerely thank the reviewer for the constructive and encouraging feedback. We appreciate your recognition of the need to update behavioral supervision as the policy evolves and the value of separately evaluating policy performance, behavioral fidelity, and test-time simulation. Following your suggestions, we clarify the contribution of each component and provide additional evidence from independent training runs, larger behavioral evaluations, and computational-cost comparisons. We address each concern below.


W1: The contribution of the central coordination mechanisms is concentrated in test-time scaling. In Table 2, freezing the behavioral supervisor changes policy-only performance by only 0.2–0.3 points, removing simulated training feedback does not reduce policy-only performance, and fixed objective weights remain within 0.1 points of the full model. Thus, the evidence supports improved usability of simulations more strongly than a substantial policy-learning benefit from the complete co-evolution mechanism.

We thank the reviewer for raising this question. We would first like to clarify that CoWISE aims to co-evolve the policy and world model through joint training. Specifically, improving policy-only performance is one part of this goal; we also aim to maintain the world model's fidelity as the policy evolves and train the policy to use its simulated observations. To explain how the three mechanisms contribute to this goal, we discuss their roles and corresponding evidence below.

  • (1) Behavioral supervision supports world-model fidelity. It trains the world model to generate observations that preserve the current policy's decisions. Consistent with this role, removing it turns TTS gains into losses in all three environments, including from +0.8 to -1.5 points on ALFWorld (Table 2). Furthermore, to directly assess its effect on behavioral fidelity, we compare the full and ablated world models using the same frozen CoWISE-Policy and evaluation samples (Sec. 5.2). In this comparison, parentheses report percentage-point changes relative to CoWISE-WM.

    EnvironmentWorld modelAction Agreement (%) ↑Task Retention (%) ↑
    SearchQACoWISE-WM46.195.3
    SearchQAw/o behavioral supervision40.7 (−5.4)92.2 (−3.1)
    ALFWorldCoWISE-WM84.798.4
    ALFWorldw/o behavioral supervision83.5 (−1.2)96.9 (−1.5)
    SciWorldCoWISE-WM90.796.9
    SciWorldw/o behavioral supervision88.0 (−2.7)93.8 (−3.1)

    Behavioral supervision therefore advances CoWISE's joint-learning objective by training a world model that better preserves the policy's decisions and task outcomes.

  • (2) Simulated training feedback helps the policy learn to use simulations. Accordingly, removing it reduces TTS gains in all three environments, including from +2.6 to +0.9 points on SciWorld (Table 2). Beyond these TTS gains, we assess its contribution to policy-only learning by additionally removing it together with real-observation supervision while retaining behavioral supervision. We report the corresponding ALFWorld policy-only success rates (%) below:

    Real-observation supervisionWith simulated feedbackWithout simulated feedback
    Retained93.493.4
    Removed91.290.5

    Simulated feedback therefore enables the world model to contribute to policy learning, improving the policy's use of simulations and yielding a 0.7-point policy-only gain when real-observation supervision is absent.

  • (3) Adaptive objective coordination improves the usefulness of simulated observations. Because real-observation and behavioral supervision follow different learning dynamics, we adapt their relative weights during training (Section 3.4). As shown in Table 2, replacing adaptive weighting with fixed weights lowers TTS scores by 0.3 to 0.8 points across all three environments. Adaptive weighting therefore improves how effectively the policy uses its jointly trained world model, supporting CoWISE's joint-learning objective.

Together, these results support jointly learning a policy and a world model that provide useful feedback to each other. These gains in world-model fidelity and the policy's use of simulations therefore directly support CoWISE's co-evolution objective.

AuthorsOpenReview ↗

Response to Reviewer c5LP (2/4)

W2: Several incremental improvements are modest: test-time scaling raises the overall score from 52.8 to 53.5, while behavioral diagnostics use relatively small samples. The reported results do not establish robustness across independent training runs. Additional world-model sampling, behavioral scoring, and test-time candidate evaluation also introduce computation whose trade-off against the observed gains needs clearer quantification.

Q1: Do the gains from adaptive weighting and current-policy behavioral supervision persist across independent training runs?

We thank the reviewer for raising these questions. We would first like to clarify that our evaluation addresses robustness across independent training runs, behavioral fidelity, and computational cost.

  • (1) Independent training runs. Following your suggestion, we retrain CoWISE and the two ablation variants using three independent random seeds per setting on both SearchQA (open-ended) and ALFWorld (rule-governed). In each case, the runs start from the same initial model and follow the corresponding settings in Section D.1. We then report task scores (%) as mean ± standard deviation across the three independent training runs for each setting.

    EnvironmentVariantw/o TTS (policy-only)w/ TTS
    SearchQACoWISE45.9 ± 0.5746.2 ± 0.52
    SearchQAw/ frozen behavioral supervisor45.7 ± 0.4045.3 ± 0.69
    SearchQAw/ fixed objective weights45.8 ± 0.4745.9 ± 0.28
    ALFWorldCoWISE93.4 ± 0.6394.2 ± 0.63
    ALFWorldw/ frozen behavioral supervisor93.1 ± 1.9092.3 ± 3.24
    ALFWorldw/ fixed objective weights93.4 ± 0.7393.4 ± 1.59

    Averaged over three independent training runs, CoWISE's TTS scores exceed the frozen-supervisor variant by 0.9 and 1.9 points on SearchQA and ALFWorld, and the fixed-weight variant by 0.3 and 0.8 points, respectively. Thus, the benefits of current-policy behavioral supervision and adaptive weighting for the policy's use of simulations persist after independent retraining.

  • (2) Behavioral fidelity on larger evaluation samples. Following your suggestion, we expand the evaluation to 200 transitions and 200 tasks per environment on SearchQA (open-ended) and ALFWorld (rule-governed). For ALFWorld, we sample the tasks from the main evaluation set described in Section C.1. To keep the comparison controlled, all world models use the same frozen CoWISE-Policy and evaluation samples within each environment. We then compare their action agreement and task retention.

    EnvironmentWorld modelAction Agreement (%) ↑Task Retention (%) ↑
    SearchQAW2W40.092.4
    SearchQABehR33.289.1
    SearchQACoWISE-WM46.195.7
    ALFWorldW2W84.396.8
    ALFWorldBehR81.795.2
    ALFWorldCoWISE-WM84.798.4

    In the expanded evaluation, CoWISE-WM leads W2W and BehR on both metrics, with task retention exceeding W2W by 3.3 points on SearchQA and 1.6 points on ALFWorld. This supports joint training of a world model whose observations better preserve the policy's decisions and task outcomes across open-ended and rule-governed environments.

AuthorsOpenReview ↗

Response to Reviewer c5LP (3/4)

Continued response to W2: computational cost.

  • (3) Computational cost. Following your suggestion, we compare the performance and cost of CoWISE with policy sampling and inference using W2W or BehR. The results show that CoWISE outperforms policy self-voting at comparable total benchmark time, while sharing weights between the policy and world model. We explain these findings through quantitative and qualitative evidence below.

    • (3.1) Quantitative evidence. We use the same frozen CoWISE-Policy, evaluation tasks, hardware, and concurrency settings for all comparisons. The policy-sampling baseline uses self-voting and has a total benchmark time comparable to CoWISE + TTS. All world-model variants follow Section D.3 with K = 4.

      EnvironmentInference configurationScore (%) ↑Total benchmark time (min) ↓Models deployed concurrently ↓
      SearchQAPolicy-only45.99.051
      SearchQAPolicy sampling (self-voting)45.631.251
      SearchQAPolicy + W2W45.630.962
      SearchQAPolicy + BehR45.430.412
      SearchQACoWISE + TTS46.230.521
      ALFWorldPolicy-only93.47.721
      ALFWorldPolicy sampling (self-voting)92.321.951
      ALFWorldPolicy + W2W93.420.832
      ALFWorldPolicy + BehR92.722.342
      ALFWorldCoWISE + TTS94.222.051

      CoWISE + TTS improves SearchQA and ALFWorld by 0.3 and 0.8 points over policy-only execution, with total benchmark time increasing by 237.2% and 185.6%, respectively. At comparable total benchmark time, its 0.6- and 1.9-point gains over policy self-voting show that the jointly learned world model improves action selection more effectively than additional policy sampling.

    • (3.2) Qualitative evidence. CoWISE uses one shared model for policy and world-model inference, whereas Policy + W2W and Policy + BehR each require two deployed models. With higher task scores in both environments, CoWISE combines more effective simulation-assisted decisions with a single model that supports both roles.

Reviewer_p9p9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a co-evolving world model and agent policy. It identifies and tackles the problem in such systems -- the world model might not keep up with the policy’s evolution and thus generate next observations that do not provide information meaningful to the evolving policy. To this end, the authors design a behavioral supervision signal, in addition to the next-observation prediction and policy optimization objectives. They evaluate the method across a large array of tasks and observe decent improvements.

Critical strengths
  1. This paper identifies an important problem in world-model-policy dual models: a static world model may become less useful as the policy model evolves. This is critical when we guide the agent’s decisions with simulated observations, because seemingly similar observations can lead to different actions.

  2. The proposed solution is sensible and intuitive. Updating behavioral supervision with the current policy directly addresses this mismatch, and retaining next-observation supervision grounds the world model in real environmental feedback.

  3. The evaluation is quite extensive, covering six question-answering datasets and two interactive environments. The authors evaluate both downstream task performance and behavioral fidelity, e.g., action agreement and task retention. The ablations also highlight the contributions of the main training components.

Critical weaknesses
  1. In my opinion, the ablations do not validate how much the proposed components improve the policy-only performance. In particular, removing behavioral supervision or simulated training feedback has little effect on these results, although the differences are larger when simulations are used at inference.

  2. The main results are not supplemented with uncertainty estimates, which are particularly important when they are close to the baselines.

  3. Although Appendix E provides examples, concrete examples showing how the world model’s predictions and the policy’s responses change across training checkpoints would help us understand how they co-evolve. Some illustrative experiments tracing these changes would be beneficial.

Questions for authors

Please address the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer p9p9 (1/3)

We sincerely thank the reviewer for the constructive and encouraging feedback. We appreciate your recognition of the importance of adapting world models to evolving policies, the rationale behind our approach, and the breadth of our evaluation. Below, we clarify the roles of the training components, report results from repeated evaluations, and illustrate co-evolution through a case study across training checkpoints.


W1: In my opinion, the ablations do not validate how much the proposed components improve the policy-only performance. In particular, removing behavioral supervision or simulated training feedback has little effect on these results, although the differences are larger when simulations are used at inference.

We thank the reviewer for raising this question. We would first like to clarify that CoWISE aims to co-evolve the policy and world model through joint training. Improving policy-only performance is one part of this objective; we also aim to maintain the world model's semantic and behavioral fidelity as the policy evolves to provide useful simulated observations for policy learning and test-time decision-making. To further explain how behavioral supervision and simulated training feedback contribute to this overall goal, we discuss their roles and the corresponding ablation results below.

  • (1) Behavioral supervision supports world-model fidelity. Behavioral supervision trains the world model to generate observations that preserve the current policy's decisions. Table 2 shows that removing it turns TTS gains into losses in all three environments. On ALFWorld, the gain drops from +0.8 to -1.5 points. To directly assess its effect on behavioral fidelity, we compare the full and ablated world models using the same frozen CoWISE-Policy and evaluation samples, following Sec. 5.2. The full-model results below are from Figure 3.

    EnvironmentWorld modelAction Agreement (%) ↑Task Retention (%) ↑
    SearchQACoWISE-WM46.195.3
    SearchQAw/o behavioral supervision40.7 (−5.4)92.2 (−3.1)
    ALFWorldCoWISE-WM84.798.4
    ALFWorldw/o behavioral supervision83.5 (−1.2)96.9 (−1.5)
    SciWorldCoWISE-WM90.796.9
    SciWorldw/o behavioral supervision88.0 (−2.7)93.8 (−3.1)

    Removing behavioral supervision lowers both action agreement and task retention in all three environments, supporting its contribution to world-model behavioral fidelity.

  • (2) Simulated training feedback improves the policy's use of simulations. Training with world-model predictions helps the policy learn to act on simulated observations. Removing this feedback reduces TTS gains in all three environments, including from +2.6 to +0.9 points on SciWorld (Table 2).

The ablations show that behavioral supervision and simulated training feedback improve performance when the policy uses world-model predictions. Both components therefore play a critical role in CoWISE's co-evolution of the policy and world model, enabling them to learn from each other's predictions.

AuthorsOpenReview ↗

Response to Reviewer p9p9 (2/3)

W2: The main results are not supplemented with uncertainty estimates, which are particularly important when they are close to the baselines.

We thank the reviewer for this suggestion. Following your suggestion, we repeat our evaluations with three different evaluation seeds, and CoWISE retains its lead in overall mean performance under repeated evaluation. We report mean ± standard deviation for these evaluations and mark results taken from prior work with †. Our SearchQA policy-only evaluations use greedy decoding, giving a standard deviation of 0.0.

MethodSearchQA EM (%)ALFWorld SR (%)SciWorld SR (%)Overall Avg. (%)
GRPO44.288.7 ± 1.3249.1 ± 0.8750.4 ± 0.20
GiGPO43.6†90.8 ± 1.3†53.4†50.7†
PaW45.2†80.6 ± 3.1†53.0 ± 0.8750.6
TAPO41.883.6 ± 3.9†54.4 ± 0.8648.6
W2W + GRPO44.3 ± 0.1289.1 ± 1.3249.7 ± 0.8750.6 ± 0.22
BehR + GRPO44.1 ± 0.1288.0 ± 1.4648.7 ± 0.8750.2 ± 0.22
CoWISE (policy-only)45.9 ± 0.093.4 ± 1.0953.7 ± 0.8752.8 ± 0.17
CoWISE + TTS46.2 ± 0.1394.2 ± 0.9756.3 ± 0.8653.5 ± 0.19

CoWISE (policy-only) achieves an overall score of 52.8 ± 0.17. TTS further raises the overall score to 53.5 ± 0.19. CoWISE therefore retains its overall performance advantage after averaging over the three evaluation seeds.

AuthorsOpenReview ↗

Response to Reviewer p9p9 (3/3)

W3: Although Appendix E provides examples, concrete examples showing how the world model’s predictions and the policy’s responses change across training checkpoints would help us understand how they co-evolve. Some illustrative experiments tracing these changes would be beneficial.

We thank the reviewer for this suggestion. Following your suggestion, we compare world-model predictions and policy responses for the same ALFWorld interaction across early, middle, and late training checkpoints. This comparison connects changes in the world model's predictions with changes in the policy's decisions. We present the overall training trends and the case study below.

  • (1) Quantitative evidence. Figures 8(a, d) and 9(a, d) in Sec. D.4 show that policy reward and world-model behavioral fidelity both improve over training on SearchQA and ALFWorld. These curves provide an overall view of their development during joint training.

  • (2) Evolution of predictions and policy responses. The ALFWorld task is: Your task is to: put a hot apple in diningtable.

    The agent is holding apple 1, which it has just picked up from countertop 1. The apple has not yet been heated, so the agent needs to go to microwave 1 and heat it before placing it on diningtable 1. We compare each checkpoint's policy and world model using this same interaction history.

    CheckpointWorld-model predictionPolicy response to real observationPolicy response to simulated observation
    Early (step 50)You pick up the apple 1 from the countertop 1. This is a hot apple 1.go to diningtable 1go to diningtable 1
    Middle (step 100)You pick up the apple 1 from the countertop 1. This is a hot apple 1.go to microwave 1go to diningtable 1
    Late (step 150)You pick up the apple 1 from the countertop 1.go to microwave 1go to microwave 1

    Early in training, the policy heads toward the dining table under both observations. At the middle checkpoint, it chooses the microwave under the real observation but still chooses the dining table under the incorrect "hot apple" prediction. At the late checkpoint, the world model removes this error, and the policy chooses the microwave under both observations. The comparison shows the policy learning to act on the heating requirement and the world model correcting a prediction that affects the evolving policy's decision.

Together, the training curves and case study illustrate how the world model adapts as the policy learns and how its predictions support the policy's decisions, making their co-evolution concrete.


Thank you again for the constructive suggestions. We sincerely hope these clarifications and additional results address your concerns about component contributions, evaluation uncertainty, and the co-evolution of the policy and world model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.