- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 3(+2)
- Presentation
- 3(+2)
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes CoWISE, a method that jointly evolves a policy and a world model while maintaining both semantic fidelity and behavioral fidelity. The method demonstrates substantial performance improvements across eight benchmarks.
Critical strengths
S1. The method is evaluated across a diverse set of benchmarks. Reporting results on eight benchmarks provides sufficient evidence that its performance gains generalize across tasks.
Critical weaknesses
W1: The method lacks novelty. As discussed in the paper’s Related Work section, an architecture in which the policy and world model share a single set of parameters has already been adopted in many prior studies. The use of behavioral fidelity for GRPO training also does not differ substantially from BehR (Huang et al., 2026). Although the fixed objective weights in Equation 5 are presented as a novel contribution, they appear to be heuristic, and no ablation study examines the contribution of each term. Moreover, the ablation results in Table 2 show no meaningful performance improvement from using fixed objective weights.
W2: The ablation results are inconclusive. On ALFWorld, the lowest score among the ablated variants is 91.2 for the variant without TTS, which is still 0.4 points higher than the strongest baseline, GiGPO, at 90.8. This raises the question of whether CoWISE’s performance gains can actually be attributed to the proposed method. A clearer explanation of these results is needed.
Questions for authors
Q1. Why was test-time scaling not applied to PaW and TAPO? If it can be applied, what performance would these methods achieve?
Q2. In Figure 1(a), behavioral fidelity decreases consistently on SearchQA but follows a roughly sinusoidal pattern on ALFWorld. What explains this difference?
14 comments
Response to Reviewer cs8i (2/5)
Continued response to W1: motivation and ablation evidence for objective weighting.
-
(2) Motivation and mathematical basis of adaptive weighting. We would first like to clarify that
CoWISE uses adaptive objective weighting, with updated during training by Eq. (5), and we consider this adaptive mechanism one of our core contributions. The fixed-weight setting () in Table 2 is an ablation used to evaluate this mechanism.Figure 1(c) shows that behavioral fidelity can continue to improve while prediction of real observations deteriorates. We therefore adapt the balance between the two objectives to improve behavioral fidelity while maintaining accurate observation prediction.
Since semantic learning trains the model to predict real observations, we measure this quality through perplexity on a fixed calibration set. Appendix B.4 shows that
where is the increase in prediction loss from its historical minimum, and is the best historical perplexity. An allowed relative increase in perplexity therefore defines a prediction-loss budget. Equation (5) increases the behavioral weight while budget remains and shifts weight toward real-observation supervision when the budget is exceeded. The update uses measured prediction loss to balance adaptation to the current policy with accurate observation prediction.
-
(3) Ablation evidence. Table 2 separately removes real-observation supervision and behavioral supervision, and compares adaptive with fixed weights. On ALFWorld, removing either supervision term lowers the TTS score from 94.2 to 91.6. Replacing adaptive weighting with fixed weights lowers TTS scores by 0.3, 0.8, and 0.8 points on SearchQA, ALFWorld, and SciWorld, respectively. ALFWorld's TTS gain increases from 0 with fixed weights to +0.8 points with adaptive weighting.
We further compare both variants using three independent training seeds per setting, reporting TTS scores (%) as mean ± standard deviation:
Environment Adaptive weighting Fixed weights SearchQA 46.2 ± 0.52 45.9 ± 0.28 ALFWorld 94.2 ± 0.63 93.4 ± 1.59 The mean gains remain in both environments after independent retraining, supporting adaptive weighting for decisions made using world-model predictions.
Response to Reviewer cs8i (3/5)
W2: The ablation results are inconclusive. On ALFWorld, the lowest score among the ablated variants is 91.2 for the variant without TTS, which is still 0.4 points higher than the strongest baseline, GiGPO, at 90.8. This raises the question of whether CoWISE’s performance gains can actually be attributed to the proposed method. A clearer explanation of these results is needed.
We thank the reviewer for raising this question. We would like to explain why the ablated variants still outperform GiGPO without world-model-assisted TTS. CoWISE improves the policy itself through joint training with the world model, using shared parameters (internalizing world knowledge from real observations) and simulated training feedback (learning about the environment and how to act on simulated observations). These training gains therefore remain when world-model-assisted TTS is disabled. Below, we explain these two mechanisms and the corresponding ablation evidence.
-
(1) Shared parameters allow world-model learning to improve the policy itself. Real-observation supervision trains the world model to predict how the environment responds to actions. These updates modify the same parameters used by the policy, allowing the learned environment knowledge to support action selection even without test-time simulation (Sec. 3.4, Eq. 4). Consistent with this mechanism, removing real-observation supervision lowers ALFWorld policy-only success from 93.4% to 91.2%, a 2.2-point drop (Table 2).
-
(2) Simulated feedback allows the policy to learn world knowledge even without real-observation supervision. As training progresses, world-model predictions are gradually incorporated into policy trajectories. The policy learns from these predicted action consequences using real task rewards, providing another route for world knowledge to improve the trained policy (Secs. 3.1–3.2, Appendix B.2). To assess this remaining contribution, we additionally remove simulated feedback from the variant without real-observation supervision, while retaining behavioral supervision.
Real-observation supervision With simulated training feedback Without simulated training feedback Retained 93.4 93.4 Removed 91.2 90.5 (new) The 91.2 variant still benefits from CoWISE’s simulated training feedback. Further removing this feedback lowers policy-only success to 90.5 (−0.7 points), close to GiGPO’s 90.8. The 2.9-point gap between full CoWISE and this joint ablation (93.4 vs. 90.5) supports the contribution of CoWISE’s training mechanisms even without world-model-assisted TTS.
Together, these results show that CoWISE’s world model improves the policy through shared parameters and simulated feedback during training. These improvements are retained in the learned policy, explaining its strong performance even without world-model-assisted TTS.
Response to Reviewer cs8i (1/5)
We sincerely thank the reviewer for the careful feedback and for recognizing our evaluation across eight benchmarks. Below, we explain CoWISE’s distinction from prior methods and the mathematical basis of adaptive weighting, and present additional ablations and test-time comparisons with PaW and TAPO.
(1) Novelty and distinction from prior methods. We thank the reviewer for raising this concern. To clarify the novelty of CoWISE, we compare it with related methods across multiple dimensions in the table below. To our knowledge, CoWISE is the first framework to jointly optimize policy and world-model capabilities and their coordination within a single model, enabling both roles to co-evolve through mutual feedback.
(1.1) Compared with world-model training methods (W2W/BehR). W2W and BehR train standalone world models. W2W uses offline interaction trajectories to learn observation prediction, while BehR derives behavioral supervision from a
fixed reference policy. Specifically, as shown in Figure 1(b), world models trained with BehR's fixed-policy supervision lose behavioral fidelity as the policy evolves. In contrast, CoWISE jointly trains policy and world-model capabilities within one model, enabling both roles to co-evolve. The evolving policy provides online interaction trajectories and updated behavioral supervision, while the world model provides simulated feedback for policy learning. Freezing the behavioral supervisor lowers ALFWorld TTS performance from 94.2 to 92.3 (Table 2).(1.2) Compared with shared-parameter and co-evolution methods (PaW, TAPO, and COMAP). These methods focus on improving individual policy and world-model capabilities through auxiliary prediction or iterative interaction. However, strong performance in policy-only task execution and next-state prediction does not ensure effective joint use: simulated observations can still induce decisions inconsistent with real feedback (Sec. 2.2). In contrast, CoWISE jointly optimizes both capabilities and their coordination within a single model. Specifically, the evolving policy supervises whether simulated observations preserve its responses to real observations, while the policy learns to use simulated feedback through real task rewards. Removing behavioral supervision lowers ALFWorld TTS performance from 94.2 to 91.6 (Table 2), supporting the value of explicitly optimizing this coordination.
Together, these comparisons highlight CoWISE's contribution: jointly improving policy and world-model capabilities and their coordination within a single model, so that both roles co-evolve through mutual feedback.