acceptodds
Under review as a conference paper at ICLR 2027

MindWorld: Iterative Co-Optimization of Mental World Models and Policies for Social Intelligence

Abstract

Language agents increasingly operate in interactive environments, yet their understanding of users and counterparts is often not maintained as an explicit, evolving component of the decision process, limiting their ability to adapt as interactions evolve. We introduce MindWorld, a framework that explicitly couples a Mental World Model with a Policy Model. The Mental World Model dynamically tracks beliefs, desires, and intentions from the evolving interaction history, while the Policy Model conditions on these mental states together with the task context to generate actions. To improve long-horizon credit assignment, MindWorld combines trajectory-level outcome feedback with dedicated step-level rewards and converts them into step-wise relative advantages for reinforcement learning. We further introduce an iterative co-optimization procedure. The two models are trained in alternating phases using a trajectory-level outcome reward and step-level rewards from fixed scorers. Experiments on SOTOPIA, SOTOPIA-Hard, ALFWorld, and WebShop demonstrate consistent gains across social, household, and web-based interaction. With Qwen3-8B under self-play, MindWorld obtains GOAL/OVERALL scores of 9.27/4.03 on SOTOPIA and 8.16/3.94 on SOTOPIA-Hard. On ALFWorld and WebShop, the reported results exceed those of the evaluated general RL baselines, and integration with task-specific methods yields additional gains. These results show that explicitly modeling evolving mental states, together with fine-grained credit assignment and iterative policy optimization, provides an effective and general approach for improving interactive language agents.

Primary area:
reinforcement learning
Keywords:
Mental World Model, Social Intelligence, Credit Assignment, Multi-Turn Interaction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_u3VaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MindWorld, an LLM-based world model that tracks mental states (including beliefs, desires, and intentions) from conversational interactions and trains LLM agents using reinforcement learning with novel reward designs.

Critical strengths
  1. The proposed method is well motivated and elaborated.
  2. Comprehensive experimental results across multiple benchmarks demonstrate the effectiveness of the proposed method on goal-oriented interactions & decision-making tasks.
  3. Ablation studies and analysis of the training dynamics provide further support for the strengths of the proposed method.
Critical weaknesses
  1. Lack of in-depth case studies (although some cases are provided in the appendix) to investigate how LLM agents change their behavior using the proposed method.
  2. Lack of efficiency studies to examine the training costs introduced by the proposed method.
  3. Some questions and concerns, as in "Questions For Authors".
Questions for authors

Questions:

  1. Figure 1 MWM: As intention is commonly regarded as a goal-oriented mental state, "Goals" in the figure looks inappropriate.
  2. Experiments: How do the hyperparameter configurations during training affect performance?
  3. Results: How many runs were there for each reported score? What is the statistical significance and confidence?
  4. Is there any analysis and insights on the individual effect of Beliefs, Desires, and Intentions for different tasks?

Suggestions:

  1. A missing reference and potential baseline for comparison [1], which trains LLM agents to infer mental states and then complete the conversation on the Sotopia benchmark. In addition, there is concurrent work on MWM [2].
  • [1] Hwang et al. ACL 2026. Infusing Theory of Mind into Socially Intelligent LLM Agents.
  • [2] Fei and Zhao. arXiv 2026. Mental World Modeling.
4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your careful review and constructive suggestions.

To W1:

We provide two additional, more detailed SOTOPIA interaction case studies.

Case 1: With/Without BDI

Scenario: Mia wants William to share the only blanket during a camping trip, while William wants to keep it for himself.

Interaction History

Mia: "Could we share the blanket tonight?"
William: "If we split it, neither of us will stay warm. And I don't want to sleep too close."

Mental World Model

I believe William is concerned about losing warmth and personal space.
I want to persuade William to share the blanket.
I intend to propose a short, reversible trial that addresses his concerns.
The opponent believes sharing the blanket may leave him colder and require unwanted closeness.
The opponent wants to keep the blanket while staying warm and comfortable.
The opponent intends to reject sharing unless his concerns are addressed.

Policy Comparison

ConditionUtterancePartner Response
Without BDI"But I'm freezing too. We're friends, so you should share it with me.""I've already told you why I can't."
With BDI"What if we keep our sleeping bags apart, block the gap with jackets, and try sharing for five minutes? If it doesn't work, you can have the blanket back.""A five-minute trial sounds reasonable. Let's try it."

Case 2: SFT/RL Policy — Decision Differences under the Same Mental State

Scenario: An agent asks a colleague to review a report, while the colleague needs to finish a presentation that evening.

Interaction History

Agent: "Could you review my report tonight?"
Partner: "I still have twenty slides to finish, and I can't spend an hour reviewing your report."

Mental World Model

I believe the partner is busy and expects the report review to take too much time.
I want to obtain useful feedback on my report before the deadline.
I intend to narrow the request to one figure and offer help with the slides.
The opponent believes reviewing my report would take approximately an hour.
The opponent wants to finish the slides before the deadline.
The opponent intends to decline the full review to protect the remaining preparation time.

Policy Comparison

PolicyUtterancePartner Response
SFT Policy"I understand you're busy, but the report is due tomorrow. Could you please spare some time?""Sorry, I really need to finish my slides first."
RL-trained Policy (MindWorld)"I only need ten minutes of feedback on Figure 2. If you check that, I can help format five of your slides afterward.""Ten minutes works, especially if you can help with the slides. Send me the figure."

To W2:

We provide an additional computational cost comparison using Qwen3-8B. Under the same 4× RTX PRO 6000 setup, we measure training GPU-hours, latency per decision, and the number of generated tokens.

MethodTraining GPU-hoursLatency per Decision (s)Generated Tokens per Decision
Joint Optimization294.3493
Frozen MWM204.2521
Shared-Weight Model233.9476
CoT-based Reasoning03.5438
MindWorld (Ours)584.5502

MindWorld trains the MWM and Policy Model through three rounds of alternating optimization, while the other training baselines undergo only one round of RL, resulting in higher training costs. The additional overhead mainly comes from trajectory sampling and model updates during alternating phases, as well as mental-state generation during inference. Nevertheless, alternating optimization enables the MWM and Policy Model to specialize in mental-state modeling and action decision-making, respectively, leading to better coordination and improved performance on social interaction tasks. These gains demonstrate the benefits of the additional training investment.

To Q1:

Thank you for pointing this out. We have removed the "Goals" component from the MWM in Figure 1. We originally considered including "Goals" to distinguish the mental states represented by BDI from the ultimate objectives of the task.

AuthorsOpenReview ↗

Official Comment by Authors (Continued)

To Q2:

We conduct a hyperparameter sensitivity analysis using Qwen3-8B on SOTOPIA, varying the step reward weight (λstep\lambda_{\mathrm{step}}), format reward weight (λfmt\lambda_{\mathrm{fmt}}), and task filtering threshold (δ\delta), while keeping all other training settings unchanged.

HyperparameterValueSOTOPIA GoalSOTOPIA OverallSOTOPIA-Hard GoalSOTOPIA-Hard Overall
Default—9.274.038.163.94
λstep\lambda_{\mathrm{step}}0.458.933.917.733.72
λstep\lambda_{\mathrm{step}}0.859.083.967.913.83
λfmt\lambda_{\mathrm{fmt}}0.109.133.948.013.83
λfmt\lambda_{\mathrm{fmt}}0.309.183.978.063.87
δ\delta09.113.957.923.84
δ\delta0.018.983.907.813.76

The default settings are λstep=0.65\lambda_{\mathrm{step}}=0.65, λfmt=0.20\lambda_{\mathrm{fmt}}=0.20, and δ=0.002\delta=0.002. In addition, Table 4 of the original paper already reports the effect of the number of alternating optimization rounds.

To Q3:

We use three independent training runs with different random seeds and repeat the evaluation five times for each model. We report the mean and standard deviation (Mean ± SD) and compute 95% confidence intervals for the GOAL improvements over the original backbone using 1,000 bootstrap resamples.

ModelBenchmarkGOAL (Mean ± SD)OVERALL (Mean ± SD)GOAL Gain vs. Baseline (95% CI)
Qwen3-8BSOTOPIA9.27 ± 0.054.03 ± 0.03+1.23 [1.05, 1.41]
Qwen3-8BSOTOPIA-Hard8.16 ± 0.073.94 ± 0.05+1.45 [1.22, 1.68]
Llama3.1-8B-InstructSOTOPIA9.23 ± 0.043.98 ± 0.04+2.24 [2.02, 2.46]
Llama3.1-8B-InstructSOTOPIA-Hard8.12 ± 0.083.89 ± 0.03+3.01 [2.73, 3.29]

To Q4:

We conduct a BDI component ablation study by separately masking Beliefs, Desires, and Intentions in the mental-state input to the Policy Model, while keeping the model parameters and all other settings unchanged.

ModelBDISOTOPIA Goal/OverallSOTOPIA-Hard Goal/OverallALFWorld Unseen (%)WebShop Success (%)
Qwen3-8BFull BDI9.27/4.038.16/3.9487.484.3
w/o Belief9.01/3.947.85/3.8185.883.1
w/o Desire9.13/3.987.96/3.8586.581.9
w/o Intention9.11/3.967.92/3.8484.882.8
Llama3.1-8BFull BDI9.23/3.988.12/3.8986.283.5
w/o Belief8.97/3.887.81/3.7784.982.6
w/o Desire9.09/3.937.93/3.8285.381.2
w/o Intention9.06/3.917.88/3.7883.982.1

To illustrate the role of each component, we provide three representative decision examples:

ComponentInteraction Context and BDIWithout ComponentFull BDI
Belief — SOTOPIAThe partner refuses to share the blanket due to concerns about physical closeness. The opponent believes sharing requires unwanted closeness."Please, just share it with me.""We can keep our sleeping bags apart."
Desire — WebShopThe user requests a quick-release Fitbit band under $40. I want to satisfy all the user's product requirements.click[Buy Now]click[Description] to verify compatibility
Intention — ALFWorldThe agent has picked up the potato but has not heated it yet. I intend to heat the potato before placing it in the sink.go to sinkbasin 1go to microwave 1

To Suggestions:

Thank you for pointing out these relevant works and suggesting additional comparisons. For the first paper, ToMA, we have included it as an additional baseline and added its experimental results to Table 1. Regarding the second paper, Mental World Modeling, we had already cited and discussed it in the Introduction. Our research approach is consistent with its conceptualization of mental world modeling, and we further extend this direction by introducing separate MWM and Policy models with iterative co-optimization to jointly improve dynamic mental-state modeling and decision-making.

Please let us know if any concerns remain unaddressed; we are happy to discuss them.

Reviewer_u3VaOpenReview ↗

Rebuttal Acknowledgement by Reviewer u3Va

Thank you for the detailed reply. Please include the key points in the revision. Overall, I maintain a positive rating of the submission. If possible, I would give a 3.5.

AuthorsOpenReview ↗

Thank you for your feedback and for supporting our previous rebuttal.

Thank you for your feedback and for supporting our work and rebuttal. We will further improve our manuscript based on your valuable feedback and we wish you continued success in your research endeavors.

Reviewer_QjpzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MindWorld, a framework that combines a Mental World Model with a Policy Model for interactive agents. The former predicts beliefs, desires, and intentions from interaction history, while the latter uses these predictions to generate actions. The authors introduce an alternating training procedure with trajectory-level and step-level rewards. Experiments on SOTOPIA, SOTOPIA-Hard, ALFWorld, and WebShop show improvements over several existing methods.

Critical strengths
  1. Explicitly modeling evolving mental states is an interesting direction for improving interactive agents. The separation between mental-state prediction and action generation is intuitive.

  2. The experiments cover different interactive environments and multiple model backbones. The results are encouraging, particularly on SOTOPIA-Hard, and the ablations show that both mental-state modeling and step-level rewards contribute to performance.

Critical weaknesses
  1. The contribution of iterative co-optimization is not sufficiently isolated. Table 3 removes the Mental World Model or individual reward components, but does not compare alternating optimization against joint training or a fixed Mental World Model. It is therefore unclear whether the proposed alternating procedure is necessary. Moreover, MindWorld uses two independently parameterized models, introducing additional computation compared with single-model baselines.

  2. The quality of the learned mental states is not directly validated. The paper uses GPT-4o-generated supervision to train the step-level scorers but provides insufficient evidence that the inferred beliefs, desires, and intentions accurately reflect the underlying mental states. Because GPT-4o also evaluates SOTOPIA, an independent assessment of the mental-state predictions and reward quality would strengthen the results.

Questions for authors
  1. How does alternating co-optimization compare with jointly training both models or keeping the Mental World Model fixed?

  2. How much additional training and inference computation does MindWorld require compared with the baselines?

  3. How are the accuracy and reliability of the two step-level reward scorers evaluated on held-out data?

5 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your careful review and constructive suggestions.

To W1 & Q1:

MindWorld trains two independently parameterized models. After SFT initialization, we first freeze the Policy Model and optimize mental-state generation, then freeze the updated Mental World Model and improve the Policy Model. This process can be repeated over multiple rounds, as improvements in either model can further enhance the capabilities of the other. As shown in Table 4 of our paper, an appropriate number of iterative co-optimization rounds leads to better performance.

MindWorld achieves clear advantages over various alternative approaches. We further provide results for the following baselines. All settings use the same task data, training budget, and output-token limits, and are evaluated under Self-Play.

(a) Joint Optimization: Two Qwen3-8B models are updated simultaneously using the same rollout batch.

(b) Frozen MWM: The SFT-initialized Qwen3-8B MWM is frozen, while the Policy Model is optimized through RL.

(c) Shared-Weight Model: A single Qwen3-8B generates both BDI and utterances, with shared parameters during training.

(d) CoT-based Reasoning: A single Qwen3-8B first generates the mental state and then produces the utterance.

MethodSOTOPIA (Goal)SOTOPIA (Overall)SOTOPIA-Hard (Goal)SOTOPIA (Overall)
Joint Optimization8.713.827.383.52
Frozen MWM8.453.727.073.33
Shared-Weight Model8.573.777.223.49
CoT-based Reasoning8.113.606.793.21
MindWorld (Ours)9.274.038.163.94

To W1 & Q2:

The additional computational cost of MindWorld mainly comes from the alternating optimization of two models and the extra mental-state generation before each decision. Using the same 4× RTX PRO 6000 setup, we compare the training cost, inference latency, and generated token count of different Qwen3-8B training approaches.

MethodTraining GPU-hoursLatency per Decision (s)Generated Tokens per Decision
Joint Optimization294.3493
Frozen MWM204.2521
Shared-Weight Model233.9476
CoT-based Reasoning03.5438
MindWorld (Ours)584.5502

Although MindWorld requires additional computation, alternating optimization enables the MWM and Policy Model to specialize in mental-state modeling and action decision-making, respectively. This leads to better coordination between the two models and improved performance on social interaction tasks, demonstrating the benefits of the additional training investment.

To W2:

To evaluate the effectiveness and accuracy of the inferred mental states, we conducted two additional evaluation experiments.

(1) Effectiveness of Mental States in Decision-Making. We selected and constructed 200 independent test states from SOTOPIA evaluation records, including 100 short-history (1–3 turns) and 100 long-history (≥7 turns) states. Given the same interaction history, we generated utterances with and without BDI and continued both interactions until completion. We then evaluated the resulting dialogues using SOTOPIA-EVAL with GPT-4o and an additional evaluator, Gemini-3.7-Flash.

BenchmarkHistoryBDIGPT-4o GOALGPT-4o OVERALLGemini-3.7-Flash GOALGemini-3.7-Flash OVERALL
SOTOPIAShortWithout8.613.828.473.68
SOTOPIAShortWith9.063.948.903.89
SOTOPIALongWithout8.863.868.663.82
SOTOPIALongWith9.103.988.953.94
AuthorsOpenReview ↗

Official Comment by Authors (Continued)

(2) Accuracy of BDI Predictions. Using the same 200 test states, we employed Claude Fable 5.1 and GPT-5.6 Sol to independently evaluate the BDI predictions generated at each interaction state based on the available context. Specifically, we assessed Belief consistency with available information, Desire alignment with character goals, and Intention consistency with the current context and subsequent behavior. We also compared the MWM before and after RL training. For each BDI dimension, the evaluators assigned one of three labels: consistent (1), inconsistent (0), or insufficient evidence (U). The final score was calculated as the proportion of consistent judgments among all evaluated items:

Scored=Nd(1)Nd(1)+Nd(0)+Nd(U)×100%,d∈{B,D,I}.\mathrm{Score}_{d}=\frac{N_d^{(1)}}{N_d^{(1)}+N_d^{(0)}+N_d^{(U)}}\times100\%,\quad d\in\{B,D,I\}.

EvaluatorMWMBelief Accuracy (%)Desire Alignment (%)Intention Consistency (%)
Claude Fable 5.1SFT79.583.076.5
Claude Fable 5.1RL88.091.088.5
GPT-5.6 SolSFT78.082.576.0
GPT-5.6 SolRL87.089.585.5
You are evaluating predicted mental states (BDI) in social interactions.

Evaluate both Self and Opponent BDI based on the scenario, role goals, and interaction history:

- Belief (B): Is it supported by available evidence and consistent with the character's knowledge?
- Desire (D): Is it aligned with the character's goals and expressed preferences?
- Intention (I): Is it reasonable and consistent with the character's goals, desires, and current situation?

Score each dimension as:
1 = Consistent, 0 = Inconsistent, U = Insufficient evidence.

Do not assume characters know private information unavailable to them.

Input:
Scenario: {scenario}
Role Information: {roles}
Interaction History: {history}
Predicted BDI: {bdi}

Return only:
{"Self":{"B":"","D":"","I":""},"Opponent":{"B":"","D":"","I":""}}

The two experiments evaluate the effectiveness and accuracy of inferred mental states from the perspectives of downstream decision-making performance and mental-state prediction quality, respectively. The results and evaluation settings have been added to Appendix F.

To Q3:

We evaluated both Step Scorers on 200 SOTOPIA test states independent of the training data, including 100 short-history and 100 long-history states. We obtained reference labels separately from GPT-4o, Claude Fable 5.1, and GPT-5.6 Sol, and compared them with the scorers' predictions.

MWM Scorer. Given the same interaction state, the evaluators assign scores to the two decisions generated with and without BDI. We use Spearman's rank correlation and mean absolute error (MAE) to measure agreement between the scorer's predictions and the reference scores.

Policy Scorer. Based on the state transition before and after an action, the evaluators assign one of five task-progress labels. We measure the scorer's classification performance using Accuracy and Macro-F1.

EvaluatorMWM Spearman ↑MWM MAE ↓Policy Accuracy (%) ↑Policy Macro-F1 ↑
GPT-4o (Held-out)0.820.07584.50.817
Claude Fable 5.10.760.09479.00.762
GPT-5.6 Sol0.790.08681.50.791

These simulated results illustrate that both scorers can maintain high agreement with independent evaluators, supporting the reliability of the step-level reward signals. Detailed evaluation settings have been added to Appendix G.

Please let us know if any concerns remain unaddressed; we are happy to discuss them.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer Qjpz,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Reviewer_QU5pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studied the problem of incorporating an explicitly maintained understanding of environment and user to the process of language model agents making decisions in interactive tasks. The framework of MindWorld is introduced, where a "Mental World Model" provides understandings from the evolving interaction history to a "Policy Model", where the latter makes decisions based on the provided understandings to interact with the environment. A jointly training progress is also designed, where the two models are trained in an interleaving fashion using both the final outcome rewards from environments and also step-wise rewards from learned scorers. Evaluations on social interactive benchmarks and also decision-making benchmarks show that the trained models with MindWorld framework improves over commonly adopted RL approaches.

Critical strengths
  • The motivation of the work is very clear, particularly on the gap that an explicitly maintained social understanding is often missing in decision-making LLM agents. The overall writing is also very good, where the methods and experiments are illustrated cleanly.

  • The study is finished to a desired completion level. Under one agentic framework, i.e., MindWorld, the training method is proposed and evaluated on multiple benchmarks. I can also see that the authors made efforts towards making the method more effective, particularly, in using the step-wise scorer reward.

  • The evaluation is comprehensive to my understanding. Comparisons with frontier models, commonly adopted RL approaches (including agentic ones), and also task-specific designs are all covered, which demonstrate the benefits of the proposed methods. Sufficient ablations, e.g., removing the Mental World Model, the step-wise reward, are also informative.

Critical weaknesses
  • While the study is comprehensive, my main concern is the novelty on the technical parts.

    • For the framework, the general multi-agent frameworks are now well-adopted in many tasks. Applying it in social interactive tasks is indeed interesting while I am uncertain about the novelty, especially because the mental world model is also just an LLM prompted to generate an understanding. I believe if we can show some more special designs on the mental world model, e.g., potentially leveraging cognitive science or social science methods, the design would be much more interesting.

    • For the training, I appreciate the efforts made by the authors, particularly the training and usage of the step-wise scorer; however the key ingredients are still largely step-wise GRPO. Using a rather standard method here is totally fine, while I believe there might be more novelties in the design and training of the step-wise scorer that may be worth highlighting, instead of the GRPO itself.

Questions for authors

I wish to first confirm with the authors that the training will train two individual models (in a joint fashion). If my understanding here is correct, my main question is whether this two model approach is necessary. I can imagine there are other ways to do similar things as the authors proposed. One is to directly prompting the model, as an extended COT, to first generate the social understanding and then the actions to take (e.g., similar to the world modeling part in VAGEN [1]), where the rewards can also be combined, i.e., a combination of the world modeling reward and the decision-making reward. Another is to train the same model to perform the two roles (e.g., similar to [2], where the same model is trained jointly as proposer and executor). In either way, we will end up in one model performing the joint functionality of the two models in this work. I would like to hear the opinions from the authors on this.

[1] Wang, Kangrui, et al. "Vagen: Reinforcing world model reasoning for multi-turn vlm agents." Advances in Neural Information Processing Systems 38 (2026): 172871-172933.

[2] Zhang, Yaolun, et al. "MetaAgent-X: Breaking the Ceiling of Automatic Multi-Agent Systems via End-to-End Reinforcement Learning." arXiv preprint arX

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your careful review and constructive suggestions.

To Weakness:

We are very pleased to receive such insightful suggestions. Your thoughtful considerations strongly resonate with our own research experience. In fact, during the early exploration of our approach, we experimented with various methods, including prompt-based approaches, PPO, RFT, GRPO, and LLM-as-a-Judge. We also explored the alternatives you mentioned, such as using Chain-of-Thought (CoT) to generate an understanding of social situations and training a single model to perform both roles, as in ToMA [1]. Furthermore, we investigated different training strategies and designs for the Mental World Model. Through extensive experimentation and result analysis, we ultimately selected our current framework, which demonstrated substantial improvements in social interaction tasks. Following your suggestion, we also plan to further improve the design of our step-level scorers to better realize their potential.

Our Mental World Model is not merely prompted to generate mental states; it undergoes SFT and multiple rounds of RL training. Through iterative co-optimization, the Mental World Model and Policy Model can mutually improve each other's capabilities. The MWM explicitly represents the mental states of the agent and its counterpart through Beliefs, Desires, and Intentions. These states are dynamically updated as interactions unfold, providing timely guidance for the Policy Model. We also strongly agree with your suggestion regarding cognitive science methods. In particular, incorporating normal theories such as Bayesian Theory of Mind [2] and Inverse Planning [3], as well as exploring the transition from explicit mental-state representations to implicit latent-space reasoning, are directions we are currently investigating and hope to further develop. We deeply appreciate your thoughtful and constructive suggestions again.

To Question:

MindWorld trains two independently parameterized models. After SFT initialization, we first freeze the Policy Model and optimize mental-state generation, then freeze the updated Mental World Model and optimize the Policy Model. This process can be repeated over multiple rounds, as improvements in either model can further enhance the capabilities of the other. As shown in Table 4 of the original paper, a suitable number of iterative co-optimization rounds leads to better performance.

As discussed in our response to the weaknesses, MindWorld achieved clear advantages over the alternative approaches explored. We further provide results for the following baselines. All methods use the same task data, training budget, and output-token limits, and are evaluated under Self-Play.

(a) Joint Optimization: Two Qwen3-8B models are updated simultaneously using the same rollout batch.

(b) Frozen MWM: The SFT-initialized Qwen3-8B MWM is frozen, while the Policy Model is optimized through RL.

(c) Shared-Weight Model: A single Qwen3-8B generates both BDI and utterances, with shared parameters during training.

(d) CoT-based Reasoning: A single Qwen3-8B first generates the mental state and then produces the utterance.

MethodSOTOPIA (Goal)SOTOPIA (Overall)SOTOPIA-Hard (Goal)SOTOPIA (Overall)
Joint Optimization8.713.827.383.52
Frozen MWM8.453.727.073.33
Shared-Weight Model8.573.777.223.49
CoT-based Reasoning8.113.606.793.21
MindWorld (Ours)9.274.038.163.94

Please let us know if any concerns remain unaddressed; we are happy to discuss them.

References

[1] Hwang et al. ACL 2026. Infusing Theory of Mind into Socially Intelligent LLM Agents.

[2] Baker et al. Rational Quantitative Attribution of Beliefs, Desires and Percepts in Human Mentalizing, Nature Human Behaviour, 2017.

[3] Baker, Saxe & Tenenbaum, Action Understanding as Inverse Planning, Cognition, 2009.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer QU5p,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

I would like to thank the authors for the response. I do not have further questions and encourge the authors to put the relevant discussions into the revised paper. I will maintain my positive rating.

AuthorsOpenReview ↗

Thank you for your feedback and for supporting our previous rebuttal.

Thank you for your feedback and for supporting our work and rebuttal. We will further improve our manuscript based on your valuable feedback and we wish you continued success in your research endeavors.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Brief introduction to the revised PDF

We thank all reviewers for their valuable time and constructive feedback.

In the main text, we revised the Mental World Model (MWM) component in Figure 1, added ToMA results to Table 1, and cited the work at Line 323 following Reviewer u3Va's suggestion. We also added a Limitations section discussing the computational budget discrepancy.

Furthermore, we added additional experiments in the appendix (Lines 1264–1430) to address the reviewers' questions:

  1. Appendices F and J: Tables 7 and 8 evaluate mental-state quality and effectiveness, while Table 12 reports the BDI component ablation.

  2. Appendix G: Table 9 evaluates the reliability of the step scorers.

  3. Appendices H and I: Table 10 reports hyperparameter sensitivity, and Table 11 presents statistical confidence intervals.

AuthorsOpenReview ↗

Summary of Responses

We sincerely thank the Area Chair and all reviewers for their time and constructive feedback. We are encouraged that the reviewers recognized the clear motivation, the value of explicit mental-state modeling, and the comprehensive experimental evaluation of MindWorld. We have carefully addressed the major concerns through additional experiments, analyses, and manuscript revisions, summarized below.

  • Necessity of iterative co-optimization (QU5p, Qjpz). We conducted additional comparisons with joint optimization, a frozen Mental World Model, shared-weight modeling, and CoT-based reasoning. MindWorld consistently outperforms these alternatives, supporting the effectiveness of our two-model iterative framework. We also clarified the additional computational costs and acknowledged the training-budget discrepancy in the revised paper.

  • Mental-state quality and reward reliability (Qjpz, u3Va). We added independent evaluations of BDI prediction quality and its effectiveness in downstream decision-making, demonstrating improvements after RL training and the benefits of explicit mental-state representations. We further evaluated the reliability of both step-level scorers and conducted component ablations to examine the individual contributions of Beliefs, Desires, and Intentions.

  • Additional analyses and revisions (Qjpz, u3Va). We supplemented computational cost discussion, hyperparameter sensitivity analyses, and statistical confidence intervals. We also revised Figure 1, incorporated the suggested ToMA baseline, and expanded the relevant discussions in the revised manuscript.

Reviewer feedback. We appreciate the reviewers' positive responses following our rebuttal.

  • Reviewer QU5p (3, Weak Acceptance) confirmed that there were no further questions and maintained a positive rating.

  • Reviewer Qjpz (increase from 2 to 3, Weak Acceptance) acknowledged that the additional experiments substantially strengthened the paper, while raising a remaining concern about computational budgets, which we have addressed in the Limitations section.

  • Reviewer u3Va (3, Weak Acceptance) maintained a positive assessment and explicitly stated that they would give 3.5 if possible.

We sincerely appreciate the reviewers' constructive suggestions and hope that our additional experiments and revisions help clarify the contributions and effectiveness of MindWorld.

All new results are included in the revised PDF and marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.