- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes MindWorld, an LLM-based world model that tracks mental states (including beliefs, desires, and intentions) from conversational interactions and trains LLM agents using reinforcement learning with novel reward designs.
Critical strengths
- The proposed method is well motivated and elaborated.
- Comprehensive experimental results across multiple benchmarks demonstrate the effectiveness of the proposed method on goal-oriented interactions & decision-making tasks.
- Ablation studies and analysis of the training dynamics provide further support for the strengths of the proposed method.
Critical weaknesses
- Lack of in-depth case studies (although some cases are provided in the appendix) to investigate how LLM agents change their behavior using the proposed method.
- Lack of efficiency studies to examine the training costs introduced by the proposed method.
- Some questions and concerns, as in "Questions For Authors".
Questions for authors
Questions:
- Figure 1 MWM: As intention is commonly regarded as a goal-oriented mental state, "Goals" in the figure looks inappropriate.
- Experiments: How do the hyperparameter configurations during training affect performance?
- Results: How many runs were there for each reported score? What is the statistical significance and confidence?
- Is there any analysis and insights on the individual effect of Beliefs, Desires, and Intentions for different tasks?
Suggestions:
- A missing reference and potential baseline for comparison [1], which trains LLM agents to infer mental states and then complete the conversation on the Sotopia benchmark. In addition, there is concurrent work on MWM [2].
- [1] Hwang et al. ACL 2026. Infusing Theory of Mind into Socially Intelligent LLM Agents.
- [2] Fei and Zhao. arXiv 2026. Mental World Modeling.
4 comments
Official Comment by Authors (Continued)
To Q2:
We conduct a hyperparameter sensitivity analysis using Qwen3-8B on SOTOPIA, varying the step reward weight (), format reward weight (), and task filtering threshold (), while keeping all other training settings unchanged.
| Hyperparameter | Value | SOTOPIA Goal | SOTOPIA Overall | SOTOPIA-Hard Goal | SOTOPIA-Hard Overall |
|---|---|---|---|---|---|
| Default | — | 9.27 | 4.03 | 8.16 | 3.94 |
| 0.45 | 8.93 | 3.91 | 7.73 | 3.72 | |
| 0.85 | 9.08 | 3.96 | 7.91 | 3.83 | |
| 0.10 | 9.13 | 3.94 | 8.01 | 3.83 | |
| 0.30 | 9.18 | 3.97 | 8.06 | 3.87 | |
| 0 | 9.11 | 3.95 | 7.92 | 3.84 | |
| 0.01 | 8.98 | 3.90 | 7.81 | 3.76 |
The default settings are , , and . In addition, Table 4 of the original paper already reports the effect of the number of alternating optimization rounds.
To Q3:
We use three independent training runs with different random seeds and repeat the evaluation five times for each model. We report the mean and standard deviation (Mean ± SD) and compute 95% confidence intervals for the GOAL improvements over the original backbone using 1,000 bootstrap resamples.
| Model | Benchmark | GOAL (Mean ± SD) | OVERALL (Mean ± SD) | GOAL Gain vs. Baseline (95% CI) |
|---|---|---|---|---|
| Qwen3-8B | SOTOPIA | 9.27 ± 0.05 | 4.03 ± 0.03 | +1.23 [1.05, 1.41] |
| Qwen3-8B | SOTOPIA-Hard | 8.16 ± 0.07 | 3.94 ± 0.05 | +1.45 [1.22, 1.68] |
| Llama3.1-8B-Instruct | SOTOPIA | 9.23 ± 0.04 | 3.98 ± 0.04 | +2.24 [2.02, 2.46] |
| Llama3.1-8B-Instruct | SOTOPIA-Hard | 8.12 ± 0.08 | 3.89 ± 0.03 | +3.01 [2.73, 3.29] |
To Q4:
We conduct a BDI component ablation study by separately masking Beliefs, Desires, and Intentions in the mental-state input to the Policy Model, while keeping the model parameters and all other settings unchanged.
| Model | BDI | SOTOPIA Goal/Overall | SOTOPIA-Hard Goal/Overall | ALFWorld Unseen (%) | WebShop Success (%) |
|---|---|---|---|---|---|
| Qwen3-8B | Full BDI | 9.27/4.03 | 8.16/3.94 | 87.4 | 84.3 |
| w/o Belief | 9.01/3.94 | 7.85/3.81 | 85.8 | 83.1 | |
| w/o Desire | 9.13/3.98 | 7.96/3.85 | 86.5 | 81.9 | |
| w/o Intention | 9.11/3.96 | 7.92/3.84 | 84.8 | 82.8 | |
| Llama3.1-8B | Full BDI | 9.23/3.98 | 8.12/3.89 | 86.2 | 83.5 |
| w/o Belief | 8.97/3.88 | 7.81/3.77 | 84.9 | 82.6 | |
| w/o Desire | 9.09/3.93 | 7.93/3.82 | 85.3 | 81.2 | |
| w/o Intention | 9.06/3.91 | 7.88/3.78 | 83.9 | 82.1 |
To illustrate the role of each component, we provide three representative decision examples:
| Component | Interaction Context and BDI | Without Component | Full BDI |
|---|---|---|---|
| Belief — SOTOPIA | The partner refuses to share the blanket due to concerns about physical closeness. The opponent believes sharing requires unwanted closeness. | "Please, just share it with me." | "We can keep our sleeping bags apart." |
| Desire — WebShop | The user requests a quick-release Fitbit band under $40. I want to satisfy all the user's product requirements. | click[Buy Now] | click[Description] to verify compatibility |
| Intention — ALFWorld | The agent has picked up the potato but has not heated it yet. I intend to heat the potato before placing it in the sink. | go to sinkbasin 1 | go to microwave 1 |
To Suggestions:
Thank you for pointing out these relevant works and suggesting additional comparisons. For the first paper, ToMA, we have included it as an additional baseline and added its experimental results to Table 1. Regarding the second paper, Mental World Modeling, we had already cited and discussed it in the Introduction. Our research approach is consistent with its conceptualization of mental world modeling, and we further extend this direction by introducing separate MWM and Policy models with iterative co-optimization to jointly improve dynamic mental-state modeling and decision-making.
Please let us know if any concerns remain unaddressed; we are happy to discuss them.
Rebuttal Acknowledgement by Reviewer u3Va
Thank you for the detailed reply. Please include the key points in the revision. Overall, I maintain a positive rating of the submission. If possible, I would give a 3.5.
Thank you for your feedback and for supporting our previous rebuttal.
Thank you for your feedback and for supporting our work and rebuttal. We will further improve our manuscript based on your valuable feedback and we wish you continued success in your research endeavors.
Official Comment by Authors
Thank you for your careful review and constructive suggestions.
To W1:
We provide two additional, more detailed SOTOPIA interaction case studies.
Case 1: With/Without BDI
Scenario: Mia wants William to share the only blanket during a camping trip, while William wants to keep it for himself.
Interaction History
Mental World Model
I believe William is concerned about losing warmth and personal space.
I want to persuade William to share the blanket.
I intend to propose a short, reversible trial that addresses his concerns.
The opponent believes sharing the blanket may leave him colder and require unwanted closeness.
The opponent wants to keep the blanket while staying warm and comfortable.
The opponent intends to reject sharing unless his concerns are addressed.
Policy Comparison
Case 2: SFT/RL Policy — Decision Differences under the Same Mental State
Scenario: An agent asks a colleague to review a report, while the colleague needs to finish a presentation that evening.
Interaction History
Mental World Model
I believe the partner is busy and expects the report review to take too much time.
I want to obtain useful feedback on my report before the deadline.
I intend to narrow the request to one figure and offer help with the slides.
The opponent believes reviewing my report would take approximately an hour.
The opponent wants to finish the slides before the deadline.
The opponent intends to decline the full review to protect the remaining preparation time.
Policy Comparison
To W2:
We provide an additional computational cost comparison using Qwen3-8B. Under the same 4× RTX PRO 6000 setup, we measure training GPU-hours, latency per decision, and the number of generated tokens.
MindWorld trains the MWM and Policy Model through three rounds of alternating optimization, while the other training baselines undergo only one round of RL, resulting in higher training costs. The additional overhead mainly comes from trajectory sampling and model updates during alternating phases, as well as mental-state generation during inference. Nevertheless, alternating optimization enables the MWM and Policy Model to specialize in mental-state modeling and action decision-making, respectively, leading to better coordination and improved performance on social interaction tasks. These gains demonstrate the benefits of the additional training investment.
To Q1:
Thank you for pointing this out. We have removed the "Goals" component from the MWM in Figure 1. We originally considered including "Goals" to distinguish the mental states represented by BDI from the ultimate objectives of the task.