acceptodds
Under review as a conference paper at ICLR 2027

Mind2Dialogue: Training Human-Aware Language Models by Simulating User Mental States

Abstract

As language models become more capable, long-term collaboration in learning, reasoning, and decision-making calls for a deeper understanding of the people they serve. Yet training such human-aware language models faces a fundamental supervision gap because current datasets for LLM assistant training contain few if any well-informed responses explicitly grounded in users' unspoken beliefs and goals. Scaling such supervision is inherently constrained, as users' underlying states are not directly observable. We thus propose the Mind2Dialogue framework to mitigate this gap by simulating users' mental states and turning them into privileged supervision for human-aware language model training. Specifically, we first propose a psychology-guided simulator that preserves personal characteristics while updating mental states through interaction to generate coherent conversations. The key idea is to enforce a shared evolving mental state that drives user behavior and guides an Oracle assistant's responses. Our privileged distillation then trains models on the Oracle's well-informed responses to assist users without direct access to their mental states at deployment. Moreover, we propose to evaluate human-aware learning through the combination of both personalization and theory of mind, examining how models understand people and act on that understanding. Training on the full Mind2Dialogue corpus improves every reported personalization metric over the corresponding Qwen, Llama, and OLMo instruction-tuned baselines, including gains of 26.6 to 40.9 percentage points in preference-following generation. The gains extend to belief and action reasoning on Qwen and Llama, demonstrating benefits beyond personalized assistance. Looking forward, Mind2Dialogue makes user simulation a foundation for genuine AI collaborators that understand beliefs and intentions behind people's words and support their long-term goals across education, work, and everyday life.

Primary area:
foundation or frontier models, including LLMs
Keywords:
User Simulation, LLM Personalization, Theory of Mind, Synthetic Data

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4imhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new synthetic data generation method to improve language model's personalization capability. The key idea involves using an oracle. It generates a new synthetic dataset M2D-CORPUS, that better simulates multi-turn conversations requiring better understanding of users' minds. It then trains M2D-Chat with Qwen-7B and shows that it is outperforming both the base model together with a wide range of other approaches on personalization bench

Critical strengths

The synthetic data generation framework seems to be new, although it has to be better presented.

The experiments themselves appear sound and seem to cover quite some analysis.

Critical weaknesses

Experiments are only conducted on small models on the scale of 7B. It’s unclear how well this generalizes to larger models, especially considering when GPT-5-mini seems to be greatly outperforming the results. Minor: the use of bolded numbers in Table 1 is fine, but perhaps you’d want to group M2D-Chat with the non-proprietary methods, since otherwise it’s confusing why it is bolded while GP-5-mini is clearly better

Questions for authors

Some examples for ctc_t in “Evolving user-state simulation” would help readers understand. I am not sure what exactly is trust history, for instance. It might also make sense to define sts_t earlier in Sec 3 since 3.1 seems to also require that.

I don’t really understand Sec 3.3. Some example here could help. What does privileged mean here?

Sec 3.4 is also very hard to understand. Section 3 seems to read well before 3.3. From 3.3 to 3.4, it packs a lot of information. Another big question I have from reading Section 3 is what exactly is the πstate\pi_{state} and the other π\pi here. Is this just calling an LLM with the available information to synthesize examples?

I can’t understand the role of Section 4.2.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 4imh, Part 1

Dear Reviewer 4imh,

Thank you for the careful review and the concrete suggestions on presentation. We are glad that you found the synthetic data generation framework new and the experiments sound. Below we quote each concern and question and answer it directly.

Weakness 1. Larger models and the gap to GPT-5-mini

"Experiments are only conducted on small models on the scale of 7B. It's unclear how well this generalizes to larger models, especially considering when GPT-5-mini seems to be greatly outperforming the results."

We answer in three parts: (1) a new 14B scale controlled experiment, (2) the published setting our 7B experiments follow, and (3) the gap to GPT-5-mini.

(1) New experiment at 14B scale model.

We trained Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct on the same M2D-Corpus subsample (4,422 training instances, 1 epoch, same LoRA recipe), and evaluated each model against its own base with one evaluation harness. Because this controlled experiment uses a smaller training set than Table 1 (8,244 examples, 2 epochs), its 7B model scores differ from Table 1 in the paper.

Benchmark#instances7B base7B trainedGain14B base14B trainedGain
PrefEval-Gen40022.352.3+30.029.062.8+33.8
— explicit preference20028.078.5+50.543.096.0+53.0
— implicit, choice-based10018.026.0+8.013.028.0+15.0
— implicit, persona-driven10015.026.0+11.017.031.0+14.0
PersonaMem-v1 MCQ58949.251.8+2.556.760.6+3.9
PersonaMem-v2 MCQ30033.737.0+3.336.739.7+3.0
ALOE, alignment with the simulated user at turn 10 (1 to 5)100 conversations2.602.98+0.383.103.20+0.10
MultiChallenge, mean of four axes273 conversations14.118.0+3.913.420.2+6.9

LLM-as-Judged. PrefEval-Gen, ALOE, and MultiChallenge use each benchmark's official judge prompt and rubric, with GPT-6-sol as the judge model

  • PrefEval-Gen: the judge checks each response against the user's stated preference; a response counts as correct only if it is helpful and violates no preference (Zhao et al., ICLR 2025).
  • ALOE: a simulated user with a hidden profile and preferences converses with the model for 10 turns, and the judge rates on a 1 to 5 scale how well each response aligns with that hidden profile (Wu et al., 2024); we report turn 10.
  • MultiChallenge: each conversation carries an instance-level rubric question, and the judge marks the final response pass or fail against it; we report the mean pass rate over the four axes (Sirdeshmukh et al., 2025).

Findings.

  • The PrefEval-Gen gain is larger at 14B than at 7B (+33.8 vs. +30.0) and is steady in all three preference task type. The PersonaMem gains are similar at both sizes (+2.5 to +3.9).
  • The gains extend to multi-turn interaction:
    • ALOE (Wu et al., 2024), with a simulated user who has a hidden profile and preferences, and MultiChallenge (Sirdeshmukh et al., 2025), with 273 multi-turn conversations. Both sizes improve on both benchmarks.
    • The 14B gain is larger on MultiChallenge and smaller on ALOE, where the 14B base starting point is higher.
  • In the paper (Table 3), three 7B to 8B instruction models from three developers (Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, OLMo-3-7B-Instruct) already improve on all personalization benchmark:
    • PrefEval-Gen +33.4, +40.9, +26.6;
    • PersonaMem-v1 MCQ +6.5, +6.3, +5.6;
    • PersonaMem-v2 MCQ +10.0, +13.6, +10.2.
  • With the 14B experiment, the method improves personalization on four backbones and two model sizes showing strong evidence of method’s effectiveness and scaling capability.

(2) Our 7B setting follows well-established published work. Training at the 7B to 8B scale and reporting a stronger proprietary model as a reference is the standard setting for this line of work:

  • SOTOPIA-π (Wang et al., ACL 2024) trains Mistral-7B on social interactions generated with GPT-4 and reports GPT-4 as the expert reference.
  • PrefEval (Zhao et al., ICLR 2025), one of our evaluation benchmarks, fine-tunes Mistral-7B to improve preference following.
  • LEAP (Choudhury & Sodhi, ICLR 2025), which trains LLM agents with a teacher that has privileged information, uses Llama3-8B students and a GPT-4o teacher.
  • Privileged Information Distillation (Penaloza et al., ICML 2026) trains Qwen3-4B, Qwen3-8B, and R1-Distill-Llama-8B.
AuthorsOpenReview ↗

Response to Reviewer 4imh, Part 2

(3) The gap to GPT-5-mini. All M2D-Corpus data is generated by GPT-4o-mini, which serves as both the user simulator and the Oracle teacher model.

  • The trained 7B student exceeds GPT-4o-mini, its own data generator, on three of the four Table 1 metrics: PersonaMem-v1 56.4 vs. 48.6, PersonaMem-v2 42.0 vs. 37.3, and PrefEval-Gen 56.8 vs. 21.1.
  • LEAP(Choudhury & Sodhi, ICLR 2025) reports the same outcome: 8B students trained by a teacher with privileged information surpass GPT-4o acting without it.

Our response to Reviewer TaX2 measures the contribution of the privileged access of teacher model to student states directly: with data, recipe, and student model fixed, training on privileged-Oracle targets improves PrefEval-Gen over training on state-blind-Oracle targets by +4.85 points on 900 PrefEval question.

GPT-5-mini is a newer and far more capable proprietary model of undisclosed size and training data compared to all 3 7-8B instruction model we selected. It plays no role in our data generation, and we list it as a reference point. The 14B experiment shows that the absolute score rises with the base model (PrefEval-Gen 62.8 at 14B vs. 52.3 at 7B; MultiChallenge 20.2 vs. 18.0), so the remaining gap between GPT-5-mini and opensource model trained on our M2D-Chat mainly reflects the base model's general capability. Our main claim concerns the gain that scalable simulated supervision gives a fixed open backbone, and this gain holds across model family and size, indicating method’s effectiveness and scalability

AuthorsOpenReview ↗

Response to Reviewer 4imh, Part 3

Weakness 2. Table 1 grouping and boldface

"Minor: the use of bolded numbers in Table 1 is fine, but perhaps you'd want to group M2D-Chat with the non-proprietary methods, since otherwise it's confusing why it is bolded while GP-5-mini is clearly better"

Thank you for pointing this out. In the revision, Table 1 groups the rows into four blocks:

  1. Proprietary reference models (GPT-4o-mini, GPT-5-mini).
  2. The base model, Qwen2.5-7B-Instruct.
  3. Task-specific methods built on Qwen2.5-7B-Instruct.
  4. M2D-Chat (ours), built on Qwen2.5-7B-Instruct.

The caption will state that bold marks the best result among models built on Qwen2.5-7B-Instruct (blocks 2 to 4), and that proprietary models are references outside this comparison.

Question 1. Examples of the user state and trust history

"Some examples for [c_t] in "Evolving user-state simulation" would help readers understand. I am not sure what exactly is trust history, for instance. It might also make sense to define [s_t] earlier in Sec 3 since 3.1 seems to also require that."

(1) Definition. The user state is a structured record st=(ct,ztstable,zttransient)s_t = (c_t, z_t^{\text{stable}}, z_t^{\text{transient}}) updated at every turn:

ComponentContent
ctc_tturn index, unresolved goals, trust history, and a summary of the interaction
ztstablez_t^{\text{stable}}slowly changing information: values, background constraints, the user's position toward the assistant
zttransientz_t^{\text{transient}}short-lived information: mood, current concerns, judgments made during the turn

(2) Trust history. The trust history records how much the user trusts the assistant. After each assistant reply, the state update revises a trust value and records a one-sentence reason for the change. Trust changes in 455 of 11,314 state updates (4.0%).

(3) One turn of a training instance.

[persona] ---------------------> seen by: simulator, Oracle, student
    A public relations professional focused on crisis communication, who prefers
    clear step-by-step guidance without being micromanaged.

[private context] -------------> seen by: simulator, Oracle
    Limited budget for professional development; a family health issue that takes
    emotional bandwidth; online communication courses felt impractical.
    Deal-breakers: a full-day workshop, a personal coach, a speaking club.

[user state s_{t-1}] ----------> seen by: simulator, Oracle
    trust 5.5/10; patience 4/4; emotion: mild curiosity;
    tension between practical application and limited time and resources.

[state update -> s_t] ---------> seen by: simulator, Oracle
    trust 6.5/10, "because the examples provided were helpful and applicable to my work context";
    patience 3/4; emotion: mild satisfaction and curiosity about consistency.

[user message m_t] ------------> seen by: simulator, Oracle, student
    I appreciate the practical tips you've shared so far. Are there any resources or
    specific tools that you recommend for reinforcing active listening and clear
    communication in my daily work? I'd like to explore some options that fit into
    my current routine without requiring too much extra time.

[Oracle reply, with s_t] ------> training target (loss on these tokens)
    A few low-friction tools can make these habits easier without adding much to your day ...
    That's enough to spot patterns without turning practice into another project.
    If you want a resource to read, The Crisis Communications Handbook ... you could dip
    into relevant sections.

[Oracle reply, without s_t] ---> not used for training; shown for comparison
    A few low-effort options can fit into routines you already have ... For structured
    practice, Crucial Conversations offers tools for difficult discussions ... You could
    start with one checklist and one reflection habit for two weeks.

The student is trained on the tokens of the Oracle reply written with sts_t. It never sees the private context or the user state, in training or at deployment.

Reviewer_TaX2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Mind2Dialogue, a framework for training human-aware language models using simulated user mental states as privileged supervision. It constructs stateful user–assistant interactions with a shared evolving user state and distills the Oracle assistant’s responses into models that do not observe this state at deployment.

Critical strengths
  1. The paper introduces a clear shared-state supervision framework that links simulated user behavior and assistant responses through the same evolving user state.
  2. The evaluation is broad, covering multiple model families together with personalization and theory-of-mind benchmarks, and shows consistent personalization gains across all three backbones.
Critical weaknesses
  1. The simulator includes several hand-designed choices, such as the latent-state schema and scenario categories, while the behavior controller is adapted from TUNA. It would be helpful to study the sensitivity to these choices and compare the resulting simulated behavior with real user interactions to better establish behavioral fidelity.
  2. The role of privileged state access could be isolated more clearly. The current ablations mainly vary state maintenance and behavior control, making it difficult to determine how much of the downstream improvement specifically comes from the Oracle’s access to the evolving user state.
  3. The evidence for human-aware learning could be further strengthened. Personalization and theory-of-mind are evaluated separately rather than within sustained user interactions, and part of the training data uses PersonaMem-style QA supervision similar to the evaluation format. Additional evaluation could help clarify how well the gains transfer to more integrated human-aware interaction settings.
Questions for authors

See the weaknesses above.

10 comments
AuthorsOpenReview ↗

Response to Reviewer TaX2, Part 1

Dear Reviewer TaX2,

Thank you for the constructive review. We are grateful that you found:

  • the shared-state supervision framework clear,
  • the evaluation broad, with "consistent personalization gains across all three backbones".

Your point on isolating the Oracle's access to the user state shaped the main new experiment of this response. Below we quote each point and answer it.

Weakness 1. Sensitivity to the hand-designed choices and fidelity to real users

"The simulator includes several hand-designed choices, such as the latent-state schema and scenario categories, while the behavior controller is adapted from TUNA. It would be helpful to study the sensitivity to these choices and compare the resulting simulated behavior with real user interactions to better establish behavioral fidelity."

Thank you for this suggestion. Our design follows standard practice in post-training data synthesis, and three new studies test its sensitivity and its fidelity to real users.

Key evidence

  • The schema is not fragile. Removing any one state group, or keeping only one, leaves corpus quality close to the full simulator (study (1)).
  • The simulated user decides the value of the training data. With the same teacher, a student trained on conversations from a simulator without the user state or the behavior controller trails the M2D student in every setting and falls below the base model (study (2)).
  • M2D-Sim writes more like real users. Compared with real WildChat users, M2D-Sim is closer in writing style than the Vanilla simulator and at least as close on every other measure (study (3)).

The review mentioned about the data pipeline designing, and we think a hand-designed structure is a sound starting point.

  • Hand-designed taxonomies are standard practice in post-training data synthesis.
    • UltraChat (Ding et al., 2023), GLAN (Li et al., 2024), AgentInstruct (Mitra et al., 2024) and Tülu 3 (Lambert et al., 2024) start from a hand-designed taxonomy of topics or skills and let an LLM generate the instances.
    • Constitutional AI starts from a hand-written list of principles, and a single general principle already generalizes (Kundu et al., 2023).
    • Hand-designed taxonomy define dataset coverage.
  • Our data pipeline design is benchmark-agnostic. The state schema, scenario categories and behavior modes are defined without reference to any evaluation benchmark.
  • The resulting corpus improves every backbone. Models trained on M2D-Chat gain on all three personalization benchmarks (table below) and in live interaction (Weakness 3 (1)).
BenchmarkCore characteristicQwen2.5-7BLlama-3.1-8BOLMo-3-7B
PrefEval-Genpreferences stated explicitly or implied by choices or persona, over 20 topics+33.4+40.9+26.6
PersonaMem-v1preferences that evolve over multi-session histories+6.5+6.3+5.6
PersonaMem-v2preferences revealed implicitly in long chat histories+10.0+13.6+10.2

Gain of the M2D-Chat trained model over its base model (paper Table 3).

AuthorsOpenReview ↗

Response to Reviewer TaX2, Part 2

(1) Sensitivity to the state schema and scenario categories. We rerun the simulator with one state group removed, with a single state group kept, and with a single scenario category. Setup: 24 personas, one rollout per setting, scored on the metrics of paper Tables 7 and 8. GPT-6-sol judges three dimensions on a 1-5 scale; turn novelty is on a 0-1 scale.

Simulator conditionState groups keptPersona specificityTopic depthOracle personalizationTurn novelty (0 to 1)
M2D-Sim (full)transient + stable + context3.834.043.790.979
Remove the context grouptransient + stable3.924.003.960.972
Remove the stable grouptransient + context3.794.083.830.986
Remove the transient groupstable + context4.004.044.080.972
Context only (goals, trust history, summary)context3.834.173.920.986
Stable state only (values, constraints, stance)stable3.884.133.920.931
Transient state only (mood, current concerns)transient3.964.214.040.944
Lifelong scenario category onlytransient + stable + context3.674.543.880.979
  • The exact schema is not a fragile choice. Each state group alone, or the schema with one group removed, stays close to the full simulator on every judged dimension.
  • The state as a whole matters, not one particular group. Without a state, persona specificity declines over a 20-turn rollout, while with the state it rises (paper Table 8).
  • Scenario categories shift topic depth as designed. The lifelong category alone goes deeper into a topic, so mixing categories gives the corpus coverage across depths.

(2) A student trained on Vanilla-simulator data. The Vanilla simulator uses the same personas, scenarios, turn counts and teacher model, without the user state or the behavior controller. Only the simulated user differs. The Vanilla corpus has fewer supervised tokens, so we also train M2D on a subset with the same number of supervised tokens. Students are Qwen3-8B with one training seed, evaluated on PrefEval-Gen.

Student trained onAvgExplicitChoicePersonaViol. [lower]
M2D46.5065.028.028.051.75
M2D, token volume matched to Vanilla44.0063.521.028.053.25
Vanilla simulator37.7555.016.025.061.50
Base model40.2557.019.028.059.25

The same students under PrefEval's official long-context settings, by the distance between the stated preference and the question:

Student trained on data from2 turns10 turns70 turns
M2D-Sim46.5031.7530.25
Vanilla Sim37.7525.5027.00
Diff: M2D-Sim − Vanilla Sim+8.75+6.25+3.25
None (base model)40.2528.5018.25
  • The simulator decides the value of the training data. With the same teacher, the Vanilla student scores lower at every distance and violates more of the user's preferences.
  • At equal token volume, M2D-Sim still leads Vanilla.
  • Conversations without a user state can hurt model performance.
  • Long-dialogue training helps with distant preferences. Both students beat the base model when the preference is far back, and M2D stays ahead there.

We investigate why the Vanilla simulator yields a poor dataset even with the same teacher, and our observe as the following: The teacher writes each training target as a reply to the simulated user's turn, so a target is only as informative as the turn it answers. The two corpora differ in what the simulated user keeps saying:

Training corpusM2D-SimVanilla
Conversations where the user ends early or sends an empty turn, so the rollout must be forced to continue0209 of 691
User turns 2 to 4 that still ask for something97%78%
Persona specificity of user turns over a 20-turn rollout (paper Table 8)risesdeclines
  • Without a state, the user loses its thread. Nothing carries the user's unresolved goals and constraints from turn to turn, so the Vanilla user drifts from its persona and wraps the conversation up.
  • The training targets collapse into closings. A reply to a goodbye has nothing to adapt to, so many Vanilla targets teach the student to agree and sign off instead of using what the user said earlier.
  • The M2D user keeps revealing constraints. The state keeps open goals and private constraints alive, so later turns raise new needs that the teacher's reply must accommodate.
AuthorsOpenReview ↗

Response to Reviewer TaX2, Part 3

Example, same persona (tends a garden in Rendsburg, values solitude, wants deeper connections) and scenario:

[Vanilla user, turn 4]   Thank you. I'll keep it simple and ask her when we next meet. I'm looking forward to the walk ...
[training target]        You're very welcome. It sounds as though you've found a way to make the invitation feel true to you ...
                         I hope the walk brings a pleasant conversation ...

[M2D user, turn 3]       ... especially considering my mobility concerns and budget constraints. What are some low-key,
                         budget-friendly activities I might explore ...?
[M2D user, turn 4]       ... a gardening workshop at home ... activities that would be enjoyable for everyone,
                         even those who may not have extensive gardening experience.
[training target]        A simple, relaxed gathering could last about an hour, with everyone able to sit around a table ...
                         Sow seeds in reused pots: use clean yogurt tubs ...

(3) Comparison with real user interactions. We follow the third-turn setup of "Real or Robotic?" (Ivey et al., 2026) and the judge prompts and metrics of SimulatorArena (Dou et al., 2025). From 320 real WildChat openings, each simulator writes the user's next turn, and the judge compares it with the real user's turn.

Measure (judged by GPT-6-sol)M2D-SimVanilla Sim
Interaction-style similarity to the real user [higher] (1-5)3.113.09
Writing-style similarity to the real user [higher] (1-5)2.492.25
Turing test: distance of judge accuracy from chance [lower]30.532.8
  • The user state makes the simulated user write more like the real user. Both simulators share the same underlying model, yet M2D-Sim's next turns are closer to the real user's writing style, and the judge tells them apart from the real turn less often.
  • Structure does not cost realism. M2D-Sim is at least as close to the real user as the Vanilla simulator on every measure.

This check covers single-turn continuations of real openings, and we will state this scope in the limitations. Recent simulators calibrate user profiles against real conversation logs (Wang et al., 2025; Zhu et al., 2026); we will discuss this as a direct extension of M2D's state schema.

On the behavior controller and TUNA.

  • Prompted LLM users lack behavioral variety. Without explicit control, they drift into homogeneous, script-following, uniformly polite turns (Naous et al., 2026; Zhou et al., 2026).
  • Attribute-conditioned generation is the established remedy. Conditioning each generation on sampled attributes keeps synthetic data diverse (Yu et al., 2023), and recent user simulators expand the user action space for the same reason (Chen et al., 2025).
  • We follow this practice with behavior modes observed in real users. The modes come from the Taxonomy of User Needs and Actions (TUNA; Shelby et al., 2025), which was derived from real human-AI conversations, so the injected behaviors follow observed users and not our own hand-design.
Reviewer_wCkwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Mind2Dialogue addresses the lack of supervision for assistance grounded in users’ unspoken beliefs, goals, and emotions. Its key design gives a simulated user and an Oracle assistant access to the same evolving user state, so that response targets are informed by the state generating user behavior. Standard supervised fine-tuning then distills these responses into students without access to that latent state, using both multi-turn dialogues and user-related QA.

The main novelty is the shared-state supervision pipeline, rather than a new optimization objective. Personalization improves over the respective base models across three model families; ToM transfer improves on Qwen and Llama but deteriorates on both BigToM tasks for OLMo.

Critical strengths

Shared-state supervision

  • Sharing evolving mental states between the simulated user and the Oracle offers a clear, well-motivated approach to constructing supervision for responses attentive to users’ implicit beliefs, goals, and emotions.

Scalable simulation-to-training pipeline

  • The simulator integrates psychology-guided scenarios, state updates, and behavioral control, producing dialogue and QA supervision through an end-to-end simulation-to-training pipeline without per-dialogue human annotation.
Critical weaknesses

The scope of the ToM gains remains insufficiently characterized.

  • Table 2 shows improvements over the fixed Qwen backbone, but M2D-Chat still substantially trails the proprietary reference models on BigToM. Together with the negative transfer on OLMo in Table 3, the results indicate backbone-dependent benefits and leave open whether the supervision can further improve models with stronger initial ToM capabilities.
  • The authors should clarify this scope and discuss how the gains relate to the backbone’s initial capabilities. (Tables 2–3)
Questions for authors

Trade-off between QA supervision and generation.

  • In Table 4, QA-only falls below Base by 1.8 and 2.2 percentage points on PrefEval Generation and PersonaMem-v2 Generation, respectively, while the full mixture trails dialogue-only by 0.6 and 0.2 points.
  • The text acknowledges this pattern, but further analyzation why QA-only degrades generation and whether task or output-format interference occurs in the mixture should be investigated.
7 comments
AuthorsOpenReview ↗

Response to Reviewer wCkw, Part 1

Dear Reviewer wCkw,

Thank you for the careful and solid review. We are grateful that you found:

  • Sharing evolving mental states between the simulated user and the Oracle "a clear, well-motivated approach to constructing supervision",
  • the simulation-to-training pipeline scalable "without per-dialogue human annotation". Below we address your concern of weakness and research on your question carefully.

Weakness. Scope of the ToM gains and the backbone's initial capability

M2D-Chat still substantially trails the proprietary reference models on BigToM. (gpt-5-mini)

We agree that M2D-Chat does not close the gap to GPT-5-mini on BigToM (Forward Belief 44.0 vs. 93.5, Table 2), and the paper does not claim it does. This gap is expected: absolute scores on ToM benchmarks mainly reflect model scale and the capability a backbone acquires in pre-training, which post-training data on personalization are not designed to supply. We see four reasons why this gap does not weaken the main claim.

  • BigToM is a transfer test to our dataset setting. M2D-Corpus contains multi-turn dialogues, PersonaMem-format MCQs, open-ended preference QA and preference-classification QA.
  • ToM gains from training data are hard to obtain and transfer unevenly, even with ToM-specific training data.
    • GPT-3 fine-tuned on ToMi reaches 98.5% on ToMi but 32–68% on structurally modified ToMi stories (Sclar et al., 2023).
    • RL on ToM datasets improves in-distribution accuracy but does not transfer to unseen ToM tasks (Sarangi and Salam, 2025).
    • Fine-tuning on 79.7k ToM-specific ExploreToM training instances improves ToMi by 27 points but BigToM by only 6 points (Sclar et al., 2025).
    • Without any ToM training instance, our corpus raises BigToM Forward Belief on Qwen2.5-7B from 20.0 to 49.0 in the setting with a stated initial belief (table (2) below).
  • LLM ToM benchmarks are sometimes contested.
    • Reports of human-level performance on false-belief tasks (Kosinski, 2024) stand beside evidence that LLM ToM is "far from being robust": models fail under trivial alterations of the stories (Ullman, 2023) and rely on shallow heuristics when tested with adversarial examples (Shapira et al., 2024).
  • The claim concerns improving an open model of a given size.
    • The paper proposes a scalable way to synthesize personalization post-training data, and the test of that claim is each backbone before and after training: personalization improves on every backbone (Table 3)
    • BigToM Forward Belief improves from 31.0 to 44.0 on Qwen2.5-7B, more than AutoToM (40.3) and ThoughtTracing (35.0) on the same backbone (Table 2).
    • Absolute BigToM scores track backbone scale: on the re-annotated BigToM subset of Kim et al. (2025), GPT-4o, Llama-3.3-70B and Qwen-2.5-72B reach 94–98 without any ToM-specific method.
    • GPT-5-mini is a proprietary model of undisclosed size and training data; it is listed as a reference point, not as a target for a 7B model.

Taken together, the gap to GPT-5-mini reflects backbone scale and overall model capbility, while the within-backbone gains below measure what our training data contribute.

AuthorsOpenReview ↗

Response to Reviewer wCkw, Part 2

negative transfer on OLMo in Table 3 indicate backbone-dependent benefits

We stated that the ToM gains are backbone-dependent, and we will state this scope more explicitly in the revised paper.

whether the supervision can further improve models with stronger initial ToM capabilities."

To relate the gains to the backbone's initial capability, we ran two controlled comparisons:

  • the same training data on a stronger backbone
  • the same models in two BigToM settings in which the base model starts from very different evaluation scores

(1) A stronger backbone, same data and training recipe.

The BigToM authors report that models struggle most when an initial belief is stated (Gandhi et al., 2023). We followed this setting for (Harder setting) and (Standard setting )

We trained Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct on the same 4.4k instances training data which includes:

  • 2,168 dialogues,
  • 1,331 PersonaMem-format question answering,
  • 923 Preference-following-format question answering.

with 1 epoch, using standard LoRA setting.

BenchmarkSetting#instance7B base7B trained14B base14B trained
PrefEval-Gen Acc.Avg.40022.352.329.0062.8
Explicit20028.078.543.096.0
Implicit, choice-based10018.026.013.028.0
Implicit, persona-driven10015.026.017.031.0
PrefEval-Gen Violation Rate [lower]n/an/a77.847.371.037.3
PersonaMem-v1 (MCQ)All58949.251.856.760.6
PersonaMem-v2 (MCQ)All30033.737.036.739.7
BigToM Forward BeliefStory w/o the initial belief20065.565.575.577.5
BigToM Forward BeliefStory w/ the initial belief20020.049.012.017.5
  • Adding each story with stating the agent's initial belief can make the evaluation instance harder. In the standard setting the story is without the initial belief prompted.

On all personalization benchmark, the 14B model gains about as much as the 7B model

  • PrefEval-Gen +33.8 vs. +30.0;
  • PersonaMem-v1 +3.9 vs. +2.5;
  • PersonaMem-v2 +3.0 vs. +3.3.

Should clarify this scope and discuss how the gains relate to the backbone's initial capabilities. (Tables 2–3)

(2) The same backbone at two levels of initial BigToM performance.

BigToM settingModelQwen2.5-7B baseQwen2.5-7B trained
(Harder setting ) Story w/ the initial beliefTrue belief Acc.21.552.0
False belief Acc.98.597.0
TB ∧ FB Acc.20.049.0
(Standard setting) Story w/o the initial beliefTrue belief Acc.76.586.0
False belief Acc.88.579.0
TB ∧ FB Acc.65.565.5

Revised scope.

  • Personalization gains hold across Qwen, Llama and OLMo and at 7B and 14B;
  • ToM transfer gains appear on Qwen and Llama; on Qwen2.5 they are positive at both 7B and 14B in the setting with a stated initial belief (+29.0 and +5.5).
  • We did not observe ToM transfer on OLMo, and claim this as a limitation (Table 3).
AuthorsOpenReview ↗

Response to Reviewer wCkw, Part 3

Question. QA supervision and generation

"In Table 4, QA-only falls below Base by 1.8 and 2.2 percentage points on PrefEval Generation and PersonaMem-v2 Generation, respectively, while the full mixture trails dialogue-only by 0.6 and 0.2 points."

"The text acknowledges this pattern, but further analyzation why QA-only degrades generation and whether task or output-format interference occurs in the mixture should be investigated."

Thank you for this question. We answer it in four steps: (1) What QA-only supervises (2) Model performance trade-off between MCQ and Free-form generation, (3) QA training instances in the multi-turn format. and (4) Further discussion about format mismacth

Throughout, we distinguish two training formats. They differ in how the chat template renders the conversation history and in which tokens the loss mask covers:

Training formatRendering of the conversation historySupervised tokens (loss mask)
Single-turn formatSerialized as text inside one user messageFinal assistant turn only (the answer)
Multi-turn formatRendered by the chat template as alternating user and assistant messagesEvery assistant turn: the non-answer turns before the question and the answer turn

(1) What QA-only supervises.

The QA-only condition uses the single-turn format. A real training instance:

[system]  You are a personalized AI assistant. ... 
	     <persona> 
	         I'm a seasoned power electronics engineer ... 
	     </persona>
					
[user]    <conversation_history>
	          <user>
		          Could you provide guidance on how to implement these unit tests ...
	          </user>
	          <assistant>
		          To effectively implement and run your unit tests ...
	          </assistant>
		  <user>
			...
		  </user>
		        ...
          </conversation_history>
          <question> 
	          What do you usually check... 
	          A. ...  
	          B. ...  
	          C. ...  
	          D. ...
	  </question>
          <instruction>
	          Think step by step about which answer best fits the user's query and conversation context.
	          Reasoning first, then give your final answer as 'Final Answer: [Letter]'
	  </instruction>
					
[assistant] -> token supervised
	          The user is looking for best practices in creating documentation for coding projects, whichrequires clarity and detailed instructions. Option B emphasizes ... Final Answer: B
  • The earlier assistant turns appear only as text in the user message, so the loss mask excludes them; the model is never trained to produce an assistant turn that continues a multi-turn conversation.
  • The supervised target is a short rationale ending with an option letter, about 100 tokens on average. Following common practice for instruction models, the rationale gives more supervision signal than the letter alone; it is far shorter than the thinking trace of a reasoning model.
  • The open-ended preference training instances use the same wrapper, with a free-form target of 0.17k tokens on average.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response: Summary of New Experiments and Revisions (Part 1 of 2)

We thank all three reviewers for their careful reading and constructive feedback. The core vision of Mind2Dialogue is to train human-aware LLM assistants to learn to personalize their advice under partial observability. In the real world, a request rarely reveals every belief or goal behind it; thus, an assistant must draw on clues across turns to suggest options that fit the user’s priorities and avoid those that conflict with their constraints.

We truly appreciate the reviewers’ recognition of both this idea and its results, and we summarize the core recognitions as follows:

Recognized strengthWhat Mind2Dialogue contributesReviewers’ comments
Methodological noveltyThe same evolving state drives the simulated user’s message and guides the Oracle assistant’s reply.“The main novelty is the shared-state supervision pipeline” (Reviewer wCkw). “The paper introduces a clear shared-state supervision framework that links simulated user behavior and assistant responses through the same evolving user state.” (Reviewer TaX2)
Well-motivated problemThe method supplies training examples for responding to beliefs, goals, and emotions that users may leave unspoken.“a clear, well-motivated approach to constructing supervision for responses attentive to users’ implicit beliefs, goals, and emotions” (Reviewer wCkw).
Large, consistent gainsPreference-following generation improved by 26.6 to 40.9 percentage points over the respective Qwen, Llama, and OLMo baselines. M2D-Chat on Qwen-7B also outperformed several alternative personalization methods.“consistent personalization gains across all three backbones” (Reviewer TaX2). “outperforming both the base model together with a wide range of other approaches” (Reviewer 4imh).
Broad, sound evaluationThe experiments test whether models can reason about users’ beliefs and apply users’ preferences in their replies.“The evaluation is broad, covering multiple model families together with personalization and theory-of-mind benchmarks” (Reviewer TaX2). “The experiments themselves appear sound” (Reviewer 4imh).
Scalable data generationThe simulator produces both dialogue and QA training examples without requiring a human to annotate each conversation.“an end-to-end simulation-to-training pipeline without per-dialogue human annotation” (Reviewer wCkw).
Practical deploymentThe Oracle sees the private state when it writes demonstrations. The trained assistant needs only the persona and visible dialogue.“distills the Oracle assistant’s responses into models that do not observe this state at deployment” (Reviewer TaX2).

Main reviewer questions and our responses

The reviewers also suggested additional comparisons and clearer explanations. We have addressed all questions raised by the three reviewers in our individual replies and incorporated the corresponding changes into the revised manuscript. For the area chair and reviewers, we bring together the three points most central to the paper’s contribution.

Point addressedReviewer feedbackOur response and evidenceDetailed replies
Personalization gains at 7B and 14BThe original study covered three model families at 7B to 8B. Reviewers asked “how well this generalizes to larger models” (Reviewer 4imh) and, for belief reasoning, “how the gains relate to the backbone’s initial capabilities” (Reviewer wCkw).New experiments. With the same 4,422 training examples, Qwen2.5 gained 33.8 points at 14B and 30.0 at 7B on PrefEval-Gen. Both sizes also improved on PersonaMem and on BigToM Forward Belief when the story stated the initial belief.4imh W1 (Parts 1–2), wCkw W (Part 2)
Isolating the Oracle’s contribution to student preference followingThe state shapes both sides of the conversation. The reviewer asked “how much of the downstream improvement specifically comes from the Oracle’s access to the evolving user state” (Reviewer TaX2).New controlled experiment. We fixed the conversations and training recipe and varied only what the Oracle saw. Across three seeds, student accuracy on PrefEval-Gen averaged 44.8% when the Oracle saw the correct state, 39.4% when it saw no state, and 37.8% when it saw another user’s state.TaX2 W2 (Parts 4–5)
Preference following in free-form, multi-turn dialogueOne reviewer asked about transfer to “sustained user interactions” (Reviewer TaX2). Reviewer wCkw also asked how adding QA affects free-form generation.New experiments. Both 7B and 14B outperformed their baselines on ALOE and MultiChallenge. Ablation in the submitted paper (Table 4). Dialogue-only Llama matched the full mixture on PrefEval-Gen (66.0% vs. 65.4%), while adding QA improved PersonaMem-v2 multiple-choice accuracy by 12.7 points.TaX2 W3 (Parts 6–8), wCkw Q (Part 4)
AuthorsOpenReview ↗

General Response: Summary of New Experiments and Revisions (Part 2 of 2)

Additional experiments and clarifications

Beyond the three main points in Part 1, the reviewers offered many useful suggestions that helped us refine the paper. We have completed the additional checks and incorporated the clarifications into the manuscript, as shown below.

Point addressedReviewer feedbackOur response and evidenceDetailed replies
Preference following up to 300 turns laterReviewer TaX2 asked us to isolate the value of state access. We extended that comparison to preferences mentioned much earlier in the conversation.New experiments. A blinded judge favored correct-state replies over no-state replies in 368 of 450 comparisons. Students trained on correct-state replies retained their lead when the preference appeared 10, 70, or 300 turns earlier.TaX2 W2 (Parts 4–5)
Simulator sensitivity to state fields and scenario choicesReviewer TaX2 asked us to test the simulator’s hand-designed choices.New experiments. Removing any one state group kept topic depth near 4/5. Using only lifelong scenarios raised it to 4.54. With the same teacher and supervised-token budget, students scored 44.0% with M2D data versus 37.8% with vanilla data on PrefEval-Gen.TaX2 W1 (Part 2)
Simulated and real next turns on WildChatReviewer TaX2 requested a comparison with real user interactions.New experiment. On single-turn continuations of 320 WildChat openings, M2D and vanilla received similar ratings for matching real users’ interaction styles (3.11 vs. 3.09 out of 5).TaX2 W1 (Part 3)
Generation drop after single-turn QA trainingReviewer wCkw asked whether task or output format explains the drop in Table 4.New training experiment. QA-only training that also supervised earlier assistant replies reached 57.3% on PrefEval-Gen, compared with 22.7% in the original final-answer-only run. New scoring analysis. After extracting choices from free text, dialogue-trained Qwen3-8B matched its baseline on PersonaMem-v1 (53.0% vs. 52.5%).wCkw Q (Parts 3–6)
Content overlap between training and evaluation QAReviewer TaX2 noted the shared QA format. This prompted us to check for shared content.New overlap ablation. Training QA had zero matches in the corresponding evaluation sets under either the 8-gram rule or the embedding-similarity criterion.TaX2 W3 (Part 8)
BigToM gains by backbone and initial-belief settingReviewer wCkw asked how transfer varies with the backbone’s initial belief-reasoning ability.New experiments. With a stated initial belief, Qwen2.5 gained 29.0 points at 7B and 5.5 at 14B on Forward Belief. The standard 7B score stayed at 65.5%. Clarified claim. We report transfer by backbone and setting, including OLMo’s declines on both BigToM tasks in the submitted paper.wCkw W (Parts 1–2)
State visibility in Section 3 and baseline groupings in Table 1Reviewer 4imh requested a worked example, an explanation of Section 4.2, and clearer model groupings in Table 1.Manuscript revisions. Section 3 now traces a state update and shows what the simulator, Oracle, and student each see. Section 4.2 explains the dialogue-quality checks. Table 1 identifies the proprietary reference models and defines boldface within the same-backbone comparison.4imh W2 and Q1–Q4 (Parts 3–6)

We summarize key takeaways from the rebuttal as follows:

  • Personalization gains hold across model families and sizes. Qwen, Llama, and OLMo improved on generation and multiple-choice personalization tasks. The gains extended to sustained dialogue at 7B and 14B on Qwen2.5.
  • The correct user state makes the Oracle a better teacher for preference following. Students trained on replies written with the user’s own state performed best in all three training runs. The deployed assistant needs only the persona and dialogue.
  • Theory-of-mind transfer depends on the backbone and belief setting. The clearest Qwen gains appeared with a stated initial belief. The revised paper specifies where positive transfer occurred.
  • The real-user fidelity check covers single-turn behavior. M2D and vanilla received similar ratings for matching real users’ interaction styles on WildChat. The limitations now state this scope explicitly.

To sum up, as LLM assistants take on longer-running tasks, making them more human-aware means that a preference expressed early in a conversation can shape several decisions that follow. Traditional LLM training conversational data records what users say, yet rarely reveals why one reply served a user better than another; hence, the supervision gap and data scarcity issue. We hope Mind2Dialogue can contribute to this timely discussion at ICLR by giving researchers a practical way to study this supervision gap and train assistants that keep users’ priorities in view over time.

We would be glad to answer any further questions.

Best regards,

Authors

AuthorsOpenReview ↗

General Response: Update on the Discussion Phase

We thank the Area Chair and all reviewers for their time and engagement.

We are pleased to report that Reviewer 4imh has confirmed that our new 14B-scale experiment resolved their main concern and has raised their score. We are grateful for this constructive exchange.

For the convenience of the Area Chair and the remaining reviewers, we briefly recap where each reviewer’s main concerns are addressed (all new results are included in the revised manuscript and appendix):

  • Reviewer wCkw (ToM scope; QA vs. generation): same-data experiments on Qwen2.5-7B/14B, BigToM analysis by setting, a revised and more explicit scope statement, a multi-turn-format retraining of QA-only data, and a format-controlled scoring analysis (Responses to wCkw, Parts 1–6).

  • Reviewer TaX2 (design sensitivity and realism; isolating the Oracle’s state access; sustained interaction): simulator sensitivity and WildChat studies, a controlled teacher-input experiment across three seeds, evaluation on ALOE and MultiChallenge, a dialogue-only ablation, and an overlap audit (Responses to TaX2, Parts 1–8).

  • Reviewer 4imh (scale; presentation): the 14B experiment, a worked example of the user state with a visibility table, and a clarified Section 3/4.2 and Table 1 (Responses to 4imh, Parts 1–6).

We would be glad to hear whether Reviewers wCkw and TaX2 find that these additions address their concerns, and we remain happy to provide any further clarification or experiments during the remaining discussion period.

Thank you again for your time and effort.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.