- Rating
- 2
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Collaborative Theory of Mind (CToM), a framework that trains an assistant to anticipate user responses and use those predictions for internal revision before presenting a draft; the policy chooses between asking a clarification question, drafting with prediction-guided revision, and committing a previously shown draft.
Training combines a supervised user-response prediction objective with GRPO on task performance and visible interaction cost, using a simulated user to provide both observed replies and counterfactual replies to discarded drafts. Across mathematical reasoning, coding, and document writing, CToM achieves the highest reported mean task scores among the evaluated methods.
The approach is promising and addresses an important problem. However, my main concerns are that the evaluation does not distinguish transferable user modeling from adaptation to a fixed simulator, and that the ablations do not isolate prediction supervision as the cause of the improvements. These gaps limit the strength of the paper’s central claim.
Critical strengths
A concrete and useful formulation of anticipatory collaboration. The method operationalizes anticipation through a specific learning objective and inference procedure rather than treating theory of mind solely as a prompting instruction or an abstract capability. The distinction between asking for genuinely missing information and internally repairing predictable mistakes is particularly sensible: the method allows both behaviors within a single policy.
Critical weaknesses
-
The evaluation does not establish generalization beyond a fixed user simulator Training and evaluation use the same Qwen3-32B-FP8 simulator with domain-specific behavioral templates; math uses a fixed expert persona, while coding and writing users follow prescribed rules for concise, cooperative disclosure of hidden requirements. The experiments demonstrate success within this simulated interaction distribution, but do not distinguish learning a broadly transferable user model from adapting to the simulator’s characteristic corrections, acceptance patterns, and disclosure rules. Task diversity is valuable, but it is not equivalent to user diversity. Likewise, stochastic sampling from one simulator does not establish robustness to users with different expertise, preferences, feedback styles, or reliability.
-
The central contribution of prediction supervision is not isolated The closest baseline, Task-Only GRPO, removes both the prediction loss and the effort penalty, whereas CToM uses both; the authors correctly describe this as measuring their joint contribution. However, the subsequent behavioral analysis attributes larger revision gains specifically to training the assistant to predict feedback. This attribution is not established by the available comparison. The small effort coefficient is not a sufficient reason to dismiss its effect, particularly because GRPO normalizes rewards within rollout groups.
Questions for authors
How well does CToM transfer to held-out simulator models, personas?
4 comments
Follow-up to Reviewer 8giV
Dear Reviewer 8giV,
Thank you once more for your careful review. We are writing to ask whether the response has resolved your concerns. If so, we would be grateful if you would consider raising your score. If anything is still open, we would appreciate hearing about it soon, so that we have enough time to address it before the discussion period ends.
Thanks for the response and additional experiments. The full ablation addresses my concern about separating prediction supervision from the effort penalty. The held-out simulator results and the talkative-user experiment also partially address my concern about generalization.
I still think the user-modeling claim should be more limited, as these experiments do not directly test users with different expertise, preferences, or feedback reliability. However, the new results provide useful evidence and improve my assessment of the contribution. I will raise my Informativeness score.
Thank you for reading our response and the new results, and for raising your informativeness score. We are glad that the full ablation addresses your concern about separating prediction supervision from the effort penalty.
Remaining concern
"I still think the user-modeling claim should be more limited, as these experiments do not directly test users with different expertise, preferences, or feedback reliability."
We would like to emphasize that this is a method paper. Its contribution is a training method that aims to improve collaborative task performance through user-response prediction with an internal draft–predict–revise loop. We do not claim to predict or model users perfectly. Our claim is that learning to predict user responses improves task performance.
In the revision, we added three new users and evaluated them in seven settings across the three domains. None of them is used in training.
| New user | What differs from the training user | Domains |
|---|---|---|
| GPT-4o-mini | the simulator model, from another model family | math, coding, writing |
| Claude Haiku 5.5 | the simulator model, from another model family | math, coding, writing |
| Talkative user | the user persona: long, informal replies with side remarks | coding |
With the held-out simulators, adding prediction supervision raises the task score in 11 of the 12 comparisons, and CToM w/o effort has the highest task score in five of the six settings (see App. B.7, Table 8). With the talkative user, CToM has the highest task score (see our response to Question 1). These evaluations show that the gains in collaborative task performance from prediction supervision generalize to a different user persona and to different model families.
Response to Reviewer 8giV
We thank the reviewer and appreciate the noted strengths: an approach that is "promising and addresses an important problem", a "concrete and useful formulation of anticipatory collaboration", and a distinction between asking for missing information and internally repairing predictable mistakes that is "particularly sensible". We ran new experiments for both concerns, in the revised PDF.
Weakness 1
We evaluated all models with two held-out user simulators from other model families, GPT-4o-mini and Claude Haiku 5.5 (see General Response, Section 2; App. B.7, Table 8).
CToM, although trained only with the Qwen3-32B user, generalizes its user prediction to held-out users it has not seen, which yields better task performance.
Weakness 2
The full 2×2 ablation (see General Response, Section 1; Table 1) separates the two signals.
The effect of the effort penalty is now measured by the two ablations that remove it, not argued from the size of its coefficient. The revision also no longer reports the combined score: we found that the effort weight of 0.01, although it has an effect in training, is too small to be used in evaluation, so we evaluate user effort directly (see General Response, Section 1).
Question 1
Transfer to held-out simulator models is addressed in Weakness 1. On personas, we ran a new evaluation on coding with a user persona not seen in training: a talkative user. It is the same Qwen3-32B simulator, but it writes long, informal replies of four to six sentences, and surrounds the information the assistant needs with side remarks. The original user replies in one or two sentences.
Coding task score (×100 ± SEM)
This shows that prediction supervision also helps with a user persona that CToM has not seen in training.
Early in the project we also tried a smaller model (Qwen3-8B) as a novice user, but it often forgot its role and the assistant could not learn from it, so we dropped that setting. Simulators from different model families (the held-out simulators) can be considered as having different personas in this setting.