acceptodds
Under review as a conference paper at ICLR 2027

Collaborative Theory of Mind: Learning to Self-Revise by Anticipating User Responses

10 views

Abstract

Large language models (LLMs) demonstrate strong individual task performance, yet still struggle to collaborate with users over extended, multi-turn interactions, for example by attempting solutions before understanding the user's needs or by failing to revise mistaken assumptions after receiving feedback. We propose Collaborative Theory of Mind (CToM), a framework that addresses these challenges by explicitly training LLMs to anticipate user responses and to use those predictions to improve their own behavior. CToM drafts a response, predicts how the user will respond, and uses this prediction to guide internal revision before presenting a response. During training, the user's reply to each draft serves as the target for predicting the next turn, and we train the assistant to minimize the negative log-likelihood of that reply. We combine this prediction objective with group-relative policy optimization (GRPO) to optimize both task performance and interaction efficiency. Across mathematical reasoning, coding, and writing tasks, CToM achieves the highest collaborative task performance and the best trade-off between task performance and user effort among the evaluated methods. For example, on a collaborative mathematical reasoning task, CToM improves final accuracy from 74.1% to 79.6% compared with naive GRPO, while reducing the number of tokens users need to process by 76%. Behavioral analyses of interaction trajectories show that user-response predictions become more accurate after training, and that more accurate predictions are associated with larger improvements in collaborative task performance through self-revision.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Alignment, Reinforcement Learning, Human-AI collaboration, theory of mind

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8giVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Collaborative Theory of Mind (CToM), a framework that trains an assistant to anticipate user responses and use those predictions for internal revision before presenting a draft; the policy chooses between asking a clarification question, drafting with prediction-guided revision, and committing a previously shown draft.

Training combines a supervised user-response prediction objective with GRPO on task performance and visible interaction cost, using a simulated user to provide both observed replies and counterfactual replies to discarded drafts. Across mathematical reasoning, coding, and document writing, CToM achieves the highest reported mean task scores among the evaluated methods.

The approach is promising and addresses an important problem. However, my main concerns are that the evaluation does not distinguish transferable user modeling from adaptation to a fixed simulator, and that the ablations do not isolate prediction supervision as the cause of the improvements. These gaps limit the strength of the paper’s central claim.

Critical strengths

A concrete and useful formulation of anticipatory collaboration. The method operationalizes anticipation through a specific learning objective and inference procedure rather than treating theory of mind solely as a prompting instruction or an abstract capability. The distinction between asking for genuinely missing information and internally repairing predictable mistakes is particularly sensible: the method allows both behaviors within a single policy.

Critical weaknesses
  • The evaluation does not establish generalization beyond a fixed user simulator Training and evaluation use the same Qwen3-32B-FP8 simulator with domain-specific behavioral templates; math uses a fixed expert persona, while coding and writing users follow prescribed rules for concise, cooperative disclosure of hidden requirements. The experiments demonstrate success within this simulated interaction distribution, but do not distinguish learning a broadly transferable user model from adapting to the simulator’s characteristic corrections, acceptance patterns, and disclosure rules. Task diversity is valuable, but it is not equivalent to user diversity. Likewise, stochastic sampling from one simulator does not establish robustness to users with different expertise, preferences, feedback styles, or reliability.

  • The central contribution of prediction supervision is not isolated The closest baseline, Task-Only GRPO, removes both the prediction loss and the effort penalty, whereas CToM uses both; the authors correctly describe this as measuring their joint contribution. However, the subsequent behavioral analysis attributes larger revision gains specifically to training the assistant to predict feedback. This attribution is not established by the available comparison. The small effort coefficient is not a sufficient reason to dismiss its effect, particularly because GRPO normalizes rewards within rollout groups.

Questions for authors

How well does CToM transfer to held-out simulator models, personas?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8giV

We thank the reviewer and appreciate the noted strengths: an approach that is "promising and addresses an important problem", a "concrete and useful formulation of anticipatory collaboration", and a distinction between asking for missing information and internally repairing predictable mistakes that is "particularly sensible". We ran new experiments for both concerns, in the revised PDF.

Weakness 1

"The evaluation does not establish generalization beyond a fixed user simulator. ... Task diversity is valuable, but it is not equivalent to user diversity."

We evaluated all models with two held-out user simulators from other model families, GPT-4o-mini and Claude Haiku 5.5 (see General Response, Section 2; App. B.7, Table 8).

  • Prediction supervision raises task score for held-out users.
  • Under the same reward, CToM needs fewer user replies than CToM w/o prediction (6–23% fewer; see Table 9).
  • CToM w/o effort has the highest task score in 5 of the 6 held-out user experiments.

CToM, although trained only with the Qwen3-32B user, generalizes its user prediction to held-out users it has not seen, which yields better task performance.

Weakness 2

"The central contribution of prediction supervision is not isolated. ... the subsequent behavioral analysis attributes larger revision gains specifically to training the assistant to predict feedback. This attribution is not established by the available comparison. The small effort coefficient is not a sufficient reason to dismiss its effect, particularly because GRPO normalizes rewards within rollout groups."

The full 2×2 ablation (see General Response, Section 1; Table 1) separates the two signals.

  • Prediction supervision raises task score in all six comparisons, with or without the effort penalty.
  • The effort penalty alone leaves task score within 1 SEM of Task-Only GRPO in every domain. Removing it from CToM makes the assistant use 15–61% more visible tokens.
  • The behavioral analyses now compare CToM with CToM w/o prediction, which share the effort penalty, prompt and loop (see Sec. 5.3, Fig. 3). The math within-round self-revision gain is +3.40 for CToM vs. +0.93 for CToM w/o prediction at revision cap 1, and +5.48 vs. +3.72 at cap 3. After a failing first program, the coding score ends at 46.2 for CToM vs. 40.5 for CToM w/o prediction.

The effect of the effort penalty is now measured by the two ablations that remove it, not argued from the size of its coefficient. The revision also no longer reports the combined score: we found that the effort weight of 0.01, although it has an effect in training, is too small to be used in evaluation, so we evaluate user effort directly (see General Response, Section 1).

Question 1

"How well does CToM transfer to held-out simulator models, personas?"

Transfer to held-out simulator models is addressed in Weakness 1. On personas, we ran a new evaluation on coding with a user persona not seen in training: a talkative user. It is the same Qwen3-32B simulator, but it writes long, informal replies of four to six sentences, and surrounds the information the assistant needs with side remarks. The original user replies in one or two sentences.

Coding task score (×100 ± SEM)

MethodOriginal user (Table 1)Talkative user
Naive GRPO48.91 ± 3.2443.15 ± 3.28
CToM51.05 ± 3.0848.95 ± 3.18
CToM w/o prediction46.98 ± 3.1342.57 ± 3.15
  • CToM has the highest task score with the talkative user: 5.8 points above Naive GRPO and 6.4 points above CToM w/o prediction.
  • Its score drops by 2.1 points with the talkative user, compared with 5.8 for Naive GRPO and 4.4 for CToM w/o prediction.

This shows that prediction supervision also helps with a user persona that CToM has not seen in training.

Early in the project we also tried a smaller model (Qwen3-8B) as a novice user, but it often forgot its role and the assistant could not learn from it, so we dropped that setting. Simulators from different model families (the held-out simulators) can be considered as having different personas in this setting.

AuthorsOpenReview ↗

Follow-up to Reviewer 8giV

Dear Reviewer 8giV,

Thank you once more for your careful review. We are writing to ask whether the response has resolved your concerns. If so, we would be grateful if you would consider raising your score. If anything is still open, we would appreciate hearing about it soon, so that we have enough time to address it before the discussion period ends.

Thanks for the response and additional experiments. The full ablation addresses my concern about separating prediction supervision from the effort penalty. The held-out simulator results and the talkative-user experiment also partially address my concern about generalization.

I still think the user-modeling claim should be more limited, as these experiments do not directly test users with different expertise, preferences, or feedback reliability. However, the new results provide useful evidence and improve my assessment of the contribution. I will raise my Informativeness score.

Thank you for reading our response and the new results, and for raising your informativeness score. We are glad that the full ablation addresses your concern about separating prediction supervision from the effort penalty.

Remaining concern

"I still think the user-modeling claim should be more limited, as these experiments do not directly test users with different expertise, preferences, or feedback reliability."

We would like to emphasize that this is a method paper. Its contribution is a training method that aims to improve collaborative task performance through user-response prediction with an internal draft–predict–revise loop. We do not claim to predict or model users perfectly. Our claim is that learning to predict user responses improves task performance.

In the revision, we added three new users and evaluated them in seven settings across the three domains. None of them is used in training.

New userWhat differs from the training userDomains
GPT-4o-minithe simulator model, from another model familymath, coding, writing
Claude Haiku 5.5the simulator model, from another model familymath, coding, writing
Talkative userthe user persona: long, informal replies with side remarkscoding

With the held-out simulators, adding prediction supervision raises the task score in 11 of the 12 comparisons, and CToM w/o effort has the highest task score in five of the six settings (see App. B.7, Table 8). With the talkative user, CToM has the highest task score (see our response to Question 1). These evaluations show that the gains in collaborative task performance from prediction supervision generalize to a different user persona and to different model families.

Reviewer_8VxeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Collaborative Theory of Mind (CToM), which trains an assistant to predict the user's reply to its own draft and to revise before showing it. Each round, the assistant writes a short plan and chooses ASK, DRAFT, or COMMIT. On DRAFT, it writes a draft and predicts the user's reply, and a fixed lexical classifier labels the prediction as satisfied or not; if not, the assistant revises, up to three times, and then shows the last draft. During training, the prediction for a shown draft is supervised by the simulated user's actual reply, and the prediction for each discarded draft by a counterfactual reply obtained by querying the simulator after the rollout. The prediction loss is added to GRPO, whose reward is the task score minus 0.01 times an interaction cost, and the prediction gradient is capped at half the GRPO gradient. Experiments use a Qwen3-8B assistant and a Qwen3-32B user simulator on MATH Level-5, BigCodeBench, and MediumDocEdit. Baselines are an untrained prompting baseline, GRPO with a minimal prompt, and Task-Only GRPO, which shares CToM's prompt but has neither the prediction loss nor the cost penalty. CToM obtains the highest task and combined scores in all three domains. The paper also reports that predictions become more accurate with training, that the prediction loss makes internal revision more useful, and that encouraging more questions lowers the combined score.

Critical strengths
  1. The idea is simple and well motivated. Predicting the user's reply to a draft becomes an explicit training target and is used at test time to decide whether to revise first. Querying the simulator for counterfactual replies to drafts the user never saw is a neat, reusable way to get supervision. The method, prompts, and classifier rules are all given, and code is released. This is the main reason I think the paper is worth attention.
  2. Some analyses are useful. An action-entropy bonus raises the ASK rate in all three domains, but the combined score drops in five of six comparisons , so asking more is not the same as collaborating better.
Critical weaknesses
  1. The paper does not show that predicting the user is what helps. Task-Only GRPO, the only baseline with CToM's prompt, removes both the prediction loss and the effort penalty, and CToM beats it by only 1.3–2.9 points, so the gap cannot be attributed to either. The prediction loss also gives every discarded draft feedback written by a 32B simulator that knows the answer or the full spec, which is extra supervision in itself.
  2. The theory-of-mind part is never tested. Each domain has one fixed simulated user. In math it holds the reference answer and only tells the assistant whether its work is right, at most naming the wrong step, so predicting its reply is essentially self-verification, closer to self-correction work such as SCoRe (Kumar et al., ICLR 2025) than to modeling a user. In coding and writing it points out mismatches with a hidden spec. Users never differ, so we cannot tell whether the model understands a person or has learned how one simulator gives feedback. The learned strategy may also be simulator-specific.
Questions for authors
  1. Is there a run with $λ_{pred} = 0 $ that keeps the effort penalty?
  2. By the counterfactual-reply equation, the simulator answering a discarded draft sees the earlier drafts and the assistant's own predicted replies, which the user never sees in real rollouts (App. C.1). How are these put into the simulator prompt, given that Prompts 8–10 only contain the visible chat history? Could they pull the counterfactual targets toward the assistant's own predictions? Suggestions:
    Please introduce variation across users, for example the novice and expert levels already in the math simulator prompt, or users with different preferences, and test whether predictions and revisions change with the user. Evaluating with a simulator from a different model family would also show whether the gains depend on Qwen3-32B.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 8Vxe (1/2)

We thank the reviewer and appreciate the noted strengths: an idea that is "simple and well motivated", counterfactual replies as "a neat, reusable way to get supervision", and the action-entropy analysis, which shows that "asking more is not the same as collaborating better". We ran new experiments to address the concerns, now in the revised PDF.

Weakness 1

"The paper does not show that predicting the user is what helps. Task-Only GRPO, the only baseline with CToM's prompt, removes both the prediction loss and the effort penalty, and CToM beats it by only 1.3–2.9 points, so the gap cannot be attributed to either."

We trained the two missing ablations (see General Response, Section 1; Table 1). CToM w/o prediction keeps the effort penalty and removes only the prediction loss; CToM w/o effort keeps the prediction loss and removes the effort penalty.

  • With the effort penalty kept, prediction supervision adds 1.8 points of task score on math, 4.1 on coding and 1.1 on writing, and cuts user replies by 8% on coding and 17% on writing.
  • Without the penalty it adds 5.0, 6.3 and 1.1 points.
  • The effort penalty alone moves task score by at most 1.1 points.

The gains are therefore attributable to prediction supervision.

Weakness 2

"The prediction loss also gives every discarded draft feedback written by a 32B simulator that knows the answer or the full spec, which is extra supervision in itself."

The counterfactual replies are the training targets of the prediction loss (see Sec. 4.2). They are how CToM is trained to predict the user, not a separate source of supervision, so the full ablation in Weakness 1 measures their effect.

  • They are used only as targets for the predicted reply. The assistant never sees them, and they are never used as targets for its drafts.
  • The simulated user does not hand over its hidden information: in math it never reveals, quotes or hints at the reference answer, and in coding and writing it never pastes the specification or the material (see Table 12).
  • At test time there are no counterfactual queries. The assistant revises from its own predictions.
  • With this supervision, prediction F1 with the Qwen3-32B user is 19.7 on math, 43.8 on coding and 35.8 on writing, against 14.0, 12.8 and 15.8 for CToM w/o prediction (see Fig. 3a; Table 6), and the self-revision gain on math is larger (see Fig. 3b).

Weakness 3

"The theory-of-mind part is never tested. Each domain has one fixed simulated user. ... predicting its reply is essentially self-verification, closer to self-correction work such as SCoRe (Kumar et al., ICLR 2025) than to modeling a user. ... The learned strategy may also be simulator-specific."

We evaluated with two held-out user simulators from other model families, GPT-4o-mini and Claude Haiku 5.5 (see General Response, Section 2; App. B.7).

  • Prediction supervision raises task score for held-out users.
  • Prediction F1 on coding is 54.2 with GPT-4o-mini and 53.1 with Haiku 5.5 for CToM, against 8.4 and 9.4 for CToM w/o prediction.

CToM, although trained only with the Qwen3-32B user, generalizes its user prediction to held-out users it has not seen. We appreciate your mention of SCoRe, and we have added it to the related work.

AuthorsOpenReview ↗

Response to Reviewer 8Vxe (2/2)

Question 1

"Is there a run with λpred=0\lambda_{\text{pred}}=0 that keeps the effort penalty?"

Yes. This is addressed in Weakness 1: CToM w/o prediction keeps the effort penalty and removes only the prediction loss. We put the results in General Response, Section 1.

Question 2

"... the simulator answering a discarded draft sees the earlier drafts and the assistant's own predicted replies ... How are these put into the simulator prompt, given that Prompts 8–10 only contain the visible chat history?"

Your reading of the equation is correct. The simulator prompt (Prompts 8–10) is the same for real and counterfactual replies. The {chat_history} slot is filled in two ways:

  • Outer conversation (real replies). The slot holds only the visible conversation. The simulator never sees discarded drafts; it sees only the draft that is shown.
  • Inner loop (counterfactual replies, the training targets for the prediction). A counterfactual query is made for each draft in the inner loop. For the first draft, the slot holds the visible conversation followed by the draft: user → draft → counterfactual reply. For a later draft, the user turn after each earlier draft holds the assistant's own predicted reply, not the earlier counterfactual reply: user → draft 1 → predicted reply 1 → draft 2 → counterfactual reply.

We use the assistant's own prediction in that turn because, at inference, the prediction serves as the simulated user inside the inner loop. For the second and later drafts of a round, the chat history therefore contains the assistant's earlier predictions.

Question 3

"Could they pull the counterfactual targets toward the assistant's own predictions?"

If the assistant predicts that the user is satisfied, the inner loop ends. If not, the simulator prompt asks the simulated user to respond only to the assistant's latest message (see Prompts 8–10), so the earlier predictions have no effect on the counterfactual reply.

Question 4

"Please introduce variation across users, for example the novice and expert levels already in the math simulator prompt, or users with different preferences ... Evaluating with a simulator from a different model family would also show whether the gains depend on Qwen3-32B."

Evaluation with simulators from two other model families, GPT-4o-mini and Claude Haiku 5.5, is addressed in Weakness 3. On the novice level in the math simulator prompt: early in the project we tried a smaller model (Qwen3-8B) as a novice user, but it often forgot its role and the assistant could not learn from it. We dropped that setting, and math uses only the expert persona. Simulators from different model families (the held-out simulators) can be considered as having different personas in this setting.

Reviewer_BZorOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Collaborative Theory of Mind (CToM), a framework that trains LLM assistants to anticipate user feedback and use those predictions to revise responses before presenting them. The assistant chooses among three actions: ASK, DRAFT, and COMMIT. DRAFT invokes an internal draft–predict–revise loop, which continues until a fixed classifier labels the predicted user response as satisfied or the revision budget is exhausted. Training combines GRPO, using a reward that balances task performance and user-visible interaction effort, with supervised prediction of user responses.

Critical strengths
  • The motivation is clear and compelling. The paper identifies meaningful distinction between correcting predictable problems internally and asking users for missing information. Anticipating avoidable corrections before showing a response is a well-motivated approach to reducing the burden of collaboration.
  • An interesting, concrete operationalization of user-response anticipation. Rather than relying only on a prompt to consider the user’s perspective, CToM makes predicting user feedback an explicit learning objective and connects those predictions to a revision mechanism.
Critical weaknesses
  • The experiments do not sufficiently isolate the contribution of prediction supervision. Task-Only GRPO uses the same prompt and revision loop but removes both prediction supervision and the effort penalty. Its comparison with CToM confounds these two components,
  • Comparisons with closely related methods such as CollabLLM [1] and DiscoverLLM [2] are missing and some baseline comparisons are unfair. A matched self-feedback baseline would also help establish whether anticipating the user’s perspective offers benefits beyond generic iterative critique. Also, the coding recovery comparison is difficult to interpret independently of the interaction protocol. The baseline coding prompt directs the assistant to produce complete code as a final submission, so Prompting Base and Naive GRPO only does single-turn. A stronger comparison would give all methods equivalent opportunities to show provisional drafts, receive feedback, and revise before committing.

[1] CollabLLM: From Passive Responders to Active Collaborators
[2] DiscoverLLM: From Executing Intents to Discovering Them

Questions for authors
  1. How does CToM perform when only prediction supervision is disabled, with the task-and-effort reward and all interaction machinery unchanged? Compare with a matched revision baseline methods that generates ordinary self-critique rather than predicted user feedback.
  2. What evidence indicates that the simulator’s feedback resembles feedback from real users? Provide cross-simulator results or a human evaluation measuring task success, satisfaction, perceived effort, and the number of substantive corrections.
  3. Can the authors report task performance against total generated tokens and latency, including hidden drafts and predictions, while separately reporting user-visible turns and tokens? How do the results change under matched total computation budgets and different revision caps?
  4. Why is response prediction restricted to DRAFT, after the action has already been selected? Could uncertainty-aware predictions of responses to candidate questions improve ASK selection, without treating predicted answers as confirmed information? A justification or targeted ablation would clarify whether the current separation is essential or simply a design choice.
5 comments
AuthorsOpenReview ↗

Response to Reviewer BZor (1/3)

We thank the reviewer and appreciate the noted strengths: a "clear and compelling" motivation, and an "interesting, concrete operationalization of user-response anticipation". We ran new experiments to address the concerns.

Weakness 1

"The experiments do not sufficiently isolate the contribution of prediction supervision. Task-Only GRPO uses the same prompt and revision loop but removes both prediction supervision and the effort penalty."

We trained the two missing ablations, which completes the 2×2 design (see General Response, Section 1; Table 1). CToM w/o prediction disables only the prediction loss.

  • Prediction supervision raises task score in all six comparisons: by 1.8 points on math, 4.1 on coding and 1.1 on writing with the effort penalty, and by 5.0, 6.3 and 1.1 points without it.
  • Under the same reward, it also reduces user replies by 8% on coding and 17% on writing.

These results show that the gains come from prediction supervision.

Weakness 2

"Comparisons with closely related methods such as CollabLLM [1] and DiscoverLLM [2] are missing and some baseline comparisons are unfair. A matched self-feedback baseline would also help establish whether anticipating the user's perspective offers benefits beyond generic iterative critique."

We did not run CollabLLM or DiscoverLLM: we do not have the training budget to train these models. CollabLLM requires forward sampling of multiple conversation paths during training.

We provide Self-Refine (Madaan et al., 2023) as the "matched self-feedback baseline" (see App. B.8; prompts in App. C.5). It is added to both Prompting Base and Naive GRPO. It runs at inference on every assistant turn: the model critiques its draft and rewrites it, and stops when its critique reports no issues. It allows up to three rewrites per turn, the same as CToM's revision cap, and each critique has the token cap of CToM's prediction.

Task score with the Qwen3-32B user (×100 ± SEM; see Table 10)

MethodMathCodingWriting
Prompting Base69.14 ± 1.9734.00 ± 3.1913.69 ± 0.36
+ Self-Refine69.62 ± 1.9534.74 ± 3.3113.81 ± 0.35
Naive GRPO74.08 ± 1.6748.91 ± 3.2417.92 ± 0.29
+ Self-Refine75.86 ± 1.6249.78 ± 3.3117.80 ± 0.30
CToM79.62 ± 1.6551.05 ± 3.0819.08 ± 0.51
  • Self-Refine improves the baselines by at most 1.8 points (Naive GRPO on math).
  • CToM is 3.8 points above the better Self-Refine variant on math, 1.3 on coding and 1.3 on writing.

This shows that training the model to predict the user's feedback improves task performance more than having the model critique itself.

AuthorsOpenReview ↗

Response to Reviewer BZor (2/3)

Weakness 3

"... the coding recovery comparison is difficult to interpret independently of the interaction protocol. ... A stronger comparison would give all methods equivalent opportunities to show provisional drafts, receive feedback, and revise before committing."

We ran this comparison (see Sec. 5.3; App. B.9, Fig. 6). On coding, Prompting Base and Naive GRPO submit their first program as the final answer, which ends the conversation. In the new evaluation (+ reply), the simulated user replies to this program as it does to a draft, and the assistant can then revise it.

Coding score (×100) after a failing first program, at each turn after it

MethodFirst program+1+2+3+4Gain
Prompting Base + reply20.936.641.542.342.5+21.6
Naive GRPO + reply30.040.743.243.243.3+13.3
CToM18.838.444.645.846.2+27.4
CToM w/o effort23.641.645.445.745.8+22.2
CToM w/o prediction18.133.839.040.240.5+22.4
Task-Only GRPO19.134.239.440.040.1+21.0
  • With the user's replies, both baselines recover: Prompting Base from 20.9 to 42.5, and Naive GRPO from 30.0 to 43.3.
  • CToM ends highest (46.2) and gains the most (+27.4).
  • The two variants trained with prediction supervision end at 46.2 and 45.8, above both baselines. The two without it end at 40.5 and 40.1, below both.

This shows that the larger recovery of CToM comes from prediction supervision, and not from the interaction protocol.

Question 1

"How does CToM perform when only prediction supervision is disabled, with the task-and-effort reward and all interaction machinery unchanged? Compare with a matched revision baseline methods that generates ordinary self-critique rather than predicted user feedback."

This is addressed in Weakness 1 and Weakness 2. CToM w/o prediction disables only prediction supervision, with the task-and-effort reward, prompt and revision loop unchanged (see Weakness 1). Self-Refine is the self-critique baseline (see Weakness 2).

Question 2

"What evidence indicates that the simulator's feedback resembles feedback from real users? Provide cross-simulator results or a human evaluation measuring task success, satisfaction, perceived effort, and the number of substantive corrections."

We evaluated all models with two held-out user simulators from other model families, GPT-4o-mini and Claude Haiku 5.5 (see General Response, Section 2; App. B.7, Table 8).

  • Prediction supervision raises task score for held-out users.
  • CToM w/o effort has the highest task score in 5 of the 6 held-out user experiments.

CToM, although trained only with the Qwen3-32B user, generalizes its user prediction to held-out users it has not seen.

AuthorsOpenReview ↗

Response to Reviewer BZor (3/3)

Question 3

"Can the authors report task performance against total generated tokens and latency, including hidden drafts and predictions, while separately reporting user-visible turns and tokens? How do the results change under matched total computation budgets and different revision caps?"

Every method has the same generation budget. In each round, CToM may write up to four drafts and four predictions: 9,728 tokens in total on math, 5,632 on coding and 7,168 on writing. Prompting Base and Naive GRPO have exactly this total as their per-response cap (see Table 3), so every model can generate a response of the same length. CToM is trained to use this budget actively: it predicts the user's reply and revises its draft before showing it. Self-Refine has the same cap for its reply, plus separate caps for its critiques and rewrites.

Task score (×100) and total generated tokens per conversation, visible plus hidden, with the Qwen3-32B user (see App. B.10, Table 11). For visible tokens alone, see Table 1.

MethodMath scoreMath tokensCoding scoreCoding tokensWriting scoreWriting tokens
Prompting Base69.141,10434.0023513.69259
+ Self-Refine69.621,15834.7424813.81269
Naive GRPO74.082,25548.9125317.92353
+ Self-Refine75.862,19249.7826917.80831
CToM79.622,15851.0597419.083,676
CToM w/o effort82.162,88554.4279019.185,287
CToM w/o prediction77.812,26846.982,23518.035,404
Task-Only GRPO77.192,96248.112,11918.072,476

Hidden generation and latency

QuantityMathCodingWriting
CToM: hidden tokens per conversation1,6336482,908
CToM: hidden share of all generated tokens76%66%79%
CToM: internal revisions per visible round (cap 3)1.740.692.40
Naive GRPO + Self-Refine: hidden tokens per conversation27116482
Naive GRPO + Self-Refine: rewrites per turn0.200.000.79
Prompting Base + Self-Refine: hidden tokens per conversation65139
CToM: generation time per conversation (s)18.911.238.6
Naive GRPO: generation time per conversation (s)14.22.67.8
CToM: generation time per visible turn (s)12.13.48.8
Naive GRPO: generation time per visible turn (s)7.31.37.8
  • CToM uses the budget to predict and revise. Its hidden tokens are plans, predictions and discarded drafts.
  • Self-Refine uses little of the budget and has little effect. Its hidden tokens are critiques and discarded drafts. Prompting Base + Self-Refine makes no rewrites, and Self-Refine improves the baselines by at most 1.8 points.
  • On math, CToM generates about as much as Naive GRPO and scores 79.6 vs. 74.1. On coding and writing it generates more.
  • CToM w/o prediction generates at least as much as CToM in every domain and scores lower.
  • Latency. The times above are the policy's generation time in the training rollouts, with simulator time excluded.
  • Revision caps. For the revision-cap study, see Fig. 3b and App. B.4. Math runs trained with revision caps 1 and 3 both show a larger self-revision gain with the prediction loss: +3.40 for CToM vs. +0.93 for CToM w/o prediction at cap 1, and +5.48 vs. +3.72 at cap 3.

Overall, CToM improves task performance not simply by generating more, but through its user-prediction and revision pipeline, which lets it use the same generation budget more effectively to improve collaborative task performance and reduce user input effort (see General Response, Section 1).

Question 4

"Why is response prediction restricted to DRAFT, after the action has already been selected? Could uncertainty-aware predictions of responses to candidate questions improve ASK selection, without treating predicted answers as confirmed information? A justification or targeted ablation would clarify whether the current separation is essential or simply a design choice."

The assistant asks when it lacks information that only the user has, so predicting the reply would mean guessing what it does not know (see Sec. 4). Uncertainty-aware predictions would raise significant issues of their own, such as a confidence gate with a threshold to tune, and the reliability of the model's self-reported confidence.

The separation is not essential to the framework: CToM does not prevent applying predictions to the ASK action as well. Future work can investigate how to incorporate predictions into ASK to further improve the pipeline. We believe the current findings are solid enough to share with the community.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response to All Reviewers (2/2)

2. Held-out user simulators

All models are trained with Qwen3-32B-FP8 as the user. We evaluate the same models with two held-out user simulators from other model families: GPT-4o-mini and Claude Haiku 5.5 (see App. B.7, Table 8; user effort and prediction F1 are in Table 9). The Qwen3-32B columns below are the results of Table 1.

Task score of each method (×100)

MethodQwen3-32B: MathQwen3-32B: CodingQwen3-32B: WritingGPT-4o-mini: MathGPT-4o-mini: CodingGPT-4o-mini: WritingHaiku 5.5: MathHaiku 5.5: CodingHaiku 5.5: Writing
Prompting Base69.1434.0013.6967.2330.0813.6271.6846.1413.85
Naive GRPO74.0848.9117.9260.3836.7117.7281.8263.1517.85
CToM79.6251.0519.0872.0937.1617.0882.3662.7420.38
CToM w/o effort82.1654.4219.1877.8442.8117.1185.1465.5020.80
CToM w/o prediction77.8146.9818.0375.4533.2316.2278.6360.3719.80
Task-Only GRPO77.1948.1118.0772.2633.9214.7080.5163.6119.93

Prediction F1 with the held-out users (×100): lexical F1 between the predicted and the actual user reply

MethodGPT-4o-mini: MathGPT-4o-mini: CodingGPT-4o-mini: WritingHaiku 5.5: MathHaiku 5.5: CodingHaiku 5.5: Writing
CToM11.454.222.012.853.116.8
CToM w/o effort12.362.224.113.169.114.0
CToM w/o prediction7.28.413.513.59.417.6
Task-Only GRPO7.57.49.415.16.611.4

With the two held-out users:

  • Prediction supervision raises task score for held-out users.
  • The models trained with the prediction loss have a higher prediction F1 than those without it in every domain with GPT-4o-mini, and on coding with Haiku 5.5.
  • Under the same reward, CToM needs fewer user replies than CToM w/o prediction (6–23% fewer; see Table 9).
  • CToM w/o effort has the highest task score in 5 of the 6 held-out user experiments. This shows that CToM, although trained only with the Qwen3-32B user, generalizes its user prediction to held-out users it has not seen, which yields better task performance.

3. Changes in the revised PDF

The main text has few changes. The new results are in the appendix. Added and changed text is in blue in the revised PDF.

  • Table 1: adds the two ablations and drops the combined score. Sec. 5.1 and 5.2 describe the ablations and their result.
  • Fig. 3 and Sec. 5.3: prediction accuracy, revision gain and coding recovery now compare CToM with CToM w/o prediction.
  • Combined score: removed from the evaluation. Fig. 4(b) and the appendix entropy tables (Tables 4–5) now use task score, which is lower in all six comparisons. It is still the training reward (see Sec. 4.4).
  • New appendix sections: user effort (see App. B.6, Table 7); held-out user simulators (see App. B.7, Tables 8–9); Self-Refine baseline (see App. B.8, Table 10; prompts in App. C.5); coding recovery when the user also replies to the program submitted by the two baselines (see App. B.9, Fig. 6); total generated tokens and generation time (see App. B.10, Table 11).
  • Corrected: the writing score of Task-Only GRPO in Table 1 is 18.07 ± 0.35 (5.79 turns, 1168 tokens). The submitted version showed 17.74 (5.92 turns, 1072 tokens), which came from the run with the action-entropy bonus. This recording issue does not affect the conclusions of the paper. The gain over the strongest baseline on writing is now 5.6% (was 6.5%), and the appendix entropy tables (Tables 4–5) are corrected to match.
  • Abstract and introduction: the claims based on the combined score are removed, and CToM is stated to outperform all baselines.
  • Related work: added a paragraph on self-correction (Self-Refine, SCoRe).
  • Fixed: App. C.3 now describes how the simulator's chat history is built for counterfactual queries.
AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We thank the reviewers for their detailed reviews. In particular, we were glad to see that they appreciated the clear motivation (BZor, 8Vxe), the concrete formulation of user-response anticipation as an explicit learning objective (BZor, 8Vxe, 8giV), the distinction between asking for missing information and internally repairing predictable mistakes (BZor, 8giV), the counterfactual replies as a reusable source of supervision (8Vxe), and the clarity of the presentation (BZor, 8Vxe). We also appreciated the clarifying questions and constructive suggestions, and have responded to each reviewer's comments in the respective threads.

The reviews share two main concerns: (1) the lack of a full ablation of prediction supervision and the effort penalty; and (2) whether the results hold beyond the user simulator used in training. We ran new experiments for both, now in the revised PDF (see Table 1, Sec. 5.2; Tables 7–9, App. B.6–B.7).

1. Full 2×2 ablation of the two training signals

We trained the two missing cells. CToM w/o prediction sets λpred=0\lambda_{\text{pred}}=0 and keeps the effort penalty. CToM w/o effort sets weffort=0w_{\text{effort}}=0 and keeps the prediction loss. Task-Only GRPO removes both.

Task score (×100, ±1 SEM clustered by task)

MethodLpred\mathcal{L}_{\text{pred}}CeffortC_{\text{effort}}MathCodingWriting
CToM✓✓79.62 ± 1.6551.05 ± 3.0819.08 ± 0.51
CToM w/o effort✓82.16 ± 1.6254.42 ± 3.1919.18 ± 0.40
CToM w/o prediction✓77.81 ± 1.7746.98 ± 3.1318.03 ± 0.44
Task-Only GRPO77.19 ± 1.7548.11 ± 3.1518.07 ± 0.35

Gain in task score from prediction supervision (points)

ComparisonMathCodingWriting
With the effort penalty: CToM − CToM w/o prediction+1.8+4.1+1.1
Without it: CToM w/o effort − Task-Only GRPO+5.0+6.3+1.1

User effort (see App. B.6, Table 7): number of user replies and user input tokens per conversation, over conversations with at least one user reply. Bold marks the lowest value.

MethodMath repliesMath input tokensCoding repliesCoding input tokensWriting repliesWriting input tokens
CToM2.2489.92.5768.63.8986.9
CToM w/o effort2.2392.82.9877.55.00113.3
CToM w/o prediction2.2885.72.7873.44.69105.8
Task-Only GRPO2.42105.83.2982.44.79104.5
  • Prediction supervision raises task score in all six comparisons.
  • Under the same reward, prediction supervision reduces user replies by 8% on coding and 17% on writing, and user input tokens by 7% and 18%. CToM needs the fewest replies and input tokens of the four variants on coding and writing.
  • Removing the effort penalty from CToM raises task score (+2.5 on math, +3.4 on coding) but lengthens the interaction: the assistant uses 15–61% more visible tokens.

Combined score. The revision no longer reports the combined score (task score − 0.01 × effort cost). We found that the effort weight of 0.01, although it has an effect in training, is too small to be used in evaluation. Instead, we evaluate user effort directly, which measures the input required from the user, and show that CToM requires less user input than its ablations on coding and writing.

For other behavioral analyses of the new ablations, see Sec. 5.3 (Fig. 3) and App. B.4–B.5 of the revised paper.

Dear Reviewers,

Thank you for your reviews and contributions. As we are now halfway through the discussion period, I would like to encourage you to carefully read the authors' responses and engage in further discussion where appropriate. Please also indicate whether the rebuttal has addressed your concerns or changed your assessment.

Thank you for your time and effort!

Thank you for encouraging the reviewers to read our responses and engage in the discussion. We appreciate your time and effort in handling our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.