- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ThoughtTrace, a dataset pairing multi-turn human–AI conversations with users' self-reported thoughts (reasons for messages and reactions to responses), and shows their use for next-message prediction and alignment.
Critical strengths
- User thoughts collected during conversations are a new and useful type of data, especially for user modeling and simulation.
- The collection protocol is carefully designed, and the paper is clearly written.
Critical weaknesses
- The prediction experiment provides limited evidence for the utility of thoughts. The input includes the user's reaction to the previous response, which often already states the intent of the next message, so the gain is largely expected and mainly reflects correlation; such thoughts are also unavailable at deployment.
- The alignment experiment does not clearly isolate the effect of thoughts. All chosen responses are GPT-5.4 rewrites, so much of the gain may come from distillation, and the thought- vs. message-guided comparison also changes how dissatisfied responses are selected.
- Arena-Hard measures general response quality without user context, so it does not directly support the claim about personalized assistants.
- With the help of an LLM, I scanned the released dataset: about one third of the thoughts were written after the conversation ended, most reactions are generic (e.g., "satisfied", "good"), and many reasons are templated (e.g., "conversation starter"). Without a human baseline, the claim that thoughts are hard for LLMs to infer may reflect this annotation noise.
Questions for authors
see above
7 comments
Rebuttal by Authors
W3. Personalized assistance can be evaluated more directly than with Arena-Hard.
Arena-Hard tests whether thought supervision improves general response quality, i.e., whether the gains hold beyond ThoughtTrace users. For user-specific evaluation, App. B.4 (see lines 1147–1163) uses 20 held-out ThoughtTrace dissatisfaction cases with five annotators each, who judge rewrites against the original user's own thought. Thought-guided rewrites better address that user's dissatisfaction in 70.0% of the 100 pairwise comparisons (20 cases × 5 annotators) and win the majority vote in 16 of 20 cases. We have included this clarification in the updated Sec. 5.2 (lines 468–469).
W4. Annotation timing and brief or templated thoughts may affect the inference-difficulty result.
Thank you for inspecting the released data closely. This level of noise is common in real-world data, most thoughts are specific to their turn, and our results do not rely on late or low-information thoughts.
(1) This level of noise is common in real-world data collection. LMSYS-Chat-1M reports "a significant amount of low-quality and duplicate data" and applies no filtering on purpose "to reflect the real-world distribution" (Zheng et al., 2023). In WildChat, many jailbreaking prompts are exact copies circulating online; the most common of them appears 3,903 times, from 211 users (Zhao et al., 2024b). Similar noise does not prevent ThoughtTrace from being informative, as our experiments and the analyses below show.
(2) Late thoughts are by design and do not drive the gain.
-
Late annotation is by design and mostly covers the final turn. The reviewer's count matches ours: 34.8% of thoughts were written after the conversation's last message. Post-conversation annotation was allowed by design (App. C.3–C.4, see lines 1783–1784, 1821), so that participants could finish recording thoughts after the 10-minute window. We run an additional analysis of the timestamps: most of these thoughts annotate the final turn, which no prediction target follows, the median delay between a message and its thought is about one minute, and 90% are written within eight minutes.
-
Late thoughts do not drive the prediction gain. We also run an additional experiment that repeats next-message prediction while controlling for when thoughts were written: thoughts improve prediction over history-only by +47% on instances where every thought was written before the target message, matching the gain on all instances (+46%). We have included these analyses and results in the updated App. B.12 and B.14.
(3) Very short and templated thoughts are uncommon. We run an additional analysis of thought content. Only 22.6% of reactions have three words or fewer, mostly explicit approvals such as "good". These are genuine satisfaction signals, which we keep rather than distort the natural distribution. Templated reasons are rare: only 4.3% of reasons repeat another reason's exact text, and "conversation starter" appears in 5 of 4,498. We have included these statistics in the updated App. B.12.
(4) Most thoughts are specific to their turn, not generic or templated. We run an additional experiment that randomly samples 100 reasons and 100 reactions and rates each on a 1–5 informativeness scale (1 = empty, 2 = generic, 3 = specific to the turn, 4 = adds information beyond the message, 5 = rich personal context). 74% of reasons and 56% of reactions are specific to the turn (≥3), and 27% of each add information beyond the message (≥4). We have included these ratings in the updated App. B.12. Neither utility experiment in Sec. 5 relies on low-information thoughts: Sec. 5.1 keeps examples whose thoughts are rated ≥4/5 (see line 2522), and Sec. 5.2 uses only dissatisfaction reactions of at least six words (see line 2625).
(5) Low-information or retrospective thoughts do not drive the inference result. We run an additional experiment that repeats Thought Property 2 by thought informativeness and timing. Informative thoughts (rated ≥4/5) are harder to infer than the average sampled thought (reasons 2.85 vs. 3.17; reactions 2.70 vs. 2.96), and thoughts written before the next user message are about as hard (3.22 and 2.85). We have included these results in the updated Sec. 4.2 (lines 316–319) and App. B.13.
(6) Human validation confirms that the thoughts are informative. Because a message admits many plausible thoughts, low inference scores reflect the gap between an observer's guess and the user's ground truth rather than noise in the reports. As a human check, 30 independent raters find that the thoughts add information beyond the messages (4.0/5) and explain user behavior (4.2/5, App. B.1, see lines 306–308, 763–770). The thoughts also improve prediction of the actual next message (+8.4 points across six models; see line 437), and humans prefer the rewrites they guide (70.0%; see lines 1162–1163).
Official Comment by Authors
We hope that our responses and additional experiments conducted during the rebuttal period have adequately addressed your questions! We would be happy to address any further questions or suggestions you may have, and we would greatly appreciate it if you could consider increasing your score if you find our responses helpful!
Rebuttal by Authors
We thank Reviewer FVXE for recognizing user thoughts as a new and useful type of data for user modeling and simulation, the carefully designed collection protocol, and the clear writing, and for inspecting the released dataset. Below, we add controls and ablations to the prediction experiment and analyze annotation timing and quality.
Our analysis in Sec. 4.2 shows that reactions typically do not state the intent of the next message directly, and our additional experiments show that most of the prediction gain remains without the immediate reaction. Thoughts also need not be available at deployment, since they can serve as training data.
(1) Reactions typically do not state the intent of the next message directly. In a shared UMAP projection of text embeddings, reaction–next-message pairs are much farther apart than consecutive user messages (median distance 6.62 vs. 1.96; centroid distance 0.320 vs. 0.120; MMD 0.257 vs. 0.096; App. B.9, see lines 1368–1400). An LLM judge rates the next message's coverage of the reaction as minimal (2.00/5; see line 305), and frontier LLMs that are given the next message still recover the reaction poorly (2.54/5; see line 315). App. B.3 (see line 1140) also shows a case where an accurate reaction does not determine what the user says next.
(2) The gain does not rely on the immediate reaction. We run an additional experiment with three ablations of the thought-augmented context:
Table. Ablations of the thought-augmented context in next-message prediction: semantic similarity to the user's actual next message (0–100) and relative gain over history-only.
Most of the gain therefore remains without the reaction that could preview the next message, including from reasons and earlier-turn thoughts. We have included these ablations in the updated Sec. 5.1 (lines 442–447) and App. B.14 (Table 9).
(3) Uses of collected thoughts.
Uses that do not need thoughts at deployment. As an extension of Sec. 5.1, where thoughts improve prediction of the user's next message (+8.4 points across six models, Table 1, see line 437), the collected thoughts can be summarized into user personas to build better user simulators. Thoughts also serve as additional training data and are not needed at inference: the model trained with thought-guided DPO runs without thoughts and outperforms message-guided DPO on Arena-Hard (+4.5 style-controlled win, Table 2, see lines 478–481).
Future direction. How to collect a minimal set of critical thoughts from users at deployment, to complement other signals for continual model improvement, is a promising direction beyond our current scope. Our contribution is to show that thoughts improve user behavior simulation and model alignment, and to release ThoughtTrace as a data source for future research.
All trained conditions share the same distillation, and the advantage of thoughts holds when selection is controlled.
(1) Distillation is shared by all trained conditions. The WildChat, ThoughtTrace (messages), and ThoughtTrace (thoughts) conditions all use GPT-5.4 rewrites as chosen responses, with the same rewriter, base model, and DPO hyperparameters (App. D.9, see lines 2630–2646 and lines 2710–2713). The gain over the base model (22.5 → 41.5–48.1 style-controlled; see lines 478–481) therefore includes learning from GPT-5.4 rewrites, and the thought-specific effect is the gap between rewrite conditions: +3.9 win and +4.5 style-controlled win over message-guided rewrites on the same conversations. We have included this distinction in the updated Sec. 5.2 (lines 469–471).
(2) Selection vs. guidance. The main comparison varies both which responses are selected and how they are rewritten, and selection is part of the value of thoughts: on the same conversations, thoughts surface 1,000 dissatisfaction instances while messages surface 450 (see line 483). Two existing results address this. At matched size (450 instances), thought guidance still leads by +3.1 win / +3.7 style-controlled (App. B.5, see lines 1184–1186). In the human evaluation (App. B.4, see lines 1147–1163), both rewrites are generated for the same dissatisfaction instances, so selection is held fixed, and the thought-guided rewrite is preferred in 70.0% of the 100 pairwise comparisons (20 cases × 5 annotators) and wins the majority vote in 16 of 20 cases.